2024-09-16T03:25:02.216 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T03:25:02.239 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T03:25:02.329 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882 branch: main description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/careful_host thrashosds-health workloads/dedup-io-mixed} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905882' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 ms bind msgr2: false ms type: async osd debug inject dispatch delay duration: 0.1 osd debug inject dispatch delay probability: 0.1 osd_async_recovery_min_cost: 1 osd_max_pg_log_entries: 2 osd_min_pg_log_entries: 1 osd_object_clean_region_max_num_intervals: 1000 osd_pg_log_trim_min: 0 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd inject bad map crc probability: 0.1 osd max backfills: 3 osd max markdown count: 1000 osd max scrubs: 5 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - failed to encode map - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - \(POOL_APP_NOT_ENABLED\) mon_bind_msgr2: false sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 min_in: 4 workunit: branch: main sha1: 7498493279982dfe87f00616198a5967475f6169 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 2855 sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 sleep_before_teardown: 0 subset: 17936/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7498493279982dfe87f00616198a5967475f6169 targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMrSlIFHn+oeg94Tleq8n2nqfaZxdYUJqDxf7yWhFl4am6RiJGZL1xZWBI2ejqZK0fe1g3mSHS01BXKxZXvHS+0= smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDz/uE2dfF6JFvVUxekLqome09IzbXavXJpbC7zieyyUUHoBpf993ZzznOmY29k6eEIyQxMiVbaLovyhjxO6fGQ= smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGD7G0C86AQxUvTZYhli9+OdNHhsdJ62jAqoKIzK7C4FaOoLqE32h8O0SwsnglWigoribcKVXN+Krv1AXV/6P2A= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHyuxrOLtYmKFpCTrCN/85jDl//yoybKDtHqE2BBtC7MGSOqRUudo0YCNt4S3ixAmfqxPmfKwUneN/hdwXWgI5k= tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 2 thrash_hosts: true timeout: 1200 - exec: client.0: - sudo ceph osd pool create low_tier 4 - rados: clients: - client.0 dedup_chunk_algo: fastcdc dedup_chunk_size: '131072' enable_dedup: true low_tier_pool: low_tier objects: 50 op_weights: read: 100 set_chunk: 30 tier_evict: 10 tier_flush: 5 tier_promote: 10 write: 50 ops: 1500 set_chunk: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_20:00:16 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T03:25:02.329 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa; will attempt to use it 2024-09-16T03:25:02.330 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks 2024-09-16T03:25:02.330 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T03:25:02.331 INFO:teuthology.task.internal:Checking packages... 2024-09-16T03:25:02.352 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'fc3e0bff7468fd55478adbae7f75508b749124b6' 2024-09-16T03:25:02.352 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T03:25:02.353 INFO:teuthology.packaging:ref: None 2024-09-16T03:25:02.353 INFO:teuthology.packaging:tag: None 2024-09-16T03:25:02.353 INFO:teuthology.packaging:branch: main 2024-09-16T03:25:02.353 INFO:teuthology.packaging:sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T03:25:02.353 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-09-16T03:25:02.568 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4937.g74984932 2024-09-16T03:25:02.569 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T03:25:02.576 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T03:25:02.576 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T03:25:02.586 INFO:teuthology.task.internal:Saving configuration 2024-09-16T03:25:02.605 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T03:25:02.611 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T03:25:02.637 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:19:32.175336', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMrSlIFHn+oeg94Tleq8n2nqfaZxdYUJqDxf7yWhFl4am6RiJGZL1xZWBI2ejqZK0fe1g3mSHS01BXKxZXvHS+0='} 2024-09-16T03:25:02.660 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:19:32.174061', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDz/uE2dfF6JFvVUxekLqome09IzbXavXJpbC7zieyyUUHoBpf993ZzznOmY29k6eEIyQxMiVbaLovyhjxO6fGQ='} 2024-09-16T03:25:02.682 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:19:32.172631', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGD7G0C86AQxUvTZYhli9+OdNHhsdJ62jAqoKIzK7C4FaOoLqE32h8O0SwsnglWigoribcKVXN+Krv1AXV/6P2A='} 2024-09-16T03:25:02.703 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:19:32.176482', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHyuxrOLtYmKFpCTrCN/85jDl//yoybKDtHqE2BBtC7MGSOqRUudo0YCNt4S3ixAmfqxPmfKwUneN/hdwXWgI5k='} 2024-09-16T03:25:02.703 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T03:25:02.708 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-09-16T03:25:02.709 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-09-16T03:25:02.709 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-09-16T03:25:02.709 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-09-16T03:25:02.709 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T03:25:02.845 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8d6d2bbac0>, signals=[15]) 2024-09-16T03:25:02.845 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T03:25:02.854 INFO:teuthology.task.internal:Opening connections... 2024-09-16T03:25:02.854 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-09-16T03:25:02.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:25:02.932 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2024-09-16T03:25:02.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:25:03.010 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-09-16T03:25:03.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:25:03.088 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2024-09-16T03:25:03.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:25:03.166 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T03:25:03.173 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-09-16T03:25:03.191 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-09-16T03:25:03.192 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-09-16T03:25:03.250 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-09-16T03:25:03.250 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-09-16T03:25:03.250 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-09-16T03:25:03.250 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:25:03.250 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-09-16T03:25:03.250 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:25:03.250 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:25:03.250 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-09-16T03:25:03.250 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:25:03.250 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:25:03.250 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:25:03.250 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:25:03.251 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:25:03.251 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:25:03.251 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-09-16T03:25:03.276 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2024-09-16T03:25:03.295 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2024-09-16T03:25:03.296 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2024-09-16T03:25:03.354 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="CentOS Stream" 2024-09-16T03:25:03.354 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="9" 2024-09-16T03:25:03.354 INFO:teuthology.orchestra.run.smithi112.stdout:ID="centos" 2024-09-16T03:25:03.354 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:25:03.354 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="9" 2024-09-16T03:25:03.354 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:25:03.355 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:25:03.355 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2024-09-16T03:25:03.355 INFO:teuthology.orchestra.run.smithi112.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:25:03.355 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:25:03.355 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:25:03.355 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:25:03.355 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:25:03.355 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:25:03.355 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2024-09-16T03:25:03.376 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-09-16T03:25:03.395 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-09-16T03:25:03.395 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-09-16T03:25:03.453 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-09-16T03:25:03.453 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="9" 2024-09-16T03:25:03.453 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-09-16T03:25:03.453 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:25:03.454 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="9" 2024-09-16T03:25:03.454 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:25:03.454 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:25:03.454 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-09-16T03:25:03.454 INFO:teuthology.orchestra.run.smithi155.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:25:03.454 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:25:03.454 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:25:03.454 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:25:03.454 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:25:03.454 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:25:03.455 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-09-16T03:25:03.476 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2024-09-16T03:25:03.494 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2024-09-16T03:25:03.495 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2024-09-16T03:25:03.553 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Stream" 2024-09-16T03:25:03.553 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="9" 2024-09-16T03:25:03.553 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos" 2024-09-16T03:25:03.554 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora" 2024-09-16T03:25:03.554 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="9" 2024-09-16T03:25:03.554 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el9" 2024-09-16T03:25:03.554 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T03:25:03.554 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31" 2024-09-16T03:25:03.554 INFO:teuthology.orchestra.run.smithi157.stdout:LOGO="fedora-logo-icon" 2024-09-16T03:25:03.554 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T03:25:03.554 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/" 2024-09-16T03:25:03.554 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T03:25:03.554 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T03:25:03.554 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T03:25:03.555 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2024-09-16T03:25:03.576 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T03:25:03.586 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T03:25:03.591 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T03:25:03.591 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:25:03.593 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:25:03.596 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:25:03.598 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:25:03.614 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T03:25:03.620 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T03:25:03.621 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:25:03.651 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:25:03.654 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:25:03.656 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:25:03.741 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T03:25:03.841 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T03:25:03.861 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T03:25:03.861 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T03:25:03.862 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:25:03.862 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:25:03.862 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:25:03.863 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:25:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:25:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:25:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:25:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:25:03.883 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-09-16T03:25:03.883 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-09-16T03:25:03.884 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-09-16T03:25:03.884 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-09-16T03:25:03.940 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:25:03.941 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-503.el9.x86_64 2024-09-16T03:25:03.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-09-16T03:25:03.944 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:25:03.944 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:25:03.944 INFO:teuthology.orchestra.run.smithi112.stdout:5.14.0-503.el9.x86_64 2024-09-16T03:25:03.945 INFO:teuthology.task.kernel:Running kernel on smithi155: 5.14.0-503.el9.x86_64 2024-09-16T03:25:03.945 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2024-09-16T03:25:03.945 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.14.0-503.el9.x86_64 2024-09-16T03:25:03.946 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum install -y kernel 2024-09-16T03:25:03.946 INFO:teuthology.task.kernel:Running kernel on smithi112: 5.14.0-503.el9.x86_64 2024-09-16T03:25:03.946 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum install -y kernel 2024-09-16T03:25:06.498 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-16T03:25:06.588 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.2 MB 00:00 2024-09-16T03:25:06.615 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 8.2 MB 00:00 2024-09-16T03:25:07.421 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.2 MB 00:01 2024-09-16T03:25:09.040 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - AppStream 15 MB/s | 20 MB 00:01 2024-09-16T03:25:09.765 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:02 2024-09-16T03:25:09.864 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 20 MB 00:02 2024-09-16T03:25:12.221 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 20 MB 00:03 2024-09-16T03:25:14.315 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.5 MB 00:00 2024-09-16T03:25:14.674 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.5 MB 00:01 2024-09-16T03:25:15.474 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-16T03:25:15.840 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 102 kB/s | 19 kB 00:00 2024-09-16T03:25:16.166 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 19 kB 00:00 2024-09-16T03:25:17.025 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 19 kB 00:00 2024-09-16T03:25:17.194 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-16T03:25:17.404 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-16T03:25:17.672 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-16T03:25:17.879 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-16T03:25:19.147 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 19 kB 00:00 2024-09-16T03:25:20.185 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-16T03:25:22.650 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-09-16T03:25:22.935 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T03:25:23.518 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-16T03:25:24.657 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:25:24.657 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:25:24.740 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:25:24.752 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:25:24.752 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-09-16T03:25:24.752 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:25:24.752 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:25:24.752 INFO:teuthology.orchestra.run.smithi157.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:25:24.753 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-16T03:25:24.753 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:25:24.753 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:25:24.753 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:25:24.753 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:25:24.753 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:25:24.753 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:25:24.753 INFO:teuthology.orchestra.run.smithi157.stdout:Install 4 Packages 2024-09-16T03:25:24.753 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:25:24.754 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 79 M 2024-09-16T03:25:24.754 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 126 M 2024-09-16T03:25:24.754 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:25:24.966 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:25:24.966 INFO:teuthology.orchestra.run.smithi112.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:25:25.071 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:25:25.080 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:25:25.080 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-09-16T03:25:25.080 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:25:25.080 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:25:25.080 INFO:teuthology.orchestra.run.smithi112.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:25:25.080 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-16T03:25:25.080 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:25:25.080 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:25:25.080 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:25:25.081 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:25:25.081 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:25:25.081 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:25:25.081 INFO:teuthology.orchestra.run.smithi112.stdout:Install 4 Packages 2024-09-16T03:25:25.081 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:25:25.081 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 79 M 2024-09-16T03:25:25.082 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 126 M 2024-09-16T03:25:25.082 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:25:25.118 INFO:teuthology.orchestra.run.smithi157.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 266 kB/s | 23 kB 00:00 2024-09-16T03:25:25.294 INFO:teuthology.orchestra.run.smithi112.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 252 kB/s | 23 kB 00:00 2024-09-16T03:25:25.452 INFO:teuthology.orchestra.run.smithi157.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-09-16T03:25:25.600 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:25:25.600 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:25:25.607 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-09-16T03:25:25.686 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:25:25.694 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:25:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-09-16T03:25:25.694 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:25:25.695 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:25:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:25:25.695 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-16T03:25:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:25:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:25:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:25:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:25:25.695 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:25:25.695 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:25:25.695 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-09-16T03:25:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:25:25.696 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 79 M 2024-09-16T03:25:25.696 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 126 M 2024-09-16T03:25:25.696 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:25:25.852 INFO:teuthology.orchestra.run.smithi157.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 42 MB/s | 35 MB 00:00 2024-09-16T03:25:25.886 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 248 kB/s | 23 kB 00:00 2024-09-16T03:25:26.294 INFO:teuthology.orchestra.run.smithi157.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 24 MB/s | 29 MB 00:01 2024-09-16T03:25:26.294 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:25:26.294 INFO:teuthology.orchestra.run.smithi157.stdout:Total 51 MB/s | 79 MB 00:01 2024-09-16T03:25:26.370 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 60 MB/s | 35 MB 00:00 2024-09-16T03:25:26.520 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-09-16T03:25:26.854 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 30 MB/s | 29 MB 00:00 2024-09-16T03:25:26.854 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:25:26.854 INFO:teuthology.orchestra.run.smithi064.stdout:Total 68 MB/s | 79 MB 00:01 2024-09-16T03:25:26.981 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:25:27.352 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:25:27.352 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:25:27.564 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:25:27.639 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T03:25:27.639 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T03:25:27.722 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:25:27.729 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:25:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-09-16T03:25:27.730 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:25:27.730 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:25:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T03:25:27.730 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-16T03:25:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T03:25:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T03:25:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T03:25:27.730 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:25:27.730 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:25:27.730 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:25:27.730 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2024-09-16T03:25:27.731 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:25:27.731 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 79 M 2024-09-16T03:25:27.731 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 126 M 2024-09-16T03:25:27.731 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:25:27.786 INFO:teuthology.orchestra.run.smithi112.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 6.1 MB/s | 16 MB 00:02 2024-09-16T03:25:27.842 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 253 kB/s | 23 kB 00:00 2024-09-16T03:25:27.937 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:25:27.937 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:25:28.200 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-09-16T03:25:28.308 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:25:28.308 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:25:28.467 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 46 MB/s | 29 MB 00:00 2024-09-16T03:25:28.785 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 33 MB/s | 35 MB 00:01 2024-09-16T03:25:28.785 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:25:28.785 INFO:teuthology.orchestra.run.smithi155.stdout:Total 75 MB/s | 79 MB 00:01 2024-09-16T03:25:28.956 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:25:28.956 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:25:29.295 INFO:teuthology.orchestra.run.smithi112.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 7.1 MB/s | 29 MB 00:03 2024-09-16T03:25:29.466 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:25:29.829 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:25:29.829 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:25:30.745 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:25:30.816 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:25:30.816 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:25:31.445 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:25:31.517 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:25:31.519 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:25:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:25:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:25:33.293 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:25:33.303 INFO:teuthology.orchestra.run.smithi112.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 4.3 MB/s | 35 MB 00:08 2024-09-16T03:25:33.303 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:25:33.303 INFO:teuthology.orchestra.run.smithi112.stdout:Total 9.6 MB/s | 79 MB 00:08 2024-09-16T03:25:33.991 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:25:34.001 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:25:34.092 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:25:34.333 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:25:34.358 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:25:34.358 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:25:34.474 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:25:35.050 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:25:35.171 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:25:35.314 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:25:35.314 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:25:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:25:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:25:37.768 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:25:38.462 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:25:38.537 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:25:39.347 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:25:39.382 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:25:40.301 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:25:40.347 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:25:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:25:41.491 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:25:41.888 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:25:42.036 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:25:44.243 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:25:45.493 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:25:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:25:46.475 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:25:46.920 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:25:51.344 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:22.653 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:22.696 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:23.024 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:23.024 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:26:23.024 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:26:23.024 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:26:23.913 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:23.914 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:26:23.914 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:26:23.914 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:26:23.914 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:26:23.914 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:26:23.914 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:26:23.914 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:26:23.914 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:26:24.234 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:26:24.236 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:24.327 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:26:24.327 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:26:24.327 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:26:24.327 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:26:24.327 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:26:24.328 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-16T03:26:24.328 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:26:24.328 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:26:24.386 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:26:24.387 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:26:24.388 DEBUG:teuthology.orchestra.run.smithi157:> sudo rpm -qi grub2-tools 2024-09-16T03:26:24.425 INFO:teuthology.orchestra.run.smithi157.stdout:Name : grub2-tools 2024-09-16T03:26:24.425 INFO:teuthology.orchestra.run.smithi157.stdout:Epoch : 1 2024-09-16T03:26:24.425 INFO:teuthology.orchestra.run.smithi157.stdout:Version : 2.06 2024-09-16T03:26:24.425 INFO:teuthology.orchestra.run.smithi157.stdout:Release : 82.el9 2024-09-16T03:26:24.425 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: x86_64 2024-09-16T03:26:24.426 INFO:teuthology.orchestra.run.smithi157.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:26:24.426 INFO:teuthology.orchestra.run.smithi157.stdout:Group : Unspecified 2024-09-16T03:26:24.426 INFO:teuthology.orchestra.run.smithi157.stdout:Size : 8274670 2024-09-16T03:26:24.426 INFO:teuthology.orchestra.run.smithi157.stdout:License : GPLv3+ 2024-09-16T03:26:24.426 INFO:teuthology.orchestra.run.smithi157.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:26:24.426 INFO:teuthology.orchestra.run.smithi157.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:26:24.426 INFO:teuthology.orchestra.run.smithi157.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:26:24.426 INFO:teuthology.orchestra.run.smithi157.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:26:24.426 INFO:teuthology.orchestra.run.smithi157.stdout:Packager : builder@centos.org 2024-09-16T03:26:24.426 INFO:teuthology.orchestra.run.smithi157.stdout:Vendor : CentOS 2024-09-16T03:26:24.426 INFO:teuthology.orchestra.run.smithi157.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:26:24.426 INFO:teuthology.orchestra.run.smithi157.stdout:Summary : Support tools for GRUB. 2024-09-16T03:26:24.426 INFO:teuthology.orchestra.run.smithi157.stdout:Description : 2024-09-16T03:26:24.427 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:26:24.427 INFO:teuthology.orchestra.run.smithi157.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:26:24.427 INFO:teuthology.orchestra.run.smithi157.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:26:24.427 INFO:teuthology.orchestra.run.smithi157.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:26:24.427 INFO:teuthology.orchestra.run.smithi157.stdout:hardware devices. 2024-09-16T03:26:24.427 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:26:24.427 INFO:teuthology.orchestra.run.smithi157.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:26:24.428 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:26:24.428 INFO:teuthology.task.kernel:Updating grub on smithi157 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:26:24.428 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:26:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:26:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:26:24.647 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:26:25.106 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2024-09-16T03:26:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:25.216 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:25.471 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:25.471 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:26:25.471 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:26:25.471 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:26:25.472 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:26:25.472 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:26:25.472 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:26:25.472 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:26:25.472 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:26:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:26:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:26:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:26:25.836 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:26:25.976 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:26:25.976 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:26:25.976 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:26:25.976 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:26:25.976 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:26:25.976 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T03:26:25.976 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:26:25.976 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:26:26.035 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:26:26.035 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:26:26.035 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2024-09-16T03:26:26.075 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2024-09-16T03:26:26.075 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2024-09-16T03:26:26.075 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.06 2024-09-16T03:26:26.075 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 82.el9 2024-09-16T03:26:26.075 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2024-09-16T03:26:26.075 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:26:26.076 INFO:teuthology.orchestra.run.smithi064.stdout:Group : Unspecified 2024-09-16T03:26:26.076 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 8274670 2024-09-16T03:26:26.076 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2024-09-16T03:26:26.076 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:26:26.076 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:26:26.076 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:26:26.076 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:26:26.076 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : builder@centos.org 2024-09-16T03:26:26.076 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : CentOS 2024-09-16T03:26:26.076 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:26:26.076 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2024-09-16T03:26:26.076 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2024-09-16T03:26:26.076 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:26:26.077 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:26:26.077 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:26:26.077 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:26:26.077 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2024-09-16T03:26:26.077 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:26:26.077 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:26:26.078 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:26:26.078 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:26:26.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:26:26.429 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:26.429 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:26:26.429 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:26:26.429 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:26:26.429 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:26:26.430 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:26:26.430 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:26:26.430 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:26:26.430 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:26:26.466 INFO:teuthology.orchestra.run.smithi157.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:26:26.505 INFO:teuthology.orchestra.run.smithi157.stderr:done 2024-09-16T03:26:26.507 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-09-16T03:26:26.523 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.vfdaYsvCOL 2024-09-16T03:26:26.524 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vfdaYsvCOL 2024-09-16T03:26:26.595 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.vfdaYsvCOL 2024-09-16T03:26:26.736 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.vfdaYsvCOL is 6KB 2024-09-16T03:26:26.742 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-09-16T03:26:26.785 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.vfdaYsvCOL 2024-09-16T03:26:26.799 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:26:26.800 DEBUG:teuthology.orchestra.run.smithi157:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:26:26.863 INFO:teuthology.orchestra.run.smithi157.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:26:26.863 INFO:teuthology.orchestra.run.smithi157.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:26:26.864 INFO:teuthology.orchestra.run.smithi157.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:26:26.864 INFO:teuthology.orchestra.run.smithi157.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:26:26.865 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:26:26.882 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:26:26.882 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:26:26.883 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:26:26.883 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:26:26.883 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:26:26.883 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-16T03:26:26.883 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:26:26.883 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:26:26.942 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:26:26.942 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:26:26.942 DEBUG:teuthology.orchestra.run.smithi155:> sudo rpm -qi grub2-tools 2024-09-16T03:26:26.983 INFO:teuthology.orchestra.run.smithi155.stdout:Name : grub2-tools 2024-09-16T03:26:26.983 INFO:teuthology.orchestra.run.smithi155.stdout:Epoch : 1 2024-09-16T03:26:26.983 INFO:teuthology.orchestra.run.smithi155.stdout:Version : 2.06 2024-09-16T03:26:26.983 INFO:teuthology.orchestra.run.smithi155.stdout:Release : 82.el9 2024-09-16T03:26:26.983 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: x86_64 2024-09-16T03:26:26.983 INFO:teuthology.orchestra.run.smithi155.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:26:26.983 INFO:teuthology.orchestra.run.smithi155.stdout:Group : Unspecified 2024-09-16T03:26:26.984 INFO:teuthology.orchestra.run.smithi155.stdout:Size : 8274670 2024-09-16T03:26:26.984 INFO:teuthology.orchestra.run.smithi155.stdout:License : GPLv3+ 2024-09-16T03:26:26.984 INFO:teuthology.orchestra.run.smithi155.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:26:26.984 INFO:teuthology.orchestra.run.smithi155.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:26:26.984 INFO:teuthology.orchestra.run.smithi155.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:26:26.985 INFO:teuthology.orchestra.run.smithi155.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:26:26.985 INFO:teuthology.orchestra.run.smithi155.stdout:Packager : builder@centos.org 2024-09-16T03:26:26.985 INFO:teuthology.orchestra.run.smithi155.stdout:Vendor : CentOS 2024-09-16T03:26:26.985 INFO:teuthology.orchestra.run.smithi155.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:26:26.985 INFO:teuthology.orchestra.run.smithi155.stdout:Summary : Support tools for GRUB. 2024-09-16T03:26:26.986 INFO:teuthology.orchestra.run.smithi155.stdout:Description : 2024-09-16T03:26:26.986 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:26:26.986 INFO:teuthology.orchestra.run.smithi155.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:26:26.986 INFO:teuthology.orchestra.run.smithi155.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:26:26.986 INFO:teuthology.orchestra.run.smithi155.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:26:26.987 INFO:teuthology.orchestra.run.smithi155.stdout:hardware devices. 2024-09-16T03:26:26.987 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:26:26.987 INFO:teuthology.orchestra.run.smithi155.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:26:26.988 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:26:26.989 INFO:teuthology.task.kernel:Updating grub on smithi155 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:26:26.989 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:26:26.999 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2024-09-16T03:26:27.683 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2024-09-16T03:26:28.085 INFO:teuthology.orchestra.run.smithi064.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:26:28.115 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-09-16T03:26:28.117 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-09-16T03:26:28.132 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.XAFGO0oGOy 2024-09-16T03:26:28.132 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XAFGO0oGOy 2024-09-16T03:26:28.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.XAFGO0oGOy 2024-09-16T03:26:28.331 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.XAFGO0oGOy is 6KB 2024-09-16T03:26:28.378 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.XAFGO0oGOy 2024-09-16T03:26:28.392 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:26:28.454 INFO:teuthology.orchestra.run.smithi064.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:26:28.455 INFO:teuthology.orchestra.run.smithi064.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:26:28.455 INFO:teuthology.orchestra.run.smithi064.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:26:28.455 INFO:teuthology.orchestra.run.smithi064.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:26:28.458 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:26:28.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-09-16T03:26:29.042 INFO:teuthology.orchestra.run.smithi155.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:26:29.073 INFO:teuthology.orchestra.run.smithi155.stderr:done 2024-09-16T03:26:29.075 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-09-16T03:26:29.091 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.2lyJMLTJUv 2024-09-16T03:26:29.092 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2lyJMLTJUv 2024-09-16T03:26:29.159 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.2lyJMLTJUv 2024-09-16T03:26:29.301 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.2lyJMLTJUv is 6KB 2024-09-16T03:26:29.351 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.2lyJMLTJUv 2024-09-16T03:26:29.366 DEBUG:teuthology.orchestra.run.smithi155:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:26:29.435 INFO:teuthology.orchestra.run.smithi155.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:26:29.435 INFO:teuthology.orchestra.run.smithi155.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:26:29.435 INFO:teuthology.orchestra.run.smithi155.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:26:29.435 INFO:teuthology.orchestra.run.smithi155.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:26:29.436 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:26:29.603 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2024-09-16T03:26:29.687 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:29.726 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T03:26:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T03:26:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T03:26:30.916 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T03:26:30.916 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:26:30.916 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:26:30.916 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T03:26:30.916 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T03:26:30.916 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T03:26:30.916 INFO:teuthology.orchestra.run.smithi112.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T03:26:30.916 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:26:30.916 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:26:31.288 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:26:31.377 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:26:31.378 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:26:31.378 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T03:26:31.378 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:26:31.378 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:26:31.378 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T03:26:31.378 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:26:31.378 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T03:26:31.436 INFO:teuthology.orchestra.run.smithi112.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T03:26:31.436 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T03:26:31.436 DEBUG:teuthology.orchestra.run.smithi112:> sudo rpm -qi grub2-tools 2024-09-16T03:26:31.476 INFO:teuthology.orchestra.run.smithi112.stdout:Name : grub2-tools 2024-09-16T03:26:31.476 INFO:teuthology.orchestra.run.smithi112.stdout:Epoch : 1 2024-09-16T03:26:31.476 INFO:teuthology.orchestra.run.smithi112.stdout:Version : 2.06 2024-09-16T03:26:31.477 INFO:teuthology.orchestra.run.smithi112.stdout:Release : 82.el9 2024-09-16T03:26:31.477 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: x86_64 2024-09-16T03:26:31.477 INFO:teuthology.orchestra.run.smithi112.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T03:26:31.477 INFO:teuthology.orchestra.run.smithi112.stdout:Group : Unspecified 2024-09-16T03:26:31.477 INFO:teuthology.orchestra.run.smithi112.stdout:Size : 8274670 2024-09-16T03:26:31.477 INFO:teuthology.orchestra.run.smithi112.stdout:License : GPLv3+ 2024-09-16T03:26:31.477 INFO:teuthology.orchestra.run.smithi112.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T03:26:31.477 INFO:teuthology.orchestra.run.smithi112.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T03:26:31.477 INFO:teuthology.orchestra.run.smithi112.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T03:26:31.477 INFO:teuthology.orchestra.run.smithi112.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T03:26:31.477 INFO:teuthology.orchestra.run.smithi112.stdout:Packager : builder@centos.org 2024-09-16T03:26:31.477 INFO:teuthology.orchestra.run.smithi112.stdout:Vendor : CentOS 2024-09-16T03:26:31.478 INFO:teuthology.orchestra.run.smithi112.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T03:26:31.478 INFO:teuthology.orchestra.run.smithi112.stdout:Summary : Support tools for GRUB. 2024-09-16T03:26:31.478 INFO:teuthology.orchestra.run.smithi112.stdout:Description : 2024-09-16T03:26:31.478 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:26:31.478 INFO:teuthology.orchestra.run.smithi112.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T03:26:31.478 INFO:teuthology.orchestra.run.smithi112.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T03:26:31.478 INFO:teuthology.orchestra.run.smithi112.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T03:26:31.478 INFO:teuthology.orchestra.run.smithi112.stdout:hardware devices. 2024-09-16T03:26:31.478 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:26:31.478 INFO:teuthology.orchestra.run.smithi112.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T03:26:31.479 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T03:26:31.479 INFO:teuthology.task.kernel:Updating grub on smithi112 to boot 5.14.0-505.el9.x86_64 2024-09-16T03:26:31.480 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T03:26:32.126 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2024-09-16T03:26:33.411 INFO:teuthology.orchestra.run.smithi112.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T03:26:33.435 INFO:teuthology.orchestra.run.smithi112.stderr:done 2024-09-16T03:26:33.437 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-09-16T03:26:33.454 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.PQfKossAG6 2024-09-16T03:26:33.454 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PQfKossAG6 2024-09-16T03:26:33.520 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.PQfKossAG6 2024-09-16T03:26:33.659 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.PQfKossAG6 is 6KB 2024-09-16T03:26:33.708 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.PQfKossAG6 2024-09-16T03:26:33.723 DEBUG:teuthology.orchestra.run.smithi112:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T03:26:33.788 INFO:teuthology.orchestra.run.smithi112.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T03:26:33.788 INFO:teuthology.orchestra.run.smithi112.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T03:26:33.788 INFO:teuthology.orchestra.run.smithi112.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T03:26:33.788 INFO:teuthology.orchestra.run.smithi112.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T03:26:33.789 DEBUG:teuthology.orchestra.run.smithi112:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T03:26:33.943 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2024-09-16T03:26:57.025 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-16T03:26:57.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-16T03:26:57.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:26:58.621 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T03:26:58.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-16T03:26:58.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:26:59.606 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-16T03:26:59.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T03:26:59.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:03.949 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T03:27:03.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-16T03:27:03.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:15.517 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-16T03:27:17.057 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-09-16T03:27:18.077 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-09-16T03:27:22.429 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-16T03:27:24.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-16T03:27:24.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:26.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-16T03:27:26.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:27.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T03:27:27.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:27.581 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-16T03:27:29.117 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-09-16T03:27:30.141 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-09-16T03:27:31.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-16T03:27:31.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:34.493 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-16T03:27:39.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-16T03:27:39.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:41.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-16T03:27:41.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:42.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T03:27:42.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:42.657 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-16T03:27:46.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-16T03:27:46.497 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:57.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-16T03:27:57.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:27:59.549 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-09-16T03:28:04.925 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-16T03:28:14.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-16T03:28:14.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:28:14.905 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-09-16T03:28:15.263 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-16T03:28:15.264 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:28:15.264 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-09-16T03:28:15.279 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:28:15.279 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:28:15.279 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:28:15.279 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:28:16.189 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-16T03:28:16.280 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T03:28:16.281 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:28:16.375 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-09-16T03:28:16.411 DEBUG:teuthology.parallel:result is None 2024-09-16T03:28:19.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-16T03:28:19.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:28:20.284 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-09-16T03:28:20.649 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-16T03:28:20.650 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:28:20.650 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-09-16T03:28:20.664 INFO:teuthology.orchestra.run.smithi112.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:28:20.664 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:28:20.664 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:28:20.665 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:28:21.665 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T03:28:21.666 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:28:21.768 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2024-09-16T03:28:21.788 DEBUG:teuthology.parallel:result is None 2024-09-16T03:28:34.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-16T03:28:34.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:28:34.532 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-09-16T03:28:34.955 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-16T03:28:34.956 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:28:34.956 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-09-16T03:28:34.970 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:28:34.971 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:28:34.971 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:28:34.971 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:28:35.971 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-16T03:28:35.972 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:28:36.091 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2024-09-16T03:28:36.113 DEBUG:teuthology.parallel:result is None 2024-09-16T03:28:42.145 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T03:28:57.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T03:28:57.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:28:57.572 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-09-16T03:28:57.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T03:28:57.926 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-505.el9.x86_64"... 2024-09-16T03:28:57.926 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-09-16T03:28:57.943 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-505.el9.x86_64 2024-09-16T03:28:57.943 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T03:28:57.943 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:28:57.943 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T03:28:58.944 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-16T03:28:58.944 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:28:59.020 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2024-09-16T03:28:59.048 DEBUG:teuthology.parallel:result is None 2024-09-16T03:28:59.048 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T03:28:59.055 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T03:28:59.056 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:28:59.059 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:28:59.061 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:28:59.064 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:28:59.083 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T03:28:59.090 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T03:28:59.098 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T03:28:59.098 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:28:59.119 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:28:59.122 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:28:59.124 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:28:59.218 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T03:28:59.224 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T03:28:59.225 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:28:59.227 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:28:59.229 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:28:59.231 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:28:59.264 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:28:59.265 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:28:59.267 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:28:59.268 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:28:59.275 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:28:59.276 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:28:59.279 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:28:59.280 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:28:59.281 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T03:28:59.289 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T03:28:59.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:28:59.320 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:28:59.323 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:28:59.326 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:28:59.355 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T03:28:59.364 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T03:28:59.364 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:28:59.390 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:28:59.393 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:28:59.396 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:28:59.414 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:28:59.495 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:28:59.538 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:28:59.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:28:59.665 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:28:59.665 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:28:59.733 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:28:59.776 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:28:59.839 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:28:59.883 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:28:59.925 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:28:59.925 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:28:59.992 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:29:00.032 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:29:00.096 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:29:00.178 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:29:00.225 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:29:00.225 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:29:00.293 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:29:00.341 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:29:00.397 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:29:00.481 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:29:00.525 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:29:00.525 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:29:00.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-09-16T03:29:00.595 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2024-09-16T03:29:00.598 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-09-16T03:29:00.600 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2024-09-16T03:29:00.644 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:29:00.651 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:29:00.652 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:29:00.689 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:29:01.041 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T03:29:01.049 INFO:teuthology.task.internal:Starting timer... 2024-09-16T03:29:01.049 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T03:29:01.075 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T03:29:01.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-09-16T03:29:01.150 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-09-16T03:29:01.152 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2024-09-16T03:29:01.222 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: 2024-09-16T03:29:01.224 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-09-16T03:29:01.274 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-09-16T03:29:01.276 DEBUG:teuthology.orchestra.run.smithi157:> sudo service auditd rotate 2024-09-16T03:29:01.351 INFO:teuthology.orchestra.run.smithi157.stdout:Rotating logs: 2024-09-16T03:29:01.353 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T03:29:01.353 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-09-16T03:29:01.378 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-09-16T03:29:01.379 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2024-09-16T03:29:01.414 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2024-09-16T03:29:01.414 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-09-16T03:29:01.435 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-09-16T03:29:01.436 DEBUG:teuthology.orchestra.run.smithi157:> /usr/sbin/getenforce 2024-09-16T03:29:01.462 INFO:teuthology.orchestra.run.smithi157.stdout:Permissive 2024-09-16T03:29:01.463 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi112.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi155.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi157.front.sepia.ceph.com': 'permissive'} 2024-09-16T03:29:01.463 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T03:29:01.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:29:01.494 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T03:29:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:29:01.525 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T03:29:01.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:29:01.554 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T03:29:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:29:01.583 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T03:29:01.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-09-16T03:29:01.619 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2024-09-16T03:29:01.649 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-09-16T03:29:01.682 DEBUG:teuthology.orchestra.run.smithi157:> sudo /usr/sbin/setenforce permissive 2024-09-16T03:29:01.716 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T03:29:01.725 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T03:29:01.728 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T03:29:01.839 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T03:29:01.877 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T03:29:01.878 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi112.front.sepia.ceph.com,smithi155.front.sepia.ceph.com,smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T03:32:52.590 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi112.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2024-09-16T03:32:52.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-16T03:32:52.593 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:32:52.672 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-09-16T03:32:52.757 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-16T03:32:52.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-16T03:32:52.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:32:52.843 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-09-16T03:32:52.925 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-16T03:32:52.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T03:32:52.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:32:53.012 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-09-16T03:32:53.098 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T03:32:53.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-16T03:32:53.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:32:53.177 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-09-16T03:32:53.270 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-16T03:32:53.271 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T03:32:53.289 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T03:32:53.289 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:32:53.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:32:53.292 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:32:53.292 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:32:53.294 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:32:53.294 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:32:53.297 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:32:53.297 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:32:53.325 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T03:32:53.326 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T03:32:53.326 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T03:32:53.342 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T03:32:53.342 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T03:32:53.343 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T03:32:53.343 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T03:32:53.358 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T03:32:53.369 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2024-09-16T03:32:53.369 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-09-16T03:32:53.370 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-09-16T03:32:53.380 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-09-16T03:32:53.381 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2024-09-16T03:32:53.381 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-09-16T03:32:53.384 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: ntpd: command not found 2024-09-16T03:32:53.395 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T03:32:53.395 INFO:teuthology.orchestra.run.smithi157.stdout:506 Cannot talk to daemon 2024-09-16T03:32:53.396 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T03:32:53.397 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T03:32:53.410 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T03:32:53.411 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T03:32:53.411 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T03:32:53.412 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T03:32:53.424 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T03:32:53.464 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:32:53.468 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:32:53.468 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-09-16T03:32:53.468 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.468 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.468 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.468 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.470 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:32:53.471 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:32:53.473 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:32:53.474 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-09-16T03:32:53.474 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.474 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.474 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.474 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.474 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:32:53.475 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-09-16T03:32:53.475 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.475 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.475 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.475 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.479 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:32:53.482 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:32:53.482 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-09-16T03:32:53.483 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.483 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.483 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.483 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:32:53.485 INFO:teuthology.run_tasks:Running task install... 2024-09-16T03:32:53.519 DEBUG:teuthology.task.install:project ceph 2024-09-16T03:32:53.519 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}} 2024-09-16T03:32:53.519 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'} 2024-09-16T03:32:53.519 INFO:teuthology.task.install:Using flavor: default 2024-09-16T03:32:53.530 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T03:32:53.530 INFO:teuthology.task.install:extra packages: [] 2024-09-16T03:32:53.530 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'tag': None, 'wait_for_package': False} 2024-09-16T03:32:53.531 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T03:32:53.534 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'tag': None, 'wait_for_package': False} 2024-09-16T03:32:53.534 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T03:32:53.536 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'tag': None, 'wait_for_package': False} 2024-09-16T03:32:53.537 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T03:32:53.538 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'tag': None, 'wait_for_package': False} 2024-09-16T03:32:53.539 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T03:32:53.699 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/ 2024-09-16T03:32:53.699 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4930.gfc3e0bff 2024-09-16T03:32:53.703 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/ 2024-09-16T03:32:53.703 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4930.gfc3e0bff 2024-09-16T03:32:53.706 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/ 2024-09-16T03:32:53.707 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4930.gfc3e0bff 2024-09-16T03:32:53.711 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/ 2024-09-16T03:32:53.712 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4930.gfc3e0bff 2024-09-16T03:32:53.813 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T03:32:53.814 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:32:53.814 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T03:32:53.823 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T03:32:53.824 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:32:53.824 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T03:32:53.826 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T03:32:53.826 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:32:53.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T03:32:53.833 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T03:32:53.833 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:32:53.833 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T03:32:53.844 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T03:32:53.844 DEBUG:teuthology.orchestra.run.smithi157:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/fc3e0bff7468fd55478adbae7f75508b749124b6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T03:32:53.857 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T03:32:53.857 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/fc3e0bff7468fd55478adbae7f75508b749124b6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T03:32:53.858 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T03:32:53.859 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/fc3e0bff7468fd55478adbae7f75508b749124b6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T03:32:53.863 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T03:32:53.864 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/fc3e0bff7468fd55478adbae7f75508b749124b6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T03:32:53.916 DEBUG:teuthology.orchestra.run.smithi157:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T03:32:53.930 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T03:32:53.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T03:32:53.938 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T03:32:53.998 DEBUG:teuthology.orchestra.run.smithi157:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T03:32:54.014 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T03:32:54.016 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T03:32:54.017 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T03:32:54.062 INFO:teuthology.orchestra.run.smithi157.stdout:check_obsoletes = 1 2024-09-16T03:32:54.063 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum clean all 2024-09-16T03:32:54.078 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-09-16T03:32:54.080 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-09-16T03:32:54.081 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2024-09-16T03:32:54.083 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2024-09-16T03:32:54.084 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-09-16T03:32:54.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-09-16T03:32:54.300 INFO:teuthology.orchestra.run.smithi157.stdout:45 files removed 2024-09-16T03:32:54.306 INFO:teuthology.orchestra.run.smithi112.stdout:45 files removed 2024-09-16T03:32:54.314 INFO:teuthology.orchestra.run.smithi155.stdout:45 files removed 2024-09-16T03:32:54.324 INFO:teuthology.orchestra.run.smithi064.stdout:45 files removed 2024-09-16T03:32:54.328 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw 2024-09-16T03:32:54.333 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw 2024-09-16T03:32:54.341 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw 2024-09-16T03:32:54.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw 2024-09-16T03:32:54.880 INFO:teuthology.orchestra.run.smithi157.stdout:ceph packages for x86_64 320 kB/s | 84 kB 00:00 2024-09-16T03:32:54.904 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 292 kB/s | 84 kB 00:00 2024-09-16T03:32:54.909 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 295 kB/s | 84 kB 00:00 2024-09-16T03:32:54.917 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 315 kB/s | 84 kB 00:00 2024-09-16T03:32:55.164 INFO:teuthology.orchestra.run.smithi157.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-16T03:32:55.188 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-16T03:32:55.201 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 101 kB/s | 24 kB 00:00 2024-09-16T03:32:55.219 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 91 kB/s | 24 kB 00:00 2024-09-16T03:32:55.439 INFO:teuthology.orchestra.run.smithi157.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-16T03:32:55.447 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 8.7 kB/s | 1.9 kB 00:00 2024-09-16T03:32:55.469 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 8.4 kB/s | 1.9 kB 00:00 2024-09-16T03:32:55.494 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-16T03:32:56.143 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-16T03:32:57.228 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.2 MB 00:01 2024-09-16T03:32:58.130 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.2 MB 00:02 2024-09-16T03:32:58.520 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 17 MB/s | 20 MB 00:01 2024-09-16T03:32:58.556 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.2 MB 00:03 2024-09-16T03:33:00.412 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 20 MB 00:02 2024-09-16T03:33:03.087 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-16T03:33:03.848 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 20 MB 00:04 2024-09-16T03:33:04.031 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 20 MB 00:04 2024-09-16T03:33:04.763 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-16T03:33:05.330 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-16T03:33:06.173 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 98 kB/s | 19 kB 00:00 2024-09-16T03:33:06.675 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-16T03:33:07.210 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-16T03:33:08.299 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-16T03:33:08.415 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.5 MB 00:00 2024-09-16T03:33:09.900 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 19 kB 00:00 2024-09-16T03:33:09.991 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 19 kB 00:00 2024-09-16T03:33:11.437 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-16T03:33:11.629 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 14 MB/s | 23 MB 00:01 2024-09-16T03:33:12.298 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-09-16T03:33:12.699 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-16T03:33:14.752 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:33:14.766 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:33:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-16T03:33:14.766 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:33:14.766 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:33:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 11 M 2024-09-16T03:33:14.766 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-09-16T03:33:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.5 M 2024-09-16T03:33:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-16T03:33:14.766 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-16T03:33:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T03:33:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-16T03:33:14.766 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 22 M 2024-09-16T03:33:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 25 k 2024-09-16T03:33:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T03:33:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T03:33:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T03:33:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 755 k 2024-09-16T03:33:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T03:33:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T03:33:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T03:33:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T03:33:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 508 k 2024-09-16T03:33:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T03:33:14.767 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-16T03:33:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T03:33:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T03:33:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T03:33:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 45 k 2024-09-16T03:33:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 143 k 2024-09-16T03:33:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-16T03:33:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T03:33:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 320 k 2024-09-16T03:33:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 305 k 2024-09-16T03:33:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 99 k 2024-09-16T03:33:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T03:33:14.768 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T03:33:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T03:33:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T03:33:14.769 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-09-16T03:33:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T03:33:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:14.769 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:33:14.769 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:33:14.769 INFO:teuthology.orchestra.run.smithi064.stdout:Install 31 Packages 2024-09-16T03:33:14.769 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 2 Packages 2024-09-16T03:33:14.769 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:14.770 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 62 M 2024-09-16T03:33:14.770 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:33:15.236 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:33:15.251 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:33:15.251 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-16T03:33:15.251 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:33:15.252 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:33:15.252 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 11 M 2024-09-16T03:33:15.252 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-09-16T03:33:15.252 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.5 M 2024-09-16T03:33:15.252 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-16T03:33:15.252 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-16T03:33:15.252 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T03:33:15.252 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-16T03:33:15.252 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 22 M 2024-09-16T03:33:15.252 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 25 k 2024-09-16T03:33:15.252 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T03:33:15.252 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T03:33:15.253 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T03:33:15.253 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 755 k 2024-09-16T03:33:15.253 INFO:teuthology.orchestra.run.smithi155.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T03:33:15.253 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T03:33:15.253 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T03:33:15.253 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T03:33:15.253 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 508 k 2024-09-16T03:33:15.253 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T03:33:15.253 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-16T03:33:15.253 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T03:33:15.253 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T03:33:15.253 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T03:33:15.253 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 45 k 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 143 k 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 320 k 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 305 k 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 99 k 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:33:15.255 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:33:15.255 INFO:teuthology.orchestra.run.smithi155.stdout:Install 31 Packages 2024-09-16T03:33:15.255 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 2 Packages 2024-09-16T03:33:15.255 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:33:15.255 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 62 M 2024-09-16T03:33:15.255 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:33:15.633 INFO:teuthology.orchestra.run.smithi064.stdout:(1/33): ceph-base-19.3.0-4930.gfc3e0bff.el9.x86 13 MB/s | 5.5 MB 00:00 2024-09-16T03:33:15.700 INFO:teuthology.orchestra.run.smithi064.stdout:(2/33): ceph-selinux-19.3.0-4930.gfc3e0bff.el9. 378 kB/s | 25 kB 00:00 2024-09-16T03:33:15.791 INFO:teuthology.orchestra.run.smithi064.stdout:(3/33): libcephfs2-19.3.0-4930.gfc3e0bff.el9.x8 8.1 MB/s | 755 kB 00:00 2024-09-16T03:33:15.883 INFO:teuthology.orchestra.run.smithi064.stdout:(4/33): libradosstriper1-19.3.0-4930.gfc3e0bff. 5.4 MB/s | 508 kB 00:00 2024-09-16T03:33:16.250 INFO:teuthology.orchestra.run.smithi064.stdout:(5/33): ceph-radosgw-19.3.0-4930.gfc3e0bff.el9. 10 MB/s | 11 MB 00:01 2024-09-16T03:33:16.326 INFO:teuthology.orchestra.run.smithi064.stdout:(6/33): librgw2-19.3.0-4930.gfc3e0bff.el9.x86_6 12 MB/s | 5.5 MB 00:00 2024-09-16T03:33:16.350 INFO:teuthology.orchestra.run.smithi064.stdout:(7/33): python3-ceph-argparse-19.3.0-4930.gfc3e 451 kB/s | 45 kB 00:00 2024-09-16T03:33:16.376 INFO:teuthology.orchestra.run.smithi064.stdout:(8/33): python3-ceph-common-19.3.0-4930.gfc3e0b 2.8 MB/s | 143 kB 00:00 2024-09-16T03:33:16.401 INFO:teuthology.orchestra.run.smithi064.stdout:(9/33): python3-cephfs-19.3.0-4930.gfc3e0bff.el 3.1 MB/s | 161 kB 00:00 2024-09-16T03:33:16.426 INFO:teuthology.orchestra.run.smithi064.stdout:(10/33): python3-rados-19.3.0-4930.gfc3e0bff.el 6.2 MB/s | 320 kB 00:00 2024-09-16T03:33:16.452 INFO:teuthology.orchestra.run.smithi064.stdout:(11/33): python3-rbd-19.3.0-4930.gfc3e0bff.el9. 6.0 MB/s | 305 kB 00:00 2024-09-16T03:33:16.477 INFO:teuthology.orchestra.run.smithi064.stdout:(12/33): python3-rgw-19.3.0-4930.gfc3e0bff.el9. 1.9 MB/s | 99 kB 00:00 2024-09-16T03:33:16.686 INFO:teuthology.orchestra.run.smithi064.stdout:(13/33): ceph-common-19.3.0-4930.gfc3e0bff.el9. 15 MB/s | 22 MB 00:01 2024-09-16T03:33:16.744 INFO:teuthology.orchestra.run.smithi064.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 366 kB/s | 107 kB 00:00 2024-09-16T03:33:16.786 INFO:teuthology.orchestra.run.smithi064.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 552 kB/s | 171 kB 00:00 2024-09-16T03:33:16.811 INFO:teuthology.orchestra.run.smithi064.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 676 kB/s | 45 kB 00:00 2024-09-16T03:33:16.837 INFO:teuthology.orchestra.run.smithi064.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-16T03:33:16.870 INFO:teuthology.orchestra.run.smithi064.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-09-16T03:33:16.895 INFO:teuthology.orchestra.run.smithi064.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 722 kB/s | 42 kB 00:00 2024-09-16T03:33:16.918 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-16T03:33:16.920 INFO:teuthology.orchestra.run.smithi064.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-09-16T03:33:16.946 INFO:teuthology.orchestra.run.smithi064.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 716 kB/s | 36 kB 00:00 2024-09-16T03:33:16.971 INFO:teuthology.orchestra.run.smithi064.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 941 kB/s | 47 kB 00:00 2024-09-16T03:33:16.997 INFO:teuthology.orchestra.run.smithi064.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-16T03:33:17.051 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-16T03:33:17.055 INFO:teuthology.orchestra.run.smithi064.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 602 kB/s | 26 kB 00:00 2024-09-16T03:33:17.080 INFO:teuthology.orchestra.run.smithi064.stdout:(25/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.2 MB/s | 308 kB 00:00 2024-09-16T03:33:17.106 INFO:teuthology.orchestra.run.smithi064.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-09-16T03:33:17.133 INFO:teuthology.orchestra.run.smithi064.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T03:33:17.175 INFO:teuthology.orchestra.run.smithi064.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-16T03:33:17.209 INFO:teuthology.orchestra.run.smithi064.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.0 MB/s | 839 kB 00:00 2024-09-16T03:33:17.359 INFO:teuthology.orchestra.run.smithi064.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-09-16T03:33:17.426 INFO:teuthology.orchestra.run.smithi064.stdout:(31/33): librados2-19.3.0-4930.gfc3e0bff.el9.x8 16 MB/s | 3.5 MB 00:00 2024-09-16T03:33:17.543 INFO:teuthology.orchestra.run.smithi064.stdout:(32/33): libarrow-9.0.0-11.el9.x86_64.rpm 7.7 MB/s | 4.4 MB 00:00 2024-09-16T03:33:17.592 INFO:teuthology.orchestra.run.smithi155.stdout:(1/33): ceph-base-19.3.0-4930.gfc3e0bff.el9.x86 2.6 MB/s | 5.5 MB 00:02 2024-09-16T03:33:17.601 INFO:teuthology.orchestra.run.smithi064.stdout:(33/33): librbd1-19.3.0-4930.gfc3e0bff.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-16T03:33:17.606 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:33:17.606 INFO:teuthology.orchestra.run.smithi064.stdout:Total 22 MB/s | 62 MB 00:02 2024-09-16T03:33:17.642 INFO:teuthology.orchestra.run.smithi155.stdout:(2/33): ceph-selinux-19.3.0-4930.gfc3e0bff.el9. 507 kB/s | 25 kB 00:00 2024-09-16T03:33:17.719 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:33:17.759 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:33:17.759 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:33:17.834 INFO:teuthology.orchestra.run.smithi155.stdout:(3/33): libcephfs2-19.3.0-4930.gfc3e0bff.el9.x8 3.8 MB/s | 755 kB 00:00 2024-09-16T03:33:17.977 INFO:teuthology.orchestra.run.smithi155.stdout:(4/33): libradosstriper1-19.3.0-4930.gfc3e0bff. 3.5 MB/s | 508 kB 00:00 2024-09-16T03:33:18.432 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:33:18.432 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:33:18.635 INFO:teuthology.orchestra.run.smithi155.stdout:(5/33): ceph-radosgw-19.3.0-4930.gfc3e0bff.el9. 3.4 MB/s | 11 MB 00:03 2024-09-16T03:33:18.677 INFO:teuthology.orchestra.run.smithi155.stdout:(6/33): python3-ceph-argparse-19.3.0-4930.gfc3e 1.1 MB/s | 45 kB 00:00 2024-09-16T03:33:18.744 INFO:teuthology.orchestra.run.smithi155.stdout:(7/33): python3-ceph-common-19.3.0-4930.gfc3e0b 2.1 MB/s | 143 kB 00:00 2024-09-16T03:33:18.811 INFO:teuthology.orchestra.run.smithi155.stdout:(8/33): python3-cephfs-19.3.0-4930.gfc3e0bff.el 2.4 MB/s | 161 kB 00:00 2024-09-16T03:33:18.911 INFO:teuthology.orchestra.run.smithi155.stdout:(9/33): python3-rados-19.3.0-4930.gfc3e0bff.el9 3.1 MB/s | 320 kB 00:00 2024-09-16T03:33:19.053 INFO:teuthology.orchestra.run.smithi155.stdout:(10/33): librgw2-19.3.0-4930.gfc3e0bff.el9.x86_ 5.1 MB/s | 5.5 MB 00:01 2024-09-16T03:33:19.080 INFO:teuthology.orchestra.run.smithi155.stdout:(11/33): python3-rbd-19.3.0-4930.gfc3e0bff.el9. 1.8 MB/s | 305 kB 00:00 2024-09-16T03:33:19.112 INFO:teuthology.orchestra.run.smithi155.stdout:(12/33): python3-rgw-19.3.0-4930.gfc3e0bff.el9. 1.7 MB/s | 99 kB 00:00 2024-09-16T03:33:19.196 INFO:teuthology.orchestra.run.smithi155.stdout:(13/33): boost-program-options-1.75.0-8.el9.x86 919 kB/s | 107 kB 00:00 2024-09-16T03:33:19.230 INFO:teuthology.orchestra.run.smithi155.stdout:(14/33): libnbd-1.20.2-2.el9.x86_64.rpm 1.4 MB/s | 171 kB 00:00 2024-09-16T03:33:19.272 INFO:teuthology.orchestra.run.smithi155.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-09-16T03:33:19.295 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:33:19.297 INFO:teuthology.orchestra.run.smithi155.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 676 kB/s | 45 kB 00:00 2024-09-16T03:33:19.309 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:33:19.309 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-16T03:33:19.309 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:33:19.309 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:33:19.309 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 11 M 2024-09-16T03:33:19.309 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2024-09-16T03:33:19.309 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.5 M 2024-09-16T03:33:19.309 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-16T03:33:19.310 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-16T03:33:19.310 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T03:33:19.310 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-16T03:33:19.310 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 22 M 2024-09-16T03:33:19.310 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 25 k 2024-09-16T03:33:19.310 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T03:33:19.310 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T03:33:19.310 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T03:33:19.310 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 755 k 2024-09-16T03:33:19.310 INFO:teuthology.orchestra.run.smithi112.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T03:33:19.310 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T03:33:19.310 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T03:33:19.310 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T03:33:19.311 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 508 k 2024-09-16T03:33:19.311 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T03:33:19.311 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-16T03:33:19.311 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T03:33:19.311 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T03:33:19.311 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T03:33:19.311 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 45 k 2024-09-16T03:33:19.311 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 143 k 2024-09-16T03:33:19.311 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-16T03:33:19.311 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T03:33:19.311 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 320 k 2024-09-16T03:33:19.311 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 305 k 2024-09-16T03:33:19.311 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 99 k 2024-09-16T03:33:19.312 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T03:33:19.312 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T03:33:19.312 INFO:teuthology.orchestra.run.smithi112.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T03:33:19.312 INFO:teuthology.orchestra.run.smithi112.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T03:33:19.312 INFO:teuthology.orchestra.run.smithi112.stdout:Installing weak dependencies: 2024-09-16T03:33:19.312 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T03:33:19.312 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:33:19.312 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:33:19.312 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:33:19.312 INFO:teuthology.orchestra.run.smithi112.stdout:Install 31 Packages 2024-09-16T03:33:19.312 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 2 Packages 2024-09-16T03:33:19.312 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:33:19.313 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 62 M 2024-09-16T03:33:19.313 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:33:19.372 INFO:teuthology.orchestra.run.smithi155.stdout:(17/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-09-16T03:33:19.398 INFO:teuthology.orchestra.run.smithi155.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-09-16T03:33:19.411 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:33:19.422 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:33:19.423 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-16T03:33:19.423 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:33:19.423 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:33:19.423 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 11 M 2024-09-16T03:33:19.423 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrading: 2024-09-16T03:33:19.423 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.5 M 2024-09-16T03:33:19.423 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-16T03:33:19.423 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-16T03:33:19.423 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T03:33:19.423 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-16T03:33:19.423 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 22 M 2024-09-16T03:33:19.423 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 25 k 2024-09-16T03:33:19.424 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T03:33:19.424 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T03:33:19.424 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T03:33:19.424 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 755 k 2024-09-16T03:33:19.424 INFO:teuthology.orchestra.run.smithi157.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T03:33:19.424 INFO:teuthology.orchestra.run.smithi157.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T03:33:19.424 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T03:33:19.424 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T03:33:19.424 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 508 k 2024-09-16T03:33:19.424 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T03:33:19.424 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-16T03:33:19.424 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T03:33:19.425 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T03:33:19.425 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T03:33:19.425 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 45 k 2024-09-16T03:33:19.425 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 143 k 2024-09-16T03:33:19.425 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-16T03:33:19.425 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T03:33:19.425 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 320 k 2024-09-16T03:33:19.425 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 305 k 2024-09-16T03:33:19.425 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 99 k 2024-09-16T03:33:19.425 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T03:33:19.425 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T03:33:19.425 INFO:teuthology.orchestra.run.smithi157.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T03:33:19.425 INFO:teuthology.orchestra.run.smithi157.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T03:33:19.426 INFO:teuthology.orchestra.run.smithi157.stdout:Installing weak dependencies: 2024-09-16T03:33:19.426 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T03:33:19.426 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:33:19.426 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:33:19.426 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:33:19.426 INFO:teuthology.orchestra.run.smithi157.stdout:Install 31 Packages 2024-09-16T03:33:19.426 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrade 2 Packages 2024-09-16T03:33:19.426 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:33:19.428 INFO:teuthology.orchestra.run.smithi155.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 839 kB/s | 42 kB 00:00 2024-09-16T03:33:19.428 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 62 M 2024-09-16T03:33:19.428 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:33:19.465 INFO:teuthology.orchestra.run.smithi155.stdout:(20/33): qatlib-service-24.02.0-1.el9.x86_64.rp 863 kB/s | 36 kB 00:00 2024-09-16T03:33:19.499 INFO:teuthology.orchestra.run.smithi155.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-09-16T03:33:19.524 INFO:teuthology.orchestra.run.smithi155.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 810 kB/s | 47 kB 00:00 2024-09-16T03:33:19.599 INFO:teuthology.orchestra.run.smithi155.stdout:(23/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-16T03:33:19.625 INFO:teuthology.orchestra.run.smithi155.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-16T03:33:19.658 INFO:teuthology.orchestra.run.smithi155.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-16T03:33:19.709 INFO:teuthology.orchestra.run.smithi155.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T03:33:19.809 INFO:teuthology.orchestra.run.smithi155.stdout:(27/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-16T03:33:19.851 INFO:teuthology.orchestra.run.smithi155.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-16T03:33:19.968 INFO:teuthology.orchestra.run.smithi155.stdout:(29/33): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-16T03:33:20.004 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:33:20.035 INFO:teuthology.orchestra.run.smithi155.stdout:(30/33): libarrow-9.0.0-11.el9.x86_64.rpm 8.6 MB/s | 4.4 MB 00:00 2024-09-16T03:33:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T03:33:20.213 INFO:teuthology.orchestra.run.smithi157.stdout:(1/33): ceph-base-19.3.0-4930.gfc3e0bff.el9.x86 9.0 MB/s | 5.5 MB 00:00 2024-09-16T03:33:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-16T03:33:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-16T03:33:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T03:33:20.286 INFO:teuthology.orchestra.run.smithi157.stdout:(2/33): ceph-selinux-19.3.0-4930.gfc3e0bff.el9. 336 kB/s | 25 kB 00:00 2024-09-16T03:33:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T03:33:20.386 INFO:teuthology.orchestra.run.smithi157.stdout:(3/33): libcephfs2-19.3.0-4930.gfc3e0bff.el9.x8 7.4 MB/s | 755 kB 00:00 2024-09-16T03:33:20.500 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T03:33:20.503 INFO:teuthology.orchestra.run.smithi157.stdout:(4/33): ceph-radosgw-19.3.0-4930.gfc3e0bff.el9. 12 MB/s | 11 MB 00:00 2024-09-16T03:33:20.528 INFO:teuthology.orchestra.run.smithi157.stdout:(5/33): libradosstriper1-19.3.0-4930.gfc3e0bff. 3.5 MB/s | 508 kB 00:00 2024-09-16T03:33:20.553 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T03:33:20.595 INFO:teuthology.orchestra.run.smithi157.stdout:(6/33): python3-ceph-argparse-19.3.0-4930.gfc3e 676 kB/s | 45 kB 00:00 2024-09-16T03:33:20.679 INFO:teuthology.orchestra.run.smithi157.stdout:(7/33): python3-ceph-common-19.3.0-4930.gfc3e0b 1.7 MB/s | 143 kB 00:00 2024-09-16T03:33:20.763 INFO:teuthology.orchestra.run.smithi157.stdout:(8/33): python3-cephfs-19.3.0-4930.gfc3e0bff.el 1.9 MB/s | 161 kB 00:00 2024-09-16T03:33:20.855 INFO:teuthology.orchestra.run.smithi157.stdout:(9/33): librgw2-19.3.0-4930.gfc3e0bff.el9.x86_6 16 MB/s | 5.5 MB 00:00 2024-09-16T03:33:20.880 INFO:teuthology.orchestra.run.smithi157.stdout:(10/33): python3-rados-19.3.0-4930.gfc3e0bff.el 2.7 MB/s | 320 kB 00:00 2024-09-16T03:33:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-16T03:33:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-16T03:33:21.035 INFO:teuthology.orchestra.run.smithi155.stdout:(31/33): ceph-common-19.3.0-4930.gfc3e0bff.el9. 3.9 MB/s | 22 MB 00:05 2024-09-16T03:33:21.088 INFO:teuthology.orchestra.run.smithi157.stdout:(11/33): ceph-common-19.3.0-4930.gfc3e0bff.el9. 15 MB/s | 22 MB 00:01 2024-09-16T03:33:21.094 INFO:teuthology.orchestra.run.smithi155.stdout:(32/33): librados2-19.3.0-4930.gfc3e0bff.el9.x8 3.1 MB/s | 3.5 MB 00:01 2024-09-16T03:33:21.101 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-16T03:33:21.122 INFO:teuthology.orchestra.run.smithi157.stdout:(12/33): python3-rgw-19.3.0-4930.gfc3e0bff.el9. 410 kB/s | 99 kB 00:00 2024-09-16T03:33:21.144 INFO:teuthology.orchestra.run.smithi155.stdout:(33/33): librbd1-19.3.0-4930.gfc3e0bff.el9.x86_ 2.9 MB/s | 3.2 MB 00:01 2024-09-16T03:33:21.147 INFO:teuthology.orchestra.run.smithi157.stdout:(13/33): python3-rbd-19.3.0-4930.gfc3e0bff.el9. 1.0 MB/s | 305 kB 00:00 2024-09-16T03:33:21.150 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:33:21.150 INFO:teuthology.orchestra.run.smithi155.stdout:Total 10 MB/s | 62 MB 00:05 2024-09-16T03:33:21.192 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 8/35 2024-09-16T03:33:21.218 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-16T03:33:21.270 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:33:21.281 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-16T03:33:21.290 INFO:teuthology.orchestra.run.smithi157.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 533 kB/s | 107 kB 00:00 2024-09-16T03:33:21.306 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:33:21.306 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:33:21.315 INFO:teuthology.orchestra.run.smithi157.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 887 kB/s | 171 kB 00:00 2024-09-16T03:33:21.318 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-16T03:33:21.340 INFO:teuthology.orchestra.run.smithi157.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 833 kB/s | 160 kB 00:00 2024-09-16T03:33:21.357 INFO:teuthology.orchestra.run.smithi157.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 678 kB/s | 45 kB 00:00 2024-09-16T03:33:21.368 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 11/35 2024-09-16T03:33:21.407 INFO:teuthology.orchestra.run.smithi157.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 843 kB/s | 42 kB 00:00 2024-09-16T03:33:21.433 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-16T03:33:21.441 INFO:teuthology.orchestra.run.smithi157.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 662 kB 00:00 2024-09-16T03:33:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-16T03:33:21.466 INFO:teuthology.orchestra.run.smithi157.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-16T03:33:21.491 INFO:teuthology.orchestra.run.smithi157.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-16T03:33:21.508 INFO:teuthology.orchestra.run.smithi157.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 536 kB/s | 36 kB 00:00 2024-09-16T03:33:21.518 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-16T03:33:21.525 INFO:teuthology.orchestra.run.smithi157.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 808 kB/s | 47 kB 00:00 2024-09-16T03:33:21.552 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-16T03:33:21.584 INFO:teuthology.orchestra.run.smithi157.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-09-16T03:33:21.609 INFO:teuthology.orchestra.run.smithi157.stdout:(25/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-16T03:33:21.634 INFO:teuthology.orchestra.run.smithi157.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 970 kB/s | 49 kB 00:00 2024-09-16T03:33:21.659 INFO:teuthology.orchestra.run.smithi157.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2024-09-16T03:33:21.663 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-16T03:33:21.695 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-16T03:33:21.718 INFO:teuthology.orchestra.run.smithi157.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-16T03:33:21.749 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-16T03:33:21.752 INFO:teuthology.orchestra.run.smithi157.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-16T03:33:21.784 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-16T03:33:21.814 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-16T03:33:21.847 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T03:33:21.871 INFO:teuthology.orchestra.run.smithi112.stdout:(1/33): ceph-base-19.3.0-4930.gfc3e0bff.el9.x86 2.6 MB/s | 5.5 MB 00:02 2024-09-16T03:33:21.890 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-16T03:33:21.913 INFO:teuthology.orchestra.run.smithi112.stdout:(2/33): ceph-selinux-19.3.0-4930.gfc3e0bff.el9. 607 kB/s | 25 kB 00:00 2024-09-16T03:33:21.952 INFO:teuthology.orchestra.run.smithi157.stdout:(30/33): librados2-19.3.0-4930.gfc3e0bff.el9.x8 17 MB/s | 3.5 MB 00:00 2024-09-16T03:33:21.990 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:33:21.990 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:33:22.077 INFO:teuthology.orchestra.run.smithi157.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-09-16T03:33:22.105 INFO:teuthology.orchestra.run.smithi112.stdout:(3/33): libcephfs2-19.3.0-4930.gfc3e0bff.el9.x8 3.8 MB/s | 755 kB 00:00 2024-09-16T03:33:22.136 INFO:teuthology.orchestra.run.smithi157.stdout:(32/33): librbd1-19.3.0-4930.gfc3e0bff.el9.x86_ 18 MB/s | 3.2 MB 00:00 2024-09-16T03:33:22.177 INFO:teuthology.orchestra.run.smithi157.stdout:(33/33): thrift-0.15.0-2.el9.x86_64.rpm 3.4 MB/s | 1.6 MB 00:00 2024-09-16T03:33:22.181 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:33:22.181 INFO:teuthology.orchestra.run.smithi157.stdout:Total 22 MB/s | 62 MB 00:02 2024-09-16T03:33:22.228 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T03:33:22.239 INFO:teuthology.orchestra.run.smithi112.stdout:(4/33): libradosstriper1-19.3.0-4930.gfc3e0bff. 3.7 MB/s | 508 kB 00:00 2024-09-16T03:33:22.269 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-16T03:33:22.294 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:33:22.331 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:33:22.331 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:33:22.361 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-16T03:33:22.397 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 24/35 2024-09-16T03:33:22.560 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T03:33:23.009 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:33:23.009 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:33:23.083 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T03:33:23.113 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-16T03:33:23.167 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-16T03:33:23.197 INFO:teuthology.orchestra.run.smithi112.stdout:(5/33): librgw2-19.3.0-4930.gfc3e0bff.el9.x86_6 5.7 MB/s | 5.5 MB 00:00 2024-09-16T03:33:23.239 INFO:teuthology.orchestra.run.smithi112.stdout:(6/33): python3-ceph-argparse-19.3.0-4930.gfc3e 1.1 MB/s | 45 kB 00:00 2024-09-16T03:33:23.240 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 28/35 2024-09-16T03:33:23.281 INFO:teuthology.orchestra.run.smithi112.stdout:(7/33): python3-ceph-common-19.3.0-4930.gfc3e0b 3.4 MB/s | 143 kB 00:00 2024-09-16T03:33:23.281 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 29/35 2024-09-16T03:33:23.414 INFO:teuthology.orchestra.run.smithi112.stdout:(8/33): ceph-radosgw-19.3.0-4930.gfc3e0bff.el9. 3.0 MB/s | 11 MB 00:03 2024-09-16T03:33:23.448 INFO:teuthology.orchestra.run.smithi112.stdout:(9/33): python3-cephfs-19.3.0-4930.gfc3e0bff.el 967 kB/s | 161 kB 00:00 2024-09-16T03:33:23.507 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:33:23.515 INFO:teuthology.orchestra.run.smithi112.stdout:(10/33): python3-rados-19.3.0-4930.gfc3e0bff.el 3.1 MB/s | 320 kB 00:00 2024-09-16T03:33:23.540 INFO:teuthology.orchestra.run.smithi112.stdout:(11/33): python3-rbd-19.3.0-4930.gfc3e0bff.el9. 3.2 MB/s | 305 kB 00:00 2024-09-16T03:33:23.566 INFO:teuthology.orchestra.run.smithi112.stdout:(12/33): python3-rgw-19.3.0-4930.gfc3e0bff.el9. 1.9 MB/s | 99 kB 00:00 2024-09-16T03:33:23.614 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T03:33:23.641 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-16T03:33:23.667 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-16T03:33:23.682 INFO:teuthology.orchestra.run.smithi112.stdout:(13/33): boost-program-options-1.75.0-8.el9.x86 754 kB/s | 107 kB 00:00 2024-09-16T03:33:23.689 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T03:33:23.712 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T03:33:23.716 INFO:teuthology.orchestra.run.smithi112.stdout:(14/33): libnbd-1.20.2-2.el9.x86_64.rpm 1.1 MB/s | 171 kB 00:00 2024-09-16T03:33:23.766 INFO:teuthology.orchestra.run.smithi112.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-09-16T03:33:23.791 INFO:teuthology.orchestra.run.smithi112.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 603 kB/s | 45 kB 00:00 2024-09-16T03:33:23.867 INFO:teuthology.orchestra.run.smithi112.stdout:(17/33): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-09-16T03:33:23.893 INFO:teuthology.orchestra.run.smithi112.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 292 kB 00:00 2024-09-16T03:33:23.907 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T03:33:23.917 INFO:teuthology.orchestra.run.smithi112.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 842 kB/s | 42 kB 00:00 2024-09-16T03:33:23.942 INFO:teuthology.orchestra.run.smithi112.stdout:(20/33): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-09-16T03:33:23.962 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T03:33:23.968 INFO:teuthology.orchestra.run.smithi112.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 714 kB/s | 36 kB 00:00 2024-09-16T03:33:23.993 INFO:teuthology.orchestra.run.smithi112.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-09-16T03:33:24.060 INFO:teuthology.orchestra.run.smithi112.stdout:(23/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.3 MB/s | 308 kB 00:00 2024-09-16T03:33:24.086 INFO:teuthology.orchestra.run.smithi112.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-16T03:33:24.119 INFO:teuthology.orchestra.run.smithi112.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-16T03:33:24.144 INFO:teuthology.orchestra.run.smithi112.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2024-09-16T03:33:24.244 INFO:teuthology.orchestra.run.smithi112.stdout:(27/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-16T03:33:24.287 INFO:teuthology.orchestra.run.smithi112.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-16T03:33:24.368 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-16T03:33:24.426 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-16T03:33:24.489 INFO:teuthology.orchestra.run.smithi112.stdout:(29/33): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-09-16T03:33:24.511 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-16T03:33:24.534 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:33:24.546 INFO:teuthology.orchestra.run.smithi112.stdout:(30/33): libarrow-9.0.0-11.el9.x86_64.rpm 8.0 MB/s | 4.4 MB 00:00 2024-09-16T03:33:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 8/35 2024-09-16T03:33:24.620 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T03:33:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-16T03:33:24.657 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-16T03:33:24.683 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-16T03:33:24.692 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-16T03:33:24.706 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T03:33:24.880 INFO:teuthology.orchestra.run.smithi112.stdout:(31/33): ceph-common-19.3.0-4930.gfc3e0bff.el9. 4.2 MB/s | 22 MB 00:05 2024-09-16T03:33:24.880 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T03:33:24.881 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-16T03:33:24.881 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 11/35 2024-09-16T03:33:24.881 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-16T03:33:24.881 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-16T03:33:24.917 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T03:33:24.919 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-16T03:33:24.953 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-16T03:33:24.971 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T03:33:25.062 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-16T03:33:25.097 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-16T03:33:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-16T03:33:25.186 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-16T03:33:25.215 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-16T03:33:25.238 INFO:teuthology.orchestra.run.smithi112.stdout:(32/33): librados2-19.3.0-4930.gfc3e0bff.el9.x8 4.6 MB/s | 3.5 MB 00:00 2024-09-16T03:33:25.248 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T03:33:25.287 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-16T03:33:25.321 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-16T03:33:25.363 INFO:teuthology.orchestra.run.smithi112.stdout:(33/33): librbd1-19.3.0-4930.gfc3e0bff.el9.x86_ 4.0 MB/s | 3.2 MB 00:00 2024-09-16T03:33:25.368 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:33:25.368 INFO:teuthology.orchestra.run.smithi112.stdout:Total 10 MB/s | 62 MB 00:06 2024-09-16T03:33:25.435 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-16T03:33:25.488 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:33:25.524 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:33:25.525 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:33:25.527 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-16T03:33:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T03:33:25.617 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 8/35 2024-09-16T03:33:25.636 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-16T03:33:25.637 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-16T03:33:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-16T03:33:25.670 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-16T03:33:25.699 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-16T03:33:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-16T03:33:25.736 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-16T03:33:25.767 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 24/35 2024-09-16T03:33:25.803 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 11/35 2024-09-16T03:33:25.811 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T03:33:25.875 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-16T03:33:25.905 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-16T03:33:25.969 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-16T03:33:26.003 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-16T03:33:26.115 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-16T03:33:26.149 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-16T03:33:26.197 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:33:26.198 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:33:26.198 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-16T03:33:26.228 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-16T03:33:26.257 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-16T03:33:26.279 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-16T03:33:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T03:33:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T03:33:26.315 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-16T03:33:26.332 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-16T03:33:26.342 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-16T03:33:26.385 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-16T03:33:26.390 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-16T03:33:26.390 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T03:33:26.390 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T03:33:26.390 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-16T03:33:26.457 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 28/35 2024-09-16T03:33:26.483 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 29/35 2024-09-16T03:33:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T03:33:26.711 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-16T03:33:26.803 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-16T03:33:26.847 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 24/35 2024-09-16T03:33:26.886 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T03:33:27.385 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T03:33:27.406 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-16T03:33:27.477 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-16T03:33:27.556 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 28/35 2024-09-16T03:33:27.593 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 29/35 2024-09-16T03:33:27.693 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:33:27.780 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T03:33:27.809 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-16T03:33:27.843 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-16T03:33:27.865 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T03:33:27.887 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T03:33:28.028 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T03:33:28.087 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T03:33:28.489 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-16T03:33:28.536 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-16T03:33:28.620 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-16T03:33:28.710 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 8/35 2024-09-16T03:33:28.749 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-16T03:33:28.809 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-16T03:33:28.837 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-16T03:33:28.846 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-16T03:33:28.881 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-16T03:33:28.887 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 11/35 2024-09-16T03:33:28.952 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-16T03:33:28.981 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-16T03:33:29.030 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-16T03:33:29.063 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-16T03:33:29.181 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-16T03:33:29.214 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-16T03:33:29.261 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-16T03:33:29.296 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-16T03:33:29.325 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-16T03:33:29.358 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T03:33:29.395 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-16T03:33:29.447 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-16T03:33:29.488 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-16T03:33:29.590 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-16T03:33:29.590 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T03:33:29.590 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T03:33:29.590 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:33:29.631 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-16T03:33:29.732 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T03:33:29.780 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-16T03:33:29.871 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-16T03:33:29.891 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-16T03:33:29.910 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 24/35 2024-09-16T03:33:29.954 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T03:33:29.988 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-16T03:33:30.443 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T03:33:30.482 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-16T03:33:30.553 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-16T03:33:30.575 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-16T03:33:30.616 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-16T03:33:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 28/35 2024-09-16T03:33:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 29/35 2024-09-16T03:33:30.708 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-16T03:33:30.709 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T03:33:30.709 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T03:33:30.709 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:33:30.743 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-16T03:33:32.977 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-16T03:33:33.005 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-16T03:33:33.587 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-16T03:33:33.625 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-16T03:33:33.726 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-16T03:33:33.726 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T03:33:33.726 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T03:33:33.727 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:33:33.760 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-16T03:33:36.945 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-16T03:33:36.945 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-09-16T03:33:36.945 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-09-16T03:33:36.945 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-09-16T03:33:36.945 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-09-16T03:33:36.945 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-09-16T03:33:36.945 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-09-16T03:33:36.945 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-09-16T03:33:36.946 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-16T03:33:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-16T03:33:37.995 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:33:37.995 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T03:33:37.995 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:33:37.995 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:33:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T03:33:38.079 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T03:33:38.079 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T03:33:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T03:33:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/35 2024-09-16T03:33:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/35 2024-09-16T03:33:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/35 2024-09-16T03:33:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/35 2024-09-16T03:33:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/35 2024-09-16T03:33:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 6/35 2024-09-16T03:33:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-16T03:33:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 8/35 2024-09-16T03:33:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 9/35 2024-09-16T03:33:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 10/35 2024-09-16T03:33:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 11/35 2024-09-16T03:33:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-16T03:33:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 13/35 2024-09-16T03:33:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T03:33:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T03:33:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T03:33:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T03:33:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T03:33:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T03:33:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T03:33:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T03:33:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T03:33:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T03:33:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T03:33:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T03:33:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T03:33:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T03:33:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T03:33:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T03:33:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T03:33:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T03:33:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-16T03:33:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T03:33:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/35 2024-09-16T03:33:39.714 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-16T03:33:39.715 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-09-16T03:33:39.715 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-09-16T03:33:39.715 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-09-16T03:33:39.715 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-09-16T03:33:39.715 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-09-16T03:33:39.715 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /root 2024-09-16T03:33:39.715 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-09-16T03:33:39.715 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:33:40.087 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T03:33:40.087 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:40.087 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-09-16T03:33:40.088 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.088 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.088 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:33:40.088 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T03:33:40.088 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.088 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.088 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.088 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.088 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T03:33:40.088 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T03:33:40.088 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T03:33:40.088 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.089 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T03:33:40.090 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.090 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.090 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:40.090 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T03:33:40.090 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T03:33:40.090 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T03:33:40.090 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T03:33:40.090 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T03:33:40.090 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:40.090 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:33:40.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-test 2024-09-16T03:33:40.667 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-16T03:33:40.667 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /sys 2024-09-16T03:33:40.668 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /proc 2024-09-16T03:33:40.668 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /mnt 2024-09-16T03:33:40.668 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /var/tmp 2024-09-16T03:33:40.668 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /home 2024-09-16T03:33:40.668 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /root 2024-09-16T03:33:40.668 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /tmp 2024-09-16T03:33:40.668 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:33:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-16T03:33:40.750 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-16T03:33:40.750 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:33:40.750 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T03:33:40.750 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:33:40.750 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:33:40.750 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:33:40.751 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T03:33:40.835 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T03:33:40.835 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T03:33:40.988 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:33:41.126 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:33:41.127 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:33:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-16T03:33:41.127 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:33:41.127 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:33:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 46 M 2024-09-16T03:33:41.127 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-16T03:33:41.127 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-16T03:33:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T03:33:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T03:33:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:41.128 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:33:41.128 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:33:41.128 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-09-16T03:33:41.128 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:41.129 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 46 M 2024-09-16T03:33:41.129 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 201 M 2024-09-16T03:33:41.129 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:33:41.376 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-16T03:33:41.421 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): libcephsqlite-19.3.0-4930.gfc3e0bff.el9. 595 kB/s | 161 kB 00:00 2024-09-16T03:33:41.451 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 848 kB/s | 64 kB 00:00 2024-09-16T03:33:41.640 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-16T03:33:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-16T03:33:41.667 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:33:41.667 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T03:33:41.667 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:33:41.667 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:33:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:33:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T03:33:41.793 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T03:33:41.793 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T03:33:42.022 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T03:33:42.022 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/35 2024-09-16T03:33:42.022 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/35 2024-09-16T03:33:42.022 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/35 2024-09-16T03:33:42.022 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/35 2024-09-16T03:33:42.023 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/35 2024-09-16T03:33:42.023 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 6/35 2024-09-16T03:33:42.023 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-16T03:33:42.023 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 8/35 2024-09-16T03:33:42.023 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 9/35 2024-09-16T03:33:42.023 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 10/35 2024-09-16T03:33:42.023 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 11/35 2024-09-16T03:33:42.023 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-16T03:33:42.023 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 13/35 2024-09-16T03:33:42.023 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T03:33:42.023 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T03:33:42.024 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T03:33:42.024 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T03:33:42.024 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T03:33:42.024 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T03:33:42.024 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T03:33:42.024 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T03:33:42.024 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T03:33:42.024 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T03:33:42.024 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T03:33:42.024 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T03:33:42.025 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T03:33:42.025 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T03:33:42.025 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T03:33:42.026 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T03:33:42.026 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T03:33:42.026 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T03:33:42.026 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-16T03:33:42.026 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T03:33:42.026 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/35 2024-09-16T03:33:42.752 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): ceph-test-19.3.0-4930.gfc3e0bff.el9.x86_ 28 MB/s | 46 MB 00:01 2024-09-16T03:33:42.753 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:33:42.754 INFO:teuthology.orchestra.run.smithi064.stdout:Total 28 MB/s | 46 MB 00:01 2024-09-16T03:33:42.781 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:33:42.803 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:33:42.803 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:33:42.826 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T03:33:42.827 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:33:42.827 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-09-16T03:33:42.827 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.827 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.827 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:33:42.827 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T03:33:42.827 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.827 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.827 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.827 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.827 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T03:33:42.827 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T03:33:42.828 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T03:33:42.828 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.828 INFO:teuthology.orchestra.run.smithi155.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T03:33:42.828 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T03:33:42.828 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T03:33:42.828 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T03:33:42.828 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.828 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T03:33:42.828 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.828 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T03:33:42.829 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T03:33:42.829 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T03:33:42.829 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.829 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.829 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.829 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T03:33:42.829 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.829 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.829 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:42.829 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T03:33:42.829 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T03:33:42.829 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T03:33:42.829 INFO:teuthology.orchestra.run.smithi155.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T03:33:42.830 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T03:33:42.830 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:33:42.830 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:33:43.097 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T03:33:43.097 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/35 2024-09-16T03:33:43.097 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/35 2024-09-16T03:33:43.097 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/35 2024-09-16T03:33:43.097 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/35 2024-09-16T03:33:43.097 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/35 2024-09-16T03:33:43.097 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 6/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 8/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 9/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 10/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 11/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 13/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T03:33:43.098 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T03:33:43.099 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T03:33:43.099 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T03:33:43.099 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T03:33:43.099 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T03:33:43.099 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T03:33:43.099 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T03:33:43.099 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T03:33:43.099 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T03:33:43.099 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T03:33:43.099 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T03:33:43.099 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T03:33:43.099 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-16T03:33:43.099 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T03:33:43.100 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/35 2024-09-16T03:33:43.158 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-test 2024-09-16T03:33:43.187 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:33:43.187 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:33:43.723 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:33:43.731 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:31 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:33:43.805 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T03:33:43.843 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T03:33:43.870 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-16T03:33:43.870 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:33:43.871 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:33:43.871 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-16T03:33:43.871 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:33:43.871 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:33:43.871 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 46 M 2024-09-16T03:33:43.871 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-16T03:33:43.871 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-16T03:33:43.871 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T03:33:43.871 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T03:33:43.872 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:33:43.872 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:33:43.872 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:33:43.872 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2024-09-16T03:33:43.872 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:33:43.872 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 46 M 2024-09-16T03:33:43.873 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 201 M 2024-09-16T03:33:43.873 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:33:43.972 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-16T03:33:43.972 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2024-09-16T03:33:43.972 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2024-09-16T03:33:43.973 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2024-09-16T03:33:43.973 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2024-09-16T03:33:43.973 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2024-09-16T03:33:43.973 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /root 2024-09-16T03:33:43.973 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2024-09-16T03:33:43.973 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:33:44.000 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T03:33:44.001 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:33:44.001 INFO:teuthology.orchestra.run.smithi157.stdout:Upgraded: 2024-09-16T03:33:44.001 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.001 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.001 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:33:44.001 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T03:33:44.001 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.001 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.001 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.001 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.001 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T03:33:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T03:33:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T03:33:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T03:33:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T03:33:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T03:33:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T03:33:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T03:33:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T03:33:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T03:33:44.002 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T03:33:44.003 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.003 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.003 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.003 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T03:33:44.003 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.003 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.003 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:44.003 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T03:33:44.003 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T03:33:44.003 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T03:33:44.003 INFO:teuthology.orchestra.run.smithi157.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T03:33:44.003 INFO:teuthology.orchestra.run.smithi157.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T03:33:44.004 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:33:44.004 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:33:44.284 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.3 MB/s | 304 kB 00:00 2024-09-16T03:33:44.334 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-16T03:33:44.358 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-test 2024-09-16T03:33:44.384 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): libcephsqlite-19.3.0-4930.gfc3e0bff.el9. 700 kB/s | 161 kB 00:00 2024-09-16T03:33:44.936 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:27 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:33:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-16T03:33:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-16T03:33:44.992 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:33:44.993 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T03:33:44.993 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:33:44.993 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:33:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:33:44.995 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T03:33:45.071 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:33:45.072 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:33:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-16T03:33:45.072 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:33:45.072 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:33:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 46 M 2024-09-16T03:33:45.072 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-16T03:33:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-16T03:33:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T03:33:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T03:33:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:33:45.073 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:33:45.073 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:33:45.073 INFO:teuthology.orchestra.run.smithi157.stdout:Install 4 Packages 2024-09-16T03:33:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:33:45.073 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 46 M 2024-09-16T03:33:45.073 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 201 M 2024-09-16T03:33:45.073 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:33:45.085 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T03:33:45.086 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T03:33:45.476 INFO:teuthology.orchestra.run.smithi157.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.4 MB/s | 304 kB 00:00 2024-09-16T03:33:45.551 INFO:teuthology.orchestra.run.smithi157.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 850 kB/s | 64 kB 00:00 2024-09-16T03:33:45.576 INFO:teuthology.orchestra.run.smithi157.stdout:(3/4): libcephsqlite-19.3.0-4930.gfc3e0bff.el9. 724 kB/s | 161 kB 00:00 2024-09-16T03:33:45.577 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): ceph-test-19.3.0-4930.gfc3e0bff.el9.x86_ 32 MB/s | 46 MB 00:01 2024-09-16T03:33:45.579 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:33:45.579 INFO:teuthology.orchestra.run.smithi155.stdout:Total 27 MB/s | 46 MB 00:01 2024-09-16T03:33:45.607 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:33:45.627 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:33:45.627 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:33:46.005 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:33:46.005 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:33:46.329 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T03:33:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/35 2024-09-16T03:33:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/35 2024-09-16T03:33:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/35 2024-09-16T03:33:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/35 2024-09-16T03:33:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/35 2024-09-16T03:33:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 6/35 2024-09-16T03:33:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-16T03:33:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 8/35 2024-09-16T03:33:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 9/35 2024-09-16T03:33:46.330 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 10/35 2024-09-16T03:33:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 11/35 2024-09-16T03:33:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-16T03:33:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 13/35 2024-09-16T03:33:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T03:33:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T03:33:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T03:33:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T03:33:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T03:33:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T03:33:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T03:33:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T03:33:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T03:33:46.331 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T03:33:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T03:33:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T03:33:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T03:33:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T03:33:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T03:33:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T03:33:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T03:33:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T03:33:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-16T03:33:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T03:33:46.332 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/35 2024-09-16T03:33:46.526 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:33:46.608 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T03:33:46.651 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T03:33:46.678 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-16T03:33:46.759 INFO:teuthology.orchestra.run.smithi157.stdout:(4/4): ceph-test-19.3.0-4930.gfc3e0bff.el9.x86_ 32 MB/s | 46 MB 00:01 2024-09-16T03:33:46.762 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:33:46.762 INFO:teuthology.orchestra.run.smithi157.stdout:Total 27 MB/s | 46 MB 00:01 2024-09-16T03:33:46.791 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:33:46.811 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:33:46.812 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:33:47.190 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:33:47.190 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:33:47.374 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T03:33:47.375 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:33:47.375 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2024-09-16T03:33:47.375 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.375 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.375 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:33:47.375 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T03:33:47.375 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.375 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.375 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.375 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.375 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T03:33:47.376 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T03:33:47.376 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T03:33:47.376 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.376 INFO:teuthology.orchestra.run.smithi112.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T03:33:47.376 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T03:33:47.376 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T03:33:47.376 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T03:33:47.376 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.376 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T03:33:47.376 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.376 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T03:33:47.377 INFO:teuthology.orchestra.run.smithi112.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T03:33:47.378 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:33:47.378 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:33:47.723 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-test 2024-09-16T03:33:47.723 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:33:47.816 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T03:33:47.867 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T03:33:47.894 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-16T03:33:48.296 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:32 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:33:48.428 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:33:48.429 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:33:48.429 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-16T03:33:48.429 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:33:48.429 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:33:48.429 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 46 M 2024-09-16T03:33:48.429 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-16T03:33:48.429 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-16T03:33:48.429 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T03:33:48.430 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T03:33:48.430 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:33:48.430 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:33:48.430 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:33:48.430 INFO:teuthology.orchestra.run.smithi112.stdout:Install 4 Packages 2024-09-16T03:33:48.430 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:33:48.430 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 46 M 2024-09-16T03:33:48.431 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 201 M 2024-09-16T03:33:48.431 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:33:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-16T03:33:48.531 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-16T03:33:48.607 INFO:teuthology.orchestra.run.smithi112.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-09-16T03:33:48.649 INFO:teuthology.orchestra.run.smithi112.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T03:33:48.692 INFO:teuthology.orchestra.run.smithi112.stdout:(3/4): libcephsqlite-19.3.0-4930.gfc3e0bff.el9. 680 kB/s | 161 kB 00:00 2024-09-16T03:33:49.858 INFO:teuthology.orchestra.run.smithi112.stdout:(4/4): ceph-test-19.3.0-4930.gfc3e0bff.el9.x86_ 32 MB/s | 46 MB 00:01 2024-09-16T03:33:49.860 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:33:49.860 INFO:teuthology.orchestra.run.smithi112.stdout:Total 32 MB/s | 46 MB 00:01 2024-09-16T03:33:49.888 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:33:49.907 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:33:49.908 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:33:50.272 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:33:50.272 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:33:50.786 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:33:50.868 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T03:33:50.907 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T03:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-16T03:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/4 2024-09-16T03:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/4 2024-09-16T03:33:50.910 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T03:33:50.934 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-16T03:33:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-16T03:33:51.126 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-16T03:33:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T03:33:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:51.226 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:33:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T03:33:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T03:33:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:51.227 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:33:51.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph 2024-09-16T03:33:52.096 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:40 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:33:52.240 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:33:52.243 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:33:52.243 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-16T03:33:52.243 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:33:52.243 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:33:52.243 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 6.6 k 2024-09-16T03:33:52.244 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-16T03:33:52.244 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 2.4 M 2024-09-16T03:33:52.244 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 1.5 M 2024-09-16T03:33:52.244 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 291 k 2024-09-16T03:33:52.244 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 4.8 M 2024-09-16T03:33:52.244 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 17 M 2024-09-16T03:33:52.244 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T03:33:52.244 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T03:33:52.244 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T03:33:52.244 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T03:33:52.245 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T03:33:52.245 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T03:33:52.245 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T03:33:52.245 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T03:33:52.245 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T03:33:52.245 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T03:33:52.245 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections 2024-09-16T03:33:52.245 INFO:teuthology.orchestra.run.smithi064.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T03:33:52.246 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T03:33:52.246 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T03:33:52.246 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T03:33:52.246 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T03:33:52.246 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T03:33:52.246 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T03:33:52.246 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T03:33:52.246 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T03:33:52.246 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T03:33:52.246 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T03:33:52.246 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T03:33:52.246 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T03:33:52.246 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T03:33:52.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T03:33:52.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T03:33:52.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T03:33:52.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T03:33:52.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T03:33:52.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T03:33:52.247 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T03:33:52.247 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:52.247 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:33:52.247 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:33:52.247 INFO:teuthology.orchestra.run.smithi064.stdout:Install 37 Packages 2024-09-16T03:33:52.248 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:52.248 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 31 M 2024-09-16T03:33:52.248 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 109 M 2024-09-16T03:33:52.248 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:33:52.279 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-16T03:33:52.359 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-16T03:33:53.069 INFO:teuthology.orchestra.run.smithi064.stdout:(1/37): ceph-19.3.0-4930.gfc3e0bff.el9.x86_64.r 52 kB/s | 6.6 kB 00:00 2024-09-16T03:33:53.254 INFO:teuthology.orchestra.run.smithi064.stdout:(2/37): ceph-mgr-19.3.0-4930.gfc3e0bff.el9.x86_ 4.9 MB/s | 1.5 MB 00:00 2024-09-16T03:33:53.312 INFO:teuthology.orchestra.run.smithi064.stdout:(3/37): ceph-mds-19.3.0-4930.gfc3e0bff.el9.x86_ 6.5 MB/s | 2.4 MB 00:00 2024-09-16T03:33:53.405 INFO:teuthology.orchestra.run.smithi064.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4930.gfc3e 3.1 MB/s | 291 kB 00:00 2024-09-16T03:33:53.454 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-16T03:33:53.454 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/4 2024-09-16T03:33:53.454 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/4 2024-09-16T03:33:53.454 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T03:33:53.470 INFO:teuthology.orchestra.run.smithi064.stdout:(5/37): ceph-mon-19.3.0-4930.gfc3e0bff.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-16T03:33:53.671 INFO:teuthology.orchestra.run.smithi064.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 949 kB/s | 253 kB 00:00 2024-09-16T03:33:53.764 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T03:33:53.764 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:33:53.764 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:33:53.764 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:53.764 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:53.765 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T03:33:53.765 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T03:33:53.765 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:33:53.765 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:33:53.838 INFO:teuthology.orchestra.run.smithi064.stdout:(7/37): ceph-osd-19.3.0-4930.gfc3e0bff.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-16T03:33:53.863 INFO:teuthology.orchestra.run.smithi064.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 556 kB/s | 106 kB 00:00 2024-09-16T03:33:53.896 INFO:teuthology.orchestra.run.smithi064.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 2.9 MB/s | 1.2 MB 00:00 2024-09-16T03:33:53.922 INFO:teuthology.orchestra.run.smithi064.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-09-16T03:33:53.972 INFO:teuthology.orchestra.run.smithi064.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-09-16T03:33:54.031 INFO:teuthology.orchestra.run.smithi064.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-16T03:33:54.042 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph 2024-09-16T03:33:54.048 INFO:teuthology.orchestra.run.smithi064.stdout:(13/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.4 MB/s | 215 kB 00:00 2024-09-16T03:33:54.073 INFO:teuthology.orchestra.run.smithi064.stdout:(14/37): python3-pycparser-2.20-6.el9.noarch.rp 576 kB/s | 135 kB 00:00 2024-09-16T03:33:54.098 INFO:teuthology.orchestra.run.smithi064.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 518 kB/s | 35 kB 00:00 2024-09-16T03:33:54.123 INFO:teuthology.orchestra.run.smithi064.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-16T03:33:54.149 INFO:teuthology.orchestra.run.smithi064.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 767 kB/s | 77 kB 00:00 2024-09-16T03:33:54.182 INFO:teuthology.orchestra.run.smithi064.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 496 kB/s | 42 kB 00:00 2024-09-16T03:33:54.233 INFO:teuthology.orchestra.run.smithi064.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-09-16T03:33:54.258 INFO:teuthology.orchestra.run.smithi064.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-09-16T03:33:54.284 INFO:teuthology.orchestra.run.smithi064.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-16T03:33:54.317 INFO:teuthology.orchestra.run.smithi064.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-16T03:33:54.350 INFO:teuthology.orchestra.run.smithi064.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 181 kB/s | 11 kB 00:00 2024-09-16T03:33:54.376 INFO:teuthology.orchestra.run.smithi064.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 313 kB/s | 18 kB 00:00 2024-09-16T03:33:54.401 INFO:teuthology.orchestra.run.smithi064.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-09-16T03:33:54.430 INFO:teuthology.orchestra.run.smithi064.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 363 kB/s | 19 kB 00:00 2024-09-16T03:33:54.455 INFO:teuthology.orchestra.run.smithi064.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 369 kB/s | 20 kB 00:00 2024-09-16T03:33:54.480 INFO:teuthology.orchestra.run.smithi064.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-09-16T03:33:54.622 INFO:teuthology.orchestra.run.smithi064.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 473 kB/s | 79 kB 00:00 2024-09-16T03:33:54.627 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:42 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:33:54.639 INFO:teuthology.orchestra.run.smithi064.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 990 kB/s | 16 kB 00:00 2024-09-16T03:33:54.673 INFO:teuthology.orchestra.run.smithi064.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 865 kB/s | 358 kB 00:00 2024-09-16T03:33:54.687 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-16T03:33:54.687 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/4 2024-09-16T03:33:54.687 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/4 2024-09-16T03:33:54.687 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T03:33:54.698 INFO:teuthology.orchestra.run.smithi064.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-16T03:33:54.731 INFO:teuthology.orchestra.run.smithi064.stdout:(33/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-09-16T03:33:54.748 INFO:teuthology.orchestra.run.smithi064.stdout:(34/37): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-09-16T03:33:54.765 INFO:teuthology.orchestra.run.smithi064.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-16T03:33:54.775 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:33:54.778 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:33:54.778 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-16T03:33:54.778 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:33:54.778 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:33:54.778 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 6.6 k 2024-09-16T03:33:54.778 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-16T03:33:54.778 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 2.4 M 2024-09-16T03:33:54.778 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 1.5 M 2024-09-16T03:33:54.778 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 291 k 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 4.8 M 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 17 M 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections 2024-09-16T03:33:54.779 INFO:teuthology.orchestra.run.smithi155.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T03:33:54.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T03:33:54.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T03:33:54.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T03:33:54.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T03:33:54.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T03:33:54.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T03:33:54.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T03:33:54.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T03:33:54.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T03:33:54.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T03:33:54.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T03:33:54.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T03:33:54.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T03:33:54.781 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T03:33:54.781 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T03:33:54.781 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T03:33:54.781 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T03:33:54.781 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T03:33:54.781 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T03:33:54.781 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T03:33:54.781 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:33:54.781 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:33:54.781 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:33:54.782 INFO:teuthology.orchestra.run.smithi155.stdout:Install 37 Packages 2024-09-16T03:33:54.782 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:33:54.782 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 31 M 2024-09-16T03:33:54.782 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 109 M 2024-09-16T03:33:54.782 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:33:54.791 INFO:teuthology.orchestra.run.smithi064.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-16T03:33:54.816 INFO:teuthology.orchestra.run.smithi064.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-16T03:33:54.821 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:33:54.821 INFO:teuthology.orchestra.run.smithi064.stdout:Total 12 MB/s | 31 MB 00:02 2024-09-16T03:33:54.977 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:33:55.022 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:33:55.022 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:33:55.024 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T03:33:55.024 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:33:55.025 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:33:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T03:33:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T03:33:55.025 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:33:55.025 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:33:55.314 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-16T03:33:55.365 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph 2024-09-16T03:33:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-16T03:33:55.458 INFO:teuthology.orchestra.run.smithi155.stdout:(1/37): ceph-19.3.0-4930.gfc3e0bff.el9.x86_64.r 42 kB/s | 6.6 kB 00:00 2024-09-16T03:33:55.521 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:33:55.521 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:33:55.625 INFO:teuthology.orchestra.run.smithi155.stdout:(2/37): ceph-mds-19.3.0-4930.gfc3e0bff.el9.x86_ 7.3 MB/s | 2.4 MB 00:00 2024-09-16T03:33:55.667 INFO:teuthology.orchestra.run.smithi155.stdout:(3/37): ceph-mgr-19.3.0-4930.gfc3e0bff.el9.x86_ 4.2 MB/s | 1.5 MB 00:00 2024-09-16T03:33:55.775 INFO:teuthology.orchestra.run.smithi155.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4930.gfc3e 2.6 MB/s | 291 kB 00:00 2024-09-16T03:33:55.884 INFO:teuthology.orchestra.run.smithi155.stdout:(5/37): ceph-mon-19.3.0-4930.gfc3e0bff.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-16T03:33:55.941 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:38 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:33:56.010 INFO:teuthology.orchestra.run.smithi155.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-16T03:33:56.089 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:33:56.092 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:33:56.092 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-16T03:33:56.092 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:33:56.092 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:33:56.092 INFO:teuthology.orchestra.run.smithi157.stdout: ceph x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 6.6 k 2024-09-16T03:33:56.092 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-16T03:33:56.092 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 2.4 M 2024-09-16T03:33:56.092 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 1.5 M 2024-09-16T03:33:56.092 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 291 k 2024-09-16T03:33:56.092 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 4.8 M 2024-09-16T03:33:56.093 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 17 M 2024-09-16T03:33:56.093 INFO:teuthology.orchestra.run.smithi157.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T03:33:56.093 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T03:33:56.093 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T03:33:56.093 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T03:33:56.093 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T03:33:56.093 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T03:33:56.093 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T03:33:56.093 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T03:33:56.093 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T03:33:56.093 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T03:33:56.093 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections 2024-09-16T03:33:56.093 INFO:teuthology.orchestra.run.smithi157.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T03:33:56.094 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T03:33:56.095 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T03:33:56.095 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T03:33:56.095 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T03:33:56.095 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T03:33:56.095 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T03:33:56.095 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T03:33:56.095 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:33:56.095 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:33:56.095 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:33:56.095 INFO:teuthology.orchestra.run.smithi157.stdout:Install 37 Packages 2024-09-16T03:33:56.095 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:33:56.096 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 31 M 2024-09-16T03:33:56.096 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 109 M 2024-09-16T03:33:56.096 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:33:56.185 INFO:teuthology.orchestra.run.smithi155.stdout:(7/37): ceph-osd-19.3.0-4930.gfc3e0bff.el9.x86_ 31 MB/s | 17 MB 00:00 2024-09-16T03:33:56.210 INFO:teuthology.orchestra.run.smithi155.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 532 kB/s | 106 kB 00:00 2024-09-16T03:33:56.298 INFO:teuthology.orchestra.run.smithi155.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 3.0 MB/s | 1.2 MB 00:00 2024-09-16T03:33:56.319 INFO:teuthology.orchestra.run.smithi155.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-09-16T03:33:56.344 INFO:teuthology.orchestra.run.smithi155.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-16T03:33:56.370 INFO:teuthology.orchestra.run.smithi155.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 734 kB/s | 135 kB 00:00 2024-09-16T03:33:56.565 INFO:teuthology.orchestra.run.smithi157.stdout:(1/37): ceph-19.3.0-4930.gfc3e0bff.el9.x86_64.r 48 kB/s | 6.6 kB 00:00 2024-09-16T03:33:56.653 INFO:teuthology.orchestra.run.smithi155.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 123 kB/s | 35 kB 00:00 2024-09-16T03:33:56.715 INFO:teuthology.orchestra.run.smithi157.stdout:(2/37): ceph-mgr-19.3.0-4930.gfc3e0bff.el9.x86_ 5.3 MB/s | 1.5 MB 00:00 2024-09-16T03:33:56.782 INFO:teuthology.orchestra.run.smithi157.stdout:(3/37): ceph-mds-19.3.0-4930.gfc3e0bff.el9.x86_ 6.8 MB/s | 2.4 MB 00:00 2024-09-16T03:33:56.865 INFO:teuthology.orchestra.run.smithi157.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4930.gfc3e 3.4 MB/s | 291 kB 00:00 2024-09-16T03:33:56.867 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:33:56.870 INFO:teuthology.orchestra.run.smithi155.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 342 kB/s | 188 kB 00:00 2024-09-16T03:33:56.895 INFO:teuthology.orchestra.run.smithi155.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 313 kB/s | 172 kB 00:00 2024-09-16T03:33:56.927 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T03:33:56.937 INFO:teuthology.orchestra.run.smithi155.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 272 kB/s | 77 kB 00:00 2024-09-16T03:33:56.961 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T03:33:56.979 INFO:teuthology.orchestra.run.smithi155.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 468 kB/s | 51 kB 00:00 2024-09-16T03:33:56.982 INFO:teuthology.orchestra.run.smithi157.stdout:(5/37): ceph-mon-19.3.0-4930.gfc3e0bff.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-16T03:33:56.988 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T03:33:57.013 INFO:teuthology.orchestra.run.smithi155.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 355 kB/s | 42 kB 00:00 2024-09-16T03:33:57.046 INFO:teuthology.orchestra.run.smithi155.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-09-16T03:33:57.049 INFO:teuthology.orchestra.run.smithi157.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-16T03:33:57.065 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T03:33:57.072 INFO:teuthology.orchestra.run.smithi155.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-09-16T03:33:57.092 INFO:teuthology.orchestra.run.smithi157.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-16T03:33:57.104 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T03:33:57.105 INFO:teuthology.orchestra.run.smithi155.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 481 kB/s | 43 kB 00:00 2024-09-16T03:33:57.131 INFO:teuthology.orchestra.run.smithi155.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 463 kB/s | 11 kB 00:00 2024-09-16T03:33:57.133 INFO:teuthology.orchestra.run.smithi157.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-16T03:33:57.137 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T03:33:57.156 INFO:teuthology.orchestra.run.smithi155.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 710 kB/s | 18 kB 00:00 2024-09-16T03:33:57.181 INFO:teuthology.orchestra.run.smithi155.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-16T03:33:57.207 INFO:teuthology.orchestra.run.smithi155.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-09-16T03:33:57.232 INFO:teuthology.orchestra.run.smithi155.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 795 kB/s | 20 kB 00:00 2024-09-16T03:33:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T03:33:57.257 INFO:teuthology.orchestra.run.smithi155.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-16T03:33:57.282 INFO:teuthology.orchestra.run.smithi155.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-16T03:33:57.308 INFO:teuthology.orchestra.run.smithi155.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2024-09-16T03:33:57.308 INFO:teuthology.orchestra.run.smithi157.stdout:(9/37): ceph-osd-19.3.0-4930.gfc3e0bff.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-16T03:33:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T03:33:57.335 INFO:teuthology.orchestra.run.smithi155.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-16T03:33:57.342 INFO:teuthology.orchestra.run.smithi157.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 598 kB/s | 125 kB 00:00 2024-09-16T03:33:57.358 INFO:teuthology.orchestra.run.smithi155.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-16T03:33:57.365 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T03:33:57.375 INFO:teuthology.orchestra.run.smithi155.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-16T03:33:57.376 INFO:teuthology.orchestra.run.smithi157.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 3.1 MB/s | 1.2 MB 00:00 2024-09-16T03:33:57.400 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T03:33:57.400 INFO:teuthology.orchestra.run.smithi155.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 855 kB/s | 36 kB 00:00 2024-09-16T03:33:57.426 INFO:teuthology.orchestra.run.smithi155.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-16T03:33:57.429 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T03:33:57.451 INFO:teuthology.orchestra.run.smithi155.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 795 kB/s | 20 kB 00:00 2024-09-16T03:33:57.464 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T03:33:57.467 INFO:teuthology.orchestra.run.smithi157.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.3 MB/s | 215 kB 00:00 2024-09-16T03:33:57.476 INFO:teuthology.orchestra.run.smithi155.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-16T03:33:57.501 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T03:33:57.518 INFO:teuthology.orchestra.run.smithi155.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-16T03:33:57.523 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:33:57.524 INFO:teuthology.orchestra.run.smithi155.stdout:Total 11 MB/s | 31 MB 00:02 2024-09-16T03:33:57.547 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T03:33:57.559 INFO:teuthology.orchestra.run.smithi157.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 938 kB/s | 172 kB 00:00 2024-09-16T03:33:57.585 INFO:teuthology.orchestra.run.smithi157.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 777 kB/s | 188 kB 00:00 2024-09-16T03:33:57.590 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T03:33:57.610 INFO:teuthology.orchestra.run.smithi157.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 244 kB/s | 35 kB 00:00 2024-09-16T03:33:57.624 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T03:33:57.635 INFO:teuthology.orchestra.run.smithi157.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-16T03:33:57.661 INFO:teuthology.orchestra.run.smithi157.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 671 kB/s | 51 kB 00:00 2024-09-16T03:33:57.673 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T03:33:57.677 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:33:57.697 INFO:teuthology.orchestra.run.smithi157.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 379 kB/s | 22 kB 00:00 2024-09-16T03:33:57.713 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T03:33:57.717 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:33:57.717 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:33:57.720 INFO:teuthology.orchestra.run.smithi157.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 381 kB/s | 42 kB 00:00 2024-09-16T03:33:57.734 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-16T03:33:57.734 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/4 2024-09-16T03:33:57.734 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/4 2024-09-16T03:33:57.735 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T03:33:57.762 INFO:teuthology.orchestra.run.smithi157.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-09-16T03:33:57.787 INFO:teuthology.orchestra.run.smithi157.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 482 kB/s | 43 kB 00:00 2024-09-16T03:33:57.789 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T03:33:57.812 INFO:teuthology.orchestra.run.smithi157.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 428 kB/s | 11 kB 00:00 2024-09-16T03:33:57.838 INFO:teuthology.orchestra.run.smithi157.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 714 kB/s | 18 kB 00:00 2024-09-16T03:33:57.863 INFO:teuthology.orchestra.run.smithi157.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-16T03:33:57.888 INFO:teuthology.orchestra.run.smithi157.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-16T03:33:57.913 INFO:teuthology.orchestra.run.smithi157.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-16T03:33:57.939 INFO:teuthology.orchestra.run.smithi157.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-16T03:33:57.940 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T03:33:57.964 INFO:teuthology.orchestra.run.smithi157.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-16T03:33:57.989 INFO:teuthology.orchestra.run.smithi157.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-09-16T03:33:58.014 INFO:teuthology.orchestra.run.smithi157.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-16T03:33:58.024 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T03:33:58.040 INFO:teuthology.orchestra.run.smithi157.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-16T03:33:58.065 INFO:teuthology.orchestra.run.smithi157.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-16T03:33:58.066 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T03:33:58.067 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:33:58.067 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:33:58.067 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:58.067 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:33:58.067 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T03:33:58.067 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T03:33:58.067 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:33:58.067 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:33:58.085 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T03:33:58.090 INFO:teuthology.orchestra.run.smithi157.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-16T03:33:58.122 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T03:33:58.141 INFO:teuthology.orchestra.run.smithi157.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-09-16T03:33:58.167 INFO:teuthology.orchestra.run.smithi157.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 807 kB/s | 20 kB 00:00 2024-09-16T03:33:58.189 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T03:33:58.191 INFO:teuthology.orchestra.run.smithi157.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-09-16T03:33:58.217 INFO:teuthology.orchestra.run.smithi157.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-09-16T03:33:58.222 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:33:58.222 INFO:teuthology.orchestra.run.smithi157.stdout:Total 14 MB/s | 31 MB 00:02 2024-09-16T03:33:58.228 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:33:58.228 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:33:58.253 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T03:33:58.378 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:33:58.387 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph 2024-09-16T03:33:58.415 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:33:58.415 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:33:58.448 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T03:33:58.501 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T03:33:58.583 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T03:33:58.680 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T03:33:58.722 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T03:33:58.862 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T03:33:58.906 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:33:58.906 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:33:58.967 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:42 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:33:59.049 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 32/37 2024-09-16T03:33:59.077 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-16T03:33:59.105 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-16T03:33:59.105 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:33:59.105 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T03:33:59.105 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:33:59.105 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:33:59.106 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:33:59.113 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:33:59.116 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:33:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-16T03:33:59.116 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:33:59.116 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:33:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 6.6 k 2024-09-16T03:33:59.116 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-16T03:33:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 2.4 M 2024-09-16T03:33:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 1.5 M 2024-09-16T03:33:59.116 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 291 k 2024-09-16T03:33:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 4.8 M 2024-09-16T03:33:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 17 M 2024-09-16T03:33:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T03:33:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T03:33:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T03:33:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T03:33:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T03:33:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T03:33:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T03:33:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T03:33:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T03:33:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T03:33:59.117 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections 2024-09-16T03:33:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T03:33:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T03:33:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T03:33:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T03:33:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T03:33:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T03:33:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T03:33:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T03:33:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T03:33:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T03:33:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T03:33:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T03:33:59.118 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T03:33:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T03:33:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T03:33:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T03:33:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T03:33:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T03:33:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T03:33:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T03:33:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T03:33:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:33:59.119 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:33:59.119 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:33:59.119 INFO:teuthology.orchestra.run.smithi112.stdout:Install 37 Packages 2024-09-16T03:33:59.119 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:33:59.120 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 31 M 2024-09-16T03:33:59.120 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 109 M 2024-09-16T03:33:59.120 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:33:59.492 INFO:teuthology.orchestra.run.smithi112.stdout:(1/37): ceph-19.3.0-4930.gfc3e0bff.el9.x86_64.r 45 kB/s | 6.6 kB 00:00 2024-09-16T03:33:59.602 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:33:59.650 INFO:teuthology.orchestra.run.smithi112.stdout:(2/37): ceph-mgr-19.3.0-4930.gfc3e0bff.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-16T03:33:59.680 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T03:33:59.712 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T03:33:59.717 INFO:teuthology.orchestra.run.smithi112.stdout:(3/37): ceph-mds-19.3.0-4930.gfc3e0bff.el9.x86_ 6.4 MB/s | 2.4 MB 00:00 2024-09-16T03:33:59.740 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T03:33:59.812 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T03:33:59.817 INFO:teuthology.orchestra.run.smithi112.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4930.gfc3e 2.8 MB/s | 291 kB 00:00 2024-09-16T03:33:59.881 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T03:33:59.893 INFO:teuthology.orchestra.run.smithi112.stdout:(5/37): ceph-mon-19.3.0-4930.gfc3e0bff.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-16T03:33:59.926 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T03:34:00.009 INFO:teuthology.orchestra.run.smithi112.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-16T03:34:00.018 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T03:34:00.068 INFO:teuthology.orchestra.run.smithi112.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 7.0 MB/s | 1.2 MB 00:00 2024-09-16T03:34:00.093 INFO:teuthology.orchestra.run.smithi112.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2024-09-16T03:34:00.096 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T03:34:00.141 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T03:34:00.181 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T03:34:00.210 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T03:34:00.245 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T03:34:00.247 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:34:00.269 INFO:teuthology.orchestra.run.smithi112.stdout:(9/37): ceph-osd-19.3.0-4930.gfc3e0bff.el9.x86_ 28 MB/s | 17 MB 00:00 2024-09-16T03:34:00.278 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T03:34:00.294 INFO:teuthology.orchestra.run.smithi112.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 622 kB/s | 125 kB 00:00 2024-09-16T03:34:00.302 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T03:34:00.319 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T03:34:00.319 INFO:teuthology.orchestra.run.smithi112.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 539 kB/s | 135 kB 00:00 2024-09-16T03:34:00.335 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T03:34:00.354 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T03:34:00.371 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T03:34:00.389 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T03:34:00.428 INFO:teuthology.orchestra.run.smithi112.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.3 MB/s | 215 kB 00:00 2024-09-16T03:34:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T03:34:00.443 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T03:34:00.470 INFO:teuthology.orchestra.run.smithi112.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-16T03:34:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T03:34:00.487 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T03:34:00.495 INFO:teuthology.orchestra.run.smithi112.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 939 kB/s | 188 kB 00:00 2024-09-16T03:34:00.520 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T03:34:00.520 INFO:teuthology.orchestra.run.smithi112.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-16T03:34:00.546 INFO:teuthology.orchestra.run.smithi112.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-16T03:34:00.559 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T03:34:00.571 INFO:teuthology.orchestra.run.smithi112.stdout:(17/37): python3-markupsafe-1.1.1-12.el9.x86_64 244 kB/s | 35 kB 00:00 2024-09-16T03:34:00.596 INFO:teuthology.orchestra.run.smithi112.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-09-16T03:34:00.612 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T03:34:00.655 INFO:teuthology.orchestra.run.smithi112.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 745 kB/s | 43 kB 00:00 2024-09-16T03:34:00.661 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-16T03:34:00.680 INFO:teuthology.orchestra.run.smithi112.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-09-16T03:34:00.691 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T03:34:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-16T03:34:00.691 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:00.691 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T03:34:00.692 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:34:00.692 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:34:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:00.697 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T03:34:00.705 INFO:teuthology.orchestra.run.smithi112.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-09-16T03:34:00.730 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T03:34:00.739 INFO:teuthology.orchestra.run.smithi112.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 411 kB/s | 11 kB 00:00 2024-09-16T03:34:00.764 INFO:teuthology.orchestra.run.smithi112.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-16T03:34:00.766 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T03:34:00.777 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T03:34:00.790 INFO:teuthology.orchestra.run.smithi112.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 363 kB/s | 18 kB 00:00 2024-09-16T03:34:00.804 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T03:34:00.815 INFO:teuthology.orchestra.run.smithi112.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-09-16T03:34:00.830 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T03:34:00.840 INFO:teuthology.orchestra.run.smithi112.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-16T03:34:00.847 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T03:34:00.865 INFO:teuthology.orchestra.run.smithi112.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-16T03:34:00.869 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T03:34:00.880 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T03:34:00.890 INFO:teuthology.orchestra.run.smithi112.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-09-16T03:34:00.916 INFO:teuthology.orchestra.run.smithi112.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T03:34:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T03:34:00.929 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T03:34:00.941 INFO:teuthology.orchestra.run.smithi112.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-16T03:34:00.964 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T03:34:00.966 INFO:teuthology.orchestra.run.smithi112.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-16T03:34:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T03:34:00.992 INFO:teuthology.orchestra.run.smithi112.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-16T03:34:00.999 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T03:34:01.017 INFO:teuthology.orchestra.run.smithi112.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-09-16T03:34:01.045 INFO:teuthology.orchestra.run.smithi112.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-16T03:34:01.047 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T03:34:01.071 INFO:teuthology.orchestra.run.smithi112.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 797 kB/s | 20 kB 00:00 2024-09-16T03:34:01.095 INFO:teuthology.orchestra.run.smithi112.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-16T03:34:01.120 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T03:34:01.121 INFO:teuthology.orchestra.run.smithi112.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.0 MB/s | 427 kB 00:00 2024-09-16T03:34:01.124 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:01.124 INFO:teuthology.orchestra.run.smithi112.stdout:Total 15 MB/s | 31 MB 00:02 2024-09-16T03:34:01.161 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-16T03:34:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T03:34:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-16T03:34:01.187 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:01.187 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T03:34:01.187 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:34:01.187 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:34:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:01.195 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T03:34:01.215 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T03:34:01.270 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:34:01.295 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T03:34:01.310 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:34:01.310 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:34:01.332 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T03:34:01.390 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T03:34:01.413 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T03:34:01.420 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-16T03:34:01.433 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T03:34:01.445 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-16T03:34:01.445 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:01.446 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T03:34:01.446 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:34:01.446 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:34:01.446 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:01.465 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T03:34:01.505 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T03:34:01.512 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-16T03:34:01.562 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T03:34:01.571 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T03:34:01.616 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T03:34:01.734 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 32/37 2024-09-16T03:34:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-16T03:34:01.778 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:34:01.779 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:34:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-16T03:34:01.786 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:01.786 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T03:34:01.786 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:34:01.786 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:34:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:01.821 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T03:34:01.875 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T03:34:01.970 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T03:34:02.069 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T03:34:02.113 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T03:34:02.245 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T03:34:02.415 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 32/37 2024-09-16T03:34:02.443 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-16T03:34:02.470 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-16T03:34:02.470 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:02.470 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T03:34:02.470 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:34:02.470 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:34:02.470 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:03.167 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:34:03.215 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T03:34:03.248 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T03:34:03.268 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-16T03:34:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-16T03:34:03.279 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/37 2024-09-16T03:34:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/37 2024-09-16T03:34:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/37 2024-09-16T03:34:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/37 2024-09-16T03:34:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/37 2024-09-16T03:34:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 6/37 2024-09-16T03:34:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T03:34:03.280 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T03:34:03.281 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T03:34:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T03:34:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T03:34:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T03:34:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T03:34:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T03:34:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T03:34:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T03:34:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T03:34:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T03:34:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T03:34:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T03:34:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T03:34:03.282 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T03:34:03.283 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T03:34:03.283 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T03:34:03.295 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-16T03:34:03.295 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:03.295 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T03:34:03.295 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:34:03.295 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:34:03.295 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:03.365 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T03:34:03.425 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T03:34:03.458 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T03:34:03.551 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T03:34:03.637 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T03:34:03.677 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T03:34:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T03:34:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T03:34:03.744 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-16T03:34:03.768 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-16T03:34:03.768 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:03.768 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T03:34:03.768 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:34:03.768 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:34:03.768 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:03.776 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T03:34:03.810 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T03:34:03.859 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T03:34:03.910 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T03:34:03.945 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T03:34:03.977 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-16T03:34:03.993 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T03:34:03.994 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-16T03:34:04.006 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-16T03:34:04.006 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:04.006 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T03:34:04.006 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:34:04.006 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:34:04.006 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:04.018 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-16T03:34:04.018 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:04.018 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T03:34:04.018 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:34:04.019 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:34:04.019 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:04.025 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T03:34:04.083 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-16T03:34:04.107 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T03:34:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T03:34:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:04.208 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:34:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T03:34:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T03:34:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T03:34:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T03:34:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T03:34:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T03:34:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T03:34:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T03:34:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T03:34:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T03:34:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T03:34:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T03:34:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T03:34:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T03:34:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T03:34:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T03:34:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T03:34:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T03:34:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T03:34:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T03:34:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T03:34:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T03:34:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T03:34:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T03:34:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:04.211 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T03:34:04.333 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T03:34:04.386 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T03:34:04.425 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T03:34:04.461 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-16T03:34:04.482 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T03:34:04.485 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-16T03:34:04.485 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:04.485 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T03:34:04.485 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:34:04.485 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:34:04.485 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:04.545 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T03:34:04.595 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-base 2024-09-16T03:34:04.720 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-16T03:34:04.734 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T03:34:04.744 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-16T03:34:04.744 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:04.744 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T03:34:04.744 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:34:04.744 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:34:04.744 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:04.786 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T03:34:04.811 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-16T03:34:04.864 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T03:34:04.952 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T03:34:05.001 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T03:34:05.130 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T03:34:05.189 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:05.282 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:05.295 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 32/37 2024-09-16T03:34:05.320 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-16T03:34:05.328 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:05.329 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-16T03:34:05.329 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:05.346 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-16T03:34:05.347 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:05.347 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T03:34:05.347 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:34:05.347 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T03:34:05.347 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:05.400 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install cephadm 2024-09-16T03:34:05.895 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-16T03:34:05.895 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/37 2024-09-16T03:34:05.895 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/37 2024-09-16T03:34:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/37 2024-09-16T03:34:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/37 2024-09-16T03:34:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/37 2024-09-16T03:34:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 6/37 2024-09-16T03:34:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T03:34:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T03:34:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T03:34:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T03:34:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T03:34:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T03:34:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T03:34:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T03:34:05.896 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T03:34:05.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T03:34:05.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T03:34:05.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T03:34:05.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T03:34:05.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T03:34:05.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T03:34:05.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T03:34:05.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T03:34:05.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T03:34:05.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T03:34:05.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T03:34:05.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T03:34:05.897 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T03:34:05.901 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T03:34:05.901 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T03:34:05.901 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T03:34:05.901 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T03:34:05.901 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T03:34:05.901 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T03:34:05.901 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T03:34:05.901 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T03:34:05.984 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:06.122 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:06.135 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:06.135 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-16T03:34:06.135 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:06.135 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:34:06.135 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 784 k 2024-09-16T03:34:06.135 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:06.135 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:34:06.136 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:06.136 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-16T03:34:06.136 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:06.136 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 784 k 2024-09-16T03:34:06.136 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 791 k 2024-09-16T03:34:06.136 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:34:06.412 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm-19.3.0-4930.gfc3e0bff.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-16T03:34:06.413 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:06.413 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.6 MB/s | 784 kB 00:00 2024-09-16T03:34:06.413 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:34:06.418 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:34:06.418 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:34:06.425 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:34:06.425 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:34:06.499 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:34:06.618 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-16T03:34:06.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/37 2024-09-16T03:34:06.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/37 2024-09-16T03:34:06.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/37 2024-09-16T03:34:06.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/37 2024-09-16T03:34:06.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/37 2024-09-16T03:34:06.618 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 6/37 2024-09-16T03:34:06.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T03:34:06.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T03:34:06.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T03:34:06.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T03:34:06.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T03:34:06.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T03:34:06.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T03:34:06.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T03:34:06.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T03:34:06.620 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T03:34:06.621 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T03:34:06.621 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T03:34:06.621 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T03:34:06.621 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T03:34:06.621 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T03:34:06.621 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T03:34:06.621 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T03:34:06.621 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T03:34:06.621 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T03:34:06.621 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T03:34:06.622 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T03:34:06.622 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T03:34:06.622 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T03:34:06.622 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T03:34:06.622 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T03:34:06.622 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T03:34:06.622 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T03:34:06.622 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T03:34:06.622 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T03:34:06.622 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T03:34:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-16T03:34:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-16T03:34:06.841 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:06.841 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T03:34:06.841 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:34:06.841 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T03:34:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T03:34:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:06.877 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:34:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T03:34:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T03:34:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T03:34:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T03:34:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T03:34:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T03:34:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T03:34:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T03:34:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T03:34:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T03:34:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T03:34:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T03:34:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T03:34:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T03:34:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T03:34:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T03:34:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T03:34:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T03:34:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:06.879 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:07.069 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-16T03:34:07.301 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base 2024-09-16T03:34:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-16T03:34:07.312 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:07.312 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T03:34:07.312 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:34:07.312 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T03:34:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:07.428 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:07.539 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-16T03:34:07.563 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-16T03:34:07.563 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:07.563 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T03:34:07.563 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:34:07.563 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T03:34:07.563 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:07.626 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-16T03:34:07.644 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:07.645 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:34:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:07.645 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:07.645 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:07.708 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T03:34:07.708 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:07.708 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:34:07.708 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:07.708 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:07.709 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:07.709 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:07.709 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:07.709 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:07.709 INFO:teuthology.orchestra.run.smithi157.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T03:34:07.709 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T03:34:07.709 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T03:34:07.709 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T03:34:07.709 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T03:34:07.709 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T03:34:07.709 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T03:34:07.710 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T03:34:07.710 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T03:34:07.710 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T03:34:07.710 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T03:34:07.710 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T03:34:07.710 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T03:34:07.710 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T03:34:07.710 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T03:34:07.710 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T03:34:07.711 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T03:34:07.711 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T03:34:07.711 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T03:34:07.711 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T03:34:07.711 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T03:34:07.711 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T03:34:07.711 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T03:34:07.711 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T03:34:07.711 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T03:34:07.711 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T03:34:07.711 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T03:34:07.712 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T03:34:07.712 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T03:34:07.712 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T03:34:07.712 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T03:34:07.712 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:07.712 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:07.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T03:34:07.865 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:07.956 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:07.998 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:07.999 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-16T03:34:07.999 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:08.081 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install cephadm 2024-09-16T03:34:08.094 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-base 2024-09-16T03:34:08.442 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:08.581 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:08.582 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:08.582 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-16T03:34:08.582 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:08.582 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:34:08.582 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 145 k 2024-09-16T03:34:08.582 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:08.582 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:34:08.583 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:08.583 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-16T03:34:08.583 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:08.583 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 145 k 2024-09-16T03:34:08.583 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 432 k 2024-09-16T03:34:08.583 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:34:08.638 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:08.669 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:51 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:08.760 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:08.769 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:08.770 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:08.770 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-16T03:34:08.770 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:08.770 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:34:08.770 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 784 k 2024-09-16T03:34:08.771 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:08.771 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:34:08.771 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:08.771 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-16T03:34:08.771 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:08.771 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 784 k 2024-09-16T03:34:08.771 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 791 k 2024-09-16T03:34:08.771 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:34:08.784 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-immutable-object-cache-19.3.0-4930.gfc3e0b 720 kB/s | 145 kB 00:00 2024-09-16T03:34:08.785 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:08.785 INFO:teuthology.orchestra.run.smithi064.stdout:Total 716 kB/s | 145 kB 00:00 2024-09-16T03:34:08.785 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:34:08.793 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:34:08.793 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:34:08.801 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:08.802 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-16T03:34:08.802 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:08.853 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:34:08.854 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:34:08.871 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install cephadm 2024-09-16T03:34:09.007 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:34:09.057 INFO:teuthology.orchestra.run.smithi155.stdout:cephadm-19.3.0-4930.gfc3e0bff.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-16T03:34:09.058 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:09.058 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-16T03:34:09.058 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:34:09.060 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-16T03:34:09.062 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:34:09.062 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:34:09.068 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:34:09.069 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:34:09.086 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-16T03:34:09.086 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:09.086 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T03:34:09.087 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:09.142 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:34:09.360 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-16T03:34:09.360 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/37 2024-09-16T03:34:09.360 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/37 2024-09-16T03:34:09.360 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/37 2024-09-16T03:34:09.360 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/37 2024-09-16T03:34:09.360 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/37 2024-09-16T03:34:09.360 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 6/37 2024-09-16T03:34:09.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T03:34:09.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T03:34:09.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T03:34:09.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T03:34:09.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T03:34:09.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T03:34:09.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T03:34:09.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T03:34:09.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T03:34:09.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T03:34:09.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T03:34:09.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T03:34:09.361 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T03:34:09.362 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T03:34:09.362 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T03:34:09.362 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T03:34:09.362 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T03:34:09.362 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T03:34:09.362 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T03:34:09.363 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T03:34:09.363 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T03:34:09.363 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T03:34:09.363 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T03:34:09.363 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T03:34:09.364 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T03:34:09.364 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T03:34:09.364 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T03:34:09.364 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T03:34:09.364 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T03:34:09.364 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T03:34:09.432 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:09.560 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:09.564 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:09.565 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:09.565 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-16T03:34:09.565 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:09.565 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:34:09.565 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 784 k 2024-09-16T03:34:09.565 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:09.565 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:34:09.565 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:09.565 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-16T03:34:09.565 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:09.566 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 784 k 2024-09-16T03:34:09.566 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 791 k 2024-09-16T03:34:09.566 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:34:09.677 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:09.846 INFO:teuthology.orchestra.run.smithi157.stdout:cephadm-19.3.0-4930.gfc3e0bff.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-16T03:34:09.847 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:09.847 INFO:teuthology.orchestra.run.smithi157.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-16T03:34:09.847 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:34:09.852 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:34:09.853 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:34:09.859 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:34:09.859 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:34:09.956 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:34:10.012 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-16T03:34:10.013 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:10.013 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:34:10.013 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:10.013 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:10.013 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:10.057 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:10.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr 2024-09-16T03:34:10.265 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T03:34:10.265 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:10.265 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:34:10.265 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:10.266 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:10.266 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:10.266 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:10.266 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:10.266 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:10.266 INFO:teuthology.orchestra.run.smithi112.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T03:34:10.266 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T03:34:10.266 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T03:34:10.266 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T03:34:10.266 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T03:34:10.266 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T03:34:10.266 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T03:34:10.266 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T03:34:10.267 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T03:34:10.268 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T03:34:10.268 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T03:34:10.268 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T03:34:10.268 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T03:34:10.268 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T03:34:10.268 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T03:34:10.268 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T03:34:10.268 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T03:34:10.268 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T03:34:10.268 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:10.268 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:34:10.306 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:10.306 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:10.306 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:34:10.307 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:10.307 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:10.307 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:10.425 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:10.517 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:10.546 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T03:34:10.644 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-base 2024-09-16T03:34:10.817 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:10.902 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:10.911 INFO:teuthology.orchestra.run.smithi064.stdout:Package ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:10.956 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:10.957 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-16T03:34:10.958 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:11.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T03:34:11.127 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:11.176 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:11.176 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:11.177 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:34:11.177 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:11.177 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:11.177 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:11.216 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:34:11.264 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:11.265 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:11.265 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-09-16T03:34:11.266 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:11.266 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:34:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 145 k 2024-09-16T03:34:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:11.266 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:34:11.266 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:11.266 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-16T03:34:11.266 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:11.266 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 145 k 2024-09-16T03:34:11.266 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 432 k 2024-09-16T03:34:11.266 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:34:11.307 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:11.348 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:34:11.349 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-16T03:34:11.349 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:34:11.397 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T03:34:11.406 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install cephadm 2024-09-16T03:34:11.497 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-immutable-object-cache-19.3.0-4930.gfc3e0b 629 kB/s | 145 kB 00:00 2024-09-16T03:34:11.497 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:11.497 INFO:teuthology.orchestra.run.smithi155.stdout:Total 625 kB/s | 145 kB 00:00 2024-09-16T03:34:11.498 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:34:11.505 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:34:11.505 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:34:11.563 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:34:11.563 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:34:11.621 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:11.711 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:34:11.761 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:11.763 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:11.763 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-16T03:34:11.763 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:11.763 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:34:11.763 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 14 M 2024-09-16T03:34:11.763 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-16T03:34:11.764 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T03:34:11.764 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 34 k 2024-09-16T03:34:11.764 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 16 k 2024-09-16T03:34:11.764 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T03:34:11.764 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T03:34:11.765 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T03:34:11.765 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T03:34:11.765 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T03:34:11.765 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T03:34:11.765 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T03:34:11.765 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T03:34:11.766 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:11.766 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:34:11.766 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:11.766 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-09-16T03:34:11.766 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:11.767 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-16T03:34:11.768 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 19 M 2024-09-16T03:34:11.768 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 159 M 2024-09-16T03:34:11.768 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:34:11.789 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-16T03:34:11.789 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:11.789 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T03:34:11.789 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:11.961 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:34:11.964 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:54 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:11.987 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4930.gfc 234 kB/s | 34 kB 00:00 2024-09-16T03:34:12.013 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4930.gfc3 96 kB/s | 16 kB 00:00 2024-09-16T03:34:12.093 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:34:12.095 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:12.095 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-16T03:34:12.095 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:12.095 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:34:12.095 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 784 k 2024-09-16T03:34:12.095 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:12.096 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:34:12.096 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:12.096 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-16T03:34:12.096 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:12.096 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 784 k 2024-09-16T03:34:12.097 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 791 k 2024-09-16T03:34:12.097 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:34:12.104 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:12.105 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:12.105 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-09-16T03:34:12.105 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:12.105 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:34:12.106 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 145 k 2024-09-16T03:34:12.106 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:12.106 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:34:12.106 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:12.106 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-16T03:34:12.107 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:12.107 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 145 k 2024-09-16T03:34:12.107 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 432 k 2024-09-16T03:34:12.107 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:34:12.238 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-16T03:34:12.386 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-immutable-object-cache-19.3.0-4930.gfc3e0b 517 kB/s | 145 kB 00:00 2024-09-16T03:34:12.387 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:12.387 INFO:teuthology.orchestra.run.smithi157.stdout:Total 514 kB/s | 145 kB 00:00 2024-09-16T03:34:12.387 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:34:12.396 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:34:12.396 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:34:12.454 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:34:12.454 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:34:12.522 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): ceph-mgr-dashboard-19.3.0-4930.gfc3e0bf 21 MB/s | 14 MB 00:00 2024-09-16T03:34:12.557 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.8 MB/s | 1.0 MB 00:00 2024-09-16T03:34:12.563 INFO:teuthology.orchestra.run.smithi112.stdout:cephadm-19.3.0-4930.gfc3e0bff.el9.noarch.rpm 1.6 MB/s | 784 kB 00:00 2024-09-16T03:34:12.564 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:12.565 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.6 MB/s | 784 kB 00:00 2024-09-16T03:34:12.565 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:34:12.570 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:34:12.570 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:34:12.576 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:34:12.577 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:34:12.596 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:34:12.605 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.3 MB/s | 863 kB 00:00 2024-09-16T03:34:12.631 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 285 kB/s | 19 kB 00:00 2024-09-16T03:34:12.648 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:34:12.670 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-16T03:34:12.672 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.6 MB/s | 548 kB 00:00 2024-09-16T03:34:12.695 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-16T03:34:12.696 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:12.696 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T03:34:12.696 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:12.699 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.2 MB/s | 144 kB 00:00 2024-09-16T03:34:12.715 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 736 kB/s | 31 kB 00:00 2024-09-16T03:34:12.736 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-16T03:34:12.736 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:12.736 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:34:12.736 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:12.736 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:12.736 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:12.748 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-16T03:34:12.815 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.8 MB/s | 2.0 MB 00:00 2024-09-16T03:34:12.819 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:12.819 INFO:teuthology.orchestra.run.smithi064.stdout:Total 18 MB/s | 19 MB 00:01 2024-09-16T03:34:12.868 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:34:12.888 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:34:12.888 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:34:13.028 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr 2024-09-16T03:34:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:34:13.175 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:34:13.191 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:13.551 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:13.610 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:01 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:13.667 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-16T03:34:13.667 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:13.667 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:34:13.667 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:13.667 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:13.668 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:13.702 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:13.745 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:13.746 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-16T03:34:13.746 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:13.778 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:34:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:13.799 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:34:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:13.799 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:13.799 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:34:13.822 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T03:34:13.850 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T03:34:13.863 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr 2024-09-16T03:34:13.889 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T03:34:13.934 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T03:34:13.964 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T03:34:13.990 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T03:34:14.054 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T03:34:14.188 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T03:34:14.277 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T03:34:14.327 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T03:34:14.355 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:34:14.386 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:02 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:14.387 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 10/12 2024-09-16T03:34:14.444 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:14.522 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:14.523 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:14.523 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-16T03:34:14.523 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:14.523 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:34:14.523 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 14 M 2024-09-16T03:34:14.523 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-16T03:34:14.524 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T03:34:14.524 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 34 k 2024-09-16T03:34:14.524 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 16 k 2024-09-16T03:34:14.524 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T03:34:14.524 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T03:34:14.524 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T03:34:14.524 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T03:34:14.524 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T03:34:14.524 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T03:34:14.524 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T03:34:14.524 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T03:34:14.524 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:14.524 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:34:14.525 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:14.525 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-09-16T03:34:14.525 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:14.525 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 19 M 2024-09-16T03:34:14.525 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 159 M 2024-09-16T03:34:14.525 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:34:14.540 INFO:teuthology.orchestra.run.smithi157.stdout:Package ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:14.558 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:34:14.582 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:14.583 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-16T03:34:14.583 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:14.647 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T03:34:14.690 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:34:14.692 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:14.692 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-09-16T03:34:14.692 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:14.692 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:34:14.692 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 145 k 2024-09-16T03:34:14.692 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:14.692 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:34:14.693 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:14.693 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-16T03:34:14.693 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:14.693 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 145 k 2024-09-16T03:34:14.693 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 432 k 2024-09-16T03:34:14.693 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:34:14.902 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-immutable-object-cache-19.3.0-4930.gfc3e0b 693 kB/s | 145 kB 00:00 2024-09-16T03:34:14.902 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:14.903 INFO:teuthology.orchestra.run.smithi112.stdout:Total 688 kB/s | 145 kB 00:00 2024-09-16T03:34:14.903 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:34:14.915 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:34:14.915 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:34:14.966 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:34:14.966 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:34:15.108 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:34:15.179 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-16T03:34:15.192 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4930.gfc3 107 kB/s | 16 kB 00:00 2024-09-16T03:34:15.203 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-16T03:34:15.203 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:15.203 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T03:34:15.203 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:15.210 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:15.217 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4930.gfc 191 kB/s | 34 kB 00:00 2024-09-16T03:34:15.345 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:15.346 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:15.346 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-16T03:34:15.346 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:15.346 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:34:15.346 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 14 M 2024-09-16T03:34:15.346 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-16T03:34:15.346 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T03:34:15.346 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 34 k 2024-09-16T03:34:15.346 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 16 k 2024-09-16T03:34:15.346 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T03:34:15.346 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T03:34:15.347 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T03:34:15.347 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T03:34:15.347 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T03:34:15.347 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T03:34:15.347 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T03:34:15.347 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T03:34:15.347 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:15.347 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:34:15.347 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:15.347 INFO:teuthology.orchestra.run.smithi157.stdout:Install 12 Packages 2024-09-16T03:34:15.347 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:15.348 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 19 M 2024-09-16T03:34:15.348 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 159 M 2024-09-16T03:34:15.348 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:34:15.410 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-09-16T03:34:15.652 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): ceph-mgr-dashboard-19.3.0-4930.gfc3e0bf 23 MB/s | 14 MB 00:00 2024-09-16T03:34:15.710 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-16T03:34:15.752 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.8 MB/s | 1.0 MB 00:00 2024-09-16T03:34:15.777 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 290 kB/s | 19 kB 00:00 2024-09-16T03:34:15.819 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.1 MB/s | 863 kB 00:00 2024-09-16T03:34:15.853 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-09-16T03:34:15.887 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.0 MB/s | 188 kB 00:00 2024-09-16T03:34:15.907 INFO:teuthology.orchestra.run.smithi157.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4930.gfc 221 kB/s | 34 kB 00:00 2024-09-16T03:34:15.920 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 311 kB/s | 31 kB 00:00 2024-09-16T03:34:15.931 INFO:teuthology.orchestra.run.smithi157.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4930.gfc3 91 kB/s | 16 kB 00:00 2024-09-16T03:34:15.970 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.4 MB/s | 2.0 MB 00:00 2024-09-16T03:34:15.975 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:15.975 INFO:teuthology.orchestra.run.smithi155.stdout:Total 13 MB/s | 19 MB 00:01 2024-09-16T03:34:16.023 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:34:16.042 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:34:16.042 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:34:16.065 INFO:teuthology.orchestra.run.smithi157.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 269 kB 00:00 2024-09-16T03:34:16.116 INFO:teuthology.orchestra.run.smithi157.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-09-16T03:34:16.137 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-16T03:34:16.137 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:16.138 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:34:16.138 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:16.138 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:16.138 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:34:16.166 INFO:teuthology.orchestra.run.smithi157.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-16T03:34:16.191 INFO:teuthology.orchestra.run.smithi157.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 777 kB/s | 19 kB 00:00 2024-09-16T03:34:16.267 INFO:teuthology.orchestra.run.smithi157.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 27 MB/s | 2.0 MB 00:00 2024-09-16T03:34:16.292 INFO:teuthology.orchestra.run.smithi157.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-16T03:34:16.317 INFO:teuthology.orchestra.run.smithi157.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T03:34:16.323 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:34:16.324 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:34:16.344 INFO:teuthology.orchestra.run.smithi157.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-16T03:34:16.356 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr 2024-09-16T03:34:16.493 INFO:teuthology.orchestra.run.smithi157.stdout:(11/12): ceph-mgr-dashboard-19.3.0-4930.gfc3e0b 19 MB/s | 14 MB 00:00 2024-09-16T03:34:16.518 INFO:teuthology.orchestra.run.smithi157.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.9 MB/s | 863 kB 00:00 2024-09-16T03:34:16.521 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:16.521 INFO:teuthology.orchestra.run.smithi157.stdout:Total 16 MB/s | 19 MB 00:01 2024-09-16T03:34:16.570 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:34:16.588 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:34:16.588 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:34:16.860 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:34:16.860 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:34:16.936 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:00 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:34:16.942 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:34:17.027 INFO:teuthology.orchestra.run.smithi112.stdout:Package ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:17.030 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T03:34:17.069 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:34:17.070 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-16T03:34:17.070 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:34:17.073 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T03:34:17.120 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T03:34:17.131 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T03:34:17.150 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T03:34:17.237 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T03:34:17.361 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T03:34:17.463 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T03:34:17.464 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:34:17.514 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T03:34:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T03:34:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T03:34:17.593 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:34:17.625 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 10/12 2024-09-16T03:34:17.634 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T03:34:17.672 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T03:34:17.699 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:01 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:34:17.743 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 11/12 2024-09-16T03:34:17.759 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T03:34:17.816 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-16T03:34:17.832 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:34:17.833 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:17.833 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-16T03:34:17.833 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:17.833 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:34:17.833 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 14 M 2024-09-16T03:34:17.833 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-16T03:34:17.834 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T03:34:17.834 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 34 k 2024-09-16T03:34:17.834 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 16 k 2024-09-16T03:34:17.834 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T03:34:17.834 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T03:34:17.834 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T03:34:17.834 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T03:34:17.834 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T03:34:17.834 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T03:34:17.834 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T03:34:17.834 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T03:34:17.834 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:17.834 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:34:17.835 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:17.835 INFO:teuthology.orchestra.run.smithi112.stdout:Install 12 Packages 2024-09-16T03:34:17.835 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:17.835 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 19 M 2024-09-16T03:34:17.835 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 159 M 2024-09-16T03:34:17.835 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:34:17.882 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T03:34:17.984 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T03:34:18.035 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T03:34:18.064 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:34:18.096 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 10/12 2024-09-16T03:34:18.323 INFO:teuthology.orchestra.run.smithi112.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4930.gfc 247 kB/s | 34 kB 00:00 2024-09-16T03:34:18.349 INFO:teuthology.orchestra.run.smithi112.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4930.gfc3 100 kB/s | 16 kB 00:00 2024-09-16T03:34:18.549 INFO:teuthology.orchestra.run.smithi112.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-16T03:34:18.641 INFO:teuthology.orchestra.run.smithi112.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-09-16T03:34:18.816 INFO:teuthology.orchestra.run.smithi112.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4930.gfc3e0bf 22 MB/s | 14 MB 00:00 2024-09-16T03:34:18.858 INFO:teuthology.orchestra.run.smithi112.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.5 MB/s | 548 kB 00:00 2024-09-16T03:34:18.883 INFO:teuthology.orchestra.run.smithi112.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 292 kB/s | 19 kB 00:00 2024-09-16T03:34:18.917 INFO:teuthology.orchestra.run.smithi112.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-16T03:34:18.934 INFO:teuthology.orchestra.run.smithi112.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-09-16T03:34:18.976 INFO:teuthology.orchestra.run.smithi112.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-16T03:34:19.034 INFO:teuthology.orchestra.run.smithi112.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 12 MB/s | 2.0 MB 00:00 2024-09-16T03:34:19.076 INFO:teuthology.orchestra.run.smithi112.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.6 MB/s | 863 kB 00:00 2024-09-16T03:34:19.080 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:19.080 INFO:teuthology.orchestra.run.smithi112.stdout:Total 15 MB/s | 19 MB 00:01 2024-09-16T03:34:19.128 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:34:19.147 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:34:19.147 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:34:19.408 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:34:19.409 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:34:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-16T03:34:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 1/12 2024-09-16T03:34:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 2/12 2024-09-16T03:34:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 3/12 2024-09-16T03:34:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T03:34:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T03:34:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T03:34:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T03:34:19.716 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T03:34:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:34:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T03:34:19.717 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T03:34:19.994 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:34:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T03:34:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:20.037 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:34:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T03:34:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:20.037 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T03:34:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T03:34:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T03:34:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T03:34:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T03:34:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T03:34:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T03:34:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T03:34:20.038 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:20.038 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:20.060 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T03:34:20.095 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T03:34:20.141 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T03:34:20.172 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T03:34:20.254 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T03:34:20.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T03:34:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T03:34:20.482 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T03:34:20.526 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T03:34:20.555 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:34:20.595 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 10/12 2024-09-16T03:34:20.818 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 11/12 2024-09-16T03:34:20.869 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:08 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:20.879 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-16T03:34:21.009 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:21.010 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-09-16T03:34:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-16T03:34:21.010 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-09-16T03:34:21.010 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:34:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 7.4 M 2024-09-16T03:34:21.010 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-16T03:34:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T03:34:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T03:34:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T03:34:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T03:34:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T03:34:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T03:34:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T03:34:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T03:34:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T03:34:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T03:34:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T03:34:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:21.011 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:34:21.011 INFO:teuthology.orchestra.run.smithi064.stdout:======================================================================================= 2024-09-16T03:34:21.012 INFO:teuthology.orchestra.run.smithi064.stdout:Install 12 Packages 2024-09-16T03:34:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:21.012 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 38 M 2024-09-16T03:34:21.012 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 212 M 2024-09-16T03:34:21.012 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:34:21.266 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 11/12 2024-09-16T03:34:21.325 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-16T03:34:21.457 INFO:teuthology.orchestra.run.smithi064.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-16T03:34:21.499 INFO:teuthology.orchestra.run.smithi064.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.5 MB/s | 801 kB 00:00 2024-09-16T03:34:21.516 INFO:teuthology.orchestra.run.smithi064.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-09-16T03:34:21.549 INFO:teuthology.orchestra.run.smithi064.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 546 kB/s | 18 kB 00:00 2024-09-16T03:34:21.591 INFO:teuthology.orchestra.run.smithi064.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 921 kB/s | 38 kB 00:00 2024-09-16T03:34:21.658 INFO:teuthology.orchestra.run.smithi064.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 19 MB/s | 3.0 MB 00:00 2024-09-16T03:34:21.709 INFO:teuthology.orchestra.run.smithi064.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-09-16T03:34:21.809 INFO:teuthology.orchestra.run.smithi064.stdout:(8/12): ceph-mgr-diskprediction-local-19.3.0-49 14 MB/s | 7.4 MB 00:00 2024-09-16T03:34:21.884 INFO:teuthology.orchestra.run.smithi064.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 17 MB/s | 4.9 MB 00:00 2024-09-16T03:34:21.959 INFO:teuthology.orchestra.run.smithi064.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-09-16T03:34:22.060 INFO:teuthology.orchestra.run.smithi064.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.0 MB/s | 250 kB 00:00 2024-09-16T03:34:22.260 INFO:teuthology.orchestra.run.smithi064.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-09-16T03:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:22.261 INFO:teuthology.orchestra.run.smithi064.stdout:Total 31 MB/s | 38 MB 00:01 2024-09-16T03:34:22.628 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:34:22.655 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:34:22.655 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:34:22.898 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-16T03:34:22.898 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 1/12 2024-09-16T03:34:22.899 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 2/12 2024-09-16T03:34:22.899 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 3/12 2024-09-16T03:34:22.899 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T03:34:22.899 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T03:34:22.899 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T03:34:22.899 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T03:34:22.899 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T03:34:22.899 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:34:22.899 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T03:34:22.899 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T03:34:23.106 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:34:23.106 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:34:23.212 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T03:34:23.212 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:23.212 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:34:23.212 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T03:34:23.212 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:23.212 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:23.213 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:23.213 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T03:34:23.213 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T03:34:23.213 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T03:34:23.213 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T03:34:23.213 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T03:34:23.213 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T03:34:23.213 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T03:34:23.213 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T03:34:23.213 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:23.213 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:23.252 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-16T03:34:23.252 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 1/12 2024-09-16T03:34:23.252 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 2/12 2024-09-16T03:34:23.252 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 3/12 2024-09-16T03:34:23.252 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T03:34:23.252 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T03:34:23.252 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T03:34:23.252 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T03:34:23.252 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T03:34:23.253 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:34:23.253 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T03:34:23.253 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T03:34:23.510 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T03:34:23.581 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T03:34:23.582 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:23.582 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:34:23.582 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T03:34:23.582 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:23.582 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:23.582 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:23.582 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T03:34:23.582 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T03:34:23.582 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T03:34:23.582 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T03:34:23.582 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T03:34:23.582 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T03:34:23.583 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T03:34:23.583 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T03:34:23.583 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:23.583 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:23.755 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 11/12 2024-09-16T03:34:23.816 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-16T03:34:23.827 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T03:34:23.902 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:34:23.980 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T03:34:24.025 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:34:24.078 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:12 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:24.131 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:34:24.168 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T03:34:24.213 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:24.214 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-09-16T03:34:24.214 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-16T03:34:24.214 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-09-16T03:34:24.214 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:34:24.214 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 7.4 M 2024-09-16T03:34:24.214 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-16T03:34:24.214 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T03:34:24.215 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T03:34:24.215 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T03:34:24.215 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T03:34:24.215 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T03:34:24.215 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T03:34:24.215 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T03:34:24.215 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T03:34:24.215 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T03:34:24.215 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T03:34:24.215 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T03:34:24.215 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:24.215 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:34:24.215 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-09-16T03:34:24.216 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-09-16T03:34:24.216 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:24.216 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 38 M 2024-09-16T03:34:24.216 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 212 M 2024-09-16T03:34:24.216 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:34:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T03:34:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T03:34:24.395 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:07 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T03:34:24.530 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:24.531 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================= 2024-09-16T03:34:24.532 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-16T03:34:24.532 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================= 2024-09-16T03:34:24.532 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:34:24.532 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 7.4 M 2024-09-16T03:34:24.532 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-16T03:34:24.532 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T03:34:24.532 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T03:34:24.532 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T03:34:24.532 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T03:34:24.532 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T03:34:24.532 INFO:teuthology.orchestra.run.smithi157.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T03:34:24.532 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T03:34:24.533 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T03:34:24.533 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T03:34:24.533 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T03:34:24.533 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T03:34:24.533 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:24.533 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:34:24.533 INFO:teuthology.orchestra.run.smithi157.stdout:======================================================================================= 2024-09-16T03:34:24.533 INFO:teuthology.orchestra.run.smithi157.stdout:Install 12 Packages 2024-09-16T03:34:24.533 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:24.534 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 38 M 2024-09-16T03:34:24.534 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 212 M 2024-09-16T03:34:24.534 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:34:24.912 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-16T03:34:25.021 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-09-16T03:34:25.057 INFO:teuthology.orchestra.run.smithi157.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-16T03:34:25.079 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.2 MB/s | 801 kB 00:00 2024-09-16T03:34:25.091 INFO:teuthology.orchestra.run.smithi157.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.3 MB/s | 801 kB 00:00 2024-09-16T03:34:25.164 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T03:34:25.196 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-16T03:34:25.222 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 128 kB/s | 18 kB 00:00 2024-09-16T03:34:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T03:34:25.297 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 384 kB/s | 38 kB 00:00 2024-09-16T03:34:25.364 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.7 MB/s | 3.0 MB 00:00 2024-09-16T03:34:25.383 INFO:teuthology.orchestra.run.smithi157.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-16T03:34:25.489 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 5.1 MB 00:00 2024-09-16T03:34:25.515 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-09-16T03:34:25.540 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 4.9 MB/s | 250 kB 00:00 2024-09-16T03:34:25.632 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 12 MB/s | 4.9 MB 00:00 2024-09-16T03:34:25.641 INFO:teuthology.orchestra.run.smithi157.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 56 kB/s | 33 kB 00:00 2024-09-16T03:34:25.792 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-16T03:34:25.792 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 1/12 2024-09-16T03:34:25.792 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 2/12 2024-09-16T03:34:25.792 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 3/12 2024-09-16T03:34:25.792 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T03:34:25.793 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T03:34:25.793 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T03:34:25.793 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T03:34:25.793 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T03:34:25.793 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T03:34:25.793 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T03:34:25.793 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T03:34:25.867 INFO:teuthology.orchestra.run.smithi157.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 38 kB/s | 18 kB 00:00 2024-09-16T03:34:25.915 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-09-16T03:34:25.916 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:25.916 INFO:teuthology.orchestra.run.smithi155.stdout:Total 22 MB/s | 38 MB 00:01 2024-09-16T03:34:26.111 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T03:34:26.112 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:26.112 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:34:26.112 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T03:34:26.112 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:26.112 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:26.112 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:26.112 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T03:34:26.112 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T03:34:26.112 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T03:34:26.112 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T03:34:26.112 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T03:34:26.112 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T03:34:26.113 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T03:34:26.113 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T03:34:26.113 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:26.113 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:34:26.242 INFO:teuthology.orchestra.run.smithi157.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-09-16T03:34:26.267 INFO:teuthology.orchestra.run.smithi157.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 62 kB/s | 38 kB 00:00 2024-09-16T03:34:26.272 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:34:26.299 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:34:26.299 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:34:26.360 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T03:34:26.569 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:34:26.741 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:34:26.741 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:34:26.924 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:10 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:34:26.977 INFO:teuthology.orchestra.run.smithi157.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.5 MB/s | 4.9 MB 00:01 2024-09-16T03:34:27.002 INFO:teuthology.orchestra.run.smithi157.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 323 kB/s | 245 kB 00:00 2024-09-16T03:34:27.055 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:34:27.056 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-09-16T03:34:27.057 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-16T03:34:27.057 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-09-16T03:34:27.057 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:34:27.057 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 7.4 M 2024-09-16T03:34:27.057 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-16T03:34:27.057 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T03:34:27.057 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T03:34:27.057 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T03:34:27.057 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T03:34:27.057 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T03:34:27.057 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T03:34:27.057 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T03:34:27.058 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T03:34:27.058 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T03:34:27.058 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T03:34:27.058 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T03:34:27.058 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:27.058 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:34:27.058 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-09-16T03:34:27.058 INFO:teuthology.orchestra.run.smithi112.stdout:Install 12 Packages 2024-09-16T03:34:27.058 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:27.059 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 38 M 2024-09-16T03:34:27.059 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 212 M 2024-09-16T03:34:27.059 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:34:27.427 INFO:teuthology.orchestra.run.smithi157.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.4 MB/s | 5.1 MB 00:01 2024-09-16T03:34:27.505 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:34:27.564 INFO:teuthology.orchestra.run.smithi112.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-16T03:34:27.582 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T03:34:27.622 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:34:27.661 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T03:34:27.691 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-16T03:34:27.706 INFO:teuthology.orchestra.run.smithi112.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 231 kB/s | 33 kB 00:00 2024-09-16T03:34:27.722 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:34:27.740 INFO:teuthology.orchestra.run.smithi112.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.2 MB/s | 801 kB 00:00 2024-09-16T03:34:27.752 INFO:teuthology.orchestra.run.smithi157.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 323 kB/s | 250 kB 00:00 2024-09-16T03:34:27.765 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T03:34:27.849 INFO:teuthology.orchestra.run.smithi112.stdout:(4/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-16T03:34:27.874 INFO:teuthology.orchestra.run.smithi112.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 136 kB/s | 18 kB 00:00 2024-09-16T03:34:27.948 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T03:34:27.957 INFO:teuthology.orchestra.run.smithi112.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 354 kB/s | 38 kB 00:00 2024-09-16T03:34:27.980 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T03:34:28.056 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T03:34:28.091 INFO:teuthology.orchestra.run.smithi112.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-09-16T03:34:28.477 INFO:teuthology.orchestra.run.smithi157.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-16T03:34:28.479 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:28.479 INFO:teuthology.orchestra.run.smithi157.stdout:Total 9.7 MB/s | 38 MB 00:03 2024-09-16T03:34:28.715 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T03:34:28.775 INFO:teuthology.orchestra.run.smithi112.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.8 MB/s | 3.0 MB 00:01 2024-09-16T03:34:28.836 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:34:28.836 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T03:34:28.862 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:34:28.862 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:34:28.883 INFO:teuthology.orchestra.run.smithi112.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-09-16T03:34:29.033 INFO:teuthology.orchestra.run.smithi112.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.3 MB/s | 4.9 MB 00:01 2024-09-16T03:34:29.302 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:34:29.302 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:34:29.375 INFO:teuthology.orchestra.run.smithi112.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.0 MB/s | 5.1 MB 00:01 2024-09-16T03:34:30.063 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:34:30.068 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:34:30.149 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T03:34:30.201 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:34:30.232 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-16T03:34:30.232 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 1/12 2024-09-16T03:34:30.232 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:34:30.232 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:34:30.232 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T03:34:30.232 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T03:34:30.232 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T03:34:30.233 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T03:34:30.233 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T03:34:30.233 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T03:34:30.233 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:34:30.233 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T03:34:30.301 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:34:30.335 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T03:34:30.522 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T03:34:30.559 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T03:34:30.637 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T03:34:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T03:34:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:30.679 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:34:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T03:34:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T03:34:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T03:34:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T03:34:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T03:34:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T03:34:30.679 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T03:34:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T03:34:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T03:34:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T03:34:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T03:34:30.680 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:30.680 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:31.075 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-rook 2024-09-16T03:34:31.125 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T03:34:31.156 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-16T03:34:31.298 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T03:34:31.415 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T03:34:31.665 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:19 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:31.807 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:31.808 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:31.808 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-16T03:34:31.808 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:31.808 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:34:31.808 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 49 k 2024-09-16T03:34:31.808 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:31.809 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:34:31.810 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:31.810 INFO:teuthology.orchestra.run.smithi064.stdout:Install 13 Packages 2024-09-16T03:34:31.810 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:31.810 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 2.2 M 2024-09-16T03:34:31.810 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 26 M 2024-09-16T03:34:31.810 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:34:31.817 INFO:teuthology.orchestra.run.smithi112.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.5 MB/s | 16 MB 00:02 2024-09-16T03:34:31.822 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:31.822 INFO:teuthology.orchestra.run.smithi112.stdout:Total 8.0 MB/s | 38 MB 00:04 2024-09-16T03:34:32.189 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:34:32.206 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:34:32.206 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:34:32.433 INFO:teuthology.orchestra.run.smithi064.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 205 kB/s | 26 kB 00:00 2024-09-16T03:34:32.450 INFO:teuthology.orchestra.run.smithi064.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 135 kB/s | 19 kB 00:00 2024-09-16T03:34:32.475 INFO:teuthology.orchestra.run.smithi064.stdout:(3/13): ceph-mgr-rook-19.3.0-4930.gfc3e0bff.el9 295 kB/s | 49 kB 00:00 2024-09-16T03:34:32.592 INFO:teuthology.orchestra.run.smithi064.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-16T03:34:32.618 INFO:teuthology.orchestra.run.smithi064.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-09-16T03:34:32.642 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:34:32.642 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:34:32.651 INFO:teuthology.orchestra.run.smithi064.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 918 kB/s | 54 kB 00:00 2024-09-16T03:34:32.668 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:34:32.684 INFO:teuthology.orchestra.run.smithi064.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-16T03:34:32.719 INFO:teuthology.orchestra.run.smithi064.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-16T03:34:32.735 INFO:teuthology.orchestra.run.smithi064.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 171 kB/s | 14 kB 00:00 2024-09-16T03:34:32.769 INFO:teuthology.orchestra.run.smithi064.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-16T03:34:32.786 INFO:teuthology.orchestra.run.smithi064.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.3 MB/s | 59 kB 00:00 2024-09-16T03:34:32.811 INFO:teuthology.orchestra.run.smithi064.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 2.2 MB/s | 90 kB 00:00 2024-09-16T03:34:32.886 INFO:teuthology.orchestra.run.smithi064.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-09-16T03:34:32.889 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:32.889 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.1 MB/s | 2.2 MB 00:01 2024-09-16T03:34:32.953 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:34:32.970 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:34:32.970 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:34:33.067 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:34:33.067 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:34:33.451 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:34:33.517 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T03:34:33.557 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:34:33.578 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:34:33.657 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:34:33.699 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T03:34:33.703 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T03:34:33.733 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-16T03:34:33.749 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T03:34:33.799 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-16T03:34:33.799 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 1/12 2024-09-16T03:34:33.799 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:34:33.799 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:34:33.799 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T03:34:33.799 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T03:34:33.799 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T03:34:33.799 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T03:34:33.800 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T03:34:33.800 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T03:34:33.800 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:34:33.800 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T03:34:33.814 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T03:34:33.870 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T03:34:33.886 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T03:34:33.905 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T03:34:33.915 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T03:34:33.947 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T03:34:33.992 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T03:34:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T03:34:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T03:34:34.218 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T03:34:34.218 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:34.218 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:34:34.218 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:34.218 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T03:34:34.218 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T03:34:34.218 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T03:34:34.218 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T03:34:34.218 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T03:34:34.219 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T03:34:34.219 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T03:34:34.219 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T03:34:34.219 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T03:34:34.219 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T03:34:34.219 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T03:34:34.219 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:34.219 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T03:34:34.635 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-rook 2024-09-16T03:34:34.663 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T03:34:34.787 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T03:34:34.992 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T03:34:35.037 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T03:34:35.071 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T03:34:35.128 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T03:34:35.157 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-16T03:34:35.215 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:23 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:35.350 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:35.351 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-16T03:34:35.352 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:35.352 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:34:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 49 k 2024-09-16T03:34:35.352 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-16T03:34:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T03:34:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T03:34:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T03:34:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T03:34:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T03:34:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T03:34:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T03:34:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T03:34:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T03:34:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T03:34:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T03:34:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T03:34:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:35.353 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:34:35.353 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:35.353 INFO:teuthology.orchestra.run.smithi155.stdout:Install 13 Packages 2024-09-16T03:34:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:35.354 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 2.2 M 2024-09-16T03:34:35.354 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 26 M 2024-09-16T03:34:35.354 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:34:35.601 INFO:teuthology.orchestra.run.smithi155.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 316 kB/s | 26 kB 00:00 2024-09-16T03:34:35.624 INFO:teuthology.orchestra.run.smithi155.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 180 kB/s | 19 kB 00:00 2024-09-16T03:34:35.683 INFO:teuthology.orchestra.run.smithi155.stdout:(3/13): ceph-mgr-rook-19.3.0-4930.gfc3e0bff.el9 299 kB/s | 49 kB 00:00 2024-09-16T03:34:35.742 INFO:teuthology.orchestra.run.smithi155.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-16T03:34:35.767 INFO:teuthology.orchestra.run.smithi155.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-09-16T03:34:35.793 INFO:teuthology.orchestra.run.smithi155.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-16T03:34:35.834 INFO:teuthology.orchestra.run.smithi155.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-16T03:34:35.860 INFO:teuthology.orchestra.run.smithi155.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 235 kB/s | 14 kB 00:00 2024-09-16T03:34:35.893 INFO:teuthology.orchestra.run.smithi155.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.8 MB/s | 220 kB 00:00 2024-09-16T03:34:35.927 INFO:teuthology.orchestra.run.smithi155.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-16T03:34:35.938 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-16T03:34:35.938 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/13 2024-09-16T03:34:35.938 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T03:34:35.938 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T03:34:35.938 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T03:34:35.938 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T03:34:35.938 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T03:34:35.939 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T03:34:35.939 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T03:34:35.939 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T03:34:35.939 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T03:34:35.939 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T03:34:35.939 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T03:34:35.952 INFO:teuthology.orchestra.run.smithi155.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.0 MB/s | 279 kB 00:00 2024-09-16T03:34:35.978 INFO:teuthology.orchestra.run.smithi155.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-16T03:34:36.013 INFO:teuthology.orchestra.run.smithi155.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-16T03:34:36.013 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:36.013 INFO:teuthology.orchestra.run.smithi155.stdout:Total 3.4 MB/s | 2.2 MB 00:00 2024-09-16T03:34:36.021 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:34:36.076 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:34:36.091 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:34:36.091 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:34:36.179 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:34:36.179 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:34:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T03:34:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:36.276 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:34:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T03:34:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T03:34:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T03:34:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T03:34:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T03:34:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T03:34:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T03:34:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T03:34:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T03:34:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T03:34:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T03:34:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T03:34:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:36.277 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:36.327 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-16T03:34:36.327 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 1/12 2024-09-16T03:34:36.327 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:34:36.327 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:34:36.327 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T03:34:36.327 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T03:34:36.327 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T03:34:36.328 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T03:34:36.328 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T03:34:36.328 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T03:34:36.328 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:34:36.328 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T03:34:36.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T03:34:36.695 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:34:36.712 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T03:34:36.712 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:36.712 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:34:36.712 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:36.712 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T03:34:36.712 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T03:34:36.713 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T03:34:36.713 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T03:34:36.713 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T03:34:36.713 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T03:34:36.713 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T03:34:36.713 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T03:34:36.713 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T03:34:36.713 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T03:34:36.713 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T03:34:36.713 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:36.713 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:36.866 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T03:34:36.912 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T03:34:36.967 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T03:34:37.005 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T03:34:37.040 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T03:34:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T03:34:37.072 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-16T03:34:37.085 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-rook 2024-09-16T03:34:37.114 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:37.148 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T03:34:37.249 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T03:34:37.254 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:37.255 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:37.255 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-16T03:34:37.255 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:37.255 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:34:37.255 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 154 k 2024-09-16T03:34:37.255 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-16T03:34:37.256 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T03:34:37.256 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T03:34:37.256 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T03:34:37.256 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T03:34:37.256 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T03:34:37.256 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:37.256 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:34:37.256 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:37.256 INFO:teuthology.orchestra.run.smithi064.stdout:Install 6 Packages 2024-09-16T03:34:37.256 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:37.257 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 7.0 M 2024-09-16T03:34:37.257 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 33 M 2024-09-16T03:34:37.257 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:34:37.298 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T03:34:37.671 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:20 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:37.788 INFO:teuthology.orchestra.run.smithi064.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-16T03:34:37.805 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:37.806 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:37.806 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-16T03:34:37.806 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:37.806 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:34:37.806 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 49 k 2024-09-16T03:34:37.807 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-16T03:34:37.807 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T03:34:37.807 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T03:34:37.807 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T03:34:37.807 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T03:34:37.807 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T03:34:37.807 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T03:34:37.807 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T03:34:37.807 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T03:34:37.807 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T03:34:37.807 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T03:34:37.807 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T03:34:37.807 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T03:34:37.808 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:37.808 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:34:37.808 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:37.808 INFO:teuthology.orchestra.run.smithi157.stdout:Install 13 Packages 2024-09-16T03:34:37.808 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:37.808 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 2.2 M 2024-09-16T03:34:37.808 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 26 M 2024-09-16T03:34:37.809 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:34:37.813 INFO:teuthology.orchestra.run.smithi064.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4930.gfc3e0bff.e 699 kB/s | 154 kB 00:00 2024-09-16T03:34:37.872 INFO:teuthology.orchestra.run.smithi064.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 990 kB/s | 58 kB 00:00 2024-09-16T03:34:37.905 INFO:teuthology.orchestra.run.smithi064.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-16T03:34:37.939 INFO:teuthology.orchestra.run.smithi064.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.6 MB/s | 548 kB 00:00 2024-09-16T03:34:38.011 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T03:34:38.054 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T03:34:38.058 INFO:teuthology.orchestra.run.smithi157.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 349 kB/s | 26 kB 00:00 2024-09-16T03:34:38.083 INFO:teuthology.orchestra.run.smithi157.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 194 kB/s | 19 kB 00:00 2024-09-16T03:34:38.088 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T03:34:38.136 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T03:34:38.158 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-16T03:34:38.158 INFO:teuthology.orchestra.run.smithi157.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 222 kB 00:00 2024-09-16T03:34:38.184 INFO:teuthology.orchestra.run.smithi157.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 159 kB 00:00 2024-09-16T03:34:38.209 INFO:teuthology.orchestra.run.smithi157.stdout:(5/13): ceph-mgr-rook-19.3.0-4930.gfc3e0bff.el9 219 kB/s | 49 kB 00:00 2024-09-16T03:34:38.235 INFO:teuthology.orchestra.run.smithi157.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-16T03:34:38.259 INFO:teuthology.orchestra.run.smithi157.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-16T03:34:38.310 INFO:teuthology.orchestra.run.smithi157.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-16T03:34:38.335 INFO:teuthology.orchestra.run.smithi157.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 141 kB/s | 14 kB 00:00 2024-09-16T03:34:38.360 INFO:teuthology.orchestra.run.smithi157.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-16T03:34:38.386 INFO:teuthology.orchestra.run.smithi157.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-16T03:34:38.411 INFO:teuthology.orchestra.run.smithi157.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-16T03:34:38.453 INFO:teuthology.orchestra.run.smithi157.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-16T03:34:38.455 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:38.455 INFO:teuthology.orchestra.run.smithi157.stdout:Total 3.4 MB/s | 2.2 MB 00:00 2024-09-16T03:34:38.520 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:34:38.535 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:34:38.535 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:34:38.625 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:34:38.626 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:34:38.943 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-16T03:34:38.943 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/13 2024-09-16T03:34:38.943 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T03:34:38.943 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T03:34:38.944 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T03:34:38.944 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T03:34:38.944 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T03:34:38.944 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T03:34:38.944 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T03:34:38.944 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T03:34:38.944 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T03:34:38.944 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T03:34:38.944 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T03:34:39.022 INFO:teuthology.orchestra.run.smithi064.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.2 MB/s | 6.0 MB 00:01 2024-09-16T03:34:39.025 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:39.025 INFO:teuthology.orchestra.run.smithi064.stdout:Total 4.0 MB/s | 7.0 MB 00:01 2024-09-16T03:34:39.102 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:34:39.115 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:34:39.115 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:34:39.139 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:34:39.222 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:34:39.222 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:34:39.269 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T03:34:39.269 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:39.269 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:34:39.269 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:39.269 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T03:34:39.269 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T03:34:39.269 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T03:34:39.269 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T03:34:39.270 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T03:34:39.270 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T03:34:39.270 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T03:34:39.270 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T03:34:39.270 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T03:34:39.270 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T03:34:39.270 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T03:34:39.270 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T03:34:39.270 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:39.270 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:39.310 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T03:34:39.371 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T03:34:39.427 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T03:34:39.463 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T03:34:39.522 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T03:34:39.546 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:34:39.556 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T03:34:39.623 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T03:34:39.666 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-16T03:34:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 1/12 2024-09-16T03:34:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T03:34:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T03:34:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T03:34:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T03:34:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T03:34:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T03:34:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T03:34:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T03:34:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T03:34:39.667 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T03:34:39.713 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T03:34:39.724 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T03:34:39.767 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T03:34:39.781 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T03:34:40.054 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T03:34:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:40.055 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:34:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T03:34:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T03:34:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T03:34:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T03:34:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T03:34:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T03:34:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T03:34:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T03:34:40.055 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T03:34:40.056 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T03:34:40.056 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T03:34:40.056 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:40.056 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:34:40.129 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:28 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:40.265 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:40.266 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:40.266 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-16T03:34:40.266 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:40.266 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:34:40.266 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 154 k 2024-09-16T03:34:40.266 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-16T03:34:40.266 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T03:34:40.266 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T03:34:40.267 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T03:34:40.267 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T03:34:40.267 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T03:34:40.267 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:40.267 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:34:40.267 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:40.267 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-09-16T03:34:40.267 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:40.268 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 7.0 M 2024-09-16T03:34:40.268 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 33 M 2024-09-16T03:34:40.268 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:34:40.324 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T03:34:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T03:34:40.450 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-rook 2024-09-16T03:34:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T03:34:40.496 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T03:34:40.519 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-16T03:34:40.545 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T03:34:40.573 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-09-16T03:34:40.579 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T03:34:40.627 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T03:34:40.631 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-16T03:34:40.649 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-16T03:34:40.657 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): ceph-mgr-cephadm-19.3.0-4930.gfc3e0bff.e 710 kB/s | 154 kB 00:00 2024-09-16T03:34:40.682 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-16T03:34:40.724 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-16T03:34:40.791 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-09-16T03:34:40.793 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:40.793 INFO:teuthology.orchestra.run.smithi155.stdout:Total 13 MB/s | 7.0 MB 00:00 2024-09-16T03:34:40.863 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:34:40.874 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:34:40.874 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:34:40.974 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:34:40.974 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:34:41.024 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:34:41.157 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:34:41.158 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:41.158 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-16T03:34:41.158 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:41.158 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:34:41.158 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 49 k 2024-09-16T03:34:41.158 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-16T03:34:41.158 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T03:34:41.158 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T03:34:41.159 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T03:34:41.159 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T03:34:41.159 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T03:34:41.159 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T03:34:41.159 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T03:34:41.159 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T03:34:41.159 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T03:34:41.159 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T03:34:41.159 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T03:34:41.159 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T03:34:41.159 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:41.159 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:34:41.160 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:41.160 INFO:teuthology.orchestra.run.smithi112.stdout:Install 13 Packages 2024-09-16T03:34:41.160 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:41.160 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 2.2 M 2024-09-16T03:34:41.160 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 26 M 2024-09-16T03:34:41.160 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:34:41.304 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-16T03:34:41.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/6 2024-09-16T03:34:41.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T03:34:41.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T03:34:41.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T03:34:41.304 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T03:34:41.307 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:34:41.324 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-16T03:34:41.324 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/13 2024-09-16T03:34:41.324 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T03:34:41.324 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T03:34:41.324 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T03:34:41.324 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T03:34:41.324 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T03:34:41.324 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T03:34:41.324 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T03:34:41.324 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T03:34:41.325 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T03:34:41.325 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T03:34:41.325 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T03:34:41.475 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T03:34:41.535 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T03:34:41.575 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T03:34:41.575 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:41.575 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:34:41.575 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:41.576 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T03:34:41.576 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T03:34:41.576 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T03:34:41.576 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T03:34:41.576 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T03:34:41.576 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:41.576 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T03:34:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:41.664 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:34:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T03:34:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T03:34:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T03:34:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T03:34:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T03:34:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T03:34:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T03:34:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T03:34:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T03:34:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T03:34:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T03:34:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T03:34:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:41.666 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:41.666 INFO:teuthology.orchestra.run.smithi112.stdout:(1/13): ceph-mgr-rook-19.3.0-4930.gfc3e0bff.el9 295 kB/s | 49 kB 00:00 2024-09-16T03:34:41.821 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-fuse 2024-09-16T03:34:41.919 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T03:34:41.997 INFO:teuthology.orchestra.run.smithi112.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 38 kB/s | 19 kB 00:00 2024-09-16T03:34:42.059 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T03:34:42.097 INFO:teuthology.orchestra.run.smithi112.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 43 kB/s | 26 kB 00:00 2024-09-16T03:34:42.154 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T03:34:42.231 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T03:34:42.255 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-16T03:34:42.413 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:30 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:42.480 INFO:teuthology.orchestra.run.smithi112.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 272 kB/s | 222 kB 00:00 2024-09-16T03:34:42.495 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:25 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:42.554 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:42.554 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:42.554 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-16T03:34:42.554 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:42.554 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:34:42.554 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 900 k 2024-09-16T03:34:42.554 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-09-16T03:34:42.555 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T03:34:42.555 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:42.555 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:34:42.555 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:42.555 INFO:teuthology.orchestra.run.smithi064.stdout:Install 2 Packages 2024-09-16T03:34:42.555 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:42.555 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 980 k 2024-09-16T03:34:42.556 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 2.9 M 2024-09-16T03:34:42.556 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:34:42.629 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:42.630 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:42.630 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-16T03:34:42.630 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:42.630 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:34:42.630 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 154 k 2024-09-16T03:34:42.630 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-16T03:34:42.630 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T03:34:42.630 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T03:34:42.630 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T03:34:42.630 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T03:34:42.631 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T03:34:42.631 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:42.631 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:34:42.631 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:42.631 INFO:teuthology.orchestra.run.smithi157.stdout:Install 6 Packages 2024-09-16T03:34:42.631 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:42.631 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 7.0 M 2024-09-16T03:34:42.632 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 33 M 2024-09-16T03:34:42.632 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:34:42.690 INFO:teuthology.orchestra.run.smithi064.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 704 kB/s | 80 kB 00:00 2024-09-16T03:34:42.772 INFO:teuthology.orchestra.run.smithi112.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 205 kB/s | 159 kB 00:00 2024-09-16T03:34:42.814 INFO:teuthology.orchestra.run.smithi112.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 794 kB/s | 32 kB 00:00 2024-09-16T03:34:42.831 INFO:teuthology.orchestra.run.smithi112.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 860 kB/s | 14 kB 00:00 2024-09-16T03:34:42.849 INFO:teuthology.orchestra.run.smithi064.stdout:(2/2): ceph-fuse-19.3.0-4930.gfc3e0bff.el9.x86_ 3.2 MB/s | 900 kB 00:00 2024-09-16T03:34:42.849 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:42.849 INFO:teuthology.orchestra.run.smithi064.stdout:Total 3.3 MB/s | 980 kB 00:00 2024-09-16T03:34:42.862 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:34:42.865 INFO:teuthology.orchestra.run.smithi112.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 6.5 MB/s | 220 kB 00:00 2024-09-16T03:34:42.885 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:34:42.885 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:34:42.890 INFO:teuthology.orchestra.run.smithi112.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 352 kB/s | 279 kB 00:00 2024-09-16T03:34:42.896 INFO:teuthology.orchestra.run.smithi157.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-16T03:34:42.921 INFO:teuthology.orchestra.run.smithi157.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4930.gfc3e0bff.e 654 kB/s | 154 kB 00:00 2024-09-16T03:34:42.940 INFO:teuthology.orchestra.run.smithi112.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-09-16T03:34:42.971 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:34:42.972 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:34:42.974 INFO:teuthology.orchestra.run.smithi112.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 9.4 MB/s | 1.0 MB 00:00 2024-09-16T03:34:42.980 INFO:teuthology.orchestra.run.smithi157.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 989 kB/s | 58 kB 00:00 2024-09-16T03:34:42.991 INFO:teuthology.orchestra.run.smithi112.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-16T03:34:43.014 INFO:teuthology.orchestra.run.smithi157.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.6 MB/s | 548 kB 00:00 2024-09-16T03:34:43.039 INFO:teuthology.orchestra.run.smithi157.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-09-16T03:34:43.108 INFO:teuthology.orchestra.run.smithi112.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 86 kB/s | 54 kB 00:00 2024-09-16T03:34:43.111 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:43.111 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.1 MB/s | 2.2 MB 00:01 2024-09-16T03:34:43.115 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-16T03:34:43.115 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/6 2024-09-16T03:34:43.115 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T03:34:43.115 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T03:34:43.116 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T03:34:43.116 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T03:34:43.166 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:34:43.181 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:34:43.197 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:34:43.197 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:34:43.288 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:34:43.288 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:34:43.306 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T03:34:43.377 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-16T03:34:43.380 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T03:34:43.381 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:43.381 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:34:43.381 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:43.381 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T03:34:43.381 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T03:34:43.381 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T03:34:43.381 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T03:34:43.381 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T03:34:43.381 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:43.381 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:43.648 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-fuse 2024-09-16T03:34:43.808 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:34:43.977 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T03:34:44.033 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T03:34:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-16T03:34:44.081 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/2 2024-09-16T03:34:44.096 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T03:34:44.122 INFO:teuthology.orchestra.run.smithi157.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.2 MB/s | 6.0 MB 00:01 2024-09-16T03:34:44.125 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:44.125 INFO:teuthology.orchestra.run.smithi157.stdout:Total 4.7 MB/s | 7.0 MB 00:01 2024-09-16T03:34:44.134 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T03:34:44.175 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T03:34:44.196 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:34:44.207 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:34:44.207 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:34:44.212 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:44.276 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T03:34:44.309 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:34:44.310 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:34:44.348 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:44.349 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:44.349 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-16T03:34:44.349 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:44.349 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:34:44.349 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 900 k 2024-09-16T03:34:44.349 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-09-16T03:34:44.350 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T03:34:44.350 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:44.350 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:34:44.350 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:44.350 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-09-16T03:34:44.350 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:44.350 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 980 k 2024-09-16T03:34:44.350 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.9 M 2024-09-16T03:34:44.350 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:34:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T03:34:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:44.361 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:34:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T03:34:44.361 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:44.362 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:44.378 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T03:34:44.444 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T03:34:44.585 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 590 kB/s | 80 kB 00:00 2024-09-16T03:34:44.625 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:34:44.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-volume 2024-09-16T03:34:44.768 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-fuse-19.3.0-4930.gfc3e0bff.el9.x86_ 2.8 MB/s | 900 kB 00:00 2024-09-16T03:34:44.769 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:44.769 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.3 MB/s | 980 kB 00:00 2024-09-16T03:34:44.781 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:34:44.787 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T03:34:44.802 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:34:44.802 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:34:44.840 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T03:34:44.883 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:34:44.883 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:34:45.070 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:34:45.152 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T03:34:45.191 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T03:34:45.222 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T03:34:45.224 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T03:34:45.258 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:45.272 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T03:34:45.295 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-16T03:34:45.302 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-16T03:34:45.364 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T03:34:45.397 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:45.398 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-09-16T03:34:45.398 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:45.398 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:34:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 285 k 2024-09-16T03:34:45.398 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:45.398 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:34:45.398 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:45.398 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-16T03:34:45.399 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:45.399 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 285 k 2024-09-16T03:34:45.399 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 1.3 M 2024-09-16T03:34:45.399 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:34:45.452 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T03:34:45.528 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T03:34:45.553 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-16T03:34:45.637 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-volume-19.3.0-4930.gfc3e0bff.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-16T03:34:45.637 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:45.638 INFO:teuthology.orchestra.run.smithi064.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-16T03:34:45.638 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:34:45.647 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:34:45.647 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:34:45.679 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:34:45.679 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:34:45.910 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:34:45.980 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:45.997 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:45.998 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:45.998 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T03:34:45.998 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:46.020 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-16T03:34:46.020 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/2 2024-09-16T03:34:46.074 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-16T03:34:46.075 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/13 2024-09-16T03:34:46.075 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T03:34:46.075 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T03:34:46.075 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T03:34:46.075 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T03:34:46.075 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T03:34:46.075 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T03:34:46.076 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T03:34:46.076 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T03:34:46.076 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T03:34:46.076 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T03:34:46.076 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T03:34:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T03:34:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:46.291 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:34:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T03:34:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:46.292 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-16T03:34:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/6 2024-09-16T03:34:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T03:34:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T03:34:46.374 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T03:34:46.375 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T03:34:46.416 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T03:34:46.417 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:46.417 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:34:46.417 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:46.417 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T03:34:46.417 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T03:34:46.417 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T03:34:46.417 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T03:34:46.417 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T03:34:46.417 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T03:34:46.417 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T03:34:46.418 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T03:34:46.418 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T03:34:46.418 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T03:34:46.418 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T03:34:46.418 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T03:34:46.418 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:46.418 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:34:46.600 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-volume 2024-09-16T03:34:46.645 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T03:34:46.645 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:46.645 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:34:46.645 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:46.646 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T03:34:46.646 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T03:34:46.646 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T03:34:46.646 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T03:34:46.646 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T03:34:46.647 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:46.647 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:46.655 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T03:34:46.896 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-fuse 2024-09-16T03:34:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:47.066 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:34:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:47.066 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:47.189 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:47.219 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:34:47.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados-devel 2024-09-16T03:34:47.326 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:47.326 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-09-16T03:34:47.326 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:47.326 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:34:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 285 k 2024-09-16T03:34:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:47.326 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:34:47.326 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:47.326 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-16T03:34:47.327 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:47.327 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 285 k 2024-09-16T03:34:47.327 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.3 M 2024-09-16T03:34:47.327 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:34:47.353 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:34:47.353 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-16T03:34:47.354 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:47.354 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:34:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 154 k 2024-09-16T03:34:47.354 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-16T03:34:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T03:34:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T03:34:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T03:34:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T03:34:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T03:34:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:47.354 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:34:47.355 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:47.355 INFO:teuthology.orchestra.run.smithi112.stdout:Install 6 Packages 2024-09-16T03:34:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:47.355 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 7.0 M 2024-09-16T03:34:47.355 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 33 M 2024-09-16T03:34:47.355 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:34:47.460 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:30 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:47.558 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-volume-19.3.0-4930.gfc3e0bff.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-16T03:34:47.558 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:47.558 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-16T03:34:47.559 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:34:47.567 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:34:47.567 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:34:47.594 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:47.594 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:47.595 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-16T03:34:47.595 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:47.595 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:34:47.595 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 900 k 2024-09-16T03:34:47.595 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-09-16T03:34:47.595 INFO:teuthology.orchestra.run.smithi157.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T03:34:47.595 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:47.595 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:34:47.595 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:47.596 INFO:teuthology.orchestra.run.smithi157.stdout:Install 2 Packages 2024-09-16T03:34:47.596 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:47.596 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 980 k 2024-09-16T03:34:47.597 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 2.9 M 2024-09-16T03:34:47.597 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:34:47.598 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:34:47.599 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:34:47.650 INFO:teuthology.orchestra.run.smithi112.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4930.gfc3e0bff.e 726 kB/s | 154 kB 00:00 2024-09-16T03:34:47.717 INFO:teuthology.orchestra.run.smithi112.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.1 MB/s | 548 kB 00:00 2024-09-16T03:34:47.743 INFO:teuthology.orchestra.run.smithi112.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-16T03:34:47.767 INFO:teuthology.orchestra.run.smithi112.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-16T03:34:47.820 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:34:47.841 INFO:teuthology.orchestra.run.smithi157.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 703 kB/s | 80 kB 00:00 2024-09-16T03:34:47.885 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:47.905 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:47.905 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:47.905 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T03:34:47.905 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:47.919 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:48.016 INFO:teuthology.orchestra.run.smithi157.stdout:(2/2): ceph-fuse-19.3.0-4930.gfc3e0bff.el9.x86_ 3.0 MB/s | 900 kB 00:00 2024-09-16T03:34:48.017 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:48.017 INFO:teuthology.orchestra.run.smithi157.stdout:Total 2.3 MB/s | 980 kB 00:00 2024-09-16T03:34:48.029 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:34:48.051 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:34:48.051 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:34:48.059 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-16T03:34:48.060 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 127 k 2024-09-16T03:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:48.061 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-16T03:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 127 k 2024-09-16T03:34:48.062 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 456 k 2024-09-16T03:34:48.063 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:34:48.076 INFO:teuthology.orchestra.run.smithi112.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 389 kB/s | 248 kB 00:00 2024-09-16T03:34:48.132 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:34:48.133 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:34:48.241 INFO:teuthology.orchestra.run.smithi064.stdout:librados-devel-19.3.0-4930.gfc3e0bff.el9.x86_64 699 kB/s | 127 kB 00:00 2024-09-16T03:34:48.242 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:48.242 INFO:teuthology.orchestra.run.smithi064.stdout:Total 694 kB/s | 127 kB 00:00 2024-09-16T03:34:48.243 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:34:48.248 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:34:48.248 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:34:48.316 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:34:48.317 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:34:48.318 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:34:48.451 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T03:34:48.477 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:34:48.523 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-16T03:34:48.584 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:48.910 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:48.910 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:48.910 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:34:48.910 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:48.910 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:48.910 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:49.021 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:49.213 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados-devel 2024-09-16T03:34:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-16T03:34:49.246 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/2 2024-09-16T03:34:49.284 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:49.284 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:49.285 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:34:49.285 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:49.285 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:49.285 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:49.451 INFO:teuthology.orchestra.run.smithi112.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.0 MB/s | 6.0 MB 00:02 2024-09-16T03:34:49.453 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:49.454 INFO:teuthology.orchestra.run.smithi112.stdout:Total 3.3 MB/s | 7.0 MB 00:02 2024-09-16T03:34:49.521 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T03:34:49.521 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:49.521 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:34:49.521 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T03:34:49.521 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:49.521 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:49.525 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:34:49.537 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:34:49.537 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:34:49.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs2 2024-09-16T03:34:49.637 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:34:49.637 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:34:49.808 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:49.837 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-volume 2024-09-16T03:34:49.937 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:34:49.943 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:49.944 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:49.944 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-09-16T03:34:49.944 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:49.944 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:34:49.944 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 127 k 2024-09-16T03:34:49.944 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:49.944 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:34:49.944 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:49.944 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-16T03:34:49.944 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:49.945 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 127 k 2024-09-16T03:34:49.945 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 456 k 2024-09-16T03:34:49.945 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:34:50.105 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T03:34:50.145 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:50.147 INFO:teuthology.orchestra.run.smithi155.stdout:librados-devel-19.3.0-4930.gfc3e0bff.el9.x86_64 625 kB/s | 127 kB 00:00 2024-09-16T03:34:50.148 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:50.148 INFO:teuthology.orchestra.run.smithi155.stdout:Total 621 kB/s | 127 kB 00:00 2024-09-16T03:34:50.148 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:34:50.156 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:34:50.156 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:34:50.161 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T03:34:50.219 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:34:50.220 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:34:50.239 INFO:teuthology.orchestra.run.smithi064.stdout:Package libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:50.286 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:50.287 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-16T03:34:50.287 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:50.361 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install libcephfs-devel 2024-09-16T03:34:50.366 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:34:50.409 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:33 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:50.487 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:50.546 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:50.546 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:50.546 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-09-16T03:34:50.546 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:50.546 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:34:50.546 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 285 k 2024-09-16T03:34:50.547 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:50.547 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:34:50.547 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:50.547 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-16T03:34:50.547 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:50.547 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 285 k 2024-09-16T03:34:50.548 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 1.3 M 2024-09-16T03:34:50.548 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:34:50.676 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T03:34:50.771 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T03:34:50.780 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-volume-19.3.0-4930.gfc3e0bff.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-16T03:34:50.780 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:50.780 INFO:teuthology.orchestra.run.smithi157.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-16T03:34:50.781 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:34:50.789 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:34:50.789 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:34:50.817 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:34:50.817 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:34:50.839 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T03:34:50.865 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-16T03:34:50.925 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:50.956 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:51.045 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:34:51.099 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:51.100 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-16T03:34:51.100 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:51.100 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:34:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 32 k 2024-09-16T03:34:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:51.100 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:34:51.100 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:51.100 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-16T03:34:51.100 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:51.100 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 32 k 2024-09-16T03:34:51.100 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 142 k 2024-09-16T03:34:51.101 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:34:51.175 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:51.175 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:51.175 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:34:51.175 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:51.175 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:51.175 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:51.190 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:51.207 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:51.207 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:51.207 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T03:34:51.207 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:51.245 INFO:teuthology.orchestra.run.smithi064.stdout:libcephfs-devel-19.3.0-4930.gfc3e0bff.el9.x86_6 219 kB/s | 32 kB 00:00 2024-09-16T03:34:51.246 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:51.246 INFO:teuthology.orchestra.run.smithi064.stdout:Total 217 kB/s | 32 kB 00:00 2024-09-16T03:34:51.246 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:34:51.250 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:34:51.250 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:34:51.268 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:34:51.268 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:34:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:34:51.399 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs2 2024-09-16T03:34:51.471 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:51.620 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-16T03:34:51.620 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/6 2024-09-16T03:34:51.620 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T03:34:51.620 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T03:34:51.620 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T03:34:51.620 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T03:34:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:51.883 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T03:34:51.883 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:51.883 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:34:51.884 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:51.884 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T03:34:51.884 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T03:34:51.884 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T03:34:51.884 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T03:34:51.884 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T03:34:51.884 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:51.884 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:34:51.975 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:52.068 INFO:teuthology.orchestra.run.smithi155.stdout:Package libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:52.071 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:52.071 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:52.072 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:34:52.072 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:52.072 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:52.072 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:52.113 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-fuse 2024-09-16T03:34:52.116 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:52.117 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-16T03:34:52.117 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:52.188 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs-devel 2024-09-16T03:34:52.209 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:52.210 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:52.210 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:34:52.210 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:52.210 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:52.210 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:52.253 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librados2 2024-09-16T03:34:52.512 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados-devel 2024-09-16T03:34:52.678 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:34:52.758 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:52.811 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:34:52.812 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:52.812 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-16T03:34:52.812 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:52.812 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:34:52.812 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 900 k 2024-09-16T03:34:52.812 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-09-16T03:34:52.812 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T03:34:52.813 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:52.813 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:34:52.813 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:52.813 INFO:teuthology.orchestra.run.smithi112.stdout:Install 2 Packages 2024-09-16T03:34:52.813 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:52.817 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 980 k 2024-09-16T03:34:52.817 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 2.9 M 2024-09-16T03:34:52.817 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:34:52.839 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:52.892 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:52.893 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:52.893 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-09-16T03:34:52.893 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:52.893 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:34:52.893 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 32 k 2024-09-16T03:34:52.893 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:52.893 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:34:52.893 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:52.893 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-16T03:34:52.893 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:52.893 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 32 k 2024-09-16T03:34:52.893 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 142 k 2024-09-16T03:34:52.893 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:34:52.932 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:52.978 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:52.978 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-16T03:34:52.979 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:53.048 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install librbd1 2024-09-16T03:34:53.051 INFO:teuthology.orchestra.run.smithi155.stdout:libcephfs-devel-19.3.0-4930.gfc3e0bff.el9.x86_6 201 kB/s | 32 kB 00:00 2024-09-16T03:34:53.052 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:53.052 INFO:teuthology.orchestra.run.smithi155.stdout:Total 199 kB/s | 32 kB 00:00 2024-09-16T03:34:53.052 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:34:53.056 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:34:53.056 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:34:53.069 INFO:teuthology.orchestra.run.smithi112.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 505 kB/s | 80 kB 00:00 2024-09-16T03:34:53.071 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:34:53.072 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:34:53.104 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:53.164 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:34:53.203 INFO:teuthology.orchestra.run.smithi112.stdout:(2/2): ceph-fuse-19.3.0-4930.gfc3e0bff.el9.x86_ 3.0 MB/s | 900 kB 00:00 2024-09-16T03:34:53.203 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:53.203 INFO:teuthology.orchestra.run.smithi112.stdout:Total 2.4 MB/s | 980 kB 00:00 2024-09-16T03:34:53.215 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:34:53.237 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:34:53.237 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:34:53.239 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:53.239 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:53.239 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-09-16T03:34:53.239 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:53.239 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:34:53.240 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 127 k 2024-09-16T03:34:53.240 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:53.240 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:34:53.240 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:53.240 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-16T03:34:53.240 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:53.240 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 127 k 2024-09-16T03:34:53.241 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 456 k 2024-09-16T03:34:53.241 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:34:53.273 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:53.317 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:34:53.317 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:34:53.451 INFO:teuthology.orchestra.run.smithi157.stdout:librados-devel-19.3.0-4930.gfc3e0bff.el9.x86_64 602 kB/s | 127 kB 00:00 2024-09-16T03:34:53.451 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:53.452 INFO:teuthology.orchestra.run.smithi157.stdout:Total 598 kB/s | 127 kB 00:00 2024-09-16T03:34:53.452 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:34:53.458 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:34:53.458 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:34:53.514 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:34:53.521 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:34:53.521 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:34:53.640 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:53.640 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:53.654 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T03:34:53.677 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:34:53.718 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-16T03:34:53.735 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:53.781 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:53.782 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-16T03:34:53.782 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:53.783 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:53.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rados 2024-09-16T03:34:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:53.872 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:34:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:53.872 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:54.065 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados2 2024-09-16T03:34:54.186 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:54.386 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-16T03:34:54.387 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/2 2024-09-16T03:34:54.399 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:54.399 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:54.399 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:34:54.399 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:54.399 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:54.399 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:54.444 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:54.537 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:54.583 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:54.584 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-16T03:34:54.584 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:54.627 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:54.638 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs2 2024-09-16T03:34:54.656 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rgw 2024-09-16T03:34:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T03:34:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:54.665 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:34:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T03:34:54.665 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:54.665 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:34:54.716 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:54.761 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:54.762 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-16T03:34:54.762 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:54.831 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librbd1 2024-09-16T03:34:54.962 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-volume 2024-09-16T03:34:55.216 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:55.249 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:55.308 INFO:teuthology.orchestra.run.smithi157.stdout:Package libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:55.345 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:55.355 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:55.356 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-16T03:34:55.356 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:55.391 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:55.392 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-16T03:34:55.392 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:55.395 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:55.419 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install libcephfs-devel 2024-09-16T03:34:55.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-cephfs 2024-09-16T03:34:55.486 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:55.529 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:55.530 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-16T03:34:55.530 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:55.544 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:34:55.597 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rados 2024-09-16T03:34:55.682 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:34:55.682 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:55.682 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-09-16T03:34:55.683 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:55.683 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:34:55.683 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 285 k 2024-09-16T03:34:55.683 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:55.683 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:34:55.683 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:55.683 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-16T03:34:55.683 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:55.683 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 285 k 2024-09-16T03:34:55.683 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 1.3 M 2024-09-16T03:34:55.683 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:34:55.915 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-volume-19.3.0-4930.gfc3e0bff.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-16T03:34:55.915 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:55.915 INFO:teuthology.orchestra.run.smithi112.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-16T03:34:55.916 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:34:55.924 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:34:55.924 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:34:55.957 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:34:55.957 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:34:55.990 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:56.066 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:56.127 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:56.127 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:56.127 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-09-16T03:34:56.127 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:56.127 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:34:56.128 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 32 k 2024-09-16T03:34:56.128 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:56.128 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:34:56.128 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:34:56.128 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-16T03:34:56.128 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:56.128 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 32 k 2024-09-16T03:34:56.129 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 142 k 2024-09-16T03:34:56.129 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:34:56.162 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:56.163 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:56.182 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:34:56.209 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:56.210 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-16T03:34:56.211 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:56.254 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:56.258 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:56.271 INFO:teuthology.orchestra.run.smithi157.stdout:libcephfs-devel-19.3.0-4930.gfc3e0bff.el9.x86_6 223 kB/s | 32 kB 00:00 2024-09-16T03:34:56.272 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:56.272 INFO:teuthology.orchestra.run.smithi157.stdout:Total 221 kB/s | 32 kB 00:00 2024-09-16T03:34:56.272 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:34:56.275 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:56.275 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:34:56.275 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T03:34:56.275 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:56.275 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:34:56.276 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:34:56.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install python3-rbd 2024-09-16T03:34:56.294 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:34:56.294 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:34:56.301 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:56.301 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-16T03:34:56.301 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:56.371 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rgw 2024-09-16T03:34:56.388 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:34:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:56.875 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:56.879 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:56.939 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:56.973 INFO:teuthology.orchestra.run.smithi064.stdout:Package python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:57.019 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:57.020 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2024-09-16T03:34:57.020 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:57.030 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:57.073 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:57.074 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-16T03:34:57.074 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:57.092 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-fuse 2024-09-16T03:34:57.107 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:57.107 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:57.107 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:34:57.107 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:57.107 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:34:57.107 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:57.148 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-cephfs 2024-09-16T03:34:57.282 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-16T03:34:57.282 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:57.282 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:34:57.283 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-16T03:34:57.283 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:57.283 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:34:57.306 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librados2 2024-09-16T03:34:57.559 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados-devel 2024-09-16T03:34:57.686 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:57.719 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:57.810 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:57.828 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:34:57.829 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:57.829 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-09-16T03:34:57.829 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:57.829 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:34:57.829 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 85 k 2024-09-16T03:34:57.830 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:57.830 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:34:57.830 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:34:57.830 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-16T03:34:57.830 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:57.830 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 85 k 2024-09-16T03:34:57.830 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 231 k 2024-09-16T03:34:57.830 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:34:57.852 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:57.853 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-16T03:34:57.853 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:57.895 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:57.920 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rbd 2024-09-16T03:34:57.995 INFO:teuthology.orchestra.run.smithi157.stdout:Package librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:58.024 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-fuse-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 441 kB/s | 85 kB 00:00 2024-09-16T03:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:58.025 INFO:teuthology.orchestra.run.smithi064.stdout:Total 436 kB/s | 85 kB 00:00 2024-09-16T03:34:58.026 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:34:58.039 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:34:58.041 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:58.043 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-16T03:34:58.043 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:58.115 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install librbd1 2024-09-16T03:34:58.116 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:34:58.116 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:34:58.141 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:34:58.275 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:34:58.275 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:58.275 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-09-16T03:34:58.276 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:58.276 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:34:58.276 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 127 k 2024-09-16T03:34:58.276 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:58.276 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:34:58.276 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:34:58.276 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-16T03:34:58.276 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:58.276 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 127 k 2024-09-16T03:34:58.276 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 456 k 2024-09-16T03:34:58.276 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:34:58.291 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:34:58.391 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:58.469 INFO:teuthology.orchestra.run.smithi112.stdout:librados-devel-19.3.0-4930.gfc3e0bff.el9.x86_64 659 kB/s | 127 kB 00:00 2024-09-16T03:34:58.469 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:58.470 INFO:teuthology.orchestra.run.smithi112.stdout:Total 654 kB/s | 127 kB 00:00 2024-09-16T03:34:58.470 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:34:58.475 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:34:58.475 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:34:58.486 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:58.538 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:34:58.538 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:34:58.577 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:58.621 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:58.622 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-09-16T03:34:58.622 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:34:58.686 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:58.686 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:34:58.697 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-fuse 2024-09-16T03:34:58.777 INFO:teuthology.orchestra.run.smithi157.stdout:Package librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:58.788 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:58.791 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:58.832 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:58.833 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-16T03:34:58.833 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:58.890 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rados 2024-09-16T03:34:59.052 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:59.053 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:34:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:59.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:34:59.053 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:34:59.263 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:59.268 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:59.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-mirror 2024-09-16T03:34:59.397 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:34:59.398 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:59.398 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-09-16T03:34:59.398 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:59.398 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:34:59.398 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 85 k 2024-09-16T03:34:59.398 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:59.399 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:34:59.399 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:34:59.399 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-16T03:34:59.399 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:34:59.399 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 85 k 2024-09-16T03:34:59.399 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 231 k 2024-09-16T03:34:59.399 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:34:59.459 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:34:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:34:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:59.518 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:34:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:34:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:34:59.519 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:34:59.550 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:34:59.589 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-fuse-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 451 kB/s | 85 kB 00:00 2024-09-16T03:34:59.589 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:34:59.589 INFO:teuthology.orchestra.run.smithi155.stdout:Total 448 kB/s | 85 kB 00:00 2024-09-16T03:34:59.589 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:34:59.593 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:34:59.595 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-16T03:34:59.595 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:34:59.596 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:34:59.596 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:34:59.661 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rgw 2024-09-16T03:34:59.661 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:34:59.662 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:34:59.746 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs2 2024-09-16T03:34:59.818 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:34:59.912 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:34:59.919 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:00.052 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:35:00.053 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:35:00.053 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repo Size 2024-09-16T03:35:00.053 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:35:00.053 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:35:00.053 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-16T03:35:00.053 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:00.053 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:35:00.053 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:35:00.053 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-16T03:35:00.054 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:00.054 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.2 M 2024-09-16T03:35:00.054 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 14 M 2024-09-16T03:35:00.054 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:35:00.237 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:35:00.322 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:00.323 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:35:00.328 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:35:00.371 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:35:00.372 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-16T03:35:00.372 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:35:00.394 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-mirror-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 9.4 MB/s | 3.2 MB 00:00 2024-09-16T03:35:00.394 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:35:00.395 INFO:teuthology.orchestra.run.smithi064.stdout:Total 9.3 MB/s | 3.2 MB 00:00 2024-09-16T03:35:00.395 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:35:00.405 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:35:00.405 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:35:00.414 INFO:teuthology.orchestra.run.smithi112.stdout:Package libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:35:00.438 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-cephfs 2024-09-16T03:35:00.457 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:35:00.459 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-16T03:35:00.459 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:35:00.492 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:35:00.492 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:35:00.533 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install libcephfs-devel 2024-09-16T03:35:00.578 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:00.579 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:00.579 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:35:00.579 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:35:00.579 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:00.579 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:35:00.881 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-mirror 2024-09-16T03:35:00.958 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:35:01.010 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:35:01.028 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:01.056 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:35:01.056 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T03:35:01.056 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:35:01.056 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:35:01.056 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:01.098 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:35:01.100 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:35:01.143 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:35:01.144 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-16T03:35:01.144 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:35:01.216 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install python3-rbd 2024-09-16T03:35:01.232 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:35:01.233 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:35:01.233 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-09-16T03:35:01.233 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:35:01.234 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:35:01.234 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 32 k 2024-09-16T03:35:01.234 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:01.234 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:35:01.234 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:35:01.234 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-16T03:35:01.234 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:01.234 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 32 k 2024-09-16T03:35:01.234 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 142 k 2024-09-16T03:35:01.234 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:35:01.380 INFO:teuthology.orchestra.run.smithi112.stdout:libcephfs-devel-19.3.0-4930.gfc3e0bff.el9.x86_6 217 kB/s | 32 kB 00:00 2024-09-16T03:35:01.380 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:35:01.381 INFO:teuthology.orchestra.run.smithi112.stdout:Total 216 kB/s | 32 kB 00:00 2024-09-16T03:35:01.381 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:35:01.384 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:35:01.384 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:35:01.398 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:35:01.399 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:35:01.472 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:49 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:35:01.486 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:35:01.577 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:01.613 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:35:01.613 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:35:01.614 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-09-16T03:35:01.614 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:35:01.614 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:35:01.614 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-16T03:35:01.614 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:01.614 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:35:01.614 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:35:01.614 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-16T03:35:01.614 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:01.615 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 3.2 M 2024-09-16T03:35:01.615 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 14 M 2024-09-16T03:35:01.615 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:35:01.788 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:35:01.878 INFO:teuthology.orchestra.run.smithi157.stdout:Package python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:35:01.922 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:35:01.923 INFO:teuthology.orchestra.run.smithi157.stdout:Nothing to do. 2024-09-16T03:35:01.923 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:35:01.942 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:01.959 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-mirror-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 9.3 MB/s | 3.2 MB 00:00 2024-09-16T03:35:01.960 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:35:01.960 INFO:teuthology.orchestra.run.smithi155.stdout:Total 9.2 MB/s | 3.2 MB 00:00 2024-09-16T03:35:01.960 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:35:01.971 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:35:01.971 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:35:01.996 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-fuse 2024-09-16T03:35:02.057 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:35:02.057 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:35:02.196 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:02.196 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:02.196 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:35:02.196 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:35:02.196 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:02.196 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:35:02.221 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:02.221 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:02.221 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:35:02.221 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:35:02.221 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:02.221 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:35:02.394 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librados2 2024-09-16T03:35:02.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install rbd-nbd 2024-09-16T03:35:02.515 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:35:02.561 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:35:02.577 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:02.604 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:02.604 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:35:02.604 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T03:35:02.604 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:35:02.604 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:35:02.605 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:02.697 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:35:02.699 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:35:02.699 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-09-16T03:35:02.699 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:35:02.699 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:35:02.699 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 85 k 2024-09-16T03:35:02.699 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:02.699 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:35:02.699 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:35:02.699 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-16T03:35:02.700 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:02.700 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 85 k 2024-09-16T03:35:02.700 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 231 k 2024-09-16T03:35:02.700 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:35:02.875 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-fuse-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 487 kB/s | 85 kB 00:00 2024-09-16T03:35:02.875 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:35:02.875 INFO:teuthology.orchestra.run.smithi157.stdout:Total 483 kB/s | 85 kB 00:00 2024-09-16T03:35:02.875 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:35:02.882 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:35:02.882 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:35:02.946 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:35:02.947 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:35:02.958 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:35:03.048 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:35:03.054 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:51 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:35:03.091 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:35:03.092 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-16T03:35:03.092 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:35:03.106 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:35:03.161 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install librbd1 2024-09-16T03:35:03.195 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-09-16T03:35:03.196 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:35:03.196 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-09-16T03:35:03.196 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:35:03.197 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-09-16T03:35:03.197 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 170 k 2024-09-16T03:35:03.197 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:03.197 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-09-16T03:35:03.197 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-09-16T03:35:03.197 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-09-16T03:35:03.197 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:03.197 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 170 k 2024-09-16T03:35:03.198 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 490 k 2024-09-16T03:35:03.198 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-09-16T03:35:03.205 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:03.405 INFO:teuthology.orchestra.run.smithi064.stdout:rbd-nbd-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 817 kB/s | 170 kB 00:00 2024-09-16T03:35:03.406 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:35:03.406 INFO:teuthology.orchestra.run.smithi064.stdout:Total 812 kB/s | 170 kB 00:00 2024-09-16T03:35:03.406 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-09-16T03:35:03.414 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-09-16T03:35:03.414 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-09-16T03:35:03.484 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-09-16T03:35:03.484 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-09-16T03:35:03.634 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:03.637 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-09-16T03:35:03.722 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:35:03.728 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:03.729 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:03.729 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:35:03.729 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:35:03.729 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:03.729 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:35:03.747 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:03.813 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:35:03.857 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:35:03.858 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-16T03:35:03.858 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:35:03.900 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:03.900 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:03.900 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:35:03.900 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:35:03.901 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:03.901 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:35:03.919 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rados 2024-09-16T03:35:03.993 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-nbd 2024-09-16T03:35:04.117 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-mirror 2024-09-16T03:35:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:04.448 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:04.448 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:04.448 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-09-16T03:35:04.448 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:35:04.448 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:04.448 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-09-16T03:35:04.481 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:35:04.571 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:35:04.581 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:52 ago on Mon 16 Sep 2024 03:33:12 AM UTC. 2024-09-16T03:35:04.614 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:35:04.615 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-16T03:35:04.615 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:35:04.673 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rgw 2024-09-16T03:35:04.699 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:35:04.718 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-09-16T03:35:04.718 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:35:04.718 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-09-16T03:35:04.719 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:35:04.719 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-09-16T03:35:04.719 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 170 k 2024-09-16T03:35:04.719 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:04.719 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-09-16T03:35:04.720 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-09-16T03:35:04.720 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-09-16T03:35:04.720 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:04.720 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 170 k 2024-09-16T03:35:04.720 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 490 k 2024-09-16T03:35:04.720 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-09-16T03:35:04.725 DEBUG:teuthology.parallel:result is None 2024-09-16T03:35:04.839 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:35:04.840 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:35:04.840 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repo Size 2024-09-16T03:35:04.840 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:35:04.840 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:35:04.840 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-16T03:35:04.840 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:04.840 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:35:04.841 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:35:04.841 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-16T03:35:04.841 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:04.841 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 3.2 M 2024-09-16T03:35:04.841 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 14 M 2024-09-16T03:35:04.841 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:35:04.931 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-nbd-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 795 kB/s | 170 kB 00:00 2024-09-16T03:35:04.931 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:35:04.932 INFO:teuthology.orchestra.run.smithi155.stdout:Total 790 kB/s | 170 kB 00:00 2024-09-16T03:35:04.932 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-09-16T03:35:04.939 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-09-16T03:35:04.940 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-09-16T03:35:05.003 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-09-16T03:35:05.003 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-09-16T03:35:05.160 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-09-16T03:35:05.186 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-mirror-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 9.2 MB/s | 3.2 MB 00:00 2024-09-16T03:35:05.187 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:35:05.187 INFO:teuthology.orchestra.run.smithi157.stdout:Total 9.2 MB/s | 3.2 MB 00:00 2024-09-16T03:35:05.187 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:35:05.197 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:35:05.197 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:35:05.235 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:49 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:35:05.262 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:05.279 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:35:05.280 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:35:05.325 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:35:05.368 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:35:05.369 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-16T03:35:05.369 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:35:05.432 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-cephfs 2024-09-16T03:35:05.681 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:05.719 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:35:05.787 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:05.813 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:05.813 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:35:05.813 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T03:35:05.813 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:35:05.813 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:35:05.813 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:05.946 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:05.946 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:05.946 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-09-16T03:35:05.946 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:35:05.946 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:05.946 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-09-16T03:35:05.997 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:49 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:35:06.088 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:35:06.131 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:35:06.133 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-16T03:35:06.133 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:35:06.191 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install python3-rbd 2024-09-16T03:35:06.227 DEBUG:teuthology.parallel:result is None 2024-09-16T03:35:06.757 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:50 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:35:06.847 INFO:teuthology.orchestra.run.smithi112.stdout:Package python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-16T03:35:06.890 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:35:06.891 INFO:teuthology.orchestra.run.smithi112.stdout:Nothing to do. 2024-09-16T03:35:06.891 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:35:06.931 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:06.931 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:06.931 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:35:06.931 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:35:06.932 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:06.932 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:35:06.953 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-fuse 2024-09-16T03:35:07.182 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install rbd-nbd 2024-09-16T03:35:07.534 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:51 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:35:07.679 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:35:07.680 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:35:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-09-16T03:35:07.680 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:35:07.680 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:35:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 85 k 2024-09-16T03:35:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:07.680 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:35:07.680 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:35:07.680 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-16T03:35:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:07.681 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 85 k 2024-09-16T03:35:07.681 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 231 k 2024-09-16T03:35:07.681 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:35:07.767 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:50 ago on Mon 16 Sep 2024 03:33:17 AM UTC. 2024-09-16T03:35:07.866 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-fuse-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 460 kB/s | 85 kB 00:00 2024-09-16T03:35:07.867 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:35:07.868 INFO:teuthology.orchestra.run.smithi112.stdout:Total 456 kB/s | 85 kB 00:00 2024-09-16T03:35:07.868 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:35:07.875 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:35:07.875 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:35:07.903 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-09-16T03:35:07.904 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:35:07.904 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-09-16T03:35:07.904 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:35:07.904 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-09-16T03:35:07.904 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 170 k 2024-09-16T03:35:07.904 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:07.904 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-09-16T03:35:07.904 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-09-16T03:35:07.904 INFO:teuthology.orchestra.run.smithi157.stdout:Install 1 Package 2024-09-16T03:35:07.904 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:07.904 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 170 k 2024-09-16T03:35:07.904 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 490 k 2024-09-16T03:35:07.905 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-09-16T03:35:07.941 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:35:07.942 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:35:08.097 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:35:08.118 INFO:teuthology.orchestra.run.smithi157.stdout:rbd-nbd-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 806 kB/s | 170 kB 00:00 2024-09-16T03:35:08.118 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:35:08.118 INFO:teuthology.orchestra.run.smithi157.stdout:Total 801 kB/s | 170 kB 00:00 2024-09-16T03:35:08.118 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-09-16T03:35:08.124 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-09-16T03:35:08.125 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-09-16T03:35:08.189 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-09-16T03:35:08.190 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-09-16T03:35:08.200 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:08.355 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-09-16T03:35:08.464 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:08.590 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:08.857 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:08.857 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:08.857 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:35:08.857 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:35:08.857 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:08.857 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:35:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:09.126 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-mirror 2024-09-16T03:35:09.182 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:09.182 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:09.182 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-09-16T03:35:09.182 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:35:09.183 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:09.183 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-09-16T03:35:09.435 DEBUG:teuthology.parallel:result is None 2024-09-16T03:35:09.708 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:53 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:35:09.846 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:35:09.847 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:35:09.847 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repo Size 2024-09-16T03:35:09.847 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:35:09.847 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:35:09.847 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-16T03:35:09.847 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:09.847 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:35:09.847 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:35:09.847 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-16T03:35:09.847 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:09.847 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 3.2 M 2024-09-16T03:35:09.848 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 14 M 2024-09-16T03:35:09.848 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:35:10.186 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-mirror-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 9.4 MB/s | 3.2 MB 00:00 2024-09-16T03:35:10.186 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:35:10.187 INFO:teuthology.orchestra.run.smithi112.stdout:Total 9.4 MB/s | 3.2 MB 00:00 2024-09-16T03:35:10.187 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:35:10.196 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:35:10.196 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:35:10.278 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:35:10.278 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:35:10.730 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:35:10.779 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:10.804 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:10.804 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T03:35:10.804 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T03:35:10.805 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:35:10.805 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T03:35:10.805 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:11.953 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:35:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:35:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:11.953 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:35:12.153 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install rbd-nbd 2024-09-16T03:35:12.728 INFO:teuthology.orchestra.run.smithi112.stdout:Last metadata expiration check: 0:01:56 ago on Mon 16 Sep 2024 03:33:16 AM UTC. 2024-09-16T03:35:12.863 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-09-16T03:35:12.864 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:35:12.864 INFO:teuthology.orchestra.run.smithi112.stdout: Package Architecture Version Repository Size 2024-09-16T03:35:12.864 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:35:12.864 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-09-16T03:35:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 170 k 2024-09-16T03:35:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:12.865 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-09-16T03:35:12.865 INFO:teuthology.orchestra.run.smithi112.stdout:================================================================================ 2024-09-16T03:35:12.865 INFO:teuthology.orchestra.run.smithi112.stdout:Install 1 Package 2024-09-16T03:35:12.865 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:12.865 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 170 k 2024-09-16T03:35:12.865 INFO:teuthology.orchestra.run.smithi112.stdout:Installed size: 490 k 2024-09-16T03:35:12.865 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-09-16T03:35:13.072 INFO:teuthology.orchestra.run.smithi112.stdout:rbd-nbd-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 824 kB/s | 170 kB 00:00 2024-09-16T03:35:13.073 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-09-16T03:35:13.073 INFO:teuthology.orchestra.run.smithi112.stdout:Total 817 kB/s | 170 kB 00:00 2024-09-16T03:35:13.073 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-09-16T03:35:13.081 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-09-16T03:35:13.081 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-09-16T03:35:13.144 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-09-16T03:35:13.144 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-09-16T03:35:13.298 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-09-16T03:35:13.403 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:13.804 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:14.064 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-16T03:35:14.064 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:14.064 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-09-16T03:35:14.064 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-16T03:35:14.064 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:14.064 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-09-16T03:35:14.333 DEBUG:teuthology.parallel:result is None 2024-09-16T03:35:14.334 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T03:35:14.484 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T03:35:14.511 INFO:teuthology.orchestra.run.smithi064.stdout:19.3.0-4930.gfc3e0bff.el9 2024-09-16T03:35:14.511 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930.gfc3e0bff.el9 2024-09-16T03:35:14.512 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930.gfc3e0bff is installed. 2024-09-16T03:35:14.513 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T03:35:14.653 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T03:35:14.675 INFO:teuthology.orchestra.run.smithi112.stdout:19.3.0-4930.gfc3e0bff.el9 2024-09-16T03:35:14.676 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930.gfc3e0bff.el9 2024-09-16T03:35:14.676 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930.gfc3e0bff is installed. 2024-09-16T03:35:14.678 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T03:35:14.817 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T03:35:14.843 INFO:teuthology.orchestra.run.smithi155.stdout:19.3.0-4930.gfc3e0bff.el9 2024-09-16T03:35:14.844 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930.gfc3e0bff.el9 2024-09-16T03:35:14.844 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930.gfc3e0bff is installed. 2024-09-16T03:35:14.846 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T03:35:14.986 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T03:35:15.013 INFO:teuthology.orchestra.run.smithi157.stdout:19.3.0-4930.gfc3e0bff.el9 2024-09-16T03:35:15.013 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930.gfc3e0bff.el9 2024-09-16T03:35:15.013 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930.gfc3e0bff is installed. 2024-09-16T03:35:15.015 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T03:35:15.016 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:35:15.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:35:15.047 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:35:15.047 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:35:15.075 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:35:15.075 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:35:15.106 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:35:15.106 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:35:15.137 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T03:35:15.137 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:35:15.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:35:15.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:35:15.240 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:35:15.240 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:35:15.267 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:35:15.334 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:35:15.334 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:35:15.364 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:35:15.434 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:35:15.434 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:35:15.463 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:35:15.535 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T03:35:15.536 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:35:15.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:35:15.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:35:15.636 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:35:15.637 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:35:15.664 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:35:15.732 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:35:15.732 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:35:15.762 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:35:15.831 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:35:15.832 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:35:15.861 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:35:15.931 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T03:35:15.931 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:35:15.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:35:15.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:35:16.035 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:35:16.035 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:35:16.064 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:35:16.134 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:35:16.134 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:35:16.164 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:35:16.234 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:35:16.234 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:35:16.265 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:35:16.335 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T03:35:16.432 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T03:35:16.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2024-09-16T03:35:16.435 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /var/log/ceph 2024-09-16T03:35:16.437 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /var/log/ceph 2024-09-16T03:35:16.439 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /var/log/ceph 2024-09-16T03:35:16.469 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T03:35:16.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:35:16.505 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:35:16.508 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:35:16.512 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:35:16.541 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T03:35:16.541 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:35:16.574 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:35:16.577 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:35:16.582 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:35:16.622 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T03:35:16.622 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'mon election default strategy': 3, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-16T03:35:16.623 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882', 'branch': 'main', 'description': 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/careful_host thrashosds-health workloads/dedup-io-mixed}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7905882', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'min_in': 4}, 'workunit': {'branch': 'main', 'sha1': '7498493279982dfe87f00616198a5967475f6169'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 2855, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'sleep_before_teardown': 0, 'subset': '17936/100000', 'suite': 'rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7498493279982dfe87f00616198a5967475f6169', 'targets': {'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMrSlIFHn+oeg94Tleq8n2nqfaZxdYUJqDxf7yWhFl4am6RiJGZL1xZWBI2ejqZK0fe1g3mSHS01BXKxZXvHS+0=', 'smithi112.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDz/uE2dfF6JFvVUxekLqome09IzbXavXJpbC7zieyyUUHoBpf993ZzznOmY29k6eEIyQxMiVbaLovyhjxO6fGQ=', 'smithi155.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGD7G0C86AQxUvTZYhli9+OdNHhsdJ62jAqoKIzK7C4FaOoLqE32h8O0SwsnglWigoribcKVXN+Krv1AXV/6P2A=', 'smithi157.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHyuxrOLtYmKFpCTrCN/85jDl//yoybKDtHqE2BBtC7MGSOqRUudo0YCNt4S3ixAmfqxPmfKwUneN/hdwXWgI5k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'global': {'mon election default strategy': 3, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'cluster': 'ceph'}}, {'exec': {'mon.a': ['while ! ceph balancer status ; do sleep 1 ; done', 'ceph balancer mode crush-compat', 'ceph balancer on']}}, {'thrashosds': {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 2, 'thrash_hosts': True, 'timeout': 1200}}, {'exec': {'client.0': ['sudo ceph osd pool create low_tier 4']}}, {'rados': {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'set_chunk': 30, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_20:00:16', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T03:35:16.623 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:35:16.643 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:35:16.646 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:35:16.652 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:35:16.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:35:16.701 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:35:16.703 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:35:16.710 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:35:16.737 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:35:16.737 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:35:16.784 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:35:16.784 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-09-16T03:35:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:35:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:16.842 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T03:35:16.842 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:16.842 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:16.842 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-16 03:35:01.320063198 +0000 2024-09-16T03:35:16.843 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-16 03:32:31.334894146 +0000 2024-09-16T03:35:16.843 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-16 03:32:31.334894146 +0000 2024-09-16T03:35:16.843 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-16 03:32:31.334894146 +0000 2024-09-16T03:35:16.843 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:35:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-16T03:35:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-16T03:35:16.907 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000263495 s, 1.9 MB/s 2024-09-16T03:35:16.908 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:35:16.966 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-09-16T03:35:17.023 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:35:17.023 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:17.023 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-16T03:35:17.023 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:17.023 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:17.023 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-16 03:35:01.320063198 +0000 2024-09-16T03:35:17.023 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-16 03:32:31.660887778 +0000 2024-09-16T03:35:17.023 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-16 03:32:31.660887778 +0000 2024-09-16T03:35:17.024 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-16 03:32:31.660887778 +0000 2024-09-16T03:35:17.024 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:35:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-16T03:35:17.089 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-16T03:35:17.090 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000196645 s, 2.6 MB/s 2024-09-16T03:35:17.090 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:35:17.148 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-09-16T03:35:17.205 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:35:17.205 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:17.205 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-16T03:35:17.205 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:17.205 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:17.205 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-16 03:35:01.321063179 +0000 2024-09-16T03:35:17.206 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-16 03:32:31.988881370 +0000 2024-09-16T03:35:17.206 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-16 03:32:31.988881370 +0000 2024-09-16T03:35:17.206 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-16 03:32:31.988881370 +0000 2024-09-16T03:35:17.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:35:17.270 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-16T03:35:17.270 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-16T03:35:17.270 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000179609 s, 2.9 MB/s 2024-09-16T03:35:17.271 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:35:17.328 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-09-16T03:35:17.384 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:35:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:17.385 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-16T03:35:17.385 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:17.385 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:17.385 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-16 03:35:01.321063179 +0000 2024-09-16T03:35:17.385 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-16 03:32:32.342874455 +0000 2024-09-16T03:35:17.385 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-16 03:32:32.342874455 +0000 2024-09-16T03:35:17.385 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-09-16 03:32:32.342874455 +0000 2024-09-16T03:35:17.385 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:35:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-16T03:35:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-16T03:35:17.452 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000176846 s, 2.9 MB/s 2024-09-16T03:35:17.453 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:35:17.510 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:17.510 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:35:17.510 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:35:17.529 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:35:17.530 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2024-09-16T03:35:17.589 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:35:17.589 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:17.589 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T03:35:17.589 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:17.589 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:17.589 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-16 03:35:11.055211882 +0000 2024-09-16T03:35:17.590 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-16 03:32:27.898591530 +0000 2024-09-16T03:35:17.590 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-16 03:32:27.898591530 +0000 2024-09-16T03:35:17.590 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-09-16 03:32:27.898591530 +0000 2024-09-16T03:35:17.590 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:35:17.660 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-16T03:35:17.660 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-16T03:35:17.660 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000240766 s, 2.1 MB/s 2024-09-16T03:35:17.660 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:35:17.718 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2024-09-16T03:35:17.776 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:35:17.776 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:17.776 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-16T03:35:17.776 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:17.776 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:17.776 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-16 03:35:11.055211882 +0000 2024-09-16T03:35:17.776 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-16 03:32:28.228582970 +0000 2024-09-16T03:35:17.777 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-16 03:32:28.229582944 +0000 2024-09-16T03:35:17.777 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-09-16 03:32:28.228582970 +0000 2024-09-16T03:35:17.777 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:35:17.842 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-16T03:35:17.842 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-16T03:35:17.842 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000239612 s, 2.1 MB/s 2024-09-16T03:35:17.843 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:35:17.902 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2024-09-16T03:35:17.958 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:35:17.958 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:17.958 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-16T03:35:17.959 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:17.959 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:17.959 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-16 03:35:11.056211856 +0000 2024-09-16T03:35:17.959 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-16 03:32:28.561574331 +0000 2024-09-16T03:35:17.959 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-16 03:32:28.561574331 +0000 2024-09-16T03:35:17.959 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-09-16 03:32:28.561574331 +0000 2024-09-16T03:35:17.959 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:35:18.023 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-16T03:35:18.023 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-16T03:35:18.023 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.00019206 s, 2.7 MB/s 2024-09-16T03:35:18.024 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:35:18.081 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2024-09-16T03:35:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:35:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:18.137 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-16T03:35:18.137 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:18.138 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:18.138 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-16 03:35:11.056211856 +0000 2024-09-16T03:35:18.138 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-16 03:32:28.879566082 +0000 2024-09-16T03:35:18.138 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-16 03:32:28.879566082 +0000 2024-09-16T03:35:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-09-16 03:32:28.879566082 +0000 2024-09-16T03:35:18.138 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:35:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-16T03:35:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-16T03:35:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000206187 s, 2.5 MB/s 2024-09-16T03:35:18.202 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:35:18.259 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:18.259 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:35:18.259 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:35:18.279 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:35:18.279 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2024-09-16T03:35:18.337 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:35:18.338 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:18.338 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-16T03:35:18.338 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:18.338 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:18.338 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-09-16 03:35:02.862718970 +0000 2024-09-16T03:35:18.338 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-09-16 03:32:29.225340682 +0000 2024-09-16T03:35:18.338 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-09-16 03:32:29.225340682 +0000 2024-09-16T03:35:18.338 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-09-16 03:32:29.225340682 +0000 2024-09-16T03:35:18.338 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:35:18.405 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-09-16T03:35:18.405 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-09-16T03:35:18.405 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000281184 s, 1.8 MB/s 2024-09-16T03:35:18.406 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:35:18.465 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2024-09-16T03:35:18.523 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:35:18.523 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:18.523 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-16T03:35:18.523 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:18.523 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:18.523 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-09-16 03:35:02.863718951 +0000 2024-09-16T03:35:18.523 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-09-16 03:32:29.546334613 +0000 2024-09-16T03:35:18.524 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-09-16 03:32:29.546334613 +0000 2024-09-16T03:35:18.524 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-09-16 03:32:29.546334613 +0000 2024-09-16T03:35:18.524 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:35:18.589 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-09-16T03:35:18.590 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-09-16T03:35:18.590 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000260872 s, 2.0 MB/s 2024-09-16T03:35:18.591 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:35:18.648 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2024-09-16T03:35:18.705 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:35:18.705 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:18.705 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-16T03:35:18.705 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:18.705 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:18.705 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-09-16 03:35:02.863718951 +0000 2024-09-16T03:35:18.705 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-09-16 03:32:29.870328487 +0000 2024-09-16T03:35:18.705 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-09-16 03:32:29.870328487 +0000 2024-09-16T03:35:18.705 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-09-16 03:32:29.870328487 +0000 2024-09-16T03:35:18.705 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:35:18.770 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-09-16T03:35:18.770 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-09-16T03:35:18.770 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000204901 s, 2.5 MB/s 2024-09-16T03:35:18.771 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:35:18.828 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2024-09-16T03:35:18.885 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:35:18.885 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:18.885 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-16T03:35:18.885 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:18.885 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:18.885 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-09-16 03:35:02.863718951 +0000 2024-09-16T03:35:18.885 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-09-16 03:32:30.220321870 +0000 2024-09-16T03:35:18.885 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-09-16 03:32:30.220321870 +0000 2024-09-16T03:35:18.886 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-09-16 03:32:30.220321870 +0000 2024-09-16T03:35:18.886 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:35:18.950 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-09-16T03:35:18.950 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-09-16T03:35:18.950 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000217076 s, 2.4 MB/s 2024-09-16T03:35:18.951 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:35:19.008 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:19.009 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:35:19.009 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:35:19.028 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:35:19.028 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2024-09-16T03:35:19.088 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:35:19.088 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:19.088 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T03:35:19.088 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:19.089 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:19.089 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-09-16 03:35:06.071225167 +0000 2024-09-16T03:35:19.089 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-09-16 03:32:32.533314399 +0000 2024-09-16T03:35:19.089 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-09-16 03:32:32.533314399 +0000 2024-09-16T03:35:19.089 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-09-16 03:32:32.533314399 +0000 2024-09-16T03:35:19.089 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:35:19.156 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-09-16T03:35:19.157 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-09-16T03:35:19.157 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.00010042 s, 5.1 MB/s 2024-09-16T03:35:19.158 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:35:19.217 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2024-09-16T03:35:19.275 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:35:19.275 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:19.275 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-16T03:35:19.276 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:19.276 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:19.276 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-09-16 03:35:06.072225147 +0000 2024-09-16T03:35:19.276 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-09-16 03:32:32.845308509 +0000 2024-09-16T03:35:19.276 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-09-16 03:32:32.845308509 +0000 2024-09-16T03:35:19.276 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-09-16 03:32:32.845308509 +0000 2024-09-16T03:35:19.277 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:35:19.341 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-09-16T03:35:19.341 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-09-16T03:35:19.341 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 9.4166e-05 s, 5.4 MB/s 2024-09-16T03:35:19.342 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:35:19.400 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2024-09-16T03:35:19.456 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:35:19.456 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:19.456 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-16T03:35:19.456 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:19.456 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:19.456 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-09-16 03:35:06.072225147 +0000 2024-09-16T03:35:19.456 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-09-16 03:32:33.173302317 +0000 2024-09-16T03:35:19.456 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-09-16 03:32:33.173302317 +0000 2024-09-16T03:35:19.456 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-09-16 03:32:33.173302317 +0000 2024-09-16T03:35:19.457 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:35:19.520 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-09-16T03:35:19.520 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-09-16T03:35:19.520 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000108081 s, 4.7 MB/s 2024-09-16T03:35:19.521 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:35:19.579 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2024-09-16T03:35:19.635 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:35:19.635 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:35:19.635 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-16T03:35:19.635 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:35:19.635 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T03:35:19.635 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-09-16 03:35:06.072225147 +0000 2024-09-16T03:35:19.635 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-09-16 03:32:33.505296049 +0000 2024-09-16T03:35:19.635 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-09-16 03:32:33.505296049 +0000 2024-09-16T03:35:19.635 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-09-16 03:32:33.505296049 +0000 2024-09-16T03:35:19.635 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:35:19.698 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-09-16T03:35:19.698 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-09-16T03:35:19.698 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000109456 s, 4.7 MB/s 2024-09-16T03:35:19.699 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:35:19.755 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:19.756 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-16T03:35:19.756 INFO:tasks.ceph:Generating config... 2024-09-16T03:35:19.757 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-16T03:35:19.757 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T03:35:19.757 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-16T03:35:19.757 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T03:35:19.758 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T03:35:19.758 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2024-09-16T03:35:19.758 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T03:35:19.758 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T03:35:19.758 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T03:35:19.758 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T03:35:19.758 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T03:35:19.758 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T03:35:19.758 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T03:35:19.758 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T03:35:19.759 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T03:35:19.759 INFO:tasks.ceph:[osd] osd backoff on degraded = True 2024-09-16T03:35:19.759 INFO:tasks.ceph:[osd] osd backoff on peering = True 2024-09-16T03:35:19.759 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-16T03:35:19.759 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-09-16T03:35:19.759 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T03:35:19.759 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T03:35:19.759 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T03:35:19.759 INFO:tasks.ceph:[osd] osd inject bad map crc probability = 0.1 2024-09-16T03:35:19.759 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-09-16T03:35:19.759 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-16T03:35:19.759 INFO:tasks.ceph:[osd] osd max scrubs = 5 2024-09-16T03:35:19.760 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T03:35:19.760 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T03:35:19.760 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T03:35:19.760 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T03:35:19.760 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T03:35:19.760 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-16T03:35:19.760 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-16T03:35:19.760 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-16T03:35:19.760 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-09-16T03:35:19.760 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-16T03:35:19.760 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-16T03:35:19.760 INFO:tasks.ceph:[global] ms type = async 2024-09-16T03:35:19.760 INFO:tasks.ceph:[global] osd debug inject dispatch delay duration = 0.1 2024-09-16T03:35:19.761 INFO:tasks.ceph:[global] osd debug inject dispatch delay probability = 0.1 2024-09-16T03:35:19.761 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2024-09-16T03:35:19.761 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 2 2024-09-16T03:35:19.761 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 1 2024-09-16T03:35:19.761 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:35:19.761 INFO:tasks.ceph:[global] osd_pg_log_trim_min = 0 2024-09-16T03:35:19.761 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-09-16T03:35:19.762 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-09-16T03:35:19.762 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T03:35:19.762 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T03:35:19.762 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T03:35:19.762 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T03:35:19.762 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T03:35:19.762 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-16T03:35:19.762 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T03:35:19.762 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T03:35:19.762 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T03:35:19.762 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T03:35:19.762 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-16T03:35:19.763 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T03:35:19.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T03:35:19.817 INFO:teuthology.orchestra.run.smithi064.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T03:35:19.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T03:35:19.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:35:19.895 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.64:6789]'), ('mon.b', '[v1:172.21.15.112:6789]'), ('mon.c', '[v1:172.21.15.155:6789]')] 2024-09-16T03:35:19.896 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.64:6789],[v1:172.21.15.112:6789],[v1:172.21.15.155:6789]', 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-16T03:35:19.897 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:35:19.897 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T03:35:19.952 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.64:6789]' --addv b '[v1:172.21.15.112:6789]' --addv c '[v1:172.21.15.155:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:20.042 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:20.042 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: generated fsid b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2 2024-09-16T03:35:20.042 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-09-16T03:35:20.042 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 0 2024-09-16T03:35:20.042 INFO:teuthology.orchestra.run.smithi064.stdout:fsid b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2 2024-09-16T03:35:20.042 INFO:teuthology.orchestra.run.smithi064.stdout:last_changed 2024-09-16T03:35:20.041154+0000 2024-09-16T03:35:20.042 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-09-16T03:35:20.041154+0000 2024-09-16T03:35:20.042 INFO:teuthology.orchestra.run.smithi064.stdout:min_mon_release 17 (quincy) 2024-09-16T03:35:20.043 INFO:teuthology.orchestra.run.smithi064.stdout:election_strategy: 3 2024-09-16T03:35:20.043 INFO:teuthology.orchestra.run.smithi064.stdout:0: v1:172.21.15.64:6789/0 mon.a 2024-09-16T03:35:20.043 INFO:teuthology.orchestra.run.smithi064.stdout:1: v1:172.21.15.112:6789/0 mon.b 2024-09-16T03:35:20.043 INFO:teuthology.orchestra.run.smithi064.stdout:2: v1:172.21.15.155:6789/0 mon.c 2024-09-16T03:35:20.043 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T03:35:20.044 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T03:35:20.060 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2... 2024-09-16T03:35:20.061 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:35:20.105 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:35:20.108 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:35:20.111 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:35:20.148 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-09-16T03:35:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2024-09-16T03:35:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:35:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2024-09-16T03:35:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2024-09-16T03:35:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.148 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.149 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2024-09-16T03:35:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2024-09-16T03:35:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2024-09-16T03:35:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:35:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:35:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2024-09-16T03:35:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:35:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2024-09-16T03:35:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:35:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:35:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2024-09-16T03:35:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.150 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:35:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2024-09-16T03:35:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2024-09-16T03:35:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2024-09-16T03:35:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2024-09-16T03:35:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2024-09-16T03:35:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = "[v1:172.21.15.64:6789],[v1:172.21.15.112:6789],[v1:172.21.15.155:6789]" 2024-09-16T03:35:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: mon election default strategy = 3 2024-09-16T03:35:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: ms bind msgr2 = False 2024-09-16T03:35:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: ms type = async 2024-09-16T03:35:20.151 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug inject dispatch delay duration = 0.1 2024-09-16T03:35:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug inject dispatch delay probability = 0.1 2024-09-16T03:35:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: osd_async_recovery_min_cost = 1 2024-09-16T03:35:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: osd_max_pg_log_entries = 2 2024-09-16T03:35:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: osd_min_pg_log_entries = 1 2024-09-16T03:35:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:35:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pg_log_trim_min = 0 2024-09-16T03:35:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_min_size = 2 2024-09-16T03:35:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_size = 3 2024-09-16T03:35:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2 2024-09-16T03:35:20.152 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.152 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2024-09-16T03:35:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2024-09-16T03:35:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2024-09-16T03:35:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 120 2024-09-16T03:35:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:35:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2024-09-16T03:35:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2024-09-16T03:35:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2024-09-16T03:35:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2024-09-16T03:35:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2024-09-16T03:35:20.153 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug trim objects = true 2024-09-16T03:35:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2024-09-16T03:35:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2024-09-16T03:35:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:35:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2024-09-16T03:35:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2024-09-16T03:35:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:35:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2024-09-16T03:35:20.154 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2024-09-16T03:35:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2024-09-16T03:35:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: debug monc = 20 2024-09-16T03:35:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2024-09-16T03:35:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression algorithm = snappy 2024-09-16T03:35:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression mode = aggressive 2024-09-16T03:35:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2024-09-16T03:35:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore write v2 random = True 2024-09-16T03:35:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore zero block detection = True 2024-09-16T03:35:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 20 2024-09-16T03:35:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 20 2024-09-16T03:35:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-09-16T03:35:20.155 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2024-09-16T03:35:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 10 2024-09-16T03:35:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:35:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2024-09-16T03:35:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:35:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: osd backoff on degraded = True 2024-09-16T03:35:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: osd backoff on peering = True 2024-09-16T03:35:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: osd blocked scrub grace period = 3600 2024-09-16T03:35:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug reject backfill probability = 0.3 2024-09-16T03:35:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify cached snaps = True 2024-09-16T03:35:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify missing on start = True 2024-09-16T03:35:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:35:20.156 INFO:teuthology.orchestra.run.smithi064.stdout: osd inject bad map crc probability = 0.1 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: osd max backfills = 3 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: osd max markdown count = 1000 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: osd max scrubs = 5 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock override recovery settings = True 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue = debug_random 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: osd op queue cut off = debug_random 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub min interval = 60 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: osd shutdown pgref assert = True 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: osd snap trim sleep = 2 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-09-16T03:35:20.157 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2024-09-16T03:35:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-09-16T03:35:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-09-16T03:35:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:35:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:35:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2024-09-16T03:35:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.158 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2024-09-16T03:35:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-09-16T03:35:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-09-16T03:35:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2024-09-16T03:35:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-09-16T03:35:20.158 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2024-09-16T03:35:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2024-09-16T03:35:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:35:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2024-09-16T03:35:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2024-09-16T03:35:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:35:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:35:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:35:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:35:20.159 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:35:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:35:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:35:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2024-09-16T03:35:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:35:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:35:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: mon min osdmap epochs = 50 2024-09-16T03:35:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune interval = 2 2024-09-16T03:35:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune min = 15 2024-09-16T03:35:20.160 INFO:teuthology.orchestra.run.smithi064.stdout: mon osdmap full prune txsize = 2 2024-09-16T03:35:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: mon scrub interval = 300 2024-09-16T03:35:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: paxos service trim min = 10 2024-09-16T03:35:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:35:20.161 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2024-09-16T03:35:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2024-09-16T03:35:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2024-09-16T03:35:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2024-09-16T03:35:20.161 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:35:20.162 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:35:20.162 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2024-09-16T03:35:20.162 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.b] 2024-09-16T03:35:20.162 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.c] 2024-09-16T03:35:20.163 INFO:teuthology.orchestra.run.smithi155.stdout:[global] 2024-09-16T03:35:20.163 INFO:teuthology.orchestra.run.smithi155.stdout: chdir = "" 2024-09-16T03:35:20.163 INFO:teuthology.orchestra.run.smithi155.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:35:20.163 INFO:teuthology.orchestra.run.smithi155.stdout: auth supported = cephx 2024-09-16T03:35:20.163 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.163 INFO:teuthology.orchestra.run.smithi155.stdout: filestore xattr use omap = true 2024-09-16T03:35:20.163 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.163 INFO:teuthology.orchestra.run.smithi155.stdout: mon clock drift allowed = 1.000 2024-09-16T03:35:20.164 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.164 INFO:teuthology.orchestra.run.smithi155.stdout: osd crush chooseleaf type = 0 2024-09-16T03:35:20.164 INFO:teuthology.orchestra.run.smithi155.stdout: auth debug = true 2024-09-16T03:35:20.164 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.164 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on old message = true 2024-09-16T03:35:20.164 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on bug = true 2024-09-16T03:35:20.164 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.164 INFO:teuthology.orchestra.run.smithi155.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:35:20.164 INFO:teuthology.orchestra.run.smithi155.stdout: mon pg warn max object skew = 0 2024-09-16T03:35:20.164 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.164 INFO:teuthology.orchestra.run.smithi155.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:35:20.164 INFO:teuthology.orchestra.run.smithi155.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:35:20.164 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.165 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default size = 2 2024-09-16T03:35:20.165 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.165 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow primary affinity = true 2024-09-16T03:35:20.165 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow pg remap = true 2024-09-16T03:35:20.165 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:35:20.165 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:35:20.165 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on no sortbitwise = false 2024-09-16T03:35:20.165 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:35:20.165 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on too few osds = false 2024-09-16T03:35:20.165 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:35:20.165 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:35:20.165 INFO:teuthology.orchestra.run.smithi155.stdout: mon_allow_pool_size_one = true 2024-09-16T03:35:20.165 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: osd default data pool replay window = 5 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: mon allow pool delete = true 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: mon cluster log file level = debug 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: debug asserts on shutdown = true 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: mon health detail to clog = false 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: mon host = "[v1:172.21.15.64:6789],[v1:172.21.15.112:6789],[v1:172.21.15.155:6789]" 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: mon election default strategy = 3 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: ms bind msgr2 = False 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: ms type = async 2024-09-16T03:35:20.166 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug inject dispatch delay duration = 0.1 2024-09-16T03:35:20.167 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug inject dispatch delay probability = 0.1 2024-09-16T03:35:20.167 INFO:teuthology.orchestra.run.smithi155.stdout: osd_async_recovery_min_cost = 1 2024-09-16T03:35:20.167 INFO:teuthology.orchestra.run.smithi155.stdout: osd_max_pg_log_entries = 2 2024-09-16T03:35:20.167 INFO:teuthology.orchestra.run.smithi155.stdout: osd_min_pg_log_entries = 1 2024-09-16T03:35:20.167 INFO:teuthology.orchestra.run.smithi155.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:35:20.167 INFO:teuthology.orchestra.run.smithi155.stdout: osd_pg_log_trim_min = 0 2024-09-16T03:35:20.167 INFO:teuthology.orchestra.run.smithi155.stdout: osd_pool_default_min_size = 2 2024-09-16T03:35:20.167 INFO:teuthology.orchestra.run.smithi155.stdout: osd_pool_default_size = 3 2024-09-16T03:35:20.167 INFO:teuthology.orchestra.run.smithi155.stdout: fsid = b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2 2024-09-16T03:35:20.167 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.167 INFO:teuthology.orchestra.run.smithi155.stdout:[osd] 2024-09-16T03:35:20.167 INFO:teuthology.orchestra.run.smithi155.stdout: osd journal size = 100 2024-09-16T03:35:20.168 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.168 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub load threshold = 5.0 2024-09-16T03:35:20.168 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub max interval = 120 2024-09-16T03:35:20.168 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:35:20.168 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.168 INFO:teuthology.orchestra.run.smithi155.stdout: osd recover clone overlap = true 2024-09-16T03:35:20.168 INFO:teuthology.orchestra.run.smithi155.stdout: osd recovery max chunk = 1048576 2024-09-16T03:35:20.168 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.168 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug shutdown = true 2024-09-16T03:35:20.168 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug op order = true 2024-09-16T03:35:20.168 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify stray on activate = true 2024-09-16T03:35:20.168 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug trim objects = true 2024-09-16T03:35:20.168 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: osd open classes on start = true 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug pg log writeout = true 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: osd map max advance = 10 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: journal zero on create = true 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: filestore apply finisher threads = 3 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: bdev debug aio = true 2024-09-16T03:35:20.169 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug misdirected ops = true 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: debug monc = 20 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore block size = 96636764160 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore compression algorithm = snappy 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore compression mode = aggressive 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore fsck on mount = True 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore write v2 random = True 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore zero block detection = True 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluefs = 20 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluestore = 20 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: debug osd = 20 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: debug rocksdb = 10 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:35:20.170 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd full ratio = 0.9 2024-09-16T03:35:20.171 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:35:20.171 INFO:teuthology.orchestra.run.smithi155.stdout: osd backoff on degraded = True 2024-09-16T03:35:20.171 INFO:teuthology.orchestra.run.smithi155.stdout: osd backoff on peering = True 2024-09-16T03:35:20.171 INFO:teuthology.orchestra.run.smithi155.stdout: osd blocked scrub grace period = 3600 2024-09-16T03:35:20.171 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug reject backfill probability = 0.3 2024-09-16T03:35:20.171 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify cached snaps = True 2024-09-16T03:35:20.171 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify missing on start = True 2024-09-16T03:35:20.171 INFO:teuthology.orchestra.run.smithi155.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:35:20.171 INFO:teuthology.orchestra.run.smithi155.stdout: osd inject bad map crc probability = 0.1 2024-09-16T03:35:20.171 INFO:teuthology.orchestra.run.smithi155.stdout: osd max backfills = 3 2024-09-16T03:35:20.172 INFO:teuthology.orchestra.run.smithi155.stdout: osd max markdown count = 1000 2024-09-16T03:35:20.172 INFO:teuthology.orchestra.run.smithi155.stdout: osd max scrubs = 5 2024-09-16T03:35:20.172 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock override recovery settings = True 2024-09-16T03:35:20.172 INFO:teuthology.orchestra.run.smithi155.stdout: osd objectstore = bluestore 2024-09-16T03:35:20.172 INFO:teuthology.orchestra.run.smithi155.stdout: osd op queue = debug_random 2024-09-16T03:35:20.172 INFO:teuthology.orchestra.run.smithi155.stdout: osd op queue cut off = debug_random 2024-09-16T03:35:20.172 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub min interval = 60 2024-09-16T03:35:20.172 INFO:teuthology.orchestra.run.smithi155.stdout: osd shutdown pgref assert = True 2024-09-16T03:35:20.172 INFO:teuthology.orchestra.run.smithi155.stdout: osd snap trim sleep = 2 2024-09-16T03:35:20.173 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.173 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr] 2024-09-16T03:35:20.173 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-09-16T03:35:20.173 INFO:teuthology.orchestra.run.smithi155.stdout: debug mgr = 20 2024-09-16T03:35:20.173 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-09-16T03:35:20.173 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-09-16T03:35:20.173 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:35:20.174 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:35:20.174 INFO:teuthology.orchestra.run.smithi155.stdout: mgr/telemetry/nag = false 2024-09-16T03:35:20.174 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.174 INFO:teuthology.orchestra.run.smithi155.stdout:[mon] 2024-09-16T03:35:20.174 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-09-16T03:35:20.174 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-09-16T03:35:20.174 INFO:teuthology.orchestra.run.smithi155.stdout: debug paxos = 20 2024-09-16T03:35:20.174 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-09-16T03:35:20.174 INFO:teuthology.orchestra.run.smithi155.stdout: mon data avail warn = 5 2024-09-16T03:35:20.174 INFO:teuthology.orchestra.run.smithi155.stdout: mon mgr mkfs grace = 240 2024-09-16T03:35:20.175 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:35:20.175 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd reporter subtree level = osd 2024-09-16T03:35:20.175 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd prime pg temp = true 2024-09-16T03:35:20.175 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:35:20.175 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.175 INFO:teuthology.orchestra.run.smithi155.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:35:20.175 INFO:teuthology.orchestra.run.smithi155.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:35:20.175 INFO:teuthology.orchestra.run.smithi155.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:35:20.175 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.176 INFO:teuthology.orchestra.run.smithi155.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:35:20.176 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:35:20.176 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:35:20.176 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.176 INFO:teuthology.orchestra.run.smithi155.stdout: # 1m isn't quite enough 2024-09-16T03:35:20.176 INFO:teuthology.orchestra.run.smithi155.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:35:20.176 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.176 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:35:20.176 INFO:teuthology.orchestra.run.smithi155.stdout: mon min osdmap epochs = 50 2024-09-16T03:35:20.176 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune interval = 2 2024-09-16T03:35:20.176 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune min = 15 2024-09-16T03:35:20.177 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune txsize = 2 2024-09-16T03:35:20.177 INFO:teuthology.orchestra.run.smithi155.stdout: mon scrub interval = 300 2024-09-16T03:35:20.177 INFO:teuthology.orchestra.run.smithi155.stdout: paxos service trim min = 10 2024-09-16T03:35:20.177 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-09-16T03:35:20.177 INFO:teuthology.orchestra.run.smithi155.stdout:[client] 2024-09-16T03:35:20.177 INFO:teuthology.orchestra.run.smithi155.stdout: rgw cache enabled = true 2024-09-16T03:35:20.177 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable ops log = true 2024-09-16T03:35:20.177 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable usage log = true 2024-09-16T03:35:20.177 INFO:teuthology.orchestra.run.smithi155.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:35:20.177 INFO:teuthology.orchestra.run.smithi155.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:35:20.177 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.a] 2024-09-16T03:35:20.177 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.b] 2024-09-16T03:35:20.177 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.c] 2024-09-16T03:35:20.178 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2024-09-16T03:35:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2024-09-16T03:35:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:35:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2024-09-16T03:35:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2024-09-16T03:35:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2024-09-16T03:35:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2024-09-16T03:35:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2024-09-16T03:35:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2024-09-16T03:35:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2024-09-16T03:35:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:35:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2024-09-16T03:35:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:35:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:35:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2024-09-16T03:35:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2024-09-16T03:35:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2024-09-16T03:35:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:35:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:35:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2024-09-16T03:35:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:35:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2024-09-16T03:35:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = "[v1:172.21.15.64:6789],[v1:172.21.15.112:6789],[v1:172.21.15.155:6789]" 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: mon election default strategy = 3 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: ms bind msgr2 = False 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: ms type = async 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug inject dispatch delay duration = 0.1 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug inject dispatch delay probability = 0.1 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: osd_async_recovery_min_cost = 1 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_pg_log_entries = 2 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: osd_min_pg_log_entries = 1 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pg_log_trim_min = 0 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_min_size = 2 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_size = 3 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2 2024-09-16T03:35:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.183 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2024-09-16T03:35:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2024-09-16T03:35:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2024-09-16T03:35:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 120 2024-09-16T03:35:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:35:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2024-09-16T03:35:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2024-09-16T03:35:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2024-09-16T03:35:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2024-09-16T03:35:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2024-09-16T03:35:20.184 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug trim objects = true 2024-09-16T03:35:20.184 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.184 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2024-09-16T03:35:20.184 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2024-09-16T03:35:20.184 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.184 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:35:20.184 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.184 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2024-09-16T03:35:20.184 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.184 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2024-09-16T03:35:20.184 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.185 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:35:20.185 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2024-09-16T03:35:20.185 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.185 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2024-09-16T03:35:20.185 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2024-09-16T03:35:20.185 INFO:teuthology.orchestra.run.smithi112.stdout: debug monc = 20 2024-09-16T03:35:20.185 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore block size = 96636764160 2024-09-16T03:35:20.185 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore compression algorithm = snappy 2024-09-16T03:35:20.185 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore compression mode = aggressive 2024-09-16T03:35:20.185 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore fsck on mount = True 2024-09-16T03:35:20.185 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore write v2 random = True 2024-09-16T03:35:20.185 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore zero block detection = True 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluefs = 20 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluestore = 20 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: debug rocksdb = 10 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd full ratio = 0.9 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: osd backoff on degraded = True 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: osd backoff on peering = True 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: osd blocked scrub grace period = 3600 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug reject backfill probability = 0.3 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify cached snaps = True 2024-09-16T03:35:20.186 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify missing on start = True 2024-09-16T03:35:20.187 INFO:teuthology.orchestra.run.smithi112.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:35:20.187 INFO:teuthology.orchestra.run.smithi112.stdout: osd inject bad map crc probability = 0.1 2024-09-16T03:35:20.187 INFO:teuthology.orchestra.run.smithi112.stdout: osd max backfills = 3 2024-09-16T03:35:20.187 INFO:teuthology.orchestra.run.smithi112.stdout: osd max markdown count = 1000 2024-09-16T03:35:20.187 INFO:teuthology.orchestra.run.smithi112.stdout: osd max scrubs = 5 2024-09-16T03:35:20.187 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock override recovery settings = True 2024-09-16T03:35:20.187 INFO:teuthology.orchestra.run.smithi112.stdout: osd objectstore = bluestore 2024-09-16T03:35:20.187 INFO:teuthology.orchestra.run.smithi112.stdout: osd op queue = debug_random 2024-09-16T03:35:20.187 INFO:teuthology.orchestra.run.smithi112.stdout: osd op queue cut off = debug_random 2024-09-16T03:35:20.187 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub min interval = 60 2024-09-16T03:35:20.187 INFO:teuthology.orchestra.run.smithi112.stdout: osd shutdown pgref assert = True 2024-09-16T03:35:20.187 INFO:teuthology.orchestra.run.smithi112.stdout: osd snap trim sleep = 2 2024-09-16T03:35:20.188 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.188 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2024-09-16T03:35:20.188 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-16T03:35:20.188 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2024-09-16T03:35:20.188 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-09-16T03:35:20.188 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-09-16T03:35:20.188 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:35:20.188 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:35:20.188 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2024-09-16T03:35:20.188 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.188 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2024-09-16T03:35:20.188 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-16T03:35:20.188 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.189 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: mon min osdmap epochs = 50 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: mon osdmap full prune interval = 2 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: mon osdmap full prune min = 15 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: mon osdmap full prune txsize = 2 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: mon scrub interval = 300 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: paxos service trim min = 10 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-16T03:35:20.190 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2024-09-16T03:35:20.191 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2024-09-16T03:35:20.191 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2024-09-16T03:35:20.191 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2024-09-16T03:35:20.191 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:35:20.191 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:35:20.191 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2024-09-16T03:35:20.191 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.b] 2024-09-16T03:35:20.191 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.c] 2024-09-16T03:35:20.192 INFO:teuthology.orchestra.run.smithi157.stdout:[global] 2024-09-16T03:35:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: chdir = "" 2024-09-16T03:35:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:35:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: auth supported = cephx 2024-09-16T03:35:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: filestore xattr use omap = true 2024-09-16T03:35:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: mon clock drift allowed = 1.000 2024-09-16T03:35:20.192 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.193 INFO:teuthology.orchestra.run.smithi157.stdout: osd crush chooseleaf type = 0 2024-09-16T03:35:20.193 INFO:teuthology.orchestra.run.smithi157.stdout: auth debug = true 2024-09-16T03:35:20.193 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.193 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on old message = true 2024-09-16T03:35:20.193 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on bug = true 2024-09-16T03:35:20.193 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.193 INFO:teuthology.orchestra.run.smithi157.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:35:20.193 INFO:teuthology.orchestra.run.smithi157.stdout: mon pg warn max object skew = 0 2024-09-16T03:35:20.193 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.194 INFO:teuthology.orchestra.run.smithi157.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:35:20.194 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:35:20.194 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.194 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default size = 2 2024-09-16T03:35:20.194 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.194 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow primary affinity = true 2024-09-16T03:35:20.194 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow pg remap = true 2024-09-16T03:35:20.194 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:35:20.194 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:35:20.195 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on no sortbitwise = false 2024-09-16T03:35:20.195 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:35:20.195 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on too few osds = false 2024-09-16T03:35:20.195 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:35:20.195 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:35:20.195 INFO:teuthology.orchestra.run.smithi157.stdout: mon_allow_pool_size_one = true 2024-09-16T03:35:20.195 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.195 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:35:20.196 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.196 INFO:teuthology.orchestra.run.smithi157.stdout: osd default data pool replay window = 5 2024-09-16T03:35:20.196 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.196 INFO:teuthology.orchestra.run.smithi157.stdout: mon allow pool delete = true 2024-09-16T03:35:20.196 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.196 INFO:teuthology.orchestra.run.smithi157.stdout: mon cluster log file level = debug 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: debug asserts on shutdown = true 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: mon health detail to clog = false 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: mon host = "[v1:172.21.15.64:6789],[v1:172.21.15.112:6789],[v1:172.21.15.155:6789]" 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: mon election default strategy = 3 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: ms bind msgr2 = False 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: ms type = async 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug inject dispatch delay duration = 0.1 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug inject dispatch delay probability = 0.1 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: osd_async_recovery_min_cost = 1 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: osd_max_pg_log_entries = 2 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: osd_min_pg_log_entries = 1 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pg_log_trim_min = 0 2024-09-16T03:35:20.197 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_min_size = 2 2024-09-16T03:35:20.198 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_size = 3 2024-09-16T03:35:20.198 INFO:teuthology.orchestra.run.smithi157.stdout: fsid = b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2 2024-09-16T03:35:20.198 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.198 INFO:teuthology.orchestra.run.smithi157.stdout:[osd] 2024-09-16T03:35:20.198 INFO:teuthology.orchestra.run.smithi157.stdout: osd journal size = 100 2024-09-16T03:35:20.198 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.198 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub load threshold = 5.0 2024-09-16T03:35:20.198 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub max interval = 120 2024-09-16T03:35:20.198 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:35:20.198 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.198 INFO:teuthology.orchestra.run.smithi157.stdout: osd recover clone overlap = true 2024-09-16T03:35:20.198 INFO:teuthology.orchestra.run.smithi157.stdout: osd recovery max chunk = 1048576 2024-09-16T03:35:20.198 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug shutdown = true 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug op order = true 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify stray on activate = true 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug trim objects = true 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: osd open classes on start = true 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug pg log writeout = true 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: osd map max advance = 10 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: journal zero on create = true 2024-09-16T03:35:20.199 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.200 INFO:teuthology.orchestra.run.smithi157.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:35:20.200 INFO:teuthology.orchestra.run.smithi157.stdout: filestore apply finisher threads = 3 2024-09-16T03:35:20.200 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.200 INFO:teuthology.orchestra.run.smithi157.stdout: bdev debug aio = true 2024-09-16T03:35:20.200 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug misdirected ops = true 2024-09-16T03:35:20.200 INFO:teuthology.orchestra.run.smithi157.stdout: debug monc = 20 2024-09-16T03:35:20.200 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore block size = 96636764160 2024-09-16T03:35:20.200 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore compression algorithm = snappy 2024-09-16T03:35:20.200 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore compression mode = aggressive 2024-09-16T03:35:20.200 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore fsck on mount = True 2024-09-16T03:35:20.201 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore write v2 random = True 2024-09-16T03:35:20.201 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore zero block detection = True 2024-09-16T03:35:20.201 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluefs = 20 2024-09-16T03:35:20.201 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluestore = 20 2024-09-16T03:35:20.201 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-09-16T03:35:20.201 INFO:teuthology.orchestra.run.smithi157.stdout: debug osd = 20 2024-09-16T03:35:20.201 INFO:teuthology.orchestra.run.smithi157.stdout: debug rocksdb = 10 2024-09-16T03:35:20.201 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:35:20.201 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd full ratio = 0.9 2024-09-16T03:35:20.201 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:35:20.201 INFO:teuthology.orchestra.run.smithi157.stdout: osd backoff on degraded = True 2024-09-16T03:35:20.201 INFO:teuthology.orchestra.run.smithi157.stdout: osd backoff on peering = True 2024-09-16T03:35:20.201 INFO:teuthology.orchestra.run.smithi157.stdout: osd blocked scrub grace period = 3600 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug reject backfill probability = 0.3 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify cached snaps = True 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify missing on start = True 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd inject bad map crc probability = 0.1 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd max backfills = 3 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd max markdown count = 1000 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd max scrubs = 5 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock override recovery settings = True 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd objectstore = bluestore 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd op queue = debug_random 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd op queue cut off = debug_random 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub min interval = 60 2024-09-16T03:35:20.202 INFO:teuthology.orchestra.run.smithi157.stdout: osd shutdown pgref assert = True 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout: osd snap trim sleep = 2 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr] 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout: debug mgr = 20 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout: mgr/telemetry/nag = false 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout:[mon] 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-09-16T03:35:20.203 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-09-16T03:35:20.204 INFO:teuthology.orchestra.run.smithi157.stdout: debug paxos = 20 2024-09-16T03:35:20.204 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-09-16T03:35:20.204 INFO:teuthology.orchestra.run.smithi157.stdout: mon data avail warn = 5 2024-09-16T03:35:20.204 INFO:teuthology.orchestra.run.smithi157.stdout: mon mgr mkfs grace = 240 2024-09-16T03:35:20.204 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:35:20.204 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd reporter subtree level = osd 2024-09-16T03:35:20.204 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd prime pg temp = true 2024-09-16T03:35:20.204 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:35:20.204 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.204 INFO:teuthology.orchestra.run.smithi157.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:35:20.204 INFO:teuthology.orchestra.run.smithi157.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:35:20.204 INFO:teuthology.orchestra.run.smithi157.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:35:20.204 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: # 1m isn't quite enough 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: mon min osdmap epochs = 50 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune interval = 2 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune min = 15 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: mon osdmap full prune txsize = 2 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: mon scrub interval = 300 2024-09-16T03:35:20.205 INFO:teuthology.orchestra.run.smithi157.stdout: paxos service trim min = 10 2024-09-16T03:35:20.206 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-16T03:35:20.206 INFO:teuthology.orchestra.run.smithi157.stdout:[client] 2024-09-16T03:35:20.206 INFO:teuthology.orchestra.run.smithi157.stdout: rgw cache enabled = true 2024-09-16T03:35:20.206 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable ops log = true 2024-09-16T03:35:20.206 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable usage log = true 2024-09-16T03:35:20.206 INFO:teuthology.orchestra.run.smithi157.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:35:20.206 INFO:teuthology.orchestra.run.smithi157.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:35:20.206 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.a] 2024-09-16T03:35:20.206 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.b] 2024-09-16T03:35:20.206 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.c] 2024-09-16T03:35:20.209 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T03:35:20.209 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T03:35:20.305 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T03:35:20.305 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:35:20.305 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T03:35:20.321 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:35:20.321 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T03:35:20.376 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2024-09-16T03:35:20.377 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:35:20.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:35:20.377 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:35:20.452 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:35:20.453 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:20.509 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2024-09-16T03:35:20.510 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:35:20.510 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:35:20.510 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:35:20.551 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:35:20.551 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:20.608 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi155.front.sepia.ceph.com 2024-09-16T03:35:20.608 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:35:20.608 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:35:20.608 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:35:20.650 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:35:20.650 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:20.709 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi157.front.sepia.ceph.com 2024-09-16T03:35:20.710 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:35:20.710 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:35:20.710 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:35:20.745 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:35:20.745 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:20.801 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T03:35:20.801 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T03:35:20.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T03:35:20.854 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T03:35:20.857 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T03:35:20.915 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T03:35:20.920 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T03:35:20.920 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T03:35:20.921 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T03:35:20.962 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T03:35:20.975 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T03:35:20.975 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-16T03:35:20.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T03:35:21.002 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:21.002 INFO:tasks.ceph:role: osd.0 2024-09-16T03:35:21.002 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-16T03:35:21.002 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:35:21.075 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:35:21.075 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:35:21.075 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:35:21.075 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:35:21.075 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:35:21.075 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:35:21.076 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:35:21.076 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:35:21.076 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:35:21.076 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:35:21.593 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-16T03:35:21.596 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-16T03:35:21.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T03:35:21.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T03:35:21.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T03:35:21.838 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:21.838 INFO:tasks.ceph:role: osd.4 2024-09-16T03:35:21.838 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-16T03:35:21.838 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:35:21.913 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:35:21.913 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:35:21.914 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:35:21.914 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:35:21.914 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:35:21.914 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:35:21.914 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:35:21.914 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:35:21.914 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:35:21.914 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:35:22.356 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-16T03:35:22.359 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-16T03:35:22.359 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T03:35:22.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T03:35:22.488 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-16T03:35:22.555 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:22.555 INFO:tasks.ceph:role: osd.8 2024-09-16T03:35:22.555 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-16T03:35:22.555 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:35:22.630 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:35:22.630 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:35:22.630 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:35:22.630 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:35:22.630 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:35:22.630 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:35:22.630 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:35:22.630 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:35:22.630 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:35:22.630 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:35:23.114 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-16T03:35:23.116 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-16T03:35:23.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-09-16T03:35:23.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-16T03:35:23.295 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-09-16T03:35:23.362 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:23.362 INFO:tasks.ceph:role: osd.12 2024-09-16T03:35:23.362 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-16T03:35:23.362 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T03:35:23.434 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:35:23.435 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:35:23.435 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:35:23.435 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:35:23.435 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:35:23.435 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:35:23.435 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:35:23.435 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:35:23.435 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:35:23.435 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:35:23.950 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-16T03:35:23.952 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-16T03:35:23.952 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-09-16T03:35:24.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-09-16T03:35:24.137 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:24.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:24.237+0000 7ff712351640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T03:35:24.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:24.237+0000 7ff712351640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T03:35:24.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:24.237+0000 7ff712351640 -1 bdev(0x5645ac3cf800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T03:35:24.239 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:24.237+0000 7ff712351640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T03:35:27.394 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T03:35:27.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:27.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:27.519+0000 7f3475a24640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T03:35:27.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:27.519+0000 7f3475a24640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T03:35:27.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:27.519+0000 7f3475a24640 -1 bdev(0x5562de7b9800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T03:35:27.521 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:27.519+0000 7f3475a24640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T03:35:30.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T03:35:30.710 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:30.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:30.802+0000 7f5def811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-16T03:35:30.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:30.802+0000 7f5def811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-16T03:35:30.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:30.802+0000 7f5def811640 -1 bdev(0x556284d83800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-16T03:35:30.805 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:30.802+0000 7f5def811640 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-16T03:35:33.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-16T03:35:33.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:34.078 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:34.076+0000 7f5b4efb2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-09-16T03:35:34.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:34.076+0000 7f5b4efb2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-09-16T03:35:34.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:34.077+0000 7f5b4efb2640 -1 bdev(0x56460fe29800 /var/lib/ceph/osd/ceph-12/block) open stat got: (1) Operation not permitted 2024-09-16T03:35:34.079 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-16T03:35:34.077+0000 7f5b4efb2640 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-09-16T03:35:36.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-09-16T03:35:37.016 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T03:35:37.047 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:37.047 INFO:tasks.ceph:role: osd.1 2024-09-16T03:35:37.047 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-16T03:35:37.047 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:35:37.123 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:35:37.123 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:35:37.123 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:35:37.123 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:35:37.123 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:35:37.123 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:35:37.123 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:35:37.124 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:35:37.124 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:35:37.124 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:35:37.639 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-09-16T03:35:37.641 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-16T03:35:37.641 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-09-16T03:35:37.763 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T03:35:37.795 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T03:35:37.861 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:37.861 INFO:tasks.ceph:role: osd.5 2024-09-16T03:35:37.861 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-16T03:35:37.861 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:35:37.932 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:35:37.932 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:35:37.933 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:35:37.933 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:35:37.933 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:35:37.933 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:35:37.933 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:35:37.933 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:35:37.933 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:35:37.933 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:35:38.396 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-09-16T03:35:38.398 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-16T03:35:38.398 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-16T03:35:38.483 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T03:35:38.512 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-16T03:35:38.576 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:38.577 INFO:tasks.ceph:role: osd.9 2024-09-16T03:35:38.577 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-16T03:35:38.577 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:35:38.649 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:35:38.649 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:35:38.649 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:35:38.649 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:35:38.649 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:35:38.649 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:35:38.649 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:35:38.649 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:35:38.649 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:35:38.649 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:35:39.146 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-09-16T03:35:39.149 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-16T03:35:39.149 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-09-16T03:35:39.289 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-16T03:35:39.319 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-09-16T03:35:39.385 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:39.385 INFO:tasks.ceph:role: osd.13 2024-09-16T03:35:39.385 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-16T03:35:39.385 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T03:35:39.458 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:35:39.458 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:35:39.458 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:35:39.458 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:35:39.458 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:35:39.458 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:35:39.458 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:35:39.459 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:35:39.459 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:35:39.459 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:35:39.980 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-09-16T03:35:39.980 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-16T03:35:39.981 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-09-16T03:35:40.089 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-09-16T03:35:40.124 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:40.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:40.224+0000 7f9e58902640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T03:35:40.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:40.224+0000 7f9e58902640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T03:35:40.226 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:40.224+0000 7f9e58902640 -1 bdev(0x55a3b63bd800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T03:35:40.227 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:40.224+0000 7f9e58902640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T03:35:43.136 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T03:35:43.164 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:43.253 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:43.251+0000 7fde3a7a4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T03:35:43.254 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:43.252+0000 7fde3a7a4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T03:35:43.254 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:43.252+0000 7fde3a7a4640 -1 bdev(0x55aeff1fb800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T03:35:43.254 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:43.252+0000 7fde3a7a4640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T03:35:46.898 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T03:35:46.929 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:47.019 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:47.017+0000 7f889a4ef640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-16T03:35:47.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:47.017+0000 7f889a4ef640 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-16T03:35:47.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:47.018+0000 7f889a4ef640 -1 bdev(0x5629addc1800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-16T03:35:47.020 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:47.018+0000 7f889a4ef640 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-16T03:35:49.933 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-16T03:35:49.961 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:50.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:50.047+0000 7fbb5fc14640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-09-16T03:35:50.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:50.047+0000 7fbb5fc14640 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-09-16T03:35:50.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:50.047+0000 7fbb5fc14640 -1 bdev(0x55e05a739800 /var/lib/ceph/osd/ceph-13/block) open stat got: (1) Operation not permitted 2024-09-16T03:35:50.049 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-16T03:35:50.047+0000 7fbb5fc14640 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-09-16T03:35:52.958 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-09-16T03:35:52.987 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T03:35:53.018 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:53.018 INFO:tasks.ceph:role: osd.2 2024-09-16T03:35:53.018 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com 2024-09-16T03:35:53.018 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:35:53.095 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:35:53.095 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:35:53.095 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:35:53.096 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:35:53.096 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:35:53.096 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:35:53.096 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:35:53.096 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:35:53.096 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:35:53.096 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:35:53.608 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-09-16T03:35:53.610 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-09-16T03:35:53.611 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-16T03:35:53.746 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T03:35:53.780 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-16T03:35:53.846 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:53.846 INFO:tasks.ceph:role: osd.6 2024-09-16T03:35:53.847 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com 2024-09-16T03:35:53.847 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:35:53.919 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:35:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:35:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:35:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:35:53.919 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:35:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:35:53.919 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:35:53.919 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:35:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:35:53.919 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:35:54.370 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-09-16T03:35:54.372 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-09-16T03:35:54.372 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-09-16T03:35:54.466 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-16T03:35:54.495 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-16T03:35:54.566 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:54.566 INFO:tasks.ceph:role: osd.10 2024-09-16T03:35:54.566 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com 2024-09-16T03:35:54.566 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:35:54.639 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:35:54.639 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:35:54.639 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:35:54.639 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:35:54.639 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:35:54.639 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:35:54.639 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:35:54.639 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:35:54.639 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:35:54.639 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:35:55.129 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-09-16T03:35:55.130 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-09-16T03:35:55.131 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-16T03:35:55.281 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-16T03:35:55.310 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-09-16T03:35:55.375 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T03:35:55.376 INFO:tasks.ceph:role: osd.14 2024-09-16T03:35:55.376 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com 2024-09-16T03:35:55.376 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T03:35:55.447 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:35:55.447 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:35:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:35:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:35:55.448 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:35:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:35:55.448 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:35:55.448 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:35:55.448 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:35:55.448 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:35:55.942 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-09-16T03:35:55.944 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-09-16T03:35:55.944 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-09-16T03:35:56.086 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-09-16T03:35:56.119 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:56.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:35:56.221+0000 7f9484e17640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T03:35:56.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:35:56.221+0000 7f9484e17640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T03:35:56.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:35:56.221+0000 7f9484e17640 -1 bdev(0x563d897af800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T03:35:56.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:35:56.222+0000 7f9484e17640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T03:35:59.385 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T03:35:59.415 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:35:59.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:35:59.504+0000 7f1a65127640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-16T03:35:59.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:35:59.504+0000 7f1a65127640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-16T03:35:59.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:35:59.504+0000 7f1a65127640 -1 bdev(0x55a4d23a5800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-16T03:35:59.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:35:59.504+0000 7f1a65127640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-16T03:36:03.181 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-16T03:36:03.210 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:36:03.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:36:03.296+0000 7f30fd82e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-16T03:36:03.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:36:03.296+0000 7f30fd82e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-16T03:36:03.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:36:03.296+0000 7f30fd82e640 -1 bdev(0x55f27072d800 /var/lib/ceph/osd/ceph-10/block) open stat got: (1) Operation not permitted 2024-09-16T03:36:03.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:36:03.296+0000 7f30fd82e640 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-16T03:36:06.220 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-16T03:36:06.250 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:36:06.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:36:06.337+0000 7f3bc2c1f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-09-16T03:36:06.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:36:06.337+0000 7f3bc2c1f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-09-16T03:36:06.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:36:06.337+0000 7f3bc2c1f640 -1 bdev(0x5614c412d800 /var/lib/ceph/osd/ceph-14/block) open stat got: (1) Operation not permitted 2024-09-16T03:36:06.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-09-16T03:36:06.337+0000 7f3bc2c1f640 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-09-16T03:36:09.510 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-09-16T03:36:09.539 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T03:36:09.570 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T03:36:09.571 INFO:tasks.ceph:role: osd.3 2024-09-16T03:36:09.571 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-16T03:36:09.571 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:36:09.638 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:36:09.638 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:36:09.638 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:36:09.639 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:36:09.639 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:36:09.639 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:36:09.639 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:36:09.639 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:36:09.639 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:36:09.639 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:36:09.671 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-09-16T03:36:09.673 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-09-16T03:36:09.674 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T03:36:09.744 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T03:36:09.814 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-16T03:36:09.881 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T03:36:09.881 INFO:tasks.ceph:role: osd.7 2024-09-16T03:36:09.881 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-16T03:36:09.881 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:36:09.947 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:36:09.947 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:36:09.947 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:36:09.947 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:36:09.948 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:36:09.948 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:36:09.948 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:36:09.948 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:36:09.948 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:36:09.948 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:36:09.980 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-09-16T03:36:09.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-09-16T03:36:09.983 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-09-16T03:36:10.053 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-16T03:36:10.124 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-16T03:36:10.200 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T03:36:10.200 INFO:tasks.ceph:role: osd.11 2024-09-16T03:36:10.200 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-16T03:36:10.200 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:36:10.284 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:36:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:36:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:36:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:36:10.285 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:36:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:36:10.285 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:36:10.285 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:36:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:36:10.285 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:36:10.317 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-09-16T03:36:10.319 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-09-16T03:36:10.319 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-09-16T03:36:10.363 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-16T03:36:10.436 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-09-16T03:36:10.503 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T03:36:10.503 INFO:tasks.ceph:role: osd.15 2024-09-16T03:36:10.504 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-16T03:36:10.504 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T03:36:10.570 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:36:10.570 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:36:10.570 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:36:10.570 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T03:36:10.570 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:36:10.571 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:36:10.571 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:36:10.571 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T03:36:10.571 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:36:10.571 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:36:10.603 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-09-16T03:36:10.605 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-09-16T03:36:10.605 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-09-16T03:36:10.681 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-09-16T03:36:10.752 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:36:10.848 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:10.846+0000 7f65645c5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T03:36:10.848 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:10.846+0000 7f65645c5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T03:36:10.849 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:10.846+0000 7f65645c5640 -1 bdev(0x5597d9c29800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T03:36:10.849 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:10.846+0000 7f65645c5640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T03:36:13.756 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T03:36:13.786 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:36:13.877 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:13.874+0000 7f3e28c0b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-16T03:36:13.877 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:13.874+0000 7f3e28c0b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-16T03:36:13.877 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:13.874+0000 7f3e28c0b640 -1 bdev(0x563ec9fcd800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-16T03:36:13.877 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:13.874+0000 7f3e28c0b640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-16T03:36:16.800 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T03:36:16.830 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:36:16.922 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:16.917+0000 7fa70fa0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-16T03:36:16.922 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:16.918+0000 7fa70fa0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-16T03:36:16.922 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:16.918+0000 7fa70fa0d640 -1 bdev(0x55dcb7935800 /var/lib/ceph/osd/ceph-11/block) open stat got: (1) Operation not permitted 2024-09-16T03:36:16.922 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:16.918+0000 7fa70fa0d640 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-16T03:36:20.100 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-16T03:36:20.129 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:36:20.220 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:20.217+0000 7fbd091bb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-09-16T03:36:20.220 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:20.217+0000 7fbd091bb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-09-16T03:36:20.220 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:20.217+0000 7fbd091bb640 -1 bdev(0x55890d0ff800 /var/lib/ceph/osd/ceph-15/block) open stat got: (1) Operation not permitted 2024-09-16T03:36:20.220 INFO:teuthology.orchestra.run.smithi157.stderr:2024-09-16T03:36:20.217+0000 7fbd091bb640 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-09-16T03:36:23.387 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-09-16T03:36:23.414 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T03:36:23.415 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:36:23.415 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-16T03:36:23.447 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:36:23.447 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T03:36:23.516 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:36:23.516 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T03:36:23.584 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:36:23.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-16T03:36:23.651 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:36:23.651 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-09-16T03:36:23.720 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:36:23.720 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T03:36:23.751 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:36:23.751 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T03:36:23.819 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:36:23.819 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-16T03:36:23.886 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:36:23.886 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-09-16T03:36:23.954 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:36:23.954 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T03:36:23.985 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:36:23.985 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-16T03:36:24.054 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:36:24.054 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-16T03:36:24.122 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:36:24.122 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-09-16T03:36:24.190 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:36:24.190 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T03:36:24.220 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:36:24.220 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T03:36:24.289 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:36:24.289 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-16T03:36:24.357 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:36:24.357 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-16T03:36:24.425 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:36:24.425 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-09-16T03:36:24.491 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:36:24.492 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T03:36:24.548 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T03:36:24.548 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T03:36:24.551 DEBUG:teuthology.orchestra.run.smithi112:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T03:36:24.554 DEBUG:teuthology.orchestra.run.smithi155:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T03:36:24.573 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.y] 2024-09-16T03:36:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQB4p+dm1rHlMhAAb9hrYO33ZkYCR2v/oLoQog== 2024-09-16T03:36:24.573 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2024-09-16T03:36:24.573 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQB8p+dmcCcwDhAAX8QmvxOGjzuBbYXruZ44Yg== 2024-09-16T03:36:24.573 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.4] 2024-09-16T03:36:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQB/p+dm3bIAHxAAJcUr+Hqh/pOYlHtXIUM5KQ== 2024-09-16T03:36:24.574 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.8] 2024-09-16T03:36:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCCp+dmdTziLxAAclB6UBPH74CMGQeifrc42w== 2024-09-16T03:36:24.574 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.12] 2024-09-16T03:36:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCGp+dmGs+eBBAAeWJSpOekUrbK2zi5N50UWA== 2024-09-16T03:36:24.574 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2024-09-16T03:36:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCMp+dmlXBtDRAA5uUpPdt+UpINKPhroX5soQ== 2024-09-16T03:36:24.574 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.5] 2024-09-16T03:36:24.574 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCPp+dmGwkPDxAAG1jtbe8ltYA5wSdj0RbCZg== 2024-09-16T03:36:24.575 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.9] 2024-09-16T03:36:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCTp+dm5wcaARAAz034htzpP7rQrvSEmGnOIA== 2024-09-16T03:36:24.575 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.13] 2024-09-16T03:36:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCWp+dmrEzfAhAAgT4Z45nIBrSvkfCAZoifWw== 2024-09-16T03:36:24.575 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2024-09-16T03:36:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCcp+dms2xLDRAAbVKfkiMzHSSPU0WTDC6xNg== 2024-09-16T03:36:24.575 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.6] 2024-09-16T03:36:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCfp+dmQpAlHhAA5YC7+EPGfCSFXFIF4cl5xA== 2024-09-16T03:36:24.575 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.10] 2024-09-16T03:36:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCjp+dmFpXBERAAf0QEIaOgHIqUlOvlfBzW2Q== 2024-09-16T03:36:24.575 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.14] 2024-09-16T03:36:24.575 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCmp+dm65gzFBAAaypS1XZT39lPcketgTF6yQ== 2024-09-16T03:36:24.576 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2024-09-16T03:36:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQB4p+dmEoyFNhAAzD7TolrE6ROWm0RmMnX2lQ== 2024-09-16T03:36:24.576 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2024-09-16T03:36:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCqp+dm8x56MhAAM7yGvVy4A0yy8xzP3COFJw== 2024-09-16T03:36:24.576 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.7] 2024-09-16T03:36:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCtp+dmjxU0NBAADC9fjwX9DOnm38uMT9Jn7w== 2024-09-16T03:36:24.576 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.11] 2024-09-16T03:36:24.576 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCwp+dm/gPRNhAA20W5pcPoxlft8Hj6D7i+xQ== 2024-09-16T03:36:24.576 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.15] 2024-09-16T03:36:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQC0p+dm4L4ODRAA1O0eD3vf6jaxYCjosaa8og== 2024-09-16T03:36:24.577 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-09-16T03:36:24.577 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQB4p+dmp0NPORAAw7V1ajINvD4+IyWgzr2aCg== 2024-09-16T03:36:24.579 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.y] 2024-09-16T03:36:24.579 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQB4p+dm1rHlMhAAb9hrYO33ZkYCR2v/oLoQog== 2024-09-16T03:36:24.579 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2024-09-16T03:36:24.579 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQB8p+dmcCcwDhAAX8QmvxOGjzuBbYXruZ44Yg== 2024-09-16T03:36:24.579 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.4] 2024-09-16T03:36:24.579 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQB/p+dm3bIAHxAAJcUr+Hqh/pOYlHtXIUM5KQ== 2024-09-16T03:36:24.579 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.8] 2024-09-16T03:36:24.580 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCCp+dmdTziLxAAclB6UBPH74CMGQeifrc42w== 2024-09-16T03:36:24.580 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.12] 2024-09-16T03:36:24.580 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCGp+dmGs+eBBAAeWJSpOekUrbK2zi5N50UWA== 2024-09-16T03:36:24.580 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2024-09-16T03:36:24.580 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCMp+dmlXBtDRAA5uUpPdt+UpINKPhroX5soQ== 2024-09-16T03:36:24.580 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.5] 2024-09-16T03:36:24.580 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCPp+dmGwkPDxAAG1jtbe8ltYA5wSdj0RbCZg== 2024-09-16T03:36:24.580 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.9] 2024-09-16T03:36:24.580 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCTp+dm5wcaARAAz034htzpP7rQrvSEmGnOIA== 2024-09-16T03:36:24.580 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.13] 2024-09-16T03:36:24.580 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCWp+dmrEzfAhAAgT4Z45nIBrSvkfCAZoifWw== 2024-09-16T03:36:24.581 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2024-09-16T03:36:24.581 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCcp+dms2xLDRAAbVKfkiMzHSSPU0WTDC6xNg== 2024-09-16T03:36:24.581 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.6] 2024-09-16T03:36:24.581 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCfp+dmQpAlHhAA5YC7+EPGfCSFXFIF4cl5xA== 2024-09-16T03:36:24.581 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.10] 2024-09-16T03:36:24.581 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCjp+dmFpXBERAAf0QEIaOgHIqUlOvlfBzW2Q== 2024-09-16T03:36:24.581 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.14] 2024-09-16T03:36:24.581 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCmp+dm65gzFBAAaypS1XZT39lPcketgTF6yQ== 2024-09-16T03:36:24.581 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.x] 2024-09-16T03:36:24.582 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQB4p+dmEoyFNhAAzD7TolrE6ROWm0RmMnX2lQ== 2024-09-16T03:36:24.582 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.3] 2024-09-16T03:36:24.582 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCqp+dm8x56MhAAM7yGvVy4A0yy8xzP3COFJw== 2024-09-16T03:36:24.582 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.7] 2024-09-16T03:36:24.582 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCtp+dmjxU0NBAADC9fjwX9DOnm38uMT9Jn7w== 2024-09-16T03:36:24.582 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.11] 2024-09-16T03:36:24.582 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCwp+dm/gPRNhAA20W5pcPoxlft8Hj6D7i+xQ== 2024-09-16T03:36:24.582 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.15] 2024-09-16T03:36:24.582 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQC0p+dm4L4ODRAA1O0eD3vf6jaxYCjosaa8og== 2024-09-16T03:36:24.582 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2024-09-16T03:36:24.583 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQB4p+dmp0NPORAAw7V1ajINvD4+IyWgzr2aCg== 2024-09-16T03:36:24.584 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.y] 2024-09-16T03:36:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQB4p+dm1rHlMhAAb9hrYO33ZkYCR2v/oLoQog== 2024-09-16T03:36:24.585 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.0] 2024-09-16T03:36:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQB8p+dmcCcwDhAAX8QmvxOGjzuBbYXruZ44Yg== 2024-09-16T03:36:24.585 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.4] 2024-09-16T03:36:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQB/p+dm3bIAHxAAJcUr+Hqh/pOYlHtXIUM5KQ== 2024-09-16T03:36:24.585 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.8] 2024-09-16T03:36:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQCCp+dmdTziLxAAclB6UBPH74CMGQeifrc42w== 2024-09-16T03:36:24.585 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.12] 2024-09-16T03:36:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQCGp+dmGs+eBBAAeWJSpOekUrbK2zi5N50UWA== 2024-09-16T03:36:24.585 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.1] 2024-09-16T03:36:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQCMp+dmlXBtDRAA5uUpPdt+UpINKPhroX5soQ== 2024-09-16T03:36:24.585 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.5] 2024-09-16T03:36:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQCPp+dmGwkPDxAAG1jtbe8ltYA5wSdj0RbCZg== 2024-09-16T03:36:24.586 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.9] 2024-09-16T03:36:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQCTp+dm5wcaARAAz034htzpP7rQrvSEmGnOIA== 2024-09-16T03:36:24.586 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.13] 2024-09-16T03:36:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQCWp+dmrEzfAhAAgT4Z45nIBrSvkfCAZoifWw== 2024-09-16T03:36:24.586 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.2] 2024-09-16T03:36:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQCcp+dms2xLDRAAbVKfkiMzHSSPU0WTDC6xNg== 2024-09-16T03:36:24.586 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.6] 2024-09-16T03:36:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQCfp+dmQpAlHhAA5YC7+EPGfCSFXFIF4cl5xA== 2024-09-16T03:36:24.586 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.10] 2024-09-16T03:36:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQCjp+dmFpXBERAAf0QEIaOgHIqUlOvlfBzW2Q== 2024-09-16T03:36:24.586 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.14] 2024-09-16T03:36:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQCmp+dm65gzFBAAaypS1XZT39lPcketgTF6yQ== 2024-09-16T03:36:24.586 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.x] 2024-09-16T03:36:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQB4p+dmEoyFNhAAzD7TolrE6ROWm0RmMnX2lQ== 2024-09-16T03:36:24.587 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.3] 2024-09-16T03:36:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQCqp+dm8x56MhAAM7yGvVy4A0yy8xzP3COFJw== 2024-09-16T03:36:24.587 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.7] 2024-09-16T03:36:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQCtp+dmjxU0NBAADC9fjwX9DOnm38uMT9Jn7w== 2024-09-16T03:36:24.587 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.11] 2024-09-16T03:36:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQCwp+dm/gPRNhAA20W5pcPoxlft8Hj6D7i+xQ== 2024-09-16T03:36:24.587 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.15] 2024-09-16T03:36:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQC0p+dm4L4ODRAA1O0eD3vf6jaxYCjosaa8og== 2024-09-16T03:36:24.587 INFO:teuthology.orchestra.run.smithi155.stdout:[client.0] 2024-09-16T03:36:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQB4p+dmp0NPORAAw7V1ajINvD4+IyWgzr2aCg== 2024-09-16T03:36:24.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:36:24.620 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:36:24.627 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:36:24.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.687 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.690 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.747 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.749 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.801 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.803 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.806 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.857 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.859 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.861 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.914 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.917 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.968 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.970 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:24.972 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.023 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.025 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.028 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.082 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.084 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.136 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.139 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.141 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.192 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.196 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.198 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.249 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.251 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.257 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.310 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.313 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.364 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:36:25.366 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:36:25.368 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:36:25.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.421 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.424 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.476 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.478 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.480 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.534 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.537 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.592 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.594 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:36:25.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:36:25.649 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:36:25.651 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:36:25.705 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T03:36:25.705 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T03:36:25.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:36:25.829 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T03:36:25.856 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T03:36:25.880 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:36:25.976 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T03:36:26.001 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T03:36:26.027 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:36:26.123 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T03:36:26.148 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:36:26.151 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:36:26.153 DEBUG:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:36:26.203 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T03:36:26.204 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T03:36:26.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T03:36:26.207 INFO:tasks.ceph.mon.a:Started 2024-09-16T03:36:26.207 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:36:26.207 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:36:26.209 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:36:26.210 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:36:26.210 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:36:26.246 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:36:26.246 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T03:36:26.246 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-16T03:36:26.246 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-16T03:36:26.249 INFO:tasks.ceph.mgr.y:Started 2024-09-16T03:36:26.249 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T03:36:26.249 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T03:36:26.251 INFO:tasks.ceph.mgr.x:Started 2024-09-16T03:36:26.252 DEBUG:tasks.ceph:set 0 configs 2024-09-16T03:36:26.252 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T03:36:26.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T03:36:37.416 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-09-16T03:36:37.429 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T03:36:37.429 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T03:36:37.429 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:36:37.429 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T03:36:37.456 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:36:37.456 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T03:36:37.522 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:36:37.522 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-16T03:36:37.588 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-16T03:36:37.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-09-16T03:36:37.654 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:36:37.654 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T03:36:37.685 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:36:37.685 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T03:36:37.754 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:36:37.754 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-16T03:36:37.822 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-16T03:36:37.822 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-09-16T03:36:37.888 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:36:37.888 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T03:36:37.918 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:36:37.919 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-16T03:36:37.987 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:36:37.988 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-16T03:36:38.056 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T03:36:38.056 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-09-16T03:36:38.124 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:36:38.124 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T03:36:38.150 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:36:38.150 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-16T03:36:38.215 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:36:38.215 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-16T03:36:38.280 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-16T03:36:38.280 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-09-16T03:36:38.345 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new bdbe961a-78d3-4f7d-88ed-ce24475d49e3 0 2024-09-16T03:36:38.565 INFO:teuthology.orchestra.run.smithi157.stdout:0 2024-09-16T03:36:38.576 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 154c8ddd-7a3d-402c-b9d6-28538f4e6bc4 1 2024-09-16T03:36:38.757 INFO:teuthology.orchestra.run.smithi157.stdout:1 2024-09-16T03:36:38.769 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 2a06de4e-0835-4a42-8c8a-49fb55aab566 2 2024-09-16T03:36:38.950 INFO:teuthology.orchestra.run.smithi157.stdout:2 2024-09-16T03:36:38.961 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 8a87f9f0-c380-4f38-bbd2-06b2def4fd2d 3 2024-09-16T03:36:39.143 INFO:teuthology.orchestra.run.smithi157.stdout:3 2024-09-16T03:36:39.154 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 5ac53318-12e8-496c-a87f-ba5de79c52aa 4 2024-09-16T03:36:39.333 INFO:teuthology.orchestra.run.smithi157.stdout:4 2024-09-16T03:36:39.344 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 675c81a4-2314-463e-82a7-0060735c78fd 5 2024-09-16T03:36:39.523 INFO:teuthology.orchestra.run.smithi157.stdout:5 2024-09-16T03:36:39.535 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new b18962a7-3b23-4e50-bf31-1d0629557423 6 2024-09-16T03:36:39.716 INFO:teuthology.orchestra.run.smithi157.stdout:6 2024-09-16T03:36:39.727 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 9706c1e8-dac0-4e16-8bc5-7cacc6f13870 7 2024-09-16T03:36:39.905 INFO:teuthology.orchestra.run.smithi157.stdout:7 2024-09-16T03:36:39.917 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new fafd06b8-c005-4471-a128-9dd51580c0f0 8 2024-09-16T03:36:40.096 INFO:teuthology.orchestra.run.smithi157.stdout:8 2024-09-16T03:36:40.106 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 79e38e57-52c2-416f-9b90-75621f06d3e3 9 2024-09-16T03:36:40.287 INFO:teuthology.orchestra.run.smithi157.stdout:9 2024-09-16T03:36:40.299 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10 10 2024-09-16T03:36:40.480 INFO:teuthology.orchestra.run.smithi157.stdout:10 2024-09-16T03:36:40.492 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 7a2a169a-ce94-4091-a792-27d0ecf9a7fa 11 2024-09-16T03:36:40.675 INFO:teuthology.orchestra.run.smithi157.stdout:11 2024-09-16T03:36:40.687 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 7a280cda-b740-40c8-9fae-1a1ff11651f6 12 2024-09-16T03:36:40.865 INFO:teuthology.orchestra.run.smithi157.stdout:12 2024-09-16T03:36:40.878 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new ce3a3148-cd90-422c-a106-045de89ab045 13 2024-09-16T03:36:41.059 INFO:teuthology.orchestra.run.smithi157.stdout:13 2024-09-16T03:36:41.070 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 461c2d79-f4e7-4b21-899c-50d4a6cf12f9 14 2024-09-16T03:36:41.282 INFO:teuthology.orchestra.run.smithi157.stdout:14 2024-09-16T03:36:41.295 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 9a8c0bbe-9389-4b2c-95e1-34d3e66461e9 15 2024-09-16T03:36:41.478 INFO:teuthology.orchestra.run.smithi157.stdout:15 2024-09-16T03:36:41.490 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T03:36:41.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T03:36:41.493 INFO:tasks.ceph.osd.0:Started 2024-09-16T03:36:41.493 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T03:36:41.493 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T03:36:41.495 INFO:tasks.ceph.osd.4:Started 2024-09-16T03:36:41.495 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-16T03:36:41.495 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-16T03:36:41.497 INFO:tasks.ceph.osd.8:Started 2024-09-16T03:36:41.497 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-16T03:36:41.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-16T03:36:41.500 INFO:tasks.ceph.osd.12:Started 2024-09-16T03:36:41.500 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T03:36:41.500 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T03:36:41.503 INFO:tasks.ceph.osd.1:Started 2024-09-16T03:36:41.503 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T03:36:41.503 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T03:36:41.505 INFO:tasks.ceph.osd.5:Started 2024-09-16T03:36:41.505 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-16T03:36:41.505 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-16T03:36:41.508 INFO:tasks.ceph.osd.9:Started 2024-09-16T03:36:41.508 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-16T03:36:41.508 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-16T03:36:41.510 INFO:tasks.ceph.osd.13:Started 2024-09-16T03:36:41.510 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T03:36:41.510 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T03:36:41.514 INFO:tasks.ceph.osd.2:Started 2024-09-16T03:36:41.514 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-16T03:36:41.514 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-16T03:36:41.516 INFO:tasks.ceph.osd.6:Started 2024-09-16T03:36:41.517 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-16T03:36:41.517 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-16T03:36:41.519 INFO:tasks.ceph.osd.10:Started 2024-09-16T03:36:41.519 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-16T03:36:41.519 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-16T03:36:41.521 INFO:tasks.ceph.osd.14:Started 2024-09-16T03:36:41.521 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T03:36:41.521 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T03:36:41.524 INFO:tasks.ceph.osd.3:Started 2024-09-16T03:36:41.524 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-16T03:36:41.524 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-16T03:36:41.526 INFO:tasks.ceph.osd.7:Started 2024-09-16T03:36:41.526 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-16T03:36:41.526 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-16T03:36:41.528 INFO:tasks.ceph.osd.11:Started 2024-09-16T03:36:41.528 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-16T03:36:41.528 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-16T03:36:41.530 INFO:tasks.ceph.osd.15:Started 2024-09-16T03:36:41.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:36:41.728 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:36:41.728 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":18,"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","created":"2024-09-16T03:36:37.310421+0000","modified":"2024-09-16T03:36:41.474235+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T03:36:41.474235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bdbe961a-78d3-4f7d-88ed-ce24475d49e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"154c8ddd-7a3d-402c-b9d6-28538f4e6bc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2a06de4e-0835-4a42-8c8a-49fb55aab566","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8a87f9f0-c380-4f38-bbd2-06b2def4fd2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5ac53318-12e8-496c-a87f-ba5de79c52aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"675c81a4-2314-463e-82a7-0060735c78fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b18962a7-3b23-4e50-bf31-1d0629557423","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9706c1e8-dac0-4e16-8bc5-7cacc6f13870","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"fafd06b8-c005-4471-a128-9dd51580c0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"79e38e57-52c2-416f-9b90-75621f06d3e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"7a2a169a-ce94-4091-a792-27d0ecf9a7fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"7a280cda-b740-40c8-9fae-1a1ff11651f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ce3a3148-cd90-422c-a106-045de89ab045","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"461c2d79-f4e7-4b21-899c-50d4a6cf12f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"9a8c0bbe-9389-4b2c-95e1-34d3e66461e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:36:41.739 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T03:36:41.739 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T03:36:42.445 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:36:42.443+0000 7f12f161b640 -1 Falling back to public interface 2024-09-16T03:36:42.451 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:36:42.449+0000 7f3d5340e640 -1 Falling back to public interface 2024-09-16T03:36:42.469 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:36:42.466+0000 7f92967c5640 -1 Falling back to public interface 2024-09-16T03:36:42.684 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-09-16T03:36:42.682+0000 7f5019b9e640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T03:36:42.686 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-09-16T03:36:42.684+0000 7f5019b9e640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T03:36:42.706 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:36:42.704+0000 7f653ba1e640 -1 Falling back to public interface 2024-09-16T03:36:42.708 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:36:42.707+0000 7eff8aeb2640 -1 Falling back to public interface 2024-09-16T03:36:42.719 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:36:42.717+0000 7f4225a15640 -1 Falling back to public interface 2024-09-16T03:36:42.731 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:36:42.729+0000 7f495002f640 -1 Falling back to public interface 2024-09-16T03:36:42.939 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:36:42.937+0000 7f0af9bb2640 -1 Falling back to public interface 2024-09-16T03:36:42.974 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:36:42.971+0000 7fa396d45640 -1 Falling back to public interface 2024-09-16T03:36:42.974 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:36:42.972+0000 7ff2749c5640 -1 Falling back to public interface 2024-09-16T03:36:43.198 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:36:43.196+0000 7efefdd34640 -1 Falling back to public interface 2024-09-16T03:36:43.200 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:36:43.198+0000 7f8bf7d34640 -1 Falling back to public interface 2024-09-16T03:36:43.204 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:36:43.201+0000 7f3a32422640 -1 Falling back to public interface 2024-09-16T03:36:43.209 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:36:43.206+0000 7fd1144d6640 -1 Falling back to public interface 2024-09-16T03:36:43.349 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T03:36:43.438 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:36:43.436+0000 7fedc4419640 -1 Falling back to public interface 2024-09-16T03:36:43.479 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:36:43.478+0000 7f6701864640 -1 Falling back to public interface 2024-09-16T03:36:43.654 INFO:teuthology.misc.health.smithi064.stdout: 2024-09-16T03:36:43.655 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":18,"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","created":"2024-09-16T03:36:37.310421+0000","modified":"2024-09-16T03:36:41.474235+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T03:36:41.474235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bdbe961a-78d3-4f7d-88ed-ce24475d49e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"154c8ddd-7a3d-402c-b9d6-28538f4e6bc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2a06de4e-0835-4a42-8c8a-49fb55aab566","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8a87f9f0-c380-4f38-bbd2-06b2def4fd2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5ac53318-12e8-496c-a87f-ba5de79c52aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"675c81a4-2314-463e-82a7-0060735c78fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b18962a7-3b23-4e50-bf31-1d0629557423","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9706c1e8-dac0-4e16-8bc5-7cacc6f13870","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"fafd06b8-c005-4471-a128-9dd51580c0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"79e38e57-52c2-416f-9b90-75621f06d3e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"7a2a169a-ce94-4091-a792-27d0ecf9a7fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"7a280cda-b740-40c8-9fae-1a1ff11651f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"ce3a3148-cd90-422c-a106-045de89ab045","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"461c2d79-f4e7-4b21-899c-50d4a6cf12f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"9a8c0bbe-9389-4b2c-95e1-34d3e66461e9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:36:43.667 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-09-16T03:36:45.978 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:36:45.977+0000 7f3d5340e640 -1 osd.12 0 log_to_monitors true 2024-09-16T03:36:45.999 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:36:45.997+0000 7f92967c5640 -1 osd.14 0 log_to_monitors true 2024-09-16T03:36:46.003 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:36:46.002+0000 7f12f161b640 -1 osd.5 0 log_to_monitors true 2024-09-16T03:36:46.501 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:36:46.499+0000 7f653ba1e640 -1 osd.1 0 log_to_monitors true 2024-09-16T03:36:46.507 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:36:46.505+0000 7f4225a15640 -1 osd.8 0 log_to_monitors true 2024-09-16T03:36:46.543 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:36:46.541+0000 7f495002f640 -1 osd.4 0 log_to_monitors true 2024-09-16T03:36:46.681 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:36:46.679+0000 7f0af9bb2640 -1 osd.0 0 log_to_monitors true 2024-09-16T03:36:46.728 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:36:46.726+0000 7efefdd34640 -1 osd.13 0 log_to_monitors true 2024-09-16T03:36:46.768 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:36:46.766+0000 7eff8aeb2640 -1 osd.3 0 log_to_monitors true 2024-09-16T03:36:46.944 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:36:46.942+0000 7fedc4419640 -1 osd.9 0 log_to_monitors true 2024-09-16T03:36:46.971 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:36:46.970+0000 7f8bf7d34640 -1 osd.7 0 log_to_monitors true 2024-09-16T03:36:47.011 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:36:47.009+0000 7fa396d45640 -1 osd.10 0 log_to_monitors true 2024-09-16T03:36:47.040 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:36:47.038+0000 7f12ed008640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T03:36:47.041 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:36:47.040+0000 7f12e4a1e640 -1 osd.5 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:47.043 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:36:47.041+0000 7f9292754640 -1 osd.14 0 waiting for initial osdmap 2024-09-16T03:36:47.047 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:36:47.045+0000 7f928a16a640 -1 osd.14 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:47.049 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:36:47.048+0000 7f12e8e2f640 -1 osd.5 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:47.051 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:36:47.048+0000 7f928e57b640 -1 osd.14 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:47.251 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:36:47.249+0000 7f3a32422640 -1 osd.6 0 log_to_monitors true 2024-09-16T03:36:47.272 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:36:47.269+0000 7fd1144d6640 -1 osd.2 0 log_to_monitors true 2024-09-16T03:36:47.478 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:36:47.476+0000 7f6701864640 -1 osd.15 0 log_to_monitors true 2024-09-16T03:36:47.526 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:36:47.525+0000 7ff2749c5640 -1 osd.11 0 log_to_monitors true 2024-09-16T03:36:47.646 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:36:47.644+0000 7f3d4ecfe640 -1 osd.12 0 waiting for initial osdmap 2024-09-16T03:36:47.647 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:36:47.645+0000 7f3d4270c640 -1 osd.12 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:47.649 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:36:47.648+0000 7f3d46b1d640 -1 osd.12 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:48.045 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:36:48.043+0000 7fa392cd4640 -1 osd.10 0 waiting for initial osdmap 2024-09-16T03:36:48.045 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:36:48.043+0000 7fa38a6ea640 -1 osd.10 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:48.046 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:36:48.044+0000 7f494bfbe640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T03:36:48.051 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:36:48.049+0000 7f49439d4640 -1 osd.4 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:48.052 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:36:48.050+0000 7fa38eafb640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:48.053 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:36:48.051+0000 7efef9cc3640 -1 osd.13 0 waiting for initial osdmap 2024-09-16T03:36:48.053 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:36:48.051+0000 7fedbfd26640 -1 osd.9 0 waiting for initial osdmap 2024-09-16T03:36:48.054 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:36:48.052+0000 7efef16d9640 -1 osd.13 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:48.055 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:36:48.053+0000 7fedb773d640 -1 osd.9 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:48.059 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:36:48.057+0000 7fedbbb4d640 -1 osd.9 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:48.059 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:36:48.058+0000 7f4947de5640 -1 osd.4 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:48.060 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:36:48.058+0000 7efef5aea640 -1 osd.13 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:48.617 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:36:48.615+0000 7f653745e640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T03:36:48.617 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:36:48.615+0000 7f652ae6c640 -1 osd.1 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:48.753 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:36:48.751+0000 7eff86e41640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T03:36:48.754 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:36:48.752+0000 7eff7a84f640 -1 osd.3 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:48.758 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:36:48.756+0000 7f8bf3cc3640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T03:36:48.758 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:36:48.757+0000 7f8be76d1640 -1 osd.7 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:48.799 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:36:48.798+0000 7f4221b09640 -1 osd.8 0 waiting for initial osdmap 2024-09-16T03:36:48.800 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:36:48.798+0000 7f4214d04640 -1 osd.8 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:48.809 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:36:48.807+0000 7f0af5b41640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T03:36:48.810 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:36:48.808+0000 7f0ae954f640 -1 osd.0 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:48.823 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:36:48.821+0000 7f652f27d640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:48.958 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:36:48.956+0000 7eff7ec60640 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:48.961 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:36:48.959+0000 7f8bebae2640 -1 osd.7 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:49.005 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:36:49.003+0000 7f4219114640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:49.013 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:36:49.012+0000 7f0aed960640 -1 osd.0 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:49.054 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:36:49.052+0000 7f66fd7f3640 -1 osd.15 0 waiting for initial osdmap 2024-09-16T03:36:49.057 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:36:49.055+0000 7ff270954640 -1 osd.11 0 waiting for initial osdmap 2024-09-16T03:36:49.058 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:36:49.055+0000 7f66f5209640 -1 osd.15 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:49.059 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:36:49.055+0000 7ff26836a640 -1 osd.11 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:49.061 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:36:49.059+0000 7f66f961a640 -1 osd.15 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:49.061 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:36:49.059+0000 7ff26c77b640 -1 osd.11 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:49.751 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:36:49.749+0000 7f3a2e528640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T03:36:49.751 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:36:49.749+0000 7f3a21722640 -1 osd.6 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:49.754 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:36:49.752+0000 7f3a25b33640 -1 osd.6 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:49.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:36:49.772+0000 7fd110465640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T03:36:49.775 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:36:49.772+0000 7fd103e73640 -1 osd.2 0 handle_osd_map injecting map crc failure 2024-09-16T03:36:49.781 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:36:49.778+0000 7fd108284640 -1 osd.2 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:36:50.061 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:36:50.059+0000 7f12e4a1e640 -1 osd.5 21 handle_osd_map injecting map crc failure 2024-09-16T03:36:50.062 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:36:50.059+0000 7f3a21722640 -1 osd.6 22 handle_osd_map injecting map crc failure 2024-09-16T03:36:50.062 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:36:50.059+0000 7fd103e73640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-09-16T03:36:50.775 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:36:50.773+0000 7f9288166640 -1 osd.14 21 handle_osd_map injecting map crc failure 2024-09-16T03:36:51.282 DEBUG:teuthology.orchestra.run.smithi064:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T03:36:51.588 INFO:teuthology.misc.health.smithi064.stdout: 2024-09-16T03:36:51.588 INFO:teuthology.misc.health.smithi064.stdout:{"epoch":24,"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","created":"2024-09-16T03:36:37.310421+0000","modified":"2024-09-16T03:36:51.067403+0000","last_up_change":"2024-09-16T03:36:50.058111+0000","last_in_change":"2024-09-16T03:36:41.474235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:36:49.696307+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.789999961853027,"score_stable":15.789999961853027,"optimal_score":0.18999999761581421,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bdbe961a-78d3-4f7d-88ed-ce24475d49e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6813","nonce":3924351635}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6814","nonce":3924351635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6816","nonce":3924351635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6815","nonce":3924351635}]},"public_addr":"172.21.15.64:6813/3924351635","cluster_addr":"172.21.15.64:6814/3924351635","heartbeat_back_addr":"172.21.15.64:6816/3924351635","heartbeat_front_addr":"172.21.15.64:6815/3924351635","state":["exists","up"]},{"osd":1,"uuid":"154c8ddd-7a3d-402c-b9d6-28538f4e6bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6804","nonce":624977804}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6805","nonce":624977804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6807","nonce":624977804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6806","nonce":624977804}]},"public_addr":"172.21.15.112:6804/624977804","cluster_addr":"172.21.15.112:6805/624977804","heartbeat_back_addr":"172.21.15.112:6807/624977804","heartbeat_front_addr":"172.21.15.112:6806/624977804","state":["exists","up"]},{"osd":2,"uuid":"2a06de4e-0835-4a42-8c8a-49fb55aab566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6812","nonce":2804648086}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6813","nonce":2804648086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6815","nonce":2804648086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6814","nonce":2804648086}]},"public_addr":"172.21.15.155:6812/2804648086","cluster_addr":"172.21.15.155:6813/2804648086","heartbeat_back_addr":"172.21.15.155:6815/2804648086","heartbeat_front_addr":"172.21.15.155:6814/2804648086","state":["exists","up"]},{"osd":3,"uuid":"8a87f9f0-c380-4f38-bbd2-06b2def4fd2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6800","nonce":1935597988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6801","nonce":1935597988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6803","nonce":1935597988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6802","nonce":1935597988}]},"public_addr":"172.21.15.157:6800/1935597988","cluster_addr":"172.21.15.157:6801/1935597988","heartbeat_back_addr":"172.21.15.157:6803/1935597988","heartbeat_front_addr":"172.21.15.157:6802/1935597988","state":["exists","up"]},{"osd":4,"uuid":"5ac53318-12e8-496c-a87f-ba5de79c52aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":2675475248}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":2675475248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6812","nonce":2675475248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":2675475248}]},"public_addr":"172.21.15.64:6809/2675475248","cluster_addr":"172.21.15.64:6810/2675475248","heartbeat_back_addr":"172.21.15.64:6812/2675475248","heartbeat_front_addr":"172.21.15.64:6811/2675475248","state":["exists","up"]},{"osd":5,"uuid":"675c81a4-2314-463e-82a7-0060735c78fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6800","nonce":499905276}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6801","nonce":499905276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6803","nonce":499905276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6802","nonce":499905276}]},"public_addr":"172.21.15.112:6800/499905276","cluster_addr":"172.21.15.112:6801/499905276","heartbeat_back_addr":"172.21.15.112:6803/499905276","heartbeat_front_addr":"172.21.15.112:6802/499905276","state":["exists","up"]},{"osd":6,"uuid":"b18962a7-3b23-4e50-bf31-1d0629557423","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6808","nonce":2839983406}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6809","nonce":2839983406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6811","nonce":2839983406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6810","nonce":2839983406}]},"public_addr":"172.21.15.155:6808/2839983406","cluster_addr":"172.21.15.155:6809/2839983406","heartbeat_back_addr":"172.21.15.155:6811/2839983406","heartbeat_front_addr":"172.21.15.155:6810/2839983406","state":["exists","up"]},{"osd":7,"uuid":"9706c1e8-dac0-4e16-8bc5-7cacc6f13870","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6808","nonce":3021982871}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6809","nonce":3021982871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6811","nonce":3021982871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6810","nonce":3021982871}]},"public_addr":"172.21.15.157:6808/3021982871","cluster_addr":"172.21.15.157:6809/3021982871","heartbeat_back_addr":"172.21.15.157:6811/3021982871","heartbeat_front_addr":"172.21.15.157:6810/3021982871","state":["exists","up"]},{"osd":8,"uuid":"fafd06b8-c005-4471-a128-9dd51580c0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":3609017627}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":3609017627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":3609017627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":3609017627}]},"public_addr":"172.21.15.64:6805/3609017627","cluster_addr":"172.21.15.64:6806/3609017627","heartbeat_back_addr":"172.21.15.64:6808/3609017627","heartbeat_front_addr":"172.21.15.64:6807/3609017627","state":["exists","up"]},{"osd":9,"uuid":"79e38e57-52c2-416f-9b90-75621f06d3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6812","nonce":701981815}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6813","nonce":701981815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6815","nonce":701981815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6814","nonce":701981815}]},"public_addr":"172.21.15.112:6812/701981815","cluster_addr":"172.21.15.112:6813/701981815","heartbeat_back_addr":"172.21.15.112:6815/701981815","heartbeat_front_addr":"172.21.15.112:6814/701981815","state":["exists","up"]},{"osd":10,"uuid":"8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6804","nonce":4138619523}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6805","nonce":4138619523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6807","nonce":4138619523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6806","nonce":4138619523}]},"public_addr":"172.21.15.155:6804/4138619523","cluster_addr":"172.21.15.155:6805/4138619523","heartbeat_back_addr":"172.21.15.155:6807/4138619523","heartbeat_front_addr":"172.21.15.155:6806/4138619523","state":["exists","up"]},{"osd":11,"uuid":"7a2a169a-ce94-4091-a792-27d0ecf9a7fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6804","nonce":3607367937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6805","nonce":3607367937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6807","nonce":3607367937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6806","nonce":3607367937}]},"public_addr":"172.21.15.157:6804/3607367937","cluster_addr":"172.21.15.157:6805/3607367937","heartbeat_back_addr":"172.21.15.157:6807/3607367937","heartbeat_front_addr":"172.21.15.157:6806/3607367937","state":["exists","up"]},{"osd":12,"uuid":"7a280cda-b740-40c8-9fae-1a1ff11651f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":2984461266}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":2984461266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":2984461266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":2984461266}]},"public_addr":"172.21.15.64:6801/2984461266","cluster_addr":"172.21.15.64:6802/2984461266","heartbeat_back_addr":"172.21.15.64:6804/2984461266","heartbeat_front_addr":"172.21.15.64:6803/2984461266","state":["exists","up"]},{"osd":13,"uuid":"ce3a3148-cd90-422c-a106-045de89ab045","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6808","nonce":1616314527}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6809","nonce":1616314527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6811","nonce":1616314527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6810","nonce":1616314527}]},"public_addr":"172.21.15.112:6808/1616314527","cluster_addr":"172.21.15.112:6809/1616314527","heartbeat_back_addr":"172.21.15.112:6811/1616314527","heartbeat_front_addr":"172.21.15.112:6810/1616314527","state":["exists","up"]},{"osd":14,"uuid":"461c2d79-f4e7-4b21-899c-50d4a6cf12f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6800","nonce":4203868121}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6801","nonce":4203868121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6803","nonce":4203868121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6802","nonce":4203868121}]},"public_addr":"172.21.15.155:6800/4203868121","cluster_addr":"172.21.15.155:6801/4203868121","heartbeat_back_addr":"172.21.15.155:6803/4203868121","heartbeat_front_addr":"172.21.15.155:6802/4203868121","state":["exists","up"]},{"osd":15,"uuid":"9a8c0bbe-9389-4b2c-95e1-34d3e66461e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6812","nonce":181354529}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6813","nonce":181354529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6815","nonce":181354529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6814","nonce":181354529}]},"public_addr":"172.21.15.157:6812/181354529","cluster_addr":"172.21.15.157:6813/181354529","heartbeat_back_addr":"172.21.15.157:6815/181354529","heartbeat_front_addr":"172.21.15.157:6814/181354529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.702114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.496387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:48.252480+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.791534+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.517661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.018557+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:48.247312+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.959242+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.506293+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.963066+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.961846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:48.574000+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:46.945017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.752207+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:46.975964+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:48.501682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:36:51.599 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-09-16T03:36:51.599 INFO:tasks.ceph:Creating RBD pool 2024-09-16T03:36:51.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T03:36:52.078 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'rbd' created 2024-09-16T03:36:52.083 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:36:52.081+0000 7eff7884b640 -1 osd.3 22 handle_osd_map injecting map crc failure 2024-09-16T03:36:52.090 DEBUG:teuthology.orchestra.run.smithi064:> rbd --cluster ceph pool init rbd 2024-09-16T03:36:53.130 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:36:53.126+0000 7f0ae754b640 -1 osd.0 25 handle_osd_map injecting map crc failure 2024-09-16T03:36:54.293 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:36:54.291+0000 7ff266366640 -1 osd.11 26 handle_osd_map injecting map crc failure 2024-09-16T03:36:55.098 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:36:55.095+0000 7f4214d04640 -1 osd.8 27 handle_osd_map injecting map crc failure 2024-09-16T03:36:55.408 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T03:36:55.409 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-16T03:36:55.409 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T03:36:55.721 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:36:55.733 INFO:teuthology.orchestra.run.smithi064.stdout:[{"version":9,"timestamp":"2024-09-16T03:36:49.773928+0000","name":"","changes":[{"name":"osd.2/osd_mclock_max_capacity_iops_ssd","new_value":"7381.762054"}]}] 2024-09-16T03:36:55.733 INFO:tasks.ceph_manager:config epoch is 9 2024-09-16T03:36:55.733 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T03:36:55.734 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T03:36:55.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T03:36:56.090 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:36:56.100 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":4,"flags":0,"active_gid":4103,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6800","nonce":496328888}]},"active_addr":"172.21.15.64:6800/496328888","active_change":"2024-09-16T03:36:41.671354+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4100,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":2211237099}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":2831778041}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":560672394}]}]} 2024-09-16T03:36:56.106 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T03:36:56.106 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T03:36:56.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:36:56.412 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:36:56.412 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":28,"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","created":"2024-09-16T03:36:37.310421+0000","modified":"2024-09-16T03:36:55.089739+0000","last_up_change":"2024-09-16T03:36:50.058111+0000","last_in_change":"2024-09-16T03:36:41.474235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:36:49.696307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.789999961853027,"score_stable":15.789999961853027,"optimal_score":0.18999999761581421,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:36:51.904455+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":5.9699997901916504,"score_stable":5.9699997901916504,"optimal_score":0.87999999523162842,"raw_score_acting":5.25,"raw_score_stable":5.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bdbe961a-78d3-4f7d-88ed-ce24475d49e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6813","nonce":3924351635}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6814","nonce":3924351635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6816","nonce":3924351635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6815","nonce":3924351635}]},"public_addr":"172.21.15.64:6813/3924351635","cluster_addr":"172.21.15.64:6814/3924351635","heartbeat_back_addr":"172.21.15.64:6816/3924351635","heartbeat_front_addr":"172.21.15.64:6815/3924351635","state":["exists","up"]},{"osd":1,"uuid":"154c8ddd-7a3d-402c-b9d6-28538f4e6bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6804","nonce":624977804}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6805","nonce":624977804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6807","nonce":624977804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6806","nonce":624977804}]},"public_addr":"172.21.15.112:6804/624977804","cluster_addr":"172.21.15.112:6805/624977804","heartbeat_back_addr":"172.21.15.112:6807/624977804","heartbeat_front_addr":"172.21.15.112:6806/624977804","state":["exists","up"]},{"osd":2,"uuid":"2a06de4e-0835-4a42-8c8a-49fb55aab566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6812","nonce":2804648086}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6813","nonce":2804648086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6815","nonce":2804648086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6814","nonce":2804648086}]},"public_addr":"172.21.15.155:6812/2804648086","cluster_addr":"172.21.15.155:6813/2804648086","heartbeat_back_addr":"172.21.15.155:6815/2804648086","heartbeat_front_addr":"172.21.15.155:6814/2804648086","state":["exists","up"]},{"osd":3,"uuid":"8a87f9f0-c380-4f38-bbd2-06b2def4fd2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6800","nonce":1935597988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6801","nonce":1935597988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6803","nonce":1935597988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6802","nonce":1935597988}]},"public_addr":"172.21.15.157:6800/1935597988","cluster_addr":"172.21.15.157:6801/1935597988","heartbeat_back_addr":"172.21.15.157:6803/1935597988","heartbeat_front_addr":"172.21.15.157:6802/1935597988","state":["exists","up"]},{"osd":4,"uuid":"5ac53318-12e8-496c-a87f-ba5de79c52aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":2675475248}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":2675475248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6812","nonce":2675475248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":2675475248}]},"public_addr":"172.21.15.64:6809/2675475248","cluster_addr":"172.21.15.64:6810/2675475248","heartbeat_back_addr":"172.21.15.64:6812/2675475248","heartbeat_front_addr":"172.21.15.64:6811/2675475248","state":["exists","up"]},{"osd":5,"uuid":"675c81a4-2314-463e-82a7-0060735c78fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6800","nonce":499905276}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6801","nonce":499905276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6803","nonce":499905276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6802","nonce":499905276}]},"public_addr":"172.21.15.112:6800/499905276","cluster_addr":"172.21.15.112:6801/499905276","heartbeat_back_addr":"172.21.15.112:6803/499905276","heartbeat_front_addr":"172.21.15.112:6802/499905276","state":["exists","up"]},{"osd":6,"uuid":"b18962a7-3b23-4e50-bf31-1d0629557423","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6808","nonce":2839983406}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6809","nonce":2839983406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6811","nonce":2839983406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6810","nonce":2839983406}]},"public_addr":"172.21.15.155:6808/2839983406","cluster_addr":"172.21.15.155:6809/2839983406","heartbeat_back_addr":"172.21.15.155:6811/2839983406","heartbeat_front_addr":"172.21.15.155:6810/2839983406","state":["exists","up"]},{"osd":7,"uuid":"9706c1e8-dac0-4e16-8bc5-7cacc6f13870","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6808","nonce":3021982871}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6809","nonce":3021982871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6811","nonce":3021982871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6810","nonce":3021982871}]},"public_addr":"172.21.15.157:6808/3021982871","cluster_addr":"172.21.15.157:6809/3021982871","heartbeat_back_addr":"172.21.15.157:6811/3021982871","heartbeat_front_addr":"172.21.15.157:6810/3021982871","state":["exists","up"]},{"osd":8,"uuid":"fafd06b8-c005-4471-a128-9dd51580c0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":3609017627}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":3609017627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":3609017627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":3609017627}]},"public_addr":"172.21.15.64:6805/3609017627","cluster_addr":"172.21.15.64:6806/3609017627","heartbeat_back_addr":"172.21.15.64:6808/3609017627","heartbeat_front_addr":"172.21.15.64:6807/3609017627","state":["exists","up"]},{"osd":9,"uuid":"79e38e57-52c2-416f-9b90-75621f06d3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6812","nonce":701981815}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6813","nonce":701981815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6815","nonce":701981815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6814","nonce":701981815}]},"public_addr":"172.21.15.112:6812/701981815","cluster_addr":"172.21.15.112:6813/701981815","heartbeat_back_addr":"172.21.15.112:6815/701981815","heartbeat_front_addr":"172.21.15.112:6814/701981815","state":["exists","up"]},{"osd":10,"uuid":"8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6804","nonce":4138619523}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6805","nonce":4138619523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6807","nonce":4138619523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6806","nonce":4138619523}]},"public_addr":"172.21.15.155:6804/4138619523","cluster_addr":"172.21.15.155:6805/4138619523","heartbeat_back_addr":"172.21.15.155:6807/4138619523","heartbeat_front_addr":"172.21.15.155:6806/4138619523","state":["exists","up"]},{"osd":11,"uuid":"7a2a169a-ce94-4091-a792-27d0ecf9a7fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6804","nonce":3607367937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6805","nonce":3607367937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6807","nonce":3607367937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6806","nonce":3607367937}]},"public_addr":"172.21.15.157:6804/3607367937","cluster_addr":"172.21.15.157:6805/3607367937","heartbeat_back_addr":"172.21.15.157:6807/3607367937","heartbeat_front_addr":"172.21.15.157:6806/3607367937","state":["exists","up"]},{"osd":12,"uuid":"7a280cda-b740-40c8-9fae-1a1ff11651f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":2984461266}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":2984461266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":2984461266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":2984461266}]},"public_addr":"172.21.15.64:6801/2984461266","cluster_addr":"172.21.15.64:6802/2984461266","heartbeat_back_addr":"172.21.15.64:6804/2984461266","heartbeat_front_addr":"172.21.15.64:6803/2984461266","state":["exists","up"]},{"osd":13,"uuid":"ce3a3148-cd90-422c-a106-045de89ab045","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6808","nonce":1616314527}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6809","nonce":1616314527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6811","nonce":1616314527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6810","nonce":1616314527}]},"public_addr":"172.21.15.112:6808/1616314527","cluster_addr":"172.21.15.112:6809/1616314527","heartbeat_back_addr":"172.21.15.112:6811/1616314527","heartbeat_front_addr":"172.21.15.112:6810/1616314527","state":["exists","up"]},{"osd":14,"uuid":"461c2d79-f4e7-4b21-899c-50d4a6cf12f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6800","nonce":4203868121}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6801","nonce":4203868121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6803","nonce":4203868121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6802","nonce":4203868121}]},"public_addr":"172.21.15.155:6800/4203868121","cluster_addr":"172.21.15.155:6801/4203868121","heartbeat_back_addr":"172.21.15.155:6803/4203868121","heartbeat_front_addr":"172.21.15.155:6802/4203868121","state":["exists","up"]},{"osd":15,"uuid":"9a8c0bbe-9389-4b2c-95e1-34d3e66461e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6812","nonce":181354529}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6813","nonce":181354529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6815","nonce":181354529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6814","nonce":181354529}]},"public_addr":"172.21.15.157:6812/181354529","cluster_addr":"172.21.15.157:6813/181354529","heartbeat_back_addr":"172.21.15.157:6815/181354529","heartbeat_front_addr":"172.21.15.157:6814/181354529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.702114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.496387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:48.252480+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.791534+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.517661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.018557+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:48.247312+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.959242+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.506293+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.963066+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.961846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:48.574000+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:46.945017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.752207+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:46.975964+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:48.501682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:36:56.425 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T03:36:56.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:36:56.724 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:36:56.725 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":28,"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","created":"2024-09-16T03:36:37.310421+0000","modified":"2024-09-16T03:36:55.089739+0000","last_up_change":"2024-09-16T03:36:50.058111+0000","last_in_change":"2024-09-16T03:36:41.474235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:36:49.696307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.789999961853027,"score_stable":15.789999961853027,"optimal_score":0.18999999761581421,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:36:51.904455+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":5.9699997901916504,"score_stable":5.9699997901916504,"optimal_score":0.87999999523162842,"raw_score_acting":5.25,"raw_score_stable":5.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bdbe961a-78d3-4f7d-88ed-ce24475d49e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6813","nonce":3924351635}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6814","nonce":3924351635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6816","nonce":3924351635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6815","nonce":3924351635}]},"public_addr":"172.21.15.64:6813/3924351635","cluster_addr":"172.21.15.64:6814/3924351635","heartbeat_back_addr":"172.21.15.64:6816/3924351635","heartbeat_front_addr":"172.21.15.64:6815/3924351635","state":["exists","up"]},{"osd":1,"uuid":"154c8ddd-7a3d-402c-b9d6-28538f4e6bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6804","nonce":624977804}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6805","nonce":624977804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6807","nonce":624977804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6806","nonce":624977804}]},"public_addr":"172.21.15.112:6804/624977804","cluster_addr":"172.21.15.112:6805/624977804","heartbeat_back_addr":"172.21.15.112:6807/624977804","heartbeat_front_addr":"172.21.15.112:6806/624977804","state":["exists","up"]},{"osd":2,"uuid":"2a06de4e-0835-4a42-8c8a-49fb55aab566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6812","nonce":2804648086}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6813","nonce":2804648086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6815","nonce":2804648086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6814","nonce":2804648086}]},"public_addr":"172.21.15.155:6812/2804648086","cluster_addr":"172.21.15.155:6813/2804648086","heartbeat_back_addr":"172.21.15.155:6815/2804648086","heartbeat_front_addr":"172.21.15.155:6814/2804648086","state":["exists","up"]},{"osd":3,"uuid":"8a87f9f0-c380-4f38-bbd2-06b2def4fd2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6800","nonce":1935597988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6801","nonce":1935597988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6803","nonce":1935597988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6802","nonce":1935597988}]},"public_addr":"172.21.15.157:6800/1935597988","cluster_addr":"172.21.15.157:6801/1935597988","heartbeat_back_addr":"172.21.15.157:6803/1935597988","heartbeat_front_addr":"172.21.15.157:6802/1935597988","state":["exists","up"]},{"osd":4,"uuid":"5ac53318-12e8-496c-a87f-ba5de79c52aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":2675475248}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":2675475248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6812","nonce":2675475248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":2675475248}]},"public_addr":"172.21.15.64:6809/2675475248","cluster_addr":"172.21.15.64:6810/2675475248","heartbeat_back_addr":"172.21.15.64:6812/2675475248","heartbeat_front_addr":"172.21.15.64:6811/2675475248","state":["exists","up"]},{"osd":5,"uuid":"675c81a4-2314-463e-82a7-0060735c78fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6800","nonce":499905276}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6801","nonce":499905276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6803","nonce":499905276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6802","nonce":499905276}]},"public_addr":"172.21.15.112:6800/499905276","cluster_addr":"172.21.15.112:6801/499905276","heartbeat_back_addr":"172.21.15.112:6803/499905276","heartbeat_front_addr":"172.21.15.112:6802/499905276","state":["exists","up"]},{"osd":6,"uuid":"b18962a7-3b23-4e50-bf31-1d0629557423","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6808","nonce":2839983406}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6809","nonce":2839983406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6811","nonce":2839983406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6810","nonce":2839983406}]},"public_addr":"172.21.15.155:6808/2839983406","cluster_addr":"172.21.15.155:6809/2839983406","heartbeat_back_addr":"172.21.15.155:6811/2839983406","heartbeat_front_addr":"172.21.15.155:6810/2839983406","state":["exists","up"]},{"osd":7,"uuid":"9706c1e8-dac0-4e16-8bc5-7cacc6f13870","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6808","nonce":3021982871}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6809","nonce":3021982871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6811","nonce":3021982871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6810","nonce":3021982871}]},"public_addr":"172.21.15.157:6808/3021982871","cluster_addr":"172.21.15.157:6809/3021982871","heartbeat_back_addr":"172.21.15.157:6811/3021982871","heartbeat_front_addr":"172.21.15.157:6810/3021982871","state":["exists","up"]},{"osd":8,"uuid":"fafd06b8-c005-4471-a128-9dd51580c0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":3609017627}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":3609017627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":3609017627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":3609017627}]},"public_addr":"172.21.15.64:6805/3609017627","cluster_addr":"172.21.15.64:6806/3609017627","heartbeat_back_addr":"172.21.15.64:6808/3609017627","heartbeat_front_addr":"172.21.15.64:6807/3609017627","state":["exists","up"]},{"osd":9,"uuid":"79e38e57-52c2-416f-9b90-75621f06d3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6812","nonce":701981815}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6813","nonce":701981815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6815","nonce":701981815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6814","nonce":701981815}]},"public_addr":"172.21.15.112:6812/701981815","cluster_addr":"172.21.15.112:6813/701981815","heartbeat_back_addr":"172.21.15.112:6815/701981815","heartbeat_front_addr":"172.21.15.112:6814/701981815","state":["exists","up"]},{"osd":10,"uuid":"8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6804","nonce":4138619523}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6805","nonce":4138619523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6807","nonce":4138619523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6806","nonce":4138619523}]},"public_addr":"172.21.15.155:6804/4138619523","cluster_addr":"172.21.15.155:6805/4138619523","heartbeat_back_addr":"172.21.15.155:6807/4138619523","heartbeat_front_addr":"172.21.15.155:6806/4138619523","state":["exists","up"]},{"osd":11,"uuid":"7a2a169a-ce94-4091-a792-27d0ecf9a7fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6804","nonce":3607367937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6805","nonce":3607367937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6807","nonce":3607367937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6806","nonce":3607367937}]},"public_addr":"172.21.15.157:6804/3607367937","cluster_addr":"172.21.15.157:6805/3607367937","heartbeat_back_addr":"172.21.15.157:6807/3607367937","heartbeat_front_addr":"172.21.15.157:6806/3607367937","state":["exists","up"]},{"osd":12,"uuid":"7a280cda-b740-40c8-9fae-1a1ff11651f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":2984461266}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":2984461266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":2984461266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":2984461266}]},"public_addr":"172.21.15.64:6801/2984461266","cluster_addr":"172.21.15.64:6802/2984461266","heartbeat_back_addr":"172.21.15.64:6804/2984461266","heartbeat_front_addr":"172.21.15.64:6803/2984461266","state":["exists","up"]},{"osd":13,"uuid":"ce3a3148-cd90-422c-a106-045de89ab045","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6808","nonce":1616314527}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6809","nonce":1616314527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6811","nonce":1616314527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6810","nonce":1616314527}]},"public_addr":"172.21.15.112:6808/1616314527","cluster_addr":"172.21.15.112:6809/1616314527","heartbeat_back_addr":"172.21.15.112:6811/1616314527","heartbeat_front_addr":"172.21.15.112:6810/1616314527","state":["exists","up"]},{"osd":14,"uuid":"461c2d79-f4e7-4b21-899c-50d4a6cf12f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6800","nonce":4203868121}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6801","nonce":4203868121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6803","nonce":4203868121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6802","nonce":4203868121}]},"public_addr":"172.21.15.155:6800/4203868121","cluster_addr":"172.21.15.155:6801/4203868121","heartbeat_back_addr":"172.21.15.155:6803/4203868121","heartbeat_front_addr":"172.21.15.155:6802/4203868121","state":["exists","up"]},{"osd":15,"uuid":"9a8c0bbe-9389-4b2c-95e1-34d3e66461e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6812","nonce":181354529}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6813","nonce":181354529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6815","nonce":181354529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6814","nonce":181354529}]},"public_addr":"172.21.15.157:6812/181354529","cluster_addr":"172.21.15.157:6813/181354529","heartbeat_back_addr":"172.21.15.157:6815/181354529","heartbeat_front_addr":"172.21.15.157:6814/181354529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.702114+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.496387+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:48.252480+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.791534+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.517661+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.018557+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:48.247312+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.959242+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.506293+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.963066+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.961846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:48.574000+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:46.945017+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:47.752207+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:46.975964+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:36:48.501682+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:36:56.737 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T03:36:56.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T03:36:56.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T03:36:56.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T03:36:56.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T03:36:56.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T03:36:56.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T03:36:56.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T03:36:56.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-16T03:36:56.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-16T03:36:56.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-16T03:36:56.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-16T03:36:56.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-16T03:36:56.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-16T03:36:56.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-16T03:36:56.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-16T03:36:57.190 INFO:teuthology.orchestra.run.smithi064.stdout:98784247812 2024-09-16T03:36:57.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T03:36:57.197 INFO:teuthology.orchestra.run.smithi064.stdout:98784247812 2024-09-16T03:36:57.197 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T03:36:57.209 INFO:teuthology.orchestra.run.smithi064.stdout:90194313220 2024-09-16T03:36:57.210 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T03:36:57.217 INFO:teuthology.orchestra.run.smithi064.stdout:98784247811 2024-09-16T03:36:57.217 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T03:36:57.224 INFO:teuthology.orchestra.run.smithi064.stdout:94489280515 2024-09-16T03:36:57.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T03:36:57.229 INFO:teuthology.orchestra.run.smithi064.stdout:90194313219 2024-09-16T03:36:57.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T03:36:57.234 INFO:teuthology.orchestra.run.smithi064.stdout:94489280516 2024-09-16T03:36:57.234 INFO:teuthology.orchestra.run.smithi064.stdout:94489280516 2024-09-16T03:36:57.235 INFO:teuthology.orchestra.run.smithi064.stdout:94489280516 2024-09-16T03:36:57.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T03:36:57.236 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T03:36:57.236 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T03:36:57.241 INFO:teuthology.orchestra.run.smithi064.stdout:90194313220 2024-09-16T03:36:57.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T03:36:57.250 INFO:teuthology.orchestra.run.smithi064.stdout:98784247811 2024-09-16T03:36:57.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T03:36:57.253 INFO:teuthology.orchestra.run.smithi064.stdout:94489280515 2024-09-16T03:36:57.253 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T03:36:57.262 INFO:teuthology.orchestra.run.smithi064.stdout:94489280515 2024-09-16T03:36:57.263 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T03:36:57.284 INFO:teuthology.orchestra.run.smithi064.stdout:94489280516 2024-09-16T03:36:57.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T03:36:57.285 INFO:teuthology.orchestra.run.smithi064.stdout:94489280515 2024-09-16T03:36:57.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T03:36:57.353 INFO:teuthology.orchestra.run.smithi064.stdout:94489280515 2024-09-16T03:36:57.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T03:36:58.162 INFO:teuthology.orchestra.run.smithi064.stdout:98784247811 2024-09-16T03:36:58.191 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247811 for osd.6 2024-09-16T03:36:58.191 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.311 INFO:teuthology.orchestra.run.smithi064.stdout:90194313220 2024-09-16T03:36:58.340 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.14 2024-09-16T03:36:58.340 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.344 INFO:teuthology.orchestra.run.smithi064.stdout:98784247812 2024-09-16T03:36:58.367 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.2 2024-09-16T03:36:58.367 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.421 INFO:teuthology.orchestra.run.smithi064.stdout:94489280515 2024-09-16T03:36:58.428 INFO:teuthology.orchestra.run.smithi064.stdout:94489280515 2024-09-16T03:36:58.438 INFO:teuthology.orchestra.run.smithi064.stdout:90194313220 2024-09-16T03:36:58.451 INFO:teuthology.orchestra.run.smithi064.stdout:98784247811 2024-09-16T03:36:58.461 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.7 2024-09-16T03:36:58.462 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.464 INFO:teuthology.orchestra.run.smithi064.stdout:94489280515 2024-09-16T03:36:58.466 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.5 2024-09-16T03:36:58.466 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.484 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247811 for osd.15 2024-09-16T03:36:58.484 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.501 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.8 2024-09-16T03:36:58.501 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.505 INFO:teuthology.orchestra.run.smithi064.stdout:94489280516 2024-09-16T03:36:58.508 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.9 2024-09-16T03:36:58.508 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.509 INFO:teuthology.orchestra.run.smithi064.stdout:90194313219 2024-09-16T03:36:58.516 INFO:teuthology.orchestra.run.smithi064.stdout:94489280516 2024-09-16T03:36:58.527 INFO:teuthology.orchestra.run.smithi064.stdout:94489280515 2024-09-16T03:36:58.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.12 2024-09-16T03:36:58.532 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.533 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.10 2024-09-16T03:36:58.533 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.534 INFO:teuthology.orchestra.run.smithi064.stdout:94489280516 2024-09-16T03:36:58.538 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.0 2024-09-16T03:36:58.538 INFO:teuthology.orchestra.run.smithi064.stdout:98784247812 2024-09-16T03:36:58.538 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.543 INFO:teuthology.orchestra.run.smithi064.stdout:94489280516 2024-09-16T03:36:58.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.1 2024-09-16T03:36:58.545 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.552 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.13 2024-09-16T03:36:58.552 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.555 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.11 2024-09-16T03:36:58.555 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.559 INFO:teuthology.orchestra.run.smithi064.stdout:94489280515 2024-09-16T03:36:58.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.3 2024-09-16T03:36:58.564 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.572 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.4 2024-09-16T03:36:58.573 DEBUG:teuthology.parallel:result is None 2024-09-16T03:36:58.573 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T03:36:58.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:36:58.909 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:36:58.909 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:36:58.921 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-16T03:36:57.677699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":82,"num_write_kb":1111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":444088,"kb_used_data":3296,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509505352,"statfs":{"total":1546188226560,"available":1545733480448,"internally_reserved":0,"allocated":3375104,"data_stored":2768374,"data_compressed":68744,"data_compressed_allocated":94208,"data_compressed_original":1228800,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.100212+0000","last_change":"2024-09-16T03:36:55.100546+0000","last_active":"2024-09-16T03:36:55.100212+0000","last_peered":"2024-09-16T03:36:55.100212+0000","last_clean":"2024-09-16T03:36:55.100212+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:36:55.100212+0000","last_undegraded":"2024-09-16T03:36:55.100212+0000","last_fullsized":"2024-09-16T03:36:55.100212+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.896892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.883127+0000","last_change":"2024-09-16T03:36:55.883348+0000","last_active":"2024-09-16T03:36:55.883127+0000","last_peered":"2024-09-16T03:36:55.883127+0000","last_clean":"2024-09-16T03:36:55.883127+0000","last_became_active":"2024-09-16T03:36:53.672370+0000","last_became_peered":"2024-09-16T03:36:53.672370+0000","last_unstale":"2024-09-16T03:36:55.883127+0000","last_undegraded":"2024-09-16T03:36:55.883127+0000","last_fullsized":"2024-09-16T03:36:55.883127+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:52.231720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.605610+0000","last_change":"2024-09-16T03:36:55.605861+0000","last_active":"2024-09-16T03:36:55.605610+0000","last_peered":"2024-09-16T03:36:55.605610+0000","last_clean":"2024-09-16T03:36:55.605610+0000","last_became_active":"2024-09-16T03:36:53.379870+0000","last_became_peered":"2024-09-16T03:36:53.379870+0000","last_unstale":"2024-09-16T03:36:55.605610+0000","last_undegraded":"2024-09-16T03:36:55.605610+0000","last_fullsized":"2024-09-16T03:36:55.605610+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:04.628405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044368400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.270467+0000","last_change":"2024-09-16T03:36:55.270467+0000","last_active":"2024-09-16T03:36:55.270467+0000","last_peered":"2024-09-16T03:36:55.270467+0000","last_clean":"2024-09-16T03:36:55.270467+0000","last_became_active":"2024-09-16T03:36:53.570778+0000","last_became_peered":"2024-09-16T03:36:53.570778+0000","last_unstale":"2024-09-16T03:36:55.270467+0000","last_undegraded":"2024-09-16T03:36:55.270467+0000","last_fullsized":"2024-09-16T03:36:55.270467+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:03.986104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.201161+0000","last_change":"2024-09-16T03:36:55.201161+0000","last_active":"2024-09-16T03:36:55.201161+0000","last_peered":"2024-09-16T03:36:55.201161+0000","last_clean":"2024-09-16T03:36:55.201161+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:36:55.201161+0000","last_undegraded":"2024-09-16T03:36:55.201161+0000","last_fullsized":"2024-09-16T03:36:55.201161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:07.449762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:56.004079+0000","last_change":"2024-09-16T03:36:56.004079+0000","last_active":"2024-09-16T03:36:56.004079+0000","last_peered":"2024-09-16T03:36:56.004079+0000","last_clean":"2024-09-16T03:36:56.004079+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:36:56.004079+0000","last_undegraded":"2024-09-16T03:36:56.004079+0000","last_fullsized":"2024-09-16T03:36:56.004079+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.599963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.200242+0000","last_change":"2024-09-16T03:36:55.200565+0000","last_active":"2024-09-16T03:36:55.200242+0000","last_peered":"2024-09-16T03:36:55.200242+0000","last_clean":"2024-09-16T03:36:55.200242+0000","last_became_active":"2024-09-16T03:36:53.182912+0000","last_became_peered":"2024-09-16T03:36:53.182912+0000","last_unstale":"2024-09-16T03:36:55.200242+0000","last_undegraded":"2024-09-16T03:36:55.200242+0000","last_fullsized":"2024-09-16T03:36:55.200242+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:08.335036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.504324+0000","last_change":"2024-09-16T03:36:55.504324+0000","last_active":"2024-09-16T03:36:55.504324+0000","last_peered":"2024-09-16T03:36:55.504324+0000","last_clean":"2024-09-16T03:36:55.504324+0000","last_became_active":"2024-09-16T03:36:53.284975+0000","last_became_peered":"2024-09-16T03:36:53.284975+0000","last_unstale":"2024-09-16T03:36:55.504324+0000","last_undegraded":"2024-09-16T03:36:55.504324+0000","last_fullsized":"2024-09-16T03:36:55.504324+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:55.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"28'54","reported_seq":87,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:56.977774+0000","last_change":"2024-09-16T03:36:51.858122+0000","last_active":"2024-09-16T03:36:56.977774+0000","last_peered":"2024-09-16T03:36:56.977774+0000","last_clean":"2024-09-16T03:36:56.977774+0000","last_became_active":"2024-09-16T03:36:51.855488+0000","last_became_peered":"2024-09-16T03:36:51.855488+0000","last_unstale":"2024-09-16T03:36:56.977774+0000","last_undegraded":"2024-09-16T03:36:56.977774+0000","last_fullsized":"2024-09-16T03:36:56.977774+0000","mapping_epoch":23,"log_start":"28'51","ondisk_log_start":"28'51","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:36:50.058111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":51,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:54.474463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":80,"num_write_kb":1109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":80,"num_write_kb":1109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1402464,"data_compressed":78696,"data_compressed_allocated":110592,"data_compressed_original":1351680,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":88574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247812,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":495827,"data_compressed":21256,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,5,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247811,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":172032,"data_stored":86569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":498958,"data_compressed":21256,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":95503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":94663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,5,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":96405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":184320,"data_stored":90880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":95503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":294912,"data_stored":561278,"data_compressed":26232,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":196608,"data_stored":94663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280515,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":94072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":95503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":95522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,7,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":97885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":86569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":26232,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":26232,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":26232,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:36:58.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:36:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:36:59.219 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:36:59.231 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-16T03:36:57.677699+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":82,"num_write_kb":1111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":444088,"kb_used_data":3296,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509505352,"statfs":{"total":1546188226560,"available":1545733480448,"internally_reserved":0,"allocated":3375104,"data_stored":2768374,"data_compressed":68744,"data_compressed_allocated":94208,"data_compressed_original":1228800,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.100212+0000","last_change":"2024-09-16T03:36:55.100546+0000","last_active":"2024-09-16T03:36:55.100212+0000","last_peered":"2024-09-16T03:36:55.100212+0000","last_clean":"2024-09-16T03:36:55.100212+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:36:55.100212+0000","last_undegraded":"2024-09-16T03:36:55.100212+0000","last_fullsized":"2024-09-16T03:36:55.100212+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.896892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.883127+0000","last_change":"2024-09-16T03:36:55.883348+0000","last_active":"2024-09-16T03:36:55.883127+0000","last_peered":"2024-09-16T03:36:55.883127+0000","last_clean":"2024-09-16T03:36:55.883127+0000","last_became_active":"2024-09-16T03:36:53.672370+0000","last_became_peered":"2024-09-16T03:36:53.672370+0000","last_unstale":"2024-09-16T03:36:55.883127+0000","last_undegraded":"2024-09-16T03:36:55.883127+0000","last_fullsized":"2024-09-16T03:36:55.883127+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:52.231720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.605610+0000","last_change":"2024-09-16T03:36:55.605861+0000","last_active":"2024-09-16T03:36:55.605610+0000","last_peered":"2024-09-16T03:36:55.605610+0000","last_clean":"2024-09-16T03:36:55.605610+0000","last_became_active":"2024-09-16T03:36:53.379870+0000","last_became_peered":"2024-09-16T03:36:53.379870+0000","last_unstale":"2024-09-16T03:36:55.605610+0000","last_undegraded":"2024-09-16T03:36:55.605610+0000","last_fullsized":"2024-09-16T03:36:55.605610+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:04.628405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044368400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.270467+0000","last_change":"2024-09-16T03:36:55.270467+0000","last_active":"2024-09-16T03:36:55.270467+0000","last_peered":"2024-09-16T03:36:55.270467+0000","last_clean":"2024-09-16T03:36:55.270467+0000","last_became_active":"2024-09-16T03:36:53.570778+0000","last_became_peered":"2024-09-16T03:36:53.570778+0000","last_unstale":"2024-09-16T03:36:55.270467+0000","last_undegraded":"2024-09-16T03:36:55.270467+0000","last_fullsized":"2024-09-16T03:36:55.270467+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:03.986104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.201161+0000","last_change":"2024-09-16T03:36:55.201161+0000","last_active":"2024-09-16T03:36:55.201161+0000","last_peered":"2024-09-16T03:36:55.201161+0000","last_clean":"2024-09-16T03:36:55.201161+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:36:55.201161+0000","last_undegraded":"2024-09-16T03:36:55.201161+0000","last_fullsized":"2024-09-16T03:36:55.201161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:07.449762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:56.004079+0000","last_change":"2024-09-16T03:36:56.004079+0000","last_active":"2024-09-16T03:36:56.004079+0000","last_peered":"2024-09-16T03:36:56.004079+0000","last_clean":"2024-09-16T03:36:56.004079+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:36:56.004079+0000","last_undegraded":"2024-09-16T03:36:56.004079+0000","last_fullsized":"2024-09-16T03:36:56.004079+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.599963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.200242+0000","last_change":"2024-09-16T03:36:55.200565+0000","last_active":"2024-09-16T03:36:55.200242+0000","last_peered":"2024-09-16T03:36:55.200242+0000","last_clean":"2024-09-16T03:36:55.200242+0000","last_became_active":"2024-09-16T03:36:53.182912+0000","last_became_peered":"2024-09-16T03:36:53.182912+0000","last_unstale":"2024-09-16T03:36:55.200242+0000","last_undegraded":"2024-09-16T03:36:55.200242+0000","last_fullsized":"2024-09-16T03:36:55.200242+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:08.335036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.504324+0000","last_change":"2024-09-16T03:36:55.504324+0000","last_active":"2024-09-16T03:36:55.504324+0000","last_peered":"2024-09-16T03:36:55.504324+0000","last_clean":"2024-09-16T03:36:55.504324+0000","last_became_active":"2024-09-16T03:36:53.284975+0000","last_became_peered":"2024-09-16T03:36:53.284975+0000","last_unstale":"2024-09-16T03:36:55.504324+0000","last_undegraded":"2024-09-16T03:36:55.504324+0000","last_fullsized":"2024-09-16T03:36:55.504324+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:55.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"28'54","reported_seq":87,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:56.977774+0000","last_change":"2024-09-16T03:36:51.858122+0000","last_active":"2024-09-16T03:36:56.977774+0000","last_peered":"2024-09-16T03:36:56.977774+0000","last_clean":"2024-09-16T03:36:56.977774+0000","last_became_active":"2024-09-16T03:36:51.855488+0000","last_became_peered":"2024-09-16T03:36:51.855488+0000","last_unstale":"2024-09-16T03:36:56.977774+0000","last_undegraded":"2024-09-16T03:36:56.977774+0000","last_fullsized":"2024-09-16T03:36:56.977774+0000","mapping_epoch":23,"log_start":"28'51","ondisk_log_start":"28'51","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:36:50.058111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":51,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:54.474463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":80,"num_write_kb":1109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":44,"num_write":80,"num_write_kb":1109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1402464,"data_compressed":78696,"data_compressed_allocated":110592,"data_compressed_original":1351680,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247811,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":88574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247812,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":245760,"data_stored":495827,"data_compressed":21256,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,5,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247811,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":172032,"data_stored":86569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":498958,"data_compressed":21256,"data_compressed_allocated":28672,"data_compressed_original":389120,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":95503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":94663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,5,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":96405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":184320,"data_stored":90880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280515,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":95503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":294912,"data_stored":561278,"data_compressed":26232,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":196608,"data_stored":94663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280515,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":94072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280515,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":95503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":95522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,7,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313219,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":97885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247812,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":86569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":26232,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":26232,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":26232,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:36:59.232 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T03:36:59.232 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T03:36:59.232 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T03:36:59.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T03:36:59.567 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:36:59.567 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T03:36:59.580 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T03:36:59.581 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T03:36:59.591 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T03:36:59.592 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-09-16T03:36:59.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! ceph balancer status ; do sleep 1 ; done' 2024-09-16T03:36:59.891 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:36:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "active": true, 2024-09-16T03:36:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "last_optimize_duration": "0:00:00.000393", 2024-09-16T03:36:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "last_optimize_started": "Mon Sep 16 03:36:41 2024", 2024-09-16T03:36:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "mode": "upmap", 2024-09-16T03:36:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "no_optimization_needed": false, 2024-09-16T03:36:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "optimize_result": "No pools available", 2024-09-16T03:36:59.891 INFO:teuthology.orchestra.run.smithi064.stdout: "plans": [] 2024-09-16T03:36:59.891 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:36:59.903 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode crush-compat' 2024-09-16T03:37:00.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-09-16T03:37:01.255 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-16T03:37:01.265 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 2, 'thrash_hosts': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-16T03:37:01.266 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'min_in': 4}, 'workunit': {'branch': 'main', 'sha1': '7498493279982dfe87f00616198a5967475f6169'}} 2024-09-16T03:37:01.266 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 4, 'thrash_hosts': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-16T03:37:01.266 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-16T03:37:01.266 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T03:37:01.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:37:01.423 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:01.421+0000 7f66f5209640 -1 osd.15 28 handle_osd_map injecting map crc failure 2024-09-16T03:37:01.564 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:37:01.564 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:37:01.575 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T03:37:01.331413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":122,"num_write_kb":2088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":444180,"kb_used_data":3312,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509505260,"statfs":{"total":1546188226560,"available":1545733386240,"internally_reserved":0,"allocated":3391488,"data_stored":2892318,"data_compressed":80605,"data_compressed_allocated":110592,"data_compressed_original":1351680,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":40,"num_write_kb":977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.653714"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.100212+0000","last_change":"2024-09-16T03:36:55.100546+0000","last_active":"2024-09-16T03:36:55.100212+0000","last_peered":"2024-09-16T03:36:55.100212+0000","last_clean":"2024-09-16T03:36:55.100212+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:36:55.100212+0000","last_undegraded":"2024-09-16T03:36:55.100212+0000","last_fullsized":"2024-09-16T03:36:55.100212+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.896892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.883127+0000","last_change":"2024-09-16T03:36:55.883348+0000","last_active":"2024-09-16T03:36:55.883127+0000","last_peered":"2024-09-16T03:36:55.883127+0000","last_clean":"2024-09-16T03:36:55.883127+0000","last_became_active":"2024-09-16T03:36:53.672370+0000","last_became_peered":"2024-09-16T03:36:53.672370+0000","last_unstale":"2024-09-16T03:36:55.883127+0000","last_undegraded":"2024-09-16T03:36:55.883127+0000","last_fullsized":"2024-09-16T03:36:55.883127+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:52.231720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.605610+0000","last_change":"2024-09-16T03:36:55.605861+0000","last_active":"2024-09-16T03:36:55.605610+0000","last_peered":"2024-09-16T03:36:55.605610+0000","last_clean":"2024-09-16T03:36:55.605610+0000","last_became_active":"2024-09-16T03:36:53.379870+0000","last_became_peered":"2024-09-16T03:36:53.379870+0000","last_unstale":"2024-09-16T03:36:55.605610+0000","last_undegraded":"2024-09-16T03:36:55.605610+0000","last_fullsized":"2024-09-16T03:36:55.605610+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:04.628405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044368400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.270467+0000","last_change":"2024-09-16T03:36:55.270467+0000","last_active":"2024-09-16T03:36:55.270467+0000","last_peered":"2024-09-16T03:36:55.270467+0000","last_clean":"2024-09-16T03:36:55.270467+0000","last_became_active":"2024-09-16T03:36:53.570778+0000","last_became_peered":"2024-09-16T03:36:53.570778+0000","last_unstale":"2024-09-16T03:36:55.270467+0000","last_undegraded":"2024-09-16T03:36:55.270467+0000","last_fullsized":"2024-09-16T03:36:55.270467+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:03.986104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.201161+0000","last_change":"2024-09-16T03:36:55.201161+0000","last_active":"2024-09-16T03:36:55.201161+0000","last_peered":"2024-09-16T03:36:55.201161+0000","last_clean":"2024-09-16T03:36:55.201161+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:36:55.201161+0000","last_undegraded":"2024-09-16T03:36:55.201161+0000","last_fullsized":"2024-09-16T03:36:55.201161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:07.449762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:56.004079+0000","last_change":"2024-09-16T03:36:56.004079+0000","last_active":"2024-09-16T03:36:56.004079+0000","last_peered":"2024-09-16T03:36:56.004079+0000","last_clean":"2024-09-16T03:36:56.004079+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:36:56.004079+0000","last_undegraded":"2024-09-16T03:36:56.004079+0000","last_fullsized":"2024-09-16T03:36:56.004079+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.599963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.200242+0000","last_change":"2024-09-16T03:36:55.200565+0000","last_active":"2024-09-16T03:36:55.200242+0000","last_peered":"2024-09-16T03:36:55.200242+0000","last_clean":"2024-09-16T03:36:55.200242+0000","last_became_active":"2024-09-16T03:36:53.182912+0000","last_became_peered":"2024-09-16T03:36:53.182912+0000","last_unstale":"2024-09-16T03:36:55.200242+0000","last_undegraded":"2024-09-16T03:36:55.200242+0000","last_fullsized":"2024-09-16T03:36:55.200242+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:08.335036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-16T03:37:01.012438+0000","last_change":"2024-09-16T03:36:55.504324+0000","last_active":"2024-09-16T03:37:01.012438+0000","last_peered":"2024-09-16T03:37:01.012438+0000","last_clean":"2024-09-16T03:37:01.012438+0000","last_became_active":"2024-09-16T03:36:53.284975+0000","last_became_peered":"2024-09-16T03:36:53.284975+0000","last_unstale":"2024-09-16T03:37:01.012438+0000","last_undegraded":"2024-09-16T03:37:01.012438+0000","last_fullsized":"2024-09-16T03:37:01.012438+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:55.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"28'90","reported_seq":129,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:37:00.966803+0000","last_change":"2024-09-16T03:36:51.858122+0000","last_active":"2024-09-16T03:37:00.966803+0000","last_peered":"2024-09-16T03:37:00.966803+0000","last_clean":"2024-09-16T03:37:00.966803+0000","last_became_active":"2024-09-16T03:36:51.855488+0000","last_became_peered":"2024-09-16T03:36:51.855488+0000","last_unstale":"2024-09-16T03:37:00.966803+0000","last_undegraded":"2024-09-16T03:37:00.966803+0000","last_fullsized":"2024-09-16T03:37:00.966803+0000","mapping_epoch":23,"log_start":"28'86","ondisk_log_start":"28'86","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:36:50.058111+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":86,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:54.474463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":120,"num_write_kb":2086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":120,"num_write_kb":2086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1402464,"data_compressed":81399,"data_compressed_allocated":110592,"data_compressed_original":1351680,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":88574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":253952,"data_stored":557799,"data_compressed":27421,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,5,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":172032,"data_stored":86569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":249856,"data_stored":560930,"data_compressed":26592,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":95503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":94663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,5,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":96405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":184320,"data_stored":90880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":95503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":294912,"data_stored":561278,"data_compressed":26592,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":196608,"data_stored":94663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":94072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":95503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":95522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,7,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":97885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":86569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":27093,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":26592,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":27714,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:37:01.577 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:37:01.577 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:01.574+0000 7f0ae754b640 -1 osd.0 29 handle_osd_map injecting map crc failure 2024-09-16T03:37:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:37:01.881 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:37:01.886 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T03:37:01.678448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":122,"num_write_kb":2088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":444180,"kb_used_data":3312,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509505260,"statfs":{"total":1546188226560,"available":1545733386240,"internally_reserved":0,"allocated":3391488,"data_stored":2892318,"data_compressed":80605,"data_compressed_allocated":110592,"data_compressed_original":1351680,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":40,"num_write_kb":977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000749"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.100212+0000","last_change":"2024-09-16T03:36:55.100546+0000","last_active":"2024-09-16T03:36:55.100212+0000","last_peered":"2024-09-16T03:36:55.100212+0000","last_clean":"2024-09-16T03:36:55.100212+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:36:55.100212+0000","last_undegraded":"2024-09-16T03:36:55.100212+0000","last_fullsized":"2024-09-16T03:36:55.100212+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.896892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.883127+0000","last_change":"2024-09-16T03:36:55.883348+0000","last_active":"2024-09-16T03:36:55.883127+0000","last_peered":"2024-09-16T03:36:55.883127+0000","last_clean":"2024-09-16T03:36:55.883127+0000","last_became_active":"2024-09-16T03:36:53.672370+0000","last_became_peered":"2024-09-16T03:36:53.672370+0000","last_unstale":"2024-09-16T03:36:55.883127+0000","last_undegraded":"2024-09-16T03:36:55.883127+0000","last_fullsized":"2024-09-16T03:36:55.883127+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:52.231720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.605610+0000","last_change":"2024-09-16T03:36:55.605861+0000","last_active":"2024-09-16T03:36:55.605610+0000","last_peered":"2024-09-16T03:36:55.605610+0000","last_clean":"2024-09-16T03:36:55.605610+0000","last_became_active":"2024-09-16T03:36:53.379870+0000","last_became_peered":"2024-09-16T03:36:53.379870+0000","last_unstale":"2024-09-16T03:36:55.605610+0000","last_undegraded":"2024-09-16T03:36:55.605610+0000","last_fullsized":"2024-09-16T03:36:55.605610+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:04.628405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044368400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.270467+0000","last_change":"2024-09-16T03:36:55.270467+0000","last_active":"2024-09-16T03:36:55.270467+0000","last_peered":"2024-09-16T03:36:55.270467+0000","last_clean":"2024-09-16T03:36:55.270467+0000","last_became_active":"2024-09-16T03:36:53.570778+0000","last_became_peered":"2024-09-16T03:36:53.570778+0000","last_unstale":"2024-09-16T03:36:55.270467+0000","last_undegraded":"2024-09-16T03:36:55.270467+0000","last_fullsized":"2024-09-16T03:36:55.270467+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:03.986104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":23,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.201161+0000","last_change":"2024-09-16T03:36:55.201161+0000","last_active":"2024-09-16T03:36:55.201161+0000","last_peered":"2024-09-16T03:36:55.201161+0000","last_clean":"2024-09-16T03:36:55.201161+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:36:55.201161+0000","last_undegraded":"2024-09-16T03:36:55.201161+0000","last_fullsized":"2024-09-16T03:36:55.201161+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:07.449762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:56.004079+0000","last_change":"2024-09-16T03:36:56.004079+0000","last_active":"2024-09-16T03:36:56.004079+0000","last_peered":"2024-09-16T03:36:56.004079+0000","last_clean":"2024-09-16T03:36:56.004079+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:36:56.004079+0000","last_undegraded":"2024-09-16T03:36:56.004079+0000","last_fullsized":"2024-09-16T03:36:56.004079+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.599963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:36:55.200242+0000","last_change":"2024-09-16T03:36:55.200565+0000","last_active":"2024-09-16T03:36:55.200242+0000","last_peered":"2024-09-16T03:36:55.200242+0000","last_clean":"2024-09-16T03:36:55.200242+0000","last_became_active":"2024-09-16T03:36:53.182912+0000","last_became_peered":"2024-09-16T03:36:53.182912+0000","last_unstale":"2024-09-16T03:36:55.200242+0000","last_undegraded":"2024-09-16T03:36:55.200242+0000","last_fullsized":"2024-09-16T03:36:55.200242+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:08.335036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-16T03:37:01.012438+0000","last_change":"2024-09-16T03:36:55.504324+0000","last_active":"2024-09-16T03:37:01.012438+0000","last_peered":"2024-09-16T03:37:01.012438+0000","last_clean":"2024-09-16T03:37:01.012438+0000","last_became_active":"2024-09-16T03:36:53.284975+0000","last_became_peered":"2024-09-16T03:36:53.284975+0000","last_unstale":"2024-09-16T03:37:01.012438+0000","last_undegraded":"2024-09-16T03:37:01.012438+0000","last_fullsized":"2024-09-16T03:37:01.012438+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:55.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"28'90","reported_seq":129,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T03:37:00.966803+0000","last_change":"2024-09-16T03:36:51.858122+0000","last_active":"2024-09-16T03:37:00.966803+0000","last_peered":"2024-09-16T03:37:00.966803+0000","last_clean":"2024-09-16T03:37:00.966803+0000","last_became_active":"2024-09-16T03:36:51.855488+0000","last_became_peered":"2024-09-16T03:36:51.855488+0000","last_unstale":"2024-09-16T03:37:00.966803+0000","last_undegraded":"2024-09-16T03:37:00.966803+0000","last_fullsized":"2024-09-16T03:37:00.966803+0000","mapping_epoch":23,"log_start":"28'86","ondisk_log_start":"28'86","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:36:50.058111+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":86,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:54.474463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":120,"num_write_kb":2086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":120,"num_write_kb":2086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":1402464,"data_compressed":81399,"data_compressed_allocated":110592,"data_compressed_original":1351680,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":88574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":253952,"data_stored":557799,"data_compressed":27421,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,5,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":172032,"data_stored":86569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":249856,"data_stored":560930,"data_compressed":26592,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":95503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":94663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,5,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":245760,"data_stored":96405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":184320,"data_stored":90880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":95503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":294912,"data_stored":561278,"data_compressed":26592,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":196608,"data_stored":94663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":94072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":208896,"data_stored":95503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":212992,"data_stored":95522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,7,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313220,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27240,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":212992,"data_stored":97885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":86569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":27093,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":26592,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":27714,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:37:01.887 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T03:37:01.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-16T03:37:02.192 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 30 2024-09-16T03:37:02.192 INFO:teuthology.orchestra.run.smithi064.stdout:fsid b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2 2024-09-16T03:37:02.192 INFO:teuthology.orchestra.run.smithi064.stdout:created 2024-09-16T03:36:37.310421+0000 2024-09-16T03:37:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:modified 2024-09-16T03:37:01.327074+0000 2024-09-16T03:37:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T03:37:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:crush_version 7 2024-09-16T03:37:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:full_ratio 0.95 2024-09-16T03:37:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:backfillfull_ratio 0.9 2024-09-16T03:37:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:nearfull_ratio 0.85 2024-09-16T03:37:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:require_min_compat_client luminous 2024-09-16T03:37:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:min_compat_client jewel 2024-09-16T03:37:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:require_osd_release squid 2024-09-16T03:37:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:stretch_mode_enabled false 2024-09-16T03:37:02.193 INFO:teuthology.orchestra.run.smithi064.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.79 2024-09-16T03:37:02.194 INFO:teuthology.orchestra.run.smithi064.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 5.97 2024-09-16T03:37:02.194 INFO:teuthology.orchestra.run.smithi064.stdout:max_osd 16 2024-09-16T03:37:02.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6813/3924351635 v1:172.21.15.64:6814/3924351635 exists,up bdbe961a-78d3-4f7d-88ed-ce24475d49e3 2024-09-16T03:37:02.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.112:6804/624977804 v1:172.21.15.112:6805/624977804 exists,up 154c8ddd-7a3d-402c-b9d6-28538f4e6bc4 2024-09-16T03:37:02.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.155:6812/2804648086 v1:172.21.15.155:6813/2804648086 exists,up 2a06de4e-0835-4a42-8c8a-49fb55aab566 2024-09-16T03:37:02.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.157:6800/1935597988 v1:172.21.15.157:6801/1935597988 exists,up 8a87f9f0-c380-4f38-bbd2-06b2def4fd2d 2024-09-16T03:37:02.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6809/2675475248 v1:172.21.15.64:6810/2675475248 exists,up 5ac53318-12e8-496c-a87f-ba5de79c52aa 2024-09-16T03:37:02.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.112:6800/499905276 v1:172.21.15.112:6801/499905276 exists,up 675c81a4-2314-463e-82a7-0060735c78fd 2024-09-16T03:37:02.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.155:6808/2839983406 v1:172.21.15.155:6809/2839983406 exists,up b18962a7-3b23-4e50-bf31-1d0629557423 2024-09-16T03:37:02.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.157:6808/3021982871 v1:172.21.15.157:6809/3021982871 exists,up 9706c1e8-dac0-4e16-8bc5-7cacc6f13870 2024-09-16T03:37:02.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6805/3609017627 v1:172.21.15.64:6806/3609017627 exists,up fafd06b8-c005-4471-a128-9dd51580c0f0 2024-09-16T03:37:02.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.112:6812/701981815 v1:172.21.15.112:6813/701981815 exists,up 79e38e57-52c2-416f-9b90-75621f06d3e3 2024-09-16T03:37:02.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.155:6804/4138619523 v1:172.21.15.155:6805/4138619523 exists,up 8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10 2024-09-16T03:37:02.195 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.157:6804/3607367937 v1:172.21.15.157:6805/3607367937 exists,up 7a2a169a-ce94-4091-a792-27d0ecf9a7fa 2024-09-16T03:37:02.195 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6801/2984461266 v1:172.21.15.64:6802/2984461266 exists,up 7a280cda-b740-40c8-9fae-1a1ff11651f6 2024-09-16T03:37:02.195 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.112:6808/1616314527 v1:172.21.15.112:6809/1616314527 exists,up ce3a3148-cd90-422c-a106-045de89ab045 2024-09-16T03:37:02.195 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.155:6800/4203868121 v1:172.21.15.155:6801/4203868121 exists,up 461c2d79-f4e7-4b21-899c-50d4a6cf12f9 2024-09-16T03:37:02.195 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.157:6812/181354529 v1:172.21.15.157:6813/181354529 exists,up 9a8c0bbe-9389-4b2c-95e1-34d3e66461e9 2024-09-16T03:37:02.205 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6813/3924351635 v1:172.21.15.64:6814/3924351635 exists,up bdbe961a-78d3-4f7d-88ed-ce24475d49e3', 'osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.112:6804/624977804 v1:172.21.15.112:6805/624977804 exists,up 154c8ddd-7a3d-402c-b9d6-28538f4e6bc4', 'osd.2 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.155:6812/2804648086 v1:172.21.15.155:6813/2804648086 exists,up 2a06de4e-0835-4a42-8c8a-49fb55aab566', 'osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.157:6800/1935597988 v1:172.21.15.157:6801/1935597988 exists,up 8a87f9f0-c380-4f38-bbd2-06b2def4fd2d', 'osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6809/2675475248 v1:172.21.15.64:6810/2675475248 exists,up 5ac53318-12e8-496c-a87f-ba5de79c52aa', 'osd.5 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.112:6800/499905276 v1:172.21.15.112:6801/499905276 exists,up 675c81a4-2314-463e-82a7-0060735c78fd', 'osd.6 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.155:6808/2839983406 v1:172.21.15.155:6809/2839983406 exists,up b18962a7-3b23-4e50-bf31-1d0629557423', 'osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.157:6808/3021982871 v1:172.21.15.157:6809/3021982871 exists,up 9706c1e8-dac0-4e16-8bc5-7cacc6f13870', 'osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6805/3609017627 v1:172.21.15.64:6806/3609017627 exists,up fafd06b8-c005-4471-a128-9dd51580c0f0', 'osd.9 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.112:6812/701981815 v1:172.21.15.112:6813/701981815 exists,up 79e38e57-52c2-416f-9b90-75621f06d3e3', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.155:6804/4138619523 v1:172.21.15.155:6805/4138619523 exists,up 8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10', 'osd.11 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.157:6804/3607367937 v1:172.21.15.157:6805/3607367937 exists,up 7a2a169a-ce94-4091-a792-27d0ecf9a7fa', 'osd.12 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.64:6801/2984461266 v1:172.21.15.64:6802/2984461266 exists,up 7a280cda-b740-40c8-9fae-1a1ff11651f6', 'osd.13 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.112:6808/1616314527 v1:172.21.15.112:6809/1616314527 exists,up ce3a3148-cd90-422c-a106-045de89ab045', 'osd.14 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.155:6800/4203868121 v1:172.21.15.155:6801/4203868121 exists,up 461c2d79-f4e7-4b21-899c-50d4a6cf12f9', 'osd.15 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.157:6812/181354529 v1:172.21.15.157:6813/181354529 exists,up 9a8c0bbe-9389-4b2c-95e1-34d3e66461e9'] 2024-09-16T03:37:02.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-16T03:37:02.309 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "name": "mon.a", 2024-09-16T03:37:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster": "ceph", 2024-09-16T03:37:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-16T03:37:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "admin_socket_mode": "", 2024-09-16T03:37:02.309 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-16T03:37:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_client_required": "cephx, none", 2024-09-16T03:37:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_cluster_required": "cephx", 2024-09-16T03:37:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_debug": "true", 2024-09-16T03:37:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-16T03:37:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-16T03:37:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_service_required": "cephx", 2024-09-16T03:37:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-16T03:37:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "auth_supported": "cephx", 2024-09-16T03:37:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio": "true", 2024-09-16T03:37:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-16T03:37:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio_poll_ms": "250", 2024-09-16T03:37:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_aio_reap_max": "16", 2024-09-16T03:37:02.310 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_async_discard_threads": "0", 2024-09-16T03:37:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_block_size": "4096", 2024-09-16T03:37:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_aio": "false", 2024-09-16T03:37:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-16T03:37:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-16T03:37:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_debug_inflight_ios": "false", 2024-09-16T03:37:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_enable_discard": "false", 2024-09-16T03:37:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_flock_retry": "3", 2024-09-16T03:37:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-16T03:37:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_inject_crash": "0", 2024-09-16T03:37:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-16T03:37:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_ioring": "false", 2024-09-16T03:37:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_ioring_hipri": "false", 2024-09-16T03:37:02.311 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-16T03:37:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-16T03:37:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-16T03:37:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-16T03:37:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-16T03:37:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-16T03:37:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "bdev_type": "", 2024-09-16T03:37:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_alloc_size": "1048576", 2024-09-16T03:37:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_allocator": "hybrid", 2024-09-16T03:37:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_buffered_io": "true", 2024-09-16T03:37:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_check_for_zeros": "false", 2024-09-16T03:37:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-16T03:37:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-16T03:37:02.312 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_compact_log_sync": "false", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_max_log_runway": "4194304", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_max_prefetch": "1048576", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_min_flush_size": "524288", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_min_log_runway": "1048576", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_replay_recovery": "false", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_sync_write": "false", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-16T03:37:02.313 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-16T03:37:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-16T03:37:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_allocation_from_file": "true", 2024-09-16T03:37:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_allocator": "hybrid", 2024-09-16T03:37:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_async_db_compaction": "true", 2024-09-16T03:37:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-16T03:37:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-16T03:37:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-16T03:37:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-16T03:37:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bdev_label_multi": "true", 2024-09-16T03:37:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-16T03:37:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-16T03:37:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-16T03:37:02.314 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-16T03:37:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-16T03:37:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_create": "true", 2024-09-16T03:37:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_db_create": "false", 2024-09-16T03:37:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_db_path": "", 2024-09-16T03:37:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_db_size": "0", 2024-09-16T03:37:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_path": "", 2024-09-16T03:37:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_preallocate_file": "false", 2024-09-16T03:37:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_size": "107374182400", 2024-09-16T03:37:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_wal_create": "false", 2024-09-16T03:37:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_wal_path": "", 2024-09-16T03:37:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_block_wal_size": "100663296", 2024-09-16T03:37:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs": "true", 2024-09-16T03:37:02.315 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-16T03:37:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-16T03:37:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-16T03:37:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-16T03:37:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-16T03:37:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T03:37:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T03:37:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-16T03:37:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T03:37:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_autotune": "true", 2024-09-16T03:37:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-16T03:37:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-16T03:37:02.316 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-16T03:37:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-16T03:37:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_size": "0", 2024-09-16T03:37:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-16T03:37:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-16T03:37:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-16T03:37:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-16T03:37:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_type": "2q", 2024-09-16T03:37:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-16T03:37:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_clone_cow": "true", 2024-09-16T03:37:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-16T03:37:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-16T03:37:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-16T03:37:02.317 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_mode": "none", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_csum_type": "crc32c", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_freelist": "false", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-16T03:37:02.318 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-16T03:37:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-16T03:37:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-16T03:37:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-16T03:37:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-16T03:37:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-16T03:37:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-16T03:37:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-16T03:37:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_small_allocations": "0", 2024-09-16T03:37:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-16T03:37:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_default_buffered_read": "true", 2024-09-16T03:37:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_default_buffered_write": "false", 2024-09-16T03:37:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-16T03:37:02.319 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-16T03:37:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-16T03:37:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-16T03:37:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-16T03:37:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-16T03:37:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-16T03:37:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-16T03:37:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-16T03:37:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fail_eio": "false", 2024-09-16T03:37:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-16T03:37:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-16T03:37:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-16T03:37:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-16T03:37:02.320 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-16T03:37:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-16T03:37:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mount": "false", 2024-09-16T03:37:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-16T03:37:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_umount": "false", 2024-09-16T03:37:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-16T03:37:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-16T03:37:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-16T03:37:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-16T03:37:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-16T03:37:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-16T03:37:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-16T03:37:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-16T03:37:02.321 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_ignore_data_csum": "false", 2024-09-16T03:37:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-16T03:37:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-16T03:37:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-16T03:37:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-16T03:37:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_log_op_age": "5.000000", 2024-09-16T03:37:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_alloc_size": "0", 2024-09-16T03:37:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_blob_size": "0", 2024-09-16T03:37:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-16T03:37:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-16T03:37:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-16T03:37:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_max_deferred_txc": "32", 2024-09-16T03:37:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_min_alloc_size": "0", 2024-09-16T03:37:02.322 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-16T03:37:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-16T03:37:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_nid_prealloc": "1024", 2024-09-16T03:37:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-16T03:37:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-16T03:37:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-16T03:37:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-16T03:37:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_retry_disk_reads": "3", 2024-09-16T03:37:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_cf": "true", 2024-09-16T03:37:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-16T03:37:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-16T03:37:02.323 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-16T03:37:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-16T03:37:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-16T03:37:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-16T03:37:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-16T03:37:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-16T03:37:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_spdk_mem": "512", 2024-09-16T03:37:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-16T03:37:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-16T03:37:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-16T03:37:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-16T03:37:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-16T03:37:02.324 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_tracing": "false", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_write_v2": "false", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_write_v2_random": "false", 2024-09-16T03:37:02.325 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_zero_block_detection": "false", 2024-09-16T03:37:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-16T03:37:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-16T03:37:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-16T03:37:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-16T03:37:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-16T03:37:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-16T03:37:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-16T03:37:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-16T03:37:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-16T03:37:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-16T03:37:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-16T03:37:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-16T03:37:02.326 INFO:teuthology.orchestra.run.smithi064.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-16T03:37:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-16T03:37:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_cluster_require_signatures": "false", 2024-09-16T03:37:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_cluster_require_version": "2", 2024-09-16T03:37:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_require_signatures": "false", 2024-09-16T03:37:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_require_version": "2", 2024-09-16T03:37:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_service_require_signatures": "false", 2024-09-16T03:37:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_service_require_version": "2", 2024-09-16T03:37:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "cephx_sign_messages": "true", 2024-09-16T03:37:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "chdir": "", 2024-09-16T03:37:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "client_acl_type": "", 2024-09-16T03:37:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "client_asio_thread_count": "2", 2024-09-16T03:37:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "client_cache_mid": "0.750000", 2024-09-16T03:37:02.327 INFO:teuthology.orchestra.run.smithi064.stdout: "client_cache_size": "16384", 2024-09-16T03:37:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_caps_release_delay": "5", 2024-09-16T03:37:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_check_pool_perm": "true", 2024-09-16T03:37:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-16T03:37:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_debug_force_sync_read": "false", 2024-09-16T03:37:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_debug_getattr_caps": "false", 2024-09-16T03:37:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_debug_inject_features": "", 2024-09-16T03:37:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_debug_inject_tick_delay": "0", 2024-09-16T03:37:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-16T03:37:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_die_on_failed_remount": "false", 2024-09-16T03:37:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_dirsize_rbytes": "true", 2024-09-16T03:37:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_force_lazyio": "false", 2024-09-16T03:37:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_fs": "", 2024-09-16T03:37:02.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-16T03:37:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_inject_release_failure": "false", 2024-09-16T03:37:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_max_inline_size": "4096", 2024-09-16T03:37:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-16T03:37:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mds_namespace": "", 2024-09-16T03:37:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_metadata": "", 2024-09-16T03:37:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mount_gid": "-1", 2024-09-16T03:37:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mount_timeout": "300", 2024-09-16T03:37:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mount_uid": "-1", 2024-09-16T03:37:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_mountpoint": "/", 2024-09-16T03:37:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_notify_timeout": "10", 2024-09-16T03:37:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc": "true", 2024-09-16T03:37:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_max_dirty": "104857600", 2024-09-16T03:37:02.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-16T03:37:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_max_objects": "1000", 2024-09-16T03:37:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_size": "209715200", 2024-09-16T03:37:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client_oc_target_dirty": "8388608", 2024-09-16T03:37:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client_permissions": "true", 2024-09-16T03:37:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client_quota": "true", 2024-09-16T03:37:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client_quota_df": "true", 2024-09-16T03:37:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client_readahead_max_bytes": "0", 2024-09-16T03:37:02.330 INFO:teuthology.orchestra.run.smithi064.stdout: "client_readahead_max_periods": "4", 2024-09-16T03:37:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_readahead_min": "131072", 2024-09-16T03:37:02.332 INFO:teuthology.orchestra.run.smithi064.stdout: "client_reconnect_stale": "false", 2024-09-16T03:37:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_shutdown_timeout": "30", 2024-09-16T03:37:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_snapdir": ".snap", 2024-09-16T03:37:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_tick_interval": "1", 2024-09-16T03:37:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_trace": "", 2024-09-16T03:37:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_try_dentry_invalidate": "false", 2024-09-16T03:37:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_use_faked_inos": "false", 2024-09-16T03:37:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_use_random_mds": "false", 2024-09-16T03:37:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_graylog": "false", 2024-09-16T03:37:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-16T03:37:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_graylog_port": "12201", 2024-09-16T03:37:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_monitors": "default=true", 2024-09-16T03:37:02.333 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_syslog": "false", 2024-09-16T03:37:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-16T03:37:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "clog_to_syslog_level": "info", 2024-09-16T03:37:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "-", 2024-09-16T03:37:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_network": "", 2024-09-16T03:37:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_network_interface": "", 2024-09-16T03:37:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zlib_isal": "false", 2024-09-16T03:37:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zlib_level": "5", 2024-09-16T03:37:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zlib_winsize": "-15", 2024-09-16T03:37:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "compressor_zstd_level": "1", 2024-09-16T03:37:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-16T03:37:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-16T03:37:02.334 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_alien_op_num_threads": "6", 2024-09-16T03:37:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-16T03:37:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-16T03:37:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-16T03:37:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_osd_stat_interval": "0", 2024-09-16T03:37:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_seastar_cpu_cores": "", 2024-09-16T03:37:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "crimson_seastar_num_threads": "0", 2024-09-16T03:37:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location": "", 2024-09-16T03:37:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location_hook": "", 2024-09-16T03:37:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_location_hook_timeout": "10", 2024-09-16T03:37:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "daemonize": "false", 2024-09-16T03:37:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "daos_pool": "tank", 2024-09-16T03:37:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-16T03:37:02.335 INFO:teuthology.orchestra.run.smithi064.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_alienstore": "0/5", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_allow_any_pool_priority": "false", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_asok": "1/5", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_asok_assert_abort": "false", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_asserts_on_shutdown": "true", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_auth": "20/20", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_bdev": "1/3", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_bluefs": "1/5", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_bluestore": "1/5", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_buffer": "0/1", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_ceph_dedup": "0/5", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_ceph_exporter": "1/5", 2024-09-16T03:37:02.336 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_cephfs_mirror": "0/5", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_cephsqlite": "0/5", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_client": "0/5", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_compressor": "1/5", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_context": "0/1", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_crimson_interrupt": "0/5", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_crush": "1/1", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_crypto": "1/5", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_cyanstore": "0/5", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_deliberately_leak_memory": "false", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_disable_randomized_ping": "false", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_dpdk": "1/5", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_eventtrace": "1/5", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_filer": "0/1", 2024-09-16T03:37:02.337 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_filestore": "1/3", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_finisher": "1/1", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_fuse": "1/5", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_heartbeat_testing_span": "0", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_heartbeatmap": "1/5", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_javaclient": "1/5", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_journal": "1/3", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_journaler": "0/5", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_kstore": "1/5", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_lockdep": "0/1", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mclock": "1/5", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds": "1/5", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_balancer": "1/5", 2024-09-16T03:37:02.338 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_locker": "1/5", 2024-09-16T03:37:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_log": "1/5", 2024-09-16T03:37:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_log_expire": "1/5", 2024-09-16T03:37:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_migrator": "1/5", 2024-09-16T03:37:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mds_quiesce": "3/5", 2024-09-16T03:37:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_memstore": "1/5", 2024-09-16T03:37:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mgr": "2/5", 2024-09-16T03:37:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mgrc": "1/5", 2024-09-16T03:37:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_mon": "20/20", 2024-09-16T03:37:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_monc": "0/10", 2024-09-16T03:37:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_ms": "1/1", 2024-09-16T03:37:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_none": "0/5", 2024-09-16T03:37:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_objclass": "0/5", 2024-09-16T03:37:02.339 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_objectcacher": "0/5", 2024-09-16T03:37:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_objecter": "0/1", 2024-09-16T03:37:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_optracker": "0/5", 2024-09-16T03:37:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_osd": "1/5", 2024-09-16T03:37:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_paxos": "20/20", 2024-09-16T03:37:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_perfcounter": "1/5", 2024-09-16T03:37:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_prioritycache": "1/5", 2024-09-16T03:37:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rados": "0/5", 2024-09-16T03:37:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd": "0/5", 2024-09-16T03:37:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd_mirror": "0/5", 2024-09-16T03:37:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd_pwl": "0/5", 2024-09-16T03:37:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rbd_replay": "0/5", 2024-09-16T03:37:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_refs": "0/0", 2024-09-16T03:37:02.340 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_reserver": "1/1", 2024-09-16T03:37:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw": "1/5", 2024-09-16T03:37:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_access": "1/5", 2024-09-16T03:37:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_datacache": "1/5", 2024-09-16T03:37:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_dbstore": "1/5", 2024-09-16T03:37:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_flight": "1/5", 2024-09-16T03:37:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-16T03:37:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_notification": "1/5", 2024-09-16T03:37:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rgw_sync": "1/5", 2024-09-16T03:37:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_rocksdb": "4/5", 2024-09-16T03:37:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore": "0/5", 2024-09-16T03:37:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_backref": "0/5", 2024-09-16T03:37:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_cache": "0/5", 2024-09-16T03:37:02.341 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_cleaner": "0/5", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_device": "0/5", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_epm": "0/5", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_journal": "0/5", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_lba": "0/5", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_odata": "0/5", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_omap": "0/5", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_onode": "0/5", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_t": "0/5", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_seastore_tm": "0/5", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_striper": "0/1", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_test": "0/5", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_throttle": "1/1", 2024-09-16T03:37:02.342 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_timer": "0/1", 2024-09-16T03:37:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_tp": "0/5", 2024-09-16T03:37:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "debug_trace": "1/5", 2024-09-16T03:37:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-16T03:37:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "device_failure_prediction_mode": "none", 2024-09-16T03:37:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-16T03:37:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-16T03:37:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_graylog": "false", 2024-09-16T03:37:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_journald": "false", 2024-09-16T03:37:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_stderr": "true", 2024-09-16T03:37:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "err_to_syslog": "false", 2024-09-16T03:37:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event_tracing": "false", 2024-09-16T03:37:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_addr": "0.0.0.0", 2024-09-16T03:37:02.343 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_cert_file": "", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_http_port": "9926", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_key_file": "", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_prio_limit": "5", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_sort_metrics": "true", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "exporter_stats_period": "5", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "fake_statfs_for_testing": "0", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "fatal_signal_handlers": "true", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "filer_max_purge_ops": "10", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "filer_max_truncate_ops": "128", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_apply_finisher_threads": "1", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_blackhole": "false", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_btrfs_clone_range": "true", 2024-09-16T03:37:02.344 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_btrfs_snap": "true", 2024-09-16T03:37:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_caller_concurrency": "10", 2024-09-16T03:37:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_collect_device_partition_information": "true", 2024-09-16T03:37:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_commit_timeout": "600.000000", 2024-09-16T03:37:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_inject_read_err": "false", 2024-09-16T03:37:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_omap_check": "false", 2024-09-16T03:37:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_debug_verify_split": "false", 2024-09-16T03:37:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_dump_file": "", 2024-09-16T03:37:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-16T03:37:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-16T03:37:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fadvise": "true", 2024-09-16T03:37:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fail_eio": "true", 2024-09-16T03:37:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fd_cache_shards": "16", 2024-09-16T03:37:02.345 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fd_cache_size": "128", 2024-09-16T03:37:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fiemap": "false", 2024-09-16T03:37:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fiemap_threshold": "4096", 2024-09-16T03:37:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-16T03:37:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-16T03:37:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_inject_stall": "0", 2024-09-16T03:37:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_journal_parallel": "false", 2024-09-16T03:37:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_journal_trailing": "false", 2024-09-16T03:37:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_journal_writeahead": "false", 2024-09-16T03:37:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_kill_at": "0", 2024-09-16T03:37:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-16T03:37:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-16T03:37:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-16T03:37:02.346 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-16T03:37:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-16T03:37:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs": "0", 2024-09-16T03:37:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-16T03:37:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-16T03:37:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-16T03:37:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-16T03:37:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size": "0", 2024-09-16T03:37:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-16T03:37:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-16T03:37:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-16T03:37:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_merge_threshold": "-10", 2024-09-16T03:37:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-16T03:37:02.347 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_odsync_write": "false", 2024-09-16T03:37:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_omap_backend": "rocksdb", 2024-09-16T03:37:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_omap_backend_path": "", 2024-09-16T03:37:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-16T03:37:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-16T03:37:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-16T03:37:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_op_thread_timeout": "60", 2024-09-16T03:37:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_op_threads": "2", 2024-09-16T03:37:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_punch_hole": "false", 2024-09-16T03:37:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-16T03:37:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-16T03:37:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-16T03:37:02.348 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-16T03:37:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-16T03:37:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-16T03:37:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-16T03:37:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-16T03:37:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-16T03:37:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_queue_max_ops": "50", 2024-09-16T03:37:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-16T03:37:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_seek_data_hole": "false", 2024-09-16T03:37:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_sloppy_crc": "false", 2024-09-16T03:37:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-16T03:37:02.349 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_splice": "false", 2024-09-16T03:37:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_split_multiple": "2", 2024-09-16T03:37:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_split_rand_factor": "20", 2024-09-16T03:37:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_update_to": "1000", 2024-09-16T03:37:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-16T03:37:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-16T03:37:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-16T03:37:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-16T03:37:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-16T03:37:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-16T03:37:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_enable": "true", 2024-09-16T03:37:02.350 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-16T03:37:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-16T03:37:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-16T03:37:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-16T03:37:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-16T03:37:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-16T03:37:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_xfs_extsize": "false", 2024-09-16T03:37:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "filestore_zfs_snap": "false", 2024-09-16T03:37:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "fio_dir": "/tmp/fio", 2024-09-16T03:37:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2", 2024-09-16T03:37:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_allow_other": "true", 2024-09-16T03:37:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_atomic_o_trunc": "true", 2024-09-16T03:37:02.351 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_big_writes": "true", 2024-09-16T03:37:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_debug": "false", 2024-09-16T03:37:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_default_permissions": "false", 2024-09-16T03:37:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_disable_pagecache": "false", 2024-09-16T03:37:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_max_write": "0", 2024-09-16T03:37:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_multithreaded": "true", 2024-09-16T03:37:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_require_active_mds": "true", 2024-09-16T03:37:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_set_user_groups": "true", 2024-09-16T03:37:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_splice_move": "true", 2024-09-16T03:37:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_splice_read": "true", 2024-09-16T03:37:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_splice_write": "true", 2024-09-16T03:37:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-16T03:37:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "fuse_use_invalidate_cb": "true", 2024-09-16T03:37:02.352 INFO:teuthology.orchestra.run.smithi064.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "gss_target_name": "ceph", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_file": "", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_inject_failure": "0", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_interval": "5", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "host": "smithi064", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-16T03:37:02.353 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-16T03:37:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-16T03:37:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-16T03:37:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-16T03:37:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-16T03:37:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "inject_early_sigterm": "false", 2024-09-16T03:37:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "jaeger_agent_port": "6799", 2024-09-16T03:37:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "jaeger_tracing_enable": "false", 2024-09-16T03:37:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_aio": "true", 2024-09-16T03:37:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_align_min_size": "65536", 2024-09-16T03:37:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_block_align": "true", 2024-09-16T03:37:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_block_size": "4096", 2024-09-16T03:37:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_dio": "true", 2024-09-16T03:37:02.354 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_discard": "false", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_force_aio": "false", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_ignore_corruption": "false", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_max_write_bytes": "10485760", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_max_write_entries": "100", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_replay_from": "0", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_write_header_frequency": "0", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journal_zero_on_create": "false", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journaler_prefetch_periods": "10", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journaler_prezero_periods": "5", 2024-09-16T03:37:02.355 INFO:teuthology.orchestra.run.smithi064.stdout: "journaler_write_head_interval": "15", 2024-09-16T03:37:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "key": "", 2024-09-16T03:37:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "keyfile": "", 2024-09-16T03:37:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-16T03:37:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_backend": "rocksdb", 2024-09-16T03:37:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_default_stripe_size": "65536", 2024-09-16T03:37:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_fsck_on_mount": "false", 2024-09-16T03:37:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-16T03:37:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_max_bytes": "67108864", 2024-09-16T03:37:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_max_ops": "512", 2024-09-16T03:37:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_nid_prealloc": "1024", 2024-09-16T03:37:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_onode_map_size": "1024", 2024-09-16T03:37:02.356 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-16T03:37:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_sync_submit_transaction": "false", 2024-09-16T03:37:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "kstore_sync_transaction": "false", 2024-09-16T03:37:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "librados_thread_count": "2", 2024-09-16T03:37:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "lockdep": "false", 2024-09-16T03:37:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "lockdep_force_backtrace": "false", 2024-09-16T03:37:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "log_coarse_timestamps": "true", 2024-09-16T03:37:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-16T03:37:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "log_flush_on_exit": "false", 2024-09-16T03:37:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "log_graylog_host": "127.0.0.1", 2024-09-16T03:37:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "log_graylog_port": "12201", 2024-09-16T03:37:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "log_max_new": "1000", 2024-09-16T03:37:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "log_max_recent": "10000", 2024-09-16T03:37:02.357 INFO:teuthology.orchestra.run.smithi064.stdout: "log_stderr_prefix": "", 2024-09-16T03:37:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "log_stop_at_utilization": "0.970000", 2024-09-16T03:37:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_file": "true", 2024-09-16T03:37:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_graylog": "false", 2024-09-16T03:37:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_journald": "false", 2024-09-16T03:37:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_stderr": "false", 2024-09-16T03:37:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "log_to_syslog": "false", 2024-09-16T03:37:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "max_rotating_auth_attempts": "10", 2024-09-16T03:37:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-16T03:37:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_action_on_write_error": "1", 2024-09-16T03:37:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_alternate_name_max": "8192", 2024-09-16T03:37:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_asio_thread_count": "2", 2024-09-16T03:37:02.358 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_export_pin": "true", 2024-09-16T03:37:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_dirs": "true", 2024-09-16T03:37:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-16T03:37:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_interval": "5", 2024-09-16T03:37:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-16T03:37:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-16T03:37:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_interval": "10", 2024-09-16T03:37:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_max": "-1", 2024-09-16T03:37:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_max_until": "-1", 2024-09-16T03:37:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_merge_size": "50", 2024-09-16T03:37:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_midchunk": "0.300000", 2024-09-16T03:37:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-16T03:37:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_min_start": "0.200000", 2024-09-16T03:37:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_minchunk": "0.001000", 2024-09-16T03:37:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_mode": "0", 2024-09-16T03:37:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_need_max": "1.200000", 2024-09-16T03:37:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_need_min": "0.800000", 2024-09-16T03:37:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_overload_epochs": "2", 2024-09-16T03:37:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-16T03:37:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-16T03:37:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_bits": "3", 2024-09-16T03:37:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-16T03:37:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_size": "10000", 2024-09-16T03:37:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-16T03:37:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_target_decay": "10.000000", 2024-09-16T03:37:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-16T03:37:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_beacon_grace": "15.000000", 2024-09-16T03:37:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_beacon_interval": "4.000000", 2024-09-16T03:37:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-16T03:37:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-16T03:37:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_mid": "0.700000", 2024-09-16T03:37:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-16T03:37:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_quiesce_delay": "0", 2024-09-16T03:37:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-16T03:37:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-16T03:37:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-16T03:37:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_release_free_interval": "10", 2024-09-16T03:37:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_reservation": "0.050000", 2024-09-16T03:37:02.361 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-16T03:37:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_trim_interval": "1", 2024-09-16T03:37:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache_trim_threshold": "262144", 2024-09-16T03:37:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-16T03:37:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-16T03:37:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-16T03:37:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_client_prealloc_inos": "1000", 2024-09-16T03:37:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-16T03:37:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_connect_bootstrapping": "false", 2024-09-16T03:37:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_damage_table_max_entries": "10000", 2024-09-16T03:37:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-16T03:37:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_auth_pins": "false", 2024-09-16T03:37:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_frag": "false", 2024-09-16T03:37:02.362 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_scatterstat": "false", 2024-09-16T03:37:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_debug_subtrees": "false", 2024-09-16T03:37:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_decay_halflife": "5.000000", 2024-09-16T03:37:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_default_dir_hash": "2", 2024-09-16T03:37:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_defer_session_stale": "true", 2024-09-16T03:37:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_deny_all_reconnect": "false", 2024-09-16T03:37:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_keys_per_op": "16384", 2024-09-16T03:37:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_max_commit_size": "10", 2024-09-16T03:37:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_max_entries": "0", 2024-09-16T03:37:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dir_prefetch": "true", 2024-09-16T03:37:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-16T03:37:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-16T03:37:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_on_map": "false", 2024-09-16T03:37:02.363 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_early_reply": "true", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_enable_op_tracker": "true", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_enforce_unique_name": "true", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_random": "true", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-16T03:37:02.364 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-16T03:37:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-16T03:37:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_health_summarize_threshold": "10", 2024-09-16T03:37:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-16T03:37:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-16T03:37:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_health_dummy": "false", 2024-09-16T03:37:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-16T03:37:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_migrator_session_race": "false", 2024-09-16T03:37:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-16T03:37:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-16T03:37:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-16T03:37:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_join_fs": "", 2024-09-16T03:37:02.365 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_journal_format": "1", 2024-09-16T03:37:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-16T03:37:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_create_at": "0", 2024-09-16T03:37:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_dirfrag_at": "0", 2024-09-16T03:37:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_export_at": "0", 2024-09-16T03:37:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_import_at": "0", 2024-09-16T03:37:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_journal_at": "0", 2024-09-16T03:37:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_journal_expire_at": "0", 2024-09-16T03:37:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_journal_replay_at": "0", 2024-09-16T03:37:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_link_at": "0", 2024-09-16T03:37:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_mdstable_at": "0", 2024-09-16T03:37:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_openc_at": "0", 2024-09-16T03:37:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_rename_at": "0", 2024-09-16T03:37:02.366 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_kill_shutdown_at": "0", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_event_large_threshold": "524288", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_events_per_segment": "1024", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_max_events": "-1", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_max_segments": "128", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_pause": "false", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_segment_size": "0", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_trim_threshold": "128", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log_warn_factor": "2.000000", 2024-09-16T03:37:02.367 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_caps_per_client": "1048576", 2024-09-16T03:37:02.368 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_completed_flushes": "100000", 2024-09-16T03:37:02.368 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_completed_requests": "100000", 2024-09-16T03:37:02.368 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_export_size": "20971520", 2024-09-16T03:37:02.368 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_file_recover": "32", 2024-09-16T03:37:02.368 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_purge_files": "64", 2024-09-16T03:37:02.368 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_purge_ops": "8192", 2024-09-16T03:37:02.368 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-16T03:37:02.368 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-16T03:37:02.368 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_max_snaps_per_dir": "100", 2024-09-16T03:37:02.368 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_metrics_update_interval": "2", 2024-09-16T03:37:02.368 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_min_caps_per_client": "100", 2024-09-16T03:37:02.368 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_min_caps_working_set": "10000", 2024-09-16T03:37:02.368 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_numa_node": "-1", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_complaint_time": "30.000000", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_duration": "600", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_size": "20", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_slow_op_size": "20", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_op_log_threshold": "5", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_ping_grace": "15", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_ping_interval": "5", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_max_caps": "30000", 2024-09-16T03:37:02.369 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-16T03:37:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-16T03:37:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-16T03:37:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_recall_warning_threshold": "262144", 2024-09-16T03:37:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-16T03:37:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_replay_interval": "1.000000", 2024-09-16T03:37:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-16T03:37:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-16T03:37:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_root_ino_gid": "0", 2024-09-16T03:37:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_root_ino_uid": "0", 2024-09-16T03:37:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-16T03:37:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_scrub_stats_review_period": "1", 2024-09-16T03:37:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-16T03:37:02.370 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_shutdown_check": "0", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_skip_ino": "0", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_snap_max_uid": "4294967294", 2024-09-16T03:37:02.371 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_snap_min_uid": "0", 2024-09-16T03:37:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_snap_rstat": "false", 2024-09-16T03:37:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_standby_replay_damaged": "false", 2024-09-16T03:37:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_symlink_recovery": "true", 2024-09-16T03:37:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-16T03:37:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_thrash_exports": "0", 2024-09-16T03:37:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_thrash_fragments": "0", 2024-09-16T03:37:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_tick_interval": "5.000000", 2024-09-16T03:37:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_valgrind_exit": "false", 2024-09-16T03:37:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_verify_backtrace": "1", 2024-09-16T03:37:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_verify_scatter": "false", 2024-09-16T03:37:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-16T03:37:02.372 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_wipe_sessions": "false", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool_debug": "false", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_device_bytes": "1073741824", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_page_set": "false", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "memstore_page_size": "65536", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_client_bytes": "134217728", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_client_messages": "512", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_disabled_modules": "", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_enable_op_tracker": "true", 2024-09-16T03:37:02.373 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-16T03:37:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_max_pg_creating": "1024", 2024-09-16T03:37:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_max_pg_num_change": "128", 2024-09-16T03:37:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mds_bytes": "134217728", 2024-09-16T03:37:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mds_messages": "128", 2024-09-16T03:37:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-16T03:37:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mon_bytes": "134217728", 2024-09-16T03:37:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_mon_messages": "128", 2024-09-16T03:37:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-16T03:37:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-16T03:37:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_op_history_duration": "600", 2024-09-16T03:37:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_op_history_size": "20", 2024-09-16T03:37:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-16T03:37:02.374 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_op_log_threshold": "5", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_osd_bytes": "536870912", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_osd_messages": "8192", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_pool": "true", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_standby_modules": "true", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_stats_period": "5", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_stats_threshold": "5", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_tick_period": "2", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_allow_pool_delete": "true", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_allow_pool_size_one": "true", 2024-09-16T03:37:02.375 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_auth_validate_all_caps": "true", 2024-09-16T03:37:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-16T03:37:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-16T03:37:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_bytes": "104857600", 2024-09-16T03:37:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_directed_command_retry": "2", 2024-09-16T03:37:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-16T03:37:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-16T03:37:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-16T03:37:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-16T03:37:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_hunt_parallel": "3", 2024-09-16T03:37:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_log_interval": "1.000000", 2024-09-16T03:37:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-16T03:37:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_ping_interval": "10.000000", 2024-09-16T03:37:02.376 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-16T03:37:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_client_target_rank": "-1", 2024-09-16T03:37:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-16T03:37:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-16T03:37:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-16T03:37:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_level": "debug", 2024-09-16T03:37:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_file": "true", 2024-09-16T03:37:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-16T03:37:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-16T03:37:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-16T03:37:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_journald": "false", 2024-09-16T03:37:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-16T03:37:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-16T03:37:02.377 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_compact_on_bootstrap": "false", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_compact_on_start": "false", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_compact_on_trim": "true", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_cpu_threads": "4", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_crush_min_required_version": "hammer", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_daemon_bytes": "419430400", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data_avail_crit": "5", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data_avail_warn": "5", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_data_size_warn": "16106127360", 2024-09-16T03:37:02.378 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-16T03:37:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-16T03:37:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_dump_json": "false", 2024-09-16T03:37:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-16T03:37:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_dump_transactions": "false", 2024-09-16T03:37:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_extra_checks": "false", 2024-09-16T03:37:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-16T03:37:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-16T03:37:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_require_reef": "false", 2024-09-16T03:37:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_no_require_squid": "false", 2024-09-16T03:37:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-16T03:37:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-16T03:37:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-16T03:37:02.379 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_down_mkfs_grace": "120", 2024-09-16T03:37:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_down_uptime_grace": "60", 2024-09-16T03:37:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_election_default_strategy": "3", 2024-09-16T03:37:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_election_timeout": "5.000000", 2024-09-16T03:37:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-16T03:37:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_elector_ping_divisor": "2", 2024-09-16T03:37:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-16T03:37:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_enable_op_tracker": "true", 2024-09-16T03:37:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_fake_pool_delete": "false", 2024-09-16T03:37:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_force_quorum_join": "false", 2024-09-16T03:37:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-16T03:37:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_globalid_prealloc": "10000", 2024-09-16T03:37:02.380 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_detail_to_clog": "false", 2024-09-16T03:37:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_log_update_period": "5", 2024-09-16T03:37:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_max_detail": "50", 2024-09-16T03:37:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_to_clog": "true", 2024-09-16T03:37:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_to_clog_interval": "600", 2024-09-16T03:37:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-16T03:37:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_host": "[v1:172.21.15.64:6789],[v1:172.21.15.112:6789],[v1:172.21.15.155:6789]", 2024-09-16T03:37:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_host_override": "", 2024-09-16T03:37:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_initial_members": "", 2024-09-16T03:37:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-16T03:37:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-16T03:37:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-16T03:37:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-16T03:37:02.381 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-16T03:37:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_lease": "5.000000", 2024-09-16T03:37:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-16T03:37:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-16T03:37:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_log_full_interval": "50", 2024-09-16T03:37:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_log_max": "10000", 2024-09-16T03:37:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_log_max_summary": "50", 2024-09-16T03:37:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-16T03:37:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_log_epochs": "500", 2024-09-16T03:37:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-16T03:37:02.382 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_osd": "10000", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_pg_per_osd": "10000", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_pool_pg_num": "65536", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mds_force_trim_to": "0", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mds_skip_sanity": "false", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_memory_autotune": "true", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_memory_target": "2147483648", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_beacon_grace": "30", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_digest_period": "5", 2024-09-16T03:37:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_inactive_grace": "60", 2024-09-16T03:37:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-16T03:37:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-16T03:37:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_min_osdmap_epochs": "50", 2024-09-16T03:37:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-16T03:37:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-16T03:37:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_complaint_time": "30", 2024-09-16T03:37:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_duration": "600", 2024-09-16T03:37:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_size": "20", 2024-09-16T03:37:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_slow_op_size": "20", 2024-09-16T03:37:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-16T03:37:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_op_log_threshold": "5", 2024-09-16T03:37:02.384 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-16T03:37:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-16T03:37:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-16T03:37:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_auto_mark_in": "false", 2024-09-16T03:37:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-16T03:37:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-16T03:37:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-16T03:37:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_cache_size": "500", 2024-09-16T03:37:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-16T03:37:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-16T03:37:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-16T03:37:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_down_out_interval": "600", 2024-09-16T03:37:02.385 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_force_trim_to": "0", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_min_down_reporters": "2", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-16T03:37:02.386 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-16T03:37:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-16T03:37:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-16T03:37:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_report_timeout": "900", 2024-09-16T03:37:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-16T03:37:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-16T03:37:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-16T03:37:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-16T03:37:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-16T03:37:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-16T03:37:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-16T03:37:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-16T03:37:02.387 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_stuck_threshold": "60", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_probe_timeout": "2.000000", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_max_change": "0.050000", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_max_osds": "4", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-16T03:37:02.388 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-16T03:37:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-16T03:37:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_interval": "300", 2024-09-16T03:37:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_max_keys": "100", 2024-09-16T03:37:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_scrub_timeout": "300", 2024-09-16T03:37:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_session_timeout": "300", 2024-09-16T03:37:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_smart_report_timeout": "5", 2024-09-16T03:37:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stat_smooth_intervals": "6", 2024-09-16T03:37:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-16T03:37:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_pool_min_size": "2", 2024-09-16T03:37:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_pool_size": "4", 2024-09-16T03:37:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-16T03:37:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-16T03:37:02.389 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_debug": "false", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_provider_kill_at": "0", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_requester_kill_at": "0", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_sync_timeout": "60.000000", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_target_pg_per_osd": "100", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_tick_interval": "5", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_timecheck_interval": "300.000000", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_use_min_delay_socket": "false", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_older_version_delay": "604800", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-16T03:37:02.390 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-16T03:37:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-16T03:37:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-16T03:37:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-16T03:37:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-16T03:37:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_misplaced": "false", 2024-09-16T03:37:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-16T03:37:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_older_version": "true", 2024-09-16T03:37:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-16T03:37:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-16T03:37:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-16T03:37:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-16T03:37:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-16T03:37:02.391 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-16T03:37:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-16T03:37:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-16T03:37:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-16T03:37:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "monmap": "", 2024-09-16T03:37:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T03:37:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-16T03:37:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-16T03:37:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T03:37:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-16T03:37:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-16T03:37:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "motr_tracing_enabled": "false", 2024-09-16T03:37:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_op_threads": "3", 2024-09-16T03:37:02.392 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-16T03:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_cm": "false", 2024-09-16T03:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_device_name": "", 2024-09-16T03:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_dscp": "96", 2024-09-16T03:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-16T03:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-16T03:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_local_gid": "", 2024-09-16T03:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-16T03:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_port_num": "1", 2024-09-16T03:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-16T03:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-16T03:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-16T03:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-16T03:37:02.393 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_sl": "3", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_support_srq": "true", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_rdma_type": "ib", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_async_reap_threshold": "5", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_before_connect": "false", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_ipv4": "true", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_ipv6": "false", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_msgr1": "true", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_msgr2": "false", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_port_max": "7568", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_port_min": "6800", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_retry_count": "3", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_bind_retry_delay": "5", 2024-09-16T03:37:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_client": "false", 2024-09-16T03:37:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_mds": "false", 2024-09-16T03:37:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_mgr": "false", 2024-09-16T03:37:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_mon": "false", 2024-09-16T03:37:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_blackhole_osd": "false", 2024-09-16T03:37:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_client_mode": "crc secure", 2024-09-16T03:37:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-16T03:37:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_cluster_mode": "crc secure", 2024-09-16T03:37:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_cluster_type": "", 2024-09-16T03:37:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_compress_secure": "false", 2024-09-16T03:37:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_connection_idle_timeout": "900", 2024-09-16T03:37:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_connection_ready_timeout": "10", 2024-09-16T03:37:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_crc_data": "true", 2024-09-16T03:37:02.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_crc_header": "true", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_bad_msg": "false", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_bug": "true", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_old_message": "true", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_skipped_message": "false", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_coremask": "0xF", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_enable_tso": "true", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_hugepages": "", 2024-09-16T03:37:02.396 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-16T03:37:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-16T03:37:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_lro": "true", 2024-09-16T03:37:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_memory_channel": "4", 2024-09-16T03:37:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-16T03:37:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_pmd": "", 2024-09-16T03:37:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_port_id": "0", 2024-09-16T03:37:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-16T03:37:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-16T03:37:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_dump_on_send": "false", 2024-09-16T03:37:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_initial_backoff": "0.200000", 2024-09-16T03:37:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_delay_max": "1.000000", 2024-09-16T03:37:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-16T03:37:02.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_delay_type": "", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_network_congestion": "0", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_inject_socket_failures": "0", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_learn_addr_from_peer": "true", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_max_accept_failures": "4", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_max_backoff": "15.000000", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_mon_client_mode": "secure crc", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_mon_service_mode": "secure crc", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_osd_compress_min_size": "1024", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_osd_compress_mode": "none", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-16T03:37:02.398 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_pq_min_cost": "65536", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_public_type": "", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_service_mode": "crc secure", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_listen_backlog": "512", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_nodelay": "true", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_tcp_rcvbuf": "0", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "ms_type": "async", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "no_config_file": "false", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_completion_locks_per_session": "32", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-16T03:37:02.399 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-16T03:37:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_inflight_ops": "1024", 2024-09-16T03:37:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-16T03:37:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-16T03:37:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_tick_interval": "5.000000", 2024-09-16T03:37:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter_timeout": "10.000000", 2024-09-16T03:37:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "objectstore_blackhole": "false", 2024-09-16T03:37:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-16T03:37:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "openssl_engine_opts": "", 2024-09-16T03:37:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_delay_time": "5.000000", 2024-09-16T03:37:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_hist_halflife": "1000", 2024-09-16T03:37:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_max_low_ops": "2", 2024-09-16T03:37:02.400 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_max_ops": "4", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_agent_slop": "0.020000", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_asio_thread_count": "2", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_async_recovery_min_cost": "1", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backfill_scan_max": "512", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backfill_scan_min": "64", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backoff_on_degraded": "false", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backoff_on_peering": "false", 2024-09-16T03:37:02.401 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_backoff_on_unfound": "true", 2024-09-16T03:37:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_beacon_report_interval": "300", 2024-09-16T03:37:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_duration": "30", 2024-09-16T03:37:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-16T03:37:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_max_block_size": "67108864", 2024-09-16T03:37:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-16T03:37:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_blkin_trace_all": "false", 2024-09-16T03:37:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-16T03:37:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-16T03:37:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-16T03:37:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-16T03:37:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_check_for_log_corruption": "false", 2024-09-16T03:37:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-16T03:37:02.402 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_class_update_on_start": "true", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_message_cap": "256", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_message_size_cap": "524288000", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_op_priority": "63", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_client_watch_timeout": "30", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_command_max_records": "256", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_compact_on_start": "false", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_update_on_start": "true", 2024-09-16T03:37:02.403 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_crush_update_weight_set": "true", 2024-09-16T03:37:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-16T03:37:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-16T03:37:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-16T03:37:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-16T03:37:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-16T03:37:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_feed_pullee": "-1", 2024-09-16T03:37:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-16T03:37:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-16T03:37:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_inject_dispatch_delay_probability": "0.100000", 2024-09-16T03:37:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_misdirected_ops": "false", 2024-09-16T03:37:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_no_acting_change": "false", 2024-09-16T03:37:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_no_purge_strays": "false", 2024-09-16T03:37:02.404 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_op_order": "false", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_shutdown": "false", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_trim_objects": "false", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_snaps": "false", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-16T03:37:02.405 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_keys": "1024", 2024-09-16T03:37:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-16T03:37:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-16T03:37:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-16T03:37:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_stride": "524288", 2024-09-16T03:37:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-16T03:37:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-16T03:37:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_default_notify_timeout": "30", 2024-09-16T03:37:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep": "0.000000", 2024-09-16T03:37:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-16T03:37:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-16T03:37:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-16T03:37:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_discard_disconnected_ops": "true", 2024-09-16T03:37:02.406 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_ec_partial_reads": "true", 2024-09-16T03:37:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_enable_op_tracker": "true", 2024-09-16T03:37:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-16T03:37:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-16T03:37:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-16T03:37:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-16T03:37:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_info": "true", 2024-09-16T03:37:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_shutdown": "true", 2024-09-16T03:37:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-16T03:37:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-16T03:37:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-16T03:37:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-16T03:37:02.407 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_function_tracing": "false", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_grace": "20", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_interval": "6", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_min_peers": "10", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_min_size": "2000", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_stale": "600", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_hit_set_max_size": "100000", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_hit_set_min_size": "1000", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-16T03:37:02.408 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-16T03:37:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-16T03:37:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-16T03:37:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_journal_size": "5120", 2024-09-16T03:37:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_kill_backfill_at": "0", 2024-09-16T03:37:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-16T03:37:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_cache_size": "50", 2024-09-16T03:37:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_dedup": "true", 2024-09-16T03:37:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_message_max": "40", 2024-09-16T03:37:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-16T03:37:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_map_share_max_epochs": "40", 2024-09-16T03:37:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_attr_name_len": "100", 2024-09-16T03:37:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_attr_size": "0", 2024-09-16T03:37:02.409 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_backfills": "1", 2024-09-16T03:37:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_markdown_count": "5", 2024-09-16T03:37:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_markdown_period": "600", 2024-09-16T03:37:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_object_name_len": "2048", 2024-09-16T03:37:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_object_namespace_len": "256", 2024-09-16T03:37:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_object_size": "134217728", 2024-09-16T03:37:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-16T03:37:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-16T03:37:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pg_blocked_by": "16", 2024-09-16T03:37:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pg_log_entries": "2", 2024-09-16T03:37:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-16T03:37:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_pgls": "1024", 2024-09-16T03:37:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_push_cost": "8388608", 2024-09-16T03:37:02.410 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_push_objects": "10", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_scrubs": "3", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_trimming_pgs": "2", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_write_op_reply_len": "64", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_max_write_size": "90", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-16T03:37:02.411 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_profile": "balanced", 2024-09-16T03:37:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-16T03:37:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-16T03:37:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-16T03:37:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-16T03:37:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-16T03:37:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-16T03:37:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-16T03:37:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-16T03:37:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-16T03:37:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-16T03:37:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-16T03:37:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_base": "805306368", 2024-09-16T03:37:02.412 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_cache_min": "134217728", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_target": "4294967296", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_target_autotune": "false", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_min_pg_log_entries": "1", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_min_recovery_priority": "0", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_report_interval": "5", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_num_cache_shards": "32", 2024-09-16T03:37:02.413 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_num_op_tracker_shard": "32", 2024-09-16T03:37:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_numa_auto_affinity": "true", 2024-09-16T03:37:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_numa_node": "-1", 2024-09-16T03:37:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_numa_prefer_iface": "true", 2024-09-16T03:37:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-09-16T03:37:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objecter_finishers": "1", 2024-09-16T03:37:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objectstore": "bluestore", 2024-09-16T03:37:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objectstore_fuse": "false", 2024-09-16T03:37:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-16T03:37:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_objectstore_tracing": "false", 2024-09-16T03:37:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_complaint_time": "30.000000", 2024-09-16T03:37:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_duration": "600", 2024-09-16T03:37:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_size": "20", 2024-09-16T03:37:02.414 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_slow_op_size": "20", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_log_threshold": "5", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_shards": "0", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_shards_hdd": "5", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_shards_ssd": "8", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_pq_min_cost": "65536", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_queue_cut_off": "high", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-16T03:37:02.415 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_op_thread_timeout": "15", 2024-09-16T03:37:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_open_classes_on_start": "true", 2024-09-16T03:37:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_os_flags": "0", 2024-09-16T03:37:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_peering_op_priority": "255", 2024-09-16T03:37:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_delete_cost": "1048576", 2024-09-16T03:37:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_delete_priority": "5", 2024-09-16T03:37:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-16T03:37:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-16T03:37:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-16T03:37:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_log_trim_max": "10000", 2024-09-16T03:37:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_log_trim_min": "0", 2024-09-16T03:37:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-16T03:37:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-16T03:37:02.416 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-16T03:37:02.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-16T03:37:02.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-16T03:37:02.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-16T03:37:02.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-16T03:37:02.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-16T03:37:02.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-16T03:37:02.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-16T03:37:02.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_crimson": "false", 2024-09-16T03:37:02.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-16T03:37:02.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-16T03:37:02.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-16T03:37:02.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-16T03:37:02.417 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-16T03:37:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-16T03:37:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-16T03:37:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-16T03:37:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_flags": "0", 2024-09-16T03:37:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-16T03:37:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_min_size": "2", 2024-09-16T03:37:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-16T03:37:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_pg_num": "32", 2024-09-16T03:37:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_pgp_num": "0", 2024-09-16T03:37:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-16T03:37:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_read_ratio": "70", 2024-09-16T03:37:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_size": "3", 2024-09-16T03:37:02.418 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_default_type": "replicated", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_push_per_object_cost": "1000", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recover_clone_overlap": "true", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_cost": "20971520", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_active": "0", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-16T03:37:02.419 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_max_single_start": "1", 2024-09-16T03:37:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_op_priority": "3", 2024-09-16T03:37:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-16T03:37:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_priority": "5", 2024-09-16T03:37:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-16T03:37:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep": "0.000000", 2024-09-16T03:37:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-16T03:37:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-16T03:37:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-16T03:37:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_repair_during_recovery": "false", 2024-09-16T03:37:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_requested_scrub_priority": "120", 2024-09-16T03:37:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-16T03:37:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-16T03:37:02.420 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_auto_repair": "false", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_begin_hour": "0", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_begin_week_day": "0", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_chunk_max": "15", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_chunk_min": "5", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_cost": "52428800", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_during_recovery": "false", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_end_hour": "0", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_end_week_day": "0", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_event_cost": "4096", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-16T03:37:02.421 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-16T03:37:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_invalid_stats": "true", 2024-09-16T03:37:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-16T03:37:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-16T03:37:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_max_preemptions": "5", 2024-09-16T03:37:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-16T03:37:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_priority": "5", 2024-09-16T03:37:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-16T03:37:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_retry_delay": "30", 2024-09-16T03:37:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-16T03:37:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-16T03:37:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_retry_trimming": "10", 2024-09-16T03:37:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_scrub_sleep": "0.000000", 2024-09-16T03:37:02.422 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_skip_data_digest": "false", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_smart_report_timeout": "5", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_cost": "1048576", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_priority": "5", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-16T03:37:02.423 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-16T03:37:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_target_transaction_size": "30", 2024-09-16T03:37:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-16T03:37:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-16T03:37:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-16T03:37:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-16T03:37:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-16T03:37:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-16T03:37:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-16T03:37:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-16T03:37:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-16T03:37:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-16T03:37:02.424 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_tracing": "false", 2024-09-16T03:37:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_use_stale_snap": "false", 2024-09-16T03:37:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-16T03:37:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "osdc_blkin_trace_all": "false", 2024-09-16T03:37:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_kill_at": "0", 2024-09-16T03:37:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_max_join_drift": "10", 2024-09-16T03:37:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_min": "500", 2024-09-16T03:37:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_min_wait": "0.050000", 2024-09-16T03:37:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_propose_interval": "1.000000", 2024-09-16T03:37:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_service_trim_max": "500", 2024-09-16T03:37:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-16T03:37:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_service_trim_min": "10", 2024-09-16T03:37:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_stash_full_interval": "25", 2024-09-16T03:37:02.425 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_trim_max": "500", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "paxos_trim_min": "250", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "perf": "true", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "-", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrv": "", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "public_bind_addr": "-", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "public_network": "", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "public_network_interface": "", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "qat_compressor_busy_polling": "false", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "qat_compressor_enabled": "false", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "qat_compressor_session_max_number": "256", 2024-09-16T03:37:02.426 INFO:teuthology.orchestra.run.smithi064.stdout: "rados_mon_op_timeout": "0", 2024-09-16T03:37:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rados_osd_op_timeout": "0", 2024-09-16T03:37:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rados_tracing": "false", 2024-09-16T03:37:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_atime_update_interval": "60", 2024-09-16T03:37:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-16T03:37:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_balance_parent_reads": "false", 2024-09-16T03:37:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_balance_snap_reads": "false", 2024-09-16T03:37:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_blkin_trace_all": "false", 2024-09-16T03:37:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-16T03:37:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-16T03:37:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache": "true", 2024-09-16T03:37:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-16T03:37:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-16T03:37:02.427 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_policy": "writearound", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_size": "33554432", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_clone_copy_on_read": "false", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_compression_hint": "none", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_concurrent_management_ops": "10", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_clone_format": "auto", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_data_pool": "", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_features": "61", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_format": "2", 2024-09-16T03:37:02.428 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_map_options": "", 2024-09-16T03:37:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_order": "22", 2024-09-16T03:37:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_pool": "rbd", 2024-09-16T03:37:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-16T03:37:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_stripe_count": "0", 2024-09-16T03:37:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_default_stripe_unit": "0", 2024-09-16T03:37:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-16T03:37:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-16T03:37:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-16T03:37:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_enable_alloc_hint": "true", 2024-09-16T03:37:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-16T03:37:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_io_scheduler": "simple", 2024-09-16T03:37:02.429 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_order": "24", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_pool": "", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_journal_splay_width": "4", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_localize_parent_reads": "false", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_localize_snap_reads": "false", 2024-09-16T03:37:02.430 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-16T03:37:02.431 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-16T03:37:02.432 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_mtime_update_interval": "60", 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_non_blocking_aio": "true", 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_op_thread_timeout": "60", 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_op_threads": "1", 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_parent_cache_enabled": "false", 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_plugins": "", 2024-09-16T03:37:02.433 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_bps_burst": "0", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_bps_limit": "0", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_exclude_ops": "0", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_iops_burst": "0", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_iops_limit": "0", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-16T03:37:02.434 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_read_from_replica_policy": "default", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_skip_partial_discard": "true", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-16T03:37:02.435 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_tracing": "false", 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_validate_names": "true", 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd_validate_pool": "true", 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "restapi_base_url": "", 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "restapi_log_level": "", 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_acl_grants_max_num": "100", 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_admin_entry": "admin", 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_asio_assert_yielding": "false", 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_backend_store": "rados", 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_barbican_url": "", 2024-09-16T03:37:02.436 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_beast_enable_async": "true", 2024-09-16T03:37:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_counters_cache": "false", 2024-09-16T03:37:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-16T03:37:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-16T03:37:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-16T03:37:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-16T03:37:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-16T03:37:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-16T03:37:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-16T03:37:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-16T03:37:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cache_enabled": "true", 2024-09-16T03:37:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cache_expiry_interval": "900", 2024-09-16T03:37:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cache_lru_size": "25000", 2024-09-16T03:37:02.437 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_config_store": "rados", 2024-09-16T03:37:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_content_length_compat": "false", 2024-09-16T03:37:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_copy_obj_progress": "true", 2024-09-16T03:37:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-16T03:37:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cors_rules_max_num": "100", 2024-09-16T03:37:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_cross_domain_policy": "", 2024-09-16T03:37:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-16T03:37:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_addr": "", 2024-09-16T03:37:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-16T03:37:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-16T03:37:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-16T03:37:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-16T03:37:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_password": "", 2024-09-16T03:37:02.438 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-16T03:37:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_kmip_username": "", 2024-09-16T03:37:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_require_ssl": "true", 2024-09-16T03:37:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-16T03:37:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-16T03:37:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-16T03:37:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-16T03:37:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-16T03:37:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-16T03:37:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-16T03:37:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-16T03:37:02.439 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-16T03:37:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-16T03:37:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-16T03:37:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-16T03:37:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-16T03:37:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-16T03:37:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-16T03:37:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_addr": "", 2024-09-16T03:37:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_auth": "token", 2024-09-16T03:37:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_namespace": "", 2024-09-16T03:37:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_prefix": "", 2024-09-16T03:37:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-16T03:37:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-16T03:37:02.440 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_token_file": "", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_buffersize": "524288", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_low_speed_time": "300", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-16T03:37:02.441 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-16T03:37:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-16T03:37:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-16T03:37:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-16T03:37:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-16T03:37:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-16T03:37:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-16T03:37:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-16T03:37:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-16T03:37:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-16T03:37:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-16T03:37:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-16T03:37:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_changes_size": "1000", 2024-09-16T03:37:02.442 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_num_shards": "128", 2024-09-16T03:37:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-16T03:37:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_log_window": "30", 2024-09-16T03:37:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-16T03:37:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-16T03:37:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-16T03:37:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-16T03:37:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-16T03:37:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-16T03:37:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-16T03:37:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-16T03:37:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-16T03:37:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-16T03:37:02.443 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-16T03:37:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-16T03:37:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-16T03:37:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_disable_s3select": "false", 2024-09-16T03:37:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-16T03:37:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-16T03:37:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-16T03:37:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-16T03:37:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-16T03:37:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-16T03:37:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-16T03:37:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-16T03:37:02.444 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-16T03:37:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-16T03:37:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-16T03:37:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-16T03:37:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dns_name": "", 2024-09-16T03:37:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dns_s3website_name": "", 2024-09-16T03:37:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dynamic_resharding": "true", 2024-09-16T03:37:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-16T03:37:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-16T03:37:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-16T03:37:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_gc_threads": "true", 2024-09-16T03:37:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_lc_threads": "true", 2024-09-16T03:37:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_mdsearch": "true", 2024-09-16T03:37:02.445 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_ops_log": "false", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_quota_threads": "true", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_static_website": "false", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enable_usage_log": "false", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_enforce_swift_acls": "true", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_exit_timeout_secs": "120", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_expose_bucket": "false", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_extended_http_attrs": "", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_filter": "none", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_frontends": "beast port=7480", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_deferred": "50", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-16T03:37:02.446 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_objs": "32", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_gc_processor_period": "3600", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_init_timeout": "300", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_kafka_connection_idle": "300", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-16T03:37:02.447 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-16T03:37:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-16T03:37:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-16T03:37:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-16T03:37:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_domain": "", 2024-09-16T03:37:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_password": "", 2024-09-16T03:37:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_password_path": "", 2024-09-16T03:37:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_project": "", 2024-09-16T03:37:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_tenant": "", 2024-09-16T03:37:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_token": "", 2024-09-16T03:37:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_token_path": "", 2024-09-16T03:37:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_admin_user": "", 2024-09-16T03:37:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_domain": "", 2024-09-16T03:37:02.448 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_password": "", 2024-09-16T03:37:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_project": "", 2024-09-16T03:37:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-16T03:37:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_barbican_user": "", 2024-09-16T03:37:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-16T03:37:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-16T03:37:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-16T03:37:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-16T03:37:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-16T03:37:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_url": "", 2024-09-16T03:37:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-16T03:37:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_debug_interval": "-1", 2024-09-16T03:37:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_lock_max_time": "90", 2024-09-16T03:37:02.449 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_objs": "32", 2024-09-16T03:37:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_rules": "1000", 2024-09-16T03:37:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_worker": "3", 2024-09-16T03:37:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-16T03:37:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lc_thread_delay": "0", 2024-09-16T03:37:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-16T03:37:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_dnattr": "uid", 2024-09-16T03:37:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-16T03:37:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_searchfilter": "", 2024-09-16T03:37:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-16T03:37:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ldap_uri": "", 2024-09-16T03:37:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-16T03:37:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-16T03:37:02.450 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-16T03:37:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-16T03:37:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_http_headers": "", 2024-09-16T03:37:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-16T03:37:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-16T03:37:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_log_object_name_utc": "false", 2024-09-16T03:37:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-16T03:37:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-16T03:37:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_attr_name_len": "0", 2024-09-16T03:37:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_attr_size": "0", 2024-09-16T03:37:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-16T03:37:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_chunk_size": "4194304", 2024-09-16T03:37:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-16T03:37:02.451 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-16T03:37:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-16T03:37:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_listing_results": "5000", 2024-09-16T03:37:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_notify_retries": "10", 2024-09-16T03:37:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-16T03:37:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_put_param_size": "1048576", 2024-09-16T03:37:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_put_size": "5368709120", 2024-09-16T03:37:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_max_slo_entries": "1000", 2024-09-16T03:37:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_md_log_max_shards": "64", 2024-09-16T03:37:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-16T03:37:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-16T03:37:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-16T03:37:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-16T03:37:02.452 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_mp_lock_max_time": "600", 2024-09-16T03:37:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-16T03:37:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-16T03:37:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-16T03:37:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-16T03:37:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-16T03:37:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-16T03:37:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-16T03:37:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-16T03:37:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_max_gc": "300", 2024-09-16T03:37:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-16T03:37:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-16T03:37:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-16T03:37:02.453 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-16T03:37:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-16T03:37:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-16T03:37:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-16T03:37:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_num_async_rados_threads": "32", 2024-09-16T03:37:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_num_control_oids": "8", 2024-09-16T03:37:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_numa_node": "-1", 2024-09-16T03:37:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-16T03:37:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-16T03:37:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_objexp_chunk_size": "100", 2024-09-16T03:37:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_objexp_gc_interval": "600", 2024-09-16T03:37:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-16T03:37:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-16T03:37:02.454 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-16T03:37:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_op_thread_timeout": "600", 2024-09-16T03:37:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_op_tracing": "false", 2024-09-16T03:37:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_opa_token": "", 2024-09-16T03:37:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_opa_url": "", 2024-09-16T03:37:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_opa_verify_ssl": "true", 2024-09-16T03:37:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-16T03:37:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-16T03:37:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_rados": "false", 2024-09-16T03:37:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_ops_log_socket_path": "", 2024-09-16T03:37:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-16T03:37:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-16T03:37:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-16T03:37:02.455 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_push_interval": "2.000000", 2024-09-16T03:37:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-16T03:37:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-16T03:37:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-16T03:37:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-16T03:37:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_lanes": "3", 2024-09-16T03:37:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-16T03:37:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-16T03:37:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_cache_partitions": "3", 2024-09-16T03:37:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-16T03:37:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_print_continue": "true", 2024-09-16T03:37:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-16T03:37:02.456 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rados_tracing": "false", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_realm": "", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_realm_id": "", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_region": "", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_request_uri": "", 2024-09-16T03:37:02.457 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_batch_size": "64", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_max_aio": "128", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_num_logs": "16", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshard_thread_interval": "600", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_resolve_cname": "false", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_run_sync_thread": "true", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-16T03:37:02.458 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_scheduler_type": "throttler", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_script_uri": "", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_service_provider_name": "", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_client_id": "", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_client_secret": "", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_entry": "sts", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_key": "sts", 2024-09-16T03:37:02.459 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-16T03:37:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_min_session_duration": "900", 2024-09-16T03:37:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sts_token_introspection_url": "", 2024-09-16T03:37:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_account_in_url": "false", 2024-09-16T03:37:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_auth_entry": "auth", 2024-09-16T03:37:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_auth_url": "", 2024-09-16T03:37:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_custom_header": "", 2024-09-16T03:37:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-16T03:37:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_need_stats": "true", 2024-09-16T03:37:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_tenant_name": "", 2024-09-16T03:37:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_token_expiration": "86400", 2024-09-16T03:37:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_url": "", 2024-09-16T03:37:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_url_prefix": "swift", 2024-09-16T03:37:02.460 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_lease_period": "120", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_thread_pool_size": "512", 2024-09-16T03:37:02.461 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_comment": "", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_createby": "", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_encoding": "", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_flag": "false", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_origin": "", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_torrent_tracker": "", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_trust_forwarded_https": "false", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-16T03:37:02.462 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-16T03:37:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_max_shards": "32", 2024-09-16T03:37:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_usage_max_user_shards": "1", 2024-09-16T03:37:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_use_opa_authz": "false", 2024-09-16T03:37:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_counters_cache": "false", 2024-09-16T03:37:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-16T03:37:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-16T03:37:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-16T03:37:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_max_buckets": "1000", 2024-09-16T03:37:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_policies_max_num": "100", 2024-09-16T03:37:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-16T03:37:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-16T03:37:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-16T03:37:02.463 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-16T03:37:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_user_unique_email": "true", 2024-09-16T03:37:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_verify_ssl": "true", 2024-09-16T03:37:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-16T03:37:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zone": "", 2024-09-16T03:37:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zone_id": "", 2024-09-16T03:37:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-16T03:37:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zonegroup": "", 2024-09-16T03:37:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zonegroup_id": "", 2024-09-16T03:37:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-16T03:37:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rgwlc_auto_session_clear": "true", 2024-09-16T03:37:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-16T03:37:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_block_size": "4096", 2024-09-16T03:37:02.464 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_size": "536870912", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-16T03:37:02.465 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_index_type": "binary_search", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_partition_filters": "false", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_perf": "false", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "run_dir": "/var/run/ceph", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_block_create": "true", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_cache_lru_size": "67108864", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_default_max_object_size": "16777216", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-16T03:37:02.466 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_device_size": "53687091200", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_full_integrity_check": "false", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_batch_capacity": "16", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_main_device_type": "SSD", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-16T03:37:02.467 INFO:teuthology.orchestra.run.smithi064.stdout: "seastore_segment_size": "67108864", 2024-09-16T03:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "set_keepcaps": "false", 2024-09-16T03:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "setgroup": "", 2024-09-16T03:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "setuser": "", 2024-09-16T03:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "setuser_match_path": "", 2024-09-16T03:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-16T03:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "thp": "false", 2024-09-16T03:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "threadpool_default_timeout": "60", 2024-09-16T03:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-16T03:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "throttler_perf_counter": "true", 2024-09-16T03:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "tmp_dir": "/tmp", 2024-09-16T03:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-16T03:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "uadk_compressor_enabled": "false", 2024-09-16T03:37:02.468 INFO:teuthology.orchestra.run.smithi064.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-16T03:37:02.469 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:02.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-16T03:37:02.587 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a: {} 2024-09-16T03:37:02.588 INFO:teuthology.orchestra.run.smithi064.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T03:37:02.597 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b: {} 2024-09-16T03:37:02.597 INFO:teuthology.orchestra.run.smithi064.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T03:37:02.607 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c: {} 2024-09-16T03:37:02.607 INFO:teuthology.orchestra.run.smithi064.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T03:37:02.617 DEBUG:teuthology.orchestra.run.smithi157:> type ceph-objectstore-tool 2024-09-16T03:37:02.636 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T03:37:02.636 DEBUG:teuthology.orchestra.run.smithi155:> type ceph-objectstore-tool 2024-09-16T03:37:02.652 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T03:37:02.652 DEBUG:teuthology.orchestra.run.smithi064:> type ceph-objectstore-tool 2024-09-16T03:37:02.667 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T03:37:02.668 DEBUG:teuthology.orchestra.run.smithi112:> type ceph-objectstore-tool 2024-09-16T03:37:02.683 INFO:teuthology.orchestra.run.smithi112.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T03:37:02.683 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T03:37:02.691 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T03:37:02.691 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi157.front.sepia.ceph.com 2024-09-16T03:37:02.692 DEBUG:teuthology.orchestra.run.smithi157:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create low_tier 4' 2024-09-16T03:37:02.692 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-16T03:37:02.692 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:37:02.692 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:37:02.692 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:37:02.693 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-16T03:37:02.693 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-16T03:37:02.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:37:02.694 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-16T03:37:02.694 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:37:02.694 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-16T03:37:02.694 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:37:02.724 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:02.722+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:02.794 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:02.793+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:02.822 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:02.822 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:37:02.822 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:02.834 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:37:02.838 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:02.838 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:02.856 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:02.857 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:02.875 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:02.875 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:02.891 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:02.891 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:02.895 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:02.893+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:02.907 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:02.907 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:02.924 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:02.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:02.944 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:02.945 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:02.948 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:02.948 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:02.957 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:37:02.958 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:02.958 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:02.977 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:02.977 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:02.994 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:02.994 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:02.996 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:02.995+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:03.011 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:03.011 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:03.028 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:03.028 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:03.045 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:03.045 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:03.061 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:03.061 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:03.067 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:03.067 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:03.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:37:03.079 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:03.080 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:03.097 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:03.095+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:03.098 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:03.099 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:03.188 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:03.188 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:03.197 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:37:03.198 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:03.196+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:03.298 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:03.297+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:03.304 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:03.304 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:03.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:37:03.400 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:03.398+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:03.420 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:03.420 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:03.429 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:37:03.500 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:03.499+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:03.535 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:03.535 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:03.544 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:37:03.602 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:03.600+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:03.651 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:03.651 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:03.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:37:03.703 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:03.701+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:03.804 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:03.802+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:03.808 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:03.808 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:03.817 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:37:03.905 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:03.903+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:03.923 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:37:03.923 INFO:teuthology.orchestra.run.smithi157.stderr:pool 'low_tier' created 2024-09-16T03:37:03.940 INFO:teuthology.run_tasks:Running task rados... 2024-09-16T03:37:03.953 INFO:tasks.rados:Beginning rados... 2024-09-16T03:37:03.953 INFO:tasks.rados:config is {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'set_chunk': 30, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True} 2024-09-16T03:37:03.953 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'min_in': 4}, 'workunit': {'branch': 'main', 'sha1': '7498493279982dfe87f00616198a5967475f6169'}} 2024-09-16T03:37:03.953 INFO:tasks.rados:config is {'clients': ['client.0'], 'dedup_chunk_algo': 'fastcdc', 'dedup_chunk_size': '131072', 'enable_dedup': True, 'low_tier_pool': 'low_tier', 'objects': 50, 'op_weights': {'read': 100, 'set_chunk': 30, 'tier_evict': 10, 'tier_flush': 5, 'tier_promote': 10, 'write': 50}, 'ops': 1500, 'set_chunk': True} 2024-09-16T03:37:03.953 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-16T03:37:03.968 INFO:tasks.rados:joining rados 2024-09-16T03:37:03.968 INFO:tasks.rados:clients are ['client.0'] 2024-09-16T03:37:03.968 INFO:tasks.rados:starting run 0 out of 1 2024-09-16T03:37:03.968 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-16T03:37:03.969 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-09-16T03:37:03.971 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:03.971 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:03.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:37:04.022 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:04.020+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:04.107 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:04.105+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:04.130 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:04.130 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:04.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:37:04.172 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:04.170+0000 7eff7884b640 -1 osd.3 30 handle_osd_map injecting map crc failure 2024-09-16T03:37:04.173 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:04.171+0000 7eff7884b640 -1 osd.3 30 handle_osd_map injecting map crc failure 2024-09-16T03:37:04.208 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:04.207+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:04.247 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:04.247 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:04.256 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:37:04.308 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:04.307+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:04.373 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:04.373 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:04.381 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:37:04.409 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:04.407+0000 7fd101e6f640 -1 osd.2 30 handle_osd_map injecting map crc failure 2024-09-16T03:37:04.409 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:04.408+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:04.484 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:04.484 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:04.493 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:37:04.510 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:04.508+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:04.596 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:04.597 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:04.605 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:37:04.650 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:04.648+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:04.708 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:04.708 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:04.711 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:04.709+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:04.716 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:37:04.813 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:04.811+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:04.860 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:04.860 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:04.868 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:37:04.914 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:04.912+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:04.929 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:04.928+0000 7f3a21722640 -1 osd.6 31 handle_osd_map injecting map crc failure 2024-09-16T03:37:04.930 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_0' created 2024-09-16T03:37:04.948 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-16T03:37:05.014 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:05.014 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:05.014 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:05.013+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:05.022 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:37:05.110 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:37:05.126 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:05.126 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:05.135 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:37:05.153 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:05.152+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:05.216 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:05.214+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:05.235 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:05.235 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:05.238 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:05.238 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:05.247 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:37:05.253 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:05.253 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:05.270 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:05.270 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:05.287 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:05.288 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:05.304 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:05.304 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:05.316 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:05.315+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:05.320 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:05.321 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:05.338 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:05.338 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:05.350 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:05.350 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:05.355 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:05.356 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:05.358 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:37:05.371 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:05.372 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:05.389 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:05.389 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:05.406 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:05.406 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:05.417 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:05.416+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:05.422 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:05.423 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:05.438 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:05.439 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:05.456 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:05.457 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:05.462 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:05.463 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:05.463 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:05.463 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:05.471 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:37:05.473 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:05.474 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:05.490 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:05.491 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:05.518 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:05.516+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:05.574 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:05.574 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:05.574 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:37:05.574 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:37:05.574 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:05.583 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:37:05.620 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:05.618+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:05.687 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [] 2024-09-16T03:37:05.687 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:05.695 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:37:05.720 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:05.719+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:05.811 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:05.811 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:05.811 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:37:05.811 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:05.820 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:37:05.821 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:05.819+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:05.922 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:05.921+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:05.924 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:05.925 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:05.925 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:05.925 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:05.925 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:05.934 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:37:05.935 INFO:teuthology.orchestra.run.smithi064.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-16T03:37:05.971 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:37:06.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:06.022+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:06.079 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:06.079 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:06.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:06.080 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [] 2024-09-16T03:37:06.080 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:06.088 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:37:06.124 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:06.122+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:06.225 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:06.224+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:06.232 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:06.232 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:06.232 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:37:06.232 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:06.241 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:37:06.327 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:06.325+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:06.346 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:06.346 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:06.346 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:06.346 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:06.346 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:06.354 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:37:06.427 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:06.426+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:06.498 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:06.498 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:06.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:06.498 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [] 2024-09-16T03:37:06.498 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:06.506 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:37:06.528 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:06.526+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:06.610 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:06.610 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:06.611 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:37:06.611 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:06.619 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:37:06.629 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:06.628+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:06.721 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:06.721 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:06.721 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:06.721 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:06.722 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:06.730 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:37:06.730 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:06.728+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:06.833 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:06.832+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:06.874 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:06.875 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:06.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:06.875 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:06.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.875 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4338.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-16T03:37:06.875 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:36:54.197032+0000", 2024-09-16T03:37:06.875 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 12.6766702, 2024-09-16T03:37:06.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00097050299999999995, 2024-09-16T03:37:06.875 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:06.875 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:06.875 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:54.197032+0000", 2024-09-16T03:37:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:54.197032+0000", 2024-09-16T03:37:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:54.197032+0000", 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5300000000000004e-07 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:54.197042+0000", 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7699999999999996e-06 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:54.197079+0000", 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7141999999999998e-05 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:54.197091+0000", 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.13e-05 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:54.197169+0000", 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8305000000000003e-05 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:54.197190+0000", 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0703000000000002e-05 2024-09-16T03:37:06.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:54.197959+0000", 2024-09-16T03:37:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076932400000000005 2024-09-16T03:37:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:54.198002+0000", 2024-09-16T03:37:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3405999999999997e-05 2024-09-16T03:37:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:06.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.880 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4338.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-16T03:37:06.880 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:36:55.200828+0000", 2024-09-16T03:37:06.880 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.67287425, 2024-09-16T03:37:06.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.201056609, 2024-09-16T03:37:06.880 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:06.880 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:06.880 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:06.880 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:06.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:06.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:55.200828+0000", 2024-09-16T03:37:06.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:06.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:06.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:55.200828+0000", 2024-09-16T03:37:06.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:06.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:06.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:55.200828+0000", 2024-09-16T03:37:06.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4000000000000001e-07 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:55.200842+0000", 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.385e-05 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:55.200864+0000", 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1933000000000002e-05 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:06.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:55.301000+0000", 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10013556 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:55.301176+0000", 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017615 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:55.301215+0000", 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8625999999999999e-05 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:06.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:55.401831+0000", 2024-09-16T03:37:06.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100616397 2024-09-16T03:37:06.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:06.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:06.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:06.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:36:55.401884+0000", 2024-09-16T03:37:06.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3452999999999998e-05 2024-09-16T03:37:06.884 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:06.884 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:06.884 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:06.884 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:06.884 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:06.884 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:06.890 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:37:06.934 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:06.933+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:06.937 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:37:06.951 DEBUG:teuthology.orchestra.run.smithi157:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0 2024-09-16T03:37:06.987 INFO:tasks.rados.rados.0.smithi157.stdout:adding op weight read -> 100 2024-09-16T03:37:06.987 INFO:tasks.rados.rados.0.smithi157.stdout:adding op weight write -> 25 2024-09-16T03:37:06.987 INFO:tasks.rados.rados.0.smithi157.stdout:adding op weight delete -> 10 2024-09-16T03:37:06.987 INFO:tasks.rados.rados.0.smithi157.stdout:adding op weight set_chunk -> 30 2024-09-16T03:37:06.987 INFO:tasks.rados.rados.0.smithi157.stdout:adding op weight tier_promote -> 10 2024-09-16T03:37:06.987 INFO:tasks.rados.rados.0.smithi157.stdout:adding op weight tier_evict -> 10 2024-09-16T03:37:06.987 INFO:tasks.rados.rados.0.smithi157.stdout:adding op weight tier_flush -> 5 2024-09-16T03:37:06.987 INFO:tasks.rados.rados.0.smithi157.stdout:adding op weight write_excl -> 25 2024-09-16T03:37:06.987 INFO:tasks.rados.rados.0.smithi157.stdout:ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev) 2024-09-16T03:37:06.987 INFO:tasks.rados.rados.0.smithi157.stdout:Configuration: 2024-09-16T03:37:06.988 INFO:tasks.rados.rados.0.smithi157.stdout: Number of operations: 1500 2024-09-16T03:37:06.988 INFO:tasks.rados.rados.0.smithi157.stdout: Number of objects: 50 2024-09-16T03:37:06.988 INFO:tasks.rados.rados.0.smithi157.stdout: Max in flight operations: 16 2024-09-16T03:37:06.988 INFO:tasks.rados.rados.0.smithi157.stdout: Object size (in bytes): 4000000 2024-09-16T03:37:06.989 INFO:tasks.rados.rados.0.smithi157.stdout: Write stride min: 400000 2024-09-16T03:37:06.989 INFO:tasks.rados.rados.0.smithi157.stdout: Write stride max: 800000 2024-09-16T03:37:07.035 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:07.034+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:07.038 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:07.038 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:07.038 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:37:07.038 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:07.047 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:37:07.139 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:07.136+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:07.202 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:07.202 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:07.202 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:07.202 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:07.202 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:07.210 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:37:07.240 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:07.235+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:07.314 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:07.314 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:07.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:07.315 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [] 2024-09-16T03:37:07.315 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:07.324 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:37:07.339 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:07.337+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:07.439 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:07.436+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:07.442 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:07.442 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:07.442 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:37:07.442 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:07.451 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:37:07.502 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:37:07.548 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:07.546+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:07.596 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:37:07.597 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:37:07.597 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:07.606 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:37:07.642 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:07.642 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:07.650 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:07.647+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:07.659 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:07.659 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:07.676 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:07.677 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:07.693 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:07.694 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:07.694 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:37:07.694 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:37:07.694 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:37:07.694 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T03:37:07.695 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T03:37:07.711 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:07.711 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:07.711 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:07.712 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:07.720 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:37:07.729 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:07.729 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:07.746 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:07.746 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:07.762 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:07.759+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:07.763 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:07.763 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:07.779 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:07.779 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:07.796 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:07.796 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:07.817 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:07.818 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:07.834 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:07.834 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:07.844 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:07.841+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:07.853 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:07.853 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:07.866 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:07.867 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:07.867 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:07.867 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:07.875 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:37:07.883 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:07.883 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:07.900 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:07.901 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:07.960 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:07.957+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:08.005 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:37:08.005 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:08.005 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 35, 2024-09-16T03:37:08.005 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2", 2024-09-16T03:37:08.005 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-16T03:36:37.310421+0000", 2024-09-16T03:37:08.005 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-16T03:37:07.938717+0000", 2024-09-16T03:37:08.005 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-09-16T03:36:50.058111+0000", 2024-09-16T03:37:08.005 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-09-16T03:36:41.474235+0000", 2024-09-16T03:37:08.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5806080, 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: "nodeep-scrub", 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: "noscrub", 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 7, 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T03:37:08.006 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-09-16T03:37:08.007 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 4, 2024-09-16T03:37:08.007 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 16, 2024-09-16T03:37:08.007 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-09-16T03:37:08.007 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "jewel", 2024-09-16T03:37:08.007 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-09-16T03:37:08.007 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-09-16T03:37:08.007 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-09-16T03:37:08.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.007 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-09-16T03:37:08.007 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-09-16T03:37:08.007 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-09-16T03:36:49.696307+0000", 2024-09-16T03:37:08.007 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-09-16T03:37:08.007 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-09-16T03:37:08.008 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-09-16T03:37:08.008 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-09-16T03:37:08.008 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-09-16T03:37:08.008 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-09-16T03:37:08.008 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:37:08.008 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:37:08.008 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:37:08.008 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:37:08.008 INFO:teuthology.orchestra.run.smithi064.stdout: "is_stretch_pool": false, 2024-09-16T03:37:08.008 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-09-16T03:37:08.008 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:37:08.008 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 1, 2024-09-16T03:37:08.009 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 1, 2024-09-16T03:37:08.009 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 1, 2024-09-16T03:37:08.009 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 1, 2024-09-16T03:37:08.009 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 1, 2024-09-16T03:37:08.009 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-09-16T03:37:08.009 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-09-16T03:37:08.009 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-09-16T03:37:08.009 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-09-16T03:37:08.009 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-09-16T03:37:08.009 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-09-16T03:37:08.009 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-09-16T03:37:08.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.009 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "25", 2024-09-16T03:37:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-09-16T03:37:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:37:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T03:37:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-09-16T03:37:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:37:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-09-16T03:37:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-09-16T03:37:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-09-16T03:37:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-09-16T03:37:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-09-16T03:37:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-09-16T03:37:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-09-16T03:37:08.010 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-09-16T03:37:08.011 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-09-16T03:37:08.011 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-09-16T03:37:08.011 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-09-16T03:37:08.011 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-09-16T03:37:08.011 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-09-16T03:37:08.011 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:37:08.011 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:37:08.011 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:37:08.011 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-09-16T03:37:08.011 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-09-16T03:37:08.011 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-09-16T03:37:08.011 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-09-16T03:37:08.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-09-16T03:37:08.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.012 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-09-16T03:37:08.012 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-09-16T03:37:08.012 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-09-16T03:37:08.012 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:37:08.012 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:37:08.012 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:37:08.012 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:37:08.012 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-09-16T03:37:08.012 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-09-16T03:37:08.012 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-09-16T03:37:08.012 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-09-16T03:37:08.012 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-09-16T03:37:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-09-16T03:37:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-09-16T03:37:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-09-16T03:37:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-09-16T03:37:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-09-16T03:37:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-09-16T03:37:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 15.789999961853027, 2024-09-16T03:37:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 15.789999961853027, 2024-09-16T03:37:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.18999999761581421, 2024-09-16T03:37:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 3, 2024-09-16T03:37:08.013 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 3, 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "rbd", 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-09-16T03:36:51.904455+0000", 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-09-16T03:37:08.014 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "is_stretch_pool": false, 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 8, 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 8, 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 8, 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 8, 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 8, 2024-09-16T03:37:08.015 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-09-16T03:37:08.016 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-09-16T03:37:08.016 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-09-16T03:37:08.016 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-09-16T03:37:08.016 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-09-16T03:37:08.016 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-09-16T03:37:08.016 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-09-16T03:37:08.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.016 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "28", 2024-09-16T03:37:08.016 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-09-16T03:37:08.016 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:37:08.016 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T03:37:08.016 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-09-16T03:37:08.017 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:37:08.017 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 2, 2024-09-16T03:37:08.017 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 28, 2024-09-16T03:37:08.017 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-09-16T03:37:08.017 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-09-16T03:37:08.017 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-09-16T03:37:08.017 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-09-16T03:37:08.017 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-09-16T03:37:08.017 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-09-16T03:37:08.017 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-09-16T03:37:08.017 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-09-16T03:37:08.017 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-09-16T03:37:08.017 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-09-16T03:37:08.018 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:37:08.019 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:37:08.019 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:37:08.019 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:37:08.019 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-09-16T03:37:08.019 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-09-16T03:37:08.019 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-09-16T03:37:08.019 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-09-16T03:37:08.019 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-09-16T03:37:08.019 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-09-16T03:37:08.019 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-09-16T03:37:08.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.019 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-09-16T03:37:08.019 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 5.9699997901916504, 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 5.9699997901916504, 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.87999999523162842, 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 5.25, 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 5.25, 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "low_tier", 2024-09-16T03:37:08.020 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-09-16T03:37:02.999795+0000", 2024-09-16T03:37:08.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-09-16T03:37:08.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-09-16T03:37:08.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-09-16T03:37:08.021 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-09-16T03:37:08.021 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-09-16T03:37:08.021 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-09-16T03:37:08.021 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:37:08.021 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:37:08.021 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:37:08.021 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:37:08.021 INFO:teuthology.orchestra.run.smithi064.stdout: "is_stretch_pool": false, 2024-09-16T03:37:08.021 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-09-16T03:37:08.021 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:37:08.022 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 4, 2024-09-16T03:37:08.022 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 4, 2024-09-16T03:37:08.022 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 4, 2024-09-16T03:37:08.022 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 4, 2024-09-16T03:37:08.022 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 4, 2024-09-16T03:37:08.022 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-09-16T03:37:08.022 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-09-16T03:37:08.022 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-09-16T03:37:08.022 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-09-16T03:37:08.022 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-09-16T03:37:08.022 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "33", 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-09-16T03:37:08.023 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-09-16T03:37:08.024 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-09-16T03:37:08.024 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-09-16T03:37:08.024 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-09-16T03:37:08.024 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-09-16T03:37:08.024 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-09-16T03:37:08.024 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-09-16T03:37:08.024 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-09-16T03:37:08.024 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:37:08.024 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:37:08.024 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:37:08.024 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-09-16T03:37:08.024 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-09-16T03:37:08.024 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-09-16T03:37:08.025 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": {}, 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 3.9900000095367432, 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 3.9900000095367432, 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.68999999761581421, 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.75, 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.75, 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.027 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 4, 2024-09-16T03:37:08.027 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "unique_pool_0", 2024-09-16T03:37:08.027 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-09-16T03:37:04.279293+0000", 2024-09-16T03:37:08.027 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-09-16T03:37:08.027 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-09-16T03:37:08.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-09-16T03:37:08.027 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-09-16T03:37:08.027 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-09-16T03:37:08.027 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-09-16T03:37:08.027 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:37:08.027 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:37:08.027 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "is_stretch_pool": false, 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 16, 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 16, 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 16, 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 16, 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 16, 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-09-16T03:37:08.028 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-09-16T03:37:08.029 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-09-16T03:37:08.029 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-09-16T03:37:08.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.029 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "35", 2024-09-16T03:37:08.029 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-09-16T03:37:08.029 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:37:08.029 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T03:37:08.029 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-09-16T03:37:08.029 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:37:08.029 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-09-16T03:37:08.029 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-09-16T03:37:08.030 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-09-16T03:37:08.030 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-09-16T03:37:08.030 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-09-16T03:37:08.030 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-09-16T03:37:08.030 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-09-16T03:37:08.030 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-09-16T03:37:08.030 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-09-16T03:37:08.030 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-09-16T03:37:08.030 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-09-16T03:37:08.030 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-09-16T03:37:08.030 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-09-16T03:37:08.030 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:37:08.031 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:37:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:37:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-09-16T03:37:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-09-16T03:37:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-09-16T03:37:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-09-16T03:37:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-09-16T03:37:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-09-16T03:37:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: "rados": {} 2024-09-16T03:37:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-09-16T03:37:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-09-16T03:37:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 3, 2024-09-16T03:37:08.032 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 3, 2024-09-16T03:37:08.033 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 1, 2024-09-16T03:37:08.033 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 3, 2024-09-16T03:37:08.033 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 3, 2024-09-16T03:37:08.033 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 1, 2024-09-16T03:37:08.033 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 1, 2024-09-16T03:37:08.033 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 1 2024-09-16T03:37:08.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.033 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:37:08.033 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:37:08.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.033 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-09-16T03:37:08.033 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "bdbe961a-78d3-4f7d-88ed-ce24475d49e3", 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 0, 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3924351635 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3924351635 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.036 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.036 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.036 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-09-16T03:37:08.036 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3924351635 2024-09-16T03:37:08.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.036 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.036 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.036 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3924351635 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6813/3924351635", 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6814/3924351635", 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6816/3924351635", 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6815/3924351635", 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "154c8ddd-7a3d-402c-b9d6-28538f4e6bc4", 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 32, 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.038 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6804", 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 624977804 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6805", 2024-09-16T03:37:08.039 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 624977804 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6807", 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 624977804 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.040 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6806", 2024-09-16T03:37:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 624977804 2024-09-16T03:37:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.112:6804/624977804", 2024-09-16T03:37:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.112:6805/624977804", 2024-09-16T03:37:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.112:6807/624977804", 2024-09-16T03:37:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.112:6806/624977804", 2024-09-16T03:37:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.041 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "2a06de4e-0835-4a42-8c8a-49fb55aab566", 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 23, 2024-09-16T03:37:08.042 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 0, 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6812", 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2804648086 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6813", 2024-09-16T03:37:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2804648086 2024-09-16T03:37:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6815", 2024-09-16T03:37:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2804648086 2024-09-16T03:37:08.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6814", 2024-09-16T03:37:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2804648086 2024-09-16T03:37:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.155:6812/2804648086", 2024-09-16T03:37:08.045 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.155:6813/2804648086", 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.155:6815/2804648086", 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.155:6814/2804648086", 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "8a87f9f0-c380-4f38-bbd2-06b2def4fd2d", 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.046 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:37:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 0, 2024-09-16T03:37:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6800", 2024-09-16T03:37:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1935597988 2024-09-16T03:37:08.047 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6801", 2024-09-16T03:37:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1935597988 2024-09-16T03:37:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.048 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6803", 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1935597988 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6802", 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1935597988 2024-09-16T03:37:08.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.157:6800/1935597988", 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.157:6801/1935597988", 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.157:6803/1935597988", 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.157:6802/1935597988", 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-09-16T03:37:08.050 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "5ac53318-12e8-496c-a87f-ba5de79c52aa", 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 32, 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-09-16T03:37:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675475248 2024-09-16T03:37:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-09-16T03:37:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675475248 2024-09-16T03:37:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.052 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-09-16T03:37:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675475248 2024-09-16T03:37:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.053 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-09-16T03:37:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675475248 2024-09-16T03:37:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/2675475248", 2024-09-16T03:37:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6810/2675475248", 2024-09-16T03:37:08.054 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6812/2675475248", 2024-09-16T03:37:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6811/2675475248", 2024-09-16T03:37:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-09-16T03:37:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "675c81a4-2314-463e-82a7-0060735c78fd", 2024-09-16T03:37:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.055 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 21, 2024-09-16T03:37:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 32, 2024-09-16T03:37:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6800", 2024-09-16T03:37:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 499905276 2024-09-16T03:37:08.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6801", 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 499905276 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6803", 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 499905276 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6802", 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 499905276 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.112:6800/499905276", 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.112:6801/499905276", 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.112:6803/499905276", 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.112:6802/499905276", 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "b18962a7-3b23-4e50-bf31-1d0629557423", 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.059 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 23, 2024-09-16T03:37:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 31, 2024-09-16T03:37:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.060 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6808", 2024-09-16T03:37:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2839983406 2024-09-16T03:37:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6809", 2024-09-16T03:37:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2839983406 2024-09-16T03:37:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6811", 2024-09-16T03:37:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2839983406 2024-09-16T03:37:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6810", 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2839983406 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.155:6808/2839983406", 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.155:6809/2839983406", 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.155:6811/2839983406", 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.155:6810/2839983406", 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-09-16T03:37:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "9706c1e8-dac0-4e16-8bc5-7cacc6f13870", 2024-09-16T03:37:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:37:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 0, 2024-09-16T03:37:08.064 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6808", 2024-09-16T03:37:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3021982871 2024-09-16T03:37:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6809", 2024-09-16T03:37:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3021982871 2024-09-16T03:37:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6811", 2024-09-16T03:37:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3021982871 2024-09-16T03:37:08.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6810", 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3021982871 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.157:6808/3021982871", 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.157:6809/3021982871", 2024-09-16T03:37:08.067 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.157:6811/3021982871", 2024-09-16T03:37:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.157:6810/3021982871", 2024-09-16T03:37:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 8, 2024-09-16T03:37:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "fafd06b8-c005-4471-a128-9dd51580c0f0", 2024-09-16T03:37:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.068 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 32, 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3609017627 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-09-16T03:37:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3609017627 2024-09-16T03:37:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.070 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-09-16T03:37:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3609017627 2024-09-16T03:37:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-09-16T03:37:08.071 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3609017627 2024-09-16T03:37:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6805/3609017627", 2024-09-16T03:37:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6806/3609017627", 2024-09-16T03:37:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6808/3609017627", 2024-09-16T03:37:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6807/3609017627", 2024-09-16T03:37:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 9, 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "79e38e57-52c2-416f-9b90-75621f06d3e3", 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 25, 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.073 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6812", 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 701981815 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6813", 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 701981815 2024-09-16T03:37:08.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6815", 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 701981815 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.075 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6814", 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 701981815 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.112:6812/701981815", 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.112:6813/701981815", 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.112:6815/701981815", 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.112:6814/701981815", 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 10, 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10", 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 32, 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.077 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6804", 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4138619523 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6805", 2024-09-16T03:37:08.078 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4138619523 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6807", 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4138619523 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.079 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6806", 2024-09-16T03:37:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4138619523 2024-09-16T03:37:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.155:6804/4138619523", 2024-09-16T03:37:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.155:6805/4138619523", 2024-09-16T03:37:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.155:6807/4138619523", 2024-09-16T03:37:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.155:6806/4138619523", 2024-09-16T03:37:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.080 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 11, 2024-09-16T03:37:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "7a2a169a-ce94-4091-a792-27d0ecf9a7fa", 2024-09-16T03:37:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.081 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 23, 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 32, 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6804", 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3607367937 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.082 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6805", 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3607367937 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6807", 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3607367937 2024-09-16T03:37:08.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6806", 2024-09-16T03:37:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3607367937 2024-09-16T03:37:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.084 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.157:6804/3607367937", 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.157:6805/3607367937", 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.157:6807/3607367937", 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.157:6806/3607367937", 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 12, 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "7a280cda-b740-40c8-9fae-1a1ff11651f6", 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.085 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 21, 2024-09-16T03:37:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 32, 2024-09-16T03:37:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-09-16T03:37:08.086 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2984461266 2024-09-16T03:37:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-09-16T03:37:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2984461266 2024-09-16T03:37:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.087 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-09-16T03:37:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2984461266 2024-09-16T03:37:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.088 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-09-16T03:37:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2984461266 2024-09-16T03:37:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/2984461266", 2024-09-16T03:37:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6802/2984461266", 2024-09-16T03:37:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6804/2984461266", 2024-09-16T03:37:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6803/2984461266", 2024-09-16T03:37:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 13, 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "ce3a3148-cd90-422c-a106-045de89ab045", 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 0, 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.090 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6808", 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1616314527 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.092 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.092 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6809", 2024-09-16T03:37:08.092 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1616314527 2024-09-16T03:37:08.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.092 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.092 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.092 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.092 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.092 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6811", 2024-09-16T03:37:08.092 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1616314527 2024-09-16T03:37:08.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6810", 2024-09-16T03:37:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1616314527 2024-09-16T03:37:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.112:6808/1616314527", 2024-09-16T03:37:08.093 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.112:6809/1616314527", 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.112:6811/1616314527", 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.112:6810/1616314527", 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 14, 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "461c2d79-f4e7-4b21-899c-50d4a6cf12f9", 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.094 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 21, 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 25, 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6800", 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4203868121 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.095 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6801", 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4203868121 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.096 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6803", 2024-09-16T03:37:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4203868121 2024-09-16T03:37:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6802", 2024-09-16T03:37:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4203868121 2024-09-16T03:37:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.097 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.155:6800/4203868121", 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.155:6801/4203868121", 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.155:6803/4203868121", 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.155:6802/4203868121", 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 15, 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "9a8c0bbe-9389-4b2c-95e1-34d3e66461e9", 2024-09-16T03:37:08.098 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 23, 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 32, 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.099 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6812", 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 181354529 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6813", 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 181354529 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6815", 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 181354529 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:37:08.101 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6814", 2024-09-16T03:37:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 181354529 2024-09-16T03:37:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.157:6812/181354529", 2024-09-16T03:37:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.157:6813/181354529", 2024-09-16T03:37:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.157:6815/181354529", 2024-09-16T03:37:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.157:6814/181354529", 2024-09-16T03:37:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:37:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:37:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:37:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:08.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:47.702114+0000", 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.103 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-09-16T03:37:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:47.496387+0000", 2024-09-16T03:37:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-09-16T03:37:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.104 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:48.252480+0000", 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.105 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:47.791534+0000", 2024-09-16T03:37:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-09-16T03:37:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:47.517661+0000", 2024-09-16T03:37:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:47.018557+0000", 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.107 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:48.247312+0000", 2024-09-16T03:37:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-09-16T03:37:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.108 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:47.959242+0000", 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 8, 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:47.506293+0000", 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.109 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 9, 2024-09-16T03:37:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:47.963066+0000", 2024-09-16T03:37:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 10, 2024-09-16T03:37:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.110 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:47.961846+0000", 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 11, 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:48.574000+0000", 2024-09-16T03:37:08.111 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 12, 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:46.945017+0000", 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 13, 2024-09-16T03:37:08.112 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:47.752207+0000", 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 14, 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.113 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:46.975964+0000", 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 15, 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:36:48.501682+0000", 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.114 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [], 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [], 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [], 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": {}, 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-09-16T03:37:08.115 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [], 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [], 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-09-16T03:37:08.116 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:08.117 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:08.118 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:08.004+0000 7f9288166640 -1 osd.14 34 handle_osd_map injecting map crc failure 2024-09-16T03:37:08.119 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:08.119 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:08.120 INFO:tasks.thrashosds.thrasher:j is {'epoch': 35, 'fsid': 'b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2', 'created': '2024-09-16T03:36:37.310421+0000', 'modified': '2024-09-16T03:37:07.938717+0000', 'last_up_change': '2024-09-16T03:36:50.058111+0000', 'last_in_change': '2024-09-16T03:36:41.474235+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T03:36:49.696307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.789999961853027, 'score_stable': 15.789999961853027, 'optimal_score': 0.1899999976158142, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T03:36:51.904455+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5.96999979019165, 'score_stable': 5.96999979019165, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 5.25, 'raw_score_stable': 5.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-09-16T03:37:02.999795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.990000009536743, 'score_stable': 3.990000009536743, 'optimal_score': 0.6899999976158142, 'raw_score_acting': 2.75, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T03:37:04.279293+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'bdbe961a-78d3-4f7d-88ed-ce24475d49e3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 3924351635}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6814', 'nonce': 3924351635}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6816', 'nonce': 3924351635}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 3924351635}]}, 'public_addr': '172.21.15.64:6813/3924351635', 'cluster_addr': '172.21.15.64:6814/3924351635', 'heartbeat_back_addr': '172.21.15.64:6816/3924351635', 'heartbeat_front_addr': '172.21.15.64:6815/3924351635', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '154c8ddd-7a3d-402c-b9d6-28538f4e6bc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6804', 'nonce': 624977804}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6805', 'nonce': 624977804}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6807', 'nonce': 624977804}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6806', 'nonce': 624977804}]}, 'public_addr': '172.21.15.112:6804/624977804', 'cluster_addr': '172.21.15.112:6805/624977804', 'heartbeat_back_addr': '172.21.15.112:6807/624977804', 'heartbeat_front_addr': '172.21.15.112:6806/624977804', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2a06de4e-0835-4a42-8c8a-49fb55aab566', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6812', 'nonce': 2804648086}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6813', 'nonce': 2804648086}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6815', 'nonce': 2804648086}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6814', 'nonce': 2804648086}]}, 'public_addr': '172.21.15.155:6812/2804648086', 'cluster_addr': '172.21.15.155:6813/2804648086', 'heartbeat_back_addr': '172.21.15.155:6815/2804648086', 'heartbeat_front_addr': '172.21.15.155:6814/2804648086', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8a87f9f0-c380-4f38-bbd2-06b2def4fd2d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6800', 'nonce': 1935597988}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6801', 'nonce': 1935597988}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6803', 'nonce': 1935597988}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6802', 'nonce': 1935597988}]}, 'public_addr': '172.21.15.157:6800/1935597988', 'cluster_addr': '172.21.15.157:6801/1935597988', 'heartbeat_back_addr': '172.21.15.157:6803/1935597988', 'heartbeat_front_addr': '172.21.15.157:6802/1935597988', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '5ac53318-12e8-496c-a87f-ba5de79c52aa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 2675475248}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6810', 'nonce': 2675475248}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6812', 'nonce': 2675475248}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 2675475248}]}, 'public_addr': '172.21.15.64:6809/2675475248', 'cluster_addr': '172.21.15.64:6810/2675475248', 'heartbeat_back_addr': '172.21.15.64:6812/2675475248', 'heartbeat_front_addr': '172.21.15.64:6811/2675475248', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '675c81a4-2314-463e-82a7-0060735c78fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6800', 'nonce': 499905276}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6801', 'nonce': 499905276}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6803', 'nonce': 499905276}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6802', 'nonce': 499905276}]}, 'public_addr': '172.21.15.112:6800/499905276', 'cluster_addr': '172.21.15.112:6801/499905276', 'heartbeat_back_addr': '172.21.15.112:6803/499905276', 'heartbeat_front_addr': '172.21.15.112:6802/499905276', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b18962a7-3b23-4e50-bf31-1d0629557423', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 31, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6808', 'nonce': 2839983406}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6809', 'nonce': 2839983406}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6811', 'nonce': 2839983406}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6810', 'nonce': 2839983406}]}, 'public_addr': '172.21.15.155:6808/2839983406', 'cluster_addr': '172.21.15.155:6809/2839983406', 'heartbeat_back_addr': '172.21.15.155:6811/2839983406', 'heartbeat_front_addr': '172.21.15.155:6810/2839983406', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '9706c1e8-dac0-4e16-8bc5-7cacc6f13870', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6808', 'nonce': 3021982871}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6809', 'nonce': 3021982871}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6811', 'nonce': 3021982871}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6810', 'nonce': 3021982871}]}, 'public_addr': '172.21.15.157:6808/3021982871', 'cluster_addr': '172.21.15.157:6809/3021982871', 'heartbeat_back_addr': '172.21.15.157:6811/3021982871', 'heartbeat_front_addr': '172.21.15.157:6810/3021982871', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'fafd06b8-c005-4471-a128-9dd51580c0f0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 3609017627}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6806', 'nonce': 3609017627}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6808', 'nonce': 3609017627}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 3609017627}]}, 'public_addr': '172.21.15.64:6805/3609017627', 'cluster_addr': '172.21.15.64:6806/3609017627', 'heartbeat_back_addr': '172.21.15.64:6808/3609017627', 'heartbeat_front_addr': '172.21.15.64:6807/3609017627', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '79e38e57-52c2-416f-9b90-75621f06d3e3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6812', 'nonce': 701981815}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6813', 'nonce': 701981815}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6815', 'nonce': 701981815}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6814', 'nonce': 701981815}]}, 'public_addr': '172.21.15.112:6812/701981815', 'cluster_addr': '172.21.15.112:6813/701981815', 'heartbeat_back_addr': '172.21.15.112:6815/701981815', 'heartbeat_front_addr': '172.21.15.112:6814/701981815', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6804', 'nonce': 4138619523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6805', 'nonce': 4138619523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6807', 'nonce': 4138619523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6806', 'nonce': 4138619523}]}, 'public_addr': '172.21.15.155:6804/4138619523', 'cluster_addr': '172.21.15.155:6805/4138619523', 'heartbeat_back_addr': '172.21.15.155:6807/4138619523', 'heartbeat_front_addr': '172.21.15.155:6806/4138619523', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '7a2a169a-ce94-4091-a792-27d0ecf9a7fa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6804', 'nonce': 3607367937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6805', 'nonce': 3607367937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6807', 'nonce': 3607367937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6806', 'nonce': 3607367937}]}, 'public_addr': '172.21.15.157:6804/3607367937', 'cluster_addr': '172.21.15.157:6805/3607367937', 'heartbeat_back_addr': '172.21.15.157:6807/3607367937', 'heartbeat_front_addr': '172.21.15.157:6806/3607367937', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': '7a280cda-b740-40c8-9fae-1a1ff11651f6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 2984461266}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6802', 'nonce': 2984461266}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6804', 'nonce': 2984461266}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 2984461266}]}, 'public_addr': '172.21.15.64:6801/2984461266', 'cluster_addr': '172.21.15.64:6802/2984461266', 'heartbeat_back_addr': '172.21.15.64:6804/2984461266', 'heartbeat_front_addr': '172.21.15.64:6803/2984461266', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': 'ce3a3148-cd90-422c-a106-045de89ab045', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6808', 'nonce': 1616314527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6809', 'nonce': 1616314527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6811', 'nonce': 1616314527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6810', 'nonce': 1616314527}]}, 'public_addr': '172.21.15.112:6808/1616314527', 'cluster_addr': '172.21.15.112:6809/1616314527', 'heartbeat_back_addr': '172.21.15.112:6811/1616314527', 'heartbeat_front_addr': '172.21.15.112:6810/1616314527', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '461c2d79-f4e7-4b21-899c-50d4a6cf12f9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6800', 'nonce': 4203868121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6801', 'nonce': 4203868121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6803', 'nonce': 4203868121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6802', 'nonce': 4203868121}]}, 'public_addr': '172.21.15.155:6800/4203868121', 'cluster_addr': '172.21.15.155:6801/4203868121', 'heartbeat_back_addr': '172.21.15.155:6803/4203868121', 'heartbeat_front_addr': '172.21.15.155:6802/4203868121', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': '9a8c0bbe-9389-4b2c-95e1-34d3e66461e9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6812', 'nonce': 181354529}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6813', 'nonce': 181354529}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6815', 'nonce': 181354529}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6814', 'nonce': 181354529}]}, 'public_addr': '172.21.15.157:6812/181354529', 'cluster_addr': '172.21.15.157:6813/181354529', 'heartbeat_back_addr': '172.21.15.157:6815/181354529', 'heartbeat_front_addr': '172.21.15.157:6814/181354529', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:47.702114+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:47.496387+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:48.252480+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:47.791534+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:47.517661+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:47.018557+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:48.247312+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:47.959242+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:47.506293+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:47.963066+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:47.961846+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:48.574000+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:46.945017+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:47.752207+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:46.975964+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:36:48.501682+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T03:37:08.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:37:08.121 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:37:08.121 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:08.116+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:08.219 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:08.218+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:08.265 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:08.265 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:08.274 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:37:08.320 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:08.319+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:08.378 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:08.378 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:08.387 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:37:08.420 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:37:08.420 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:37:08.422 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:08.421+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:08.432 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-16T03:37:07.944501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":54,"acting":54,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":14,"kb":1509949440,"kb_used":444648,"kb_used_data":3776,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504792,"statfs":{"total":1546188226560,"available":1545732907008,"internally_reserved":0,"allocated":3866624,"data_stored":3405933,"data_compressed":93744,"data_compressed_allocated":122880,"data_compressed_original":1536000,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":11,"num_write":25,"num_write_kb":586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.023734"},"pg_stats":[{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:37:04.919920+0000","last_change":"2024-09-16T03:37:04.919920+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:04.919920+0000","last_undegraded":"2024-09-16T03:37:04.919920+0000","last_fullsized":"2024-09-16T03:37:04.919920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:37:04.919920+0000","last_change":"2024-09-16T03:37:04.919920+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:04.919920+0000","last_undegraded":"2024-09-16T03:37:04.919920+0000","last_fullsized":"2024-09-16T03:37:04.919920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:37:04.919920+0000","last_change":"2024-09-16T03:37:04.919920+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:04.919920+0000","last_undegraded":"2024-09-16T03:37:04.919920+0000","last_fullsized":"2024-09-16T03:37:04.919920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:37:04.919920+0000","last_change":"2024-09-16T03:37:04.919920+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:04.919920+0000","last_undegraded":"2024-09-16T03:37:04.919920+0000","last_fullsized":"2024-09-16T03:37:04.919920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":4,"reported_epoch":33,"state":"creating+activating","last_fresh":"2024-09-16T03:37:05.942730+0000","last_change":"2024-09-16T03:37:05.937515+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:05.942730+0000","last_undegraded":"2024-09-16T03:37:05.942730+0000","last_fullsized":"2024-09-16T03:37:05.942730+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":["15","4","9"],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:37:04.919920+0000","last_change":"2024-09-16T03:37:04.919920+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:04.919920+0000","last_undegraded":"2024-09-16T03:37:04.919920+0000","last_fullsized":"2024-09-16T03:37:04.919920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:37:04.919920+0000","last_change":"2024-09-16T03:37:04.919920+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:04.919920+0000","last_undegraded":"2024-09-16T03:37:04.919920+0000","last_fullsized":"2024-09-16T03:37:04.919920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:37:04.919920+0000","last_change":"2024-09-16T03:37:04.919920+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:04.919920+0000","last_undegraded":"2024-09-16T03:37:04.919920+0000","last_fullsized":"2024-09-16T03:37:04.919920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":4,"reported_epoch":33,"state":"creating+activating","last_fresh":"2024-09-16T03:37:06.139090+0000","last_change":"2024-09-16T03:37:05.937160+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:06.139090+0000","last_undegraded":"2024-09-16T03:37:06.139090+0000","last_fullsized":"2024-09-16T03:37:06.139090+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":["15","7","11"],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":166,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T03:37:06.036578+0000","last_change":"2024-09-16T03:36:51.858122+0000","last_active":"2024-09-16T03:37:06.036578+0000","last_peered":"2024-09-16T03:37:06.036578+0000","last_clean":"2024-09-16T03:37:06.036578+0000","last_became_active":"2024-09-16T03:36:51.855488+0000","last_became_peered":"2024-09-16T03:36:51.855488+0000","last_unstale":"2024-09-16T03:37:06.036578+0000","last_undegraded":"2024-09-16T03:37:06.036578+0000","last_fullsized":"2024-09-16T03:37:06.036578+0000","mapping_epoch":23,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:36:50.058111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:54.474463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":32,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T03:37:06.037046+0000","last_change":"2024-09-16T03:36:55.504324+0000","last_active":"2024-09-16T03:37:06.037046+0000","last_peered":"2024-09-16T03:37:06.037046+0000","last_clean":"2024-09-16T03:37:06.037046+0000","last_became_active":"2024-09-16T03:36:53.284975+0000","last_became_peered":"2024-09-16T03:36:53.284975+0000","last_unstale":"2024-09-16T03:37:06.037046+0000","last_undegraded":"2024-09-16T03:37:06.037046+0000","last_fullsized":"2024-09-16T03:37:06.037046+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:55.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T03:37:06.036654+0000","last_change":"2024-09-16T03:37:04.942264+0000","last_active":"2024-09-16T03:37:06.036654+0000","last_peered":"2024-09-16T03:37:06.036654+0000","last_clean":"2024-09-16T03:37:06.036654+0000","last_became_active":"2024-09-16T03:37:04.941953+0000","last_became_peered":"2024-09-16T03:37:04.941953+0000","last_unstale":"2024-09-16T03:37:06.036654+0000","last_undegraded":"2024-09-16T03:37:06.036654+0000","last_fullsized":"2024-09-16T03:37:06.036654+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:04.021943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:37:04.919920+0000","last_change":"2024-09-16T03:37:04.919920+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:04.919920+0000","last_undegraded":"2024-09-16T03:37:04.919920+0000","last_fullsized":"2024-09-16T03:37:04.919920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T03:37:02.286371+0000","last_change":"2024-09-16T03:36:55.200565+0000","last_active":"2024-09-16T03:37:02.286371+0000","last_peered":"2024-09-16T03:37:02.286371+0000","last_clean":"2024-09-16T03:37:02.286371+0000","last_became_active":"2024-09-16T03:36:53.182912+0000","last_became_peered":"2024-09-16T03:36:53.182912+0000","last_unstale":"2024-09-16T03:37:02.286371+0000","last_undegraded":"2024-09-16T03:37:02.286371+0000","last_fullsized":"2024-09-16T03:37:02.286371+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:08.335036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T03:37:05.933194+0000","last_change":"2024-09-16T03:37:05.134238+0000","last_active":"2024-09-16T03:37:05.933194+0000","last_peered":"2024-09-16T03:37:05.933194+0000","last_clean":"2024-09-16T03:37:05.933194+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:37:05.933194+0000","last_undegraded":"2024-09-16T03:37:05.933194+0000","last_fullsized":"2024-09-16T03:37:05.933194+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:13.684062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:37:04.919920+0000","last_change":"2024-09-16T03:37:04.919920+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:04.919920+0000","last_undegraded":"2024-09-16T03:37:04.919920+0000","last_fullsized":"2024-09-16T03:37:04.919920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":28,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T03:37:05.936808+0000","last_change":"2024-09-16T03:36:56.004079+0000","last_active":"2024-09-16T03:37:05.936808+0000","last_peered":"2024-09-16T03:37:05.936808+0000","last_clean":"2024-09-16T03:37:05.936808+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:37:05.936808+0000","last_undegraded":"2024-09-16T03:37:05.936808+0000","last_fullsized":"2024-09-16T03:37:05.936808+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.599963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:37:03.913037+0000","last_change":"2024-09-16T03:37:03.913037+0000","last_active":"2024-09-16T03:37:03.913037+0000","last_peered":"2024-09-16T03:37:03.913037+0000","last_clean":"2024-09-16T03:37:03.913037+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:03.913037+0000","last_undegraded":"2024-09-16T03:37:03.913037+0000","last_fullsized":"2024-09-16T03:37:03.913037+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":4,"reported_epoch":33,"state":"creating+activating","last_fresh":"2024-09-16T03:37:06.038867+0000","last_change":"2024-09-16T03:37:05.936513+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:06.038867+0000","last_undegraded":"2024-09-16T03:37:06.038867+0000","last_fullsized":"2024-09-16T03:37:06.038867+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":["11","2","15"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":27,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T03:37:02.286382+0000","last_change":"2024-09-16T03:36:55.201161+0000","last_active":"2024-09-16T03:37:02.286382+0000","last_peered":"2024-09-16T03:37:02.286382+0000","last_clean":"2024-09-16T03:37:02.286382+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:37:02.286382+0000","last_undegraded":"2024-09-16T03:37:02.286382+0000","last_fullsized":"2024-09-16T03:37:02.286382+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:07.449762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T03:37:05.937578+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:37:05.937578+0000","last_peered":"2024-09-16T03:37:05.937578+0000","last_clean":"2024-09-16T03:37:05.937578+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:37:05.937578+0000","last_undegraded":"2024-09-16T03:37:05.937578+0000","last_fullsized":"2024-09-16T03:37:05.937578+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:19.199339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T03:37:06.043855+0000","last_change":"2024-09-16T03:37:06.043381+0000","last_active":"2024-09-16T03:37:06.043855+0000","last_peered":"2024-09-16T03:37:06.043855+0000","last_clean":"2024-09-16T03:37:06.043855+0000","last_became_active":"2024-09-16T03:37:06.043063+0000","last_became_peered":"2024-09-16T03:37:06.043063+0000","last_unstale":"2024-09-16T03:37:06.043855+0000","last_undegraded":"2024-09-16T03:37:06.043855+0000","last_fullsized":"2024-09-16T03:37:06.043855+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:05.001572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T03:37:06.033137+0000","last_change":"2024-09-16T03:36:55.270467+0000","last_active":"2024-09-16T03:37:06.033137+0000","last_peered":"2024-09-16T03:37:06.033137+0000","last_clean":"2024-09-16T03:37:06.033137+0000","last_became_active":"2024-09-16T03:36:53.570778+0000","last_became_peered":"2024-09-16T03:36:53.570778+0000","last_unstale":"2024-09-16T03:37:06.033137+0000","last_undegraded":"2024-09-16T03:37:06.033137+0000","last_fullsized":"2024-09-16T03:37:06.033137+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:03.986104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-16T03:37:06.038792+0000","last_change":"2024-09-16T03:37:06.038263+0000","last_active":"2024-09-16T03:37:06.038792+0000","last_peered":"2024-09-16T03:37:06.038792+0000","last_clean":"2024-09-16T03:37:06.038792+0000","last_became_active":"2024-09-16T03:37:06.036785+0000","last_became_peered":"2024-09-16T03:37:06.036785+0000","last_unstale":"2024-09-16T03:37:06.038792+0000","last_undegraded":"2024-09-16T03:37:06.038792+0000","last_fullsized":"2024-09-16T03:37:06.038792+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:22.904281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T03:37:01.951057+0000","last_change":"2024-09-16T03:36:55.605861+0000","last_active":"2024-09-16T03:37:01.951057+0000","last_peered":"2024-09-16T03:37:01.951057+0000","last_clean":"2024-09-16T03:37:01.951057+0000","last_became_active":"2024-09-16T03:36:53.379870+0000","last_became_peered":"2024-09-16T03:36:53.379870+0000","last_unstale":"2024-09-16T03:37:01.951057+0000","last_undegraded":"2024-09-16T03:37:01.951057+0000","last_fullsized":"2024-09-16T03:37:01.951057+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:04.628405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044368400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:37:04.919920+0000","last_change":"2024-09-16T03:37:04.919920+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:04.919920+0000","last_undegraded":"2024-09-16T03:37:04.919920+0000","last_fullsized":"2024-09-16T03:37:04.919920+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T03:37:01.786246+0000","last_change":"2024-09-16T03:36:55.883348+0000","last_active":"2024-09-16T03:37:01.786246+0000","last_peered":"2024-09-16T03:37:01.786246+0000","last_clean":"2024-09-16T03:37:01.786246+0000","last_became_active":"2024-09-16T03:36:53.672370+0000","last_became_peered":"2024-09-16T03:36:53.672370+0000","last_unstale":"2024-09-16T03:37:01.786246+0000","last_undegraded":"2024-09-16T03:37:01.786246+0000","last_fullsized":"2024-09-16T03:37:01.786246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:52.231720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":4,"reported_epoch":33,"state":"creating+activating","last_fresh":"2024-09-16T03:37:05.942909+0000","last_change":"2024-09-16T03:37:05.936534+0000","last_active":"2024-09-16T03:37:04.919920+0000","last_peered":"2024-09-16T03:37:04.919920+0000","last_clean":"2024-09-16T03:37:04.919920+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:37:05.942909+0000","last_undegraded":"2024-09-16T03:37:05.942909+0000","last_fullsized":"2024-09-16T03:37:05.942909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,12],"acting":[11,8,12],"avail_no_missing":["11","8","12"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-16T03:37:02.386429+0000","last_change":"2024-09-16T03:36:55.100546+0000","last_active":"2024-09-16T03:37:02.386429+0000","last_peered":"2024-09-16T03:37:02.386429+0000","last_clean":"2024-09-16T03:37:02.386429+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:37:02.386429+0000","last_undegraded":"2024-09-16T03:37:02.386429+0000","last_fullsized":"2024-09-16T03:37:02.386429+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.896892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":3},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":4},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":1586784,"data_compressed":93744,"data_compressed_allocated":122880,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":118823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":307200,"data_stored":651836,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,5,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247813,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":118538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280518,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":274432,"data_stored":638446,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":111851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":110739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,5,7,8,9,10,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":270336,"data_stored":112753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":208896,"data_stored":107228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":111851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":335872,"data_stored":646996,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":217088,"data_stored":110739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":110148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":111851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[2,3,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":111598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,7,9,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":113961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":217088,"data_stored":118575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:37:08.434 INFO:tasks.thrashosds.thrasher:Setting 4.1 to [4, 7, 2] 2024-09-16T03:37:08.434 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.1', '4', '7', '2'] 2024-09-16T03:37:08.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 4.1 4 7 2 2024-09-16T03:37:08.491 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:08.491 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:08.500 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:37:08.523 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:08.522+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:08.604 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:08.604 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:08.612 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:37:08.623 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:08.622+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:08.716 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:08.716 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:08.724 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:08.723+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:08.725 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:37:08.826 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:08.823+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:08.828 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:08.828 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:08.837 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:37:08.926 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:08.925+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:08.946 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:08.945+0000 7f3a21722640 -1 osd.6 35 handle_osd_map injecting map crc failure 2024-09-16T03:37:08.948 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:08.946+0000 7f49419d0640 -1 osd.4 35 handle_osd_map injecting map crc failure 2024-09-16T03:37:08.948 INFO:teuthology.orchestra.run.smithi064.stderr:set 4.1 pg_upmap mapping to [4,7,2] 2024-09-16T03:37:08.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:37:08.983 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:08.983 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:08.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:37:09.027 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:09.025+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:09.114 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:09.115 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:09.123 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:37:09.166 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:09.165+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:09.229 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:09.228+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:09.230 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:09.230 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:09.239 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:37:09.330 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:09.328+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:09.387 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:09.387 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:09.395 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:37:09.431 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:09.428+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:09.502 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:09.503 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:09.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:37:09.532 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:09.529+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:09.618 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:09.618 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:09.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:37:09.632 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:09.630+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:09.733 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:09.731+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:09.734 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:09.734 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:09.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:37:09.834 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:09.832+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:09.850 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:09.850 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:09.859 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:37:09.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:37:09.935 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:09.933+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:09.950 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:09.948+0000 7f49419d0640 -1 osd.4 36 handle_osd_map injecting map crc failure 2024-09-16T03:37:09.950 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:37:10.012 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:10.012 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:10.021 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:37:10.035 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:10.034+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:10.048 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:10.046+0000 7f66f3205640 -1 osd.15 36 handle_osd_map injecting map crc failure 2024-09-16T03:37:10.091 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:10.091 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:10.111 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:10.111 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:10.128 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:10.128 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:10.136 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:10.135+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:10.145 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:10.145 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:10.146 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:10.146 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:10.155 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:37:10.163 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:10.164 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:10.183 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:10.183 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:10.199 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:10.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:10.217 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:10.217 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:10.233 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:10.233 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:10.237 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:10.236+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:10.250 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:10.250 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:10.268 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:10.269 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:10.273 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:10.273 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:10.282 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:37:10.285 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:10.285 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:10.303 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:10.304 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:10.321 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:10.321 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:10.338 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:10.339 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:10.339 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:10.337+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:10.355 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:10.356 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:10.395 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:10.395 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:10.404 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:37:10.439 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:10.438+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:10.509 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:10.507+0000 7fedb5739640 -1 osd.9 36 handle_osd_map injecting map crc failure 2024-09-16T03:37:10.540 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:10.538+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:10.551 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:10.551 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:10.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:10.551 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-09-16T03:37:10.552 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:10.561 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:37:10.642 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:10.639+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:10.678 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:10.679 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:10.679 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:10.679 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:10.687 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:37:10.743 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:10.741+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:10.791 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:10.791 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:10.791 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:37:10.791 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:37:10.791 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:10.799 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:37:10.884 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:10.882+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:10.942 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:10.942 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:10.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:10.942 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [] 2024-09-16T03:37:10.943 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:10.944 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:10.942+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:10.949 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:10.947+0000 7ff26836a640 -1 osd.11 37 handle_osd_map injecting map crc failure 2024-09-16T03:37:10.951 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:37:11.045 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:11.042+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:11.093 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:11.093 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:11.093 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:11.094 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:11.102 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:37:11.145 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:11.143+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:11.244 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:11.244 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:11.245 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:37:11.245 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:37:11.245 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:11.246 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:11.244+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:11.253 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:37:11.347 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:11.345+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:11.358 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:11.358 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:11.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:11.358 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:37:11.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.358 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:18 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'8, mlcod=25'5)", 2024-09-16T03:37:11.358 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:52.878066+0000", 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.476132497999998, 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102244637, 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:52.878066+0000", 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:52.878066+0000", 2024-09-16T03:37:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:52.878068+0000", 2024-09-16T03:37:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.81e-06 2024-09-16T03:37:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:52.878102+0000", 2024-09-16T03:37:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4022000000000001e-05 2024-09-16T03:37:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:52.878120+0000", 2024-09-16T03:37:11.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7640000000000001e-05 2024-09-16T03:37:11.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:52.878138+0000", 2024-09-16T03:37:11.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8150000000000001e-05 2024-09-16T03:37:11.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:52.978181+0000", 2024-09-16T03:37:11.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100042934 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:52.978201+0000", 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0497e-05 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:52.980273+0000", 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020723069999999998 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:52.980311+0000", 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7277000000000002e-05 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:46 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'28, mlcod=26'19)", 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:54.698600+0000", 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.655598804, 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20236221500000001, 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.363 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.698600+0000", 2024-09-16T03:37:11.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.698600+0000", 2024-09-16T03:37:11.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.698601+0000", 2024-09-16T03:37:11.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9400000000000004e-07 2024-09-16T03:37:11.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.698609+0000", 2024-09-16T03:37:11.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6670000000000002e-06 2024-09-16T03:37:11.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.698618+0000", 2024-09-16T03:37:11.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4209999999999995e-06 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798730+0000", 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10011264 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798868+0000", 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000137813 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798915+0000", 2024-09-16T03:37:11.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7015000000000003e-05 2024-09-16T03:37:11.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.900932+0000", 2024-09-16T03:37:11.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102016828 2024-09-16T03:37:11.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.900962+0000", 2024-09-16T03:37:11.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9937000000000002e-05 2024-09-16T03:37:11.368 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.368 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.369 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.369 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:47 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'29, mlcod=26'19)", 2024-09-16T03:37:11.369 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:54.798768+0000", 2024-09-16T03:37:11.369 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.555430130000001, 2024-09-16T03:37:11.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10221208599999999, 2024-09-16T03:37:11.369 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.369 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.369 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.370 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798768+0000", 2024-09-16T03:37:11.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798768+0000", 2024-09-16T03:37:11.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798770+0000", 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.813e-06 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798914+0000", 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000143504 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798925+0000", 2024-09-16T03:37:11.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0941000000000001e-05 2024-09-16T03:37:11.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798930+0000", 2024-09-16T03:37:11.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2079999999999999e-06 2024-09-16T03:37:11.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.899442+0000", 2024-09-16T03:37:11.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100512119 2024-09-16T03:37:11.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.899476+0000", 2024-09-16T03:37:11.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3798999999999999e-05 2024-09-16T03:37:11.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.900965+0000", 2024-09-16T03:37:11.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00148902 2024-09-16T03:37:11.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.900981+0000", 2024-09-16T03:37:11.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5682e-05 2024-09-16T03:37:11.374 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.374 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.374 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.374 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:48 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'30, mlcod=26'19)", 2024-09-16T03:37:11.374 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:54.798943+0000", 2024-09-16T03:37:11.374 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.555255781, 2024-09-16T03:37:11.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104342214, 2024-09-16T03:37:11.374 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.374 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.375 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.375 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798943+0000", 2024-09-16T03:37:11.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798943+0000", 2024-09-16T03:37:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798943+0000", 2024-09-16T03:37:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2e-07 2024-09-16T03:37:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798954+0000", 2024-09-16T03:37:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0883999999999999e-05 2024-09-16T03:37:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798960+0000", 2024-09-16T03:37:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8089999999999998e-06 2024-09-16T03:37:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798972+0000", 2024-09-16T03:37:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.183e-05 2024-09-16T03:37:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.899803+0000", 2024-09-16T03:37:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10083091600000001 2024-09-16T03:37:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.899823+0000", 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0003000000000001e-05 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.903243+0000", 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0034197300000000002 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.903285+0000", 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2521999999999997e-05 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:49 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'31, mlcod=26'19)", 2024-09-16T03:37:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:54.798979+0000", 2024-09-16T03:37:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.555219994000002, 2024-09-16T03:37:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.104688724, 2024-09-16T03:37:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798979+0000", 2024-09-16T03:37:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798979+0000", 2024-09-16T03:37:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798979+0000", 2024-09-16T03:37:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:37:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798985+0000", 2024-09-16T03:37:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4450000000000001e-06 2024-09-16T03:37:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798989+0000", 2024-09-16T03:37:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7919999999999999e-06 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.798993+0000", 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7560000000000001e-06 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.900144+0000", 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101150851 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.900159+0000", 2024-09-16T03:37:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5526e-05 2024-09-16T03:37:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.903626+0000", 2024-09-16T03:37:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0034661900000000001 2024-09-16T03:37:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.903667+0000", 2024-09-16T03:37:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1811999999999999e-05 2024-09-16T03:37:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.384 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:50 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'32, mlcod=26'19)", 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:54.801027+0000", 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.553172033999999, 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102662747, 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.801027+0000", 2024-09-16T03:37:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.801027+0000", 2024-09-16T03:37:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.801028+0000", 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3540000000000001e-06 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.801103+0000", 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5485e-05 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.801117+0000", 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3919e-05 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.801124+0000", 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4869999999999999e-06 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.900374+0000", 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099250242000000002 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.900391+0000", 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7048999999999999e-05 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.903670+0000", 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003278745 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.903689+0000", 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9466e-05 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.389 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:51 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'33, mlcod=26'19)", 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:54.801756+0000", 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.552442251999999, 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101951496, 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.801756+0000", 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.801756+0000", 2024-09-16T03:37:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.801758+0000", 2024-09-16T03:37:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4360000000000001e-06 2024-09-16T03:37:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.801775+0000", 2024-09-16T03:37:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6864e-05 2024-09-16T03:37:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.801788+0000", 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2979999999999999e-05 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.801795+0000", 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8889999999999996e-06 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.900747+0000", 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098952949999999998 2024-09-16T03:37:11.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.900761+0000", 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3915e-05 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.903691+0000", 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029295789999999999 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:54.903708+0000", 2024-09-16T03:37:11.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6883000000000001e-05 2024-09-16T03:37:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:70 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'48, mlcod=28'40)", 2024-09-16T03:37:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:56.573255+0000", 2024-09-16T03:37:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.780943619, 2024-09-16T03:37:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20136780100000001, 2024-09-16T03:37:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.394 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.395 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:56.573255+0000", 2024-09-16T03:37:11.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:56.573255+0000", 2024-09-16T03:37:11.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:56.573256+0000", 2024-09-16T03:37:11.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4470000000000001e-06 2024-09-16T03:37:11.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:56.573263+0000", 2024-09-16T03:37:11.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5860000000000001e-06 2024-09-16T03:37:11.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:56.573275+0000", 2024-09-16T03:37:11.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2466000000000001e-05 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:56.573283+0000", 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7190000000000001e-06 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:56.773221+0000", 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19993765199999999 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:56.773277+0000", 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6335999999999997e-05 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:56.774588+0000", 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013112829999999999 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:56.774623+0000", 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4311999999999999e-05 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:77 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'55, mlcod=28'53)", 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:57.284389+0000", 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.069809373, 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101559154, 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.284389+0000", 2024-09-16T03:37:11.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.284389+0000", 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.284391+0000", 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.685e-06 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.284456+0000", 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4875000000000005e-05 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.284469+0000", 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3284e-05 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.284495+0000", 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5582e-05 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.284573+0000", 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8819999999999994e-05 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.284595+0000", 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1905000000000001e-05 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.385913+0000", 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10131802500000001 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.385948+0000", 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4978000000000003e-05 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:80 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'58, mlcod=28'53)", 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:57.286378+0000", 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.067820243, 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102888364, 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.403 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.286378+0000", 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.286378+0000", 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.286379+0000", 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.071e-06 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.286498+0000", 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000118743 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.286507+0000", 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5029999999999997e-06 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.286513+0000", 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9789999999999998e-06 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.386689+0000", 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100176364 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.386704+0000", 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5218000000000001e-05 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.389223+0000", 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025182830000000001 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.389267+0000", 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4202999999999997e-05 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.407 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:81 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'59, mlcod=28'57)", 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:57.491946+0000", 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.862252722000001, 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20114105800000001, 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.491946+0000", 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.491946+0000", 2024-09-16T03:37:11.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.491948+0000", 2024-09-16T03:37:11.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6129999999999998e-06 2024-09-16T03:37:11.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.491964+0000", 2024-09-16T03:37:11.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5044e-05 2024-09-16T03:37:11.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.491989+0000", 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5269000000000002e-05 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.592117+0000", 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10012858700000001 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.692271+0000", 2024-09-16T03:37:11.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100153213 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.692288+0000", 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7258999999999999e-05 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.693034+0000", 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074585999999999999 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:57.693087+0000", 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3213e-05 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:92 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'66, mlcod=28'60)", 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:58.786417+0000", 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 12.567781588999999, 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20163715199999999, 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.412 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.786417+0000", 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.786417+0000", 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.786418+0000", 2024-09-16T03:37:11.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4470000000000001e-06 2024-09-16T03:37:11.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.786428+0000", 2024-09-16T03:37:11.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4360000000000008e-06 2024-09-16T03:37:11.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.786443+0000", 2024-09-16T03:37:11.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5163e-05 2024-09-16T03:37:11.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886572+0000", 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10012942900000001 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886706+0000", 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013349 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886765+0000", 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9364999999999998e-05 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.988020+0000", 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101254263 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.988054+0000", 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4558999999999999e-05 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:93 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'67, mlcod=28'61)", 2024-09-16T03:37:11.416 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:58.886603+0000", 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 12.467595935, 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101770498, 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886603+0000", 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886603+0000", 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886605+0000", 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1799999999999999e-06 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886680+0000", 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4935e-05 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886700+0000", 2024-09-16T03:37:11.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0506e-05 2024-09-16T03:37:11.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886717+0000", 2024-09-16T03:37:11.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6296000000000001e-05 2024-09-16T03:37:11.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.987197+0000", 2024-09-16T03:37:11.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100480737 2024-09-16T03:37:11.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.987232+0000", 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4415999999999997e-05 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.988343+0000", 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011112330000000001 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.988373+0000", 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0195000000000001e-05 2024-09-16T03:37:11.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:94 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'68, mlcod=28'61)", 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:36:58.886742+0000", 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 12.467456322, 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10181921100000001, 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886742+0000", 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886742+0000", 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886744+0000", 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.367e-06 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886753+0000", 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4460000000000007e-06 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886767+0000", 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3833e-05 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.886778+0000", 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1420000000000001e-05 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.987813+0000", 2024-09-16T03:37:11.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101034347 2024-09-16T03:37:11.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.987826+0000", 2024-09-16T03:37:11.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3607e-05 2024-09-16T03:37:11.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.988534+0000", 2024-09-16T03:37:11.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000707437 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:36:58.988561+0000", 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7753999999999999e-05 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:116 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'86, mlcod=28'80)", 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:00.662963+0000", 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.691235303999999, 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101616046, 2024-09-16T03:37:11.425 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.662963+0000", 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.662963+0000", 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.662964+0000", 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2900000000000002e-07 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.662970+0000", 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4349999999999996e-06 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.662980+0000", 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8810000000000008e-06 2024-09-16T03:37:11.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.763068+0000", 2024-09-16T03:37:11.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100088473 2024-09-16T03:37:11.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.763907+0000", 2024-09-16T03:37:11.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00083923899999999998 2024-09-16T03:37:11.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.763924+0000", 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7124000000000001e-05 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.764540+0000", 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000615125 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.764579+0000", 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9839999999999998e-05 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.429 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:117 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'87, mlcod=28'80)", 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:00.763763+0000", 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.5904357, 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10161619500000001, 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.763763+0000", 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.763763+0000", 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.763764+0000", 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.392e-06 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.763799+0000", 2024-09-16T03:37:11.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4762999999999999e-05 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.763811+0000", 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2184000000000001e-05 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.763817+0000", 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5910000000000003e-06 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.863281+0000", 2024-09-16T03:37:11.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099464204000000001 2024-09-16T03:37:11.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.863315+0000", 2024-09-16T03:37:11.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3506000000000001e-05 2024-09-16T03:37:11.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.865340+0000", 2024-09-16T03:37:11.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020252260000000002 2024-09-16T03:37:11.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.865379+0000", 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9329000000000002e-05 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:119 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'89, mlcod=28'85)", 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:00.965923+0000", 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.388275415000001, 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102308368, 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.434 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.965923+0000", 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.965923+0000", 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.965924+0000", 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8099999999999998e-07 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.965934+0000", 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9709999999999999e-06 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.965949+0000", 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5218000000000001e-05 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.965976+0000", 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6914999999999999e-05 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.966102+0000", 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012641200000000001 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.966127+0000", 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5355e-05 2024-09-16T03:37:11.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.068158+0000", 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102030361 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.068232+0000", 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3755000000000001e-05 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:120 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'90, mlcod=28'85)", 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:00.966307+0000", 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.387891166999999, 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103432519, 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.966307+0000", 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.966307+0000", 2024-09-16T03:37:11.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.966308+0000", 2024-09-16T03:37:11.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:37:11.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.966401+0000", 2024-09-16T03:37:11.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2727999999999997e-05 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.966409+0000", 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6170000000000001e-06 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.966414+0000", 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.694e-06 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.066618+0000", 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100203378 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.066676+0000", 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8362000000000001e-05 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.069699+0000", 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030231490000000001 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.069740+0000", 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0867e-05 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:121 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'91, mlcod=28'87)", 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:00.967285+0000", 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.386913826000001, 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10248593, 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.443 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.967285+0000", 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.967285+0000", 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.967285+0000", 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7800000000000001e-07 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.967290+0000", 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3680000000000004e-06 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:00.967297+0000", 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1629999999999999e-06 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.067391+0000", 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100094433 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.067567+0000", 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017594400000000001 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.067588+0000", 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0709000000000001e-05 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.069742+0000", 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002154127 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.069771+0000", 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8507999999999999e-05 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.447 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4260.0:128 1.0 e29/23 1:30306672:devicehealth::main.db.0000000000000000:head v 29'98, mlcod=28'93)", 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:01.372936+0000", 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.9812630789999996, 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.16095910599999999, 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.372936+0000", 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.372936+0000", 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.372937+0000", 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6610000000000001e-06 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.372982+0000", 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4333999999999999e-05 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.372995+0000", 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3645999999999999e-05 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.373002+0000", 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3489999999999997e-06 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.533085+0000", 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.160083962 2024-09-16T03:37:11.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.533141+0000", 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5791999999999997e-05 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.533848+0000", 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00070697100000000003 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:01.533895+0000", 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6390999999999997e-05 2024-09-16T03:37:11.451 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.452 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:11.452 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:11.452 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:11.453 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:37:11.454 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:11.451+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:11.554 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:11.553+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:11.557 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:11.557 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:11.557 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:11.557 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:11.566 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:37:11.655 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:11.654+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:11.668 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:11.668 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:11.668 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:37:11.668 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:37:11.668 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:11.678 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:37:11.756 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:11.755+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:11.821 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:11.821 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:11.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:11.821 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [] 2024-09-16T03:37:11.821 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:11.830 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:37:11.858 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:11.856+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:11.933 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:11.933 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:11.933 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:11.933 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:11.942 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:37:11.950 INFO:tasks.rados.rados.0.smithi157.stdout:1: write initial oid 1 2024-09-16T03:37:11.951 INFO:tasks.rados.rados.0.smithi157.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-09-16T03:37:11.955 INFO:tasks.rados.rados.0.smithi157.stdout:1: writing smithi15736696-1 from 416807 to 892056 tid 1 2024-09-16T03:37:11.958 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:11.956+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:11.962 INFO:tasks.rados.rados.0.smithi157.stdout:1: writing smithi15736696-1 from 1542129 to 2016807 tid 2 2024-09-16T03:37:11.962 INFO:tasks.rados.rados.0.smithi157.stdout:2: write initial oid 2 2024-09-16T03:37:11.962 INFO:tasks.rados.rados.0.smithi157.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-09-16T03:37:11.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:37:11.967 INFO:tasks.rados.rados.0.smithi157.stdout:2: writing smithi15736696-2 from 416807 to 892056 tid 1 2024-09-16T03:37:11.973 INFO:tasks.rados.rados.0.smithi157.stdout:2: writing smithi15736696-2 from 1542129 to 2016807 tid 2 2024-09-16T03:37:11.973 INFO:tasks.rados.rados.0.smithi157.stdout:3: write initial oid 3 2024-09-16T03:37:11.974 INFO:tasks.rados.rados.0.smithi157.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-09-16T03:37:11.979 INFO:tasks.rados.rados.0.smithi157.stdout:3: writing smithi15736696-3 from 433614 to 984112 tid 1 2024-09-16T03:37:11.985 INFO:tasks.rados.rados.0.smithi157.stdout:3: writing smithi15736696-3 from 1600611 to 2033614 tid 2 2024-09-16T03:37:11.985 INFO:tasks.rados.rados.0.smithi157.stdout:4: write initial oid 4 2024-09-16T03:37:11.985 INFO:tasks.rados.rados.0.smithi157.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-09-16T03:37:11.991 INFO:tasks.rados.rados.0.smithi157.stdout:4: writing smithi15736696-4 from 450421 to 1076168 tid 1 2024-09-16T03:37:11.995 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:11.993+0000 7eff7884b640 -1 osd.3 37 handle_osd_map injecting map crc failure 2024-09-16T03:37:11.997 INFO:tasks.rados.rados.0.smithi157.stdout:4: writing smithi15736696-4 from 1659093 to 2050421 tid 2 2024-09-16T03:37:11.997 INFO:tasks.rados.rados.0.smithi157.stdout:5: write initial oid 5 2024-09-16T03:37:11.997 INFO:tasks.rados.rados.0.smithi157.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-09-16T03:37:12.003 INFO:tasks.rados.rados.0.smithi157.stdout:5: writing smithi15736696-5 from 467228 to 1168224 tid 1 2024-09-16T03:37:12.009 INFO:tasks.rados.rados.0.smithi157.stdout:5: writing smithi15736696-5 from 1717575 to 2067228 tid 2 2024-09-16T03:37:12.009 INFO:tasks.rados.rados.0.smithi157.stdout:6: write initial oid 6 2024-09-16T03:37:12.009 INFO:tasks.rados.rados.0.smithi157.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-09-16T03:37:12.015 INFO:tasks.rados.rados.0.smithi157.stdout:6: writing smithi15736696-6 from 484035 to 1260280 tid 1 2024-09-16T03:37:12.020 INFO:tasks.rados.rados.0.smithi157.stdout:6: writing smithi15736696-6 from 2059704 to 2084035 tid 2 2024-09-16T03:37:12.020 INFO:tasks.rados.rados.0.smithi157.stdout:7: write initial oid 7 2024-09-16T03:37:12.020 INFO:tasks.rados.rados.0.smithi157.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-09-16T03:37:12.025 INFO:tasks.rados.rados.0.smithi157.stdout:7: writing smithi15736696-7 from 500842 to 952336 tid 1 2024-09-16T03:37:12.031 INFO:tasks.rados.rados.0.smithi157.stdout:7: writing smithi15736696-7 from 1718186 to 2100842 tid 2 2024-09-16T03:37:12.031 INFO:tasks.rados.rados.0.smithi157.stdout:8: write initial oid 8 2024-09-16T03:37:12.031 INFO:tasks.rados.rados.0.smithi157.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-09-16T03:37:12.037 INFO:tasks.rados.rados.0.smithi157.stdout:8: writing smithi15736696-8 from 517649 to 1044392 tid 1 2024-09-16T03:37:12.043 INFO:tasks.rados.rados.0.smithi157.stdout:8: writing smithi15736696-8 from 1776668 to 2117649 tid 2 2024-09-16T03:37:12.043 INFO:tasks.rados.rados.0.smithi157.stdout:9: write initial oid 9 2024-09-16T03:37:12.043 INFO:tasks.rados.rados.0.smithi157.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-09-16T03:37:12.045 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:37:12.046 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:12.052 INFO:tasks.rados.rados.0.smithi157.stdout:9: writing smithi15736696-9 from 534456 to 1252801 tid 1 2024-09-16T03:37:12.055 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:37:12.056 INFO:tasks.rados.rados.0.smithi157.stdout:9: writing smithi15736696-9 from 1951503 to 2134456 tid 2 2024-09-16T03:37:12.056 INFO:tasks.rados.rados.0.smithi157.stdout:10: write initial oid 10 2024-09-16T03:37:12.056 INFO:tasks.rados.rados.0.smithi157.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-09-16T03:37:12.059 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:12.057+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:12.062 INFO:tasks.rados.rados.0.smithi157.stdout:10: writing smithi15736696-10 from 551263 to 1344857 tid 1 2024-09-16T03:37:12.066 INFO:tasks.rados.rados.0.smithi157.stdout:10: writing smithi15736696-10 from 1893632 to 2151263 tid 2 2024-09-16T03:37:12.066 INFO:tasks.rados.rados.0.smithi157.stdout:11: write initial oid 11 2024-09-16T03:37:12.066 INFO:tasks.rados.rados.0.smithi157.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-09-16T03:37:12.072 INFO:tasks.rados.rados.0.smithi157.stdout:11: writing smithi15736696-11 from 568070 to 1036913 tid 1 2024-09-16T03:37:12.076 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:12.075+0000 7fd101e6f640 -1 osd.2 38 handle_osd_map injecting map crc failure 2024-09-16T03:37:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:11: writing smithi15736696-11 from 1552114 to 2168070 tid 2 2024-09-16T03:37:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:12: write initial oid 12 2024-09-16T03:37:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-09-16T03:37:12.085 INFO:tasks.rados.rados.0.smithi157.stdout:12: writing smithi15736696-12 from 584877 to 1128969 tid 1 2024-09-16T03:37:12.092 INFO:tasks.rados.rados.0.smithi157.stdout:12: writing smithi15736696-12 from 1610596 to 2172599 tid 2 2024-09-16T03:37:12.092 INFO:tasks.rados.rados.0.smithi157.stdout:13: write initial oid 13 2024-09-16T03:37:12.092 INFO:tasks.rados.rados.0.smithi157.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-09-16T03:37:12.098 INFO:tasks.rados.rados.0.smithi157.stdout:13: writing smithi15736696-13 from 601684 to 1221025 tid 1 2024-09-16T03:37:12.103 INFO:tasks.rados.rados.0.smithi157.stdout:13: writing smithi15736696-13 from 1669078 to 2201684 tid 2 2024-09-16T03:37:12.104 INFO:tasks.rados.rados.0.smithi157.stdout:14: write initial oid 14 2024-09-16T03:37:12.104 INFO:tasks.rados.rados.0.smithi157.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-09-16T03:37:12.111 INFO:tasks.rados.rados.0.smithi157.stdout:14: writing smithi15736696-14 from 618491 to 1313081 tid 1 2024-09-16T03:37:12.116 INFO:tasks.rados.rados.0.smithi157.stdout:14: writing smithi15736696-14 from 2011207 to 2218491 tid 2 2024-09-16T03:37:12.116 INFO:tasks.rados.rados.0.smithi157.stdout:15: write initial oid 15 2024-09-16T03:37:12.116 INFO:tasks.rados.rados.0.smithi157.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-09-16T03:37:12.124 INFO:tasks.rados.rados.0.smithi157.stdout:15: writing smithi15736696-15 from 635298 to 1405137 tid 1 2024-09-16T03:37:12.129 INFO:tasks.rados.rados.0.smithi157.stdout:15: writing smithi15736696-15 from 2069689 to 2235298 tid 2 2024-09-16T03:37:12.129 INFO:tasks.rados.rados.0.smithi157.stdout:16: write initial oid 16 2024-09-16T03:37:12.129 INFO:tasks.rados.rados.0.smithi157.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-09-16T03:37:12.134 INFO:tasks.rados.rados.0.smithi157.stdout:16: writing smithi15736696-16 from 652105 to 1097193 tid 1 2024-09-16T03:37:12.141 INFO:tasks.rados.rados.0.smithi157.stdout:16: writing smithi15736696-16 from 1728171 to 2181159 tid 2 2024-09-16T03:37:12.141 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.141 INFO:tasks.rados.rados.0.smithi157.stdout:2: finishing write tid 1 to smithi15736696-2 2024-09-16T03:37:12.141 INFO:tasks.rados.rados.0.smithi157.stdout:3: finishing write tid 1 to smithi15736696-3 2024-09-16T03:37:12.144 INFO:tasks.rados.rados.0.smithi157.stdout:8: finishing write tid 1 to smithi15736696-8 2024-09-16T03:37:12.161 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:12.159+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:12.162 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:12.162 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [] 2024-09-16T03:37:12.162 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:12.171 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:37:12.188 INFO:tasks.rados.rados.0.smithi157.stdout:3: finishing write tid 2 to smithi15736696-3 2024-09-16T03:37:12.189 INFO:tasks.rados.rados.0.smithi157.stdout:3: finishing write tid 3 to smithi15736696-3 2024-09-16T03:37:12.189 INFO:tasks.rados.rados.0.smithi157.stdout:3: finishing write tid 4 to smithi15736696-3 2024-09-16T03:37:12.189 INFO:tasks.rados.rados.0.smithi157.stdout:3: oid 3 updating version 0 to 2 2024-09-16T03:37:12.190 INFO:tasks.rados.rados.0.smithi157.stdout:3: oid 3 version 2 is already newer than 1 2024-09-16T03:37:12.190 INFO:tasks.rados.rados.0.smithi157.stdout:3: oid 3 updating version 2 to 3 2024-09-16T03:37:12.190 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T03:37:12.190 INFO:tasks.rados.rados.0.smithi157.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-16T03:37:12.190 INFO:tasks.rados.rados.0.smithi157.stdout:3: done (15 left) 2024-09-16T03:37:12.190 INFO:tasks.rados.rados.0.smithi157.stdout:17: write initial oid 17 2024-09-16T03:37:12.190 INFO:tasks.rados.rados.0.smithi157.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-09-16T03:37:12.196 INFO:tasks.rados.rados.0.smithi157.stdout:17: writing smithi15736696-17 from 668912 to 1305602 tid 1 2024-09-16T03:37:12.201 INFO:tasks.rados.rados.0.smithi157.stdout:17: writing smithi15736696-17 from 1903006 to 2268912 tid 2 2024-09-16T03:37:12.201 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.202 INFO:tasks.rados.rados.0.smithi157.stdout:4: finishing write tid 1 to smithi15736696-4 2024-09-16T03:37:12.202 INFO:tasks.rados.rados.0.smithi157.stdout:4: finishing write tid 2 to smithi15736696-4 2024-09-16T03:37:12.258 INFO:tasks.rados.rados.0.smithi157.stdout:11: finishing write tid 1 to smithi15736696-11 2024-09-16T03:37:12.336 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:12.336 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:12.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.337 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:12.337 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:37:12.337 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.097858534999999996, 2024-09-16T03:37:12.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.097892895999999993, 2024-09-16T03:37:12.337 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:12.337 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:12.337 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:12.337 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:12.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:12.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186761+0000", 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076880999999999998 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186772+0000", 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0986e-05 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186799+0000", 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6254e-05 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:12.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:12.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186876+0000", 2024-09-16T03:37:12.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7523999999999998e-05 2024-09-16T03:37:12.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:12.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:12.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186896+0000", 2024-09-16T03:37:12.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0302999999999999e-05 2024-09-16T03:37:12.340 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:12.340 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:12.340 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:12.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:12.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36)", 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.096652745999999998, 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.096743724000000003, 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:12.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187199+0000", 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6199999999999996e-07 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188184+0000", 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000985442 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:12.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:12.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188194+0000", 2024-09-16T03:37:12.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5659999999999996e-06 2024-09-16T03:37:12.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:12.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:12.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:12.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188210+0000", 2024-09-16T03:37:12.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5858999999999998e-05 2024-09-16T03:37:12.344 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:12.344 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:12.344 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:12.344 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:12.344 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2024-09-16T03:37:12.344 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 2 2024-09-16T03:37:12.344 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:12.344 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:37:12.347 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:12.334+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:12.348 INFO:tasks.rados.rados.0.smithi157.stdout:12: finishing write tid 1 to smithi15736696-12 2024-09-16T03:37:12.349 INFO:tasks.rados.rados.0.smithi157.stdout:6: finishing write tid 1 to smithi15736696-6 2024-09-16T03:37:12.349 INFO:tasks.rados.rados.0.smithi157.stdout:6: finishing write tid 2 to smithi15736696-6 2024-09-16T03:37:12.349 INFO:tasks.rados.rados.0.smithi157.stdout:6: finishing write tid 3 to smithi15736696-6 2024-09-16T03:37:12.349 INFO:tasks.rados.rados.0.smithi157.stdout:6: finishing write tid 4 to smithi15736696-6 2024-09-16T03:37:12.349 INFO:tasks.rados.rados.0.smithi157.stdout:6: oid 6 updating version 0 to 4 2024-09-16T03:37:12.349 INFO:tasks.rados.rados.0.smithi157.stdout:6: oid 6 updating version 4 to 5 2024-09-16T03:37:12.349 INFO:tasks.rados.rados.0.smithi157.stdout:6: oid 6 updating version 5 to 6 2024-09-16T03:37:12.349 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:37:12.349 INFO:tasks.rados.rados.0.smithi157.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-16T03:37:12.349 INFO:tasks.rados.rados.0.smithi157.stdout:6: done (15 left) 2024-09-16T03:37:12.349 INFO:tasks.rados.rados.0.smithi157.stdout:18: write initial oid 18 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout:18: writing smithi15736696-18 from 685719 to 1397658 tid 1 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout:18: writing smithi15736696-18 from 1845135 to 2285719 tid 2 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout:4: finishing write tid 3 to smithi15736696-4 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout:4: finishing write tid 4 to smithi15736696-4 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout:4: oid 4 updating version 0 to 2 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout:4: oid 4 version 2 is already newer than 1 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout:4: oid 4 updating version 2 to 3 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout:17: finishing write tid 1 to smithi15736696-17 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout:4: done (15 left) 2024-09-16T03:37:12.350 INFO:tasks.rados.rados.0.smithi157.stdout:19: write initial oid 19 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout:19: writing smithi15736696-19 from 702526 to 1489714 tid 1 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout:19: writing smithi15736696-19 from 1903617 to 2302526 tid 2 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout:15: finishing write tid 1 to smithi15736696-15 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout:15: finishing write tid 2 to smithi15736696-15 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout:15: finishing write tid 3 to smithi15736696-15 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout:2: finishing write tid 2 to smithi15736696-2 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout:2: finishing write tid 3 to smithi15736696-2 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout:2: finishing write tid 4 to smithi15736696-2 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout:2: oid 2 updating version 0 to 2 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout:2: oid 2 updating version 2 to 3 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout:2: oid 2 version 3 is already newer than 1 2024-09-16T03:37:12.351 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T03:37:12.352 INFO:tasks.rados.rados.0.smithi157.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-16T03:37:12.352 INFO:tasks.rados.rados.0.smithi157.stdout:5: finishing write tid 1 to smithi15736696-5 2024-09-16T03:37:12.352 INFO:tasks.rados.rados.0.smithi157.stdout:2: done (15 left) 2024-09-16T03:37:12.352 INFO:tasks.rados.rados.0.smithi157.stdout:20: write initial oid 20 2024-09-16T03:37:12.352 INFO:tasks.rados.rados.0.smithi157.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-09-16T03:37:12.353 INFO:tasks.rados.rados.0.smithi157.stdout:20: writing smithi15736696-20 from 719333 to 1181770 tid 1 2024-09-16T03:37:12.361 INFO:tasks.rados.rados.0.smithi157.stdout:20: writing smithi15736696-20 from 1962099 to 2319333 tid 2 2024-09-16T03:37:12.361 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.361 INFO:tasks.rados.rados.0.smithi157.stdout:11: finishing write tid 2 to smithi15736696-11 2024-09-16T03:37:12.361 INFO:tasks.rados.rados.0.smithi157.stdout:1: finishing write tid 1 to smithi15736696-1 2024-09-16T03:37:12.361 INFO:tasks.rados.rados.0.smithi157.stdout:1: finishing write tid 2 to smithi15736696-1 2024-09-16T03:37:12.368 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:37:12.391 INFO:tasks.rados.rados.0.smithi157.stdout:12: finishing write tid 2 to smithi15736696-12 2024-09-16T03:37:12.391 INFO:tasks.rados.rados.0.smithi157.stdout:12: finishing write tid 3 to smithi15736696-12 2024-09-16T03:37:12.392 INFO:tasks.rados.rados.0.smithi157.stdout:12: finishing write tid 4 to smithi15736696-12 2024-09-16T03:37:12.392 INFO:tasks.rados.rados.0.smithi157.stdout:12: oid 12 updating version 0 to 1 2024-09-16T03:37:12.392 INFO:tasks.rados.rados.0.smithi157.stdout:12: oid 12 updating version 1 to 2 2024-09-16T03:37:12.392 INFO:tasks.rados.rados.0.smithi157.stdout:12: oid 12 updating version 2 to 3 2024-09-16T03:37:12.392 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:37:12.392 INFO:tasks.rados.rados.0.smithi157.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-16T03:37:12.392 INFO:tasks.rados.rados.0.smithi157.stdout:12: done (15 left) 2024-09-16T03:37:12.392 INFO:tasks.rados.rados.0.smithi157.stdout:21: write initial oid 21 2024-09-16T03:37:12.392 INFO:tasks.rados.rados.0.smithi157.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-09-16T03:37:12.398 INFO:tasks.rados.rados.0.smithi157.stdout:21: writing smithi15736696-21 from 736140 to 1273826 tid 1 2024-09-16T03:37:12.404 INFO:tasks.rados.rados.0.smithi157.stdout:21: writing smithi15736696-21 from 2020581 to 2336140 tid 2 2024-09-16T03:37:12.404 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.404 INFO:tasks.rados.rados.0.smithi157.stdout:5: finishing write tid 2 to smithi15736696-5 2024-09-16T03:37:12.405 INFO:tasks.rados.rados.0.smithi157.stdout:5: finishing write tid 3 to smithi15736696-5 2024-09-16T03:37:12.405 INFO:tasks.rados.rados.0.smithi157.stdout:5: finishing write tid 4 to smithi15736696-5 2024-09-16T03:37:12.405 INFO:tasks.rados.rados.0.smithi157.stdout:5: oid 5 updating version 0 to 2 2024-09-16T03:37:12.405 INFO:tasks.rados.rados.0.smithi157.stdout:5: oid 5 version 2 is already newer than 1 2024-09-16T03:37:12.405 INFO:tasks.rados.rados.0.smithi157.stdout:5: oid 5 updating version 2 to 3 2024-09-16T03:37:12.405 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T03:37:12.405 INFO:tasks.rados.rados.0.smithi157.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-16T03:37:12.405 INFO:tasks.rados.rados.0.smithi157.stdout:5: done (15 left) 2024-09-16T03:37:12.405 INFO:tasks.rados.rados.0.smithi157.stdout:22: write initial oid 22 2024-09-16T03:37:12.405 INFO:tasks.rados.rados.0.smithi157.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-09-16T03:37:12.411 INFO:tasks.rados.rados.0.smithi157.stdout:22: writing smithi15736696-22 from 752947 to 1365882 tid 1 2024-09-16T03:37:12.416 INFO:tasks.rados.rados.0.smithi157.stdout:22: writing smithi15736696-22 from 1962710 to 2352947 tid 2 2024-09-16T03:37:12.416 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.431 INFO:tasks.rados.rados.0.smithi157.stdout:15: finishing write tid 4 to smithi15736696-15 2024-09-16T03:37:12.431 INFO:tasks.rados.rados.0.smithi157.stdout:15: oid 15 updating version 0 to 3 2024-09-16T03:37:12.431 INFO:tasks.rados.rados.0.smithi157.stdout:15: oid 15 version 3 is already newer than 1 2024-09-16T03:37:12.431 INFO:tasks.rados.rados.0.smithi157.stdout:15: oid 15 version 3 is already newer than 2 2024-09-16T03:37:12.431 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:12.431 INFO:tasks.rados.rados.0.smithi157.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-16T03:37:12.431 INFO:tasks.rados.rados.0.smithi157.stdout:15: done (15 left) 2024-09-16T03:37:12.432 INFO:tasks.rados.rados.0.smithi157.stdout:23: write initial oid 23 2024-09-16T03:37:12.432 INFO:tasks.rados.rados.0.smithi157.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-09-16T03:37:12.436 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:12.435+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:12.438 INFO:tasks.rados.rados.0.smithi157.stdout:23: writing smithi15736696-23 from 769754 to 1457938 tid 1 2024-09-16T03:37:12.444 INFO:tasks.rados.rados.0.smithi157.stdout:23: writing smithi15736696-23 from 2021192 to 2369754 tid 2 2024-09-16T03:37:12.444 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.453 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:12.453 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:12.453 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:12.458 INFO:tasks.rados.rados.0.smithi157.stdout:9: finishing write tid 1 to smithi15736696-9 2024-09-16T03:37:12.463 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:37:12.463 INFO:tasks.rados.rados.0.smithi157.stdout:1: finishing write tid 3 to smithi15736696-1 2024-09-16T03:37:12.463 INFO:tasks.rados.rados.0.smithi157.stdout:11: finishing write tid 3 to smithi15736696-11 2024-09-16T03:37:12.463 INFO:tasks.rados.rados.0.smithi157.stdout:1: finishing write tid 4 to smithi15736696-1 2024-09-16T03:37:12.463 INFO:tasks.rados.rados.0.smithi157.stdout:1: oid 1 updating version 0 to 1 2024-09-16T03:37:12.463 INFO:tasks.rados.rados.0.smithi157.stdout:1: oid 1 updating version 1 to 3 2024-09-16T03:37:12.463 INFO:tasks.rados.rados.0.smithi157.stdout:1: oid 1 version 3 is already newer than 2 2024-09-16T03:37:12.463 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T03:37:12.464 INFO:tasks.rados.rados.0.smithi157.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-16T03:37:12.464 INFO:tasks.rados.rados.0.smithi157.stdout:1: done (15 left) 2024-09-16T03:37:12.464 INFO:tasks.rados.rados.0.smithi157.stdout:24: write initial oid 24 2024-09-16T03:37:12.464 INFO:tasks.rados.rados.0.smithi157.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-09-16T03:37:12.468 INFO:tasks.rados.rados.0.smithi157.stdout:24: writing smithi15736696-24 from 786561 to 1266347 tid 1 2024-09-16T03:37:12.474 INFO:tasks.rados.rados.0.smithi157.stdout:24: writing smithi15736696-24 from 1796027 to 2263691 tid 2 2024-09-16T03:37:12.474 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.474 INFO:tasks.rados.rados.0.smithi157.stdout:11: finishing write tid 4 to smithi15736696-11 2024-09-16T03:37:12.474 INFO:tasks.rados.rados.0.smithi157.stdout:11: oid 11 updating version 0 to 1 2024-09-16T03:37:12.474 INFO:tasks.rados.rados.0.smithi157.stdout:11: oid 11 updating version 1 to 3 2024-09-16T03:37:12.475 INFO:tasks.rados.rados.0.smithi157.stdout:11: oid 11 version 3 is already newer than 2 2024-09-16T03:37:12.475 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T03:37:12.475 INFO:tasks.rados.rados.0.smithi157.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-16T03:37:12.475 INFO:tasks.rados.rados.0.smithi157.stdout:11: done (15 left) 2024-09-16T03:37:12.475 INFO:tasks.rados.rados.0.smithi157.stdout:25: write initial oid 25 2024-09-16T03:37:12.475 INFO:tasks.rados.rados.0.smithi157.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-09-16T03:37:12.479 INFO:tasks.rados.rados.0.smithi157.stdout:25: writing smithi15736696-25 from 403368 to 958403 tid 1 2024-09-16T03:37:12.486 INFO:tasks.rados.rados.0.smithi157.stdout:25: writing smithi15736696-25 from 1454509 to 2182184 tid 2 2024-09-16T03:37:12.486 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.497 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:12.497 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:12.507 INFO:tasks.rados.rados.0.smithi157.stdout:8: finishing write tid 2 to smithi15736696-8 2024-09-16T03:37:12.515 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:12.516 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:12.527 INFO:tasks.rados.rados.0.smithi157.stdout:7: finishing write tid 1 to smithi15736696-7 2024-09-16T03:37:12.527 INFO:tasks.rados.rados.0.smithi157.stdout:7: finishing write tid 2 to smithi15736696-7 2024-09-16T03:37:12.527 INFO:tasks.rados.rados.0.smithi157.stdout:7: finishing write tid 3 to smithi15736696-7 2024-09-16T03:37:12.527 INFO:tasks.rados.rados.0.smithi157.stdout:7: finishing write tid 4 to smithi15736696-7 2024-09-16T03:37:12.527 INFO:tasks.rados.rados.0.smithi157.stdout:7: oid 7 updating version 0 to 3 2024-09-16T03:37:12.527 INFO:tasks.rados.rados.0.smithi157.stdout:7: oid 7 version 3 is already newer than 1 2024-09-16T03:37:12.527 INFO:tasks.rados.rados.0.smithi157.stdout:7: oid 7 version 3 is already newer than 2 2024-09-16T03:37:12.527 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T03:37:12.527 INFO:tasks.rados.rados.0.smithi157.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-16T03:37:12.527 INFO:tasks.rados.rados.0.smithi157.stdout:13: finishing write tid 1 to smithi15736696-13 2024-09-16T03:37:12.528 INFO:tasks.rados.rados.0.smithi157.stdout:7: done (15 left) 2024-09-16T03:37:12.528 INFO:tasks.rados.rados.0.smithi157.stdout:26: write initial oid 26 2024-09-16T03:37:12.528 INFO:tasks.rados.rados.0.smithi157.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-09-16T03:37:12.531 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:12.532 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:12.532 INFO:tasks.rados.rados.0.smithi157.stdout:26: writing smithi15736696-26 from 420175 to 1050459 tid 1 2024-09-16T03:37:12.536 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:12.534+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:12.539 INFO:tasks.rados.rados.0.smithi157.stdout:26: writing smithi15736696-26 from 1796638 to 2267971 tid 2 2024-09-16T03:37:12.539 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.548 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:12.559 INFO:tasks.rados.rados.0.smithi157.stdout:9: finishing write tid 2 to smithi15736696-9 2024-09-16T03:37:12.559 INFO:tasks.rados.rados.0.smithi157.stdout:9: finishing write tid 3 to smithi15736696-9 2024-09-16T03:37:12.559 INFO:tasks.rados.rados.0.smithi157.stdout:9: finishing write tid 4 to smithi15736696-9 2024-09-16T03:37:12.559 INFO:tasks.rados.rados.0.smithi157.stdout:9: oid 9 updating version 0 to 1 2024-09-16T03:37:12.559 INFO:tasks.rados.rados.0.smithi157.stdout:9: oid 9 updating version 1 to 2 2024-09-16T03:37:12.559 INFO:tasks.rados.rados.0.smithi157.stdout:9: oid 9 updating version 2 to 3 2024-09-16T03:37:12.559 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:37:12.560 INFO:tasks.rados.rados.0.smithi157.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-16T03:37:12.560 INFO:tasks.rados.rados.0.smithi157.stdout:9: done (15 left) 2024-09-16T03:37:12.560 INFO:tasks.rados.rados.0.smithi157.stdout:27: write initial oid 27 2024-09-16T03:37:12.560 INFO:tasks.rados.rados.0.smithi157.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-09-16T03:37:12.565 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:12.565 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:12.566 INFO:tasks.rados.rados.0.smithi157.stdout:27: writing smithi15736696-27 from 436982 to 1142515 tid 1 2024-09-16T03:37:12.569 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:12.569 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:12.573 INFO:tasks.rados.rados.0.smithi157.stdout:27: writing smithi15736696-27 from 1855120 to 2436982 tid 2 2024-09-16T03:37:12.573 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.573 INFO:tasks.rados.rados.0.smithi157.stdout:16: finishing write tid 1 to smithi15736696-16 2024-09-16T03:37:12.573 INFO:tasks.rados.rados.0.smithi157.stdout:16: finishing write tid 2 to smithi15736696-16 2024-09-16T03:37:12.573 INFO:tasks.rados.rados.0.smithi157.stdout:16: finishing write tid 3 to smithi15736696-16 2024-09-16T03:37:12.578 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:37:12.609 INFO:tasks.rados.rados.0.smithi157.stdout:8: finishing write tid 3 to smithi15736696-8 2024-09-16T03:37:12.610 INFO:tasks.rados.rados.0.smithi157.stdout:8: finishing write tid 4 to smithi15736696-8 2024-09-16T03:37:12.610 INFO:tasks.rados.rados.0.smithi157.stdout:8: oid 8 updating version 0 to 2 2024-09-16T03:37:12.610 INFO:tasks.rados.rados.0.smithi157.stdout:8: oid 8 version 2 is already newer than 1 2024-09-16T03:37:12.610 INFO:tasks.rados.rados.0.smithi157.stdout:8: oid 8 updating version 2 to 3 2024-09-16T03:37:12.610 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:37:12.610 INFO:tasks.rados.rados.0.smithi157.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:37:12.610 INFO:tasks.rados.rados.0.smithi157.stdout:10: finishing write tid 1 to smithi15736696-10 2024-09-16T03:37:12.610 INFO:tasks.rados.rados.0.smithi157.stdout:8: done (15 left) 2024-09-16T03:37:12.610 INFO:tasks.rados.rados.0.smithi157.stdout:28: write initial oid 28 2024-09-16T03:37:12.610 INFO:tasks.rados.rados.0.smithi157.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-09-16T03:37:12.616 INFO:tasks.rados.rados.0.smithi157.stdout:28: writing smithi15736696-28 from 453789 to 1234571 tid 1 2024-09-16T03:37:12.622 INFO:tasks.rados.rados.0.smithi157.stdout:28: writing smithi15736696-28 from 1913602 to 2388604 tid 2 2024-09-16T03:37:12.622 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.622 INFO:tasks.rados.rados.0.smithi157.stdout:14: finishing write tid 1 to smithi15736696-14 2024-09-16T03:37:12.622 INFO:tasks.rados.rados.0.smithi157.stdout:10: finishing write tid 2 to smithi15736696-10 2024-09-16T03:37:12.622 INFO:tasks.rados.rados.0.smithi157.stdout:14: finishing write tid 2 to smithi15736696-14 2024-09-16T03:37:12.622 INFO:tasks.rados.rados.0.smithi157.stdout:10: finishing write tid 3 to smithi15736696-10 2024-09-16T03:37:12.622 INFO:tasks.rados.rados.0.smithi157.stdout:14: finishing write tid 3 to smithi15736696-14 2024-09-16T03:37:12.622 INFO:tasks.rados.rados.0.smithi157.stdout:10: finishing write tid 4 to smithi15736696-10 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:10: oid 10 updating version 0 to 5 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:10: oid 10 version 5 is already newer than 4 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:10: oid 10 updating version 5 to 6 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:14: finishing write tid 4 to smithi15736696-14 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:14: oid 14 updating version 0 to 1 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:14: oid 14 updating version 1 to 3 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:14: oid 14 version 3 is already newer than 2 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:10: done (15 left) 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:14: done (14 left) 2024-09-16T03:37:12.623 INFO:tasks.rados.rados.0.smithi157.stdout:29: write initial oid 29 2024-09-16T03:37:12.624 INFO:tasks.rados.rados.0.smithi157.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-09-16T03:37:12.627 INFO:tasks.rados.rados.0.smithi157.stdout:29: writing smithi15736696-29 from 470596 to 926627 tid 1 2024-09-16T03:37:12.634 INFO:tasks.rados.rados.0.smithi157.stdout:29: writing smithi15736696-29 from 1572084 to 2190744 tid 2 2024-09-16T03:37:12.634 INFO:tasks.rados.rados.0.smithi157.stdout:30: write initial oid 30 2024-09-16T03:37:12.634 INFO:tasks.rados.rados.0.smithi157.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-09-16T03:37:12.637 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:12.636+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:12.639 INFO:tasks.rados.rados.0.smithi157.stdout:30: writing smithi15736696-30 from 487403 to 1018683 tid 1 2024-09-16T03:37:12.643 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:12.643 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:12.644 INFO:tasks.rados.rados.0.smithi157.stdout:30: writing smithi15736696-30 from 1514213 to 1992884 tid 2 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout:23: finishing write tid 1 to smithi15736696-23 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout:13: finishing write tid 2 to smithi15736696-13 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout:13: finishing write tid 3 to smithi15736696-13 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout:13: finishing write tid 4 to smithi15736696-13 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout:13: oid 13 updating version 0 to 4 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout:13: oid 13 updating version 4 to 5 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout:13: oid 13 updating version 5 to 6 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout:20: finishing write tid 1 to smithi15736696-20 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout:20: finishing write tid 2 to smithi15736696-20 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout:13: done (15 left) 2024-09-16T03:37:12.645 INFO:tasks.rados.rados.0.smithi157.stdout:31: write initial oid 31 2024-09-16T03:37:12.646 INFO:tasks.rados.rados.0.smithi157.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-09-16T03:37:12.650 INFO:tasks.rados.rados.0.smithi157.stdout:31: writing smithi15736696-31 from 504210 to 1110739 tid 1 2024-09-16T03:37:12.657 INFO:tasks.rados.rados.0.smithi157.stdout:31: writing smithi15736696-31 from 1572695 to 2195024 tid 2 2024-09-16T03:37:12.657 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.661 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:12.661 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:12.675 INFO:tasks.rados.rados.0.smithi157.stdout:19: finishing write tid 1 to smithi15736696-19 2024-09-16T03:37:12.679 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:12.679 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:12.686 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:12.686 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:12.694 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:37:12.695 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:12.695 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:12.703 INFO:tasks.rados.rados.0.smithi157.stdout:21: finishing write tid 1 to smithi15736696-21 2024-09-16T03:37:12.703 INFO:tasks.rados.rados.0.smithi157.stdout:21: finishing write tid 2 to smithi15736696-21 2024-09-16T03:37:12.704 INFO:tasks.rados.rados.0.smithi157.stdout:21: finishing write tid 3 to smithi15736696-21 2024-09-16T03:37:12.704 INFO:tasks.rados.rados.0.smithi157.stdout:21: finishing write tid 4 to smithi15736696-21 2024-09-16T03:37:12.704 INFO:tasks.rados.rados.0.smithi157.stdout:21: oid 21 updating version 0 to 2 2024-09-16T03:37:12.704 INFO:tasks.rados.rados.0.smithi157.stdout:21: oid 21 updating version 2 to 3 2024-09-16T03:37:12.704 INFO:tasks.rados.rados.0.smithi157.stdout:21: oid 21 version 3 is already newer than 1 2024-09-16T03:37:12.704 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T03:37:12.704 INFO:tasks.rados.rados.0.smithi157.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-16T03:37:12.704 INFO:tasks.rados.rados.0.smithi157.stdout:21: done (15 left) 2024-09-16T03:37:12.704 INFO:tasks.rados.rados.0.smithi157.stdout:32: write initial oid 32 2024-09-16T03:37:12.704 INFO:tasks.rados.rados.0.smithi157.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-09-16T03:37:12.709 INFO:tasks.rados.rados.0.smithi157.stdout:32: writing smithi15736696-32 from 521017 to 1319148 tid 1 2024-09-16T03:37:12.715 INFO:tasks.rados.rados.0.smithi157.stdout:32: writing smithi15736696-32 from 1747530 to 2229870 tid 2 2024-09-16T03:37:12.715 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.715 INFO:tasks.rados.rados.0.smithi157.stdout:17: finishing write tid 2 to smithi15736696-17 2024-09-16T03:37:12.715 INFO:tasks.rados.rados.0.smithi157.stdout:17: finishing write tid 3 to smithi15736696-17 2024-09-16T03:37:12.715 INFO:tasks.rados.rados.0.smithi157.stdout:17: finishing write tid 4 to smithi15736696-17 2024-09-16T03:37:12.715 INFO:tasks.rados.rados.0.smithi157.stdout:17: oid 17 updating version 0 to 1 2024-09-16T03:37:12.716 INFO:tasks.rados.rados.0.smithi157.stdout:17: oid 17 updating version 1 to 2 2024-09-16T03:37:12.716 INFO:tasks.rados.rados.0.smithi157.stdout:17: oid 17 updating version 2 to 3 2024-09-16T03:37:12.716 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T03:37:12.716 INFO:tasks.rados.rados.0.smithi157.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-16T03:37:12.716 INFO:tasks.rados.rados.0.smithi157.stdout:17: done (15 left) 2024-09-16T03:37:12.716 INFO:tasks.rados.rados.0.smithi157.stdout:33: write initial oid 33 2024-09-16T03:37:12.716 INFO:tasks.rados.rados.0.smithi157.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-09-16T03:37:12.720 INFO:tasks.rados.rados.0.smithi157.stdout:33: writing smithi15736696-33 from 537824 to 1011204 tid 1 2024-09-16T03:37:12.728 INFO:tasks.rados.rados.0.smithi157.stdout:33: writing smithi15736696-33 from 1806012 to 2432010 tid 2 2024-09-16T03:37:12.728 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.728 INFO:tasks.rados.rados.0.smithi157.stdout:22: finishing write tid 1 to smithi15736696-22 2024-09-16T03:37:12.728 INFO:tasks.rados.rados.0.smithi157.stdout:19: finishing write tid 2 to smithi15736696-19 2024-09-16T03:37:12.729 INFO:tasks.rados.rados.0.smithi157.stdout:19: finishing write tid 3 to smithi15736696-19 2024-09-16T03:37:12.729 INFO:tasks.rados.rados.0.smithi157.stdout:28: finishing write tid 1 to smithi15736696-28 2024-09-16T03:37:12.729 INFO:tasks.rados.rados.0.smithi157.stdout:28: finishing write tid 2 to smithi15736696-28 2024-09-16T03:37:12.738 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:12.736+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:12.742 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:12.742 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:12.745 INFO:tasks.rados.rados.0.smithi157.stdout:23: finishing write tid 2 to smithi15736696-23 2024-09-16T03:37:12.745 INFO:tasks.rados.rados.0.smithi157.stdout:23: finishing write tid 3 to smithi15736696-23 2024-09-16T03:37:12.745 INFO:tasks.rados.rados.0.smithi157.stdout:23: finishing write tid 4 to smithi15736696-23 2024-09-16T03:37:12.746 INFO:tasks.rados.rados.0.smithi157.stdout:23: oid 23 updating version 0 to 8 2024-09-16T03:37:12.746 INFO:tasks.rados.rados.0.smithi157.stdout:23: oid 23 updating version 8 to 9 2024-09-16T03:37:12.746 INFO:tasks.rados.rados.0.smithi157.stdout:23: oid 23 version 9 is already newer than 7 2024-09-16T03:37:12.746 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T03:37:12.746 INFO:tasks.rados.rados.0.smithi157.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-16T03:37:12.746 INFO:tasks.rados.rados.0.smithi157.stdout:20: finishing write tid 3 to smithi15736696-20 2024-09-16T03:37:12.746 INFO:tasks.rados.rados.0.smithi157.stdout:20: finishing write tid 4 to smithi15736696-20 2024-09-16T03:37:12.746 INFO:tasks.rados.rados.0.smithi157.stdout:20: oid 20 updating version 0 to 8 2024-09-16T03:37:12.746 INFO:tasks.rados.rados.0.smithi157.stdout:20: oid 20 version 8 is already newer than 7 2024-09-16T03:37:12.746 INFO:tasks.rados.rados.0.smithi157.stdout:20: oid 20 updating version 8 to 9 2024-09-16T03:37:12.746 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 9 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T03:37:12.746 INFO:tasks.rados.rados.0.smithi157.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-16T03:37:12.746 INFO:tasks.rados.rados.0.smithi157.stdout:20: done (15 left) 2024-09-16T03:37:12.747 INFO:tasks.rados.rados.0.smithi157.stdout:23: done (14 left) 2024-09-16T03:37:12.747 INFO:tasks.rados.rados.0.smithi157.stdout:34: write initial oid 34 2024-09-16T03:37:12.747 INFO:tasks.rados.rados.0.smithi157.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-09-16T03:37:12.751 INFO:tasks.rados.rados.0.smithi157.stdout:34: writing smithi15736696-34 from 554631 to 1103260 tid 1 2024-09-16T03:37:12.757 INFO:tasks.rados.rados.0.smithi157.stdout:34: writing smithi15736696-34 from 1748141 to 2234150 tid 2 2024-09-16T03:37:12.757 INFO:tasks.rados.rados.0.smithi157.stdout:35: write initial oid 35 2024-09-16T03:37:12.757 INFO:tasks.rados.rados.0.smithi157.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-09-16T03:37:12.764 INFO:tasks.rados.rados.0.smithi157.stdout:35: writing smithi15736696-35 from 571438 to 1195316 tid 1 2024-09-16T03:37:12.771 INFO:tasks.rados.rados.0.smithi157.stdout:35: writing smithi15736696-35 from 1806623 to 2436290 tid 2 2024-09-16T03:37:12.771 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.771 INFO:tasks.rados.rados.0.smithi157.stdout:31: finishing write tid 1 to smithi15736696-31 2024-09-16T03:37:12.772 INFO:tasks.rados.rados.0.smithi157.stdout:16: finishing write tid 4 to smithi15736696-16 2024-09-16T03:37:12.772 INFO:tasks.rados.rados.0.smithi157.stdout:16: oid 16 updating version 0 to 5 2024-09-16T03:37:12.772 INFO:tasks.rados.rados.0.smithi157.stdout:16: oid 16 version 5 is already newer than 4 2024-09-16T03:37:12.772 INFO:tasks.rados.rados.0.smithi157.stdout:16: oid 16 updating version 5 to 6 2024-09-16T03:37:12.772 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:37:12.772 INFO:tasks.rados.rados.0.smithi157.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-16T03:37:12.772 INFO:tasks.rados.rados.0.smithi157.stdout:16: done (15 left) 2024-09-16T03:37:12.772 INFO:tasks.rados.rados.0.smithi157.stdout:36: write initial oid 36 2024-09-16T03:37:12.773 INFO:tasks.rados.rados.0.smithi157.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-09-16T03:37:12.778 INFO:tasks.rados.rados.0.smithi157.stdout:36: writing smithi15736696-36 from 588245 to 1287372 tid 1 2024-09-16T03:37:12.784 INFO:tasks.rados.rados.0.smithi157.stdout:36: writing smithi15736696-36 from 1865105 to 2354783 tid 2 2024-09-16T03:37:12.784 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.810 INFO:tasks.rados.rados.0.smithi157.stdout:18: finishing write tid 1 to smithi15736696-18 2024-09-16T03:37:12.817 INFO:tasks.rados.rados.0.smithi157.stdout:19: finishing write tid 4 to smithi15736696-19 2024-09-16T03:37:12.818 INFO:tasks.rados.rados.0.smithi157.stdout:19: oid 19 updating version 0 to 4 2024-09-16T03:37:12.818 INFO:tasks.rados.rados.0.smithi157.stdout:19: oid 19 updating version 4 to 6 2024-09-16T03:37:12.818 INFO:tasks.rados.rados.0.smithi157.stdout:19: oid 19 version 6 is already newer than 5 2024-09-16T03:37:12.818 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T03:37:12.818 INFO:tasks.rados.rados.0.smithi157.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-16T03:37:12.818 INFO:tasks.rados.rados.0.smithi157.stdout:19: done (15 left) 2024-09-16T03:37:12.818 INFO:tasks.rados.rados.0.smithi157.stdout:37: write initial oid 37 2024-09-16T03:37:12.818 INFO:tasks.rados.rados.0.smithi157.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-09-16T03:37:12.824 INFO:tasks.rados.rados.0.smithi157.stdout:37: writing smithi15736696-37 from 605052 to 1379428 tid 1 2024-09-16T03:37:12.831 INFO:tasks.rados.rados.0.smithi157.stdout:37: writing smithi15736696-37 from 1923587 to 2556923 tid 2 2024-09-16T03:37:12.832 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.832 INFO:tasks.rados.rados.0.smithi157.stdout:24: finishing write tid 1 to smithi15736696-24 2024-09-16T03:37:12.832 INFO:tasks.rados.rados.0.smithi157.stdout:24: finishing write tid 2 to smithi15736696-24 2024-09-16T03:37:12.832 INFO:tasks.rados.rados.0.smithi157.stdout:24: finishing write tid 3 to smithi15736696-24 2024-09-16T03:37:12.832 INFO:tasks.rados.rados.0.smithi157.stdout:24: finishing write tid 4 to smithi15736696-24 2024-09-16T03:37:12.832 INFO:tasks.rados.rados.0.smithi157.stdout:24: oid 24 updating version 0 to 8 2024-09-16T03:37:12.832 INFO:tasks.rados.rados.0.smithi157.stdout:24: oid 24 version 8 is already newer than 7 2024-09-16T03:37:12.832 INFO:tasks.rados.rados.0.smithi157.stdout:24: oid 24 updating version 8 to 9 2024-09-16T03:37:12.832 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T03:37:12.832 INFO:tasks.rados.rados.0.smithi157.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:37:12.832 INFO:tasks.rados.rados.0.smithi157.stdout:27: finishing write tid 1 to smithi15736696-27 2024-09-16T03:37:12.832 INFO:tasks.rados.rados.0.smithi157.stdout:32: finishing write tid 1 to smithi15736696-32 2024-09-16T03:37:12.832 INFO:tasks.rados.rados.0.smithi157.stdout:33: finishing write tid 1 to smithi15736696-33 2024-09-16T03:37:12.833 INFO:tasks.rados.rados.0.smithi157.stdout:24: done (15 left) 2024-09-16T03:37:12.833 INFO:tasks.rados.rados.0.smithi157.stdout:38: write initial oid 38 2024-09-16T03:37:12.833 INFO:tasks.rados.rados.0.smithi157.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-09-16T03:37:12.837 INFO:tasks.rados.rados.0.smithi157.stdout:38: writing smithi15736696-38 from 621859 to 1071484 tid 1 2024-09-16T03:37:12.839 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:12.838+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:12.840 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:12.840 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:12.845 INFO:tasks.rados.rados.0.smithi157.stdout:38: writing smithi15736696-38 from 1865716 to 2621859 tid 2 2024-09-16T03:37:12.845 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.846 INFO:tasks.rados.rados.0.smithi157.stdout:38: finishing write tid 1 to smithi15736696-38 2024-09-16T03:37:12.849 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:37:12.853 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:12.854 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:12.860 INFO:tasks.rados.rados.0.smithi157.stdout:31: finishing write tid 2 to smithi15736696-31 2024-09-16T03:37:12.860 INFO:tasks.rados.rados.0.smithi157.stdout:31: finishing write tid 3 to smithi15736696-31 2024-09-16T03:37:12.860 INFO:tasks.rados.rados.0.smithi157.stdout:31: finishing write tid 4 to smithi15736696-31 2024-09-16T03:37:12.860 INFO:tasks.rados.rados.0.smithi157.stdout:31: oid 31 updating version 0 to 5 2024-09-16T03:37:12.860 INFO:tasks.rados.rados.0.smithi157.stdout:31: oid 31 updating version 5 to 6 2024-09-16T03:37:12.860 INFO:tasks.rados.rados.0.smithi157.stdout:31: oid 31 version 6 is already newer than 4 2024-09-16T03:37:12.860 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:37:12.860 INFO:tasks.rados.rados.0.smithi157.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-16T03:37:12.860 INFO:tasks.rados.rados.0.smithi157.stdout:31: done (15 left) 2024-09-16T03:37:12.861 INFO:tasks.rados.rados.0.smithi157.stdout:39: write initial oid 39 2024-09-16T03:37:12.861 INFO:tasks.rados.rados.0.smithi157.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-09-16T03:37:12.866 INFO:tasks.rados.rados.0.smithi157.stdout:39: writing smithi15736696-39 from 638666 to 1163540 tid 1 2024-09-16T03:37:12.870 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:12.870 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:12.874 INFO:tasks.rados.rados.0.smithi157.stdout:39: writing smithi15736696-39 from 1924198 to 2561203 tid 2 2024-09-16T03:37:12.874 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.886 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:12.887 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:12.905 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:12.906 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:12.913 INFO:tasks.rados.rados.0.smithi157.stdout:18: finishing write tid 2 to smithi15736696-18 2024-09-16T03:37:12.917 INFO:tasks.rados.rados.0.smithi157.stdout:27: finishing write tid 2 to smithi15736696-27 2024-09-16T03:37:12.919 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:12.919 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:12.930 INFO:tasks.rados.rados.0.smithi157.stdout:33: finishing write tid 2 to smithi15736696-33 2024-09-16T03:37:12.930 INFO:tasks.rados.rados.0.smithi157.stdout:33: finishing write tid 3 to smithi15736696-33 2024-09-16T03:37:12.931 INFO:tasks.rados.rados.0.smithi157.stdout:33: finishing write tid 4 to smithi15736696-33 2024-09-16T03:37:12.931 INFO:tasks.rados.rados.0.smithi157.stdout:33: oid 33 updating version 0 to 5 2024-09-16T03:37:12.931 INFO:tasks.rados.rados.0.smithi157.stdout:33: oid 33 updating version 5 to 6 2024-09-16T03:37:12.931 INFO:tasks.rados.rados.0.smithi157.stdout:33: oid 33 version 6 is already newer than 4 2024-09-16T03:37:12.932 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:37:12.932 INFO:tasks.rados.rados.0.smithi157.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-16T03:37:12.932 INFO:tasks.rados.rados.0.smithi157.stdout:33: done (15 left) 2024-09-16T03:37:12.932 INFO:tasks.rados.rados.0.smithi157.stdout:40: write initial oid 40 2024-09-16T03:37:12.932 INFO:tasks.rados.rados.0.smithi157.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-09-16T03:37:12.937 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:12.937 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:12.938 INFO:tasks.rados.rados.0.smithi157.stdout:40: writing smithi15736696-40 from 655473 to 1371949 tid 1 2024-09-16T03:37:12.939 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:12.937+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:12.945 INFO:tasks.rados.rados.0.smithi157.stdout:40: writing smithi15736696-40 from 2099033 to 2655473 tid 2 2024-09-16T03:37:12.945 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.947 INFO:tasks.rados.rados.0.smithi157.stdout:30: finishing write tid 1 to smithi15736696-30 2024-09-16T03:37:12.947 INFO:tasks.rados.rados.0.smithi157.stdout:30: finishing write tid 2 to smithi15736696-30 2024-09-16T03:37:12.960 INFO:tasks.rados.rados.0.smithi157.stdout:28: finishing write tid 3 to smithi15736696-28 2024-09-16T03:37:12.960 INFO:tasks.rados.rados.0.smithi157.stdout:28: finishing write tid 4 to smithi15736696-28 2024-09-16T03:37:12.960 INFO:tasks.rados.rados.0.smithi157.stdout:28: oid 28 updating version 0 to 2 2024-09-16T03:37:12.960 INFO:tasks.rados.rados.0.smithi157.stdout:28: oid 28 version 2 is already newer than 1 2024-09-16T03:37:12.960 INFO:tasks.rados.rados.0.smithi157.stdout:28: oid 28 updating version 2 to 3 2024-09-16T03:37:12.960 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 3 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T03:37:12.960 INFO:tasks.rados.rados.0.smithi157.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-16T03:37:12.960 INFO:tasks.rados.rados.0.smithi157.stdout:28: done (15 left) 2024-09-16T03:37:12.960 INFO:tasks.rados.rados.0.smithi157.stdout:41: write initial oid 41 2024-09-16T03:37:12.961 INFO:tasks.rados.rados.0.smithi157.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-09-16T03:37:12.963 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:37:12.967 INFO:tasks.rados.rados.0.smithi157.stdout:41: writing smithi15736696-41 from 672280 to 1464005 tid 1 2024-09-16T03:37:12.975 INFO:tasks.rados.rados.0.smithi157.stdout:41: writing smithi15736696-41 from 2157515 to 2672280 tid 2 2024-09-16T03:37:12.975 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.975 INFO:tasks.rados.rados.0.smithi157.stdout:35: finishing write tid 1 to smithi15736696-35 2024-09-16T03:37:12.975 INFO:tasks.rados.rados.0.smithi157.stdout:35: finishing write tid 2 to smithi15736696-35 2024-09-16T03:37:12.976 INFO:tasks.rados.rados.0.smithi157.stdout:39: finishing write tid 1 to smithi15736696-39 2024-09-16T03:37:12.976 INFO:tasks.rados.rados.0.smithi157.stdout:39: finishing write tid 2 to smithi15736696-39 2024-09-16T03:37:12.976 INFO:tasks.rados.rados.0.smithi157.stdout:39: finishing write tid 3 to smithi15736696-39 2024-09-16T03:37:12.976 INFO:tasks.rados.rados.0.smithi157.stdout:39: finishing write tid 4 to smithi15736696-39 2024-09-16T03:37:12.976 INFO:tasks.rados.rados.0.smithi157.stdout:39: oid 39 updating version 0 to 8 2024-09-16T03:37:12.976 INFO:tasks.rados.rados.0.smithi157.stdout:39: oid 39 updating version 8 to 9 2024-09-16T03:37:12.976 INFO:tasks.rados.rados.0.smithi157.stdout:39: oid 39 version 9 is already newer than 7 2024-09-16T03:37:12.976 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T03:37:12.976 INFO:tasks.rados.rados.0.smithi157.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-16T03:37:12.976 INFO:tasks.rados.rados.0.smithi157.stdout:39: done (15 left) 2024-09-16T03:37:12.976 INFO:tasks.rados.rados.0.smithi157.stdout:42: write initial oid 42 2024-09-16T03:37:12.976 INFO:tasks.rados.rados.0.smithi157.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-09-16T03:37:12.981 INFO:tasks.rados.rados.0.smithi157.stdout:42: writing smithi15736696-42 from 689087 to 1156061 tid 1 2024-09-16T03:37:12.988 INFO:tasks.rados.rados.0.smithi157.stdout:42: writing smithi15736696-42 from 1699644 to 2483976 tid 2 2024-09-16T03:37:12.988 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:12.998 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:12.998 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:13.007 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:37:13.015 INFO:tasks.rados.rados.0.smithi157.stdout:22: finishing write tid 2 to smithi15736696-22 2024-09-16T03:37:13.015 INFO:tasks.rados.rados.0.smithi157.stdout:22: finishing write tid 3 to smithi15736696-22 2024-09-16T03:37:13.015 INFO:tasks.rados.rados.0.smithi157.stdout:22: finishing write tid 4 to smithi15736696-22 2024-09-16T03:37:13.015 INFO:tasks.rados.rados.0.smithi157.stdout:22: oid 22 updating version 0 to 8 2024-09-16T03:37:13.015 INFO:tasks.rados.rados.0.smithi157.stdout:22: oid 22 updating version 8 to 9 2024-09-16T03:37:13.015 INFO:tasks.rados.rados.0.smithi157.stdout:22: oid 22 version 9 is already newer than 7 2024-09-16T03:37:13.015 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T03:37:13.015 INFO:tasks.rados.rados.0.smithi157.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-16T03:37:13.015 INFO:tasks.rados.rados.0.smithi157.stdout:26: finishing write tid 1 to smithi15736696-26 2024-09-16T03:37:13.016 INFO:tasks.rados.rados.0.smithi157.stdout:26: finishing write tid 2 to smithi15736696-26 2024-09-16T03:37:13.016 INFO:tasks.rados.rados.0.smithi157.stdout:18: finishing write tid 3 to smithi15736696-18 2024-09-16T03:37:13.016 INFO:tasks.rados.rados.0.smithi157.stdout:22: done (15 left) 2024-09-16T03:37:13.016 INFO:tasks.rados.rados.0.smithi157.stdout:43: write initial oid 43 2024-09-16T03:37:13.016 INFO:tasks.rados.rados.0.smithi157.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-09-16T03:37:13.021 INFO:tasks.rados.rados.0.smithi157.stdout:43: writing smithi15736696-43 from 705894 to 1248117 tid 1 2024-09-16T03:37:13.028 INFO:tasks.rados.rados.0.smithi157.stdout:43: writing smithi15736696-43 from 1758126 to 2402469 tid 2 2024-09-16T03:37:13.028 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.028 INFO:tasks.rados.rados.0.smithi157.stdout:18: finishing write tid 4 to smithi15736696-18 2024-09-16T03:37:13.028 INFO:tasks.rados.rados.0.smithi157.stdout:18: oid 18 updating version 0 to 4 2024-09-16T03:37:13.028 INFO:tasks.rados.rados.0.smithi157.stdout:18: oid 18 updating version 4 to 5 2024-09-16T03:37:13.028 INFO:tasks.rados.rados.0.smithi157.stdout:18: oid 18 updating version 5 to 6 2024-09-16T03:37:13.028 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T03:37:13.029 INFO:tasks.rados.rados.0.smithi157.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-16T03:37:13.029 INFO:tasks.rados.rados.0.smithi157.stdout:18: done (15 left) 2024-09-16T03:37:13.029 INFO:tasks.rados.rados.0.smithi157.stdout:44: write initial oid 44 2024-09-16T03:37:13.029 INFO:tasks.rados.rados.0.smithi157.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-09-16T03:37:13.035 INFO:tasks.rados.rados.0.smithi157.stdout:44: writing smithi15736696-44 from 722701 to 1340173 tid 1 2024-09-16T03:37:13.040 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:13.037+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:13.043 INFO:tasks.rados.rados.0.smithi157.stdout:44: writing smithi15736696-44 from 1816608 to 2604609 tid 2 2024-09-16T03:37:13.043 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.043 INFO:tasks.rados.rados.0.smithi157.stdout:25: finishing write tid 1 to smithi15736696-25 2024-09-16T03:37:13.043 INFO:tasks.rados.rados.0.smithi157.stdout:27: finishing write tid 3 to smithi15736696-27 2024-09-16T03:37:13.043 INFO:tasks.rados.rados.0.smithi157.stdout:27: finishing write tid 4 to smithi15736696-27 2024-09-16T03:37:13.043 INFO:tasks.rados.rados.0.smithi157.stdout:27: oid 27 updating version 0 to 11 2024-09-16T03:37:13.043 INFO:tasks.rados.rados.0.smithi157.stdout:27: oid 27 version 11 is already newer than 10 2024-09-16T03:37:13.043 INFO:tasks.rados.rados.0.smithi157.stdout:27: oid 27 updating version 11 to 12 2024-09-16T03:37:13.043 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:13.043 INFO:tasks.rados.rados.0.smithi157.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:13.043 INFO:tasks.rados.rados.0.smithi157.stdout:27: done (15 left) 2024-09-16T03:37:13.044 INFO:tasks.rados.rados.0.smithi157.stdout:45: write initial oid 45 2024-09-16T03:37:13.044 INFO:tasks.rados.rados.0.smithi157.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-09-16T03:37:13.050 INFO:tasks.rados.rados.0.smithi157.stdout:45: writing smithi15736696-45 from 739508 to 1432229 tid 1 2024-09-16T03:37:13.057 INFO:tasks.rados.rados.0.smithi157.stdout:45: writing smithi15736696-45 from 1875090 to 2523102 tid 2 2024-09-16T03:37:13.057 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.057 INFO:tasks.rados.rados.0.smithi157.stdout:30: finishing write tid 3 to smithi15736696-30 2024-09-16T03:37:13.057 INFO:tasks.rados.rados.0.smithi157.stdout:30: finishing write tid 4 to smithi15736696-30 2024-09-16T03:37:13.057 INFO:tasks.rados.rados.0.smithi157.stdout:30: oid 30 updating version 0 to 10 2024-09-16T03:37:13.057 INFO:tasks.rados.rados.0.smithi157.stdout:30: oid 30 updating version 10 to 11 2024-09-16T03:37:13.057 INFO:tasks.rados.rados.0.smithi157.stdout:30: oid 30 updating version 11 to 12 2024-09-16T03:37:13.057 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:37:13.058 INFO:tasks.rados.rados.0.smithi157.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-16T03:37:13.058 INFO:tasks.rados.rados.0.smithi157.stdout:30: done (15 left) 2024-09-16T03:37:13.058 INFO:tasks.rados.rados.0.smithi157.stdout:46: write initial oid 46 2024-09-16T03:37:13.058 INFO:tasks.rados.rados.0.smithi157.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-09-16T03:37:13.065 INFO:tasks.rados.rados.0.smithi157.stdout:46: writing smithi15736696-46 from 756315 to 1524285 tid 1 2024-09-16T03:37:13.071 INFO:tasks.rados.rados.0.smithi157.stdout:46: writing smithi15736696-46 from 1933572 to 2725242 tid 2 2024-09-16T03:37:13.071 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.084 INFO:tasks.rados.rados.0.smithi157.stdout:32: finishing write tid 2 to smithi15736696-32 2024-09-16T03:37:13.084 INFO:tasks.rados.rados.0.smithi157.stdout:32: finishing write tid 3 to smithi15736696-32 2024-09-16T03:37:13.084 INFO:tasks.rados.rados.0.smithi157.stdout:32: finishing write tid 4 to smithi15736696-32 2024-09-16T03:37:13.084 INFO:tasks.rados.rados.0.smithi157.stdout:32: oid 32 updating version 0 to 6 2024-09-16T03:37:13.084 INFO:tasks.rados.rados.0.smithi157.stdout:32: oid 32 version 6 is already newer than 4 2024-09-16T03:37:13.084 INFO:tasks.rados.rados.0.smithi157.stdout:32: oid 32 version 6 is already newer than 5 2024-09-16T03:37:13.084 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:37:13.084 INFO:tasks.rados.rados.0.smithi157.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-16T03:37:13.085 INFO:tasks.rados.rados.0.smithi157.stdout:38: finishing write tid 2 to smithi15736696-38 2024-09-16T03:37:13.085 INFO:tasks.rados.rados.0.smithi157.stdout:38: finishing write tid 3 to smithi15736696-38 2024-09-16T03:37:13.085 INFO:tasks.rados.rados.0.smithi157.stdout:38: finishing write tid 4 to smithi15736696-38 2024-09-16T03:37:13.085 INFO:tasks.rados.rados.0.smithi157.stdout:38: oid 38 updating version 0 to 4 2024-09-16T03:37:13.085 INFO:tasks.rados.rados.0.smithi157.stdout:38: oid 38 updating version 4 to 5 2024-09-16T03:37:13.085 INFO:tasks.rados.rados.0.smithi157.stdout:38: oid 38 updating version 5 to 6 2024-09-16T03:37:13.085 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:37:13.085 INFO:tasks.rados.rados.0.smithi157.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-16T03:37:13.085 INFO:tasks.rados.rados.0.smithi157.stdout:32: done (15 left) 2024-09-16T03:37:13.085 INFO:tasks.rados.rados.0.smithi157.stdout:38: done (14 left) 2024-09-16T03:37:13.085 INFO:tasks.rados.rados.0.smithi157.stdout:47: write initial oid 47 2024-09-16T03:37:13.085 INFO:tasks.rados.rados.0.smithi157.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-09-16T03:37:13.091 INFO:tasks.rados.rados.0.smithi157.stdout:47: writing smithi15736696-47 from 773122 to 1332694 tid 1 2024-09-16T03:37:13.098 INFO:tasks.rados.rados.0.smithi157.stdout:47: writing smithi15736696-47 from 1992054 to 2643735 tid 2 2024-09-16T03:37:13.098 INFO:tasks.rados.rados.0.smithi157.stdout:48: write initial oid 48 2024-09-16T03:37:13.098 INFO:tasks.rados.rados.0.smithi157.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-09-16T03:37:13.105 INFO:tasks.rados.rados.0.smithi157.stdout:48: writing smithi15736696-48 from 789929 to 1424750 tid 1 2024-09-16T03:37:13.113 INFO:tasks.rados.rados.0.smithi157.stdout:48: writing smithi15736696-48 from 2050536 to 2789929 tid 2 2024-09-16T03:37:13.113 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.113 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:13.113 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:26: finishing write tid 3 to smithi15736696-26 2024-09-16T03:37:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:26: finishing write tid 4 to smithi15736696-26 2024-09-16T03:37:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:26: oid 26 updating version 0 to 12 2024-09-16T03:37:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:26: oid 26 version 12 is already newer than 11 2024-09-16T03:37:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:26: oid 26 version 12 is already newer than 10 2024-09-16T03:37:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 12 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:37:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-16T03:37:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:26: done (15 left) 2024-09-16T03:37:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:49: write initial oid 49 2024-09-16T03:37:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-09-16T03:37:13.123 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:37:13.124 INFO:tasks.rados.rados.0.smithi157.stdout:49: writing smithi15736696-49 from 406736 to 1116806 tid 1 2024-09-16T03:37:13.130 INFO:tasks.rados.rados.0.smithi157.stdout:49: writing smithi15736696-49 from 1709018 to 2364368 tid 2 2024-09-16T03:37:13.131 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.137 INFO:tasks.rados.rados.0.smithi157.stdout:34: finishing write tid 1 to smithi15736696-34 2024-09-16T03:37:13.141 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:13.140+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:13.150 INFO:tasks.rados.rados.0.smithi157.stdout:42: finishing write tid 1 to smithi15736696-42 2024-09-16T03:37:13.229 INFO:tasks.rados.rados.0.smithi157.stdout:35: finishing write tid 3 to smithi15736696-35 2024-09-16T03:37:13.230 INFO:tasks.rados.rados.0.smithi157.stdout:35: finishing write tid 4 to smithi15736696-35 2024-09-16T03:37:13.230 INFO:tasks.rados.rados.0.smithi157.stdout:35: oid 35 updating version 0 to 5 2024-09-16T03:37:13.230 INFO:tasks.rados.rados.0.smithi157.stdout:35: oid 35 version 5 is already newer than 4 2024-09-16T03:37:13.230 INFO:tasks.rados.rados.0.smithi157.stdout:35: oid 35 updating version 5 to 6 2024-09-16T03:37:13.230 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:37:13.230 INFO:tasks.rados.rados.0.smithi157.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:37:13.230 INFO:tasks.rados.rados.0.smithi157.stdout:35: done (15 left) 2024-09-16T03:37:13.230 INFO:tasks.rados.rados.0.smithi157.stdout:50: write initial oid 50 2024-09-16T03:37:13.230 INFO:tasks.rados.rados.0.smithi157.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-09-16T03:37:13.231 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:13.231 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:13.236 INFO:tasks.rados.rados.0.smithi157.stdout:50: writing smithi15736696-50 from 423543 to 1208862 tid 1 2024-09-16T03:37:13.240 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:37:13.241 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:13.239+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:13.244 INFO:tasks.rados.rados.0.smithi157.stdout:50: writing smithi15736696-50 from 1767500 to 2566508 tid 2 2024-09-16T03:37:13.244 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.244 INFO:tasks.rados.rados.0.smithi157.stdout:46: finishing write tid 1 to smithi15736696-46 2024-09-16T03:37:13.254 INFO:tasks.rados.rados.0.smithi157.stdout:42: finishing write tid 2 to smithi15736696-42 2024-09-16T03:37:13.254 INFO:tasks.rados.rados.0.smithi157.stdout:42: finishing write tid 3 to smithi15736696-42 2024-09-16T03:37:13.321 INFO:tasks.rados.rados.0.smithi157.stdout:43: finishing write tid 1 to smithi15736696-43 2024-09-16T03:37:13.321 INFO:tasks.rados.rados.0.smithi157.stdout:43: finishing write tid 2 to smithi15736696-43 2024-09-16T03:37:13.342 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:13.340+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:13.346 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:13.346 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:13.348 INFO:tasks.rados.rados.0.smithi157.stdout:44: finishing write tid 1 to smithi15736696-44 2024-09-16T03:37:13.348 INFO:tasks.rados.rados.0.smithi157.stdout:44: finishing write tid 2 to smithi15736696-44 2024-09-16T03:37:13.348 INFO:tasks.rados.rados.0.smithi157.stdout:44: finishing write tid 3 to smithi15736696-44 2024-09-16T03:37:13.355 INFO:tasks.rados.rados.0.smithi157.stdout:42: finishing write tid 4 to smithi15736696-42 2024-09-16T03:37:13.355 INFO:tasks.rados.rados.0.smithi157.stdout:42: oid 42 updating version 0 to 7 2024-09-16T03:37:13.355 INFO:tasks.rados.rados.0.smithi157.stdout:42: oid 42 updating version 7 to 8 2024-09-16T03:37:13.355 INFO:tasks.rados.rados.0.smithi157.stdout:42: oid 42 updating version 8 to 9 2024-09-16T03:37:13.355 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T03:37:13.355 INFO:tasks.rados.rados.0.smithi157.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-16T03:37:13.355 INFO:tasks.rados.rados.0.smithi157.stdout:42: done (15 left) 2024-09-16T03:37:13.355 INFO:tasks.rados.rados.0.smithi157.stdout:51: copy oid 2 target oid 2 low_tier 2024-09-16T03:37:13.355 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 2 to 2 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.355 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.356 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:37:13.366 INFO:tasks.rados.rados.0.smithi157.stdout:51: finishing copy op to oid 2 low_tier 2024-09-16T03:37:13.366 INFO:tasks.rados.rados.0.smithi157.stdout:51: done (15 left) 2024-09-16T03:37:13.366 INFO:tasks.rados.rados.0.smithi157.stdout:52: copy oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.366 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 3 to 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.366 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:43: finishing write tid 3 to smithi15736696-43 2024-09-16T03:37:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:34: finishing write tid 2 to smithi15736696-34 2024-09-16T03:37:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:43: finishing write tid 4 to smithi15736696-43 2024-09-16T03:37:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:43: oid 43 updating version 0 to 10 2024-09-16T03:37:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:43: oid 43 updating version 10 to 11 2024-09-16T03:37:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:43: oid 43 updating version 11 to 12 2024-09-16T03:37:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 12 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T03:37:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-16T03:37:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:43: done (15 left) 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:53: copy oid 4 target oid 4 low_tier 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 4 to 4 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:37: finishing write tid 1 to smithi15736696-37 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:37: finishing write tid 2 to smithi15736696-37 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:25: finishing write tid 2 to smithi15736696-25 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:25: finishing write tid 3 to smithi15736696-25 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:25: finishing write tid 4 to smithi15736696-25 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:25: oid 25 updating version 0 to 9 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:25: oid 25 version 9 is already newer than 7 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:25: oid 25 version 9 is already newer than 8 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-16T03:37:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:25: done (15 left) 2024-09-16T03:37:13.425 INFO:tasks.rados.rados.0.smithi157.stdout:54: copy oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.425 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 5 to 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.425 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.425 INFO:tasks.rados.rados.0.smithi157.stdout:34: finishing write tid 3 to smithi15736696-34 2024-09-16T03:37:13.426 INFO:tasks.rados.rados.0.smithi157.stdout:34: finishing write tid 4 to smithi15736696-34 2024-09-16T03:37:13.426 INFO:tasks.rados.rados.0.smithi157.stdout:34: oid 34 updating version 0 to 15 2024-09-16T03:37:13.426 INFO:tasks.rados.rados.0.smithi157.stdout:34: oid 34 version 15 is already newer than 14 2024-09-16T03:37:13.426 INFO:tasks.rados.rados.0.smithi157.stdout:34: oid 34 version 15 is already newer than 13 2024-09-16T03:37:13.426 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 15 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T03:37:13.426 INFO:tasks.rados.rados.0.smithi157.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-16T03:37:13.426 INFO:tasks.rados.rados.0.smithi157.stdout:34: done (15 left) 2024-09-16T03:37:13.426 INFO:tasks.rados.rados.0.smithi157.stdout:55: copy oid 6 target oid 6 low_tier 2024-09-16T03:37:13.426 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 6 to 6 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.426 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.443 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:13.442+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:13.449 INFO:tasks.rados.rados.0.smithi157.stdout:44: finishing write tid 4 to smithi15736696-44 2024-09-16T03:37:13.449 INFO:tasks.rados.rados.0.smithi157.stdout:44: oid 44 updating version 0 to 9 2024-09-16T03:37:13.449 INFO:tasks.rados.rados.0.smithi157.stdout:44: oid 44 version 9 is already newer than 7 2024-09-16T03:37:13.449 INFO:tasks.rados.rados.0.smithi157.stdout:44: oid 44 version 9 is already newer than 8 2024-09-16T03:37:13.449 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T03:37:13.449 INFO:tasks.rados.rados.0.smithi157.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-16T03:37:13.449 INFO:tasks.rados.rados.0.smithi157.stdout:44: done (15 left) 2024-09-16T03:37:13.449 INFO:tasks.rados.rados.0.smithi157.stdout:56: copy oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.449 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 7 to 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.449 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.464 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:13.464 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:13.473 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:37:13.518 INFO:tasks.rados.rados.0.smithi157.stdout:48: finishing write tid 1 to smithi15736696-48 2024-09-16T03:37:13.518 INFO:tasks.rados.rados.0.smithi157.stdout:48: finishing write tid 2 to smithi15736696-48 2024-09-16T03:37:13.522 INFO:tasks.rados.rados.0.smithi157.stdout:36: finishing write tid 1 to smithi15736696-36 2024-09-16T03:37:13.522 INFO:tasks.rados.rados.0.smithi157.stdout:36: finishing write tid 2 to smithi15736696-36 2024-09-16T03:37:13.525 INFO:tasks.rados.rados.0.smithi157.stdout:37: finishing write tid 3 to smithi15736696-37 2024-09-16T03:37:13.525 INFO:tasks.rados.rados.0.smithi157.stdout:45: finishing write tid 1 to smithi15736696-45 2024-09-16T03:37:13.525 INFO:tasks.rados.rados.0.smithi157.stdout:37: finishing write tid 4 to smithi15736696-37 2024-09-16T03:37:13.525 INFO:tasks.rados.rados.0.smithi157.stdout:37: oid 37 updating version 0 to 5 2024-09-16T03:37:13.525 INFO:tasks.rados.rados.0.smithi157.stdout:37: oid 37 version 5 is already newer than 4 2024-09-16T03:37:13.525 INFO:tasks.rados.rados.0.smithi157.stdout:37: oid 37 updating version 5 to 6 2024-09-16T03:37:13.525 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T03:37:13.525 INFO:tasks.rados.rados.0.smithi157.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-16T03:37:13.525 INFO:tasks.rados.rados.0.smithi157.stdout:29: finishing write tid 1 to smithi15736696-29 2024-09-16T03:37:13.525 INFO:tasks.rados.rados.0.smithi157.stdout:29: finishing write tid 2 to smithi15736696-29 2024-09-16T03:37:13.525 INFO:tasks.rados.rados.0.smithi157.stdout:37: done (15 left) 2024-09-16T03:37:13.525 INFO:tasks.rados.rados.0.smithi157.stdout:57: copy oid 8 target oid 8 low_tier 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 8 to 8 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout:29: finishing write tid 3 to smithi15736696-29 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout:29: finishing write tid 4 to smithi15736696-29 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout:29: oid 29 updating version 0 to 12 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout:29: oid 29 version 12 is already newer than 10 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout:29: oid 29 version 12 is already newer than 11 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout:48: finishing write tid 3 to smithi15736696-48 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout:29: done (15 left) 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout:58: copy oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 9 to 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.526 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.527 INFO:tasks.rados.rados.0.smithi157.stdout:36: finishing write tid 3 to smithi15736696-36 2024-09-16T03:37:13.527 INFO:tasks.rados.rados.0.smithi157.stdout:48: finishing write tid 4 to smithi15736696-48 2024-09-16T03:37:13.527 INFO:tasks.rados.rados.0.smithi157.stdout:48: oid 48 updating version 0 to 12 2024-09-16T03:37:13.527 INFO:tasks.rados.rados.0.smithi157.stdout:48: oid 48 version 12 is already newer than 11 2024-09-16T03:37:13.527 INFO:tasks.rados.rados.0.smithi157.stdout:48: oid 48 version 12 is already newer than 10 2024-09-16T03:37:13.527 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:37:13.527 INFO:tasks.rados.rados.0.smithi157.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-16T03:37:13.527 INFO:tasks.rados.rados.0.smithi157.stdout:48: done (15 left) 2024-09-16T03:37:13.527 INFO:tasks.rados.rados.0.smithi157.stdout:59: copy oid 10 target oid 10 low_tier 2024-09-16T03:37:13.527 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 10 to 10 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.527 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.527 INFO:tasks.rados.rados.0.smithi157.stdout:36: finishing write tid 4 to smithi15736696-36 2024-09-16T03:37:13.527 INFO:tasks.rados.rados.0.smithi157.stdout:36: oid 36 updating version 0 to 17 2024-09-16T03:37:13.528 INFO:tasks.rados.rados.0.smithi157.stdout:36: oid 36 updating version 17 to 18 2024-09-16T03:37:13.528 INFO:tasks.rados.rados.0.smithi157.stdout:36: oid 36 version 18 is already newer than 16 2024-09-16T03:37:13.528 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 18 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:37:13.528 INFO:tasks.rados.rados.0.smithi157.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-16T03:37:13.528 INFO:tasks.rados.rados.0.smithi157.stdout:36: done (15 left) 2024-09-16T03:37:13.528 INFO:tasks.rados.rados.0.smithi157.stdout:60: copy oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.528 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 11 to 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.528 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.529 INFO:tasks.rados.rados.0.smithi157.stdout:52: finishing copy op to oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.529 INFO:tasks.rados.rados.0.smithi157.stdout:52: done (15 left) 2024-09-16T03:37:13.529 INFO:tasks.rados.rados.0.smithi157.stdout:61: copy oid 12 target oid 12 low_tier 2024-09-16T03:37:13.529 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 12 to 12 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.529 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.536 INFO:tasks.rados.rados.0.smithi157.stdout:54: finishing copy op to oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.537 INFO:tasks.rados.rados.0.smithi157.stdout:54: done (15 left) 2024-09-16T03:37:13.537 INFO:tasks.rados.rados.0.smithi157.stdout:62: copy oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.537 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 13 to 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.537 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.544 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:13.543+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:13.551 INFO:tasks.rados.rados.0.smithi157.stdout:46: finishing write tid 2 to smithi15736696-46 2024-09-16T03:37:13.551 INFO:tasks.rados.rados.0.smithi157.stdout:46: finishing write tid 3 to smithi15736696-46 2024-09-16T03:37:13.551 INFO:tasks.rados.rados.0.smithi157.stdout:46: finishing write tid 4 to smithi15736696-46 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout:46: oid 46 updating version 0 to 9 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout:46: oid 46 version 9 is already newer than 8 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout:46: oid 46 version 9 is already newer than 7 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout:46: done (15 left) 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout:63: copy oid 14 target oid 14 low_tier 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 14 to 14 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout:47: finishing write tid 1 to smithi15736696-47 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout:53: finishing copy op to oid 4 low_tier 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout:53: done (15 left) 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout:64: copy oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.552 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 15 to 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.553 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.580 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:13.580 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:13.588 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:37:13.591 INFO:tasks.rados.rados.0.smithi157.stdout:41: finishing write tid 1 to smithi15736696-41 2024-09-16T03:37:13.645 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:13.644+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:13.650 INFO:tasks.rados.rados.0.smithi157.stdout:47: finishing write tid 2 to smithi15736696-47 2024-09-16T03:37:13.669 INFO:tasks.rados.rados.0.smithi157.stdout:62: finishing copy op to oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.669 INFO:tasks.rados.rados.0.smithi157.stdout:62: done (15 left) 2024-09-16T03:37:13.669 INFO:tasks.rados.rados.0.smithi157.stdout:65: copy oid 16 target oid 16 low_tier 2024-09-16T03:37:13.669 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 16 to 16 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.669 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.669 INFO:tasks.rados.rados.0.smithi157.stdout:58: finishing copy op to oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.670 INFO:tasks.rados.rados.0.smithi157.stdout:58: done (15 left) 2024-09-16T03:37:13.670 INFO:tasks.rados.rados.0.smithi157.stdout:66: copy oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.670 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 17 to 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.670 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.691 INFO:tasks.rados.rados.0.smithi157.stdout:45: finishing write tid 2 to smithi15736696-45 2024-09-16T03:37:13.692 INFO:tasks.rados.rados.0.smithi157.stdout:41: finishing write tid 2 to smithi15736696-41 2024-09-16T03:37:13.692 INFO:tasks.rados.rados.0.smithi157.stdout:41: finishing write tid 3 to smithi15736696-41 2024-09-16T03:37:13.692 INFO:tasks.rados.rados.0.smithi157.stdout:41: finishing write tid 4 to smithi15736696-41 2024-09-16T03:37:13.692 INFO:tasks.rados.rados.0.smithi157.stdout:41: oid 41 updating version 0 to 8 2024-09-16T03:37:13.692 INFO:tasks.rados.rados.0.smithi157.stdout:41: oid 41 version 8 is already newer than 7 2024-09-16T03:37:13.692 INFO:tasks.rados.rados.0.smithi157.stdout:41: oid 41 updating version 8 to 9 2024-09-16T03:37:13.692 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:37:13.692 INFO:tasks.rados.rados.0.smithi157.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-16T03:37:13.692 INFO:tasks.rados.rados.0.smithi157.stdout:41: done (15 left) 2024-09-16T03:37:13.692 INFO:tasks.rados.rados.0.smithi157.stdout:67: copy oid 18 target oid 18 low_tier 2024-09-16T03:37:13.693 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 18 to 18 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.693 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.697 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:13.697 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:13.706 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:37:13.724 INFO:tasks.rados.rados.0.smithi157.stdout:40: finishing write tid 1 to smithi15736696-40 2024-09-16T03:37:13.724 INFO:tasks.rados.rados.0.smithi157.stdout:40: finishing write tid 2 to smithi15736696-40 2024-09-16T03:37:13.734 INFO:tasks.rados.rados.0.smithi157.stdout:60: finishing copy op to oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.735 INFO:tasks.rados.rados.0.smithi157.stdout:60: done (15 left) 2024-09-16T03:37:13.735 INFO:tasks.rados.rados.0.smithi157.stdout:68: copy oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.735 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 19 to 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.735 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.739 INFO:tasks.rados.rados.0.smithi157.stdout:55: finishing copy op to oid 6 low_tier 2024-09-16T03:37:13.739 INFO:tasks.rados.rados.0.smithi157.stdout:55: done (15 left) 2024-09-16T03:37:13.739 INFO:tasks.rados.rados.0.smithi157.stdout:69: copy oid 20 target oid 20 low_tier 2024-09-16T03:37:13.739 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 20 to 20 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.739 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.739 INFO:tasks.rados.rados.0.smithi157.stdout:56: finishing copy op to oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.739 INFO:tasks.rados.rados.0.smithi157.stdout:56: done (15 left) 2024-09-16T03:37:13.739 INFO:tasks.rados.rados.0.smithi157.stdout:70: copy oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.740 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 21 to 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.740 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.742 INFO:tasks.rados.rados.0.smithi157.stdout:64: finishing copy op to oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.742 INFO:tasks.rados.rados.0.smithi157.stdout:64: done (15 left) 2024-09-16T03:37:13.742 INFO:tasks.rados.rados.0.smithi157.stdout:71: copy oid 22 target oid 22 low_tier 2024-09-16T03:37:13.743 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 22 to 22 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.743 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.746 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:13.744+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:13.751 INFO:tasks.rados.rados.0.smithi157.stdout:47: finishing write tid 3 to smithi15736696-47 2024-09-16T03:37:13.751 INFO:tasks.rados.rados.0.smithi157.stdout:47: finishing write tid 4 to smithi15736696-47 2024-09-16T03:37:13.751 INFO:tasks.rados.rados.0.smithi157.stdout:47: oid 47 updating version 0 to 11 2024-09-16T03:37:13.751 INFO:tasks.rados.rados.0.smithi157.stdout:47: oid 47 version 11 is already newer than 10 2024-09-16T03:37:13.751 INFO:tasks.rados.rados.0.smithi157.stdout:47: oid 47 updating version 11 to 12 2024-09-16T03:37:13.751 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T03:37:13.751 INFO:tasks.rados.rados.0.smithi157.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-16T03:37:13.752 INFO:tasks.rados.rados.0.smithi157.stdout:47: done (15 left) 2024-09-16T03:37:13.752 INFO:tasks.rados.rados.0.smithi157.stdout:72: copy oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.752 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 23 to 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.752 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.796 INFO:tasks.rados.rados.0.smithi157.stdout:45: finishing write tid 3 to smithi15736696-45 2024-09-16T03:37:13.796 INFO:tasks.rados.rados.0.smithi157.stdout:45: finishing write tid 4 to smithi15736696-45 2024-09-16T03:37:13.796 INFO:tasks.rados.rados.0.smithi157.stdout:45: oid 45 updating version 0 to 13 2024-09-16T03:37:13.796 INFO:tasks.rados.rados.0.smithi157.stdout:45: oid 45 updating version 13 to 14 2024-09-16T03:37:13.796 INFO:tasks.rados.rados.0.smithi157.stdout:45: oid 45 updating version 14 to 15 2024-09-16T03:37:13.796 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 15 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T03:37:13.796 INFO:tasks.rados.rados.0.smithi157.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-16T03:37:13.796 INFO:tasks.rados.rados.0.smithi157.stdout:45: done (15 left) 2024-09-16T03:37:13.796 INFO:tasks.rados.rados.0.smithi157.stdout:73: copy oid 24 target oid 24 low_tier 2024-09-16T03:37:13.796 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 24 to 24 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.797 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.816 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:13.816 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:13.824 INFO:tasks.rados.rados.0.smithi157.stdout:40: finishing write tid 3 to smithi15736696-40 2024-09-16T03:37:13.825 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:37:13.826 INFO:tasks.rados.rados.0.smithi157.stdout:40: finishing write tid 4 to smithi15736696-40 2024-09-16T03:37:13.826 INFO:tasks.rados.rados.0.smithi157.stdout:40: oid 40 updating version 0 to 15 2024-09-16T03:37:13.826 INFO:tasks.rados.rados.0.smithi157.stdout:40: oid 40 version 15 is already newer than 13 2024-09-16T03:37:13.826 INFO:tasks.rados.rados.0.smithi157.stdout:40: oid 40 version 15 is already newer than 14 2024-09-16T03:37:13.826 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 15 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T03:37:13.826 INFO:tasks.rados.rados.0.smithi157.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-16T03:37:13.827 INFO:tasks.rados.rados.0.smithi157.stdout:40: done (15 left) 2024-09-16T03:37:13.827 INFO:tasks.rados.rados.0.smithi157.stdout:74: copy oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.827 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 25 to 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.827 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.827 INFO:tasks.rados.rados.0.smithi157.stdout:50: finishing write tid 1 to smithi15736696-50 2024-09-16T03:37:13.827 INFO:tasks.rados.rados.0.smithi157.stdout:50: finishing write tid 2 to smithi15736696-50 2024-09-16T03:37:13.841 INFO:tasks.rados.rados.0.smithi157.stdout:49: finishing write tid 1 to smithi15736696-49 2024-09-16T03:37:13.870 INFO:tasks.rados.rados.0.smithi157.stdout:61: finishing copy op to oid 12 low_tier 2024-09-16T03:37:13.870 INFO:tasks.rados.rados.0.smithi157.stdout:61: done (15 left) 2024-09-16T03:37:13.870 INFO:tasks.rados.rados.0.smithi157.stdout:75: copy oid 26 target oid 26 low_tier 2024-09-16T03:37:13.870 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 26 to 26 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.870 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.871 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:13.867+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:13.877 INFO:tasks.rados.rados.0.smithi157.stdout:57: finishing copy op to oid 8 low_tier 2024-09-16T03:37:13.877 INFO:tasks.rados.rados.0.smithi157.stdout:57: done (15 left) 2024-09-16T03:37:13.877 INFO:tasks.rados.rados.0.smithi157.stdout:76: copy oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.877 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 27 to 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.877 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.942 INFO:tasks.rados.rados.0.smithi157.stdout:69: finishing copy op to oid 20 low_tier 2024-09-16T03:37:13.942 INFO:tasks.rados.rados.0.smithi157.stdout:70: finishing copy op to oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.942 INFO:tasks.rados.rados.0.smithi157.stdout:69: done (15 left) 2024-09-16T03:37:13.942 INFO:tasks.rados.rados.0.smithi157.stdout:70: done (14 left) 2024-09-16T03:37:13.942 INFO:tasks.rados.rados.0.smithi157.stdout:77: copy oid 28 target oid 28 low_tier 2024-09-16T03:37:13.942 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 28 to 28 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.942 INFO:tasks.rados.rados.0.smithi157.stdout:78: copy oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.942 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 29 to 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.943 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.945 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:13.946 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:13.947 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:13.944+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:13.954 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:37:13.958 INFO:tasks.rados.rados.0.smithi157.stdout:63: finishing copy op to oid 14 low_tier 2024-09-16T03:37:13.958 INFO:tasks.rados.rados.0.smithi157.stdout:63: done (15 left) 2024-09-16T03:37:13.959 INFO:tasks.rados.rados.0.smithi157.stdout:79: copy oid 30 target oid 30 low_tier 2024-09-16T03:37:13.959 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 30 to 30 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.959 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.965 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:37:13.965 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:37:13.965 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:37:13.965 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T03:37:13.965 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.000000 2024-09-16T03:37:13.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 0 2024-09-16T03:37:13.982 INFO:tasks.rados.rados.0.smithi157.stdout:65: finishing copy op to oid 16 low_tier 2024-09-16T03:37:13.983 INFO:tasks.rados.rados.0.smithi157.stdout:65: done (15 left) 2024-09-16T03:37:13.983 INFO:tasks.rados.rados.0.smithi157.stdout:80: copy oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.983 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 31 to 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:13.983 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:13.983 INFO:tasks.rados.rados.0.smithi157.stdout:68: finishing copy op to oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:13.983 INFO:tasks.rados.rados.0.smithi157.stdout:68: done (15 left) 2024-09-16T03:37:13.983 INFO:tasks.rados.rados.0.smithi157.stdout:81: copy oid 32 target oid 32 low_tier 2024-09-16T03:37:13.983 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 32 to 32 low_tier tgt_pool_name low_tier 2024-09-16T03:37:13.983 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.048 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:14.045+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:14.054 INFO:tasks.rados.rados.0.smithi157.stdout:66: finishing copy op to oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.054 INFO:tasks.rados.rados.0.smithi157.stdout:66: done (15 left) 2024-09-16T03:37:14.054 INFO:tasks.rados.rados.0.smithi157.stdout:82: copy oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.054 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 33 to 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:14.054 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.068 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:14.068 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:14.071 INFO:tasks.rados.rados.0.smithi157.stdout:50: finishing write tid 3 to smithi15736696-50 2024-09-16T03:37:14.071 INFO:tasks.rados.rados.0.smithi157.stdout:50: finishing write tid 4 to smithi15736696-50 2024-09-16T03:37:14.071 INFO:tasks.rados.rados.0.smithi157.stdout:50: oid 50 updating version 0 to 17 2024-09-16T03:37:14.071 INFO:tasks.rados.rados.0.smithi157.stdout:50: oid 50 version 17 is already newer than 16 2024-09-16T03:37:14.071 INFO:tasks.rados.rados.0.smithi157.stdout:50: oid 50 updating version 17 to 18 2024-09-16T03:37:14.072 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 18 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:37:14.072 INFO:tasks.rados.rados.0.smithi157.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-16T03:37:14.072 INFO:tasks.rados.rados.0.smithi157.stdout:49: finishing write tid 2 to smithi15736696-49 2024-09-16T03:37:14.072 INFO:tasks.rados.rados.0.smithi157.stdout:50: done (15 left) 2024-09-16T03:37:14.072 INFO:tasks.rados.rados.0.smithi157.stdout:83: copy oid 34 target oid 34 low_tier 2024-09-16T03:37:14.072 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 34 to 34 low_tier tgt_pool_name low_tier 2024-09-16T03:37:14.072 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.077 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:37:14.092 INFO:tasks.rados.rados.0.smithi157.stdout:76: finishing copy op to oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.092 INFO:tasks.rados.rados.0.smithi157.stdout:76: done (15 left) 2024-09-16T03:37:14.093 INFO:tasks.rados.rados.0.smithi157.stdout:84: copy oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.093 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 35 to 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:14.093 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.144 INFO:tasks.rados.rados.0.smithi157.stdout:49: finishing write tid 3 to smithi15736696-49 2024-09-16T03:37:14.144 INFO:tasks.rados.rados.0.smithi157.stdout:49: finishing write tid 4 to smithi15736696-49 2024-09-16T03:37:14.144 INFO:tasks.rados.rados.0.smithi157.stdout:49: oid 49 updating version 0 to 11 2024-09-16T03:37:14.144 INFO:tasks.rados.rados.0.smithi157.stdout:49: oid 49 version 11 is already newer than 10 2024-09-16T03:37:14.144 INFO:tasks.rados.rados.0.smithi157.stdout:49: oid 49 updating version 11 to 12 2024-09-16T03:37:14.145 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T03:37:14.145 INFO:tasks.rados.rados.0.smithi157.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-16T03:37:14.145 INFO:tasks.rados.rados.0.smithi157.stdout:49: done (15 left) 2024-09-16T03:37:14.145 INFO:tasks.rados.rados.0.smithi157.stdout:85: copy oid 36 target oid 36 low_tier 2024-09-16T03:37:14.145 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 36 to 36 low_tier tgt_pool_name low_tier 2024-09-16T03:37:14.145 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.149 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:14.147+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:14.155 INFO:tasks.rados.rados.0.smithi157.stdout:77: finishing copy op to oid 28 low_tier 2024-09-16T03:37:14.156 INFO:tasks.rados.rados.0.smithi157.stdout:77: done (15 left) 2024-09-16T03:37:14.156 INFO:tasks.rados.rados.0.smithi157.stdout:86: copy oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.156 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 37 to 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:14.156 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.179 INFO:tasks.rados.rados.0.smithi157.stdout:71: finishing copy op to oid 22 low_tier 2024-09-16T03:37:14.180 INFO:tasks.rados.rados.0.smithi157.stdout:79: finishing copy op to oid 30 low_tier 2024-09-16T03:37:14.180 INFO:tasks.rados.rados.0.smithi157.stdout:71: done (15 left) 2024-09-16T03:37:14.180 INFO:tasks.rados.rados.0.smithi157.stdout:79: done (14 left) 2024-09-16T03:37:14.180 INFO:tasks.rados.rados.0.smithi157.stdout:87: copy oid 38 target oid 38 low_tier 2024-09-16T03:37:14.180 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 38 to 38 low_tier tgt_pool_name low_tier 2024-09-16T03:37:14.180 INFO:tasks.rados.rados.0.smithi157.stdout:88: copy oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.180 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 39 to 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:14.180 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.183 INFO:tasks.rados.rados.0.smithi157.stdout:59: finishing copy op to oid 10 low_tier 2024-09-16T03:37:14.183 INFO:tasks.rados.rados.0.smithi157.stdout:59: done (15 left) 2024-09-16T03:37:14.183 INFO:tasks.rados.rados.0.smithi157.stdout:89: copy oid 40 target oid 40 low_tier 2024-09-16T03:37:14.183 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 40 to 40 low_tier tgt_pool_name low_tier 2024-09-16T03:37:14.183 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.183 INFO:tasks.rados.rados.0.smithi157.stdout:72: finishing copy op to oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.183 INFO:tasks.rados.rados.0.smithi157.stdout:72: done (15 left) 2024-09-16T03:37:14.183 INFO:tasks.rados.rados.0.smithi157.stdout:90: copy oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.183 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 41 to 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:14.184 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.184 INFO:tasks.rados.rados.0.smithi157.stdout:73: finishing copy op to oid 24 low_tier 2024-09-16T03:37:14.184 INFO:tasks.rados.rados.0.smithi157.stdout:73: done (15 left) 2024-09-16T03:37:14.184 INFO:tasks.rados.rados.0.smithi157.stdout:91: copy oid 42 target oid 42 low_tier 2024-09-16T03:37:14.184 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 42 to 42 low_tier tgt_pool_name low_tier 2024-09-16T03:37:14.184 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.201 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:14.201 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:14.210 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:37:14.250 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:14.249+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:14.280 INFO:tasks.rados.rados.0.smithi157.stdout:80: finishing copy op to oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.280 INFO:tasks.rados.rados.0.smithi157.stdout:80: done (15 left) 2024-09-16T03:37:14.280 INFO:tasks.rados.rados.0.smithi157.stdout:92: copy oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.280 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 43 to 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:14.280 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.280 INFO:tasks.rados.rados.0.smithi157.stdout:75: finishing copy op to oid 26 low_tier 2024-09-16T03:37:14.280 INFO:tasks.rados.rados.0.smithi157.stdout:75: done (15 left) 2024-09-16T03:37:14.280 INFO:tasks.rados.rados.0.smithi157.stdout:93: copy oid 44 target oid 44 low_tier 2024-09-16T03:37:14.280 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 44 to 44 low_tier tgt_pool_name low_tier 2024-09-16T03:37:14.281 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.283 INFO:tasks.rados.rados.0.smithi157.stdout:86: finishing copy op to oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.283 INFO:tasks.rados.rados.0.smithi157.stdout:86: done (15 left) 2024-09-16T03:37:14.283 INFO:tasks.rados.rados.0.smithi157.stdout:94: copy oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.284 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 45 to 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:14.284 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.284 INFO:tasks.rados.rados.0.smithi157.stdout:78: finishing copy op to oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.284 INFO:tasks.rados.rados.0.smithi157.stdout:78: done (15 left) 2024-09-16T03:37:14.284 INFO:tasks.rados.rados.0.smithi157.stdout:95: copy oid 46 target oid 46 low_tier 2024-09-16T03:37:14.284 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 46 to 46 low_tier tgt_pool_name low_tier 2024-09-16T03:37:14.284 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.284 INFO:tasks.rados.rados.0.smithi157.stdout:83: finishing copy op to oid 34 low_tier 2024-09-16T03:37:14.284 INFO:tasks.rados.rados.0.smithi157.stdout:83: done (15 left) 2024-09-16T03:37:14.284 INFO:tasks.rados.rados.0.smithi157.stdout:96: copy oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.284 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 47 to 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:14.284 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.301 INFO:tasks.rados.rados.0.smithi157.stdout:85: finishing copy op to oid 36 low_tier 2024-09-16T03:37:14.301 INFO:tasks.rados.rados.0.smithi157.stdout:85: done (15 left) 2024-09-16T03:37:14.301 INFO:tasks.rados.rados.0.smithi157.stdout:97: copy oid 48 target oid 48 low_tier 2024-09-16T03:37:14.301 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 48 to 48 low_tier tgt_pool_name low_tier 2024-09-16T03:37:14.301 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.351 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:14.350+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:14.357 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:14.357 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:14.366 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:37:14.386 INFO:tasks.rados.rados.0.smithi157.stdout:67: finishing copy op to oid 18 low_tier 2024-09-16T03:37:14.386 INFO:tasks.rados.rados.0.smithi157.stdout:67: done (15 left) 2024-09-16T03:37:14.386 INFO:tasks.rados.rados.0.smithi157.stdout:98: copy oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.387 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 49 to 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:14.387 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.396 INFO:tasks.rados.rados.0.smithi157.stdout:81: finishing copy op to oid 32 low_tier 2024-09-16T03:37:14.396 INFO:tasks.rados.rados.0.smithi157.stdout:81: done (15 left) 2024-09-16T03:37:14.396 INFO:tasks.rados.rados.0.smithi157.stdout:99: copy oid 50 target oid 50 low_tier 2024-09-16T03:37:14.396 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 50 to 50 low_tier tgt_pool_name low_tier 2024-09-16T03:37:14.397 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.401 INFO:tasks.rados.rados.0.smithi157.stdout:82: finishing copy op to oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.401 INFO:tasks.rados.rados.0.smithi157.stdout:82: done (15 left) 2024-09-16T03:37:14.401 INFO:tasks.rados.rados.0.smithi157.stdout:100: copy oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.401 INFO:tasks.rados.rados.0.smithi157.stdout:copy op oid 1 to 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2024-09-16T03:37:14.401 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.408 INFO:tasks.rados.rados.0.smithi157.stdout:88: finishing copy op to oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.409 INFO:tasks.rados.rados.0.smithi157.stdout:88: done (15 left) 2024-09-16T03:37:14.409 INFO:tasks.rados.rados.0.smithi157.stdout:101: set_chunk oid 2 target oid 2 low_tier 2024-09-16T03:37:14.409 INFO:tasks.rados.rados.0.smithi157.stdout:101: set_chunk oid 2 offset: 350720 length: 1444864 target oid smithi15736696-2 low_tier offset: 350720 snap -1 2024-09-16T03:37:14.409 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.411 INFO:tasks.rados.rados.0.smithi157.stdout:95: finishing copy op to oid 46 low_tier 2024-09-16T03:37:14.411 INFO:tasks.rados.rados.0.smithi157.stdout:95: done (15 left) 2024-09-16T03:37:14.411 INFO:tasks.rados.rados.0.smithi157.stdout:102: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.411 INFO:tasks.rados.rados.0.smithi157.stdout:102: set_chunk oid 3 offset: 260608 length: 169984 target oid smithi15736696-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 260608 snap -1 2024-09-16T03:37:14.412 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.452 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:14.450+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:14.486 INFO:tasks.rados.rados.0.smithi157.stdout:74: finishing copy op to oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.486 INFO:tasks.rados.rados.0.smithi157.stdout:74: done (15 left) 2024-09-16T03:37:14.486 INFO:tasks.rados.rados.0.smithi157.stdout:103: set_chunk oid 4 target oid 4 low_tier 2024-09-16T03:37:14.487 INFO:tasks.rados.rados.0.smithi157.stdout:103: set_chunk oid 4 offset: 517120 length: 626688 target oid smithi15736696-4 low_tier offset: 517120 snap -1 2024-09-16T03:37:14.487 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.487 INFO:tasks.rados.rados.0.smithi157.stdout:96: finishing copy op to oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.487 INFO:tasks.rados.rados.0.smithi157.stdout:96: done (15 left) 2024-09-16T03:37:14.488 INFO:tasks.rados.rados.0.smithi157.stdout:104: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.488 INFO:tasks.rados.rados.0.smithi157.stdout:104: set_chunk oid 5 offset: 1156096 length: 820736 target oid smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1156096 snap -1 2024-09-16T03:37:14.488 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.489 INFO:tasks.rados.rados.0.smithi157.stdout:84: finishing copy op to oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.489 INFO:tasks.rados.rados.0.smithi157.stdout:84: done (15 left) 2024-09-16T03:37:14.489 INFO:tasks.rados.rados.0.smithi157.stdout:105: set_chunk oid 6 target oid 6 low_tier 2024-09-16T03:37:14.489 INFO:tasks.rados.rados.0.smithi157.stdout:105: set_chunk oid 6 offset: 314880 length: 1545728 target oid smithi15736696-6 low_tier offset: 314880 snap -1 2024-09-16T03:37:14.489 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.494 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:14.495 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:14.501 INFO:tasks.rados.rados.0.smithi157.stdout:92: finishing copy op to oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.501 INFO:tasks.rados.rados.0.smithi157.stdout:92: done (15 left) 2024-09-16T03:37:14.503 INFO:tasks.rados.rados.0.smithi157.stdout:106: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.503 INFO:tasks.rados.rados.0.smithi157.stdout:106: set_chunk oid 7 offset: 214016 length: 215040 target oid smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 214016 snap -1 2024-09-16T03:37:14.503 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.504 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:37:14.510 INFO:tasks.rados.rados.0.smithi157.stdout:90: finishing copy op to oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.510 INFO:tasks.rados.rados.0.smithi157.stdout:90: done (15 left) 2024-09-16T03:37:14.510 INFO:tasks.rados.rados.0.smithi157.stdout:107: set_chunk oid 8 target oid 8 low_tier 2024-09-16T03:37:14.510 INFO:tasks.rados.rados.0.smithi157.stdout:107: set_chunk oid 8 offset: 428032 length: 1131520 target oid smithi15736696-8 low_tier offset: 428032 snap -1 2024-09-16T03:37:14.511 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.512 INFO:tasks.rados.rados.0.smithi157.stdout:93: finishing copy op to oid 44 low_tier 2024-09-16T03:37:14.512 INFO:tasks.rados.rados.0.smithi157.stdout:93: done (15 left) 2024-09-16T03:37:14.512 INFO:tasks.rados.rados.0.smithi157.stdout:108: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.512 INFO:tasks.rados.rados.0.smithi157.stdout:108: set_chunk oid 9 offset: 644608 length: 139776 target oid smithi15736696-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 644608 snap -1 2024-09-16T03:37:14.512 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.515 INFO:tasks.rados.rados.0.smithi157.stdout:100: finishing copy op to oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.515 INFO:tasks.rados.rados.0.smithi157.stdout:100: done (15 left) 2024-09-16T03:37:14.515 INFO:tasks.rados.rados.0.smithi157.stdout:109: set_chunk oid 10 target oid 10 low_tier 2024-09-16T03:37:14.515 INFO:tasks.rados.rados.0.smithi157.stdout:109: set_chunk oid 10 offset: 1418240 length: 153088 target oid smithi15736696-10 low_tier offset: 1418240 snap -1 2024-09-16T03:37:14.515 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.553 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:14.551+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:14.589 INFO:tasks.rados.rados.0.smithi157.stdout:87: finishing copy op to oid 38 low_tier 2024-09-16T03:37:14.589 INFO:tasks.rados.rados.0.smithi157.stdout:87: done (15 left) 2024-09-16T03:37:14.589 INFO:tasks.rados.rados.0.smithi157.stdout:110: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.589 INFO:tasks.rados.rados.0.smithi157.stdout:110: set_chunk oid 11 offset: 771072 length: 1076736 target oid smithi15736696-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 771072 snap -1 2024-09-16T03:37:14.589 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.614 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:14.614 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:14.616 INFO:tasks.rados.rados.0.smithi157.stdout:101: finishing set_chunk to oid 2 2024-09-16T03:37:14.616 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T03:37:14.616 INFO:tasks.rados.rados.0.smithi157.stdout:101: done (15 left) 2024-09-16T03:37:14.616 INFO:tasks.rados.rados.0.smithi157.stdout:111: set_chunk oid 12 target oid 12 low_tier 2024-09-16T03:37:14.617 INFO:tasks.rados.rados.0.smithi157.stdout:111: set_chunk oid 12 offset: 48128 length: 1675776 target oid smithi15736696-12 low_tier offset: 48128 snap -1 2024-09-16T03:37:14.617 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.623 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:37:14.653 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:14.652+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:14.689 INFO:tasks.rados.rados.0.smithi157.stdout:91: finishing copy op to oid 42 low_tier 2024-09-16T03:37:14.689 INFO:tasks.rados.rados.0.smithi157.stdout:91: done (15 left) 2024-09-16T03:37:14.689 INFO:tasks.rados.rados.0.smithi157.stdout:112: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.689 INFO:tasks.rados.rados.0.smithi157.stdout:112: set_chunk oid 13 offset: 222208 length: 1040384 target oid smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 222208 snap -1 2024-09-16T03:37:14.690 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.712 INFO:tasks.rados.rados.0.smithi157.stdout:89: finishing copy op to oid 40 low_tier 2024-09-16T03:37:14.713 INFO:tasks.rados.rados.0.smithi157.stdout:89: done (15 left) 2024-09-16T03:37:14.713 INFO:tasks.rados.rados.0.smithi157.stdout:113: set_chunk oid 14 target oid 14 low_tier 2024-09-16T03:37:14.713 INFO:tasks.rados.rados.0.smithi157.stdout:113: set_chunk oid 14 offset: 780288 length: 1311744 target oid smithi15736696-14 low_tier offset: 780288 snap -1 2024-09-16T03:37:14.713 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.715 INFO:tasks.rados.rados.0.smithi157.stdout:97: finishing copy op to oid 48 low_tier 2024-09-16T03:37:14.715 INFO:tasks.rados.rados.0.smithi157.stdout:97: done (15 left) 2024-09-16T03:37:14.715 INFO:tasks.rados.rados.0.smithi157.stdout:114: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.715 INFO:tasks.rados.rados.0.smithi157.stdout:114: set_chunk oid 15 offset: 994304 length: 508416 target oid smithi15736696-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 994304 snap -1 2024-09-16T03:37:14.715 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.717 INFO:tasks.rados.rados.0.smithi157.stdout:98: finishing copy op to oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.717 INFO:tasks.rados.rados.0.smithi157.stdout:98: done (15 left) 2024-09-16T03:37:14.717 INFO:tasks.rados.rados.0.smithi157.stdout:115: set_chunk oid 16 target oid 16 low_tier 2024-09-16T03:37:14.717 INFO:tasks.rados.rados.0.smithi157.stdout:115: set_chunk oid 16 offset: 1036800 length: 194560 target oid smithi15736696-16 low_tier offset: 1036800 snap -1 2024-09-16T03:37:14.717 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.754 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:14.752+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:14.770 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:14.770 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:14.779 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:37:14.820 INFO:tasks.rados.rados.0.smithi157.stdout:102: finishing set_chunk to oid 3 2024-09-16T03:37:14.821 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T03:37:14.821 INFO:tasks.rados.rados.0.smithi157.stdout:102: done (15 left) 2024-09-16T03:37:14.821 INFO:tasks.rados.rados.0.smithi157.stdout:116: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.821 INFO:tasks.rados.rados.0.smithi157.stdout:116: set_chunk oid 17 offset: 1652224 length: 227840 target oid smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1652224 snap -1 2024-09-16T03:37:14.821 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.821 INFO:tasks.rados.rados.0.smithi157.stdout:105: finishing set_chunk to oid 6 2024-09-16T03:37:14.821 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:37:14.821 INFO:tasks.rados.rados.0.smithi157.stdout:106: finishing set_chunk to oid 7 2024-09-16T03:37:14.821 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T03:37:14.822 INFO:tasks.rados.rados.0.smithi157.stdout:105: done (15 left) 2024-09-16T03:37:14.822 INFO:tasks.rados.rados.0.smithi157.stdout:106: done (14 left) 2024-09-16T03:37:14.822 INFO:tasks.rados.rados.0.smithi157.stdout:117: set_chunk oid 18 target oid 18 low_tier 2024-09-16T03:37:14.822 INFO:tasks.rados.rados.0.smithi157.stdout:117: set_chunk oid 18 offset: 1943040 length: 120320 target oid smithi15736696-18 low_tier offset: 1943040 snap -1 2024-09-16T03:37:14.822 INFO:tasks.rados.rados.0.smithi157.stdout:118: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.822 INFO:tasks.rados.rados.0.smithi157.stdout:118: set_chunk oid 19 offset: 1267200 length: 498176 target oid smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1267200 snap -1 2024-09-16T03:37:14.822 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.822 INFO:tasks.rados.rados.0.smithi157.stdout:94: finishing copy op to oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.822 INFO:tasks.rados.rados.0.smithi157.stdout:94: done (15 left) 2024-09-16T03:37:14.822 INFO:tasks.rados.rados.0.smithi157.stdout:119: set_chunk oid 20 target oid 20 low_tier 2024-09-16T03:37:14.822 INFO:tasks.rados.rados.0.smithi157.stdout:119: set_chunk oid 20 offset: 58368 length: 1395200 target oid smithi15736696-20 low_tier offset: 58368 snap -1 2024-09-16T03:37:14.823 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.865 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:14.862+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:14.896 INFO:tasks.rados.rados.0.smithi157.stdout:104: finishing set_chunk to oid 5 2024-09-16T03:37:14.896 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T03:37:14.896 INFO:tasks.rados.rados.0.smithi157.stdout:104: done (15 left) 2024-09-16T03:37:14.896 INFO:tasks.rados.rados.0.smithi157.stdout:120: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.896 INFO:tasks.rados.rados.0.smithi157.stdout:120: set_chunk oid 21 offset: 146944 length: 826880 target oid smithi15736696-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 146944 snap -1 2024-09-16T03:37:14.896 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.923 INFO:tasks.rados.rados.0.smithi157.stdout:111: finishing set_chunk to oid 12 2024-09-16T03:37:14.924 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:37:14.924 INFO:tasks.rados.rados.0.smithi157.stdout:111: done (15 left) 2024-09-16T03:37:14.924 INFO:tasks.rados.rados.0.smithi157.stdout:121: set_chunk oid 22 target oid 22 low_tier 2024-09-16T03:37:14.924 INFO:tasks.rados.rados.0.smithi157.stdout:121: set_chunk oid 22 offset: 998400 length: 843776 target oid smithi15736696-22 low_tier offset: 998400 snap -1 2024-09-16T03:37:14.924 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.925 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:14.925 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:14.926 INFO:tasks.rados.rados.0.smithi157.stdout:107: finishing set_chunk to oid 8 2024-09-16T03:37:14.926 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:37:14.926 INFO:tasks.rados.rados.0.smithi157.stdout:107: done (15 left) 2024-09-16T03:37:14.926 INFO:tasks.rados.rados.0.smithi157.stdout:122: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.926 INFO:tasks.rados.rados.0.smithi157.stdout:122: set_chunk oid 23 offset: 292864 length: 1954816 target oid smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 292864 snap -1 2024-09-16T03:37:14.927 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.927 INFO:tasks.rados.rados.0.smithi157.stdout:109: finishing set_chunk to oid 10 2024-09-16T03:37:14.927 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:37:14.927 INFO:tasks.rados.rados.0.smithi157.stdout:109: done (15 left) 2024-09-16T03:37:14.927 INFO:tasks.rados.rados.0.smithi157.stdout:123: set_chunk oid 24 target oid 24 low_tier 2024-09-16T03:37:14.927 INFO:tasks.rados.rados.0.smithi157.stdout:123: set_chunk oid 24 offset: 1569792 length: 599552 target oid smithi15736696-24 low_tier offset: 1569792 snap -1 2024-09-16T03:37:14.927 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.927 INFO:tasks.rados.rados.0.smithi157.stdout:112: finishing set_chunk to oid 13 2024-09-16T03:37:14.927 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T03:37:14.927 INFO:tasks.rados.rados.0.smithi157.stdout:112: done (15 left) 2024-09-16T03:37:14.927 INFO:tasks.rados.rados.0.smithi157.stdout:124: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:14.927 INFO:tasks.rados.rados.0.smithi157.stdout:124: set_chunk oid 25 offset: 479744 length: 376320 target oid smithi15736696-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 479744 snap -1 2024-09-16T03:37:14.927 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:14.933 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:37:14.948 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:37:14.956 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:14.954+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:14.976 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:14.973+0000 7f652ae6c640 -1 osd.1 40 handle_osd_map injecting map crc failure 2024-09-16T03:37:14.977 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.1 primary-affinity to 0 (802) 2024-09-16T03:37:14.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:37:15.017 INFO:tasks.rados.rados.0.smithi157.stdout:108: finishing set_chunk to oid 9 2024-09-16T03:37:15.017 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:37:15.018 INFO:tasks.rados.rados.0.smithi157.stdout:108: done (15 left) 2024-09-16T03:37:15.018 INFO:tasks.rados.rados.0.smithi157.stdout:125: set_chunk oid 26 target oid 26 low_tier 2024-09-16T03:37:15.018 INFO:tasks.rados.rados.0.smithi157.stdout:125: set_chunk oid 26 offset: 1580544 length: 787968 target oid smithi15736696-26 low_tier offset: 1580544 snap -1 2024-09-16T03:37:15.018 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.056 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:15.053+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:15.078 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:15.078 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:15.079 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:15.079 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:15.087 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:37:15.097 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:15.098 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:15.114 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:15.115 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:15.121 INFO:tasks.rados.rados.0.smithi157.stdout:116: finishing set_chunk to oid 17 2024-09-16T03:37:15.121 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T03:37:15.121 INFO:tasks.rados.rados.0.smithi157.stdout:114: finishing set_chunk to oid 15 2024-09-16T03:37:15.121 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:15.121 INFO:tasks.rados.rados.0.smithi157.stdout:114: done (15 left) 2024-09-16T03:37:15.121 INFO:tasks.rados.rados.0.smithi157.stdout:116: done (14 left) 2024-09-16T03:37:15.121 INFO:tasks.rados.rados.0.smithi157.stdout:126: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:15.121 INFO:tasks.rados.rados.0.smithi157.stdout:126: set_chunk oid 27 offset: 525824 length: 1633280 target oid smithi15736696-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 525824 snap -1 2024-09-16T03:37:15.122 INFO:tasks.rados.rados.0.smithi157.stdout:127: set_chunk oid 28 target oid 28 low_tier 2024-09-16T03:37:15.122 INFO:tasks.rados.rados.0.smithi157.stdout:127: set_chunk oid 28 offset: 157696 length: 555008 target oid smithi15736696-28 low_tier offset: 157696 snap -1 2024-09-16T03:37:15.122 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.131 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:15.131 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:15.147 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:15.148 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:15.157 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:15.155+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:15.164 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:15.164 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:15.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:15.181 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:15.197 INFO:tasks.rados.rados.0.smithi157.stdout:99: finishing copy op to oid 50 low_tier 2024-09-16T03:37:15.198 INFO:tasks.rados.rados.0.smithi157.stdout:99: done (15 left) 2024-09-16T03:37:15.198 INFO:tasks.rados.rados.0.smithi157.stdout:128: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:15.198 INFO:tasks.rados.rados.0.smithi157.stdout:128: set_chunk oid 29 offset: 408064 length: 1645568 target oid smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 408064 snap -1 2024-09-16T03:37:15.198 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.198 INFO:tasks.rados.rados.0.smithi157.stdout:103: finishing set_chunk to oid 4 2024-09-16T03:37:15.198 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:37:15.198 INFO:tasks.rados.rados.0.smithi157.stdout:103: done (15 left) 2024-09-16T03:37:15.198 INFO:tasks.rados.rados.0.smithi157.stdout:129: set_chunk oid 30 target oid 30 low_tier 2024-09-16T03:37:15.198 INFO:tasks.rados.rados.0.smithi157.stdout:129: set_chunk oid 30 offset: 504832 length: 179712 target oid smithi15736696-30 low_tier offset: 504832 snap -1 2024-09-16T03:37:15.198 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.199 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:15.199 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:15.213 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:15.213 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:15.225 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:15.225 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:15.229 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:15.230 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:15.234 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:37:15.246 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:15.246 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:15.258 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:15.256+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:15.299 INFO:tasks.rados.rados.0.smithi157.stdout:113: finishing set_chunk to oid 14 2024-09-16T03:37:15.299 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T03:37:15.299 INFO:tasks.rados.rados.0.smithi157.stdout:113: done (15 left) 2024-09-16T03:37:15.299 INFO:tasks.rados.rados.0.smithi157.stdout:130: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:15.299 INFO:tasks.rados.rados.0.smithi157.stdout:130: set_chunk oid 31 offset: 410624 length: 1716224 target oid smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 410624 snap -1 2024-09-16T03:37:15.300 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.315 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:15.315 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:15.331 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:15.331 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:15.339 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:15.339 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:15.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:37:15.348 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:15.349 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:15.359 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:15.357+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:15.364 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:15.364 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:15.381 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:15.381 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:15.400 INFO:tasks.rados.rados.0.smithi157.stdout:121: finishing set_chunk to oid 22 2024-09-16T03:37:15.400 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T03:37:15.400 INFO:tasks.rados.rados.0.smithi157.stdout:117: finishing set_chunk to oid 18 2024-09-16T03:37:15.400 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T03:37:15.400 INFO:tasks.rados.rados.0.smithi157.stdout:125: finishing set_chunk to oid 26 2024-09-16T03:37:15.400 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 12 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:37:15.400 INFO:tasks.rados.rados.0.smithi157.stdout:117: done (15 left) 2024-09-16T03:37:15.400 INFO:tasks.rados.rados.0.smithi157.stdout:121: done (14 left) 2024-09-16T03:37:15.400 INFO:tasks.rados.rados.0.smithi157.stdout:125: done (13 left) 2024-09-16T03:37:15.400 INFO:tasks.rados.rados.0.smithi157.stdout:131: set_chunk oid 32 target oid 32 low_tier 2024-09-16T03:37:15.400 INFO:tasks.rados.rados.0.smithi157.stdout:131: set_chunk oid 32 offset: 836096 length: 133120 target oid smithi15736696-32 low_tier offset: 836096 snap -1 2024-09-16T03:37:15.401 INFO:tasks.rados.rados.0.smithi157.stdout:132: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:15.401 INFO:tasks.rados.rados.0.smithi157.stdout:132: set_chunk oid 33 offset: 347648 length: 927744 target oid smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 347648 snap -1 2024-09-16T03:37:15.401 INFO:tasks.rados.rados.0.smithi157.stdout:133: set_chunk oid 34 target oid 34 low_tier 2024-09-16T03:37:15.401 INFO:tasks.rados.rados.0.smithi157.stdout:133: set_chunk oid 34 offset: 1950208 length: 301056 target oid smithi15736696-34 low_tier offset: 1950208 snap -1 2024-09-16T03:37:15.401 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.422 INFO:tasks.rados.rados.0.smithi157.stdout:110: finishing set_chunk to oid 11 2024-09-16T03:37:15.422 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T03:37:15.422 INFO:tasks.rados.rados.0.smithi157.stdout:110: done (15 left) 2024-09-16T03:37:15.422 INFO:tasks.rados.rados.0.smithi157.stdout:134: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:15.422 INFO:tasks.rados.rados.0.smithi157.stdout:134: set_chunk oid 35 offset: 749056 length: 372736 target oid smithi15736696-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 749056 snap -1 2024-09-16T03:37:15.422 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.457 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:15.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.8.22:4 3.2 e41/31)", 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.397142+0000", 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.059225923, 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059242389999999999, 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397142+0000", 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397142+0000", 2024-09-16T03:37:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397142+0000", 2024-09-16T03:37:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.03e-07 2024-09-16T03:37:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397147+0000", 2024-09-16T03:37:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9239999999999998e-06 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397161+0000", 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3638000000000001e-05 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397170+0000", 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9560000000000003e-06 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.10.22:8 3.2 e41/31)", 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.397188+0000", 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.05917998, 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059221422000000003, 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397188+0000", 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397188+0000", 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397189+0000", 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4299999999999999e-07 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397197+0000", 2024-09-16T03:37:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4829999999999999e-06 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397201+0000", 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.613e-06 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397203+0000", 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.858e-06 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:37:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 2 2024-09-16T03:37:15.464 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:15.465 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:15.463+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:15.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:37:15.499 INFO:tasks.rados.rados.0.smithi157.stdout:119: finishing set_chunk to oid 20 2024-09-16T03:37:15.499 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 9 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T03:37:15.499 INFO:tasks.rados.rados.0.smithi157.stdout:119: done (15 left) 2024-09-16T03:37:15.499 INFO:tasks.rados.rados.0.smithi157.stdout:135: set_chunk oid 36 target oid 36 low_tier 2024-09-16T03:37:15.500 INFO:tasks.rados.rados.0.smithi157.stdout:135: set_chunk oid 36 offset: 30208 length: 1171968 target oid smithi15736696-36 low_tier offset: 30208 snap -1 2024-09-16T03:37:15.500 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.566 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:15.564+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:15.602 INFO:tasks.rados.rados.0.smithi157.stdout:120: finishing set_chunk to oid 21 2024-09-16T03:37:15.602 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T03:37:15.602 INFO:tasks.rados.rados.0.smithi157.stdout:120: done (15 left) 2024-09-16T03:37:15.602 INFO:tasks.rados.rados.0.smithi157.stdout:136: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:15.602 INFO:tasks.rados.rados.0.smithi157.stdout:136: set_chunk oid 37 offset: 55808 length: 1153536 target oid smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 55808 snap -1 2024-09-16T03:37:15.602 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.610 INFO:tasks.rados.rados.0.smithi157.stdout:127: finishing set_chunk to oid 28 2024-09-16T03:37:15.610 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 3 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T03:37:15.610 INFO:tasks.rados.rados.0.smithi157.stdout:127: done (15 left) 2024-09-16T03:37:15.610 INFO:tasks.rados.rados.0.smithi157.stdout:137: set_chunk oid 38 target oid 38 low_tier 2024-09-16T03:37:15.610 INFO:tasks.rados.rados.0.smithi157.stdout:137: set_chunk oid 38 offset: 1810944 length: 46592 target oid smithi15736696-38 low_tier offset: 1810944 snap -1 2024-09-16T03:37:15.610 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.610 INFO:tasks.rados.rados.0.smithi157.stdout:130: finishing set_chunk to oid 31 2024-09-16T03:37:15.611 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:37:15.611 INFO:tasks.rados.rados.0.smithi157.stdout:130: done (15 left) 2024-09-16T03:37:15.611 INFO:tasks.rados.rados.0.smithi157.stdout:138: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:15.611 INFO:tasks.rados.rados.0.smithi157.stdout:138: set_chunk oid 39 offset: 1501184 length: 653312 target oid smithi15736696-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1501184 snap -1 2024-09-16T03:37:15.611 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.613 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:15.614 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:15.614 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:15.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:37:15.667 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:15.665+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:15.702 INFO:tasks.rados.rados.0.smithi157.stdout:129: finishing set_chunk to oid 30 2024-09-16T03:37:15.702 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:37:15.702 INFO:tasks.rados.rados.0.smithi157.stdout:129: done (15 left) 2024-09-16T03:37:15.703 INFO:tasks.rados.rados.0.smithi157.stdout:139: set_chunk oid 40 target oid 40 low_tier 2024-09-16T03:37:15.703 INFO:tasks.rados.rados.0.smithi157.stdout:139: set_chunk oid 40 offset: 936960 length: 571904 target oid smithi15736696-40 low_tier offset: 936960 snap -1 2024-09-16T03:37:15.703 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.707 INFO:tasks.rados.rados.0.smithi157.stdout:134: finishing set_chunk to oid 35 2024-09-16T03:37:15.707 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:37:15.707 INFO:tasks.rados.rados.0.smithi157.stdout:134: done (15 left) 2024-09-16T03:37:15.707 INFO:tasks.rados.rados.0.smithi157.stdout:140: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:15.707 INFO:tasks.rados.rados.0.smithi157.stdout:140: set_chunk oid 41 offset: 233984 length: 1971200 target oid smithi15736696-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 233984 snap -1 2024-09-16T03:37:15.707 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.733 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:17 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'7, mlcod=25'5)", 2024-09-16T03:37:15.733 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:52.877437+0000", 2024-09-16T03:37:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.853332658999999, 2024-09-16T03:37:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201252123, 2024-09-16T03:37:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877437+0000", 2024-09-16T03:37:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877437+0000", 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877439+0000", 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877446+0000", 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877459+0000", 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3124e-05 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977610+0000", 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100150878 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977758+0000", 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147644 2024-09-16T03:37:15.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977819+0000", 2024-09-16T03:37:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0798999999999998e-05 2024-09-16T03:37:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.078644+0000", 2024-09-16T03:37:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10082574399999999 2024-09-16T03:37:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.078689+0000", 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5016999999999999e-05 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:18 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'8, mlcod=25'5)", 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:52.977655+0000", 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.753115270999999, 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10141719, 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.738 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977655+0000", 2024-09-16T03:37:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977655+0000", 2024-09-16T03:37:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977657+0000", 2024-09-16T03:37:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.973e-06 2024-09-16T03:37:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977670+0000", 2024-09-16T03:37:15.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.322e-05 2024-09-16T03:37:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977685+0000", 2024-09-16T03:37:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5492999999999999e-05 2024-09-16T03:37:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977716+0000", 2024-09-16T03:37:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0979000000000003e-05 2024-09-16T03:37:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.078362+0000", 2024-09-16T03:37:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100645185 2024-09-16T03:37:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.078390+0000", 2024-09-16T03:37:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8850999999999999e-05 2024-09-16T03:37:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.079040+0000", 2024-09-16T03:37:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064971500000000004 2024-09-16T03:37:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.079072+0000", 2024-09-16T03:37:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1773999999999998e-05 2024-09-16T03:37:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:22 1.0 e26/23 1:30306672:devicehealth::main.db.0000000000000000:head v 26'12, mlcod=25'8)", 2024-09-16T03:37:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:53.483871+0000", 2024-09-16T03:37:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.246899236000001, 2024-09-16T03:37:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101867204, 2024-09-16T03:37:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483871+0000", 2024-09-16T03:37:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483871+0000", 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483892+0000", 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0962999999999999e-05 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483977+0000", 2024-09-16T03:37:15.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4975000000000005e-05 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483986+0000", 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1339999999999993e-06 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.584127+0000", 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10014146 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.584272+0000", 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014464100000000001 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.584327+0000", 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5012000000000001e-05 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.585688+0000", 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013607890000000001 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.585738+0000", 2024-09-16T03:37:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0229999999999998e-05 2024-09-16T03:37:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.748 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:39 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'21, mlcod=26'19)", 2024-09-16T03:37:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:37:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.135451934999999, 2024-09-16T03:37:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30197182500000003, 2024-09-16T03:37:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:37:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:37:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595320+0000", 2024-09-16T03:37:15.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.962e-06 2024-09-16T03:37:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595336+0000", 2024-09-16T03:37:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5568000000000001e-05 2024-09-16T03:37:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595349+0000", 2024-09-16T03:37:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3203e-05 2024-09-16T03:37:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695424+0000", 2024-09-16T03:37:15.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007542899999999 2024-09-16T03:37:15.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695495+0000", 2024-09-16T03:37:15.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0450999999999993e-05 2024-09-16T03:37:15.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695538+0000", 2024-09-16T03:37:15.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3433000000000002e-05 2024-09-16T03:37:15.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897256+0000", 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20171840699999999 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897290+0000", 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3371999999999998e-05 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.754 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:40 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'22, mlcod=26'19)", 2024-09-16T03:37:15.755 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695446+0000", 2024-09-16T03:37:15.755 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.035323886, 2024-09-16T03:37:15.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20236575600000001, 2024-09-16T03:37:15.755 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.755 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.755 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.755 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695446+0000", 2024-09-16T03:37:15.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695446+0000", 2024-09-16T03:37:15.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695447+0000", 2024-09-16T03:37:15.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:37:15.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695461+0000", 2024-09-16T03:37:15.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3915e-05 2024-09-16T03:37:15.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695470+0000", 2024-09-16T03:37:15.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6959999999999994e-06 2024-09-16T03:37:15.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695478+0000", 2024-09-16T03:37:15.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2050000000000002e-06 2024-09-16T03:37:15.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695933+0000", 2024-09-16T03:37:15.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00045501500000000002 2024-09-16T03:37:15.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695958+0000", 2024-09-16T03:37:15.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4578000000000001e-05 2024-09-16T03:37:15.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897782+0000", 2024-09-16T03:37:15.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20182425700000001 2024-09-16T03:37:15.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897812+0000", 2024-09-16T03:37:15.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9947e-05 2024-09-16T03:37:15.759 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.759 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.759 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.760 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:41 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'23, mlcod=26'19)", 2024-09-16T03:37:15.760 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695515+0000", 2024-09-16T03:37:15.760 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.035255318000001, 2024-09-16T03:37:15.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20231417400000001, 2024-09-16T03:37:15.760 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.760 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.760 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.760 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695515+0000", 2024-09-16T03:37:15.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695515+0000", 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695516+0000", 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7499999999999998e-07 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695590+0000", 2024-09-16T03:37:15.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3924999999999994e-05 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695601+0000", 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1253999999999999e-05 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695611+0000", 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9059999999999997e-06 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696250+0000", 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063903099999999997 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696263+0000", 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.276e-05 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897815+0000", 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201552074 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897829+0000", 2024-09-16T03:37:15.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4249e-05 2024-09-16T03:37:15.764 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.764 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.764 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.764 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:42 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'24, mlcod=26'19)", 2024-09-16T03:37:15.764 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:37:15.764 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.03514741, 2024-09-16T03:37:15.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.202223289, 2024-09-16T03:37:15.764 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.764 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.765 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.765 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:37:15.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:37:15.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8599999999999997e-07 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695638+0000", 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4844000000000001e-05 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695645+0000", 2024-09-16T03:37:15.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8510000000000001e-06 2024-09-16T03:37:15.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695651+0000", 2024-09-16T03:37:15.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5600000000000001e-06 2024-09-16T03:37:15.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696535+0000", 2024-09-16T03:37:15.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088443500000000004 2024-09-16T03:37:15.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696550+0000", 2024-09-16T03:37:15.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4788e-05 2024-09-16T03:37:15.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897831+0000", 2024-09-16T03:37:15.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201281234 2024-09-16T03:37:15.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897846+0000", 2024-09-16T03:37:15.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4691e-05 2024-09-16T03:37:15.768 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.769 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.769 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.769 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:43 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'25, mlcod=26'19)", 2024-09-16T03:37:15.769 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695660+0000", 2024-09-16T03:37:15.769 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.035110039999999, 2024-09-16T03:37:15.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.202209369, 2024-09-16T03:37:15.769 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.769 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.769 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.769 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695660+0000", 2024-09-16T03:37:15.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695660+0000", 2024-09-16T03:37:15.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695661+0000", 2024-09-16T03:37:15.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:37:15.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695669+0000", 2024-09-16T03:37:15.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.32e-06 2024-09-16T03:37:15.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695676+0000", 2024-09-16T03:37:15.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3610000000000003e-06 2024-09-16T03:37:15.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695681+0000", 2024-09-16T03:37:15.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0889999999999999e-06 2024-09-16T03:37:15.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696770+0000", 2024-09-16T03:37:15.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001088852 2024-09-16T03:37:15.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696784+0000", 2024-09-16T03:37:15.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4178999999999999e-05 2024-09-16T03:37:15.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897849+0000", 2024-09-16T03:37:15.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20106433700000001 2024-09-16T03:37:15.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897869+0000", 2024-09-16T03:37:15.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0652000000000001e-05 2024-09-16T03:37:15.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.773 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.774 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:44 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'26, mlcod=26'19)", 2024-09-16T03:37:15.774 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:37:15.774 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.035079372999999, 2024-09-16T03:37:15.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20224351600000001, 2024-09-16T03:37:15.774 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.774 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.774 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.774 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:37:15.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:37:15.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:37:15.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1099999999999996e-07 2024-09-16T03:37:15.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695765+0000", 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3973000000000005e-05 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695775+0000", 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5400000000000001e-06 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695781+0000", 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8369999999999999e-06 2024-09-16T03:37:15.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698245+0000", 2024-09-16T03:37:15.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024633480000000002 2024-09-16T03:37:15.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698260+0000", 2024-09-16T03:37:15.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4859e-05 2024-09-16T03:37:15.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897880+0000", 2024-09-16T03:37:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199619933 2024-09-16T03:37:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897934+0000", 2024-09-16T03:37:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4515000000000002e-05 2024-09-16T03:37:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:45 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'27, mlcod=26'19)", 2024-09-16T03:37:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.698152+0000", 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.032618197000001, 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200415062, 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698152+0000", 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698152+0000", 2024-09-16T03:37:15.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698153+0000", 2024-09-16T03:37:15.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.525e-06 2024-09-16T03:37:15.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698165+0000", 2024-09-16T03:37:15.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1861e-05 2024-09-16T03:37:15.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698178+0000", 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3275000000000001e-05 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698186+0000", 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4139999999999997e-06 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897000+0000", 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.198814149 2024-09-16T03:37:15.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897035+0000", 2024-09-16T03:37:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5039000000000001e-05 2024-09-16T03:37:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.898524+0000", 2024-09-16T03:37:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001488952 2024-09-16T03:37:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.898567+0000", 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2846999999999998e-05 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:46 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'28, mlcod=26'19)", 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.698626+0000", 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.03214393, 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20067976400000001, 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698626+0000", 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698626+0000", 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698627+0000", 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.395e-06 2024-09-16T03:37:15.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698641+0000", 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698653+0000", 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1846e-05 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.798792+0000", 2024-09-16T03:37:15.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100139397 2024-09-16T03:37:15.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897344+0000", 2024-09-16T03:37:15.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098552085999999997 2024-09-16T03:37:15.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897361+0000", 2024-09-16T03:37:15.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6716000000000001e-05 2024-09-16T03:37:15.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.899266+0000", 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019056520000000001 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.899306+0000", 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9322000000000001e-05 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:77 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'55, mlcod=28'53)", 2024-09-16T03:37:15.787 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:57.284321+0000", 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.446448697000001, 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101501645, 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284321+0000", 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284321+0000", 2024-09-16T03:37:15.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284326+0000", 2024-09-16T03:37:15.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4020000000000004e-06 2024-09-16T03:37:15.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284408+0000", 2024-09-16T03:37:15.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2547999999999996e-05 2024-09-16T03:37:15.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284435+0000", 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6455e-05 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284449+0000", 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3822000000000001e-05 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284552+0000", 2024-09-16T03:37:15.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010370999999999999 2024-09-16T03:37:15.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284580+0000", 2024-09-16T03:37:15.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7375000000000001e-05 2024-09-16T03:37:15.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.385774+0000", 2024-09-16T03:37:15.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101194703 2024-09-16T03:37:15.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.385823+0000", 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.863e-05 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:78 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'56, mlcod=28'53)", 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:57.285053+0000", 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.445716491999999, 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101371368, 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.792 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.285053+0000", 2024-09-16T03:37:15.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.285053+0000", 2024-09-16T03:37:15.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.285055+0000", 2024-09-16T03:37:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2610000000000001e-06 2024-09-16T03:37:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.285110+0000", 2024-09-16T03:37:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5560000000000003e-05 2024-09-16T03:37:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.285120+0000", 2024-09-16T03:37:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6490000000000003e-06 2024-09-16T03:37:15.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.285127+0000", 2024-09-16T03:37:15.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5020000000000003e-06 2024-09-16T03:37:15.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.385287+0000", 2024-09-16T03:37:15.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015977600000001 2024-09-16T03:37:15.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.385323+0000", 2024-09-16T03:37:15.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5311000000000001e-05 2024-09-16T03:37:15.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.386389+0000", 2024-09-16T03:37:15.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001065999 2024-09-16T03:37:15.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.386425+0000", 2024-09-16T03:37:15.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6310000000000003e-05 2024-09-16T03:37:15.797 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.797 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.797 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.797 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:79 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'57, mlcod=28'53)", 2024-09-16T03:37:15.797 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:57.288914+0000", 2024-09-16T03:37:15.798 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.441856291000001, 2024-09-16T03:37:15.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10187072799999999, 2024-09-16T03:37:15.798 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.798 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.798 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.798 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.288914+0000", 2024-09-16T03:37:15.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.288914+0000", 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.288917+0000", 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5049999999999999e-06 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.288998+0000", 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1298000000000007e-05 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.289019+0000", 2024-09-16T03:37:15.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0234999999999999e-05 2024-09-16T03:37:15.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.389117+0000", 2024-09-16T03:37:15.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10009857 2024-09-16T03:37:15.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.389340+0000", 2024-09-16T03:37:15.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022286100000000001 2024-09-16T03:37:15.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.389372+0000", 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2057000000000002e-05 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.390733+0000", 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013608769999999999 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.390784+0000", 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1325000000000001e-05 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.802 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.803 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.803 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:94 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'68, mlcod=28'61)", 2024-09-16T03:37:15.803 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:58.787786+0000", 2024-09-16T03:37:15.803 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.942984031999998, 2024-09-16T03:37:15.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101551135, 2024-09-16T03:37:15.803 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.803 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.803 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.803 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.787786+0000", 2024-09-16T03:37:15.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.787786+0000", 2024-09-16T03:37:15.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.787789+0000", 2024-09-16T03:37:15.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8009999999999999e-06 2024-09-16T03:37:15.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.787804+0000", 2024-09-16T03:37:15.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5668999999999999e-05 2024-09-16T03:37:15.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.787820+0000", 2024-09-16T03:37:15.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5772000000000001e-05 2024-09-16T03:37:15.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.787832+0000", 2024-09-16T03:37:15.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2128e-05 2024-09-16T03:37:15.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.887977+0000", 2024-09-16T03:37:15.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100145172 2024-09-16T03:37:15.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.887996+0000", 2024-09-16T03:37:15.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8077e-05 2024-09-16T03:37:15.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.889308+0000", 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00131285 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.889337+0000", 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8666e-05 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.807 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:99 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'73, mlcod=28'66)", 2024-09-16T03:37:15.808 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:37:15.808 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.639478132000001, 2024-09-16T03:37:15.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201204829, 2024-09-16T03:37:15.808 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.808 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.808 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.808 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:37:15.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:37:15.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:37:15.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9800000000000004e-07 2024-09-16T03:37:15.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091329+0000", 2024-09-16T03:37:15.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7039000000000002e-05 2024-09-16T03:37:15.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091336+0000", 2024-09-16T03:37:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:37:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191408+0000", 2024-09-16T03:37:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100072733 2024-09-16T03:37:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191470+0000", 2024-09-16T03:37:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2087000000000004e-05 2024-09-16T03:37:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191504+0000", 2024-09-16T03:37:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3453e-05 2024-09-16T03:37:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292441+0000", 2024-09-16T03:37:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100937374 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292497+0000", 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5365999999999998e-05 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:100 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'74, mlcod=28'66)", 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:59.191451+0000", 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.539319425999999, 2024-09-16T03:37:15.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10146166, 2024-09-16T03:37:15.814 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.814 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.814 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.814 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191451+0000", 2024-09-16T03:37:15.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191451+0000", 2024-09-16T03:37:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191452+0000", 2024-09-16T03:37:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6509999999999999e-06 2024-09-16T03:37:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191464+0000", 2024-09-16T03:37:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1901e-05 2024-09-16T03:37:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191473+0000", 2024-09-16T03:37:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4360000000000002e-06 2024-09-16T03:37:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191483+0000", 2024-09-16T03:37:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0570999999999999e-05 2024-09-16T03:37:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292103+0000", 2024-09-16T03:37:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100620336 2024-09-16T03:37:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292136+0000", 2024-09-16T03:37:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2809999999999999e-05 2024-09-16T03:37:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292884+0000", 2024-09-16T03:37:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00074768000000000002 2024-09-16T03:37:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292912+0000", 2024-09-16T03:37:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8274999999999999e-05 2024-09-16T03:37:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:120 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'90, mlcod=28'85)", 2024-09-16T03:37:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:00.966325+0000", 2024-09-16T03:37:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.764444578999999, 2024-09-16T03:37:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101423443, 2024-09-16T03:37:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966325+0000", 2024-09-16T03:37:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966325+0000", 2024-09-16T03:37:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966326+0000", 2024-09-16T03:37:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:37:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966402+0000", 2024-09-16T03:37:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6174999999999995e-05 2024-09-16T03:37:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966410+0000", 2024-09-16T03:37:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8350000000000004e-06 2024-09-16T03:37:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966414+0000", 2024-09-16T03:37:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1350000000000002e-06 2024-09-16T03:37:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966771+0000", 2024-09-16T03:37:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035708499999999999 2024-09-16T03:37:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966793+0000", 2024-09-16T03:37:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1424e-05 2024-09-16T03:37:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:01.067671+0000", 2024-09-16T03:37:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10087803400000001 2024-09-16T03:37:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:01.067749+0000", 2024-09-16T03:37:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8030999999999999e-05 2024-09-16T03:37:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:9 4.8 e38/32 4:13b03fc9:::smithi15736696-3:head v 38'1, mlcod=0'0)", 2024-09-16T03:37:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.980745+0000", 2024-09-16T03:37:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.750024765, 2024-09-16T03:37:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102593847, 2024-09-16T03:37:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.980745+0000", 2024-09-16T03:37:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.980745+0000", 2024-09-16T03:37:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.980746+0000", 2024-09-16T03:37:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-09-16T03:37:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.981323+0000", 2024-09-16T03:37:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057713399999999998 2024-09-16T03:37:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.981335+0000", 2024-09-16T03:37:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2257e-05 2024-09-16T03:37:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.981345+0000", 2024-09-16T03:37:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0399e-05 2024-09-16T03:37:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.081502+0000", 2024-09-16T03:37:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015674400000001 2024-09-16T03:37:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.081527+0000", 2024-09-16T03:37:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5106e-05 2024-09-16T03:37:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.083241+0000", 2024-09-16T03:37:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017139410000000001 2024-09-16T03:37:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.083339+0000", 2024-09-16T03:37:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7874999999999994e-05 2024-09-16T03:37:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.10.22:8 3.2 e41/31)", 2024-09-16T03:37:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.397188+0000", 2024-09-16T03:37:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.33358178799999999, 2024-09-16T03:37:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101229079, 2024-09-16T03:37:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397188+0000", 2024-09-16T03:37:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397188+0000", 2024-09-16T03:37:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397189+0000", 2024-09-16T03:37:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4299999999999999e-07 2024-09-16T03:37:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397197+0000", 2024-09-16T03:37:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4829999999999999e-06 2024-09-16T03:37:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397201+0000", 2024-09-16T03:37:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.613e-06 2024-09-16T03:37:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397203+0000", 2024-09-16T03:37:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.858e-06 2024-09-16T03:37:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.497649+0000", 2024-09-16T03:37:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10044581399999999 2024-09-16T03:37:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.497662+0000", 2024-09-16T03:37:15.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2261e-05 2024-09-16T03:37:15.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.498390+0000", 2024-09-16T03:37:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000728095 2024-09-16T03:37:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.498417+0000", 2024-09-16T03:37:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7611999999999999e-05 2024-09-16T03:37:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.838 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:15.839 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:15.836+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:15.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:37:15.842 INFO:tasks.rados.rados.0.smithi157.stdout:124: finishing set_chunk to oid 25 2024-09-16T03:37:15.842 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:15.842 INFO:tasks.rados.rados.0.smithi157.stdout:124: done (15 left) 2024-09-16T03:37:15.842 INFO:tasks.rados.rados.0.smithi157.stdout:141: set_chunk oid 42 target oid 42 low_tier 2024-09-16T03:37:15.842 INFO:tasks.rados.rados.0.smithi157.stdout:141: set_chunk oid 42 offset: 231424 length: 1875456 target oid smithi15736696-42 low_tier offset: 231424 snap -1 2024-09-16T03:37:15.842 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.842 INFO:tasks.rados.rados.0.smithi157.stdout:126: finishing set_chunk to oid 27 2024-09-16T03:37:15.842 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:15.842 INFO:tasks.rados.rados.0.smithi157.stdout:126: done (15 left) 2024-09-16T03:37:15.842 INFO:tasks.rados.rados.0.smithi157.stdout:142: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:15.843 INFO:tasks.rados.rados.0.smithi157.stdout:142: set_chunk oid 43 offset: 92672 length: 1287168 target oid smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 92672 snap -1 2024-09-16T03:37:15.843 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.903 INFO:tasks.rados.rados.0.smithi157.stdout:131: finishing set_chunk to oid 32 2024-09-16T03:37:15.903 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:37:15.903 INFO:tasks.rados.rados.0.smithi157.stdout:131: done (15 left) 2024-09-16T03:37:15.903 INFO:tasks.rados.rados.0.smithi157.stdout:143: set_chunk oid 44 target oid 44 low_tier 2024-09-16T03:37:15.903 INFO:tasks.rados.rados.0.smithi157.stdout:143: set_chunk oid 44 offset: 884224 length: 1102848 target oid smithi15736696-44 low_tier offset: 884224 snap -1 2024-09-16T03:37:15.903 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.940 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:15.939+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:15.950 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:293 4.e 4:765e9179:::smithi15736696-44:head [assert-version v9,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.902028+0000", 2024-09-16T03:37:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.047394617, 2024-09-16T03:37:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047419059999999999, 2024-09-16T03:37:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 293 2024-09-16T03:37:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902028+0000", 2024-09-16T03:37:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902028+0000", 2024-09-16T03:37:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902031+0000", 2024-09-16T03:37:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9440000000000001e-06 2024-09-16T03:37:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902041+0000", 2024-09-16T03:37:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0477e-05 2024-09-16T03:37:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902058+0000", 2024-09-16T03:37:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7227e-05 2024-09-16T03:37:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902069+0000", 2024-09-16T03:37:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0165e-05 2024-09-16T03:37:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902130+0000", 2024-09-16T03:37:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1043000000000005e-05 2024-09-16T03:37:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902202+0000", 2024-09-16T03:37:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1884000000000005e-05 2024-09-16T03:37:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:37:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-09-16T03:37:15.955 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:15.959 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:37:15.982 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:15.980+0000 7ff266366640 -1 osd.11 41 handle_osd_map injecting map crc failure 2024-09-16T03:37:15.983 INFO:tasks.rados.rados.0.smithi157.stdout:137: finishing set_chunk to oid 38 2024-09-16T03:37:15.983 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:37:15.983 INFO:tasks.rados.rados.0.smithi157.stdout:137: done (15 left) 2024-09-16T03:37:15.983 INFO:tasks.rados.rados.0.smithi157.stdout:144: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:15.983 INFO:tasks.rados.rados.0.smithi157.stdout:144: set_chunk oid 45 offset: 886272 length: 1440768 target oid smithi15736696-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 886272 snap -1 2024-09-16T03:37:15.983 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:15.984 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:37:16.040 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:16.039+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:16.067 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:16.067 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:16.067 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:16.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:37:16.086 INFO:tasks.rados.rados.0.smithi157.stdout:142: finishing set_chunk to oid 43 2024-09-16T03:37:16.086 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 12 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T03:37:16.086 INFO:tasks.rados.rados.0.smithi157.stdout:142: done (15 left) 2024-09-16T03:37:16.086 INFO:tasks.rados.rados.0.smithi157.stdout:145: set_chunk oid 46 target oid 46 low_tier 2024-09-16T03:37:16.086 INFO:tasks.rados.rados.0.smithi157.stdout:145: set_chunk oid 46 offset: 1321472 length: 1080832 target oid smithi15736696-46 low_tier offset: 1321472 snap -1 2024-09-16T03:37:16.086 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.108 INFO:tasks.rados.rados.0.smithi157.stdout:136: finishing set_chunk to oid 37 2024-09-16T03:37:16.109 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T03:37:16.109 INFO:tasks.rados.rados.0.smithi157.stdout:140: finishing set_chunk to oid 41 2024-09-16T03:37:16.109 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:37:16.109 INFO:tasks.rados.rados.0.smithi157.stdout:136: done (15 left) 2024-09-16T03:37:16.109 INFO:tasks.rados.rados.0.smithi157.stdout:140: done (14 left) 2024-09-16T03:37:16.109 INFO:tasks.rados.rados.0.smithi157.stdout:146: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:16.109 INFO:tasks.rados.rados.0.smithi157.stdout:146: set_chunk oid 47 offset: 313344 length: 2379776 target oid smithi15736696-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 313344 snap -1 2024-09-16T03:37:16.109 INFO:tasks.rados.rados.0.smithi157.stdout:147: set_chunk oid 48 target oid 48 low_tier 2024-09-16T03:37:16.109 INFO:tasks.rados.rados.0.smithi157.stdout:147: set_chunk oid 48 offset: 232448 length: 2506752 target oid smithi15736696-48 low_tier offset: 232448 snap -1 2024-09-16T03:37:16.109 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.142 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:16.140+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:16.210 INFO:tasks.rados.rados.0.smithi157.stdout:141: finishing set_chunk to oid 42 2024-09-16T03:37:16.210 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T03:37:16.210 INFO:tasks.rados.rados.0.smithi157.stdout:141: done (15 left) 2024-09-16T03:37:16.210 INFO:tasks.rados.rados.0.smithi157.stdout:148: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:16.210 INFO:tasks.rados.rados.0.smithi157.stdout:148: set_chunk oid 49 offset: 1273344 length: 1325568 target oid smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1273344 snap -1 2024-09-16T03:37:16.210 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.226 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:16.226 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:16.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:16.226 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:16.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.226 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:33 4.e 4:773b934b:::smithi15736696-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:16.226 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.050039+0000", 2024-09-16T03:37:16.226 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1732413639999999, 2024-09-16T03:37:16.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40763112099999999, 2024-09-16T03:37:16.226 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050039+0000", 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050039+0000", 2024-09-16T03:37:16.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050042+0000", 2024-09-16T03:37:16.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5009999999999999e-06 2024-09-16T03:37:16.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050806+0000", 2024-09-16T03:37:16.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076374199999999998 2024-09-16T03:37:16.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050819+0000", 2024-09-16T03:37:16.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3692999999999999e-05 2024-09-16T03:37:16.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050829+0000", 2024-09-16T03:37:16.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8630000000000003e-06 2024-09-16T03:37:16.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050917+0000", 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8196999999999996e-05 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050922+0000", 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.831e-06 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153032+0000", 2024-09-16T03:37:16.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102109992 2024-09-16T03:37:16.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153123+0000", 2024-09-16T03:37:16.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1037000000000003e-05 2024-09-16T03:37:16.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:16.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153218+0000", 2024-09-16T03:37:16.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5047e-05 2024-09-16T03:37:16.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:16.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156107+0000", 2024-09-16T03:37:16.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028893299999999998 2024-09-16T03:37:16.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255606+0000", 2024-09-16T03:37:16.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099498438999999994 2024-09-16T03:37:16.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457591+0000", 2024-09-16T03:37:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201984996 2024-09-16T03:37:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457631+0000", 2024-09-16T03:37:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0110000000000001e-05 2024-09-16T03:37:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457670+0000", 2024-09-16T03:37:16.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9342999999999999e-05 2024-09-16T03:37:16.234 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.234 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.234 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.234 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:34 4.e 4:773b934b:::smithi15736696-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:16.234 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:37:16.234 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1688693460000001, 2024-09-16T03:37:16.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50346527299999999, 2024-09-16T03:37:16.234 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.234 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.234 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.234 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:37:16.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054413+0000", 2024-09-16T03:37:16.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0669999999999999e-06 2024-09-16T03:37:16.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054525+0000", 2024-09-16T03:37:16.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111462 2024-09-16T03:37:16.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054538+0000", 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3341e-05 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054545+0000", 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.584e-06 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054608+0000", 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2953999999999994e-05 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054613+0000", 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6110000000000001e-06 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154852+0000", 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10023838 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154915+0000", 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3138000000000004e-05 2024-09-16T03:37:16.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155001+0000", 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5730999999999996e-05 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156166+0000", 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001165485 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255909+0000", 2024-09-16T03:37:16.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099742558999999995 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557800+0000", 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30189130600000003 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557836+0000", 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6566000000000002e-05 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557877+0000", 2024-09-16T03:37:16.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0089000000000002e-05 2024-09-16T03:37:16.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.241 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.245 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:35 4.e 4:773b934b:::smithi15736696-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:16.245 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:37:16.245 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.168721015, 2024-09-16T03:37:16.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50344320499999995, 2024-09-16T03:37:16.245 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.245 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.245 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.245 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.245 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.245 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.245 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-16T03:37:16.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.246 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:37:16.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:37:16.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054564+0000", 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5710000000000002e-06 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054569+0000", 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0490000000000003e-06 2024-09-16T03:37:16.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054578+0000", 2024-09-16T03:37:16.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:37:16.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054639+0000", 2024-09-16T03:37:16.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1061000000000004e-05 2024-09-16T03:37:16.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:37:16.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054644+0000", 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999996e-06 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155607+0000", 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100963372 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155652+0000", 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:16.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155722+0000", 2024-09-16T03:37:16.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0130999999999997e-05 2024-09-16T03:37:16.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:16.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.157921+0000", 2024-09-16T03:37:16.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021990159999999998 2024-09-16T03:37:16.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255940+0000", 2024-09-16T03:37:16.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098018569999999999 2024-09-16T03:37:16.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557930+0000", 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.301990333 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557963+0000", 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2348000000000003e-05 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558003+0000", 2024-09-16T03:37:16.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:37:16.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.252 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.252 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:36 4.e 4:773b934b:::smithi15736696-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:16.252 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:37:16.252 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1686823259999999, 2024-09-16T03:37:16.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50353811800000003, 2024-09-16T03:37:16.252 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.252 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.252 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-16T03:37:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:37:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054599+0000", 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054601+0000", 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4360000000000001e-06 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054606+0000", 2024-09-16T03:37:16.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1019999999999996e-06 2024-09-16T03:37:16.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054610+0000", 2024-09-16T03:37:16.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.681e-06 2024-09-16T03:37:16.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054662+0000", 2024-09-16T03:37:16.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1545000000000002e-05 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054664+0000", 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1919999999999999e-06 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155903+0000", 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101239068 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:16.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155970+0000", 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6934999999999995e-05 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558029+0000", 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40205888899999997 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558078+0000", 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9842999999999997e-05 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558136+0000", 2024-09-16T03:37:16.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7985000000000001e-05 2024-09-16T03:37:16.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.258 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.258 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:45 4.1 4:89839b7e:::smithi15736696-12:head [write 584877~544092,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:16.258 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.084704+0000", 2024-09-16T03:37:16.258 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1385762279999998, 2024-09-16T03:37:16.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.204597581, 2024-09-16T03:37:16.258 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 45 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.084704+0000", 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.084704+0000", 2024-09-16T03:37:16.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.084707+0000", 2024-09-16T03:37:16.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6520000000000002e-06 2024-09-16T03:37:16.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.085209+0000", 2024-09-16T03:37:16.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050241700000000001 2024-09-16T03:37:16.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.085227+0000", 2024-09-16T03:37:16.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7229999999999999e-05 2024-09-16T03:37:16.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.085236+0000", 2024-09-16T03:37:16.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7000000000000003e-06 2024-09-16T03:37:16.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.185361+0000", 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100124484 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.185365+0000", 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2520000000000001e-06 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.185647+0000", 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028175999999999999 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.185733+0000", 2024-09-16T03:37:16.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6550000000000003e-05 2024-09-16T03:37:16.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-09-16T03:37:16.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.185821+0000", 2024-09-16T03:37:16.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8049999999999999e-05 2024-09-16T03:37:16.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:16.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.288487+0000", 2024-09-16T03:37:16.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102665916 2024-09-16T03:37:16.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.288836+0000", 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034844800000000001 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289226+0000", 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039014799999999999 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289258+0000", 2024-09-16T03:37:16.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1628999999999998e-05 2024-09-16T03:37:16.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289302+0000", 2024-09-16T03:37:16.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4345000000000003e-05 2024-09-16T03:37:16.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.265 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.265 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:46 4.1 4:89839b7e:::smithi15736696-12:head [write 1610596~562003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:16.265 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.090699+0000", 2024-09-16T03:37:16.265 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1325820320000002, 2024-09-16T03:37:16.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29921881900000002, 2024-09-16T03:37:16.266 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.266 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.266 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.266 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.266 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.266 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.266 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 46 2024-09-16T03:37:16.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.266 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.090699+0000", 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.090699+0000", 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.090701+0000", 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.745e-06 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091336+0000", 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000634784 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091355+0000", 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9003000000000001e-05 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091366+0000", 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0855e-05 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.185397+0000", 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094030631000000003 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.185399+0000", 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7659999999999999e-06 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.186908+0000", 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015083010000000001 2024-09-16T03:37:16.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.186963+0000", 2024-09-16T03:37:16.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5773000000000003e-05 2024-09-16T03:37:16.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-09-16T03:37:16.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.187043+0000", 2024-09-16T03:37:16.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9664000000000002e-05 2024-09-16T03:37:16.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:16.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.288490+0000", 2024-09-16T03:37:16.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101447038 2024-09-16T03:37:16.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289337+0000", 2024-09-16T03:37:16.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00084686500000000003 2024-09-16T03:37:16.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389813+0000", 2024-09-16T03:37:16.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10047641 2024-09-16T03:37:16.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389863+0000", 2024-09-16T03:37:16.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9416000000000003e-05 2024-09-16T03:37:16.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389917+0000", 2024-09-16T03:37:16.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4568000000000003e-05 2024-09-16T03:37:16.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.274 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.274 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:47 4.1 4:89839b7e:::smithi15736696-12:head [setxattr _header (56) in=7b,truncate 2184877] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:16.274 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.091384+0000", 2024-09-16T03:37:16.274 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1318968800000002, 2024-09-16T03:37:16.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29870844299999999, 2024-09-16T03:37:16.274 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.274 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 47 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091384+0000", 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091384+0000", 2024-09-16T03:37:16.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091385+0000", 2024-09-16T03:37:16.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8299999999999996e-07 2024-09-16T03:37:16.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091389+0000", 2024-09-16T03:37:16.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5360000000000003e-06 2024-09-16T03:37:16.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091394+0000", 2024-09-16T03:37:16.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0459999999999996e-06 2024-09-16T03:37:16.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091398+0000", 2024-09-16T03:37:16.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5279999999999999e-06 2024-09-16T03:37:16.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.288106+0000", 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19670805599999999 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.288166+0000", 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0455000000000003e-05 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,7", 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.288250+0000", 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3368000000000005e-05 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:16.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.288629+0000", 2024-09-16T03:37:16.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037977900000000001 2024-09-16T03:37:16.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389454+0000", 2024-09-16T03:37:16.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100824434 2024-09-16T03:37:16.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389972+0000", 2024-09-16T03:37:16.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00051786400000000004 2024-09-16T03:37:16.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.390030+0000", 2024-09-16T03:37:16.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8464999999999997e-05 2024-09-16T03:37:16.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.390092+0000", 2024-09-16T03:37:16.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2129000000000001e-05 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:48 4.1 4:89839b7e:::smithi15736696-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.091404+0000", 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1318762979999999, 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29894670400000001, 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 48 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091404+0000", 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091404+0000", 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091405+0000", 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5999999999999999e-07 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091406+0000", 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7209999999999999e-06 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091411+0000", 2024-09-16T03:37:16.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6580000000000001e-06 2024-09-16T03:37:16.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.091414+0000", 2024-09-16T03:37:16.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2389999999999998e-06 2024-09-16T03:37:16.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.288439+0000", 2024-09-16T03:37:16.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19702472800000001 2024-09-16T03:37:16.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:16.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.288479+0000", 2024-09-16T03:37:16.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9719999999999999e-05 2024-09-16T03:37:16.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.390142+0000", 2024-09-16T03:37:16.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101663513 2024-09-16T03:37:16.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.390235+0000", 2024-09-16T03:37:16.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2268000000000005e-05 2024-09-16T03:37:16.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.390351+0000", 2024-09-16T03:37:16.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011649700000000001 2024-09-16T03:37:16.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.286 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.286 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:65 4.0 e39/32 4:023abada:::smithi15736696-17:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:16.286 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.243633+0000", 2024-09-16T03:37:16.286 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9796474750000002, 2024-09-16T03:37:16.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034624920000000002, 2024-09-16T03:37:16.287 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.287 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.287 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.287 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.243633+0000", 2024-09-16T03:37:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.243633+0000", 2024-09-16T03:37:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.243634+0000", 2024-09-16T03:37:16.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8500000000000002e-07 2024-09-16T03:37:16.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.245514+0000", 2024-09-16T03:37:16.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001880933 2024-09-16T03:37:16.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.245527+0000", 2024-09-16T03:37:16.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3012e-05 2024-09-16T03:37:16.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.245538+0000", 2024-09-16T03:37:16.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0349e-05 2024-09-16T03:37:16.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.245606+0000", 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.826e-05 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.245628+0000", 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2059e-05 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.247065+0000", 2024-09-16T03:37:16.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001436559 2024-09-16T03:37:16.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.247096+0000", 2024-09-16T03:37:16.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0935000000000001e-05 2024-09-16T03:37:16.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.292 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:47 4.1 e39/36 ondisk, result = 0)", 2024-09-16T03:37:16.292 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.289268+0000", 2024-09-16T03:37:16.292 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.934013035, 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100192514, 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289268+0000", 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289268+0000", 2024-09-16T03:37:16.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289268+0000", 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3200000000000001e-07 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289270+0000", 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6390000000000001e-06 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289273+0000", 2024-09-16T03:37:16.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6330000000000002e-06 2024-09-16T03:37:16.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289276+0000", 2024-09-16T03:37:16.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5390000000000002e-06 2024-09-16T03:37:16.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389440+0000", 2024-09-16T03:37:16.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10016428600000001 2024-09-16T03:37:16.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389448+0000", 2024-09-16T03:37:16.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8210000000000002e-06 2024-09-16T03:37:16.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389460+0000", 2024-09-16T03:37:16.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2264e-05 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:46 4.1 e39/36 ondisk, result = 0)", 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.289671+0000", 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.933609379, 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10039656299999999, 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289671+0000", 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289671+0000", 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289672+0000", 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5000000000000001e-07 2024-09-16T03:37:16.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289676+0000", 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4300000000000002e-06 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.289685+0000", 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2590000000000006e-06 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389754+0000", 2024-09-16T03:37:16.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100068774 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389805+0000", 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1464000000000001e-05 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389810+0000", 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9980000000000003e-06 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.390068+0000", 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000257688 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:33 4.e e39/32 ondisk, result = 0)", 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.357344+0000", 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8659370069999999, 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100338025, 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.301 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357344+0000", 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357344+0000", 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357346+0000", 2024-09-16T03:37:16.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5770000000000001e-06 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357355+0000", 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8060000000000001e-06 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357380+0000", 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4586000000000001e-05 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357398+0000", 2024-09-16T03:37:16.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8873e-05 2024-09-16T03:37:16.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457578+0000", 2024-09-16T03:37:16.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100179829 2024-09-16T03:37:16.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457586+0000", 2024-09-16T03:37:16.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8210000000000002e-06 2024-09-16T03:37:16.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457682+0000", 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5532999999999995e-05 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:34 4.e e39/32 ondisk, result = 0)", 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.357439+0000", 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.865841299, 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20045163999999999, 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.305 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357439+0000", 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357439+0000", 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357441+0000", 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357448+0000", 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2370000000000003e-06 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357465+0000", 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6753e-05 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357483+0000", 2024-09-16T03:37:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8729e-05 2024-09-16T03:37:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557791+0000", 2024-09-16T03:37:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20030721100000001 2024-09-16T03:37:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557796+0000", 2024-09-16T03:37:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.626e-06 2024-09-16T03:37:16.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557891+0000", 2024-09-16T03:37:16.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4833999999999994e-05 2024-09-16T03:37:16.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.309 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.309 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:35 4.e e39/32 ondisk, result = 0)", 2024-09-16T03:37:16.309 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.357539+0000", 2024-09-16T03:37:16.309 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8657416740000001, 2024-09-16T03:37:16.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20047055999999999, 2024-09-16T03:37:16.310 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.310 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.310 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.310 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357539+0000", 2024-09-16T03:37:16.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357539+0000", 2024-09-16T03:37:16.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357540+0000", 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0100000000000001e-06 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357545+0000", 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9370000000000003e-06 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357558+0000", 2024-09-16T03:37:16.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3303e-05 2024-09-16T03:37:16.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.357572+0000", 2024-09-16T03:37:16.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3556e-05 2024-09-16T03:37:16.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557920+0000", 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20034832599999999 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557926+0000", 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5310000000000001e-06 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558010+0000", 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3897e-05 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.314 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:66 4.0 e39/32 4:023abada:::smithi15736696-17:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:16.314 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.389922+0000", 2024-09-16T03:37:16.314 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8333582239999999, 2024-09-16T03:37:16.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10180618399999999, 2024-09-16T03:37:16.314 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.314 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.314 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.314 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389922+0000", 2024-09-16T03:37:16.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389922+0000", 2024-09-16T03:37:16.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.389923+0000", 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0500000000000002e-07 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.390080+0000", 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015668299999999999 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.390087+0000", 2024-09-16T03:37:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9410000000000001e-06 2024-09-16T03:37:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.490163+0000", 2024-09-16T03:37:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007639 2024-09-16T03:37:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.490218+0000", 2024-09-16T03:37:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4345999999999998e-05 2024-09-16T03:37:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.490234+0000", 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6892e-05 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.491703+0000", 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001468977 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.491729+0000", 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5150000000000001e-05 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.318 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:70 4.0 e39/32 4:0a55bbf2:::smithi15736696-18:head v 39'5, mlcod=0'0)", 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.547090+0000", 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6761901969999999, 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021513629999999999, 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.547090+0000", 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.547090+0000", 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.547094+0000", 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.23e-06 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.547504+0000", 2024-09-16T03:37:16.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041017199999999999 2024-09-16T03:37:16.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.547516+0000", 2024-09-16T03:37:16.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500999999999999e-05 2024-09-16T03:37:16.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.547523+0000", 2024-09-16T03:37:16.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1069999999999999e-06 2024-09-16T03:37:16.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.547576+0000", 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2932000000000001e-05 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.547593+0000", 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7048e-05 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.549207+0000", 2024-09-16T03:37:16.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00161389 2024-09-16T03:37:16.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.549242+0000", 2024-09-16T03:37:16.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4483000000000002e-05 2024-09-16T03:37:16.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.323 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.323 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:267 4.0 e41/32 4:0a55bbf2:::smithi15736696-18:head v 41'20, mlcod=40'18)", 2024-09-16T03:37:16.323 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.197363+0000", 2024-09-16T03:37:16.323 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.0259172990000001, 2024-09-16T03:37:16.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200820739, 2024-09-16T03:37:16.324 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.324 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.324 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.324 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.197363+0000", 2024-09-16T03:37:16.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.197363+0000", 2024-09-16T03:37:16.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.197364+0000", 2024-09-16T03:37:16.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.23e-07 2024-09-16T03:37:16.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.197369+0000", 2024-09-16T03:37:16.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4820000000000002e-06 2024-09-16T03:37:16.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.197380+0000", 2024-09-16T03:37:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0524e-05 2024-09-16T03:37:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.297449+0000", 2024-09-16T03:37:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10006904799999999 2024-09-16T03:37:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397584+0000", 2024-09-16T03:37:16.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100135321 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.397614+0000", 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0270999999999999e-05 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398139+0000", 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052445599999999997 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398184+0000", 2024-09-16T03:37:16.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5414000000000003e-05 2024-09-16T03:37:16.328 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.328 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.328 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.328 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:280 4.e 4:70297eed:::smithi15736696-31:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:16.328 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.298531+0000", 2024-09-16T03:37:16.328 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.92474992099999997, 2024-09-16T03:37:16.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30954304300000002, 2024-09-16T03:37:16.328 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.328 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.328 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 280 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298531+0000", 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298531+0000", 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298534+0000", 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5769999999999998e-06 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298548+0000", 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.377e-05 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298576+0000", 2024-09-16T03:37:16.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8181999999999999e-05 2024-09-16T03:37:16.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398649+0000", 2024-09-16T03:37:16.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007253200000001 2024-09-16T03:37:16.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.498772+0000", 2024-09-16T03:37:16.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100123273 2024-09-16T03:37:16.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.498838+0000", 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5927999999999996e-05 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.605923+0000", 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10708543399999999 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.606636+0000", 2024-09-16T03:37:16.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000712836 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.607909+0000", 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00127289 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.608017+0000", 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108004 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.608048+0000", 2024-09-16T03:37:16.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.048e-05 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.608074+0000", 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6137000000000001e-05 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.4.22:3 3.1 e41/31 3:bffb5658:::smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'22, mlcod=41'20)", 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.504709+0000", 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.71857120500000005, 2024-09-16T03:37:16.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100843815, 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.504709+0000", 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.504709+0000", 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.504711+0000", 2024-09-16T03:37:16.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.691e-06 2024-09-16T03:37:16.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.504718+0000", 2024-09-16T03:37:16.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1459999999999999e-06 2024-09-16T03:37:16.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.504730+0000", 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1287999999999999e-05 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.604826+0000", 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100096304 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.604896+0000", 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0335999999999999e-05 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.604925+0000", 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9184e-05 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.605512+0000", 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058628800000000002 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.605553+0000", 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1578000000000001e-05 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:277 4.b e41/32 4:dcb6965e:::smithi15736696-28:head v 41'13, mlcod=40'11)", 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.604894+0000", 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.61838706399999999, 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029266069999999999, 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.604894+0000", 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.604894+0000", 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.604894+0000", 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9799999999999999e-07 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.604903+0000", 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9870000000000005e-06 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.604911+0000", 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1780000000000003e-06 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.604917+0000", 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9059999999999999e-06 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.604977+0000", 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9806000000000003e-05 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.604991+0000", 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3687e-05 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.607781+0000", 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027908630000000002 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.607820+0000", 2024-09-16T03:37:16.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8782000000000002e-05 2024-09-16T03:37:16.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.343 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.343 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.343 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:16.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:37:16.346 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:16.342+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:16.347 INFO:tasks.rados.rados.0.smithi157.stdout:143: finishing set_chunk to oid 44 2024-09-16T03:37:16.347 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T03:37:16.347 INFO:tasks.rados.rados.0.smithi157.stdout:143: done (15 left) 2024-09-16T03:37:16.347 INFO:tasks.rados.rados.0.smithi157.stdout:149: set_chunk oid 50 target oid 50 low_tier 2024-09-16T03:37:16.347 INFO:tasks.rados.rados.0.smithi157.stdout:149: set_chunk oid 50 offset: 516608 length: 2073600 target oid smithi15736696-50 low_tier offset: 516608 snap -1 2024-09-16T03:37:16.347 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.347 INFO:tasks.rados.rados.0.smithi157.stdout:144: finishing set_chunk to oid 45 2024-09-16T03:37:16.347 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 15 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T03:37:16.347 INFO:tasks.rados.rados.0.smithi157.stdout:144: done (15 left) 2024-09-16T03:37:16.347 INFO:tasks.rados.rados.0.smithi157.stdout:150: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:16.348 INFO:tasks.rados.rados.0.smithi157.stdout:150: set_chunk oid 1 offset: 488960 length: 488960 target oid smithi15736696-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 488960 snap -1 2024-09-16T03:37:16.348 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.348 INFO:tasks.rados.rados.0.smithi157.stdout:150: finishing set_chunk to oid 1 2024-09-16T03:37:16.348 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T03:37:16.348 INFO:tasks.rados.rados.0.smithi157.stdout:150: done (15 left) 2024-09-16T03:37:16.348 INFO:tasks.rados.rados.0.smithi157.stdout:151: set_chunk oid 2 target oid 2 low_tier 2024-09-16T03:37:16.348 INFO:tasks.rados.rados.0.smithi157.stdout:151: set_chunk oid 2 offset: 1441792 length: 455680 target oid smithi15736696-2 low_tier offset: 1441792 snap -1 2024-09-16T03:37:16.348 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.348 INFO:tasks.rados.rados.0.smithi157.stdout:151: finishing set_chunk to oid 2 2024-09-16T03:37:16.348 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 1441792 length: 455680 2024-09-16T03:37:16.348 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T03:37:16.348 INFO:tasks.rados.rados.0.smithi157.stdout:151: done (15 left) 2024-09-16T03:37:16.348 INFO:tasks.rados.rados.0.smithi157.stdout:152: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:16.349 INFO:tasks.rados.rados.0.smithi157.stdout:152: set_chunk oid 3 offset: 1303552 length: 309760 target oid smithi15736696-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1303552 snap -1 2024-09-16T03:37:16.349 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.412 INFO:tasks.rados.rados.0.smithi157.stdout:147: finishing set_chunk to oid 48 2024-09-16T03:37:16.412 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:37:16.413 INFO:tasks.rados.rados.0.smithi157.stdout:147: done (15 left) 2024-09-16T03:37:16.413 INFO:tasks.rados.rados.0.smithi157.stdout:153: set_chunk oid 4 target oid 4 low_tier 2024-09-16T03:37:16.413 INFO:tasks.rados.rados.0.smithi157.stdout:153: set_chunk oid 4 offset: 351232 length: 98304 target oid smithi15736696-4 low_tier offset: 351232 snap -1 2024-09-16T03:37:16.413 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.419 INFO:tasks.rados.rados.0.smithi157.stdout:152: finishing set_chunk to oid 3 2024-09-16T03:37:16.419 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T03:37:16.419 INFO:tasks.rados.rados.0.smithi157.stdout:152: done (15 left) 2024-09-16T03:37:16.419 INFO:tasks.rados.rados.0.smithi157.stdout:154: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:16.419 INFO:tasks.rados.rados.0.smithi157.stdout:154: set_chunk oid 5 offset: 230912 length: 95232 target oid smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 230912 snap -1 2024-09-16T03:37:16.419 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.449 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:16.444+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:16.495 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:16.495 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:16.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.495 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:303 4.a 4.a681cc8a (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:16.495 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:37:16.495 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.082174282000000001, 2024-09-16T03:37:16.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082192453999999998, 2024-09-16T03:37:16.495 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.495 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.495 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:16.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.496 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 303 2024-09-16T03:37:16.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.496 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:37:16.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:37:16.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411721+0000", 2024-09-16T03:37:16.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0200000000000001e-06 2024-09-16T03:37:16.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411729+0000", 2024-09-16T03:37:16.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1240000000000005e-06 2024-09-16T03:37:16.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411744+0000", 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099000000000001e-05 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411751+0000", 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.029e-06 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:283 4.a 4:503187b3:::smithi15736696-34:head [assert-version v15,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.398853+0000", 2024-09-16T03:37:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.0950401540000001, 2024-09-16T03:37:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0950980029999999, 2024-09-16T03:37:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 283 2024-09-16T03:37:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.499 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398853+0000", 2024-09-16T03:37:16.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398853+0000", 2024-09-16T03:37:16.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398855+0000", 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0080000000000001e-06 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398862+0000", 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6129999999999998e-06 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398885+0000", 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2988000000000001e-05 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.498959+0000", 2024-09-16T03:37:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100073548 2024-09-16T03:37:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.499016+0000", 2024-09-16T03:37:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7024000000000002e-05 2024-09-16T03:37:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.499066+0000", 2024-09-16T03:37:16.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9808000000000002e-05 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:285 4.a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v18,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.498989+0000", 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.99490359799999994, 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.994993409, 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.503 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 285 2024-09-16T03:37:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.498989+0000", 2024-09-16T03:37:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.498989+0000", 2024-09-16T03:37:16.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.498991+0000", 2024-09-16T03:37:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9350000000000001e-06 2024-09-16T03:37:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.499001+0000", 2024-09-16T03:37:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8810000000000008e-06 2024-09-16T03:37:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.499013+0000", 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1970000000000001e-05 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.499021+0000", 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3240000000000003e-06 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.499165+0000", 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143442 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.499218+0000", 2024-09-16T03:37:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3267999999999999e-05 2024-09-16T03:37:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:37:16.507 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 3 2024-09-16T03:37:16.507 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:16.508 INFO:tasks.rados.rados.0.smithi157.stdout:145: finishing set_chunk to oid 46 2024-09-16T03:37:16.509 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:37:16.509 INFO:tasks.rados.rados.0.smithi157.stdout:145: done (15 left) 2024-09-16T03:37:16.509 INFO:tasks.rados.rados.0.smithi157.stdout:155: set_chunk oid 6 target oid 6 low_tier 2024-09-16T03:37:16.509 INFO:tasks.rados.rados.0.smithi157.stdout:155: set_chunk oid 6 offset: 199680 length: 29184 target oid smithi15736696-6 low_tier offset: 199680 snap -1 2024-09-16T03:37:16.509 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:37:16.514 INFO:tasks.rados.rados.0.smithi157.stdout:149: finishing set_chunk to oid 50 2024-09-16T03:37:16.514 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 18 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:37:16.514 INFO:tasks.rados.rados.0.smithi157.stdout:149: done (15 left) 2024-09-16T03:37:16.514 INFO:tasks.rados.rados.0.smithi157.stdout:156: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:16.514 INFO:tasks.rados.rados.0.smithi157.stdout:156: set_chunk oid 7 offset: 1037312 length: 801280 target oid smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1037312 snap -1 2024-09-16T03:37:16.514 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.547 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:16.545+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:16.619 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:16.619 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:16.619 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:16.619 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:16.619 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:16.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:37:16.648 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:16.647+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:16.722 INFO:tasks.rados.rados.0.smithi157.stdout:154: finishing set_chunk to oid 5 2024-09-16T03:37:16.723 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T03:37:16.723 INFO:tasks.rados.rados.0.smithi157.stdout:154: done (15 left) 2024-09-16T03:37:16.723 INFO:tasks.rados.rados.0.smithi157.stdout:157: set_chunk oid 8 target oid 8 low_tier 2024-09-16T03:37:16.723 INFO:tasks.rados.rados.0.smithi157.stdout:157: set_chunk oid 8 offset: 936448 length: 605696 target oid smithi15736696-8 low_tier offset: 936448 snap -1 2024-09-16T03:37:16.723 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.723 INFO:tasks.rados.rados.0.smithi157.stdout:157: finishing set_chunk to oid 8 2024-09-16T03:37:16.723 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 936448 length: 605696 2024-09-16T03:37:16.723 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:37:16.723 INFO:tasks.rados.rados.0.smithi157.stdout:157: done (15 left) 2024-09-16T03:37:16.723 INFO:tasks.rados.rados.0.smithi157.stdout:158: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:16.723 INFO:tasks.rados.rados.0.smithi157.stdout:158: set_chunk oid 9 offset: 792064 length: 1281536 target oid smithi15736696-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 792064 snap -1 2024-09-16T03:37:16.723 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.736 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:16.736 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:16.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:16.737 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:16.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.737 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4338.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:37:16.737 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.094389+0000", 2024-09-16T03:37:16.737 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.639707757, 2024-09-16T03:37:16.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203832385, 2024-09-16T03:37:16.737 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.737 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.737 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.737 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.737 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4338", 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3073758685", 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094389+0000", 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094389+0000", 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094393+0000", 2024-09-16T03:37:16.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.467e-06 2024-09-16T03:37:16.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094405+0000", 2024-09-16T03:37:16.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1793e-05 2024-09-16T03:37:16.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094425+0000", 2024-09-16T03:37:16.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0412999999999999e-05 2024-09-16T03:37:16.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094436+0000", 2024-09-16T03:37:16.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1178e-05 2024-09-16T03:37:16.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094516+0000", 2024-09-16T03:37:16.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0109000000000002e-05 2024-09-16T03:37:16.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:16.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094522+0000", 2024-09-16T03:37:16.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8950000000000003e-06 2024-09-16T03:37:16.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.196488+0000", 2024-09-16T03:37:16.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10196596400000001 2024-09-16T03:37:16.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.196600+0000", 2024-09-16T03:37:16.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111703 2024-09-16T03:37:16.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,10", 2024-09-16T03:37:16.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.196738+0000", 2024-09-16T03:37:16.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013858800000000001 2024-09-16T03:37:16.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:16.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.197729+0000", 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00099082199999999993 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.198192+0000", 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046261099999999998 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.298103+0000", 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099911137999999997 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.298167+0000", 2024-09-16T03:37:16.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4289999999999996e-05 2024-09-16T03:37:16.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.298222+0000", 2024-09-16T03:37:16.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4413999999999998e-05 2024-09-16T03:37:16.744 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.744 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.744 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.744 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4338.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:16.745 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:37:16.745 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.636999435, 2024-09-16T03:37:16.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30502727499999999, 2024-09-16T03:37:16.745 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.745 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.745 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.745 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.745 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4338", 2024-09-16T03:37:16.745 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3073758685", 2024-09-16T03:37:16.745 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-16T03:37:16.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.745 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:37:16.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:37:16.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097101+0000", 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.915e-06 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097111+0000", 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0589e-05 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097130+0000", 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8958999999999999e-05 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097141+0000", 2024-09-16T03:37:16.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0616e-05 2024-09-16T03:37:16.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097227+0000", 2024-09-16T03:37:16.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6322000000000005e-05 2024-09-16T03:37:16.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:16.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097232+0000", 2024-09-16T03:37:16.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2059999999999998e-06 2024-09-16T03:37:16.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200235+0000", 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103002945 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200318+0000", 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2515999999999994e-05 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,10", 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200429+0000", 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011158999999999999 2024-09-16T03:37:16.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201478+0000", 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00104893 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201942+0000", 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046354700000000003 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402048+0000", 2024-09-16T03:37:16.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20010616000000001 2024-09-16T03:37:16.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402089+0000", 2024-09-16T03:37:16.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0762e-05 2024-09-16T03:37:16.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402125+0000", 2024-09-16T03:37:16.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6217999999999998e-05 2024-09-16T03:37:16.752 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.752 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.752 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.753 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:13 4.a 4:51338165:::smithi15736696-4:head [write 450421~625747,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:37:16.753 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.991194+0000", 2024-09-16T03:37:16.753 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.7429031120000005, 2024-09-16T03:37:16.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20656139100000001, 2024-09-16T03:37:16.753 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.753 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.753 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.754 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 13 2024-09-16T03:37:16.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.754 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991194+0000", 2024-09-16T03:37:16.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991194+0000", 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991197+0000", 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1470000000000001e-06 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991908+0000", 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071085899999999999 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991925+0000", 2024-09-16T03:37:16.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7368e-05 2024-09-16T03:37:16.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991936+0000", 2024-09-16T03:37:16.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0891e-05 2024-09-16T03:37:16.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.992011+0000", 2024-09-16T03:37:16.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4814000000000006e-05 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.992018+0000", 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2239999999999998e-06 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.994165+0000", 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002147043 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.994259+0000", 2024-09-16T03:37:16.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3764999999999994e-05 2024-09-16T03:37:16.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:16.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.994365+0000", 2024-09-16T03:37:16.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105701 2024-09-16T03:37:16.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:16.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996323+0000", 2024-09-16T03:37:16.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001958212 2024-09-16T03:37:16.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.000278+0000", 2024-09-16T03:37:16.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039546690000000001 2024-09-16T03:37:16.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197685+0000", 2024-09-16T03:37:16.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19740706 2024-09-16T03:37:16.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197721+0000", 2024-09-16T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6631000000000002e-05 2024-09-16T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197755+0000", 2024-09-16T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4007000000000002e-05 2024-09-16T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:14 4.a 4:51338165:::smithi15736696-4:head [write 1659093~391328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:16.760 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.996458+0000", 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.7376393820000002, 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20147354000000001, 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 14 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996458+0000", 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996458+0000", 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996476+0000", 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8681999999999998e-05 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996922+0000", 2024-09-16T03:37:16.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044590599999999998 2024-09-16T03:37:16.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996937+0000", 2024-09-16T03:37:16.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5214e-05 2024-09-16T03:37:16.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996948+0000", 2024-09-16T03:37:16.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0825000000000001e-05 2024-09-16T03:37:16.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.997004+0000", 2024-09-16T03:37:16.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5943000000000003e-05 2024-09-16T03:37:16.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.997079+0000", 2024-09-16T03:37:16.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5018999999999994e-05 2024-09-16T03:37:16.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:16.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.997179+0000", 2024-09-16T03:37:16.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9597000000000002e-05 2024-09-16T03:37:16.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:16.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.998553+0000", 2024-09-16T03:37:16.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001373649 2024-09-16T03:37:16.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197800+0000", 2024-09-16T03:37:16.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199247911 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197871+0000", 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0599000000000006e-05 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197906+0000", 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5006999999999999e-05 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197931+0000", 2024-09-16T03:37:16.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5188e-05 2024-09-16T03:37:16.767 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.767 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.767 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.767 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:15 4.a 4:51338165:::smithi15736696-4:head [setxattr _header (55) in=7b,truncate 2050421] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:16.767 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:37:16.767 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.7371275019999999, 2024-09-16T03:37:16.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30116835400000003, 2024-09-16T03:37:16.767 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.767 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.767 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.768 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 15 2024-09-16T03:37:16.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.768 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:37:16.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:37:16.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996971+0000", 2024-09-16T03:37:16.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.066e-06 2024-09-16T03:37:16.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996984+0000", 2024-09-16T03:37:16.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3246e-05 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996994+0000", 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.713e-06 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097070+0000", 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100076313 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097141+0000", 2024-09-16T03:37:16.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1298999999999996e-05 2024-09-16T03:37:16.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097205+0000", 2024-09-16T03:37:16.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3907999999999999e-05 2024-09-16T03:37:16.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:16.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097300+0000", 2024-09-16T03:37:16.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5209000000000003e-05 2024-09-16T03:37:16.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197645+0000", 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10034422899999999 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197837+0000", 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000192454 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298049+0000", 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100211868 2024-09-16T03:37:16.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298082+0000", 2024-09-16T03:37:16.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3467000000000003e-05 2024-09-16T03:37:16.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298138+0000", 2024-09-16T03:37:16.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5581999999999997e-05 2024-09-16T03:37:16.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.773 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.773 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:16 4.a 4:51338165:::smithi15736696-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.097092+0000", 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6370051170000002, 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20130547800000001, 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 16 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.774 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097092+0000", 2024-09-16T03:37:16.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097092+0000", 2024-09-16T03:37:16.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097094+0000", 2024-09-16T03:37:16.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0159999999999998e-06 2024-09-16T03:37:16.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097101+0000", 2024-09-16T03:37:16.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0489999999999998e-06 2024-09-16T03:37:16.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097109+0000", 2024-09-16T03:37:16.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6899999999999992e-06 2024-09-16T03:37:16.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097114+0000", 2024-09-16T03:37:16.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8450000000000002e-06 2024-09-16T03:37:16.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197569+0000", 2024-09-16T03:37:16.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100455534 2024-09-16T03:37:16.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:16.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197626+0000", 2024-09-16T03:37:16.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6548999999999997e-05 2024-09-16T03:37:16.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298197+0000", 2024-09-16T03:37:16.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10057105099999999 2024-09-16T03:37:16.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298281+0000", 2024-09-16T03:37:16.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4307000000000004e-05 2024-09-16T03:37:16.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298397+0000", 2024-09-16T03:37:16.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116437 2024-09-16T03:37:16.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.780 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.780 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:13 4.a e37/32 ondisk, result = 0)", 2024-09-16T03:37:16.780 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.097961+0000", 2024-09-16T03:37:16.780 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6361359540000002, 2024-09-16T03:37:16.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099803646999999995, 2024-09-16T03:37:16.780 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.780 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097961+0000", 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097961+0000", 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097962+0000", 2024-09-16T03:37:16.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.173e-06 2024-09-16T03:37:16.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097966+0000", 2024-09-16T03:37:16.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9709999999999998e-06 2024-09-16T03:37:16.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097976+0000", 2024-09-16T03:37:16.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0010000000000001e-05 2024-09-16T03:37:16.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097981+0000", 2024-09-16T03:37:16.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5639999999999998e-06 2024-09-16T03:37:16.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197676+0000", 2024-09-16T03:37:16.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099694783999999995 2024-09-16T03:37:16.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197680+0000", 2024-09-16T03:37:16.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3270000000000002e-06 2024-09-16T03:37:16.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197765+0000", 2024-09-16T03:37:16.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4818e-05 2024-09-16T03:37:16.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.784 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.784 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:14 4.a e39/32 ondisk, result = 0)", 2024-09-16T03:37:16.785 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.099182+0000", 2024-09-16T03:37:16.785 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6349153589999998, 2024-09-16T03:37:16.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098624421000000004, 2024-09-16T03:37:16.785 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.785 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.785 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.785 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.099182+0000", 2024-09-16T03:37:16.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.099182+0000", 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.099183+0000", 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3600000000000002e-07 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.099185+0000", 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9459999999999998e-06 2024-09-16T03:37:16.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.099193+0000", 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.099197+0000", 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5870000000000001e-06 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197793+0000", 2024-09-16T03:37:16.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098596707000000006 2024-09-16T03:37:16.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197797+0000", 2024-09-16T03:37:16.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9439999999999998e-06 2024-09-16T03:37:16.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197806+0000", 2024-09-16T03:37:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9679999999999995e-06 2024-09-16T03:37:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:17 4.3 e39/32 4:cf22405f:::smithi15736696-5:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.105947+0000", 2024-09-16T03:37:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6281502769999996, 2024-09-16T03:37:16.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104850028, 2024-09-16T03:37:16.790 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.790 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.791 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.791 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.105947+0000", 2024-09-16T03:37:16.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.105947+0000", 2024-09-16T03:37:16.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.105947+0000", 2024-09-16T03:37:16.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:37:16.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.106721+0000", 2024-09-16T03:37:16.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077385699999999997 2024-09-16T03:37:16.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.106731+0000", 2024-09-16T03:37:16.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0137e-05 2024-09-16T03:37:16.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.106740+0000", 2024-09-16T03:37:16.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1900000000000001e-06 2024-09-16T03:37:16.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.206893+0000", 2024-09-16T03:37:16.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100152222 2024-09-16T03:37:16.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.206905+0000", 2024-09-16T03:37:16.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2087e-05 2024-09-16T03:37:16.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.210755+0000", 2024-09-16T03:37:16.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038502779999999999 2024-09-16T03:37:16.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.210797+0000", 2024-09-16T03:37:16.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1915000000000001e-05 2024-09-16T03:37:16.795 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:18 4.3 e39/32 4:cf22405f:::smithi15736696-5:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.107285+0000", 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6268115730000003, 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103915285, 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.107285+0000", 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.107285+0000", 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.107286+0000", 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6400000000000001e-07 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.107624+0000", 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000337602 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.107632+0000", 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9100000000000005e-06 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.107638+0000", 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1129999999999997e-06 2024-09-16T03:37:16.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.207615+0000", 2024-09-16T03:37:16.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099977110999999994 2024-09-16T03:37:16.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.207626+0000", 2024-09-16T03:37:16.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1017e-05 2024-09-16T03:37:16.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.211168+0000", 2024-09-16T03:37:16.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0035419800000000001 2024-09-16T03:37:16.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.211201+0000", 2024-09-16T03:37:16.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2787999999999998e-05 2024-09-16T03:37:16.801 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.801 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.801 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.801 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:19 4.3 e39/32 4:cf22405f:::smithi15736696-5:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:16.801 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.108113+0000", 2024-09-16T03:37:16.801 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6259845229999996, 2024-09-16T03:37:16.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103128811, 2024-09-16T03:37:16.801 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.801 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.801 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.801 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.108113+0000", 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.108113+0000", 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.108113+0000", 2024-09-16T03:37:16.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5000000000000002e-07 2024-09-16T03:37:16.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.108117+0000", 2024-09-16T03:37:16.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9700000000000001e-06 2024-09-16T03:37:16.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.108124+0000", 2024-09-16T03:37:16.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5529999999999998e-06 2024-09-16T03:37:16.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.108127+0000", 2024-09-16T03:37:16.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2049999999999998e-06 2024-09-16T03:37:16.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.208252+0000", 2024-09-16T03:37:16.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100124537 2024-09-16T03:37:16.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.208272+0000", 2024-09-16T03:37:16.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0171000000000001e-05 2024-09-16T03:37:16.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.211221+0000", 2024-09-16T03:37:16.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029492950000000002 2024-09-16T03:37:16.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.211241+0000", 2024-09-16T03:37:16.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.033e-05 2024-09-16T03:37:16.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.806 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.807 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:15 4.a e39/32 ondisk, result = 0)", 2024-09-16T03:37:16.807 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.145839+0000", 2024-09-16T03:37:16.807 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.5882584150000003, 2024-09-16T03:37:16.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.152306893, 2024-09-16T03:37:16.807 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.807 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.807 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.807 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.145839+0000", 2024-09-16T03:37:16.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.145839+0000", 2024-09-16T03:37:16.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.145839+0000", 2024-09-16T03:37:16.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0999999999999997e-07 2024-09-16T03:37:16.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.145843+0000", 2024-09-16T03:37:16.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.151e-06 2024-09-16T03:37:16.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.145853+0000", 2024-09-16T03:37:16.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0721e-05 2024-09-16T03:37:16.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.145858+0000", 2024-09-16T03:37:16.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7090000000000001e-06 2024-09-16T03:37:16.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298040+0000", 2024-09-16T03:37:16.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15218193699999999 2024-09-16T03:37:16.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298045+0000", 2024-09-16T03:37:16.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3959999999999997e-06 2024-09-16T03:37:16.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298146+0000", 2024-09-16T03:37:16.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100169 2024-09-16T03:37:16.812 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.812 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.812 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.812 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:33 4.e e39/32 4:773b934b:::smithi15736696-9:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:16.812 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.153368+0000", 2024-09-16T03:37:16.813 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.5807286270000001, 2024-09-16T03:37:16.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10211094, 2024-09-16T03:37:16.813 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.813 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.813 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.813 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153368+0000", 2024-09-16T03:37:16.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153368+0000", 2024-09-16T03:37:16.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153369+0000", 2024-09-16T03:37:16.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2399999999999999e-07 2024-09-16T03:37:16.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153898+0000", 2024-09-16T03:37:16.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052914200000000002 2024-09-16T03:37:16.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153912+0000", 2024-09-16T03:37:16.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4292e-05 2024-09-16T03:37:16.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153921+0000", 2024-09-16T03:37:16.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6510000000000007e-06 2024-09-16T03:37:16.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.254037+0000", 2024-09-16T03:37:16.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10011605699999999 2024-09-16T03:37:16.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.254049+0000", 2024-09-16T03:37:16.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2563e-05 2024-09-16T03:37:16.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255451+0000", 2024-09-16T03:37:16.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014011939999999999 2024-09-16T03:37:16.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255479+0000", 2024-09-16T03:37:16.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8717e-05 2024-09-16T03:37:16.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.818 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.818 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.818 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:34 4.e e39/32 4:773b934b:::smithi15736696-9:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:16.819 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.155077+0000", 2024-09-16T03:37:16.819 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.5790205010000005, 2024-09-16T03:37:16.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10073102, 2024-09-16T03:37:16.819 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.819 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.819 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.819 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155077+0000", 2024-09-16T03:37:16.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155077+0000", 2024-09-16T03:37:16.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155078+0000", 2024-09-16T03:37:16.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.203e-06 2024-09-16T03:37:16.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155169+0000", 2024-09-16T03:37:16.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0829999999999996e-05 2024-09-16T03:37:16.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155181+0000", 2024-09-16T03:37:16.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2489e-05 2024-09-16T03:37:16.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155188+0000", 2024-09-16T03:37:16.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3690000000000001e-06 2024-09-16T03:37:16.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.254763+0000", 2024-09-16T03:37:16.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099574489000000002 2024-09-16T03:37:16.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.254779+0000", 2024-09-16T03:37:16.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6036e-05 2024-09-16T03:37:16.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255781+0000", 2024-09-16T03:37:16.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001001844 2024-09-16T03:37:16.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255808+0000", 2024-09-16T03:37:16.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6760000000000001e-05 2024-09-16T03:37:16.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.823 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.824 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:35 4.e e39/32 4:773b934b:::smithi15736696-9:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:16.824 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.155779+0000", 2024-09-16T03:37:16.824 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.578318222, 2024-09-16T03:37:16.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100056543, 2024-09-16T03:37:16.824 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.824 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.824 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.824 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155779+0000", 2024-09-16T03:37:16.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155779+0000", 2024-09-16T03:37:16.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155780+0000", 2024-09-16T03:37:16.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.079e-06 2024-09-16T03:37:16.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155785+0000", 2024-09-16T03:37:16.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6070000000000001e-06 2024-09-16T03:37:16.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155794+0000", 2024-09-16T03:37:16.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9029999999999999e-06 2024-09-16T03:37:16.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155800+0000", 2024-09-16T03:37:16.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.682e-06 2024-09-16T03:37:16.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255313+0000", 2024-09-16T03:37:16.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099512456999999999 2024-09-16T03:37:16.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255325+0000", 2024-09-16T03:37:16.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2469999999999999e-05 2024-09-16T03:37:16.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255810+0000", 2024-09-16T03:37:16.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000485212 2024-09-16T03:37:16.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255835+0000", 2024-09-16T03:37:16.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5133000000000001e-05 2024-09-16T03:37:16.829 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.829 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.829 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.829 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:73 4.a 4:532814f2:::smithi15736696-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:16.829 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:37:16.830 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.4228372990000002, 2024-09-16T03:37:16.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.362642662, 2024-09-16T03:37:16.830 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.830 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.830 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.830 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.830 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.830 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.830 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73 2024-09-16T03:37:16.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.830 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:37:16.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:37:16.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311261+0000", 2024-09-16T03:37:16.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.562e-06 2024-09-16T03:37:16.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311925+0000", 2024-09-16T03:37:16.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066385499999999998 2024-09-16T03:37:16.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311937+0000", 2024-09-16T03:37:16.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1881e-05 2024-09-16T03:37:16.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311943+0000", 2024-09-16T03:37:16.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6999999999999996e-06 2024-09-16T03:37:16.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311988+0000", 2024-09-16T03:37:16.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5358000000000002e-05 2024-09-16T03:37:16.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312070+0000", 2024-09-16T03:37:16.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1735000000000003e-05 2024-09-16T03:37:16.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:16.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312167+0000", 2024-09-16T03:37:16.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6685e-05 2024-09-16T03:37:16.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:16.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.313661+0000", 2024-09-16T03:37:16.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001494324 2024-09-16T03:37:16.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.414759+0000", 2024-09-16T03:37:16.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10109850300000001 2024-09-16T03:37:16.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:16.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673821+0000", 2024-09-16T03:37:16.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.25906127400000001 2024-09-16T03:37:16.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673858+0000", 2024-09-16T03:37:16.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7620999999999998e-05 2024-09-16T03:37:16.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673902+0000", 2024-09-16T03:37:16.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4163999999999999e-05 2024-09-16T03:37:16.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.837 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.837 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:73 4.a e39/32 ondisk, result = 0)", 2024-09-16T03:37:16.837 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.314603+0000", 2024-09-16T03:37:16.837 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.4194940310000002, 2024-09-16T03:37:16.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100166465, 2024-09-16T03:37:16.837 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.837 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.838 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.838 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.314603+0000", 2024-09-16T03:37:16.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.314603+0000", 2024-09-16T03:37:16.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.314604+0000", 2024-09-16T03:37:16.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.265e-06 2024-09-16T03:37:16.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.314608+0000", 2024-09-16T03:37:16.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8260000000000003e-06 2024-09-16T03:37:16.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.314617+0000", 2024-09-16T03:37:16.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6619999999999994e-06 2024-09-16T03:37:16.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.314622+0000", 2024-09-16T03:37:16.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9080000000000003e-06 2024-09-16T03:37:16.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.414739+0000", 2024-09-16T03:37:16.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100117016 2024-09-16T03:37:16.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.414749+0000", 2024-09-16T03:37:16.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0736e-05 2024-09-16T03:37:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.414770+0000", 2024-09-16T03:37:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0052e-05 2024-09-16T03:37:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.4.22:4 3.3 e41/31 3:deac805c:::smithi15736696-44 low_tier:head v 41'19, mlcod=41'14)", 2024-09-16T03:37:16.842 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.902970+0000", 2024-09-16T03:37:16.843 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.83112690899999997, 2024-09-16T03:37:16.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200721028, 2024-09-16T03:37:16.843 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.843 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.843 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.843 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902970+0000", 2024-09-16T03:37:16.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902970+0000", 2024-09-16T03:37:16.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902972+0000", 2024-09-16T03:37:16.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.705e-06 2024-09-16T03:37:16.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902981+0000", 2024-09-16T03:37:16.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6740000000000003e-06 2024-09-16T03:37:16.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902995+0000", 2024-09-16T03:37:16.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4039000000000001e-05 2024-09-16T03:37:16.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.003072+0000", 2024-09-16T03:37:16.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007719599999999 2024-09-16T03:37:16.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.103239+0000", 2024-09-16T03:37:16.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100167336 2024-09-16T03:37:16.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.103253+0000", 2024-09-16T03:37:16.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-09-16T03:37:16.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.103657+0000", 2024-09-16T03:37:16.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040399499999999998 2024-09-16T03:37:16.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.103691+0000", 2024-09-16T03:37:16.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3846000000000001e-05 2024-09-16T03:37:16.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.847 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.848 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:291 4.3 e42/32 4:c8492bfc:::smithi15736696-42:head v 42'12, mlcod=40'10)", 2024-09-16T03:37:16.848 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.108219+0000", 2024-09-16T03:37:16.848 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.62587757399999999, 2024-09-16T03:37:16.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100638062, 2024-09-16T03:37:16.848 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.848 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.848 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.848 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.108219+0000", 2024-09-16T03:37:16.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.108219+0000", 2024-09-16T03:37:16.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.108220+0000", 2024-09-16T03:37:16.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8200000000000001e-07 2024-09-16T03:37:16.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.108225+0000", 2024-09-16T03:37:16.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7400000000000004e-06 2024-09-16T03:37:16.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.108232+0000", 2024-09-16T03:37:16.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9570000000000002e-06 2024-09-16T03:37:16.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.108241+0000", 2024-09-16T03:37:16.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7190000000000007e-06 2024-09-16T03:37:16.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.208366+0000", 2024-09-16T03:37:16.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100125123 2024-09-16T03:37:16.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.208383+0000", 2024-09-16T03:37:16.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6733000000000001e-05 2024-09-16T03:37:16.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.208812+0000", 2024-09-16T03:37:16.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042915199999999997 2024-09-16T03:37:16.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.208858+0000", 2024-09-16T03:37:16.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5355999999999997e-05 2024-09-16T03:37:16.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.853 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.857 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.5.21:8 3.3 e42/31 3:c38d2449:::smithi15736696-48 low_tier:head v 42'20, mlcod=41'18)", 2024-09-16T03:37:16.857 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.204890+0000", 2024-09-16T03:37:16.857 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.52920663300000004, 2024-09-16T03:37:16.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104615123, 2024-09-16T03:37:16.857 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.857 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.858 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:16.858 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.204890+0000", 2024-09-16T03:37:16.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.204890+0000", 2024-09-16T03:37:16.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.204891+0000", 2024-09-16T03:37:16.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5300000000000004e-07 2024-09-16T03:37:16.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.204895+0000", 2024-09-16T03:37:16.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2479999999999998e-06 2024-09-16T03:37:16.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.204901+0000", 2024-09-16T03:37:16.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4659999999999998e-06 2024-09-16T03:37:16.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.204904+0000", 2024-09-16T03:37:16.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6839999999999998e-06 2024-09-16T03:37:16.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.308692+0000", 2024-09-16T03:37:16.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103787855 2024-09-16T03:37:16.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.308732+0000", 2024-09-16T03:37:16.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9721999999999997e-05 2024-09-16T03:37:16.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:16.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.309471+0000", 2024-09-16T03:37:16.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073856900000000001 2024-09-16T03:37:16.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:16.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.309506+0000", 2024-09-16T03:37:16.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4925999999999997e-05 2024-09-16T03:37:16.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.862 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.862 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.862 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:16.864 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:16.862+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:16.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:37:16.866 INFO:tasks.rados.rados.0.smithi157.stdout:155: finishing set_chunk to oid 6 2024-09-16T03:37:16.866 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:37:16.866 INFO:tasks.rados.rados.0.smithi157.stdout:155: done (15 left) 2024-09-16T03:37:16.867 INFO:tasks.rados.rados.0.smithi157.stdout:159: set_chunk oid 10 target oid 10 low_tier 2024-09-16T03:37:16.867 INFO:tasks.rados.rados.0.smithi157.stdout:159: set_chunk oid 10 offset: 679424 length: 1135104 target oid smithi15736696-10 low_tier offset: 679424 snap -1 2024-09-16T03:37:16.867 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.867 INFO:tasks.rados.rados.0.smithi157.stdout:159: finishing set_chunk to oid 10 2024-09-16T03:37:16.867 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 679424 length: 1135104 2024-09-16T03:37:16.867 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:37:16.867 INFO:tasks.rados.rados.0.smithi157.stdout:159: done (15 left) 2024-09-16T03:37:16.867 INFO:tasks.rados.rados.0.smithi157.stdout:160: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:16.868 INFO:tasks.rados.rados.0.smithi157.stdout:160: set_chunk oid 11 offset: 209408 length: 1241600 target oid smithi15736696-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 209408 snap -1 2024-09-16T03:37:16.868 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.868 INFO:tasks.rados.rados.0.smithi157.stdout:160: finishing set_chunk to oid 11 2024-09-16T03:37:16.868 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 209408 length: 1241600 2024-09-16T03:37:16.868 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T03:37:16.868 INFO:tasks.rados.rados.0.smithi157.stdout:160: done (15 left) 2024-09-16T03:37:16.868 INFO:tasks.rados.rados.0.smithi157.stdout:161: set_chunk oid 12 target oid 12 low_tier 2024-09-16T03:37:16.868 INFO:tasks.rados.rados.0.smithi157.stdout:161: set_chunk oid 12 offset: 140800 length: 160768 target oid smithi15736696-12 low_tier offset: 140800 snap -1 2024-09-16T03:37:16.868 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.924 INFO:tasks.rados.rados.0.smithi157.stdout:153: finishing set_chunk to oid 4 2024-09-16T03:37:16.924 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:37:16.924 INFO:tasks.rados.rados.0.smithi157.stdout:153: done (15 left) 2024-09-16T03:37:16.924 INFO:tasks.rados.rados.0.smithi157.stdout:162: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:16.924 INFO:tasks.rados.rados.0.smithi157.stdout:162: set_chunk oid 13 offset: 549376 length: 1020416 target oid smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 549376 snap -1 2024-09-16T03:37:16.925 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.925 INFO:tasks.rados.rados.0.smithi157.stdout:156: finishing set_chunk to oid 7 2024-09-16T03:37:16.925 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T03:37:16.925 INFO:tasks.rados.rados.0.smithi157.stdout:156: done (15 left) 2024-09-16T03:37:16.925 INFO:tasks.rados.rados.0.smithi157.stdout:163: set_chunk oid 14 target oid 14 low_tier 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout:163: set_chunk oid 14 offset: 1708032 length: 227328 target oid smithi15736696-14 low_tier offset: 1708032 snap -1 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout:162: finishing set_chunk to oid 13 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 549376 length: 1020416 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout:162: done (15 left) 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout:164: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout:164: set_chunk oid 15 offset: 744960 length: 670720 target oid smithi15736696-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 744960 snap -1 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout:163: finishing set_chunk to oid 14 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 1708032 length: 227328 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout:163: done (15 left) 2024-09-16T03:37:16.926 INFO:tasks.rados.rados.0.smithi157.stdout:164 retry set_chunk ! 2024-09-16T03:37:16.927 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 15 2024-09-16T03:37:16.927 INFO:tasks.rados.rados.0.smithi157.stdout:161: finishing set_chunk to oid 12 2024-09-16T03:37:16.927 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 140800 length: 160768 2024-09-16T03:37:16.927 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:37:16.927 INFO:tasks.rados.rados.0.smithi157.stdout:161: done (14 left) 2024-09-16T03:37:16.927 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 14 2024-09-16T03:37:16.964 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:16.962+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:16.976 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:16.976 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:16.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.976 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:282 4.c 4:34492f38:::smithi15736696-33:head [assert-version v6,set-chunk in=389b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:16.977 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.398842+0000", 2024-09-16T03:37:16.977 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.5767363909999998, 2024-09-16T03:37:16.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5767614779999999, 2024-09-16T03:37:16.977 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.977 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.977 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:16.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.977 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.977 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 282 2024-09-16T03:37:16.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.977 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398842+0000", 2024-09-16T03:37:16.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398842+0000", 2024-09-16T03:37:16.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398844+0000", 2024-09-16T03:37:16.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.322e-06 2024-09-16T03:37:16.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398853+0000", 2024-09-16T03:37:16.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6999999999999997e-06 2024-09-16T03:37:16.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398870+0000", 2024-09-16T03:37:16.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6665000000000001e-05 2024-09-16T03:37:16.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398884+0000", 2024-09-16T03:37:16.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4144e-05 2024-09-16T03:37:16.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:16.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398937+0000", 2024-09-16T03:37:16.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3554999999999998e-05 2024-09-16T03:37:16.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:16.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.399000+0000", 2024-09-16T03:37:16.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2077999999999998e-05 2024-09-16T03:37:16.983 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.983 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.983 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:314 4.d 4.4da07fcd (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.922540+0000", 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.053038574999999998, 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053100361999999998, 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 314 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.984 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.922540+0000", 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.922540+0000", 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.922542+0000", 2024-09-16T03:37:16.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9889999999999999e-06 2024-09-16T03:37:16.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:16.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.922550+0000", 2024-09-16T03:37:16.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3990000000000004e-06 2024-09-16T03:37:16.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:16.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.922564+0000", 2024-09-16T03:37:16.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4109e-05 2024-09-16T03:37:16.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:16.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:16.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:16.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.922573+0000", 2024-09-16T03:37:16.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4330000000000004e-06 2024-09-16T03:37:16.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:16.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:16.987 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:37:16.987 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 2 2024-09-16T03:37:16.987 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:16.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:37:17.024 INFO:tasks.rados.rados.0.smithi157.stdout:164: finishing set_chunk to oid 15 2024-09-16T03:37:17.024 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 744960 length: 670720 2024-09-16T03:37:17.024 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:17.024 INFO:tasks.rados.rados.0.smithi157.stdout:164: done (13 left) 2024-09-16T03:37:17.024 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 13 2024-09-16T03:37:17.026 INFO:tasks.rados.rados.0.smithi157.stdout:158: finishing set_chunk to oid 9 2024-09-16T03:37:17.026 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:37:17.026 INFO:tasks.rados.rados.0.smithi157.stdout:158: done (12 left) 2024-09-16T03:37:17.027 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 12 2024-09-16T03:37:17.049 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:17.047+0000 7f8be56cd640 -1 osd.7 42 handle_osd_map injecting map crc failure 2024-09-16T03:37:17.066 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:17.064+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:17.096 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:17.096 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:17.096 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:17.096 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:17.096 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:17.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:37:17.189 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:17.187+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:17.254 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:17.254 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:17.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:17.254 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:17.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.254 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:57 4.d 4:b3fe05b2:::smithi15736696-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:17.254 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:37:17.254 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1273356339999996, 2024-09-16T03:37:17.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.205477148, 2024-09-16T03:37:17.255 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.255 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.255 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:17.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:17.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:17.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:17.255 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 57 2024-09-16T03:37:17.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.255 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:37:17.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:37:17.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124125+0000", 2024-09-16T03:37:17.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.604e-06 2024-09-16T03:37:17.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124908+0000", 2024-09-16T03:37:17.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000783086 2024-09-16T03:37:17.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124919+0000", 2024-09-16T03:37:17.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0591000000000001e-05 2024-09-16T03:37:17.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124925+0000", 2024-09-16T03:37:17.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.55e-06 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124985+0000", 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9817e-05 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.125113+0000", 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127429 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:37:17.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.125229+0000", 2024-09-16T03:37:17.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116784 2024-09-16T03:37:17.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:17.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.127169+0000", 2024-09-16T03:37:17.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001939187 2024-09-16T03:37:17.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128052+0000", 2024-09-16T03:37:17.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088291800000000001 2024-09-16T03:37:17.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329495+0000", 2024-09-16T03:37:17.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201443328 2024-09-16T03:37:17.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:17.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329535+0000", 2024-09-16T03:37:17.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9823999999999997e-05 2024-09-16T03:37:17.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329601+0000", 2024-09-16T03:37:17.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000003e-05 2024-09-16T03:37:17.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.265 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.266 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:58 4.d 4:b3fe05b2:::smithi15736696-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:17.266 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.128177+0000", 2024-09-16T03:37:17.266 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1232821370000003, 2024-09-16T03:37:17.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201511992, 2024-09-16T03:37:17.267 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.267 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.267 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:17.267 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:17.267 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:17.267 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:17.267 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 58 2024-09-16T03:37:17.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.267 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128177+0000", 2024-09-16T03:37:17.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128177+0000", 2024-09-16T03:37:17.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128178+0000", 2024-09-16T03:37:17.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0920000000000001e-06 2024-09-16T03:37:17.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128330+0000", 2024-09-16T03:37:17.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000151889 2024-09-16T03:37:17.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128339+0000", 2024-09-16T03:37:17.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9320000000000002e-06 2024-09-16T03:37:17.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128344+0000", 2024-09-16T03:37:17.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7140000000000001e-06 2024-09-16T03:37:17.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128416+0000", 2024-09-16T03:37:17.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2711000000000002e-05 2024-09-16T03:37:17.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128476+0000", 2024-09-16T03:37:17.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9682999999999998e-05 2024-09-16T03:37:17.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128556+0000", 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0167999999999995e-05 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129938+0000", 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001381969 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.228471+0000", 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098533234999999997 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329630+0000", 2024-09-16T03:37:17.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101158717 2024-09-16T03:37:17.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:17.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329664+0000", 2024-09-16T03:37:17.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3759e-05 2024-09-16T03:37:17.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329689+0000", 2024-09-16T03:37:17.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5123e-05 2024-09-16T03:37:17.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.274 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.274 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:59 4.d 4:b3fe05b2:::smithi15736696-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:17.274 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.128354+0000", 2024-09-16T03:37:17.274 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1231047829999996, 2024-09-16T03:37:17.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20143926300000001, 2024-09-16T03:37:17.274 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 59 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128354+0000", 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128354+0000", 2024-09-16T03:37:17.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128355+0000", 2024-09-16T03:37:17.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9299999999999997e-07 2024-09-16T03:37:17.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128366+0000", 2024-09-16T03:37:17.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1072e-05 2024-09-16T03:37:17.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128373+0000", 2024-09-16T03:37:17.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.268e-06 2024-09-16T03:37:17.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128379+0000", 2024-09-16T03:37:17.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6039999999999998e-06 2024-09-16T03:37:17.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128972+0000", 2024-09-16T03:37:17.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000592732 2024-09-16T03:37:17.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129024+0000", 2024-09-16T03:37:17.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2024999999999997e-05 2024-09-16T03:37:17.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:37:17.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129095+0000", 2024-09-16T03:37:17.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1371999999999994e-05 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.228461+0000", 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099365518 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.228474+0000", 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3212e-05 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329718+0000", 2024-09-16T03:37:17.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101243888 2024-09-16T03:37:17.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:17.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329748+0000", 2024-09-16T03:37:17.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9992e-05 2024-09-16T03:37:17.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329794+0000", 2024-09-16T03:37:17.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5787000000000001e-05 2024-09-16T03:37:17.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.281 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.281 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:60 4.d 4:b3fe05b2:::smithi15736696-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:17.281 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:37:17.281 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1230729300000002, 2024-09-16T03:37:17.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30164545100000001, 2024-09-16T03:37:17.281 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.281 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 60 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128387+0000", 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3599999999999999e-07 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128388+0000", 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7659999999999999e-06 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128394+0000", 2024-09-16T03:37:17.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0679999999999996e-06 2024-09-16T03:37:17.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128398+0000", 2024-09-16T03:37:17.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4220000000000002e-06 2024-09-16T03:37:17.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129762+0000", 2024-09-16T03:37:17.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001364285 2024-09-16T03:37:17.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:17.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129804+0000", 2024-09-16T03:37:17.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2058000000000003e-05 2024-09-16T03:37:17.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.429903+0000", 2024-09-16T03:37:17.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30009841900000001 2024-09-16T03:37:17.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.429952+0000", 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9066000000000001e-05 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.430032+0000", 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9931000000000004e-05 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.286 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:58 4.d e39/32 ondisk, result = 0)", 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.129714+0000", 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1217451629999999, 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022771, 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129714+0000", 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129714+0000", 2024-09-16T03:37:17.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129715+0000", 2024-09-16T03:37:17.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1299999999999999e-07 2024-09-16T03:37:17.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129718+0000", 2024-09-16T03:37:17.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9780000000000001e-06 2024-09-16T03:37:17.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129725+0000", 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6420000000000007e-06 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129729+0000", 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.98e-06 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129932+0000", 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020234300000000001 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129935+0000", 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6040000000000001e-06 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129942+0000", 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3500000000000002e-06 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.290 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:59 4.d e39/32 ondisk, result = 0)", 2024-09-16T03:37:17.291 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.129893+0000", 2024-09-16T03:37:17.291 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1215663039999999, 2024-09-16T03:37:17.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098572227999999998, 2024-09-16T03:37:17.291 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.291 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.291 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:17.291 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129893+0000", 2024-09-16T03:37:17.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129893+0000", 2024-09-16T03:37:17.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129894+0000", 2024-09-16T03:37:17.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5600000000000005e-07 2024-09-16T03:37:17.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129896+0000", 2024-09-16T03:37:17.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.52e-06 2024-09-16T03:37:17.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129902+0000", 2024-09-16T03:37:17.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1310000000000003e-06 2024-09-16T03:37:17.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129906+0000", 2024-09-16T03:37:17.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7390000000000001e-06 2024-09-16T03:37:17.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.228451+0000", 2024-09-16T03:37:17.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098545196000000002 2024-09-16T03:37:17.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.228457+0000", 2024-09-16T03:37:17.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6149999999999996e-06 2024-09-16T03:37:17.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.228465+0000", 2024-09-16T03:37:17.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.371e-06 2024-09-16T03:37:17.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.296 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.296 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:57 4.d e39/32 ondisk, result = 0)", 2024-09-16T03:37:17.297 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.329271+0000", 2024-09-16T03:37:17.297 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.922187718, 2024-09-16T03:37:17.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000335543, 2024-09-16T03:37:17.297 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.297 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.297 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:17.297 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329271+0000", 2024-09-16T03:37:17.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329271+0000", 2024-09-16T03:37:17.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329275+0000", 2024-09-16T03:37:17.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.783e-06 2024-09-16T03:37:17.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329288+0000", 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.223e-05 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329318+0000", 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0589999999999997e-05 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329344+0000", 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5432999999999999e-05 2024-09-16T03:37:17.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329483+0000", 2024-09-16T03:37:17.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013974199999999999 2024-09-16T03:37:17.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329491+0000", 2024-09-16T03:37:17.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5410000000000003e-06 2024-09-16T03:37:17.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329607+0000", 2024-09-16T03:37:17.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116224 2024-09-16T03:37:17.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.301 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.301 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:81 4.c 4:3313e97c:::smithi15736696-21:head [write 736140~537686,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:17.301 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:37:17.301 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8538329060000001, 2024-09-16T03:37:17.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203525015, 2024-09-16T03:37:17.302 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.302 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.302 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:17.302 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:17.302 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:17.302 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:17.302 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81 2024-09-16T03:37:17.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.302 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397629+0000", 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1889999999999999e-06 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398085+0000", 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00045514200000000001 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398100+0000", 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5637999999999998e-05 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398110+0000", 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6549999999999999e-06 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398190+0000", 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0272999999999995e-05 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398339+0000", 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014924200000000001 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398504+0000", 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016471500000000001 2024-09-16T03:37:17.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.400267+0000", 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017625169999999999 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.401104+0000", 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083686399999999999 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601084+0000", 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199980397 2024-09-16T03:37:17.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:17.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601114+0000", 2024-09-16T03:37:17.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0484000000000002e-05 2024-09-16T03:37:17.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601151+0000", 2024-09-16T03:37:17.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6899e-05 2024-09-16T03:37:17.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.308 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.308 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:82 4.c 4:3313e97c:::smithi15736696-21:head [write 2020581~315559] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:17.308 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.403482+0000", 2024-09-16T03:37:17.308 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8479772890000001, 2024-09-16T03:37:17.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197763785, 2024-09-16T03:37:17.308 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.308 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.308 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:17.308 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403482+0000", 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403482+0000", 2024-09-16T03:37:17.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403484+0000", 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6640000000000001e-06 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403791+0000", 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000307287 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403802+0000", 2024-09-16T03:37:17.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1482e-05 2024-09-16T03:37:17.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403808+0000", 2024-09-16T03:37:17.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7049999999999996e-06 2024-09-16T03:37:17.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403860+0000", 2024-09-16T03:37:17.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1678999999999998e-05 2024-09-16T03:37:17.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403965+0000", 2024-09-16T03:37:17.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105 2024-09-16T03:37:17.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:17.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.404111+0000", 2024-09-16T03:37:17.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014622600000000001 2024-09-16T03:37:17.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.503942+0000", 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099831195999999997 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.503956+0000", 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3363000000000001e-05 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601187+0000", 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097231834000000003 2024-09-16T03:37:17.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:17.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601216+0000", 2024-09-16T03:37:17.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8986e-05 2024-09-16T03:37:17.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601246+0000", 2024-09-16T03:37:17.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9363e-05 2024-09-16T03:37:17.316 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.316 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.316 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.316 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:83 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 2336140] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:17.316 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:37:17.316 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8476381110000002, 2024-09-16T03:37:17.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29747773399999999, 2024-09-16T03:37:17.316 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.316 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.317 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:17.317 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:17.317 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:17.317 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:17.317 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-09-16T03:37:17.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.317 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:37:17.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403822+0000", 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-07 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403825+0000", 2024-09-16T03:37:17.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000001e-06 2024-09-16T03:37:17.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403831+0000", 2024-09-16T03:37:17.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9399999999999999e-06 2024-09-16T03:37:17.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403834+0000", 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1939999999999998e-06 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.404994+0000", 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011601249999999999 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405083+0000", 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8809999999999998e-05 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:17.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405217+0000", 2024-09-16T03:37:17.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013401900000000001 2024-09-16T03:37:17.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:17.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.503958+0000", 2024-09-16T03:37:17.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098741342999999995 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.506033+0000", 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020749660000000001 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701227+0000", 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.195194171 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:17.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701262+0000", 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4768999999999999e-05 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701299+0000", 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6884000000000001e-05 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:84 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:37:17.323 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8476063859999998, 2024-09-16T03:37:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29760038700000002, 2024-09-16T03:37:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-09-16T03:37:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:37:17.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:37:17.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:37:17.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.58e-07 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403861+0000", 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7689999999999996e-06 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403866+0000", 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6330000000000004e-06 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403869+0000", 2024-09-16T03:37:17.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7390000000000001e-06 2024-09-16T03:37:17.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405790+0000", 2024-09-16T03:37:17.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001920351 2024-09-16T03:37:17.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:17.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405865+0000", 2024-09-16T03:37:17.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5234999999999994e-05 2024-09-16T03:37:17.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701328+0000", 2024-09-16T03:37:17.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29546253300000003 2024-09-16T03:37:17.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701382+0000", 2024-09-16T03:37:17.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4481999999999998e-05 2024-09-16T03:37:17.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701453+0000", 2024-09-16T03:37:17.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1186999999999995e-05 2024-09-16T03:37:17.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.329 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.330 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:82 4.c e39/32 ondisk, result = 0)", 2024-09-16T03:37:17.330 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.405724+0000", 2024-09-16T03:37:17.330 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8457351849999997, 2024-09-16T03:37:17.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098225404000000002, 2024-09-16T03:37:17.330 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405724+0000", 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405724+0000", 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405725+0000", 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3399999999999997e-07 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405729+0000", 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3470000000000001e-06 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405738+0000", 2024-09-16T03:37:17.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.225e-06 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405742+0000", 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7400000000000004e-06 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.503931+0000", 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098189188999999996 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.503938+0000", 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.156e-06 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.503949+0000", 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1813e-05 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:83 4.c e39/32 ondisk, result = 0)", 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.405905+0000", 2024-09-16T03:37:17.334 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8455544640000001, 2024-09-16T03:37:17.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100136081, 2024-09-16T03:37:17.335 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.335 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:17.335 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405905+0000", 2024-09-16T03:37:17.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405905+0000", 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405906+0000", 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5499999999999997e-07 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405908+0000", 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5749999999999999e-06 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405915+0000", 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405919+0000", 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8039999999999999e-06 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.506020+0000", 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10010116400000001 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.506027+0000", 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3880000000000002e-06 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.506041+0000", 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3713000000000001e-05 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.338 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:81 4.c e39/32 ondisk, result = 0)", 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.500927+0000", 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.7505321800000004, 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10023235799999999, 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.500927+0000", 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.500927+0000", 2024-09-16T03:37:17.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.500928+0000", 2024-09-16T03:37:17.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4339999999999999e-06 2024-09-16T03:37:17.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.500934+0000", 2024-09-16T03:37:17.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7420000000000003e-06 2024-09-16T03:37:17.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.500947+0000", 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3217000000000001e-05 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601020+0000", 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100072378 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601074+0000", 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4527999999999997e-05 2024-09-16T03:37:17.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601080+0000", 2024-09-16T03:37:17.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1369999999999999e-06 2024-09-16T03:37:17.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601159+0000", 2024-09-16T03:37:17.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8922000000000001e-05 2024-09-16T03:37:17.342 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.342 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.343 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:82 4.c e39/32 ondisk, result = 0)", 2024-09-16T03:37:17.343 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.601051+0000", 2024-09-16T03:37:17.343 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6504078629999999, 2024-09-16T03:37:17.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020286600000000001, 2024-09-16T03:37:17.343 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601051+0000", 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601051+0000", 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601053+0000", 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.454e-06 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601059+0000", 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0299999999999999e-06 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601068+0000", 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4960000000000002e-06 2024-09-16T03:37:17.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601075+0000", 2024-09-16T03:37:17.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8299999999999998e-06 2024-09-16T03:37:17.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601178+0000", 2024-09-16T03:37:17.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102972 2024-09-16T03:37:17.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601183+0000", 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8969999999999999e-06 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601254+0000", 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1186999999999995e-05 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:83 4.c e39/32 ondisk, result = 0)", 2024-09-16T03:37:17.347 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.601086+0000", 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6503733250000003, 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100218748, 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601086+0000", 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601086+0000", 2024-09-16T03:37:17.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601086+0000", 2024-09-16T03:37:17.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6600000000000002e-07 2024-09-16T03:37:17.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601089+0000", 2024-09-16T03:37:17.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4279999999999999e-06 2024-09-16T03:37:17.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601094+0000", 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6160000000000001e-06 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701166+0000", 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100071271 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701218+0000", 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2364000000000002e-05 2024-09-16T03:37:17.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701224+0000", 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6280000000000002e-06 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701305+0000", 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0975000000000003e-05 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:129 4.c 4:34492f38:::smithi15736696-33:head [write 537824~473380,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.719747+0000", 2024-09-16T03:37:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.5317123989999999, 2024-09-16T03:37:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10914728899999999, 2024-09-16T03:37:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 129 2024-09-16T03:37:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.352 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.719747+0000", 2024-09-16T03:37:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.719747+0000", 2024-09-16T03:37:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.719750+0000", 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2729999999999998e-06 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.720187+0000", 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043690100000000001 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.720198+0000", 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1242e-05 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.720208+0000", 2024-09-16T03:37:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3700000000000001e-06 2024-09-16T03:37:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.720263+0000", 2024-09-16T03:37:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5268e-05 2024-09-16T03:37:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.720364+0000", 2024-09-16T03:37:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010131100000000001 2024-09-16T03:37:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.720468+0000", 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103853 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.721631+0000", 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00116289 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.822884+0000", 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10125332300000001 2024-09-16T03:37:17.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828818+0000", 2024-09-16T03:37:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059341020000000001 2024-09-16T03:37:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828851+0000", 2024-09-16T03:37:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2880999999999999e-05 2024-09-16T03:37:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828894+0000", 2024-09-16T03:37:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2874999999999999e-05 2024-09-16T03:37:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:129 4.c e39/32 ondisk, result = 0)", 2024-09-16T03:37:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.722655+0000", 2024-09-16T03:37:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.5288040360000004, 2024-09-16T03:37:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100235244, 2024-09-16T03:37:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.358 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.722655+0000", 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.722655+0000", 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.722656+0000", 2024-09-16T03:37:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.057e-06 2024-09-16T03:37:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.722660+0000", 2024-09-16T03:37:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4580000000000002e-06 2024-09-16T03:37:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.722668+0000", 2024-09-16T03:37:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6139999999999992e-06 2024-09-16T03:37:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.722675+0000", 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.822867+0000", 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100192325 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.822880+0000", 2024-09-16T03:37:17.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3033e-05 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.822890+0000", 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0030999999999999e-05 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:129 4.c e39/32 ondisk, result = 0)", 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.828700+0000", 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.4227595160000002, 2024-09-16T03:37:17.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000201497, 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828700+0000", 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828700+0000", 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828701+0000", 2024-09-16T03:37:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.02e-06 2024-09-16T03:37:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828704+0000", 2024-09-16T03:37:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3790000000000002e-06 2024-09-16T03:37:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828713+0000", 2024-09-16T03:37:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5250000000000005e-06 2024-09-16T03:37:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828719+0000", 2024-09-16T03:37:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:37:17.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828809+0000", 2024-09-16T03:37:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9726000000000001e-05 2024-09-16T03:37:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828813+0000", 2024-09-16T03:37:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9990000000000002e-06 2024-09-16T03:37:17.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828901+0000", 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8671000000000005e-05 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:314 4.d 4:b3fe05b2:::smithi15736696-15:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.922540+0000", 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.328919342, 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10032601400000001, 2024-09-16T03:37:17.367 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 314 2024-09-16T03:37:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.922540+0000", 2024-09-16T03:37:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.922540+0000", 2024-09-16T03:37:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.922542+0000", 2024-09-16T03:37:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9889999999999999e-06 2024-09-16T03:37:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:17.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.922550+0000", 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3990000000000004e-06 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.922564+0000", 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4109e-05 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.922573+0000", 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4330000000000004e-06 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:17.022732+0000", 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100159529 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:17.022806+0000", 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3849000000000004e-05 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:17.022866+0000", 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9706000000000001e-05 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.371 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:17.372 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:17.372 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:17.373 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:37:17.374 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:17.371+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:17.388 INFO:tasks.rados.rados.0.smithi157.stdout:118: finishing set_chunk to oid 19 2024-09-16T03:37:17.388 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T03:37:17.388 INFO:tasks.rados.rados.0.smithi157.stdout:118: done (11 left) 2024-09-16T03:37:17.388 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 11 2024-09-16T03:37:17.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:37:17.474 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:17.472+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:17.478 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:17.478 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:17.478 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:17.478 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:17.487 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:37:17.537 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:17.538 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:17.555 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:17.555 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:17.572 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:17.572 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:17.574 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:17.573+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:17.589 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:17.589 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:17.591 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:17.591 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:17.600 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:37:17.606 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:17.606 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:17.623 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:17.624 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:17.639 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:17.640 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:17.656 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:17.657 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:17.672 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:17.673 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:17.678 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:17.676+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:17.689 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:17.689 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:17.690 INFO:tasks.rados.rados.0.smithi157.stdout:122: finishing set_chunk to oid 23 2024-09-16T03:37:17.690 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T03:37:17.690 INFO:tasks.rados.rados.0.smithi157.stdout:122: done (10 left) 2024-09-16T03:37:17.690 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 10 2024-09-16T03:37:17.692 INFO:tasks.rados.rados.0.smithi157.stdout:115: finishing set_chunk to oid 16 2024-09-16T03:37:17.692 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:37:17.692 INFO:tasks.rados.rados.0.smithi157.stdout:115: done (9 left) 2024-09-16T03:37:17.692 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 9 2024-09-16T03:37:17.705 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:17.705 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:17.706 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:17.706 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:17.714 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:37:17.724 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:17.724 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:17.740 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:17.740 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:17.756 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:17.757 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:17.773 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:17.774 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:17.778 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:17.777+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:17.790 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:17.791 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:17.819 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:17.819 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:17.828 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:37:17.881 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:17.880+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:17.891 INFO:tasks.rados.rados.0.smithi157.stdout:128: finishing set_chunk to oid 29 2024-09-16T03:37:17.891 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:37:17.891 INFO:tasks.rados.rados.0.smithi157.stdout:128: done (8 left) 2024-09-16T03:37:17.891 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 8 2024-09-16T03:37:17.893 INFO:tasks.rados.rados.0.smithi157.stdout:138: finishing set_chunk to oid 39 2024-09-16T03:37:17.893 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T03:37:17.893 INFO:tasks.rados.rados.0.smithi157.stdout:138: done (7 left) 2024-09-16T03:37:17.893 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 7 2024-09-16T03:37:17.894 INFO:tasks.rados.rados.0.smithi157.stdout:132: finishing set_chunk to oid 33 2024-09-16T03:37:17.894 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:37:17.894 INFO:tasks.rados.rados.0.smithi157.stdout:132: done (6 left) 2024-09-16T03:37:17.894 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 6 2024-09-16T03:37:17.930 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:17.930 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:17.938 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:37:17.985 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:17.979+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:17.993 INFO:tasks.rados.rados.0.smithi157.stdout:139: finishing set_chunk to oid 40 2024-09-16T03:37:17.993 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 15 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T03:37:17.993 INFO:tasks.rados.rados.0.smithi157.stdout:139: done (5 left) 2024-09-16T03:37:17.993 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 5 2024-09-16T03:37:17.993 INFO:tasks.rados.rados.0.smithi157.stdout:123: finishing set_chunk to oid 24 2024-09-16T03:37:17.993 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T03:37:17.993 INFO:tasks.rados.rados.0.smithi157.stdout:123: done (4 left) 2024-09-16T03:37:17.993 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 4 2024-09-16T03:37:17.993 INFO:tasks.rados.rados.0.smithi157.stdout:133: finishing set_chunk to oid 34 2024-09-16T03:37:17.993 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 15 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T03:37:17.994 INFO:tasks.rados.rados.0.smithi157.stdout:133: done (3 left) 2024-09-16T03:37:17.994 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 3 2024-09-16T03:37:17.996 INFO:tasks.rados.rados.0.smithi157.stdout:146: finishing set_chunk to oid 47 2024-09-16T03:37:17.996 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T03:37:17.996 INFO:tasks.rados.rados.0.smithi157.stdout:146: done (2 left) 2024-09-16T03:37:17.996 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 2 2024-09-16T03:37:17.997 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:37:18.042 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:18.042 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:18.051 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:37:18.083 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:18.081+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:18.174 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:18.172+0000 7eff7884b640 -1 osd.3 43 handle_osd_map injecting map crc failure 2024-09-16T03:37:18.175 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:18.173+0000 7eff7884b640 -1 osd.3 43 handle_osd_map injecting map crc failure 2024-09-16T03:37:18.183 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:18.181+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:18.194 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:18.194 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:18.195 INFO:tasks.rados.rados.0.smithi157.stdout:135: finishing set_chunk to oid 36 2024-09-16T03:37:18.195 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 18 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:37:18.195 INFO:tasks.rados.rados.0.smithi157.stdout:135: done (1 left) 2024-09-16T03:37:18.195 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 1 2024-09-16T03:37:18.203 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:37:18.284 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:18.283+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:18.300 INFO:tasks.rados.rados.0.smithi157.stdout:148: finishing set_chunk to oid 49 2024-09-16T03:37:18.300 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T03:37:18.300 INFO:tasks.rados.rados.0.smithi157.stdout:148: done (0 left) 2024-09-16T03:37:18.300 INFO:tasks.rados.rados.0.smithi157.stdout:165: set_chunk oid 16 target oid 16 low_tier 2024-09-16T03:37:18.300 INFO:tasks.rados.rados.0.smithi157.stdout:165: set_chunk oid 16 offset: 1347072 length: 205824 target oid smithi15736696-16 low_tier offset: 1347072 snap -1 2024-09-16T03:37:18.301 INFO:tasks.rados.rados.0.smithi157.stdout:166: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.301 INFO:tasks.rados.rados.0.smithi157.stdout:166: set_chunk oid 17 offset: 630784 length: 1114112 target oid smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 630784 snap -1 2024-09-16T03:37:18.301 INFO:tasks.rados.rados.0.smithi157.stdout:167: set_chunk oid 18 target oid 18 low_tier 2024-09-16T03:37:18.301 INFO:tasks.rados.rados.0.smithi157.stdout:167: set_chunk oid 18 offset: 1237504 length: 858624 target oid smithi15736696-18 low_tier offset: 1237504 snap -1 2024-09-16T03:37:18.301 INFO:tasks.rados.rados.0.smithi157.stdout:168: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.301 INFO:tasks.rados.rados.0.smithi157.stdout:168: set_chunk oid 19 offset: 716288 length: 474624 target oid smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 716288 snap -1 2024-09-16T03:37:18.301 INFO:tasks.rados.rados.0.smithi157.stdout:169: set_chunk oid 20 target oid 20 low_tier 2024-09-16T03:37:18.301 INFO:tasks.rados.rados.0.smithi157.stdout:169: set_chunk oid 20 offset: 1180160 length: 737792 target oid smithi15736696-20 low_tier offset: 1180160 snap -1 2024-09-16T03:37:18.301 INFO:tasks.rados.rados.0.smithi157.stdout:170: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.301 INFO:tasks.rados.rados.0.smithi157.stdout:170: set_chunk oid 21 offset: 40448 length: 1603584 target oid smithi15736696-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 40448 snap -1 2024-09-16T03:37:18.301 INFO:tasks.rados.rados.0.smithi157.stdout:171: set_chunk oid 22 target oid 22 low_tier 2024-09-16T03:37:18.301 INFO:tasks.rados.rados.0.smithi157.stdout:171: set_chunk oid 22 offset: 228864 length: 1055232 target oid smithi15736696-22 low_tier offset: 228864 snap -1 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:172: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:172: set_chunk oid 23 offset: 369152 length: 826880 target oid smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 369152 snap -1 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:173: set_chunk oid 24 target oid 24 low_tier 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:173: set_chunk oid 24 offset: 845824 length: 704512 target oid smithi15736696-24 low_tier offset: 845824 snap -1 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:174: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:174: set_chunk oid 25 offset: 123904 length: 2000896 target oid smithi15736696-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 123904 snap -1 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:175: set_chunk oid 26 target oid 26 low_tier 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:175: set_chunk oid 26 offset: 321536 length: 1422848 target oid smithi15736696-26 low_tier offset: 321536 snap -1 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:176: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:176: set_chunk oid 27 offset: 2230784 length: 141312 target oid smithi15736696-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2230784 snap -1 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:177: set_chunk oid 28 target oid 28 low_tier 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:177: set_chunk oid 28 offset: 642048 length: 192512 target oid smithi15736696-28 low_tier offset: 642048 snap -1 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:178: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.302 INFO:tasks.rados.rados.0.smithi157.stdout:178: set_chunk oid 29 offset: 551424 length: 1443328 target oid smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 551424 snap -1 2024-09-16T03:37:18.303 INFO:tasks.rados.rados.0.smithi157.stdout:179: set_chunk oid 30 target oid 30 low_tier 2024-09-16T03:37:18.303 INFO:tasks.rados.rados.0.smithi157.stdout:179: set_chunk oid 30 offset: 680448 length: 1135104 target oid smithi15736696-30 low_tier offset: 680448 snap -1 2024-09-16T03:37:18.303 INFO:tasks.rados.rados.0.smithi157.stdout:180: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.303 INFO:tasks.rados.rados.0.smithi157.stdout:180: set_chunk oid 31 offset: 930816 length: 1506304 target oid smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 930816 snap -1 2024-09-16T03:37:18.303 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.303 INFO:tasks.rados.rados.0.smithi157.stdout:169: finishing set_chunk to oid 20 2024-09-16T03:37:18.303 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1180160 length: 737792 2024-09-16T03:37:18.303 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T03:37:18.303 INFO:tasks.rados.rados.0.smithi157.stdout:172: finishing set_chunk to oid 23 2024-09-16T03:37:18.303 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 369152 length: 826880 2024-09-16T03:37:18.303 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T03:37:18.303 INFO:tasks.rados.rados.0.smithi157.stdout:179: finishing set_chunk to oid 30 2024-09-16T03:37:18.303 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 680448 length: 1135104 2024-09-16T03:37:18.304 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:37:18.304 INFO:tasks.rados.rados.0.smithi157.stdout:169: done (15 left) 2024-09-16T03:37:18.304 INFO:tasks.rados.rados.0.smithi157.stdout:172: done (14 left) 2024-09-16T03:37:18.304 INFO:tasks.rados.rados.0.smithi157.stdout:179: done (13 left) 2024-09-16T03:37:18.304 INFO:tasks.rados.rados.0.smithi157.stdout:181: set_chunk oid 32 target oid 32 low_tier 2024-09-16T03:37:18.304 INFO:tasks.rados.rados.0.smithi157.stdout:181: set_chunk oid 32 offset: 1292800 length: 897024 target oid smithi15736696-32 low_tier offset: 1292800 snap -1 2024-09-16T03:37:18.304 INFO:tasks.rados.rados.0.smithi157.stdout:182: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.304 INFO:tasks.rados.rados.0.smithi157.stdout:182: set_chunk oid 33 offset: 1534464 length: 244736 target oid smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1534464 snap -1 2024-09-16T03:37:18.304 INFO:tasks.rados.rados.0.smithi157.stdout:183: set_chunk oid 34 target oid 34 low_tier 2024-09-16T03:37:18.304 INFO:tasks.rados.rados.0.smithi157.stdout:183: set_chunk oid 34 offset: 769024 length: 984576 target oid smithi15736696-34 low_tier offset: 769024 snap -1 2024-09-16T03:37:18.304 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.304 INFO:tasks.rados.rados.0.smithi157.stdout:177: finishing set_chunk to oid 28 2024-09-16T03:37:18.304 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 642048 length: 192512 2024-09-16T03:37:18.305 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T03:37:18.305 INFO:tasks.rados.rados.0.smithi157.stdout:177: done (15 left) 2024-09-16T03:37:18.305 INFO:tasks.rados.rados.0.smithi157.stdout:184: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.305 INFO:tasks.rados.rados.0.smithi157.stdout:184: set_chunk oid 35 offset: 1213952 length: 868864 target oid smithi15736696-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1213952 snap -1 2024-09-16T03:37:18.305 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.305 INFO:tasks.rados.rados.0.smithi157.stdout:166: finishing set_chunk to oid 17 2024-09-16T03:37:18.305 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 630784 length: 1114112 2024-09-16T03:37:18.305 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T03:37:18.305 INFO:tasks.rados.rados.0.smithi157.stdout:171: finishing set_chunk to oid 22 2024-09-16T03:37:18.305 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 228864 length: 1055232 2024-09-16T03:37:18.305 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T03:37:18.305 INFO:tasks.rados.rados.0.smithi157.stdout:180: finishing set_chunk to oid 31 2024-09-16T03:37:18.305 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 930816 length: 1506304 2024-09-16T03:37:18.306 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:37:18.306 INFO:tasks.rados.rados.0.smithi157.stdout:167: finishing set_chunk to oid 18 2024-09-16T03:37:18.306 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1237504 length: 858624 2024-09-16T03:37:18.306 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T03:37:18.306 INFO:tasks.rados.rados.0.smithi157.stdout:166: done (15 left) 2024-09-16T03:37:18.306 INFO:tasks.rados.rados.0.smithi157.stdout:167: done (14 left) 2024-09-16T03:37:18.306 INFO:tasks.rados.rados.0.smithi157.stdout:171: done (13 left) 2024-09-16T03:37:18.306 INFO:tasks.rados.rados.0.smithi157.stdout:180: done (12 left) 2024-09-16T03:37:18.306 INFO:tasks.rados.rados.0.smithi157.stdout:185: set_chunk oid 36 target oid 36 low_tier 2024-09-16T03:37:18.306 INFO:tasks.rados.rados.0.smithi157.stdout:185: set_chunk oid 36 offset: 965120 length: 773632 target oid smithi15736696-36 low_tier offset: 965120 snap -1 2024-09-16T03:37:18.306 INFO:tasks.rados.rados.0.smithi157.stdout:186: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.306 INFO:tasks.rados.rados.0.smithi157.stdout:186: set_chunk oid 37 offset: 1193984 length: 294400 target oid smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1193984 snap -1 2024-09-16T03:37:18.306 INFO:tasks.rados.rados.0.smithi157.stdout:187: set_chunk oid 38 target oid 38 low_tier 2024-09-16T03:37:18.307 INFO:tasks.rados.rados.0.smithi157.stdout:187: set_chunk oid 38 offset: 757248 length: 268288 target oid smithi15736696-38 low_tier offset: 757248 snap -1 2024-09-16T03:37:18.307 INFO:tasks.rados.rados.0.smithi157.stdout:188: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.307 INFO:tasks.rados.rados.0.smithi157.stdout:188: set_chunk oid 39 offset: 30208 length: 2283520 target oid smithi15736696-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 30208 snap -1 2024-09-16T03:37:18.307 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.307 INFO:tasks.rados.rados.0.smithi157.stdout:174: finishing set_chunk to oid 25 2024-09-16T03:37:18.307 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 123904 length: 2000896 2024-09-16T03:37:18.307 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:18.307 INFO:tasks.rados.rados.0.smithi157.stdout:178: finishing set_chunk to oid 29 2024-09-16T03:37:18.307 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 551424 length: 1443328 2024-09-16T03:37:18.307 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:37:18.307 INFO:tasks.rados.rados.0.smithi157.stdout:174: done (15 left) 2024-09-16T03:37:18.307 INFO:tasks.rados.rados.0.smithi157.stdout:178: done (14 left) 2024-09-16T03:37:18.307 INFO:tasks.rados.rados.0.smithi157.stdout:189: set_chunk oid 40 target oid 40 low_tier 2024-09-16T03:37:18.308 INFO:tasks.rados.rados.0.smithi157.stdout:189: set_chunk oid 40 offset: 908800 length: 1651200 target oid smithi15736696-40 low_tier offset: 908800 snap -1 2024-09-16T03:37:18.308 INFO:tasks.rados.rados.0.smithi157.stdout:190: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.308 INFO:tasks.rados.rados.0.smithi157.stdout:190: set_chunk oid 41 offset: 122368 length: 2205696 target oid smithi15736696-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 122368 snap -1 2024-09-16T03:37:18.308 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.308 INFO:tasks.rados.rados.0.smithi157.stdout:189: finishing set_chunk to oid 40 2024-09-16T03:37:18.308 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 908800 length: 1651200 2024-09-16T03:37:18.308 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T03:37:18.308 INFO:tasks.rados.rados.0.smithi157.stdout:188: finishing set_chunk to oid 39 2024-09-16T03:37:18.308 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 30208 length: 2283520 2024-09-16T03:37:18.308 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T03:37:18.308 INFO:tasks.rados.rados.0.smithi157.stdout:188: done (15 left) 2024-09-16T03:37:18.308 INFO:tasks.rados.rados.0.smithi157.stdout:189: done (14 left) 2024-09-16T03:37:18.308 INFO:tasks.rados.rados.0.smithi157.stdout:191: set_chunk oid 42 target oid 42 low_tier 2024-09-16T03:37:18.309 INFO:tasks.rados.rados.0.smithi157.stdout:191: set_chunk oid 42 offset: 669696 length: 220160 target oid smithi15736696-42 low_tier offset: 669696 snap -1 2024-09-16T03:37:18.309 INFO:tasks.rados.rados.0.smithi157.stdout:192: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.309 INFO:tasks.rados.rados.0.smithi157.stdout:192: set_chunk oid 43 offset: 1128448 length: 1395712 target oid smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1128448 snap -1 2024-09-16T03:37:18.309 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.309 INFO:tasks.rados.rados.0.smithi157.stdout:192: finishing set_chunk to oid 43 2024-09-16T03:37:18.309 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1128448 length: 1395712 2024-09-16T03:37:18.309 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T03:37:18.309 INFO:tasks.rados.rados.0.smithi157.stdout:192: done (15 left) 2024-09-16T03:37:18.309 INFO:tasks.rados.rados.0.smithi157.stdout:193: set_chunk oid 44 target oid 44 low_tier 2024-09-16T03:37:18.309 INFO:tasks.rados.rados.0.smithi157.stdout:193: set_chunk oid 44 offset: 524800 length: 953856 target oid smithi15736696-44 low_tier offset: 524800 snap -1 2024-09-16T03:37:18.309 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.310 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:18.310 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:18.311 INFO:tasks.rados.rados.0.smithi157.stdout:170: finishing set_chunk to oid 21 2024-09-16T03:37:18.311 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 40448 length: 1603584 2024-09-16T03:37:18.311 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T03:37:18.311 INFO:tasks.rados.rados.0.smithi157.stdout:170: done (15 left) 2024-09-16T03:37:18.311 INFO:tasks.rados.rados.0.smithi157.stdout:194: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.311 INFO:tasks.rados.rados.0.smithi157.stdout:194: set_chunk oid 45 offset: 752640 length: 118784 target oid smithi15736696-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 752640 snap -1 2024-09-16T03:37:18.311 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.314 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:37:18.384 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:18.383+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:18.397 INFO:tasks.rados.rados.0.smithi157.stdout:193: finishing set_chunk to oid 44 2024-09-16T03:37:18.397 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 524800 length: 953856 2024-09-16T03:37:18.397 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T03:37:18.397 INFO:tasks.rados.rados.0.smithi157.stdout:193: done (15 left) 2024-09-16T03:37:18.397 INFO:tasks.rados.rados.0.smithi157.stdout:195: set_chunk oid 46 target oid 46 low_tier 2024-09-16T03:37:18.398 INFO:tasks.rados.rados.0.smithi157.stdout:195: set_chunk oid 46 offset: 367104 length: 399872 target oid smithi15736696-46 low_tier offset: 367104 snap -1 2024-09-16T03:37:18.398 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.412 INFO:tasks.rados.rados.0.smithi157.stdout:186: finishing set_chunk to oid 37 2024-09-16T03:37:18.412 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1193984 length: 294400 2024-09-16T03:37:18.412 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T03:37:18.412 INFO:tasks.rados.rados.0.smithi157.stdout:175: finishing set_chunk to oid 26 2024-09-16T03:37:18.413 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 321536 length: 1422848 2024-09-16T03:37:18.413 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:37:18.413 INFO:tasks.rados.rados.0.smithi157.stdout:175: done (15 left) 2024-09-16T03:37:18.413 INFO:tasks.rados.rados.0.smithi157.stdout:186: done (14 left) 2024-09-16T03:37:18.413 INFO:tasks.rados.rados.0.smithi157.stdout:196: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.413 INFO:tasks.rados.rados.0.smithi157.stdout:196: set_chunk oid 47 offset: 964608 length: 670720 target oid smithi15736696-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 964608 snap -1 2024-09-16T03:37:18.413 INFO:tasks.rados.rados.0.smithi157.stdout:197: set_chunk oid 48 target oid 48 low_tier 2024-09-16T03:37:18.413 INFO:tasks.rados.rados.0.smithi157.stdout:197: set_chunk oid 48 offset: 1471488 length: 197120 target oid smithi15736696-48 low_tier offset: 1471488 snap -1 2024-09-16T03:37:18.413 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.413 INFO:tasks.rados.rados.0.smithi157.stdout:196: finishing set_chunk to oid 47 2024-09-16T03:37:18.413 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 964608 length: 670720 2024-09-16T03:37:18.413 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T03:37:18.413 INFO:tasks.rados.rados.0.smithi157.stdout:196: done (15 left) 2024-09-16T03:37:18.414 INFO:tasks.rados.rados.0.smithi157.stdout:198: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.414 INFO:tasks.rados.rados.0.smithi157.stdout:198: set_chunk oid 49 offset: 1097216 length: 1292288 target oid smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1097216 snap -1 2024-09-16T03:37:18.414 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.418 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:18.418 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:18.427 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:37:18.485 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:18.484+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:18.496 INFO:tasks.rados.rados.0.smithi157.stdout:185: finishing set_chunk to oid 36 2024-09-16T03:37:18.496 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 965120 length: 773632 2024-09-16T03:37:18.496 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:37:18.496 INFO:tasks.rados.rados.0.smithi157.stdout:185: done (15 left) 2024-09-16T03:37:18.496 INFO:tasks.rados.rados.0.smithi157.stdout:199: set_chunk oid 50 target oid 50 low_tier 2024-09-16T03:37:18.496 INFO:tasks.rados.rados.0.smithi157.stdout:199: set_chunk oid 50 offset: 2612224 length: 78336 target oid smithi15736696-50 low_tier offset: 2612224 snap -1 2024-09-16T03:37:18.497 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.513 INFO:tasks.rados.rados.0.smithi157.stdout:197: finishing set_chunk to oid 48 2024-09-16T03:37:18.513 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1471488 length: 197120 2024-09-16T03:37:18.513 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:37:18.513 INFO:tasks.rados.rados.0.smithi157.stdout:197: done (15 left) 2024-09-16T03:37:18.514 INFO:tasks.rados.rados.0.smithi157.stdout:200: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.514 INFO:tasks.rados.rados.0.smithi157.stdout:200: set_chunk oid 1 offset: 1794560 length: 34304 target oid smithi15736696-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1794560 snap -1 2024-09-16T03:37:18.514 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.514 INFO:tasks.rados.rados.0.smithi157.stdout:190: finishing set_chunk to oid 41 2024-09-16T03:37:18.514 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 122368 length: 2205696 2024-09-16T03:37:18.514 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:37:18.514 INFO:tasks.rados.rados.0.smithi157.stdout:190: done (15 left) 2024-09-16T03:37:18.514 INFO:tasks.rados.rados.0.smithi157.stdout:201: set_chunk oid 2 target oid 2 low_tier 2024-09-16T03:37:18.514 INFO:tasks.rados.rados.0.smithi157.stdout:201: set_chunk oid 2 offset: 666112 length: 190464 target oid smithi15736696-2 low_tier offset: 666112 snap -1 2024-09-16T03:37:18.514 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.530 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:18.530 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:18.538 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:37:18.586 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:18.585+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:18.601 INFO:tasks.rados.rados.0.smithi157.stdout:195: finishing set_chunk to oid 46 2024-09-16T03:37:18.601 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:37:18.601 INFO:tasks.rados.rados.0.smithi157.stdout:195: done (15 left) 2024-09-16T03:37:18.601 INFO:tasks.rados.rados.0.smithi157.stdout:202: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.601 INFO:tasks.rados.rados.0.smithi157.stdout:202: set_chunk oid 3 offset: 1014272 length: 447488 target oid smithi15736696-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1014272 snap -1 2024-09-16T03:37:18.601 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.601 INFO:tasks.rados.rados.0.smithi157.stdout:202: finishing set_chunk to oid 3 2024-09-16T03:37:18.601 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1014272 length: 447488 2024-09-16T03:37:18.602 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T03:37:18.602 INFO:tasks.rados.rados.0.smithi157.stdout:202: done (15 left) 2024-09-16T03:37:18.602 INFO:tasks.rados.rados.0.smithi157.stdout:203: set_chunk oid 4 target oid 4 low_tier 2024-09-16T03:37:18.602 INFO:tasks.rados.rados.0.smithi157.stdout:203: set_chunk oid 4 offset: 75264 length: 714240 target oid smithi15736696-4 low_tier offset: 75264 snap -1 2024-09-16T03:37:18.602 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.615 INFO:tasks.rados.rados.0.smithi157.stdout:198: finishing set_chunk to oid 49 2024-09-16T03:37:18.615 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1097216 length: 1292288 2024-09-16T03:37:18.615 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T03:37:18.616 INFO:tasks.rados.rados.0.smithi157.stdout:198: done (15 left) 2024-09-16T03:37:18.616 INFO:tasks.rados.rados.0.smithi157.stdout:204: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.616 INFO:tasks.rados.rados.0.smithi157.stdout:204: set_chunk oid 5 offset: 148992 length: 1375744 target oid smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 148992 snap -1 2024-09-16T03:37:18.616 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.616 INFO:tasks.rados.rados.0.smithi157.stdout:191: finishing set_chunk to oid 42 2024-09-16T03:37:18.616 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 669696 length: 220160 2024-09-16T03:37:18.616 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T03:37:18.617 INFO:tasks.rados.rados.0.smithi157.stdout:191: done (15 left) 2024-09-16T03:37:18.617 INFO:tasks.rados.rados.0.smithi157.stdout:205: set_chunk oid 6 target oid 6 low_tier 2024-09-16T03:37:18.617 INFO:tasks.rados.rados.0.smithi157.stdout:205: set_chunk oid 6 offset: 1520640 length: 161280 target oid smithi15736696-6 low_tier offset: 1520640 snap -1 2024-09-16T03:37:18.617 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.617 INFO:tasks.rados.rados.0.smithi157.stdout:201: finishing set_chunk to oid 2 2024-09-16T03:37:18.617 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 666112 length: 190464 2024-09-16T03:37:18.617 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T03:37:18.617 INFO:tasks.rados.rados.0.smithi157.stdout:201: done (15 left) 2024-09-16T03:37:18.617 INFO:tasks.rados.rados.0.smithi157.stdout:206: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.617 INFO:tasks.rados.rados.0.smithi157.stdout:206: set_chunk oid 7 offset: 905728 length: 416256 target oid smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 905728 snap -1 2024-09-16T03:37:18.617 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.617 INFO:tasks.rados.rados.0.smithi157.stdout:205: finishing set_chunk to oid 6 2024-09-16T03:37:18.617 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1520640 length: 161280 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout:205: done (15 left) 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout:207: set_chunk oid 8 target oid 8 low_tier 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout:207: set_chunk oid 8 offset: 1041920 length: 22016 target oid smithi15736696-8 low_tier offset: 1041920 snap -1 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout:206: finishing set_chunk to oid 7 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 905728 length: 416256 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout:206: done (15 left) 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout:208: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout:208: set_chunk oid 9 offset: 1153024 length: 531968 target oid smithi15736696-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1153024 snap -1 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout:207: finishing set_chunk to oid 8 2024-09-16T03:37:18.618 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 1041920 length: 22016 2024-09-16T03:37:18.619 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:37:18.619 INFO:tasks.rados.rados.0.smithi157.stdout:207: done (15 left) 2024-09-16T03:37:18.619 INFO:tasks.rados.rados.0.smithi157.stdout:209: set_chunk oid 10 target oid 10 low_tier 2024-09-16T03:37:18.619 INFO:tasks.rados.rados.0.smithi157.stdout:209: set_chunk oid 10 offset: 279552 length: 1377792 target oid smithi15736696-10 low_tier offset: 279552 snap -1 2024-09-16T03:37:18.619 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.641 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:18.641 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:18.650 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:37:18.693 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:18.692+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:18.703 INFO:tasks.rados.rados.0.smithi157.stdout:176: finishing set_chunk to oid 27 2024-09-16T03:37:18.703 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:18.703 INFO:tasks.rados.rados.0.smithi157.stdout:203: finishing set_chunk to oid 4 2024-09-16T03:37:18.703 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 75264 length: 714240 2024-09-16T03:37:18.703 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:37:18.703 INFO:tasks.rados.rados.0.smithi157.stdout:176: done (15 left) 2024-09-16T03:37:18.703 INFO:tasks.rados.rados.0.smithi157.stdout:203: done (14 left) 2024-09-16T03:37:18.703 INFO:tasks.rados.rados.0.smithi157.stdout:210: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.704 INFO:tasks.rados.rados.0.smithi157.stdout:210: set_chunk oid 11 offset: 861696 length: 1105408 target oid smithi15736696-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 861696 snap -1 2024-09-16T03:37:18.704 INFO:tasks.rados.rados.0.smithi157.stdout:211: set_chunk oid 12 target oid 12 low_tier 2024-09-16T03:37:18.704 INFO:tasks.rados.rados.0.smithi157.stdout:211: set_chunk oid 12 offset: 1215488 length: 842240 target oid smithi15736696-12 low_tier offset: 1215488 snap -1 2024-09-16T03:37:18.704 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.704 INFO:tasks.rados.rados.0.smithi157.stdout:210: finishing set_chunk to oid 11 2024-09-16T03:37:18.704 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 861696 length: 1105408 2024-09-16T03:37:18.704 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T03:37:18.704 INFO:tasks.rados.rados.0.smithi157.stdout:210: done (15 left) 2024-09-16T03:37:18.704 INFO:tasks.rados.rados.0.smithi157.stdout:212: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.704 INFO:tasks.rados.rados.0.smithi157.stdout:212: set_chunk oid 13 offset: 36352 length: 1275392 target oid smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 36352 snap -1 2024-09-16T03:37:18.704 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.715 INFO:tasks.rados.rados.0.smithi157.stdout:204: finishing set_chunk to oid 5 2024-09-16T03:37:18.715 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 148992 length: 1375744 2024-09-16T03:37:18.715 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T03:37:18.716 INFO:tasks.rados.rados.0.smithi157.stdout:204: done (15 left) 2024-09-16T03:37:18.716 INFO:tasks.rados.rados.0.smithi157.stdout:213: set_chunk oid 14 target oid 14 low_tier 2024-09-16T03:37:18.716 INFO:tasks.rados.rados.0.smithi157.stdout:213: set_chunk oid 14 offset: 755712 length: 522240 target oid smithi15736696-14 low_tier offset: 755712 snap -1 2024-09-16T03:37:18.716 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.716 INFO:tasks.rados.rados.0.smithi157.stdout:208: finishing set_chunk to oid 9 2024-09-16T03:37:18.716 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1153024 length: 531968 2024-09-16T03:37:18.716 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:37:18.716 INFO:tasks.rados.rados.0.smithi157.stdout:208: done (15 left) 2024-09-16T03:37:18.716 INFO:tasks.rados.rados.0.smithi157.stdout:214: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:18.716 INFO:tasks.rados.rados.0.smithi157.stdout:214: set_chunk oid 15 offset: 429056 length: 1155072 target oid smithi15736696-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 429056 snap -1 2024-09-16T03:37:18.716 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:18.716 INFO:tasks.rados.rados.0.smithi157.stdout:209: finishing set_chunk to oid 10 2024-09-16T03:37:18.716 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 279552 length: 1377792 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout:213: finishing set_chunk to oid 14 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 755712 length: 522240 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout:209: done (15 left) 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout:213: done (14 left) 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout:214 retry set_chunk ! 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 14 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout:194: finishing set_chunk to oid 45 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 15 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout:214: finishing set_chunk to oid 15 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 429056 length: 1155072 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:18.717 INFO:tasks.rados.rados.0.smithi157.stdout:194: done (13 left) 2024-09-16T03:37:18.718 INFO:tasks.rados.rados.0.smithi157.stdout:214: done (12 left) 2024-09-16T03:37:18.718 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 12 2024-09-16T03:37:18.718 INFO:tasks.rados.rados.0.smithi157.stdout:181: finishing set_chunk to oid 32 2024-09-16T03:37:18.718 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:37:18.718 INFO:tasks.rados.rados.0.smithi157.stdout:181: done (11 left) 2024-09-16T03:37:18.718 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 11 2024-09-16T03:37:18.753 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:18.753 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:18.762 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:37:18.802 INFO:tasks.rados.rados.0.smithi157.stdout:211: finishing set_chunk to oid 12 2024-09-16T03:37:18.803 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1215488 length: 842240 2024-09-16T03:37:18.803 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:37:18.803 INFO:tasks.rados.rados.0.smithi157.stdout:212: finishing set_chunk to oid 13 2024-09-16T03:37:18.803 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 36352 length: 1275392 2024-09-16T03:37:18.803 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T03:37:18.803 INFO:tasks.rados.rados.0.smithi157.stdout:211: done (10 left) 2024-09-16T03:37:18.803 INFO:tasks.rados.rados.0.smithi157.stdout:212: done (9 left) 2024-09-16T03:37:18.803 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 9 2024-09-16T03:37:18.804 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:18.800+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:18.805 INFO:tasks.rados.rados.0.smithi157.stdout:184: finishing set_chunk to oid 35 2024-09-16T03:37:18.805 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:37:18.805 INFO:tasks.rados.rados.0.smithi157.stdout:184: done (8 left) 2024-09-16T03:37:18.805 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 8 2024-09-16T03:37:18.805 INFO:tasks.rados.rados.0.smithi157.stdout:199: finishing set_chunk to oid 50 2024-09-16T03:37:18.805 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 18 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:37:18.805 INFO:tasks.rados.rados.0.smithi157.stdout:199: done (7 left) 2024-09-16T03:37:18.805 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 7 2024-09-16T03:37:18.877 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:18.878 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:18.888 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:37:18.889 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:18.888+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:18.990 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:18.988+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:18.992 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:18.992 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:18.994 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:37:18.994 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:18.991+0000 7fedb773d640 -1 osd.9 44 handle_osd_map injecting map crc failure 2024-09-16T03:37:19.001 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:37:19.004 INFO:tasks.rados.rados.0.smithi157.stdout:200: finishing set_chunk to oid 1 2024-09-16T03:37:19.004 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T03:37:19.004 INFO:tasks.rados.rados.0.smithi157.stdout:200: done (6 left) 2024-09-16T03:37:19.004 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 6 2024-09-16T03:37:19.016 INFO:tasks.rados.rados.0.smithi157.stdout:187: finishing set_chunk to oid 38 2024-09-16T03:37:19.016 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:37:19.016 INFO:tasks.rados.rados.0.smithi157.stdout:187: done (5 left) 2024-09-16T03:37:19.016 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 5 2024-09-16T03:37:19.091 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:19.089+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:19.104 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:19.104 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:19.113 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:37:19.191 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:19.189+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:19.216 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:19.216 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:19.224 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:37:19.292 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:19.290+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:19.327 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:19.327 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:19.336 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:37:19.393 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:19.391+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:19.438 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:19.439 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:19.447 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:37:19.494 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:19.492+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:19.551 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:19.551 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:19.559 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:37:19.602 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:19.601+0000 7f66f3205640 -1 osd.15 44 handle_osd_map injecting map crc failure 2024-09-16T03:37:19.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:19.602+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:19.603 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:19.602+0000 7f4212d00640 -1 osd.8 44 handle_osd_map injecting map crc failure 2024-09-16T03:37:19.663 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:19.663 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:19.671 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:37:19.695 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:19.693+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:19.703 INFO:tasks.rados.rados.0.smithi157.stdout:182: finishing set_chunk to oid 33 2024-09-16T03:37:19.704 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:37:19.704 INFO:tasks.rados.rados.0.smithi157.stdout:182: done (4 left) 2024-09-16T03:37:19.704 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 4 2024-09-16T03:37:19.704 INFO:tasks.rados.rados.0.smithi157.stdout:168: finishing set_chunk to oid 19 2024-09-16T03:37:19.704 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T03:37:19.705 INFO:tasks.rados.rados.0.smithi157.stdout:168: done (3 left) 2024-09-16T03:37:19.705 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 3 2024-09-16T03:37:19.796 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:19.794+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:19.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:37:19.804 INFO:tasks.rados.rados.0.smithi157.stdout:173: finishing set_chunk to oid 24 2024-09-16T03:37:19.805 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T03:37:19.805 INFO:tasks.rados.rados.0.smithi157.stdout:173: done (2 left) 2024-09-16T03:37:19.805 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 2 2024-09-16T03:37:19.815 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:19.815 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:19.824 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:37:19.897 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:19.895+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:19.904 INFO:tasks.rados.rados.0.smithi157.stdout:165: finishing set_chunk to oid 16 2024-09-16T03:37:19.905 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:37:19.905 INFO:tasks.rados.rados.0.smithi157.stdout:165: done (1 left) 2024-09-16T03:37:19.905 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 1 2024-09-16T03:37:19.928 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:19.928 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:19.937 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:37:19.939 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:19.939 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:19.957 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:19.958 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:19.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:19.975 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:19.992 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:19.992 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:19.992 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:37:19.992 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:37:19.992 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:37:19.993 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-16T03:37:19.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-16T03:37:19.997 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:19.995+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:20.010 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:20.010 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:20.026 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:20.026 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:20.041 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:20.041 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:20.043 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:20.043 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:20.049 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:37:20.059 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:20.060 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:20.076 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:20.076 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:20.093 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:20.093 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:20.098 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:20.097+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:20.111 INFO:tasks.rados.rados.0.smithi157.stdout:183: finishing set_chunk to oid 34 2024-09-16T03:37:20.111 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 15 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T03:37:20.111 INFO:tasks.rados.rados.0.smithi157.stdout:183: done (0 left) 2024-09-16T03:37:20.111 INFO:tasks.rados.rados.0.smithi157.stdout:215: set_chunk oid 16 target oid 16 low_tier 2024-09-16T03:37:20.111 INFO:tasks.rados.rados.0.smithi157.stdout:215: set_chunk oid 16 offset: 822272 length: 517632 target oid smithi15736696-16 low_tier offset: 822272 snap -1 2024-09-16T03:37:20.111 INFO:tasks.rados.rados.0.smithi157.stdout:216: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.111 INFO:tasks.rados.rados.0.smithi157.stdout:216: set_chunk oid 17 offset: 1278976 length: 799232 target oid smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1278976 snap -1 2024-09-16T03:37:20.112 INFO:tasks.rados.rados.0.smithi157.stdout:217: set_chunk oid 18 target oid 18 low_tier 2024-09-16T03:37:20.112 INFO:tasks.rados.rados.0.smithi157.stdout:217: set_chunk oid 18 offset: 628736 length: 687104 target oid smithi15736696-18 low_tier offset: 628736 snap -1 2024-09-16T03:37:20.112 INFO:tasks.rados.rados.0.smithi157.stdout:218: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.112 INFO:tasks.rados.rados.0.smithi157.stdout:218: set_chunk oid 19 offset: 1532928 length: 664064 target oid smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1532928 snap -1 2024-09-16T03:37:20.112 INFO:tasks.rados.rados.0.smithi157.stdout:219: set_chunk oid 20 target oid 20 low_tier 2024-09-16T03:37:20.112 INFO:tasks.rados.rados.0.smithi157.stdout:219: set_chunk oid 20 offset: 845824 length: 930304 target oid smithi15736696-20 low_tier offset: 845824 snap -1 2024-09-16T03:37:20.112 INFO:tasks.rados.rados.0.smithi157.stdout:220: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.112 INFO:tasks.rados.rados.0.smithi157.stdout:220: set_chunk oid 21 offset: 206848 length: 1879040 target oid smithi15736696-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 206848 snap -1 2024-09-16T03:37:20.112 INFO:tasks.rados.rados.0.smithi157.stdout:221: set_chunk oid 22 target oid 22 low_tier 2024-09-16T03:37:20.112 INFO:tasks.rados.rados.0.smithi157.stdout:221: set_chunk oid 22 offset: 1859072 length: 493568 target oid smithi15736696-22 low_tier offset: 1859072 snap -1 2024-09-16T03:37:20.112 INFO:tasks.rados.rados.0.smithi157.stdout:222: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.112 INFO:tasks.rados.rados.0.smithi157.stdout:222: set_chunk oid 23 offset: 164352 length: 1216000 target oid smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 164352 snap -1 2024-09-16T03:37:20.113 INFO:tasks.rados.rados.0.smithi157.stdout:223: set_chunk oid 24 target oid 24 low_tier 2024-09-16T03:37:20.113 INFO:tasks.rados.rados.0.smithi157.stdout:223: set_chunk oid 24 offset: 12800 length: 1000448 target oid smithi15736696-24 low_tier offset: 12800 snap -1 2024-09-16T03:37:20.113 INFO:tasks.rados.rados.0.smithi157.stdout:224: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.113 INFO:tasks.rados.rados.0.smithi157.stdout:224: set_chunk oid 25 offset: 1110528 length: 1140736 target oid smithi15736696-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1110528 snap -1 2024-09-16T03:37:20.113 INFO:tasks.rados.rados.0.smithi157.stdout:225: set_chunk oid 26 target oid 26 low_tier 2024-09-16T03:37:20.113 INFO:tasks.rados.rados.0.smithi157.stdout:225: set_chunk oid 26 offset: 1139200 length: 239104 target oid smithi15736696-26 low_tier offset: 1139200 snap -1 2024-09-16T03:37:20.113 INFO:tasks.rados.rados.0.smithi157.stdout:226: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.113 INFO:tasks.rados.rados.0.smithi157.stdout:226: set_chunk oid 27 offset: 634368 length: 427008 target oid smithi15736696-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 634368 snap -1 2024-09-16T03:37:20.113 INFO:tasks.rados.rados.0.smithi157.stdout:227: set_chunk oid 28 target oid 28 low_tier 2024-09-16T03:37:20.113 INFO:tasks.rados.rados.0.smithi157.stdout:227: set_chunk oid 28 offset: 1132544 length: 211456 target oid smithi15736696-28 low_tier offset: 1132544 snap -1 2024-09-16T03:37:20.113 INFO:tasks.rados.rados.0.smithi157.stdout:228: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.113 INFO:tasks.rados.rados.0.smithi157.stdout:228: set_chunk oid 29 offset: 367104 length: 160768 target oid smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 367104 snap -1 2024-09-16T03:37:20.113 INFO:tasks.rados.rados.0.smithi157.stdout:229: set_chunk oid 30 target oid 30 low_tier 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout:229: set_chunk oid 30 offset: 5120 length: 1209344 target oid smithi15736696-30 low_tier offset: 5120 snap -1 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout:230: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout:230: set_chunk oid 31 offset: 2098176 length: 53248 target oid smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2098176 snap -1 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout:215: finishing set_chunk to oid 16 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 822272 length: 517632 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout:216: finishing set_chunk to oid 17 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1278976 length: 799232 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout:220: finishing set_chunk to oid 21 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 206848 length: 1879040 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T03:37:20.114 INFO:tasks.rados.rados.0.smithi157.stdout:215: done (15 left) 2024-09-16T03:37:20.115 INFO:tasks.rados.rados.0.smithi157.stdout:216: done (14 left) 2024-09-16T03:37:20.115 INFO:tasks.rados.rados.0.smithi157.stdout:220: done (13 left) 2024-09-16T03:37:20.115 INFO:tasks.rados.rados.0.smithi157.stdout:231: set_chunk oid 32 target oid 32 low_tier 2024-09-16T03:37:20.115 INFO:tasks.rados.rados.0.smithi157.stdout:231: set_chunk oid 32 offset: 762368 length: 469504 target oid smithi15736696-32 low_tier offset: 762368 snap -1 2024-09-16T03:37:20.115 INFO:tasks.rados.rados.0.smithi157.stdout:232: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.115 INFO:tasks.rados.rados.0.smithi157.stdout:232: set_chunk oid 33 offset: 291328 length: 2046976 target oid smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 291328 snap -1 2024-09-16T03:37:20.115 INFO:tasks.rados.rados.0.smithi157.stdout:233: set_chunk oid 34 target oid 34 low_tier 2024-09-16T03:37:20.115 INFO:tasks.rados.rados.0.smithi157.stdout:233: set_chunk oid 34 offset: 1148928 length: 961024 target oid smithi15736696-34 low_tier offset: 1148928 snap -1 2024-09-16T03:37:20.115 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.115 INFO:tasks.rados.rados.0.smithi157.stdout:218: finishing set_chunk to oid 19 2024-09-16T03:37:20.115 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1532928 length: 664064 2024-09-16T03:37:20.115 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T03:37:20.115 INFO:tasks.rados.rados.0.smithi157.stdout:222: finishing set_chunk to oid 23 2024-09-16T03:37:20.116 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 164352 length: 1216000 2024-09-16T03:37:20.116 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T03:37:20.116 INFO:tasks.rados.rados.0.smithi157.stdout:219: finishing set_chunk to oid 20 2024-09-16T03:37:20.116 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 845824 length: 930304 2024-09-16T03:37:20.117 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T03:37:20.117 INFO:tasks.rados.rados.0.smithi157.stdout:229: finishing set_chunk to oid 30 2024-09-16T03:37:20.117 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 5120 length: 1209344 2024-09-16T03:37:20.117 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:37:20.117 INFO:tasks.rados.rados.0.smithi157.stdout:223: finishing set_chunk to oid 24 2024-09-16T03:37:20.117 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 12800 length: 1000448 2024-09-16T03:37:20.117 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T03:37:20.117 INFO:tasks.rados.rados.0.smithi157.stdout:218: done (15 left) 2024-09-16T03:37:20.117 INFO:tasks.rados.rados.0.smithi157.stdout:219: done (14 left) 2024-09-16T03:37:20.117 INFO:tasks.rados.rados.0.smithi157.stdout:222: done (13 left) 2024-09-16T03:37:20.117 INFO:tasks.rados.rados.0.smithi157.stdout:223: done (12 left) 2024-09-16T03:37:20.117 INFO:tasks.rados.rados.0.smithi157.stdout:229: done (11 left) 2024-09-16T03:37:20.118 INFO:tasks.rados.rados.0.smithi157.stdout:234: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.118 INFO:tasks.rados.rados.0.smithi157.stdout:234: set_chunk oid 35 offset: 591872 length: 2048 target oid smithi15736696-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 591872 snap -1 2024-09-16T03:37:20.118 INFO:tasks.rados.rados.0.smithi157.stdout:235: set_chunk oid 36 target oid 36 low_tier 2024-09-16T03:37:20.118 INFO:tasks.rados.rados.0.smithi157.stdout:235: set_chunk oid 36 offset: 606720 length: 1724416 target oid smithi15736696-36 low_tier offset: 606720 snap -1 2024-09-16T03:37:20.118 INFO:tasks.rados.rados.0.smithi157.stdout:236: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.118 INFO:tasks.rados.rados.0.smithi157.stdout:236: set_chunk oid 37 offset: 733184 length: 1754624 target oid smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 733184 snap -1 2024-09-16T03:37:20.118 INFO:tasks.rados.rados.0.smithi157.stdout:237: set_chunk oid 38 target oid 38 low_tier 2024-09-16T03:37:20.118 INFO:tasks.rados.rados.0.smithi157.stdout:237: set_chunk oid 38 offset: 180736 length: 1139712 target oid smithi15736696-38 low_tier offset: 180736 snap -1 2024-09-16T03:37:20.118 INFO:tasks.rados.rados.0.smithi157.stdout:238: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.118 INFO:tasks.rados.rados.0.smithi157.stdout:238: set_chunk oid 39 offset: 899584 length: 114688 target oid smithi15736696-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 899584 snap -1 2024-09-16T03:37:20.118 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.118 INFO:tasks.rados.rados.0.smithi157.stdout:232: finishing set_chunk to oid 33 2024-09-16T03:37:20.118 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 291328 length: 2046976 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout:232: done (15 left) 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout:239: set_chunk oid 40 target oid 40 low_tier 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout:239: set_chunk oid 40 offset: 557568 length: 1289728 target oid smithi15736696-40 low_tier offset: 557568 snap -1 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout:236: finishing set_chunk to oid 37 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 733184 length: 1754624 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout:236: done (15 left) 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout:240: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout:240: set_chunk oid 41 offset: 1128448 length: 882176 target oid smithi15736696-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1128448 snap -1 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout:240: finishing set_chunk to oid 41 2024-09-16T03:37:20.119 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1128448 length: 882176 2024-09-16T03:37:20.120 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:37:20.120 INFO:tasks.rados.rados.0.smithi157.stdout:237: finishing set_chunk to oid 38 2024-09-16T03:37:20.120 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 180736 length: 1139712 2024-09-16T03:37:20.120 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:37:20.120 INFO:tasks.rados.rados.0.smithi157.stdout:231: finishing set_chunk to oid 32 2024-09-16T03:37:20.120 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 762368 length: 469504 2024-09-16T03:37:20.120 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:37:20.120 INFO:tasks.rados.rados.0.smithi157.stdout:231: done (15 left) 2024-09-16T03:37:20.121 INFO:tasks.rados.rados.0.smithi157.stdout:237: done (14 left) 2024-09-16T03:37:20.121 INFO:tasks.rados.rados.0.smithi157.stdout:240: done (13 left) 2024-09-16T03:37:20.121 INFO:tasks.rados.rados.0.smithi157.stdout:241: set_chunk oid 42 target oid 42 low_tier 2024-09-16T03:37:20.121 INFO:tasks.rados.rados.0.smithi157.stdout:241: set_chunk oid 42 offset: 720384 length: 210944 target oid smithi15736696-42 low_tier offset: 720384 snap -1 2024-09-16T03:37:20.121 INFO:tasks.rados.rados.0.smithi157.stdout:242: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.121 INFO:tasks.rados.rados.0.smithi157.stdout:242: set_chunk oid 43 offset: 158720 length: 1909760 target oid smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 158720 snap -1 2024-09-16T03:37:20.121 INFO:tasks.rados.rados.0.smithi157.stdout:243: set_chunk oid 44 target oid 44 low_tier 2024-09-16T03:37:20.121 INFO:tasks.rados.rados.0.smithi157.stdout:243: set_chunk oid 44 offset: 2236928 length: 10240 target oid smithi15736696-44 low_tier offset: 2236928 snap -1 2024-09-16T03:37:20.121 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.121 INFO:tasks.rados.rados.0.smithi157.stdout:241: finishing set_chunk to oid 42 2024-09-16T03:37:20.121 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 720384 length: 210944 2024-09-16T03:37:20.122 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T03:37:20.122 INFO:tasks.rados.rados.0.smithi157.stdout:241: done (15 left) 2024-09-16T03:37:20.122 INFO:tasks.rados.rados.0.smithi157.stdout:244: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.122 INFO:tasks.rados.rados.0.smithi157.stdout:244: set_chunk oid 45 offset: 1619456 length: 555008 target oid smithi15736696-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1619456 snap -1 2024-09-16T03:37:20.122 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.122 INFO:tasks.rados.rados.0.smithi157.stdout:244: finishing set_chunk to oid 45 2024-09-16T03:37:20.122 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1619456 length: 555008 2024-09-16T03:37:20.122 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T03:37:20.122 INFO:tasks.rados.rados.0.smithi157.stdout:244: done (15 left) 2024-09-16T03:37:20.122 INFO:tasks.rados.rados.0.smithi157.stdout:245: set_chunk oid 46 target oid 46 low_tier 2024-09-16T03:37:20.122 INFO:tasks.rados.rados.0.smithi157.stdout:245: set_chunk oid 46 offset: 908800 length: 587264 target oid smithi15736696-46 low_tier offset: 908800 snap -1 2024-09-16T03:37:20.122 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.123 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:20.123 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:20.128 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:20.128 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:20.134 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-16T03:37:20.145 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:20.146 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:20.154 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:20.154 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:20.162 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:20.162 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:20.163 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:37:20.179 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:20.179 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:20.196 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:20.197 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:20.199 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:20.198+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:20.210 INFO:tasks.rados.rados.0.smithi157.stdout:226: finishing set_chunk to oid 27 2024-09-16T03:37:20.210 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 634368 length: 427008 2024-09-16T03:37:20.210 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:20.210 INFO:tasks.rados.rados.0.smithi157.stdout:226: done (15 left) 2024-09-16T03:37:20.210 INFO:tasks.rados.rados.0.smithi157.stdout:246: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.210 INFO:tasks.rados.rados.0.smithi157.stdout:246: set_chunk oid 47 offset: 162816 length: 1676288 target oid smithi15736696-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 162816 snap -1 2024-09-16T03:37:20.211 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.211 INFO:tasks.rados.rados.0.smithi157.stdout:233: finishing set_chunk to oid 34 2024-09-16T03:37:20.211 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 1148928 length: 961024 2024-09-16T03:37:20.211 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T03:37:20.211 INFO:tasks.rados.rados.0.smithi157.stdout:233: done (15 left) 2024-09-16T03:37:20.211 INFO:tasks.rados.rados.0.smithi157.stdout:247: set_chunk oid 48 target oid 48 low_tier 2024-09-16T03:37:20.211 INFO:tasks.rados.rados.0.smithi157.stdout:247: set_chunk oid 48 offset: 1363456 length: 1326592 target oid smithi15736696-48 low_tier offset: 1363456 snap -1 2024-09-16T03:37:20.211 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.211 INFO:tasks.rados.rados.0.smithi157.stdout:230: finishing set_chunk to oid 31 2024-09-16T03:37:20.211 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2098176 length: 53248 2024-09-16T03:37:20.211 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:37:20.211 INFO:tasks.rados.rados.0.smithi157.stdout:228: finishing set_chunk to oid 29 2024-09-16T03:37:20.211 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 367104 length: 160768 2024-09-16T03:37:20.212 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:37:20.212 INFO:tasks.rados.rados.0.smithi157.stdout:228: done (15 left) 2024-09-16T03:37:20.212 INFO:tasks.rados.rados.0.smithi157.stdout:230: done (14 left) 2024-09-16T03:37:20.212 INFO:tasks.rados.rados.0.smithi157.stdout:248: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.212 INFO:tasks.rados.rados.0.smithi157.stdout:248: set_chunk oid 49 offset: 2009600 length: 228352 target oid smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2009600 snap -1 2024-09-16T03:37:20.212 INFO:tasks.rados.rados.0.smithi157.stdout:249: set_chunk oid 50 target oid 50 low_tier 2024-09-16T03:37:20.212 INFO:tasks.rados.rados.0.smithi157.stdout:249: set_chunk oid 50 offset: 871936 length: 201728 target oid smithi15736696-50 low_tier offset: 871936 snap -1 2024-09-16T03:37:20.212 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.212 INFO:tasks.rados.rados.0.smithi157.stdout:247: finishing set_chunk to oid 48 2024-09-16T03:37:20.212 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1363456 length: 1326592 2024-09-16T03:37:20.212 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:37:20.212 INFO:tasks.rados.rados.0.smithi157.stdout:247: done (15 left) 2024-09-16T03:37:20.213 INFO:tasks.rados.rados.0.smithi157.stdout:250: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.213 INFO:tasks.rados.rados.0.smithi157.stdout:250: set_chunk oid 1 offset: 1529344 length: 112128 target oid smithi15736696-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1529344 snap -1 2024-09-16T03:37:20.213 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.213 INFO:tasks.rados.rados.0.smithi157.stdout:245: finishing set_chunk to oid 46 2024-09-16T03:37:20.213 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 908800 length: 587264 2024-09-16T03:37:20.213 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:37:20.213 INFO:tasks.rados.rados.0.smithi157.stdout:245: done (15 left) 2024-09-16T03:37:20.213 INFO:tasks.rados.rados.0.smithi157.stdout:251: set_chunk oid 2 target oid 2 low_tier 2024-09-16T03:37:20.213 INFO:tasks.rados.rados.0.smithi157.stdout:251: set_chunk oid 2 offset: 404992 length: 229376 target oid smithi15736696-2 low_tier offset: 404992 snap -1 2024-09-16T03:37:20.213 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.213 INFO:tasks.rados.rados.0.smithi157.stdout:242: finishing set_chunk to oid 43 2024-09-16T03:37:20.213 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 158720 length: 1909760 2024-09-16T03:37:20.213 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout:242: done (15 left) 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout:252: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout:252: set_chunk oid 3 offset: 8704 length: 446976 target oid smithi15736696-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 8704 snap -1 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout:251: finishing set_chunk to oid 2 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 404992 length: 229376 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout:251: done (15 left) 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout:253: set_chunk oid 4 target oid 4 low_tier 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout:253: set_chunk oid 4 offset: 528896 length: 1132032 target oid smithi15736696-4 low_tier offset: 528896 snap -1 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout:252: finishing set_chunk to oid 3 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 8704 length: 446976 2024-09-16T03:37:20.214 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T03:37:20.215 INFO:tasks.rados.rados.0.smithi157.stdout:252: done (15 left) 2024-09-16T03:37:20.215 INFO:tasks.rados.rados.0.smithi157.stdout:254: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.215 INFO:tasks.rados.rados.0.smithi157.stdout:254: set_chunk oid 5 offset: 490496 length: 542720 target oid smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 490496 snap -1 2024-09-16T03:37:20.215 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.215 INFO:tasks.rados.rados.0.smithi157.stdout:246: finishing set_chunk to oid 47 2024-09-16T03:37:20.215 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 162816 length: 1676288 2024-09-16T03:37:20.215 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T03:37:20.215 INFO:tasks.rados.rados.0.smithi157.stdout:246: done (15 left) 2024-09-16T03:37:20.215 INFO:tasks.rados.rados.0.smithi157.stdout:255: set_chunk oid 6 target oid 6 low_tier 2024-09-16T03:37:20.215 INFO:tasks.rados.rados.0.smithi157.stdout:255: set_chunk oid 6 offset: 1052672 length: 261632 target oid smithi15736696-6 low_tier offset: 1052672 snap -1 2024-09-16T03:37:20.215 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.215 INFO:tasks.rados.rados.0.smithi157.stdout:255: finishing set_chunk to oid 6 2024-09-16T03:37:20.215 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1052672 length: 261632 2024-09-16T03:37:20.216 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:37:20.216 INFO:tasks.rados.rados.0.smithi157.stdout:255: done (15 left) 2024-09-16T03:37:20.216 INFO:tasks.rados.rados.0.smithi157.stdout:256: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.216 INFO:tasks.rados.rados.0.smithi157.stdout:256: set_chunk oid 7 offset: 0 length: 494592 target oid smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 0 snap -1 2024-09-16T03:37:20.216 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.216 INFO:tasks.rados.rados.0.smithi157.stdout:256: finishing set_chunk to oid 7 2024-09-16T03:37:20.216 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 0 length: 494592 2024-09-16T03:37:20.216 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T03:37:20.216 INFO:tasks.rados.rados.0.smithi157.stdout:256: done (15 left) 2024-09-16T03:37:20.216 INFO:tasks.rados.rados.0.smithi157.stdout:257: set_chunk oid 8 target oid 8 low_tier 2024-09-16T03:37:20.216 INFO:tasks.rados.rados.0.smithi157.stdout:257: set_chunk oid 8 offset: 1081344 length: 60416 target oid smithi15736696-8 low_tier offset: 1081344 snap -1 2024-09-16T03:37:20.216 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.282 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:20.282 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:20.282 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:37:20.282 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:20.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-16T03:37:20.292 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:37:20.307 INFO:tasks.rados.rados.0.smithi157.stdout:235: finishing set_chunk to oid 36 2024-09-16T03:37:20.308 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 606720 length: 1724416 2024-09-16T03:37:20.308 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:37:20.308 INFO:tasks.rados.rados.0.smithi157.stdout:239: finishing set_chunk to oid 40 2024-09-16T03:37:20.308 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 557568 length: 1289728 2024-09-16T03:37:20.308 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T03:37:20.308 INFO:tasks.rados.rados.0.smithi157.stdout:253: finishing set_chunk to oid 4 2024-09-16T03:37:20.308 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 528896 length: 1132032 2024-09-16T03:37:20.308 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:37:20.308 INFO:tasks.rados.rados.0.smithi157.stdout:248: finishing set_chunk to oid 49 2024-09-16T03:37:20.308 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2009600 length: 228352 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:249: finishing set_chunk to oid 50 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 871936 length: 201728 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:235: done (15 left) 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:239: done (14 left) 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:248: done (13 left) 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:249: done (12 left) 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:253: done (11 left) 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:258: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:258: set_chunk oid 9 offset: 1063424 length: 167936 target oid smithi15736696-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1063424 snap -1 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:259: set_chunk oid 10 target oid 10 low_tier 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:259: set_chunk oid 10 offset: 295424 length: 78848 target oid smithi15736696-10 low_tier offset: 295424 snap -1 2024-09-16T03:37:20.309 INFO:tasks.rados.rados.0.smithi157.stdout:260: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.310 INFO:tasks.rados.rados.0.smithi157.stdout:260: set_chunk oid 11 offset: 1017856 length: 380416 target oid smithi15736696-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1017856 snap -1 2024-09-16T03:37:20.310 INFO:tasks.rados.rados.0.smithi157.stdout:261: set_chunk oid 12 target oid 12 low_tier 2024-09-16T03:37:20.310 INFO:tasks.rados.rados.0.smithi157.stdout:261: set_chunk oid 12 offset: 776704 length: 1313280 target oid smithi15736696-12 low_tier offset: 776704 snap -1 2024-09-16T03:37:20.310 INFO:tasks.rados.rados.0.smithi157.stdout:262: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.310 INFO:tasks.rados.rados.0.smithi157.stdout:262: set_chunk oid 13 offset: 219136 length: 1700352 target oid smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 219136 snap -1 2024-09-16T03:37:20.310 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.310 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:20.305+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:20.311 INFO:tasks.rados.rados.0.smithi157.stdout:260: finishing set_chunk to oid 11 2024-09-16T03:37:20.311 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1017856 length: 380416 2024-09-16T03:37:20.311 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T03:37:20.311 INFO:tasks.rados.rados.0.smithi157.stdout:260: done (15 left) 2024-09-16T03:37:20.312 INFO:tasks.rados.rados.0.smithi157.stdout:263: set_chunk oid 14 target oid 14 low_tier 2024-09-16T03:37:20.312 INFO:tasks.rados.rados.0.smithi157.stdout:263: set_chunk oid 14 offset: 41984 length: 1281024 target oid smithi15736696-14 low_tier offset: 41984 snap -1 2024-09-16T03:37:20.312 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.315 INFO:tasks.rados.rados.0.smithi157.stdout:258: finishing set_chunk to oid 9 2024-09-16T03:37:20.315 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1063424 length: 167936 2024-09-16T03:37:20.316 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:37:20.316 INFO:tasks.rados.rados.0.smithi157.stdout:258: done (15 left) 2024-09-16T03:37:20.316 INFO:tasks.rados.rados.0.smithi157.stdout:264: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.316 INFO:tasks.rados.rados.0.smithi157.stdout:264: set_chunk oid 15 offset: 496640 length: 1432576 target oid smithi15736696-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 496640 snap -1 2024-09-16T03:37:20.316 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.316 INFO:tasks.rados.rados.0.smithi157.stdout:261: finishing set_chunk to oid 12 2024-09-16T03:37:20.316 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 776704 length: 1313280 2024-09-16T03:37:20.316 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:37:20.316 INFO:tasks.rados.rados.0.smithi157.stdout:261: done (15 left) 2024-09-16T03:37:20.316 INFO:tasks.rados.rados.0.smithi157.stdout:265: set_chunk oid 16 target oid 16 low_tier 2024-09-16T03:37:20.316 INFO:tasks.rados.rados.0.smithi157.stdout:265: set_chunk oid 16 offset: 1840640 length: 331264 target oid smithi15736696-16 low_tier offset: 1840640 snap -1 2024-09-16T03:37:20.316 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.316 INFO:tasks.rados.rados.0.smithi157.stdout:264: finishing set_chunk to oid 15 2024-09-16T03:37:20.317 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 496640 length: 1432576 2024-09-16T03:37:20.317 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:20.317 INFO:tasks.rados.rados.0.smithi157.stdout:264: done (15 left) 2024-09-16T03:37:20.317 INFO:tasks.rados.rados.0.smithi157.stdout:266: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.317 INFO:tasks.rados.rados.0.smithi157.stdout:266: set_chunk oid 17 offset: 644608 length: 420352 target oid smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 644608 snap -1 2024-09-16T03:37:20.317 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.317 INFO:tasks.rados.rados.0.smithi157.stdout:234: finishing set_chunk to oid 35 2024-09-16T03:37:20.317 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:37:20.317 INFO:tasks.rados.rados.0.smithi157.stdout:234: done (15 left) 2024-09-16T03:37:20.317 INFO:tasks.rados.rados.0.smithi157.stdout:266 retry set_chunk ! 2024-09-16T03:37:20.317 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 15 2024-09-16T03:37:20.318 INFO:tasks.rados.rados.0.smithi157.stdout:238: finishing set_chunk to oid 39 2024-09-16T03:37:20.318 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T03:37:20.318 INFO:tasks.rados.rados.0.smithi157.stdout:238: done (14 left) 2024-09-16T03:37:20.318 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 14 2024-09-16T03:37:20.400 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:20.399+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:20.407 INFO:tasks.rados.rados.0.smithi157.stdout:257: finishing set_chunk to oid 8 2024-09-16T03:37:20.407 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 1081344 length: 60416 2024-09-16T03:37:20.407 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:37:20.407 INFO:tasks.rados.rados.0.smithi157.stdout:257: done (13 left) 2024-09-16T03:37:20.408 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 13 2024-09-16T03:37:20.408 INFO:tasks.rados.rados.0.smithi157.stdout:262: finishing set_chunk to oid 13 2024-09-16T03:37:20.408 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 219136 length: 1700352 2024-09-16T03:37:20.408 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T03:37:20.408 INFO:tasks.rados.rados.0.smithi157.stdout:262: done (12 left) 2024-09-16T03:37:20.408 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 12 2024-09-16T03:37:20.410 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:37:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:37:20.410 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:20.419 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:37:20.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-16T03:37:20.502 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:20.500+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:20.508 INFO:tasks.rados.rados.0.smithi157.stdout:221: finishing set_chunk to oid 22 2024-09-16T03:37:20.508 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T03:37:20.508 INFO:tasks.rados.rados.0.smithi157.stdout:221: done (11 left) 2024-09-16T03:37:20.508 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 11 2024-09-16T03:37:20.508 INFO:tasks.rados.rados.0.smithi157.stdout:263: finishing set_chunk to oid 14 2024-09-16T03:37:20.508 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 41984 length: 1281024 2024-09-16T03:37:20.508 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T03:37:20.508 INFO:tasks.rados.rados.0.smithi157.stdout:263: done (10 left) 2024-09-16T03:37:20.508 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 10 2024-09-16T03:37:20.509 INFO:tasks.rados.rados.0.smithi157.stdout:227: finishing set_chunk to oid 28 2024-09-16T03:37:20.509 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 3 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T03:37:20.509 INFO:tasks.rados.rados.0.smithi157.stdout:227: done (9 left) 2024-09-16T03:37:20.509 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 9 2024-09-16T03:37:20.525 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:20.525 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:37:20.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:37:20.525 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:20.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.525 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:13 4.a e39/32 4:51338165:::smithi15736696-4:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:20.525 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:11.994554+0000", 2024-09-16T03:37:20.526 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.5278071410000003, 2024-09-16T03:37:20.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034462199999999998, 2024-09-16T03:37:20.526 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.526 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.526 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.526 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.994554+0000", 2024-09-16T03:37:20.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.994554+0000", 2024-09-16T03:37:20.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.994554+0000", 2024-09-16T03:37:20.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8799999999999998e-07 2024-09-16T03:37:20.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.995984+0000", 2024-09-16T03:37:20.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014298850000000001 2024-09-16T03:37:20.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.995995+0000", 2024-09-16T03:37:20.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0937e-05 2024-09-16T03:37:20.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.996008+0000", 2024-09-16T03:37:20.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3373000000000001e-05 2024-09-16T03:37:20.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.996074+0000", 2024-09-16T03:37:20.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5137999999999999e-05 2024-09-16T03:37:20.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.996101+0000", 2024-09-16T03:37:20.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7659999999999999e-05 2024-09-16T03:37:20.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997962+0000", 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018607109999999999 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.998000+0000", 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8127999999999998e-05 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:14 4.a e39/32 4:51338165:::smithi15736696-4:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:11.997261+0000", 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.5250998799999991, 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185554500000001, 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997261+0000", 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997261+0000", 2024-09-16T03:37:20.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997263+0000", 2024-09-16T03:37:20.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.373e-06 2024-09-16T03:37:20.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997764+0000", 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00050155299999999996 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997775+0000", 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1066999999999999e-05 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097841+0000", 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100066316 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097882+0000", 2024-09-16T03:37:20.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0850999999999999e-05 2024-09-16T03:37:20.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097904+0000", 2024-09-16T03:37:20.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1347000000000001e-05 2024-09-16T03:37:20.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.099089+0000", 2024-09-16T03:37:20.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001185195 2024-09-16T03:37:20.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.099117+0000", 2024-09-16T03:37:20.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7843000000000001e-05 2024-09-16T03:37:20.536 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.536 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.536 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.536 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:29 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:20.536 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.038885+0000", 2024-09-16T03:37:20.536 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.4834758830000006, 2024-09-16T03:37:20.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021559560000000001, 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.038885+0000", 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.038885+0000", 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.038885+0000", 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3700000000000001e-07 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.039585+0000", 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069925100000000002 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.039598+0000", 2024-09-16T03:37:20.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3303e-05 2024-09-16T03:37:20.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.039609+0000", 2024-09-16T03:37:20.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1208e-05 2024-09-16T03:37:20.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.039671+0000", 2024-09-16T03:37:20.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1739e-05 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.039690+0000", 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9320000000000001e-05 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.040998+0000", 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001307241 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.041041+0000", 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3557000000000003e-05 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:15 4.a e39/32 4:51338165:::smithi15736696-4:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.097860+0000", 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.4245007849999993, 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001302213, 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.541 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097860+0000", 2024-09-16T03:37:20.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097860+0000", 2024-09-16T03:37:20.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097861+0000", 2024-09-16T03:37:20.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7100000000000011e-07 2024-09-16T03:37:20.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097868+0000", 2024-09-16T03:37:20.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7909999999999999e-06 2024-09-16T03:37:20.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097874+0000", 2024-09-16T03:37:20.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.348e-06 2024-09-16T03:37:20.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097879+0000", 2024-09-16T03:37:20.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7929999999999997e-06 2024-09-16T03:37:20.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.098619+0000", 2024-09-16T03:37:20.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073992700000000003 2024-09-16T03:37:20.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.098632+0000", 2024-09-16T03:37:20.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2958e-05 2024-09-16T03:37:20.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.099146+0000", 2024-09-16T03:37:20.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00051384499999999997 2024-09-16T03:37:20.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.099162+0000", 2024-09-16T03:37:20.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.658e-05 2024-09-16T03:37:20.546 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.546 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.546 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.546 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:20.547 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.141452+0000", 2024-09-16T03:37:20.547 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.3809091470000006, 2024-09-16T03:37:20.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013578970000000001, 2024-09-16T03:37:20.547 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.547 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.547 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.547 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.141452+0000", 2024-09-16T03:37:20.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.141452+0000", 2024-09-16T03:37:20.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.141453+0000", 2024-09-16T03:37:20.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5179999999999999e-06 2024-09-16T03:37:20.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.141629+0000", 2024-09-16T03:37:20.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017514000000000001 2024-09-16T03:37:20.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.141638+0000", 2024-09-16T03:37:20.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7280000000000007e-06 2024-09-16T03:37:20.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.141645+0000", 2024-09-16T03:37:20.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0720000000000002e-06 2024-09-16T03:37:20.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.141719+0000", 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3740999999999998e-05 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.141735+0000", 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6186000000000001e-05 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.142769+0000", 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001033766 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.142810+0000", 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0745999999999999e-05 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.243415+0000", 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.2789458860000007, 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027177450000000001, 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.551 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.243415+0000", 2024-09-16T03:37:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.243415+0000", 2024-09-16T03:37:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.243416+0000", 2024-09-16T03:37:20.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5600000000000001e-07 2024-09-16T03:37:20.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244398+0000", 2024-09-16T03:37:20.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00098253199999999994 2024-09-16T03:37:20.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244428+0000", 2024-09-16T03:37:20.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9921e-05 2024-09-16T03:37:20.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244437+0000", 2024-09-16T03:37:20.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9609999999999994e-06 2024-09-16T03:37:20.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244480+0000", 2024-09-16T03:37:20.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2718e-05 2024-09-16T03:37:20.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244493+0000", 2024-09-16T03:37:20.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3519999999999999e-05 2024-09-16T03:37:20.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.246103+0000", 2024-09-16T03:37:20.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001610043 2024-09-16T03:37:20.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.246133+0000", 2024-09-16T03:37:20.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9594000000000002e-05 2024-09-16T03:37:20.556 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.556 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.556 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.556 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:37:20.556 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.244449+0000", 2024-09-16T03:37:20.556 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.2779117220000007, 2024-09-16T03:37:20.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018711890000000001, 2024-09-16T03:37:20.557 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.557 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.557 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.557 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244449+0000", 2024-09-16T03:37:20.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244449+0000", 2024-09-16T03:37:20.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244450+0000", 2024-09-16T03:37:20.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5499999999999998e-07 2024-09-16T03:37:20.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244795+0000", 2024-09-16T03:37:20.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034479399999999999 2024-09-16T03:37:20.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244803+0000", 2024-09-16T03:37:20.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1710000000000002e-06 2024-09-16T03:37:20.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244809+0000", 2024-09-16T03:37:20.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1859999999999997e-06 2024-09-16T03:37:20.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.245523+0000", 2024-09-16T03:37:20.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00071372099999999995 2024-09-16T03:37:20.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.245536+0000", 2024-09-16T03:37:20.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3217999999999999e-05 2024-09-16T03:37:20.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.246300+0000", 2024-09-16T03:37:20.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00076391600000000001 2024-09-16T03:37:20.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.246321+0000", 2024-09-16T03:37:20.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0727999999999999e-05 2024-09-16T03:37:20.562 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.562 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.562 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.562 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:37:20.562 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.244820+0000", 2024-09-16T03:37:20.562 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.2775410059999999, 2024-09-16T03:37:20.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015550060000000001, 2024-09-16T03:37:20.562 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.563 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.563 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.563 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244820+0000", 2024-09-16T03:37:20.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244820+0000", 2024-09-16T03:37:20.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244821+0000", 2024-09-16T03:37:20.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1900000000000002e-07 2024-09-16T03:37:20.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244828+0000", 2024-09-16T03:37:20.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1450000000000002e-06 2024-09-16T03:37:20.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244834+0000", 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5930000000000002e-06 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.244841+0000", 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1929999999999998e-06 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.245942+0000", 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001100923 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.245955+0000", 2024-09-16T03:37:20.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3101e-05 2024-09-16T03:37:20.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.246361+0000", 2024-09-16T03:37:20.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040612299999999999 2024-09-16T03:37:20.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.246375+0000", 2024-09-16T03:37:20.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4309e-05 2024-09-16T03:37:20.566 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.567 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.567 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.567 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:73 4.a e39/32 4:532814f2:::smithi15736696-19:head v 39'4, mlcod=39'2)", 2024-09-16T03:37:20.567 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.312294+0000", 2024-09-16T03:37:20.567 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.210067102, 2024-09-16T03:37:20.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002243854, 2024-09-16T03:37:20.567 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.567 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.567 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.567 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.312294+0000", 2024-09-16T03:37:20.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.312294+0000", 2024-09-16T03:37:20.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.312296+0000", 2024-09-16T03:37:20.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.826e-06 2024-09-16T03:37:20.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.313036+0000", 2024-09-16T03:37:20.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00074037599999999995 2024-09-16T03:37:20.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.313050+0000", 2024-09-16T03:37:20.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3492000000000001e-05 2024-09-16T03:37:20.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.313056+0000", 2024-09-16T03:37:20.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.584e-06 2024-09-16T03:37:20.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.313105+0000", 2024-09-16T03:37:20.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8875999999999998e-05 2024-09-16T03:37:20.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.313119+0000", 2024-09-16T03:37:20.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3978000000000001e-05 2024-09-16T03:37:20.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.314493+0000", 2024-09-16T03:37:20.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001373499 2024-09-16T03:37:20.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.314538+0000", 2024-09-16T03:37:20.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5222999999999997e-05 2024-09-16T03:37:20.572 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.572 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.572 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.573 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:74 4.a e39/32 4:532814f2:::smithi15736696-19:head v 39'5, mlcod=39'2)", 2024-09-16T03:37:20.573 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.416979+0000", 2024-09-16T03:37:20.573 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.1053815409999999, 2024-09-16T03:37:20.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013705970000000001, 2024-09-16T03:37:20.573 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.573 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.573 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.573 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.416979+0000", 2024-09-16T03:37:20.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.416979+0000", 2024-09-16T03:37:20.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.416982+0000", 2024-09-16T03:37:20.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9799999999999998e-06 2024-09-16T03:37:20.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.417192+0000", 2024-09-16T03:37:20.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020973699999999999 2024-09-16T03:37:20.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.417205+0000", 2024-09-16T03:37:20.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3247999999999999e-05 2024-09-16T03:37:20.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.417230+0000", 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4494999999999998e-05 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.417281+0000", 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0677999999999998e-05 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.417294+0000", 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.347e-05 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.418322+0000", 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001028227 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.418350+0000", 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7761999999999999e-05 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.577 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:93 4.a e39/32 4:5e3c0760:::smithi15736696-24:head v 39'7, mlcod=39'2)", 2024-09-16T03:37:20.578 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.474319+0000", 2024-09-16T03:37:20.578 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.0480415599999997, 2024-09-16T03:37:20.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001634356, 2024-09-16T03:37:20.578 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.578 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.578 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.578 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.474319+0000", 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.474319+0000", 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.474321+0000", 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9120000000000001e-06 2024-09-16T03:37:20.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.474718+0000", 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00039684300000000002 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.474733+0000", 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4942e-05 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.474741+0000", 2024-09-16T03:37:20.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1559999999999995e-06 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.474804+0000", 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3136e-05 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.474827+0000", 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2096999999999999e-05 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.475921+0000", 2024-09-16T03:37:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010943019999999999 2024-09-16T03:37:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.475954+0000", 2024-09-16T03:37:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2968e-05 2024-09-16T03:37:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.4.22:3 3.1 e41/31 3:bffb5658:::smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'22, mlcod=41'20)", 2024-09-16T03:37:20.582 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:15.504663+0000", 2024-09-16T03:37:20.583 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.0176982299999997, 2024-09-16T03:37:20.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100603585, 2024-09-16T03:37:20.583 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.583 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.583 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.583 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.504663+0000", 2024-09-16T03:37:20.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.504663+0000", 2024-09-16T03:37:20.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.504664+0000", 2024-09-16T03:37:20.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1969999999999999e-06 2024-09-16T03:37:20.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.504678+0000", 2024-09-16T03:37:20.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3793e-05 2024-09-16T03:37:20.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.504688+0000", 2024-09-16T03:37:20.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.01e-05 2024-09-16T03:37:20.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.604761+0000", 2024-09-16T03:37:20.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100072681 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.604805+0000", 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4035000000000001e-05 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.604821+0000", 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6095000000000001e-05 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.605217+0000", 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00039639200000000002 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.605266+0000", 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9292000000000002e-05 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:276 4.a e41/32 4:50107449:::smithi15736696-27:head v 41'20, mlcod=39'18)", 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:15.701824+0000", 2024-09-16T03:37:20.587 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.8205365599999999, 2024-09-16T03:37:20.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10067820700000001, 2024-09-16T03:37:20.588 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.588 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.588 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.588 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.701824+0000", 2024-09-16T03:37:20.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.701824+0000", 2024-09-16T03:37:20.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.701826+0000", 2024-09-16T03:37:20.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.232e-06 2024-09-16T03:37:20.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.701832+0000", 2024-09-16T03:37:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9479999999999996e-06 2024-09-16T03:37:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.701843+0000", 2024-09-16T03:37:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0987e-05 2024-09-16T03:37:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.701848+0000", 2024-09-16T03:37:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2809999999999999e-06 2024-09-16T03:37:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.801996+0000", 2024-09-16T03:37:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100148312 2024-09-16T03:37:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.802032+0000", 2024-09-16T03:37:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5642000000000002e-05 2024-09-16T03:37:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.802470+0000", 2024-09-16T03:37:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043790499999999999 2024-09-16T03:37:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.802503+0000", 2024-09-16T03:37:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.29e-05 2024-09-16T03:37:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.592 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.15.23:17 3.1 e42/31 3:86477bf9:::smithi15736696-46 low_tier:head v 42'23, mlcod=41'21)", 2024-09-16T03:37:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.187034+0000", 2024-09-16T03:37:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.3353265700000003, 2024-09-16T03:37:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10100469400000001, 2024-09-16T03:37:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.593 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187034+0000", 2024-09-16T03:37:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187034+0000", 2024-09-16T03:37:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187035+0000", 2024-09-16T03:37:20.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5400000000000001e-07 2024-09-16T03:37:20.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187042+0000", 2024-09-16T03:37:20.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1729999999999998e-06 2024-09-16T03:37:20.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187063+0000", 2024-09-16T03:37:20.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1160000000000001e-05 2024-09-16T03:37:20.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187081+0000", 2024-09-16T03:37:20.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7935999999999999e-05 2024-09-16T03:37:20.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.287313+0000", 2024-09-16T03:37:20.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100231325 2024-09-16T03:37:20.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.287345+0000", 2024-09-16T03:37:20.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2119000000000002e-05 2024-09-16T03:37:20.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.288013+0000", 2024-09-16T03:37:20.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00066847800000000002 2024-09-16T03:37:20.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.288039+0000", 2024-09-16T03:37:20.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5948999999999998e-05 2024-09-16T03:37:20.598 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.599 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.599 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.599 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.11.23:19 3.1 e42/31 3:9aba8983:::smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'26, mlcod=42'24)", 2024-09-16T03:37:20.599 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.419116+0000", 2024-09-16T03:37:20.599 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.1032455260000003, 2024-09-16T03:37:20.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10061766900000001, 2024-09-16T03:37:20.599 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.600 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.600 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.600 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.419116+0000", 2024-09-16T03:37:20.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.419116+0000", 2024-09-16T03:37:20.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.419117+0000", 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0869999999999999e-06 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.419130+0000", 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3256e-05 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.419140+0000", 2024-09-16T03:37:20.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0397000000000001e-05 2024-09-16T03:37:20.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519236+0000", 2024-09-16T03:37:20.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100095583 2024-09-16T03:37:20.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519277+0000", 2024-09-16T03:37:20.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1631000000000002e-05 2024-09-16T03:37:20.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519298+0000", 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0472e-05 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519706+0000", 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000408 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519733+0000", 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7243e-05 2024-09-16T03:37:20.603 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.604 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.604 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.604 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.5.21:10 3.1 e42/31 3:8f38b759:::smithi15736696-6 low_tier:head v 42'27, mlcod=42'24)", 2024-09-16T03:37:20.604 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.519254+0000", 2024-09-16T03:37:20.604 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.0031074990000004, 2024-09-16T03:37:20.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100789212, 2024-09-16T03:37:20.604 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.604 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.604 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.604 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519254+0000", 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519254+0000", 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519254+0000", 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9999999999999997e-07 2024-09-16T03:37:20.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519259+0000", 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6489999999999999e-06 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519264+0000", 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5289999999999999e-06 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619388+0000", 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100123484 2024-09-16T03:37:20.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619501+0000", 2024-09-16T03:37:20.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011266500000000001 2024-09-16T03:37:20.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619536+0000", 2024-09-16T03:37:20.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5352999999999998e-05 2024-09-16T03:37:20.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.620019+0000", 2024-09-16T03:37:20.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00048265200000000003 2024-09-16T03:37:20.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.608 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.620043+0000", 2024-09-16T03:37:20.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4179999999999999e-05 2024-09-16T03:37:20.608 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.608 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.608 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.608 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.5.21:11 3.1 e42/31 3:adc537a6:::smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'28, mlcod=42'24)", 2024-09-16T03:37:20.608 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.619416+0000", 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.9029449000000001, 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100981345, 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619416+0000", 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619416+0000", 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619418+0000", 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.99e-06 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619445+0000", 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6755e-05 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619465+0000", 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0097000000000001e-05 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719589+0000", 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10012367799999999 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719706+0000", 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117259 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719784+0000", 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7714999999999997e-05 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.720366+0000", 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058208999999999995 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.720397+0000", 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1761000000000003e-05 2024-09-16T03:37:20.612 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:303 4.a e42/32 4:51338165:::smithi15736696-4:head v 42'21, mlcod=40'19)", 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.719733+0000", 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.8026282880000002, 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000964027, 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719733+0000", 2024-09-16T03:37:20.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719733+0000", 2024-09-16T03:37:20.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719734+0000", 2024-09-16T03:37:20.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.246e-06 2024-09-16T03:37:20.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719791+0000", 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6715999999999997e-05 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719811+0000", 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0207000000000002e-05 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719827+0000", 2024-09-16T03:37:20.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5892e-05 2024-09-16T03:37:20.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719945+0000", 2024-09-16T03:37:20.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011824699999999999 2024-09-16T03:37:20.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719958+0000", 2024-09-16T03:37:20.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3033e-05 2024-09-16T03:37:20.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.720669+0000", 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000710646 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.720697+0000", 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8039999999999999e-05 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.617 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:333 4.a e45/32 4:503187b3:::smithi15736696-34:head v 45'29, mlcod=44'27)", 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.003000+0000", 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.51936140200000003, 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10086444999999999, 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003000+0000", 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003000+0000", 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003001+0000", 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.35e-06 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003019+0000", 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8182999999999999e-05 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003033+0000", 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4307999999999999e-05 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003040+0000", 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4359999999999999e-06 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.103213+0000", 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100172795 2024-09-16T03:37:20.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.103232+0000", 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9746999999999999e-05 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.103822+0000", 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058940000000000002 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.103864+0000", 2024-09-16T03:37:20.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2231000000000003e-05 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:19 3.1 e45/31 3:aca5d7ac:::smithi15736696-26 low_tier:head v 45'32, mlcod=44'29)", 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.206860+0000", 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.315500909, 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00096942399999999998, 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:20.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.206860+0000", 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.206860+0000", 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.206873+0000", 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3337e-05 2024-09-16T03:37:20.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.206887+0000", 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.36e-05 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.206903+0000", 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5838e-05 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.206913+0000", 2024-09-16T03:37:20.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0309000000000001e-05 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.206993+0000", 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9297999999999999e-05 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.207020+0000", 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7250000000000002e-05 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:20.625 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.207779+0000", 2024-09-16T03:37:20.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000759667 2024-09-16T03:37:20.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:20.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:20.626 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:20.626 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.207830+0000", 2024-09-16T03:37:20.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0124999999999999e-05 2024-09-16T03:37:20.626 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.626 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.626 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.626 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:20.626 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:20.626 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:20.628 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:20.626+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:20.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-16T03:37:20.629 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:37:20.631 INFO:tasks.rados.rados.0.smithi157.stdout:225: finishing set_chunk to oid 26 2024-09-16T03:37:20.631 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 12 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:37:20.631 INFO:tasks.rados.rados.0.smithi157.stdout:225: done (8 left) 2024-09-16T03:37:20.631 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 8 2024-09-16T03:37:20.631 INFO:tasks.rados.rados.0.smithi157.stdout:265: finishing set_chunk to oid 16 2024-09-16T03:37:20.631 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:37:20.631 INFO:tasks.rados.rados.0.smithi157.stdout:265: done (7 left) 2024-09-16T03:37:20.631 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 7 2024-09-16T03:37:20.631 INFO:tasks.rados.rados.0.smithi157.stdout:250: finishing set_chunk to oid 1 2024-09-16T03:37:20.631 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T03:37:20.632 INFO:tasks.rados.rados.0.smithi157.stdout:250: done (6 left) 2024-09-16T03:37:20.632 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 6 2024-09-16T03:37:20.632 INFO:tasks.rados.rados.0.smithi157.stdout:254: finishing set_chunk to oid 5 2024-09-16T03:37:20.632 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T03:37:20.632 INFO:tasks.rados.rados.0.smithi157.stdout:254: done (5 left) 2024-09-16T03:37:20.632 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 5 2024-09-16T03:37:20.632 INFO:tasks.rados.rados.0.smithi157.stdout:243: finishing set_chunk to oid 44 2024-09-16T03:37:20.632 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T03:37:20.632 INFO:tasks.rados.rados.0.smithi157.stdout:243: done (4 left) 2024-09-16T03:37:20.632 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 4 2024-09-16T03:37:20.710 INFO:tasks.rados.rados.0.smithi157.stdout:217: finishing set_chunk to oid 18 2024-09-16T03:37:20.711 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T03:37:20.711 INFO:tasks.rados.rados.0.smithi157.stdout:217: done (3 left) 2024-09-16T03:37:20.711 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 3 2024-09-16T03:37:20.711 INFO:tasks.rados.rados.0.smithi157.stdout:224: finishing set_chunk to oid 25 2024-09-16T03:37:20.711 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:20.711 INFO:tasks.rados.rados.0.smithi157.stdout:224: done (2 left) 2024-09-16T03:37:20.711 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 2 2024-09-16T03:37:20.728 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:20.727+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:20.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-09-16T03:37:20.771 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:20.771 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:20.771 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:37:20.771 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:20.780 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:37:20.811 INFO:tasks.rados.rados.0.smithi157.stdout:259: finishing set_chunk to oid 10 2024-09-16T03:37:20.811 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:37:20.812 INFO:tasks.rados.rados.0.smithi157.stdout:259: done (1 left) 2024-09-16T03:37:20.812 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 1 2024-09-16T03:37:20.830 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:20.828+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:20.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-09-16T03:37:20.918 INFO:tasks.rados.rados.0.smithi157.stdout:266: finishing set_chunk to oid 17 2024-09-16T03:37:20.918 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T03:37:20.918 INFO:tasks.rados.rados.0.smithi157.stdout:266: done (0 left) 2024-09-16T03:37:20.918 INFO:tasks.rados.rados.0.smithi157.stdout:267: set_chunk oid 18 target oid 18 low_tier 2024-09-16T03:37:20.918 INFO:tasks.rados.rados.0.smithi157.stdout:267: set_chunk oid 18 offset: 403968 length: 512512 target oid smithi15736696-18 low_tier offset: 403968 snap -1 2024-09-16T03:37:20.918 INFO:tasks.rados.rados.0.smithi157.stdout:268: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.918 INFO:tasks.rados.rados.0.smithi157.stdout:268: set_chunk oid 19 offset: 883712 length: 1372672 target oid smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 883712 snap -1 2024-09-16T03:37:20.918 INFO:tasks.rados.rados.0.smithi157.stdout:269: set_chunk oid 20 target oid 20 low_tier 2024-09-16T03:37:20.919 INFO:tasks.rados.rados.0.smithi157.stdout:269: set_chunk oid 20 offset: 2146816 length: 141824 target oid smithi15736696-20 low_tier offset: 2146816 snap -1 2024-09-16T03:37:20.919 INFO:tasks.rados.rados.0.smithi157.stdout:270: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.919 INFO:tasks.rados.rados.0.smithi157.stdout:270: set_chunk oid 21 offset: 182272 length: 1471488 target oid smithi15736696-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 182272 snap -1 2024-09-16T03:37:20.919 INFO:tasks.rados.rados.0.smithi157.stdout:271: set_chunk oid 22 target oid 22 low_tier 2024-09-16T03:37:20.919 INFO:tasks.rados.rados.0.smithi157.stdout:271: set_chunk oid 22 offset: 1683456 length: 667136 target oid smithi15736696-22 low_tier offset: 1683456 snap -1 2024-09-16T03:37:20.919 INFO:tasks.rados.rados.0.smithi157.stdout:272: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.919 INFO:tasks.rados.rados.0.smithi157.stdout:272: set_chunk oid 23 offset: 243712 length: 873472 target oid smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 243712 snap -1 2024-09-16T03:37:20.919 INFO:tasks.rados.rados.0.smithi157.stdout:273: set_chunk oid 24 target oid 24 low_tier 2024-09-16T03:37:20.919 INFO:tasks.rados.rados.0.smithi157.stdout:273: set_chunk oid 24 offset: 212480 length: 1927168 target oid smithi15736696-24 low_tier offset: 212480 snap -1 2024-09-16T03:37:20.919 INFO:tasks.rados.rados.0.smithi157.stdout:274: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.919 INFO:tasks.rados.rados.0.smithi157.stdout:274: set_chunk oid 25 offset: 1014784 length: 136704 target oid smithi15736696-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1014784 snap -1 2024-09-16T03:37:20.919 INFO:tasks.rados.rados.0.smithi157.stdout:275: set_chunk oid 26 target oid 26 low_tier 2024-09-16T03:37:20.919 INFO:tasks.rados.rados.0.smithi157.stdout:275: set_chunk oid 26 offset: 951296 length: 410624 target oid smithi15736696-26 low_tier offset: 951296 snap -1 2024-09-16T03:37:20.920 INFO:tasks.rados.rados.0.smithi157.stdout:276: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.920 INFO:tasks.rados.rados.0.smithi157.stdout:276: set_chunk oid 27 offset: 658944 length: 529408 target oid smithi15736696-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 658944 snap -1 2024-09-16T03:37:20.920 INFO:tasks.rados.rados.0.smithi157.stdout:277: set_chunk oid 28 target oid 28 low_tier 2024-09-16T03:37:20.920 INFO:tasks.rados.rados.0.smithi157.stdout:277: set_chunk oid 28 offset: 422400 length: 386560 target oid smithi15736696-28 low_tier offset: 422400 snap -1 2024-09-16T03:37:20.920 INFO:tasks.rados.rados.0.smithi157.stdout:278: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.920 INFO:tasks.rados.rados.0.smithi157.stdout:278: set_chunk oid 29 offset: 247296 length: 448000 target oid smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 247296 snap -1 2024-09-16T03:37:20.920 INFO:tasks.rados.rados.0.smithi157.stdout:279: set_chunk oid 30 target oid 30 low_tier 2024-09-16T03:37:20.920 INFO:tasks.rados.rados.0.smithi157.stdout:279: set_chunk oid 30 offset: 658944 length: 696832 target oid smithi15736696-30 low_tier offset: 658944 snap -1 2024-09-16T03:37:20.920 INFO:tasks.rados.rados.0.smithi157.stdout:280: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.920 INFO:tasks.rados.rados.0.smithi157.stdout:280: set_chunk oid 31 offset: 198144 length: 1040896 target oid smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 198144 snap -1 2024-09-16T03:37:20.921 INFO:tasks.rados.rados.0.smithi157.stdout:281: set_chunk oid 32 target oid 32 low_tier 2024-09-16T03:37:20.921 INFO:tasks.rados.rados.0.smithi157.stdout:281: set_chunk oid 32 offset: 406016 length: 1667584 target oid smithi15736696-32 low_tier offset: 406016 snap -1 2024-09-16T03:37:20.921 INFO:tasks.rados.rados.0.smithi157.stdout:282: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.921 INFO:tasks.rados.rados.0.smithi157.stdout:282: set_chunk oid 33 offset: 2028032 length: 244736 target oid smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2028032 snap -1 2024-09-16T03:37:20.921 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.921 INFO:tasks.rados.rados.0.smithi157.stdout:270: finishing set_chunk to oid 21 2024-09-16T03:37:20.921 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 182272 length: 1471488 2024-09-16T03:37:20.921 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T03:37:20.921 INFO:tasks.rados.rados.0.smithi157.stdout:268: finishing set_chunk to oid 19 2024-09-16T03:37:20.921 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 883712 length: 1372672 2024-09-16T03:37:20.921 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T03:37:20.921 INFO:tasks.rados.rados.0.smithi157.stdout:268: done (15 left) 2024-09-16T03:37:20.921 INFO:tasks.rados.rados.0.smithi157.stdout:270: done (14 left) 2024-09-16T03:37:20.922 INFO:tasks.rados.rados.0.smithi157.stdout:283: set_chunk oid 34 target oid 34 low_tier 2024-09-16T03:37:20.922 INFO:tasks.rados.rados.0.smithi157.stdout:283: set_chunk oid 34 offset: 2057216 length: 433152 target oid smithi15736696-34 low_tier offset: 2057216 snap -1 2024-09-16T03:37:20.922 INFO:tasks.rados.rados.0.smithi157.stdout:284: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.922 INFO:tasks.rados.rados.0.smithi157.stdout:284: set_chunk oid 35 offset: 679424 length: 1018880 target oid smithi15736696-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 679424 snap -1 2024-09-16T03:37:20.922 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.922 INFO:tasks.rados.rados.0.smithi157.stdout:277: finishing set_chunk to oid 28 2024-09-16T03:37:20.922 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 422400 length: 386560 2024-09-16T03:37:20.922 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T03:37:20.922 INFO:tasks.rados.rados.0.smithi157.stdout:280: finishing set_chunk to oid 31 2024-09-16T03:37:20.922 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 198144 length: 1040896 2024-09-16T03:37:20.922 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:37:20.922 INFO:tasks.rados.rados.0.smithi157.stdout:277: done (15 left) 2024-09-16T03:37:20.922 INFO:tasks.rados.rados.0.smithi157.stdout:280: done (14 left) 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout:285: set_chunk oid 36 target oid 36 low_tier 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout:285: set_chunk oid 36 offset: 375296 length: 896000 target oid smithi15736696-36 low_tier offset: 375296 snap -1 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout:286: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout:286: set_chunk oid 37 offset: 86528 length: 1526272 target oid smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 86528 snap -1 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout:272: finishing set_chunk to oid 23 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 243712 length: 873472 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout:272: done (15 left) 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout:287: set_chunk oid 38 target oid 38 low_tier 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout:287: set_chunk oid 38 offset: 346112 length: 1770496 target oid smithi15736696-38 low_tier offset: 346112 snap -1 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout:273: finishing set_chunk to oid 24 2024-09-16T03:37:20.923 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 212480 length: 1927168 2024-09-16T03:37:20.924 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T03:37:20.924 INFO:tasks.rados.rados.0.smithi157.stdout:273: done (15 left) 2024-09-16T03:37:20.924 INFO:tasks.rados.rados.0.smithi157.stdout:288: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.924 INFO:tasks.rados.rados.0.smithi157.stdout:288: set_chunk oid 39 offset: 176640 length: 2063360 target oid smithi15736696-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 176640 snap -1 2024-09-16T03:37:20.924 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.924 INFO:tasks.rados.rados.0.smithi157.stdout:279: finishing set_chunk to oid 30 2024-09-16T03:37:20.924 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 658944 length: 696832 2024-09-16T03:37:20.924 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:37:20.925 INFO:tasks.rados.rados.0.smithi157.stdout:279: done (15 left) 2024-09-16T03:37:20.925 INFO:tasks.rados.rados.0.smithi157.stdout:289: set_chunk oid 40 target oid 40 low_tier 2024-09-16T03:37:20.925 INFO:tasks.rados.rados.0.smithi157.stdout:289: set_chunk oid 40 offset: 18944 length: 1477120 target oid smithi15736696-40 low_tier offset: 18944 snap -1 2024-09-16T03:37:20.925 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:37:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:37:20.925 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:20.928 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.928 INFO:tasks.rados.rados.0.smithi157.stdout:276: finishing set_chunk to oid 27 2024-09-16T03:37:20.928 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 658944 length: 529408 2024-09-16T03:37:20.928 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:20.928 INFO:tasks.rados.rados.0.smithi157.stdout:276: done (15 left) 2024-09-16T03:37:20.928 INFO:tasks.rados.rados.0.smithi157.stdout:290: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.928 INFO:tasks.rados.rados.0.smithi157.stdout:290: set_chunk oid 41 offset: 225792 length: 262144 target oid smithi15736696-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 225792 snap -1 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout:283: finishing set_chunk to oid 34 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 2057216 length: 433152 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout:283: done (15 left) 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout:291: set_chunk oid 42 target oid 42 low_tier 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout:291: set_chunk oid 42 offset: 1402880 length: 561664 target oid smithi15736696-42 low_tier offset: 1402880 snap -1 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout:287: finishing set_chunk to oid 38 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 346112 length: 1770496 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout:287: done (15 left) 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout:292: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.929 INFO:tasks.rados.rados.0.smithi157.stdout:292: set_chunk oid 43 offset: 1700864 length: 51200 target oid smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1700864 snap -1 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout:285: finishing set_chunk to oid 36 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 375296 length: 896000 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout:285: done (15 left) 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout:293: set_chunk oid 44 target oid 44 low_tier 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout:293: set_chunk oid 44 offset: 30720 length: 1741824 target oid smithi15736696-44 low_tier offset: 30720 snap -1 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout:291: finishing set_chunk to oid 42 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1402880 length: 561664 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout:291: done (15 left) 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout:294: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:20.930 INFO:tasks.rados.rados.0.smithi157.stdout:294: set_chunk oid 45 offset: 338432 length: 1764352 target oid smithi15736696-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 338432 snap -1 2024-09-16T03:37:20.931 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:20.932 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:37:20.951 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:20.949+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:21.010 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:37:21.020 INFO:tasks.rados.rados.0.smithi157.stdout:271: finishing set_chunk to oid 22 2024-09-16T03:37:21.020 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 1683456 length: 667136 2024-09-16T03:37:21.020 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T03:37:21.021 INFO:tasks.rados.rados.0.smithi157.stdout:267: finishing set_chunk to oid 18 2024-09-16T03:37:21.021 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 403968 length: 512512 2024-09-16T03:37:21.021 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T03:37:21.021 INFO:tasks.rados.rados.0.smithi157.stdout:286: finishing set_chunk to oid 37 2024-09-16T03:37:21.021 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 86528 length: 1526272 2024-09-16T03:37:21.021 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T03:37:21.021 INFO:tasks.rados.rados.0.smithi157.stdout:275: finishing set_chunk to oid 26 2024-09-16T03:37:21.021 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 951296 length: 410624 2024-09-16T03:37:21.021 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:37:21.021 INFO:tasks.rados.rados.0.smithi157.stdout:290: finishing set_chunk to oid 41 2024-09-16T03:37:21.021 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 225792 length: 262144 2024-09-16T03:37:21.021 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:37:21.021 INFO:tasks.rados.rados.0.smithi157.stdout:274: finishing set_chunk to oid 25 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1014784 length: 136704 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:267: done (15 left) 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:271: done (14 left) 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:274: done (13 left) 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:275: done (12 left) 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:286: done (11 left) 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:290: done (10 left) 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:295: set_chunk oid 46 target oid 46 low_tier 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:295: set_chunk oid 46 offset: 1522688 length: 924672 target oid smithi15736696-46 low_tier offset: 1522688 snap -1 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:296: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:296: set_chunk oid 47 offset: 97280 length: 2065920 target oid smithi15736696-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 97280 snap -1 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:297: set_chunk oid 48 target oid 48 low_tier 2024-09-16T03:37:21.022 INFO:tasks.rados.rados.0.smithi157.stdout:297: set_chunk oid 48 offset: 614912 length: 1162240 target oid smithi15736696-48 low_tier offset: 614912 snap -1 2024-09-16T03:37:21.023 INFO:tasks.rados.rados.0.smithi157.stdout:298: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.023 INFO:tasks.rados.rados.0.smithi157.stdout:298: set_chunk oid 49 offset: 464896 length: 706560 target oid smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 464896 snap -1 2024-09-16T03:37:21.023 INFO:tasks.rados.rados.0.smithi157.stdout:299: set_chunk oid 50 target oid 50 low_tier 2024-09-16T03:37:21.023 INFO:tasks.rados.rados.0.smithi157.stdout:299: set_chunk oid 50 offset: 1143808 length: 1275904 target oid smithi15736696-50 low_tier offset: 1143808 snap -1 2024-09-16T03:37:21.023 INFO:tasks.rados.rados.0.smithi157.stdout:300: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.023 INFO:tasks.rados.rados.0.smithi157.stdout:300: set_chunk oid 1 offset: 875520 length: 210432 target oid smithi15736696-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 875520 snap -1 2024-09-16T03:37:21.023 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.023 INFO:tasks.rados.rados.0.smithi157.stdout:281: finishing set_chunk to oid 32 2024-09-16T03:37:21.023 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 406016 length: 1667584 2024-09-16T03:37:21.023 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:37:21.023 INFO:tasks.rados.rados.0.smithi157.stdout:294: finishing set_chunk to oid 45 2024-09-16T03:37:21.023 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 338432 length: 1764352 2024-09-16T03:37:21.023 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout:281: done (15 left) 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout:294: done (14 left) 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout:301: set_chunk oid 2 target oid 2 low_tier 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout:301: set_chunk oid 2 offset: 255488 length: 662016 target oid smithi15736696-2 low_tier offset: 255488 snap -1 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout:302: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout:302: set_chunk oid 3 offset: 892416 length: 364544 target oid smithi15736696-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 892416 snap -1 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout:284: finishing set_chunk to oid 35 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 679424 length: 1018880 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout:284: done (15 left) 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout:303: set_chunk oid 4 target oid 4 low_tier 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout:303: set_chunk oid 4 offset: 1030656 length: 828928 target oid smithi15736696-4 low_tier offset: 1030656 snap -1 2024-09-16T03:37:21.024 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.025 INFO:tasks.rados.rados.0.smithi157.stdout:288: finishing set_chunk to oid 39 2024-09-16T03:37:21.025 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 176640 length: 2063360 2024-09-16T03:37:21.025 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T03:37:21.025 INFO:tasks.rados.rados.0.smithi157.stdout:288: done (15 left) 2024-09-16T03:37:21.025 INFO:tasks.rados.rados.0.smithi157.stdout:304: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.025 INFO:tasks.rados.rados.0.smithi157.stdout:304: set_chunk oid 5 offset: 714752 length: 94208 target oid smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 714752 snap -1 2024-09-16T03:37:21.025 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.025 INFO:tasks.rados.rados.0.smithi157.stdout:295: finishing set_chunk to oid 46 2024-09-16T03:37:21.025 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1522688 length: 924672 2024-09-16T03:37:21.025 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:37:21.025 INFO:tasks.rados.rados.0.smithi157.stdout:297: finishing set_chunk to oid 48 2024-09-16T03:37:21.025 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 614912 length: 1162240 2024-09-16T03:37:21.025 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout:300: finishing set_chunk to oid 1 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 875520 length: 210432 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout:295: done (15 left) 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout:297: done (14 left) 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout:300: done (13 left) 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout:305: set_chunk oid 6 target oid 6 low_tier 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout:305: set_chunk oid 6 offset: 465408 length: 276480 target oid smithi15736696-6 low_tier offset: 465408 snap -1 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout:306: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout:306: set_chunk oid 7 offset: 121344 length: 310272 target oid smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 121344 snap -1 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout:307: set_chunk oid 8 target oid 8 low_tier 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout:307: set_chunk oid 8 offset: 643584 length: 557056 target oid smithi15736696-8 low_tier offset: 643584 snap -1 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.026 INFO:tasks.rados.rados.0.smithi157.stdout:296: finishing set_chunk to oid 47 2024-09-16T03:37:21.027 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 97280 length: 2065920 2024-09-16T03:37:21.027 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T03:37:21.027 INFO:tasks.rados.rados.0.smithi157.stdout:296: done (15 left) 2024-09-16T03:37:21.027 INFO:tasks.rados.rados.0.smithi157.stdout:308: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.027 INFO:tasks.rados.rados.0.smithi157.stdout:308: set_chunk oid 9 offset: 275456 length: 1419264 target oid smithi15736696-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 275456 snap -1 2024-09-16T03:37:21.027 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.027 INFO:tasks.rados.rados.0.smithi157.stdout:293: finishing set_chunk to oid 44 2024-09-16T03:37:21.027 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 30720 length: 1741824 2024-09-16T03:37:21.027 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T03:37:21.027 INFO:tasks.rados.rados.0.smithi157.stdout:293: done (15 left) 2024-09-16T03:37:21.027 INFO:tasks.rados.rados.0.smithi157.stdout:309: set_chunk oid 10 target oid 10 low_tier 2024-09-16T03:37:21.027 INFO:tasks.rados.rados.0.smithi157.stdout:309: set_chunk oid 10 offset: 470528 length: 516608 target oid smithi15736696-10 low_tier offset: 470528 snap -1 2024-09-16T03:37:21.027 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.028 INFO:tasks.rados.rados.0.smithi157.stdout:303: finishing set_chunk to oid 4 2024-09-16T03:37:21.028 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 1030656 length: 828928 2024-09-16T03:37:21.028 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:37:21.028 INFO:tasks.rados.rados.0.smithi157.stdout:303: done (15 left) 2024-09-16T03:37:21.028 INFO:tasks.rados.rados.0.smithi157.stdout:310: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.028 INFO:tasks.rados.rados.0.smithi157.stdout:310: set_chunk oid 11 offset: 741376 length: 629248 target oid smithi15736696-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 741376 snap -1 2024-09-16T03:37:21.028 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.028 INFO:tasks.rados.rados.0.smithi157.stdout:305: finishing set_chunk to oid 6 2024-09-16T03:37:21.028 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 465408 length: 276480 2024-09-16T03:37:21.028 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:37:21.028 INFO:tasks.rados.rados.0.smithi157.stdout:307: finishing set_chunk to oid 8 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 643584 length: 557056 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout:305: done (15 left) 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout:307: done (14 left) 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout:311: set_chunk oid 12 target oid 12 low_tier 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout:311: set_chunk oid 12 offset: 263168 length: 1913344 target oid smithi15736696-12 low_tier offset: 263168 snap -1 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout:312: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout:312: set_chunk oid 13 offset: 601600 length: 1342976 target oid smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 601600 snap -1 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout:306: finishing set_chunk to oid 7 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 121344 length: 310272 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout:306: done (15 left) 2024-09-16T03:37:21.029 INFO:tasks.rados.rados.0.smithi157.stdout:313: set_chunk oid 14 target oid 14 low_tier 2024-09-16T03:37:21.030 INFO:tasks.rados.rados.0.smithi157.stdout:313: set_chunk oid 14 offset: 635392 length: 260096 target oid smithi15736696-14 low_tier offset: 635392 snap -1 2024-09-16T03:37:21.030 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.030 INFO:tasks.rados.rados.0.smithi157.stdout:308: finishing set_chunk to oid 9 2024-09-16T03:37:21.030 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 275456 length: 1419264 2024-09-16T03:37:21.030 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:37:21.030 INFO:tasks.rados.rados.0.smithi157.stdout:308: done (15 left) 2024-09-16T03:37:21.030 INFO:tasks.rados.rados.0.smithi157.stdout:314: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.030 INFO:tasks.rados.rados.0.smithi157.stdout:314: set_chunk oid 15 offset: 51200 length: 1626112 target oid smithi15736696-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 51200 snap -1 2024-09-16T03:37:21.030 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.030 INFO:tasks.rados.rados.0.smithi157.stdout:312: finishing set_chunk to oid 13 2024-09-16T03:37:21.030 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 601600 length: 1342976 2024-09-16T03:37:21.030 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T03:37:21.030 INFO:tasks.rados.rados.0.smithi157.stdout:312: done (15 left) 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout:315: set_chunk oid 16 target oid 16 low_tier 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout:315: set_chunk oid 16 offset: 155136 length: 398336 target oid smithi15736696-16 low_tier offset: 155136 snap -1 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout:311: finishing set_chunk to oid 12 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 263168 length: 1913344 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout:311: done (15 left) 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout:316: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout:316: set_chunk oid 17 offset: 249344 length: 1667584 target oid smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 249344 snap -1 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout:302: finishing set_chunk to oid 3 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout:302: done (15 left) 2024-09-16T03:37:21.031 INFO:tasks.rados.rados.0.smithi157.stdout:317: set_chunk oid 18 target oid 18 low_tier 2024-09-16T03:37:21.032 INFO:tasks.rados.rados.0.smithi157.stdout:317: set_chunk oid 18 offset: 1033216 length: 641024 target oid smithi15736696-18 low_tier offset: 1033216 snap -1 2024-09-16T03:37:21.032 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.032 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:21.031+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:21.038 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:21.038 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:37:21.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:37:21.038 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:21.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.038 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:41 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:21.038 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.156949+0000", 2024-09-16T03:37:21.038 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.879191939, 2024-09-16T03:37:21.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021202600000000001, 2024-09-16T03:37:21.038 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.038 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156949+0000", 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156949+0000", 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156949+0000", 2024-09-16T03:37:21.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9100000000000004e-07 2024-09-16T03:37:21.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157324+0000", 2024-09-16T03:37:21.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037452500000000001 2024-09-16T03:37:21.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157340+0000", 2024-09-16T03:37:21.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6426000000000002e-05 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157350+0000", 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6150000000000003e-06 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157415+0000", 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5425999999999993e-05 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157441+0000", 2024-09-16T03:37:21.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6177999999999999e-05 2024-09-16T03:37:21.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.159020+0000", 2024-09-16T03:37:21.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001578325 2024-09-16T03:37:21.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.159069+0000", 2024-09-16T03:37:21.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9274000000000003e-05 2024-09-16T03:37:21.042 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.042 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.043 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.043 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:21.043 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:37:21.043 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.8500602290000003, 2024-09-16T03:37:21.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102630163, 2024-09-16T03:37:21.043 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.044 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.044 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.044 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:37:21.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:37:21.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186081+0000", 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5600000000000005e-07 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186970+0000", 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00088889000000000004 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186979+0000", 2024-09-16T03:37:21.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0620000000000007e-06 2024-09-16T03:37:21.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186986+0000", 2024-09-16T03:37:21.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:37:21.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287184+0000", 2024-09-16T03:37:21.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100198036 2024-09-16T03:37:21.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287199+0000", 2024-09-16T03:37:21.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5458999999999999e-05 2024-09-16T03:37:21.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288659+0000", 2024-09-16T03:37:21.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014601900000000001 2024-09-16T03:37:21.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288711+0000", 2024-09-16T03:37:21.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1088000000000003e-05 2024-09-16T03:37:21.048 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.048 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.048 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.049 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:21.049 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:37:21.049 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.8490237960000009, 2024-09-16T03:37:21.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10205370900000001, 2024-09-16T03:37:21.049 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.049 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.049 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.049 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:37:21.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:37:21.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187118+0000", 2024-09-16T03:37:21.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8599999999999998e-07 2024-09-16T03:37:21.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188062+0000", 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094447300000000004 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188070+0000", 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.695e-06 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188074+0000", 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0709999999999996e-06 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287842+0000", 2024-09-16T03:37:21.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099768478999999993 2024-09-16T03:37:21.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287864+0000", 2024-09-16T03:37:21.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1679999999999999e-05 2024-09-16T03:37:21.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289137+0000", 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012732430000000001 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289171+0000", 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3382e-05 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:42 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:21.053 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:37:21.054 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.7789418880000003, 2024-09-16T03:37:21.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10172858, 2024-09-16T03:37:21.054 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.054 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.054 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.054 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:37:21.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:37:21.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257201+0000", 2024-09-16T03:37:21.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:37:21.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257536+0000", 2024-09-16T03:37:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033513899999999998 2024-09-16T03:37:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257555+0000", 2024-09-16T03:37:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8547999999999999e-05 2024-09-16T03:37:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257566+0000", 2024-09-16T03:37:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1379999999999999e-05 2024-09-16T03:37:21.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357720+0000", 2024-09-16T03:37:21.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100153532 2024-09-16T03:37:21.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357734+0000", 2024-09-16T03:37:21.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4491000000000001e-05 2024-09-16T03:37:21.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358896+0000", 2024-09-16T03:37:21.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011620599999999999 2024-09-16T03:37:21.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358927+0000", 2024-09-16T03:37:21.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1158999999999998e-05 2024-09-16T03:37:21.058 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.058 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.058 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.058 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:43 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:21.058 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:37:21.059 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.7771579759999998, 2024-09-16T03:37:21.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101764453, 2024-09-16T03:37:21.059 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.059 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.059 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.059 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:37:21.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:37:21.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258984+0000", 2024-09-16T03:37:21.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5689999999999999e-06 2024-09-16T03:37:21.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258994+0000", 2024-09-16T03:37:21.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259008+0000", 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4151999999999999e-05 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359073+0000", 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100065628 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359110+0000", 2024-09-16T03:37:21.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7045999999999997e-05 2024-09-16T03:37:21.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359124+0000", 2024-09-16T03:37:21.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.382e-05 2024-09-16T03:37:21.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360716+0000", 2024-09-16T03:37:21.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015919829999999999 2024-09-16T03:37:21.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360747+0000", 2024-09-16T03:37:21.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0874999999999998e-05 2024-09-16T03:37:21.063 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.063 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.063 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.063 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:47 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:21.063 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.288337+0000", 2024-09-16T03:37:21.063 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.7478037460000007, 2024-09-16T03:37:21.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00086119400000000002, 2024-09-16T03:37:21.063 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.064 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.064 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.064 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288337+0000", 2024-09-16T03:37:21.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288337+0000", 2024-09-16T03:37:21.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288338+0000", 2024-09-16T03:37:21.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.539e-06 2024-09-16T03:37:21.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288346+0000", 2024-09-16T03:37:21.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5009999999999998e-06 2024-09-16T03:37:21.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288355+0000", 2024-09-16T03:37:21.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9709999999999993e-06 2024-09-16T03:37:21.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288365+0000", 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0141e-05 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288735+0000", 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037035300000000002 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288747+0000", 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1765e-05 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289173+0000", 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042624699999999999 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289198+0000", 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4677000000000001e-05 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.068 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:61 4.5 e39/32 4:ad0b9852:::smithi15736696-16:head v 39'4, mlcod=0'0)", 2024-09-16T03:37:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.359093+0000", 2024-09-16T03:37:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.6770477069999998, 2024-09-16T03:37:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001688403, 2024-09-16T03:37:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359093+0000", 2024-09-16T03:37:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359093+0000", 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359094+0000", 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0999999999999997e-07 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359205+0000", 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000111025 2024-09-16T03:37:21.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359211+0000", 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8930000000000002e-06 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359215+0000", 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2300000000000002e-06 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359251+0000", 2024-09-16T03:37:21.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6585000000000003e-05 2024-09-16T03:37:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359270+0000", 2024-09-16T03:37:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8785000000000001e-05 2024-09-16T03:37:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360754+0000", 2024-09-16T03:37:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014841240000000001 2024-09-16T03:37:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360781+0000", 2024-09-16T03:37:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6951e-05 2024-09-16T03:37:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:62 4.5 e39/32 4:ad0b9852:::smithi15736696-16:head v 39'5, mlcod=0'0)", 2024-09-16T03:37:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.359222+0000", 2024-09-16T03:37:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.6769185800000006, 2024-09-16T03:37:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020277289999999998, 2024-09-16T03:37:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.073 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359222+0000", 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359222+0000", 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359222+0000", 2024-09-16T03:37:21.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3999999999999997e-07 2024-09-16T03:37:21.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359341+0000", 2024-09-16T03:37:21.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011825899999999999 2024-09-16T03:37:21.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359345+0000", 2024-09-16T03:37:21.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1180000000000002e-06 2024-09-16T03:37:21.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359349+0000", 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.941e-06 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359991+0000", 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00064214700000000003 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360010+0000", 2024-09-16T03:37:21.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9028000000000001e-05 2024-09-16T03:37:21.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.361209+0000", 2024-09-16T03:37:21.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011994460000000001 2024-09-16T03:37:21.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.361250+0000", 2024-09-16T03:37:21.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0450000000000001e-05 2024-09-16T03:37:21.077 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.077 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.077 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.078 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:63 4.5 e39/32 4:ad0b9852:::smithi15736696-16:head v 39'6, mlcod=0'0)", 2024-09-16T03:37:21.078 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.359355+0000", 2024-09-16T03:37:21.078 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.6767859440000006, 2024-09-16T03:37:21.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019271519999999999, 2024-09-16T03:37:21.078 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.078 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.078 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.078 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359355+0000", 2024-09-16T03:37:21.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359355+0000", 2024-09-16T03:37:21.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359355+0000", 2024-09-16T03:37:21.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.22e-07 2024-09-16T03:37:21.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359357+0000", 2024-09-16T03:37:21.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.187e-06 2024-09-16T03:37:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359360+0000", 2024-09-16T03:37:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.745e-06 2024-09-16T03:37:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359363+0000", 2024-09-16T03:37:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7039999999999999e-06 2024-09-16T03:37:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360603+0000", 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012399850000000001 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360614+0000", 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1599e-05 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.361254+0000", 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063991999999999999 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.361282+0000", 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7690000000000001e-05 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:260 4.5 e40/32 4:aff4cc41:::smithi15736696-11:head v 40'13, mlcod=40'11)", 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:15.196507+0000", 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.83963403, 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016026790000000001, 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.083 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.084 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.084 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.196507+0000", 2024-09-16T03:37:21.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.196507+0000", 2024-09-16T03:37:21.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.196507+0000", 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3200000000000001e-07 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.196516+0000", 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.115e-06 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.196526+0000", 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8439999999999993e-06 2024-09-16T03:37:21.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.196539+0000", 2024-09-16T03:37:21.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.342e-05 2024-09-16T03:37:21.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.197579+0000", 2024-09-16T03:37:21.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010397029999999999 2024-09-16T03:37:21.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.197595+0000", 2024-09-16T03:37:21.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5471000000000001e-05 2024-09-16T03:37:21.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.198002+0000", 2024-09-16T03:37:21.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000407315 2024-09-16T03:37:21.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.198109+0000", 2024-09-16T03:37:21.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010747900000000001 2024-09-16T03:37:21.087 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.087 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.088 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.088 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:292 4.5 e41/32 4:a11e1d7c:::smithi15736696-43:head v 41'14, mlcod=40'12)", 2024-09-16T03:37:21.088 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.005011+0000", 2024-09-16T03:37:21.088 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.0311299700000003, 2024-09-16T03:37:21.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00055603300000000005, 2024-09-16T03:37:21.088 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.088 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.088 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.088 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.005011+0000", 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.005011+0000", 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.005012+0000", 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.005019+0000", 2024-09-16T03:37:21.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4420000000000001e-06 2024-09-16T03:37:21.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.005031+0000", 2024-09-16T03:37:21.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1323e-05 2024-09-16T03:37:21.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.005037+0000", 2024-09-16T03:37:21.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4359999999999999e-06 2024-09-16T03:37:21.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.005101+0000", 2024-09-16T03:37:21.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4286000000000001e-05 2024-09-16T03:37:21.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.005123+0000", 2024-09-16T03:37:21.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1336e-05 2024-09-16T03:37:21.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.092 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.005534+0000", 2024-09-16T03:37:21.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00041144400000000001 2024-09-16T03:37:21.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.092 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.092 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.005567+0000", 2024-09-16T03:37:21.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2623000000000003e-05 2024-09-16T03:37:21.092 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.092 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.092 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.092 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:265 4.5 e43/32 4:ad0b9852:::smithi15736696-16:head v 43'15, mlcod=40'13)", 2024-09-16T03:37:21.093 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:17.589957+0000", 2024-09-16T03:37:21.093 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.4461836450000001, 2024-09-16T03:37:21.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00076321699999999995, 2024-09-16T03:37:21.093 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.093 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.093 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.093 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.093 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.093 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:17.589957+0000", 2024-09-16T03:37:21.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:17.589957+0000", 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:17.589957+0000", 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1899999999999998e-07 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:17.589962+0000", 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6079999999999998e-06 2024-09-16T03:37:21.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:17.589975+0000", 2024-09-16T03:37:21.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3309000000000001e-05 2024-09-16T03:37:21.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:17.589987+0000", 2024-09-16T03:37:21.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2051e-05 2024-09-16T03:37:21.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:17.590066+0000", 2024-09-16T03:37:21.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8726000000000005e-05 2024-09-16T03:37:21.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:17.590092+0000", 2024-09-16T03:37:21.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5914999999999998e-05 2024-09-16T03:37:21.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:17.590665+0000", 2024-09-16T03:37:21.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057277499999999998 2024-09-16T03:37:21.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:17.590720+0000", 2024-09-16T03:37:21.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5513999999999997e-05 2024-09-16T03:37:21.097 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.097 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.097 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.097 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:388 4.5 e45/32 4:a4fbca73:::smithi15736696-39:head v 45'18, mlcod=43'16)", 2024-09-16T03:37:21.097 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.207615+0000", 2024-09-16T03:37:21.097 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.82852600300000001, 2024-09-16T03:37:21.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00090024899999999999, 2024-09-16T03:37:21.097 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.207615+0000", 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.207615+0000", 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.207617+0000", 2024-09-16T03:37:21.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6220000000000001e-06 2024-09-16T03:37:21.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.207629+0000", 2024-09-16T03:37:21.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1409e-05 2024-09-16T03:37:21.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.207647+0000", 2024-09-16T03:37:21.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8525e-05 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.207658+0000", 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0638e-05 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.207744+0000", 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5914000000000004e-05 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.207774+0000", 2024-09-16T03:37:21.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0463999999999998e-05 2024-09-16T03:37:21.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.208468+0000", 2024-09-16T03:37:21.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069374599999999997 2024-09-16T03:37:21.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.208515+0000", 2024-09-16T03:37:21.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6931000000000002e-05 2024-09-16T03:37:21.101 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.101 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:415 4.5 e45/32 4:ad0b9852:::smithi15736696-16:head v 45'19, mlcod=45'17)", 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.507757+0000", 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.52838412000000001, 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00061881399999999997, 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.507757+0000", 2024-09-16T03:37:21.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.507757+0000", 2024-09-16T03:37:21.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.507759+0000", 2024-09-16T03:37:21.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4150000000000002e-06 2024-09-16T03:37:21.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.507773+0000", 2024-09-16T03:37:21.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3607e-05 2024-09-16T03:37:21.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.507786+0000", 2024-09-16T03:37:21.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3533000000000001e-05 2024-09-16T03:37:21.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.507796+0000", 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0370000000000001e-05 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.507871+0000", 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4868000000000002e-05 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.507908+0000", 2024-09-16T03:37:21.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6789000000000003e-05 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.508346+0000", 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043748000000000002 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.508375+0000", 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9751999999999999e-05 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.106 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.107 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.107 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:21.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-09-16T03:37:21.109 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:37:21.118 INFO:tasks.rados.rados.0.smithi157.stdout:278: finishing set_chunk to oid 29 2024-09-16T03:37:21.118 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 247296 length: 448000 2024-09-16T03:37:21.118 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:37:21.118 INFO:tasks.rados.rados.0.smithi157.stdout:289: finishing set_chunk to oid 40 2024-09-16T03:37:21.118 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 18944 length: 1477120 2024-09-16T03:37:21.118 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T03:37:21.118 INFO:tasks.rados.rados.0.smithi157.stdout:278: done (15 left) 2024-09-16T03:37:21.118 INFO:tasks.rados.rados.0.smithi157.stdout:289: done (14 left) 2024-09-16T03:37:21.118 INFO:tasks.rados.rados.0.smithi157.stdout:318: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout:318: set_chunk oid 19 offset: 198656 length: 1055232 target oid smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 198656 snap -1 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout:318 retry set_chunk ! 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 15 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout:299: finishing set_chunk to oid 50 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1143808 length: 1275904 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout:299: done (14 left) 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 14 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout:316: finishing set_chunk to oid 17 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 249344 length: 1667584 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout:316: done (13 left) 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 13 2024-09-16T03:37:21.119 INFO:tasks.rados.rados.0.smithi157.stdout:301: finishing set_chunk to oid 2 2024-09-16T03:37:21.120 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 255488 length: 662016 2024-09-16T03:37:21.120 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T03:37:21.120 INFO:tasks.rados.rados.0.smithi157.stdout:301: done (12 left) 2024-09-16T03:37:21.120 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 12 2024-09-16T03:37:21.120 INFO:tasks.rados.rados.0.smithi157.stdout:317: finishing set_chunk to oid 18 2024-09-16T03:37:21.120 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 1033216 length: 641024 2024-09-16T03:37:21.120 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T03:37:21.120 INFO:tasks.rados.rados.0.smithi157.stdout:317: done (11 left) 2024-09-16T03:37:21.120 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 11 2024-09-16T03:37:21.120 INFO:tasks.rados.rados.0.smithi157.stdout:304: finishing set_chunk to oid 5 2024-09-16T03:37:21.120 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 714752 length: 94208 2024-09-16T03:37:21.120 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T03:37:21.120 INFO:tasks.rados.rados.0.smithi157.stdout:304: done (10 left) 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 10 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout:318: finishing set_chunk to oid 19 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 198656 length: 1055232 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout:318: done (9 left) 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 9 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout:313: finishing set_chunk to oid 14 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 635392 length: 260096 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout:313: done (8 left) 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 8 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout:314: finishing set_chunk to oid 15 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 51200 length: 1626112 2024-09-16T03:37:21.121 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:21.122 INFO:tasks.rados.rados.0.smithi157.stdout:314: done (7 left) 2024-09-16T03:37:21.122 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 7 2024-09-16T03:37:21.122 INFO:tasks.rados.rados.0.smithi157.stdout:310: finishing set_chunk to oid 11 2024-09-16T03:37:21.122 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 741376 length: 629248 2024-09-16T03:37:21.122 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T03:37:21.122 INFO:tasks.rados.rados.0.smithi157.stdout:310: done (6 left) 2024-09-16T03:37:21.122 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 6 2024-09-16T03:37:21.153 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:21.151+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:21.219 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:21.219 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:21.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.219 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:442 4.5 e45/32)", 2024-09-16T03:37:21.219 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.217548+0000", 2024-09-16T03:37:21.219 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.00067139999999999995, 2024-09-16T03:37:21.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00068386800000000002, 2024-09-16T03:37:21.220 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.220 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.220 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "no flag points reached", 2024-09-16T03:37:21.220 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.217548+0000", 2024-09-16T03:37:21.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.217548+0000", 2024-09-16T03:37:21.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.217549+0000", 2024-09-16T03:37:21.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.232e-06 2024-09-16T03:37:21.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.217556+0000", 2024-09-16T03:37:21.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0550000000000002e-06 2024-09-16T03:37:21.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.217568+0000", 2024-09-16T03:37:21.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1503e-05 2024-09-16T03:37:21.222 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.222 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.222 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.222 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.222 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-16T03:37:21.222 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 1 2024-09-16T03:37:21.223 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:21.230 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:37:21.250 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 reset_purged_snaps_last 2024-09-16T03:37:21.275 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:21.273+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:21.319 INFO:tasks.rados.rados.0.smithi157.stdout:269: finishing set_chunk to oid 20 2024-09-16T03:37:21.319 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 9 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T03:37:21.320 INFO:tasks.rados.rados.0.smithi157.stdout:269: done (5 left) 2024-09-16T03:37:21.320 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 5 2024-09-16T03:37:21.320 INFO:tasks.rados.rados.0.smithi157.stdout:282: finishing set_chunk to oid 33 2024-09-16T03:37:21.320 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:37:21.320 INFO:tasks.rados.rados.0.smithi157.stdout:282: done (4 left) 2024-09-16T03:37:21.320 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 4 2024-09-16T03:37:21.320 INFO:tasks.rados.rados.0.smithi157.stdout:292: finishing set_chunk to oid 43 2024-09-16T03:37:21.320 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 12 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T03:37:21.320 INFO:tasks.rados.rados.0.smithi157.stdout:292: done (3 left) 2024-09-16T03:37:21.320 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 3 2024-09-16T03:37:21.334 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:21.334 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:21.334 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:37:21.334 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:37:21.334 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:21.335 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:21.334+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:21.343 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:37:21.420 INFO:tasks.rados.rados.0.smithi157.stdout:309: finishing set_chunk to oid 10 2024-09-16T03:37:21.420 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:37:21.420 INFO:tasks.rados.rados.0.smithi157.stdout:309: done (2 left) 2024-09-16T03:37:21.420 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 2 2024-09-16T03:37:21.421 INFO:tasks.rados.rados.0.smithi157.stdout:298: finishing set_chunk to oid 49 2024-09-16T03:37:21.422 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T03:37:21.422 INFO:tasks.rados.rados.0.smithi157.stdout:298: done (1 left) 2024-09-16T03:37:21.422 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 1 2024-09-16T03:37:21.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 reset_purged_snaps_last 2024-09-16T03:37:21.437 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:21.435+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:21.488 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:37:21.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:37:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T03:37:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:37:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.408477433000002, 2024-09-16T03:37:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.88203620400000005, 2024-09-16T03:37:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.489 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:37:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:37:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:37:21.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076767+0000", 2024-09-16T03:37:21.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9300000000000002e-06 2024-09-16T03:37:21.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076774+0000", 2024-09-16T03:37:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6630000000000004e-06 2024-09-16T03:37:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076785+0000", 2024-09-16T03:37:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1221e-05 2024-09-16T03:37:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076796+0000", 2024-09-16T03:37:21.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0428000000000001e-05 2024-09-16T03:37:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076856+0000", 2024-09-16T03:37:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:37:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:37:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076863+0000", 2024-09-16T03:37:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9840000000000004e-06 2024-09-16T03:37:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855770+0000", 2024-09-16T03:37:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.77890736999999999 2024-09-16T03:37:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855902+0000", 2024-09-16T03:37:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013207 2024-09-16T03:37:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.856054+0000", 2024-09-16T03:37:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015220899999999999 2024-09-16T03:37:21.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.858407+0000", 2024-09-16T03:37:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023530069999999998 2024-09-16T03:37:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958598+0000", 2024-09-16T03:37:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100190194 2024-09-16T03:37:21.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958716+0000", 2024-09-16T03:37:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118589 2024-09-16T03:37:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958761+0000", 2024-09-16T03:37:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5120000000000002e-05 2024-09-16T03:37:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958802+0000", 2024-09-16T03:37:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0383999999999998e-05 2024-09-16T03:37:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:37:21.497 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:37:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 26.890061255999999, 2024-09-16T03:37:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70508991300000001, 2024-09-16T03:37:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 52 2024-09-16T03:37:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:37:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:37:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:37:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6300000000000005e-07 2024-09-16T03:37:21.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595187+0000", 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9790000000000001e-06 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595194+0000", 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6490000000000002e-06 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595208+0000", 2024-09-16T03:37:21.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4139e-05 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801931+0000", 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.206723083 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801990+0000", 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8625999999999997e-05 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802066+0000", 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6553999999999999e-05 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802504+0000", 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043775200000000002 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.199998+0000", 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39749365599999997 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300151+0000", 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015310099999999 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300210+0000", 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9376999999999997e-05 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300271+0000", 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1334000000000006e-05 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.503 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.369956+0000", 2024-09-16T03:37:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.115286942000001, 2024-09-16T03:37:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60610467099999998, 2024-09-16T03:37:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.504 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 69 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.369956+0000", 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.369956+0000", 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.369957+0000", 2024-09-16T03:37:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3099999999999997e-07 2024-09-16T03:37:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370028+0000", 2024-09-16T03:37:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1404999999999998e-05 2024-09-16T03:37:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370036+0000", 2024-09-16T03:37:21.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.79e-06 2024-09-16T03:37:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370042+0000", 2024-09-16T03:37:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.494e-06 2024-09-16T03:37:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572455+0000", 2024-09-16T03:37:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20241295100000001 2024-09-16T03:37:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572512+0000", 2024-09-16T03:37:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7071000000000003e-05 2024-09-16T03:37:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572602+0000", 2024-09-16T03:37:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9275000000000002e-05 2024-09-16T03:37:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673919+0000", 2024-09-16T03:37:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10131736099999999 2024-09-16T03:37:21.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674651+0000", 2024-09-16T03:37:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073190500000000003 2024-09-16T03:37:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975976+0000", 2024-09-16T03:37:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.301325391 2024-09-16T03:37:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976014+0000", 2024-09-16T03:37:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8226000000000003e-05 2024-09-16T03:37:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976061+0000", 2024-09-16T03:37:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6071e-05 2024-09-16T03:37:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.510 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.115191484, 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60616296300000005, 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 70 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:37:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:37:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370052+0000", 2024-09-16T03:37:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8899999999999999e-07 2024-09-16T03:37:21.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370055+0000", 2024-09-16T03:37:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7709999999999998e-06 2024-09-16T03:37:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370060+0000", 2024-09-16T03:37:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6910000000000002e-06 2024-09-16T03:37:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370064+0000", 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.619e-06 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572988+0000", 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20292439300000001 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.573049+0000", 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0418999999999999e-05 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.573133+0000", 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4324000000000001e-05 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674789+0000", 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101655658 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.875723+0000", 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20093406699999999 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976140+0000", 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10041718199999999 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976172+0000", 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2169000000000003e-05 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976214+0000", 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2080999999999999e-05 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.516 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:37:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.115170273, 2024-09-16T03:37:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60623559900000001, 2024-09-16T03:37:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.518 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 71 2024-09-16T03:37:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:37:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:37:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:37:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:37:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370077+0000", 2024-09-16T03:37:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9940000000000002e-06 2024-09-16T03:37:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370081+0000", 2024-09-16T03:37:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5850000000000001e-06 2024-09-16T03:37:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370087+0000", 2024-09-16T03:37:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:37:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673522+0000", 2024-09-16T03:37:21.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.303434969 2024-09-16T03:37:21.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673567+0000", 2024-09-16T03:37:21.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5080000000000002e-05 2024-09-16T03:37:21.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673651+0000", 2024-09-16T03:37:21.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3652999999999993e-05 2024-09-16T03:37:21.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775115+0000", 2024-09-16T03:37:21.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101464178 2024-09-16T03:37:21.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975918+0000", 2024-09-16T03:37:21.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20080318699999999 2024-09-16T03:37:21.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976254+0000", 2024-09-16T03:37:21.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033619200000000002 2024-09-16T03:37:21.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976285+0000", 2024-09-16T03:37:21.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0679000000000002e-05 2024-09-16T03:37:21.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976308+0000", 2024-09-16T03:37:21.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3051e-05 2024-09-16T03:37:21.528 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.528 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.528 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.529 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.529 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:37:21.530 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.115146913, 2024-09-16T03:37:21.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60632925100000001, 2024-09-16T03:37:21.530 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.530 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.530 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.530 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.530 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.531 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.531 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 72 2024-09-16T03:37:21.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.531 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:37:21.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:37:21.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:37:21.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:37:21.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370150+0000", 2024-09-16T03:37:21.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3906999999999997e-05 2024-09-16T03:37:21.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370157+0000", 2024-09-16T03:37:21.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:37:21.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370162+0000", 2024-09-16T03:37:21.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7249999999999997e-06 2024-09-16T03:37:21.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673969+0000", 2024-09-16T03:37:21.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30380713199999998 2024-09-16T03:37:21.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674030+0000", 2024-09-16T03:37:21.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1141000000000003e-05 2024-09-16T03:37:21.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674124+0000", 2024-09-16T03:37:21.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3859999999999999e-05 2024-09-16T03:37:21.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775687+0000", 2024-09-16T03:37:21.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10156293299999999 2024-09-16T03:37:21.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976103+0000", 2024-09-16T03:37:21.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20041631400000001 2024-09-16T03:37:21.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976355+0000", 2024-09-16T03:37:21.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000251611 2024-09-16T03:37:21.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976385+0000", 2024-09-16T03:37:21.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0883999999999998e-05 2024-09-16T03:37:21.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976425+0000", 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9666000000000003e-05 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.115074429, 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60763126999999995, 2024-09-16T03:37:21.543 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.544 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.544 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.544 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.544 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.544 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.544 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 73 2024-09-16T03:37:21.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.544 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:37:21.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:37:21.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370169+0000", 2024-09-16T03:37:21.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6199999999999998e-07 2024-09-16T03:37:21.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370171+0000", 2024-09-16T03:37:21.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:37:21.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370175+0000", 2024-09-16T03:37:21.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.596e-06 2024-09-16T03:37:21.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470295+0000", 2024-09-16T03:37:21.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100119967 2024-09-16T03:37:21.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775148+0000", 2024-09-16T03:37:21.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30485311799999998 2024-09-16T03:37:21.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775203+0000", 2024-09-16T03:37:21.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4592999999999997e-05 2024-09-16T03:37:21.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775293+0000", 2024-09-16T03:37:21.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0859999999999994e-05 2024-09-16T03:37:21.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976223+0000", 2024-09-16T03:37:21.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20092996599999999 2024-09-16T03:37:21.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976463+0000", 2024-09-16T03:37:21.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000239695 2024-09-16T03:37:21.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977730+0000", 2024-09-16T03:37:21.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012667690000000001 2024-09-16T03:37:21.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977773+0000", 2024-09-16T03:37:21.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2604000000000001e-05 2024-09-16T03:37:21.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977800+0000", 2024-09-16T03:37:21.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7137000000000001e-05 2024-09-16T03:37:21.557 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.557 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.558 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.558 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.558 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:37:21.558 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.014935573999999, 2024-09-16T03:37:21.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60633810099999996, 2024-09-16T03:37:21.559 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.559 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.559 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.559 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.559 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.559 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.560 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 74 2024-09-16T03:37:21.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.560 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:37:21.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:37:21.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470308+0000", 2024-09-16T03:37:21.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0950000000000001e-06 2024-09-16T03:37:21.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470313+0000", 2024-09-16T03:37:21.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:37:21.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470320+0000", 2024-09-16T03:37:21.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5089999999999999e-06 2024-09-16T03:37:21.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470327+0000", 2024-09-16T03:37:21.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:37:21.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976483+0000", 2024-09-16T03:37:21.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50615579499999996 2024-09-16T03:37:21.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976533+0000", 2024-09-16T03:37:21.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0096000000000003e-05 2024-09-16T03:37:21.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976643+0000", 2024-09-16T03:37:21.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109481 2024-09-16T03:37:21.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977808+0000", 2024-09-16T03:37:21.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011649869999999999 2024-09-16T03:37:21.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.981699+0000", 2024-09-16T03:37:21.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003891579 2024-09-16T03:37:21.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076496+0000", 2024-09-16T03:37:21.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.094797333999999997 2024-09-16T03:37:21.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076583+0000", 2024-09-16T03:37:21.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6571999999999997e-05 2024-09-16T03:37:21.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076645+0000", 2024-09-16T03:37:21.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2331999999999999e-05 2024-09-16T03:37:21.571 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.571 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.571 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.572 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.572 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:37:21.572 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.014905439, 2024-09-16T03:37:21.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71251765099999997, 2024-09-16T03:37:21.572 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.573 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.573 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.573 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.573 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.573 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.573 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 75 2024-09-16T03:37:21.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.574 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:37:21.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:37:21.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470338+0000", 2024-09-16T03:37:21.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2500000000000005e-07 2024-09-16T03:37:21.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470389+0000", 2024-09-16T03:37:21.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.134e-05 2024-09-16T03:37:21.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470397+0000", 2024-09-16T03:37:21.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8140000000000001e-06 2024-09-16T03:37:21.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570537+0000", 2024-09-16T03:37:21.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100139425 2024-09-16T03:37:21.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977018+0000", 2024-09-16T03:37:21.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40648102899999999 2024-09-16T03:37:21.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977063+0000", 2024-09-16T03:37:21.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5247000000000002e-05 2024-09-16T03:37:21.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977143+0000", 2024-09-16T03:37:21.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0054000000000003e-05 2024-09-16T03:37:21.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.978132+0000", 2024-09-16T03:37:21.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00098896399999999995 2024-09-16T03:37:21.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182725+0000", 2024-09-16T03:37:21.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.204592626 2024-09-16T03:37:21.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182783+0000", 2024-09-16T03:37:21.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8285000000000002e-05 2024-09-16T03:37:21.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182825+0000", 2024-09-16T03:37:21.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2398999999999998e-05 2024-09-16T03:37:21.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182855+0000", 2024-09-16T03:37:21.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9842999999999999e-05 2024-09-16T03:37:21.585 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.585 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.585 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.585 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.586 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:37:21.586 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 24.914678872, 2024-09-16T03:37:21.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71263271800000005, 2024-09-16T03:37:21.586 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.586 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.586 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.586 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.586 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.586 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.587 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 76 2024-09-16T03:37:21.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.587 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:37:21.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:37:21.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570566+0000", 2024-09-16T03:37:21.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:37:21.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570577+0000", 2024-09-16T03:37:21.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0672999999999999e-05 2024-09-16T03:37:21.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570593+0000", 2024-09-16T03:37:21.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5979999999999999e-05 2024-09-16T03:37:21.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570604+0000", 2024-09-16T03:37:21.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.077e-05 2024-09-16T03:37:21.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979890+0000", 2024-09-16T03:37:21.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40928624200000002 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979945+0000", 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5651e-05 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980038+0000", 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2603999999999996e-05 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980669+0000", 2024-09-16T03:37:21.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063121900000000003 2024-09-16T03:37:21.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283050+0000", 2024-09-16T03:37:21.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30238120800000001 2024-09-16T03:37:21.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283132+0000", 2024-09-16T03:37:21.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1841999999999993e-05 2024-09-16T03:37:21.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283168+0000", 2024-09-16T03:37:21.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5438999999999997e-05 2024-09-16T03:37:21.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283197+0000", 2024-09-16T03:37:21.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8960000000000001e-05 2024-09-16T03:37:21.592 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.592 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:93 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385015+0000", 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.100228184999999, 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60601307199999999, 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.593 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 93 2024-09-16T03:37:21.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.594 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385015+0000", 2024-09-16T03:37:21.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385015+0000", 2024-09-16T03:37:21.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385015+0000", 2024-09-16T03:37:21.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6899999999999997e-07 2024-09-16T03:37:21.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385071+0000", 2024-09-16T03:37:21.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5674999999999998e-05 2024-09-16T03:37:21.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385081+0000", 2024-09-16T03:37:21.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6320000000000003e-06 2024-09-16T03:37:21.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385087+0000", 2024-09-16T03:37:21.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3080000000000004e-06 2024-09-16T03:37:21.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.686864+0000", 2024-09-16T03:37:21.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30177694399999999 2024-09-16T03:37:21.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.686911+0000", 2024-09-16T03:37:21.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7261e-05 2024-09-16T03:37:21.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.687014+0000", 2024-09-16T03:37:21.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103138 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.788048+0000", 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10103405999999999 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888243+0000", 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10019507 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990973+0000", 2024-09-16T03:37:21.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10273007100000001 2024-09-16T03:37:21.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991003+0000", 2024-09-16T03:37:21.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9734999999999999e-05 2024-09-16T03:37:21.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991028+0000", 2024-09-16T03:37:21.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4508999999999999e-05 2024-09-16T03:37:21.600 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.600 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.601 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.601 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.601 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:37:21.601 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.100143772999999, 2024-09-16T03:37:21.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70799990400000001, 2024-09-16T03:37:21.601 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.601 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.601 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.601 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.601 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.601 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 94 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385100+0000", 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9599999999999998e-07 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385104+0000", 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7579999999999998e-06 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385128+0000", 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4700999999999999e-05 2024-09-16T03:37:21.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385139+0000", 2024-09-16T03:37:21.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0587e-05 2024-09-16T03:37:21.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787499+0000", 2024-09-16T03:37:21.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40235976099999998 2024-09-16T03:37:21.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787545+0000", 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6418999999999997e-05 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787623+0000", 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7897000000000003e-05 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888605+0000", 2024-09-16T03:37:21.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100982321 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990721+0000", 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102115922 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093019+0000", 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10229788500000001 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093051+0000", 2024-09-16T03:37:21.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2255999999999997e-05 2024-09-16T03:37:21.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093099+0000", 2024-09-16T03:37:21.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7500999999999998e-05 2024-09-16T03:37:21.607 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.607 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.607 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.608 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.608 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:37:21.608 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.100092449999998, 2024-09-16T03:37:21.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80404594699999998, 2024-09-16T03:37:21.608 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.608 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.609 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.609 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.609 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.609 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.609 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 95 2024-09-16T03:37:21.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.609 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:37:21.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:37:21.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385151+0000", 2024-09-16T03:37:21.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:37:21.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385165+0000", 2024-09-16T03:37:21.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4175999999999999e-05 2024-09-16T03:37:21.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385175+0000", 2024-09-16T03:37:21.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:37:21.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385183+0000", 2024-09-16T03:37:21.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:37:21.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888643+0000", 2024-09-16T03:37:21.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50345988200000003 2024-09-16T03:37:21.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888695+0000", 2024-09-16T03:37:21.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2834999999999998e-05 2024-09-16T03:37:21.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888802+0000", 2024-09-16T03:37:21.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010697899999999999 2024-09-16T03:37:21.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990658+0000", 2024-09-16T03:37:21.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185585699999999 2024-09-16T03:37:21.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990917+0000", 2024-09-16T03:37:21.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025853600000000002 2024-09-16T03:37:21.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189054+0000", 2024-09-16T03:37:21.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198136812 2024-09-16T03:37:21.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189135+0000", 2024-09-16T03:37:21.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1482000000000004e-05 2024-09-16T03:37:21.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189196+0000", 2024-09-16T03:37:21.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1175999999999998e-05 2024-09-16T03:37:21.616 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.616 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.616 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.616 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.616 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:37:21.617 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.100051823000001, 2024-09-16T03:37:21.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80412488400000004, 2024-09-16T03:37:21.617 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.617 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.617 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.617 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.617 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.617 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.617 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 96 2024-09-16T03:37:21.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.618 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:37:21.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:37:21.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385192+0000", 2024-09-16T03:37:21.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:37:21.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385267+0000", 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5535000000000001e-05 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385277+0000", 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6239999999999997e-06 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385284+0000", 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4800000000000004e-06 2024-09-16T03:37:21.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989283+0000", 2024-09-16T03:37:21.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60399849500000002 2024-09-16T03:37:21.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989410+0000", 2024-09-16T03:37:21.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127298 2024-09-16T03:37:21.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989615+0000", 2024-09-16T03:37:21.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000205045 2024-09-16T03:37:21.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991036+0000", 2024-09-16T03:37:21.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001421042 2024-09-16T03:37:21.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189228+0000", 2024-09-16T03:37:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19819188500000001 2024-09-16T03:37:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189249+0000", 2024-09-16T03:37:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0996999999999999e-05 2024-09-16T03:37:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189293+0000", 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3912000000000002e-05 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189316+0000", 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2739e-05 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:37:21.624 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.099948092000002, 2024-09-16T03:37:21.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80642264100000005, 2024-09-16T03:37:21.625 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.625 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.625 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.625 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.625 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.625 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.625 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 97 2024-09-16T03:37:21.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.626 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.626 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.626 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:37:21.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.626 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.626 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:37:21.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385296+0000", 2024-09-16T03:37:21.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.006e-06 2024-09-16T03:37:21.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385301+0000", 2024-09-16T03:37:21.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2630000000000003e-06 2024-09-16T03:37:21.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385308+0000", 2024-09-16T03:37:21.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6030000000000001e-06 2024-09-16T03:37:21.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485459+0000", 2024-09-16T03:37:21.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150929 2024-09-16T03:37:21.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990238+0000", 2024-09-16T03:37:21.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50477992000000005 2024-09-16T03:37:21.630 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.630 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.630 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.630 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990287+0000", 2024-09-16T03:37:21.630 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8863999999999998e-05 2024-09-16T03:37:21.630 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.630 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.630 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990377+0000", 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9900000000000003e-05 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093109+0000", 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102732055 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189351+0000", 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096241766000000006 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.631 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.632 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.632 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191570+0000", 2024-09-16T03:37:21.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00221915 2024-09-16T03:37:21.632 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.632 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.632 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.632 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191653+0000", 2024-09-16T03:37:21.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3046000000000004e-05 2024-09-16T03:37:21.632 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.633 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.633 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191717+0000", 2024-09-16T03:37:21.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4139000000000004e-05 2024-09-16T03:37:21.633 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.633 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.633 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.633 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.633 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:37:21.633 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 22.999757635000002, 2024-09-16T03:37:21.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90688294999999997, 2024-09-16T03:37:21.634 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.634 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.634 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.634 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.634 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.634 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.634 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 98 2024-09-16T03:37:21.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.634 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:37:21.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:37:21.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485487+0000", 2024-09-16T03:37:21.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1799999999999999e-06 2024-09-16T03:37:21.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485498+0000", 2024-09-16T03:37:21.636 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0648999999999999e-05 2024-09-16T03:37:21.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485514+0000", 2024-09-16T03:37:21.637 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5943999999999998e-05 2024-09-16T03:37:21.637 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.637 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.637 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.637 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485525+0000", 2024-09-16T03:37:21.637 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.065e-05 2024-09-16T03:37:21.637 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.637 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.637 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.637 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991102+0000", 2024-09-16T03:37:21.637 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50557748400000002 2024-09-16T03:37:21.637 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.638 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.638 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.638 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991161+0000", 2024-09-16T03:37:21.638 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8801999999999997e-05 2024-09-16T03:37:21.638 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.638 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.638 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.638 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991255+0000", 2024-09-16T03:37:21.638 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3936000000000003e-05 2024-09-16T03:37:21.638 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093112+0000", 2024-09-16T03:37:21.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185709499999999 2024-09-16T03:37:21.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291911+0000", 2024-09-16T03:37:21.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19879897699999999 2024-09-16T03:37:21.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392167+0000", 2024-09-16T03:37:21.640 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10025634999999999 2024-09-16T03:37:21.640 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.640 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.640 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.640 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392249+0000", 2024-09-16T03:37:21.640 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2175000000000005e-05 2024-09-16T03:37:21.640 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.640 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.640 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.640 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392368+0000", 2024-09-16T03:37:21.640 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118708 2024-09-16T03:37:21.641 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.641 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.641 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.641 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.641 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.641 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.641 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:37:21.641 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 22.999704011999999, 2024-09-16T03:37:21.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90707203199999997, 2024-09-16T03:37:21.641 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.641 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.642 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.642 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.642 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.642 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.642 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 99 2024-09-16T03:37:21.642 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.643 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.643 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.643 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.643 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:37:21.643 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.643 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.644 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.644 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.644 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:37:21.644 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.644 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.644 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.644 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.644 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485540+0000", 2024-09-16T03:37:21.644 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-06 2024-09-16T03:37:21.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.645 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.645 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.645 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485661+0000", 2024-09-16T03:37:21.645 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120835 2024-09-16T03:37:21.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.645 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.645 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.645 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485680+0000", 2024-09-16T03:37:21.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8885999999999998e-05 2024-09-16T03:37:21.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.646 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.646 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485693+0000", 2024-09-16T03:37:21.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3297e-05 2024-09-16T03:37:21.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.646 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.646 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991546+0000", 2024-09-16T03:37:21.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.505853098 2024-09-16T03:37:21.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.647 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.647 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991598+0000", 2024-09-16T03:37:21.647 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1921e-05 2024-09-16T03:37:21.647 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.647 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.647 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991681+0000", 2024-09-16T03:37:21.647 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2502000000000004e-05 2024-09-16T03:37:21.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.648 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.648 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093115+0000", 2024-09-16T03:37:21.648 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10143424299999999 2024-09-16T03:37:21.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.648 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.648 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291977+0000", 2024-09-16T03:37:21.648 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198862172 2024-09-16T03:37:21.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.649 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.649 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392541+0000", 2024-09-16T03:37:21.649 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100563758 2024-09-16T03:37:21.649 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.649 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.649 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.649 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392579+0000", 2024-09-16T03:37:21.649 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7796999999999997e-05 2024-09-16T03:37:21.649 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.649 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.649 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392611+0000", 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2073999999999998e-05 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 22.999525119000001, 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0073149939999999, 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.650 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.651 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.651 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.651 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.651 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 100 2024-09-16T03:37:21.651 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.651 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.651 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:37:21.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:37:21.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:37:21.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7400000000000003e-07 2024-09-16T03:37:21.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485721+0000", 2024-09-16T03:37:21.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4530000000000001e-06 2024-09-16T03:37:21.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485726+0000", 2024-09-16T03:37:21.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0590000000000002e-06 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485729+0000", 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092342+0000", 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60661343700000003 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.654 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092469+0000", 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012658799999999999 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092671+0000", 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020206 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392452+0000", 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29978163699999999 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392486+0000", 2024-09-16T03:37:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3736999999999999e-05 2024-09-16T03:37:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492812+0000", 2024-09-16T03:37:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100325621 2024-09-16T03:37:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492901+0000", 2024-09-16T03:37:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8702000000000006e-05 2024-09-16T03:37:21.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.493033+0000", 2024-09-16T03:37:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000132185 2024-09-16T03:37:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:37:21.657 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 20.824374597999999, 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60874581500000002, 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 123 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:37:21.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.659 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.659 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:37:21.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.659 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.659 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660869+0000", 2024-09-16T03:37:21.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9400000000000005e-07 2024-09-16T03:37:21.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.660 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.660 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660921+0000", 2024-09-16T03:37:21.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2312000000000003e-05 2024-09-16T03:37:21.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.660 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.660 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660927+0000", 2024-09-16T03:37:21.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9329999999999998e-06 2024-09-16T03:37:21.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.660 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660931+0000", 2024-09-16T03:37:21.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:37:21.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.967991+0000", 2024-09-16T03:37:21.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30706032700000002 2024-09-16T03:37:21.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968039+0000", 2024-09-16T03:37:21.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7477e-05 2024-09-16T03:37:21.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968117+0000", 2024-09-16T03:37:21.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8589000000000003e-05 2024-09-16T03:37:21.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.663 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067678+0000", 2024-09-16T03:37:21.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099560406000000004 2024-09-16T03:37:21.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170175+0000", 2024-09-16T03:37:21.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102496927 2024-09-16T03:37:21.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.664 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269491+0000", 2024-09-16T03:37:21.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099316205000000005 2024-09-16T03:37:21.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.665 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269574+0000", 2024-09-16T03:37:21.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2903000000000002e-05 2024-09-16T03:37:21.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.666 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269614+0000", 2024-09-16T03:37:21.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0188000000000002e-05 2024-09-16T03:37:21.666 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.666 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.666 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.667 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.667 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.667 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:21.667 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:37:21.667 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 20.824305899999999, 2024-09-16T03:37:21.667 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60879245500000001, 2024-09-16T03:37:21.667 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.667 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.668 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:21.668 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.668 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:21.668 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:21.668 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 124 2024-09-16T03:37:21.668 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.668 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:37:21.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:37:21.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660938+0000", 2024-09-16T03:37:21.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:37:21.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.671 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660940+0000", 2024-09-16T03:37:21.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2529999999999999e-06 2024-09-16T03:37:21.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.672 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.672 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660943+0000", 2024-09-16T03:37:21.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.01e-06 2024-09-16T03:37:21.673 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.673 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.673 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.673 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660946+0000", 2024-09-16T03:37:21.673 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8159999999999998e-06 2024-09-16T03:37:21.673 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.674 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.674 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012417+0000", 2024-09-16T03:37:21.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.35147120599999998 2024-09-16T03:37:21.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012497+0000", 2024-09-16T03:37:21.675 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9963000000000007e-05 2024-09-16T03:37:21.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.676 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:21.676 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012585+0000", 2024-09-16T03:37:21.676 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:37:21.676 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.676 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.676 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.676 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067680+0000", 2024-09-16T03:37:21.676 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055095203000000002 2024-09-16T03:37:21.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.677 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.677 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.677 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170243+0000", 2024-09-16T03:37:21.677 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10256290799999999 2024-09-16T03:37:21.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.677 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.677 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:21.678 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269662+0000", 2024-09-16T03:37:21.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099418896000000007 2024-09-16T03:37:21.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.678 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:21.678 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269701+0000", 2024-09-16T03:37:21.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8829999999999999e-05 2024-09-16T03:37:21.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:21.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269729+0000", 2024-09-16T03:37:21.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8569e-05 2024-09-16T03:37:21.679 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.679 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.679 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.679 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.679 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.680 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:21.682 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:21.679+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:21.683 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 reset_purged_snaps_last 2024-09-16T03:37:21.684 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:37:21.690 INFO:tasks.rados.rados.0.smithi157.stdout:315: finishing set_chunk to oid 16 2024-09-16T03:37:21.690 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:37:21.691 INFO:tasks.rados.rados.0.smithi157.stdout:315: done (0 left) 2024-09-16T03:37:21.691 INFO:tasks.rados.rados.0.smithi157.stdout:319: set_chunk oid 20 target oid 20 low_tier 2024-09-16T03:37:21.691 INFO:tasks.rados.rados.0.smithi157.stdout:319: set_chunk oid 20 offset: 1017856 length: 861184 target oid smithi15736696-20 low_tier offset: 1017856 snap -1 2024-09-16T03:37:21.691 INFO:tasks.rados.rados.0.smithi157.stdout:320: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.691 INFO:tasks.rados.rados.0.smithi157.stdout:320: set_chunk oid 21 offset: 606720 length: 1516544 target oid smithi15736696-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 606720 snap -1 2024-09-16T03:37:21.691 INFO:tasks.rados.rados.0.smithi157.stdout:321: set_chunk oid 22 target oid 22 low_tier 2024-09-16T03:37:21.691 INFO:tasks.rados.rados.0.smithi157.stdout:321: set_chunk oid 22 offset: 183808 length: 1866240 target oid smithi15736696-22 low_tier offset: 183808 snap -1 2024-09-16T03:37:21.691 INFO:tasks.rados.rados.0.smithi157.stdout:322: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.691 INFO:tasks.rados.rados.0.smithi157.stdout:322: set_chunk oid 23 offset: 151040 length: 759808 target oid smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 151040 snap -1 2024-09-16T03:37:21.691 INFO:tasks.rados.rados.0.smithi157.stdout:323: set_chunk oid 24 target oid 24 low_tier 2024-09-16T03:37:21.691 INFO:tasks.rados.rados.0.smithi157.stdout:323: set_chunk oid 24 offset: 1321472 length: 261632 target oid smithi15736696-24 low_tier offset: 1321472 snap -1 2024-09-16T03:37:21.692 INFO:tasks.rados.rados.0.smithi157.stdout:324: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.692 INFO:tasks.rados.rados.0.smithi157.stdout:324: set_chunk oid 25 offset: 1203712 length: 755712 target oid smithi15736696-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1203712 snap -1 2024-09-16T03:37:21.692 INFO:tasks.rados.rados.0.smithi157.stdout:325: set_chunk oid 26 target oid 26 low_tier 2024-09-16T03:37:21.692 INFO:tasks.rados.rados.0.smithi157.stdout:325: set_chunk oid 26 offset: 1318912 length: 770560 target oid smithi15736696-26 low_tier offset: 1318912 snap -1 2024-09-16T03:37:21.692 INFO:tasks.rados.rados.0.smithi157.stdout:326: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.692 INFO:tasks.rados.rados.0.smithi157.stdout:326: set_chunk oid 27 offset: 1008640 length: 710656 target oid smithi15736696-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1008640 snap -1 2024-09-16T03:37:21.692 INFO:tasks.rados.rados.0.smithi157.stdout:327: set_chunk oid 28 target oid 28 low_tier 2024-09-16T03:37:21.692 INFO:tasks.rados.rados.0.smithi157.stdout:327: set_chunk oid 28 offset: 1260544 length: 469504 target oid smithi15736696-28 low_tier offset: 1260544 snap -1 2024-09-16T03:37:21.692 INFO:tasks.rados.rados.0.smithi157.stdout:328: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.692 INFO:tasks.rados.rados.0.smithi157.stdout:328: set_chunk oid 29 offset: 1590272 length: 118272 target oid smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1590272 snap -1 2024-09-16T03:37:21.692 INFO:tasks.rados.rados.0.smithi157.stdout:329: set_chunk oid 30 target oid 30 low_tier 2024-09-16T03:37:21.692 INFO:tasks.rados.rados.0.smithi157.stdout:329: set_chunk oid 30 offset: 2225152 length: 101376 target oid smithi15736696-30 low_tier offset: 2225152 snap -1 2024-09-16T03:37:21.692 INFO:tasks.rados.rados.0.smithi157.stdout:330: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.693 INFO:tasks.rados.rados.0.smithi157.stdout:330: set_chunk oid 31 offset: 62976 length: 301056 target oid smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 62976 snap -1 2024-09-16T03:37:21.693 INFO:tasks.rados.rados.0.smithi157.stdout:331: set_chunk oid 32 target oid 32 low_tier 2024-09-16T03:37:21.693 INFO:tasks.rados.rados.0.smithi157.stdout:331: set_chunk oid 32 offset: 1324544 length: 980480 target oid smithi15736696-32 low_tier offset: 1324544 snap -1 2024-09-16T03:37:21.693 INFO:tasks.rados.rados.0.smithi157.stdout:332: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.693 INFO:tasks.rados.rados.0.smithi157.stdout:332: set_chunk oid 33 offset: 179712 length: 2018816 target oid smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 179712 snap -1 2024-09-16T03:37:21.693 INFO:tasks.rados.rados.0.smithi157.stdout:333: set_chunk oid 34 target oid 34 low_tier 2024-09-16T03:37:21.693 INFO:tasks.rados.rados.0.smithi157.stdout:333: set_chunk oid 34 offset: 249856 length: 1118720 target oid smithi15736696-34 low_tier offset: 249856 snap -1 2024-09-16T03:37:21.693 INFO:tasks.rados.rados.0.smithi157.stdout:334: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.693 INFO:tasks.rados.rados.0.smithi157.stdout:334: set_chunk oid 35 offset: 1508864 length: 734720 target oid smithi15736696-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1508864 snap -1 2024-09-16T03:37:21.693 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.693 INFO:tasks.rados.rados.0.smithi157.stdout:327: finishing set_chunk to oid 28 2024-09-16T03:37:21.693 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 1260544 length: 469504 2024-09-16T03:37:21.694 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T03:37:21.694 INFO:tasks.rados.rados.0.smithi157.stdout:327: done (15 left) 2024-09-16T03:37:21.694 INFO:tasks.rados.rados.0.smithi157.stdout:335: set_chunk oid 36 target oid 36 low_tier 2024-09-16T03:37:21.694 INFO:tasks.rados.rados.0.smithi157.stdout:335: set_chunk oid 36 offset: 679936 length: 671232 target oid smithi15736696-36 low_tier offset: 679936 snap -1 2024-09-16T03:37:21.694 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.694 INFO:tasks.rados.rados.0.smithi157.stdout:331: finishing set_chunk to oid 32 2024-09-16T03:37:21.694 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1324544 length: 980480 2024-09-16T03:37:21.694 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:37:21.694 INFO:tasks.rados.rados.0.smithi157.stdout:331: done (15 left) 2024-09-16T03:37:21.694 INFO:tasks.rados.rados.0.smithi157.stdout:336: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.694 INFO:tasks.rados.rados.0.smithi157.stdout:336: set_chunk oid 37 offset: 482816 length: 1530880 target oid smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 482816 snap -1 2024-09-16T03:37:21.694 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.694 INFO:tasks.rados.rados.0.smithi157.stdout:321: finishing set_chunk to oid 22 2024-09-16T03:37:21.695 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 183808 length: 1866240 2024-09-16T03:37:21.695 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T03:37:21.695 INFO:tasks.rados.rados.0.smithi157.stdout:321: done (15 left) 2024-09-16T03:37:21.695 INFO:tasks.rados.rados.0.smithi157.stdout:337: set_chunk oid 38 target oid 38 low_tier 2024-09-16T03:37:21.695 INFO:tasks.rados.rados.0.smithi157.stdout:337: set_chunk oid 38 offset: 982016 length: 746496 target oid smithi15736696-38 low_tier offset: 982016 snap -1 2024-09-16T03:37:21.695 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.695 INFO:tasks.rados.rados.0.smithi157.stdout:324: finishing set_chunk to oid 25 2024-09-16T03:37:21.695 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1203712 length: 755712 2024-09-16T03:37:21.695 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:21.695 INFO:tasks.rados.rados.0.smithi157.stdout:334: finishing set_chunk to oid 35 2024-09-16T03:37:21.696 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1508864 length: 734720 2024-09-16T03:37:21.696 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:37:21.696 INFO:tasks.rados.rados.0.smithi157.stdout:324: done (15 left) 2024-09-16T03:37:21.696 INFO:tasks.rados.rados.0.smithi157.stdout:334: done (14 left) 2024-09-16T03:37:21.696 INFO:tasks.rados.rados.0.smithi157.stdout:338: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.696 INFO:tasks.rados.rados.0.smithi157.stdout:338: set_chunk oid 39 offset: 673792 length: 933888 target oid smithi15736696-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 673792 snap -1 2024-09-16T03:37:21.696 INFO:tasks.rados.rados.0.smithi157.stdout:339: set_chunk oid 40 target oid 40 low_tier 2024-09-16T03:37:21.696 INFO:tasks.rados.rados.0.smithi157.stdout:339: set_chunk oid 40 offset: 1848320 length: 577536 target oid smithi15736696-40 low_tier offset: 1848320 snap -1 2024-09-16T03:37:21.696 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.696 INFO:tasks.rados.rados.0.smithi157.stdout:325: finishing set_chunk to oid 26 2024-09-16T03:37:21.696 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1318912 length: 770560 2024-09-16T03:37:21.696 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:37:21.697 INFO:tasks.rados.rados.0.smithi157.stdout:325: done (15 left) 2024-09-16T03:37:21.697 INFO:tasks.rados.rados.0.smithi157.stdout:340: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.697 INFO:tasks.rados.rados.0.smithi157.stdout:340: set_chunk oid 41 offset: 2275840 length: 378880 target oid smithi15736696-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2275840 snap -1 2024-09-16T03:37:21.697 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.697 INFO:tasks.rados.rados.0.smithi157.stdout:337: finishing set_chunk to oid 38 2024-09-16T03:37:21.697 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 982016 length: 746496 2024-09-16T03:37:21.697 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:37:21.697 INFO:tasks.rados.rados.0.smithi157.stdout:337: done (15 left) 2024-09-16T03:37:21.697 INFO:tasks.rados.rados.0.smithi157.stdout:341: set_chunk oid 42 target oid 42 low_tier 2024-09-16T03:37:21.697 INFO:tasks.rados.rados.0.smithi157.stdout:341: set_chunk oid 42 offset: 1362944 length: 1152512 target oid smithi15736696-42 low_tier offset: 1362944 snap -1 2024-09-16T03:37:21.697 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.697 INFO:tasks.rados.rados.0.smithi157.stdout:341: finishing set_chunk to oid 42 2024-09-16T03:37:21.697 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1362944 length: 1152512 2024-09-16T03:37:21.698 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T03:37:21.698 INFO:tasks.rados.rados.0.smithi157.stdout:341: done (15 left) 2024-09-16T03:37:21.698 INFO:tasks.rados.rados.0.smithi157.stdout:342: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.698 INFO:tasks.rados.rados.0.smithi157.stdout:342: set_chunk oid 43 offset: 1920000 length: 186368 target oid smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1920000 snap -1 2024-09-16T03:37:21.698 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.727 INFO:tasks.rados.rados.0.smithi157.stdout:320: finishing set_chunk to oid 21 2024-09-16T03:37:21.727 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 606720 length: 1516544 2024-09-16T03:37:21.727 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T03:37:21.727 INFO:tasks.rados.rados.0.smithi157.stdout:319: finishing set_chunk to oid 20 2024-09-16T03:37:21.727 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1017856 length: 861184 2024-09-16T03:37:21.727 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T03:37:21.727 INFO:tasks.rados.rados.0.smithi157.stdout:322: finishing set_chunk to oid 23 2024-09-16T03:37:21.727 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 151040 length: 759808 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout:319: done (15 left) 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout:320: done (14 left) 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout:322: done (13 left) 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout:343: set_chunk oid 44 target oid 44 low_tier 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout:343: set_chunk oid 44 offset: 59392 length: 1303040 target oid smithi15736696-44 low_tier offset: 59392 snap -1 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout:344: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout:344: set_chunk oid 45 offset: 201728 length: 484864 target oid smithi15736696-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 201728 snap -1 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout:345: set_chunk oid 46 target oid 46 low_tier 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout:345: set_chunk oid 46 offset: 1533440 length: 731136 target oid smithi15736696-46 low_tier offset: 1533440 snap -1 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout:332: finishing set_chunk to oid 33 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 179712 length: 2018816 2024-09-16T03:37:21.728 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:37:21.729 INFO:tasks.rados.rados.0.smithi157.stdout:332: done (15 left) 2024-09-16T03:37:21.729 INFO:tasks.rados.rados.0.smithi157.stdout:346: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.729 INFO:tasks.rados.rados.0.smithi157.stdout:346: set_chunk oid 47 offset: 545792 length: 1414656 target oid smithi15736696-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 545792 snap -1 2024-09-16T03:37:21.729 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.729 INFO:tasks.rados.rados.0.smithi157.stdout:338: finishing set_chunk to oid 39 2024-09-16T03:37:21.730 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 673792 length: 933888 2024-09-16T03:37:21.730 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T03:37:21.730 INFO:tasks.rados.rados.0.smithi157.stdout:338: done (15 left) 2024-09-16T03:37:21.730 INFO:tasks.rados.rados.0.smithi157.stdout:347: set_chunk oid 48 target oid 48 low_tier 2024-09-16T03:37:21.730 INFO:tasks.rados.rados.0.smithi157.stdout:347: set_chunk oid 48 offset: 196608 length: 48128 target oid smithi15736696-48 low_tier offset: 196608 snap -1 2024-09-16T03:37:21.730 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.730 INFO:tasks.rados.rados.0.smithi157.stdout:345: finishing set_chunk to oid 46 2024-09-16T03:37:21.730 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1533440 length: 731136 2024-09-16T03:37:21.730 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:37:21.730 INFO:tasks.rados.rados.0.smithi157.stdout:345: done (15 left) 2024-09-16T03:37:21.730 INFO:tasks.rados.rados.0.smithi157.stdout:348: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.731 INFO:tasks.rados.rados.0.smithi157.stdout:348: set_chunk oid 49 offset: 176640 length: 344576 target oid smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 176640 snap -1 2024-09-16T03:37:21.731 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.731 INFO:tasks.rados.rados.0.smithi157.stdout:336: finishing set_chunk to oid 37 2024-09-16T03:37:21.731 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 482816 length: 1530880 2024-09-16T03:37:21.731 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T03:37:21.731 INFO:tasks.rados.rados.0.smithi157.stdout:328: finishing set_chunk to oid 29 2024-09-16T03:37:21.731 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1590272 length: 118272 2024-09-16T03:37:21.731 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:37:21.731 INFO:tasks.rados.rados.0.smithi157.stdout:346: finishing set_chunk to oid 47 2024-09-16T03:37:21.731 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 545792 length: 1414656 2024-09-16T03:37:21.731 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T03:37:21.731 INFO:tasks.rados.rados.0.smithi157.stdout:328: done (15 left) 2024-09-16T03:37:21.731 INFO:tasks.rados.rados.0.smithi157.stdout:336: done (14 left) 2024-09-16T03:37:21.732 INFO:tasks.rados.rados.0.smithi157.stdout:346: done (13 left) 2024-09-16T03:37:21.732 INFO:tasks.rados.rados.0.smithi157.stdout:349: set_chunk oid 50 target oid 50 low_tier 2024-09-16T03:37:21.732 INFO:tasks.rados.rados.0.smithi157.stdout:349: set_chunk oid 50 offset: 150016 length: 661504 target oid smithi15736696-50 low_tier offset: 150016 snap -1 2024-09-16T03:37:21.732 INFO:tasks.rados.rados.0.smithi157.stdout:350: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.732 INFO:tasks.rados.rados.0.smithi157.stdout:350: set_chunk oid 1 offset: 643072 length: 257536 target oid smithi15736696-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 643072 snap -1 2024-09-16T03:37:21.732 INFO:tasks.rados.rados.0.smithi157.stdout:351: set_chunk oid 2 target oid 2 low_tier 2024-09-16T03:37:21.732 INFO:tasks.rados.rados.0.smithi157.stdout:351: set_chunk oid 2 offset: 661504 length: 1045504 target oid smithi15736696-2 low_tier offset: 661504 snap -1 2024-09-16T03:37:21.732 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.732 INFO:tasks.rados.rados.0.smithi157.stdout:348: finishing set_chunk to oid 49 2024-09-16T03:37:21.732 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 176640 length: 344576 2024-09-16T03:37:21.732 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T03:37:21.732 INFO:tasks.rados.rados.0.smithi157.stdout:348: done (15 left) 2024-09-16T03:37:21.732 INFO:tasks.rados.rados.0.smithi157.stdout:352: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout:352: set_chunk oid 3 offset: 1009152 length: 596992 target oid smithi15736696-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1009152 snap -1 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout:350: finishing set_chunk to oid 1 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 643072 length: 257536 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout:350: done (15 left) 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout:353: set_chunk oid 4 target oid 4 low_tier 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout:353: set_chunk oid 4 offset: 1240576 length: 100352 target oid smithi15736696-4 low_tier offset: 1240576 snap -1 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout:351: finishing set_chunk to oid 2 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 661504 length: 1045504 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout:351: done (15 left) 2024-09-16T03:37:21.733 INFO:tasks.rados.rados.0.smithi157.stdout:354: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.734 INFO:tasks.rados.rados.0.smithi157.stdout:354: set_chunk oid 5 offset: 934400 length: 825344 target oid smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 934400 snap -1 2024-09-16T03:37:21.734 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.734 INFO:tasks.rados.rados.0.smithi157.stdout:354: finishing set_chunk to oid 5 2024-09-16T03:37:21.734 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 934400 length: 825344 2024-09-16T03:37:21.734 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T03:37:21.734 INFO:tasks.rados.rados.0.smithi157.stdout:354: done (15 left) 2024-09-16T03:37:21.734 INFO:tasks.rados.rados.0.smithi157.stdout:355: set_chunk oid 6 target oid 6 low_tier 2024-09-16T03:37:21.734 INFO:tasks.rados.rados.0.smithi157.stdout:355: set_chunk oid 6 offset: 154624 length: 557056 target oid smithi15736696-6 low_tier offset: 154624 snap -1 2024-09-16T03:37:21.734 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.782 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:21.780+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:21.826 INFO:tasks.rados.rados.0.smithi157.stdout:323: finishing set_chunk to oid 24 2024-09-16T03:37:21.826 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 1321472 length: 261632 2024-09-16T03:37:21.826 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T03:37:21.826 INFO:tasks.rados.rados.0.smithi157.stdout:326: finishing set_chunk to oid 27 2024-09-16T03:37:21.826 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1008640 length: 710656 2024-09-16T03:37:21.826 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:21.826 INFO:tasks.rados.rados.0.smithi157.stdout:333: finishing set_chunk to oid 34 2024-09-16T03:37:21.826 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 249856 length: 1118720 2024-09-16T03:37:21.826 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T03:37:21.827 INFO:tasks.rados.rados.0.smithi157.stdout:323: done (15 left) 2024-09-16T03:37:21.827 INFO:tasks.rados.rados.0.smithi157.stdout:326: done (14 left) 2024-09-16T03:37:21.827 INFO:tasks.rados.rados.0.smithi157.stdout:333: done (13 left) 2024-09-16T03:37:21.827 INFO:tasks.rados.rados.0.smithi157.stdout:356: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.827 INFO:tasks.rados.rados.0.smithi157.stdout:356: set_chunk oid 7 offset: 32256 length: 1777152 target oid smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 32256 snap -1 2024-09-16T03:37:21.827 INFO:tasks.rados.rados.0.smithi157.stdout:357: set_chunk oid 8 target oid 8 low_tier 2024-09-16T03:37:21.827 INFO:tasks.rados.rados.0.smithi157.stdout:357: set_chunk oid 8 offset: 122368 length: 512000 target oid smithi15736696-8 low_tier offset: 122368 snap -1 2024-09-16T03:37:21.827 INFO:tasks.rados.rados.0.smithi157.stdout:358: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.827 INFO:tasks.rados.rados.0.smithi157.stdout:358: set_chunk oid 9 offset: 172544 length: 336384 target oid smithi15736696-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 172544 snap -1 2024-09-16T03:37:21.827 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.827 INFO:tasks.rados.rados.0.smithi157.stdout:349: finishing set_chunk to oid 50 2024-09-16T03:37:21.827 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 150016 length: 661504 2024-09-16T03:37:21.828 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:37:21.828 INFO:tasks.rados.rados.0.smithi157.stdout:349: done (15 left) 2024-09-16T03:37:21.828 INFO:tasks.rados.rados.0.smithi157.stdout:359: set_chunk oid 10 target oid 10 low_tier 2024-09-16T03:37:21.828 INFO:tasks.rados.rados.0.smithi157.stdout:359: set_chunk oid 10 offset: 1726976 length: 81408 target oid smithi15736696-10 low_tier offset: 1726976 snap -1 2024-09-16T03:37:21.828 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.828 INFO:tasks.rados.rados.0.smithi157.stdout:357: finishing set_chunk to oid 8 2024-09-16T03:37:21.828 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 122368 length: 512000 2024-09-16T03:37:21.828 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:37:21.828 INFO:tasks.rados.rados.0.smithi157.stdout:357: done (15 left) 2024-09-16T03:37:21.828 INFO:tasks.rados.rados.0.smithi157.stdout:360: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.828 INFO:tasks.rados.rados.0.smithi157.stdout:360: set_chunk oid 11 offset: 208384 length: 1937920 target oid smithi15736696-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 208384 snap -1 2024-09-16T03:37:21.828 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.828 INFO:tasks.rados.rados.0.smithi157.stdout:347: finishing set_chunk to oid 48 2024-09-16T03:37:21.829 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 196608 length: 48128 2024-09-16T03:37:21.829 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:37:21.829 INFO:tasks.rados.rados.0.smithi157.stdout:347: done (15 left) 2024-09-16T03:37:21.829 INFO:tasks.rados.rados.0.smithi157.stdout:361: set_chunk oid 12 target oid 12 low_tier 2024-09-16T03:37:21.829 INFO:tasks.rados.rados.0.smithi157.stdout:361: set_chunk oid 12 offset: 1108992 length: 74240 target oid smithi15736696-12 low_tier offset: 1108992 snap -1 2024-09-16T03:37:21.829 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.829 INFO:tasks.rados.rados.0.smithi157.stdout:356: finishing set_chunk to oid 7 2024-09-16T03:37:21.829 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 32256 length: 1777152 2024-09-16T03:37:21.829 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T03:37:21.829 INFO:tasks.rados.rados.0.smithi157.stdout:356: done (15 left) 2024-09-16T03:37:21.829 INFO:tasks.rados.rados.0.smithi157.stdout:362: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.829 INFO:tasks.rados.rados.0.smithi157.stdout:362: set_chunk oid 13 offset: 1516544 length: 529408 target oid smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1516544 snap -1 2024-09-16T03:37:21.829 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout:360: finishing set_chunk to oid 11 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 208384 length: 1937920 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout:360: done (15 left) 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout:363: set_chunk oid 14 target oid 14 low_tier 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout:363: set_chunk oid 14 offset: 708608 length: 716800 target oid smithi15736696-14 low_tier offset: 708608 snap -1 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout:363: finishing set_chunk to oid 14 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 708608 length: 716800 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout:363: done (15 left) 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout:364: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout:364: set_chunk oid 15 offset: 216576 length: 1318400 target oid smithi15736696-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 216576 snap -1 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.830 INFO:tasks.rados.rados.0.smithi157.stdout:364: finishing set_chunk to oid 15 2024-09-16T03:37:21.831 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 216576 length: 1318400 2024-09-16T03:37:21.831 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:21.831 INFO:tasks.rados.rados.0.smithi157.stdout:364: done (15 left) 2024-09-16T03:37:21.831 INFO:tasks.rados.rados.0.smithi157.stdout:365: set_chunk oid 16 target oid 16 low_tier 2024-09-16T03:37:21.831 INFO:tasks.rados.rados.0.smithi157.stdout:365: set_chunk oid 16 offset: 2099712 length: 118784 target oid smithi15736696-16 low_tier offset: 2099712 snap -1 2024-09-16T03:37:21.831 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.831 INFO:tasks.rados.rados.0.smithi157.stdout:365: finishing set_chunk to oid 16 2024-09-16T03:37:21.831 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 2099712 length: 118784 2024-09-16T03:37:21.831 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:37:21.831 INFO:tasks.rados.rados.0.smithi157.stdout:365: done (15 left) 2024-09-16T03:37:21.831 INFO:tasks.rados.rados.0.smithi157.stdout:366: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.831 INFO:tasks.rados.rados.0.smithi157.stdout:366: set_chunk oid 17 offset: 185344 length: 1230336 target oid smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 185344 snap -1 2024-09-16T03:37:21.831 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.832 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:21.832 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:21.832 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop_reply(client.4628.0:492 4.5 e45/32)", 2024-09-16T03:37:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.824159+0000", 2024-09-16T03:37:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0015869040000000001, 2024-09-16T03:37:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016075149999999999, 2024-09-16T03:37:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "no flag points reached", 2024-09-16T03:37:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.824159+0000", 2024-09-16T03:37:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.824159+0000", 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.824160+0000", 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0100000000000001e-06 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.824164+0000", 2024-09-16T03:37:21.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.281e-06 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.824172+0000", 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2060000000000007e-06 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:492 4.5 4:a11e1d7c:::smithi15736696-43:head [assert-version v12,set-chunk in=389b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.722004+0000", 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.103742048, 2024-09-16T03:37:21.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103790413, 2024-09-16T03:37:21.836 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.836 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.836 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:37:21.836 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:21.836 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:21.836 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:21.836 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 492 2024-09-16T03:37:21.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.836 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.722004+0000", 2024-09-16T03:37:21.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.722004+0000", 2024-09-16T03:37:21.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.722005+0000", 2024-09-16T03:37:21.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:37:21.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.722012+0000", 2024-09-16T03:37:21.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7170000000000002e-06 2024-09-16T03:37:21.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.722022+0000", 2024-09-16T03:37:21.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6479999999999998e-06 2024-09-16T03:37:21.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.722028+0000", 2024-09-16T03:37:21.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4980000000000003e-06 2024-09-16T03:37:21.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:21.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.722191+0000", 2024-09-16T03:37:21.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016286299999999999 2024-09-16T03:37:21.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.722265+0000", 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4275000000000003e-05 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823285+0000", 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101019187 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823920+0000", 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063574599999999997 2024-09-16T03:37:21.840 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.5.21:14 3.3 3.62c96cdf (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.823507+0000", 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.0022391059999999998, 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023254830000000002, 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.841 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:21.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823507+0000", 2024-09-16T03:37:21.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:21.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823507+0000", 2024-09-16T03:37:21.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:21.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:21.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823509+0000", 2024-09-16T03:37:21.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.362e-06 2024-09-16T03:37:21.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823514+0000", 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6919999999999999e-06 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823525+0000", 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.03e-05 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823532+0000", 2024-09-16T03:37:21.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4959999999999999e-06 2024-09-16T03:37:21.844 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.844 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:21.844 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.844 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:21.844 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-16T03:37:21.844 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 3 2024-09-16T03:37:21.844 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:21.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 reset_purged_snaps_last 2024-09-16T03:37:21.846 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:37:21.886 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:21.884+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:21.928 INFO:tasks.rados.rados.0.smithi157.stdout:335: finishing set_chunk to oid 36 2024-09-16T03:37:21.928 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 679936 length: 671232 2024-09-16T03:37:21.928 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:37:21.928 INFO:tasks.rados.rados.0.smithi157.stdout:335: done (15 left) 2024-09-16T03:37:21.928 INFO:tasks.rados.rados.0.smithi157.stdout:367: set_chunk oid 18 target oid 18 low_tier 2024-09-16T03:37:21.929 INFO:tasks.rados.rados.0.smithi157.stdout:367: set_chunk oid 18 offset: 1350656 length: 475648 target oid smithi15736696-18 low_tier offset: 1350656 snap -1 2024-09-16T03:37:21.929 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.929 INFO:tasks.rados.rados.0.smithi157.stdout:352: finishing set_chunk to oid 3 2024-09-16T03:37:21.929 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1009152 length: 596992 2024-09-16T03:37:21.929 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T03:37:21.929 INFO:tasks.rados.rados.0.smithi157.stdout:352: done (15 left) 2024-09-16T03:37:21.929 INFO:tasks.rados.rados.0.smithi157.stdout:368: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.929 INFO:tasks.rados.rados.0.smithi157.stdout:368: set_chunk oid 19 offset: 655872 length: 897536 target oid smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 655872 snap -1 2024-09-16T03:37:21.929 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.929 INFO:tasks.rados.rados.0.smithi157.stdout:343: finishing set_chunk to oid 44 2024-09-16T03:37:21.929 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 59392 length: 1303040 2024-09-16T03:37:21.929 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T03:37:21.929 INFO:tasks.rados.rados.0.smithi157.stdout:355: finishing set_chunk to oid 6 2024-09-16T03:37:21.930 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 154624 length: 557056 2024-09-16T03:37:21.930 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:37:21.930 INFO:tasks.rados.rados.0.smithi157.stdout:343: done (15 left) 2024-09-16T03:37:21.930 INFO:tasks.rados.rados.0.smithi157.stdout:355: done (14 left) 2024-09-16T03:37:21.930 INFO:tasks.rados.rados.0.smithi157.stdout:369: set_chunk oid 20 target oid 20 low_tier 2024-09-16T03:37:21.930 INFO:tasks.rados.rados.0.smithi157.stdout:369: set_chunk oid 20 offset: 105472 length: 574976 target oid smithi15736696-20 low_tier offset: 105472 snap -1 2024-09-16T03:37:21.930 INFO:tasks.rados.rados.0.smithi157.stdout:370: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.930 INFO:tasks.rados.rados.0.smithi157.stdout:370: set_chunk oid 21 offset: 823296 length: 632320 target oid smithi15736696-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 823296 snap -1 2024-09-16T03:37:21.930 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.930 INFO:tasks.rados.rados.0.smithi157.stdout:361: finishing set_chunk to oid 12 2024-09-16T03:37:21.930 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1108992 length: 74240 2024-09-16T03:37:21.930 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:37:21.930 INFO:tasks.rados.rados.0.smithi157.stdout:361: done (15 left) 2024-09-16T03:37:21.931 INFO:tasks.rados.rados.0.smithi157.stdout:371: set_chunk oid 22 target oid 22 low_tier 2024-09-16T03:37:21.931 INFO:tasks.rados.rados.0.smithi157.stdout:371: set_chunk oid 22 offset: 593408 length: 801792 target oid smithi15736696-22 low_tier offset: 593408 snap -1 2024-09-16T03:37:21.931 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.931 INFO:tasks.rados.rados.0.smithi157.stdout:366: finishing set_chunk to oid 17 2024-09-16T03:37:21.931 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 185344 length: 1230336 2024-09-16T03:37:21.931 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T03:37:21.931 INFO:tasks.rados.rados.0.smithi157.stdout:366: done (15 left) 2024-09-16T03:37:21.931 INFO:tasks.rados.rados.0.smithi157.stdout:372: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.931 INFO:tasks.rados.rados.0.smithi157.stdout:372: set_chunk oid 23 offset: 979456 length: 562688 target oid smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 979456 snap -1 2024-09-16T03:37:21.931 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.931 INFO:tasks.rados.rados.0.smithi157.stdout:369: finishing set_chunk to oid 20 2024-09-16T03:37:21.931 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 105472 length: 574976 2024-09-16T03:37:21.931 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T03:37:21.932 INFO:tasks.rados.rados.0.smithi157.stdout:369: done (15 left) 2024-09-16T03:37:21.932 INFO:tasks.rados.rados.0.smithi157.stdout:373: set_chunk oid 24 target oid 24 low_tier 2024-09-16T03:37:21.932 INFO:tasks.rados.rados.0.smithi157.stdout:373: set_chunk oid 24 offset: 2076160 length: 134656 target oid smithi15736696-24 low_tier offset: 2076160 snap -1 2024-09-16T03:37:21.932 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.932 INFO:tasks.rados.rados.0.smithi157.stdout:368: finishing set_chunk to oid 19 2024-09-16T03:37:21.932 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 655872 length: 897536 2024-09-16T03:37:21.932 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T03:37:21.932 INFO:tasks.rados.rados.0.smithi157.stdout:368: done (15 left) 2024-09-16T03:37:21.932 INFO:tasks.rados.rados.0.smithi157.stdout:374: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout:374: set_chunk oid 25 offset: 1509376 length: 324096 target oid smithi15736696-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1509376 snap -1 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout:371: finishing set_chunk to oid 22 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 593408 length: 801792 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout:371: done (15 left) 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout:375: set_chunk oid 26 target oid 26 low_tier 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout:375: set_chunk oid 26 offset: 500736 length: 906752 target oid smithi15736696-26 low_tier offset: 500736 snap -1 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout:370: finishing set_chunk to oid 21 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 823296 length: 632320 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout:370: done (15 left) 2024-09-16T03:37:21.933 INFO:tasks.rados.rados.0.smithi157.stdout:376: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.934 INFO:tasks.rados.rados.0.smithi157.stdout:376: set_chunk oid 27 offset: 635904 length: 17920 target oid smithi15736696-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 635904 snap -1 2024-09-16T03:37:21.934 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.934 INFO:tasks.rados.rados.0.smithi157.stdout:372: finishing set_chunk to oid 23 2024-09-16T03:37:21.934 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 979456 length: 562688 2024-09-16T03:37:21.934 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T03:37:21.934 INFO:tasks.rados.rados.0.smithi157.stdout:372: done (15 left) 2024-09-16T03:37:21.934 INFO:tasks.rados.rados.0.smithi157.stdout:377: set_chunk oid 28 target oid 28 low_tier 2024-09-16T03:37:21.934 INFO:tasks.rados.rados.0.smithi157.stdout:377: set_chunk oid 28 offset: 3072 length: 135680 target oid smithi15736696-28 low_tier offset: 3072 snap -1 2024-09-16T03:37:21.934 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.934 INFO:tasks.rados.rados.0.smithi157.stdout:373: finishing set_chunk to oid 24 2024-09-16T03:37:21.934 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 2076160 length: 134656 2024-09-16T03:37:21.934 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T03:37:21.934 INFO:tasks.rados.rados.0.smithi157.stdout:373: done (15 left) 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout:378: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout:378: set_chunk oid 29 offset: 326656 length: 1812480 target oid smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 326656 snap -1 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout:374: finishing set_chunk to oid 25 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1509376 length: 324096 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout:374: done (15 left) 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout:378 retry set_chunk ! 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 15 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout:378: finishing set_chunk to oid 29 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 326656 length: 1812480 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout:378: done (14 left) 2024-09-16T03:37:21.935 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 14 2024-09-16T03:37:21.936 INFO:tasks.rados.rados.0.smithi157.stdout:342: finishing set_chunk to oid 43 2024-09-16T03:37:21.936 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 12 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T03:37:21.936 INFO:tasks.rados.rados.0.smithi157.stdout:342: done (13 left) 2024-09-16T03:37:21.936 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 13 2024-09-16T03:37:21.936 INFO:tasks.rados.rados.0.smithi157.stdout:340: finishing set_chunk to oid 41 2024-09-16T03:37:21.936 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:37:21.936 INFO:tasks.rados.rados.0.smithi157.stdout:340: done (12 left) 2024-09-16T03:37:21.936 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 12 2024-09-16T03:37:21.936 INFO:tasks.rados.rados.0.smithi157.stdout:339: finishing set_chunk to oid 40 2024-09-16T03:37:21.936 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 15 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T03:37:21.936 INFO:tasks.rados.rados.0.smithi157.stdout:339: done (11 left) 2024-09-16T03:37:21.936 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 11 2024-09-16T03:37:21.952 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:21.952 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:21.952 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:37:21.952 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:37:21.952 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:21.960 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:37:21.985 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:21.983+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:22.009 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:37:22.024 INFO:tasks.rados.rados.0.smithi157.stdout:376: finishing set_chunk to oid 27 2024-09-16T03:37:22.025 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 635904 length: 17920 2024-09-16T03:37:22.025 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:22.025 INFO:tasks.rados.rados.0.smithi157.stdout:376: done (10 left) 2024-09-16T03:37:22.025 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 10 2024-09-16T03:37:22.025 INFO:tasks.rados.rados.0.smithi157.stdout:375: finishing set_chunk to oid 26 2024-09-16T03:37:22.025 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 500736 length: 906752 2024-09-16T03:37:22.025 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:37:22.025 INFO:tasks.rados.rados.0.smithi157.stdout:375: done (9 left) 2024-09-16T03:37:22.025 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 9 2024-09-16T03:37:22.025 INFO:tasks.rados.rados.0.smithi157.stdout:330: finishing set_chunk to oid 31 2024-09-16T03:37:22.025 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:37:22.025 INFO:tasks.rados.rados.0.smithi157.stdout:330: done (8 left) 2024-09-16T03:37:22.026 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 8 2024-09-16T03:37:22.027 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:22.024+0000 7f4212d00640 -1 osd.8 45 handle_osd_map injecting map crc failure 2024-09-16T03:37:22.028 INFO:tasks.rados.rados.0.smithi157.stdout:362: finishing set_chunk to oid 13 2024-09-16T03:37:22.028 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T03:37:22.028 INFO:tasks.rados.rados.0.smithi157.stdout:362: done (7 left) 2024-09-16T03:37:22.028 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 7 2024-09-16T03:37:22.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 reset_purged_snaps_last 2024-09-16T03:37:22.068 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:22.068 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:37:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:37:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1 4.2 4:4a9d235d:::smithi15736696-1:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T03:37:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:11.955475+0000", 2024-09-16T03:37:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.109527383, 2024-09-16T03:37:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.405292714, 2024-09-16T03:37:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.069 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:37:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955475+0000", 2024-09-16T03:37:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955475+0000", 2024-09-16T03:37:22.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955477+0000", 2024-09-16T03:37:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5040000000000001e-06 2024-09-16T03:37:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955870+0000", 2024-09-16T03:37:22.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000393082 2024-09-16T03:37:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955883+0000", 2024-09-16T03:37:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3173000000000001e-05 2024-09-16T03:37:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.055959+0000", 2024-09-16T03:37:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10007564300000001 2024-09-16T03:37:22.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156811+0000", 2024-09-16T03:37:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100852073 2024-09-16T03:37:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156907+0000", 2024-09-16T03:37:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5945000000000004e-05 2024-09-16T03:37:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 8,11", 2024-09-16T03:37:22.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157029+0000", 2024-09-16T03:37:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012207099999999999 2024-09-16T03:37:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.160049+0000", 2024-09-16T03:37:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030194409999999999 2024-09-16T03:37:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260277+0000", 2024-09-16T03:37:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10022834 2024-09-16T03:37:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360663+0000", 2024-09-16T03:37:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10038625599999999 2024-09-16T03:37:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360710+0000", 2024-09-16T03:37:22.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7165e-05 2024-09-16T03:37:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360767+0000", 2024-09-16T03:37:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7021000000000002e-05 2024-09-16T03:37:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2 4.2 4:4a9d235d:::smithi15736696-1:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T03:37:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.056000+0000", 2024-09-16T03:37:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.009001779, 2024-09-16T03:37:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30489393999999997, 2024-09-16T03:37:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.077 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2 2024-09-16T03:37:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056000+0000", 2024-09-16T03:37:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056000+0000", 2024-09-16T03:37:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056003+0000", 2024-09-16T03:37:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3250000000000002e-06 2024-09-16T03:37:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056130+0000", 2024-09-16T03:37:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127059 2024-09-16T03:37:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056139+0000", 2024-09-16T03:37:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1419999999999999e-06 2024-09-16T03:37:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156244+0000", 2024-09-16T03:37:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10010532799999999 2024-09-16T03:37:22.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.158582+0000", 2024-09-16T03:37:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023374759999999998 2024-09-16T03:37:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.158698+0000", 2024-09-16T03:37:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000116051 2024-09-16T03:37:22.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 8,11", 2024-09-16T03:37:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.158807+0000", 2024-09-16T03:37:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010915400000000001 2024-09-16T03:37:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260247+0000", 2024-09-16T03:37:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10144022900000001 2024-09-16T03:37:22.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360554+0000", 2024-09-16T03:37:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100307113 2024-09-16T03:37:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360813+0000", 2024-09-16T03:37:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025846499999999999 2024-09-16T03:37:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360857+0000", 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3954e-05 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360894+0000", 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7644000000000001e-05 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:33 4.e e39/32 4:773b934b:::smithi15736696-9:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:22.085 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.153464+0000", 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.9115382010000008, 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20369546199999999, 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.153464+0000", 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.153464+0000", 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.153464+0000", 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.91e-07 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.154243+0000", 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00077855299999999997 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.154251+0000", 2024-09-16T03:37:22.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7530000000000007e-06 2024-09-16T03:37:22.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.154275+0000", 2024-09-16T03:37:22.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3954000000000001e-05 2024-09-16T03:37:22.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.255402+0000", 2024-09-16T03:37:22.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101126569 2024-09-16T03:37:22.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.255415+0000", 2024-09-16T03:37:22.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.271e-05 2024-09-16T03:37:22.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357108+0000", 2024-09-16T03:37:22.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101693282 2024-09-16T03:37:22.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357159+0000", 2024-09-16T03:37:22.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1350000000000001e-05 2024-09-16T03:37:22.091 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.091 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.091 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.091 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3 4.2 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T03:37:22.091 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.156258+0000", 2024-09-16T03:37:22.091 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.9087443589999999, 2024-09-16T03:37:22.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30472436000000003, 2024-09-16T03:37:22.091 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.091 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3 2024-09-16T03:37:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156258+0000", 2024-09-16T03:37:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.093 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.093 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156258+0000", 2024-09-16T03:37:22.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.093 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.093 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156259+0000", 2024-09-16T03:37:22.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5269999999999999e-06 2024-09-16T03:37:22.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.093 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.093 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156266+0000", 2024-09-16T03:37:22.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:37:22.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156273+0000", 2024-09-16T03:37:22.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9299999999999997e-06 2024-09-16T03:37:22.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156277+0000", 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.019e-06 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.159689+0000", 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034124329999999999 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.159744+0000", 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4821999999999998e-05 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 8,11", 2024-09-16T03:37:22.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.159858+0000", 2024-09-16T03:37:22.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011353699999999999 2024-09-16T03:37:22.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260289+0000", 2024-09-16T03:37:22.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10043086 2024-09-16T03:37:22.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360606+0000", 2024-09-16T03:37:22.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100317502 2024-09-16T03:37:22.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.460834+0000", 2024-09-16T03:37:22.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100228334 2024-09-16T03:37:22.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.460938+0000", 2024-09-16T03:37:22.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010365999999999999 2024-09-16T03:37:22.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.460982+0000", 2024-09-16T03:37:22.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4054000000000002e-05 2024-09-16T03:37:22.098 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.098 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.098 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.098 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4 4.2 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T03:37:22.098 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.156281+0000", 2024-09-16T03:37:22.099 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.9087209400000003, 2024-09-16T03:37:22.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30482965400000001, 2024-09-16T03:37:22.099 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.099 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.099 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:22.099 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.099 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.099 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.099 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4 2024-09-16T03:37:22.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.099 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156281+0000", 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156281+0000", 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156282+0000", 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.08e-07 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156284+0000", 2024-09-16T03:37:22.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7159999999999999e-06 2024-09-16T03:37:22.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156288+0000", 2024-09-16T03:37:22.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8139999999999998e-06 2024-09-16T03:37:22.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156291+0000", 2024-09-16T03:37:22.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.971e-06 2024-09-16T03:37:22.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260143+0000", 2024-09-16T03:37:22.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103851517 2024-09-16T03:37:22.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:22.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260223+0000", 2024-09-16T03:37:22.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0439999999999996e-05 2024-09-16T03:37:22.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461007+0000", 2024-09-16T03:37:22.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20078405799999999 2024-09-16T03:37:22.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461053+0000", 2024-09-16T03:37:22.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5667000000000002e-05 2024-09-16T03:37:22.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461111+0000", 2024-09-16T03:37:22.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8063000000000003e-05 2024-09-16T03:37:22.104 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.104 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.104 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.104 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:43 4.5 4:aff4cc41:::smithi15736696-11:head [setxattr _header (56) in=7b,truncate 2168070] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.256919+0000", 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.8080832000000004, 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20413487399999999, 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 43 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256919+0000", 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256919+0000", 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256921+0000", 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7179999999999999e-06 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256928+0000", 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0260000000000002e-06 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256936+0000", 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8550000000000008e-06 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256942+0000", 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.366e-06 2024-09-16T03:37:22.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258713+0000", 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001770981 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258808+0000", 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5657999999999998e-05 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258916+0000", 2024-09-16T03:37:22.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000107614 2024-09-16T03:37:22.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.261237+0000", 2024-09-16T03:37:22.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023207169999999999 2024-09-16T03:37:22.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360267+0000", 2024-09-16T03:37:22.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099029928000000003 2024-09-16T03:37:22.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.460976+0000", 2024-09-16T03:37:22.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100709222 2024-09-16T03:37:22.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461019+0000", 2024-09-16T03:37:22.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3080000000000001e-05 2024-09-16T03:37:22.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461054+0000", 2024-09-16T03:37:22.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4709000000000003e-05 2024-09-16T03:37:22.111 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.112 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.112 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.112 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:44 4.5 4:aff4cc41:::smithi15736696-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:22.112 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.256951+0000", 2024-09-16T03:37:22.112 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.8080515290000001, 2024-09-16T03:37:22.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20432772199999999, 2024-09-16T03:37:22.112 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.112 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.112 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:22.112 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.112 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 44 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256951+0000", 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256951+0000", 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256951+0000", 2024-09-16T03:37:22.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8899999999999999e-07 2024-09-16T03:37:22.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256955+0000", 2024-09-16T03:37:22.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0090000000000001e-06 2024-09-16T03:37:22.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256961+0000", 2024-09-16T03:37:22.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1179999999999997e-06 2024-09-16T03:37:22.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256967+0000", 2024-09-16T03:37:22.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3319999999999999e-06 2024-09-16T03:37:22.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259192+0000", 2024-09-16T03:37:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022248200000000002 2024-09-16T03:37:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259265+0000", 2024-09-16T03:37:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3016999999999996e-05 2024-09-16T03:37:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461104+0000", 2024-09-16T03:37:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20183947899999999 2024-09-16T03:37:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461178+0000", 2024-09-16T03:37:22.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4053000000000004e-05 2024-09-16T03:37:22.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461278+0000", 2024-09-16T03:37:22.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010020499999999999 2024-09-16T03:37:22.117 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.117 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.118 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.118 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:61 4.5 4:ad0b9852:::smithi15736696-16:head [write 652105~445088,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:22.118 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.256974+0000", 2024-09-16T03:37:22.118 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.8080277460000005, 2024-09-16T03:37:22.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30469791499999999, 2024-09-16T03:37:22.118 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.118 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.118 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.118 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.118 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 61 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256974+0000", 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256974+0000", 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256975+0000", 2024-09-16T03:37:22.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0299999999999998e-07 2024-09-16T03:37:22.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257113+0000", 2024-09-16T03:37:22.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013768100000000001 2024-09-16T03:37:22.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257121+0000", 2024-09-16T03:37:22.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1680000000000004e-06 2024-09-16T03:37:22.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257127+0000", 2024-09-16T03:37:22.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3099999999999997e-06 2024-09-16T03:37:22.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259291+0000", 2024-09-16T03:37:22.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021634839999999998 2024-09-16T03:37:22.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259396+0000", 2024-09-16T03:37:22.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105364 2024-09-16T03:37:22.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:37:22.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259545+0000", 2024-09-16T03:37:22.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014917700000000001 2024-09-16T03:37:22.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.261929+0000", 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023834709999999999 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561404+0000", 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.299475251 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561560+0000", 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015627800000000001 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561619+0000", 2024-09-16T03:37:22.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8202999999999998e-05 2024-09-16T03:37:22.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561672+0000", 2024-09-16T03:37:22.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3724999999999998e-05 2024-09-16T03:37:22.124 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.124 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.124 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.124 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:62 4.5 4:ad0b9852:::smithi15736696-16:head [write 1728171~452988] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257137+0000", 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.8078647649999997, 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30464145300000001, 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 62 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257137+0000", 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257137+0000", 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257138+0000", 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3600000000000002e-07 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257307+0000", 2024-09-16T03:37:22.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016848899999999999 2024-09-16T03:37:22.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257315+0000", 2024-09-16T03:37:22.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1110000000000008e-06 2024-09-16T03:37:22.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257319+0000", 2024-09-16T03:37:22.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:37:22.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260333+0000", 2024-09-16T03:37:22.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030136160000000002 2024-09-16T03:37:22.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260388+0000", 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4762999999999997e-05 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260485+0000", 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7314999999999999e-05 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.262171+0000", 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001685892 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561467+0000", 2024-09-16T03:37:22.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29929608299999999 2024-09-16T03:37:22.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561709+0000", 2024-09-16T03:37:22.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024165800000000001 2024-09-16T03:37:22.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561748+0000", 2024-09-16T03:37:22.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8977999999999998e-05 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561779+0000", 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0959e-05 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:63 4.5 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=7b,truncate 2252105] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257326+0000", 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.8076762780000006, 2024-09-16T03:37:22.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30459017500000002, 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 63 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257326+0000", 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257326+0000", 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257326+0000", 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.37e-07 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257329+0000", 2024-09-16T03:37:22.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.729e-06 2024-09-16T03:37:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257333+0000", 2024-09-16T03:37:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2509999999999996e-06 2024-09-16T03:37:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257337+0000", 2024-09-16T03:37:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2839999999999998e-06 2024-09-16T03:37:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.261292+0000", 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003955404 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.261340+0000", 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.778e-05 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.261419+0000", 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9716999999999996e-05 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.262200+0000", 2024-09-16T03:37:22.136 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078059600000000005 2024-09-16T03:37:22.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561515+0000", 2024-09-16T03:37:22.136 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29931443299999999 2024-09-16T03:37:22.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561816+0000", 2024-09-16T03:37:22.136 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030171499999999998 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561863+0000", 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6927e-05 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561916+0000", 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2802e-05 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.138 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.138 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:64 4.5 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:22.138 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:37:22.138 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.8076617689999992, 2024-09-16T03:37:22.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50484290399999998, 2024-09-16T03:37:22.139 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.139 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.139 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:22.139 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.139 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.139 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.139 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 64 2024-09-16T03:37:22.139 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.139 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.139 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.139 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.139 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:37:22.139 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257341+0000", 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.140 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257343+0000", 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3889999999999999e-06 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257346+0000", 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257349+0000", 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8559999999999998e-06 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357057+0000", 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099708121999999996 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357156+0000", 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9162999999999999e-05 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762060+0000", 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40490364400000001 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762112+0000", 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1882000000000002e-05 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762183+0000", 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1297000000000005e-05 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.143 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:284 4.b 4:d7a855d3:::smithi15736696-35:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:22.144 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:15.421062+0000", 2024-09-16T03:37:22.144 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.6439404880000001, 2024-09-16T03:37:22.144 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.28498220899999999, 2024-09-16T03:37:22.144 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.144 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.144 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.144 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.144 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.144 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.144 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 284 2024-09-16T03:37:22.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.144 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.421062+0000", 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.421062+0000", 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.421063+0000", 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.142e-06 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.421069+0000", 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9680000000000003e-06 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.421076+0000", 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7540000000000006e-06 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.421084+0000", 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.374e-06 2024-09-16T03:37:22.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.421116+0000", 2024-09-16T03:37:22.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2227999999999997e-05 2024-09-16T03:37:22.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.421165+0000", 2024-09-16T03:37:22.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:37:22.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,9", 2024-09-16T03:37:22.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.705059+0000", 2024-09-16T03:37:22.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.28389384400000001 2024-09-16T03:37:22.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.705537+0000", 2024-09-16T03:37:22.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00047772800000000002 2024-09-16T03:37:22.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.705871+0000", 2024-09-16T03:37:22.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033415099999999998 2024-09-16T03:37:22.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.705990+0000", 2024-09-16T03:37:22.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118539 2024-09-16T03:37:22.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.706019+0000", 2024-09-16T03:37:22.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8768e-05 2024-09-16T03:37:22.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.706044+0000", 2024-09-16T03:37:22.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5281000000000001e-05 2024-09-16T03:37:22.150 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.150 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.150 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.150 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:292 4.5 4:a11e1d7c:::smithi15736696-43:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:22.150 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:15.803137+0000", 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.2618651170000001, 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.28187556400000002, 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 292 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.803137+0000", 2024-09-16T03:37:22.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.803137+0000", 2024-09-16T03:37:22.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.803138+0000", 2024-09-16T03:37:22.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.454e-06 2024-09-16T03:37:22.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.803143+0000", 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0560000000000004e-06 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.803153+0000", 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9820000000000003e-06 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.803159+0000", 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8309999999999997e-06 2024-09-16T03:37:22.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.803209+0000", 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9452e-05 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.803259+0000", 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0266000000000003e-05 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.004962+0000", 2024-09-16T03:37:22.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20170253399999999 2024-09-16T03:37:22.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.005415+0000", 2024-09-16T03:37:22.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00045342800000000002 2024-09-16T03:37:22.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.005661+0000", 2024-09-16T03:37:22.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024615100000000001 2024-09-16T03:37:22.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.084936+0000", 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.079275171000000005 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.084986+0000", 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9305000000000003e-05 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085013+0000", 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6934e-05 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.156 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.157 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.157 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:4 3.3 3:deac805c:::smithi15736696-44 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:22.157 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:15.902365+0000", 2024-09-16T03:37:22.157 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.1626375040000001, 2024-09-16T03:37:22.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30203107200000001, 2024-09-16T03:37:22.157 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.157 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.157 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.157 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.902365+0000", 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.902365+0000", 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.902368+0000", 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9830000000000001e-06 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.902378+0000", 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0773e-05 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.902405+0000", 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6876000000000002e-05 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.902420+0000", 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4953999999999999e-05 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.902571+0000", 2024-09-16T03:37:22.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015106000000000001 2024-09-16T03:37:22.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.902718+0000", 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000147207 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.902827+0000", 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010882699999999999 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.903326+0000", 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00049833099999999997 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.006945+0000", 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103619764 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.204197+0000", 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19725152400000001 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.204286+0000", 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8987999999999996e-05 2024-09-16T03:37:22.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.204396+0000", 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010978499999999999 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:295 4.b 4:dbb349e8:::smithi15736696-46:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.085151+0000", 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.979850839, 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.42198747199999997, 2024-09-16T03:37:22.163 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.164 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.164 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.164 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.164 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.164 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.164 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 295 2024-09-16T03:37:22.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.164 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085151+0000", 2024-09-16T03:37:22.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085151+0000", 2024-09-16T03:37:22.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085153+0000", 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.421e-06 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085158+0000", 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0239999999999997e-06 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085166+0000", 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6400000000000003e-06 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085172+0000", 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2499999999999997e-06 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085293+0000", 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000121853 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085366+0000", 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2872000000000003e-05 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,9", 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.288783+0000", 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20341624799999999 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.289205+0000", 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042208099999999999 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.289914+0000", 2024-09-16T03:37:22.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00070928600000000005 2024-09-16T03:37:22.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.507063+0000", 2024-09-16T03:37:22.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.21714887299999999 2024-09-16T03:37:22.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.507095+0000", 2024-09-16T03:37:22.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2570000000000002e-05 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.507139+0000", 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3353999999999999e-05 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:400 4.2 4:4a9d235d:::smithi15736696-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.205970+0000", 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.859031686, 2024-09-16T03:37:22.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40465369000000001, 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 400 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205970+0000", 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205970+0000", 2024-09-16T03:37:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205972+0000", 2024-09-16T03:37:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.841e-06 2024-09-16T03:37:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205979+0000", 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9090000000000003e-06 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205991+0000", 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.165e-05 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.206013+0000", 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2079e-05 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.306137+0000", 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10012456 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.306223+0000", 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5181999999999999e-05 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 8,11", 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309852+0000", 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036292189999999999 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.310473+0000", 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062141500000000005 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.608696+0000", 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29822300400000001 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.610553+0000", 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018567989999999999 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.610587+0000", 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3995999999999998e-05 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.610624+0000", 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7036000000000003e-05 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.176 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:415 4.5 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.308823+0000", 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.7561790230000001, 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30007591700000003, 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 415 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.308823+0000", 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.308823+0000", 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.308824+0000", 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3459999999999999e-06 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.308829+0000", 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9169999999999997e-06 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.308840+0000", 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0223e-05 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.308847+0000", 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.96e-06 2024-09-16T03:37:22.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.308898+0000", 2024-09-16T03:37:22.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0117000000000001e-05 2024-09-16T03:37:22.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.308983+0000", 2024-09-16T03:37:22.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4994000000000007e-05 2024-09-16T03:37:22.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:37:22.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.410736+0000", 2024-09-16T03:37:22.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101753129 2024-09-16T03:37:22.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.411164+0000", 2024-09-16T03:37:22.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042846900000000001 2024-09-16T03:37:22.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.608555+0000", 2024-09-16T03:37:22.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.197390544 2024-09-16T03:37:22.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.608775+0000", 2024-09-16T03:37:22.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022056500000000001 2024-09-16T03:37:22.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.608843+0000", 2024-09-16T03:37:22.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7478999999999995e-05 2024-09-16T03:37:22.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.608899+0000", 2024-09-16T03:37:22.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6174000000000001e-05 2024-09-16T03:37:22.183 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.183 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.183 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.183 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.10.22:22 3.3 3:f84e13a5:::smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:22.183 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:37:22.183 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.755514534, 2024-09-16T03:37:22.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50224271099999995, 2024-09-16T03:37:22.183 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.183 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.183 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.183 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309489+0000", 2024-09-16T03:37:22.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4309999999999999e-06 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309496+0000", 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5810000000000002e-06 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309505+0000", 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6889999999999999e-06 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309513+0000", 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3749999999999997e-06 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.406932+0000", 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.097419631000000007 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407009+0000", 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7220999999999998e-05 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407143+0000", 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013323799999999999 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407732+0000", 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058938399999999998 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.408133+0000", 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040090500000000001 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811599+0000", 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40346633399999998 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811682+0000", 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2463e-05 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811730+0000", 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8458999999999998e-05 2024-09-16T03:37:22.188 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:442 4.5 4:a11e1d7c:::smithi15736696-43:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.014491+0000", 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.050510936, 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30407201099999998, 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.189 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 442 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.014491+0000", 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.014491+0000", 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.014493+0000", 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6899999999999999e-06 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.014500+0000", 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0550000000000002e-06 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.014508+0000", 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1589999999999993e-06 2024-09-16T03:37:22.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.014521+0000", 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2529e-05 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015412+0000", 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00089155799999999998 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015494+0000", 2024-09-16T03:37:22.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1963000000000001e-05 2024-09-16T03:37:22.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:37:22.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.217465+0000", 2024-09-16T03:37:22.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20197077499999999 2024-09-16T03:37:22.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.218135+0000", 2024-09-16T03:37:22.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00067010599999999996 2024-09-16T03:37:22.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.318389+0000", 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100253811 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.318477+0000", 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8189000000000006e-05 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.318526+0000", 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8761999999999998e-05 2024-09-16T03:37:22.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.318563+0000", 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7413999999999998e-05 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:465 4.5 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.049296996, 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60529649299999999, 2024-09-16T03:37:22.195 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:22.196 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:22.196 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:22.196 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:22.196 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:22.196 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:22.196 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 465 2024-09-16T03:37:22.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.196 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:22.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:22.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:37:22.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015706+0000", 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.091e-06 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015712+0000", 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-16T03:37:22.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:22.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015724+0000", 2024-09-16T03:37:22.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1408e-05 2024-09-16T03:37:22.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:22.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015731+0000", 2024-09-16T03:37:22.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:37:22.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:22.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216049+0000", 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200317895 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216178+0000", 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012848800000000001 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620010+0000", 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40383207199999999 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620710+0000", 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069988400000000001 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620902+0000", 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019197800000000001 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620933+0000", 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1513999999999997e-05 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620968+0000", 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4524999999999999e-05 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.621002+0000", 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3955000000000003e-05 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:22.201 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:22.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:37:22.204 INFO:tasks.rados.rados.0.smithi157.stdout:377: finishing set_chunk to oid 28 2024-09-16T03:37:22.204 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 3 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T03:37:22.204 INFO:tasks.rados.rados.0.smithi157.stdout:377: done (6 left) 2024-09-16T03:37:22.204 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 6 2024-09-16T03:37:22.204 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:22.201+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:22.219 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:37:22.226 INFO:tasks.rados.rados.0.smithi157.stdout:329: finishing set_chunk to oid 30 2024-09-16T03:37:22.226 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:37:22.226 INFO:tasks.rados.rados.0.smithi157.stdout:329: done (5 left) 2024-09-16T03:37:22.226 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 5 2024-09-16T03:37:22.227 INFO:tasks.rados.rados.0.smithi157.stdout:359: finishing set_chunk to oid 10 2024-09-16T03:37:22.227 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:37:22.227 INFO:tasks.rados.rados.0.smithi157.stdout:359: done (4 left) 2024-09-16T03:37:22.227 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 4 2024-09-16T03:37:22.239 INFO:tasks.rados.rados.0.smithi157.stdout:358: finishing set_chunk to oid 9 2024-09-16T03:37:22.239 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:37:22.239 INFO:tasks.rados.rados.0.smithi157.stdout:358: done (3 left) 2024-09-16T03:37:22.239 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 3 2024-09-16T03:37:22.303 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:22.302+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:22.315 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:22.315 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:22.315 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:37:22.315 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:22.325 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:37:22.328 INFO:tasks.rados.rados.0.smithi157.stdout:344: finishing set_chunk to oid 45 2024-09-16T03:37:22.328 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 15 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T03:37:22.328 INFO:tasks.rados.rados.0.smithi157.stdout:344: done (2 left) 2024-09-16T03:37:22.328 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 2 2024-09-16T03:37:22.346 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 reset_purged_snaps_last 2024-09-16T03:37:22.377 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:22.377 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:22.396 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:22.396 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:22.404 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:22.403+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:22.413 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:22.413 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:22.428 INFO:tasks.rados.rados.0.smithi157.stdout:353: finishing set_chunk to oid 4 2024-09-16T03:37:22.428 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:37:22.428 INFO:tasks.rados.rados.0.smithi157.stdout:353: done (1 left) 2024-09-16T03:37:22.428 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 1 2024-09-16T03:37:22.433 INFO:tasks.rados.rados.0.smithi157.stdout:367: finishing set_chunk to oid 18 2024-09-16T03:37:22.433 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T03:37:22.433 INFO:tasks.rados.rados.0.smithi157.stdout:367: done (0 left) 2024-09-16T03:37:22.433 INFO:tasks.rados.rados.0.smithi157.stdout:379: set_chunk oid 30 target oid 30 low_tier 2024-09-16T03:37:22.433 INFO:tasks.rados.rados.0.smithi157.stdout:379: set_chunk oid 30 offset: 505856 length: 1895936 target oid smithi15736696-30 low_tier offset: 505856 snap -1 2024-09-16T03:37:22.434 INFO:tasks.rados.rados.0.smithi157.stdout:380: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.434 INFO:tasks.rados.rados.0.smithi157.stdout:380: set_chunk oid 31 offset: 1474048 length: 174592 target oid smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1474048 snap -1 2024-09-16T03:37:22.434 INFO:tasks.rados.rados.0.smithi157.stdout:381: set_chunk oid 32 target oid 32 low_tier 2024-09-16T03:37:22.434 INFO:tasks.rados.rados.0.smithi157.stdout:381: set_chunk oid 32 offset: 1372160 length: 336384 target oid smithi15736696-32 low_tier offset: 1372160 snap -1 2024-09-16T03:37:22.434 INFO:tasks.rados.rados.0.smithi157.stdout:382: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.434 INFO:tasks.rados.rados.0.smithi157.stdout:382: set_chunk oid 33 offset: 696320 length: 527872 target oid smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 696320 snap -1 2024-09-16T03:37:22.434 INFO:tasks.rados.rados.0.smithi157.stdout:383: set_chunk oid 34 target oid 34 low_tier 2024-09-16T03:37:22.434 INFO:tasks.rados.rados.0.smithi157.stdout:383: set_chunk oid 34 offset: 467456 length: 1874944 target oid smithi15736696-34 low_tier offset: 467456 snap -1 2024-09-16T03:37:22.434 INFO:tasks.rados.rados.0.smithi157.stdout:384: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.434 INFO:tasks.rados.rados.0.smithi157.stdout:384: set_chunk oid 35 offset: 393216 length: 296960 target oid smithi15736696-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 393216 snap -1 2024-09-16T03:37:22.434 INFO:tasks.rados.rados.0.smithi157.stdout:385: set_chunk oid 36 target oid 36 low_tier 2024-09-16T03:37:22.434 INFO:tasks.rados.rados.0.smithi157.stdout:385: set_chunk oid 36 offset: 385024 length: 697856 target oid smithi15736696-36 low_tier offset: 385024 snap -1 2024-09-16T03:37:22.434 INFO:tasks.rados.rados.0.smithi157.stdout:386: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.435 INFO:tasks.rados.rados.0.smithi157.stdout:386: set_chunk oid 37 offset: 576000 length: 124928 target oid smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 576000 snap -1 2024-09-16T03:37:22.435 INFO:tasks.rados.rados.0.smithi157.stdout:387: set_chunk oid 38 target oid 38 low_tier 2024-09-16T03:37:22.435 INFO:tasks.rados.rados.0.smithi157.stdout:387: set_chunk oid 38 offset: 232960 length: 1852928 target oid smithi15736696-38 low_tier offset: 232960 snap -1 2024-09-16T03:37:22.435 INFO:tasks.rados.rados.0.smithi157.stdout:388: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.435 INFO:tasks.rados.rados.0.smithi157.stdout:388: set_chunk oid 39 offset: 230400 length: 1939968 target oid smithi15736696-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 230400 snap -1 2024-09-16T03:37:22.435 INFO:tasks.rados.rados.0.smithi157.stdout:389: set_chunk oid 40 target oid 40 low_tier 2024-09-16T03:37:22.435 INFO:tasks.rados.rados.0.smithi157.stdout:389: set_chunk oid 40 offset: 129536 length: 27136 target oid smithi15736696-40 low_tier offset: 129536 snap -1 2024-09-16T03:37:22.435 INFO:tasks.rados.rados.0.smithi157.stdout:390: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.435 INFO:tasks.rados.rados.0.smithi157.stdout:390: set_chunk oid 41 offset: 2404352 length: 150016 target oid smithi15736696-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2404352 snap -1 2024-09-16T03:37:22.435 INFO:tasks.rados.rados.0.smithi157.stdout:391: set_chunk oid 42 target oid 42 low_tier 2024-09-16T03:37:22.435 INFO:tasks.rados.rados.0.smithi157.stdout:391: set_chunk oid 42 offset: 850944 length: 1150976 target oid smithi15736696-42 low_tier offset: 850944 snap -1 2024-09-16T03:37:22.435 INFO:tasks.rados.rados.0.smithi157.stdout:392: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.435 INFO:tasks.rados.rados.0.smithi157.stdout:392: set_chunk oid 43 offset: 1084928 length: 946176 target oid smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1084928 snap -1 2024-09-16T03:37:22.436 INFO:tasks.rados.rados.0.smithi157.stdout:393: set_chunk oid 44 target oid 44 low_tier 2024-09-16T03:37:22.436 INFO:tasks.rados.rados.0.smithi157.stdout:393: set_chunk oid 44 offset: 1202176 length: 1363968 target oid smithi15736696-44 low_tier offset: 1202176 snap -1 2024-09-16T03:37:22.436 INFO:tasks.rados.rados.0.smithi157.stdout:394: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.436 INFO:tasks.rados.rados.0.smithi157.stdout:394: set_chunk oid 45 offset: 975872 length: 606208 target oid smithi15736696-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 975872 snap -1 2024-09-16T03:37:22.436 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.436 INFO:tasks.rados.rados.0.smithi157.stdout:380: finishing set_chunk to oid 31 2024-09-16T03:37:22.436 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1474048 length: 174592 2024-09-16T03:37:22.436 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:37:22.436 INFO:tasks.rados.rados.0.smithi157.stdout:380: done (15 left) 2024-09-16T03:37:22.436 INFO:tasks.rados.rados.0.smithi157.stdout:395: set_chunk oid 46 target oid 46 low_tier 2024-09-16T03:37:22.436 INFO:tasks.rados.rados.0.smithi157.stdout:395: set_chunk oid 46 offset: 467456 length: 2064384 target oid smithi15736696-46 low_tier offset: 467456 snap -1 2024-09-16T03:37:22.436 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.437 INFO:tasks.rados.rados.0.smithi157.stdout:383: finishing set_chunk to oid 34 2024-09-16T03:37:22.437 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 467456 length: 1874944 2024-09-16T03:37:22.437 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T03:37:22.437 INFO:tasks.rados.rados.0.smithi157.stdout:383: done (15 left) 2024-09-16T03:37:22.437 INFO:tasks.rados.rados.0.smithi157.stdout:396: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.437 INFO:tasks.rados.rados.0.smithi157.stdout:396: set_chunk oid 47 offset: 2350080 length: 86016 target oid smithi15736696-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2350080 snap -1 2024-09-16T03:37:22.437 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.437 INFO:tasks.rados.rados.0.smithi157.stdout:384: finishing set_chunk to oid 35 2024-09-16T03:37:22.437 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 393216 length: 296960 2024-09-16T03:37:22.437 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:37:22.437 INFO:tasks.rados.rados.0.smithi157.stdout:388: finishing set_chunk to oid 39 2024-09-16T03:37:22.437 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 230400 length: 1939968 2024-09-16T03:37:22.437 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T03:37:22.438 INFO:tasks.rados.rados.0.smithi157.stdout:384: done (15 left) 2024-09-16T03:37:22.438 INFO:tasks.rados.rados.0.smithi157.stdout:388: done (14 left) 2024-09-16T03:37:22.438 INFO:tasks.rados.rados.0.smithi157.stdout:397: set_chunk oid 48 target oid 48 low_tier 2024-09-16T03:37:22.438 INFO:tasks.rados.rados.0.smithi157.stdout:397: set_chunk oid 48 offset: 992256 length: 1429504 target oid smithi15736696-48 low_tier offset: 992256 snap -1 2024-09-16T03:37:22.438 INFO:tasks.rados.rados.0.smithi157.stdout:398: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.438 INFO:tasks.rados.rados.0.smithi157.stdout:398: set_chunk oid 49 offset: 883712 length: 884224 target oid smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 883712 snap -1 2024-09-16T03:37:22.438 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.438 INFO:tasks.rados.rados.0.smithi157.stdout:379: finishing set_chunk to oid 30 2024-09-16T03:37:22.438 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 505856 length: 1895936 2024-09-16T03:37:22.438 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:37:22.439 INFO:tasks.rados.rados.0.smithi157.stdout:379: done (15 left) 2024-09-16T03:37:22.439 INFO:tasks.rados.rados.0.smithi157.stdout:399: set_chunk oid 50 target oid 50 low_tier 2024-09-16T03:37:22.439 INFO:tasks.rados.rados.0.smithi157.stdout:399: set_chunk oid 50 offset: 346112 length: 1980928 target oid smithi15736696-50 low_tier offset: 346112 snap -1 2024-09-16T03:37:22.439 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.439 INFO:tasks.rados.rados.0.smithi157.stdout:399: finishing set_chunk to oid 50 2024-09-16T03:37:22.439 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 346112 length: 1980928 2024-09-16T03:37:22.440 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:37:22.440 INFO:tasks.rados.rados.0.smithi157.stdout:397: finishing set_chunk to oid 48 2024-09-16T03:37:22.440 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 992256 length: 1429504 2024-09-16T03:37:22.440 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:37:22.440 INFO:tasks.rados.rados.0.smithi157.stdout:397: done (15 left) 2024-09-16T03:37:22.440 INFO:tasks.rados.rados.0.smithi157.stdout:399: done (14 left) 2024-09-16T03:37:22.440 INFO:tasks.rados.rados.0.smithi157.stdout:400: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.440 INFO:tasks.rados.rados.0.smithi157.stdout:400: set_chunk oid 1 offset: 466944 length: 947200 target oid smithi15736696-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 466944 snap -1 2024-09-16T03:37:22.440 INFO:tasks.rados.rados.0.smithi157.stdout:401: set_chunk oid 2 target oid 2 low_tier 2024-09-16T03:37:22.440 INFO:tasks.rados.rados.0.smithi157.stdout:401: set_chunk oid 2 offset: 537088 length: 1444864 target oid smithi15736696-2 low_tier offset: 537088 snap -1 2024-09-16T03:37:22.440 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.441 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:22.441 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:22.441 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:22.441 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:22.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:37:22.462 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:22.463 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:22.480 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:22.491 INFO:tasks.rados.rados.0.smithi157.stdout:386: finishing set_chunk to oid 37 2024-09-16T03:37:22.491 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 576000 length: 124928 2024-09-16T03:37:22.491 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T03:37:22.491 INFO:tasks.rados.rados.0.smithi157.stdout:386: done (15 left) 2024-09-16T03:37:22.491 INFO:tasks.rados.rados.0.smithi157.stdout:402: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.491 INFO:tasks.rados.rados.0.smithi157.stdout:402: set_chunk oid 3 offset: 516096 length: 913920 target oid smithi15736696-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 516096 snap -1 2024-09-16T03:37:22.491 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.491 INFO:tasks.rados.rados.0.smithi157.stdout:394: finishing set_chunk to oid 45 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 975872 length: 606208 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout:394: done (15 left) 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout:403: set_chunk oid 4 target oid 4 low_tier 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout:403: set_chunk oid 4 offset: 908800 length: 437248 target oid smithi15736696-4 low_tier offset: 908800 snap -1 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout:390: finishing set_chunk to oid 41 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2404352 length: 150016 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout:390: done (15 left) 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout:404: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout:404: set_chunk oid 5 offset: 890368 length: 857600 target oid smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 890368 snap -1 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.492 INFO:tasks.rados.rados.0.smithi157.stdout:402: finishing set_chunk to oid 3 2024-09-16T03:37:22.493 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 516096 length: 913920 2024-09-16T03:37:22.493 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T03:37:22.493 INFO:tasks.rados.rados.0.smithi157.stdout:402: done (15 left) 2024-09-16T03:37:22.493 INFO:tasks.rados.rados.0.smithi157.stdout:405: set_chunk oid 6 target oid 6 low_tier 2024-09-16T03:37:22.493 INFO:tasks.rados.rados.0.smithi157.stdout:405: set_chunk oid 6 offset: 511488 length: 1139712 target oid smithi15736696-6 low_tier offset: 511488 snap -1 2024-09-16T03:37:22.493 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.493 INFO:tasks.rados.rados.0.smithi157.stdout:385: finishing set_chunk to oid 36 2024-09-16T03:37:22.493 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 385024 length: 697856 2024-09-16T03:37:22.493 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:37:22.493 INFO:tasks.rados.rados.0.smithi157.stdout:385: done (15 left) 2024-09-16T03:37:22.493 INFO:tasks.rados.rados.0.smithi157.stdout:406: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.493 INFO:tasks.rados.rados.0.smithi157.stdout:406: set_chunk oid 7 offset: 1522688 length: 484352 target oid smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1522688 snap -1 2024-09-16T03:37:22.493 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.497 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:22.497 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:22.505 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:22.503+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:22.515 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:22.515 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:22.533 INFO:tasks.rados.rados.0.smithi157.stdout:403: finishing set_chunk to oid 4 2024-09-16T03:37:22.533 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 908800 length: 437248 2024-09-16T03:37:22.533 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:37:22.533 INFO:tasks.rados.rados.0.smithi157.stdout:403: done (15 left) 2024-09-16T03:37:22.533 INFO:tasks.rados.rados.0.smithi157.stdout:407: set_chunk oid 8 target oid 8 low_tier 2024-09-16T03:37:22.533 INFO:tasks.rados.rados.0.smithi157.stdout:407: set_chunk oid 8 offset: 701952 length: 1253888 target oid smithi15736696-8 low_tier offset: 701952 snap -1 2024-09-16T03:37:22.533 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout:407: finishing set_chunk to oid 8 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 701952 length: 1253888 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout:407: done (15 left) 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout:408: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout:408: set_chunk oid 9 offset: 93696 length: 264192 target oid smithi15736696-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 93696 snap -1 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout:398: finishing set_chunk to oid 49 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 883712 length: 884224 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout:382: finishing set_chunk to oid 33 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 696320 length: 527872 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:37:22.534 INFO:tasks.rados.rados.0.smithi157.stdout:382: done (15 left) 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout:398: done (14 left) 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout:409: set_chunk oid 10 target oid 10 low_tier 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout:409: set_chunk oid 10 offset: 524288 length: 1492480 target oid smithi15736696-10 low_tier offset: 524288 snap -1 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout:410: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout:410: set_chunk oid 11 offset: 1144320 length: 35840 target oid smithi15736696-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1144320 snap -1 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout:393: finishing set_chunk to oid 44 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 1202176 length: 1363968 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout:408: finishing set_chunk to oid 9 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 93696 length: 264192 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout:393: done (15 left) 2024-09-16T03:37:22.535 INFO:tasks.rados.rados.0.smithi157.stdout:408: done (14 left) 2024-09-16T03:37:22.536 INFO:tasks.rados.rados.0.smithi157.stdout:411: set_chunk oid 12 target oid 12 low_tier 2024-09-16T03:37:22.536 INFO:tasks.rados.rados.0.smithi157.stdout:411: set_chunk oid 12 offset: 1188864 length: 264192 target oid smithi15736696-12 low_tier offset: 1188864 snap -1 2024-09-16T03:37:22.536 INFO:tasks.rados.rados.0.smithi157.stdout:412: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.536 INFO:tasks.rados.rados.0.smithi157.stdout:412: set_chunk oid 13 offset: 501248 length: 490496 target oid smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 501248 snap -1 2024-09-16T03:37:22.536 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.536 INFO:tasks.rados.rados.0.smithi157.stdout:381: finishing set_chunk to oid 32 2024-09-16T03:37:22.536 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1372160 length: 336384 2024-09-16T03:37:22.536 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:37:22.536 INFO:tasks.rados.rados.0.smithi157.stdout:387: finishing set_chunk to oid 38 2024-09-16T03:37:22.536 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 232960 length: 1852928 2024-09-16T03:37:22.536 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:37:22.536 INFO:tasks.rados.rados.0.smithi157.stdout:392: finishing set_chunk to oid 43 2024-09-16T03:37:22.536 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1084928 length: 946176 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout:381: done (15 left) 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout:387: done (14 left) 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout:392: done (13 left) 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout:413: set_chunk oid 14 target oid 14 low_tier 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout:413: set_chunk oid 14 offset: 626688 length: 1476608 target oid smithi15736696-14 low_tier offset: 626688 snap -1 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout:414: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout:414: set_chunk oid 15 offset: 722432 length: 1417216 target oid smithi15736696-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 722432 snap -1 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout:415: set_chunk oid 16 target oid 16 low_tier 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout:415: set_chunk oid 16 offset: 580608 length: 504832 target oid smithi15736696-16 low_tier offset: 580608 snap -1 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout:409: finishing set_chunk to oid 10 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 524288 length: 1492480 2024-09-16T03:37:22.537 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:391: finishing set_chunk to oid 42 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 850944 length: 1150976 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:395: finishing set_chunk to oid 46 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 467456 length: 2064384 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:391: done (15 left) 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:395: done (14 left) 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:409: done (13 left) 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:416: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:416: set_chunk oid 17 offset: 1095168 length: 98816 target oid smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1095168 snap -1 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:417: set_chunk oid 18 target oid 18 low_tier 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:417: set_chunk oid 18 offset: 585216 length: 28672 target oid smithi15736696-18 low_tier offset: 585216 snap -1 2024-09-16T03:37:22.538 INFO:tasks.rados.rados.0.smithi157.stdout:418: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.539 INFO:tasks.rados.rados.0.smithi157.stdout:418: set_chunk oid 19 offset: 26624 length: 2191360 target oid smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 26624 snap -1 2024-09-16T03:37:22.539 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.539 INFO:tasks.rados.rados.0.smithi157.stdout:410: finishing set_chunk to oid 11 2024-09-16T03:37:22.539 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1144320 length: 35840 2024-09-16T03:37:22.539 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T03:37:22.539 INFO:tasks.rados.rados.0.smithi157.stdout:396: finishing set_chunk to oid 47 2024-09-16T03:37:22.539 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 2350080 length: 86016 2024-09-16T03:37:22.539 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T03:37:22.539 INFO:tasks.rados.rados.0.smithi157.stdout:404: finishing set_chunk to oid 5 2024-09-16T03:37:22.539 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 890368 length: 857600 2024-09-16T03:37:22.539 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T03:37:22.539 INFO:tasks.rados.rados.0.smithi157.stdout:396: done (15 left) 2024-09-16T03:37:22.539 INFO:tasks.rados.rados.0.smithi157.stdout:404: done (14 left) 2024-09-16T03:37:22.540 INFO:tasks.rados.rados.0.smithi157.stdout:410: done (13 left) 2024-09-16T03:37:22.540 INFO:tasks.rados.rados.0.smithi157.stdout:419: set_chunk oid 20 target oid 20 low_tier 2024-09-16T03:37:22.540 INFO:tasks.rados.rados.0.smithi157.stdout:419: set_chunk oid 20 offset: 358400 length: 1290752 target oid smithi15736696-20 low_tier offset: 358400 snap -1 2024-09-16T03:37:22.540 INFO:tasks.rados.rados.0.smithi157.stdout:420: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.540 INFO:tasks.rados.rados.0.smithi157.stdout:420: set_chunk oid 21 offset: 1296896 length: 666624 target oid smithi15736696-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1296896 snap -1 2024-09-16T03:37:22.540 INFO:tasks.rados.rados.0.smithi157.stdout:421: set_chunk oid 22 target oid 22 low_tier 2024-09-16T03:37:22.540 INFO:tasks.rados.rados.0.smithi157.stdout:421: set_chunk oid 22 offset: 508928 length: 280576 target oid smithi15736696-22 low_tier offset: 508928 snap -1 2024-09-16T03:37:22.540 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout:411: finishing set_chunk to oid 12 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1188864 length: 264192 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout:411: done (15 left) 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout:422: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout:422: set_chunk oid 23 offset: 885248 length: 96768 target oid smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 885248 snap -1 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout:414: finishing set_chunk to oid 15 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 722432 length: 1417216 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout:414: done (15 left) 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout:423: set_chunk oid 24 target oid 24 low_tier 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout:423: set_chunk oid 24 offset: 897024 length: 543744 target oid smithi15736696-24 low_tier offset: 897024 snap -1 2024-09-16T03:37:22.541 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.542 INFO:tasks.rados.rados.0.smithi157.stdout:415: finishing set_chunk to oid 16 2024-09-16T03:37:22.542 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 580608 length: 504832 2024-09-16T03:37:22.542 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:37:22.542 INFO:tasks.rados.rados.0.smithi157.stdout:415: done (15 left) 2024-09-16T03:37:22.542 INFO:tasks.rados.rados.0.smithi157.stdout:424: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.542 INFO:tasks.rados.rados.0.smithi157.stdout:424: set_chunk oid 25 offset: 504832 length: 204800 target oid smithi15736696-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 504832 snap -1 2024-09-16T03:37:22.542 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.542 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:22.543 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:22.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 reset_purged_snaps_last 2024-09-16T03:37:22.551 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:22.551 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:22.567 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:22.568 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:22.578 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:22.578 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:22.584 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:22.584 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:22.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:37:22.601 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:22.602 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:22.605 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:22.604+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:22.618 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:22.618 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:22.633 INFO:tasks.rados.rados.0.smithi157.stdout:400: finishing set_chunk to oid 1 2024-09-16T03:37:22.634 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 466944 length: 947200 2024-09-16T03:37:22.634 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T03:37:22.634 INFO:tasks.rados.rados.0.smithi157.stdout:400: done (15 left) 2024-09-16T03:37:22.634 INFO:tasks.rados.rados.0.smithi157.stdout:425: set_chunk oid 26 target oid 26 low_tier 2024-09-16T03:37:22.634 INFO:tasks.rados.rados.0.smithi157.stdout:425: set_chunk oid 26 offset: 9216 length: 1729536 target oid smithi15736696-26 low_tier offset: 9216 snap -1 2024-09-16T03:37:22.634 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.634 INFO:tasks.rados.rados.0.smithi157.stdout:401: finishing set_chunk to oid 2 2024-09-16T03:37:22.634 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 537088 length: 1444864 2024-09-16T03:37:22.634 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T03:37:22.634 INFO:tasks.rados.rados.0.smithi157.stdout:401: done (15 left) 2024-09-16T03:37:22.634 INFO:tasks.rados.rados.0.smithi157.stdout:426: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.634 INFO:tasks.rados.rados.0.smithi157.stdout:426: set_chunk oid 27 offset: 320000 length: 23040 target oid smithi15736696-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 320000 snap -1 2024-09-16T03:37:22.635 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.635 INFO:tasks.rados.rados.0.smithi157.stdout:413: finishing set_chunk to oid 14 2024-09-16T03:37:22.635 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 626688 length: 1476608 2024-09-16T03:37:22.635 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T03:37:22.635 INFO:tasks.rados.rados.0.smithi157.stdout:413: done (15 left) 2024-09-16T03:37:22.635 INFO:tasks.rados.rados.0.smithi157.stdout:427: set_chunk oid 28 target oid 28 low_tier 2024-09-16T03:37:22.635 INFO:tasks.rados.rados.0.smithi157.stdout:427: set_chunk oid 28 offset: 918528 length: 58368 target oid smithi15736696-28 low_tier offset: 918528 snap -1 2024-09-16T03:37:22.635 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.635 INFO:tasks.rados.rados.0.smithi157.stdout:418: finishing set_chunk to oid 19 2024-09-16T03:37:22.635 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 26624 length: 2191360 2024-09-16T03:37:22.635 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T03:37:22.635 INFO:tasks.rados.rados.0.smithi157.stdout:418: done (15 left) 2024-09-16T03:37:22.635 INFO:tasks.rados.rados.0.smithi157.stdout:428: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout:428: set_chunk oid 29 offset: 981504 length: 1398784 target oid smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 981504 snap -1 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout:423: finishing set_chunk to oid 24 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 897024 length: 543744 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout:423: done (15 left) 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout:429: set_chunk oid 30 target oid 30 low_tier 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout:429: set_chunk oid 30 offset: 96768 length: 1764864 target oid smithi15736696-30 low_tier offset: 96768 snap -1 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout:424: finishing set_chunk to oid 25 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 504832 length: 204800 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout:424: done (15 left) 2024-09-16T03:37:22.636 INFO:tasks.rados.rados.0.smithi157.stdout:430: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.637 INFO:tasks.rados.rados.0.smithi157.stdout:430: set_chunk oid 31 offset: 114176 length: 555008 target oid smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 114176 snap -1 2024-09-16T03:37:22.637 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.637 INFO:tasks.rados.rados.0.smithi157.stdout:428: finishing set_chunk to oid 29 2024-09-16T03:37:22.637 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 981504 length: 1398784 2024-09-16T03:37:22.637 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:37:22.637 INFO:tasks.rados.rados.0.smithi157.stdout:428: done (15 left) 2024-09-16T03:37:22.637 INFO:tasks.rados.rados.0.smithi157.stdout:431: set_chunk oid 32 target oid 32 low_tier 2024-09-16T03:37:22.637 INFO:tasks.rados.rados.0.smithi157.stdout:431: set_chunk oid 32 offset: 124928 length: 2165248 target oid smithi15736696-32 low_tier offset: 124928 snap -1 2024-09-16T03:37:22.637 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.637 INFO:tasks.rados.rados.0.smithi157.stdout:430: finishing set_chunk to oid 31 2024-09-16T03:37:22.637 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 114176 length: 555008 2024-09-16T03:37:22.637 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:37:22.637 INFO:tasks.rados.rados.0.smithi157.stdout:430: done (15 left) 2024-09-16T03:37:22.638 INFO:tasks.rados.rados.0.smithi157.stdout:432: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.638 INFO:tasks.rados.rados.0.smithi157.stdout:432: set_chunk oid 33 offset: 494592 length: 915456 target oid smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 494592 snap -1 2024-09-16T03:37:22.638 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.639 INFO:tasks.rados.rados.0.smithi157.stdout:431: finishing set_chunk to oid 32 2024-09-16T03:37:22.639 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 124928 length: 2165248 2024-09-16T03:37:22.639 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:37:22.639 INFO:tasks.rados.rados.0.smithi157.stdout:431: done (15 left) 2024-09-16T03:37:22.639 INFO:tasks.rados.rados.0.smithi157.stdout:433: set_chunk oid 34 target oid 34 low_tier 2024-09-16T03:37:22.639 INFO:tasks.rados.rados.0.smithi157.stdout:433: set_chunk oid 34 offset: 619520 length: 708608 target oid smithi15736696-34 low_tier offset: 619520 snap -1 2024-09-16T03:37:22.639 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.639 INFO:tasks.rados.rados.0.smithi157.stdout:432: finishing set_chunk to oid 33 2024-09-16T03:37:22.639 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 494592 length: 915456 2024-09-16T03:37:22.639 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:37:22.639 INFO:tasks.rados.rados.0.smithi157.stdout:432: done (15 left) 2024-09-16T03:37:22.639 INFO:tasks.rados.rados.0.smithi157.stdout:434: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.639 INFO:tasks.rados.rados.0.smithi157.stdout:434: set_chunk oid 35 offset: 1147392 length: 1330176 target oid smithi15736696-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1147392 snap -1 2024-09-16T03:37:22.640 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.640 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:22.640 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:22.690 INFO:tasks.rados.rados.0.smithi157.stdout:405: finishing set_chunk to oid 6 2024-09-16T03:37:22.691 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 511488 length: 1139712 2024-09-16T03:37:22.691 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:37:22.691 INFO:tasks.rados.rados.0.smithi157.stdout:406: finishing set_chunk to oid 7 2024-09-16T03:37:22.691 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1522688 length: 484352 2024-09-16T03:37:22.691 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T03:37:22.691 INFO:tasks.rados.rados.0.smithi157.stdout:422: finishing set_chunk to oid 23 2024-09-16T03:37:22.691 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 885248 length: 96768 2024-09-16T03:37:22.691 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T03:37:22.691 INFO:tasks.rados.rados.0.smithi157.stdout:405: done (15 left) 2024-09-16T03:37:22.691 INFO:tasks.rados.rados.0.smithi157.stdout:406: done (14 left) 2024-09-16T03:37:22.691 INFO:tasks.rados.rados.0.smithi157.stdout:422: done (13 left) 2024-09-16T03:37:22.691 INFO:tasks.rados.rados.0.smithi157.stdout:435: set_chunk oid 36 target oid 36 low_tier 2024-09-16T03:37:22.691 INFO:tasks.rados.rados.0.smithi157.stdout:435: set_chunk oid 36 offset: 101376 length: 2413568 target oid smithi15736696-36 low_tier offset: 101376 snap -1 2024-09-16T03:37:22.692 INFO:tasks.rados.rados.0.smithi157.stdout:436: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.692 INFO:tasks.rados.rados.0.smithi157.stdout:436: set_chunk oid 37 offset: 2405888 length: 182784 target oid smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 2405888 snap -1 2024-09-16T03:37:22.692 INFO:tasks.rados.rados.0.smithi157.stdout:437: set_chunk oid 38 target oid 38 low_tier 2024-09-16T03:37:22.692 INFO:tasks.rados.rados.0.smithi157.stdout:437: set_chunk oid 38 offset: 2305536 length: 238592 target oid smithi15736696-38 low_tier offset: 2305536 snap -1 2024-09-16T03:37:22.692 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.692 INFO:tasks.rados.rados.0.smithi157.stdout:412: finishing set_chunk to oid 13 2024-09-16T03:37:22.692 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 501248 length: 490496 2024-09-16T03:37:22.692 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T03:37:22.692 INFO:tasks.rados.rados.0.smithi157.stdout:412: done (15 left) 2024-09-16T03:37:22.692 INFO:tasks.rados.rados.0.smithi157.stdout:438: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:22.692 INFO:tasks.rados.rados.0.smithi157.stdout:438: set_chunk oid 39 offset: 708608 length: 1688064 target oid smithi15736696-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 708608 snap -1 2024-09-16T03:37:22.692 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:22.692 INFO:tasks.rados.rados.0.smithi157.stdout:419: finishing set_chunk to oid 20 2024-09-16T03:37:22.693 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 358400 length: 1290752 2024-09-16T03:37:22.693 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T03:37:22.693 INFO:tasks.rados.rados.0.smithi157.stdout:419: done (15 left) 2024-09-16T03:37:22.693 INFO:tasks.rados.rados.0.smithi157.stdout:438 retry set_chunk ! 2024-09-16T03:37:22.693 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 15 2024-09-16T03:37:22.693 INFO:tasks.rados.rados.0.smithi157.stdout:433: finishing set_chunk to oid 34 2024-09-16T03:37:22.693 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 619520 length: 708608 2024-09-16T03:37:22.693 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T03:37:22.693 INFO:tasks.rados.rados.0.smithi157.stdout:433: done (14 left) 2024-09-16T03:37:22.693 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 14 2024-09-16T03:37:22.694 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 reset_purged_snaps_last 2024-09-16T03:37:22.707 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:22.705+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:22.711 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:22.711 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:22.720 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:37:22.731 INFO:tasks.rados.rados.0.smithi157.stdout:435: finishing set_chunk to oid 36 2024-09-16T03:37:22.731 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 101376 length: 2413568 2024-09-16T03:37:22.731 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:37:22.731 INFO:tasks.rados.rados.0.smithi157.stdout:435: done (13 left) 2024-09-16T03:37:22.731 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 13 2024-09-16T03:37:22.732 INFO:tasks.rados.rados.0.smithi157.stdout:389: finishing set_chunk to oid 40 2024-09-16T03:37:22.732 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 15 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T03:37:22.732 INFO:tasks.rados.rados.0.smithi157.stdout:389: done (12 left) 2024-09-16T03:37:22.732 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 12 2024-09-16T03:37:22.749 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:22.749 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:22.789 INFO:tasks.rados.rados.0.smithi157.stdout:429: finishing set_chunk to oid 30 2024-09-16T03:37:22.790 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 96768 length: 1764864 2024-09-16T03:37:22.790 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:37:22.790 INFO:tasks.rados.rados.0.smithi157.stdout:429: done (11 left) 2024-09-16T03:37:22.790 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 11 2024-09-16T03:37:22.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:22.806+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:22.832 INFO:tasks.rados.rados.0.smithi157.stdout:425: finishing set_chunk to oid 26 2024-09-16T03:37:22.832 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 9216 length: 1729536 2024-09-16T03:37:22.832 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:37:22.832 INFO:tasks.rados.rados.0.smithi157.stdout:425: done (10 left) 2024-09-16T03:37:22.832 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 10 2024-09-16T03:37:22.833 INFO:tasks.rados.rados.0.smithi157.stdout:434: finishing set_chunk to oid 35 2024-09-16T03:37:22.833 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1147392 length: 1330176 2024-09-16T03:37:22.833 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:37:22.833 INFO:tasks.rados.rados.0.smithi157.stdout:434: done (9 left) 2024-09-16T03:37:22.833 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 9 2024-09-16T03:37:22.833 INFO:tasks.rados.rados.0.smithi157.stdout:438: finishing set_chunk to oid 39 2024-09-16T03:37:22.833 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 708608 length: 1688064 2024-09-16T03:37:22.834 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T03:37:22.834 INFO:tasks.rados.rados.0.smithi157.stdout:438: done (8 left) 2024-09-16T03:37:22.834 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 8 2024-09-16T03:37:22.834 INFO:tasks.rados.rados.0.smithi157.stdout:421: finishing set_chunk to oid 22 2024-09-16T03:37:22.834 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T03:37:22.834 INFO:tasks.rados.rados.0.smithi157.stdout:421: done (7 left) 2024-09-16T03:37:22.834 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 7 2024-09-16T03:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:22.859 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:22.868 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:37:22.908 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:22.907+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:23.009 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:23.008+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:23.016 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:23.016 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:23.025 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:37:23.034 INFO:tasks.rados.rados.0.smithi157.stdout:426: finishing set_chunk to oid 27 2024-09-16T03:37:23.034 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:23.034 INFO:tasks.rados.rados.0.smithi157.stdout:426: done (6 left) 2024-09-16T03:37:23.034 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 6 2024-09-16T03:37:23.035 INFO:tasks.rados.rados.0.smithi157.stdout:417: finishing set_chunk to oid 18 2024-09-16T03:37:23.035 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T03:37:23.035 INFO:tasks.rados.rados.0.smithi157.stdout:417: done (5 left) 2024-09-16T03:37:23.036 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 5 2024-09-16T03:37:23.036 INFO:tasks.rados.rados.0.smithi157.stdout:437: finishing set_chunk to oid 38 2024-09-16T03:37:23.036 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:37:23.036 INFO:tasks.rados.rados.0.smithi157.stdout:437: done (4 left) 2024-09-16T03:37:23.036 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 4 2024-09-16T03:37:23.036 INFO:tasks.rados.rados.0.smithi157.stdout:416: finishing set_chunk to oid 17 2024-09-16T03:37:23.036 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T03:37:23.036 INFO:tasks.rados.rados.0.smithi157.stdout:416: done (3 left) 2024-09-16T03:37:23.036 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 3 2024-09-16T03:37:23.036 INFO:tasks.rados.rados.0.smithi157.stdout:427: finishing set_chunk to oid 28 2024-09-16T03:37:23.036 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 3 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T03:37:23.036 INFO:tasks.rados.rados.0.smithi157.stdout:427: done (2 left) 2024-09-16T03:37:23.036 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 2 2024-09-16T03:37:23.111 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:23.109+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:23.133 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:23.133 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:23.134 INFO:tasks.rados.rados.0.smithi157.stdout:420: finishing set_chunk to oid 21 2024-09-16T03:37:23.134 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T03:37:23.134 INFO:tasks.rados.rados.0.smithi157.stdout:420: done (1 left) 2024-09-16T03:37:23.134 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 1 2024-09-16T03:37:23.142 INFO:tasks.rados.rados.0.smithi157.stdout:436: finishing set_chunk to oid 37 2024-09-16T03:37:23.142 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T03:37:23.142 INFO:tasks.rados.rados.0.smithi157.stdout:436: done (0 left) 2024-09-16T03:37:23.142 INFO:tasks.rados.rados.0.smithi157.stdout:439: set_chunk oid 40 target oid 40 low_tier 2024-09-16T03:37:23.142 INFO:tasks.rados.rados.0.smithi157.stdout:439: set_chunk oid 40 offset: 1896448 length: 653824 target oid smithi15736696-40 low_tier offset: 1896448 snap -1 2024-09-16T03:37:23.142 INFO:tasks.rados.rados.0.smithi157.stdout:440: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.143 INFO:tasks.rados.rados.0.smithi157.stdout:440: set_chunk oid 41 offset: 367616 length: 1289728 target oid smithi15736696-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 367616 snap -1 2024-09-16T03:37:23.143 INFO:tasks.rados.rados.0.smithi157.stdout:441: set_chunk oid 42 target oid 42 low_tier 2024-09-16T03:37:23.143 INFO:tasks.rados.rados.0.smithi157.stdout:441: set_chunk oid 42 offset: 254464 length: 1257472 target oid smithi15736696-42 low_tier offset: 254464 snap -1 2024-09-16T03:37:23.143 INFO:tasks.rados.rados.0.smithi157.stdout:442: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.143 INFO:tasks.rados.rados.0.smithi157.stdout:442: set_chunk oid 43 offset: 315392 length: 1780224 target oid smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 315392 snap -1 2024-09-16T03:37:23.143 INFO:tasks.rados.rados.0.smithi157.stdout:443: set_chunk oid 44 target oid 44 low_tier 2024-09-16T03:37:23.143 INFO:tasks.rados.rados.0.smithi157.stdout:443: set_chunk oid 44 offset: 314368 length: 2057216 target oid smithi15736696-44 low_tier offset: 314368 snap -1 2024-09-16T03:37:23.143 INFO:tasks.rados.rados.0.smithi157.stdout:444: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.143 INFO:tasks.rados.rados.0.smithi157.stdout:444: set_chunk oid 45 offset: 479744 length: 1887232 target oid smithi15736696-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 479744 snap -1 2024-09-16T03:37:23.143 INFO:tasks.rados.rados.0.smithi157.stdout:445: set_chunk oid 46 target oid 46 low_tier 2024-09-16T03:37:23.143 INFO:tasks.rados.rados.0.smithi157.stdout:445: set_chunk oid 46 offset: 104448 length: 1995776 target oid smithi15736696-46 low_tier offset: 104448 snap -1 2024-09-16T03:37:23.143 INFO:tasks.rados.rados.0.smithi157.stdout:446: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.143 INFO:tasks.rados.rados.0.smithi157.stdout:446: set_chunk oid 47 offset: 663040 length: 1270784 target oid smithi15736696-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 663040 snap -1 2024-09-16T03:37:23.144 INFO:tasks.rados.rados.0.smithi157.stdout:447: set_chunk oid 48 target oid 48 low_tier 2024-09-16T03:37:23.144 INFO:tasks.rados.rados.0.smithi157.stdout:447: set_chunk oid 48 offset: 985600 length: 763392 target oid smithi15736696-48 low_tier offset: 985600 snap -1 2024-09-16T03:37:23.144 INFO:tasks.rados.rados.0.smithi157.stdout:448: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.144 INFO:tasks.rados.rados.0.smithi157.stdout:448: set_chunk oid 49 offset: 689152 length: 860672 target oid smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 689152 snap -1 2024-09-16T03:37:23.144 INFO:tasks.rados.rados.0.smithi157.stdout:449: set_chunk oid 50 target oid 50 low_tier 2024-09-16T03:37:23.144 INFO:tasks.rados.rados.0.smithi157.stdout:449: set_chunk oid 50 offset: 233984 length: 22016 target oid smithi15736696-50 low_tier offset: 233984 snap -1 2024-09-16T03:37:23.144 INFO:tasks.rados.rados.0.smithi157.stdout:450: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.144 INFO:tasks.rados.rados.0.smithi157.stdout:450: set_chunk oid 1 offset: 1253376 length: 628224 target oid smithi15736696-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1253376 snap -1 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout:451: set_chunk oid 2 target oid 2 low_tier 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout:451: set_chunk oid 2 offset: 1024 length: 1159680 target oid smithi15736696-2 low_tier offset: 1024 snap -1 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout:452: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout:452: set_chunk oid 3 offset: 276992 length: 639488 target oid smithi15736696-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 276992 snap -1 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout:453: set_chunk oid 4 target oid 4 low_tier 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout:453: set_chunk oid 4 offset: 1127936 length: 419840 target oid smithi15736696-4 low_tier offset: 1127936 snap -1 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout:454: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout:454: set_chunk oid 5 offset: 1125376 length: 932352 target oid smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1125376 snap -1 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout:441: finishing set_chunk to oid 42 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 254464 length: 1257472 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout:441: done (15 left) 2024-09-16T03:37:23.145 INFO:tasks.rados.rados.0.smithi157.stdout:455: set_chunk oid 6 target oid 6 low_tier 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout:455: set_chunk oid 6 offset: 782848 length: 654848 target oid smithi15736696-6 low_tier offset: 782848 snap -1 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout:442: finishing set_chunk to oid 43 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 315392 length: 1780224 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout:445: finishing set_chunk to oid 46 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 104448 length: 1995776 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout:442: done (15 left) 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout:445: done (14 left) 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout:456: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout:456: set_chunk oid 7 offset: 309760 length: 1470976 target oid smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 309760 snap -1 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout:457: set_chunk oid 8 target oid 8 low_tier 2024-09-16T03:37:23.146 INFO:tasks.rados.rados.0.smithi157.stdout:457: set_chunk oid 8 offset: 43008 length: 1449472 target oid smithi15736696-8 low_tier offset: 43008 snap -1 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout:450: finishing set_chunk to oid 1 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1253376 length: 628224 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout:450: done (15 left) 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout:458: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout:458: set_chunk oid 9 offset: 495104 length: 1247232 target oid smithi15736696-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 495104 snap -1 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout:453: finishing set_chunk to oid 4 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 1127936 length: 419840 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout:453: done (15 left) 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout:459: set_chunk oid 10 target oid 10 low_tier 2024-09-16T03:37:23.147 INFO:tasks.rados.rados.0.smithi157.stdout:459: set_chunk oid 10 offset: 676864 length: 800768 target oid smithi15736696-10 low_tier offset: 676864 snap -1 2024-09-16T03:37:23.148 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.148 INFO:tasks.rados.rados.0.smithi157.stdout:446: finishing set_chunk to oid 47 2024-09-16T03:37:23.148 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 663040 length: 1270784 2024-09-16T03:37:23.148 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T03:37:23.148 INFO:tasks.rados.rados.0.smithi157.stdout:446: done (15 left) 2024-09-16T03:37:23.148 INFO:tasks.rados.rados.0.smithi157.stdout:460: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.148 INFO:tasks.rados.rados.0.smithi157.stdout:460: set_chunk oid 11 offset: 133632 length: 856064 target oid smithi15736696-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 133632 snap -1 2024-09-16T03:37:23.148 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.148 INFO:tasks.rados.rados.0.smithi157.stdout:447: finishing set_chunk to oid 48 2024-09-16T03:37:23.148 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 985600 length: 763392 2024-09-16T03:37:23.148 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:37:23.148 INFO:tasks.rados.rados.0.smithi157.stdout:447: done (15 left) 2024-09-16T03:37:23.148 INFO:tasks.rados.rados.0.smithi157.stdout:461: set_chunk oid 12 target oid 12 low_tier 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout:461: set_chunk oid 12 offset: 1144832 length: 732672 target oid smithi15736696-12 low_tier offset: 1144832 snap -1 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout:452: finishing set_chunk to oid 3 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 276992 length: 639488 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout:452: done (15 left) 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout:462: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout:462: set_chunk oid 13 offset: 1054208 length: 593920 target oid smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1054208 snap -1 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout:456: finishing set_chunk to oid 7 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 309760 length: 1470976 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout:456: done (15 left) 2024-09-16T03:37:23.149 INFO:tasks.rados.rados.0.smithi157.stdout:463: set_chunk oid 14 target oid 14 low_tier 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout:463: set_chunk oid 14 offset: 884224 length: 1264128 target oid smithi15736696-14 low_tier offset: 884224 snap -1 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout:451: finishing set_chunk to oid 2 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 1024 length: 1159680 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout:451: done (15 left) 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout:464: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout:464: set_chunk oid 15 offset: 366592 length: 847360 target oid smithi15736696-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 366592 snap -1 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout:454: finishing set_chunk to oid 5 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1125376 length: 932352 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout:454: done (15 left) 2024-09-16T03:37:23.150 INFO:tasks.rados.rados.0.smithi157.stdout:465: set_chunk oid 16 target oid 16 low_tier 2024-09-16T03:37:23.151 INFO:tasks.rados.rados.0.smithi157.stdout:465: set_chunk oid 16 offset: 494592 length: 296960 target oid smithi15736696-16 low_tier offset: 494592 snap -1 2024-09-16T03:37:23.151 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.151 INFO:tasks.rados.rados.0.smithi157.stdout:460: finishing set_chunk to oid 11 2024-09-16T03:37:23.151 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 133632 length: 856064 2024-09-16T03:37:23.151 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T03:37:23.151 INFO:tasks.rados.rados.0.smithi157.stdout:460: done (15 left) 2024-09-16T03:37:23.151 INFO:tasks.rados.rados.0.smithi157.stdout:466: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.151 INFO:tasks.rados.rados.0.smithi157.stdout:466: set_chunk oid 17 offset: 1106944 length: 987648 target oid smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1106944 snap -1 2024-09-16T03:37:23.151 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.151 INFO:tasks.rados.rados.0.smithi157.stdout:465: finishing set_chunk to oid 16 2024-09-16T03:37:23.151 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 494592 length: 296960 2024-09-16T03:37:23.151 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:37:23.151 INFO:tasks.rados.rados.0.smithi157.stdout:465: done (15 left) 2024-09-16T03:37:23.152 INFO:tasks.rados.rados.0.smithi157.stdout:467: set_chunk oid 18 target oid 18 low_tier 2024-09-16T03:37:23.152 INFO:tasks.rados.rados.0.smithi157.stdout:467: set_chunk oid 18 offset: 227840 length: 1840128 target oid smithi15736696-18 low_tier offset: 227840 snap -1 2024-09-16T03:37:23.152 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:37:23.213 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:23.211+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:23.241 INFO:tasks.rados.rados.0.smithi157.stdout:439: finishing set_chunk to oid 40 2024-09-16T03:37:23.241 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 1896448 length: 653824 2024-09-16T03:37:23.241 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T03:37:23.241 INFO:tasks.rados.rados.0.smithi157.stdout:440: finishing set_chunk to oid 41 2024-09-16T03:37:23.241 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 367616 length: 1289728 2024-09-16T03:37:23.241 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:37:23.241 INFO:tasks.rados.rados.0.smithi157.stdout:448: finishing set_chunk to oid 49 2024-09-16T03:37:23.241 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 689152 length: 860672 2024-09-16T03:37:23.241 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T03:37:23.242 INFO:tasks.rados.rados.0.smithi157.stdout:443: finishing set_chunk to oid 44 2024-09-16T03:37:23.242 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 314368 length: 2057216 2024-09-16T03:37:23.242 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T03:37:23.242 INFO:tasks.rados.rados.0.smithi157.stdout:439: done (15 left) 2024-09-16T03:37:23.242 INFO:tasks.rados.rados.0.smithi157.stdout:440: done (14 left) 2024-09-16T03:37:23.242 INFO:tasks.rados.rados.0.smithi157.stdout:443: done (13 left) 2024-09-16T03:37:23.242 INFO:tasks.rados.rados.0.smithi157.stdout:448: done (12 left) 2024-09-16T03:37:23.242 INFO:tasks.rados.rados.0.smithi157.stdout:468: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.242 INFO:tasks.rados.rados.0.smithi157.stdout:468: set_chunk oid 19 offset: 929792 length: 1360896 target oid smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 929792 snap -1 2024-09-16T03:37:23.242 INFO:tasks.rados.rados.0.smithi157.stdout:469: set_chunk oid 20 target oid 20 low_tier 2024-09-16T03:37:23.242 INFO:tasks.rados.rados.0.smithi157.stdout:469: set_chunk oid 20 offset: 75264 length: 1689088 target oid smithi15736696-20 low_tier offset: 75264 snap -1 2024-09-16T03:37:23.242 INFO:tasks.rados.rados.0.smithi157.stdout:470: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.242 INFO:tasks.rados.rados.0.smithi157.stdout:470: set_chunk oid 21 offset: 417280 length: 339968 target oid smithi15736696-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 417280 snap -1 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout:471: set_chunk oid 22 target oid 22 low_tier 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout:471: set_chunk oid 22 offset: 550912 length: 1386496 target oid smithi15736696-22 low_tier offset: 550912 snap -1 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout:466: finishing set_chunk to oid 17 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1106944 length: 987648 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout:466: done (15 left) 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout:472: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout:472: set_chunk oid 23 offset: 221184 length: 1485312 target oid smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 221184 snap -1 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout:455: finishing set_chunk to oid 6 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 782848 length: 654848 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:37:23.243 INFO:tasks.rados.rados.0.smithi157.stdout:455: done (15 left) 2024-09-16T03:37:23.244 INFO:tasks.rados.rados.0.smithi157.stdout:473: set_chunk oid 24 target oid 24 low_tier 2024-09-16T03:37:23.244 INFO:tasks.rados.rados.0.smithi157.stdout:473: set_chunk oid 24 offset: 1111040 length: 813568 target oid smithi15736696-24 low_tier offset: 1111040 snap -1 2024-09-16T03:37:23.244 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.244 INFO:tasks.rados.rados.0.smithi157.stdout:468: finishing set_chunk to oid 19 2024-09-16T03:37:23.244 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 929792 length: 1360896 2024-09-16T03:37:23.244 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T03:37:23.244 INFO:tasks.rados.rados.0.smithi157.stdout:470: finishing set_chunk to oid 21 2024-09-16T03:37:23.244 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 417280 length: 339968 2024-09-16T03:37:23.245 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T03:37:23.245 INFO:tasks.rados.rados.0.smithi157.stdout:468: done (15 left) 2024-09-16T03:37:23.245 INFO:tasks.rados.rados.0.smithi157.stdout:470: done (14 left) 2024-09-16T03:37:23.245 INFO:tasks.rados.rados.0.smithi157.stdout:474: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.245 INFO:tasks.rados.rados.0.smithi157.stdout:474: set_chunk oid 25 offset: 1590784 length: 452096 target oid smithi15736696-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1590784 snap -1 2024-09-16T03:37:23.245 INFO:tasks.rados.rados.0.smithi157.stdout:475: set_chunk oid 26 target oid 26 low_tier 2024-09-16T03:37:23.245 INFO:tasks.rados.rados.0.smithi157.stdout:475: set_chunk oid 26 offset: 1226240 length: 720896 target oid smithi15736696-26 low_tier offset: 1226240 snap -1 2024-09-16T03:37:23.245 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.245 INFO:tasks.rados.rados.0.smithi157.stdout:462: finishing set_chunk to oid 13 2024-09-16T03:37:23.245 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1054208 length: 593920 2024-09-16T03:37:23.245 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T03:37:23.245 INFO:tasks.rados.rados.0.smithi157.stdout:462: done (15 left) 2024-09-16T03:37:23.246 INFO:tasks.rados.rados.0.smithi157.stdout:476: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.246 INFO:tasks.rados.rados.0.smithi157.stdout:476: set_chunk oid 27 offset: 732672 length: 1688576 target oid smithi15736696-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 732672 snap -1 2024-09-16T03:37:23.246 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.246 INFO:tasks.rados.rados.0.smithi157.stdout:472: finishing set_chunk to oid 23 2024-09-16T03:37:23.246 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 221184 length: 1485312 2024-09-16T03:37:23.246 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T03:37:23.246 INFO:tasks.rados.rados.0.smithi157.stdout:472: done (15 left) 2024-09-16T03:37:23.246 INFO:tasks.rados.rados.0.smithi157.stdout:477: set_chunk oid 28 target oid 28 low_tier 2024-09-16T03:37:23.246 INFO:tasks.rados.rados.0.smithi157.stdout:477: set_chunk oid 28 offset: 1397248 length: 124928 target oid smithi15736696-28 low_tier offset: 1397248 snap -1 2024-09-16T03:37:23.246 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.246 INFO:tasks.rados.rados.0.smithi157.stdout:469: finishing set_chunk to oid 20 2024-09-16T03:37:23.246 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 75264 length: 1689088 2024-09-16T03:37:23.246 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout:473: finishing set_chunk to oid 24 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 1111040 length: 813568 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout:469: done (15 left) 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout:473: done (14 left) 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout:478: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout:478: set_chunk oid 29 offset: 1078272 length: 214528 target oid smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1078272 snap -1 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout:479: set_chunk oid 30 target oid 30 low_tier 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout:479: set_chunk oid 30 offset: 885248 length: 740864 target oid smithi15736696-30 low_tier offset: 885248 snap -1 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout:464: finishing set_chunk to oid 15 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 366592 length: 847360 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:23.247 INFO:tasks.rados.rados.0.smithi157.stdout:464: done (15 left) 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout:480: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout:480: set_chunk oid 31 offset: 647680 length: 1817088 target oid smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 647680 snap -1 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout:476: finishing set_chunk to oid 27 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 732672 length: 1688576 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout:476: done (15 left) 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout:481: set_chunk oid 32 target oid 32 low_tier 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout:481: set_chunk oid 32 offset: 1072640 length: 806400 target oid smithi15736696-32 low_tier offset: 1072640 snap -1 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout:458: finishing set_chunk to oid 9 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 495104 length: 1247232 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:37:23.248 INFO:tasks.rados.rados.0.smithi157.stdout:458: done (15 left) 2024-09-16T03:37:23.249 INFO:tasks.rados.rados.0.smithi157.stdout:482: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.249 INFO:tasks.rados.rados.0.smithi157.stdout:482: set_chunk oid 33 offset: 182784 length: 269824 target oid smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 182784 snap -1 2024-09-16T03:37:23.249 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.260 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:23.260 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:23.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:37:23.342 INFO:tasks.rados.rados.0.smithi157.stdout:461: finishing set_chunk to oid 12 2024-09-16T03:37:23.343 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1144832 length: 732672 2024-09-16T03:37:23.343 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:37:23.343 INFO:tasks.rados.rados.0.smithi157.stdout:480: finishing set_chunk to oid 31 2024-09-16T03:37:23.343 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 647680 length: 1817088 2024-09-16T03:37:23.343 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:37:23.343 INFO:tasks.rados.rados.0.smithi157.stdout:444: finishing set_chunk to oid 45 2024-09-16T03:37:23.343 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 479744 length: 1887232 2024-09-16T03:37:23.343 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T03:37:23.343 INFO:tasks.rados.rados.0.smithi157.stdout:457: finishing set_chunk to oid 8 2024-09-16T03:37:23.343 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 43008 length: 1449472 2024-09-16T03:37:23.343 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:37:23.343 INFO:tasks.rados.rados.0.smithi157.stdout:467: finishing set_chunk to oid 18 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 227840 length: 1840128 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:463: finishing set_chunk to oid 14 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 884224 length: 1264128 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:459: finishing set_chunk to oid 10 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 676864 length: 800768 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:444: done (15 left) 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:457: done (14 left) 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:459: done (13 left) 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:461: done (12 left) 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:463: done (11 left) 2024-09-16T03:37:23.344 INFO:tasks.rados.rados.0.smithi157.stdout:467: done (10 left) 2024-09-16T03:37:23.345 INFO:tasks.rados.rados.0.smithi157.stdout:480: done (9 left) 2024-09-16T03:37:23.345 INFO:tasks.rados.rados.0.smithi157.stdout:483: set_chunk oid 34 target oid 34 low_tier 2024-09-16T03:37:23.345 INFO:tasks.rados.rados.0.smithi157.stdout:483: set_chunk oid 34 offset: 9216 length: 592384 target oid smithi15736696-34 low_tier offset: 9216 snap -1 2024-09-16T03:37:23.345 INFO:tasks.rados.rados.0.smithi157.stdout:484: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.345 INFO:tasks.rados.rados.0.smithi157.stdout:484: set_chunk oid 35 offset: 913408 length: 1375232 target oid smithi15736696-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 913408 snap -1 2024-09-16T03:37:23.345 INFO:tasks.rados.rados.0.smithi157.stdout:485: set_chunk oid 36 target oid 36 low_tier 2024-09-16T03:37:23.345 INFO:tasks.rados.rados.0.smithi157.stdout:485: set_chunk oid 36 offset: 1021440 length: 118784 target oid smithi15736696-36 low_tier offset: 1021440 snap -1 2024-09-16T03:37:23.345 INFO:tasks.rados.rados.0.smithi157.stdout:486: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.345 INFO:tasks.rados.rados.0.smithi157.stdout:486: set_chunk oid 37 offset: 899584 length: 517120 target oid smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 899584 snap -1 2024-09-16T03:37:23.345 INFO:tasks.rados.rados.0.smithi157.stdout:487: set_chunk oid 38 target oid 38 low_tier 2024-09-16T03:37:23.345 INFO:tasks.rados.rados.0.smithi157.stdout:487: set_chunk oid 38 offset: 832512 length: 120832 target oid smithi15736696-38 low_tier offset: 832512 snap -1 2024-09-16T03:37:23.345 INFO:tasks.rados.rados.0.smithi157.stdout:488: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.345 INFO:tasks.rados.rados.0.smithi157.stdout:488: set_chunk oid 39 offset: 698880 length: 1675776 target oid smithi15736696-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 698880 snap -1 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout:489: set_chunk oid 40 target oid 40 low_tier 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout:489: set_chunk oid 40 offset: 615936 length: 634368 target oid smithi15736696-40 low_tier offset: 615936 snap -1 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout:474: finishing set_chunk to oid 25 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1590784 length: 452096 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout:471: finishing set_chunk to oid 22 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 550912 length: 1386496 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout:471: done (15 left) 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout:474: done (14 left) 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout:490: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout:490: set_chunk oid 41 offset: 761856 length: 716800 target oid smithi15736696-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 761856 snap -1 2024-09-16T03:37:23.346 INFO:tasks.rados.rados.0.smithi157.stdout:491: set_chunk oid 42 target oid 42 low_tier 2024-09-16T03:37:23.347 INFO:tasks.rados.rados.0.smithi157.stdout:491: set_chunk oid 42 offset: 40960 length: 2460160 target oid smithi15736696-42 low_tier offset: 40960 snap -1 2024-09-16T03:37:23.347 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.347 INFO:tasks.rados.rados.0.smithi157.stdout:475: finishing set_chunk to oid 26 2024-09-16T03:37:23.347 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1226240 length: 720896 2024-09-16T03:37:23.347 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:37:23.347 INFO:tasks.rados.rados.0.smithi157.stdout:475: done (15 left) 2024-09-16T03:37:23.347 INFO:tasks.rados.rados.0.smithi157.stdout:492: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.347 INFO:tasks.rados.rados.0.smithi157.stdout:492: set_chunk oid 43 offset: 169984 length: 613376 target oid smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 169984 snap -1 2024-09-16T03:37:23.347 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.347 INFO:tasks.rados.rados.0.smithi157.stdout:486: finishing set_chunk to oid 37 2024-09-16T03:37:23.347 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 899584 length: 517120 2024-09-16T03:37:23.347 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T03:37:23.347 INFO:tasks.rados.rados.0.smithi157.stdout:486: done (15 left) 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout:493: set_chunk oid 44 target oid 44 low_tier 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout:493: set_chunk oid 44 offset: 38400 length: 244736 target oid smithi15736696-44 low_tier offset: 38400 snap -1 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout:478: finishing set_chunk to oid 29 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1078272 length: 214528 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout:478: done (15 left) 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout:494: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout:494: set_chunk oid 45 offset: 937472 length: 226304 target oid smithi15736696-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 937472 snap -1 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout:485: finishing set_chunk to oid 36 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 1021440 length: 118784 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:37:23.348 INFO:tasks.rados.rados.0.smithi157.stdout:485: done (15 left) 2024-09-16T03:37:23.349 INFO:tasks.rados.rados.0.smithi157.stdout:495: set_chunk oid 46 target oid 46 low_tier 2024-09-16T03:37:23.349 INFO:tasks.rados.rados.0.smithi157.stdout:495: set_chunk oid 46 offset: 1845248 length: 308224 target oid smithi15736696-46 low_tier offset: 1845248 snap -1 2024-09-16T03:37:23.349 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.349 INFO:tasks.rados.rados.0.smithi157.stdout:489: finishing set_chunk to oid 40 2024-09-16T03:37:23.349 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 615936 length: 634368 2024-09-16T03:37:23.349 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T03:37:23.349 INFO:tasks.rados.rados.0.smithi157.stdout:489: done (15 left) 2024-09-16T03:37:23.349 INFO:tasks.rados.rados.0.smithi157.stdout:496: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.349 INFO:tasks.rados.rados.0.smithi157.stdout:496: set_chunk oid 47 offset: 1754112 length: 68608 target oid smithi15736696-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1754112 snap -1 2024-09-16T03:37:23.349 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.349 INFO:tasks.rados.rados.0.smithi157.stdout:494: finishing set_chunk to oid 45 2024-09-16T03:37:23.349 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 937472 length: 226304 2024-09-16T03:37:23.349 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout:494: done (15 left) 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout:497: set_chunk oid 48 target oid 48 low_tier 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout:497: set_chunk oid 48 offset: 353280 length: 2054656 target oid smithi15736696-48 low_tier offset: 353280 snap -1 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout:497: finishing set_chunk to oid 48 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 353280 length: 2054656 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout:497: done (15 left) 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout:498: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout:498: set_chunk oid 49 offset: 365568 length: 1596416 target oid smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 365568 snap -1 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout:482: finishing set_chunk to oid 33 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 182784 length: 269824 2024-09-16T03:37:23.350 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout:482: done (15 left) 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout:498 retry set_chunk ! 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 15 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout:490: finishing set_chunk to oid 41 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 761856 length: 716800 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout:490: done (14 left) 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 14 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout:488: finishing set_chunk to oid 39 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 698880 length: 1675776 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout:484: finishing set_chunk to oid 35 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 913408 length: 1375232 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:37:23.351 INFO:tasks.rados.rados.0.smithi157.stdout:484: done (13 left) 2024-09-16T03:37:23.352 INFO:tasks.rados.rados.0.smithi157.stdout:488: done (12 left) 2024-09-16T03:37:23.352 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 12 2024-09-16T03:37:23.352 INFO:tasks.rados.rados.0.smithi157.stdout:495: finishing set_chunk to oid 46 2024-09-16T03:37:23.352 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 1845248 length: 308224 2024-09-16T03:37:23.352 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:37:23.352 INFO:tasks.rados.rados.0.smithi157.stdout:495: done (11 left) 2024-09-16T03:37:23.352 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 11 2024-09-16T03:37:23.352 INFO:tasks.rados.rados.0.smithi157.stdout:492: finishing set_chunk to oid 43 2024-09-16T03:37:23.352 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 169984 length: 613376 2024-09-16T03:37:23.353 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T03:37:23.353 INFO:tasks.rados.rados.0.smithi157.stdout:492: done (10 left) 2024-09-16T03:37:23.353 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 10 2024-09-16T03:37:23.353 INFO:tasks.rados.rados.0.smithi157.stdout:496: finishing set_chunk to oid 47 2024-09-16T03:37:23.353 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1754112 length: 68608 2024-09-16T03:37:23.353 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T03:37:23.353 INFO:tasks.rados.rados.0.smithi157.stdout:496: done (9 left) 2024-09-16T03:37:23.353 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 9 2024-09-16T03:37:23.353 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:23.352+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:23.415 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:23.413+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:23.416 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:23.416 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:23.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:37:23.438 INFO:tasks.rados.rados.0.smithi157.stdout:498: finishing set_chunk to oid 49 2024-09-16T03:37:23.438 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 365568 length: 1596416 2024-09-16T03:37:23.438 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout:498: done (8 left) 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 8 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout:481: finishing set_chunk to oid 32 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 1072640 length: 806400 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout:481: done (7 left) 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 7 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout:487: finishing set_chunk to oid 38 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 832512 length: 120832 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout:487: done (6 left) 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 6 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout:491: finishing set_chunk to oid 42 2024-09-16T03:37:23.439 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 40960 length: 2460160 2024-09-16T03:37:23.440 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T03:37:23.440 INFO:tasks.rados.rados.0.smithi157.stdout:491: done (5 left) 2024-09-16T03:37:23.440 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 5 2024-09-16T03:37:23.516 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:23.514+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:23.540 INFO:tasks.rados.rados.0.smithi157.stdout:449: finishing set_chunk to oid 50 2024-09-16T03:37:23.540 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 18 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:37:23.540 INFO:tasks.rados.rados.0.smithi157.stdout:449: done (4 left) 2024-09-16T03:37:23.540 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 4 2024-09-16T03:37:23.541 INFO:tasks.rados.rados.0.smithi157.stdout:479: finishing set_chunk to oid 30 2024-09-16T03:37:23.541 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:37:23.541 INFO:tasks.rados.rados.0.smithi157.stdout:479: done (3 left) 2024-09-16T03:37:23.541 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 3 2024-09-16T03:37:23.542 INFO:tasks.rados.rados.0.smithi157.stdout:477: finishing set_chunk to oid 28 2024-09-16T03:37:23.543 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 3 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T03:37:23.543 INFO:tasks.rados.rados.0.smithi157.stdout:477: done (2 left) 2024-09-16T03:37:23.543 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 2 2024-09-16T03:37:23.543 INFO:tasks.rados.rados.0.smithi157.stdout:493: finishing set_chunk to oid 44 2024-09-16T03:37:23.543 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T03:37:23.543 INFO:tasks.rados.rados.0.smithi157.stdout:493: done (1 left) 2024-09-16T03:37:23.543 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 1 2024-09-16T03:37:23.573 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:23.573 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:23.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:37:23.617 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:23.615+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:23.646 INFO:tasks.rados.rados.0.smithi157.stdout:483: finishing set_chunk to oid 34 2024-09-16T03:37:23.646 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 15 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T03:37:23.646 INFO:tasks.rados.rados.0.smithi157.stdout:483: done (0 left) 2024-09-16T03:37:23.646 INFO:tasks.rados.rados.0.smithi157.stdout:499: set_chunk oid 50 target oid 50 low_tier 2024-09-16T03:37:23.646 INFO:tasks.rados.rados.0.smithi157.stdout:499: set_chunk oid 50 offset: 6656 length: 1547776 target oid smithi15736696-50 low_tier offset: 6656 snap -1 2024-09-16T03:37:23.646 INFO:tasks.rados.rados.0.smithi157.stdout:500: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.646 INFO:tasks.rados.rados.0.smithi157.stdout:500: set_chunk oid 1 offset: 860160 length: 639488 target oid smithi15736696-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 860160 snap -1 2024-09-16T03:37:23.646 INFO:tasks.rados.rados.0.smithi157.stdout:501: set_chunk oid 2 target oid 2 low_tier 2024-09-16T03:37:23.647 INFO:tasks.rados.rados.0.smithi157.stdout:501: set_chunk oid 2 offset: 108544 length: 1862656 target oid smithi15736696-2 low_tier offset: 108544 snap -1 2024-09-16T03:37:23.647 INFO:tasks.rados.rados.0.smithi157.stdout:502: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.647 INFO:tasks.rados.rados.0.smithi157.stdout:502: set_chunk oid 3 offset: 290304 length: 388096 target oid smithi15736696-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 290304 snap -1 2024-09-16T03:37:23.647 INFO:tasks.rados.rados.0.smithi157.stdout:503: set_chunk oid 4 target oid 4 low_tier 2024-09-16T03:37:23.647 INFO:tasks.rados.rados.0.smithi157.stdout:503: set_chunk oid 4 offset: 1450496 length: 322560 target oid smithi15736696-4 low_tier offset: 1450496 snap -1 2024-09-16T03:37:23.647 INFO:tasks.rados.rados.0.smithi157.stdout:504: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.647 INFO:tasks.rados.rados.0.smithi157.stdout:504: set_chunk oid 5 offset: 451072 length: 1498624 target oid smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 451072 snap -1 2024-09-16T03:37:23.647 INFO:tasks.rados.rados.0.smithi157.stdout:505: set_chunk oid 6 target oid 6 low_tier 2024-09-16T03:37:23.647 INFO:tasks.rados.rados.0.smithi157.stdout:505: set_chunk oid 6 offset: 184832 length: 515072 target oid smithi15736696-6 low_tier offset: 184832 snap -1 2024-09-16T03:37:23.647 INFO:tasks.rados.rados.0.smithi157.stdout:506: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.647 INFO:tasks.rados.rados.0.smithi157.stdout:506: set_chunk oid 7 offset: 692736 length: 289792 target oid smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 692736 snap -1 2024-09-16T03:37:23.647 INFO:tasks.rados.rados.0.smithi157.stdout:507: set_chunk oid 8 target oid 8 low_tier 2024-09-16T03:37:23.647 INFO:tasks.rados.rados.0.smithi157.stdout:507: set_chunk oid 8 offset: 648704 length: 204800 target oid smithi15736696-8 low_tier offset: 648704 snap -1 2024-09-16T03:37:23.648 INFO:tasks.rados.rados.0.smithi157.stdout:508: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.648 INFO:tasks.rados.rados.0.smithi157.stdout:508: set_chunk oid 9 offset: 200192 length: 339968 target oid smithi15736696-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 200192 snap -1 2024-09-16T03:37:23.648 INFO:tasks.rados.rados.0.smithi157.stdout:509: set_chunk oid 10 target oid 10 low_tier 2024-09-16T03:37:23.648 INFO:tasks.rados.rados.0.smithi157.stdout:509: set_chunk oid 10 offset: 47616 length: 360960 target oid smithi15736696-10 low_tier offset: 47616 snap -1 2024-09-16T03:37:23.648 INFO:tasks.rados.rados.0.smithi157.stdout:510: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.648 INFO:tasks.rados.rados.0.smithi157.stdout:510: set_chunk oid 11 offset: 403456 length: 43008 target oid smithi15736696-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 403456 snap -1 2024-09-16T03:37:23.648 INFO:tasks.rados.rados.0.smithi157.stdout:511: set_chunk oid 12 target oid 12 low_tier 2024-09-16T03:37:23.648 INFO:tasks.rados.rados.0.smithi157.stdout:511: set_chunk oid 12 offset: 11776 length: 1006592 target oid smithi15736696-12 low_tier offset: 11776 snap -1 2024-09-16T03:37:23.648 INFO:tasks.rados.rados.0.smithi157.stdout:512: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.648 INFO:tasks.rados.rados.0.smithi157.stdout:512: set_chunk oid 13 offset: 440832 length: 1727488 target oid smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 440832 snap -1 2024-09-16T03:37:23.648 INFO:tasks.rados.rados.0.smithi157.stdout:513: set_chunk oid 14 target oid 14 low_tier 2024-09-16T03:37:23.648 INFO:tasks.rados.rados.0.smithi157.stdout:513: set_chunk oid 14 offset: 1428992 length: 707584 target oid smithi15736696-14 low_tier offset: 1428992 snap -1 2024-09-16T03:37:23.649 INFO:tasks.rados.rados.0.smithi157.stdout:514: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.649 INFO:tasks.rados.rados.0.smithi157.stdout:514: set_chunk oid 15 offset: 1354752 length: 268800 target oid smithi15736696-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1354752 snap -1 2024-09-16T03:37:23.649 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.649 INFO:tasks.rados.rados.0.smithi157.stdout:501: finishing set_chunk to oid 2 2024-09-16T03:37:23.649 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 108544 length: 1862656 2024-09-16T03:37:23.649 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T03:37:23.649 INFO:tasks.rados.rados.0.smithi157.stdout:504: finishing set_chunk to oid 5 2024-09-16T03:37:23.649 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 451072 length: 1498624 2024-09-16T03:37:23.649 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T03:37:23.649 INFO:tasks.rados.rados.0.smithi157.stdout:501: done (15 left) 2024-09-16T03:37:23.649 INFO:tasks.rados.rados.0.smithi157.stdout:504: done (14 left) 2024-09-16T03:37:23.649 INFO:tasks.rados.rados.0.smithi157.stdout:515: set_chunk oid 16 target oid 16 low_tier 2024-09-16T03:37:23.650 INFO:tasks.rados.rados.0.smithi157.stdout:515: set_chunk oid 16 offset: 396800 length: 1521664 target oid smithi15736696-16 low_tier offset: 396800 snap -1 2024-09-16T03:37:23.650 INFO:tasks.rados.rados.0.smithi157.stdout:516: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.650 INFO:tasks.rados.rados.0.smithi157.stdout:516: set_chunk oid 17 offset: 645120 length: 290304 target oid smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 645120 snap -1 2024-09-16T03:37:23.650 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.650 INFO:tasks.rados.rados.0.smithi157.stdout:500: finishing set_chunk to oid 1 2024-09-16T03:37:23.650 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 860160 length: 639488 2024-09-16T03:37:23.650 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T03:37:23.650 INFO:tasks.rados.rados.0.smithi157.stdout:500: done (15 left) 2024-09-16T03:37:23.650 INFO:tasks.rados.rados.0.smithi157.stdout:517: set_chunk oid 18 target oid 18 low_tier 2024-09-16T03:37:23.650 INFO:tasks.rados.rados.0.smithi157.stdout:517: set_chunk oid 18 offset: 57856 length: 1333248 target oid smithi15736696-18 low_tier offset: 57856 snap -1 2024-09-16T03:37:23.650 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.650 INFO:tasks.rados.rados.0.smithi157.stdout:511: finishing set_chunk to oid 12 2024-09-16T03:37:23.650 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 11776 length: 1006592 2024-09-16T03:37:23.651 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:37:23.651 INFO:tasks.rados.rados.0.smithi157.stdout:511: done (15 left) 2024-09-16T03:37:23.651 INFO:tasks.rados.rados.0.smithi157.stdout:518: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.651 INFO:tasks.rados.rados.0.smithi157.stdout:518: set_chunk oid 19 offset: 918016 length: 485376 target oid smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 918016 snap -1 2024-09-16T03:37:23.651 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.651 INFO:tasks.rados.rados.0.smithi157.stdout:502: finishing set_chunk to oid 3 2024-09-16T03:37:23.651 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 290304 length: 388096 2024-09-16T03:37:23.651 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T03:37:23.651 INFO:tasks.rados.rados.0.smithi157.stdout:502: done (15 left) 2024-09-16T03:37:23.651 INFO:tasks.rados.rados.0.smithi157.stdout:519: set_chunk oid 20 target oid 20 low_tier 2024-09-16T03:37:23.651 INFO:tasks.rados.rados.0.smithi157.stdout:519: set_chunk oid 20 offset: 1026048 length: 1106944 target oid smithi15736696-20 low_tier offset: 1026048 snap -1 2024-09-16T03:37:23.651 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.651 INFO:tasks.rados.rados.0.smithi157.stdout:518: finishing set_chunk to oid 19 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 918016 length: 485376 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout:518: done (15 left) 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout:520: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout:520: set_chunk oid 21 offset: 210432 length: 1100800 target oid smithi15736696-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 210432 snap -1 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout:505: finishing set_chunk to oid 6 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 184832 length: 515072 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout:505: done (15 left) 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout:521: set_chunk oid 22 target oid 22 low_tier 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout:521: set_chunk oid 22 offset: 349696 length: 1127424 target oid smithi15736696-22 low_tier offset: 349696 snap -1 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.652 INFO:tasks.rados.rados.0.smithi157.stdout:512: finishing set_chunk to oid 13 2024-09-16T03:37:23.653 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 440832 length: 1727488 2024-09-16T03:37:23.653 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T03:37:23.653 INFO:tasks.rados.rados.0.smithi157.stdout:512: done (15 left) 2024-09-16T03:37:23.653 INFO:tasks.rados.rados.0.smithi157.stdout:522: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.653 INFO:tasks.rados.rados.0.smithi157.stdout:522: set_chunk oid 23 offset: 572416 length: 1593344 target oid smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 572416 snap -1 2024-09-16T03:37:23.653 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.653 INFO:tasks.rados.rados.0.smithi157.stdout:519: finishing set_chunk to oid 20 2024-09-16T03:37:23.653 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 1026048 length: 1106944 2024-09-16T03:37:23.653 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T03:37:23.653 INFO:tasks.rados.rados.0.smithi157.stdout:519: done (15 left) 2024-09-16T03:37:23.653 INFO:tasks.rados.rados.0.smithi157.stdout:523: set_chunk oid 24 target oid 24 low_tier 2024-09-16T03:37:23.653 INFO:tasks.rados.rados.0.smithi157.stdout:523: set_chunk oid 24 offset: 178176 length: 1040384 target oid smithi15736696-24 low_tier offset: 178176 snap -1 2024-09-16T03:37:23.653 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.654 INFO:tasks.rados.rados.0.smithi157.stdout:522: finishing set_chunk to oid 23 2024-09-16T03:37:23.654 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 572416 length: 1593344 2024-09-16T03:37:23.654 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T03:37:23.654 INFO:tasks.rados.rados.0.smithi157.stdout:522: done (15 left) 2024-09-16T03:37:23.654 INFO:tasks.rados.rados.0.smithi157.stdout:524: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.654 INFO:tasks.rados.rados.0.smithi157.stdout:524: set_chunk oid 25 offset: 867328 length: 566784 target oid smithi15736696-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 867328 snap -1 2024-09-16T03:37:23.654 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.689 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:23.689 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:23.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:37:23.717 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:23.716+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:23.746 INFO:tasks.rados.rados.0.smithi157.stdout:499: finishing set_chunk to oid 50 2024-09-16T03:37:23.746 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 6656 length: 1547776 2024-09-16T03:37:23.746 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:37:23.746 INFO:tasks.rados.rados.0.smithi157.stdout:507: finishing set_chunk to oid 8 2024-09-16T03:37:23.746 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 648704 length: 204800 2024-09-16T03:37:23.746 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:37:23.747 INFO:tasks.rados.rados.0.smithi157.stdout:513: finishing set_chunk to oid 14 2024-09-16T03:37:23.747 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 14 set_chunk 14 low_tier returned error code -95 offset: 1428992 length: 707584 2024-09-16T03:37:23.747 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 0 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T03:37:23.747 INFO:tasks.rados.rados.0.smithi157.stdout:509: finishing set_chunk to oid 10 2024-09-16T03:37:23.747 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 10 set_chunk 10 low_tier returned error code -95 offset: 47616 length: 360960 2024-09-16T03:37:23.747 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 0 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:37:23.747 INFO:tasks.rados.rados.0.smithi157.stdout:516: finishing set_chunk to oid 17 2024-09-16T03:37:23.747 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 17 set_chunk 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 645120 length: 290304 2024-09-16T03:37:23.747 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 0 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T03:37:23.747 INFO:tasks.rados.rados.0.smithi157.stdout:520: finishing set_chunk to oid 21 2024-09-16T03:37:23.747 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 210432 length: 1100800 2024-09-16T03:37:23.747 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T03:37:23.747 INFO:tasks.rados.rados.0.smithi157.stdout:514: finishing set_chunk to oid 15 2024-09-16T03:37:23.748 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1354752 length: 268800 2024-09-16T03:37:23.748 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:23.748 INFO:tasks.rados.rados.0.smithi157.stdout:521: finishing set_chunk to oid 22 2024-09-16T03:37:23.748 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 349696 length: 1127424 2024-09-16T03:37:23.748 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T03:37:23.748 INFO:tasks.rados.rados.0.smithi157.stdout:508: finishing set_chunk to oid 9 2024-09-16T03:37:23.748 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 200192 length: 339968 2024-09-16T03:37:23.748 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:37:23.748 INFO:tasks.rados.rados.0.smithi157.stdout:499: done (15 left) 2024-09-16T03:37:23.748 INFO:tasks.rados.rados.0.smithi157.stdout:507: done (14 left) 2024-09-16T03:37:23.748 INFO:tasks.rados.rados.0.smithi157.stdout:508: done (13 left) 2024-09-16T03:37:23.748 INFO:tasks.rados.rados.0.smithi157.stdout:509: done (12 left) 2024-09-16T03:37:23.748 INFO:tasks.rados.rados.0.smithi157.stdout:513: done (11 left) 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:514: done (10 left) 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:516: done (9 left) 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:520: done (8 left) 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:521: done (7 left) 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:525: set_chunk oid 26 target oid 26 low_tier 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:525: set_chunk oid 26 offset: 1281024 length: 299008 target oid smithi15736696-26 low_tier offset: 1281024 snap -1 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:526: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:526: set_chunk oid 27 offset: 1096704 length: 1070080 target oid smithi15736696-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1096704 snap -1 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:527: set_chunk oid 28 target oid 28 low_tier 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:527: set_chunk oid 28 offset: 1843200 length: 45568 target oid smithi15736696-28 low_tier offset: 1843200 snap -1 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:528: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:528: set_chunk oid 29 offset: 1313792 length: 879616 target oid smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1313792 snap -1 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:529: set_chunk oid 30 target oid 30 low_tier 2024-09-16T03:37:23.749 INFO:tasks.rados.rados.0.smithi157.stdout:529: set_chunk oid 30 offset: 915968 length: 217600 target oid smithi15736696-30 low_tier offset: 915968 snap -1 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:530: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:530: set_chunk oid 31 offset: 104960 length: 37376 target oid smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 104960 snap -1 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:531: set_chunk oid 32 target oid 32 low_tier 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:531: set_chunk oid 32 offset: 565760 length: 505856 target oid smithi15736696-32 low_tier offset: 565760 snap -1 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:532: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:532: set_chunk oid 33 offset: 295936 length: 1655296 target oid smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 295936 snap -1 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:533: set_chunk oid 34 target oid 34 low_tier 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:515: finishing set_chunk to oid 16 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 396800 length: 1521664 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:517: finishing set_chunk to oid 18 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 57856 length: 1333248 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T03:37:23.750 INFO:tasks.rados.rados.0.smithi157.stdout:533: set_chunk oid 34 offset: 1920000 length: 280576 target oid smithi15736696-34 low_tier offset: 1920000 snap -1 2024-09-16T03:37:23.751 INFO:tasks.rados.rados.0.smithi157.stdout:515: done (15 left) 2024-09-16T03:37:23.751 INFO:tasks.rados.rados.0.smithi157.stdout:517: done (14 left) 2024-09-16T03:37:23.751 INFO:tasks.rados.rados.0.smithi157.stdout:534: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.751 INFO:tasks.rados.rados.0.smithi157.stdout:534: set_chunk oid 35 offset: 950272 length: 1204736 target oid smithi15736696-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 950272 snap -1 2024-09-16T03:37:23.751 INFO:tasks.rados.rados.0.smithi157.stdout:535: set_chunk oid 36 target oid 36 low_tier 2024-09-16T03:37:23.751 INFO:tasks.rados.rados.0.smithi157.stdout:535: set_chunk oid 36 offset: 394752 length: 1797632 target oid smithi15736696-36 low_tier offset: 394752 snap -1 2024-09-16T03:37:23.751 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.751 INFO:tasks.rados.rados.0.smithi157.stdout:524: finishing set_chunk to oid 25 2024-09-16T03:37:23.752 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 867328 length: 566784 2024-09-16T03:37:23.752 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:23.752 INFO:tasks.rados.rados.0.smithi157.stdout:524: done (15 left) 2024-09-16T03:37:23.752 INFO:tasks.rados.rados.0.smithi157.stdout:536: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.752 INFO:tasks.rados.rados.0.smithi157.stdout:536: set_chunk oid 37 offset: 771584 length: 1401856 target oid smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 771584 snap -1 2024-09-16T03:37:23.752 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.752 INFO:tasks.rados.rados.0.smithi157.stdout:530: finishing set_chunk to oid 31 2024-09-16T03:37:23.752 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 104960 length: 37376 2024-09-16T03:37:23.752 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:37:23.752 INFO:tasks.rados.rados.0.smithi157.stdout:529: finishing set_chunk to oid 30 2024-09-16T03:37:23.752 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 915968 length: 217600 2024-09-16T03:37:23.752 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:37:23.752 INFO:tasks.rados.rados.0.smithi157.stdout:523: finishing set_chunk to oid 24 2024-09-16T03:37:23.753 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 178176 length: 1040384 2024-09-16T03:37:23.753 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T03:37:23.753 INFO:tasks.rados.rados.0.smithi157.stdout:523: done (15 left) 2024-09-16T03:37:23.753 INFO:tasks.rados.rados.0.smithi157.stdout:529: done (14 left) 2024-09-16T03:37:23.753 INFO:tasks.rados.rados.0.smithi157.stdout:530: done (13 left) 2024-09-16T03:37:23.753 INFO:tasks.rados.rados.0.smithi157.stdout:537: set_chunk oid 38 target oid 38 low_tier 2024-09-16T03:37:23.753 INFO:tasks.rados.rados.0.smithi157.stdout:537: set_chunk oid 38 offset: 2024960 length: 155136 target oid smithi15736696-38 low_tier offset: 2024960 snap -1 2024-09-16T03:37:23.753 INFO:tasks.rados.rados.0.smithi157.stdout:538: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.753 INFO:tasks.rados.rados.0.smithi157.stdout:538: set_chunk oid 39 offset: 1482240 length: 441344 target oid smithi15736696-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1482240 snap -1 2024-09-16T03:37:23.753 INFO:tasks.rados.rados.0.smithi157.stdout:539: set_chunk oid 40 target oid 40 low_tier 2024-09-16T03:37:23.753 INFO:tasks.rados.rados.0.smithi157.stdout:539: set_chunk oid 40 offset: 1825280 length: 768512 target oid smithi15736696-40 low_tier offset: 1825280 snap -1 2024-09-16T03:37:23.753 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.753 INFO:tasks.rados.rados.0.smithi157.stdout:532: finishing set_chunk to oid 33 2024-09-16T03:37:23.754 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 295936 length: 1655296 2024-09-16T03:37:23.754 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:37:23.754 INFO:tasks.rados.rados.0.smithi157.stdout:526: finishing set_chunk to oid 27 2024-09-16T03:37:23.754 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1096704 length: 1070080 2024-09-16T03:37:23.754 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:23.754 INFO:tasks.rados.rados.0.smithi157.stdout:531: finishing set_chunk to oid 32 2024-09-16T03:37:23.754 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 565760 length: 505856 2024-09-16T03:37:23.754 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:37:23.754 INFO:tasks.rados.rados.0.smithi157.stdout:534: finishing set_chunk to oid 35 2024-09-16T03:37:23.754 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 950272 length: 1204736 2024-09-16T03:37:23.754 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:37:23.755 INFO:tasks.rados.rados.0.smithi157.stdout:533: finishing set_chunk to oid 34 2024-09-16T03:37:23.755 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 1920000 length: 280576 2024-09-16T03:37:23.756 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T03:37:23.756 INFO:tasks.rados.rados.0.smithi157.stdout:538: finishing set_chunk to oid 39 2024-09-16T03:37:23.756 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1482240 length: 441344 2024-09-16T03:37:23.756 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T03:37:23.756 INFO:tasks.rados.rados.0.smithi157.stdout:526: done (15 left) 2024-09-16T03:37:23.756 INFO:tasks.rados.rados.0.smithi157.stdout:531: done (14 left) 2024-09-16T03:37:23.756 INFO:tasks.rados.rados.0.smithi157.stdout:532: done (13 left) 2024-09-16T03:37:23.756 INFO:tasks.rados.rados.0.smithi157.stdout:533: done (12 left) 2024-09-16T03:37:23.756 INFO:tasks.rados.rados.0.smithi157.stdout:534: done (11 left) 2024-09-16T03:37:23.756 INFO:tasks.rados.rados.0.smithi157.stdout:538: done (10 left) 2024-09-16T03:37:23.756 INFO:tasks.rados.rados.0.smithi157.stdout:540: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.756 INFO:tasks.rados.rados.0.smithi157.stdout:540: set_chunk oid 41 offset: 249344 length: 262144 target oid smithi15736696-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 249344 snap -1 2024-09-16T03:37:23.757 INFO:tasks.rados.rados.0.smithi157.stdout:541: set_chunk oid 42 target oid 42 low_tier 2024-09-16T03:37:23.757 INFO:tasks.rados.rados.0.smithi157.stdout:541: set_chunk oid 42 offset: 1589248 length: 169472 target oid smithi15736696-42 low_tier offset: 1589248 snap -1 2024-09-16T03:37:23.757 INFO:tasks.rados.rados.0.smithi157.stdout:542: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.757 INFO:tasks.rados.rados.0.smithi157.stdout:542: set_chunk oid 43 offset: 818688 length: 1377280 target oid smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 818688 snap -1 2024-09-16T03:37:23.757 INFO:tasks.rados.rados.0.smithi157.stdout:543: set_chunk oid 44 target oid 44 low_tier 2024-09-16T03:37:23.757 INFO:tasks.rados.rados.0.smithi157.stdout:543: set_chunk oid 44 offset: 1445888 length: 751104 target oid smithi15736696-44 low_tier offset: 1445888 snap -1 2024-09-16T03:37:23.757 INFO:tasks.rados.rados.0.smithi157.stdout:544: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.757 INFO:tasks.rados.rados.0.smithi157.stdout:544: set_chunk oid 45 offset: 1615872 length: 383488 target oid smithi15736696-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1615872 snap -1 2024-09-16T03:37:23.757 INFO:tasks.rados.rados.0.smithi157.stdout:545: set_chunk oid 46 target oid 46 low_tier 2024-09-16T03:37:23.757 INFO:tasks.rados.rados.0.smithi157.stdout:545: set_chunk oid 46 offset: 186880 length: 881664 target oid smithi15736696-46 low_tier offset: 186880 snap -1 2024-09-16T03:37:23.757 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.757 INFO:tasks.rados.rados.0.smithi157.stdout:535: finishing set_chunk to oid 36 2024-09-16T03:37:23.757 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 394752 length: 1797632 2024-09-16T03:37:23.758 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:37:23.758 INFO:tasks.rados.rados.0.smithi157.stdout:535: done (15 left) 2024-09-16T03:37:23.758 INFO:tasks.rados.rados.0.smithi157.stdout:546: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.758 INFO:tasks.rados.rados.0.smithi157.stdout:546: set_chunk oid 47 offset: 50688 length: 520192 target oid smithi15736696-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 50688 snap -1 2024-09-16T03:37:23.758 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.806 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:23.806 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:23.814 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:37:23.818 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:23.816+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:23.846 INFO:tasks.rados.rados.0.smithi157.stdout:543: finishing set_chunk to oid 44 2024-09-16T03:37:23.846 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 1445888 length: 751104 2024-09-16T03:37:23.846 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T03:37:23.846 INFO:tasks.rados.rados.0.smithi157.stdout:543: done (15 left) 2024-09-16T03:37:23.847 INFO:tasks.rados.rados.0.smithi157.stdout:547: set_chunk oid 48 target oid 48 low_tier 2024-09-16T03:37:23.847 INFO:tasks.rados.rados.0.smithi157.stdout:547: set_chunk oid 48 offset: 1953280 length: 677888 target oid smithi15736696-48 low_tier offset: 1953280 snap -1 2024-09-16T03:37:23.847 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.847 INFO:tasks.rados.rados.0.smithi157.stdout:545: finishing set_chunk to oid 46 2024-09-16T03:37:23.847 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 186880 length: 881664 2024-09-16T03:37:23.847 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:37:23.847 INFO:tasks.rados.rados.0.smithi157.stdout:545: done (15 left) 2024-09-16T03:37:23.847 INFO:tasks.rados.rados.0.smithi157.stdout:548: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.847 INFO:tasks.rados.rados.0.smithi157.stdout:548: set_chunk oid 49 offset: 1988096 length: 24064 target oid smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1988096 snap -1 2024-09-16T03:37:23.847 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.847 INFO:tasks.rados.rados.0.smithi157.stdout:547: finishing set_chunk to oid 48 2024-09-16T03:37:23.847 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 1953280 length: 677888 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout:547: done (15 left) 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout:549: set_chunk oid 50 target oid 50 low_tier 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout:549: set_chunk oid 50 offset: 1613312 length: 498688 target oid smithi15736696-50 low_tier offset: 1613312 snap -1 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout:546: finishing set_chunk to oid 47 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 50688 length: 520192 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout:546: done (15 left) 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout:550: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout:550: set_chunk oid 1 offset: 1046016 length: 729088 target oid smithi15736696-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1046016 snap -1 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout:550: finishing set_chunk to oid 1 2024-09-16T03:37:23.848 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1046016 length: 729088 2024-09-16T03:37:23.849 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T03:37:23.849 INFO:tasks.rados.rados.0.smithi157.stdout:550: done (15 left) 2024-09-16T03:37:23.849 INFO:tasks.rados.rados.0.smithi157.stdout:551: set_chunk oid 2 target oid 2 low_tier 2024-09-16T03:37:23.849 INFO:tasks.rados.rados.0.smithi157.stdout:551: set_chunk oid 2 offset: 977408 length: 82944 target oid smithi15736696-2 low_tier offset: 977408 snap -1 2024-09-16T03:37:23.849 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.849 INFO:tasks.rados.rados.0.smithi157.stdout:542: finishing set_chunk to oid 43 2024-09-16T03:37:23.849 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 818688 length: 1377280 2024-09-16T03:37:23.849 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T03:37:23.849 INFO:tasks.rados.rados.0.smithi157.stdout:542: done (15 left) 2024-09-16T03:37:23.849 INFO:tasks.rados.rados.0.smithi157.stdout:552: set_chunk oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:23.849 INFO:tasks.rados.rados.0.smithi157.stdout:552: set_chunk oid 3 offset: 1256448 length: 749056 target oid smithi15736696-3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1256448 snap -1 2024-09-16T03:37:23.849 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:23.849 INFO:tasks.rados.rados.0.smithi157.stdout:552: finishing set_chunk to oid 3 2024-09-16T03:37:23.850 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 3 set_chunk 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1256448 length: 749056 2024-09-16T03:37:23.850 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 0 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T03:37:23.850 INFO:tasks.rados.rados.0.smithi157.stdout:552: done (15 left) 2024-09-16T03:37:23.850 INFO:tasks.rados.rados.0.smithi157.stdout:552 retry set_chunk ! 2024-09-16T03:37:23.850 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 15 2024-09-16T03:37:23.919 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:23.917+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:23.931 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:23.931 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:23.940 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:37:23.943 INFO:tasks.rados.rados.0.smithi157.stdout:528: finishing set_chunk to oid 29 2024-09-16T03:37:23.943 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1313792 length: 879616 2024-09-16T03:37:23.944 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:37:23.944 INFO:tasks.rados.rados.0.smithi157.stdout:525: finishing set_chunk to oid 26 2024-09-16T03:37:23.944 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 1281024 length: 299008 2024-09-16T03:37:23.944 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:37:23.944 INFO:tasks.rados.rados.0.smithi157.stdout:525: done (14 left) 2024-09-16T03:37:23.944 INFO:tasks.rados.rados.0.smithi157.stdout:528: done (13 left) 2024-09-16T03:37:23.944 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 13 2024-09-16T03:37:23.944 INFO:tasks.rados.rados.0.smithi157.stdout:536: finishing set_chunk to oid 37 2024-09-16T03:37:23.944 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 771584 length: 1401856 2024-09-16T03:37:23.944 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T03:37:23.944 INFO:tasks.rados.rados.0.smithi157.stdout:536: done (12 left) 2024-09-16T03:37:23.944 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 12 2024-09-16T03:37:23.945 INFO:tasks.rados.rados.0.smithi157.stdout:510: finishing set_chunk to oid 11 2024-09-16T03:37:23.945 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T03:37:23.945 INFO:tasks.rados.rados.0.smithi157.stdout:510: done (11 left) 2024-09-16T03:37:23.945 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 11 2024-09-16T03:37:23.946 INFO:tasks.rados.rados.0.smithi157.stdout:541: finishing set_chunk to oid 42 2024-09-16T03:37:23.946 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 1589248 length: 169472 2024-09-16T03:37:23.946 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T03:37:23.946 INFO:tasks.rados.rados.0.smithi157.stdout:541: done (10 left) 2024-09-16T03:37:23.946 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 10 2024-09-16T03:37:23.946 INFO:tasks.rados.rados.0.smithi157.stdout:551: finishing set_chunk to oid 2 2024-09-16T03:37:23.946 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 2 set_chunk 2 low_tier returned error code -95 offset: 977408 length: 82944 2024-09-16T03:37:23.946 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T03:37:23.947 INFO:tasks.rados.rados.0.smithi157.stdout:551: done (9 left) 2024-09-16T03:37:23.947 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 9 2024-09-16T03:37:23.947 INFO:tasks.rados.rados.0.smithi157.stdout:537: finishing set_chunk to oid 38 2024-09-16T03:37:23.948 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:37:23.948 INFO:tasks.rados.rados.0.smithi157.stdout:537: done (8 left) 2024-09-16T03:37:23.948 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 8 2024-09-16T03:37:24.019 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:24.018+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:24.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:37:24.044 INFO:tasks.rados.rados.0.smithi157.stdout:539: finishing set_chunk to oid 40 2024-09-16T03:37:24.044 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 1825280 length: 768512 2024-09-16T03:37:24.044 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T03:37:24.044 INFO:tasks.rados.rados.0.smithi157.stdout:540: finishing set_chunk to oid 41 2024-09-16T03:37:24.044 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 249344 length: 262144 2024-09-16T03:37:24.044 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:37:24.044 INFO:tasks.rados.rados.0.smithi157.stdout:544: finishing set_chunk to oid 45 2024-09-16T03:37:24.044 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1615872 length: 383488 2024-09-16T03:37:24.044 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T03:37:24.045 INFO:tasks.rados.rados.0.smithi157.stdout:549: finishing set_chunk to oid 50 2024-09-16T03:37:24.045 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 1613312 length: 498688 2024-09-16T03:37:24.045 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:37:24.045 INFO:tasks.rados.rados.0.smithi157.stdout:548: finishing set_chunk to oid 49 2024-09-16T03:37:24.045 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1988096 length: 24064 2024-09-16T03:37:24.045 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T03:37:24.045 INFO:tasks.rados.rados.0.smithi157.stdout:539: done (7 left) 2024-09-16T03:37:24.045 INFO:tasks.rados.rados.0.smithi157.stdout:540: done (6 left) 2024-09-16T03:37:24.045 INFO:tasks.rados.rados.0.smithi157.stdout:544: done (5 left) 2024-09-16T03:37:24.045 INFO:tasks.rados.rados.0.smithi157.stdout:548: done (4 left) 2024-09-16T03:37:24.045 INFO:tasks.rados.rados.0.smithi157.stdout:549: done (3 left) 2024-09-16T03:37:24.045 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 3 2024-09-16T03:37:24.046 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:24.046 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:24.051 INFO:tasks.rados.rados.0.smithi157.stdout:506: finishing set_chunk to oid 7 2024-09-16T03:37:24.051 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T03:37:24.051 INFO:tasks.rados.rados.0.smithi157.stdout:503: finishing set_chunk to oid 4 2024-09-16T03:37:24.052 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:37:24.052 INFO:tasks.rados.rados.0.smithi157.stdout:503: done (2 left) 2024-09-16T03:37:24.052 INFO:tasks.rados.rados.0.smithi157.stdout:506: done (1 left) 2024-09-16T03:37:24.052 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 1 2024-09-16T03:37:24.052 INFO:tasks.rados.rados.0.smithi157.stdout:527: finishing set_chunk to oid 28 2024-09-16T03:37:24.052 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 3 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T03:37:24.052 INFO:tasks.rados.rados.0.smithi157.stdout:527: done (0 left) 2024-09-16T03:37:24.052 INFO:tasks.rados.rados.0.smithi157.stdout:553: set_chunk oid 4 target oid 4 low_tier 2024-09-16T03:37:24.052 INFO:tasks.rados.rados.0.smithi157.stdout:553: set_chunk oid 4 offset: 260608 length: 253440 target oid smithi15736696-4 low_tier offset: 260608 snap -1 2024-09-16T03:37:24.052 INFO:tasks.rados.rados.0.smithi157.stdout:554: set_chunk oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.052 INFO:tasks.rados.rados.0.smithi157.stdout:554: set_chunk oid 5 offset: 89600 length: 1742848 target oid smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 89600 snap -1 2024-09-16T03:37:24.052 INFO:tasks.rados.rados.0.smithi157.stdout:555: set_chunk oid 6 target oid 6 low_tier 2024-09-16T03:37:24.052 INFO:tasks.rados.rados.0.smithi157.stdout:555: set_chunk oid 6 offset: 1185280 length: 30208 target oid smithi15736696-6 low_tier offset: 1185280 snap -1 2024-09-16T03:37:24.053 INFO:tasks.rados.rados.0.smithi157.stdout:556: set_chunk oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.053 INFO:tasks.rados.rados.0.smithi157.stdout:556: set_chunk oid 7 offset: 669184 length: 400384 target oid smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 669184 snap -1 2024-09-16T03:37:24.053 INFO:tasks.rados.rados.0.smithi157.stdout:557: set_chunk oid 8 target oid 8 low_tier 2024-09-16T03:37:24.053 INFO:tasks.rados.rados.0.smithi157.stdout:557: set_chunk oid 8 offset: 173568 length: 645632 target oid smithi15736696-8 low_tier offset: 173568 snap -1 2024-09-16T03:37:24.053 INFO:tasks.rados.rados.0.smithi157.stdout:558: set_chunk oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.053 INFO:tasks.rados.rados.0.smithi157.stdout:558: set_chunk oid 9 offset: 232960 length: 1597440 target oid smithi15736696-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 232960 snap -1 2024-09-16T03:37:24.053 INFO:tasks.rados.rados.0.smithi157.stdout:559: set_chunk oid 10 target oid 10 low_tier 2024-09-16T03:37:24.053 INFO:tasks.rados.rados.0.smithi157.stdout:559: set_chunk oid 10 offset: 1931264 length: 68608 target oid smithi15736696-10 low_tier offset: 1931264 snap -1 2024-09-16T03:37:24.053 INFO:tasks.rados.rados.0.smithi157.stdout:560: set_chunk oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:560: set_chunk oid 11 offset: 796672 length: 494080 target oid smithi15736696-11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 796672 snap -1 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:561: set_chunk oid 12 target oid 12 low_tier 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:561: set_chunk oid 12 offset: 1060352 length: 392704 target oid smithi15736696-12 low_tier offset: 1060352 snap -1 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:562: set_chunk oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:562: set_chunk oid 13 offset: 411136 length: 722944 target oid smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 411136 snap -1 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:563: set_chunk oid 14 target oid 14 low_tier 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:563: set_chunk oid 14 offset: 541696 length: 131584 target oid smithi15736696-14 low_tier offset: 541696 snap -1 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:553: finishing set_chunk to oid 4 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 4 set_chunk 4 low_tier returned error code -95 offset: 260608 length: 253440 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 0 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:555: finishing set_chunk to oid 6 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 6 set_chunk 6 low_tier returned error code -95 offset: 1185280 length: 30208 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:37:24.054 INFO:tasks.rados.rados.0.smithi157.stdout:556: finishing set_chunk to oid 7 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 7 set_chunk 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 669184 length: 400384 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:553: done (10 left) 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:555: done (9 left) 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:556: done (8 left) 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:564: set_chunk oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:564: set_chunk oid 15 offset: 960000 length: 64000 target oid smithi15736696-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 960000 snap -1 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:565: set_chunk oid 16 target oid 16 low_tier 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:565: set_chunk oid 16 offset: 349184 length: 482816 target oid smithi15736696-16 low_tier offset: 349184 snap -1 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:566: set_chunk oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:566: set_chunk oid 17 offset: 1953280 length: 57856 target oid smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1953280 snap -1 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:567: set_chunk oid 18 target oid 18 low_tier 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:567: set_chunk oid 18 offset: 392192 length: 890880 target oid smithi15736696-18 low_tier offset: 392192 snap -1 2024-09-16T03:37:24.055 INFO:tasks.rados.rados.0.smithi157.stdout:568: set_chunk oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.056 INFO:tasks.rados.rados.0.smithi157.stdout:568: set_chunk oid 19 offset: 968192 length: 667136 target oid smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 968192 snap -1 2024-09-16T03:37:24.056 INFO:tasks.rados.rados.0.smithi157.stdout:569: set_chunk oid 20 target oid 20 low_tier 2024-09-16T03:37:24.056 INFO:tasks.rados.rados.0.smithi157.stdout:569: set_chunk oid 20 offset: 309760 length: 54784 target oid smithi15736696-20 low_tier offset: 309760 snap -1 2024-09-16T03:37:24.056 INFO:tasks.rados.rados.0.smithi157.stdout:570: set_chunk oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.056 INFO:tasks.rados.rados.0.smithi157.stdout:570: set_chunk oid 21 offset: 1265152 length: 1035264 target oid smithi15736696-21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1265152 snap -1 2024-09-16T03:37:24.056 INFO:tasks.rados.rados.0.smithi157.stdout:571: set_chunk oid 22 target oid 22 low_tier 2024-09-16T03:37:24.056 INFO:tasks.rados.rados.0.smithi157.stdout:571: set_chunk oid 22 offset: 726016 length: 947200 target oid smithi15736696-22 low_tier offset: 726016 snap -1 2024-09-16T03:37:24.056 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.056 INFO:tasks.rados.rados.0.smithi157.stdout:554: finishing set_chunk to oid 5 2024-09-16T03:37:24.056 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 5 set_chunk 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 89600 length: 1742848 2024-09-16T03:37:24.056 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T03:37:24.056 INFO:tasks.rados.rados.0.smithi157.stdout:557: finishing set_chunk to oid 8 2024-09-16T03:37:24.056 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 8 set_chunk 8 low_tier returned error code -95 offset: 173568 length: 645632 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:558: finishing set_chunk to oid 9 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 9 set_chunk 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 232960 length: 1597440 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:561: finishing set_chunk to oid 12 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 12 set_chunk 12 low_tier returned error code -95 offset: 1060352 length: 392704 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 0 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:554: done (15 left) 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:557: done (14 left) 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:558: done (13 left) 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:561: done (12 left) 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:572: set_chunk oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:572: set_chunk oid 23 offset: 818688 length: 879616 target oid smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 818688 snap -1 2024-09-16T03:37:24.057 INFO:tasks.rados.rados.0.smithi157.stdout:573: set_chunk oid 24 target oid 24 low_tier 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout:573: set_chunk oid 24 offset: 1235968 length: 402944 target oid smithi15736696-24 low_tier offset: 1235968 snap -1 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout:574: set_chunk oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout:574: set_chunk oid 25 offset: 186880 length: 1035264 target oid smithi15736696-25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 186880 snap -1 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout:575: set_chunk oid 26 target oid 26 low_tier 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout:575: set_chunk oid 26 offset: 287744 length: 1893888 target oid smithi15736696-26 low_tier offset: 287744 snap -1 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout:568: finishing set_chunk to oid 19 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 19 set_chunk 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 968192 length: 667136 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 0 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout:568: done (15 left) 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout:576: set_chunk oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout:576: set_chunk oid 27 offset: 637440 length: 621056 target oid smithi15736696-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 637440 snap -1 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.058 INFO:tasks.rados.rados.0.smithi157.stdout:573: finishing set_chunk to oid 24 2024-09-16T03:37:24.059 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 24 set_chunk 24 low_tier returned error code -95 offset: 1235968 length: 402944 2024-09-16T03:37:24.059 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T03:37:24.059 INFO:tasks.rados.rados.0.smithi157.stdout:573: done (15 left) 2024-09-16T03:37:24.059 INFO:tasks.rados.rados.0.smithi157.stdout:577: set_chunk oid 28 target oid 28 low_tier 2024-09-16T03:37:24.059 INFO:tasks.rados.rados.0.smithi157.stdout:577: set_chunk oid 28 offset: 878080 length: 951808 target oid smithi15736696-28 low_tier offset: 878080 snap -1 2024-09-16T03:37:24.059 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.059 INFO:tasks.rados.rados.0.smithi157.stdout:576: finishing set_chunk to oid 27 2024-09-16T03:37:24.059 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 27 set_chunk 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 637440 length: 621056 2024-09-16T03:37:24.059 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:24.059 INFO:tasks.rados.rados.0.smithi157.stdout:576: done (15 left) 2024-09-16T03:37:24.059 INFO:tasks.rados.rados.0.smithi157.stdout:578: set_chunk oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.059 INFO:tasks.rados.rados.0.smithi157.stdout:578: set_chunk oid 29 offset: 461312 length: 1781248 target oid smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 461312 snap -1 2024-09-16T03:37:24.059 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.060 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:37:24.120 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:24.119+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:24.151 INFO:tasks.rados.rados.0.smithi157.stdout:562: finishing set_chunk to oid 13 2024-09-16T03:37:24.152 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 13 set_chunk 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 411136 length: 722944 2024-09-16T03:37:24.152 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T03:37:24.152 INFO:tasks.rados.rados.0.smithi157.stdout:564: finishing set_chunk to oid 15 2024-09-16T03:37:24.152 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 15 set_chunk 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 960000 length: 64000 2024-09-16T03:37:24.152 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:24.152 INFO:tasks.rados.rados.0.smithi157.stdout:562: done (15 left) 2024-09-16T03:37:24.152 INFO:tasks.rados.rados.0.smithi157.stdout:564: done (14 left) 2024-09-16T03:37:24.152 INFO:tasks.rados.rados.0.smithi157.stdout:579: set_chunk oid 30 target oid 30 low_tier 2024-09-16T03:37:24.152 INFO:tasks.rados.rados.0.smithi157.stdout:579: set_chunk oid 30 offset: 1070080 length: 748544 target oid smithi15736696-30 low_tier offset: 1070080 snap -1 2024-09-16T03:37:24.152 INFO:tasks.rados.rados.0.smithi157.stdout:580: set_chunk oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.152 INFO:tasks.rados.rados.0.smithi157.stdout:580: set_chunk oid 31 offset: 966656 length: 1062400 target oid smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 966656 snap -1 2024-09-16T03:37:24.152 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.152 INFO:tasks.rados.rados.0.smithi157.stdout:569: finishing set_chunk to oid 20 2024-09-16T03:37:24.153 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 20 set_chunk 20 low_tier returned error code -95 offset: 309760 length: 54784 2024-09-16T03:37:24.153 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 0 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T03:37:24.153 INFO:tasks.rados.rados.0.smithi157.stdout:569: done (15 left) 2024-09-16T03:37:24.153 INFO:tasks.rados.rados.0.smithi157.stdout:581: set_chunk oid 32 target oid 32 low_tier 2024-09-16T03:37:24.153 INFO:tasks.rados.rados.0.smithi157.stdout:560: finishing set_chunk to oid 11 2024-09-16T03:37:24.153 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 11 set_chunk 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 796672 length: 494080 2024-09-16T03:37:24.153 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 0 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T03:37:24.153 INFO:tasks.rados.rados.0.smithi157.stdout:581: set_chunk oid 32 offset: 331264 length: 1492480 target oid smithi15736696-32 low_tier offset: 331264 snap -1 2024-09-16T03:37:24.153 INFO:tasks.rados.rados.0.smithi157.stdout:560: done (15 left) 2024-09-16T03:37:24.153 INFO:tasks.rados.rados.0.smithi157.stdout:582: set_chunk oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.153 INFO:tasks.rados.rados.0.smithi157.stdout:582: set_chunk oid 33 offset: 674816 length: 1653760 target oid smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 674816 snap -1 2024-09-16T03:37:24.153 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.154 INFO:tasks.rados.rados.0.smithi157.stdout:570: finishing set_chunk to oid 21 2024-09-16T03:37:24.154 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 21 set_chunk 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1265152 length: 1035264 2024-09-16T03:37:24.154 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 0 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T03:37:24.154 INFO:tasks.rados.rados.0.smithi157.stdout:572: finishing set_chunk to oid 23 2024-09-16T03:37:24.154 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 23 set_chunk 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 818688 length: 879616 2024-09-16T03:37:24.154 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T03:37:24.154 INFO:tasks.rados.rados.0.smithi157.stdout:567: finishing set_chunk to oid 18 2024-09-16T03:37:24.154 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 18 set_chunk 18 low_tier returned error code -95 offset: 392192 length: 890880 2024-09-16T03:37:24.154 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 0 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T03:37:24.154 INFO:tasks.rados.rados.0.smithi157.stdout:571: finishing set_chunk to oid 22 2024-09-16T03:37:24.154 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 22 set_chunk 22 low_tier returned error code -95 offset: 726016 length: 947200 2024-09-16T03:37:24.154 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T03:37:24.154 INFO:tasks.rados.rados.0.smithi157.stdout:574: finishing set_chunk to oid 25 2024-09-16T03:37:24.155 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 186880 length: 1035264 2024-09-16T03:37:24.155 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:24.155 INFO:tasks.rados.rados.0.smithi157.stdout:567: done (15 left) 2024-09-16T03:37:24.155 INFO:tasks.rados.rados.0.smithi157.stdout:570: done (14 left) 2024-09-16T03:37:24.155 INFO:tasks.rados.rados.0.smithi157.stdout:571: done (13 left) 2024-09-16T03:37:24.155 INFO:tasks.rados.rados.0.smithi157.stdout:572: done (12 left) 2024-09-16T03:37:24.155 INFO:tasks.rados.rados.0.smithi157.stdout:574: done (11 left) 2024-09-16T03:37:24.155 INFO:tasks.rados.rados.0.smithi157.stdout:583: set_chunk oid 34 target oid 34 low_tier 2024-09-16T03:37:24.155 INFO:tasks.rados.rados.0.smithi157.stdout:583: set_chunk oid 34 offset: 405504 length: 1588224 target oid smithi15736696-34 low_tier offset: 405504 snap -1 2024-09-16T03:37:24.155 INFO:tasks.rados.rados.0.smithi157.stdout:584: set_chunk oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.156 INFO:tasks.rados.rados.0.smithi157.stdout:584: set_chunk oid 35 offset: 458240 length: 1418240 target oid smithi15736696-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 458240 snap -1 2024-09-16T03:37:24.156 INFO:tasks.rados.rados.0.smithi157.stdout:585: set_chunk oid 36 target oid 36 low_tier 2024-09-16T03:37:24.156 INFO:tasks.rados.rados.0.smithi157.stdout:585: set_chunk oid 36 offset: 164864 length: 506368 target oid smithi15736696-36 low_tier offset: 164864 snap -1 2024-09-16T03:37:24.156 INFO:tasks.rados.rados.0.smithi157.stdout:586: set_chunk oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.156 INFO:tasks.rados.rados.0.smithi157.stdout:586: set_chunk oid 37 offset: 973824 length: 950784 target oid smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 973824 snap -1 2024-09-16T03:37:24.156 INFO:tasks.rados.rados.0.smithi157.stdout:587: set_chunk oid 38 target oid 38 low_tier 2024-09-16T03:37:24.156 INFO:tasks.rados.rados.0.smithi157.stdout:587: set_chunk oid 38 offset: 556544 length: 994304 target oid smithi15736696-38 low_tier offset: 556544 snap -1 2024-09-16T03:37:24.156 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.156 INFO:tasks.rados.rados.0.smithi157.stdout:579: finishing set_chunk to oid 30 2024-09-16T03:37:24.156 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 30 set_chunk 30 low_tier returned error code -95 offset: 1070080 length: 748544 2024-09-16T03:37:24.156 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:37:24.156 INFO:tasks.rados.rados.0.smithi157.stdout:575: finishing set_chunk to oid 26 2024-09-16T03:37:24.156 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 26 set_chunk 26 low_tier returned error code -95 offset: 287744 length: 1893888 2024-09-16T03:37:24.157 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:37:24.157 INFO:tasks.rados.rados.0.smithi157.stdout:578: finishing set_chunk to oid 29 2024-09-16T03:37:24.157 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 29 set_chunk 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 461312 length: 1781248 2024-09-16T03:37:24.157 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 0 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T03:37:24.157 INFO:tasks.rados.rados.0.smithi157.stdout:580: finishing set_chunk to oid 31 2024-09-16T03:37:24.157 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 31 set_chunk 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 966656 length: 1062400 2024-09-16T03:37:24.157 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:37:24.157 INFO:tasks.rados.rados.0.smithi157.stdout:575: done (15 left) 2024-09-16T03:37:24.157 INFO:tasks.rados.rados.0.smithi157.stdout:578: done (14 left) 2024-09-16T03:37:24.157 INFO:tasks.rados.rados.0.smithi157.stdout:579: done (13 left) 2024-09-16T03:37:24.157 INFO:tasks.rados.rados.0.smithi157.stdout:580: done (12 left) 2024-09-16T03:37:24.157 INFO:tasks.rados.rados.0.smithi157.stdout:588: set_chunk oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.157 INFO:tasks.rados.rados.0.smithi157.stdout:588: set_chunk oid 39 offset: 676352 length: 1778688 target oid smithi15736696-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 676352 snap -1 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout:589: set_chunk oid 40 target oid 40 low_tier 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout:589: set_chunk oid 40 offset: 1066496 length: 970752 target oid smithi15736696-40 low_tier offset: 1066496 snap -1 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout:590: set_chunk oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout:590: set_chunk oid 41 offset: 1321472 length: 73216 target oid smithi15736696-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1321472 snap -1 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout:591: set_chunk oid 42 target oid 42 low_tier 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout:591: set_chunk oid 42 offset: 18944 length: 1910272 target oid smithi15736696-42 low_tier offset: 18944 snap -1 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout:582: finishing set_chunk to oid 33 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 33 set_chunk 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 674816 length: 1653760 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout:582: done (15 left) 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout:592: set_chunk oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout:592: set_chunk oid 43 offset: 1650688 length: 804864 target oid smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1650688 snap -1 2024-09-16T03:37:24.158 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout:583: finishing set_chunk to oid 34 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 34 set_chunk 34 low_tier returned error code -95 offset: 405504 length: 1588224 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 0 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout:583: done (15 left) 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout:593: set_chunk oid 44 target oid 44 low_tier 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout:593: set_chunk oid 44 offset: 932864 length: 66560 target oid smithi15736696-44 low_tier offset: 932864 snap -1 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout:589: finishing set_chunk to oid 40 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 40 set_chunk 40 low_tier returned error code -95 offset: 1066496 length: 970752 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 0 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout:589: done (15 left) 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout:594: set_chunk oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout:594: set_chunk oid 45 offset: 280576 length: 2447872 target oid smithi15736696-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 280576 snap -1 2024-09-16T03:37:24.159 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.160 INFO:tasks.rados.rados.0.smithi157.stdout:593: finishing set_chunk to oid 44 2024-09-16T03:37:24.160 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 44 set_chunk 44 low_tier returned error code -95 offset: 932864 length: 66560 2024-09-16T03:37:24.160 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 0 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T03:37:24.160 INFO:tasks.rados.rados.0.smithi157.stdout:593: done (15 left) 2024-09-16T03:37:24.160 INFO:tasks.rados.rados.0.smithi157.stdout:595: set_chunk oid 46 target oid 46 low_tier 2024-09-16T03:37:24.160 INFO:tasks.rados.rados.0.smithi157.stdout:595: set_chunk oid 46 offset: 737792 length: 932864 target oid smithi15736696-46 low_tier offset: 737792 snap -1 2024-09-16T03:37:24.160 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.165 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:24.165 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:24.174 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:37:24.221 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:24.220+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:24.249 INFO:tasks.rados.rados.0.smithi157.stdout:585: finishing set_chunk to oid 36 2024-09-16T03:37:24.249 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 36 set_chunk 36 low_tier returned error code -95 offset: 164864 length: 506368 2024-09-16T03:37:24.249 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:37:24.249 INFO:tasks.rados.rados.0.smithi157.stdout:585: done (15 left) 2024-09-16T03:37:24.249 INFO:tasks.rados.rados.0.smithi157.stdout:596: set_chunk oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.249 INFO:tasks.rados.rados.0.smithi157.stdout:596: set_chunk oid 47 offset: 1046528 length: 1121280 target oid smithi15736696-47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 1046528 snap -1 2024-09-16T03:37:24.249 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.250 INFO:tasks.rados.rados.0.smithi157.stdout:586: finishing set_chunk to oid 37 2024-09-16T03:37:24.250 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 37 set_chunk 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 973824 length: 950784 2024-09-16T03:37:24.250 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 0 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T03:37:24.250 INFO:tasks.rados.rados.0.smithi157.stdout:590: finishing set_chunk to oid 41 2024-09-16T03:37:24.250 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 41 set_chunk 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1321472 length: 73216 2024-09-16T03:37:24.250 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T03:37:24.250 INFO:tasks.rados.rados.0.smithi157.stdout:586: done (15 left) 2024-09-16T03:37:24.250 INFO:tasks.rados.rados.0.smithi157.stdout:590: done (14 left) 2024-09-16T03:37:24.250 INFO:tasks.rados.rados.0.smithi157.stdout:597: set_chunk oid 48 target oid 48 low_tier 2024-09-16T03:37:24.250 INFO:tasks.rados.rados.0.smithi157.stdout:597: set_chunk oid 48 offset: 334336 length: 56832 target oid smithi15736696-48 low_tier offset: 334336 snap -1 2024-09-16T03:37:24.250 INFO:tasks.rados.rados.0.smithi157.stdout:598: set_chunk oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.250 INFO:tasks.rados.rados.0.smithi157.stdout:598: set_chunk oid 49 offset: 509952 length: 1938432 target oid smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 509952 snap -1 2024-09-16T03:37:24.250 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout:594: finishing set_chunk to oid 45 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 45 set_chunk 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 280576 length: 2447872 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 0 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout:594: done (15 left) 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout:599: set_chunk oid 50 target oid 50 low_tier 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout:599: set_chunk oid 50 offset: 59904 length: 961536 target oid smithi15736696-50 low_tier offset: 59904 snap -1 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout:597: finishing set_chunk to oid 48 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 48 set_chunk 48 low_tier returned error code -95 offset: 334336 length: 56832 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 0 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout:597: done (15 left) 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout:600: set_chunk oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout:600: set_chunk oid 1 offset: 180224 length: 1463808 target oid smithi15736696-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm offset: 180224 snap -1 2024-09-16T03:37:24.251 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:24.276 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:24.276 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:24.284 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:37:24.322 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:24.320+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:24.347 INFO:tasks.rados.rados.0.smithi157.stdout:581: finishing set_chunk to oid 32 2024-09-16T03:37:24.347 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 32 set_chunk 32 low_tier returned error code -95 offset: 331264 length: 1492480 2024-09-16T03:37:24.348 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:37:24.348 INFO:tasks.rados.rados.0.smithi157.stdout:581: done (15 left) 2024-09-16T03:37:24.348 INFO:tasks.rados.rados.0.smithi157.stdout:601 oid_not_in_use 35 oid_redirect_not_in_use 0 2024-09-16T03:37:24.348 INFO:tasks.rados.rados.0.smithi157.stdout:600 wait for manifest initialization 2024-09-16T03:37:24.348 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 15 2024-09-16T03:37:24.348 INFO:tasks.rados.rados.0.smithi157.stdout:587: finishing set_chunk to oid 38 2024-09-16T03:37:24.348 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 38 set_chunk 38 low_tier returned error code -95 offset: 556544 length: 994304 2024-09-16T03:37:24.348 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:37:24.348 INFO:tasks.rados.rados.0.smithi157.stdout:587: done (14 left) 2024-09-16T03:37:24.348 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 14 2024-09-16T03:37:24.349 INFO:tasks.rados.rados.0.smithi157.stdout:598: finishing set_chunk to oid 49 2024-09-16T03:37:24.349 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 49 set_chunk 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 509952 length: 1938432 2024-09-16T03:37:24.349 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T03:37:24.349 INFO:tasks.rados.rados.0.smithi157.stdout:598: done (13 left) 2024-09-16T03:37:24.349 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 13 2024-09-16T03:37:24.349 INFO:tasks.rados.rados.0.smithi157.stdout:599: finishing set_chunk to oid 50 2024-09-16T03:37:24.349 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 50 set_chunk 50 low_tier returned error code -95 offset: 59904 length: 961536 2024-09-16T03:37:24.349 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T03:37:24.349 INFO:tasks.rados.rados.0.smithi157.stdout:599: done (12 left) 2024-09-16T03:37:24.350 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 12 2024-09-16T03:37:24.350 INFO:tasks.rados.rados.0.smithi157.stdout:563: finishing set_chunk to oid 14 2024-09-16T03:37:24.350 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T03:37:24.350 INFO:tasks.rados.rados.0.smithi157.stdout:563: done (11 left) 2024-09-16T03:37:24.350 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 11 2024-09-16T03:37:24.352 INFO:tasks.rados.rados.0.smithi157.stdout:559: finishing set_chunk to oid 10 2024-09-16T03:37:24.352 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T03:37:24.352 INFO:tasks.rados.rados.0.smithi157.stdout:559: done (10 left) 2024-09-16T03:37:24.352 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 10 2024-09-16T03:37:24.387 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:24.387 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:24.395 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:37:24.423 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:24.421+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:24.448 INFO:tasks.rados.rados.0.smithi157.stdout:565: finishing set_chunk to oid 16 2024-09-16T03:37:24.449 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 16 set_chunk 16 low_tier returned error code -95 offset: 349184 length: 482816 2024-09-16T03:37:24.449 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:37:24.449 INFO:tasks.rados.rados.0.smithi157.stdout:577: finishing set_chunk to oid 28 2024-09-16T03:37:24.449 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 28 set_chunk 28 low_tier returned error code -95 offset: 878080 length: 951808 2024-09-16T03:37:24.449 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T03:37:24.449 INFO:tasks.rados.rados.0.smithi157.stdout:588: finishing set_chunk to oid 39 2024-09-16T03:37:24.449 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 39 set_chunk 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 676352 length: 1778688 2024-09-16T03:37:24.449 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 0 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T03:37:24.449 INFO:tasks.rados.rados.0.smithi157.stdout:592: finishing set_chunk to oid 43 2024-09-16T03:37:24.450 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 43 set_chunk 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1650688 length: 804864 2024-09-16T03:37:24.450 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T03:37:24.450 INFO:tasks.rados.rados.0.smithi157.stdout:584: finishing set_chunk to oid 35 2024-09-16T03:37:24.450 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 35 set_chunk 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 458240 length: 1418240 2024-09-16T03:37:24.450 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:37:24.450 INFO:tasks.rados.rados.0.smithi157.stdout:565: done (9 left) 2024-09-16T03:37:24.450 INFO:tasks.rados.rados.0.smithi157.stdout:577: done (8 left) 2024-09-16T03:37:24.450 INFO:tasks.rados.rados.0.smithi157.stdout:584: done (7 left) 2024-09-16T03:37:24.450 INFO:tasks.rados.rados.0.smithi157.stdout:588: done (6 left) 2024-09-16T03:37:24.450 INFO:tasks.rados.rados.0.smithi157.stdout:592: done (5 left) 2024-09-16T03:37:24.450 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 5 2024-09-16T03:37:24.450 INFO:tasks.rados.rados.0.smithi157.stdout:591: finishing set_chunk to oid 42 2024-09-16T03:37:24.450 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 42 set_chunk 42 low_tier returned error code -95 offset: 18944 length: 1910272 2024-09-16T03:37:24.451 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 0 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T03:37:24.451 INFO:tasks.rados.rados.0.smithi157.stdout:591: done (4 left) 2024-09-16T03:37:24.451 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 4 2024-09-16T03:37:24.451 INFO:tasks.rados.rados.0.smithi157.stdout:600: finishing set_chunk to oid 1 2024-09-16T03:37:24.451 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 1 set_chunk 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 180224 length: 1463808 2024-09-16T03:37:24.451 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T03:37:24.451 INFO:tasks.rados.rados.0.smithi157.stdout:600: done (3 left) 2024-09-16T03:37:24.451 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 3 2024-09-16T03:37:24.451 INFO:tasks.rados.rados.0.smithi157.stdout:595: finishing set_chunk to oid 46 2024-09-16T03:37:24.451 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 46 set_chunk 46 low_tier returned error code -95 offset: 737792 length: 932864 2024-09-16T03:37:24.451 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T03:37:24.451 INFO:tasks.rados.rados.0.smithi157.stdout:595: done (2 left) 2024-09-16T03:37:24.452 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 2 2024-09-16T03:37:24.498 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:24.499 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:24.507 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:37:24.524 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:24.522+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:24.548 INFO:tasks.rados.rados.0.smithi157.stdout:596: finishing set_chunk to oid 47 2024-09-16T03:37:24.548 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 47 set_chunk 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm returned error code -95 offset: 1046528 length: 1121280 2024-09-16T03:37:24.548 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 0 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T03:37:24.548 INFO:tasks.rados.rados.0.smithi157.stdout:596: done (1 left) 2024-09-16T03:37:24.549 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 1 2024-09-16T03:37:24.610 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:24.610 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:24.619 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:37:24.663 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:24.661+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:24.722 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:24.722 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:24.725 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:24.724+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:24.730 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:37:24.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:37:24.826 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:24.825+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:24.853 INFO:tasks.rados.rados.0.smithi157.stdout:566: finishing set_chunk to oid 17 2024-09-16T03:37:24.853 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T03:37:24.853 INFO:tasks.rados.rados.0.smithi157.stdout:566: done (0 left) 2024-09-16T03:37:24.853 INFO:tasks.rados.rados.0.smithi157.stdout:601 oid_not_in_use 50 oid_redirect_not_in_use 0 2024-09-16T03:37:24.854 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 51 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.854 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 52 low_tier 2024-09-16T03:37:24.854 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 53 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.854 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 54 low_tier 2024-09-16T03:37:24.854 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 55 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.854 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 56 low_tier 2024-09-16T03:37:24.856 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 57 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.856 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 58 low_tier 2024-09-16T03:37:24.856 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 59 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.856 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 60 low_tier 2024-09-16T03:37:24.856 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 61 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.856 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 62 low_tier 2024-09-16T03:37:24.856 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 63 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.856 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 64 low_tier 2024-09-16T03:37:24.856 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 65 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.857 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 66 low_tier 2024-09-16T03:37:24.857 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 67 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.857 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 68 low_tier 2024-09-16T03:37:24.857 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 69 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.857 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 70 low_tier 2024-09-16T03:37:24.857 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 71 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.857 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 72 low_tier 2024-09-16T03:37:24.857 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 73 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.857 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 74 low_tier 2024-09-16T03:37:24.857 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 75 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.857 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 76 low_tier 2024-09-16T03:37:24.857 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 77 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.858 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 78 low_tier 2024-09-16T03:37:24.858 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 79 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.858 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 80 low_tier 2024-09-16T03:37:24.858 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 81 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.858 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 82 low_tier 2024-09-16T03:37:24.858 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 83 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.858 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 84 low_tier 2024-09-16T03:37:24.858 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 85 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.858 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 86 low_tier 2024-09-16T03:37:24.858 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 87 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.858 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 88 low_tier 2024-09-16T03:37:24.858 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 89 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.858 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 90 low_tier 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 91 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 92 low_tier 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 93 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 94 low_tier 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 95 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 96 low_tier 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 97 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 98 low_tier 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 99 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout: redirect_not_in_use: 100 low_tier 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout:601: read oid 13 snap -1 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout:601: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout:602: read oid 2 snap -1 2024-09-16T03:37:24.859 INFO:tasks.rados.rados.0.smithi157.stdout:602: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-16T03:37:24.860 INFO:tasks.rados.rados.0.smithi157.stdout:603: read oid 50 snap -1 2024-09-16T03:37:24.860 INFO:tasks.rados.rados.0.smithi157.stdout:603: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-16T03:37:24.860 INFO:tasks.rados.rados.0.smithi157.stdout:604: read oid 24 snap -1 2024-09-16T03:37:24.860 INFO:tasks.rados.rados.0.smithi157.stdout:604: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:37:24.860 INFO:tasks.rados.rados.0.smithi157.stdout:605: tier_evict oid 6 2024-09-16T03:37:24.860 INFO:tasks.rados.rados.0.smithi157.stdout:605: tier_evict oid 6 snap -1 2024-09-16T03:37:24.860 INFO:tasks.rados.rados.0.smithi157.stdout:606: read oid 1 snap -1 2024-09-16T03:37:24.860 INFO:tasks.rados.rados.0.smithi157.stdout:606: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T03:37:24.860 INFO:tasks.rados.rados.0.smithi157.stdout:607: set_chunk oid 8 target oid 2024-09-16T03:37:24.875 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:24.875 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:24.884 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:37:24.889 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:24.889 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:24.906 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:24.907 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:24.924 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:24.924 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:24.927 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:24.925+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:24.941 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:24.942 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:24.958 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:24.958 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:24.974 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:24.974 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:24.988 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:24.988 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:24.991 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:24.992 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:24.997 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:37:25.008 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:25.009 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:25.025 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:25.026 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:25.026 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:37:25.029 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:25.027+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:25.044 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:25.044 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:25.061 INFO:tasks.rados.rados.0.smithi157.stdout:607: set_chunk oid 8 offset: 1526784 length: 423936 target oid 2e5d5ef52ec15d410ea89ac4791f8d7027cdb564a460b6ed1c3e9770fac44c79 offset: 0 snap -1 2024-09-16T03:37:25.061 INFO:tasks.rados.rados.0.smithi157.stdout:608: read oid 18 snap -1 2024-09-16T03:37:25.062 INFO:tasks.rados.rados.0.smithi157.stdout:608: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T03:37:25.062 INFO:tasks.rados.rados.0.smithi157.stdout:609: tier_evict oid 38 2024-09-16T03:37:25.062 INFO:tasks.rados.rados.0.smithi157.stdout:609: tier_evict oid 38 snap -1 2024-09-16T03:37:25.062 INFO:tasks.rados.rados.0.smithi157.stdout:605: got (22) Invalid argument 2024-09-16T03:37:25.062 INFO:tasks.rados.rados.0.smithi157.stdout:605: done (8 left) 2024-09-16T03:37:25.062 INFO:tasks.rados.rados.0.smithi157.stdout:610: tier_promote oid 14 2024-09-16T03:37:25.062 INFO:tasks.rados.rados.0.smithi157.stdout:604: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:37:25.062 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 651748 2024-09-16T03:37:25.063 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:25.063 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:25.078 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:25.078 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:25.083 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:25.094 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:25.094 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:25.100 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:25.101 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:25.109 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:37:25.110 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:25.111 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:25.128 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:25.128 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:25.135 INFO:tasks.rados.rados.0.smithi157.stdout:601: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-16T03:37:25.144 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:25.144 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:25.152 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:25.151+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:25.185 INFO:tasks.rados.rados.0.smithi157.stdout:602: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-16T03:37:25.185 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:25.211 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:25.212 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:25.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:25.212 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:37:25.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.212 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:57 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:25.212 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:37:25.212 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.085355423999999, 2024-09-16T03:37:25.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20372090900000001, 2024-09-16T03:37:25.212 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:25.212 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:25.212 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:25.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125313+0000", 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7180000000000001e-06 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126730+0000", 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00141731 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126746+0000", 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6455000000000001e-05 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126756+0000", 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227068+0000", 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100312308 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227085+0000", 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6898e-05 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:25.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328995+0000", 2024-09-16T03:37:25.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101909903 2024-09-16T03:37:25.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:25.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329031+0000", 2024-09-16T03:37:25.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5717999999999999e-05 2024-09-16T03:37:25.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:25.216 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:25.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:25.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:58 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.082022294, 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084524500000001, 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128644+0000", 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.251e-06 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128913+0000", 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026856200000000002 2024-09-16T03:37:25.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128925+0000", 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2109999999999999e-05 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128933+0000", 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6260000000000003e-06 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328196+0000", 2024-09-16T03:37:25.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19926303200000001 2024-09-16T03:37:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328253+0000", 2024-09-16T03:37:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7262999999999997e-05 2024-09-16T03:37:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329458+0000", 2024-09-16T03:37:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012051939999999999 2024-09-16T03:37:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:25.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329488+0000", 2024-09-16T03:37:25.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0207000000000001e-05 2024-09-16T03:37:25.221 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:25.221 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:25.221 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:25.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.221 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:59 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:25.221 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:37:25.221 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.081526861, 2024-09-16T03:37:25.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036954900000001, 2024-09-16T03:37:25.222 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:25.222 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:25.222 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.222 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:25.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:25.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:37:25.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:25.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.223 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:25.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:37:25.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:25.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:25.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129139+0000", 2024-09-16T03:37:25.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6099999999999999e-07 2024-09-16T03:37:25.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:25.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129158+0000", 2024-09-16T03:37:25.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088e-05 2024-09-16T03:37:25.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:25.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129168+0000", 2024-09-16T03:37:25.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.048e-05 2024-09-16T03:37:25.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.229233+0000", 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100065168 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328815+0000", 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099581694999999998 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328845+0000", 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0193e-05 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:25.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329492+0000", 2024-09-16T03:37:25.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064671800000000001 2024-09-16T03:37:25.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:25.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:25.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:25.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329508+0000", 2024-09-16T03:37:25.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6246e-05 2024-09-16T03:37:25.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:25.227 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:25.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:25.228 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:25.228 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:25.228 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:25.229 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:37:25.230 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:25.228+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:25.251 INFO:tasks.rados.rados.0.smithi157.stdout:606: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T03:37:25.254 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 623860 2024-09-16T03:37:25.270 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:25.314 INFO:tasks.rados.rados.0.smithi157.stdout:603: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-16T03:37:25.314 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:25.331 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:25.329+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:25.338 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:25.338 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:25.338 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:37:25.338 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:25.346 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:37:25.374 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:25.404 INFO:tasks.rados.rados.0.smithi157.stdout:607: finishing set_chunk to oid 8 2024-09-16T03:37:25.404 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 8 set_chunk returned error code -95 offset: 1526784 length: 423936 2024-09-16T03:37:25.405 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T03:37:25.405 INFO:tasks.rados.rados.0.smithi157.stdout:608: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T03:37:25.405 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:25.429 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 359002 2024-09-16T03:37:25.432 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:25.430+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:25.449 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:25.449 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:25.449 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:25.450 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:25.450 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:25.458 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:37:25.469 INFO:tasks.rados.rados.0.smithi157.stdout:609: got (22) Invalid argument 2024-09-16T03:37:25.470 INFO:tasks.rados.rados.0.smithi157.stdout:610: got (0) Success 2024-09-16T03:37:25.470 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T03:37:25.470 INFO:tasks.rados.rados.0.smithi157.stdout:601: done (8 left) 2024-09-16T03:37:25.470 INFO:tasks.rados.rados.0.smithi157.stdout:602: done (7 left) 2024-09-16T03:37:25.470 INFO:tasks.rados.rados.0.smithi157.stdout:603: done (6 left) 2024-09-16T03:37:25.470 INFO:tasks.rados.rados.0.smithi157.stdout:604: done (5 left) 2024-09-16T03:37:25.470 INFO:tasks.rados.rados.0.smithi157.stdout:606: done (4 left) 2024-09-16T03:37:25.470 INFO:tasks.rados.rados.0.smithi157.stdout:607: done (3 left) 2024-09-16T03:37:25.470 INFO:tasks.rados.rados.0.smithi157.stdout:608: done (2 left) 2024-09-16T03:37:25.470 INFO:tasks.rados.rados.0.smithi157.stdout:609: done (1 left) 2024-09-16T03:37:25.470 INFO:tasks.rados.rados.0.smithi157.stdout:610: done (0 left) 2024-09-16T03:37:25.470 INFO:tasks.rados.rados.0.smithi157.stdout:611: read oid 36 snap -1 2024-09-16T03:37:25.470 INFO:tasks.rados.rados.0.smithi157.stdout:611: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-16T03:37:25.471 INFO:tasks.rados.rados.0.smithi157.stdout:612: set_chunk oid 33 target oid 2024-09-16T03:37:25.533 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:25.532+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:25.564 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:25.564 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:25.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:25.564 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:25.564 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.564 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:5 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'1, mlcod=0'0)", 2024-09-16T03:37:25.564 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:37:25.564 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.592420345000001, 2024-09-16T03:37:25.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.105742802, 2024-09-16T03:37:25.564 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968147+0000", 2024-09-16T03:37:25.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4e-07 2024-09-16T03:37:25.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968752+0000", 2024-09-16T03:37:25.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060437900000000005 2024-09-16T03:37:25.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968762+0000", 2024-09-16T03:37:25.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0284e-05 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968778+0000", 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6124e-05 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069797+0000", 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10101895700000001 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069833+0000", 2024-09-16T03:37:25.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6013000000000002e-05 2024-09-16T03:37:25.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.568 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.568 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.568 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073838+0000", 2024-09-16T03:37:25.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040049049999999996 2024-09-16T03:37:25.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.568 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.568 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.568 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073889+0000", 2024-09-16T03:37:25.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.13e-05 2024-09-16T03:37:25.568 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.568 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.569 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.569 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:6 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'2, mlcod=0'0)", 2024-09-16T03:37:25.569 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.973669+0000", 2024-09-16T03:37:25.569 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.586897615, 2024-09-16T03:37:25.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100614443, 2024-09-16T03:37:25.569 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.569 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.569 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.569 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.569 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.973669+0000", 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.973669+0000", 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.973671+0000", 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.463e-06 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.974070+0000", 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000399334 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.974082+0000", 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1523e-05 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.974087+0000", 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8069999999999996e-06 2024-09-16T03:37:25.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.572 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.071099+0000", 2024-09-16T03:37:25.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.097011981999999997 2024-09-16T03:37:25.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.572 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.071116+0000", 2024-09-16T03:37:25.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6566e-05 2024-09-16T03:37:25.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.572 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.074255+0000", 2024-09-16T03:37:25.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031387540000000001 2024-09-16T03:37:25.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.074284+0000", 2024-09-16T03:37:25.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9014e-05 2024-09-16T03:37:25.573 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.573 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.573 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.573 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:7 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'3, mlcod=0'0)", 2024-09-16T03:37:25.573 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.585527846, 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20061241399999999, 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975042+0000", 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5289999999999998e-06 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975079+0000", 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7626999999999997e-05 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975104+0000", 2024-09-16T03:37:25.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4958999999999999e-05 2024-09-16T03:37:25.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.576 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.075211+0000", 2024-09-16T03:37:25.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100106814 2024-09-16T03:37:25.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.576 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080573+0000", 2024-09-16T03:37:25.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0053618650000000004 2024-09-16T03:37:25.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080592+0000", 2024-09-16T03:37:25.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9079000000000001e-05 2024-09-16T03:37:25.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175497+0000", 2024-09-16T03:37:25.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.094904832999999994 2024-09-16T03:37:25.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175651+0000", 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015470800000000001 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.374574674, 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10311670100000001, 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.578 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.579 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.579 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.579 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:37:25.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.579 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.579 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.579 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.579 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:37:25.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.579 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.579 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.579 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.579 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186761+0000", 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076880999999999998 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186772+0000", 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0986e-05 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186799+0000", 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6254e-05 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186876+0000", 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7523999999999998e-05 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186896+0000", 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0302999999999999e-05 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289077+0000", 2024-09-16T03:37:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10218086899999999 2024-09-16T03:37:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289109+0000", 2024-09-16T03:37:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1625000000000003e-05 2024-09-16T03:37:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36)", 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.373368885, 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102408122, 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:37:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187199+0000", 2024-09-16T03:37:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6199999999999996e-07 2024-09-16T03:37:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188184+0000", 2024-09-16T03:37:25.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000985442 2024-09-16T03:37:25.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188194+0000", 2024-09-16T03:37:25.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5659999999999996e-06 2024-09-16T03:37:25.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188210+0000", 2024-09-16T03:37:25.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5858999999999998e-05 2024-09-16T03:37:25.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288363+0000", 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100152872 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288390+0000", 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7081999999999999e-05 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289577+0000", 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011875740000000001 2024-09-16T03:37:25.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289606+0000", 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8864999999999999e-05 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:47 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.288361+0000", 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.272205343, 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100659055, 2024-09-16T03:37:25.587 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288361+0000", 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288361+0000", 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288364+0000", 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2570000000000002e-06 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288376+0000", 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2444e-05 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288387+0000", 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0706999999999999e-05 2024-09-16T03:37:25.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288395+0000", 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1419999999999993e-06 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.388498+0000", 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100103236 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.388520+0000", 2024-09-16T03:37:25.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1815e-05 2024-09-16T03:37:25.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.388985+0000", 2024-09-16T03:37:25.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00046473100000000002 2024-09-16T03:37:25.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.389021+0000", 2024-09-16T03:37:25.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5722999999999997e-05 2024-09-16T03:37:25.591 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.591 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.591 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:13 3.3 e41/31 3:dc284e8d:::smithi15736696-28 low_tier:head v 41'14, mlcod=40'12)", 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.421422+0000", 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.139145052, 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000782356, 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.421422+0000", 2024-09-16T03:37:25.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.421422+0000", 2024-09-16T03:37:25.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.421422+0000", 2024-09-16T03:37:25.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7999999999999995e-07 2024-09-16T03:37:25.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.421456+0000", 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3458999999999999e-05 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.421478+0000", 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1843000000000001e-05 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.421493+0000", 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5477000000000001e-05 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.421626+0000", 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013297999999999999 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.421667+0000", 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0717000000000003e-05 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.422167+0000", 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050008699999999997 2024-09-16T03:37:25.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.422204+0000", 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7212999999999998e-05 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:281 4.4 e41/32 4:2cb5f2b8:::smithi15736696-32:head v 41'8, mlcod=39'6)", 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.600545+0000", 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.9600222810000005, 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00074336400000000005, 2024-09-16T03:37:25.596 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.597 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.597 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.597 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600545+0000", 2024-09-16T03:37:25.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600545+0000", 2024-09-16T03:37:25.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600545+0000", 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999997e-07 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600555+0000", 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7399999999999999e-06 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600595+0000", 2024-09-16T03:37:25.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0318999999999998e-05 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600616+0000", 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0336999999999999e-05 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600734+0000", 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000118841 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600771+0000", 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6123e-05 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.601255+0000", 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048475500000000002 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.601288+0000", 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2549000000000003e-05 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.600 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:10 3.3 e41/31 3:c069f14c:::smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'16, mlcod=41'14)", 2024-09-16T03:37:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.701871+0000", 2024-09-16T03:37:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.8586954020000004, 2024-09-16T03:37:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100943475, 2024-09-16T03:37:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701871+0000", 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701871+0000", 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701873+0000", 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1659999999999999e-06 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701879+0000", 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6540000000000002e-06 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701891+0000", 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.182e-05 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701898+0000", 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3270000000000003e-06 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802103+0000", 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10020419899999999 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802141+0000", 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8411999999999997e-05 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802782+0000", 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00064137500000000002 2024-09-16T03:37:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802815+0000", 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2521999999999998e-05 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:12 3.3 e41/31 3:deb54e08:::smithi15736696-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'17, mlcod=41'14)", 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.706846+0000", 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.8537207389999999, 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.096271079999999995, 2024-09-16T03:37:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.706846+0000", 2024-09-16T03:37:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.706846+0000", 2024-09-16T03:37:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.706847+0000", 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.105e-06 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.706865+0000", 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8099999999999999e-05 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.706876+0000", 2024-09-16T03:37:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0479000000000001e-05 2024-09-16T03:37:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.706882+0000", 2024-09-16T03:37:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4910000000000002e-06 2024-09-16T03:37:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802431+0000", 2024-09-16T03:37:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.095548925000000007 2024-09-16T03:37:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802465+0000", 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3476999999999998e-05 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.803092+0000", 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00062730200000000005 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.803117+0000", 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5201000000000001e-05 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:16 3.3 e41/31 3:e44244d1:::smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'18, mlcod=41'14)", 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.903154+0000", 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.6574129620000004, 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010928190000000001, 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903154+0000", 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903154+0000", 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903157+0000", 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7530000000000002e-06 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903169+0000", 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1970000000000001e-05 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903195+0000", 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5939e-05 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903208+0000", 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3677e-05 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903339+0000", 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013092 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903401+0000", 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2175999999999996e-05 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.904192+0000", 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00079067600000000003 2024-09-16T03:37:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.904247+0000", 2024-09-16T03:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4707999999999999e-05 2024-09-16T03:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.4.22:4 3.3 e41/31 3:deac805c:::smithi15736696-44 low_tier:head v 41'19, mlcod=41'14)", 2024-09-16T03:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.903235+0000", 2024-09-16T03:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.6573316640000009, 2024-09-16T03:37:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003515579, 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903235+0000", 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903235+0000", 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903236+0000", 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2600000000000001e-07 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903251+0000", 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5130000000000001e-05 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903260+0000", 2024-09-16T03:37:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0939999999999997e-06 2024-09-16T03:37:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903267+0000", 2024-09-16T03:37:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2200000000000003e-06 2024-09-16T03:37:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903714+0000", 2024-09-16T03:37:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00044627300000000001 2024-09-16T03:37:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903734+0000", 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0411000000000002e-05 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.906713+0000", 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0029788480000000001 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.906751+0000", 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7777000000000001e-05 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.4.22:5 3.3 e42/31 3:cb519cc1:::smithi15736696-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'21, mlcod=41'19)", 2024-09-16T03:37:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.823198+0000", 2024-09-16T03:37:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 8.7373690889999995, 2024-09-16T03:37:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00075431900000000004, 2024-09-16T03:37:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.823198+0000", 2024-09-16T03:37:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.823198+0000", 2024-09-16T03:37:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.823200+0000", 2024-09-16T03:37:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.846e-06 2024-09-16T03:37:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.823208+0000", 2024-09-16T03:37:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0960000000000001e-06 2024-09-16T03:37:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.823221+0000", 2024-09-16T03:37:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3373000000000001e-05 2024-09-16T03:37:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.823228+0000", 2024-09-16T03:37:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8040000000000004e-06 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.823283+0000", 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4886000000000003e-05 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.823303+0000", 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0387999999999999e-05 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.823909+0000", 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060546600000000003 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.823952+0000", 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3460000000000001e-05 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:24 3.3 e45/31 3:dc284e8d:::smithi15736696-28 low_tier:head v 45'23, mlcod=42'21)", 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.205792+0000", 2024-09-16T03:37:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.3547744000000002, 2024-09-16T03:37:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100620102, 2024-09-16T03:37:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205792+0000", 2024-09-16T03:37:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205792+0000", 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205793+0000", 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-07 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205799+0000", 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3409999999999999e-06 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205808+0000", 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7800000000000006e-06 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205814+0000", 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2330000000000003e-06 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.305921+0000", 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100107423 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.305935+0000", 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3529e-05 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306385+0000", 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045013900000000001 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306413+0000", 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7424999999999999e-05 2024-09-16T03:37:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.4.22:6 3.3 e45/31 3:deac805c:::smithi15736696-44 low_tier:head v 45'24, mlcod=42'21)", 2024-09-16T03:37:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.206475+0000", 2024-09-16T03:37:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.3540917889999999, 2024-09-16T03:37:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10029721, 2024-09-16T03:37:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206475+0000", 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206475+0000", 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206476+0000", 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.032e-06 2024-09-16T03:37:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206484+0000", 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7659999999999998e-06 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206493+0000", 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0010000000000007e-06 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206498+0000", 2024-09-16T03:37:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7620000000000003e-06 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306126+0000", 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099628128999999996 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306137+0000", 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0815999999999999e-05 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306750+0000", 2024-09-16T03:37:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061334499999999995 2024-09-16T03:37:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306772+0000", 2024-09-16T03:37:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2359000000000001e-05 2024-09-16T03:37:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:31 3.3 e45/31 3:e44244d1:::smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'27, mlcod=45'25)", 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.722784+0000", 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.8377827839999998, 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0012230380000000001, 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722784+0000", 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722784+0000", 2024-09-16T03:37:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722785+0000", 2024-09-16T03:37:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.387e-06 2024-09-16T03:37:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722801+0000", 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.56e-05 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722811+0000", 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6330000000000008e-06 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722816+0000", 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7819999999999999e-06 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.723454+0000", 2024-09-16T03:37:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000637442 2024-09-16T03:37:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.723471+0000", 2024-09-16T03:37:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7444000000000001e-05 2024-09-16T03:37:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.723979+0000", 2024-09-16T03:37:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050730199999999995 2024-09-16T03:37:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.724007+0000", 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8447999999999999e-05 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:26 3.3 e45/31 3:deb54e08:::smithi15736696-41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'28, mlcod=45'25)", 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.723292+0000", 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.8372751899999997, 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10011096999999999, 2024-09-16T03:37:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.723292+0000", 2024-09-16T03:37:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.723292+0000", 2024-09-16T03:37:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.723294+0000", 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4729999999999999e-06 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.723307+0000", 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3363000000000001e-05 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.723333+0000", 2024-09-16T03:37:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5296999999999999e-05 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.723353+0000", 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9802000000000001e-05 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822924+0000", 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099571316000000007 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822938+0000", 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3628999999999999e-05 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.823376+0000", 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00043805 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.823403+0000", 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7039999999999999e-05 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.5.21:14 3.3 e45/31 3:fb369346:::smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'29, mlcod=45'27)", 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.924074+0000", 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.6364925029999999, 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100324169, 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924074+0000", 2024-09-16T03:37:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924074+0000", 2024-09-16T03:37:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924076+0000", 2024-09-16T03:37:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2780000000000001e-06 2024-09-16T03:37:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924081+0000", 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4190000000000001e-06 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924090+0000", 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4700000000000008e-06 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924096+0000", 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2689999999999999e-06 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924929+0000", 2024-09-16T03:37:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00083369399999999995 2024-09-16T03:37:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924955+0000", 2024-09-16T03:37:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5893000000000001e-05 2024-09-16T03:37:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024359+0000", 2024-09-16T03:37:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099403434999999998 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024398+0000", 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9711e-05 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.4.22:8 3.3 e45/31 3:cb519cc1:::smithi15736696-9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'30, mlcod=45'27)", 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.924855+0000", 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.6357116110000001, 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099570465999999996, 2024-09-16T03:37:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924855+0000", 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924855+0000", 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924856+0000", 2024-09-16T03:37:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.012e-06 2024-09-16T03:37:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924861+0000", 2024-09-16T03:37:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4889999999999997e-06 2024-09-16T03:37:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924869+0000", 2024-09-16T03:37:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3699999999999995e-06 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924874+0000", 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.383e-06 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.925493+0000", 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061804600000000003 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.925505+0000", 2024-09-16T03:37:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2422999999999999e-05 2024-09-16T03:37:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024402+0000", 2024-09-16T03:37:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098896566000000005 2024-09-16T03:37:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024426+0000", 2024-09-16T03:37:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4176999999999999e-05 2024-09-16T03:37:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:32 3.3 e45/31 3:dc284e8d:::smithi15736696-28 low_tier:head v 45'31, mlcod=45'27)", 2024-09-16T03:37:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.925429+0000", 2024-09-16T03:37:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.635137823, 2024-09-16T03:37:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099359329999999996, 2024-09-16T03:37:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.925429+0000", 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.925429+0000", 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.925430+0000", 2024-09-16T03:37:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1500000000000003e-07 2024-09-16T03:37:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.925440+0000", 2024-09-16T03:37:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0526999999999999e-05 2024-09-16T03:37:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.925448+0000", 2024-09-16T03:37:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.588e-06 2024-09-16T03:37:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.925461+0000", 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2690999999999999e-05 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024196+0000", 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098735162000000001 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024213+0000", 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6660999999999999e-05 2024-09-16T03:37:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024749+0000", 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000536467 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024788+0000", 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9319000000000001e-05 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:25.655 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:25.657 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:37:25.657 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:25.655+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:25.757 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:25.756+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:25.791 INFO:tasks.rados.rados.0.smithi157.stdout:612: set_chunk oid 33 offset: 1987584 length: 45056 target oid 2782d12a88f9842f7f754fca32c4222cf10fb780cd136830d32faaa700e687ba offset: 0 snap -1 2024-09-16T03:37:25.792 INFO:tasks.rados.rados.0.smithi157.stdout:613: write oid 50 current snap is 0 2024-09-16T03:37:25.792 INFO:tasks.rados.rados.0.smithi157.stdout:613: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-09-16T03:37:25.796 INFO:tasks.rados.rados.0.smithi157.stdout:613: writing smithi15736696-50 from 440350 to 900918 tid 1 2024-09-16T03:37:25.801 INFO:tasks.rados.rados.0.smithi157.stdout:613: writing smithi15736696-50 from 1309629 to 1852295 tid 2 2024-09-16T03:37:25.801 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:25.802 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:25.802 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:37:25.802 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:25.806 INFO:tasks.rados.rados.0.smithi157.stdout:613: writing smithi15736696-50 from 2323973 to 2840350 tid 3 2024-09-16T03:37:25.807 INFO:tasks.rados.rados.0.smithi157.stdout:614: read oid 24 snap -1 2024-09-16T03:37:25.807 INFO:tasks.rados.rados.0.smithi157.stdout:614: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:37:25.807 INFO:tasks.rados.rados.0.smithi157.stdout:615: tier_evict oid 46 2024-09-16T03:37:25.807 INFO:tasks.rados.rados.0.smithi157.stdout:615: tier_evict oid 46 snap -1 2024-09-16T03:37:25.807 INFO:tasks.rados.rados.0.smithi157.stdout:611: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-16T03:37:25.811 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:37:25.858 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:25.857+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:25.888 INFO:tasks.rados.rados.0.smithi157.stdout:612: finishing set_chunk to oid 33 2024-09-16T03:37:25.889 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 33 set_chunk returned error code -95 offset: 1987584 length: 45056 2024-09-16T03:37:25.889 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:37:25.889 INFO:tasks.rados.rados.0.smithi157.stdout:615: got (22) Invalid argument 2024-09-16T03:37:25.889 INFO:tasks.rados.rados.0.smithi157.stdout:611: done (4 left) 2024-09-16T03:37:25.889 INFO:tasks.rados.rados.0.smithi157.stdout:612: done (3 left) 2024-09-16T03:37:25.889 INFO:tasks.rados.rados.0.smithi157.stdout:615: done (2 left) 2024-09-16T03:37:25.889 INFO:tasks.rados.rados.0.smithi157.stdout:616: tier_flush oid 15 2024-09-16T03:37:25.889 INFO:tasks.rados.rados.0.smithi157.stdout:616: tier_flush oid 15 snap -1 2024-09-16T03:37:25.889 INFO:tasks.rados.rados.0.smithi157.stdout:617: set_chunk oid 33 target oid 2024-09-16T03:37:25.916 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:25.916 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:25.917 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:25.917 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:25.917 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:25.925 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:37:25.966 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:25.965+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:25.995 INFO:tasks.rados.rados.0.smithi157.stdout:617: set_chunk oid 33 offset: 1868288 length: 561152 target oid d9bff02604770a4b90866c0f37900b237515e0dcc1cf1cf54bd0ef4ea514bc62 offset: 0 snap -1 2024-09-16T03:37:25.995 INFO:tasks.rados.rados.0.smithi157.stdout:618: read oid 37 snap -1 2024-09-16T03:37:25.995 INFO:tasks.rados.rados.0.smithi157.stdout:618: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T03:37:25.995 INFO:tasks.rados.rados.0.smithi157.stdout:619: write (excl) oid 44 current snap is 0 2024-09-16T03:37:25.995 INFO:tasks.rados.rados.0.smithi157.stdout:619: seq_num 51 ranges {457157=535817,1768111=402677} 2024-09-16T03:37:26.000 INFO:tasks.rados.rados.0.smithi157.stdout:619: writing smithi15736696-44 from 457157 to 992974 tid 1 2024-09-16T03:37:26.007 INFO:tasks.rados.rados.0.smithi157.stdout:619: writing smithi15736696-44 from 1768111 to 2170788 tid 2 2024-09-16T03:37:26.007 INFO:tasks.rados.rados.0.smithi157.stdout:620: tier_promote oid 25 2024-09-16T03:37:26.007 INFO:tasks.rados.rados.0.smithi157.stdout:621: read oid 31 snap -1 2024-09-16T03:37:26.007 INFO:tasks.rados.rados.0.smithi157.stdout:621: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-16T03:37:26.007 INFO:tasks.rados.rados.0.smithi157.stdout:622: write (excl) oid 11 current snap is 0 2024-09-16T03:37:26.007 INFO:tasks.rados.rados.0.smithi157.stdout:622: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2024-09-16T03:37:26.012 INFO:tasks.rados.rados.0.smithi157.stdout:622: writing smithi15736696-11 from 473964 to 1085030 tid 1 2024-09-16T03:37:26.019 INFO:tasks.rados.rados.0.smithi157.stdout:622: writing smithi15736696-11 from 1826593 to 2372928 tid 2 2024-09-16T03:37:26.022 INFO:tasks.rados.rados.0.smithi157.stdout:622: writing smithi15736696-11 from 2778819 to 2873964 tid 3 2024-09-16T03:37:26.022 INFO:tasks.rados.rados.0.smithi157.stdout:623: read oid 42 snap -1 2024-09-16T03:37:26.022 INFO:tasks.rados.rados.0.smithi157.stdout:623: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-16T03:37:26.023 INFO:tasks.rados.rados.0.smithi157.stdout:624: read oid 1 snap -1 2024-09-16T03:37:26.023 INFO:tasks.rados.rados.0.smithi157.stdout:624: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T03:37:26.023 INFO:tasks.rados.rados.0.smithi157.stdout:625: write oid 13 current snap is 0 2024-09-16T03:37:26.023 INFO:tasks.rados.rados.0.smithi157.stdout:625: seq_num 53 ranges {490771=686315,1885075=406346} 2024-09-16T03:37:26.029 INFO:tasks.rados.rados.0.smithi157.stdout:625: writing smithi15736696-13 from 490771 to 1177086 tid 1 2024-09-16T03:37:26.035 INFO:tasks.rados.rados.0.smithi157.stdout:625: writing smithi15736696-13 from 1885075 to 2291421 tid 2 2024-09-16T03:37:26.035 INFO:tasks.rados.rados.0.smithi157.stdout:626: read oid 18 snap -1 2024-09-16T03:37:26.035 INFO:tasks.rados.rados.0.smithi157.stdout:626: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T03:37:26.035 INFO:tasks.rados.rados.0.smithi157.stdout:627: read oid 27 snap -1 2024-09-16T03:37:26.035 INFO:tasks.rados.rados.0.smithi157.stdout:627: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:26.035 INFO:tasks.rados.rados.0.smithi157.stdout:628: write (excl) oid 46 current snap is 0 2024-09-16T03:37:26.035 INFO:tasks.rados.rados.0.smithi157.stdout:614: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:37:26.039 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1313976 2024-09-16T03:37:26.059 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:26.057+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:26.072 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:50 4.6 e39/32 4:6df3bbbe:::smithi15736696-13:head v 39'5, mlcod=0'0)", 2024-09-16T03:37:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.224976+0000", 2024-09-16T03:37:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.845056195, 2024-09-16T03:37:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100935701, 2024-09-16T03:37:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.224976+0000", 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.224976+0000", 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.224978+0000", 2024-09-16T03:37:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.105e-06 2024-09-16T03:37:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.225494+0000", 2024-09-16T03:37:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051689699999999999 2024-09-16T03:37:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.225506+0000", 2024-09-16T03:37:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1576999999999999e-05 2024-09-16T03:37:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.225511+0000", 2024-09-16T03:37:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1259999999999997e-06 2024-09-16T03:37:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.226058+0000", 2024-09-16T03:37:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054723599999999999 2024-09-16T03:37:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.226070+0000", 2024-09-16T03:37:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1800000000000001e-05 2024-09-16T03:37:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.325877+0000", 2024-09-16T03:37:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099806414999999996 2024-09-16T03:37:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.325912+0000", 2024-09-16T03:37:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5544999999999999e-05 2024-09-16T03:37:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:53 4.7 e39/32 4:ed874e32:::smithi15736696-14:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.826709572, 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20229554399999999, 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:37:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:37:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9600000000000001e-07 2024-09-16T03:37:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243671+0000", 2024-09-16T03:37:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034761699999999999 2024-09-16T03:37:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243681+0000", 2024-09-16T03:37:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0505e-05 2024-09-16T03:37:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343786+0000", 2024-09-16T03:37:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10010472300000001 2024-09-16T03:37:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.443948+0000", 2024-09-16T03:37:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100161739 2024-09-16T03:37:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.443966+0000", 2024-09-16T03:37:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8045000000000001e-05 2024-09-16T03:37:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445575+0000", 2024-09-16T03:37:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001608841 2024-09-16T03:37:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445619+0000", 2024-09-16T03:37:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3878000000000002e-05 2024-09-16T03:37:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:54 4.7 e39/32 4:ed874e32:::smithi15736696-14:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.343820+0000", 2024-09-16T03:37:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.726212403, 2024-09-16T03:37:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10196166, 2024-09-16T03:37:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343820+0000", 2024-09-16T03:37:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343820+0000", 2024-09-16T03:37:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343821+0000", 2024-09-16T03:37:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.181e-06 2024-09-16T03:37:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343931+0000", 2024-09-16T03:37:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000109572 2024-09-16T03:37:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343939+0000", 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3690000000000007e-06 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343945+0000", 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2549999999999997e-06 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444867+0000", 2024-09-16T03:37:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.1009227 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444885+0000", 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7852000000000001e-05 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445753+0000", 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086790399999999996 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445782+0000", 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8827e-05 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:55 4.7 e39/32 4:ed874e32:::smithi15736696-14:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.343952+0000", 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.72608018, 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10231132599999999, 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343952+0000", 2024-09-16T03:37:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343952+0000", 2024-09-16T03:37:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343953+0000", 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000002e-07 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343955+0000", 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4210000000000002e-06 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343959+0000", 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6389999999999998e-06 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343962+0000", 2024-09-16T03:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.632e-06 2024-09-16T03:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445398+0000", 2024-09-16T03:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101436019 2024-09-16T03:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445418+0000", 2024-09-16T03:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0815e-05 2024-09-16T03:37:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446223+0000", 2024-09-16T03:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00080508100000000003 2024-09-16T03:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446264+0000", 2024-09-16T03:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0352999999999998e-05 2024-09-16T03:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:269 4.6 e40/32 4:64ce87bc:::smithi15736696-20:head v 40'15, mlcod=40'13)", 2024-09-16T03:37:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.196770+0000", 2024-09-16T03:37:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.873262489, 2024-09-16T03:37:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20089027500000001, 2024-09-16T03:37:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196770+0000", 2024-09-16T03:37:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196770+0000", 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196771+0000", 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.288e-06 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196777+0000", 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7169999999999996e-06 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196787+0000", 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.028e-05 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.296854+0000", 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100066848 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.397050+0000", 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100195387 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.397082+0000", 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1977000000000002e-05 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.397619+0000", 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000537734 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.397660+0000", 2024-09-16T03:37:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1044000000000002e-05 2024-09-16T03:37:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:7 3.1 3:a961e47d:::smithi15736696-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.302964+0000", 2024-09-16T03:37:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.767068194, 2024-09-16T03:37:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19676334100000001, 2024-09-16T03:37:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302964+0000", 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302964+0000", 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302966+0000", 2024-09-16T03:37:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5889999999999999e-06 2024-09-16T03:37:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302972+0000", 2024-09-16T03:37:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.682e-06 2024-09-16T03:37:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302983+0000", 2024-09-16T03:37:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1147e-05 2024-09-16T03:37:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403076+0000", 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100093178 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403178+0000", 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101525 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403248+0000", 2024-09-16T03:37:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0548000000000003e-05 2024-09-16T03:37:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403370+0000", 2024-09-16T03:37:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012235100000000001 2024-09-16T03:37:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403908+0000", 2024-09-16T03:37:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053726999999999996 2024-09-16T03:37:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.404170+0000", 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00026224899999999999 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499671+0000", 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.095501213000000001 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499704+0000", 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2641000000000001e-05 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499728+0000", 2024-09-16T03:37:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3947999999999998e-05 2024-09-16T03:37:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.4.22:3 3.1 3:bffb5658:::smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.499594+0000", 2024-09-16T03:37:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.570439041, 2024-09-16T03:37:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.106172458, 2024-09-16T03:37:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499594+0000", 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499594+0000", 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499595+0000", 2024-09-16T03:37:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.361e-06 2024-09-16T03:37:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499600+0000", 2024-09-16T03:37:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.429e-06 2024-09-16T03:37:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499610+0000", 2024-09-16T03:37:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1460000000000002e-06 2024-09-16T03:37:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499617+0000", 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7919999999999993e-06 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.504406+0000", 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0047886129999999997 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.504474+0000", 2024-09-16T03:37:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7587000000000002e-05 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.504600+0000", 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012637799999999999 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.505058+0000", 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000457579 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.605493+0000", 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100435943 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.605677+0000", 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018323999999999999 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.605725+0000", 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8748999999999997e-05 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.605766+0000", 2024-09-16T03:37:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0640999999999999e-05 2024-09-16T03:37:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:17 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.086208+0000", 2024-09-16T03:37:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.9838243880000004, 2024-09-16T03:37:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20243159599999999, 2024-09-16T03:37:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086208+0000", 2024-09-16T03:37:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086208+0000", 2024-09-16T03:37:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086210+0000", 2024-09-16T03:37:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.058e-06 2024-09-16T03:37:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086216+0000", 2024-09-16T03:37:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1399999999999997e-06 2024-09-16T03:37:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086226+0000", 2024-09-16T03:37:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4110000000000002e-06 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086235+0000", 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4420000000000004e-06 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.186482+0000", 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10024678200000001 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.186574+0000", 2024-09-16T03:37:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2035999999999998e-05 2024-09-16T03:37:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.186725+0000", 2024-09-16T03:37:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150629 2024-09-16T03:37:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.187384+0000", 2024-09-16T03:37:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065916599999999998 2024-09-16T03:37:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.187823+0000", 2024-09-16T03:37:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000439018 2024-09-16T03:37:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.288509+0000", 2024-09-16T03:37:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100685624 2024-09-16T03:37:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.288611+0000", 2024-09-16T03:37:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102155 2024-09-16T03:37:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.288640+0000", 2024-09-16T03:37:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9135000000000001e-05 2024-09-16T03:37:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:19 3.1 3:9aba8983:::smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.6516009030000003, 2024-09-16T03:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30153978799999998, 2024-09-16T03:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:37:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418452+0000", 2024-09-16T03:37:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0363000000000001e-05 2024-09-16T03:37:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418470+0000", 2024-09-16T03:37:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7506000000000001e-05 2024-09-16T03:37:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418504+0000", 2024-09-16T03:37:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4542999999999998e-05 2024-09-16T03:37:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418522+0000", 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8070000000000001e-05 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418662+0000", 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013946299999999999 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418826+0000", 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016469299999999999 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418989+0000", 2024-09-16T03:37:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000162759 2024-09-16T03:37:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.419750+0000", 2024-09-16T03:37:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076104299999999998 2024-09-16T03:37:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.419865+0000", 2024-09-16T03:37:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114338 2024-09-16T03:37:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719876+0000", 2024-09-16T03:37:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30001127300000002 2024-09-16T03:37:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719920+0000", 2024-09-16T03:37:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4549999999999999e-05 2024-09-16T03:37:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719972+0000", 2024-09-16T03:37:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1187000000000003e-05 2024-09-16T03:37:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:10 3.1 3:8f38b759:::smithi15736696-6 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:37:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.5621682650000004, 2024-09-16T03:37:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.21220708699999999, 2024-09-16T03:37:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507866+0000", 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9999999999999999e-06 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507873+0000", 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7900000000000002e-06 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507888+0000", 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4554e-05 2024-09-16T03:37:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507896+0000", 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1130000000000001e-06 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507967+0000", 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0808000000000004e-05 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508028+0000", 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1325e-05 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508146+0000", 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000117922 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508722+0000", 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000575779 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.609567+0000", 2024-09-16T03:37:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100845448 2024-09-16T03:37:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720013+0000", 2024-09-16T03:37:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.110445644 2024-09-16T03:37:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720045+0000", 2024-09-16T03:37:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2186e-05 2024-09-16T03:37:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720071+0000", 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6517999999999999e-05 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:11 3.1 3:adc537a6:::smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.5568250349999992, 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20740488100000001, 2024-09-16T03:37:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513209+0000", 2024-09-16T03:37:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7439999999999999e-06 2024-09-16T03:37:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513216+0000", 2024-09-16T03:37:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9890000000000003e-06 2024-09-16T03:37:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513230+0000", 2024-09-16T03:37:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3263e-05 2024-09-16T03:37:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513236+0000", 2024-09-16T03:37:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7270000000000001e-06 2024-09-16T03:37:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513297+0000", 2024-09-16T03:37:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0767999999999998e-05 2024-09-16T03:37:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513375+0000", 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8014000000000002e-05 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513529+0000", 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015342900000000001 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.514258+0000", 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072896799999999998 2024-09-16T03:37:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.709867+0000", 2024-09-16T03:37:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.195609016 2024-09-16T03:37:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720546+0000", 2024-09-16T03:37:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010679905999999999 2024-09-16T03:37:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720583+0000", 2024-09-16T03:37:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6659999999999998e-05 2024-09-16T03:37:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720613+0000", 2024-09-16T03:37:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9397e-05 2024-09-16T03:37:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:18 3.1 3:9f9f4ec5:::smithi15736696-22 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.104882+0000", 2024-09-16T03:37:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.9651502729999999, 2024-09-16T03:37:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20218581599999999, 2024-09-16T03:37:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104882+0000", 2024-09-16T03:37:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104882+0000", 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104885+0000", 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7760000000000002e-06 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104895+0000", 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9909999999999997e-06 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104911+0000", 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5654999999999998e-05 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104927+0000", 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6620999999999999e-05 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104985+0000", 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.787e-05 2024-09-16T03:37:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105043+0000", 2024-09-16T03:37:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7720999999999998e-05 2024-09-16T03:37:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105174+0000", 2024-09-16T03:37:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013075599999999999 2024-09-16T03:37:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105846+0000", 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067215899999999997 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206477+0000", 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10063103299999999 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307001+0000", 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10052391200000001 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307035+0000", 2024-09-16T03:37:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3674999999999999e-05 2024-09-16T03:37:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307068+0000", 2024-09-16T03:37:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3646999999999998e-05 2024-09-16T03:37:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:19 3.1 3:aca5d7ac:::smithi15736696-26 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.9649680820000004, 2024-09-16T03:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264779099999999, 2024-09-16T03:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105068+0000", 2024-09-16T03:37:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.751e-06 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105072+0000", 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5730000000000001e-06 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205143+0000", 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100070665 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206515+0000", 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013715789999999999 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206570+0000", 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5049000000000001e-05 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206695+0000", 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012521599999999999 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307091+0000", 2024-09-16T03:37:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10039582 2024-09-16T03:37:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307128+0000", 2024-09-16T03:37:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7290999999999999e-05 2024-09-16T03:37:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307656+0000", 2024-09-16T03:37:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00052813199999999995 2024-09-16T03:37:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307689+0000", 2024-09-16T03:37:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3056000000000003e-05 2024-09-16T03:37:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307712+0000", 2024-09-16T03:37:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3099999999999999e-05 2024-09-16T03:37:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop_reply(osd.10.22:18 3.1 e45/31 ondisk, result = 0)", 2024-09-16T03:37:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.206414+0000", 2024-09-16T03:37:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.8636185830000001, 2024-09-16T03:37:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100665984, 2024-09-16T03:37:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206414+0000", 2024-09-16T03:37:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206414+0000", 2024-09-16T03:37:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206416+0000", 2024-09-16T03:37:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6479999999999999e-06 2024-09-16T03:37:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206421+0000", 2024-09-16T03:37:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7049999999999996e-06 2024-09-16T03:37:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206440+0000", 2024-09-16T03:37:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8689e-05 2024-09-16T03:37:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206448+0000", 2024-09-16T03:37:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5460000000000003e-06 2024-09-16T03:37:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306993+0000", 2024-09-16T03:37:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100545391 2024-09-16T03:37:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306998+0000", 2024-09-16T03:37:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5580000000000002e-06 2024-09-16T03:37:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307080+0000", 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2447000000000005e-05 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:22 3.1 3:9aba8983:::smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.207469+0000", 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.8625641139999995, 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10211275, 2024-09-16T03:37:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.207469+0000", 2024-09-16T03:37:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.207469+0000", 2024-09-16T03:37:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.207470+0000", 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5069999999999999e-06 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.207475+0000", 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9849999999999997e-06 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.207484+0000", 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4509999999999998e-06 2024-09-16T03:37:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.207490+0000", 2024-09-16T03:37:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4870000000000002e-06 2024-09-16T03:37:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307150+0000", 2024-09-16T03:37:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099659532999999995 2024-09-16T03:37:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307212+0000", 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2577999999999996e-05 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307329+0000", 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116841 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308257+0000", 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00092779699999999995 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308485+0000", 2024-09-16T03:37:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000228013 2024-09-16T03:37:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309524+0000", 2024-09-16T03:37:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010396680000000001 2024-09-16T03:37:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309559+0000", 2024-09-16T03:37:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4557999999999997e-05 2024-09-16T03:37:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309581+0000", 2024-09-16T03:37:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2331999999999999e-05 2024-09-16T03:37:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:419 4.6 e45/32 4:64ce87bc:::smithi15736696-20:head v 45'18, mlcod=42'16)", 2024-09-16T03:37:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.216745+0000", 2024-09-16T03:37:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.8532878559999997, 2024-09-16T03:37:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101185523, 2024-09-16T03:37:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.216745+0000", 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.216745+0000", 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.216746+0000", 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4559999999999999e-06 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.216754+0000", 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7749999999999993e-06 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.216782+0000", 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7668e-05 2024-09-16T03:37:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.316908+0000", 2024-09-16T03:37:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100126431 2024-09-16T03:37:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.317045+0000", 2024-09-16T03:37:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013727999999999999 2024-09-16T03:37:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.317119+0000", 2024-09-16T03:37:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3758999999999996e-05 2024-09-16T03:37:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.317891+0000", 2024-09-16T03:37:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00077201400000000001 2024-09-16T03:37:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.317930+0000", 2024-09-16T03:37:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9140000000000001e-05 2024-09-16T03:37:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.4.22:7 3.1 3:bffb5658:::smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.621946+0000", 2024-09-16T03:37:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.4480870770000003, 2024-09-16T03:37:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10152401599999999, 2024-09-16T03:37:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.621946+0000", 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.621946+0000", 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.621948+0000", 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.103e-06 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.621954+0000", 2024-09-16T03:37:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3749999999999999e-06 2024-09-16T03:37:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.621966+0000", 2024-09-16T03:37:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1538e-05 2024-09-16T03:37:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.621974+0000", 2024-09-16T03:37:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2640000000000008e-06 2024-09-16T03:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.622034+0000", 2024-09-16T03:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0554000000000003e-05 2024-09-16T03:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.622103+0000", 2024-09-16T03:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8450999999999999e-05 2024-09-16T03:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.622226+0000", 2024-09-16T03:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012333900000000001 2024-09-16T03:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.622902+0000", 2024-09-16T03:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000675941 2024-09-16T03:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.623023+0000", 2024-09-16T03:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012054500000000001 2024-09-16T03:37:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.723254+0000", 2024-09-16T03:37:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10023154099999999 2024-09-16T03:37:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.723398+0000", 2024-09-16T03:37:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000144024 2024-09-16T03:37:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.723470+0000", 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1340999999999993e-05 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:15 3.1 3:a961e47d:::smithi15736696-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.922299+0000", 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.1477337609999996, 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101304141, 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922299+0000", 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922299+0000", 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922301+0000", 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.689e-06 2024-09-16T03:37:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922306+0000", 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0170000000000004e-06 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922317+0000", 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1318000000000001e-05 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922340+0000", 2024-09-16T03:37:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3510999999999999e-05 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922387+0000", 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6368000000000001e-05 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922455+0000", 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8112000000000001e-05 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922564+0000", 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000109516 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.923085+0000", 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00052038900000000001 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.923529+0000", 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00044453199999999998 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.023533+0000", 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100003208 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.023571+0000", 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8090999999999998e-05 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.023603+0000", 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.239e-05 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.0:84 4.7 e46/32 4:ed874e32:::smithi15736696-14:head v 46'21, mlcod=46'19)", 2024-09-16T03:37:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.063250+0000", 2024-09-16T03:37:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.006782182, 2024-09-16T03:37:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101936576, 2024-09-16T03:37:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.063250+0000", 2024-09-16T03:37:26.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.063250+0000", 2024-09-16T03:37:26.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.063253+0000", 2024-09-16T03:37:26.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7049999999999999e-06 2024-09-16T03:37:26.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.064627+0000", 2024-09-16T03:37:26.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001373686 2024-09-16T03:37:26.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.064657+0000", 2024-09-16T03:37:26.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0159999999999999e-05 2024-09-16T03:37:26.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.064716+0000", 2024-09-16T03:37:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9259999999999998e-05 2024-09-16T03:37:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.066420+0000", 2024-09-16T03:37:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017040169999999999 2024-09-16T03:37:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.066450+0000", 2024-09-16T03:37:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9397999999999999e-05 2024-09-16T03:37:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.165147+0000", 2024-09-16T03:37:26.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098697464999999998 2024-09-16T03:37:26.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.165187+0000", 2024-09-16T03:37:26.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9885000000000002e-05 2024-09-16T03:37:26.177 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.177 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.177 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:779 3.1 3:abdaa5c9:::2782d12a88f9842f7f754fca32c4222cf10fb780cd136830d32faaa700e687ba:head [write 0~45056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.58345258700000002, 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30397390600000002, 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 779 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486583+0000", 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.762e-06 2024-09-16T03:37:26.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486649+0000", 2024-09-16T03:37:26.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6143999999999996e-05 2024-09-16T03:37:26.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486669+0000", 2024-09-16T03:37:26.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9837e-05 2024-09-16T03:37:26.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486681+0000", 2024-09-16T03:37:26.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2194999999999999e-05 2024-09-16T03:37:26.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486746+0000", 2024-09-16T03:37:26.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5480000000000003e-05 2024-09-16T03:37:26.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-09-16T03:37:26.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486752+0000", 2024-09-16T03:37:26.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4659999999999998e-06 2024-09-16T03:37:26.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588316+0000", 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10156393399999999 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588539+0000", 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022308300000000001 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588647+0000", 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000108047 2024-09-16T03:37:26.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.589516+0000", 2024-09-16T03:37:26.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086926899999999999 2024-09-16T03:37:26.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.789989+0000", 2024-09-16T03:37:26.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20047296100000001 2024-09-16T03:37:26.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790478+0000", 2024-09-16T03:37:26.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048882099999999996 2024-09-16T03:37:26.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790518+0000", 2024-09-16T03:37:26.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0343000000000003e-05 2024-09-16T03:37:26.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790554+0000", 2024-09-16T03:37:26.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5564e-05 2024-09-16T03:37:26.184 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.184 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.185 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.185 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.185 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.185 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:26.186 INFO:tasks.rados.rados.0.smithi157.stdout:617: finishing set_chunk to oid 33 2024-09-16T03:37:26.186 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 33 set_chunk returned error code -95 offset: 1868288 length: 561152 2024-09-16T03:37:26.186 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 0 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T03:37:26.187 INFO:tasks.rados.rados.0.smithi157.stdout:613: finishing write tid 1 to smithi15736696-50 2024-09-16T03:37:26.187 INFO:tasks.rados.rados.0.smithi157.stdout:623: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-16T03:37:26.187 INFO:tasks.rados.rados.0.smithi157.stdout:627: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:26.187 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:26.184+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:26.187 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:37:26.204 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 90157 2024-09-16T03:37:26.231 INFO:tasks.rados.rados.0.smithi157.stdout:624: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T03:37:26.256 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 947011 2024-09-16T03:37:26.288 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:26.287+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:26.291 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:26.292 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:26.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.292 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:801 4.0 4:07ad0560:::smithi15736696-25:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:26.292 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:37:26.292 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.28474952599999998, 2024-09-16T03:37:26.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.28477121399999999, 2024-09-16T03:37:26.292 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.292 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.292 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:26.292 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.292 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.293 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.293 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 801 2024-09-16T03:37:26.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.293 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:37:26.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:37:26.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.294 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006084+0000", 2024-09-16T03:37:26.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.322e-06 2024-09-16T03:37:26.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.294 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006088+0000", 2024-09-16T03:37:26.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000001e-06 2024-09-16T03:37:26.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006098+0000", 2024-09-16T03:37:26.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0186999999999999e-05 2024-09-16T03:37:26.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006104+0000", 2024-09-16T03:37:26.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.271e-06 2024-09-16T03:37:26.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.199952+0000", 2024-09-16T03:37:26.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.193847935 2024-09-16T03:37:26.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200027+0000", 2024-09-16T03:37:26.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4987000000000006e-05 2024-09-16T03:37:26.296 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.296 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.296 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.296 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:784 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=63b,truncate 2840350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:26.296 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:37:26.297 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.48424334899999999, 2024-09-16T03:37:26.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.48431354300000001, 2024-09-16T03:37:26.297 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.297 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.297 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:37:26.297 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.297 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.297 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.297 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 784 2024-09-16T03:37:26.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.297 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.298 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.298 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:37:26.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.298 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.298 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.298 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:37:26.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.298 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.298 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.298 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806590+0000", 2024-09-16T03:37:26.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6000000000000003e-07 2024-09-16T03:37:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806593+0000", 2024-09-16T03:37:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6649999999999999e-06 2024-09-16T03:37:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806597+0000", 2024-09-16T03:37:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.245e-06 2024-09-16T03:37:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806600+0000", 2024-09-16T03:37:26.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.23e-06 2024-09-16T03:37:26.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.300 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099381+0000", 2024-09-16T03:37:26.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29278119400000002 2024-09-16T03:37:26.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099438+0000", 2024-09-16T03:37:26.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6198e-05 2024-09-16T03:37:26.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:26.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099527+0000", 2024-09-16T03:37:26.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9505000000000004e-05 2024-09-16T03:37:26.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200775+0000", 2024-09-16T03:37:26.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101247904 2024-09-16T03:37:26.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200889+0000", 2024-09-16T03:37:26.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114301 2024-09-16T03:37:26.302 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.302 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.302 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.302 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:785 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:26.302 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.48422742400000002, 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.48433543499999998, 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "waiting for rw locks", 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 785 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:37:26.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:37:26.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806606+0000", 2024-09-16T03:37:26.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:37:26.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806607+0000", 2024-09-16T03:37:26.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.424e-06 2024-09-16T03:37:26.305 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806611+0000", 2024-09-16T03:37:26.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1080000000000003e-06 2024-09-16T03:37:26.305 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806614+0000", 2024-09-16T03:37:26.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.067e-06 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099813+0000", 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29319853699999998 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099858+0000", 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5667999999999997e-05 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.306 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.307 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2024-09-16T03:37:26.307 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 3 2024-09-16T03:37:26.307 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:26.307 INFO:tasks.rados.rados.0.smithi157.stdout:618: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T03:37:26.308 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:37:26.378 INFO:tasks.rados.rados.0.smithi157.stdout:613: finishing write tid 2 to smithi15736696-50 2024-09-16T03:37:26.378 INFO:tasks.rados.rados.0.smithi157.stdout:626: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T03:37:26.390 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:26.388+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:26.397 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:26.412 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:26.412 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:26.412 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:26.412 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:26.412 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:26.421 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:37:26.422 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:26.447 INFO:tasks.rados.rados.0.smithi157.stdout:622: finishing write tid 1 to smithi15736696-11 2024-09-16T03:37:26.447 INFO:tasks.rados.rados.0.smithi157.stdout:613: finishing write tid 3 to smithi15736696-50 2024-09-16T03:37:26.447 INFO:tasks.rados.rados.0.smithi157.stdout:622: finishing write tid 2 to smithi15736696-11 2024-09-16T03:37:26.447 INFO:tasks.rados.rados.0.smithi157.stdout:622: finishing write tid 3 to smithi15736696-11 2024-09-16T03:37:26.447 INFO:tasks.rados.rados.0.smithi157.stdout:622: finishing write tid 4 to smithi15736696-11 2024-09-16T03:37:26.447 INFO:tasks.rados.rados.0.smithi157.stdout:622: finishing write tid 5 to smithi15736696-11 2024-09-16T03:37:26.448 INFO:tasks.rados.rados.0.smithi157.stdout:622: oid 11 updating version 0 to 24 2024-09-16T03:37:26.448 INFO:tasks.rados.rados.0.smithi157.stdout:622: oid 11 updating version 24 to 26 2024-09-16T03:37:26.448 INFO:tasks.rados.rados.0.smithi157.stdout:622: oid 11 version 26 is already newer than 25 2024-09-16T03:37:26.448 INFO:tasks.rados.rados.0.smithi157.stdout:622: oid 11 updating version 26 to 27 2024-09-16T03:37:26.448 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 27 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-16T03:37:26.448 INFO:tasks.rados.rados.0.smithi157.stdout:622: left oid 11 (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:37:26.448 INFO:tasks.rados.rados.0.smithi157.stdout:616: got (0) Success 2024-09-16T03:37:26.448 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:26.448 INFO:tasks.rados.rados.0.smithi157.stdout:621: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-16T03:37:26.470 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1240173 2024-09-16T03:37:26.492 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:26.489+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:26.515 INFO:tasks.rados.rados.0.smithi157.stdout:625: finishing write tid 1 to smithi15736696-13 2024-09-16T03:37:26.515 INFO:tasks.rados.rados.0.smithi157.stdout:625: finishing write tid 2 to smithi15736696-13 2024-09-16T03:37:26.515 INFO:tasks.rados.rados.0.smithi157.stdout:625: finishing write tid 3 to smithi15736696-13 2024-09-16T03:37:26.515 INFO:tasks.rados.rados.0.smithi157.stdout:625: finishing write tid 4 to smithi15736696-13 2024-09-16T03:37:26.515 INFO:tasks.rados.rados.0.smithi157.stdout:625: oid 13 updating version 0 to 21 2024-09-16T03:37:26.515 INFO:tasks.rados.rados.0.smithi157.stdout:625: oid 13 updating version 21 to 23 2024-09-16T03:37:26.515 INFO:tasks.rados.rados.0.smithi157.stdout:625: oid 13 version 23 is already newer than 22 2024-09-16T03:37:26.516 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 23 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-09-16T03:37:26.516 INFO:tasks.rados.rados.0.smithi157.stdout:625: left oid 13 (ObjNum 53 snap 0 seq_num 53) 2024-09-16T03:37:26.516 INFO:tasks.rados.rados.0.smithi157.stdout:619: finishing write tid 1 to smithi15736696-44 2024-09-16T03:37:26.516 INFO:tasks.rados.rados.0.smithi157.stdout:619: finishing write tid 2 to smithi15736696-44 2024-09-16T03:37:26.516 INFO:tasks.rados.rados.0.smithi157.stdout:613: finishing write tid 4 to smithi15736696-50 2024-09-16T03:37:26.516 INFO:tasks.rados.rados.0.smithi157.stdout:628: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-09-16T03:37:26.520 INFO:tasks.rados.rados.0.smithi157.stdout:628: writing smithi15736696-46 from 507578 to 985495 tid 1 2024-09-16T03:37:26.526 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:26.527 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:26.527 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:26.527 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:26.527 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.527 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:30 4.f 4:f0d14c71:::smithi15736696-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:26.527 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.042719+0000", 2024-09-16T03:37:26.527 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.481388149000001, 2024-09-16T03:37:26.527 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.46375592199999999, 2024-09-16T03:37:26.527 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.527 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.527 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.528 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.528 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.528 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.528 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 30 2024-09-16T03:37:26.528 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.528 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.528 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.528 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.528 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.042719+0000", 2024-09-16T03:37:26.528 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.528 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.528 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.528 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.529 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.042719+0000", 2024-09-16T03:37:26.529 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.529 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.529 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.529 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.042721+0000", 2024-09-16T03:37:26.529 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8250000000000001e-06 2024-09-16T03:37:26.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.529 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.529 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.529 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043034+0000", 2024-09-16T03:37:26.529 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00031377599999999999 2024-09-16T03:37:26.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043046+0000", 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1301000000000001e-05 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043058+0000", 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2670999999999999e-05 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043109+0000", 2024-09-16T03:37:26.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1107999999999999e-05 2024-09-16T03:37:26.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.531 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.531 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043169+0000", 2024-09-16T03:37:26.531 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9929999999999997e-05 2024-09-16T03:37:26.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.531 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:37:26.531 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043268+0000", 2024-09-16T03:37:26.531 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8906000000000005e-05 2024-09-16T03:37:26.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.531 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.532 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.045851+0000", 2024-09-16T03:37:26.532 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002582553 2024-09-16T03:37:26.532 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.532 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.532 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.532 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243119+0000", 2024-09-16T03:37:26.532 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19726857 2024-09-16T03:37:26.532 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.532 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.532 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.532 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.506327+0000", 2024-09-16T03:37:26.532 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.26320757 2024-09-16T03:37:26.532 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.533 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.533 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.533 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.506405+0000", 2024-09-16T03:37:26.533 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7874999999999996e-05 2024-09-16T03:37:26.533 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.533 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.533 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.533 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.506475+0000", 2024-09-16T03:37:26.533 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9837000000000002e-05 2024-09-16T03:37:26.533 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.533 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.533 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.533 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.534 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.534 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:31 4.f 4:f0d14c71:::smithi15736696-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:26.534 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.043076+0000", 2024-09-16T03:37:26.534 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.481030427, 2024-09-16T03:37:26.534 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50359721000000002, 2024-09-16T03:37:26.534 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.534 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.534 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.534 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.534 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.534 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.534 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 31 2024-09-16T03:37:26.534 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043076+0000", 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043076+0000", 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043078+0000", 2024-09-16T03:37:26.536 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.189e-06 2024-09-16T03:37:26.536 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.536 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.536 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.536 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043084+0000", 2024-09-16T03:37:26.536 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.896e-06 2024-09-16T03:37:26.536 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.536 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.536 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.536 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043091+0000", 2024-09-16T03:37:26.536 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9770000000000003e-06 2024-09-16T03:37:26.536 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.536 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.537 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.537 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143160+0000", 2024-09-16T03:37:26.537 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10006909 2024-09-16T03:37:26.537 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.537 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.537 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.537 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143222+0000", 2024-09-16T03:37:26.537 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2049000000000002e-05 2024-09-16T03:37:26.537 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.537 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.537 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.537 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143324+0000", 2024-09-16T03:37:26.537 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102737 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143485+0000", 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016040100000000001 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.147043+0000", 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0035581609999999998 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.538 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.244108+0000", 2024-09-16T03:37:26.539 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.097064967000000002 2024-09-16T03:37:26.539 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.539 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.539 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.546541+0000", 2024-09-16T03:37:26.539 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30243345199999999 2024-09-16T03:37:26.539 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.539 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.539 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.546605+0000", 2024-09-16T03:37:26.539 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4039000000000001e-05 2024-09-16T03:37:26.539 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.539 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.540 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.540 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.546674+0000", 2024-09-16T03:37:26.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8251999999999996e-05 2024-09-16T03:37:26.540 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.540 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.540 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.540 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.540 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.540 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:32 4.f 4:f0d14c71:::smithi15736696-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:26.540 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143177+0000", 2024-09-16T03:37:26.540 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.380929536, 2024-09-16T03:37:26.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40375996400000003, 2024-09-16T03:37:26.540 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.541 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.541 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:26.541 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.541 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.541 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.541 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 32 2024-09-16T03:37:26.541 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.541 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.541 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.541 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.541 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143177+0000", 2024-09-16T03:37:26.541 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.541 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.542 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.542 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143177+0000", 2024-09-16T03:37:26.542 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.542 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.542 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.542 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143179+0000", 2024-09-16T03:37:26.542 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5e-06 2024-09-16T03:37:26.542 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.542 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.542 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143187+0000", 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9990000000000008e-06 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143192+0000", 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5480000000000001e-06 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143196+0000", 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0949999999999998e-06 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.543 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.544 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143813+0000", 2024-09-16T03:37:26.544 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061680999999999997 2024-09-16T03:37:26.544 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.544 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.544 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:26.544 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143877+0000", 2024-09-16T03:37:26.544 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3548999999999998e-05 2024-09-16T03:37:26.544 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.544 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.544 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.544 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.546720+0000", 2024-09-16T03:37:26.544 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.402843589 2024-09-16T03:37:26.544 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.546818+0000", 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7715999999999998e-05 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.546937+0000", 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000119158 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:37 4.f 4:fa163101:::smithi15736696-10:head [write 551263~793594,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143223+0000", 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.380884084, 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40383498299999998, 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 37 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.546 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143223+0000", 2024-09-16T03:37:26.547 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.547 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143223+0000", 2024-09-16T03:37:26.547 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.547 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143223+0000", 2024-09-16T03:37:26.547 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2699999999999999e-07 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144026+0000", 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00080251300000000003 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144037+0000", 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1372000000000001e-05 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.548 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144045+0000", 2024-09-16T03:37:26.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0029999999999994e-06 2024-09-16T03:37:26.549 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.549 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.549 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144095+0000", 2024-09-16T03:37:26.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9684000000000001e-05 2024-09-16T03:37:26.549 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.549 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.549 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144197+0000", 2024-09-16T03:37:26.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102011 2024-09-16T03:37:26.549 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.549 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144308+0000", 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000110826 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.149794+0000", 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0054864609999999998 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.246255+0000", 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.096460876000000001 2024-09-16T03:37:26.550 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.546976+0000", 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30072067499999999 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.547024+0000", 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8473000000000002e-05 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.547058+0000", 2024-09-16T03:37:26.551 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3461999999999999e-05 2024-09-16T03:37:26.552 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.552 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.552 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.552 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.552 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:38 4.f 4:fa163101:::smithi15736696-10:head [write 1893632~257631] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:26.552 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144061+0000", 2024-09-16T03:37:26.552 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.380045795000001, 2024-09-16T03:37:26.552 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40322499899999997, 2024-09-16T03:37:26.552 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.552 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.552 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.552 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 38 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144061+0000", 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144061+0000", 2024-09-16T03:37:26.553 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.554 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.554 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.554 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.554 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144062+0000", 2024-09-16T03:37:26.554 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.24e-06 2024-09-16T03:37:26.554 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.554 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.554 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.554 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144283+0000", 2024-09-16T03:37:26.554 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022108499999999999 2024-09-16T03:37:26.554 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.554 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.554 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144292+0000", 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.064e-06 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144299+0000", 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1109999999999996e-06 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145960+0000", 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016615110000000001 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.555 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.556 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146023+0000", 2024-09-16T03:37:26.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2775999999999997e-05 2024-09-16T03:37:26.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:37:26.556 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146113+0000", 2024-09-16T03:37:26.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9946999999999998e-05 2024-09-16T03:37:26.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.556 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.149799+0000", 2024-09-16T03:37:26.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036857610000000001 2024-09-16T03:37:26.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.246416+0000", 2024-09-16T03:37:26.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.096617465 2024-09-16T03:37:26.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.547210+0000", 2024-09-16T03:37:26.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30079376800000002 2024-09-16T03:37:26.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.547253+0000", 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3040999999999997e-05 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.547286+0000", 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3229999999999999e-05 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:39 4.f 4:fa163101:::smithi15736696-10:head [setxattr _header (56) in=7b,truncate 2151263] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:26.558 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144309+0000", 2024-09-16T03:37:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.379797656999999, 2024-09-16T03:37:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40324600900000002, 2024-09-16T03:37:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 39 2024-09-16T03:37:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.559 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144309+0000", 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144309+0000", 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144310+0000", 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1399999999999996e-07 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.560 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144314+0000", 2024-09-16T03:37:26.561 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6389999999999998e-06 2024-09-16T03:37:26.561 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144319+0000", 2024-09-16T03:37:26.561 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2490000000000001e-06 2024-09-16T03:37:26.561 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144325+0000", 2024-09-16T03:37:26.561 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8470000000000001e-06 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146713+0000", 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023881660000000002 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146765+0000", 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2617000000000001e-05 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:37:26.562 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146851+0000", 2024-09-16T03:37:26.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5857000000000001e-05 2024-09-16T03:37:26.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.563 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.563 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.149801+0000", 2024-09-16T03:37:26.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0029500110000000002 2024-09-16T03:37:26.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.563 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.563 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.346608+0000", 2024-09-16T03:37:26.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19680705300000001 2024-09-16T03:37:26.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.564 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.547462+0000", 2024-09-16T03:37:26.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20085319800000001 2024-09-16T03:37:26.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.564 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.564 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.547504+0000", 2024-09-16T03:37:26.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2216999999999999e-05 2024-09-16T03:37:26.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.564 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.564 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.547555+0000", 2024-09-16T03:37:26.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1440999999999998e-05 2024-09-16T03:37:26.565 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.565 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.565 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.565 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:40 4.f 4:fa163101:::smithi15736696-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:26.565 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144331+0000", 2024-09-16T03:37:26.565 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.379776117, 2024-09-16T03:37:26.565 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40354157699999998, 2024-09-16T03:37:26.565 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.565 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.565 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:26.565 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 40 2024-09-16T03:37:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144331+0000", 2024-09-16T03:37:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144331+0000", 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144331+0000", 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7099999999999997e-07 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144333+0000", 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0810000000000001e-06 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.568 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144336+0000", 2024-09-16T03:37:26.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2289999999999999e-06 2024-09-16T03:37:26.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.568 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.568 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.568 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144341+0000", 2024-09-16T03:37:26.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7319999999999998e-06 2024-09-16T03:37:26.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.568 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.568 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.568 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.147076+0000", 2024-09-16T03:37:26.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027350389999999999 2024-09-16T03:37:26.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.569 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:26.569 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.147123+0000", 2024-09-16T03:37:26.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7234999999999998e-05 2024-09-16T03:37:26.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.569 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.569 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.547718+0000", 2024-09-16T03:37:26.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40059412500000002 2024-09-16T03:37:26.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.569 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.569 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.547787+0000", 2024-09-16T03:37:26.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9400000000000006e-05 2024-09-16T03:37:26.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.570 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.570 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.547872+0000", 2024-09-16T03:37:26.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5364999999999993e-05 2024-09-16T03:37:26.570 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.570 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.570 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.570 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:56 4.7 4:ed874e32:::smithi15736696-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:26.570 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145066+0000", 2024-09-16T03:37:26.571 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.379041364000001, 2024-09-16T03:37:26.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40298087300000002, 2024-09-16T03:37:26.571 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.571 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.571 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:26.571 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.571 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.571 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.571 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 56 2024-09-16T03:37:26.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.571 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145066+0000", 2024-09-16T03:37:26.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.572 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145066+0000", 2024-09-16T03:37:26.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.572 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145066+0000", 2024-09-16T03:37:26.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0199999999999998e-07 2024-09-16T03:37:26.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145071+0000", 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4840000000000003e-06 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145075+0000", 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.207e-06 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145077+0000", 2024-09-16T03:37:26.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7329999999999999e-06 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.149723+0000", 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046454119999999998 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.149787+0000", 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4070000000000002e-05 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.547900+0000", 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39811357200000003 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.547967+0000", 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6475000000000003e-05 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.548046+0000", 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9617999999999996e-05 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.575 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.576 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:66 4.0 4:023abada:::smithi15736696-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:26.576 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.200996+0000", 2024-09-16T03:37:26.576 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.323110813, 2024-09-16T03:37:26.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50798839299999998, 2024-09-16T03:37:26.576 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.576 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.576 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.576 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.576 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.576 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.576 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 66 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.200996+0000", 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.200996+0000", 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.200998+0000", 2024-09-16T03:37:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5179999999999999e-06 2024-09-16T03:37:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.201330+0000", 2024-09-16T03:37:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000332672 2024-09-16T03:37:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.201340+0000", 2024-09-16T03:37:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8099999999999992e-06 2024-09-16T03:37:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301446+0000", 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100105877 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301543+0000", 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7058000000000005e-05 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301628+0000", 2024-09-16T03:37:26.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5078000000000001e-05 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301770+0000", 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014212400000000001 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.303247+0000", 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001476595 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.580 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.606126+0000", 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30287937799999998 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.708902+0000", 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102775402 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.708938+0000", 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5947e-05 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.708984+0000", 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6934000000000002e-05 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:67 4.0 4:023abada:::smithi15736696-17:head [setxattr _header (56) in=7b,truncate 2268912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.301470+0000", 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.222636687, 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40761297299999999, 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.582 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 67 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301470+0000", 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301470+0000", 2024-09-16T03:37:26.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301472+0000", 2024-09-16T03:37:26.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2060000000000001e-06 2024-09-16T03:37:26.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301483+0000", 2024-09-16T03:37:26.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0974e-05 2024-09-16T03:37:26.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301494+0000", 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1127e-05 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301503+0000", 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9910000000000008e-06 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302641+0000", 2024-09-16T03:37:26.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011377200000000001 2024-09-16T03:37:26.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302711+0000", 2024-09-16T03:37:26.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9707000000000002e-05 2024-09-16T03:37:26.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:26.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302829+0000", 2024-09-16T03:37:26.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000117716 2024-09-16T03:37:26.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.303441+0000", 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061266900000000004 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.606160+0000", 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302719185 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.709017+0000", 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102856657 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.588 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.588 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.709047+0000", 2024-09-16T03:37:26.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9444000000000001e-05 2024-09-16T03:37:26.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.588 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.588 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.709083+0000", 2024-09-16T03:37:26.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6576999999999999e-05 2024-09-16T03:37:26.588 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.588 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.588 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.589 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:68 4.0 4:023abada:::smithi15736696-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:26.589 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.301513+0000", 2024-09-16T03:37:26.589 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.222593863, 2024-09-16T03:37:26.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40773087600000002, 2024-09-16T03:37:26.589 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.589 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.589 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:26.589 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.589 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.589 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.589 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 68 2024-09-16T03:37:26.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.589 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301513+0000", 2024-09-16T03:37:26.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301513+0000", 2024-09-16T03:37:26.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301514+0000", 2024-09-16T03:37:26.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7599999999999997e-07 2024-09-16T03:37:26.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301517+0000", 2024-09-16T03:37:26.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0750000000000002e-06 2024-09-16T03:37:26.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301523+0000", 2024-09-16T03:37:26.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8699999999999997e-06 2024-09-16T03:37:26.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301528+0000", 2024-09-16T03:37:26.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5330000000000002e-06 2024-09-16T03:37:26.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.303115+0000", 2024-09-16T03:37:26.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001586856 2024-09-16T03:37:26.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:26.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.303226+0000", 2024-09-16T03:37:26.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000111049 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.709116+0000", 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40588961899999998 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.709161+0000", 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5568000000000001e-05 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.709244+0000", 2024-09-16T03:37:26.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.263e-05 2024-09-16T03:37:26.594 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.594 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.594 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.594 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:69 4.0 4:0a55bbf2:::smithi15736696-18:head [write 685719~711939,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:26.594 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.301546+0000", 2024-09-16T03:37:26.594 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.22256089, 2024-09-16T03:37:26.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50794553600000003, 2024-09-16T03:37:26.594 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.594 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.594 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.595 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.595 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.595 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.595 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 69 2024-09-16T03:37:26.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.595 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301546+0000", 2024-09-16T03:37:26.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301546+0000", 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301547+0000", 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4799999999999997e-07 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302372+0000", 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000824828 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302381+0000", 2024-09-16T03:37:26.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1759999999999999e-06 2024-09-16T03:37:26.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302389+0000", 2024-09-16T03:37:26.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4710000000000007e-06 2024-09-16T03:37:26.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.401577+0000", 2024-09-16T03:37:26.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099188002999999997 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.401681+0000", 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000103618 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.401797+0000", 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116177 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.403396+0000", 2024-09-16T03:37:26.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001598857 2024-09-16T03:37:26.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.606185+0000", 2024-09-16T03:37:26.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202788996 2024-09-16T03:37:26.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.809397+0000", 2024-09-16T03:37:26.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20321167700000001 2024-09-16T03:37:26.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.809449+0000", 2024-09-16T03:37:26.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2389000000000003e-05 2024-09-16T03:37:26.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.809492+0000", 2024-09-16T03:37:26.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2496000000000001e-05 2024-09-16T03:37:26.600 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.600 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.600 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.601 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:286 4.7 4:e2f8c24a:::smithi15736696-37:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:26.601 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.600823+0000", 2024-09-16T03:37:26.601 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.923283625, 2024-09-16T03:37:26.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50653485200000004, 2024-09-16T03:37:26.601 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.601 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.601 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.601 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.601 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.601 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.601 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 286 2024-09-16T03:37:26.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.601 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600823+0000", 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600823+0000", 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600825+0000", 2024-09-16T03:37:26.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.736e-06 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600830+0000", 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2480000000000004e-06 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600842+0000", 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1309e-05 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.700982+0000", 2024-09-16T03:37:26.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100140487 2024-09-16T03:37:26.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701107+0000", 2024-09-16T03:37:26.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000124671 2024-09-16T03:37:26.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701165+0000", 2024-09-16T03:37:26.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7995000000000003e-05 2024-09-16T03:37:26.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903559+0000", 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20239401100000001 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.904281+0000", 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072182699999999997 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.904415+0000", 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013477300000000001 2024-09-16T03:37:26.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.107259+0000", 2024-09-16T03:37:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202843209 2024-09-16T03:37:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.107321+0000", 2024-09-16T03:37:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2242000000000005e-05 2024-09-16T03:37:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.107358+0000", 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7344e-05 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:409 4.f 4:fa163101:::smithi15736696-10:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.306109+0000", 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.2179975199999999, 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50402713799999999, 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.607 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.608 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.608 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.608 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 409 2024-09-16T03:37:26.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.608 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306109+0000", 2024-09-16T03:37:26.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306109+0000", 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306111+0000", 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.429e-06 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306116+0000", 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1019999999999996e-06 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306129+0000", 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2840000000000001e-05 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306137+0000", 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1370000000000002e-06 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.506224+0000", 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20008714899999999 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.506352+0000", 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000128424 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709180+0000", 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20282708299999999 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709780+0000", 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060068800000000005 2024-09-16T03:37:26.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.710070+0000", 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000289683 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.809989+0000", 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099918689000000005 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.810070+0000", 2024-09-16T03:37:26.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1135000000000002e-05 2024-09-16T03:37:26.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.810136+0000", 2024-09-16T03:37:26.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6779000000000006e-05 2024-09-16T03:37:26.613 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.613 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.613 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.613 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:416 4.0 4:023abada:::smithi15736696-17:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:26.613 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:37:26.613 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.2148651660000001, 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60401983699999995, 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 416 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:37:26.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309243+0000", 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.178e-06 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309248+0000", 2024-09-16T03:37:26.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:37:26.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309257+0000", 2024-09-16T03:37:26.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4720000000000001e-06 2024-09-16T03:37:26.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309262+0000", 2024-09-16T03:37:26.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0839999999999999e-06 2024-09-16T03:37:26.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309297+0000", 2024-09-16T03:37:26.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4997999999999999e-05 2024-09-16T03:37:26.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309367+0000", 2024-09-16T03:37:26.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0124999999999997e-05 2024-09-16T03:37:26.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812055+0000", 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50268796900000001 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812614+0000", 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000558309 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.813040+0000", 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042652800000000002 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913189+0000", 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10014880399999999 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913220+0000", 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1353000000000003e-05 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913262+0000", 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1284e-05 2024-09-16T03:37:26.619 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.620 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.620 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.620 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:448 4.7 4:e1a4f3bd:::smithi15736696-49:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:26.620 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.014361+0000", 2024-09-16T03:37:26.620 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.5097455540000002, 2024-09-16T03:37:26.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.406490711, 2024-09-16T03:37:26.620 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.620 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.620 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.620 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.620 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 448 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.014361+0000", 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.014361+0000", 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.014363+0000", 2024-09-16T03:37:26.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.33e-06 2024-09-16T03:37:26.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.014367+0000", 2024-09-16T03:37:26.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4819999999999996e-06 2024-09-16T03:37:26.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.014376+0000", 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1349999999999998e-06 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.014382+0000", 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4249999999999997e-06 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.014410+0000", 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8649999999999998e-05 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.014461+0000", 2024-09-16T03:37:26.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0918000000000003e-05 2024-09-16T03:37:26.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:37:26.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.317546+0000", 2024-09-16T03:37:26.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30308435900000003 2024-09-16T03:37:26.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318822+0000", 2024-09-16T03:37:26.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0012767900000000001 2024-09-16T03:37:26.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.320386+0000", 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001563884 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.420803+0000", 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100416396 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.420829+0000", 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5879000000000001e-05 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.420852+0000", 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3462999999999999e-05 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:459 4.f 4:fa163101:::smithi15736696-10:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.015258+0000", 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.5088488130000002, 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40384624800000002, 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.627 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.628 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.628 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.628 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.628 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.628 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 459 2024-09-16T03:37:26.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.628 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.628 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.015258+0000", 2024-09-16T03:37:26.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.015258+0000", 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.015259+0000", 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0100000000000001e-06 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.015264+0000", 2024-09-16T03:37:26.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8500000000000002e-06 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.015277+0000", 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2838e-05 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.015285+0000", 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.636e-06 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.115390+0000", 2024-09-16T03:37:26.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100104133 2024-09-16T03:37:26.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.115448+0000", 2024-09-16T03:37:26.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8894999999999998e-05 2024-09-16T03:37:26.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:37:26.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.317909+0000", 2024-09-16T03:37:26.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202460219 2024-09-16T03:37:26.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318764+0000", 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00085538300000000003 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.320654+0000", 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0018902859999999999 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.419039+0000", 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098384761000000001 2024-09-16T03:37:26.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.419072+0000", 2024-09-16T03:37:26.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3084999999999999e-05 2024-09-16T03:37:26.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.419104+0000", 2024-09-16T03:37:26.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2152e-05 2024-09-16T03:37:26.633 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.633 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.633 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.634 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:494 4.f 4:f817def9:::smithi15736696-45:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:26.634 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:37:26.634 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.8019254309999999, 2024-09-16T03:37:26.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60451760399999999, 2024-09-16T03:37:26.634 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.634 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.634 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.634 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.634 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.634 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.634 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 494 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722183+0000", 2024-09-16T03:37:26.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:37:26.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722189+0000", 2024-09-16T03:37:26.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9429999999999997e-06 2024-09-16T03:37:26.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722203+0000", 2024-09-16T03:37:26.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4367e-05 2024-09-16T03:37:26.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722210+0000", 2024-09-16T03:37:26.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:37:26.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822367+0000", 2024-09-16T03:37:26.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100156933 2024-09-16T03:37:26.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822461+0000", 2024-09-16T03:37:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3216999999999999e-05 2024-09-16T03:37:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:37:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125593+0000", 2024-09-16T03:37:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30313253699999998 2024-09-16T03:37:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126307+0000", 2024-09-16T03:37:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071350000000000005 2024-09-16T03:37:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126825+0000", 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051846100000000001 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326612+0000", 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19978721999999999 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326663+0000", 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0460000000000001e-05 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326699+0000", 2024-09-16T03:37:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6340000000000001e-05 2024-09-16T03:37:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:517 4.0 4:0a55bbf2:::smithi15736696-18:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.922514+0000", 2024-09-16T03:37:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.6015928009999998, 2024-09-16T03:37:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50578298600000005, 2024-09-16T03:37:26.640 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.641 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.641 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.641 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.641 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.641 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.641 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 517 2024-09-16T03:37:26.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.641 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922514+0000", 2024-09-16T03:37:26.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922514+0000", 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922515+0000", 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.381e-06 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922520+0000", 2024-09-16T03:37:26.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8590000000000004e-06 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922531+0000", 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0783e-05 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922539+0000", 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4220000000000003e-06 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.923072+0000", 2024-09-16T03:37:26.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053305500000000005 2024-09-16T03:37:26.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.923145+0000", 2024-09-16T03:37:26.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3572000000000006e-05 2024-09-16T03:37:26.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:26.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.327214+0000", 2024-09-16T03:37:26.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.404069177 2024-09-16T03:37:26.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.328231+0000", 2024-09-16T03:37:26.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010163240000000001 2024-09-16T03:37:26.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.330360+0000", 2024-09-16T03:37:26.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021293029999999999 2024-09-16T03:37:26.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.428103+0000", 2024-09-16T03:37:26.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.097742869999999996 2024-09-16T03:37:26.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.428244+0000", 2024-09-16T03:37:26.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014071999999999999 2024-09-16T03:37:26.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.428297+0000", 2024-09-16T03:37:26.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3520000000000003e-05 2024-09-16T03:37:26.646 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.646 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.646 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.647 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:783 4.0 4:06b6d050:::smithi15736696-50:head [write 2323973~516377] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:26.647 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806116+0000", 2024-09-16T03:37:26.647 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.71799101899999995, 2024-09-16T03:37:26.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.43371549700000001, 2024-09-16T03:37:26.647 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:26.647 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:26.647 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:26.647 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:26.647 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:26.647 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:26.647 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 783 2024-09-16T03:37:26.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.648 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:26.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:26.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806116+0000", 2024-09-16T03:37:26.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:26.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806116+0000", 2024-09-16T03:37:26.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:26.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806118+0000", 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7379999999999999e-06 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806558+0000", 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000440708 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806571+0000", 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.27e-05 2024-09-16T03:37:26.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806577+0000", 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3659999999999997e-06 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.098215+0000", 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29163721500000001 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.098270+0000", 2024-09-16T03:37:26.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5583e-05 2024-09-16T03:37:26.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:26.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.098376+0000", 2024-09-16T03:37:26.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000106216 2024-09-16T03:37:26.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:26.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200767+0000", 2024-09-16T03:37:26.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10239073899999999 2024-09-16T03:37:26.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200814+0000", 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6409999999999998e-05 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.239704+0000", 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.038890599999999997 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.239796+0000", 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1923999999999996e-05 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:26.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:26.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:26.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.239831+0000", 2024-09-16T03:37:26.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5298e-05 2024-09-16T03:37:26.653 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.653 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.653 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.653 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:26.653 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:26.653 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:26.656 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:37:26.656 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:26.652+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:26.659 INFO:tasks.rados.rados.0.smithi157.stdout:628: writing smithi15736696-46 from 1543557 to 2093561 tid 2 2024-09-16T03:37:26.659 INFO:tasks.rados.rados.0.smithi157.stdout:628: writing smithi15736696-46 from 2833665 to 2907578 tid 3 2024-09-16T03:37:26.659 INFO:tasks.rados.rados.0.smithi157.stdout:614: done (14 left) 2024-09-16T03:37:26.660 INFO:tasks.rados.rados.0.smithi157.stdout:616: done (13 left) 2024-09-16T03:37:26.660 INFO:tasks.rados.rados.0.smithi157.stdout:617: done (12 left) 2024-09-16T03:37:26.660 INFO:tasks.rados.rados.0.smithi157.stdout:618: done (11 left) 2024-09-16T03:37:26.660 INFO:tasks.rados.rados.0.smithi157.stdout:621: done (10 left) 2024-09-16T03:37:26.660 INFO:tasks.rados.rados.0.smithi157.stdout:622: done (9 left) 2024-09-16T03:37:26.660 INFO:tasks.rados.rados.0.smithi157.stdout:623: done (8 left) 2024-09-16T03:37:26.660 INFO:tasks.rados.rados.0.smithi157.stdout:624: done (7 left) 2024-09-16T03:37:26.660 INFO:tasks.rados.rados.0.smithi157.stdout:625: done (6 left) 2024-09-16T03:37:26.660 INFO:tasks.rados.rados.0.smithi157.stdout:626: done (5 left) 2024-09-16T03:37:26.660 INFO:tasks.rados.rados.0.smithi157.stdout:627: done (4 left) 2024-09-16T03:37:26.660 INFO:tasks.rados.rados.0.smithi157.stdout:629: read oid 28 snap -1 2024-09-16T03:37:26.660 INFO:tasks.rados.rados.0.smithi157.stdout:629: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-16T03:37:26.660 INFO:tasks.rados.rados.0.smithi157.stdout:630: read oid 26 snap -1 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:630: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:631: read oid 31 snap -1 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:631: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:632: read oid 45 snap -1 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:632: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:633: write oid 9 current snap is 0 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:613: finishing write tid 5 to smithi15736696-50 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:613: oid 50 updating version 0 to 37 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:613: oid 50 updating version 37 to 39 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:613: oid 50 version 39 is already newer than 38 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:613: oid 50 version 39 is already newer than 36 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 39 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:613: left oid 50 (ObjNum 50 snap 0 seq_num 50) 2024-09-16T03:37:26.661 INFO:tasks.rados.rados.0.smithi157.stdout:619: finishing write tid 3 to smithi15736696-44 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:619: finishing write tid 4 to smithi15736696-44 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:619: oid 44 updating version 0 to 19 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:619: oid 44 updating version 19 to 20 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:619: oid 44 version 20 is already newer than 18 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 20 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:619: left oid 44 (ObjNum 51 snap 0 seq_num 51) 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:633: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:633: writing smithi15736696-9 from 524385 to 1077551 tid 1 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:633: writing smithi15736696-9 from 1602039 to 2012054 tid 2 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:633: writing smithi15736696-9 from 2577441 to 2924385 tid 3 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:613: done (8 left) 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:619: done (7 left) 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:634: write oid 34 current snap is 0 2024-09-16T03:37:26.662 INFO:tasks.rados.rados.0.smithi157.stdout:634: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:634: writing smithi15736696-34 from 541192 to 1169607 tid 1 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:634: writing smithi15736696-34 from 1660521 to 2214194 tid 2 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:634: writing smithi15736696-34 from 2888511 to 2941192 tid 3 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:635: write (excl) oid 33 current snap is 0 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:620: got (0) Success 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:635: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:635: writing smithi15736696-33 from 557999 to 1261663 tid 1 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:635: writing smithi15736696-33 from 1719003 to 2132687 tid 2 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:635: writing smithi15736696-33 from 2632287 to 2957999 tid 3 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:620: done (8 left) 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:636: read oid 3 snap -1 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:636: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-16T03:37:26.663 INFO:tasks.rados.rados.0.smithi157.stdout:637: tier_evict oid 15 2024-09-16T03:37:26.664 INFO:tasks.rados.rados.0.smithi157.stdout:637: tier_evict oid 15 snap -1 2024-09-16T03:37:26.664 INFO:tasks.rados.rados.0.smithi157.stdout:638: read oid 13 snap -1 2024-09-16T03:37:26.664 INFO:tasks.rados.rados.0.smithi157.stdout:638: expect (ObjNum 53 snap 0 seq_num 53) 2024-09-16T03:37:26.664 INFO:tasks.rados.rados.0.smithi157.stdout:639: read oid 8 snap -1 2024-09-16T03:37:26.664 INFO:tasks.rados.rados.0.smithi157.stdout:639: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:37:26.664 INFO:tasks.rados.rados.0.smithi157.stdout:640: delete oid 7 current snap is 0 2024-09-16T03:37:26.664 INFO:tasks.rados.rados.0.smithi157.stdout:637: got (22) Invalid argument 2024-09-16T03:37:26.664 INFO:tasks.rados.rados.0.smithi157.stdout:638: expect (ObjNum 53 snap 0 seq_num 53) 2024-09-16T03:37:26.664 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:26.664 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 588544 2024-09-16T03:37:26.718 INFO:tasks.rados.rados.0.smithi157.stdout:635: finishing write tid 1 to smithi15736696-33 2024-09-16T03:37:26.718 INFO:tasks.rados.rados.0.smithi157.stdout:629: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-16T03:37:26.718 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 310274 2024-09-16T03:37:26.755 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:26.754+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:26.798 INFO:tasks.rados.rados.0.smithi157.stdout:636: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-16T03:37:26.800 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:26.800 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:26.800 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:37:26.800 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:26.809 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:37:26.818 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1197634 2024-09-16T03:37:26.851 INFO:tasks.rados.rados.0.smithi157.stdout:630: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-16T03:37:26.856 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:26.854+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:26.891 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:26.912 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:26.913 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:26.913 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:26.913 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:26.913 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:26.918 INFO:tasks.rados.rados.0.smithi157.stdout:635: finishing write tid 2 to smithi15736696-33 2024-09-16T03:37:26.918 INFO:tasks.rados.rados.0.smithi157.stdout:635: finishing write tid 3 to smithi15736696-33 2024-09-16T03:37:26.918 INFO:tasks.rados.rados.0.smithi157.stdout:635: finishing write tid 4 to smithi15736696-33 2024-09-16T03:37:26.918 INFO:tasks.rados.rados.0.smithi157.stdout:635: finishing write tid 5 to smithi15736696-33 2024-09-16T03:37:26.919 INFO:tasks.rados.rados.0.smithi157.stdout:635: oid 33 updating version 0 to 12 2024-09-16T03:37:26.919 INFO:tasks.rados.rados.0.smithi157.stdout:635: oid 33 updating version 12 to 15 2024-09-16T03:37:26.919 INFO:tasks.rados.rados.0.smithi157.stdout:635: oid 33 version 15 is already newer than 13 2024-09-16T03:37:26.919 INFO:tasks.rados.rados.0.smithi157.stdout:635: oid 33 version 15 is already newer than 14 2024-09-16T03:37:26.919 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 15 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-09-16T03:37:26.919 INFO:tasks.rados.rados.0.smithi157.stdout:635: left oid 33 (ObjNum 57 snap 0 seq_num 57) 2024-09-16T03:37:26.919 INFO:tasks.rados.rados.0.smithi157.stdout:629: done (12 left) 2024-09-16T03:37:26.919 INFO:tasks.rados.rados.0.smithi157.stdout:630: done (11 left) 2024-09-16T03:37:26.919 INFO:tasks.rados.rados.0.smithi157.stdout:635: done (10 left) 2024-09-16T03:37:26.919 INFO:tasks.rados.rados.0.smithi157.stdout:636: done (9 left) 2024-09-16T03:37:26.919 INFO:tasks.rados.rados.0.smithi157.stdout:637: done (8 left) 2024-09-16T03:37:26.919 INFO:tasks.rados.rados.0.smithi157.stdout:638: done (7 left) 2024-09-16T03:37:26.919 INFO:tasks.rados.rados.0.smithi157.stdout:640: done (6 left) 2024-09-16T03:37:26.920 INFO:tasks.rados.rados.0.smithi157.stdout:641: read oid 32 snap -1 2024-09-16T03:37:26.920 INFO:tasks.rados.rados.0.smithi157.stdout:641: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-16T03:37:26.920 INFO:tasks.rados.rados.0.smithi157.stdout:642: tier_evict oid 18 2024-09-16T03:37:26.920 INFO:tasks.rados.rados.0.smithi157.stdout:642: tier_evict oid 18 snap -1 2024-09-16T03:37:26.920 INFO:tasks.rados.rados.0.smithi157.stdout:631: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-16T03:37:26.920 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:26.921 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:37:26.946 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:26.957 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:26.955+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:26.984 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2041000 2024-09-16T03:37:26.989 INFO:tasks.rados.rados.0.smithi157.stdout:632: expect (ObjNum 44 snap 0 seq_num 44) 2024-09-16T03:37:26.989 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:27.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:37:27.057 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:27.056+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:27.066 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:29 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.038945+0000", 2024-09-16T03:37:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.025551162999999, 2024-09-16T03:37:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023791260000000001, 2024-09-16T03:37:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.067 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.038945+0000", 2024-09-16T03:37:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.038945+0000", 2024-09-16T03:37:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.038945+0000", 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1999999999999999e-07 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.039493+0000", 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054706600000000002 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.039505+0000", 2024-09-16T03:37:27.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1993e-05 2024-09-16T03:37:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.039521+0000", 2024-09-16T03:37:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6401000000000001e-05 2024-09-16T03:37:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.039605+0000", 2024-09-16T03:37:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4578000000000002e-05 2024-09-16T03:37:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.039630+0000", 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4817999999999999e-05 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.041281+0000", 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016508810000000001 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.041324+0000", 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2769000000000003e-05 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.071 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:37:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.920763896, 2024-09-16T03:37:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:37:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:37:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:37:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:37:27.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:37:27.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:37:27.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:37:27.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:37:27.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:37:27.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:37:27.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:37:27.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:37:27.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:37:27.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:37:27.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:37:27.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:37:27.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:37:27.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.920410536, 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.077 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.078 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.078 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:37:27.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:37:27.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:37:27.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:37:27.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:37:27.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:37:27.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:37:27.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.081 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.082 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:37:27.082 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:37:27.082 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.919282738, 2024-09-16T03:37:27.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:37:27.082 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.082 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.082 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.082 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:37:27.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:37:27.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:37:27.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:37:27.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:37:27.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:37:27.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:37:27.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:37:27.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:37:27.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:37:27.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:37:27.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:37:27.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:37:27.087 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.087 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.087 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.087 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:37:27.087 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:37:27.087 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.918295297, 2024-09-16T03:37:27.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:37:27.087 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.087 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.087 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:37:27.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:37:27.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:37:27.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:37:27.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:37:27.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:37:27.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:37:27.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:37:27.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:37:27.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:37:27.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:37:27.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:37:27.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:37:27.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:37:27.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:37:27.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:37:27.092 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.092 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.092 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.092 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:37:27.092 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:37:27.092 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.917538553, 2024-09-16T03:37:27.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:37:27.092 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.092 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.092 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.092 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:37:27.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:37:27.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:37:27.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:37:27.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:37:27.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:37:27.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:37:27.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:37:27.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:85 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'7, mlcod=0'0)", 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.505621+0000", 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.558875349999999, 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016838669999999999, 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.097 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.098 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.505621+0000", 2024-09-16T03:37:27.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.505621+0000", 2024-09-16T03:37:27.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.505623+0000", 2024-09-16T03:37:27.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8819999999999999e-06 2024-09-16T03:37:27.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.505804+0000", 2024-09-16T03:37:27.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018145499999999999 2024-09-16T03:37:27.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.505821+0000", 2024-09-16T03:37:27.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6892e-05 2024-09-16T03:37:27.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.505832+0000", 2024-09-16T03:37:27.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1443e-05 2024-09-16T03:37:27.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.505927+0000", 2024-09-16T03:37:27.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4247999999999996e-05 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.505965+0000", 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8182999999999997e-05 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.507254+0000", 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0012892260000000001 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.507305+0000", 2024-09-16T03:37:27.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0538000000000003e-05 2024-09-16T03:37:27.102 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.102 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.102 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.103 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:86 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'8, mlcod=39'5)", 2024-09-16T03:37:27.103 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:37:27.103 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.456026316999999, 2024-09-16T03:37:27.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20388281899999999, 2024-09-16T03:37:27.103 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.103 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.103 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.103 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:37:27.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:37:27.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608472+0000", 2024-09-16T03:37:27.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.227e-06 2024-09-16T03:37:27.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608616+0000", 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000144177 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608638+0000", 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1543e-05 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608645+0000", 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5839999999999997e-06 2024-09-16T03:37:27.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608740+0000", 2024-09-16T03:37:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4716999999999995e-05 2024-09-16T03:37:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608763+0000", 2024-09-16T03:37:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2671e-05 2024-09-16T03:37:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812264+0000", 2024-09-16T03:37:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203501194 2024-09-16T03:37:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812353+0000", 2024-09-16T03:37:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8706000000000001e-05 2024-09-16T03:37:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:87 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'9, mlcod=39'5)", 2024-09-16T03:37:27.107 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:37:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.455812943, 2024-09-16T03:37:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20372206400000001, 2024-09-16T03:37:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:37:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608684+0000", 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6700000000000003e-07 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608689+0000", 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.412e-06 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608704+0000", 2024-09-16T03:37:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4397e-05 2024-09-16T03:37:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608713+0000", 2024-09-16T03:37:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:37:27.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710025+0000", 2024-09-16T03:37:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101312032 2024-09-16T03:37:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710042+0000", 2024-09-16T03:37:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6889e-05 2024-09-16T03:37:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812361+0000", 2024-09-16T03:37:27.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10231889600000001 2024-09-16T03:37:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812405+0000", 2024-09-16T03:37:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4101999999999999e-05 2024-09-16T03:37:27.115 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:101 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'10, mlcod=39'5)", 2024-09-16T03:37:27.116 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:37:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.454799506000001, 2024-09-16T03:37:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20275586900000001, 2024-09-16T03:37:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.117 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:37:27.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:37:27.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609698+0000", 2024-09-16T03:37:27.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.24e-06 2024-09-16T03:37:27.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609892+0000", 2024-09-16T03:37:27.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019465899999999999 2024-09-16T03:37:27.121 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.121 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609904+0000", 2024-09-16T03:37:27.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2031000000000001e-05 2024-09-16T03:37:27.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609912+0000", 2024-09-16T03:37:27.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7160000000000003e-06 2024-09-16T03:37:27.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710228+0000", 2024-09-16T03:37:27.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100315682 2024-09-16T03:37:27.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710242+0000", 2024-09-16T03:37:27.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4201e-05 2024-09-16T03:37:27.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812411+0000", 2024-09-16T03:37:27.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10216881799999999 2024-09-16T03:37:27.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812452+0000", 2024-09-16T03:37:27.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1522e-05 2024-09-16T03:37:27.126 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.126 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.127 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.127 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:102 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'11, mlcod=39'5)", 2024-09-16T03:37:27.127 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:37:27.127 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.452783481000001, 2024-09-16T03:37:27.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20134195399999999, 2024-09-16T03:37:27.128 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.128 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.128 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.128 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:37:27.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:37:27.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611714+0000", 2024-09-16T03:37:27.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.269e-06 2024-09-16T03:37:27.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611926+0000", 2024-09-16T03:37:27.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021252700000000001 2024-09-16T03:37:27.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611938+0000", 2024-09-16T03:37:27.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1437000000000001e-05 2024-09-16T03:37:27.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611944+0000", 2024-09-16T03:37:27.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.993e-06 2024-09-16T03:37:27.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811266+0000", 2024-09-16T03:37:27.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.199322306 2024-09-16T03:37:27.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811291+0000", 2024-09-16T03:37:27.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4542999999999999e-05 2024-09-16T03:37:27.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813021+0000", 2024-09-16T03:37:27.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001730363 2024-09-16T03:37:27.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813054+0000", 2024-09-16T03:37:27.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3516000000000002e-05 2024-09-16T03:37:27.137 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.138 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.138 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.138 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:103 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'12, mlcod=39'5)", 2024-09-16T03:37:27.138 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:37:27.139 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.44957698, 2024-09-16T03:37:27.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19838012999999999, 2024-09-16T03:37:27.139 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.139 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.139 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.140 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:37:27.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:37:27.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614920+0000", 2024-09-16T03:37:27.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.192e-06 2024-09-16T03:37:27.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614927+0000", 2024-09-16T03:37:27.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4640000000000003e-06 2024-09-16T03:37:27.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614937+0000", 2024-09-16T03:37:27.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9599999999999995e-06 2024-09-16T03:37:27.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614942+0000", 2024-09-16T03:37:27.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.31e-06 2024-09-16T03:37:27.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812489+0000", 2024-09-16T03:37:27.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.197546629 2024-09-16T03:37:27.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812511+0000", 2024-09-16T03:37:27.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2385e-05 2024-09-16T03:37:27.145 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813279+0000", 2024-09-16T03:37:27.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076818100000000005 2024-09-16T03:37:27.145 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813299+0000", 2024-09-16T03:37:27.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0009000000000001e-05 2024-09-16T03:37:27.145 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:294 4.f e41/32 4:f817def9:::smithi15736696-45:head v 41'20, mlcod=41'18)", 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.208633+0000", 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.855863331, 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00089574900000000005, 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.208633+0000", 2024-09-16T03:37:27.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.208633+0000", 2024-09-16T03:37:27.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.208635+0000", 2024-09-16T03:37:27.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.978e-06 2024-09-16T03:37:27.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.208647+0000", 2024-09-16T03:37:27.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2778e-05 2024-09-16T03:37:27.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.208664+0000", 2024-09-16T03:37:27.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6169000000000001e-05 2024-09-16T03:37:27.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.208674+0000", 2024-09-16T03:37:27.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0332e-05 2024-09-16T03:37:27.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.208783+0000", 2024-09-16T03:37:27.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000109239 2024-09-16T03:37:27.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.208801+0000", 2024-09-16T03:37:27.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7997999999999999e-05 2024-09-16T03:37:27.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.209474+0000", 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067269599999999997 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.209528+0000", 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4558999999999997e-05 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.151 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:371 4.f e45/32 4:f101f67d:::smithi15736696-22:head v 45'22, mlcod=41'20)", 2024-09-16T03:37:27.151 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.307373+0000", 2024-09-16T03:37:27.151 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.757123021, 2024-09-16T03:37:27.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00062312200000000004, 2024-09-16T03:37:27.151 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.151 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.151 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.151 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.151 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307373+0000", 2024-09-16T03:37:27.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307373+0000", 2024-09-16T03:37:27.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307373+0000", 2024-09-16T03:37:27.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3700000000000001e-07 2024-09-16T03:37:27.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307384+0000", 2024-09-16T03:37:27.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0621e-05 2024-09-16T03:37:27.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307394+0000", 2024-09-16T03:37:27.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5449999999999992e-06 2024-09-16T03:37:27.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307401+0000", 2024-09-16T03:37:27.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.746e-06 2024-09-16T03:37:27.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307464+0000", 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3044000000000002e-05 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307486+0000", 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2044000000000001e-05 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307951+0000", 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00046408799999999998 2024-09-16T03:37:27.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307996+0000", 2024-09-16T03:37:27.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5596999999999997e-05 2024-09-16T03:37:27.155 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.155 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.155 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.155 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:375 4.f e45/32 4:fcb75e43:::smithi15736696-26:head v 45'23, mlcod=41'20)", 2024-09-16T03:37:27.155 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:37:27.155 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.756455753, 2024-09-16T03:37:27.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200786568, 2024-09-16T03:37:27.156 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.156 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.156 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.156 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:37:27.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:37:27.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308041+0000", 2024-09-16T03:37:27.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8299999999999995e-07 2024-09-16T03:37:27.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308046+0000", 2024-09-16T03:37:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.904e-06 2024-09-16T03:37:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308056+0000", 2024-09-16T03:37:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9210000000000004e-06 2024-09-16T03:37:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.408122+0000", 2024-09-16T03:37:27.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100066302 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508289+0000", 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166531 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508302+0000", 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2641e-05 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508794+0000", 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00049272400000000005 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508827+0000", 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2561999999999998e-05 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:409 4.f e45/32 4:fa163101:::smithi15736696-10:head v 45'24, mlcod=45'22)", 2024-09-16T03:37:27.160 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.709292+0000", 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.3552038140000002, 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00049257499999999998, 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709292+0000", 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709292+0000", 2024-09-16T03:37:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709293+0000", 2024-09-16T03:37:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.263e-06 2024-09-16T03:37:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709299+0000", 2024-09-16T03:37:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5890000000000002e-06 2024-09-16T03:37:27.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709310+0000", 2024-09-16T03:37:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0847e-05 2024-09-16T03:37:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709324+0000", 2024-09-16T03:37:27.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3648e-05 2024-09-16T03:37:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709364+0000", 2024-09-16T03:37:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0948999999999997e-05 2024-09-16T03:37:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709377+0000", 2024-09-16T03:37:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2863e-05 2024-09-16T03:37:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709752+0000", 2024-09-16T03:37:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000375151 2024-09-16T03:37:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709785+0000", 2024-09-16T03:37:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2264999999999997e-05 2024-09-16T03:37:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.165 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:459 4.f e45/32 4:fa163101:::smithi15736696-10:head v 45'25, mlcod=45'23)", 2024-09-16T03:37:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.318000+0000", 2024-09-16T03:37:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.7464958490000004, 2024-09-16T03:37:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10092035100000001, 2024-09-16T03:37:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318000+0000", 2024-09-16T03:37:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318000+0000", 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318002+0000", 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.624e-06 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318009+0000", 2024-09-16T03:37:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9310000000000002e-06 2024-09-16T03:37:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318018+0000", 2024-09-16T03:37:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4949999999999997e-06 2024-09-16T03:37:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318031+0000", 2024-09-16T03:37:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2869e-05 2024-09-16T03:37:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418228+0000", 2024-09-16T03:37:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10019649799999999 2024-09-16T03:37:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418286+0000", 2024-09-16T03:37:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8904999999999999e-05 2024-09-16T03:37:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418884+0000", 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059763899999999996 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418921+0000", 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6390000000000002e-05 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:509 4.f e45/32 4:fa163101:::smithi15736696-10:head v 45'26, mlcod=45'24)", 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:22.024930+0000", 2024-09-16T03:37:27.170 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.0395663190000004, 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10084305, 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024930+0000", 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024930+0000", 2024-09-16T03:37:27.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024932+0000", 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.863e-06 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024942+0000", 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9960000000000005e-06 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024954+0000", 2024-09-16T03:37:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2128e-05 2024-09-16T03:37:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024962+0000", 2024-09-16T03:37:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4209999999999995e-06 2024-09-16T03:37:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125116+0000", 2024-09-16T03:37:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015399899999999 2024-09-16T03:37:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125160+0000", 2024-09-16T03:37:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4359e-05 2024-09-16T03:37:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125734+0000", 2024-09-16T03:37:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00057321599999999998 2024-09-16T03:37:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125773+0000", 2024-09-16T03:37:27.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9067999999999999e-05 2024-09-16T03:37:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.176 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:494 4.f e45/32 4:f817def9:::smithi15736696-45:head v 45'27, mlcod=45'24)", 2024-09-16T03:37:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:22.125704+0000", 2024-09-16T03:37:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.9387919599999996, 2024-09-16T03:37:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00069965700000000001, 2024-09-16T03:37:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:27.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125704+0000", 2024-09-16T03:37:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125704+0000", 2024-09-16T03:37:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125705+0000", 2024-09-16T03:37:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.393e-06 2024-09-16T03:37:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125714+0000", 2024-09-16T03:37:27.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3839999999999997e-06 2024-09-16T03:37:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125724+0000", 2024-09-16T03:37:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8099999999999992e-06 2024-09-16T03:37:27.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125731+0000", 2024-09-16T03:37:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1189999999999999e-06 2024-09-16T03:37:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125795+0000", 2024-09-16T03:37:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4068999999999999e-05 2024-09-16T03:37:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:27.182 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125814+0000", 2024-09-16T03:37:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8995999999999999e-05 2024-09-16T03:37:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126360+0000", 2024-09-16T03:37:27.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054649099999999997 2024-09-16T03:37:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126404+0000", 2024-09-16T03:37:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3395e-05 2024-09-16T03:37:27.184 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:27.185 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:27.185 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:27.186 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:37:27.189 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:27.186+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:27.191 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:37:27.195 INFO:tasks.rados.rados.0.smithi157.stdout:628: finishing write tid 1 to smithi15736696-46 2024-09-16T03:37:27.195 INFO:tasks.rados.rados.0.smithi157.stdout:628: finishing write tid 2 to smithi15736696-46 2024-09-16T03:37:27.195 INFO:tasks.rados.rados.0.smithi157.stdout:628: finishing write tid 3 to smithi15736696-46 2024-09-16T03:37:27.195 INFO:tasks.rados.rados.0.smithi157.stdout:628: finishing write tid 4 to smithi15736696-46 2024-09-16T03:37:27.195 INFO:tasks.rados.rados.0.smithi157.stdout:628: finishing write tid 5 to smithi15736696-46 2024-09-16T03:37:27.196 INFO:tasks.rados.rados.0.smithi157.stdout:628: oid 46 updating version 0 to 25 2024-09-16T03:37:27.196 INFO:tasks.rados.rados.0.smithi157.stdout:628: oid 46 updating version 25 to 26 2024-09-16T03:37:27.196 INFO:tasks.rados.rados.0.smithi157.stdout:628: oid 46 updating version 26 to 28 2024-09-16T03:37:27.196 INFO:tasks.rados.rados.0.smithi157.stdout:628: oid 46 version 28 is already newer than 27 2024-09-16T03:37:27.196 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 28 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-09-16T03:37:27.196 INFO:tasks.rados.rados.0.smithi157.stdout:628: left oid 46 (ObjNum 54 snap 0 seq_num 54) 2024-09-16T03:37:27.197 INFO:tasks.rados.rados.0.smithi157.stdout:628: done (7 left) 2024-09-16T03:37:27.197 INFO:tasks.rados.rados.0.smithi157.stdout:631: done (6 left) 2024-09-16T03:37:27.197 INFO:tasks.rados.rados.0.smithi157.stdout:632: done (5 left) 2024-09-16T03:37:27.197 INFO:tasks.rados.rados.0.smithi157.stdout:643: read oid 27 snap -1 2024-09-16T03:37:27.197 INFO:tasks.rados.rados.0.smithi157.stdout:643: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:27.198 INFO:tasks.rados.rados.0.smithi157.stdout:644: read oid 6 snap -1 2024-09-16T03:37:27.198 INFO:tasks.rados.rados.0.smithi157.stdout:644: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T03:37:27.198 INFO:tasks.rados.rados.0.smithi157.stdout:645: read oid 29 snap -1 2024-09-16T03:37:27.198 INFO:tasks.rados.rados.0.smithi157.stdout:645: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-16T03:37:27.198 INFO:tasks.rados.rados.0.smithi157.stdout:646: read oid 47 snap -1 2024-09-16T03:37:27.198 INFO:tasks.rados.rados.0.smithi157.stdout:646: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-16T03:37:27.199 INFO:tasks.rados.rados.0.smithi157.stdout:647: read oid 11 snap -1 2024-09-16T03:37:27.199 INFO:tasks.rados.rados.0.smithi157.stdout:647: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:37:27.199 INFO:tasks.rados.rados.0.smithi157.stdout:642: got (22) Invalid argument 2024-09-16T03:37:27.199 INFO:tasks.rados.rados.0.smithi157.stdout:639: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:37:27.199 INFO:tasks.rados.rados.0.smithi157.stdout:634: finishing write tid 1 to smithi15736696-34 2024-09-16T03:37:27.199 INFO:tasks.rados.rados.0.smithi157.stdout:634: finishing write tid 2 to smithi15736696-34 2024-09-16T03:37:27.200 INFO:tasks.rados.rados.0.smithi157.stdout:634: finishing write tid 3 to smithi15736696-34 2024-09-16T03:37:27.200 INFO:tasks.rados.rados.0.smithi157.stdout:634: finishing write tid 4 to smithi15736696-34 2024-09-16T03:37:27.200 INFO:tasks.rados.rados.0.smithi157.stdout:634: finishing write tid 5 to smithi15736696-34 2024-09-16T03:37:27.200 INFO:tasks.rados.rados.0.smithi157.stdout:634: oid 34 updating version 0 to 36 2024-09-16T03:37:27.200 INFO:tasks.rados.rados.0.smithi157.stdout:634: oid 34 updating version 36 to 37 2024-09-16T03:37:27.201 INFO:tasks.rados.rados.0.smithi157.stdout:634: oid 34 version 37 is already newer than 35 2024-09-16T03:37:27.201 INFO:tasks.rados.rados.0.smithi157.stdout:634: oid 34 version 37 is already newer than 34 2024-09-16T03:37:27.201 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 37 (ObjNum 56 snap 0 seq_num 56) dirty exists 2024-09-16T03:37:27.201 INFO:tasks.rados.rados.0.smithi157.stdout:634: left oid 34 (ObjNum 56 snap 0 seq_num 56) 2024-09-16T03:37:27.201 INFO:tasks.rados.rados.0.smithi157.stdout:633: finishing write tid 1 to smithi15736696-9 2024-09-16T03:37:27.201 INFO:tasks.rados.rados.0.smithi157.stdout:633: finishing write tid 2 to smithi15736696-9 2024-09-16T03:37:27.202 INFO:tasks.rados.rados.0.smithi157.stdout:633: finishing write tid 3 to smithi15736696-9 2024-09-16T03:37:27.202 INFO:tasks.rados.rados.0.smithi157.stdout:633: finishing write tid 4 to smithi15736696-9 2024-09-16T03:37:27.202 INFO:tasks.rados.rados.0.smithi157.stdout:633: finishing write tid 5 to smithi15736696-9 2024-09-16T03:37:27.202 INFO:tasks.rados.rados.0.smithi157.stdout:633: oid 9 updating version 0 to 24 2024-09-16T03:37:27.202 INFO:tasks.rados.rados.0.smithi157.stdout:633: oid 9 version 24 is already newer than 22 2024-09-16T03:37:27.203 INFO:tasks.rados.rados.0.smithi157.stdout:633: oid 9 version 24 is already newer than 21 2024-09-16T03:37:27.203 INFO:tasks.rados.rados.0.smithi157.stdout:633: oid 9 version 24 is already newer than 23 2024-09-16T03:37:27.203 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 24 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-09-16T03:37:27.203 INFO:tasks.rados.rados.0.smithi157.stdout:633: left oid 9 (ObjNum 55 snap 0 seq_num 55) 2024-09-16T03:37:27.203 INFO:tasks.rados.rados.0.smithi157.stdout:648: read oid 36 snap -1 2024-09-16T03:37:27.203 INFO:tasks.rados.rados.0.smithi157.stdout:648: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-16T03:37:27.203 INFO:tasks.rados.rados.0.smithi157.stdout:633: done (10 left) 2024-09-16T03:37:27.204 INFO:tasks.rados.rados.0.smithi157.stdout:634: done (9 left) 2024-09-16T03:37:27.204 INFO:tasks.rados.rados.0.smithi157.stdout:639: done (8 left) 2024-09-16T03:37:27.204 INFO:tasks.rados.rados.0.smithi157.stdout:642: done (7 left) 2024-09-16T03:37:27.204 INFO:tasks.rados.rados.0.smithi157.stdout:649: read oid 50 snap -1 2024-09-16T03:37:27.204 INFO:tasks.rados.rados.0.smithi157.stdout:649: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-16T03:37:27.205 INFO:tasks.rados.rados.0.smithi157.stdout:650: set_chunk oid 31 target oid 2024-09-16T03:37:27.288 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:27.287+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:27.311 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:27.311 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:27.311 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:37:27.311 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:27.316 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:27.316 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:27.320 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:37:27.333 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:27.334 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:27.349 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:27.350 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:27.366 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:27.367 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:27.382 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:27.383 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:27.389 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:27.387+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:27.399 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:27.400 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:27.417 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:27.417 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:27.434 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:27.434 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:27.447 INFO:tasks.rados.rados.0.smithi157.stdout:650: set_chunk oid 31 offset: 380928 length: 1687040 target oid 675bc5e8a50a34a7eeb520150d39162631b8f3cdc97df6520c43f7bb654e372b offset: 0 snap -1 2024-09-16T03:37:27.448 INFO:tasks.rados.rados.0.smithi157.stdout:651: read oid 13 snap -1 2024-09-16T03:37:27.448 INFO:tasks.rados.rados.0.smithi157.stdout:651: expect (ObjNum 53 snap 0 seq_num 53) 2024-09-16T03:37:27.448 INFO:tasks.rados.rados.0.smithi157.stdout:652: read oid 22 snap -1 2024-09-16T03:37:27.448 INFO:tasks.rados.rados.0.smithi157.stdout:652: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T03:37:27.448 INFO:tasks.rados.rados.0.smithi157.stdout:653: write (excl) oid 3 current snap is 0 2024-09-16T03:37:27.448 INFO:tasks.rados.rados.0.smithi157.stdout:653: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-09-16T03:37:27.450 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:27.450 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:27.453 INFO:tasks.rados.rados.0.smithi157.stdout:653: writing smithi15736696-3 from 591613 to 1045775 tid 1 2024-09-16T03:37:27.459 INFO:tasks.rados.rados.0.smithi157.stdout:653: writing smithi15736696-3 from 1719614 to 2136967 tid 2 2024-09-16T03:37:27.464 INFO:tasks.rados.rados.0.smithi157.stdout:653: writing smithi15736696-3 from 2570780 to 2991613 tid 3 2024-09-16T03:37:27.464 INFO:tasks.rados.rados.0.smithi157.stdout:654: write (excl) oid 33 current snap is 0 2024-09-16T03:37:27.464 INFO:tasks.rados.rados.0.smithi157.stdout:643: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:27.465 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:27.465 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:27.466 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:27.467 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:27.469 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 982608 2024-09-16T03:37:27.477 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:37:27.484 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:27.484 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:27.488 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:27.490 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:27.488+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:27.500 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:27.501 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:27.516 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:27.516 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:27.533 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:27.533 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:27.542 INFO:tasks.rados.rados.0.smithi157.stdout:641: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-16T03:37:27.550 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:27.550 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:27.552 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1859458 2024-09-16T03:37:27.568 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:27.568 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:27.583 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:27.584 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:27.591 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:27.589+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:27.592 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:37:27.616 INFO:tasks.rados.rados.0.smithi157.stdout:654: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-09-16T03:37:27.622 INFO:tasks.rados.rados.0.smithi157.stdout:654: writing smithi15736696-33 from 608420 to 1137831 tid 1 2024-09-16T03:37:27.629 INFO:tasks.rados.rados.0.smithi157.stdout:654: writing smithi15736696-33 from 1778096 to 2339107 tid 2 2024-09-16T03:37:27.633 INFO:tasks.rados.rados.0.smithi157.stdout:654: writing smithi15736696-33 from 2881850 to 3008420 tid 3 2024-09-16T03:37:27.633 INFO:tasks.rados.rados.0.smithi157.stdout:641: done (12 left) 2024-09-16T03:37:27.633 INFO:tasks.rados.rados.0.smithi157.stdout:643: done (11 left) 2024-09-16T03:37:27.633 INFO:tasks.rados.rados.0.smithi157.stdout:655: tier_evict oid 41 2024-09-16T03:37:27.633 INFO:tasks.rados.rados.0.smithi157.stdout:655: tier_evict oid 41 snap -1 2024-09-16T03:37:27.633 INFO:tasks.rados.rados.0.smithi157.stdout:648: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-16T03:37:27.644 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2090187 2024-09-16T03:37:27.653 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1404526 2024-09-16T03:37:27.674 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1549350 2024-09-16T03:37:27.692 INFO:tasks.rados.rados.0.smithi157.stdout:645: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-16T03:37:27.692 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:27.690+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:27.721 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:27.722 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:27.730 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:37:27.776 INFO:tasks.rados.rados.0.smithi157.stdout:646: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-16T03:37:27.792 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:27.791+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:27.807 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:27.844 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1400896 2024-09-16T03:37:27.855 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:37:27.855 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:37:27.855 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:37:27.856 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T03:37:27.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:37:27.863 INFO:tasks.rados.rados.0.smithi157.stdout:647: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:37:27.874 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:27.874 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:27.883 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:37:27.894 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:27.892+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:27.980 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2813165 2024-09-16T03:37:27.980 INFO:tasks.rados.rados.0.smithi157.stdout:644: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T03:37:27.981 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:27.995 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:27.993+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:28.028 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:28.025+0000 7fa38a6ea640 -1 osd.10 47 handle_osd_map injecting map crc failure 2024-09-16T03:37:28.029 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:28.029 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:28.029 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:28.026+0000 7f3d4270c640 -1 osd.12 47 handle_osd_map injecting map crc failure 2024-09-16T03:37:28.032 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:37:28.033 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 621150 2024-09-16T03:37:28.037 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:37:28.050 INFO:tasks.rados.rados.0.smithi157.stdout:649: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-16T03:37:28.050 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:28.095 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:28.094+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:28.155 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:28.155 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:28.163 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:37:28.163 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:37:28.165 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:37:28.173 INFO:tasks.rados.rados.0.smithi157.stdout:650: finishing set_chunk to oid 31 2024-09-16T03:37:28.174 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 31 set_chunk returned error code -95 offset: 380928 length: 1687040 2024-09-16T03:37:28.174 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T03:37:28.174 INFO:tasks.rados.rados.0.smithi157.stdout:644: done (11 left) 2024-09-16T03:37:28.174 INFO:tasks.rados.rados.0.smithi157.stdout:645: done (10 left) 2024-09-16T03:37:28.174 INFO:tasks.rados.rados.0.smithi157.stdout:646: done (9 left) 2024-09-16T03:37:28.174 INFO:tasks.rados.rados.0.smithi157.stdout:647: done (8 left) 2024-09-16T03:37:28.174 INFO:tasks.rados.rados.0.smithi157.stdout:648: done (7 left) 2024-09-16T03:37:28.174 INFO:tasks.rados.rados.0.smithi157.stdout:649: done (6 left) 2024-09-16T03:37:28.174 INFO:tasks.rados.rados.0.smithi157.stdout:650: done (5 left) 2024-09-16T03:37:28.174 INFO:tasks.rados.rados.0.smithi157.stdout:656: delete oid 21 current snap is 0 2024-09-16T03:37:28.175 INFO:tasks.rados.rados.0.smithi157.stdout:651: expect (ObjNum 53 snap 0 seq_num 53) 2024-09-16T03:37:28.176 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-09-16T03:37:28.026130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":244441084,"num_objects":114,"num_object_clones":0,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":350,"num_read_kb":20958,"num_write":537,"num_write_kb":178008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":50,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":87,"acting":87,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":87,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":781860,"kb_used_data":339796,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509167580,"statfs":{"total":1546188226560,"available":1545387601920,"internally_reserved":0,"allocated":347951104,"data_stored":361019610,"data_compressed":13321922,"data_compressed_allocated":14241792,"data_compressed_original":31428608,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,0,4,1,3,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2639933,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":89,"num_read_kb":20810,"num_write":65,"num_write_kb":2581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.344340"},"pg_stats":[{"pgid":"4.f","version":"46'28","reported_seq":79,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.326664+0000","last_change":"2024-09-16T03:37:06.386966+0000","last_active":"2024-09-16T03:37:22.326664+0000","last_peered":"2024-09-16T03:37:22.326664+0000","last_clean":"2024-09-16T03:37:22.326664+0000","last_became_active":"2024-09-16T03:37:06.386648+0000","last_became_peered":"2024-09-16T03:37:06.386648+0000","last_unstale":"2024-09-16T03:37:22.326664+0000","last_undegraded":"2024-09-16T03:37:22.326664+0000","last_fullsized":"2024-09-16T03:37:22.326664+0000","mapping_epoch":32,"log_start":"45'25","ondisk_log_start":"45'25","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":25,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:24.159491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11781542,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.e","version":"46'17","reported_seq":64,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.237861+0000","last_change":"2024-09-16T03:37:06.037443+0000","last_active":"2024-09-16T03:37:22.237861+0000","last_peered":"2024-09-16T03:37:22.237861+0000","last_clean":"2024-09-16T03:37:22.237861+0000","last_became_active":"2024-09-16T03:37:06.037035+0000","last_became_peered":"2024-09-16T03:37:06.037035+0000","last_unstale":"2024-09-16T03:37:22.237861+0000","last_undegraded":"2024-09-16T03:37:22.237861+0000","last_fullsized":"2024-09-16T03:37:22.237861+0000","mapping_epoch":32,"log_start":"45'14","ondisk_log_start":"45'14","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:09.106169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7361367,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"40'4","reported_seq":49,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.257982+0000","last_change":"2024-09-16T03:37:06.033957+0000","last_active":"2024-09-16T03:37:22.257982+0000","last_peered":"2024-09-16T03:37:22.257982+0000","last_clean":"2024-09-16T03:37:22.257982+0000","last_became_active":"2024-09-16T03:37:06.033717+0000","last_became_peered":"2024-09-16T03:37:06.033717+0000","last_unstale":"2024-09-16T03:37:22.257982+0000","last_undegraded":"2024-09-16T03:37:22.257982+0000","last_fullsized":"2024-09-16T03:37:22.257982+0000","mapping_epoch":32,"log_start":"39'1","ondisk_log_start":"39'1","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":1,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:17.751368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,1,13],"acting":[12,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"46'11","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.258019+0000","last_change":"2024-09-16T03:37:05.939190+0000","last_active":"2024-09-16T03:37:22.258019+0000","last_peered":"2024-09-16T03:37:22.258019+0000","last_clean":"2024-09-16T03:37:22.258019+0000","last_became_active":"2024-09-16T03:37:05.938951+0000","last_became_peered":"2024-09-16T03:37:05.938951+0000","last_unstale":"2024-09-16T03:37:22.258019+0000","last_undegraded":"2024-09-16T03:37:22.258019+0000","last_fullsized":"2024-09-16T03:37:22.258019+0000","mapping_epoch":32,"log_start":"43'8","ondisk_log_start":"43'8","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":8,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:11.644510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4873964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,1],"acting":[12,15,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"46'24","reported_seq":75,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T03:37:25.120118+0000","last_change":"2024-09-16T03:37:06.268430+0000","last_active":"2024-09-16T03:37:25.120118+0000","last_peered":"2024-09-16T03:37:25.120118+0000","last_clean":"2024-09-16T03:37:25.120118+0000","last_became_active":"2024-09-16T03:37:06.268115+0000","last_became_peered":"2024-09-16T03:37:06.268115+0000","last_unstale":"2024-09-16T03:37:25.120118+0000","last_undegraded":"2024-09-16T03:37:25.120118+0000","last_fullsized":"2024-09-16T03:37:25.120118+0000","mapping_epoch":32,"log_start":"46'21","ondisk_log_start":"46'21","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":21,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:33.776341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10554664,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":5160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.a","version":"46'33","reported_seq":86,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:23.639783+0000","last_change":"2024-09-16T03:37:06.386519+0000","last_active":"2024-09-16T03:37:23.639783+0000","last_peered":"2024-09-16T03:37:23.639783+0000","last_clean":"2024-09-16T03:37:23.639783+0000","last_became_active":"2024-09-16T03:37:06.286063+0000","last_became_peered":"2024-09-16T03:37:06.286063+0000","last_unstale":"2024-09-16T03:37:23.639783+0000","last_undegraded":"2024-09-16T03:37:23.639783+0000","last_fullsized":"2024-09-16T03:37:23.639783+0000","mapping_epoch":32,"log_start":"45'30","ondisk_log_start":"45'30","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":30,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:33.460844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14319366,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":24,"num_write_kb":6511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.330497+0000","last_change":"2024-09-16T03:37:16.310169+0000","last_active":"2024-09-16T03:37:22.330497+0000","last_peered":"2024-09-16T03:37:22.330497+0000","last_clean":"2024-09-16T03:37:22.330497+0000","last_became_active":"2024-09-16T03:37:16.209693+0000","last_became_peered":"2024-09-16T03:37:16.209693+0000","last_unstale":"2024-09-16T03:37:22.330497+0000","last_undegraded":"2024-09-16T03:37:22.330497+0000","last_fullsized":"2024-09-16T03:37:22.330497+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:13.573712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,8],"acting":[14,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.8","version":"46'21","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.225513+0000","last_change":"2024-09-16T03:37:06.610580+0000","last_active":"2024-09-16T03:37:22.225513+0000","last_peered":"2024-09-16T03:37:22.225513+0000","last_clean":"2024-09-16T03:37:22.225513+0000","last_became_active":"2024-09-16T03:37:06.409995+0000","last_became_peered":"2024-09-16T03:37:06.409995+0000","last_unstale":"2024-09-16T03:37:22.225513+0000","last_undegraded":"2024-09-16T03:37:22.225513+0000","last_fullsized":"2024-09-16T03:37:22.225513+0000","mapping_epoch":32,"log_start":"43'18","ondisk_log_start":"43'18","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":18,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:07.283134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8974806,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"46'23","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T03:37:25.120161+0000","last_change":"2024-09-16T03:37:06.186953+0000","last_active":"2024-09-16T03:37:25.120161+0000","last_peered":"2024-09-16T03:37:25.120161+0000","last_clean":"2024-09-16T03:37:25.120161+0000","last_became_active":"2024-09-16T03:37:06.186581+0000","last_became_peered":"2024-09-16T03:37:06.186581+0000","last_unstale":"2024-09-16T03:37:25.120161+0000","last_undegraded":"2024-09-16T03:37:25.120161+0000","last_fullsized":"2024-09-16T03:37:25.120161+0000","mapping_epoch":32,"log_start":"45'20","ondisk_log_start":"45'20","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":20,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:15.103737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9764735,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":194,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T03:37:25.485574+0000","last_change":"2024-09-16T03:36:51.858122+0000","last_active":"2024-09-16T03:37:25.485574+0000","last_peered":"2024-09-16T03:37:25.485574+0000","last_clean":"2024-09-16T03:37:25.485574+0000","last_became_active":"2024-09-16T03:36:51.855488+0000","last_became_peered":"2024-09-16T03:36:51.855488+0000","last_unstale":"2024-09-16T03:37:25.485574+0000","last_undegraded":"2024-09-16T03:37:25.485574+0000","last_fullsized":"2024-09-16T03:37:25.485574+0000","mapping_epoch":23,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:36:50.058111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:54.474463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T03:37:25.386019+0000","last_change":"2024-09-16T03:36:55.504324+0000","last_active":"2024-09-16T03:37:25.386019+0000","last_peered":"2024-09-16T03:37:25.386019+0000","last_clean":"2024-09-16T03:37:25.386019+0000","last_became_active":"2024-09-16T03:36:53.284975+0000","last_became_peered":"2024-09-16T03:36:53.284975+0000","last_unstale":"2024-09-16T03:37:25.386019+0000","last_undegraded":"2024-09-16T03:37:25.386019+0000","last_fullsized":"2024-09-16T03:37:25.386019+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:55.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"47'53","reported_seq":96,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T03:37:26.098821+0000","last_change":"2024-09-16T03:37:04.942264+0000","last_active":"2024-09-16T03:37:26.098821+0000","last_peered":"2024-09-16T03:37:26.098821+0000","last_clean":"2024-09-16T03:37:26.098821+0000","last_became_active":"2024-09-16T03:37:04.941953+0000","last_became_peered":"2024-09-16T03:37:04.941953+0000","last_unstale":"2024-09-16T03:37:26.098821+0000","last_undegraded":"2024-09-16T03:37:26.098821+0000","last_fullsized":"2024-09-16T03:37:26.098821+0000","mapping_epoch":31,"log_start":"46'49","ondisk_log_start":"46'49","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":58,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:04.021943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33125264,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":41,"num_read_kb":1433,"num_write":56,"num_write_kb":32358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.6","version":"46'20","reported_seq":71,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.028034+0000","last_change":"2024-09-16T03:37:06.037542+0000","last_active":"2024-09-16T03:37:22.028034+0000","last_peered":"2024-09-16T03:37:22.028034+0000","last_clean":"2024-09-16T03:37:22.028034+0000","last_became_active":"2024-09-16T03:37:06.037353+0000","last_became_peered":"2024-09-16T03:37:06.037353+0000","last_unstale":"2024-09-16T03:37:22.028034+0000","last_undegraded":"2024-09-16T03:37:22.028034+0000","last_fullsized":"2024-09-16T03:37:22.028034+0000","mapping_epoch":32,"log_start":"42'17","ondisk_log_start":"42'17","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:14.514916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9411788,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,6],"acting":[5,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":56,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.027441+0000","last_change":"2024-09-16T03:36:55.200565+0000","last_active":"2024-09-16T03:37:22.027441+0000","last_peered":"2024-09-16T03:37:22.027441+0000","last_clean":"2024-09-16T03:37:22.027441+0000","last_became_active":"2024-09-16T03:36:53.182912+0000","last_became_peered":"2024-09-16T03:36:53.182912+0000","last_unstale":"2024-09-16T03:37:22.027441+0000","last_undegraded":"2024-09-16T03:37:22.027441+0000","last_fullsized":"2024-09-16T03:37:22.027441+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:08.335036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"47'42","reported_seq":84,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T03:37:25.790521+0000","last_change":"2024-09-16T03:37:05.134238+0000","last_active":"2024-09-16T03:37:25.790521+0000","last_peered":"2024-09-16T03:37:25.790521+0000","last_clean":"2024-09-16T03:37:25.790521+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:37:25.790521+0000","last_undegraded":"2024-09-16T03:37:25.790521+0000","last_fullsized":"2024-09-16T03:37:25.790521+0000","mapping_epoch":31,"log_start":"46'39","ondisk_log_start":"46'39","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":50,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:13.684062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27387290,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":43,"num_write_kb":26751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"46'19","reported_seq":68,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.226985+0000","last_change":"2024-09-16T03:37:06.237537+0000","last_active":"2024-09-16T03:37:22.226985+0000","last_peered":"2024-09-16T03:37:22.226985+0000","last_clean":"2024-09-16T03:37:22.226985+0000","last_became_active":"2024-09-16T03:37:06.237172+0000","last_became_peered":"2024-09-16T03:37:06.237172+0000","last_unstale":"2024-09-16T03:37:22.226985+0000","last_undegraded":"2024-09-16T03:37:22.226985+0000","last_fullsized":"2024-09-16T03:37:22.226985+0000","mapping_epoch":32,"log_start":"43'16","ondisk_log_start":"43'16","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":16,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:31.055014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10302559,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":55,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T03:37:25.120118+0000","last_change":"2024-09-16T03:36:56.004079+0000","last_active":"2024-09-16T03:37:25.120118+0000","last_peered":"2024-09-16T03:37:25.120118+0000","last_clean":"2024-09-16T03:37:25.120118+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:37:25.120118+0000","last_undegraded":"2024-09-16T03:37:25.120118+0000","last_fullsized":"2024-09-16T03:37:25.120118+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.599963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"46'46","reported_seq":100,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.530550+0000","last_change":"2024-09-16T03:37:19.299850+0000","last_active":"2024-09-16T03:37:22.530550+0000","last_peered":"2024-09-16T03:37:22.530550+0000","last_clean":"2024-09-16T03:37:22.530550+0000","last_became_active":"2024-09-16T03:37:19.298876+0000","last_became_peered":"2024-09-16T03:37:19.298876+0000","last_unstale":"2024-09-16T03:37:22.530550+0000","last_undegraded":"2024-09-16T03:37:22.530550+0000","last_fullsized":"2024-09-16T03:37:22.530550+0000","mapping_epoch":44,"log_start":"45'43","ondisk_log_start":"45'43","created":31,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":64,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:13.639406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36790028,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":31,"num_read_kb":16,"num_write":46,"num_write_kb":35935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,4],"acting":[9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"44'9","reported_seq":61,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T03:37:25.485614+0000","last_change":"2024-09-16T03:37:06.411638+0000","last_active":"2024-09-16T03:37:25.485614+0000","last_peered":"2024-09-16T03:37:25.485614+0000","last_clean":"2024-09-16T03:37:25.485614+0000","last_became_active":"2024-09-16T03:37:06.411280+0000","last_became_peered":"2024-09-16T03:37:06.411280+0000","last_unstale":"2024-09-16T03:37:25.485614+0000","last_undegraded":"2024-09-16T03:37:25.485614+0000","last_fullsized":"2024-09-16T03:37:25.485614+0000","mapping_epoch":32,"log_start":"39'6","ondisk_log_start":"39'6","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":6,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:07.256337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4537824,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":5802,"num_write":8,"num_write_kb":2181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.127553+0000","last_change":"2024-09-16T03:36:55.201161+0000","last_active":"2024-09-16T03:37:22.127553+0000","last_peered":"2024-09-16T03:37:22.127553+0000","last_clean":"2024-09-16T03:37:22.127553+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:37:22.127553+0000","last_undegraded":"2024-09-16T03:37:22.127553+0000","last_fullsized":"2024-09-16T03:37:22.127553+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:07.449762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"47'42","reported_seq":83,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T03:37:25.995451+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:37:25.995451+0000","last_peered":"2024-09-16T03:37:25.995451+0000","last_clean":"2024-09-16T03:37:25.995451+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:37:25.995451+0000","last_undegraded":"2024-09-16T03:37:25.995451+0000","last_fullsized":"2024-09-16T03:37:25.995451+0000","mapping_epoch":31,"log_start":"46'38","ondisk_log_start":"46'38","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":50,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:19.199339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25942733,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":31,"num_read_kb":18,"num_write":45,"num_write_kb":25340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":55,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T03:37:26.123260+0000","last_change":"2024-09-16T03:37:06.043381+0000","last_active":"2024-09-16T03:37:26.123260+0000","last_peered":"2024-09-16T03:37:26.123260+0000","last_clean":"2024-09-16T03:37:26.123260+0000","last_became_active":"2024-09-16T03:37:06.043063+0000","last_became_peered":"2024-09-16T03:37:06.043063+0000","last_unstale":"2024-09-16T03:37:26.123260+0000","last_undegraded":"2024-09-16T03:37:26.123260+0000","last_fullsized":"2024-09-16T03:37:26.123260+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:09.934529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016807,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":11545,"num_write":4,"num_write_kb":929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":59,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T03:37:25.488074+0000","last_change":"2024-09-16T03:36:55.270467+0000","last_active":"2024-09-16T03:37:25.488074+0000","last_peered":"2024-09-16T03:37:25.488074+0000","last_clean":"2024-09-16T03:37:25.488074+0000","last_became_active":"2024-09-16T03:36:53.570778+0000","last_became_peered":"2024-09-16T03:36:53.570778+0000","last_unstale":"2024-09-16T03:37:25.488074+0000","last_undegraded":"2024-09-16T03:37:25.488074+0000","last_fullsized":"2024-09-16T03:37:25.488074+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:03.986104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"46'17","reported_seq":71,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T03:37:26.022363+0000","last_change":"2024-09-16T03:37:06.038263+0000","last_active":"2024-09-16T03:37:26.022363+0000","last_peered":"2024-09-16T03:37:26.022363+0000","last_clean":"2024-09-16T03:37:26.022363+0000","last_became_active":"2024-09-16T03:37:06.036785+0000","last_became_peered":"2024-09-16T03:37:06.036785+0000","last_unstale":"2024-09-16T03:37:26.022363+0000","last_undegraded":"2024-09-16T03:37:26.022363+0000","last_fullsized":"2024-09-16T03:37:26.022363+0000","mapping_epoch":32,"log_start":"44'14","ondisk_log_start":"44'14","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:24.275995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7378174,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":2011,"num_write":12,"num_write_kb":3429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":56,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.025191+0000","last_change":"2024-09-16T03:36:55.605861+0000","last_active":"2024-09-16T03:37:22.025191+0000","last_peered":"2024-09-16T03:37:22.025191+0000","last_clean":"2024-09-16T03:37:22.025191+0000","last_became_active":"2024-09-16T03:36:53.379870+0000","last_became_peered":"2024-09-16T03:36:53.379870+0000","last_unstale":"2024-09-16T03:37:22.025191+0000","last_undegraded":"2024-09-16T03:37:22.025191+0000","last_fullsized":"2024-09-16T03:37:22.025191+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:04.628405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044368400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"46'34","reported_seq":87,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:23.034015+0000","last_change":"2024-09-16T03:37:06.238073+0000","last_active":"2024-09-16T03:37:23.034015+0000","last_peered":"2024-09-16T03:37:23.034015+0000","last_clean":"2024-09-16T03:37:23.034015+0000","last_became_active":"2024-09-16T03:37:06.237774+0000","last_became_peered":"2024-09-16T03:37:06.237774+0000","last_unstale":"2024-09-16T03:37:23.034015+0000","last_undegraded":"2024-09-16T03:37:23.034015+0000","last_fullsized":"2024-09-16T03:37:23.034015+0000","mapping_epoch":32,"log_start":"46'31","ondisk_log_start":"46'31","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":31,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:23.908414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14907611,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":24,"num_write_kb":7204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":57,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.330489+0000","last_change":"2024-09-16T03:36:55.883348+0000","last_active":"2024-09-16T03:37:22.330489+0000","last_peered":"2024-09-16T03:37:22.330489+0000","last_clean":"2024-09-16T03:37:22.330489+0000","last_became_active":"2024-09-16T03:36:53.672370+0000","last_became_peered":"2024-09-16T03:36:53.672370+0000","last_unstale":"2024-09-16T03:37:22.330489+0000","last_undegraded":"2024-09-16T03:37:22.330489+0000","last_fullsized":"2024-09-16T03:37:22.330489+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:52.231720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"40'4","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.012262+0000","last_change":"2024-09-16T03:37:10.150785+0000","last_active":"2024-09-16T03:37:22.012262+0000","last_peered":"2024-09-16T03:37:22.012262+0000","last_clean":"2024-09-16T03:37:22.012262+0000","last_became_active":"2024-09-16T03:37:10.150479+0000","last_became_peered":"2024-09-16T03:37:10.150479+0000","last_unstale":"2024-09-16T03:37:22.012262+0000","last_undegraded":"2024-09-16T03:37:22.012262+0000","last_fullsized":"2024-09-16T03:37:22.012262+0000","mapping_epoch":36,"log_start":"39'1","ondisk_log_start":"39'1","created":32,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":1,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:23.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2184877,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T03:37:22.027521+0000","last_change":"2024-09-16T03:36:55.100546+0000","last_active":"2024-09-16T03:37:22.027521+0000","last_peered":"2024-09-16T03:37:22.027521+0000","last_clean":"2024-09-16T03:37:22.027521+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:37:22.027521+0000","last_undegraded":"2024-09-16T03:37:22.027521+0000","last_fullsized":"2024-09-16T03:37:22.027521+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.896892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":132,"num_read_kb":19402,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":50,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172044288,"data_stored":171015827,"data_compressed":163792,"data_compressed_allocated":180224,"data_compressed_original":385024,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":48,"acting":48,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":123245315,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":132,"num_read_kb":1483,"num_write":190,"num_write_kb":120384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":173654016,"data_stored":189087874,"data_compressed":13116263,"data_compressed_allocated":13996032,"data_compressed_original":29638656,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":1586784,"data_compressed":93744,"data_compressed_allocated":122880,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247817,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64756,"kb_used_data":37728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94307084,"statfs":{"total":96636764160,"available":96570454016,"internally_reserved":0,"allocated":38633472,"data_stored":39312683,"data_compressed":884059,"data_compressed_allocated":937984,"data_compressed_original":1945600,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247818,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58976,"kb_used_data":31296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94312864,"statfs":{"total":96636764160,"available":96576372736,"internally_reserved":0,"allocated":32047104,"data_stored":33786131,"data_compressed":1301616,"data_compressed_allocated":1400832,"data_compressed_original":3457024,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,5,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247817,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49376,"kb_used_data":21700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94322464,"statfs":{"total":96636764160,"available":96586203136,"internally_reserved":0,"allocated":22220800,"data_stored":23207470,"data_compressed":1184964,"data_compressed_allocated":1261568,"data_compressed_original":2469888,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280522,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46148,"kb_used_data":19120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325692,"statfs":{"total":96636764160,"available":96589508608,"internally_reserved":0,"allocated":19578880,"data_stored":21267374,"data_compressed":1204941,"data_compressed_allocated":1294336,"data_compressed_original":3198976,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280522,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28368,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343472,"statfs":{"total":96636764160,"available":96607715328,"internally_reserved":0,"allocated":1372160,"data_stored":1179086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50404,"kb_used_data":23376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94321436,"statfs":{"total":96636764160,"available":96585150464,"internally_reserved":0,"allocated":23937024,"data_stored":25328220,"data_compressed":1189049,"data_compressed_allocated":1277952,"data_compressed_original":2904064,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313226,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32752,"kb_used_data":5724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339088,"statfs":{"total":96636764160,"available":96603226112,"internally_reserved":0,"allocated":5861376,"data_stored":5607540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280522,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55712,"kb_used_data":24524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316128,"statfs":{"total":96636764160,"available":96579715072,"internally_reserved":0,"allocated":25112576,"data_stored":25959335,"data_compressed":1103068,"data_compressed_allocated":1171456,"data_compressed_original":2277376,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73096,"kb_used_data":46072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94298744,"statfs":{"total":96636764160,"available":96561913856,"internally_reserved":0,"allocated":47177728,"data_stored":49597875,"data_compressed":2292117,"data_compressed_allocated":2449408,"data_compressed_original":5181440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,7,8,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313226,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53764,"kb_used_data":26736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94318076,"statfs":{"total":96636764160,"available":96581709824,"internally_reserved":0,"allocated":27377664,"data_stored":28955736,"data_compressed":1204941,"data_compressed_allocated":1294336,"data_compressed_original":3198976,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,1,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280521,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36932,"kb_used_data":5744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334908,"statfs":{"total":96636764160,"available":96598945792,"internally_reserved":0,"allocated":5881856,"data_stored":5678550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53968,"kb_used_data":26940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94317872,"statfs":{"total":96636764160,"available":96581500928,"internally_reserved":0,"allocated":27586560,"data_stored":28318291,"data_compressed":884059,"data_compressed_allocated":937984,"data_compressed_original":1945600,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280521,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48980,"kb_used_data":21952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94322860,"statfs":{"total":96636764160,"available":96586608640,"internally_reserved":0,"allocated":22478848,"data_stored":23888811,"data_compressed":1189049,"data_compressed_allocated":1277952,"data_compressed_original":2904064,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280522,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55520,"kb_used_data":28492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94316320,"statfs":{"total":96636764160,"available":96579911680,"internally_reserved":0,"allocated":29175808,"data_stored":28799418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313225,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30288,"kb_used_data":3260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341552,"statfs":{"total":96636764160,"available":96605749248,"internally_reserved":0,"allocated":3338240,"data_stored":3133819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,5,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247818,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42820,"kb_used_data":15792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329020,"statfs":{"total":96636764160,"available":96592916480,"internally_reserved":0,"allocated":16171008,"data_stored":16999271,"data_compressed":884059,"data_compressed_allocated":937984,"data_compressed_original":1945600,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":15253504,"data_stored":16667543,"data_compressed":1173693,"data_compressed_allocated":1253376,"data_compressed_original":2686976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16756736,"data_stored":18367353,"data_compressed":1189049,"data_compressed_allocated":1277952,"data_compressed_original":2904064,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13303808,"data_stored":14290056,"data_compressed":884059,"data_compressed_allocated":937984,"data_compressed_original":1945600,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12603392,"data_stored":13696314,"data_compressed":1103068,"data_compressed_allocated":1171456,"data_compressed_original":2277376,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":29360128,"data_stored":32063667,"data_compressed":2292117,"data_compressed_allocated":2449408,"data_compressed_original":5181440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":15253504,"data_stored":16667543,"data_compressed":1173693,"data_compressed_allocated":1253376,"data_compressed_original":2686976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12689408,"data_stored":13811315,"data_compressed":1132730,"data_compressed_allocated":1204224,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12427264,"data_stored":13423037,"data_compressed":884059,"data_compressed_allocated":937984,"data_compressed_original":1945600,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16756736,"data_stored":18367353,"data_compressed":1189049,"data_compressed_allocated":1277952,"data_compressed_original":2904064,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":15945728,"data_stored":17443637,"data_compressed":1210687,"data_compressed_allocated":1294336,"data_compressed_original":2818048,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":13303808,"data_stored":14290056,"data_compressed":884059,"data_compressed_allocated":937984,"data_compressed_original":1945600,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3866624,"data_stored":3830774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6774784,"data_stored":6718845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3362816,"data_stored":3336493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12115968,"data_stored":12025931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":17399808,"data_stored":17291709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11616256,"data_stored":11518479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9437184,"data_stored":9478617,"data_compressed":81896,"data_compressed_allocated":90112,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5484544,"data_stored":5437415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14761984,"data_stored":14654739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":5312512,"data_stored":5280222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":28758016,"data_stored":28557172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16576512,"data_stored":16559579,"data_compressed":81896,"data_compressed_allocated":90112,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2908160,"data_stored":2888071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":935448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5406720,"data_stored":5364471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":27320320,"data_stored":27137862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:37:28.178 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-16T03:37:28.178 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-09-16T03:37:28.223 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:28.221+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:28.263 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 454161 2024-09-16T03:37:28.269 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:28.270 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:28.280 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:37:28.297 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:28.296+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:28.305 INFO:tasks.rados.rados.0.smithi157.stdout:654: finishing write tid 1 to smithi15736696-33 2024-09-16T03:37:28.305 INFO:tasks.rados.rados.0.smithi157.stdout:655: got (22) Invalid argument 2024-09-16T03:37:28.305 INFO:tasks.rados.rados.0.smithi157.stdout:652: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T03:37:28.316 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1972875 2024-09-16T03:37:28.325 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 982684 2024-09-16T03:37:28.340 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 96977 2024-09-16T03:37:28.367 INFO:tasks.rados.rados.0.smithi157.stdout:653: finishing write tid 1 to smithi15736696-3 2024-09-16T03:37:28.367 INFO:tasks.rados.rados.0.smithi157.stdout:654: finishing write tid 2 to smithi15736696-33 2024-09-16T03:37:28.367 INFO:tasks.rados.rados.0.smithi157.stdout:654: finishing write tid 3 to smithi15736696-33 2024-09-16T03:37:28.367 INFO:tasks.rados.rados.0.smithi157.stdout:651: done (5 left) 2024-09-16T03:37:28.367 INFO:tasks.rados.rados.0.smithi157.stdout:652: done (4 left) 2024-09-16T03:37:28.367 INFO:tasks.rados.rados.0.smithi157.stdout:655: done (3 left) 2024-09-16T03:37:28.367 INFO:tasks.rados.rados.0.smithi157.stdout:656: done (2 left) 2024-09-16T03:37:28.367 INFO:tasks.rados.rados.0.smithi157.stdout:657: set_chunk oid 9 target oid 2024-09-16T03:37:28.398 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:28.396+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:28.408 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:28.408 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:28.416 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:37:28.499 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:28.498+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:28.560 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:28.560 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:28.569 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:37:28.588 INFO:tasks.rados.rados.0.smithi157.stdout:657: set_chunk oid 9 offset: 397312 length: 2469888 target oid b4a34999805109aed1d5173d54999b61c2373eefd155e6467b3d49f5db74c158 offset: 0 snap -1 2024-09-16T03:37:28.589 INFO:tasks.rados.rados.0.smithi157.stdout:658: write (excl) oid 12 current snap is 0 2024-09-16T03:37:28.589 INFO:tasks.rados.rados.0.smithi157.stdout:653: finishing write tid 2 to smithi15736696-3 2024-09-16T03:37:28.589 INFO:tasks.rados.rados.0.smithi157.stdout:658: seq_num 62 ranges {642034=796262,1895060=564680} 2024-09-16T03:37:28.596 INFO:tasks.rados.rados.0.smithi157.stdout:658: writing smithi15736696-12 from 642034 to 1438296 tid 1 2024-09-16T03:37:28.600 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:28.599+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:28.602 INFO:tasks.rados.rados.0.smithi157.stdout:658: writing smithi15736696-12 from 1895060 to 2459740 tid 2 2024-09-16T03:37:28.602 INFO:tasks.rados.rados.0.smithi157.stdout:659: set_chunk oid 37 target oid 2024-09-16T03:37:28.605 INFO:tasks.rados.rados.0.smithi157.stdout:659: set_chunk oid 37 offset: 2129920 length: 89600 target oid f15a4bc1b12b17a23021c0e3457eb377158b231decb6e67343dc236bab942ef9 offset: 0 snap -1 2024-09-16T03:37:28.605 INFO:tasks.rados.rados.0.smithi157.stdout:660: write (excl) oid 44 current snap is 0 2024-09-16T03:37:28.605 INFO:tasks.rados.rados.0.smithi157.stdout:653: finishing write tid 3 to smithi15736696-3 2024-09-16T03:37:28.605 INFO:tasks.rados.rados.0.smithi157.stdout:660: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-09-16T03:37:28.611 INFO:tasks.rados.rados.0.smithi157.stdout:660: writing smithi15736696-44 from 658841 to 1130352 tid 1 2024-09-16T03:37:28.617 INFO:tasks.rados.rados.0.smithi157.stdout:660: writing smithi15736696-44 from 1553542 to 2261880 tid 2 2024-09-16T03:37:28.623 INFO:tasks.rados.rados.0.smithi157.stdout:660: writing smithi15736696-44 from 2964119 to 3058841 tid 3 2024-09-16T03:37:28.623 INFO:tasks.rados.rados.0.smithi157.stdout:653: finishing write tid 4 to smithi15736696-3 2024-09-16T03:37:28.623 INFO:tasks.rados.rados.0.smithi157.stdout:653: finishing write tid 5 to smithi15736696-3 2024-09-16T03:37:28.623 INFO:tasks.rados.rados.0.smithi157.stdout:653: oid 3 updating version 0 to 24 2024-09-16T03:37:28.623 INFO:tasks.rados.rados.0.smithi157.stdout:653: oid 3 version 24 is already newer than 22 2024-09-16T03:37:28.623 INFO:tasks.rados.rados.0.smithi157.stdout:653: oid 3 version 24 is already newer than 23 2024-09-16T03:37:28.623 INFO:tasks.rados.rados.0.smithi157.stdout:653: oid 3 updating version 24 to 25 2024-09-16T03:37:28.623 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 25 (ObjNum 59 snap 0 seq_num 59) dirty exists 2024-09-16T03:37:28.623 INFO:tasks.rados.rados.0.smithi157.stdout:653: left oid 3 (ObjNum 59 snap 0 seq_num 59) 2024-09-16T03:37:28.623 INFO:tasks.rados.rados.0.smithi157.stdout:654: finishing write tid 4 to smithi15736696-33 2024-09-16T03:37:28.623 INFO:tasks.rados.rados.0.smithi157.stdout:654: finishing write tid 5 to smithi15736696-33 2024-09-16T03:37:28.624 INFO:tasks.rados.rados.0.smithi157.stdout:654: oid 33 updating version 0 to 16 2024-09-16T03:37:28.624 INFO:tasks.rados.rados.0.smithi157.stdout:654: oid 33 updating version 16 to 17 2024-09-16T03:37:28.624 INFO:tasks.rados.rados.0.smithi157.stdout:654: oid 33 updating version 17 to 19 2024-09-16T03:37:28.624 INFO:tasks.rados.rados.0.smithi157.stdout:654: oid 33 version 19 is already newer than 18 2024-09-16T03:37:28.624 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 19 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-09-16T03:37:28.624 INFO:tasks.rados.rados.0.smithi157.stdout:654: left oid 33 (ObjNum 60 snap 0 seq_num 60) 2024-09-16T03:37:28.624 INFO:tasks.rados.rados.0.smithi157.stdout:661: read oid 20 snap -1 2024-09-16T03:37:28.624 INFO:tasks.rados.rados.0.smithi157.stdout:661: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-16T03:37:28.624 INFO:tasks.rados.rados.0.smithi157.stdout:653: done (6 left) 2024-09-16T03:37:28.624 INFO:tasks.rados.rados.0.smithi157.stdout:654: done (5 left) 2024-09-16T03:37:28.624 INFO:tasks.rados.rados.0.smithi157.stdout:662: read oid 10 snap -1 2024-09-16T03:37:28.624 INFO:tasks.rados.rados.0.smithi157.stdout:662: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:37:28.625 INFO:tasks.rados.rados.0.smithi157.stdout:663: read oid 4 snap -1 2024-09-16T03:37:28.625 INFO:tasks.rados.rados.0.smithi157.stdout:663: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-16T03:37:28.625 INFO:tasks.rados.rados.0.smithi157.stdout:664: write (excl) oid 2 current snap is 0 2024-09-16T03:37:28.625 INFO:tasks.rados.rados.0.smithi157.stdout:657: finishing set_chunk to oid 9 2024-09-16T03:37:28.625 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 9 set_chunk returned error code -95 offset: 397312 length: 2469888 2024-09-16T03:37:28.625 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 0 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-09-16T03:37:28.625 INFO:tasks.rados.rados.0.smithi157.stdout:664: seq_num 64 ranges {675648=546760,2012024=568349} 2024-09-16T03:37:28.628 INFO:tasks.rados.rados.0.smithi157.stdout:664: writing smithi15736696-2 from 675648 to 1222408 tid 1 2024-09-16T03:37:28.637 INFO:tasks.rados.rados.0.smithi157.stdout:664: writing smithi15736696-2 from 2012024 to 2580373 tid 2 2024-09-16T03:37:28.637 INFO:tasks.rados.rados.0.smithi157.stdout:657: done (7 left) 2024-09-16T03:37:28.637 INFO:tasks.rados.rados.0.smithi157.stdout:665: read oid 35 snap -1 2024-09-16T03:37:28.637 INFO:tasks.rados.rados.0.smithi157.stdout:665: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:37:28.637 INFO:tasks.rados.rados.0.smithi157.stdout:666: tier_promote oid 39 2024-09-16T03:37:28.637 INFO:tasks.rados.rados.0.smithi157.stdout:661: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-16T03:37:28.637 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:28.685 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:28.685 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:28.693 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:37:28.717 INFO:tasks.rados.rados.0.smithi157.stdout:665: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:37:28.717 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:28.715+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:28.719 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1444875 2024-09-16T03:37:28.765 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:28.794 INFO:tasks.rados.rados.0.smithi157.stdout:663: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-16T03:37:28.800 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1026801 2024-09-16T03:37:28.802 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:28.801+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:28.823 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:28.823 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:28.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:37:28.843 INFO:tasks.rados.rados.0.smithi157.stdout:662: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:37:28.857 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:28.880 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:28.903 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:28.901+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:28.904 INFO:tasks.rados.rados.0.smithi157.stdout:664: finishing write tid 1 to smithi15736696-2 2024-09-16T03:37:28.904 INFO:tasks.rados.rados.0.smithi157.stdout:664: finishing write tid 2 to smithi15736696-2 2024-09-16T03:37:28.905 INFO:tasks.rados.rados.0.smithi157.stdout:661: done (8 left) 2024-09-16T03:37:28.905 INFO:tasks.rados.rados.0.smithi157.stdout:662: done (7 left) 2024-09-16T03:37:28.905 INFO:tasks.rados.rados.0.smithi157.stdout:663: done (6 left) 2024-09-16T03:37:28.905 INFO:tasks.rados.rados.0.smithi157.stdout:665: done (5 left) 2024-09-16T03:37:28.905 INFO:tasks.rados.rados.0.smithi157.stdout:667: read oid 34 snap -1 2024-09-16T03:37:28.905 INFO:tasks.rados.rados.0.smithi157.stdout:667: expect (ObjNum 56 snap 0 seq_num 56) 2024-09-16T03:37:28.905 INFO:tasks.rados.rados.0.smithi157.stdout:668: write (excl) oid 20 current snap is 0 2024-09-16T03:37:28.905 INFO:tasks.rados.rados.0.smithi157.stdout:664: finishing write tid 3 to smithi15736696-2 2024-09-16T03:37:28.905 INFO:tasks.rados.rados.0.smithi157.stdout:668: seq_num 65 ranges {692455=622009,2070506=712007} 2024-09-16T03:37:28.911 INFO:tasks.rados.rados.0.smithi157.stdout:668: writing smithi15736696-20 from 692455 to 1314464 tid 1 2024-09-16T03:37:28.919 INFO:tasks.rados.rados.0.smithi157.stdout:668: writing smithi15736696-20 from 2070506 to 2782513 tid 2 2024-09-16T03:37:28.919 INFO:tasks.rados.rados.0.smithi157.stdout:669: set_chunk oid 30 target oid 2024-09-16T03:37:28.942 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:28.942 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:28.952 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:37:29.004 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:29.002+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:29.034 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 4 pg_num to 26 2024-09-16T03:37:29.036 INFO:tasks.rados.rados.0.smithi157.stdout:669: set_chunk oid 30 offset: 722944 length: 427520 target oid a8ae0c435cdf8eadb12f134dd3fddbfaad7579ac4c051b2b96f68620165d6384 offset: 0 snap -1 2024-09-16T03:37:29.036 INFO:tasks.rados.rados.0.smithi157.stdout:670: delete oid 31 current snap is 0 2024-09-16T03:37:29.036 INFO:tasks.rados.rados.0.smithi157.stdout:664: finishing write tid 4 to smithi15736696-2 2024-09-16T03:37:29.036 INFO:tasks.rados.rados.0.smithi157.stdout:664: oid 2 updating version 0 to 12 2024-09-16T03:37:29.036 INFO:tasks.rados.rados.0.smithi157.stdout:664: oid 2 version 12 is already newer than 11 2024-09-16T03:37:29.036 INFO:tasks.rados.rados.0.smithi157.stdout:664: oid 2 version 12 is already newer than 10 2024-09-16T03:37:29.037 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 12 (ObjNum 64 snap 0 seq_num 64) dirty exists 2024-09-16T03:37:29.037 INFO:tasks.rados.rados.0.smithi157.stdout:664: left oid 2 (ObjNum 64 snap 0 seq_num 64) 2024-09-16T03:37:29.037 INFO:tasks.rados.rados.0.smithi157.stdout:658: finishing write tid 1 to smithi15736696-12 2024-09-16T03:37:29.037 INFO:tasks.rados.rados.0.smithi157.stdout:667: expect (ObjNum 56 snap 0 seq_num 56) 2024-09-16T03:37:29.037 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:29.100 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:29.100 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:29.104 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:29.103+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:29.109 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:37:29.174 INFO:tasks.rados.rados.0.smithi157.stdout:666: got (0) Success 2024-09-16T03:37:29.174 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T03:37:29.174 INFO:tasks.rados.rados.0.smithi157.stdout:658: finishing write tid 2 to smithi15736696-12 2024-09-16T03:37:29.174 INFO:tasks.rados.rados.0.smithi157.stdout:658: finishing write tid 3 to smithi15736696-12 2024-09-16T03:37:29.175 INFO:tasks.rados.rados.0.smithi157.stdout:658: finishing write tid 4 to smithi15736696-12 2024-09-16T03:37:29.175 INFO:tasks.rados.rados.0.smithi157.stdout:658: oid 12 updating version 0 to 7 2024-09-16T03:37:29.175 INFO:tasks.rados.rados.0.smithi157.stdout:658: oid 12 version 7 is already newer than 5 2024-09-16T03:37:29.175 INFO:tasks.rados.rados.0.smithi157.stdout:658: oid 12 version 7 is already newer than 6 2024-09-16T03:37:29.175 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 7 (ObjNum 62 snap 0 seq_num 62) dirty exists 2024-09-16T03:37:29.175 INFO:tasks.rados.rados.0.smithi157.stdout:658: left oid 12 (ObjNum 62 snap 0 seq_num 62) 2024-09-16T03:37:29.175 INFO:tasks.rados.rados.0.smithi157.stdout:659: finishing set_chunk to oid 37 2024-09-16T03:37:29.175 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T03:37:29.175 INFO:tasks.rados.rados.0.smithi157.stdout:660: finishing write tid 1 to smithi15736696-44 2024-09-16T03:37:29.175 INFO:tasks.rados.rados.0.smithi157.stdout:668: finishing write tid 1 to smithi15736696-20 2024-09-16T03:37:29.203 INFO:tasks.rados.rados.0.smithi157.stdout:660: finishing write tid 2 to smithi15736696-44 2024-09-16T03:37:29.205 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:29.203+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:29.218 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:29.218 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:29.220 INFO:tasks.rados.rados.0.smithi157.stdout:668: finishing write tid 2 to smithi15736696-20 2024-09-16T03:37:29.220 INFO:tasks.rados.rados.0.smithi157.stdout:668: finishing write tid 3 to smithi15736696-20 2024-09-16T03:37:29.220 INFO:tasks.rados.rados.0.smithi157.stdout:668: finishing write tid 4 to smithi15736696-20 2024-09-16T03:37:29.220 INFO:tasks.rados.rados.0.smithi157.stdout:668: oid 20 updating version 0 to 27 2024-09-16T03:37:29.220 INFO:tasks.rados.rados.0.smithi157.stdout:668: oid 20 version 27 is already newer than 25 2024-09-16T03:37:29.221 INFO:tasks.rados.rados.0.smithi157.stdout:668: oid 20 version 27 is already newer than 26 2024-09-16T03:37:29.221 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 27 (ObjNum 65 snap 0 seq_num 65) dirty exists 2024-09-16T03:37:29.221 INFO:tasks.rados.rados.0.smithi157.stdout:668: left oid 20 (ObjNum 65 snap 0 seq_num 65) 2024-09-16T03:37:29.227 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:37:29.235 INFO:tasks.rados.rados.0.smithi157.stdout:669: finishing set_chunk to oid 30 2024-09-16T03:37:29.235 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 30 set_chunk returned error code -95 offset: 722944 length: 427520 2024-09-16T03:37:29.235 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 0 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T03:37:29.304 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:29.301+0000 7fd101e6f640 -1 osd.2 48 handle_osd_map injecting map crc failure 2024-09-16T03:37:29.304 INFO:tasks.rados.rados.0.smithi157.stdout:660: finishing write tid 3 to smithi15736696-44 2024-09-16T03:37:29.305 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:29.303+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:29.334 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:29.335 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:29.343 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:37:29.405 INFO:tasks.rados.rados.0.smithi157.stdout:660: finishing write tid 4 to smithi15736696-44 2024-09-16T03:37:29.405 INFO:tasks.rados.rados.0.smithi157.stdout:660: finishing write tid 5 to smithi15736696-44 2024-09-16T03:37:29.405 INFO:tasks.rados.rados.0.smithi157.stdout:660: oid 44 updating version 0 to 26 2024-09-16T03:37:29.405 INFO:tasks.rados.rados.0.smithi157.stdout:660: oid 44 version 26 is already newer than 25 2024-09-16T03:37:29.405 INFO:tasks.rados.rados.0.smithi157.stdout:660: oid 44 updating version 26 to 27 2024-09-16T03:37:29.405 INFO:tasks.rados.rados.0.smithi157.stdout:660: oid 44 updating version 27 to 28 2024-09-16T03:37:29.405 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 28 (ObjNum 63 snap 0 seq_num 63) dirty exists 2024-09-16T03:37:29.405 INFO:tasks.rados.rados.0.smithi157.stdout:660: left oid 44 (ObjNum 63 snap 0 seq_num 63) 2024-09-16T03:37:29.407 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:29.405+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:29.491 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:29.491 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:29.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:37:29.506 INFO:tasks.rados.rados.0.smithi157.stdout:658: done (8 left) 2024-09-16T03:37:29.506 INFO:tasks.rados.rados.0.smithi157.stdout:659: done (7 left) 2024-09-16T03:37:29.506 INFO:tasks.rados.rados.0.smithi157.stdout:660: done (6 left) 2024-09-16T03:37:29.506 INFO:tasks.rados.rados.0.smithi157.stdout:664: done (5 left) 2024-09-16T03:37:29.506 INFO:tasks.rados.rados.0.smithi157.stdout:666: done (4 left) 2024-09-16T03:37:29.506 INFO:tasks.rados.rados.0.smithi157.stdout:667: done (3 left) 2024-09-16T03:37:29.506 INFO:tasks.rados.rados.0.smithi157.stdout:668: done (2 left) 2024-09-16T03:37:29.507 INFO:tasks.rados.rados.0.smithi157.stdout:669: done (1 left) 2024-09-16T03:37:29.507 INFO:tasks.rados.rados.0.smithi157.stdout:670: done (0 left) 2024-09-16T03:37:29.507 INFO:tasks.rados.rados.0.smithi157.stdout:671: tier_promote oid 4 2024-09-16T03:37:29.507 INFO:tasks.rados.rados.0.smithi157.stdout:672: read oid 18 snap -1 2024-09-16T03:37:29.507 INFO:tasks.rados.rados.0.smithi157.stdout:672: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T03:37:29.507 INFO:tasks.rados.rados.0.smithi157.stdout:673: read oid 22 snap -1 2024-09-16T03:37:29.507 INFO:tasks.rados.rados.0.smithi157.stdout:673: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T03:37:29.507 INFO:tasks.rados.rados.0.smithi157.stdout:674: read oid 10 snap -1 2024-09-16T03:37:29.507 INFO:tasks.rados.rados.0.smithi157.stdout:674: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:37:29.507 INFO:tasks.rados.rados.0.smithi157.stdout:675: write oid 28 current snap is 0 2024-09-16T03:37:29.507 INFO:tasks.rados.rados.0.smithi157.stdout:675: seq_num 67 ranges {726069=772507,2071117=715676} 2024-09-16T03:37:29.508 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:29.507+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:29.514 INFO:tasks.rados.rados.0.smithi157.stdout:675: writing smithi15736696-28 from 726069 to 1498576 tid 1 2024-09-16T03:37:29.521 INFO:tasks.rados.rados.0.smithi157.stdout:675: writing smithi15736696-28 from 2071117 to 2786793 tid 2 2024-09-16T03:37:29.521 INFO:tasks.rados.rados.0.smithi157.stdout:676: set_chunk oid 27 target oid 2024-09-16T03:37:29.531 INFO:tasks.rados.rados.0.smithi157.stdout:676: set_chunk oid 27 offset: 838656 length: 594944 target oid 78b08a9fbefb18826ac02face5069f3d28f7e3b354c1de8a19f7e30b80592cee offset: 0 snap -1 2024-09-16T03:37:29.531 INFO:tasks.rados.rados.0.smithi157.stdout:677: delete oid 48 current snap is 0 2024-09-16T03:37:29.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:37:29.609 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:29.608+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:29.610 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:29.610 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:29.613 INFO:tasks.rados.rados.0.smithi157.stdout:676: finishing set_chunk to oid 27 2024-09-16T03:37:29.613 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 838656 length: 594944 2024-09-16T03:37:29.613 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:29.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:37:29.622 INFO:tasks.rados.rados.0.smithi157.stdout:675: finishing write tid 1 to smithi15736696-28 2024-09-16T03:37:29.706 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:29.707 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:29.710 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:29.708+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:29.724 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:29.725 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:29.742 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:29.742 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:29.759 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:29.760 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:29.770 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:29.770 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:29.776 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:29.776 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:29.779 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:37:29.793 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:29.793 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:29.810 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:29.811 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:29.811 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:29.809+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:29.814 INFO:tasks.rados.rados.0.smithi157.stdout:672: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T03:37:29.814 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 655167 2024-09-16T03:37:29.827 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:29.828 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:29.834 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:29.832+0000 7f3d40708640 -1 osd.12 48 handle_osd_map injecting map crc failure 2024-09-16T03:37:29.844 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:29.845 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:29.859 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:29.862 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:29.862 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:29.879 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:29.880 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:29.884 INFO:tasks.rados.rados.0.smithi157.stdout:673: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T03:37:29.884 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:29.897 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:29.897 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:29.901 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:29.901 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:29.901 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:29.902 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:29.902 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:29.911 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:37:29.911 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:29.909+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:29.913 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:29.914 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:29.930 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:29.931 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:29.931 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:29.952 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:29.952 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:29.958 INFO:tasks.rados.rados.0.smithi157.stdout:671: got (0) Success 2024-09-16T03:37:29.958 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T03:37:29.958 INFO:tasks.rados.rados.0.smithi157.stdout:671: done (6 left) 2024-09-16T03:37:29.958 INFO:tasks.rados.rados.0.smithi157.stdout:672: done (5 left) 2024-09-16T03:37:29.958 INFO:tasks.rados.rados.0.smithi157.stdout:673: done (4 left) 2024-09-16T03:37:29.958 INFO:tasks.rados.rados.0.smithi157.stdout:676: done (3 left) 2024-09-16T03:37:29.958 INFO:tasks.rados.rados.0.smithi157.stdout:677: done (2 left) 2024-09-16T03:37:29.959 INFO:tasks.rados.rados.0.smithi157.stdout:678: read oid 8 snap -1 2024-09-16T03:37:29.959 INFO:tasks.rados.rados.0.smithi157.stdout:678: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:37:29.959 INFO:tasks.rados.rados.0.smithi157.stdout:679: read oid 43 snap -1 2024-09-16T03:37:29.959 INFO:tasks.rados.rados.0.smithi157.stdout:679: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T03:37:29.959 INFO:tasks.rados.rados.0.smithi157.stdout:680: set_chunk oid 16 target oid 2024-09-16T03:37:30.011 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:30.010+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:30.023 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4338.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:37:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.094389+0000", 2024-09-16T03:37:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.925365393999996, 2024-09-16T03:37:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203832385, 2024-09-16T03:37:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4338", 2024-09-16T03:37:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3073758685", 2024-09-16T03:37:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5 2024-09-16T03:37:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094389+0000", 2024-09-16T03:37:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094389+0000", 2024-09-16T03:37:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094393+0000", 2024-09-16T03:37:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.467e-06 2024-09-16T03:37:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094405+0000", 2024-09-16T03:37:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1793e-05 2024-09-16T03:37:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094425+0000", 2024-09-16T03:37:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0412999999999999e-05 2024-09-16T03:37:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094436+0000", 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1178e-05 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094516+0000", 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0109000000000002e-05 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094522+0000", 2024-09-16T03:37:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8950000000000003e-06 2024-09-16T03:37:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.196488+0000", 2024-09-16T03:37:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10196596400000001 2024-09-16T03:37:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.196600+0000", 2024-09-16T03:37:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111703 2024-09-16T03:37:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,10", 2024-09-16T03:37:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.196738+0000", 2024-09-16T03:37:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013858800000000001 2024-09-16T03:37:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.197729+0000", 2024-09-16T03:37:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00099082199999999993 2024-09-16T03:37:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.198192+0000", 2024-09-16T03:37:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046261099999999998 2024-09-16T03:37:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.298103+0000", 2024-09-16T03:37:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099911137999999997 2024-09-16T03:37:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.298167+0000", 2024-09-16T03:37:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4289999999999996e-05 2024-09-16T03:37:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.298222+0000", 2024-09-16T03:37:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4413999999999998e-05 2024-09-16T03:37:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4338.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:37:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.922657072, 2024-09-16T03:37:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30502727499999999, 2024-09-16T03:37:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4338", 2024-09-16T03:37:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3073758685", 2024-09-16T03:37:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-16T03:37:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:37:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:37:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097101+0000", 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.915e-06 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097111+0000", 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0589e-05 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097130+0000", 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8958999999999999e-05 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097141+0000", 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0616e-05 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097227+0000", 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6322000000000005e-05 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097232+0000", 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2059999999999998e-06 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200235+0000", 2024-09-16T03:37:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103002945 2024-09-16T03:37:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200318+0000", 2024-09-16T03:37:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2515999999999994e-05 2024-09-16T03:37:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,10", 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200429+0000", 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011158999999999999 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201478+0000", 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00104893 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201942+0000", 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046354700000000003 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402048+0000", 2024-09-16T03:37:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20010616000000001 2024-09-16T03:37:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402089+0000", 2024-09-16T03:37:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0762e-05 2024-09-16T03:37:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402125+0000", 2024-09-16T03:37:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6217999999999998e-05 2024-09-16T03:37:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:13 4.a 4:51338165:::smithi15736696-4:head [write 450421~625747,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:37:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.991194+0000", 2024-09-16T03:37:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.028560749, 2024-09-16T03:37:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20656139100000001, 2024-09-16T03:37:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 13 2024-09-16T03:37:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991194+0000", 2024-09-16T03:37:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991194+0000", 2024-09-16T03:37:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991197+0000", 2024-09-16T03:37:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1470000000000001e-06 2024-09-16T03:37:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991908+0000", 2024-09-16T03:37:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071085899999999999 2024-09-16T03:37:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991925+0000", 2024-09-16T03:37:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7368e-05 2024-09-16T03:37:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991936+0000", 2024-09-16T03:37:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0891e-05 2024-09-16T03:37:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.992011+0000", 2024-09-16T03:37:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4814000000000006e-05 2024-09-16T03:37:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.992018+0000", 2024-09-16T03:37:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2239999999999998e-06 2024-09-16T03:37:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.994165+0000", 2024-09-16T03:37:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002147043 2024-09-16T03:37:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.994259+0000", 2024-09-16T03:37:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3764999999999994e-05 2024-09-16T03:37:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.994365+0000", 2024-09-16T03:37:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105701 2024-09-16T03:37:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996323+0000", 2024-09-16T03:37:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001958212 2024-09-16T03:37:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.000278+0000", 2024-09-16T03:37:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039546690000000001 2024-09-16T03:37:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197685+0000", 2024-09-16T03:37:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19740706 2024-09-16T03:37:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197721+0000", 2024-09-16T03:37:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6631000000000002e-05 2024-09-16T03:37:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197755+0000", 2024-09-16T03:37:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4007000000000002e-05 2024-09-16T03:37:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:14 4.a 4:51338165:::smithi15736696-4:head [write 1659093~391328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.996458+0000", 2024-09-16T03:37:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.023297019000001, 2024-09-16T03:37:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20147354000000001, 2024-09-16T03:37:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 14 2024-09-16T03:37:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996458+0000", 2024-09-16T03:37:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996458+0000", 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996476+0000", 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8681999999999998e-05 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996922+0000", 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044590599999999998 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996937+0000", 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5214e-05 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996948+0000", 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0825000000000001e-05 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.997004+0000", 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5943000000000003e-05 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.997079+0000", 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5018999999999994e-05 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.997179+0000", 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9597000000000002e-05 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.998553+0000", 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001373649 2024-09-16T03:37:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197800+0000", 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199247911 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197871+0000", 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0599000000000006e-05 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197906+0000", 2024-09-16T03:37:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5006999999999999e-05 2024-09-16T03:37:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197931+0000", 2024-09-16T03:37:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5188e-05 2024-09-16T03:37:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:15 4.a 4:51338165:::smithi15736696-4:head [setxattr _header (55) in=7b,truncate 2050421] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:37:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.022785139, 2024-09-16T03:37:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30116835400000003, 2024-09-16T03:37:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 15 2024-09-16T03:37:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:37:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:37:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996971+0000", 2024-09-16T03:37:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.066e-06 2024-09-16T03:37:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996984+0000", 2024-09-16T03:37:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3246e-05 2024-09-16T03:37:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996994+0000", 2024-09-16T03:37:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.713e-06 2024-09-16T03:37:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097070+0000", 2024-09-16T03:37:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100076313 2024-09-16T03:37:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097141+0000", 2024-09-16T03:37:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1298999999999996e-05 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097205+0000", 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3907999999999999e-05 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097300+0000", 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5209000000000003e-05 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197645+0000", 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10034422899999999 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197837+0000", 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000192454 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298049+0000", 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100211868 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298082+0000", 2024-09-16T03:37:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3467000000000003e-05 2024-09-16T03:37:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298138+0000", 2024-09-16T03:37:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5581999999999997e-05 2024-09-16T03:37:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:16 4.a 4:51338165:::smithi15736696-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.097092+0000", 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.922662754000001, 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20130547800000001, 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 16 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097092+0000", 2024-09-16T03:37:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097092+0000", 2024-09-16T03:37:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097094+0000", 2024-09-16T03:37:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0159999999999998e-06 2024-09-16T03:37:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097101+0000", 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0489999999999998e-06 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097109+0000", 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6899999999999992e-06 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097114+0000", 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8450000000000002e-06 2024-09-16T03:37:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197569+0000", 2024-09-16T03:37:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100455534 2024-09-16T03:37:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197626+0000", 2024-09-16T03:37:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6548999999999997e-05 2024-09-16T03:37:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298197+0000", 2024-09-16T03:37:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10057105099999999 2024-09-16T03:37:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298281+0000", 2024-09-16T03:37:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4307000000000004e-05 2024-09-16T03:37:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298397+0000", 2024-09-16T03:37:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116437 2024-09-16T03:37:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:73 4.a 4:532814f2:::smithi15736696-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:37:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.708494936000001, 2024-09-16T03:37:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.362642662, 2024-09-16T03:37:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73 2024-09-16T03:37:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:37:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:37:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311261+0000", 2024-09-16T03:37:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.562e-06 2024-09-16T03:37:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311925+0000", 2024-09-16T03:37:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066385499999999998 2024-09-16T03:37:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311937+0000", 2024-09-16T03:37:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1881e-05 2024-09-16T03:37:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311943+0000", 2024-09-16T03:37:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6999999999999996e-06 2024-09-16T03:37:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311988+0000", 2024-09-16T03:37:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5358000000000002e-05 2024-09-16T03:37:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312070+0000", 2024-09-16T03:37:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1735000000000003e-05 2024-09-16T03:37:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312167+0000", 2024-09-16T03:37:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6685e-05 2024-09-16T03:37:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.313661+0000", 2024-09-16T03:37:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001494324 2024-09-16T03:37:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.414759+0000", 2024-09-16T03:37:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10109850300000001 2024-09-16T03:37:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673821+0000", 2024-09-16T03:37:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.25906127400000001 2024-09-16T03:37:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673858+0000", 2024-09-16T03:37:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7620999999999998e-05 2024-09-16T03:37:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673902+0000", 2024-09-16T03:37:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4163999999999999e-05 2024-09-16T03:37:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.4.22:4 3.3 e41/31 3:deac805c:::smithi15736696-44 low_tier:head v 41'19, mlcod=41'14)", 2024-09-16T03:37:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.902970+0000", 2024-09-16T03:37:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.116784546, 2024-09-16T03:37:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200721028, 2024-09-16T03:37:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902970+0000", 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902970+0000", 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902972+0000", 2024-09-16T03:37:30.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.705e-06 2024-09-16T03:37:30.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902981+0000", 2024-09-16T03:37:30.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6740000000000003e-06 2024-09-16T03:37:30.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902995+0000", 2024-09-16T03:37:30.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4039000000000001e-05 2024-09-16T03:37:30.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.003072+0000", 2024-09-16T03:37:30.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007719599999999 2024-09-16T03:37:30.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.103239+0000", 2024-09-16T03:37:30.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100167336 2024-09-16T03:37:30.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.103253+0000", 2024-09-16T03:37:30.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4236999999999999e-05 2024-09-16T03:37:30.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.103657+0000", 2024-09-16T03:37:30.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040399499999999998 2024-09-16T03:37:30.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.103691+0000", 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3846000000000001e-05 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:303 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.608036106, 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50956986000000004, 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 303 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:37:30.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411721+0000", 2024-09-16T03:37:30.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0200000000000001e-06 2024-09-16T03:37:30.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411729+0000", 2024-09-16T03:37:30.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1240000000000005e-06 2024-09-16T03:37:30.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411744+0000", 2024-09-16T03:37:30.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099000000000001e-05 2024-09-16T03:37:30.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411751+0000", 2024-09-16T03:37:30.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.029e-06 2024-09-16T03:37:30.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511892+0000", 2024-09-16T03:37:30.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100140879 2024-09-16T03:37:30.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511961+0000", 2024-09-16T03:37:30.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9379000000000001e-05 2024-09-16T03:37:30.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:30.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.613647+0000", 2024-09-16T03:37:30.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101685994 2024-09-16T03:37:30.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.614294+0000", 2024-09-16T03:37:30.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064692799999999996 2024-09-16T03:37:30.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.714962+0000", 2024-09-16T03:37:30.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066765 2024-09-16T03:37:30.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921157+0000", 2024-09-16T03:37:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206195463 2024-09-16T03:37:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921233+0000", 2024-09-16T03:37:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5446999999999998e-05 2024-09-16T03:37:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921288+0000", 2024-09-16T03:37:30.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5847999999999998e-05 2024-09-16T03:37:30.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.085 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.085 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.085 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:303 4.a e42/32 ondisk, result = 0)", 2024-09-16T03:37:30.085 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.720776+0000", 2024-09-16T03:37:30.085 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.298979038000001, 2024-09-16T03:37:30.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200522282, 2024-09-16T03:37:30.085 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.085 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.720776+0000", 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.720776+0000", 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.720777+0000", 2024-09-16T03:37:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3829999999999999e-06 2024-09-16T03:37:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.720786+0000", 2024-09-16T03:37:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6969999999999999e-06 2024-09-16T03:37:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.720799+0000", 2024-09-16T03:37:30.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2897e-05 2024-09-16T03:37:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.820951+0000", 2024-09-16T03:37:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015253 2024-09-16T03:37:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921138+0000", 2024-09-16T03:37:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100187073 2024-09-16T03:37:30.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921150+0000", 2024-09-16T03:37:30.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1754e-05 2024-09-16T03:37:30.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921298+0000", 2024-09-16T03:37:30.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014794799999999999 2024-09-16T03:37:30.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.090 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.090 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:404 4.3 e45/32 4:cf22405f:::smithi15736696-5:head v 45'15, mlcod=42'13)", 2024-09-16T03:37:30.090 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.309931+0000", 2024-09-16T03:37:30.090 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.7098235190000004, 2024-09-16T03:37:30.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30105416800000001, 2024-09-16T03:37:30.091 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.091 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.091 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.091 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309931+0000", 2024-09-16T03:37:30.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309931+0000", 2024-09-16T03:37:30.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309933+0000", 2024-09-16T03:37:30.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0099999999999998e-06 2024-09-16T03:37:30.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309943+0000", 2024-09-16T03:37:30.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4050000000000006e-06 2024-09-16T03:37:30.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309959+0000", 2024-09-16T03:37:30.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6365e-05 2024-09-16T03:37:30.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.410036+0000", 2024-09-16T03:37:30.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100077477 2024-09-16T03:37:30.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610252+0000", 2024-09-16T03:37:30.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20021509700000001 2024-09-16T03:37:30.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610290+0000", 2024-09-16T03:37:30.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8068999999999998e-05 2024-09-16T03:37:30.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610947+0000", 2024-09-16T03:37:30.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065726700000000001 2024-09-16T03:37:30.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610985+0000", 2024-09-16T03:37:30.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8478e-05 2024-09-16T03:37:30.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.097 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.097 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.10.22:22 3.3 e45/31 3:f84e13a5:::smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'25, mlcod=45'23)", 2024-09-16T03:37:30.097 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.410130+0000", 2024-09-16T03:37:30.097 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.6096248699999993, 2024-09-16T03:37:30.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20118708499999999, 2024-09-16T03:37:30.097 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.098 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.098 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.410130+0000", 2024-09-16T03:37:30.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.410130+0000", 2024-09-16T03:37:30.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.410130+0000", 2024-09-16T03:37:30.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4700000000000002e-07 2024-09-16T03:37:30.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.410151+0000", 2024-09-16T03:37:30.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0474000000000001e-05 2024-09-16T03:37:30.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.410158+0000", 2024-09-16T03:37:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1690000000000003e-06 2024-09-16T03:37:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.510237+0000", 2024-09-16T03:37:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100079156 2024-09-16T03:37:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610415+0000", 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10017830599999999 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610475+0000", 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0022000000000002e-05 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.611284+0000", 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080844 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.611317+0000", 2024-09-16T03:37:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3071000000000002e-05 2024-09-16T03:37:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:503 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:37:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1979615540000008, 2024-09-16T03:37:30.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60521621599999997, 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 503 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:37:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821794+0000", 2024-09-16T03:37:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-09-16T03:37:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821798+0000", 2024-09-16T03:37:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2030000000000002e-06 2024-09-16T03:37:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821804+0000", 2024-09-16T03:37:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:37:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821810+0000", 2024-09-16T03:37:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4249999999999997e-06 2024-09-16T03:37:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922212+0000", 2024-09-16T03:37:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040228800000001 2024-09-16T03:37:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922307+0000", 2024-09-16T03:37:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5089999999999999e-05 2024-09-16T03:37:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.126007+0000", 2024-09-16T03:37:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20369931499999999 2024-09-16T03:37:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.127605+0000", 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001598213 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.327263+0000", 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199657905 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426900+0000", 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099637217 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426963+0000", 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.427009+0000", 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6533000000000003e-05 2024-09-16T03:37:30.109 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.110 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.110 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:777 4.a 4:5a5e9bc2:::smithi15736696-36:head [sparse-read 0~1188805] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46)", 2024-09-16T03:37:30.110 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.469159+0000", 2024-09-16T03:37:30.110 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.5505956279999999, 2024-09-16T03:37:30.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20073789, 2024-09-16T03:37:30.110 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.110 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.110 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:30.110 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.110 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 777 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.469159+0000", 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.469159+0000", 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.469160+0000", 2024-09-16T03:37:30.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9200000000000001e-07 2024-09-16T03:37:30.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.469162+0000", 2024-09-16T03:37:30.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6539999999999999e-06 2024-09-16T03:37:30.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.469169+0000", 2024-09-16T03:37:30.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4420000000000003e-06 2024-09-16T03:37:30.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.569288+0000", 2024-09-16T03:37:30.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100119507 2024-09-16T03:37:30.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.669504+0000", 2024-09-16T03:37:30.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100215715 2024-09-16T03:37:30.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.669628+0000", 2024-09-16T03:37:30.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124194 2024-09-16T03:37:30.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.669897+0000", 2024-09-16T03:37:30.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026878600000000002 2024-09-16T03:37:30.115 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.115 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.115 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.115 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:848 4.a 4:503187b3:::smithi15736696-34:head [write 541192~628415] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:30.115 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.553032+0000", 2024-09-16T03:37:30.115 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.466722522, 2024-09-16T03:37:30.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20855610899999999, 2024-09-16T03:37:30.115 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.115 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.116 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.116 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.116 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.116 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.116 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 848 2024-09-16T03:37:30.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.116 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.553032+0000", 2024-09-16T03:37:30.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.553032+0000", 2024-09-16T03:37:30.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.553042+0000", 2024-09-16T03:37:30.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0186999999999999e-05 2024-09-16T03:37:30.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.553852+0000", 2024-09-16T03:37:30.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000809211 2024-09-16T03:37:30.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.553866+0000", 2024-09-16T03:37:30.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4404e-05 2024-09-16T03:37:30.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.553882+0000", 2024-09-16T03:37:30.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5985999999999999e-05 2024-09-16T03:37:30.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.654103+0000", 2024-09-16T03:37:30.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100221269 2024-09-16T03:37:30.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.654219+0000", 2024-09-16T03:37:30.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011623700000000001 2024-09-16T03:37:30.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:30.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.654411+0000", 2024-09-16T03:37:30.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019102999999999999 2024-09-16T03:37:30.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658655+0000", 2024-09-16T03:37:30.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042448140000000004 2024-09-16T03:37:30.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658665+0000", 2024-09-16T03:37:30.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3910000000000004e-06 2024-09-16T03:37:30.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.761400+0000", 2024-09-16T03:37:30.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10273526600000001 2024-09-16T03:37:30.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.761552+0000", 2024-09-16T03:37:30.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015201400000000001 2024-09-16T03:37:30.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.761588+0000", 2024-09-16T03:37:30.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6300000000000001e-05 2024-09-16T03:37:30.122 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.122 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.122 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.122 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:849 4.a 4:503187b3:::smithi15736696-34:head [write 1660521~553673] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.558487+0000", 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.4612678949999998, 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203229874, 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 849 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.558487+0000", 2024-09-16T03:37:30.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.558487+0000", 2024-09-16T03:37:30.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.558488+0000", 2024-09-16T03:37:30.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.367e-06 2024-09-16T03:37:30.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.559021+0000", 2024-09-16T03:37:30.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053312699999999999 2024-09-16T03:37:30.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.559033+0000", 2024-09-16T03:37:30.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2128e-05 2024-09-16T03:37:30.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.559039+0000", 2024-09-16T03:37:30.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0340000000000002e-06 2024-09-16T03:37:30.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.656443+0000", 2024-09-16T03:37:30.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097403643999999998 2024-09-16T03:37:30.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.656499+0000", 2024-09-16T03:37:30.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6388999999999998e-05 2024-09-16T03:37:30.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:30.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.656606+0000", 2024-09-16T03:37:30.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106708 2024-09-16T03:37:30.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759296+0000", 2024-09-16T03:37:30.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10268964899999999 2024-09-16T03:37:30.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759350+0000", 2024-09-16T03:37:30.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3767999999999997e-05 2024-09-16T03:37:30.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.761620+0000", 2024-09-16T03:37:30.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022707790000000001 2024-09-16T03:37:30.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.761694+0000", 2024-09-16T03:37:30.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3402e-05 2024-09-16T03:37:30.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.761717+0000", 2024-09-16T03:37:30.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2878999999999999e-05 2024-09-16T03:37:30.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.129 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.130 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.130 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:850 4.a 4:503187b3:::smithi15736696-34:head [write 2888511~52681] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:30.130 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.562268+0000", 2024-09-16T03:37:30.130 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.4574862479999999, 2024-09-16T03:37:30.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29934466599999998, 2024-09-16T03:37:30.130 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.130 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.130 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 850 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562268+0000", 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562268+0000", 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562274+0000", 2024-09-16T03:37:30.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3460000000000001e-06 2024-09-16T03:37:30.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562334+0000", 2024-09-16T03:37:30.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9993e-05 2024-09-16T03:37:30.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562346+0000", 2024-09-16T03:37:30.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2048999999999999e-05 2024-09-16T03:37:30.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562351+0000", 2024-09-16T03:37:30.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6879999999999998e-06 2024-09-16T03:37:30.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.657867+0000", 2024-09-16T03:37:30.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095516648999999995 2024-09-16T03:37:30.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.657964+0000", 2024-09-16T03:37:30.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6646999999999998e-05 2024-09-16T03:37:30.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:30.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658121+0000", 2024-09-16T03:37:30.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015688 2024-09-16T03:37:30.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759327+0000", 2024-09-16T03:37:30.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101206726 2024-09-16T03:37:30.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759490+0000", 2024-09-16T03:37:30.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016302600000000001 2024-09-16T03:37:30.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861466+0000", 2024-09-16T03:37:30.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101975808 2024-09-16T03:37:30.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861543+0000", 2024-09-16T03:37:30.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6316000000000006e-05 2024-09-16T03:37:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861613+0000", 2024-09-16T03:37:30.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0537999999999994e-05 2024-09-16T03:37:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:851 4.a 4:503187b3:::smithi15736696-34:head [setxattr _header (56) in=7b,truncate 2941192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.562360+0000", 2024-09-16T03:37:30.138 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.4573951190000001, 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29963510700000001, 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 851 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562360+0000", 2024-09-16T03:37:30.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562360+0000", 2024-09-16T03:37:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562360+0000", 2024-09-16T03:37:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-16T03:37:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562363+0000", 2024-09-16T03:37:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8100000000000002e-06 2024-09-16T03:37:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562367+0000", 2024-09-16T03:37:30.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.873e-06 2024-09-16T03:37:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562370+0000", 2024-09-16T03:37:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1820000000000002e-06 2024-09-16T03:37:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658694+0000", 2024-09-16T03:37:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096324009000000002 2024-09-16T03:37:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658785+0000", 2024-09-16T03:37:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0370000000000004e-05 2024-09-16T03:37:30.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658928+0000", 2024-09-16T03:37:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014321099999999999 2024-09-16T03:37:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759398+0000", 2024-09-16T03:37:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10046983900000001 2024-09-16T03:37:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759493+0000", 2024-09-16T03:37:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5639999999999999e-05 2024-09-16T03:37:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861917+0000", 2024-09-16T03:37:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102423575 2024-09-16T03:37:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861953+0000", 2024-09-16T03:37:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6004999999999998e-05 2024-09-16T03:37:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861995+0000", 2024-09-16T03:37:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1848000000000003e-05 2024-09-16T03:37:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:852 4.a 4:503187b3:::smithi15736696-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.562374+0000", 2024-09-16T03:37:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.4573804610000001, 2024-09-16T03:37:30.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29976180800000002, 2024-09-16T03:37:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 852 2024-09-16T03:37:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562374+0000", 2024-09-16T03:37:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562374+0000", 2024-09-16T03:37:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562375+0000", 2024-09-16T03:37:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:37:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562376+0000", 2024-09-16T03:37:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4360000000000001e-06 2024-09-16T03:37:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562379+0000", 2024-09-16T03:37:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3610000000000001e-06 2024-09-16T03:37:30.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.662476+0000", 2024-09-16T03:37:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10009655100000001 2024-09-16T03:37:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759421+0000", 2024-09-16T03:37:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096945552000000004 2024-09-16T03:37:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759482+0000", 2024-09-16T03:37:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0058e-05 2024-09-16T03:37:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.862023+0000", 2024-09-16T03:37:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10254131599999999 2024-09-16T03:37:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.862072+0000", 2024-09-16T03:37:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9104999999999998e-05 2024-09-16T03:37:30.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.862136+0000", 2024-09-16T03:37:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4098999999999997e-05 2024-09-16T03:37:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:843 4.e e47/32 4:773b934b:::smithi15736696-9:head v 47'21, mlcod=47'19)", 2024-09-16T03:37:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.734480+0000", 2024-09-16T03:37:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.2852748009999999, 2024-09-16T03:37:30.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.204094623, 2024-09-16T03:37:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734480+0000", 2024-09-16T03:37:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734480+0000", 2024-09-16T03:37:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734483+0000", 2024-09-16T03:37:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1149999999999998e-06 2024-09-16T03:37:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734967+0000", 2024-09-16T03:37:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048381800000000001 2024-09-16T03:37:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734988+0000", 2024-09-16T03:37:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1101e-05 2024-09-16T03:37:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.835063+0000", 2024-09-16T03:37:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007540400000001 2024-09-16T03:37:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.935278+0000", 2024-09-16T03:37:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10021431 2024-09-16T03:37:30.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.935326+0000", 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8186000000000002e-05 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.938544+0000", 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00321857 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.938575+0000", 2024-09-16T03:37:30.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0119000000000001e-05 2024-09-16T03:37:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.161 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:30.161 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:37:30.163 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:30.069+0000 7f66f3205640 -1 osd.15 49 handle_osd_map injecting map crc failure 2024-09-16T03:37:30.163 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:30.067+0000 7fd101e6f640 -1 osd.2 49 handle_osd_map injecting map crc failure 2024-09-16T03:37:30.163 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:30.032+0000 7f12e4a1e640 -1 osd.5 49 handle_osd_map injecting map crc failure 2024-09-16T03:37:30.163 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:30.163 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:30.164 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:30.072+0000 7ff266366640 -1 osd.11 49 handle_osd_map injecting map crc failure 2024-09-16T03:37:30.164 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:30.161+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:30.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:37:30.165 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:30.032+0000 7f3d4270c640 -1 osd.12 49 handle_osd_map injecting map crc failure 2024-09-16T03:37:30.264 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:30.262+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:30.314 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:30.314 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:30.315 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:37:30.315 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:30.324 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:37:30.365 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:30.363+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:30.373 INFO:tasks.rados.rados.0.smithi157.stdout:680: set_chunk oid 16 offset: 82432 length: 2138112 target oid d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2 offset: 0 snap -1 2024-09-16T03:37:30.373 INFO:tasks.rados.rados.0.smithi157.stdout:681: read oid 21 snap -1 2024-09-16T03:37:30.373 INFO:tasks.rados.rados.0.smithi157.stdout:681: expect deleted 2024-09-16T03:37:30.373 INFO:tasks.rados.rados.0.smithi157.stdout:682: write oid 45 current snap is 0 2024-09-16T03:37:30.373 INFO:tasks.rados.rados.0.smithi157.stdout:682: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-09-16T03:37:30.380 INFO:tasks.rados.rados.0.smithi157.stdout:682: writing smithi15736696-45 from 759683 to 1399041 tid 1 2024-09-16T03:37:30.388 INFO:tasks.rados.rados.0.smithi157.stdout:682: writing smithi15736696-45 from 1904434 to 2623779 tid 2 2024-09-16T03:37:30.390 INFO:tasks.rados.rados.0.smithi157.stdout:682: writing smithi15736696-45 from 3128657 to 3159683 tid 3 2024-09-16T03:37:30.390 INFO:tasks.rados.rados.0.smithi157.stdout:683: read oid 47 snap -1 2024-09-16T03:37:30.390 INFO:tasks.rados.rados.0.smithi157.stdout:683: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-16T03:37:30.391 INFO:tasks.rados.rados.0.smithi157.stdout:684: tier_flush oid 23 2024-09-16T03:37:30.391 INFO:tasks.rados.rados.0.smithi157.stdout:684: tier_flush oid 23 snap -1 2024-09-16T03:37:30.391 INFO:tasks.rados.rados.0.smithi157.stdout:679: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T03:37:30.432 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:30.432 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:30.432 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:30.432 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:30.432 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:30.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:37:30.459 INFO:tasks.rados.rados.0.smithi157.stdout:674: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:37:30.462 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1358491 2024-09-16T03:37:30.465 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:30.463+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:30.507 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1097464 2024-09-16T03:37:30.518 INFO:tasks.rados.rados.0.smithi157.stdout:678: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-16T03:37:30.550 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:30.550 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:30.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:30.551 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:30.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.551 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:57 4.d 4:b3fe05b2:::smithi15736696-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:30.551 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:37:30.551 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.423758776, 2024-09-16T03:37:30.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.205477148, 2024-09-16T03:37:30.551 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.551 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.551 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.551 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.551 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.552 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.552 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 57 2024-09-16T03:37:30.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.552 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:37:30.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:37:30.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124125+0000", 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.604e-06 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124908+0000", 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000783086 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124919+0000", 2024-09-16T03:37:30.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0591000000000001e-05 2024-09-16T03:37:30.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124925+0000", 2024-09-16T03:37:30.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.55e-06 2024-09-16T03:37:30.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124985+0000", 2024-09-16T03:37:30.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9817e-05 2024-09-16T03:37:30.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.125113+0000", 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127429 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.125229+0000", 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116784 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.127169+0000", 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001939187 2024-09-16T03:37:30.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128052+0000", 2024-09-16T03:37:30.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088291800000000001 2024-09-16T03:37:30.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329495+0000", 2024-09-16T03:37:30.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201443328 2024-09-16T03:37:30.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329535+0000", 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9823999999999997e-05 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329601+0000", 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000003e-05 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:58 4.d 4:b3fe05b2:::smithi15736696-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.128177+0000", 2024-09-16T03:37:30.557 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.419705278999999, 2024-09-16T03:37:30.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201511992, 2024-09-16T03:37:30.558 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.558 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.558 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 58 2024-09-16T03:37:30.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.558 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128177+0000", 2024-09-16T03:37:30.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128177+0000", 2024-09-16T03:37:30.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128178+0000", 2024-09-16T03:37:30.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0920000000000001e-06 2024-09-16T03:37:30.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128330+0000", 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000151889 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128339+0000", 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9320000000000002e-06 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128344+0000", 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7140000000000001e-06 2024-09-16T03:37:30.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128416+0000", 2024-09-16T03:37:30.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2711000000000002e-05 2024-09-16T03:37:30.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128476+0000", 2024-09-16T03:37:30.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9682999999999998e-05 2024-09-16T03:37:30.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:37:30.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128556+0000", 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0167999999999995e-05 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129938+0000", 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001381969 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.228471+0000", 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098533234999999997 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329630+0000", 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101158717 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329664+0000", 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3759e-05 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329689+0000", 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5123e-05 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.563 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:59 4.d 4:b3fe05b2:::smithi15736696-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.128354+0000", 2024-09-16T03:37:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.419527925000001, 2024-09-16T03:37:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20143926300000001, 2024-09-16T03:37:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.564 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.565 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 59 2024-09-16T03:37:30.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.565 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128354+0000", 2024-09-16T03:37:30.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128354+0000", 2024-09-16T03:37:30.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128355+0000", 2024-09-16T03:37:30.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9299999999999997e-07 2024-09-16T03:37:30.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128366+0000", 2024-09-16T03:37:30.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1072e-05 2024-09-16T03:37:30.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128373+0000", 2024-09-16T03:37:30.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.268e-06 2024-09-16T03:37:30.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128379+0000", 2024-09-16T03:37:30.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6039999999999998e-06 2024-09-16T03:37:30.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128972+0000", 2024-09-16T03:37:30.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000592732 2024-09-16T03:37:30.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129024+0000", 2024-09-16T03:37:30.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2024999999999997e-05 2024-09-16T03:37:30.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:37:30.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129095+0000", 2024-09-16T03:37:30.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1371999999999994e-05 2024-09-16T03:37:30.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.228461+0000", 2024-09-16T03:37:30.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099365518 2024-09-16T03:37:30.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.228474+0000", 2024-09-16T03:37:30.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3212e-05 2024-09-16T03:37:30.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329718+0000", 2024-09-16T03:37:30.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101243888 2024-09-16T03:37:30.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329748+0000", 2024-09-16T03:37:30.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9992e-05 2024-09-16T03:37:30.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329794+0000", 2024-09-16T03:37:30.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5787000000000001e-05 2024-09-16T03:37:30.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.570 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.571 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:60 4.d 4:b3fe05b2:::smithi15736696-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:30.571 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:37:30.571 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.419496072000001, 2024-09-16T03:37:30.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30164545100000001, 2024-09-16T03:37:30.571 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.571 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:30.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.571 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 60 2024-09-16T03:37:30.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.572 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:37:30.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:37:30.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128387+0000", 2024-09-16T03:37:30.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3599999999999999e-07 2024-09-16T03:37:30.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128388+0000", 2024-09-16T03:37:30.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7659999999999999e-06 2024-09-16T03:37:30.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128394+0000", 2024-09-16T03:37:30.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0679999999999996e-06 2024-09-16T03:37:30.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128398+0000", 2024-09-16T03:37:30.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4220000000000002e-06 2024-09-16T03:37:30.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129762+0000", 2024-09-16T03:37:30.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001364285 2024-09-16T03:37:30.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:30.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129804+0000", 2024-09-16T03:37:30.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2058000000000003e-05 2024-09-16T03:37:30.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.429903+0000", 2024-09-16T03:37:30.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30009841900000001 2024-09-16T03:37:30.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.429952+0000", 2024-09-16T03:37:30.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9066000000000001e-05 2024-09-16T03:37:30.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.430032+0000", 2024-09-16T03:37:30.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9931000000000004e-05 2024-09-16T03:37:30.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.576 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:81 4.c 4:3313e97c:::smithi15736696-21:head [write 736140~537686,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:30.576 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:37:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.150256047999999, 2024-09-16T03:37:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203525015, 2024-09-16T03:37:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81 2024-09-16T03:37:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:37:30.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:37:30.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397629+0000", 2024-09-16T03:37:30.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1889999999999999e-06 2024-09-16T03:37:30.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398085+0000", 2024-09-16T03:37:30.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00045514200000000001 2024-09-16T03:37:30.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398100+0000", 2024-09-16T03:37:30.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5637999999999998e-05 2024-09-16T03:37:30.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398110+0000", 2024-09-16T03:37:30.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6549999999999999e-06 2024-09-16T03:37:30.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398190+0000", 2024-09-16T03:37:30.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0272999999999995e-05 2024-09-16T03:37:30.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398339+0000", 2024-09-16T03:37:30.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014924200000000001 2024-09-16T03:37:30.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:30.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398504+0000", 2024-09-16T03:37:30.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016471500000000001 2024-09-16T03:37:30.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.400267+0000", 2024-09-16T03:37:30.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017625169999999999 2024-09-16T03:37:30.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.401104+0000", 2024-09-16T03:37:30.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083686399999999999 2024-09-16T03:37:30.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601084+0000", 2024-09-16T03:37:30.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199980397 2024-09-16T03:37:30.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601114+0000", 2024-09-16T03:37:30.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0484000000000002e-05 2024-09-16T03:37:30.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601151+0000", 2024-09-16T03:37:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6899e-05 2024-09-16T03:37:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:82 4.c 4:3313e97c:::smithi15736696-21:head [write 2020581~315559] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.403482+0000", 2024-09-16T03:37:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.144400431000001, 2024-09-16T03:37:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197763785, 2024-09-16T03:37:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.584 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.584 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.584 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.584 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.584 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.584 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 82 2024-09-16T03:37:30.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.584 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403482+0000", 2024-09-16T03:37:30.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403482+0000", 2024-09-16T03:37:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403484+0000", 2024-09-16T03:37:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6640000000000001e-06 2024-09-16T03:37:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403791+0000", 2024-09-16T03:37:30.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000307287 2024-09-16T03:37:30.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403802+0000", 2024-09-16T03:37:30.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1482e-05 2024-09-16T03:37:30.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403808+0000", 2024-09-16T03:37:30.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7049999999999996e-06 2024-09-16T03:37:30.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403860+0000", 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1678999999999998e-05 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403965+0000", 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.404111+0000", 2024-09-16T03:37:30.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014622600000000001 2024-09-16T03:37:30.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.503942+0000", 2024-09-16T03:37:30.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099831195999999997 2024-09-16T03:37:30.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.503956+0000", 2024-09-16T03:37:30.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3363000000000001e-05 2024-09-16T03:37:30.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601187+0000", 2024-09-16T03:37:30.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097231834000000003 2024-09-16T03:37:30.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601216+0000", 2024-09-16T03:37:30.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8986e-05 2024-09-16T03:37:30.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601246+0000", 2024-09-16T03:37:30.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9363e-05 2024-09-16T03:37:30.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.590 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.590 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:83 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 2336140] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:30.590 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:37:30.590 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.144061253, 2024-09-16T03:37:30.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29747773399999999, 2024-09-16T03:37:30.590 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.590 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.590 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.591 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.591 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-09-16T03:37:30.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.591 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:37:30.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:37:30.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403822+0000", 2024-09-16T03:37:30.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-07 2024-09-16T03:37:30.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403825+0000", 2024-09-16T03:37:30.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000001e-06 2024-09-16T03:37:30.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403831+0000", 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9399999999999999e-06 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403834+0000", 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1939999999999998e-06 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.404994+0000", 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011601249999999999 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405083+0000", 2024-09-16T03:37:30.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8809999999999998e-05 2024-09-16T03:37:30.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:30.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405217+0000", 2024-09-16T03:37:30.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013401900000000001 2024-09-16T03:37:30.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.503958+0000", 2024-09-16T03:37:30.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098741342999999995 2024-09-16T03:37:30.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.506033+0000", 2024-09-16T03:37:30.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020749660000000001 2024-09-16T03:37:30.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701227+0000", 2024-09-16T03:37:30.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.195194171 2024-09-16T03:37:30.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701262+0000", 2024-09-16T03:37:30.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4768999999999999e-05 2024-09-16T03:37:30.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701299+0000", 2024-09-16T03:37:30.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6884000000000001e-05 2024-09-16T03:37:30.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.596 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.596 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:84 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:30.597 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:37:30.597 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.144029528000001, 2024-09-16T03:37:30.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29760038700000002, 2024-09-16T03:37:30.597 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.597 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:30.597 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.597 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.597 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.597 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-09-16T03:37:30.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.597 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.58e-07 2024-09-16T03:37:30.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403861+0000", 2024-09-16T03:37:30.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7689999999999996e-06 2024-09-16T03:37:30.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403866+0000", 2024-09-16T03:37:30.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6330000000000004e-06 2024-09-16T03:37:30.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403869+0000", 2024-09-16T03:37:30.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7390000000000001e-06 2024-09-16T03:37:30.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405790+0000", 2024-09-16T03:37:30.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001920351 2024-09-16T03:37:30.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:30.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405865+0000", 2024-09-16T03:37:30.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5234999999999994e-05 2024-09-16T03:37:30.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701328+0000", 2024-09-16T03:37:30.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29546253300000003 2024-09-16T03:37:30.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701382+0000", 2024-09-16T03:37:30.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4481999999999998e-05 2024-09-16T03:37:30.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701453+0000", 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1186999999999995e-05 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:129 4.c 4:34492f38:::smithi15736696-33:head [write 537824~473380,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.719747+0000", 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.828135541000002, 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10914728899999999, 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.602 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.603 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 129 2024-09-16T03:37:30.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.603 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.719747+0000", 2024-09-16T03:37:30.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.719747+0000", 2024-09-16T03:37:30.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.719750+0000", 2024-09-16T03:37:30.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2729999999999998e-06 2024-09-16T03:37:30.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.720187+0000", 2024-09-16T03:37:30.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043690100000000001 2024-09-16T03:37:30.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.720198+0000", 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1242e-05 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.720208+0000", 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3700000000000001e-06 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.720263+0000", 2024-09-16T03:37:30.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5268e-05 2024-09-16T03:37:30.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.720364+0000", 2024-09-16T03:37:30.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010131100000000001 2024-09-16T03:37:30.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:30.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.720468+0000", 2024-09-16T03:37:30.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103853 2024-09-16T03:37:30.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.721631+0000", 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00116289 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.822884+0000", 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10125332300000001 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828818+0000", 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059341020000000001 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828851+0000", 2024-09-16T03:37:30.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2880999999999999e-05 2024-09-16T03:37:30.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.828894+0000", 2024-09-16T03:37:30.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2874999999999999e-05 2024-09-16T03:37:30.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.608 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.609 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:432 4.c 4:34492f38:::smithi15736696-33:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:30.609 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:37:30.609 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.6340502879999992, 2024-09-16T03:37:30.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.404460861, 2024-09-16T03:37:30.609 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.609 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.609 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.609 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 432 2024-09-16T03:37:30.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.610 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:37:30.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:37:30.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913833+0000", 2024-09-16T03:37:30.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0610000000000001e-06 2024-09-16T03:37:30.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913839+0000", 2024-09-16T03:37:30.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9479999999999996e-06 2024-09-16T03:37:30.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913848+0000", 2024-09-16T03:37:30.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5629999999999992e-06 2024-09-16T03:37:30.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913857+0000", 2024-09-16T03:37:30.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0399999999999998e-06 2024-09-16T03:37:30.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914022+0000", 2024-09-16T03:37:30.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000165422 2024-09-16T03:37:30.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914101+0000", 2024-09-16T03:37:30.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8462999999999997e-05 2024-09-16T03:37:30.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:30.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217171+0000", 2024-09-16T03:37:30.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30307040000000002 2024-09-16T03:37:30.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217854+0000", 2024-09-16T03:37:30.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068278700000000002 2024-09-16T03:37:30.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.218098+0000", 2024-09-16T03:37:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243825 2024-09-16T03:37:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318164+0000", 2024-09-16T03:37:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100066451 2024-09-16T03:37:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318232+0000", 2024-09-16T03:37:30.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8113000000000003e-05 2024-09-16T03:37:30.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318293+0000", 2024-09-16T03:37:30.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0788000000000001e-05 2024-09-16T03:37:30.615 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.615 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.615 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.615 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:482 4.c 4:34492f38:::smithi15736696-33:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:30.615 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.621522+0000", 2024-09-16T03:37:30.615 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9263608049999998, 2024-09-16T03:37:30.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100442027, 2024-09-16T03:37:30.616 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.616 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.616 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:30.616 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.616 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.616 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.616 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 482 2024-09-16T03:37:30.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.616 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621522+0000", 2024-09-16T03:37:30.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621522+0000", 2024-09-16T03:37:30.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621524+0000", 2024-09-16T03:37:30.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:37:30.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621535+0000", 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1593000000000001e-05 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621556+0000", 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0366000000000001e-05 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.721667+0000", 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100111716 2024-09-16T03:37:30.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.721733+0000", 2024-09-16T03:37:30.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5228999999999995e-05 2024-09-16T03:37:30.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.721851+0000", 2024-09-16T03:37:30.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118678 2024-09-16T03:37:30.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.721964+0000", 2024-09-16T03:37:30.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112315 2024-09-16T03:37:30.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.620 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.620 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:532 4.c 4:34492f38:::smithi15736696-33:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:30.620 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:22.428519+0000", 2024-09-16T03:37:30.620 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1193636050000002, 2024-09-16T03:37:30.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100438534, 2024-09-16T03:37:30.620 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.620 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.620 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 532 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.428519+0000", 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.428519+0000", 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.428522+0000", 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7930000000000002e-06 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.428532+0000", 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0241000000000001e-05 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.428555+0000", 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3377e-05 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.528645+0000", 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100090053 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.528781+0000", 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135329 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.528885+0000", 2024-09-16T03:37:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010495199999999999 2024-09-16T03:37:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.528957+0000", 2024-09-16T03:37:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1789e-05 2024-09-16T03:37:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.625 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:790 4.d 4:b3fe05b2:::smithi15736696-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:30.625 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:37:30.625 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6599102139999999, 2024-09-16T03:37:30.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41456747799999999, 2024-09-16T03:37:30.625 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.625 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.625 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:30.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.625 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 790 2024-09-16T03:37:30.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.625 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:37:30.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:37:30.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887975+0000", 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0369999999999999e-06 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887987+0000", 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1606e-05 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888016+0000", 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.917e-05 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888032+0000", 2024-09-16T03:37:30.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5616000000000001e-05 2024-09-16T03:37:30.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888178+0000", 2024-09-16T03:37:30.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612100000000001 2024-09-16T03:37:30.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888307+0000", 2024-09-16T03:37:30.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012953899999999999 2024-09-16T03:37:30.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.302540+0000", 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41423238899999998 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(osd.12.0:15 4.d e47/32 ondisk, result = 0)", 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.303293+0000", 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2445894910000002, 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20036232300000001, 2024-09-16T03:37:30.629 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.630 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.630 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:30.630 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.303293+0000", 2024-09-16T03:37:30.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.303293+0000", 2024-09-16T03:37:30.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.303294+0000", 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2440000000000001e-06 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.303298+0000", 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8689999999999997e-06 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.303308+0000", 2024-09-16T03:37:30.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9860000000000006e-06 2024-09-16T03:37:30.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.303316+0000", 2024-09-16T03:37:30.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.354e-06 2024-09-16T03:37:30.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.503625+0000", 2024-09-16T03:37:30.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200308348 2024-09-16T03:37:30.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.503635+0000", 2024-09-16T03:37:30.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0158e-05 2024-09-16T03:37:30.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.503655+0000", 2024-09-16T03:37:30.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0364e-05 2024-09-16T03:37:30.633 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.633 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.633 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.633 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:854 4.c 4:34492f38:::smithi15736696-33:head [write 1719003~413684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:30.634 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.574001+0000", 2024-09-16T03:37:30.634 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9738815770000002, 2024-09-16T03:37:30.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.161642646, 2024-09-16T03:37:30.634 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.634 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.634 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.634 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.634 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 854 2024-09-16T03:37:30.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.634 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.574001+0000", 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.574001+0000", 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.574003+0000", 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.441e-06 2024-09-16T03:37:30.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.574455+0000", 2024-09-16T03:37:30.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00045155199999999998 2024-09-16T03:37:30.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.574466+0000", 2024-09-16T03:37:30.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1484000000000001e-05 2024-09-16T03:37:30.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.574474+0000", 2024-09-16T03:37:30.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8310000000000001e-06 2024-09-16T03:37:30.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578800+0000", 2024-09-16T03:37:30.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043262700000000001 2024-09-16T03:37:30.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578889+0000", 2024-09-16T03:37:30.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8823e-05 2024-09-16T03:37:30.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:30.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.579080+0000", 2024-09-16T03:37:30.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019086799999999999 2024-09-16T03:37:30.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.674713+0000", 2024-09-16T03:37:30.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095632582999999993 2024-09-16T03:37:30.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.674741+0000", 2024-09-16T03:37:30.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8229e-05 2024-09-16T03:37:30.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735345+0000", 2024-09-16T03:37:30.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060603919999999999 2024-09-16T03:37:30.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735587+0000", 2024-09-16T03:37:30.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024180499999999999 2024-09-16T03:37:30.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735643+0000", 2024-09-16T03:37:30.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6839999999999998e-05 2024-09-16T03:37:30.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.640 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.640 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:855 4.c 4:34492f38:::smithi15736696-33:head [write 2632287~325712] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:30.640 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.578453+0000", 2024-09-16T03:37:30.641 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9694291370000001, 2024-09-16T03:37:30.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.157366173, 2024-09-16T03:37:30.641 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.641 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.641 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.641 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 855 2024-09-16T03:37:30.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.641 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578453+0000", 2024-09-16T03:37:30.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578453+0000", 2024-09-16T03:37:30.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578455+0000", 2024-09-16T03:37:30.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7439999999999999e-06 2024-09-16T03:37:30.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578719+0000", 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026412300000000001 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578733+0000", 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4079e-05 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578742+0000", 2024-09-16T03:37:30.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9339999999999995e-06 2024-09-16T03:37:30.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.580343+0000", 2024-09-16T03:37:30.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001600527 2024-09-16T03:37:30.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.580416+0000", 2024-09-16T03:37:30.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3590000000000005e-05 2024-09-16T03:37:30.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:30.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.580514+0000", 2024-09-16T03:37:30.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8065000000000004e-05 2024-09-16T03:37:30.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.674748+0000", 2024-09-16T03:37:30.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094233773000000007 2024-09-16T03:37:30.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735235+0000", 2024-09-16T03:37:30.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060487372999999997 2024-09-16T03:37:30.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735708+0000", 2024-09-16T03:37:30.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047290800000000002 2024-09-16T03:37:30.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735766+0000", 2024-09-16T03:37:30.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7294999999999999e-05 2024-09-16T03:37:30.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735819+0000", 2024-09-16T03:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3761999999999998e-05 2024-09-16T03:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:856 4.c 4:34492f38:::smithi15736696-33:head [setxattr _header (56) in=7b,truncate 2957999] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.578760+0000", 2024-09-16T03:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9691224470000002, 2024-09-16T03:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.157244578, 2024-09-16T03:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.647 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 856 2024-09-16T03:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578760+0000", 2024-09-16T03:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578760+0000", 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578761+0000", 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.15e-06 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578767+0000", 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4469999999999997e-06 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578776+0000", 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0469999999999999e-06 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578783+0000", 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2690000000000002e-06 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.581163+0000", 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023805499999999999 2024-09-16T03:37:30.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.581244+0000", 2024-09-16T03:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0698000000000006e-05 2024-09-16T03:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.581326+0000", 2024-09-16T03:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2303000000000001e-05 2024-09-16T03:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:30.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.674755+0000", 2024-09-16T03:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093428323999999993 2024-09-16T03:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735276+0000", 2024-09-16T03:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060521681000000001 2024-09-16T03:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735892+0000", 2024-09-16T03:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061523800000000005 2024-09-16T03:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735944+0000", 2024-09-16T03:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1938999999999999e-05 2024-09-16T03:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.736004+0000", 2024-09-16T03:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0931999999999999e-05 2024-09-16T03:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:857 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.578794+0000", 2024-09-16T03:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9690886480000001, 2024-09-16T03:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15739841900000001, 2024-09-16T03:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 857 2024-09-16T03:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.654 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578794+0000", 2024-09-16T03:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578794+0000", 2024-09-16T03:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578795+0000", 2024-09-16T03:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.54e-07 2024-09-16T03:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578798+0000", 2024-09-16T03:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.765e-06 2024-09-16T03:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578807+0000", 2024-09-16T03:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:37:30.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.578813+0000", 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3389999999999998e-06 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.634560+0000", 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055746750999999997 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.634620+0000", 2024-09-16T03:37:30.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9969000000000001e-05 2024-09-16T03:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.736045+0000", 2024-09-16T03:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10142435 2024-09-16T03:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.736109+0000", 2024-09-16T03:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4840999999999998e-05 2024-09-16T03:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.736192+0000", 2024-09-16T03:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2752999999999999e-05 2024-09-16T03:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:855 4.c e47/32 ondisk, result = 0)", 2024-09-16T03:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.634907+0000", 2024-09-16T03:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9129754610000003, 2024-09-16T03:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100338173, 2024-09-16T03:37:30.659 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.634907+0000", 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.634907+0000", 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.634908+0000", 2024-09-16T03:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0529999999999999e-06 2024-09-16T03:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.634912+0000", 2024-09-16T03:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6500000000000002e-06 2024-09-16T03:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.634921+0000", 2024-09-16T03:37:30.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4700000000000008e-06 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735010+0000", 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10008927500000001 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735213+0000", 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020301499999999999 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:30.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735225+0000", 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.129e-05 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.735245+0000", 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0420000000000001e-05 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:987 4.c 4:3313e97c:::smithi15736696-21:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:30.372308+0000", 2024-09-16T03:37:30.663 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.17557413599999999, 2024-09-16T03:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100545467, 2024-09-16T03:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 987 2024-09-16T03:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:30.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.372308+0000", 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.372308+0000", 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.372309+0000", 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7899999999999999e-07 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.372313+0000", 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.031e-06 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.372320+0000", 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9870000000000002e-06 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.372323+0000", 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6009999999999999e-06 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.472721+0000", 2024-09-16T03:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100397815 2024-09-16T03:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.472854+0000", 2024-09-16T03:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013255400000000001 2024-09-16T03:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:30.667 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:30.668 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:30.670 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:37:30.670 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:30.670 INFO:tasks.rados.rados.0.smithi157.stdout:675: finishing write tid 2 to smithi15736696-28 2024-09-16T03:37:30.670 INFO:tasks.rados.rados.0.smithi157.stdout:675: finishing write tid 3 to smithi15736696-28 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:675: finishing write tid 4 to smithi15736696-28 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:675: oid 28 updating version 0 to 30 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:675: oid 28 version 30 is already newer than 29 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:675: oid 28 updating version 30 to 31 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 31 (ObjNum 67 snap 0 seq_num 67) dirty exists 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:675: left oid 28 (ObjNum 67 snap 0 seq_num 67) 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:674: done (8 left) 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:675: done (7 left) 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:678: done (6 left) 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:679: done (5 left) 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:685: read oid 46 snap -1 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:685: expect (ObjNum 54 snap 0 seq_num 54) 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:686: read oid 11 snap -1 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:686: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:37:30.671 INFO:tasks.rados.rados.0.smithi157.stdout:687: read oid 17 snap -1 2024-09-16T03:37:30.672 INFO:tasks.rados.rados.0.smithi157.stdout:687: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-16T03:37:30.672 INFO:tasks.rados.rados.0.smithi157.stdout:688: set_chunk oid 26 target oid 2024-09-16T03:37:30.672 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:30.667+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:30.769 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:30.768+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:30.787 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:30.787 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:30.787 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:30.787 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:30.797 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:37:30.798 INFO:tasks.rados.rados.0.smithi157.stdout:688: set_chunk oid 26 offset: 1779712 length: 605184 target oid 920f29fe85d0745d4cd24a887714ac53f1b9a978ec2208ff473c99287fdf8feb offset: 0 snap -1 2024-09-16T03:37:30.798 INFO:tasks.rados.rados.0.smithi157.stdout:689: tier_evict oid 28 2024-09-16T03:37:30.798 INFO:tasks.rados.rados.0.smithi157.stdout:689: tier_evict oid 28 snap -1 2024-09-16T03:37:30.798 INFO:tasks.rados.rados.0.smithi157.stdout:682: finishing write tid 1 to smithi15736696-45 2024-09-16T03:37:30.798 INFO:tasks.rados.rados.0.smithi157.stdout:683: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-16T03:37:30.829 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:30.871 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:30.869+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:30.890 INFO:tasks.rados.rados.0.smithi157.stdout:685: expect (ObjNum 54 snap 0 seq_num 54) 2024-09-16T03:37:30.932 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:30.941 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:37:30.942 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:37:30.942 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:30.951 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:37:30.996 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:30.994+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:31.007 INFO:tasks.rados.rados.0.smithi157.stdout:682: finishing write tid 2 to smithi15736696-45 2024-09-16T03:37:31.007 INFO:tasks.rados.rados.0.smithi157.stdout:689: got (22) Invalid argument 2024-09-16T03:37:31.007 INFO:tasks.rados.rados.0.smithi157.stdout:682: finishing write tid 3 to smithi15736696-45 2024-09-16T03:37:31.007 INFO:tasks.rados.rados.0.smithi157.stdout:682: finishing write tid 4 to smithi15736696-45 2024-09-16T03:37:31.007 INFO:tasks.rados.rados.0.smithi157.stdout:682: finishing write tid 5 to smithi15736696-45 2024-09-16T03:37:31.007 INFO:tasks.rados.rados.0.smithi157.stdout:682: oid 45 updating version 0 to 33 2024-09-16T03:37:31.007 INFO:tasks.rados.rados.0.smithi157.stdout:682: oid 45 version 33 is already newer than 31 2024-09-16T03:37:31.007 INFO:tasks.rados.rados.0.smithi157.stdout:682: oid 45 version 33 is already newer than 32 2024-09-16T03:37:31.007 INFO:tasks.rados.rados.0.smithi157.stdout:682: oid 45 version 33 is already newer than 30 2024-09-16T03:37:31.007 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 33 (ObjNum 69 snap 0 seq_num 69) dirty exists 2024-09-16T03:37:31.007 INFO:tasks.rados.rados.0.smithi157.stdout:682: left oid 45 (ObjNum 69 snap 0 seq_num 69) 2024-09-16T03:37:31.008 INFO:tasks.rados.rados.0.smithi157.stdout:688: finishing set_chunk to oid 26 2024-09-16T03:37:31.008 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 26 set_chunk returned error code -95 offset: 1779712 length: 605184 2024-09-16T03:37:31.008 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:37:31.008 INFO:tasks.rados.rados.0.smithi157.stdout:681: done (9 left) 2024-09-16T03:37:31.008 INFO:tasks.rados.rados.0.smithi157.stdout:682: done (8 left) 2024-09-16T03:37:31.008 INFO:tasks.rados.rados.0.smithi157.stdout:683: done (7 left) 2024-09-16T03:37:31.008 INFO:tasks.rados.rados.0.smithi157.stdout:685: done (6 left) 2024-09-16T03:37:31.008 INFO:tasks.rados.rados.0.smithi157.stdout:688: done (5 left) 2024-09-16T03:37:31.008 INFO:tasks.rados.rados.0.smithi157.stdout:689: done (4 left) 2024-09-16T03:37:31.008 INFO:tasks.rados.rados.0.smithi157.stdout:690: set_chunk oid 15 target oid 2024-09-16T03:37:31.042 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:31.040+0000 7f652ae6c640 -1 osd.1 50 handle_osd_map injecting map crc failure 2024-09-16T03:37:31.044 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:37:31.046 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:31.044+0000 7f49439d4640 -1 osd.4 50 handle_osd_map injecting map crc failure 2024-09-16T03:37:31.048 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:31.046+0000 7f3a1f71e640 -1 osd.6 50 handle_osd_map injecting map crc failure 2024-09-16T03:37:31.059 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:31.059 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:31.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:31.059 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:37:31.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.059 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:10 4.8 e38/32 4:13b03fc9:::smithi15736696-3:head v 38'2, mlcod=0'0)", 2024-09-16T03:37:31.059 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.085506+0000", 2024-09-16T03:37:31.059 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.970874593000001, 2024-09-16T03:37:31.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038863740000000002, 2024-09-16T03:37:31.059 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.060 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.060 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.060 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.085506+0000", 2024-09-16T03:37:31.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.085506+0000", 2024-09-16T03:37:31.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.085508+0000", 2024-09-16T03:37:31.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5719999999999999e-06 2024-09-16T03:37:31.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.085704+0000", 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000195832 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.085718+0000", 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3933e-05 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.085728+0000", 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0325e-05 2024-09-16T03:37:31.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.085785+0000", 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7250000000000002e-05 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.085802+0000", 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6994e-05 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.089343+0000", 2024-09-16T03:37:31.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035411499999999999 2024-09-16T03:37:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.089393+0000", 2024-09-16T03:37:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9317999999999998e-05 2024-09-16T03:37:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:11 4.8 e38/32 4:13b03fc9:::smithi15736696-3:head v 38'3, mlcod=0'0)", 2024-09-16T03:37:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.086542+0000", 2024-09-16T03:37:31.064 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.969839018999998, 2024-09-16T03:37:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0031939020000000002, 2024-09-16T03:37:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.086542+0000", 2024-09-16T03:37:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.065 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.086542+0000", 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.086543+0000", 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2300000000000001e-06 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.086550+0000", 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7240000000000003e-06 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.086557+0000", 2024-09-16T03:37:31.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4460000000000004e-06 2024-09-16T03:37:31.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.086563+0000", 2024-09-16T03:37:31.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.558e-06 2024-09-16T03:37:31.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.086605+0000", 2024-09-16T03:37:31.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2311000000000002e-05 2024-09-16T03:37:31.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.086623+0000", 2024-09-16T03:37:31.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7598e-05 2024-09-16T03:37:31.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.089699+0000", 2024-09-16T03:37:31.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030757530000000001 2024-09-16T03:37:31.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.089736+0000", 2024-09-16T03:37:31.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7282e-05 2024-09-16T03:37:31.069 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.069 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.069 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:21 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'4, mlcod=0'0)", 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.869082795000001, 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10231100799999999, 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:37:31.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187300+0000", 2024-09-16T03:37:31.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0360000000000001e-06 2024-09-16T03:37:31.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187905+0000", 2024-09-16T03:37:31.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060508100000000004 2024-09-16T03:37:31.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187920+0000", 2024-09-16T03:37:31.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4314000000000001e-05 2024-09-16T03:37:31.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187928+0000", 2024-09-16T03:37:31.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7150000000000004e-06 2024-09-16T03:37:31.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187984+0000", 2024-09-16T03:37:31.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5872999999999999e-05 2024-09-16T03:37:31.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188016+0000", 2024-09-16T03:37:31.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2045000000000002e-05 2024-09-16T03:37:31.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289567+0000", 2024-09-16T03:37:31.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101550875 2024-09-16T03:37:31.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.076 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.076 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289609+0000", 2024-09-16T03:37:31.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2069e-05 2024-09-16T03:37:31.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.076 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.076 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:22 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'5, mlcod=0'0)", 2024-09-16T03:37:31.076 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187945+0000", 2024-09-16T03:37:31.077 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.868436029000001, 2024-09-16T03:37:31.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101698394, 2024-09-16T03:37:31.077 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.077 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.077 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.077 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.077 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187945+0000", 2024-09-16T03:37:31.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187945+0000", 2024-09-16T03:37:31.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187946+0000", 2024-09-16T03:37:31.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9900000000000001e-07 2024-09-16T03:37:31.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187959+0000", 2024-09-16T03:37:31.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3242e-05 2024-09-16T03:37:31.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187963+0000", 2024-09-16T03:37:31.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8410000000000002e-06 2024-09-16T03:37:31.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187966+0000", 2024-09-16T03:37:31.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9390000000000002e-06 2024-09-16T03:37:31.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188816+0000", 2024-09-16T03:37:31.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085090000000000003 2024-09-16T03:37:31.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188836+0000", 2024-09-16T03:37:31.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9638000000000001e-05 2024-09-16T03:37:31.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289615+0000", 2024-09-16T03:37:31.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10077847299999999 2024-09-16T03:37:31.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289643+0000", 2024-09-16T03:37:31.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8861999999999999e-05 2024-09-16T03:37:31.081 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.082 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.082 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.082 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:23 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'6, mlcod=0'0)", 2024-09-16T03:37:31.082 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187987+0000", 2024-09-16T03:37:31.082 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.868393750999999, 2024-09-16T03:37:31.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102020717, 2024-09-16T03:37:31.082 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.082 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.082 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.082 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187987+0000", 2024-09-16T03:37:31.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187987+0000", 2024-09-16T03:37:31.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187988+0000", 2024-09-16T03:37:31.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9900000000000001e-07 2024-09-16T03:37:31.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187992+0000", 2024-09-16T03:37:31.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.707e-06 2024-09-16T03:37:31.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187998+0000", 2024-09-16T03:37:31.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7699999999999998e-06 2024-09-16T03:37:31.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188003+0000", 2024-09-16T03:37:31.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8459999999999999e-06 2024-09-16T03:37:31.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289285+0000", 2024-09-16T03:37:31.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101282107 2024-09-16T03:37:31.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289343+0000", 2024-09-16T03:37:31.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7852000000000001e-05 2024-09-16T03:37:31.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289968+0000", 2024-09-16T03:37:31.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062506400000000004 2024-09-16T03:37:31.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.087 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.290008+0000", 2024-09-16T03:37:31.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9872e-05 2024-09-16T03:37:31.087 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.087 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.087 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.087 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:81 4.c e39/32 4:3313e97c:::smithi15736696-21:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:31.087 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.398719+0000", 2024-09-16T03:37:31.087 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.657661774000001, 2024-09-16T03:37:31.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102114725, 2024-09-16T03:37:31.088 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.088 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.088 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.088 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.398719+0000", 2024-09-16T03:37:31.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.398719+0000", 2024-09-16T03:37:31.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.398722+0000", 2024-09-16T03:37:31.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7199999999999998e-06 2024-09-16T03:37:31.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.399189+0000", 2024-09-16T03:37:31.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046724000000000001 2024-09-16T03:37:31.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.090 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.399200+0000", 2024-09-16T03:37:31.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1068e-05 2024-09-16T03:37:31.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.090 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499313+0000", 2024-09-16T03:37:31.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100112249 2024-09-16T03:37:31.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499411+0000", 2024-09-16T03:37:31.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8930000000000003e-05 2024-09-16T03:37:31.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499444+0000", 2024-09-16T03:37:31.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2966000000000003e-05 2024-09-16T03:37:31.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.500783+0000", 2024-09-16T03:37:31.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013380880000000001 2024-09-16T03:37:31.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.500834+0000", 2024-09-16T03:37:31.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1464000000000001e-05 2024-09-16T03:37:31.092 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.092 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.092 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.093 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:82 4.c e39/32 4:3313e97c:::smithi15736696-21:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:31.093 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.499352+0000", 2024-09-16T03:37:31.093 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.557029478, 2024-09-16T03:37:31.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003828676, 2024-09-16T03:37:31.093 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.093 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.093 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.093 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499352+0000", 2024-09-16T03:37:31.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499352+0000", 2024-09-16T03:37:31.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499353+0000", 2024-09-16T03:37:31.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8330000000000001e-06 2024-09-16T03:37:31.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499550+0000", 2024-09-16T03:37:31.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019670199999999999 2024-09-16T03:37:31.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499565+0000", 2024-09-16T03:37:31.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5173e-05 2024-09-16T03:37:31.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499575+0000", 2024-09-16T03:37:31.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7289999999999995e-06 2024-09-16T03:37:31.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.500127+0000", 2024-09-16T03:37:31.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055166899999999997 2024-09-16T03:37:31.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.500146+0000", 2024-09-16T03:37:31.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8975e-05 2024-09-16T03:37:31.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.503147+0000", 2024-09-16T03:37:31.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.003001105 2024-09-16T03:37:31.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.503180+0000", 2024-09-16T03:37:31.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.349e-05 2024-09-16T03:37:31.098 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.098 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.098 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.098 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:83 4.c e39/32 4:3313e97c:::smithi15736696-21:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:31.098 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.499603+0000", 2024-09-16T03:37:31.098 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.556778089000002, 2024-09-16T03:37:31.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0066039699999999998, 2024-09-16T03:37:31.099 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.099 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.099 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.099 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499603+0000", 2024-09-16T03:37:31.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499603+0000", 2024-09-16T03:37:31.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499603+0000", 2024-09-16T03:37:31.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9400000000000001e-07 2024-09-16T03:37:31.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499606+0000", 2024-09-16T03:37:31.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7140000000000002e-06 2024-09-16T03:37:31.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499609+0000", 2024-09-16T03:37:31.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8660000000000002e-06 2024-09-16T03:37:31.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499611+0000", 2024-09-16T03:37:31.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5210000000000001e-06 2024-09-16T03:37:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.500860+0000", 2024-09-16T03:37:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012489109999999999 2024-09-16T03:37:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.500878+0000", 2024-09-16T03:37:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7571000000000002e-05 2024-09-16T03:37:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.506175+0000", 2024-09-16T03:37:31.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052967839999999997 2024-09-16T03:37:31.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.506207+0000", 2024-09-16T03:37:31.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2308999999999998e-05 2024-09-16T03:37:31.103 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.103 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.103 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.104 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:279 4.8 e41/32 4:1f63731d:::smithi15736696-30:head v 41'15, mlcod=40'13)", 2024-09-16T03:37:31.104 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.500221+0000", 2024-09-16T03:37:31.104 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.556160209, 2024-09-16T03:37:31.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100673015, 2024-09-16T03:37:31.104 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.104 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.104 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.104 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500221+0000", 2024-09-16T03:37:31.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500221+0000", 2024-09-16T03:37:31.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500221+0000", 2024-09-16T03:37:31.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3500000000000002e-07 2024-09-16T03:37:31.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500229+0000", 2024-09-16T03:37:31.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7209999999999994e-06 2024-09-16T03:37:31.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500242+0000", 2024-09-16T03:37:31.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2699e-05 2024-09-16T03:37:31.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.600342+0000", 2024-09-16T03:37:31.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100100734 2024-09-16T03:37:31.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.600425+0000", 2024-09-16T03:37:31.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2952000000000001e-05 2024-09-16T03:37:31.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.600447+0000", 2024-09-16T03:37:31.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.181e-05 2024-09-16T03:37:31.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.600853+0000", 2024-09-16T03:37:31.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040597400000000003 2024-09-16T03:37:31.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.600894+0000", 2024-09-16T03:37:31.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0790000000000001e-05 2024-09-16T03:37:31.109 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.109 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.109 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.109 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:302 4.8 e42/32 4:13b03fc9:::smithi15736696-3:head v 42'16, mlcod=40'14)", 2024-09-16T03:37:31.109 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.317007+0000", 2024-09-16T03:37:31.109 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.739374250000001, 2024-09-16T03:37:31.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100648134, 2024-09-16T03:37:31.109 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.109 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.109 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.110 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.317007+0000", 2024-09-16T03:37:31.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.317007+0000", 2024-09-16T03:37:31.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.317007+0000", 2024-09-16T03:37:31.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4499999999999997e-07 2024-09-16T03:37:31.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.317012+0000", 2024-09-16T03:37:31.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5730000000000001e-06 2024-09-16T03:37:31.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.317024+0000", 2024-09-16T03:37:31.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2659e-05 2024-09-16T03:37:31.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.417120+0000", 2024-09-16T03:37:31.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100095954 2024-09-16T03:37:31.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.417163+0000", 2024-09-16T03:37:31.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2546999999999998e-05 2024-09-16T03:37:31.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.417177+0000", 2024-09-16T03:37:31.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3754000000000001e-05 2024-09-16T03:37:31.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.417620+0000", 2024-09-16T03:37:31.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044358400000000001 2024-09-16T03:37:31.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.417655+0000", 2024-09-16T03:37:31.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4618e-05 2024-09-16T03:37:31.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.114 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.114 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.5.21:6 3.0 e43/42 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 43'21, mlcod=0'0)", 2024-09-16T03:37:31.114 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:17.284972+0000", 2024-09-16T03:37:31.114 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.771409183999999, 2024-09-16T03:37:31.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.1008151, 2024-09-16T03:37:31.115 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.115 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.115 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.115 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284972+0000", 2024-09-16T03:37:31.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284972+0000", 2024-09-16T03:37:31.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.116 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284973+0000", 2024-09-16T03:37:31.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8299999999999995e-07 2024-09-16T03:37:31.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284988+0000", 2024-09-16T03:37:31.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4946000000000001e-05 2024-09-16T03:37:31.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284997+0000", 2024-09-16T03:37:31.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3440000000000007e-06 2024-09-16T03:37:31.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385069+0000", 2024-09-16T03:37:31.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10007182000000001 2024-09-16T03:37:31.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385127+0000", 2024-09-16T03:37:31.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7676999999999997e-05 2024-09-16T03:37:31.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385152+0000", 2024-09-16T03:37:31.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5395999999999999e-05 2024-09-16T03:37:31.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385754+0000", 2024-09-16T03:37:31.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060247899999999995 2024-09-16T03:37:31.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.119 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385787+0000", 2024-09-16T03:37:31.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2555000000000003e-05 2024-09-16T03:37:31.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.119 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.120 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:26 3.0 e45/44 3:114c937e:::smithi15736696-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'36, mlcod=45'34)", 2024-09-16T03:37:31.120 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.106326+0000", 2024-09-16T03:37:31.120 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.950054645, 2024-09-16T03:37:31.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100770135, 2024-09-16T03:37:31.120 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.120 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.120 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.120 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.106326+0000", 2024-09-16T03:37:31.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.106326+0000", 2024-09-16T03:37:31.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.106328+0000", 2024-09-16T03:37:31.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7680000000000001e-06 2024-09-16T03:37:31.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.122 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.106341+0000", 2024-09-16T03:37:31.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2727e-05 2024-09-16T03:37:31.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.122 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.106354+0000", 2024-09-16T03:37:31.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3569e-05 2024-09-16T03:37:31.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.206423+0000", 2024-09-16T03:37:31.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100068303 2024-09-16T03:37:31.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.206486+0000", 2024-09-16T03:37:31.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3242000000000002e-05 2024-09-16T03:37:31.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.206511+0000", 2024-09-16T03:37:31.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5015999999999999e-05 2024-09-16T03:37:31.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.207066+0000", 2024-09-16T03:37:31.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055467199999999998 2024-09-16T03:37:31.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.207097+0000", 2024-09-16T03:37:31.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0837999999999998e-05 2024-09-16T03:37:31.125 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.125 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.125 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.125 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.21:5 3.0 e45/44 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'39, mlcod=45'37)", 2024-09-16T03:37:31.125 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:37:31.126 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.141785761, 2024-09-16T03:37:31.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20119541799999999, 2024-09-16T03:37:31.126 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.126 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.126 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.126 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:37:31.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:37:31.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914597+0000", 2024-09-16T03:37:31.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.494e-06 2024-09-16T03:37:31.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914603+0000", 2024-09-16T03:37:31.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7349999999999999e-06 2024-09-16T03:37:31.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914614+0000", 2024-09-16T03:37:31.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0995999999999999e-05 2024-09-16T03:37:31.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914623+0000", 2024-09-16T03:37:31.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3899999999999993e-06 2024-09-16T03:37:31.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.114941+0000", 2024-09-16T03:37:31.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20031812399999999 2024-09-16T03:37:31.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115001+0000", 2024-09-16T03:37:31.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0161999999999998e-05 2024-09-16T03:37:31.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115750+0000", 2024-09-16T03:37:31.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074851699999999998 2024-09-16T03:37:31.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115791+0000", 2024-09-16T03:37:31.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1e-05 2024-09-16T03:37:31.130 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.131 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.131 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.131 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:25 3.0 e45/44 3:0a4aa513:::smithi15736696-10 low_tier:head v 45'41, mlcod=45'37)", 2024-09-16T03:37:31.131 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.116538+0000", 2024-09-16T03:37:31.131 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.9398435260000007, 2024-09-16T03:37:31.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.005099063, 2024-09-16T03:37:31.131 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.131 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.132 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.132 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.132 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.116538+0000", 2024-09-16T03:37:31.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.132 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.116538+0000", 2024-09-16T03:37:31.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.133 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.116539+0000", 2024-09-16T03:37:31.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5340000000000001e-06 2024-09-16T03:37:31.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.133 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.116548+0000", 2024-09-16T03:37:31.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6209999999999993e-06 2024-09-16T03:37:31.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.116560+0000", 2024-09-16T03:37:31.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2804e-05 2024-09-16T03:37:31.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.116569+0000", 2024-09-16T03:37:31.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5439999999999999e-06 2024-09-16T03:37:31.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.116620+0000", 2024-09-16T03:37:31.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1174000000000002e-05 2024-09-16T03:37:31.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.116635+0000", 2024-09-16T03:37:31.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4541e-05 2024-09-16T03:37:31.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.121588+0000", 2024-09-16T03:37:31.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0049534180000000002 2024-09-16T03:37:31.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.121637+0000", 2024-09-16T03:37:31.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8427000000000002e-05 2024-09-16T03:37:31.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.136 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.136 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:432 4.c e45/32 4:34492f38:::smithi15736696-33:head v 45'10, mlcod=43'8)", 2024-09-16T03:37:31.136 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.217233+0000", 2024-09-16T03:37:31.136 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8391483579999992, 2024-09-16T03:37:31.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100706, 2024-09-16T03:37:31.137 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.137 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.137 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.137 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217233+0000", 2024-09-16T03:37:31.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217233+0000", 2024-09-16T03:37:31.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217234+0000", 2024-09-16T03:37:31.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6750000000000001e-06 2024-09-16T03:37:31.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217243+0000", 2024-09-16T03:37:31.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4600000000000003e-06 2024-09-16T03:37:31.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217256+0000", 2024-09-16T03:37:31.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3417999999999999e-05 2024-09-16T03:37:31.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317325+0000", 2024-09-16T03:37:31.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100068803 2024-09-16T03:37:31.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317390+0000", 2024-09-16T03:37:31.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5381000000000003e-05 2024-09-16T03:37:31.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317415+0000", 2024-09-16T03:37:31.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4487000000000001e-05 2024-09-16T03:37:31.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317904+0000", 2024-09-16T03:37:31.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00048901100000000002 2024-09-16T03:37:31.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.141 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.141 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317939+0000", 2024-09-16T03:37:31.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4764999999999997e-05 2024-09-16T03:37:31.141 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.141 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.141 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.142 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:854 4.c e47/32 4:34492f38:::smithi15736696-33:head v 47'13, mlcod=45'10)", 2024-09-16T03:37:31.142 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.579183+0000", 2024-09-16T03:37:31.142 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.4771984570000001, 2024-09-16T03:37:31.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10085426, 2024-09-16T03:37:31.142 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.142 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.142 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.142 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.142 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.142 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.579183+0000", 2024-09-16T03:37:31.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.579183+0000", 2024-09-16T03:37:31.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.579184+0000", 2024-09-16T03:37:31.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.838e-06 2024-09-16T03:37:31.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.579554+0000", 2024-09-16T03:37:31.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036928699999999999 2024-09-16T03:37:31.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.579569+0000", 2024-09-16T03:37:31.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5688e-05 2024-09-16T03:37:31.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.579578+0000", 2024-09-16T03:37:31.145 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1310000000000006e-06 2024-09-16T03:37:31.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580977+0000", 2024-09-16T03:37:31.145 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013990750000000001 2024-09-16T03:37:31.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580995+0000", 2024-09-16T03:37:31.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8598e-05 2024-09-16T03:37:31.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.679994+0000", 2024-09-16T03:37:31.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098998728999999994 2024-09-16T03:37:31.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.146 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.146 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.680037+0000", 2024-09-16T03:37:31.146 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2914000000000003e-05 2024-09-16T03:37:31.146 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.147 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.147 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:855 4.c e47/32 4:34492f38:::smithi15736696-33:head v 47'14, mlcod=45'10)", 2024-09-16T03:37:31.147 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.580582+0000", 2024-09-16T03:37:31.147 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.4757986409999999, 2024-09-16T03:37:31.147 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099491817999999996, 2024-09-16T03:37:31.147 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.147 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.147 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.147 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580582+0000", 2024-09-16T03:37:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580582+0000", 2024-09-16T03:37:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580584+0000", 2024-09-16T03:37:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.172e-06 2024-09-16T03:37:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580901+0000", 2024-09-16T03:37:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031786999999999999 2024-09-16T03:37:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580913+0000", 2024-09-16T03:37:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1884e-05 2024-09-16T03:37:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580920+0000", 2024-09-16T03:37:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9040000000000003e-06 2024-09-16T03:37:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.582092+0000", 2024-09-16T03:37:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011721469999999999 2024-09-16T03:37:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.582119+0000", 2024-09-16T03:37:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6437000000000001e-05 2024-09-16T03:37:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.680039+0000", 2024-09-16T03:37:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097920046999999996 2024-09-16T03:37:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.680074+0000", 2024-09-16T03:37:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5357e-05 2024-09-16T03:37:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:856 4.c e47/32 4:34492f38:::smithi15736696-33:head v 47'15, mlcod=45'10)", 2024-09-16T03:37:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.581380+0000", 2024-09-16T03:37:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.4750011509999998, 2024-09-16T03:37:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099086726999999999, 2024-09-16T03:37:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.581380+0000", 2024-09-16T03:37:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.581380+0000", 2024-09-16T03:37:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.581381+0000", 2024-09-16T03:37:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.362e-06 2024-09-16T03:37:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.581395+0000", 2024-09-16T03:37:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.379e-05 2024-09-16T03:37:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.581405+0000", 2024-09-16T03:37:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7610000000000002e-06 2024-09-16T03:37:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.581412+0000", 2024-09-16T03:37:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6070000000000002e-06 2024-09-16T03:37:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.679721+0000", 2024-09-16T03:37:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098308393999999993 2024-09-16T03:37:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.679744+0000", 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3495000000000001e-05 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.680433+0000", 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068914900000000001 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.680467+0000", 2024-09-16T03:37:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3169e-05 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.8.22:16 3.0 e47/44 3:037ddf75:::smithi15736696-34 low_tier:head v 47'52, mlcod=47'50)", 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.939321+0000", 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.1170597449999997, 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10108816800000001, 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.939321+0000", 2024-09-16T03:37:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.939321+0000", 2024-09-16T03:37:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.939323+0000", 2024-09-16T03:37:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.553e-06 2024-09-16T03:37:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.939330+0000", 2024-09-16T03:37:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4200000000000001e-06 2024-09-16T03:37:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.939347+0000", 2024-09-16T03:37:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6481999999999999e-05 2024-09-16T03:37:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.039479+0000", 2024-09-16T03:37:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100131839 2024-09-16T03:37:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.039641+0000", 2024-09-16T03:37:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016204899999999999 2024-09-16T03:37:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.039705+0000", 2024-09-16T03:37:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4146999999999994e-05 2024-09-16T03:37:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.040350+0000", 2024-09-16T03:37:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064533099999999996 2024-09-16T03:37:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.040409+0000", 2024-09-16T03:37:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9346999999999999e-05 2024-09-16T03:37:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.8.22:18 3.0 e47/44 3:037ddf75:::smithi15736696-34 low_tier:head v 47'54, mlcod=47'50)", 2024-09-16T03:37:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:27.140611+0000", 2024-09-16T03:37:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.9157696180000001, 2024-09-16T03:37:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101057298, 2024-09-16T03:37:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140611+0000", 2024-09-16T03:37:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140611+0000", 2024-09-16T03:37:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140614+0000", 2024-09-16T03:37:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2989999999999999e-06 2024-09-16T03:37:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140628+0000", 2024-09-16T03:37:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4571e-05 2024-09-16T03:37:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140645+0000", 2024-09-16T03:37:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6568000000000001e-05 2024-09-16T03:37:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140655+0000", 2024-09-16T03:37:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7620000000000007e-06 2024-09-16T03:37:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.240832+0000", 2024-09-16T03:37:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100177195 2024-09-16T03:37:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.240849+0000", 2024-09-16T03:37:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7323e-05 2024-09-16T03:37:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.241630+0000", 2024-09-16T03:37:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000780545 2024-09-16T03:37:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.241669+0000", 2024-09-16T03:37:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9035000000000001e-05 2024-09-16T03:37:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.169 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:31.171 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:37:31.172 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:31.168+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:31.214 INFO:tasks.rados.rados.0.smithi157.stdout:690: set_chunk oid 15 offset: 1634304 length: 193536 target oid 8de5e216df7365778a9df2a50922cc390a2a900c1854ce3c1d19b1da882722e1 offset: 0 snap -1 2024-09-16T03:37:31.214 INFO:tasks.rados.rados.0.smithi157.stdout:691: read oid 24 snap -1 2024-09-16T03:37:31.214 INFO:tasks.rados.rados.0.smithi157.stdout:691: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:37:31.214 INFO:tasks.rados.rados.0.smithi157.stdout:692: read oid 27 snap -1 2024-09-16T03:37:31.214 INFO:tasks.rados.rados.0.smithi157.stdout:692: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:31.215 INFO:tasks.rados.rados.0.smithi157.stdout:693: read oid 40 snap -1 2024-09-16T03:37:31.215 INFO:tasks.rados.rados.0.smithi157.stdout:693: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-16T03:37:31.215 INFO:tasks.rados.rados.0.smithi157.stdout:694: write oid 50 current snap is 0 2024-09-16T03:37:31.215 INFO:tasks.rados.rados.0.smithi157.stdout:694: seq_num 70 ranges {776490=714607,2246563=579356} 2024-09-16T03:37:31.218 INFO:tasks.rados.rados.0.smithi157.stdout:694: writing smithi15736696-50 from 776490 to 1491097 tid 1 2024-09-16T03:37:31.225 INFO:tasks.rados.rados.0.smithi157.stdout:694: writing smithi15736696-50 from 2246563 to 2825919 tid 2 2024-09-16T03:37:31.225 INFO:tasks.rados.rados.0.smithi157.stdout:695: read oid 29 snap -1 2024-09-16T03:37:31.226 INFO:tasks.rados.rados.0.smithi157.stdout:695: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-16T03:37:31.226 INFO:tasks.rados.rados.0.smithi157.stdout:696: write oid 46 current snap is 0 2024-09-16T03:37:31.226 INFO:tasks.rados.rados.0.smithi157.stdout:690: finishing set_chunk to oid 15 2024-09-16T03:37:31.226 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 15 set_chunk returned error code -95 offset: 1634304 length: 193536 2024-09-16T03:37:31.226 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:31.226 INFO:tasks.rados.rados.0.smithi157.stdout:696: seq_num 71 ranges {793297=789856,2305045=723014} 2024-09-16T03:37:31.234 INFO:tasks.rados.rados.0.smithi157.stdout:696: writing smithi15736696-46 from 793297 to 1583153 tid 1 2024-09-16T03:37:31.242 INFO:tasks.rados.rados.0.smithi157.stdout:696: writing smithi15736696-46 from 2305045 to 3028059 tid 2 2024-09-16T03:37:31.243 INFO:tasks.rados.rados.0.smithi157.stdout:690: done (10 left) 2024-09-16T03:37:31.243 INFO:tasks.rados.rados.0.smithi157.stdout:697: read oid 20 snap -1 2024-09-16T03:37:31.243 INFO:tasks.rados.rados.0.smithi157.stdout:697: expect (ObjNum 65 snap 0 seq_num 65) 2024-09-16T03:37:31.243 INFO:tasks.rados.rados.0.smithi157.stdout:698: read oid 1 snap -1 2024-09-16T03:37:31.243 INFO:tasks.rados.rados.0.smithi157.stdout:698: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T03:37:31.243 INFO:tasks.rados.rados.0.smithi157.stdout:699: read oid 41 snap -1 2024-09-16T03:37:31.243 INFO:tasks.rados.rados.0.smithi157.stdout:699: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T03:37:31.243 INFO:tasks.rados.rados.0.smithi157.stdout:700: write oid 42 current snap is 0 2024-09-16T03:37:31.243 INFO:tasks.rados.rados.0.smithi157.stdout:700: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-09-16T03:37:31.247 INFO:tasks.rados.rados.0.smithi157.stdout:700: writing smithi15736696-42 from 410104 to 875209 tid 1 2024-09-16T03:37:31.254 INFO:tasks.rados.rados.0.smithi157.stdout:700: writing smithi15736696-42 from 1563527 to 2146552 tid 2 2024-09-16T03:37:31.260 INFO:tasks.rados.rados.0.smithi157.stdout:700: writing smithi15736696-42 from 2810926 to 3210104 tid 3 2024-09-16T03:37:31.261 INFO:tasks.rados.rados.0.smithi157.stdout:701: write oid 13 current snap is 0 2024-09-16T03:37:31.261 INFO:tasks.rados.rados.0.smithi157.stdout:701: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-09-16T03:37:31.265 INFO:tasks.rados.rados.0.smithi157.stdout:701: writing smithi15736696-13 from 426911 to 967265 tid 1 2024-09-16T03:37:31.271 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:31.269+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:31.274 INFO:tasks.rados.rados.0.smithi157.stdout:701: writing smithi15736696-13 from 1622009 to 2348692 tid 2 2024-09-16T03:37:31.278 INFO:tasks.rados.rados.0.smithi157.stdout:701: writing smithi15736696-13 from 3121996 to 3226911 tid 3 2024-09-16T03:37:31.278 INFO:tasks.rados.rados.0.smithi157.stdout:702: write oid 18 current snap is 0 2024-09-16T03:37:31.278 INFO:tasks.rados.rados.0.smithi157.stdout:702: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-09-16T03:37:31.283 INFO:tasks.rados.rados.0.smithi157.stdout:702: writing smithi15736696-18 from 443718 to 1059321 tid 1 2024-09-16T03:37:31.288 INFO:tasks.rados.rados.0.smithi157.stdout:702: writing smithi15736696-18 from 1564138 to 2034479 tid 2 2024-09-16T03:37:31.295 INFO:tasks.rados.rados.0.smithi157.stdout:702: writing smithi15736696-18 from 2633066 to 3243718 tid 3 2024-09-16T03:37:31.295 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:31.295 INFO:tasks.rados.rados.0.smithi157.stdout:697: expect (ObjNum 65 snap 0 seq_num 65) 2024-09-16T03:37:31.295 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:31.318 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:37:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.318 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:996 4.8 4.be467128 (undecoded) ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:37:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:37:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.26951576100000002, 2024-09-16T03:37:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.26954203500000001, 2024-09-16T03:37:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.319 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "waiting for peered", 2024-09-16T03:37:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.320 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 996 2024-09-16T03:37:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:37:31.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:37:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047344+0000", 2024-09-16T03:37:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.846e-06 2024-09-16T03:37:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047351+0000", 2024-09-16T03:37:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.973e-06 2024-09-16T03:37:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047365+0000", 2024-09-16T03:37:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3556e-05 2024-09-16T03:37:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047382+0000", 2024-09-16T03:37:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6994e-05 2024-09-16T03:37:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047453+0000", 2024-09-16T03:37:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1458000000000006e-05 2024-09-16T03:37:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:31.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047465+0000", 2024-09-16T03:37:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1119e-05 2024-09-16T03:37:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1047 4.16 4.7dddcfb6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:31.328 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.264788+0000", 2024-09-16T03:37:31.329 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.052070772000000001, 2024-09-16T03:37:31.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.052134368, 2024-09-16T03:37:31.329 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.329 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.329 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:31.329 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.329 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.329 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.329 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1047 2024-09-16T03:37:31.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.329 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.264788+0000", 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.264788+0000", 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.264791+0000", 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0970000000000002e-06 2024-09-16T03:37:31.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.265404+0000", 2024-09-16T03:37:31.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061312599999999997 2024-09-16T03:37:31.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.265423+0000", 2024-09-16T03:37:31.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8688000000000001e-05 2024-09-16T03:37:31.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.265434+0000", 2024-09-16T03:37:31.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1899000000000001e-05 2024-09-16T03:37:31.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.332 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.332 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1048 4.16 4.7dddcfb6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:31.332 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.272300+0000", 2024-09-16T03:37:31.332 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.044558334999999998, 2024-09-16T03:37:31.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.044645511999999998, 2024-09-16T03:37:31.332 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.332 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.333 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "no flag points reached", 2024-09-16T03:37:31.333 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.333 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.333 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.333 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1048 2024-09-16T03:37:31.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.333 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.272300+0000", 2024-09-16T03:37:31.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.272300+0000", 2024-09-16T03:37:31.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.272302+0000", 2024-09-16T03:37:31.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8369999999999999e-06 2024-09-16T03:37:31.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.272946+0000", 2024-09-16T03:37:31.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064372300000000002 2024-09-16T03:37:31.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.272960+0000", 2024-09-16T03:37:31.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4341e-05 2024-09-16T03:37:31.335 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.335 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.335 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.335 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.335 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-16T03:37:31.336 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 3 2024-09-16T03:37:31.336 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:31.337 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:37:31.337 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:31.333+0000 7efef16d9640 -1 osd.13 50 handle_osd_map injecting map crc failure 2024-09-16T03:37:31.348 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1732055 2024-09-16T03:37:31.371 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:31.370+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:31.433 INFO:tasks.rados.rados.0.smithi157.stdout:696: finishing write tid 1 to smithi15736696-46 2024-09-16T03:37:31.433 INFO:tasks.rados.rados.0.smithi157.stdout:696: finishing write tid 2 to smithi15736696-46 2024-09-16T03:37:31.433 INFO:tasks.rados.rados.0.smithi157.stdout:697: done (15 left) 2024-09-16T03:37:31.433 INFO:tasks.rados.rados.0.smithi157.stdout:703: read oid 2 snap -1 2024-09-16T03:37:31.434 INFO:tasks.rados.rados.0.smithi157.stdout:703: expect (ObjNum 64 snap 0 seq_num 64) 2024-09-16T03:37:31.434 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:31.434 INFO:tasks.rados.rados.0.smithi157.stdout:696: finishing write tid 3 to smithi15736696-46 2024-09-16T03:37:31.434 INFO:tasks.rados.rados.0.smithi157.stdout:696: finishing write tid 4 to smithi15736696-46 2024-09-16T03:37:31.434 INFO:tasks.rados.rados.0.smithi157.stdout:696: oid 46 updating version 0 to 32 2024-09-16T03:37:31.434 INFO:tasks.rados.rados.0.smithi157.stdout:696: oid 46 updating version 32 to 34 2024-09-16T03:37:31.434 INFO:tasks.rados.rados.0.smithi157.stdout:696: oid 46 version 34 is already newer than 33 2024-09-16T03:37:31.434 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 34 (ObjNum 71 snap 0 seq_num 71) dirty exists 2024-09-16T03:37:31.435 INFO:tasks.rados.rados.0.smithi157.stdout:696: left oid 46 (ObjNum 71 snap 0 seq_num 71) 2024-09-16T03:37:31.435 INFO:tasks.rados.rados.0.smithi157.stdout:680: finishing set_chunk to oid 16 2024-09-16T03:37:31.435 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 16 set_chunk returned error code -95 offset: 82432 length: 2138112 2024-09-16T03:37:31.435 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:37:31.435 INFO:tasks.rados.rados.0.smithi157.stdout:691: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:37:31.439 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:31.439 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:31.439 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:37:31.439 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:37:31.439 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:31.448 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:37:31.449 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:31.492 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:31.490+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:31.498 INFO:tasks.rados.rados.0.smithi157.stdout:692: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:31.555 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:31.555 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:31.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:31.555 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:37:31.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.555 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:21 4.8 4:146f84f0:::smithi15736696-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:31.555 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.015211+0000", 2024-09-16T03:37:31.555 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.536811196999999, 2024-09-16T03:37:31.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.27460038399999998, 2024-09-16T03:37:31.555 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 21 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.015211+0000", 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.015211+0000", 2024-09-16T03:37:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.015214+0000", 2024-09-16T03:37:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3689999999999998e-06 2024-09-16T03:37:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.016145+0000", 2024-09-16T03:37:31.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00093080599999999995 2024-09-16T03:37:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.016162+0000", 2024-09-16T03:37:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6772000000000001e-05 2024-09-16T03:37:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.016171+0000", 2024-09-16T03:37:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4409999999999999e-06 2024-09-16T03:37:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.087198+0000", 2024-09-16T03:37:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.071026557000000004 2024-09-16T03:37:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.087274+0000", 2024-09-16T03:37:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6489000000000006e-05 2024-09-16T03:37:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:37:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.087361+0000", 2024-09-16T03:37:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6351e-05 2024-09-16T03:37:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.090242+0000", 2024-09-16T03:37:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00288134 2024-09-16T03:37:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.190484+0000", 2024-09-16T03:37:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100241434 2024-09-16T03:37:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289742+0000", 2024-09-16T03:37:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099258588999999994 2024-09-16T03:37:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289780+0000", 2024-09-16T03:37:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8297000000000003e-05 2024-09-16T03:37:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289811+0000", 2024-09-16T03:37:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0939000000000003e-05 2024-09-16T03:37:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.562 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:23 4.8 4:146f84f0:::smithi15736696-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.019609+0000", 2024-09-16T03:37:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.532413208000001, 2024-09-16T03:37:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.27054655500000002, 2024-09-16T03:37:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.563 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 23 2024-09-16T03:37:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.019609+0000", 2024-09-16T03:37:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.019609+0000", 2024-09-16T03:37:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.019609+0000", 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7100000000000002e-07 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.019615+0000", 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6729999999999997e-06 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.019619+0000", 2024-09-16T03:37:31.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.811e-06 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119718+0000", 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10009918 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119826+0000", 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010791800000000001 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119909+0000", 2024-09-16T03:37:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3033000000000003e-05 2024-09-16T03:37:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:37:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120012+0000", 2024-09-16T03:37:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000102639 2024-09-16T03:37:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120692+0000", 2024-09-16T03:37:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068009199999999998 2024-09-16T03:37:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.190794+0000", 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.070102629999999999 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.290091+0000", 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099296197000000003 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.290123+0000", 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.201e-05 2024-09-16T03:37:31.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.569 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.569 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.290156+0000", 2024-09-16T03:37:31.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2901000000000002e-05 2024-09-16T03:37:31.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.569 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.570 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.571 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:25 4.6 4:6ece7830:::smithi15736696-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:31.571 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:37:31.571 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.432193035000001, 2024-09-16T03:37:31.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40559452699999998, 2024-09-16T03:37:31.572 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.572 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.573 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.573 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.573 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.573 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.573 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 25 2024-09-16T03:37:31.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.573 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:37:31.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.574 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:37:31.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119830+0000", 2024-09-16T03:37:31.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9200000000000001e-07 2024-09-16T03:37:31.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120013+0000", 2024-09-16T03:37:31.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018367200000000001 2024-09-16T03:37:31.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120021+0000", 2024-09-16T03:37:31.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9179999999999994e-06 2024-09-16T03:37:31.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120027+0000", 2024-09-16T03:37:31.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6710000000000004e-06 2024-09-16T03:37:31.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220194+0000", 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10016689500000001 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220399+0000", 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020502899999999999 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220613+0000", 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021383900000000001 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224640+0000", 2024-09-16T03:37:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040268099999999996 2024-09-16T03:37:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423348+0000", 2024-09-16T03:37:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19870853399999999 2024-09-16T03:37:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525343+0000", 2024-09-16T03:37:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101995081 2024-09-16T03:37:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525381+0000", 2024-09-16T03:37:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7842000000000001e-05 2024-09-16T03:37:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525424+0000", 2024-09-16T03:37:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2644e-05 2024-09-16T03:37:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:26 4.6 4:6ece7830:::smithi15736696-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:37:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.431980807999999, 2024-09-16T03:37:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550111300000002, 2024-09-16T03:37:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 26 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120042+0000", 2024-09-16T03:37:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7999999999999995e-07 2024-09-16T03:37:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120210+0000", 2024-09-16T03:37:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016779499999999999 2024-09-16T03:37:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120219+0000", 2024-09-16T03:37:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6859999999999995e-06 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120223+0000", 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0330000000000002e-06 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221494+0000", 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101271812 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221572+0000", 2024-09-16T03:37:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7733999999999998e-05 2024-09-16T03:37:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221697+0000", 2024-09-16T03:37:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012474900000000001 2024-09-16T03:37:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224670+0000", 2024-09-16T03:37:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029729589999999998 2024-09-16T03:37:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423398+0000", 2024-09-16T03:37:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19872784900000001 2024-09-16T03:37:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525465+0000", 2024-09-16T03:37:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102067557 2024-09-16T03:37:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525494+0000", 2024-09-16T03:37:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9113999999999999e-05 2024-09-16T03:37:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525542+0000", 2024-09-16T03:37:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8145000000000001e-05 2024-09-16T03:37:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:27 4.6 4:6ece7830:::smithi15736696-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:37:31.586 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.431793223, 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40541507399999999, 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:37:31.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.588 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.588 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:37:31.588 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.588 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.588 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120230+0000", 2024-09-16T03:37:31.588 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5799999999999999e-07 2024-09-16T03:37:31.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.589 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.589 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120233+0000", 2024-09-16T03:37:31.589 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.055e-06 2024-09-16T03:37:31.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.589 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.589 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120237+0000", 2024-09-16T03:37:31.589 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7099999999999998e-06 2024-09-16T03:37:31.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.589 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.589 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120244+0000", 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4810000000000003e-06 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222660+0000", 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102416089 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222705+0000", 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5368999999999999e-05 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:31.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222782+0000", 2024-09-16T03:37:31.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6623000000000001e-05 2024-09-16T03:37:31.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224696+0000", 2024-09-16T03:37:31.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019143929999999999 2024-09-16T03:37:31.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423402+0000", 2024-09-16T03:37:31.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198705305 2024-09-16T03:37:31.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525582+0000", 2024-09-16T03:37:31.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102180178 2024-09-16T03:37:31.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525615+0000", 2024-09-16T03:37:31.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3303000000000003e-05 2024-09-16T03:37:31.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525644+0000", 2024-09-16T03:37:31.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8909999999999999e-05 2024-09-16T03:37:31.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.593 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.593 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:28 4.6 4:6ece7830:::smithi15736696-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:31.593 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:37:31.593 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.431772474999999, 2024-09-16T03:37:31.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550787799999999, 2024-09-16T03:37:31.593 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.593 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 28 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:37:31.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:37:31.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7999999999999996e-07 2024-09-16T03:37:31.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120252+0000", 2024-09-16T03:37:31.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.249e-06 2024-09-16T03:37:31.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120257+0000", 2024-09-16T03:37:31.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1969999999999998e-06 2024-09-16T03:37:31.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.596 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120261+0000", 2024-09-16T03:37:31.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1130000000000003e-06 2024-09-16T03:37:31.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.596 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222976+0000", 2024-09-16T03:37:31.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102715397 2024-09-16T03:37:31.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:31.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223049+0000", 2024-09-16T03:37:31.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2846e-05 2024-09-16T03:37:31.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525666+0000", 2024-09-16T03:37:31.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30261668800000002 2024-09-16T03:37:31.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525704+0000", 2024-09-16T03:37:31.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8025999999999998e-05 2024-09-16T03:37:31.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525758+0000", 2024-09-16T03:37:31.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3881999999999997e-05 2024-09-16T03:37:31.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.598 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:49 4.6 4:6df3bbbe:::smithi15736696-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:37:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.431757411, 2024-09-16T03:37:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40556222800000002, 2024-09-16T03:37:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49 2024-09-16T03:37:31.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:37:31.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8700000000000002e-07 2024-09-16T03:37:31.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120454+0000", 2024-09-16T03:37:31.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018909299999999999 2024-09-16T03:37:31.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120465+0000", 2024-09-16T03:37:31.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.083e-05 2024-09-16T03:37:31.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120468+0000", 2024-09-16T03:37:31.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3869999999999999e-06 2024-09-16T03:37:31.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223082+0000", 2024-09-16T03:37:31.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102613379 2024-09-16T03:37:31.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223189+0000", 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107728 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223368+0000", 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179011 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.226087+0000", 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027183390000000002 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423408+0000", 2024-09-16T03:37:31.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19732106499999999 2024-09-16T03:37:31.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.604 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525779+0000", 2024-09-16T03:37:31.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10237133 2024-09-16T03:37:31.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.604 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.604 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525805+0000", 2024-09-16T03:37:31.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5967000000000001e-05 2024-09-16T03:37:31.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525827+0000", 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1812000000000001e-05 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:50 4.6 4:6df3bbbe:::smithi15736696-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.431549102000002, 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50526462999999999, 2024-09-16T03:37:31.605 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.606 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.606 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.606 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.606 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.606 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.606 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 50 2024-09-16T03:37:31.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.606 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:37:31.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120474+0000", 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4600000000000005e-07 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120901+0000", 2024-09-16T03:37:31.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00042690000000000002 2024-09-16T03:37:31.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120907+0000", 2024-09-16T03:37:31.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7710000000000001e-06 2024-09-16T03:37:31.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120911+0000", 2024-09-16T03:37:31.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0500000000000002e-06 2024-09-16T03:37:31.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224717+0000", 2024-09-16T03:37:31.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103806175 2024-09-16T03:37:31.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224763+0000", 2024-09-16T03:37:31.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5629e-05 2024-09-16T03:37:31.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:31.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224838+0000", 2024-09-16T03:37:31.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4548999999999994e-05 2024-09-16T03:37:31.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423390+0000", 2024-09-16T03:37:31.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19855242400000001 2024-09-16T03:37:31.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.610 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.610 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423411+0000", 2024-09-16T03:37:31.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0888000000000001e-05 2024-09-16T03:37:31.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625680+0000", 2024-09-16T03:37:31.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20226923099999999 2024-09-16T03:37:31.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625709+0000", 2024-09-16T03:37:31.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9232000000000001e-05 2024-09-16T03:37:31.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625738+0000", 2024-09-16T03:37:31.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8235e-05 2024-09-16T03:37:31.613 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.613 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.613 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.614 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:51 4.6 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:31.614 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:37:31.614 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.431104520000002, 2024-09-16T03:37:31.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50490003699999997, 2024-09-16T03:37:31.614 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.614 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.614 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.614 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.614 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.614 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.614 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 51 2024-09-16T03:37:31.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:37:31.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8699999999999995e-07 2024-09-16T03:37:31.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120920+0000", 2024-09-16T03:37:31.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0820000000000001e-06 2024-09-16T03:37:31.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120924+0000", 2024-09-16T03:37:31.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:37:31.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120927+0000", 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5809999999999999e-06 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225783+0000", 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10485588699999999 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225832+0000", 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9750000000000003e-05 2024-09-16T03:37:31.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225910+0000", 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7084999999999998e-05 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423414+0000", 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19750438200000001 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423456+0000", 2024-09-16T03:37:31.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2386999999999999e-05 2024-09-16T03:37:31.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625766+0000", 2024-09-16T03:37:31.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20231011300000001 2024-09-16T03:37:31.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625790+0000", 2024-09-16T03:37:31.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3672999999999998e-05 2024-09-16T03:37:31.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625818+0000", 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7576000000000002e-05 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:52 4.6 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.431092324000002, 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50502012900000004, 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.620 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:37:31.621 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.621 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.621 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.621 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 52 2024-09-16T03:37:31.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.621 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:37:31.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:37:31.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:37:31.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:37:31.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120932+0000", 2024-09-16T03:37:31.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.181e-06 2024-09-16T03:37:31.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120935+0000", 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7690000000000001e-06 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120938+0000", 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6639999999999998e-06 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423188+0000", 2024-09-16T03:37:31.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30224989200000002 2024-09-16T03:37:31.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:31.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423318+0000", 2024-09-16T03:37:31.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013045999999999999 2024-09-16T03:37:31.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625844+0000", 2024-09-16T03:37:31.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20252547600000001 2024-09-16T03:37:31.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625895+0000", 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1242000000000002e-05 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625950+0000", 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5164999999999998e-05 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:77 4.6 4:64ce87bc:::smithi15736696-20:head [write 719333~462437,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:31.625 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.352046+0000", 2024-09-16T03:37:31.626 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.199976250999999, 2024-09-16T03:37:31.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.27449950299999998, 2024-09-16T03:37:31.626 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.626 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.626 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.626 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.626 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.626 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.626 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 77 2024-09-16T03:37:31.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.626 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.352046+0000", 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.352046+0000", 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.352056+0000", 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.022e-05 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.352437+0000", 2024-09-16T03:37:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038118000000000001 2024-09-16T03:37:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.352448+0000", 2024-09-16T03:37:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0596e-05 2024-09-16T03:37:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.352453+0000", 2024-09-16T03:37:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.558e-06 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.523622+0000", 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.17116875600000001 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.523718+0000", 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5418e-05 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.523813+0000", 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5080000000000004e-05 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525553+0000", 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001740674 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.626451+0000", 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100897971 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.626480+0000", 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8192999999999999e-05 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.626512+0000", 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2694000000000002e-05 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.626545+0000", 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3163000000000001e-05 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:279 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.355140537, 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50446570800000001, 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 279 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196883+0000", 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.48e-06 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196889+0000", 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6239999999999999e-06 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196902+0000", 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2804e-05 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196909+0000", 2024-09-16T03:37:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9379999999999992e-06 2024-09-16T03:37:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196963+0000", 2024-09-16T03:37:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3437999999999999e-05 2024-09-16T03:37:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op must wait for map", 2024-09-16T03:37:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196969+0000", 2024-09-16T03:37:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1709999999999999e-06 2024-09-16T03:37:31.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.636 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.636 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302538+0000", 2024-09-16T03:37:31.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10556889899999999 2024-09-16T03:37:31.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.636 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.636 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302659+0000", 2024-09-16T03:37:31.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121441 2024-09-16T03:37:31.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.636 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500040+0000", 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19738082900000001 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500589+0000", 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00054827000000000001 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701118+0000", 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20052917100000001 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701205+0000", 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7120999999999995e-05 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701289+0000", 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4457999999999997e-05 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701347+0000", 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8063999999999998e-05 2024-09-16T03:37:31.638 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:297 4.6 4:6ef3b53d:::smithi15736696-48:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.107720+0000", 2024-09-16T03:37:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.444301944999999, 2024-09-16T03:37:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30379255799999999, 2024-09-16T03:37:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.639 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 297 2024-09-16T03:37:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.107720+0000", 2024-09-16T03:37:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.107720+0000", 2024-09-16T03:37:31.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.107723+0000", 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.463e-06 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.107730+0000", 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1760000000000004e-06 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.641 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.107742+0000", 2024-09-16T03:37:31.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1651000000000001e-05 2024-09-16T03:37:31.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.107749+0000", 2024-09-16T03:37:31.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8699999999999992e-06 2024-09-16T03:37:31.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.107832+0000", 2024-09-16T03:37:31.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2434000000000004e-05 2024-09-16T03:37:31.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.107906+0000", 2024-09-16T03:37:31.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4183000000000004e-05 2024-09-16T03:37:31.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:31.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.309933+0000", 2024-09-16T03:37:31.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20202667399999999 2024-09-16T03:37:31.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.310431+0000", 2024-09-16T03:37:31.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00049883299999999996 2024-09-16T03:37:31.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.311051+0000", 2024-09-16T03:37:31.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061982200000000002 2024-09-16T03:37:31.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.411442+0000", 2024-09-16T03:37:31.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100390347 2024-09-16T03:37:31.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.411480+0000", 2024-09-16T03:37:31.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8084000000000003e-05 2024-09-16T03:37:31.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.411513+0000", 2024-09-16T03:37:31.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3021000000000001e-05 2024-09-16T03:37:31.645 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.645 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.645 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.645 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:306 4.6 4:6ece7830:::smithi15736696-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:31.646 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:37:31.646 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.039138222, 2024-09-16T03:37:31.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40927214699999998, 2024-09-16T03:37:31.646 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.646 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.646 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.646 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.646 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.646 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.646 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 306 2024-09-16T03:37:31.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.646 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512885+0000", 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.277e-06 2024-09-16T03:37:31.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512890+0000", 2024-09-16T03:37:31.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6689999999999997e-06 2024-09-16T03:37:31.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512900+0000", 2024-09-16T03:37:31.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0393e-05 2024-09-16T03:37:31.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512906+0000", 2024-09-16T03:37:31.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4569999999999996e-06 2024-09-16T03:37:31.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512951+0000", 2024-09-16T03:37:31.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4891000000000001e-05 2024-09-16T03:37:31.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.649 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.649 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.513000+0000", 2024-09-16T03:37:31.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9187999999999997e-05 2024-09-16T03:37:31.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.720758+0000", 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20775775699999999 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.721960+0000", 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001202107 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922010+0000", 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20004993900000001 2024-09-16T03:37:31.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922055+0000", 2024-09-16T03:37:31.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4981000000000002e-05 2024-09-16T03:37:31.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922103+0000", 2024-09-16T03:37:31.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8097999999999999e-05 2024-09-16T03:37:31.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922156+0000", 2024-09-16T03:37:31.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3390000000000002e-05 2024-09-16T03:37:31.652 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.652 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.652 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.652 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:419 4.6 4:64ce87bc:::smithi15736696-20:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:31.652 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:37:31.652 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.638338608, 2024-09-16T03:37:31.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40437787200000003, 2024-09-16T03:37:31.652 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.652 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 419 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.653 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:37:31.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913687+0000", 2024-09-16T03:37:31.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3299999999999999e-06 2024-09-16T03:37:31.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913700+0000", 2024-09-16T03:37:31.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2830000000000001e-05 2024-09-16T03:37:31.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913733+0000", 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2815999999999999e-05 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913750+0000", 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7073000000000001e-05 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913920+0000", 2024-09-16T03:37:31.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000170015 2024-09-16T03:37:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913981+0000", 2024-09-16T03:37:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1631999999999995e-05 2024-09-16T03:37:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216580+0000", 2024-09-16T03:37:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30259888200000001 2024-09-16T03:37:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217205+0000", 2024-09-16T03:37:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062461099999999996 2024-09-16T03:37:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317915+0000", 2024-09-16T03:37:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10071053000000001 2024-09-16T03:37:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317996+0000", 2024-09-16T03:37:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0440999999999998e-05 2024-09-16T03:37:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318032+0000", 2024-09-16T03:37:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5944e-05 2024-09-16T03:37:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318061+0000", 2024-09-16T03:37:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9768e-05 2024-09-16T03:37:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:479 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:37:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.8304824619999991, 2024-09-16T03:37:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.504011706, 2024-09-16T03:37:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 479 2024-09-16T03:37:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:37:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:37:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721542+0000", 2024-09-16T03:37:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3199999999999998e-06 2024-09-16T03:37:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721553+0000", 2024-09-16T03:37:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1413000000000001e-05 2024-09-16T03:37:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721570+0000", 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.685e-05 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721596+0000", 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5647999999999999e-05 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.921950+0000", 2024-09-16T03:37:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20035402899999999 2024-09-16T03:37:31.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.922009+0000", 2024-09-16T03:37:31.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9046999999999999e-05 2024-09-16T03:37:31.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:37:31.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.023850+0000", 2024-09-16T03:37:31.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10184130199999999 2024-09-16T03:37:31.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.024503+0000", 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065257399999999995 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.125105+0000", 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100602046 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225461+0000", 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100355835 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225512+0000", 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0862999999999997e-05 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225551+0000", 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9779e-05 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.665 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:816 4.6 4:6df3bbbe:::smithi15736696-13:head [write 490771~686315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:31.666 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.028411+0000", 2024-09-16T03:37:31.666 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.5236112989999997, 2024-09-16T03:37:31.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30955166899999997, 2024-09-16T03:37:31.666 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.666 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.666 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.666 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.666 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.666 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.666 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 816 2024-09-16T03:37:31.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.666 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.028411+0000", 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.028411+0000", 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.028414+0000", 2024-09-16T03:37:31.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.869e-06 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.029221+0000", 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00080734600000000002 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.029239+0000", 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8097999999999999e-05 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.029247+0000", 2024-09-16T03:37:31.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1759999999999993e-06 2024-09-16T03:37:31.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034716+0000", 2024-09-16T03:37:31.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0054690939999999999 2024-09-16T03:37:31.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034803+0000", 2024-09-16T03:37:31.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7033999999999994e-05 2024-09-16T03:37:31.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:31.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034925+0000", 2024-09-16T03:37:31.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121885 2024-09-16T03:37:31.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.129550+0000", 2024-09-16T03:37:31.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.094624616999999994 2024-09-16T03:37:31.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.137186+0000", 2024-09-16T03:37:31.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0076361989999999998 2024-09-16T03:37:31.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.337693+0000", 2024-09-16T03:37:31.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200507138 2024-09-16T03:37:31.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.337902+0000", 2024-09-16T03:37:31.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000208641 2024-09-16T03:37:31.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.337963+0000", 2024-09-16T03:37:31.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0572000000000002e-05 2024-09-16T03:37:31.672 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.672 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.672 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.672 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:817 4.6 4:6df3bbbe:::smithi15736696-13:head [write 1885075~406346] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:31.672 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.034244+0000", 2024-09-16T03:37:31.672 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.5177781269999997, 2024-09-16T03:37:31.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30387697699999999, 2024-09-16T03:37:31.672 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.673 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.673 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.673 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.673 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.673 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.673 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 817 2024-09-16T03:37:31.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.673 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034244+0000", 2024-09-16T03:37:31.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034244+0000", 2024-09-16T03:37:31.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034245+0000", 2024-09-16T03:37:31.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0210000000000001e-06 2024-09-16T03:37:31.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034619+0000", 2024-09-16T03:37:31.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00037414799999999998 2024-09-16T03:37:31.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034629+0000", 2024-09-16T03:37:31.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3649999999999993e-06 2024-09-16T03:37:31.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034633+0000", 2024-09-16T03:37:31.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5929999999999999e-06 2024-09-16T03:37:31.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.036532+0000", 2024-09-16T03:37:31.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0018992239999999999 2024-09-16T03:37:31.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.036606+0000", 2024-09-16T03:37:31.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3354000000000003e-05 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.036756+0000", 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000150106 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.129553+0000", 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.092797176999999995 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.237360+0000", 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.107807153 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.338003+0000", 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100643013 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.338089+0000", 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5673000000000005e-05 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.338121+0000", 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2150000000000002e-05 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:818 4.6 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 2890771] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.034645+0000", 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.5173776270000001, 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30358909099999998, 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.679 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 818 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034645+0000", 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034645+0000", 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034645+0000", 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0699999999999997e-07 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034648+0000", 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6529999999999998e-06 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034669+0000", 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1029000000000001e-05 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034672+0000", 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3239999999999999e-06 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.037667+0000", 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029948319999999998 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.037728+0000", 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0693999999999999e-05 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.037822+0000", 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4420999999999996e-05 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:31.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.129555+0000", 2024-09-16T03:37:31.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.091733281999999999 2024-09-16T03:37:31.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.237450+0000", 2024-09-16T03:37:31.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.107894462 2024-09-16T03:37:31.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:31.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.338159+0000", 2024-09-16T03:37:31.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100709698 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.338194+0000", 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4789999999999997e-05 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.338234+0000", 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9399e-05 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:819 4.6 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.034676+0000", 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.5173464890000004, 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.303703785, 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 819 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.686 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034676+0000", 2024-09-16T03:37:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034676+0000", 2024-09-16T03:37:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:31.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034676+0000", 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.84e-07 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034678+0000", 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5320000000000001e-06 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034683+0000", 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9910000000000002e-06 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.034687+0000", 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1520000000000001e-06 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.129473+0000", 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.094786578999999996 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.129540+0000", 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6882000000000001e-05 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.338267+0000", 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20872724300000001 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.338316+0000", 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8266000000000002e-05 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.338379+0000", 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3856000000000007e-05 2024-09-16T03:37:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.691 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.691 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.691 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:31.691 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:31.691 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:31.692 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1584095 2024-09-16T03:37:31.692 INFO:tasks.rados.rados.0.smithi157.stdout:686: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:37:31.693 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:31.693 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1254674 2024-09-16T03:37:31.693 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:37:31.694 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:31.690+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:31.694 INFO:tasks.rados.rados.0.smithi157.stdout:687: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-16T03:37:31.695 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:31.720 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:31.752 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1188463 2024-09-16T03:37:31.764 INFO:tasks.rados.rados.0.smithi157.stdout:693: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-16T03:37:31.793 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:31.791+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:31.839 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:31.839 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:31.840 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:31.840 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:31.849 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:37:31.853 INFO:tasks.rados.rados.0.smithi157.stdout:695: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-16T03:37:31.884 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 617807 2024-09-16T03:37:31.894 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:31.892+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:31.932 INFO:tasks.rados.rados.0.smithi157.stdout:703: expect (ObjNum 64 snap 0 seq_num 64) 2024-09-16T03:37:31.932 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:31.993 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:31.993 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:31.993 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:37:31.993 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:37:31.993 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:31.995 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:31.994+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:32.002 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:37:32.042 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2622454 2024-09-16T03:37:32.051 INFO:tasks.rados.rados.0.smithi157.stdout:698: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T03:37:32.061 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1962164 2024-09-16T03:37:32.062 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 236668 2024-09-16T03:37:32.096 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:32.094+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:32.099 INFO:tasks.rados.rados.0.smithi157.stdout:700: finishing write tid 1 to smithi15736696-42 2024-09-16T03:37:32.099 INFO:tasks.rados.rados.0.smithi157.stdout:700: finishing write tid 2 to smithi15736696-42 2024-09-16T03:37:32.099 INFO:tasks.rados.rados.0.smithi157.stdout:700: finishing write tid 3 to smithi15736696-42 2024-09-16T03:37:32.099 INFO:tasks.rados.rados.0.smithi157.stdout:684: got (0) Success 2024-09-16T03:37:32.100 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 0 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T03:37:32.100 INFO:tasks.rados.rados.0.smithi157.stdout:680: done (15 left) 2024-09-16T03:37:32.100 INFO:tasks.rados.rados.0.smithi157.stdout:684: done (14 left) 2024-09-16T03:37:32.100 INFO:tasks.rados.rados.0.smithi157.stdout:686: done (13 left) 2024-09-16T03:37:32.100 INFO:tasks.rados.rados.0.smithi157.stdout:687: done (12 left) 2024-09-16T03:37:32.100 INFO:tasks.rados.rados.0.smithi157.stdout:691: done (11 left) 2024-09-16T03:37:32.100 INFO:tasks.rados.rados.0.smithi157.stdout:692: done (10 left) 2024-09-16T03:37:32.100 INFO:tasks.rados.rados.0.smithi157.stdout:693: done (9 left) 2024-09-16T03:37:32.100 INFO:tasks.rados.rados.0.smithi157.stdout:695: done (8 left) 2024-09-16T03:37:32.100 INFO:tasks.rados.rados.0.smithi157.stdout:696: done (7 left) 2024-09-16T03:37:32.100 INFO:tasks.rados.rados.0.smithi157.stdout:698: done (6 left) 2024-09-16T03:37:32.100 INFO:tasks.rados.rados.0.smithi157.stdout:703: done (5 left) 2024-09-16T03:37:32.100 INFO:tasks.rados.rados.0.smithi157.stdout:704: read oid 31 snap -1 2024-09-16T03:37:32.101 INFO:tasks.rados.rados.0.smithi157.stdout:704: expect deleted 2024-09-16T03:37:32.101 INFO:tasks.rados.rados.0.smithi157.stdout:705: read oid 27 snap -1 2024-09-16T03:37:32.101 INFO:tasks.rados.rados.0.smithi157.stdout:705: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:32.101 INFO:tasks.rados.rados.0.smithi157.stdout:706: write (excl) oid 29 current snap is 0 2024-09-16T03:37:32.101 INFO:tasks.rados.rados.0.smithi157.stdout:699: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T03:37:32.148 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:37:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:3 3.0 3.f934d18 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:37:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.745529848, 2024-09-16T03:37:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78381636700000001, 2024-09-16T03:37:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.148 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:37:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:37:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399429+0000", 2024-09-16T03:37:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:37:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399434+0000", 2024-09-16T03:37:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1050000000000003e-06 2024-09-16T03:37:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399442+0000", 2024-09-16T03:37:32.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0160000000000008e-06 2024-09-16T03:37:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399448+0000", 2024-09-16T03:37:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.879e-06 2024-09-16T03:37:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399489+0000", 2024-09-16T03:37:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0815999999999997e-05 2024-09-16T03:37:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399495+0000", 2024-09-16T03:37:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3439999999999997e-06 2024-09-16T03:37:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183226+0000", 2024-09-16T03:37:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78373077400000002 2024-09-16T03:37:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183244+0000", 2024-09-16T03:37:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8128999999999999e-05 2024-09-16T03:37:32.152 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:5 3.0 3.aefbbec0 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:37:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.645762386000001, 2024-09-16T03:37:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68447923499999996, 2024-09-16T03:37:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:32.153 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499196+0000", 2024-09-16T03:37:32.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3480000000000001e-06 2024-09-16T03:37:32.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499202+0000", 2024-09-16T03:37:32.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4219999999999999e-06 2024-09-16T03:37:32.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499212+0000", 2024-09-16T03:37:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:37:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499218+0000", 2024-09-16T03:37:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4920000000000001e-06 2024-09-16T03:37:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499283+0000", 2024-09-16T03:37:32.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4906000000000005e-05 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499293+0000", 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0513000000000001e-05 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183665+0000", 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68437196600000005 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183674+0000", 2024-09-16T03:37:32.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9700000000000005e-06 2024-09-16T03:37:32.158 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.158 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.158 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.158 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:6 3.0 3.3bcda120 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:32.158 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:37:32.158 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.645701776999999, 2024-09-16T03:37:32.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78474037699999999, 2024-09-16T03:37:32.158 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.158 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:37:32.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0500000000000002e-07 2024-09-16T03:37:32.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499260+0000", 2024-09-16T03:37:32.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6669999999999998e-06 2024-09-16T03:37:32.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499266+0000", 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499274+0000", 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2250000000000003e-06 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499316+0000", 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2670000000000003e-05 2024-09-16T03:37:32.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499323+0000", 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.37e-06 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283987+0000", 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78466433700000005 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283996+0000", 2024-09-16T03:37:32.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8650000000000007e-06 2024-09-16T03:37:32.163 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.163 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.163 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.163 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:15 3.0 3.7ec93288 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:32.163 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:37:32.163 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.536509766999998, 2024-09-16T03:37:32.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57518172599999995, 2024-09-16T03:37:32.163 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.163 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.164 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:32.164 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:37:32.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:37:32.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608449+0000", 2024-09-16T03:37:32.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.184e-06 2024-09-16T03:37:32.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608459+0000", 2024-09-16T03:37:32.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0553999999999999e-05 2024-09-16T03:37:32.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608470+0000", 2024-09-16T03:37:32.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0524e-05 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608493+0000", 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2980999999999999e-05 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608537+0000", 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3850000000000002e-05 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:32.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608542+0000", 2024-09-16T03:37:32.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:37:32.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183623+0000", 2024-09-16T03:37:32.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.575080649 2024-09-16T03:37:32.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183629+0000", 2024-09-16T03:37:32.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6950000000000003e-06 2024-09-16T03:37:32.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.167 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.168 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.168 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:11 3.0 3.77005160 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:32.168 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:37:32.168 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.443009456999999, 2024-09-16T03:37:32.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.48153733399999998, 2024-09-16T03:37:32.168 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.169 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.169 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:32.169 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:37:32.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:37:32.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701949+0000", 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.314e-06 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701954+0000", 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701964+0000", 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9550000000000004e-06 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701986+0000", 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1931999999999999e-05 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802129+0000", 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100142986 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:32.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802135+0000", 2024-09-16T03:37:32.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6350000000000003e-06 2024-09-16T03:37:32.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.172 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183475+0000", 2024-09-16T03:37:32.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38133958099999998 2024-09-16T03:37:32.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.172 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183485+0000", 2024-09-16T03:37:32.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0642000000000001e-05 2024-09-16T03:37:32.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:6 3.0 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.162105543999999, 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60469085300000003, 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:37:32.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:37:32.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982853+0000", 2024-09-16T03:37:32.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.393e-06 2024-09-16T03:37:32.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982858+0000", 2024-09-16T03:37:32.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.062e-06 2024-09-16T03:37:32.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982867+0000", 2024-09-16T03:37:32.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:37:32.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982879+0000", 2024-09-16T03:37:32.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2241e-05 2024-09-16T03:37:32.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982996+0000", 2024-09-16T03:37:32.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116996 2024-09-16T03:37:32.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:32.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983007+0000", 2024-09-16T03:37:32.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0694e-05 2024-09-16T03:37:32.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284499+0000", 2024-09-16T03:37:32.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30149171400000002 2024-09-16T03:37:32.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284690+0000", 2024-09-16T03:37:32.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000191032 2024-09-16T03:37:32.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:32.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284908+0000", 2024-09-16T03:37:32.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021827399999999999 2024-09-16T03:37:32.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487092+0000", 2024-09-16T03:37:32.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.202184156 2024-09-16T03:37:32.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:32.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487110+0000", 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7856999999999999e-05 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587335+0000", 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100224802 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587410+0000", 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5343e-05 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587543+0000", 2024-09-16T03:37:32.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000132492 2024-09-16T03:37:32.181 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.181 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.181 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.181 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:2 3.0 3:16550cfd:::smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:32.181 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:37:32.181 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.162033352, 2024-09-16T03:37:32.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40289696000000003, 2024-09-16T03:37:32.181 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.181 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.182 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:32.182 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:37:32.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:37:32.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982925+0000", 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982928+0000", 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4130000000000002e-06 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982936+0000", 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.875e-06 2024-09-16T03:37:32.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982942+0000", 2024-09-16T03:37:32.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:37:32.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983093+0000", 2024-09-16T03:37:32.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015144400000000001 2024-09-16T03:37:32.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:32.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983098+0000", 2024-09-16T03:37:32.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4909999999999999e-06 2024-09-16T03:37:32.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385695+0000", 2024-09-16T03:37:32.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402597539 2024-09-16T03:37:32.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385821+0000", 2024-09-16T03:37:32.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125418 2024-09-16T03:37:32.185 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.185 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:9 3.0 3:2c9b9e7c:::smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.162017163, 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60615655800000001, 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982941+0000", 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0860000000000001e-06 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982946+0000", 2024-09-16T03:37:32.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:37:32.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982954+0000", 2024-09-16T03:37:32.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8520000000000004e-06 2024-09-16T03:37:32.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982958+0000", 2024-09-16T03:37:32.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4870000000000004e-06 2024-09-16T03:37:32.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983116+0000", 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015774699999999999 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983120+0000", 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486023+0000", 2024-09-16T03:37:32.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50290281400000003 2024-09-16T03:37:32.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486423+0000", 2024-09-16T03:37:32.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040079299999999998 2024-09-16T03:37:32.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486707+0000", 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028336600000000001 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588861+0000", 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102154756 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588952+0000", 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0630000000000005e-05 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588984+0000", 2024-09-16T03:37:32.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2116999999999998e-05 2024-09-16T03:37:32.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589022+0000", 2024-09-16T03:37:32.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7910000000000001e-05 2024-09-16T03:37:32.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589097+0000", 2024-09-16T03:37:32.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4536999999999995e-05 2024-09-16T03:37:32.193 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.193 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.193 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.193 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:12 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:32.193 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:37:32.193 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.161941815, 2024-09-16T03:37:32.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60616102900000002, 2024-09-16T03:37:32.193 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.193 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.193 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983017+0000", 2024-09-16T03:37:32.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.483e-06 2024-09-16T03:37:32.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983022+0000", 2024-09-16T03:37:32.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:37:32.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983032+0000", 2024-09-16T03:37:32.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0036e-05 2024-09-16T03:37:32.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983038+0000", 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6800000000000004e-06 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983155+0000", 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116401 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983163+0000", 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6219999999999998e-06 2024-09-16T03:37:32.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589120+0000", 2024-09-16T03:37:32.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60595653199999999 2024-09-16T03:37:32.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589176+0000", 2024-09-16T03:37:32.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6542000000000002e-05 2024-09-16T03:37:32.197 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.197 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.197 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.198 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:5 3.0 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:32.198 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:37:32.198 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 11.230752885999999, 2024-09-16T03:37:32.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30268081699999999, 2024-09-16T03:37:32.198 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.198 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.198 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.198 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:37:32.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:37:32.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914207+0000", 2024-09-16T03:37:32.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1500000000000002e-06 2024-09-16T03:37:32.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914215+0000", 2024-09-16T03:37:32.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7530000000000007e-06 2024-09-16T03:37:32.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914232+0000", 2024-09-16T03:37:32.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.628e-05 2024-09-16T03:37:32.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914242+0000", 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.058e-05 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914305+0000", 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3294999999999996e-05 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914377+0000", 2024-09-16T03:37:32.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1698000000000004e-05 2024-09-16T03:37:32.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:32.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914517+0000", 2024-09-16T03:37:32.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000139443 2024-09-16T03:37:32.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.015450+0000", 2024-09-16T03:37:32.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100933668 2024-09-16T03:37:32.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:32.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.015480+0000", 2024-09-16T03:37:32.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9467000000000001e-05 2024-09-16T03:37:32.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216766+0000", 2024-09-16T03:37:32.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20128665300000001 2024-09-16T03:37:32.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216831+0000", 2024-09-16T03:37:32.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4220999999999994e-05 2024-09-16T03:37:32.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216885+0000", 2024-09-16T03:37:32.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4608999999999998e-05 2024-09-16T03:37:32.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.204 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.204 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:24 3.0 3:2dd0f25f:::smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:32.204 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:37:32.204 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 11.13041849, 2024-09-16T03:37:32.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30270049300000001, 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014541+0000", 2024-09-16T03:37:32.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2479999999999999e-06 2024-09-16T03:37:32.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014550+0000", 2024-09-16T03:37:32.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.115e-06 2024-09-16T03:37:32.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014567+0000", 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7292999999999999e-05 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014577+0000", 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8020000000000003e-06 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115566+0000", 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10098905900000001 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115702+0000", 2024-09-16T03:37:32.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013602500000000001 2024-09-16T03:37:32.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:32.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115893+0000", 2024-09-16T03:37:32.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019095700000000001 2024-09-16T03:37:32.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216936+0000", 2024-09-16T03:37:32.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101042992 2024-09-16T03:37:32.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317125+0000", 2024-09-16T03:37:32.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100188757 2024-09-16T03:37:32.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:32.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317155+0000", 2024-09-16T03:37:32.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0009999999999999e-05 2024-09-16T03:37:32.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317210+0000", 2024-09-16T03:37:32.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4589999999999997e-05 2024-09-16T03:37:32.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317239+0000", 2024-09-16T03:37:32.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9645000000000002e-05 2024-09-16T03:37:32.210 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.210 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.210 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.210 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:30 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:32.210 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:37:32.211 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.928623309000001, 2024-09-16T03:37:32.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40312788999999999, 2024-09-16T03:37:32.211 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.211 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.211 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.211 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:37:32.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:37:32.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216338+0000", 2024-09-16T03:37:32.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0799999999999999e-06 2024-09-16T03:37:32.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216354+0000", 2024-09-16T03:37:32.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6439e-05 2024-09-16T03:37:32.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216392+0000", 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7162000000000001e-05 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216444+0000", 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2082e-05 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417447+0000", 2024-09-16T03:37:32.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20100358400000001 2024-09-16T03:37:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417583+0000", 2024-09-16T03:37:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135537 2024-09-16T03:37:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417777+0000", 2024-09-16T03:37:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019431200000000001 2024-09-16T03:37:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:32.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418479+0000", 2024-09-16T03:37:32.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00070148499999999996 2024-09-16T03:37:32.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418755+0000", 2024-09-16T03:37:32.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027627100000000003 2024-09-16T03:37:32.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619285+0000", 2024-09-16T03:37:32.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200530138 2024-09-16T03:37:32.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619370+0000", 2024-09-16T03:37:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4567999999999994e-05 2024-09-16T03:37:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619462+0000", 2024-09-16T03:37:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2231999999999994e-05 2024-09-16T03:37:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:16 3.0 3:037ddf75:::smithi15736696-34 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.935786+0000", 2024-09-16T03:37:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.2091712750000001, 2024-09-16T03:37:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20566089100000001, 2024-09-16T03:37:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935786+0000", 2024-09-16T03:37:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935786+0000", 2024-09-16T03:37:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935789+0000", 2024-09-16T03:37:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3670000000000001e-06 2024-09-16T03:37:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935802+0000", 2024-09-16T03:37:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3079e-05 2024-09-16T03:37:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935832+0000", 2024-09-16T03:37:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0006e-05 2024-09-16T03:37:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935848+0000", 2024-09-16T03:37:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5812999999999999e-05 2024-09-16T03:37:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.938925+0000", 2024-09-16T03:37:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030769840000000001 2024-09-16T03:37:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.939017+0000", 2024-09-16T03:37:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1666e-05 2024-09-16T03:37:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.939223+0000", 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020624200000000001 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.040123+0000", 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10089940999999999 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140975+0000", 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100852699 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.223 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.141371+0000", 2024-09-16T03:37:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039579799999999999 2024-09-16T03:37:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.141411+0000", 2024-09-16T03:37:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0170999999999999e-05 2024-09-16T03:37:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.141447+0000", 2024-09-16T03:37:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5655999999999999e-05 2024-09-16T03:37:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:17 3.0 3:037ddf75:::smithi15736696-34 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.935889+0000", 2024-09-16T03:37:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.2090681070000002, 2024-09-16T03:37:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20623545400000001, 2024-09-16T03:37:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935889+0000", 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935889+0000", 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935891+0000", 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.39e-06 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935897+0000", 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3029999999999996e-06 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935908+0000", 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0756e-05 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935916+0000", 2024-09-16T03:37:32.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6629999999999999e-06 2024-09-16T03:37:32.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.039673+0000", 2024-09-16T03:37:32.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103757029 2024-09-16T03:37:32.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.039746+0000", 2024-09-16T03:37:32.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2972999999999994e-05 2024-09-16T03:37:32.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:32.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.039869+0000", 2024-09-16T03:37:32.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012310700000000001 2024-09-16T03:37:32.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:32.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140934+0000", 2024-09-16T03:37:32.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101064666 2024-09-16T03:37:32.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.141007+0000", 2024-09-16T03:37:32.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3118e-05 2024-09-16T03:37:32.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.142058+0000", 2024-09-16T03:37:32.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010510459999999999 2024-09-16T03:37:32.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.142095+0000", 2024-09-16T03:37:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6539999999999999e-05 2024-09-16T03:37:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.142125+0000", 2024-09-16T03:37:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9862999999999999e-05 2024-09-16T03:37:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:18 3.0 3:037ddf75:::smithi15736696-34 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:37:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.2090290379999997, 2024-09-16T03:37:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30645336299999998, 2024-09-16T03:37:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:37:32.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:37:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935929+0000", 2024-09-16T03:37:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4600000000000003e-07 2024-09-16T03:37:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935936+0000", 2024-09-16T03:37:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3199999999999996e-06 2024-09-16T03:37:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935943+0000", 2024-09-16T03:37:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8900000000000007e-06 2024-09-16T03:37:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935954+0000", 2024-09-16T03:37:32.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0246e-05 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140278+0000", 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20432396999999999 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140342+0000", 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4442000000000004e-05 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:32.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140468+0000", 2024-09-16T03:37:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125718 2024-09-16T03:37:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.141460+0000", 2024-09-16T03:37:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00099208400000000006 2024-09-16T03:37:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242240+0000", 2024-09-16T03:37:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100779733 2024-09-16T03:37:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242316+0000", 2024-09-16T03:37:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6038999999999995e-05 2024-09-16T03:37:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242351+0000", 2024-09-16T03:37:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4928999999999997e-05 2024-09-16T03:37:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242382+0000", 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1046000000000001e-05 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:28 3.0 3:26d46705:::076d7e878f580d0ea556d11054cb0027df80eadb9845d4d121514c92db92c9b8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.473539+0000", 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.67141827200000004, 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.207364408, 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.238 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473539+0000", 2024-09-16T03:37:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473539+0000", 2024-09-16T03:37:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473542+0000", 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9210000000000001e-06 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473828+0000", 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000286085 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473847+0000", 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.946e-05 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473858+0000", 2024-09-16T03:37:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0285999999999999e-05 2024-09-16T03:37:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474128+0000", 2024-09-16T03:37:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000269886 2024-09-16T03:37:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474242+0000", 2024-09-16T03:37:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000114714 2024-09-16T03:37:32.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474404+0000", 2024-09-16T03:37:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000161595 2024-09-16T03:37:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.480364+0000", 2024-09-16T03:37:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0059603629999999998 2024-09-16T03:37:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680752+0000", 2024-09-16T03:37:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20038762800000001 2024-09-16T03:37:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680811+0000", 2024-09-16T03:37:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9502000000000001e-05 2024-09-16T03:37:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680850+0000", 2024-09-16T03:37:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8717000000000002e-05 2024-09-16T03:37:32.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680903+0000", 2024-09-16T03:37:32.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3251000000000002e-05 2024-09-16T03:37:32.244 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.244 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.244 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.244 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:29 3.0 3:363c19ae:::d7c46b815608541b41d2b012df2d45d94bbc9173d36826cf8a21a48f304023a1:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:32.245 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.473885+0000", 2024-09-16T03:37:32.245 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.67107225199999998, 2024-09-16T03:37:32.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20712613699999999, 2024-09-16T03:37:32.245 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.245 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.245 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.245 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473885+0000", 2024-09-16T03:37:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473885+0000", 2024-09-16T03:37:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473886+0000", 2024-09-16T03:37:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.64e-07 2024-09-16T03:37:32.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474063+0000", 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000177441 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474074+0000", 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.059e-05 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474082+0000", 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0909999999999993e-06 2024-09-16T03:37:32.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.475039+0000", 2024-09-16T03:37:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000956886 2024-09-16T03:37:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.475123+0000", 2024-09-16T03:37:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4462000000000005e-05 2024-09-16T03:37:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.475298+0000", 2024-09-16T03:37:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000174742 2024-09-16T03:37:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.580527+0000", 2024-09-16T03:37:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.105228435 2024-09-16T03:37:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680758+0000", 2024-09-16T03:37:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100231606 2024-09-16T03:37:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680953+0000", 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019436 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680986+0000", 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3760999999999997e-05 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681011+0000", 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4799000000000001e-05 2024-09-16T03:37:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.251 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.251 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.251 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:30 3.0 3:1e508a18:::6088440c44690292c78eb2d574d796d457a957c4bf203df90b3df0f9d0ca33b8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:32.251 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474098+0000", 2024-09-16T03:37:32.251 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.67085942499999995, 2024-09-16T03:37:32.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.207015378, 2024-09-16T03:37:32.251 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.251 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.251 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.251 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474098+0000", 2024-09-16T03:37:32.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474098+0000", 2024-09-16T03:37:32.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474099+0000", 2024-09-16T03:37:32.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8700000000000005e-07 2024-09-16T03:37:32.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474147+0000", 2024-09-16T03:37:32.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8300999999999997e-05 2024-09-16T03:37:32.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474156+0000", 2024-09-16T03:37:32.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5740000000000002e-06 2024-09-16T03:37:32.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474168+0000", 2024-09-16T03:37:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1294999999999999e-05 2024-09-16T03:37:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.476209+0000", 2024-09-16T03:37:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020416240000000001 2024-09-16T03:37:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.476369+0000", 2024-09-16T03:37:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015955200000000001 2024-09-16T03:37:32.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.476591+0000", 2024-09-16T03:37:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000222065 2024-09-16T03:37:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680697+0000", 2024-09-16T03:37:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20410637000000001 2024-09-16T03:37:32.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680763+0000", 2024-09-16T03:37:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6118999999999995e-05 2024-09-16T03:37:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681056+0000", 2024-09-16T03:37:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029263599999999998 2024-09-16T03:37:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681088+0000", 2024-09-16T03:37:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1520999999999999e-05 2024-09-16T03:37:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681113+0000", 2024-09-16T03:37:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5633999999999999e-05 2024-09-16T03:37:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.257 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:33 3.0 3:1e59f26f:::f36fb4795b66b358920b15346e41df644c7e2d9386f5f260380eec916b18bcae:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474176+0000", 2024-09-16T03:37:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.67078136899999996, 2024-09-16T03:37:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20702289600000001, 2024-09-16T03:37:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474176+0000", 2024-09-16T03:37:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474176+0000", 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474177+0000", 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8999999999999995e-07 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474740+0000", 2024-09-16T03:37:32.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056329899999999996 2024-09-16T03:37:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474750+0000", 2024-09-16T03:37:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5599999999999999e-06 2024-09-16T03:37:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474758+0000", 2024-09-16T03:37:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7779999999999996e-06 2024-09-16T03:37:32.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.477151+0000", 2024-09-16T03:37:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023921900000000002 2024-09-16T03:37:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.477265+0000", 2024-09-16T03:37:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011442000000000001 2024-09-16T03:37:32.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.477487+0000", 2024-09-16T03:37:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022229500000000001 2024-09-16T03:37:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680740+0000", 2024-09-16T03:37:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20325288599999999 2024-09-16T03:37:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:32.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680769+0000", 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8694999999999999e-05 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681151+0000", 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000382622 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681177+0000", 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5581000000000001e-05 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681199+0000", 2024-09-16T03:37:32.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1780000000000002e-05 2024-09-16T03:37:32.264 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.264 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.264 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.264 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:34 3.0 3:08307d25:::2b3cb797ae2d3abc24b2cae4a012f26325debe2fc38ccb08bd4503e949d305ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:32.264 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:37:32.264 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.67016725200000005, 2024-09-16T03:37:32.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30798127600000003, 2024-09-16T03:37:32.264 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.264 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474791+0000", 2024-09-16T03:37:32.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6000000000000003e-07 2024-09-16T03:37:32.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474865+0000", 2024-09-16T03:37:32.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4504000000000004e-05 2024-09-16T03:37:32.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474882+0000", 2024-09-16T03:37:32.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6765999999999998e-05 2024-09-16T03:37:32.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474890+0000", 2024-09-16T03:37:32.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.14e-06 2024-09-16T03:37:32.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681242+0000", 2024-09-16T03:37:32.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20635221200000001 2024-09-16T03:37:32.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681355+0000", 2024-09-16T03:37:32.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000112211 2024-09-16T03:37:32.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:32.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681505+0000", 2024-09-16T03:37:32.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015040100000000001 2024-09-16T03:37:32.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:32.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682352+0000", 2024-09-16T03:37:32.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084686400000000002 2024-09-16T03:37:32.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682989+0000", 2024-09-16T03:37:32.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063749800000000001 2024-09-16T03:37:32.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:32.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782695+0000", 2024-09-16T03:37:32.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099706063999999997 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782735+0000", 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9304000000000002e-05 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782771+0000", 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6551999999999998e-05 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.270 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.271 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:32.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:37:32.274 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:37:32.276 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:32.276 INFO:tasks.rados.rados.0.smithi157.stdout:694: finishing write tid 1 to smithi15736696-50 2024-09-16T03:37:32.276 INFO:tasks.rados.rados.0.smithi157.stdout:700: finishing write tid 4 to smithi15736696-42 2024-09-16T03:37:32.276 INFO:tasks.rados.rados.0.smithi157.stdout:701: finishing write tid 1 to smithi15736696-13 2024-09-16T03:37:32.276 INFO:tasks.rados.rados.0.smithi157.stdout:694: finishing write tid 2 to smithi15736696-50 2024-09-16T03:37:32.276 INFO:tasks.rados.rados.0.smithi157.stdout:694: finishing write tid 3 to smithi15736696-50 2024-09-16T03:37:32.276 INFO:tasks.rados.rados.0.smithi157.stdout:694: finishing write tid 4 to smithi15736696-50 2024-09-16T03:37:32.276 INFO:tasks.rados.rados.0.smithi157.stdout:694: oid 50 updating version 0 to 43 2024-09-16T03:37:32.276 INFO:tasks.rados.rados.0.smithi157.stdout:694: oid 50 version 43 is already newer than 42 2024-09-16T03:37:32.277 INFO:tasks.rados.rados.0.smithi157.stdout:694: oid 50 updating version 43 to 44 2024-09-16T03:37:32.277 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 44 (ObjNum 70 snap 0 seq_num 70) dirty exists 2024-09-16T03:37:32.277 INFO:tasks.rados.rados.0.smithi157.stdout:694: left oid 50 (ObjNum 70 snap 0 seq_num 70) 2024-09-16T03:37:32.277 INFO:tasks.rados.rados.0.smithi157.stdout:702: finishing write tid 1 to smithi15736696-18 2024-09-16T03:37:32.277 INFO:tasks.rados.rados.0.smithi157.stdout:706: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-09-16T03:37:32.277 INFO:tasks.rados.rados.0.smithi157.stdout:706: writing smithi15736696-29 from 460525 to 1151377 tid 1 2024-09-16T03:37:32.277 INFO:tasks.rados.rados.0.smithi157.stdout:706: writing smithi15736696-29 from 1622620 to 2352972 tid 2 2024-09-16T03:37:32.277 INFO:tasks.rados.rados.0.smithi157.stdout:706: writing smithi15736696-29 from 3060489 to 3260525 tid 3 2024-09-16T03:37:32.277 INFO:tasks.rados.rados.0.smithi157.stdout:694: done (7 left) 2024-09-16T03:37:32.277 INFO:tasks.rados.rados.0.smithi157.stdout:699: done (6 left) 2024-09-16T03:37:32.277 INFO:tasks.rados.rados.0.smithi157.stdout:707: write oid 8 current snap is 0 2024-09-16T03:37:32.277 INFO:tasks.rados.rados.0.smithi157.stdout:702: finishing write tid 2 to smithi15736696-18 2024-09-16T03:37:32.277 INFO:tasks.rados.rados.0.smithi157.stdout:700: finishing write tid 5 to smithi15736696-42 2024-09-16T03:37:32.278 INFO:tasks.rados.rados.0.smithi157.stdout:700: oid 42 updating version 0 to 20 2024-09-16T03:37:32.278 INFO:tasks.rados.rados.0.smithi157.stdout:700: oid 42 version 20 is already newer than 19 2024-09-16T03:37:32.278 INFO:tasks.rados.rados.0.smithi157.stdout:700: oid 42 updating version 20 to 21 2024-09-16T03:37:32.278 INFO:tasks.rados.rados.0.smithi157.stdout:700: oid 42 version 21 is already newer than 18 2024-09-16T03:37:32.278 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 21 (ObjNum 72 snap 0 seq_num 72) dirty exists 2024-09-16T03:37:32.278 INFO:tasks.rados.rados.0.smithi157.stdout:700: left oid 42 (ObjNum 72 snap 0 seq_num 72) 2024-09-16T03:37:32.278 INFO:tasks.rados.rados.0.smithi157.stdout:701: finishing write tid 2 to smithi15736696-13 2024-09-16T03:37:32.278 INFO:tasks.rados.rados.0.smithi157.stdout:701: finishing write tid 3 to smithi15736696-13 2024-09-16T03:37:32.278 INFO:tasks.rados.rados.0.smithi157.stdout:707: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-09-16T03:37:32.278 INFO:tasks.rados.rados.0.smithi157.stdout:707: writing smithi15736696-8 from 477332 to 1243433 tid 1 2024-09-16T03:37:32.278 INFO:tasks.rados.rados.0.smithi157.stdout:707: writing smithi15736696-8 from 1681102 to 2155112 tid 2 2024-09-16T03:37:32.279 INFO:tasks.rados.rados.0.smithi157.stdout:707: writing smithi15736696-8 from 2687912 to 3277332 tid 3 2024-09-16T03:37:32.279 INFO:tasks.rados.rados.0.smithi157.stdout:700: done (6 left) 2024-09-16T03:37:32.279 INFO:tasks.rados.rados.0.smithi157.stdout:708: read oid 44 snap -1 2024-09-16T03:37:32.279 INFO:tasks.rados.rados.0.smithi157.stdout:708: expect (ObjNum 63 snap 0 seq_num 63) 2024-09-16T03:37:32.279 INFO:tasks.rados.rados.0.smithi157.stdout:709: read oid 41 snap -1 2024-09-16T03:37:32.279 INFO:tasks.rados.rados.0.smithi157.stdout:709: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T03:37:32.279 INFO:tasks.rados.rados.0.smithi157.stdout:710: delete oid 19 current snap is 0 2024-09-16T03:37:32.279 INFO:tasks.rados.rados.0.smithi157.stdout:701: finishing write tid 4 to smithi15736696-13 2024-09-16T03:37:32.279 INFO:tasks.rados.rados.0.smithi157.stdout:702: finishing write tid 3 to smithi15736696-18 2024-09-16T03:37:32.279 INFO:tasks.rados.rados.0.smithi157.stdout:701: finishing write tid 5 to smithi15736696-13 2024-09-16T03:37:32.280 INFO:tasks.rados.rados.0.smithi157.stdout:701: oid 13 updating version 0 to 31 2024-09-16T03:37:32.280 INFO:tasks.rados.rados.0.smithi157.stdout:701: oid 13 version 31 is already newer than 30 2024-09-16T03:37:32.280 INFO:tasks.rados.rados.0.smithi157.stdout:701: oid 13 updating version 31 to 32 2024-09-16T03:37:32.280 INFO:tasks.rados.rados.0.smithi157.stdout:701: oid 13 version 32 is already newer than 29 2024-09-16T03:37:32.280 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 32 (ObjNum 73 snap 0 seq_num 73) dirty exists 2024-09-16T03:37:32.280 INFO:tasks.rados.rados.0.smithi157.stdout:701: left oid 13 (ObjNum 73 snap 0 seq_num 73) 2024-09-16T03:37:32.280 INFO:tasks.rados.rados.0.smithi157.stdout:705: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:32.280 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1277144 2024-09-16T03:37:32.280 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1522017 2024-09-16T03:37:32.280 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:32.281 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:32.270+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:32.282 INFO:tasks.rados.rados.0.smithi157.stdout:702: finishing write tid 4 to smithi15736696-18 2024-09-16T03:37:32.282 INFO:tasks.rados.rados.0.smithi157.stdout:702: finishing write tid 5 to smithi15736696-18 2024-09-16T03:37:32.282 INFO:tasks.rados.rados.0.smithi157.stdout:702: oid 18 updating version 0 to 45 2024-09-16T03:37:32.282 INFO:tasks.rados.rados.0.smithi157.stdout:702: oid 18 version 45 is already newer than 43 2024-09-16T03:37:32.282 INFO:tasks.rados.rados.0.smithi157.stdout:702: oid 18 version 45 is already newer than 44 2024-09-16T03:37:32.282 INFO:tasks.rados.rados.0.smithi157.stdout:702: oid 18 version 45 is already newer than 42 2024-09-16T03:37:32.282 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 45 (ObjNum 74 snap 0 seq_num 74) dirty exists 2024-09-16T03:37:32.282 INFO:tasks.rados.rados.0.smithi157.stdout:702: left oid 18 (ObjNum 74 snap 0 seq_num 74) 2024-09-16T03:37:32.282 INFO:tasks.rados.rados.0.smithi157.stdout:706: finishing write tid 1 to smithi15736696-29 2024-09-16T03:37:32.307 INFO:tasks.rados.rados.0.smithi157.stdout:706: finishing write tid 2 to smithi15736696-29 2024-09-16T03:37:32.308 INFO:tasks.rados.rados.0.smithi157.stdout:706: finishing write tid 3 to smithi15736696-29 2024-09-16T03:37:32.308 INFO:tasks.rados.rados.0.smithi157.stdout:706: finishing write tid 4 to smithi15736696-29 2024-09-16T03:37:32.308 INFO:tasks.rados.rados.0.smithi157.stdout:706: finishing write tid 5 to smithi15736696-29 2024-09-16T03:37:32.308 INFO:tasks.rados.rados.0.smithi157.stdout:706: oid 29 updating version 0 to 47 2024-09-16T03:37:32.308 INFO:tasks.rados.rados.0.smithi157.stdout:706: oid 29 version 47 is already newer than 45 2024-09-16T03:37:32.308 INFO:tasks.rados.rados.0.smithi157.stdout:706: oid 29 updating version 47 to 48 2024-09-16T03:37:32.308 INFO:tasks.rados.rados.0.smithi157.stdout:706: oid 29 version 48 is already newer than 46 2024-09-16T03:37:32.308 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 48 (ObjNum 75 snap 0 seq_num 75) dirty exists 2024-09-16T03:37:32.308 INFO:tasks.rados.rados.0.smithi157.stdout:706: left oid 29 (ObjNum 75 snap 0 seq_num 75) 2024-09-16T03:37:32.323 INFO:tasks.rados.rados.0.smithi157.stdout:708: expect (ObjNum 63 snap 0 seq_num 63) 2024-09-16T03:37:32.323 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:32.342 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:32.340+0000 7efef16d9640 -1 osd.13 52 handle_osd_map injecting map crc failure 2024-09-16T03:37:32.373 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:32.371+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:32.386 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:32.386 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:32.386 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:32.395 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:37:32.403 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:32.403 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:32.421 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:32.421 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:32.438 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:32.439 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:32.440 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:32.438+0000 7f4214d04640 -1 osd.8 52 handle_osd_map injecting map crc failure 2024-09-16T03:37:32.455 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:32.455 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:32.471 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:32.471 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:32.474 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:32.473+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:32.487 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:32.488 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:32.498 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:32.499 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:32.499 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:37:32.499 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:37:32.499 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:32.500 INFO:tasks.rados.rados.0.smithi157.stdout:709: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T03:37:32.504 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:32.505 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:32.507 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:37:32.508 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1459553 2024-09-16T03:37:32.522 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:32.522 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:32.538 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:32.538 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:32.551 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:32.563 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:32.564 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:32.576 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:32.574+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:32.580 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:32.580 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:32.581 INFO:tasks.rados.rados.0.smithi157.stdout:707: finishing write tid 1 to smithi15736696-8 2024-09-16T03:37:32.581 INFO:tasks.rados.rados.0.smithi157.stdout:707: finishing write tid 2 to smithi15736696-8 2024-09-16T03:37:32.582 INFO:tasks.rados.rados.0.smithi157.stdout:707: finishing write tid 3 to smithi15736696-8 2024-09-16T03:37:32.582 INFO:tasks.rados.rados.0.smithi157.stdout:701: done (8 left) 2024-09-16T03:37:32.582 INFO:tasks.rados.rados.0.smithi157.stdout:702: done (7 left) 2024-09-16T03:37:32.582 INFO:tasks.rados.rados.0.smithi157.stdout:704: done (6 left) 2024-09-16T03:37:32.582 INFO:tasks.rados.rados.0.smithi157.stdout:705: done (5 left) 2024-09-16T03:37:32.583 INFO:tasks.rados.rados.0.smithi157.stdout:706: done (4 left) 2024-09-16T03:37:32.583 INFO:tasks.rados.rados.0.smithi157.stdout:708: done (3 left) 2024-09-16T03:37:32.583 INFO:tasks.rados.rados.0.smithi157.stdout:709: done (2 left) 2024-09-16T03:37:32.583 INFO:tasks.rados.rados.0.smithi157.stdout:710: done (1 left) 2024-09-16T03:37:32.583 INFO:tasks.rados.rados.0.smithi157.stdout:711: read oid 14 snap -1 2024-09-16T03:37:32.583 INFO:tasks.rados.rados.0.smithi157.stdout:711: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T03:37:32.583 INFO:tasks.rados.rados.0.smithi157.stdout:712: tier_flush oid 44 2024-09-16T03:37:32.583 INFO:tasks.rados.rados.0.smithi157.stdout:712: tier_flush oid 44 snap -1 2024-09-16T03:37:32.583 INFO:tasks.rados.rados.0.smithi157.stdout:713: read oid 48 snap -1 2024-09-16T03:37:32.583 INFO:tasks.rados.rados.0.smithi157.stdout:713: expect deleted 2024-09-16T03:37:32.583 INFO:tasks.rados.rados.0.smithi157.stdout:714: delete oid 4 current snap is 0 2024-09-16T03:37:32.586 INFO:tasks.rados.rados.0.smithi157.stdout:711: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T03:37:32.590 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 834223 2024-09-16T03:37:32.597 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:32.597 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:32.612 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:32.612 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:32.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:32.612 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:37:32.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.612 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:57 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:32.612 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:37:32.613 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.485762300000001, 2024-09-16T03:37:32.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20372090900000001, 2024-09-16T03:37:32.613 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.613 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.613 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.613 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:37:32.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:37:32.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125313+0000", 2024-09-16T03:37:32.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7180000000000001e-06 2024-09-16T03:37:32.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126730+0000", 2024-09-16T03:37:32.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00141731 2024-09-16T03:37:32.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126746+0000", 2024-09-16T03:37:32.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6455000000000001e-05 2024-09-16T03:37:32.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126756+0000", 2024-09-16T03:37:32.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:37:32.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227068+0000", 2024-09-16T03:37:32.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100312308 2024-09-16T03:37:32.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227085+0000", 2024-09-16T03:37:32.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6898e-05 2024-09-16T03:37:32.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328995+0000", 2024-09-16T03:37:32.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101909903 2024-09-16T03:37:32.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.617 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.617 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329031+0000", 2024-09-16T03:37:32.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5717999999999999e-05 2024-09-16T03:37:32.617 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.617 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.617 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.617 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:58 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:32.618 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:37:32.618 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.48242917, 2024-09-16T03:37:32.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084524500000001, 2024-09-16T03:37:32.618 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.618 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.618 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.618 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.618 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.618 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:37:32.618 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:37:32.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128644+0000", 2024-09-16T03:37:32.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.251e-06 2024-09-16T03:37:32.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128913+0000", 2024-09-16T03:37:32.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026856200000000002 2024-09-16T03:37:32.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.620 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128925+0000", 2024-09-16T03:37:32.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2109999999999999e-05 2024-09-16T03:37:32.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.620 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128933+0000", 2024-09-16T03:37:32.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6260000000000003e-06 2024-09-16T03:37:32.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328196+0000", 2024-09-16T03:37:32.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19926303200000001 2024-09-16T03:37:32.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328253+0000", 2024-09-16T03:37:32.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7262999999999997e-05 2024-09-16T03:37:32.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329458+0000", 2024-09-16T03:37:32.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012051939999999999 2024-09-16T03:37:32.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329488+0000", 2024-09-16T03:37:32.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0207000000000001e-05 2024-09-16T03:37:32.622 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.622 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.622 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.622 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:59 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:32.622 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:37:32.622 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.481933736999999, 2024-09-16T03:37:32.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036954900000001, 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129139+0000", 2024-09-16T03:37:32.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6099999999999999e-07 2024-09-16T03:37:32.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129158+0000", 2024-09-16T03:37:32.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088e-05 2024-09-16T03:37:32.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129168+0000", 2024-09-16T03:37:32.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.048e-05 2024-09-16T03:37:32.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.625 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.229233+0000", 2024-09-16T03:37:32.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100065168 2024-09-16T03:37:32.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.625 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.625 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328815+0000", 2024-09-16T03:37:32.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099581694999999998 2024-09-16T03:37:32.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.626 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328845+0000", 2024-09-16T03:37:32.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0193e-05 2024-09-16T03:37:32.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.626 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329492+0000", 2024-09-16T03:37:32.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064671800000000001 2024-09-16T03:37:32.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.626 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329508+0000", 2024-09-16T03:37:32.626 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6246e-05 2024-09-16T03:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.0:15 4.d e47/32 4:b3fe05b2:::smithi15736696-15:head v 47'5, mlcod=39'3)", 2024-09-16T03:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.3086390110000004, 2024-09-16T03:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100920751, 2024-09-16T03:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:32.627 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:32.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:32.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:37:32.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:32.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:37:32.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:32.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:32.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302435+0000", 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2340000000000001e-06 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302446+0000", 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0596e-05 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302461+0000", 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5082000000000001e-05 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:32.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402592+0000", 2024-09-16T03:37:32.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100130488 2024-09-16T03:37:32.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:32.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402712+0000", 2024-09-16T03:37:32.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012013100000000001 2024-09-16T03:37:32.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:32.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402741+0000", 2024-09-16T03:37:32.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9717999999999999e-05 2024-09-16T03:37:32.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:32.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403309+0000", 2024-09-16T03:37:32.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056725700000000005 2024-09-16T03:37:32.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:32.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:32.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:32.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403354+0000", 2024-09-16T03:37:32.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5244999999999998e-05 2024-09-16T03:37:32.631 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.631 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.631 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.631 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:32.632 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:32.632 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:32.632 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:32.633 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:32.633 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:32.634 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:32.634 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:37:32.635 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 915845 2024-09-16T03:37:32.646 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:32.647 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:32.649 INFO:tasks.rados.rados.0.smithi157.stdout:707: finishing write tid 4 to smithi15736696-8 2024-09-16T03:37:32.649 INFO:tasks.rados.rados.0.smithi157.stdout:707: finishing write tid 5 to smithi15736696-8 2024-09-16T03:37:32.649 INFO:tasks.rados.rados.0.smithi157.stdout:707: oid 8 updating version 0 to 36 2024-09-16T03:37:32.649 INFO:tasks.rados.rados.0.smithi157.stdout:707: oid 8 updating version 36 to 37 2024-09-16T03:37:32.649 INFO:tasks.rados.rados.0.smithi157.stdout:707: oid 8 version 37 is already newer than 34 2024-09-16T03:37:32.649 INFO:tasks.rados.rados.0.smithi157.stdout:707: oid 8 version 37 is already newer than 35 2024-09-16T03:37:32.649 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 37 (ObjNum 76 snap 0 seq_num 76) dirty exists 2024-09-16T03:37:32.649 INFO:tasks.rados.rados.0.smithi157.stdout:707: left oid 8 (ObjNum 76 snap 0 seq_num 76) 2024-09-16T03:37:32.663 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:32.663 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:32.677 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:32.675+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:32.684 INFO:tasks.rados.rados.0.smithi157.stdout:707: done (4 left) 2024-09-16T03:37:32.684 INFO:tasks.rados.rados.0.smithi157.stdout:711: done (3 left) 2024-09-16T03:37:32.684 INFO:tasks.rados.rados.0.smithi157.stdout:713: done (2 left) 2024-09-16T03:37:32.684 INFO:tasks.rados.rados.0.smithi157.stdout:714: done (1 left) 2024-09-16T03:37:32.684 INFO:tasks.rados.rados.0.smithi157.stdout:715: read oid 50 snap -1 2024-09-16T03:37:32.684 INFO:tasks.rados.rados.0.smithi157.stdout:715: expect (ObjNum 70 snap 0 seq_num 70) 2024-09-16T03:37:32.684 INFO:tasks.rados.rados.0.smithi157.stdout:716: read oid 5 snap -1 2024-09-16T03:37:32.684 INFO:tasks.rados.rados.0.smithi157.stdout:716: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T03:37:32.685 INFO:tasks.rados.rados.0.smithi157.stdout:717: read oid 20 snap -1 2024-09-16T03:37:32.685 INFO:tasks.rados.rados.0.smithi157.stdout:717: expect (ObjNum 65 snap 0 seq_num 65) 2024-09-16T03:37:32.685 INFO:tasks.rados.rados.0.smithi157.stdout:718: read oid 32 snap -1 2024-09-16T03:37:32.685 INFO:tasks.rados.rados.0.smithi157.stdout:718: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-16T03:37:32.685 INFO:tasks.rados.rados.0.smithi157.stdout:719: set_chunk oid 11 target oid 2024-09-16T03:37:32.750 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:32.751 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:32.759 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:37:32.778 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:32.777+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:32.792 INFO:tasks.rados.rados.0.smithi157.stdout:719: set_chunk oid 11 offset: 740352 length: 552960 target oid 35dc25b4c5e4bcbfeaa0545c63363444a17367e6103648141dd56c34be30023f offset: 0 snap -1 2024-09-16T03:37:32.792 INFO:tasks.rados.rados.0.smithi157.stdout:720: read oid 30 snap -1 2024-09-16T03:37:32.793 INFO:tasks.rados.rados.0.smithi157.stdout:720: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-16T03:37:32.793 INFO:tasks.rados.rados.0.smithi157.stdout:721: write oid 3 current snap is 0 2024-09-16T03:37:32.793 INFO:tasks.rados.rados.0.smithi157.stdout:721: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-09-16T03:37:32.798 INFO:tasks.rados.rados.0.smithi157.stdout:721: writing smithi15736696-3 from 527753 to 1235954 tid 1 2024-09-16T03:37:32.805 INFO:tasks.rados.rados.0.smithi157.stdout:721: writing smithi15736696-3 from 1856548 to 2594238 tid 2 2024-09-16T03:37:32.809 INFO:tasks.rados.rados.0.smithi157.stdout:721: writing smithi15736696-3 from 3286534 to 3327753 tid 3 2024-09-16T03:37:32.809 INFO:tasks.rados.rados.0.smithi157.stdout:722: read oid 10 snap -1 2024-09-16T03:37:32.809 INFO:tasks.rados.rados.0.smithi157.stdout:722: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:37:32.809 INFO:tasks.rados.rados.0.smithi157.stdout:723: set_chunk oid 38 target oid 2024-09-16T03:37:32.863 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:32.863 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:32.872 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:37:32.879 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:32.878+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:32.976 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:32.976 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:32.980 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:32.978+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:32.985 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:37:33.081 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:33.080+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:33.089 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:33.089 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:33.097 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:37:33.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:37:33.198 INFO:tasks.rados.rados.0.smithi157.stdout:723: set_chunk oid 38 offset: 881152 length: 1496576 target oid e1088dd1abad9616f276ffe845e0947f7b4b34f11e46b6546680d2fcc339a6b2 offset: 0 snap -1 2024-09-16T03:37:33.198 INFO:tasks.rados.rados.0.smithi157.stdout:724: read oid 23 snap -1 2024-09-16T03:37:33.198 INFO:tasks.rados.rados.0.smithi157.stdout:724: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T03:37:33.199 INFO:tasks.rados.rados.0.smithi157.stdout:725: read oid 1 snap -1 2024-09-16T03:37:33.199 INFO:tasks.rados.rados.0.smithi157.stdout:725: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T03:37:33.199 INFO:tasks.rados.rados.0.smithi157.stdout:726: set_chunk oid 6 target oid 2024-09-16T03:37:33.200 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:33.200 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:33.209 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:37:33.216 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:33.215+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:33.283 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:33.281+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:33.314 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:33.314 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:33.322 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:37:33.384 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:33.382+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:33.426 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:33.427 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:33.435 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:37:33.485 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:33.484+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:33.579 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:33.579 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:33.586 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:33.584+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:33.587 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:37:33.605 INFO:tasks.rados.rados.0.smithi157.stdout:726: set_chunk oid 6 offset: 1509888 length: 480768 target oid d3309c925a62c10defe8ba98995e2e3e312634a0aff7f15db029c20734882555 offset: 0 snap -1 2024-09-16T03:37:33.605 INFO:tasks.rados.rados.0.smithi157.stdout:727: read oid 16 snap -1 2024-09-16T03:37:33.605 INFO:tasks.rados.rados.0.smithi157.stdout:727: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-16T03:37:33.605 INFO:tasks.rados.rados.0.smithi157.stdout:728: tier_evict oid 49 2024-09-16T03:37:33.605 INFO:tasks.rados.rados.0.smithi157.stdout:728: tier_evict oid 49 snap -1 2024-09-16T03:37:33.605 INFO:tasks.rados.rados.0.smithi157.stdout:716: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-16T03:37:33.627 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:33.650 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:33.672 INFO:tasks.rados.rados.0.smithi157.stdout:715: expect (ObjNum 70 snap 0 seq_num 70) 2024-09-16T03:37:33.687 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:33.685+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:33.688 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1649619 2024-09-16T03:37:33.691 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:33.691 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:33.700 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:37:33.736 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1387785 2024-09-16T03:37:33.786 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1138642 2024-09-16T03:37:33.788 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:33.786+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:33.828 INFO:tasks.rados.rados.0.smithi157.stdout:722: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T03:37:33.843 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:33.844 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:33.844 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:33.853 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:37:33.868 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:33.889 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:33.888+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:33.892 INFO:tasks.rados.rados.0.smithi157.stdout:718: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-16T03:37:33.892 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:33.955 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:33.956 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:33.964 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:37:33.967 INFO:tasks.rados.rados.0.smithi157.stdout:717: expect (ObjNum 65 snap 0 seq_num 65) 2024-09-16T03:37:33.972 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 959243 2024-09-16T03:37:33.990 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:33.988+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:34.024 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2031784 2024-09-16T03:37:34.046 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:34.048 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:37:34.048 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:37:34.048 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:37:34.048 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-09-16T03:37:34.048 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T03:37:34.048 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T03:37:34.049 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok config set filestore_inject_stall 3 2024-09-16T03:37:34.067 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:34.067 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:34.076 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:37:34.096 INFO:tasks.rados.rados.0.smithi157.stdout:719: finishing set_chunk to oid 11 2024-09-16T03:37:34.096 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 27 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-16T03:37:34.096 INFO:tasks.rados.rados.0.smithi157.stdout:715: done (14 left) 2024-09-16T03:37:34.096 INFO:tasks.rados.rados.0.smithi157.stdout:716: done (13 left) 2024-09-16T03:37:34.096 INFO:tasks.rados.rados.0.smithi157.stdout:717: done (12 left) 2024-09-16T03:37:34.096 INFO:tasks.rados.rados.0.smithi157.stdout:718: done (11 left) 2024-09-16T03:37:34.096 INFO:tasks.rados.rados.0.smithi157.stdout:719: done (10 left) 2024-09-16T03:37:34.096 INFO:tasks.rados.rados.0.smithi157.stdout:722: done (9 left) 2024-09-16T03:37:34.096 INFO:tasks.rados.rados.0.smithi157.stdout:729: read oid 39 snap -1 2024-09-16T03:37:34.096 INFO:tasks.rados.rados.0.smithi157.stdout:729: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-16T03:37:34.096 INFO:tasks.rados.rados.0.smithi157.stdout:730: read oid 29 snap -1 2024-09-16T03:37:34.096 INFO:tasks.rados.rados.0.smithi157.stdout:730: expect (ObjNum 75 snap 0 seq_num 75) 2024-09-16T03:37:34.097 INFO:tasks.rados.rados.0.smithi157.stdout:731: read oid 41 snap -1 2024-09-16T03:37:34.097 INFO:tasks.rados.rados.0.smithi157.stdout:731: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T03:37:34.097 INFO:tasks.rados.rados.0.smithi157.stdout:732: read oid 20 snap -1 2024-09-16T03:37:34.097 INFO:tasks.rados.rados.0.smithi157.stdout:732: expect (ObjNum 65 snap 0 seq_num 65) 2024-09-16T03:37:34.097 INFO:tasks.rados.rados.0.smithi157.stdout:733: read oid 7 snap -1 2024-09-16T03:37:34.097 INFO:tasks.rados.rados.0.smithi157.stdout:733: expect deleted 2024-09-16T03:37:34.097 INFO:tasks.rados.rados.0.smithi157.stdout:734: set_chunk oid 18 target oid 2024-09-16T03:37:34.097 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:34.094+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:34.159 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T03:37:34.160 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:34.179 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:34.180 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:34.188 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:37:34.192 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:34.190+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:34.293 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:34.292+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:34.316 INFO:tasks.rados.rados.0.smithi157.stdout:734: set_chunk oid 18 offset: 897024 length: 2102272 target oid f01998a8b1965ad32daec964a94b91288095f9602c5d8b1d292b22628e852b49 offset: 0 snap -1 2024-09-16T03:37:34.317 INFO:tasks.rados.rados.0.smithi157.stdout:735: read oid 2 snap -1 2024-09-16T03:37:34.317 INFO:tasks.rados.rados.0.smithi157.stdout:735: expect (ObjNum 64 snap 0 seq_num 64) 2024-09-16T03:37:34.317 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:34.317 INFO:tasks.rados.rados.0.smithi157.stdout:720: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-16T03:37:34.337 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:34.337 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:34.346 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:37:34.346 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:37:34.395 INFO:tasks.rados.rados.0.smithi157.stdout:723: finishing set_chunk to oid 38 2024-09-16T03:37:34.396 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 38 set_chunk returned error code -95 offset: 881152 length: 1496576 2024-09-16T03:37:34.396 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:37:34.396 INFO:tasks.rados.rados.0.smithi157.stdout:725: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T03:37:34.400 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:34.399+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:34.419 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1265226 2024-09-16T03:37:34.452 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:34.452 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:34.454 INFO:tasks.rados.rados.0.smithi157.stdout:721: finishing write tid 1 to smithi15736696-3 2024-09-16T03:37:34.454 INFO:tasks.rados.rados.0.smithi157.stdout:721: finishing write tid 2 to smithi15736696-3 2024-09-16T03:37:34.454 INFO:tasks.rados.rados.0.smithi157.stdout:721: finishing write tid 3 to smithi15736696-3 2024-09-16T03:37:34.454 INFO:tasks.rados.rados.0.smithi157.stdout:720: done (15 left) 2024-09-16T03:37:34.454 INFO:tasks.rados.rados.0.smithi157.stdout:723: done (14 left) 2024-09-16T03:37:34.454 INFO:tasks.rados.rados.0.smithi157.stdout:725: done (13 left) 2024-09-16T03:37:34.454 INFO:tasks.rados.rados.0.smithi157.stdout:736: write oid 45 current snap is 0 2024-09-16T03:37:34.454 INFO:tasks.rados.rados.0.smithi157.stdout:721: finishing write tid 4 to smithi15736696-3 2024-09-16T03:37:34.454 INFO:tasks.rados.rados.0.smithi157.stdout:721: finishing write tid 5 to smithi15736696-3 2024-09-16T03:37:34.455 INFO:tasks.rados.rados.0.smithi157.stdout:721: oid 3 updating version 0 to 29 2024-09-16T03:37:34.455 INFO:tasks.rados.rados.0.smithi157.stdout:721: oid 3 version 29 is already newer than 28 2024-09-16T03:37:34.455 INFO:tasks.rados.rados.0.smithi157.stdout:721: oid 3 version 29 is already newer than 27 2024-09-16T03:37:34.455 INFO:tasks.rados.rados.0.smithi157.stdout:721: oid 3 updating version 29 to 30 2024-09-16T03:37:34.455 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 30 (ObjNum 79 snap 0 seq_num 79) dirty exists 2024-09-16T03:37:34.455 INFO:tasks.rados.rados.0.smithi157.stdout:721: left oid 3 (ObjNum 79 snap 0 seq_num 79) 2024-09-16T03:37:34.455 INFO:tasks.rados.rados.0.smithi157.stdout:724: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T03:37:34.456 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1746032 2024-09-16T03:37:34.461 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:37:34.504 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:34.503+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:34.516 INFO:tasks.rados.rados.0.smithi157.stdout:712: got (0) Success 2024-09-16T03:37:34.517 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 0 (ObjNum 63 snap 0 seq_num 63) dirty exists 2024-09-16T03:37:34.517 INFO:tasks.rados.rados.0.smithi157.stdout:728: got (22) Invalid argument 2024-09-16T03:37:34.517 INFO:tasks.rados.rados.0.smithi157.stdout:727: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-16T03:37:34.520 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:34.519+0000 7ff266366640 -1 osd.11 53 handle_osd_map injecting map crc failure 2024-09-16T03:37:34.567 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:34.567 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:34.576 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:37:34.596 INFO:tasks.rados.rados.0.smithi157.stdout:726: finishing set_chunk to oid 6 2024-09-16T03:37:34.596 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 1509888 length: 480768 2024-09-16T03:37:34.597 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:37:34.597 INFO:tasks.rados.rados.0.smithi157.stdout:736: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-09-16T03:37:34.597 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:34.594+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:34.603 INFO:tasks.rados.rados.0.smithi157.stdout:736: writing smithi15736696-45 from 544560 to 1328010 tid 1 2024-09-16T03:37:34.609 INFO:tasks.rados.rados.0.smithi157.stdout:736: writing smithi15736696-45 from 1915030 to 2396378 tid 2 2024-09-16T03:37:34.614 INFO:tasks.rados.rados.0.smithi157.stdout:736: writing smithi15736696-45 from 2797604 to 3277365 tid 3 2024-09-16T03:37:34.614 INFO:tasks.rados.rados.0.smithi157.stdout:712: done (13 left) 2024-09-16T03:37:34.614 INFO:tasks.rados.rados.0.smithi157.stdout:721: done (12 left) 2024-09-16T03:37:34.614 INFO:tasks.rados.rados.0.smithi157.stdout:724: done (11 left) 2024-09-16T03:37:34.614 INFO:tasks.rados.rados.0.smithi157.stdout:726: done (10 left) 2024-09-16T03:37:34.614 INFO:tasks.rados.rados.0.smithi157.stdout:727: done (9 left) 2024-09-16T03:37:34.614 INFO:tasks.rados.rados.0.smithi157.stdout:728: done (8 left) 2024-09-16T03:37:34.614 INFO:tasks.rados.rados.0.smithi157.stdout:737: write (excl) oid 37 current snap is 0 2024-09-16T03:37:34.614 INFO:tasks.rados.rados.0.smithi157.stdout:737: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-09-16T03:37:34.619 INFO:tasks.rados.rados.0.smithi157.stdout:737: writing smithi15736696-37 from 561367 to 1020066 tid 1 2024-09-16T03:37:34.626 INFO:tasks.rados.rados.0.smithi157.stdout:737: writing smithi15736696-37 from 1573512 to 2314871 tid 2 2024-09-16T03:37:34.632 INFO:tasks.rados.rados.0.smithi157.stdout:737: writing smithi15736696-37 from 2941380 to 3361367 tid 3 2024-09-16T03:37:34.632 INFO:tasks.rados.rados.0.smithi157.stdout:738: read oid 46 snap -1 2024-09-16T03:37:34.633 INFO:tasks.rados.rados.0.smithi157.stdout:738: expect (ObjNum 71 snap 0 seq_num 71) 2024-09-16T03:37:34.633 INFO:tasks.rados.rados.0.smithi157.stdout:739: read oid 40 snap -1 2024-09-16T03:37:34.633 INFO:tasks.rados.rados.0.smithi157.stdout:739: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-16T03:37:34.633 INFO:tasks.rados.rados.0.smithi157.stdout:740: read oid 33 snap -1 2024-09-16T03:37:34.633 INFO:tasks.rados.rados.0.smithi157.stdout:740: expect (ObjNum 60 snap 0 seq_num 60) 2024-09-16T03:37:34.633 INFO:tasks.rados.rados.0.smithi157.stdout:741: set_chunk oid 11 target oid 2024-09-16T03:37:34.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:37:34.702 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:34.702 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:34.711 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:37:34.720 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:34.718+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:34.752 INFO:tasks.rados.rados.0.smithi157.stdout:741: set_chunk oid 11 offset: 352256 length: 2168320 target oid 3fe30c185a9e639eda809550d42af155ca8e7e1b727c11f74cf38485832f48e4 offset: 0 snap -1 2024-09-16T03:37:34.752 INFO:tasks.rados.rados.0.smithi157.stdout:742: tier_promote oid 47 2024-09-16T03:37:34.752 INFO:tasks.rados.rados.0.smithi157.stdout:729: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-16T03:37:34.752 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:34.799 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:34.797+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:34.801 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:34.802 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:34.812 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:34.819 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:34.819 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:34.836 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:34.837 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:34.843 INFO:tasks.rados.rados.0.smithi157.stdout:732: expect (ObjNum 65 snap 0 seq_num 65) 2024-09-16T03:37:34.853 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:34.854 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:34.857 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:34.857 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:34.867 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:37:34.871 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:34.871 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:34.888 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:34.889 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:34.905 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:34.905 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:34.910 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:34.909+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:34.922 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:34.922 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:34.935 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:34.938 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:34.939 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:34.971 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:34.972 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:34.978 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:34.978 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:34.980 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:37:34.986 INFO:tasks.rados.rados.0.smithi157.stdout:729: done (13 left) 2024-09-16T03:37:34.986 INFO:tasks.rados.rados.0.smithi157.stdout:732: done (12 left) 2024-09-16T03:37:34.986 INFO:tasks.rados.rados.0.smithi157.stdout:733: done (11 left) 2024-09-16T03:37:34.986 INFO:tasks.rados.rados.0.smithi157.stdout:743: read oid 21 snap -1 2024-09-16T03:37:34.986 INFO:tasks.rados.rados.0.smithi157.stdout:743: expect deleted 2024-09-16T03:37:34.986 INFO:tasks.rados.rados.0.smithi157.stdout:744: read oid 30 snap -1 2024-09-16T03:37:34.987 INFO:tasks.rados.rados.0.smithi157.stdout:744: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-16T03:37:34.987 INFO:tasks.rados.rados.0.smithi157.stdout:745: read oid 26 snap -1 2024-09-16T03:37:34.987 INFO:tasks.rados.rados.0.smithi157.stdout:745: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-16T03:37:34.987 INFO:tasks.rados.rados.0.smithi157.stdout:746: set_chunk oid 6 target oid 2024-09-16T03:37:34.995 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:34.995 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:34.999 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:34.998+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:35.037 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:35.037 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:35.090 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:35.090 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:35.090 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:37:35.090 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:37:35.090 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:35.100 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:37:35.101 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:35.099+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:35.102 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:35.102 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:35.119 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:35.119 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:35.136 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:35.136 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:35.182 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:35.183 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:35.202 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:35.200+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:35.207 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:35.207 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:37:35.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:37:35.207 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:35.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.207 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T03:37:35.207 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:37:35.207 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.126931712000001, 2024-09-16T03:37:35.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.88203620400000005, 2024-09-16T03:37:35.207 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.207 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.207 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.208 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.208 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.208 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.208 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:37:35.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.208 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:37:35.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:37:35.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076767+0000", 2024-09-16T03:37:35.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9300000000000002e-06 2024-09-16T03:37:35.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076774+0000", 2024-09-16T03:37:35.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6630000000000004e-06 2024-09-16T03:37:35.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076785+0000", 2024-09-16T03:37:35.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1221e-05 2024-09-16T03:37:35.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076796+0000", 2024-09-16T03:37:35.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0428000000000001e-05 2024-09-16T03:37:35.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076856+0000", 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076863+0000", 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9840000000000004e-06 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855770+0000", 2024-09-16T03:37:35.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.77890736999999999 2024-09-16T03:37:35.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855902+0000", 2024-09-16T03:37:35.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013207 2024-09-16T03:37:35.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.856054+0000", 2024-09-16T03:37:35.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015220899999999999 2024-09-16T03:37:35.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.858407+0000", 2024-09-16T03:37:35.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023530069999999998 2024-09-16T03:37:35.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958598+0000", 2024-09-16T03:37:35.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100190194 2024-09-16T03:37:35.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958716+0000", 2024-09-16T03:37:35.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118589 2024-09-16T03:37:35.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958761+0000", 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5120000000000002e-05 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958802+0000", 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0383999999999998e-05 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 40.608515535000002, 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70508991300000001, 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 52 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:37:35.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:37:35.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6300000000000005e-07 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595187+0000", 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9790000000000001e-06 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595194+0000", 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6490000000000002e-06 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595208+0000", 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4139e-05 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801931+0000", 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.206723083 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801990+0000", 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8625999999999997e-05 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802066+0000", 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6553999999999999e-05 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802504+0000", 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043775200000000002 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.199998+0000", 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39749365599999997 2024-09-16T03:37:35.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300151+0000", 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015310099999999 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300210+0000", 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9376999999999997e-05 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300271+0000", 2024-09-16T03:37:35.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1334000000000006e-05 2024-09-16T03:37:35.221 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.221 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.221 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.221 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.221 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.369956+0000", 2024-09-16T03:37:35.221 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 38.833741220999997, 2024-09-16T03:37:35.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60610467099999998, 2024-09-16T03:37:35.221 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.221 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.221 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.221 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.222 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.222 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.222 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 69 2024-09-16T03:37:35.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.222 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.369956+0000", 2024-09-16T03:37:35.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.369956+0000", 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.369957+0000", 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3099999999999997e-07 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370028+0000", 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1404999999999998e-05 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370036+0000", 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.79e-06 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370042+0000", 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.494e-06 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572455+0000", 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20241295100000001 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572512+0000", 2024-09-16T03:37:35.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7071000000000003e-05 2024-09-16T03:37:35.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572602+0000", 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9275000000000002e-05 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673919+0000", 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10131736099999999 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674651+0000", 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073190500000000003 2024-09-16T03:37:35.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975976+0000", 2024-09-16T03:37:35.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.301325391 2024-09-16T03:37:35.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976014+0000", 2024-09-16T03:37:35.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8226000000000003e-05 2024-09-16T03:37:35.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976061+0000", 2024-09-16T03:37:35.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6071e-05 2024-09-16T03:37:35.228 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.228 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.228 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.228 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.228 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:37:35.228 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 38.833645763, 2024-09-16T03:37:35.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60616296300000005, 2024-09-16T03:37:35.229 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.229 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.229 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.229 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.229 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.229 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.229 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 70 2024-09-16T03:37:35.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.229 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:37:35.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370052+0000", 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8899999999999999e-07 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370055+0000", 2024-09-16T03:37:35.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7709999999999998e-06 2024-09-16T03:37:35.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370060+0000", 2024-09-16T03:37:35.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6910000000000002e-06 2024-09-16T03:37:35.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370064+0000", 2024-09-16T03:37:35.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.619e-06 2024-09-16T03:37:35.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572988+0000", 2024-09-16T03:37:35.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20292439300000001 2024-09-16T03:37:35.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.573049+0000", 2024-09-16T03:37:35.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0418999999999999e-05 2024-09-16T03:37:35.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.573133+0000", 2024-09-16T03:37:35.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4324000000000001e-05 2024-09-16T03:37:35.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674789+0000", 2024-09-16T03:37:35.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101655658 2024-09-16T03:37:35.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.875723+0000", 2024-09-16T03:37:35.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20093406699999999 2024-09-16T03:37:35.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976140+0000", 2024-09-16T03:37:35.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10041718199999999 2024-09-16T03:37:35.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976172+0000", 2024-09-16T03:37:35.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2169000000000003e-05 2024-09-16T03:37:35.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976214+0000", 2024-09-16T03:37:35.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2080999999999999e-05 2024-09-16T03:37:35.236 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.236 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.236 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.236 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.236 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:37:35.236 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 38.833624552000003, 2024-09-16T03:37:35.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60623559900000001, 2024-09-16T03:37:35.237 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.237 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.237 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.237 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.237 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.237 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.237 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 71 2024-09-16T03:37:35.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.237 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:37:35.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370077+0000", 2024-09-16T03:37:35.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9940000000000002e-06 2024-09-16T03:37:35.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370081+0000", 2024-09-16T03:37:35.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5850000000000001e-06 2024-09-16T03:37:35.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370087+0000", 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673522+0000", 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.303434969 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673567+0000", 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5080000000000002e-05 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673651+0000", 2024-09-16T03:37:35.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3652999999999993e-05 2024-09-16T03:37:35.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775115+0000", 2024-09-16T03:37:35.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101464178 2024-09-16T03:37:35.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975918+0000", 2024-09-16T03:37:35.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20080318699999999 2024-09-16T03:37:35.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976254+0000", 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033619200000000002 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976285+0000", 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0679000000000002e-05 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976308+0000", 2024-09-16T03:37:35.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3051e-05 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 38.833601192000003, 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60632925100000001, 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.243 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.244 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.244 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 72 2024-09-16T03:37:35.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.244 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:37:35.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:37:35.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:37:35.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:37:35.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370150+0000", 2024-09-16T03:37:35.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3906999999999997e-05 2024-09-16T03:37:35.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370157+0000", 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370162+0000", 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7249999999999997e-06 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673969+0000", 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30380713199999998 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674030+0000", 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1141000000000003e-05 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674124+0000", 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3859999999999999e-05 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775687+0000", 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10156293299999999 2024-09-16T03:37:35.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976103+0000", 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20041631400000001 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976355+0000", 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000251611 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976385+0000", 2024-09-16T03:37:35.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0883999999999998e-05 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976425+0000", 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9666000000000003e-05 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 38.833528708000003, 2024-09-16T03:37:35.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60763126999999995, 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 73 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:37:35.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370169+0000", 2024-09-16T03:37:35.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6199999999999998e-07 2024-09-16T03:37:35.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370171+0000", 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370175+0000", 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.596e-06 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470295+0000", 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100119967 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775148+0000", 2024-09-16T03:37:35.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30485311799999998 2024-09-16T03:37:35.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775203+0000", 2024-09-16T03:37:35.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4592999999999997e-05 2024-09-16T03:37:35.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775293+0000", 2024-09-16T03:37:35.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0859999999999994e-05 2024-09-16T03:37:35.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976223+0000", 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20092996599999999 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976463+0000", 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000239695 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977730+0000", 2024-09-16T03:37:35.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012667690000000001 2024-09-16T03:37:35.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977773+0000", 2024-09-16T03:37:35.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2604000000000001e-05 2024-09-16T03:37:35.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977800+0000", 2024-09-16T03:37:35.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7137000000000001e-05 2024-09-16T03:37:35.255 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.255 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.255 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.256 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.256 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:37:35.256 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 38.733389852999998, 2024-09-16T03:37:35.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60633810099999996, 2024-09-16T03:37:35.256 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.256 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.256 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.256 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.256 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.256 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.256 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 74 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470308+0000", 2024-09-16T03:37:35.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0950000000000001e-06 2024-09-16T03:37:35.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.258 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.258 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470313+0000", 2024-09-16T03:37:35.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:37:35.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.258 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.258 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470320+0000", 2024-09-16T03:37:35.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5089999999999999e-06 2024-09-16T03:37:35.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470327+0000", 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976483+0000", 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50615579499999996 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976533+0000", 2024-09-16T03:37:35.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0096000000000003e-05 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976643+0000", 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109481 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977808+0000", 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011649869999999999 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.981699+0000", 2024-09-16T03:37:35.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003891579 2024-09-16T03:37:35.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076496+0000", 2024-09-16T03:37:35.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.094797333999999997 2024-09-16T03:37:35.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076583+0000", 2024-09-16T03:37:35.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6571999999999997e-05 2024-09-16T03:37:35.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.262 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.262 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076645+0000", 2024-09-16T03:37:35.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2331999999999999e-05 2024-09-16T03:37:35.262 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.262 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.262 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.262 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.262 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 38.733359718000003, 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71251765099999997, 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 75 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470338+0000", 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2500000000000005e-07 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470389+0000", 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.134e-05 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470397+0000", 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8140000000000001e-06 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570537+0000", 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100139425 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977018+0000", 2024-09-16T03:37:35.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40648102899999999 2024-09-16T03:37:35.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977063+0000", 2024-09-16T03:37:35.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5247000000000002e-05 2024-09-16T03:37:35.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977143+0000", 2024-09-16T03:37:35.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0054000000000003e-05 2024-09-16T03:37:35.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.978132+0000", 2024-09-16T03:37:35.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00098896399999999995 2024-09-16T03:37:35.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182725+0000", 2024-09-16T03:37:35.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.204592626 2024-09-16T03:37:35.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182783+0000", 2024-09-16T03:37:35.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8285000000000002e-05 2024-09-16T03:37:35.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182825+0000", 2024-09-16T03:37:35.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2398999999999998e-05 2024-09-16T03:37:35.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182855+0000", 2024-09-16T03:37:35.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9842999999999999e-05 2024-09-16T03:37:35.268 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 38.633133151000003, 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71263271800000005, 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.269 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 76 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570566+0000", 2024-09-16T03:37:35.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:37:35.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570577+0000", 2024-09-16T03:37:35.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0672999999999999e-05 2024-09-16T03:37:35.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570593+0000", 2024-09-16T03:37:35.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5979999999999999e-05 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570604+0000", 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.077e-05 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979890+0000", 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40928624200000002 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.272 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979945+0000", 2024-09-16T03:37:35.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5651e-05 2024-09-16T03:37:35.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980038+0000", 2024-09-16T03:37:35.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2603999999999996e-05 2024-09-16T03:37:35.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980669+0000", 2024-09-16T03:37:35.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063121900000000003 2024-09-16T03:37:35.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283050+0000", 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30238120800000001 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283132+0000", 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1841999999999993e-05 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283168+0000", 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5438999999999997e-05 2024-09-16T03:37:35.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283197+0000", 2024-09-16T03:37:35.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8960000000000001e-05 2024-09-16T03:37:35.275 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.275 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.275 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.275 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:93 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.275 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385015+0000", 2024-09-16T03:37:35.275 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.818682463999998, 2024-09-16T03:37:35.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60601307199999999, 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 93 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385015+0000", 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385015+0000", 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385015+0000", 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6899999999999997e-07 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385071+0000", 2024-09-16T03:37:35.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5674999999999998e-05 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385081+0000", 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6320000000000003e-06 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385087+0000", 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3080000000000004e-06 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.686864+0000", 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30177694399999999 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.686911+0000", 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7261e-05 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.687014+0000", 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103138 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.788048+0000", 2024-09-16T03:37:35.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10103405999999999 2024-09-16T03:37:35.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888243+0000", 2024-09-16T03:37:35.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10019507 2024-09-16T03:37:35.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990973+0000", 2024-09-16T03:37:35.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10273007100000001 2024-09-16T03:37:35.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991003+0000", 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9734999999999999e-05 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991028+0000", 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4508999999999999e-05 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.282 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.282 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:37:35.282 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.818598051999999, 2024-09-16T03:37:35.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70799990400000001, 2024-09-16T03:37:35.282 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.282 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.282 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.282 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.282 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.282 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.282 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 94 2024-09-16T03:37:35.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.282 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385100+0000", 2024-09-16T03:37:35.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9599999999999998e-07 2024-09-16T03:37:35.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385104+0000", 2024-09-16T03:37:35.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7579999999999998e-06 2024-09-16T03:37:35.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385128+0000", 2024-09-16T03:37:35.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4700999999999999e-05 2024-09-16T03:37:35.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385139+0000", 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0587e-05 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787499+0000", 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40235976099999998 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787545+0000", 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6418999999999997e-05 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787623+0000", 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7897000000000003e-05 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888605+0000", 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100982321 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990721+0000", 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102115922 2024-09-16T03:37:35.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093019+0000", 2024-09-16T03:37:35.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10229788500000001 2024-09-16T03:37:35.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093051+0000", 2024-09-16T03:37:35.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2255999999999997e-05 2024-09-16T03:37:35.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093099+0000", 2024-09-16T03:37:35.288 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7500999999999998e-05 2024-09-16T03:37:35.288 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.288 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.288 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.288 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.288 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.288 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.288 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:37:35.288 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.818546728999998, 2024-09-16T03:37:35.288 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80404594699999998, 2024-09-16T03:37:35.288 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.288 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.288 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.289 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.289 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.289 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.289 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 95 2024-09-16T03:37:35.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.289 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.289 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.289 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:37:35.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385151+0000", 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385165+0000", 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4175999999999999e-05 2024-09-16T03:37:35.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385175+0000", 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385183+0000", 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888643+0000", 2024-09-16T03:37:35.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50345988200000003 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888695+0000", 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2834999999999998e-05 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888802+0000", 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010697899999999999 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990658+0000", 2024-09-16T03:37:35.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185585699999999 2024-09-16T03:37:35.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990917+0000", 2024-09-16T03:37:35.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025853600000000002 2024-09-16T03:37:35.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189054+0000", 2024-09-16T03:37:35.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198136812 2024-09-16T03:37:35.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189135+0000", 2024-09-16T03:37:35.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1482000000000004e-05 2024-09-16T03:37:35.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189196+0000", 2024-09-16T03:37:35.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1175999999999998e-05 2024-09-16T03:37:35.294 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.818506102000001, 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80412488400000004, 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.295 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 96 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385192+0000", 2024-09-16T03:37:35.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:37:35.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385267+0000", 2024-09-16T03:37:35.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5535000000000001e-05 2024-09-16T03:37:35.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385277+0000", 2024-09-16T03:37:35.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6239999999999997e-06 2024-09-16T03:37:35.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385284+0000", 2024-09-16T03:37:35.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4800000000000004e-06 2024-09-16T03:37:35.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989283+0000", 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60399849500000002 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989410+0000", 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127298 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989615+0000", 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000205045 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991036+0000", 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001421042 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189228+0000", 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19819188500000001 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189249+0000", 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0996999999999999e-05 2024-09-16T03:37:35.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189293+0000", 2024-09-16T03:37:35.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3912000000000002e-05 2024-09-16T03:37:35.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189316+0000", 2024-09-16T03:37:35.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2739e-05 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.818402370999998, 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80642264100000005, 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.305 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 97 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385296+0000", 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.006e-06 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385301+0000", 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2630000000000003e-06 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385308+0000", 2024-09-16T03:37:35.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6030000000000001e-06 2024-09-16T03:37:35.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485459+0000", 2024-09-16T03:37:35.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150929 2024-09-16T03:37:35.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990238+0000", 2024-09-16T03:37:35.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50477992000000005 2024-09-16T03:37:35.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990287+0000", 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8863999999999998e-05 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990377+0000", 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9900000000000003e-05 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093109+0000", 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102732055 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189351+0000", 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096241766000000006 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191570+0000", 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00221915 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191653+0000", 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3046000000000004e-05 2024-09-16T03:37:35.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191717+0000", 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4139000000000004e-05 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.718211914000001, 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90688294999999997, 2024-09-16T03:37:35.311 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 98 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:37:35.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485487+0000", 2024-09-16T03:37:35.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1799999999999999e-06 2024-09-16T03:37:35.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485498+0000", 2024-09-16T03:37:35.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0648999999999999e-05 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485514+0000", 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5943999999999998e-05 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485525+0000", 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.065e-05 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991102+0000", 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50557748400000002 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991161+0000", 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8801999999999997e-05 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991255+0000", 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3936000000000003e-05 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093112+0000", 2024-09-16T03:37:35.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185709499999999 2024-09-16T03:37:35.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291911+0000", 2024-09-16T03:37:35.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19879897699999999 2024-09-16T03:37:35.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392167+0000", 2024-09-16T03:37:35.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10025634999999999 2024-09-16T03:37:35.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392249+0000", 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2175000000000005e-05 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392368+0000", 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118708 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.318 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.318 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:37:35.318 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.718158291000002, 2024-09-16T03:37:35.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90707203199999997, 2024-09-16T03:37:35.318 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.318 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.318 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.318 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.318 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.318 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.318 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 99 2024-09-16T03:37:35.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.318 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485540+0000", 2024-09-16T03:37:35.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-06 2024-09-16T03:37:35.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485661+0000", 2024-09-16T03:37:35.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120835 2024-09-16T03:37:35.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485680+0000", 2024-09-16T03:37:35.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8885999999999998e-05 2024-09-16T03:37:35.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485693+0000", 2024-09-16T03:37:35.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3297e-05 2024-09-16T03:37:35.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991546+0000", 2024-09-16T03:37:35.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.505853098 2024-09-16T03:37:35.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991598+0000", 2024-09-16T03:37:35.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1921e-05 2024-09-16T03:37:35.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991681+0000", 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2502000000000004e-05 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093115+0000", 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10143424299999999 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291977+0000", 2024-09-16T03:37:35.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198862172 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392541+0000", 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100563758 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392579+0000", 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7796999999999997e-05 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392611+0000", 2024-09-16T03:37:35.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2073999999999998e-05 2024-09-16T03:37:35.324 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.324 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.324 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.324 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.324 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:37:35.324 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.717979397999997, 2024-09-16T03:37:35.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0073149939999999, 2024-09-16T03:37:35.324 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.324 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.324 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 100 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:37:35.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:37:35.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7400000000000003e-07 2024-09-16T03:37:35.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485721+0000", 2024-09-16T03:37:35.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4530000000000001e-06 2024-09-16T03:37:35.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485726+0000", 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0590000000000002e-06 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485729+0000", 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092342+0000", 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60661343700000003 2024-09-16T03:37:35.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092469+0000", 2024-09-16T03:37:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012658799999999999 2024-09-16T03:37:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092671+0000", 2024-09-16T03:37:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020206 2024-09-16T03:37:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392452+0000", 2024-09-16T03:37:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29978163699999999 2024-09-16T03:37:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392486+0000", 2024-09-16T03:37:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3736999999999999e-05 2024-09-16T03:37:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492812+0000", 2024-09-16T03:37:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100325621 2024-09-16T03:37:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492901+0000", 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8702000000000006e-05 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.493033+0000", 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000132185 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.330 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.331 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:37:35.331 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.542828876999998, 2024-09-16T03:37:35.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60874581500000002, 2024-09-16T03:37:35.331 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.331 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.331 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.331 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.331 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.331 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.331 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 123 2024-09-16T03:37:35.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.331 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:37:35.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:37:35.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660869+0000", 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9400000000000005e-07 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660921+0000", 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2312000000000003e-05 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660927+0000", 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9329999999999998e-06 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660931+0000", 2024-09-16T03:37:35.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:37:35.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.967991+0000", 2024-09-16T03:37:35.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30706032700000002 2024-09-16T03:37:35.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968039+0000", 2024-09-16T03:37:35.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7477e-05 2024-09-16T03:37:35.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968117+0000", 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8589000000000003e-05 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067678+0000", 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099560406000000004 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170175+0000", 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102496927 2024-09-16T03:37:35.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269491+0000", 2024-09-16T03:37:35.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099316205000000005 2024-09-16T03:37:35.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269574+0000", 2024-09-16T03:37:35.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2903000000000002e-05 2024-09-16T03:37:35.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269614+0000", 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0188000000000002e-05 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.542760178999998, 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60879245500000001, 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.337 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.338 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:35.338 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:35.338 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 124 2024-09-16T03:37:35.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.338 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:37:35.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660938+0000", 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660940+0000", 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2529999999999999e-06 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660943+0000", 2024-09-16T03:37:35.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.01e-06 2024-09-16T03:37:35.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660946+0000", 2024-09-16T03:37:35.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8159999999999998e-06 2024-09-16T03:37:35.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012417+0000", 2024-09-16T03:37:35.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.35147120599999998 2024-09-16T03:37:35.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012497+0000", 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9963000000000007e-05 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012585+0000", 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067680+0000", 2024-09-16T03:37:35.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055095203000000002 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170243+0000", 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10256290799999999 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269662+0000", 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099418896000000007 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269701+0000", 2024-09-16T03:37:35.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8829999999999999e-05 2024-09-16T03:37:35.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269729+0000", 2024-09-16T03:37:35.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8569e-05 2024-09-16T03:37:35.343 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.343 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.343 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.343 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.343 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.343 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:35.345 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:37:35.346 INFO:tasks.rados.rados.0.smithi157.stdout:746: set_chunk oid 6 offset: 114688 length: 1920512 target oid fb2abb5d8594ae081bd7a522ca068bd21ee8449fbf7bef92e8b363b8b3f628bf offset: 0 snap -1 2024-09-16T03:37:35.346 INFO:tasks.rados.rados.0.smithi157.stdout:747: read oid 42 snap -1 2024-09-16T03:37:35.346 INFO:tasks.rados.rados.0.smithi157.stdout:747: expect (ObjNum 72 snap 0 seq_num 72) 2024-09-16T03:37:35.346 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:35.346 INFO:tasks.rados.rados.0.smithi157.stdout:730: expect (ObjNum 75 snap 0 seq_num 75) 2024-09-16T03:37:35.347 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:35.343+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:35.440 INFO:tasks.rados.rados.0.smithi157.stdout:731: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-16T03:37:35.445 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:35.444+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:35.450 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:35.451 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:35.451 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:37:35.451 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:35.459 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:37:35.470 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:35.500 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:35.530 INFO:tasks.rados.rados.0.smithi157.stdout:735: expect (ObjNum 64 snap 0 seq_num 64) 2024-09-16T03:37:35.530 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:35.546 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:35.544+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:35.562 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:37:35.563 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:37:35.563 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:35.571 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:37:35.577 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:35.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:35.646+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:35.672 INFO:tasks.rados.rados.0.smithi157.stdout:730: done (15 left) 2024-09-16T03:37:35.672 INFO:tasks.rados.rados.0.smithi157.stdout:731: done (14 left) 2024-09-16T03:37:35.673 INFO:tasks.rados.rados.0.smithi157.stdout:735: done (13 left) 2024-09-16T03:37:35.673 INFO:tasks.rados.rados.0.smithi157.stdout:748: write oid 34 current snap is 0 2024-09-16T03:37:35.673 INFO:tasks.rados.rados.0.smithi157.stdout:740: expect (ObjNum 60 snap 0 seq_num 60) 2024-09-16T03:37:35.682 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:35.683 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:37:35.683 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:37:35.683 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:35.683 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.683 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1 4.2 4:4a9d235d:::smithi15736696-1:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T03:37:35.683 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:11.955475+0000", 2024-09-16T03:37:35.683 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.724480393, 2024-09-16T03:37:35.683 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.405292714, 2024-09-16T03:37:35.683 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.684 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.684 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.684 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.684 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.684 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.684 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:37:35.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.684 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.684 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955475+0000", 2024-09-16T03:37:35.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.685 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955475+0000", 2024-09-16T03:37:35.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.685 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955477+0000", 2024-09-16T03:37:35.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5040000000000001e-06 2024-09-16T03:37:35.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.686 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.686 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955870+0000", 2024-09-16T03:37:35.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000393082 2024-09-16T03:37:35.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.686 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.687 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955883+0000", 2024-09-16T03:37:35.687 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3173000000000001e-05 2024-09-16T03:37:35.687 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.687 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.687 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.055959+0000", 2024-09-16T03:37:35.687 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10007564300000001 2024-09-16T03:37:35.688 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.688 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.688 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.688 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156811+0000", 2024-09-16T03:37:35.688 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100852073 2024-09-16T03:37:35.688 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.688 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.689 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.689 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156907+0000", 2024-09-16T03:37:35.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5945000000000004e-05 2024-09-16T03:37:35.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.689 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.689 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 8,11", 2024-09-16T03:37:35.689 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157029+0000", 2024-09-16T03:37:35.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012207099999999999 2024-09-16T03:37:35.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.689 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.689 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.160049+0000", 2024-09-16T03:37:35.690 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030194409999999999 2024-09-16T03:37:35.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260277+0000", 2024-09-16T03:37:35.690 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10022834 2024-09-16T03:37:35.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360663+0000", 2024-09-16T03:37:35.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10038625599999999 2024-09-16T03:37:35.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.691 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.691 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.691 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360710+0000", 2024-09-16T03:37:35.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7165e-05 2024-09-16T03:37:35.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.691 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.691 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.691 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360767+0000", 2024-09-16T03:37:35.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7021000000000002e-05 2024-09-16T03:37:35.692 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.692 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.692 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.692 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2 4.2 4:4a9d235d:::smithi15736696-1:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T03:37:35.692 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.056000+0000", 2024-09-16T03:37:35.692 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.623954788999999, 2024-09-16T03:37:35.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30489393999999997, 2024-09-16T03:37:35.692 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.693 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.693 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.693 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.693 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.693 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.693 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2 2024-09-16T03:37:35.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.694 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056000+0000", 2024-09-16T03:37:35.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056000+0000", 2024-09-16T03:37:35.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056003+0000", 2024-09-16T03:37:35.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3250000000000002e-06 2024-09-16T03:37:35.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056130+0000", 2024-09-16T03:37:35.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127059 2024-09-16T03:37:35.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056139+0000", 2024-09-16T03:37:35.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1419999999999999e-06 2024-09-16T03:37:35.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156244+0000", 2024-09-16T03:37:35.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10010532799999999 2024-09-16T03:37:35.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.158582+0000", 2024-09-16T03:37:35.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023374759999999998 2024-09-16T03:37:35.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.698 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.158698+0000", 2024-09-16T03:37:35.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000116051 2024-09-16T03:37:35.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.698 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 8,11", 2024-09-16T03:37:35.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.158807+0000", 2024-09-16T03:37:35.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010915400000000001 2024-09-16T03:37:35.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260247+0000", 2024-09-16T03:37:35.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10144022900000001 2024-09-16T03:37:35.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360554+0000", 2024-09-16T03:37:35.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100307113 2024-09-16T03:37:35.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360813+0000", 2024-09-16T03:37:35.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025846499999999999 2024-09-16T03:37:35.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360857+0000", 2024-09-16T03:37:35.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3954e-05 2024-09-16T03:37:35.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.701 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.701 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360894+0000", 2024-09-16T03:37:35.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7644000000000001e-05 2024-09-16T03:37:35.701 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.701 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.701 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.701 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3 4.2 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T03:37:35.701 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.156258+0000", 2024-09-16T03:37:35.701 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.523697369000001, 2024-09-16T03:37:35.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30472436000000003, 2024-09-16T03:37:35.702 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.702 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.702 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.702 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.702 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.702 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.702 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3 2024-09-16T03:37:35.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.702 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156258+0000", 2024-09-16T03:37:35.703 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156258+0000", 2024-09-16T03:37:35.703 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156259+0000", 2024-09-16T03:37:35.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5269999999999999e-06 2024-09-16T03:37:35.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156266+0000", 2024-09-16T03:37:35.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:37:35.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156273+0000", 2024-09-16T03:37:35.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9299999999999997e-06 2024-09-16T03:37:35.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156277+0000", 2024-09-16T03:37:35.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.019e-06 2024-09-16T03:37:35.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.159689+0000", 2024-09-16T03:37:35.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034124329999999999 2024-09-16T03:37:35.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.708 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.159744+0000", 2024-09-16T03:37:35.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4821999999999998e-05 2024-09-16T03:37:35.709 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.709 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.709 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 8,11", 2024-09-16T03:37:35.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.159858+0000", 2024-09-16T03:37:35.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011353699999999999 2024-09-16T03:37:35.709 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.709 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.709 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260289+0000", 2024-09-16T03:37:35.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10043086 2024-09-16T03:37:35.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360606+0000", 2024-09-16T03:37:35.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100317502 2024-09-16T03:37:35.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.460834+0000", 2024-09-16T03:37:35.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100228334 2024-09-16T03:37:35.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.460938+0000", 2024-09-16T03:37:35.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010365999999999999 2024-09-16T03:37:35.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.460982+0000", 2024-09-16T03:37:35.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4054000000000002e-05 2024-09-16T03:37:35.711 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.712 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.712 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.712 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4 4.2 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T03:37:35.712 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.156281+0000", 2024-09-16T03:37:35.712 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.523673949999999, 2024-09-16T03:37:35.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30482965400000001, 2024-09-16T03:37:35.712 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.712 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.713 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:35.713 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.713 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.713 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.713 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4 2024-09-16T03:37:35.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.713 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.713 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.713 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156281+0000", 2024-09-16T03:37:35.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156281+0000", 2024-09-16T03:37:35.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156282+0000", 2024-09-16T03:37:35.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.08e-07 2024-09-16T03:37:35.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156284+0000", 2024-09-16T03:37:35.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7159999999999999e-06 2024-09-16T03:37:35.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156288+0000", 2024-09-16T03:37:35.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8139999999999998e-06 2024-09-16T03:37:35.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156291+0000", 2024-09-16T03:37:35.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.971e-06 2024-09-16T03:37:35.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260143+0000", 2024-09-16T03:37:35.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103851517 2024-09-16T03:37:35.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:35.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260223+0000", 2024-09-16T03:37:35.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0439999999999996e-05 2024-09-16T03:37:35.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461007+0000", 2024-09-16T03:37:35.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20078405799999999 2024-09-16T03:37:35.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461053+0000", 2024-09-16T03:37:35.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5667000000000002e-05 2024-09-16T03:37:35.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461111+0000", 2024-09-16T03:37:35.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8063000000000003e-05 2024-09-16T03:37:35.718 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.719 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.719 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.719 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:61 4.5 4:ad0b9852:::smithi15736696-16:head [write 652105~445088,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:35.719 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.256974+0000", 2024-09-16T03:37:35.719 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.422980756000001, 2024-09-16T03:37:35.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30469791499999999, 2024-09-16T03:37:35.719 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.719 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.719 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.720 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.720 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.720 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.720 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 61 2024-09-16T03:37:35.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.720 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256974+0000", 2024-09-16T03:37:35.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256974+0000", 2024-09-16T03:37:35.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.256975+0000", 2024-09-16T03:37:35.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0299999999999998e-07 2024-09-16T03:37:35.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.722 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.722 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257113+0000", 2024-09-16T03:37:35.722 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013768100000000001 2024-09-16T03:37:35.722 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.722 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.722 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257121+0000", 2024-09-16T03:37:35.722 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1680000000000004e-06 2024-09-16T03:37:35.722 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.722 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257127+0000", 2024-09-16T03:37:35.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3099999999999997e-06 2024-09-16T03:37:35.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259291+0000", 2024-09-16T03:37:35.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021634839999999998 2024-09-16T03:37:35.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259396+0000", 2024-09-16T03:37:35.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105364 2024-09-16T03:37:35.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:37:35.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259545+0000", 2024-09-16T03:37:35.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014917700000000001 2024-09-16T03:37:35.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.261929+0000", 2024-09-16T03:37:35.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023834709999999999 2024-09-16T03:37:35.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.726 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.726 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.726 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561404+0000", 2024-09-16T03:37:35.726 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.299475251 2024-09-16T03:37:35.726 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.726 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.726 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.726 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561560+0000", 2024-09-16T03:37:35.726 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015627800000000001 2024-09-16T03:37:35.726 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.726 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.727 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.727 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561619+0000", 2024-09-16T03:37:35.727 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8202999999999998e-05 2024-09-16T03:37:35.727 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.727 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.727 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.727 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.561672+0000", 2024-09-16T03:37:35.727 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3724999999999998e-05 2024-09-16T03:37:35.727 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.727 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.728 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.728 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:64 4.5 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:35.728 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:37:35.728 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.422614779, 2024-09-16T03:37:35.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50484290399999998, 2024-09-16T03:37:35.728 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.728 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.728 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:35.729 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.729 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.729 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.729 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 64 2024-09-16T03:37:35.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.729 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:37:35.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.730 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.730 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.730 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:37:35.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.730 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.730 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.730 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257341+0000", 2024-09-16T03:37:35.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:37:35.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.730 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257343+0000", 2024-09-16T03:37:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3889999999999999e-06 2024-09-16T03:37:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257346+0000", 2024-09-16T03:37:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:37:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.731 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257349+0000", 2024-09-16T03:37:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8559999999999998e-06 2024-09-16T03:37:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357057+0000", 2024-09-16T03:37:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099708121999999996 2024-09-16T03:37:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357156+0000", 2024-09-16T03:37:35.732 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9162999999999999e-05 2024-09-16T03:37:35.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.733 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762060+0000", 2024-09-16T03:37:35.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40490364400000001 2024-09-16T03:37:35.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.733 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762112+0000", 2024-09-16T03:37:35.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1882000000000002e-05 2024-09-16T03:37:35.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.734 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762183+0000", 2024-09-16T03:37:35.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1297000000000005e-05 2024-09-16T03:37:35.734 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.734 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.734 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.735 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.735 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:295 4.b 4:dbb349e8:::smithi15736696-46:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:35.735 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.085151+0000", 2024-09-16T03:37:35.735 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 19.594803849000002, 2024-09-16T03:37:35.735 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.42198747199999997, 2024-09-16T03:37:35.735 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.735 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.735 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.735 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.735 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.735 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.735 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 295 2024-09-16T03:37:35.736 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.736 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.736 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.736 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.736 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085151+0000", 2024-09-16T03:37:35.736 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.736 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.736 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.736 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.736 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085151+0000", 2024-09-16T03:37:35.736 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.736 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085153+0000", 2024-09-16T03:37:35.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.421e-06 2024-09-16T03:37:35.737 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085158+0000", 2024-09-16T03:37:35.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0239999999999997e-06 2024-09-16T03:37:35.737 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.738 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.738 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085166+0000", 2024-09-16T03:37:35.738 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6400000000000003e-06 2024-09-16T03:37:35.738 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.738 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.738 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085172+0000", 2024-09-16T03:37:35.738 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2499999999999997e-06 2024-09-16T03:37:35.738 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.738 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.739 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085293+0000", 2024-09-16T03:37:35.739 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000121853 2024-09-16T03:37:35.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.739 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.739 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.739 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085366+0000", 2024-09-16T03:37:35.740 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2872000000000003e-05 2024-09-16T03:37:35.740 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.740 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.740 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,9", 2024-09-16T03:37:35.741 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.288783+0000", 2024-09-16T03:37:35.741 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20341624799999999 2024-09-16T03:37:35.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.741 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.741 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.741 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.289205+0000", 2024-09-16T03:37:35.741 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042208099999999999 2024-09-16T03:37:35.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.741 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.289914+0000", 2024-09-16T03:37:35.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00070928600000000005 2024-09-16T03:37:35.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.743 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.507063+0000", 2024-09-16T03:37:35.743 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.21714887299999999 2024-09-16T03:37:35.744 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.744 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.744 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.744 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.507095+0000", 2024-09-16T03:37:35.744 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2570000000000002e-05 2024-09-16T03:37:35.744 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.744 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.744 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.744 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.507139+0000", 2024-09-16T03:37:35.745 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3353999999999999e-05 2024-09-16T03:37:35.745 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.745 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.745 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.745 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.745 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.745 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:400 4.2 4:4a9d235d:::smithi15736696-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:35.745 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.205970+0000", 2024-09-16T03:37:35.745 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.473984696, 2024-09-16T03:37:35.745 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40465369000000001, 2024-09-16T03:37:35.745 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.745 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.745 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.746 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.746 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.746 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.746 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 400 2024-09-16T03:37:35.746 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.746 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.746 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.746 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.747 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205970+0000", 2024-09-16T03:37:35.747 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.747 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.747 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.747 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.747 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205970+0000", 2024-09-16T03:37:35.747 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.748 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.748 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.748 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205972+0000", 2024-09-16T03:37:35.748 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.841e-06 2024-09-16T03:37:35.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.748 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205979+0000", 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9090000000000003e-06 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205991+0000", 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.165e-05 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.206013+0000", 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2079e-05 2024-09-16T03:37:35.749 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.750 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.750 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.750 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.306137+0000", 2024-09-16T03:37:35.750 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10012456 2024-09-16T03:37:35.750 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.750 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.750 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.750 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.306223+0000", 2024-09-16T03:37:35.750 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5181999999999999e-05 2024-09-16T03:37:35.750 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.750 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.751 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 8,11", 2024-09-16T03:37:35.751 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309852+0000", 2024-09-16T03:37:35.751 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036292189999999999 2024-09-16T03:37:35.751 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.751 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.751 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.751 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.310473+0000", 2024-09-16T03:37:35.751 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062141500000000005 2024-09-16T03:37:35.751 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.751 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.752 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.752 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.608696+0000", 2024-09-16T03:37:35.752 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29822300400000001 2024-09-16T03:37:35.752 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.752 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.752 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.752 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.610553+0000", 2024-09-16T03:37:35.752 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018567989999999999 2024-09-16T03:37:35.752 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.752 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.752 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.753 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.610587+0000", 2024-09-16T03:37:35.753 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3995999999999998e-05 2024-09-16T03:37:35.753 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.753 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.753 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.753 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.610624+0000", 2024-09-16T03:37:35.753 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7036000000000003e-05 2024-09-16T03:37:35.753 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.754 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.754 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.754 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.754 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.754 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.10.22:22 3.3 3:f84e13a5:::smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:35.754 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:37:35.754 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.370467544, 2024-09-16T03:37:35.754 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50224271099999995, 2024-09-16T03:37:35.754 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.754 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.754 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.754 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.754 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309489+0000", 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4309999999999999e-06 2024-09-16T03:37:35.755 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309496+0000", 2024-09-16T03:37:35.756 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5810000000000002e-06 2024-09-16T03:37:35.756 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309505+0000", 2024-09-16T03:37:35.756 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6889999999999999e-06 2024-09-16T03:37:35.756 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309513+0000", 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3749999999999997e-06 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.406932+0000", 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.097419631000000007 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407009+0000", 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7220999999999998e-05 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.757 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:37:35.758 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407143+0000", 2024-09-16T03:37:35.758 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013323799999999999 2024-09-16T03:37:35.758 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.758 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.758 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.758 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407732+0000", 2024-09-16T03:37:35.758 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058938399999999998 2024-09-16T03:37:35.758 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.758 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.758 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.758 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.408133+0000", 2024-09-16T03:37:35.758 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040090500000000001 2024-09-16T03:37:35.759 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.759 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.759 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.760 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811599+0000", 2024-09-16T03:37:35.760 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40346633399999998 2024-09-16T03:37:35.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.760 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.760 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811682+0000", 2024-09-16T03:37:35.760 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2463e-05 2024-09-16T03:37:35.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.760 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.760 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811730+0000", 2024-09-16T03:37:35.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8458999999999998e-05 2024-09-16T03:37:35.761 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.761 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.761 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.761 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.761 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.761 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:465 4.5 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:35.761 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:37:35.761 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.664250006, 2024-09-16T03:37:35.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60529649299999999, 2024-09-16T03:37:35.761 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.761 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.761 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 465 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:37:35.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015706+0000", 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.091e-06 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015712+0000", 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015724+0000", 2024-09-16T03:37:35.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1408e-05 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015731+0000", 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216049+0000", 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200317895 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216178+0000", 2024-09-16T03:37:35.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012848800000000001 2024-09-16T03:37:35.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:37:35.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620010+0000", 2024-09-16T03:37:35.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40383207199999999 2024-09-16T03:37:35.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620710+0000", 2024-09-16T03:37:35.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069988400000000001 2024-09-16T03:37:35.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620902+0000", 2024-09-16T03:37:35.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019197800000000001 2024-09-16T03:37:35.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620933+0000", 2024-09-16T03:37:35.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1513999999999997e-05 2024-09-16T03:37:35.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620968+0000", 2024-09-16T03:37:35.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4524999999999999e-05 2024-09-16T03:37:35.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.621002+0000", 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3955000000000003e-05 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:10 3.3 3:deac805c:::smithi15736696-44 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.2727550300000008, 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53191748100000003, 2024-09-16T03:37:35.767 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.768 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.768 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.768 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:37:35.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:37:35.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407202+0000", 2024-09-16T03:37:35.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.855e-06 2024-09-16T03:37:35.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407208+0000", 2024-09-16T03:37:35.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9819999999999996e-06 2024-09-16T03:37:35.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.770 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407220+0000", 2024-09-16T03:37:35.770 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2191999999999999e-05 2024-09-16T03:37:35.770 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.770 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.770 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.770 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407236+0000", 2024-09-16T03:37:35.770 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5698000000000001e-05 2024-09-16T03:37:35.770 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.770 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.770 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.770 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538167+0000", 2024-09-16T03:37:35.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.13093111499999999 2024-09-16T03:37:35.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.771 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.771 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538229+0000", 2024-09-16T03:37:35.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2441999999999996e-05 2024-09-16T03:37:35.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.771 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:37:35.771 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538346+0000", 2024-09-16T03:37:35.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011716400000000001 2024-09-16T03:37:35.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.772 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540757+0000", 2024-09-16T03:37:35.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024105089999999999 2024-09-16T03:37:35.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.772 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.772 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.639501+0000", 2024-09-16T03:37:35.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098743545000000002 2024-09-16T03:37:35.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.772 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.772 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939038+0000", 2024-09-16T03:37:35.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29953765700000001 2024-09-16T03:37:35.773 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.773 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.773 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939075+0000", 2024-09-16T03:37:35.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6600999999999997e-05 2024-09-16T03:37:35.773 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.773 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.773 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939118+0000", 2024-09-16T03:37:35.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2721e-05 2024-09-16T03:37:35.773 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.773 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.773 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:826 4.b 4:dbb349e8:::smithi15736696-46:head [write 507578~477917,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.519716+0000", 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.1602392269999999, 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.31863338600000002, 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 826 2024-09-16T03:37:35.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.775 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.775 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.775 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.775 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.519716+0000", 2024-09-16T03:37:35.775 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.775 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.775 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.775 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.775 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.519716+0000", 2024-09-16T03:37:35.775 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.775 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.775 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.775 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.776 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.519718+0000", 2024-09-16T03:37:35.776 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2069999999999998e-06 2024-09-16T03:37:35.776 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.776 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.776 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.776 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.519844+0000", 2024-09-16T03:37:35.776 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012567899999999999 2024-09-16T03:37:35.776 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.776 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.776 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.776 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.519856+0000", 2024-09-16T03:37:35.776 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2649e-05 2024-09-16T03:37:35.776 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.519863+0000", 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6039999999999998e-06 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538663+0000", 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.018799607999999999 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538712+0000", 2024-09-16T03:37:35.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9579000000000001e-05 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,9", 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538814+0000", 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000101813 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.541185+0000", 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023706790000000001 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.543732+0000", 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0025473420000000002 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.838214+0000", 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29448223099999998 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.838314+0000", 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9327999999999995e-05 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.780 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.838349+0000", 2024-09-16T03:37:35.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5667000000000003e-05 2024-09-16T03:37:35.780 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.780 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.780 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.780 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.780 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.780 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:827 4.b 4:dbb349e8:::smithi15736696-46:head [write 1543557~550004] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:35.780 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.525668+0000", 2024-09-16T03:37:35.780 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.1542873199999999, 2024-09-16T03:37:35.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.31401412899999998, 2024-09-16T03:37:35.780 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.780 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 827 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.525668+0000", 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.781 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.525668+0000", 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.525669+0000", 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1489999999999999e-06 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.525806+0000", 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000136802 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.525816+0000", 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0725e-05 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.525822+0000", 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9959999999999999e-06 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.539804+0000", 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013981650999999999 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.539860+0000", 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6131999999999997e-05 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,9", 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.539953+0000", 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2998999999999995e-05 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.541560+0000", 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001606569 2024-09-16T03:37:35.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.785 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.785 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.544223+0000", 2024-09-16T03:37:35.785 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026632719999999999 2024-09-16T03:37:35.785 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.785 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.785 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839587+0000", 2024-09-16T03:37:35.785 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29536388699999999 2024-09-16T03:37:35.785 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.785 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.785 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839657+0000", 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9630999999999998e-05 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839682+0000", 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5316e-05 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:828 4.b 4:dbb349e8:::smithi15736696-46:head [write 2833665~73913] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.529956+0000", 2024-09-16T03:37:35.786 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.1499989240000001, 2024-09-16T03:37:35.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30987494399999999, 2024-09-16T03:37:35.787 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.787 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.787 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.787 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.787 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.787 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.787 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 828 2024-09-16T03:37:35.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.787 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.529956+0000", 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.529956+0000", 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.529957+0000", 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2699999999999999e-06 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.529980+0000", 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2515999999999999e-05 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.529990+0000", 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0356999999999999e-05 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.529996+0000", 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4439999999999999e-06 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540781+0000", 2024-09-16T03:37:35.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.010784782 2024-09-16T03:37:35.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540826+0000", 2024-09-16T03:37:35.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5901e-05 2024-09-16T03:37:35.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,9", 2024-09-16T03:37:35.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540903+0000", 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6629000000000001e-05 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.541745+0000", 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00084188500000000001 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.732006+0000", 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19026100600000001 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839735+0000", 2024-09-16T03:37:35.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.107728796 2024-09-16T03:37:35.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839790+0000", 2024-09-16T03:37:35.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4766999999999999e-05 2024-09-16T03:37:35.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839831+0000", 2024-09-16T03:37:35.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1591000000000002e-05 2024-09-16T03:37:35.792 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:829 4.b 4:dbb349e8:::smithi15736696-46:head [setxattr _header (56) in=7b,truncate 2907578] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.530013+0000", 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.1499423560000004, 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30994171300000001, 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.793 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 829 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.530013+0000", 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.530013+0000", 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.530013+0000", 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8800000000000002e-07 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.530016+0000", 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6599999999999999e-06 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.530021+0000", 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.271e-06 2024-09-16T03:37:35.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.630115+0000", 2024-09-16T03:37:35.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100094003 2024-09-16T03:37:35.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.630214+0000", 2024-09-16T03:37:35.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9085000000000004e-05 2024-09-16T03:37:35.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.630326+0000", 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011141099999999999 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,9", 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.630486+0000", 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016000500000000001 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.633653+0000", 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031677480000000002 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.732280+0000", 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098626948000000006 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839882+0000", 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10760173200000001 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.798 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839909+0000", 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6891000000000001e-05 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839954+0000", 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5371000000000003e-05 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.799 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:984 4.15 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.3078351020000003, 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.97836159599999994, 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 984 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.800 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:37:35.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:37:35.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372122+0000", 2024-09-16T03:37:35.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.934e-06 2024-09-16T03:37:35.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372129+0000", 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8029999999999999e-06 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372141+0000", 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2006e-05 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472234+0000", 2024-09-16T03:37:35.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100092985 2024-09-16T03:37:35.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472318+0000", 2024-09-16T03:37:35.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4257000000000003e-05 2024-09-16T03:37:35.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:37:35.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472326+0000", 2024-09-16T03:37:35.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8960000000000003e-06 2024-09-16T03:37:35.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350293+0000", 2024-09-16T03:37:35.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.87796699899999997 2024-09-16T03:37:35.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350433+0000", 2024-09-16T03:37:35.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013999600000000001 2024-09-16T03:37:35.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350482+0000", 2024-09-16T03:37:35.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8720000000000001e-05 2024-09-16T03:37:35.804 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.805 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.805 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.805 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.805 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1000 4.15 4:aff4cc41:::smithi15736696-11:head [read 0~2873964,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:37:35.805 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:37:35.805 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.0893960170000003, 2024-09-16T03:37:35.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.761707841, 2024-09-16T03:37:35.805 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.805 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.805 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:35.805 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.805 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1000 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590560+0000", 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590562+0000", 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.17e-06 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590568+0000", 2024-09-16T03:37:35.807 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7939999999999997e-06 2024-09-16T03:37:35.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590572+0000", 2024-09-16T03:37:35.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1300000000000001e-06 2024-09-16T03:37:35.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590619+0000", 2024-09-16T03:37:35.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7148999999999999e-05 2024-09-16T03:37:35.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590625+0000", 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5679999999999996e-06 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350858+0000", 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76023295000000002 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350973+0000", 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115023 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352267+0000", 2024-09-16T03:37:35.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012935970000000001 2024-09-16T03:37:35.810 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.810 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.810 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.810 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1001 4.15 4:aff4cc41:::smithi15736696-11:head [read 1254674~1619290,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:37:35.810 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:37:35.810 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.089377432, 2024-09-16T03:37:35.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76275827799999996, 2024-09-16T03:37:35.810 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1001 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.812 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.812 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:37:35.812 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.812 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.812 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.812 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.812 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:37:35.812 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:37:35.812 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.812 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.812 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.812 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590580+0000", 2024-09-16T03:37:35.812 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:37:35.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.813 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590584+0000", 2024-09-16T03:37:35.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8800000000000001e-06 2024-09-16T03:37:35.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.813 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590587+0000", 2024-09-16T03:37:35.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6630000000000002e-06 2024-09-16T03:37:35.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590650+0000", 2024-09-16T03:37:35.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3950999999999998e-05 2024-09-16T03:37:35.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:37:35.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590654+0000", 2024-09-16T03:37:35.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.89e-06 2024-09-16T03:37:35.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352309+0000", 2024-09-16T03:37:35.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76165495699999997 2024-09-16T03:37:35.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352407+0000", 2024-09-16T03:37:35.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7188999999999994e-05 2024-09-16T03:37:35.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.353336+0000", 2024-09-16T03:37:35.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092939400000000005 2024-09-16T03:37:35.815 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.815 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.815 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1002 4.15 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~1527025] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.0893481779999998, 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85963523600000002, 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1002 2024-09-16T03:37:35.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.817 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:37:35.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.817 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:37:35.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.817 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.15e-07 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590609+0000", 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.606e-06 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590615+0000", 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3040000000000001e-06 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590618+0000", 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.309e-06 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590669+0000", 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1048000000000003e-05 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590673+0000", 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.061e-06 2024-09-16T03:37:35.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.449932+0000", 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85925979699999999 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450032+0000", 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9858000000000005e-05 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450242+0000", 2024-09-16T03:37:35.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020993800000000001 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1038 4.12 4:4a9d235d:::smithi15736696-1:head [sparse-read 1073257~474678] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:31.241887+0000", 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.4380679939999998, 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.31208923199999999, 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:35.821 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1038 2024-09-16T03:37:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.241887+0000", 2024-09-16T03:37:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.241887+0000", 2024-09-16T03:37:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.241887+0000", 2024-09-16T03:37:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.15e-07 2024-09-16T03:37:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.241889+0000", 2024-09-16T03:37:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9640000000000002e-06 2024-09-16T03:37:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.241894+0000", 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9960000000000001e-06 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.241899+0000", 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.155e-06 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.351769+0000", 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.109870366 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.351779+0000", 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0069e-05 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.553770+0000", 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.201990593 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.553847+0000", 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7294999999999998e-05 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:35.825 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:35.826 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:35.826 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.553976+0000", 2024-09-16T03:37:35.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000129379 2024-09-16T03:37:35.826 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.826 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.826 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.826 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:35.826 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:35.826 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:35.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:37:35.828 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2250696 2024-09-16T03:37:35.828 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2611183 2024-09-16T03:37:35.829 INFO:tasks.rados.rados.0.smithi157.stdout:734: finishing set_chunk to oid 18 2024-09-16T03:37:35.829 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 45 (ObjNum 74 snap 0 seq_num 74) dirty exists 2024-09-16T03:37:35.829 INFO:tasks.rados.rados.0.smithi157.stdout:739: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-16T03:37:35.829 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:35.825+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:35.831 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2027598 2024-09-16T03:37:35.895 INFO:tasks.rados.rados.0.smithi157.stdout:738: expect (ObjNum 71 snap 0 seq_num 71) 2024-09-16T03:37:35.909 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1878749 2024-09-16T03:37:35.928 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:35.926+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:35.945 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:35.945 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:35.945 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:37:35.946 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:35.951 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 829214 2024-09-16T03:37:35.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:37:36.008 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1416831 2024-09-16T03:37:36.029 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:36.027+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:36.049 INFO:tasks.rados.rados.0.smithi157.stdout:736: finishing write tid 1 to smithi15736696-45 2024-09-16T03:37:36.049 INFO:tasks.rados.rados.0.smithi157.stdout:748: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-09-16T03:37:36.055 INFO:tasks.rados.rados.0.smithi157.stdout:748: writing smithi15736696-34 from 578174 to 1112122 tid 1 2024-09-16T03:37:36.060 INFO:tasks.rados.rados.0.smithi157.stdout:748: writing smithi15736696-34 from 1515641 to 2000658 tid 2 2024-09-16T03:37:36.063 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:36.063 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:36.067 INFO:tasks.rados.rados.0.smithi157.stdout:748: writing smithi15736696-34 from 2736097 to 3238402 tid 3 2024-09-16T03:37:36.067 INFO:tasks.rados.rados.0.smithi157.stdout:734: done (13 left) 2024-09-16T03:37:36.067 INFO:tasks.rados.rados.0.smithi157.stdout:738: done (12 left) 2024-09-16T03:37:36.067 INFO:tasks.rados.rados.0.smithi157.stdout:739: done (11 left) 2024-09-16T03:37:36.068 INFO:tasks.rados.rados.0.smithi157.stdout:740: done (10 left) 2024-09-16T03:37:36.068 INFO:tasks.rados.rados.0.smithi157.stdout:749: write oid 28 current snap is 0 2024-09-16T03:37:36.068 INFO:tasks.rados.rados.0.smithi157.stdout:736: finishing write tid 2 to smithi15736696-45 2024-09-16T03:37:36.068 INFO:tasks.rados.rados.0.smithi157.stdout:741: finishing set_chunk to oid 11 2024-09-16T03:37:36.068 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 11 set_chunk returned error code -95 offset: 352256 length: 2168320 2024-09-16T03:37:36.068 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 0 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-16T03:37:36.068 INFO:tasks.rados.rados.0.smithi157.stdout:736: finishing write tid 3 to smithi15736696-45 2024-09-16T03:37:36.068 INFO:tasks.rados.rados.0.smithi157.stdout:749: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-09-16T03:37:36.072 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:37:36.073 INFO:tasks.rados.rados.0.smithi157.stdout:749: writing smithi15736696-28 from 594981 to 1204178 tid 1 2024-09-16T03:37:36.081 INFO:tasks.rados.rados.0.smithi157.stdout:749: writing smithi15736696-28 from 1974123 to 2719151 tid 2 2024-09-16T03:37:36.085 INFO:tasks.rados.rados.0.smithi157.stdout:749: writing smithi15736696-28 from 3279873 to 3394981 tid 3 2024-09-16T03:37:36.085 INFO:tasks.rados.rados.0.smithi157.stdout:741: done (10 left) 2024-09-16T03:37:36.085 INFO:tasks.rados.rados.0.smithi157.stdout:750: set_chunk oid 32 target oid 2024-09-16T03:37:36.130 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:36.128+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:36.182 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:17 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'7, mlcod=25'5)", 2024-09-16T03:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:52.877437+0000", 2024-09-16T03:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.302621715999997, 2024-09-16T03:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201252123, 2024-09-16T03:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877437+0000", 2024-09-16T03:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877437+0000", 2024-09-16T03:37:36.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877439+0000", 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877446+0000", 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877459+0000", 2024-09-16T03:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3124e-05 2024-09-16T03:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977610+0000", 2024-09-16T03:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100150878 2024-09-16T03:37:36.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977758+0000", 2024-09-16T03:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147644 2024-09-16T03:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977819+0000", 2024-09-16T03:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0798999999999998e-05 2024-09-16T03:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.078644+0000", 2024-09-16T03:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10082574399999999 2024-09-16T03:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.078689+0000", 2024-09-16T03:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5016999999999999e-05 2024-09-16T03:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.189 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:22 1.0 e26/23 1:30306672:devicehealth::main.db.0000000000000000:head v 26'12, mlcod=25'8)", 2024-09-16T03:37:36.190 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:53.483871+0000", 2024-09-16T03:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.696188292999999, 2024-09-16T03:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101867204, 2024-09-16T03:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483871+0000", 2024-09-16T03:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483871+0000", 2024-09-16T03:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483892+0000", 2024-09-16T03:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0962999999999999e-05 2024-09-16T03:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483977+0000", 2024-09-16T03:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4975000000000005e-05 2024-09-16T03:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483986+0000", 2024-09-16T03:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1339999999999993e-06 2024-09-16T03:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.584127+0000", 2024-09-16T03:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10014146 2024-09-16T03:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.584272+0000", 2024-09-16T03:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014464100000000001 2024-09-16T03:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.584327+0000", 2024-09-16T03:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5012000000000001e-05 2024-09-16T03:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.585688+0000", 2024-09-16T03:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013607890000000001 2024-09-16T03:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.585738+0000", 2024-09-16T03:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0229999999999998e-05 2024-09-16T03:37:36.195 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:39 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'21, mlcod=26'19)", 2024-09-16T03:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.584740992, 2024-09-16T03:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30197182500000003, 2024-09-16T03:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.196 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595320+0000", 2024-09-16T03:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.962e-06 2024-09-16T03:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595336+0000", 2024-09-16T03:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5568000000000001e-05 2024-09-16T03:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595349+0000", 2024-09-16T03:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3203e-05 2024-09-16T03:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695424+0000", 2024-09-16T03:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007542899999999 2024-09-16T03:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695495+0000", 2024-09-16T03:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0450999999999993e-05 2024-09-16T03:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695538+0000", 2024-09-16T03:37:36.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3433000000000002e-05 2024-09-16T03:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897256+0000", 2024-09-16T03:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20171840699999999 2024-09-16T03:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897290+0000", 2024-09-16T03:37:36.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3371999999999998e-05 2024-09-16T03:37:36.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.202 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.203 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:40 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'22, mlcod=26'19)", 2024-09-16T03:37:36.203 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695446+0000", 2024-09-16T03:37:36.203 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.484612943000002, 2024-09-16T03:37:36.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20236575600000001, 2024-09-16T03:37:36.204 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.204 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.204 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.204 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695446+0000", 2024-09-16T03:37:36.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695446+0000", 2024-09-16T03:37:36.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695447+0000", 2024-09-16T03:37:36.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:37:36.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695461+0000", 2024-09-16T03:37:36.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3915e-05 2024-09-16T03:37:36.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695470+0000", 2024-09-16T03:37:36.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6959999999999994e-06 2024-09-16T03:37:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695478+0000", 2024-09-16T03:37:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2050000000000002e-06 2024-09-16T03:37:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695933+0000", 2024-09-16T03:37:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00045501500000000002 2024-09-16T03:37:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695958+0000", 2024-09-16T03:37:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4578000000000001e-05 2024-09-16T03:37:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897782+0000", 2024-09-16T03:37:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20182425700000001 2024-09-16T03:37:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897812+0000", 2024-09-16T03:37:36.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9947e-05 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:41 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'23, mlcod=26'19)", 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695515+0000", 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.484544374999999, 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20231417400000001, 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695515+0000", 2024-09-16T03:37:36.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695515+0000", 2024-09-16T03:37:36.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695516+0000", 2024-09-16T03:37:36.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7499999999999998e-07 2024-09-16T03:37:36.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695590+0000", 2024-09-16T03:37:36.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3924999999999994e-05 2024-09-16T03:37:36.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695601+0000", 2024-09-16T03:37:36.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1253999999999999e-05 2024-09-16T03:37:36.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695611+0000", 2024-09-16T03:37:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9059999999999997e-06 2024-09-16T03:37:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696250+0000", 2024-09-16T03:37:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063903099999999997 2024-09-16T03:37:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696263+0000", 2024-09-16T03:37:36.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.276e-05 2024-09-16T03:37:36.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897815+0000", 2024-09-16T03:37:36.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201552074 2024-09-16T03:37:36.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897829+0000", 2024-09-16T03:37:36.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4249e-05 2024-09-16T03:37:36.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.213 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:42 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'24, mlcod=26'19)", 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.484436467000002, 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.202223289, 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:37:36.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8599999999999997e-07 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695638+0000", 2024-09-16T03:37:36.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4844000000000001e-05 2024-09-16T03:37:36.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695645+0000", 2024-09-16T03:37:36.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8510000000000001e-06 2024-09-16T03:37:36.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695651+0000", 2024-09-16T03:37:36.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5600000000000001e-06 2024-09-16T03:37:36.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696535+0000", 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088443500000000004 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696550+0000", 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4788e-05 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897831+0000", 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201281234 2024-09-16T03:37:36.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897846+0000", 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4691e-05 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:43 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'25, mlcod=26'19)", 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695660+0000", 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.484399097000001, 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.202209369, 2024-09-16T03:37:36.218 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.219 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.219 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.219 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695660+0000", 2024-09-16T03:37:36.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695660+0000", 2024-09-16T03:37:36.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695661+0000", 2024-09-16T03:37:36.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:37:36.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695669+0000", 2024-09-16T03:37:36.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.32e-06 2024-09-16T03:37:36.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695676+0000", 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3610000000000003e-06 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695681+0000", 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0889999999999999e-06 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696770+0000", 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001088852 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696784+0000", 2024-09-16T03:37:36.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4178999999999999e-05 2024-09-16T03:37:36.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897849+0000", 2024-09-16T03:37:36.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20106433700000001 2024-09-16T03:37:36.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897869+0000", 2024-09-16T03:37:36.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0652000000000001e-05 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:44 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'26, mlcod=26'19)", 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.484368430000004, 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20224351600000001, 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:37:36.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:37:36.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:37:36.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1099999999999996e-07 2024-09-16T03:37:36.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695765+0000", 2024-09-16T03:37:36.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3973000000000005e-05 2024-09-16T03:37:36.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695775+0000", 2024-09-16T03:37:36.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5400000000000001e-06 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695781+0000", 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8369999999999999e-06 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698245+0000", 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024633480000000002 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698260+0000", 2024-09-16T03:37:36.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4859e-05 2024-09-16T03:37:36.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897880+0000", 2024-09-16T03:37:36.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199619933 2024-09-16T03:37:36.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897934+0000", 2024-09-16T03:37:36.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4515000000000002e-05 2024-09-16T03:37:36.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.227 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:45 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'27, mlcod=26'19)", 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.698152+0000", 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.481907253999999, 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200415062, 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698152+0000", 2024-09-16T03:37:36.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698152+0000", 2024-09-16T03:37:36.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698153+0000", 2024-09-16T03:37:36.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.525e-06 2024-09-16T03:37:36.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698165+0000", 2024-09-16T03:37:36.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1861e-05 2024-09-16T03:37:36.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698178+0000", 2024-09-16T03:37:36.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3275000000000001e-05 2024-09-16T03:37:36.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698186+0000", 2024-09-16T03:37:36.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4139999999999997e-06 2024-09-16T03:37:36.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897000+0000", 2024-09-16T03:37:36.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.198814149 2024-09-16T03:37:36.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897035+0000", 2024-09-16T03:37:36.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5039000000000001e-05 2024-09-16T03:37:36.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.898524+0000", 2024-09-16T03:37:36.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001488952 2024-09-16T03:37:36.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.898567+0000", 2024-09-16T03:37:36.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2846999999999998e-05 2024-09-16T03:37:36.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.232 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.233 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:46 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'28, mlcod=26'19)", 2024-09-16T03:37:36.233 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.698626+0000", 2024-09-16T03:37:36.233 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.481432986999998, 2024-09-16T03:37:36.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20067976400000001, 2024-09-16T03:37:36.233 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.233 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.233 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.233 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698626+0000", 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698626+0000", 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698627+0000", 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.395e-06 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698641+0000", 2024-09-16T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698653+0000", 2024-09-16T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1846e-05 2024-09-16T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.798792+0000", 2024-09-16T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100139397 2024-09-16T03:37:36.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897344+0000", 2024-09-16T03:37:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098552085999999997 2024-09-16T03:37:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897361+0000", 2024-09-16T03:37:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6716000000000001e-05 2024-09-16T03:37:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.899266+0000", 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019056520000000001 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.899306+0000", 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9322000000000001e-05 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:77 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'55, mlcod=28'53)", 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:57.284321+0000", 2024-09-16T03:37:36.237 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.895737754000002, 2024-09-16T03:37:36.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101501645, 2024-09-16T03:37:36.238 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.238 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.238 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.238 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284321+0000", 2024-09-16T03:37:36.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284321+0000", 2024-09-16T03:37:36.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284326+0000", 2024-09-16T03:37:36.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4020000000000004e-06 2024-09-16T03:37:36.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284408+0000", 2024-09-16T03:37:36.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2547999999999996e-05 2024-09-16T03:37:36.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284435+0000", 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6455e-05 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284449+0000", 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3822000000000001e-05 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284552+0000", 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010370999999999999 2024-09-16T03:37:36.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.284580+0000", 2024-09-16T03:37:36.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7375000000000001e-05 2024-09-16T03:37:36.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.385774+0000", 2024-09-16T03:37:36.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101194703 2024-09-16T03:37:36.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.385823+0000", 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.863e-05 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:79 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'57, mlcod=28'53)", 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:57.288914+0000", 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.891145348000002, 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10187072799999999, 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.242 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.288914+0000", 2024-09-16T03:37:36.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.288914+0000", 2024-09-16T03:37:36.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.288917+0000", 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5049999999999999e-06 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.288998+0000", 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1298000000000007e-05 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.289019+0000", 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0234999999999999e-05 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.389117+0000", 2024-09-16T03:37:36.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10009857 2024-09-16T03:37:36.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.389340+0000", 2024-09-16T03:37:36.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022286100000000001 2024-09-16T03:37:36.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.389372+0000", 2024-09-16T03:37:36.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2057000000000002e-05 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.390733+0000", 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013608769999999999 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.390784+0000", 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1325000000000001e-05 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.247 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:94 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'68, mlcod=28'61)", 2024-09-16T03:37:36.247 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:58.787786+0000", 2024-09-16T03:37:36.247 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.392273089, 2024-09-16T03:37:36.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101551135, 2024-09-16T03:37:36.247 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.247 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.247 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.247 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.787786+0000", 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.787786+0000", 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.787789+0000", 2024-09-16T03:37:36.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8009999999999999e-06 2024-09-16T03:37:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.787804+0000", 2024-09-16T03:37:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5668999999999999e-05 2024-09-16T03:37:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.787820+0000", 2024-09-16T03:37:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5772000000000001e-05 2024-09-16T03:37:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.787832+0000", 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2128e-05 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.887977+0000", 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100145172 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.887996+0000", 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8077e-05 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.889308+0000", 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00131285 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:58.889337+0000", 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8666e-05 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.251 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:99 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'73, mlcod=28'66)", 2024-09-16T03:37:36.252 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:37:36.252 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.088767189000002, 2024-09-16T03:37:36.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201204829, 2024-09-16T03:37:36.252 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.252 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.252 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.252 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:37:36.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:37:36.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:37:36.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9800000000000004e-07 2024-09-16T03:37:36.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091329+0000", 2024-09-16T03:37:36.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7039000000000002e-05 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091336+0000", 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191408+0000", 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100072733 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191470+0000", 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2087000000000004e-05 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191504+0000", 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3453e-05 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292441+0000", 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100937374 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292497+0000", 2024-09-16T03:37:36.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5365999999999998e-05 2024-09-16T03:37:36.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.256 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.256 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:100 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'74, mlcod=28'66)", 2024-09-16T03:37:36.256 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:59.191451+0000", 2024-09-16T03:37:36.256 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.988608483, 2024-09-16T03:37:36.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10146166, 2024-09-16T03:37:36.256 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.256 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.257 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.257 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191451+0000", 2024-09-16T03:37:36.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191451+0000", 2024-09-16T03:37:36.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191452+0000", 2024-09-16T03:37:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6509999999999999e-06 2024-09-16T03:37:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191464+0000", 2024-09-16T03:37:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1901e-05 2024-09-16T03:37:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191473+0000", 2024-09-16T03:37:36.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4360000000000002e-06 2024-09-16T03:37:36.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191483+0000", 2024-09-16T03:37:36.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0570999999999999e-05 2024-09-16T03:37:36.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292103+0000", 2024-09-16T03:37:36.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100620336 2024-09-16T03:37:36.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292136+0000", 2024-09-16T03:37:36.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2809999999999999e-05 2024-09-16T03:37:36.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292884+0000", 2024-09-16T03:37:36.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00074768000000000002 2024-09-16T03:37:36.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292912+0000", 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8274999999999999e-05 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:120 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'90, mlcod=28'85)", 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:00.966325+0000", 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.213733636000001, 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101423443, 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.261 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966325+0000", 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966325+0000", 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966326+0000", 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966402+0000", 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6174999999999995e-05 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966410+0000", 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8350000000000004e-06 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966414+0000", 2024-09-16T03:37:36.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1350000000000002e-06 2024-09-16T03:37:36.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966771+0000", 2024-09-16T03:37:36.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035708499999999999 2024-09-16T03:37:36.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:00.966793+0000", 2024-09-16T03:37:36.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1424e-05 2024-09-16T03:37:36.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:01.067671+0000", 2024-09-16T03:37:36.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10087803400000001 2024-09-16T03:37:36.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:01.067749+0000", 2024-09-16T03:37:36.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8030999999999999e-05 2024-09-16T03:37:36.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.265 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.266 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:9 4.8 e38/32 4:13b03fc9:::smithi15736696-3:head v 38'1, mlcod=0'0)", 2024-09-16T03:37:36.266 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.980745+0000", 2024-09-16T03:37:36.266 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.199313822000001, 2024-09-16T03:37:36.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102593847, 2024-09-16T03:37:36.266 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.266 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.266 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.266 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.980745+0000", 2024-09-16T03:37:36.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.980745+0000", 2024-09-16T03:37:36.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.980746+0000", 2024-09-16T03:37:36.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-09-16T03:37:36.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.981323+0000", 2024-09-16T03:37:36.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057713399999999998 2024-09-16T03:37:36.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.981335+0000", 2024-09-16T03:37:36.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2257e-05 2024-09-16T03:37:36.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.981345+0000", 2024-09-16T03:37:36.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0399e-05 2024-09-16T03:37:36.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.081502+0000", 2024-09-16T03:37:36.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015674400000001 2024-09-16T03:37:36.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.081527+0000", 2024-09-16T03:37:36.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5106e-05 2024-09-16T03:37:36.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.083241+0000", 2024-09-16T03:37:36.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017139410000000001 2024-09-16T03:37:36.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.083339+0000", 2024-09-16T03:37:36.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7874999999999994e-05 2024-09-16T03:37:36.270 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.270 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.270 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.271 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.5.21:12 3.2 e45/31 3:43460c1e:::smithi15736696-20 low_tier:head v 45'36, mlcod=45'34)", 2024-09-16T03:37:36.271 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.014759+0000", 2024-09-16T03:37:36.271 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.165299956, 2024-09-16T03:37:36.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20057897499999999, 2024-09-16T03:37:36.271 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.271 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.271 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.271 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014759+0000", 2024-09-16T03:37:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014759+0000", 2024-09-16T03:37:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014762+0000", 2024-09-16T03:37:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7429999999999998e-06 2024-09-16T03:37:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014774+0000", 2024-09-16T03:37:36.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1790999999999999e-05 2024-09-16T03:37:36.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014790+0000", 2024-09-16T03:37:36.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6409000000000002e-05 2024-09-16T03:37:36.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.114865+0000", 2024-09-16T03:37:36.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100074861 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.114929+0000", 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3769000000000006e-05 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.114947+0000", 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8085999999999999e-05 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.215300+0000", 2024-09-16T03:37:36.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100353422 2024-09-16T03:37:36.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.215338+0000", 2024-09-16T03:37:36.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7894e-05 2024-09-16T03:37:36.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.275 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.275 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.5.21:31 3.2 e52/31 3:67ce2f7a:::06be5359d971ff56f22d604c69e0981b0483bce0167608d941f628a505d4af75:head v 52'64, mlcod=50'61)", 2024-09-16T03:37:36.275 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:31.575591+0000", 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6044682999999997, 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10152863099999999, 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:31.575591+0000", 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:31.575591+0000", 2024-09-16T03:37:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:31.575592+0000", 2024-09-16T03:37:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.097e-06 2024-09-16T03:37:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:31.576124+0000", 2024-09-16T03:37:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053184399999999996 2024-09-16T03:37:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:31.576137+0000", 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3446e-05 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:31.576143+0000", 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2149999999999998e-06 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:31.676307+0000", 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100163748 2024-09-16T03:37:36.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:31.676323+0000", 2024-09-16T03:37:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6116e-05 2024-09-16T03:37:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:31.677074+0000", 2024-09-16T03:37:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075068900000000002 2024-09-16T03:37:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:31.677119+0000", 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5476000000000003e-05 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1187 3.2 e54/31 3:4b00b361:::fb2abb5d8594ae081bd7a522ca068bd21ee8449fbf7bef92e8b363b8b3f628bf:head v 54'81, mlcod=53'79)", 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:35.100308+0000", 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.0797509430000001, 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10508909499999999, 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.280 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.100308+0000", 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.100308+0000", 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.100310+0000", 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.48e-06 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.103823+0000", 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003512987 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.103835+0000", 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2763e-05 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.203922+0000", 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100086403 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.203987+0000", 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5300000000000002e-05 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.204005+0000", 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7631000000000001e-05 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.205362+0000", 2024-09-16T03:37:36.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013569470000000001 2024-09-16T03:37:36.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.205397+0000", 2024-09-16T03:37:36.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5583999999999997e-05 2024-09-16T03:37:36.284 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.284 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.284 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.284 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.284 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.285 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:36.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:37:36.287 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:36.284+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:36.295 INFO:tasks.rados.rados.0.smithi157.stdout:750: set_chunk oid 32 offset: 724480 length: 778240 target oid 28f3dbd02a2494b004e7cef3f0db24ce043047807c298214f7b10042c9638a72 offset: 0 snap -1 2024-09-16T03:37:36.295 INFO:tasks.rados.rados.0.smithi157.stdout:751: read oid 35 snap -1 2024-09-16T03:37:36.296 INFO:tasks.rados.rados.0.smithi157.stdout:751: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:37:36.296 INFO:tasks.rados.rados.0.smithi157.stdout:752: tier_promote oid 33 2024-09-16T03:37:36.296 INFO:tasks.rados.rados.0.smithi157.stdout:736: finishing write tid 4 to smithi15736696-45 2024-09-16T03:37:36.296 INFO:tasks.rados.rados.0.smithi157.stdout:737: finishing write tid 1 to smithi15736696-37 2024-09-16T03:37:36.296 INFO:tasks.rados.rados.0.smithi157.stdout:753: write (excl) oid 3 current snap is 0 2024-09-16T03:37:36.296 INFO:tasks.rados.rados.0.smithi157.stdout:736: finishing write tid 5 to smithi15736696-45 2024-09-16T03:37:36.296 INFO:tasks.rados.rados.0.smithi157.stdout:736: oid 45 updating version 0 to 40 2024-09-16T03:37:36.296 INFO:tasks.rados.rados.0.smithi157.stdout:736: oid 45 version 40 is already newer than 38 2024-09-16T03:37:36.296 INFO:tasks.rados.rados.0.smithi157.stdout:736: oid 45 updating version 40 to 41 2024-09-16T03:37:36.296 INFO:tasks.rados.rados.0.smithi157.stdout:736: oid 45 version 41 is already newer than 39 2024-09-16T03:37:36.296 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 41 (ObjNum 80 snap 0 seq_num 80) dirty exists 2024-09-16T03:37:36.296 INFO:tasks.rados.rados.0.smithi157.stdout:736: left oid 45 (ObjNum 80 snap 0 seq_num 80) 2024-09-16T03:37:36.296 INFO:tasks.rados.rados.0.smithi157.stdout:737: finishing write tid 2 to smithi15736696-37 2024-09-16T03:37:36.297 INFO:tasks.rados.rados.0.smithi157.stdout:753: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-09-16T03:37:36.300 INFO:tasks.rados.rados.0.smithi157.stdout:753: writing smithi15736696-3 from 611788 to 1012587 tid 1 2024-09-16T03:37:36.306 INFO:tasks.rados.rados.0.smithi157.stdout:753: writing smithi15736696-3 from 1748958 to 2237644 tid 2 2024-09-16T03:37:36.313 INFO:tasks.rados.rados.0.smithi157.stdout:753: writing smithi15736696-3 from 2907296 to 3411788 tid 3 2024-09-16T03:37:36.313 INFO:tasks.rados.rados.0.smithi157.stdout:736: done (13 left) 2024-09-16T03:37:36.313 INFO:tasks.rados.rados.0.smithi157.stdout:754: set_chunk oid 7 target oid 2024-09-16T03:37:36.313 INFO:tasks.rados.rados.0.smithi157.stdout:754: set_chunk oid 7 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:37:36.313 INFO:tasks.rados.rados.0.smithi157.stdout:755: write oid 45 current snap is 0 2024-09-16T03:37:36.313 INFO:tasks.rados.rados.0.smithi157.stdout:755: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-09-16T03:37:36.318 INFO:tasks.rados.rados.0.smithi157.stdout:755: writing smithi15736696-45 from 628595 to 1104643 tid 1 2024-09-16T03:37:36.325 INFO:tasks.rados.rados.0.smithi157.stdout:755: writing smithi15736696-45 from 1807440 to 2439784 tid 2 2024-09-16T03:37:36.331 INFO:tasks.rados.rados.0.smithi157.stdout:755: writing smithi15736696-45 from 2934719 to 3387193 tid 3 2024-09-16T03:37:36.331 INFO:tasks.rados.rados.0.smithi157.stdout:756: write (excl) oid 17 current snap is 0 2024-09-16T03:37:36.331 INFO:tasks.rados.rados.0.smithi157.stdout:756: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-09-16T03:37:36.337 INFO:tasks.rados.rados.0.smithi157.stdout:756: writing smithi15736696-17 from 645402 to 1196699 tid 1 2024-09-16T03:37:36.342 INFO:tasks.rados.rados.0.smithi157.stdout:756: writing smithi15736696-17 from 1749569 to 2241924 tid 2 2024-09-16T03:37:36.349 INFO:tasks.rados.rados.0.smithi157.stdout:756: writing smithi15736696-17 from 2845789 to 3445402 tid 3 2024-09-16T03:37:36.349 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:36.349 INFO:tasks.rados.rados.0.smithi157.stdout:744: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-16T03:37:36.349 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:36.360 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:37:36.387 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:36.385+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:36.430 INFO:tasks.rados.rados.0.smithi157.stdout:737: finishing write tid 3 to smithi15736696-37 2024-09-16T03:37:36.431 INFO:tasks.rados.rados.0.smithi157.stdout:742: got (0) Success 2024-09-16T03:37:36.431 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T03:37:36.431 INFO:tasks.rados.rados.0.smithi157.stdout:737: finishing write tid 4 to smithi15736696-37 2024-09-16T03:37:36.431 INFO:tasks.rados.rados.0.smithi157.stdout:742: done (15 left) 2024-09-16T03:37:36.431 INFO:tasks.rados.rados.0.smithi157.stdout:743: done (14 left) 2024-09-16T03:37:36.431 INFO:tasks.rados.rados.0.smithi157.stdout:744: done (13 left) 2024-09-16T03:37:36.431 INFO:tasks.rados.rados.0.smithi157.stdout:757: set_chunk oid 36 target oid 2024-09-16T03:37:36.436 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:36.436 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:36.437 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:37:36.437 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:36.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:37:36.490 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:36.489+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:36.555 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:36.555 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:36.555 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:36.555 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:36.556 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:36.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:37:36.606 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:36.604+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:36.677 INFO:tasks.rados.rados.0.smithi157.stdout:757: set_chunk oid 36 offset: 1993216 length: 403456 target oid 7f01c73e0f253c35d7776d11c46f7e5d9edb0e9d6a760d7305f052eba477b6c2 offset: 0 snap -1 2024-09-16T03:37:36.677 INFO:tasks.rados.rados.0.smithi157.stdout:758: read oid 29 snap -1 2024-09-16T03:37:36.677 INFO:tasks.rados.rados.0.smithi157.stdout:758: expect (ObjNum 75 snap 0 seq_num 75) 2024-09-16T03:37:36.677 INFO:tasks.rados.rados.0.smithi157.stdout:759: delete oid 39 current snap is 0 2024-09-16T03:37:36.677 INFO:tasks.rados.rados.0.smithi157.stdout:737: finishing write tid 5 to smithi15736696-37 2024-09-16T03:37:36.677 INFO:tasks.rados.rados.0.smithi157.stdout:737: oid 37 updating version 0 to 24 2024-09-16T03:37:36.678 INFO:tasks.rados.rados.0.smithi157.stdout:737: oid 37 updating version 24 to 26 2024-09-16T03:37:36.678 INFO:tasks.rados.rados.0.smithi157.stdout:737: oid 37 updating version 26 to 27 2024-09-16T03:37:36.678 INFO:tasks.rados.rados.0.smithi157.stdout:737: oid 37 version 27 is already newer than 25 2024-09-16T03:37:36.678 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 27 (ObjNum 81 snap 0 seq_num 81) dirty exists 2024-09-16T03:37:36.678 INFO:tasks.rados.rados.0.smithi157.stdout:737: left oid 37 (ObjNum 81 snap 0 seq_num 81) 2024-09-16T03:37:36.678 INFO:tasks.rados.rados.0.smithi157.stdout:745: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-16T03:37:36.682 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1003538 2024-09-16T03:37:36.690 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:36.688+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:36.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:33 4.e 4:773b934b:::smithi15736696-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.050039+0000", 2024-09-16T03:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.662539711000001, 2024-09-16T03:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40763112099999999, 2024-09-16T03:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.716 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050039+0000", 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050039+0000", 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050042+0000", 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5009999999999999e-06 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050806+0000", 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076374199999999998 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050819+0000", 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3692999999999999e-05 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050829+0000", 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8630000000000003e-06 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050917+0000", 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8196999999999996e-05 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050922+0000", 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.831e-06 2024-09-16T03:37:36.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153032+0000", 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102109992 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153123+0000", 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1037000000000003e-05 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153218+0000", 2024-09-16T03:37:36.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5047e-05 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156107+0000", 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028893299999999998 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255606+0000", 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099498438999999994 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457591+0000", 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201984996 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457631+0000", 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0110000000000001e-05 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457670+0000", 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9342999999999999e-05 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:34 4.e 4:773b934b:::smithi15736696-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.658167692999999, 2024-09-16T03:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50346527299999999, 2024-09-16T03:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.723 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-16T03:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054413+0000", 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0669999999999999e-06 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054525+0000", 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111462 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054538+0000", 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3341e-05 2024-09-16T03:37:36.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054545+0000", 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.584e-06 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054608+0000", 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2953999999999994e-05 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054613+0000", 2024-09-16T03:37:36.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6110000000000001e-06 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154852+0000", 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10023838 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154915+0000", 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3138000000000004e-05 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155001+0000", 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5730999999999996e-05 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156166+0000", 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001165485 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255909+0000", 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099742558999999995 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557800+0000", 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30189130600000003 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557836+0000", 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6566000000000002e-05 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557877+0000", 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0089000000000002e-05 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.729 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:35 4.e 4:773b934b:::smithi15736696-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.658019362000001, 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50344320499999995, 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.730 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054564+0000", 2024-09-16T03:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5710000000000002e-06 2024-09-16T03:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054569+0000", 2024-09-16T03:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0490000000000003e-06 2024-09-16T03:37:36.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054578+0000", 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054639+0000", 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1061000000000004e-05 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054644+0000", 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999996e-06 2024-09-16T03:37:36.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155607+0000", 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100963372 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155652+0000", 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155722+0000", 2024-09-16T03:37:36.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0130999999999997e-05 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.157921+0000", 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021990159999999998 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255940+0000", 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098018569999999999 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557930+0000", 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.301990333 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557963+0000", 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2348000000000003e-05 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558003+0000", 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:36 4.e 4:773b934b:::smithi15736696-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.657980673000001, 2024-09-16T03:37:36.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50353811800000003, 2024-09-16T03:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.740 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-16T03:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.741 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054599+0000", 2024-09-16T03:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054601+0000", 2024-09-16T03:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4360000000000001e-06 2024-09-16T03:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054606+0000", 2024-09-16T03:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1019999999999996e-06 2024-09-16T03:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054610+0000", 2024-09-16T03:37:36.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.681e-06 2024-09-16T03:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054662+0000", 2024-09-16T03:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1545000000000002e-05 2024-09-16T03:37:36.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054664+0000", 2024-09-16T03:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1919999999999999e-06 2024-09-16T03:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155903+0000", 2024-09-16T03:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101239068 2024-09-16T03:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155970+0000", 2024-09-16T03:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6934999999999995e-05 2024-09-16T03:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558029+0000", 2024-09-16T03:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40205888899999997 2024-09-16T03:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558078+0000", 2024-09-16T03:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9842999999999997e-05 2024-09-16T03:37:36.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558136+0000", 2024-09-16T03:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7985000000000001e-05 2024-09-16T03:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.752 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:280 4.e 4:70297eed:::smithi15736696-31:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.298531+0000", 2024-09-16T03:37:36.753 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.414048267999998, 2024-09-16T03:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30954304300000002, 2024-09-16T03:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.754 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 280 2024-09-16T03:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298531+0000", 2024-09-16T03:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298531+0000", 2024-09-16T03:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298534+0000", 2024-09-16T03:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5769999999999998e-06 2024-09-16T03:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298548+0000", 2024-09-16T03:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.377e-05 2024-09-16T03:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298576+0000", 2024-09-16T03:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8181999999999999e-05 2024-09-16T03:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398649+0000", 2024-09-16T03:37:36.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007253200000001 2024-09-16T03:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.498772+0000", 2024-09-16T03:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100123273 2024-09-16T03:37:36.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.498838+0000", 2024-09-16T03:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5927999999999996e-05 2024-09-16T03:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.605923+0000", 2024-09-16T03:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10708543399999999 2024-09-16T03:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.606636+0000", 2024-09-16T03:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000712836 2024-09-16T03:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.607909+0000", 2024-09-16T03:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00127289 2024-09-16T03:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.608017+0000", 2024-09-16T03:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108004 2024-09-16T03:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.608048+0000", 2024-09-16T03:37:36.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.048e-05 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.608074+0000", 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6137000000000001e-05 2024-09-16T03:37:36.767 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.768 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:293 4.e 4:765e9179:::smithi15736696-44:head [assert-version v9,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.902028+0000", 2024-09-16T03:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.810551165, 2024-09-16T03:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40392601299999997, 2024-09-16T03:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.769 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 293 2024-09-16T03:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.770 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902028+0000", 2024-09-16T03:37:36.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902028+0000", 2024-09-16T03:37:36.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902031+0000", 2024-09-16T03:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9440000000000001e-06 2024-09-16T03:37:36.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902041+0000", 2024-09-16T03:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0477e-05 2024-09-16T03:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902058+0000", 2024-09-16T03:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7227e-05 2024-09-16T03:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902069+0000", 2024-09-16T03:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0165e-05 2024-09-16T03:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902130+0000", 2024-09-16T03:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1043000000000005e-05 2024-09-16T03:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902202+0000", 2024-09-16T03:37:36.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1884000000000005e-05 2024-09-16T03:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.204680+0000", 2024-09-16T03:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30247867099999998 2024-09-16T03:37:36.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.205381+0000", 2024-09-16T03:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000700675 2024-09-16T03:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.305797+0000", 2024-09-16T03:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10041573099999999 2024-09-16T03:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.305865+0000", 2024-09-16T03:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8712000000000002e-05 2024-09-16T03:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.305913+0000", 2024-09-16T03:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7669e-05 2024-09-16T03:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.305954+0000", 2024-09-16T03:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0815000000000001e-05 2024-09-16T03:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:308 4.e 4:773b934b:::smithi15736696-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:36.784 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.721774+0000", 2024-09-16T03:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.990804846, 2024-09-16T03:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30397478300000003, 2024-09-16T03:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.785 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 308 2024-09-16T03:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.721774+0000", 2024-09-16T03:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.721774+0000", 2024-09-16T03:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.721777+0000", 2024-09-16T03:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4159999999999998e-06 2024-09-16T03:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.721787+0000", 2024-09-16T03:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0162e-05 2024-09-16T03:37:36.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.721802+0000", 2024-09-16T03:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4810000000000001e-05 2024-09-16T03:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.721810+0000", 2024-09-16T03:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4489999999999999e-06 2024-09-16T03:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.822017+0000", 2024-09-16T03:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10020680999999999 2024-09-16T03:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.822182+0000", 2024-09-16T03:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016551000000000001 2024-09-16T03:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.824437+0000", 2024-09-16T03:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00225435 2024-09-16T03:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.824912+0000", 2024-09-16T03:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047550700000000001 2024-09-16T03:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.825108+0000", 2024-09-16T03:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019587399999999999 2024-09-16T03:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:17.025669+0000", 2024-09-16T03:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20056116500000001 2024-09-16T03:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:17.025713+0000", 2024-09-16T03:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4079000000000003e-05 2024-09-16T03:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:17.025749+0000", 2024-09-16T03:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5651000000000001e-05 2024-09-16T03:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:393 4.e 4:765e9179:::smithi15736696-44:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.204958+0000", 2024-09-16T03:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.507621464, 2024-09-16T03:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40651974299999999, 2024-09-16T03:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.793 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 393 2024-09-16T03:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.204958+0000", 2024-09-16T03:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.204958+0000", 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.204960+0000", 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0660000000000002e-06 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.204970+0000", 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0752e-05 2024-09-16T03:37:36.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.204981+0000", 2024-09-16T03:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-16T03:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.204989+0000", 2024-09-16T03:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4549999999999998e-06 2024-09-16T03:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.205292+0000", 2024-09-16T03:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030287400000000001 2024-09-16T03:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.205363+0000", 2024-09-16T03:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1194999999999999e-05 2024-09-16T03:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309936+0000", 2024-09-16T03:37:36.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10457277 2024-09-16T03:37:36.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.310501+0000", 2024-09-16T03:37:36.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056559799999999995 2024-09-16T03:37:36.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.310890+0000", 2024-09-16T03:37:36.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038853899999999997 2024-09-16T03:37:36.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.611406+0000", 2024-09-16T03:37:36.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30051645900000001 2024-09-16T03:37:36.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.611442+0000", 2024-09-16T03:37:36.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6077e-05 2024-09-16T03:37:36.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.611477+0000", 2024-09-16T03:37:36.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5011000000000001e-05 2024-09-16T03:37:36.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.800 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.800 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:480 4.e 4:70297eed:::smithi15736696-31:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:36.800 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.621490+0000", 2024-09-16T03:37:36.800 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.091089148, 2024-09-16T03:37:36.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40240205499999998, 2024-09-16T03:37:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 480 2024-09-16T03:37:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621490+0000", 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621490+0000", 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621493+0000", 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5099999999999999e-06 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621508+0000", 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.489e-05 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621536+0000", 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8232e-05 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621560+0000", 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3757e-05 2024-09-16T03:37:36.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621675+0000", 2024-09-16T03:37:36.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114401 2024-09-16T03:37:36.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621814+0000", 2024-09-16T03:37:36.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013906500000000001 2024-09-16T03:37:36.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822507+0000", 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20069381 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.825430+0000", 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029227799999999998 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.825754+0000", 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032421700000000002 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.023790+0000", 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.198035397 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.023842+0000", 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1958e-05 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.023892+0000", 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0037999999999998e-05 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.806 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:508 4.e 4:773b934b:::smithi15736696-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.822186+0000", 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.890392650000001, 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41573648800000002, 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.807 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 508 2024-09-16T03:37:36.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.808 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822186+0000", 2024-09-16T03:37:36.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822186+0000", 2024-09-16T03:37:36.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822189+0000", 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6010000000000002e-06 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822198+0000", 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0990000000000005e-06 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822213+0000", 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4481000000000001e-05 2024-09-16T03:37:36.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922294+0000", 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10008130799999999 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922365+0000", 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0973e-05 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922457+0000", 2024-09-16T03:37:36.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2393999999999997e-05 2024-09-16T03:37:36.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.025017+0000", 2024-09-16T03:37:36.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102560259 2024-09-16T03:37:36.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.025625+0000", 2024-09-16T03:37:36.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060773999999999995 2024-09-16T03:37:36.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.025985+0000", 2024-09-16T03:37:36.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036012000000000001 2024-09-16T03:37:36.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.237775+0000", 2024-09-16T03:37:36.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.211790115 2024-09-16T03:37:36.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.237859+0000", 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3387000000000006e-05 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.237923+0000", 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4010999999999994e-05 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:797 4.e 4:765e9179:::smithi15736696-44:head [write 457157~535817,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:36.813 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.999730+0000", 2024-09-16T03:37:36.814 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.712849118999999, 2024-09-16T03:37:36.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40739979599999998, 2024-09-16T03:37:36.814 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.814 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.814 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.814 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.814 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.814 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.814 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 797 2024-09-16T03:37:36.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.815 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.999730+0000", 2024-09-16T03:37:36.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.999730+0000", 2024-09-16T03:37:36.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.999732+0000", 2024-09-16T03:37:36.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6479999999999999e-06 2024-09-16T03:37:36.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000380+0000", 2024-09-16T03:37:36.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064818899999999997 2024-09-16T03:37:36.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000392+0000", 2024-09-16T03:37:36.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2757e-05 2024-09-16T03:37:36.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000399+0000", 2024-09-16T03:37:36.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.511e-06 2024-09-16T03:37:36.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000462+0000", 2024-09-16T03:37:36.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2874999999999997e-05 2024-09-16T03:37:36.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000564+0000", 2024-09-16T03:37:36.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010215100000000001 2024-09-16T03:37:36.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000746+0000", 2024-09-16T03:37:36.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018165600000000001 2024-09-16T03:37:36.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.103052+0000", 2024-09-16T03:37:36.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102305858 2024-09-16T03:37:36.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205188+0000", 2024-09-16T03:37:36.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10213634100000001 2024-09-16T03:37:36.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.406992+0000", 2024-09-16T03:37:36.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20180416300000001 2024-09-16T03:37:36.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.407098+0000", 2024-09-16T03:37:36.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010646599999999999 2024-09-16T03:37:36.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.407130+0000", 2024-09-16T03:37:36.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1180999999999998e-05 2024-09-16T03:37:36.826 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.826 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.827 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:798 4.e 4:765e9179:::smithi15736696-44:head [write 1768111~402677] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:36.827 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006162+0000", 2024-09-16T03:37:36.827 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.706417197, 2024-09-16T03:37:36.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40110245500000002, 2024-09-16T03:37:36.828 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.828 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.828 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.828 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.828 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.828 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.829 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 798 2024-09-16T03:37:36.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.829 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006162+0000", 2024-09-16T03:37:36.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006162+0000", 2024-09-16T03:37:36.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006163+0000", 2024-09-16T03:37:36.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4789999999999999e-06 2024-09-16T03:37:36.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006421+0000", 2024-09-16T03:37:36.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025733100000000002 2024-09-16T03:37:36.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006434+0000", 2024-09-16T03:37:36.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2982e-05 2024-09-16T03:37:36.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006440+0000", 2024-09-16T03:37:36.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0839999999999997e-06 2024-09-16T03:37:36.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.103103+0000", 2024-09-16T03:37:36.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096663157999999999 2024-09-16T03:37:36.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.103173+0000", 2024-09-16T03:37:36.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9968999999999994e-05 2024-09-16T03:37:36.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.103271+0000", 2024-09-16T03:37:36.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8428e-05 2024-09-16T03:37:36.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205344+0000", 2024-09-16T03:37:36.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102072367 2024-09-16T03:37:36.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205357+0000", 2024-09-16T03:37:36.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3351e-05 2024-09-16T03:37:36.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.407194+0000", 2024-09-16T03:37:36.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201836714 2024-09-16T03:37:36.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.407231+0000", 2024-09-16T03:37:36.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6931999999999998e-05 2024-09-16T03:37:36.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.407264+0000", 2024-09-16T03:37:36.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.366e-05 2024-09-16T03:37:36.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.842 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.842 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:799 4.e 4:765e9179:::smithi15736696-44:head [setxattr _header (56) in=7b,truncate 2857157] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:36.843 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:37:36.843 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.706128445999999, 2024-09-16T03:37:36.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50104159800000003, 2024-09-16T03:37:36.843 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.843 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.843 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.843 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.844 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.844 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.844 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 799 2024-09-16T03:37:36.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.844 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:37:36.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:37:36.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:37:36.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:37:36.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006454+0000", 2024-09-16T03:37:36.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5739999999999998e-06 2024-09-16T03:37:36.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006458+0000", 2024-09-16T03:37:36.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6179999999999997e-06 2024-09-16T03:37:36.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006462+0000", 2024-09-16T03:37:36.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4999999999999999e-06 2024-09-16T03:37:36.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204504+0000", 2024-09-16T03:37:36.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19804184999999999 2024-09-16T03:37:36.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204702+0000", 2024-09-16T03:37:36.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019868199999999999 2024-09-16T03:37:36.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204888+0000", 2024-09-16T03:37:36.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185701 2024-09-16T03:37:36.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.206549+0000", 2024-09-16T03:37:36.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016612790000000001 2024-09-16T03:37:36.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507371+0000", 2024-09-16T03:37:36.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30082157399999998 2024-09-16T03:37:36.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507414+0000", 2024-09-16T03:37:36.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3053999999999998e-05 2024-09-16T03:37:36.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507451+0000", 2024-09-16T03:37:36.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6847999999999997e-05 2024-09-16T03:37:36.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507492+0000", 2024-09-16T03:37:36.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1267999999999999e-05 2024-09-16T03:37:36.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.856 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.857 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:800 4.e 4:765e9179:::smithi15736696-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:36.857 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:37:36.857 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.706105110999999, 2024-09-16T03:37:36.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50119258099999997, 2024-09-16T03:37:36.858 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.858 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.858 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:36.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.859 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.859 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 800 2024-09-16T03:37:36.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.859 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:37:36.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:37:36.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:37:36.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1899999999999998e-07 2024-09-16T03:37:36.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006478+0000", 2024-09-16T03:37:36.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:37:36.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006484+0000", 2024-09-16T03:37:36.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6620000000000002e-06 2024-09-16T03:37:36.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006488+0000", 2024-09-16T03:37:36.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8700000000000002e-06 2024-09-16T03:37:36.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205228+0000", 2024-09-16T03:37:36.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19873985899999999 2024-09-16T03:37:36.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:36.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205283+0000", 2024-09-16T03:37:36.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5878999999999998e-05 2024-09-16T03:37:36.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507527+0000", 2024-09-16T03:37:36.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302243924 2024-09-16T03:37:36.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507593+0000", 2024-09-16T03:37:36.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6061000000000003e-05 2024-09-16T03:37:36.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507666+0000", 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3059000000000006e-05 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:843 4.e 4:773b934b:::smithi15736696-9:head [write 524385~553166] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.630592+0000", 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.081987097000001, 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30844299400000003, 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.867 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.867 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.867 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 843 2024-09-16T03:37:36.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.867 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.630592+0000", 2024-09-16T03:37:36.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.630592+0000", 2024-09-16T03:37:36.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.630593+0000", 2024-09-16T03:37:36.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0100000000000003e-07 2024-09-16T03:37:36.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.630993+0000", 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040056299999999999 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631015+0000", 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1614999999999999e-05 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631025+0000", 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6840000000000008e-06 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734154+0000", 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10312974900000001 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734219+0000", 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4425999999999996e-05 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734352+0000", 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013289700000000001 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739572+0000", 2024-09-16T03:37:36.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005220616 2024-09-16T03:37:36.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.839754+0000", 2024-09-16T03:37:36.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10018216100000001 2024-09-16T03:37:36.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.938748+0000", 2024-09-16T03:37:36.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098993780000000003 2024-09-16T03:37:36.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.938981+0000", 2024-09-16T03:37:36.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023294799999999999 2024-09-16T03:37:36.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.939035+0000", 2024-09-16T03:37:36.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3653999999999998e-05 2024-09-16T03:37:36.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.873 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.873 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.873 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:844 4.e 4:773b934b:::smithi15736696-9:head [write 1602039~410015] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:36.873 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.631035+0000", 2024-09-16T03:37:36.873 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.081543558, 2024-09-16T03:37:36.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30812540599999999, 2024-09-16T03:37:36.873 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.873 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.873 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.873 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 844 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631035+0000", 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631035+0000", 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631036+0000", 2024-09-16T03:37:36.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6300000000000004e-07 2024-09-16T03:37:36.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631729+0000", 2024-09-16T03:37:36.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069264200000000004 2024-09-16T03:37:36.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631740+0000", 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0713999999999999e-05 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631745+0000", 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7729999999999996e-06 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.736236+0000", 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104490641 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.736332+0000", 2024-09-16T03:37:36.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5878000000000006e-05 2024-09-16T03:37:36.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.736536+0000", 2024-09-16T03:37:36.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020407800000000001 2024-09-16T03:37:36.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739577+0000", 2024-09-16T03:37:36.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030408219999999999 2024-09-16T03:37:36.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.839797+0000", 2024-09-16T03:37:36.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100220029 2024-09-16T03:37:36.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.939096+0000", 2024-09-16T03:37:36.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099299540000000006 2024-09-16T03:37:36.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.939131+0000", 2024-09-16T03:37:36.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4588999999999997e-05 2024-09-16T03:37:36.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.939161+0000", 2024-09-16T03:37:36.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9836999999999999e-05 2024-09-16T03:37:36.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.879 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.880 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:845 4.e 4:773b934b:::smithi15736696-9:head [write 2577441~346944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:36.880 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.631766+0000", 2024-09-16T03:37:36.880 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.080813168000001, 2024-09-16T03:37:36.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40847320799999998, 2024-09-16T03:37:36.880 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.880 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.880 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.880 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.880 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 845 2024-09-16T03:37:36.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.880 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631766+0000", 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631766+0000", 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631767+0000", 2024-09-16T03:37:36.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2699999999999999e-07 2024-09-16T03:37:36.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733448+0000", 2024-09-16T03:37:36.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101681228 2024-09-16T03:37:36.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733458+0000", 2024-09-16T03:37:36.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0573e-05 2024-09-16T03:37:36.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733473+0000", 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4953999999999999e-05 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.737776+0000", 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043023940000000002 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.737868+0000", 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2223000000000001e-05 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.738066+0000", 2024-09-16T03:37:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000197951 2024-09-16T03:37:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.740101+0000", 2024-09-16T03:37:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020355379999999999 2024-09-16T03:37:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.939218+0000", 2024-09-16T03:37:36.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19911701800000001 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040070+0000", 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100851658 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040179+0000", 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109085 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040239+0000", 2024-09-16T03:37:36.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9858999999999997e-05 2024-09-16T03:37:36.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.886 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.886 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:846 4.e 4:773b934b:::smithi15736696-9:head [setxattr _header (55) in=7b,truncate 2924385] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:36.886 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.733487+0000", 2024-09-16T03:37:36.886 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9790918469999994, 2024-09-16T03:37:36.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.306915575, 2024-09-16T03:37:36.887 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.887 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.887 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.887 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 846 2024-09-16T03:37:36.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.887 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733487+0000", 2024-09-16T03:37:36.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733487+0000", 2024-09-16T03:37:36.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733489+0000", 2024-09-16T03:37:36.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.491e-06 2024-09-16T03:37:36.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733492+0000", 2024-09-16T03:37:36.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3850000000000002e-06 2024-09-16T03:37:36.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733497+0000", 2024-09-16T03:37:36.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2610000000000001e-06 2024-09-16T03:37:36.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733503+0000", 2024-09-16T03:37:36.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0990000000000004e-06 2024-09-16T03:37:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739098+0000", 2024-09-16T03:37:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005594646 2024-09-16T03:37:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739185+0000", 2024-09-16T03:37:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6910000000000006e-05 2024-09-16T03:37:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:36.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739306+0000", 2024-09-16T03:37:36.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121348 2024-09-16T03:37:36.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:36.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.740904+0000", 2024-09-16T03:37:36.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015982170000000001 2024-09-16T03:37:36.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.939432+0000", 2024-09-16T03:37:36.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.198527603 2024-09-16T03:37:36.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:36.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040299+0000", 2024-09-16T03:37:36.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100866576 2024-09-16T03:37:36.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040343+0000", 2024-09-16T03:37:36.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3937999999999998e-05 2024-09-16T03:37:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040403+0000", 2024-09-16T03:37:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0100999999999999e-05 2024-09-16T03:37:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:847 4.e 4:773b934b:::smithi15736696-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:36.893 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.733507+0000", 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9790715500000005, 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30716211100000002, 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 847 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733507+0000", 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733507+0000", 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733508+0000", 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733511+0000", 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.757e-06 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733516+0000", 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0780000000000003e-06 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733519+0000", 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6160000000000002e-06 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739510+0000", 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00599068 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739564+0000", 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3844000000000002e-05 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040449+0000", 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30088547399999999 2024-09-16T03:37:36.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040562+0000", 2024-09-16T03:37:36.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112389 2024-09-16T03:37:36.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040670+0000", 2024-09-16T03:37:36.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010792499999999999 2024-09-16T03:37:36.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.898 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:983 3.1 e50/31 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head v 50'54, mlcod=49'52)", 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:30.066426+0000", 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.6461528239999996, 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30487072599999998, 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:36.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.066426+0000", 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.066426+0000", 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.066426+0000", 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6600000000000003e-07 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.070698+0000", 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042713880000000001 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.070707+0000", 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3279999999999995e-06 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.070715+0000", 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1920000000000005e-06 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.369583+0000", 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29886805599999999 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.369601+0000", 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7176999999999998e-05 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.371260+0000", 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001659352 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:36.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:36.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:36.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.371297+0000", 2024-09-16T03:37:36.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6967e-05 2024-09-16T03:37:36.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.903 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:36.903 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:36.903 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:36.905 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:37:36.906 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:36.903+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:36.907 INFO:tasks.rados.rados.0.smithi157.stdout:746: finishing set_chunk to oid 6 2024-09-16T03:37:36.907 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 6 set_chunk returned error code -95 offset: 114688 length: 1920512 2024-09-16T03:37:36.907 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T03:37:36.907 INFO:tasks.rados.rados.0.smithi157.stdout:747: expect (ObjNum 72 snap 0 seq_num 72) 2024-09-16T03:37:36.907 INFO:tasks.rados.rados.0.smithi157.stdout:748: finishing write tid 1 to smithi15736696-34 2024-09-16T03:37:36.907 INFO:tasks.rados.rados.0.smithi157.stdout:748: finishing write tid 2 to smithi15736696-34 2024-09-16T03:37:36.907 INFO:tasks.rados.rados.0.smithi157.stdout:749: finishing write tid 1 to smithi15736696-28 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:737: done (15 left) 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:745: done (14 left) 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:746: done (13 left) 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:747: done (12 left) 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:759: done (11 left) 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:760: read oid 8 snap -1 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:760: expect (ObjNum 76 snap 0 seq_num 76) 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:761: read oid 38 snap -1 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:761: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:762: read oid 42 snap -1 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:762: expect (ObjNum 72 snap 0 seq_num 72) 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:763: read oid 50 snap -1 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:763: expect (ObjNum 70 snap 0 seq_num 70) 2024-09-16T03:37:36.908 INFO:tasks.rados.rados.0.smithi157.stdout:764: read oid 24 snap -1 2024-09-16T03:37:36.909 INFO:tasks.rados.rados.0.smithi157.stdout:764: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:37:36.909 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:36.909 INFO:tasks.rados.rados.0.smithi157.stdout:749: finishing write tid 2 to smithi15736696-28 2024-09-16T03:37:36.909 INFO:tasks.rados.rados.0.smithi157.stdout:749: finishing write tid 3 to smithi15736696-28 2024-09-16T03:37:36.909 INFO:tasks.rados.rados.0.smithi157.stdout:749: finishing write tid 4 to smithi15736696-28 2024-09-16T03:37:36.909 INFO:tasks.rados.rados.0.smithi157.stdout:753: finishing write tid 1 to smithi15736696-3 2024-09-16T03:37:36.909 INFO:tasks.rados.rados.0.smithi157.stdout:754: finishing set_chunk to oid 7 2024-09-16T03:37:36.909 INFO:tasks.rados.rados.0.smithi157.stdout:754: got expected ENOENT (src dne) 2024-09-16T03:37:36.909 INFO:tasks.rados.rados.0.smithi157.stdout:753: finishing write tid 2 to smithi15736696-3 2024-09-16T03:37:36.909 INFO:tasks.rados.rados.0.smithi157.stdout:748: finishing write tid 3 to smithi15736696-34 2024-09-16T03:37:36.910 INFO:tasks.rados.rados.0.smithi157.stdout:750: finishing set_chunk to oid 32 2024-09-16T03:37:36.910 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 32 set_chunk returned error code -95 offset: 724480 length: 778240 2024-09-16T03:37:36.910 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:37:36.910 INFO:tasks.rados.rados.0.smithi157.stdout:749: finishing write tid 5 to smithi15736696-28 2024-09-16T03:37:36.910 INFO:tasks.rados.rados.0.smithi157.stdout:749: oid 28 updating version 0 to 40 2024-09-16T03:37:36.910 INFO:tasks.rados.rados.0.smithi157.stdout:749: oid 28 version 40 is already newer than 39 2024-09-16T03:37:36.911 INFO:tasks.rados.rados.0.smithi157.stdout:749: oid 28 version 40 is already newer than 37 2024-09-16T03:37:36.911 INFO:tasks.rados.rados.0.smithi157.stdout:749: oid 28 version 40 is already newer than 38 2024-09-16T03:37:36.911 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 40 (ObjNum 83 snap 0 seq_num 83) dirty exists 2024-09-16T03:37:36.911 INFO:tasks.rados.rados.0.smithi157.stdout:749: left oid 28 (ObjNum 83 snap 0 seq_num 83) 2024-09-16T03:37:36.911 INFO:tasks.rados.rados.0.smithi157.stdout:753: finishing write tid 3 to smithi15736696-3 2024-09-16T03:37:36.911 INFO:tasks.rados.rados.0.smithi157.stdout:748: finishing write tid 4 to smithi15736696-34 2024-09-16T03:37:36.911 INFO:tasks.rados.rados.0.smithi157.stdout:748: finishing write tid 5 to smithi15736696-34 2024-09-16T03:37:36.912 INFO:tasks.rados.rados.0.smithi157.stdout:748: oid 34 updating version 0 to 43 2024-09-16T03:37:36.912 INFO:tasks.rados.rados.0.smithi157.stdout:748: oid 34 updating version 43 to 45 2024-09-16T03:37:36.912 INFO:tasks.rados.rados.0.smithi157.stdout:748: oid 34 version 45 is already newer than 44 2024-09-16T03:37:36.912 INFO:tasks.rados.rados.0.smithi157.stdout:748: oid 34 version 45 is already newer than 42 2024-09-16T03:37:36.912 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 45 (ObjNum 82 snap 0 seq_num 82) dirty exists 2024-09-16T03:37:36.912 INFO:tasks.rados.rados.0.smithi157.stdout:748: left oid 34 (ObjNum 82 snap 0 seq_num 82) 2024-09-16T03:37:36.913 INFO:tasks.rados.rados.0.smithi157.stdout:751: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:37:36.924 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2239547 2024-09-16T03:37:36.957 INFO:tasks.rados.rados.0.smithi157.stdout:752: got (0) Success 2024-09-16T03:37:36.958 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 19 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-09-16T03:37:36.958 INFO:tasks.rados.rados.0.smithi157.stdout:753: finishing write tid 4 to smithi15736696-3 2024-09-16T03:37:36.958 INFO:tasks.rados.rados.0.smithi157.stdout:748: done (15 left) 2024-09-16T03:37:36.958 INFO:tasks.rados.rados.0.smithi157.stdout:749: done (14 left) 2024-09-16T03:37:36.958 INFO:tasks.rados.rados.0.smithi157.stdout:750: done (13 left) 2024-09-16T03:37:36.958 INFO:tasks.rados.rados.0.smithi157.stdout:751: done (12 left) 2024-09-16T03:37:36.959 INFO:tasks.rados.rados.0.smithi157.stdout:752: done (11 left) 2024-09-16T03:37:36.959 INFO:tasks.rados.rados.0.smithi157.stdout:754: done (10 left) 2024-09-16T03:37:36.959 INFO:tasks.rados.rados.0.smithi157.stdout:765: delete oid 12 current snap is 0 2024-09-16T03:37:36.959 INFO:tasks.rados.rados.0.smithi157.stdout:755: finishing write tid 1 to smithi15736696-45 2024-09-16T03:37:36.959 INFO:tasks.rados.rados.0.smithi157.stdout:756: finishing write tid 1 to smithi15736696-17 2024-09-16T03:37:36.959 INFO:tasks.rados.rados.0.smithi157.stdout:756: finishing write tid 2 to smithi15736696-17 2024-09-16T03:37:36.959 INFO:tasks.rados.rados.0.smithi157.stdout:753: finishing write tid 5 to smithi15736696-3 2024-09-16T03:37:36.960 INFO:tasks.rados.rados.0.smithi157.stdout:753: oid 3 updating version 0 to 32 2024-09-16T03:37:36.960 INFO:tasks.rados.rados.0.smithi157.stdout:753: oid 3 updating version 32 to 34 2024-09-16T03:37:36.960 INFO:tasks.rados.rados.0.smithi157.stdout:753: oid 3 version 34 is already newer than 33 2024-09-16T03:37:36.960 INFO:tasks.rados.rados.0.smithi157.stdout:753: oid 3 version 34 is already newer than 31 2024-09-16T03:37:36.960 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 34 (ObjNum 84 snap 0 seq_num 84) dirty exists 2024-09-16T03:37:36.960 INFO:tasks.rados.rados.0.smithi157.stdout:753: left oid 3 (ObjNum 84 snap 0 seq_num 84) 2024-09-16T03:37:36.960 INFO:tasks.rados.rados.0.smithi157.stdout:756: finishing write tid 3 to smithi15736696-17 2024-09-16T03:37:36.961 INFO:tasks.rados.rados.0.smithi157.stdout:756: finishing write tid 4 to smithi15736696-17 2024-09-16T03:37:36.961 INFO:tasks.rados.rados.0.smithi157.stdout:756: finishing write tid 5 to smithi15736696-17 2024-09-16T03:37:36.961 INFO:tasks.rados.rados.0.smithi157.stdout:756: oid 17 updating version 0 to 49 2024-09-16T03:37:36.961 INFO:tasks.rados.rados.0.smithi157.stdout:756: oid 17 updating version 49 to 50 2024-09-16T03:37:36.961 INFO:tasks.rados.rados.0.smithi157.stdout:756: oid 17 updating version 50 to 52 2024-09-16T03:37:36.961 INFO:tasks.rados.rados.0.smithi157.stdout:756: oid 17 version 52 is already newer than 51 2024-09-16T03:37:36.961 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 52 (ObjNum 86 snap 0 seq_num 86) dirty exists 2024-09-16T03:37:36.962 INFO:tasks.rados.rados.0.smithi157.stdout:756: left oid 17 (ObjNum 86 snap 0 seq_num 86) 2024-09-16T03:37:36.962 INFO:tasks.rados.rados.0.smithi157.stdout:755: finishing write tid 2 to smithi15736696-45 2024-09-16T03:37:36.962 INFO:tasks.rados.rados.0.smithi157.stdout:755: finishing write tid 3 to smithi15736696-45 2024-09-16T03:37:36.962 INFO:tasks.rados.rados.0.smithi157.stdout:755: finishing write tid 4 to smithi15736696-45 2024-09-16T03:37:36.962 INFO:tasks.rados.rados.0.smithi157.stdout:755: finishing write tid 5 to smithi15736696-45 2024-09-16T03:37:36.962 INFO:tasks.rados.rados.0.smithi157.stdout:755: oid 45 updating version 0 to 44 2024-09-16T03:37:36.963 INFO:tasks.rados.rados.0.smithi157.stdout:755: oid 45 updating version 44 to 45 2024-09-16T03:37:36.963 INFO:tasks.rados.rados.0.smithi157.stdout:755: oid 45 version 45 is already newer than 42 2024-09-16T03:37:36.963 INFO:tasks.rados.rados.0.smithi157.stdout:755: oid 45 version 45 is already newer than 43 2024-09-16T03:37:36.963 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 45 (ObjNum 85 snap 0 seq_num 85) dirty exists 2024-09-16T03:37:36.963 INFO:tasks.rados.rados.0.smithi157.stdout:755: left oid 45 (ObjNum 85 snap 0 seq_num 85) 2024-09-16T03:37:36.963 INFO:tasks.rados.rados.0.smithi157.stdout:758: expect (ObjNum 75 snap 0 seq_num 75) 2024-09-16T03:37:37.005 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:37.003+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:37.057 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:37.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1227 4.a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v18,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:37:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:37:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.38051262000000002, 2024-09-16T03:37:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38053478200000002, 2024-09-16T03:37:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:37:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1227 2024-09-16T03:37:37.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:37:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:37:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675972+0000", 2024-09-16T03:37:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7269999999999999e-06 2024-09-16T03:37:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675981+0000", 2024-09-16T03:37:37.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.166e-06 2024-09-16T03:37:37.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675996+0000", 2024-09-16T03:37:37.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4901e-05 2024-09-16T03:37:37.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676005+0000", 2024-09-16T03:37:37.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4579999999999993e-06 2024-09-16T03:37:37.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676081+0000", 2024-09-16T03:37:37.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6157999999999998e-05 2024-09-16T03:37:37.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676188+0000", 2024-09-16T03:37:37.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106871 2024-09-16T03:37:37.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.016594+0000", 2024-09-16T03:37:37.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34040622300000001 2024-09-16T03:37:37.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017089+0000", 2024-09-16T03:37:37.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049523899999999997 2024-09-16T03:37:37.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017779+0000", 2024-09-16T03:37:37.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069032200000000005 2024-09-16T03:37:37.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.066 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.067 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.10.22:62 3.3 e54/31)", 2024-09-16T03:37:37.067 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.874280+0000", 2024-09-16T03:37:37.067 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.18220170899999999, 2024-09-16T03:37:37.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18226661799999999, 2024-09-16T03:37:37.068 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:37.068 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.874280+0000", 2024-09-16T03:37:37.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.874280+0000", 2024-09-16T03:37:37.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.874281+0000", 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4300000000000001e-07 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.874286+0000", 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8339999999999998e-06 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.874294+0000", 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4660000000000002e-06 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.874310+0000", 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6477000000000002e-05 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1247 4.11 e54/52)", 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:37.056297+0000", 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.00018485299999999999, 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000270275, 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.071 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:37.072 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.056297+0000", 2024-09-16T03:37:37.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.056297+0000", 2024-09-16T03:37:37.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.056298+0000", 2024-09-16T03:37:37.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0699999999999998e-07 2024-09-16T03:37:37.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.056306+0000", 2024-09-16T03:37:37.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1680000000000004e-06 2024-09-16T03:37:37.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.056317+0000", 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1555e-05 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.056323+0000", 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0530000000000003e-06 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:37:37.074 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 3 2024-09-16T03:37:37.075 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:37.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:37:37.087 INFO:tasks.rados.rados.0.smithi157.stdout:761: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T03:37:37.106 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:37.104+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:37.149 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1556562 2024-09-16T03:37:37.164 INFO:tasks.rados.rados.0.smithi157.stdout:763: expect (ObjNum 70 snap 0 seq_num 70) 2024-09-16T03:37:37.183 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:37.183 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:37.183 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:37.183 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:37.183 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:37.187 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2121015 2024-09-16T03:37:37.192 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:37:37.194 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:37:37.207 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:37.205+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:37.304 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4338.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:37:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.094389+0000", 2024-09-16T03:37:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.207064070000001, 2024-09-16T03:37:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203832385, 2024-09-16T03:37:37.305 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4338", 2024-09-16T03:37:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3073758685", 2024-09-16T03:37:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5 2024-09-16T03:37:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094389+0000", 2024-09-16T03:37:37.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094389+0000", 2024-09-16T03:37:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094393+0000", 2024-09-16T03:37:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.467e-06 2024-09-16T03:37:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094405+0000", 2024-09-16T03:37:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1793e-05 2024-09-16T03:37:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094425+0000", 2024-09-16T03:37:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0412999999999999e-05 2024-09-16T03:37:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094436+0000", 2024-09-16T03:37:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1178e-05 2024-09-16T03:37:37.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094516+0000", 2024-09-16T03:37:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0109000000000002e-05 2024-09-16T03:37:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.094522+0000", 2024-09-16T03:37:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8950000000000003e-06 2024-09-16T03:37:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.196488+0000", 2024-09-16T03:37:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10196596400000001 2024-09-16T03:37:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.196600+0000", 2024-09-16T03:37:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111703 2024-09-16T03:37:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,10", 2024-09-16T03:37:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.196738+0000", 2024-09-16T03:37:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013858800000000001 2024-09-16T03:37:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.197729+0000", 2024-09-16T03:37:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00099082199999999993 2024-09-16T03:37:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.198192+0000", 2024-09-16T03:37:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046261099999999998 2024-09-16T03:37:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.298103+0000", 2024-09-16T03:37:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099911137999999997 2024-09-16T03:37:37.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.298167+0000", 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4289999999999996e-05 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.298222+0000", 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4413999999999998e-05 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4338.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:37:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.204355747999998, 2024-09-16T03:37:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30502727499999999, 2024-09-16T03:37:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4338", 2024-09-16T03:37:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3073758685", 2024-09-16T03:37:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-16T03:37:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.314 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097101+0000", 2024-09-16T03:37:37.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.915e-06 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097111+0000", 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0589e-05 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097130+0000", 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8958999999999999e-05 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097141+0000", 2024-09-16T03:37:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0616e-05 2024-09-16T03:37:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097227+0000", 2024-09-16T03:37:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6322000000000005e-05 2024-09-16T03:37:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097232+0000", 2024-09-16T03:37:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2059999999999998e-06 2024-09-16T03:37:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200235+0000", 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103002945 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200318+0000", 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2515999999999994e-05 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,10", 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200429+0000", 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011158999999999999 2024-09-16T03:37:37.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201478+0000", 2024-09-16T03:37:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00104893 2024-09-16T03:37:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201942+0000", 2024-09-16T03:37:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046354700000000003 2024-09-16T03:37:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402048+0000", 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20010616000000001 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402089+0000", 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0762e-05 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402125+0000", 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6217999999999998e-05 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.320 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:13 4.a 4:51338165:::smithi15736696-4:head [write 450421~625747,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T03:37:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.991194+0000", 2024-09-16T03:37:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.310259425000002, 2024-09-16T03:37:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20656139100000001, 2024-09-16T03:37:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.321 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 13 2024-09-16T03:37:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991194+0000", 2024-09-16T03:37:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991194+0000", 2024-09-16T03:37:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991197+0000", 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1470000000000001e-06 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991908+0000", 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071085899999999999 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991925+0000", 2024-09-16T03:37:37.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7368e-05 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.991936+0000", 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0891e-05 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.992011+0000", 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4814000000000006e-05 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.992018+0000", 2024-09-16T03:37:37.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2239999999999998e-06 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.994165+0000", 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002147043 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.994259+0000", 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3764999999999994e-05 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.994365+0000", 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105701 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996323+0000", 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001958212 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.000278+0000", 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039546690000000001 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197685+0000", 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19740706 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197721+0000", 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6631000000000002e-05 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197755+0000", 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4007000000000002e-05 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:14 4.a 4:51338165:::smithi15736696-4:head [write 1659093~391328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.996458+0000", 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.304995694999999, 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20147354000000001, 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 14 2024-09-16T03:37:37.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996458+0000", 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996458+0000", 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996476+0000", 2024-09-16T03:37:37.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8681999999999998e-05 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996922+0000", 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044590599999999998 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996937+0000", 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5214e-05 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996948+0000", 2024-09-16T03:37:37.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0825000000000001e-05 2024-09-16T03:37:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.997004+0000", 2024-09-16T03:37:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5943000000000003e-05 2024-09-16T03:37:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.997079+0000", 2024-09-16T03:37:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5018999999999994e-05 2024-09-16T03:37:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.997179+0000", 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9597000000000002e-05 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.998553+0000", 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001373649 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197800+0000", 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199247911 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197871+0000", 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0599000000000006e-05 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197906+0000", 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5006999999999999e-05 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197931+0000", 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5188e-05 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.333 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:15 4.a 4:51338165:::smithi15736696-4:head [setxattr _header (55) in=7b,truncate 2050421] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:37:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.304483815000001, 2024-09-16T03:37:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30116835400000003, 2024-09-16T03:37:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.334 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 15 2024-09-16T03:37:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:37:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:37:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996971+0000", 2024-09-16T03:37:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.066e-06 2024-09-16T03:37:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996984+0000", 2024-09-16T03:37:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3246e-05 2024-09-16T03:37:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996994+0000", 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.713e-06 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097070+0000", 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100076313 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097141+0000", 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1298999999999996e-05 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097205+0000", 2024-09-16T03:37:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3907999999999999e-05 2024-09-16T03:37:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097300+0000", 2024-09-16T03:37:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5209000000000003e-05 2024-09-16T03:37:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197645+0000", 2024-09-16T03:37:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10034422899999999 2024-09-16T03:37:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197837+0000", 2024-09-16T03:37:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000192454 2024-09-16T03:37:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298049+0000", 2024-09-16T03:37:37.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100211868 2024-09-16T03:37:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298082+0000", 2024-09-16T03:37:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3467000000000003e-05 2024-09-16T03:37:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298138+0000", 2024-09-16T03:37:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5581999999999997e-05 2024-09-16T03:37:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:73 4.a 4:532814f2:::smithi15736696-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.990193611999999, 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.362642662, 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73 2024-09-16T03:37:37.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311261+0000", 2024-09-16T03:37:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.562e-06 2024-09-16T03:37:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311925+0000", 2024-09-16T03:37:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066385499999999998 2024-09-16T03:37:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311937+0000", 2024-09-16T03:37:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1881e-05 2024-09-16T03:37:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311943+0000", 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6999999999999996e-06 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311988+0000", 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5358000000000002e-05 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312070+0000", 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1735000000000003e-05 2024-09-16T03:37:37.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312167+0000", 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6685e-05 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.313661+0000", 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001494324 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.414759+0000", 2024-09-16T03:37:37.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10109850300000001 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673821+0000", 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.25906127400000001 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673858+0000", 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7620999999999998e-05 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673902+0000", 2024-09-16T03:37:37.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4163999999999999e-05 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:303 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.889734782000001, 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50956986000000004, 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.347 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 303 2024-09-16T03:37:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:37:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411721+0000", 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0200000000000001e-06 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411729+0000", 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1240000000000005e-06 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411744+0000", 2024-09-16T03:37:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099000000000001e-05 2024-09-16T03:37:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411751+0000", 2024-09-16T03:37:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.029e-06 2024-09-16T03:37:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511892+0000", 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100140879 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511961+0000", 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9379000000000001e-05 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.613647+0000", 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101685994 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.614294+0000", 2024-09-16T03:37:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064692799999999996 2024-09-16T03:37:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.714962+0000", 2024-09-16T03:37:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066765 2024-09-16T03:37:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921157+0000", 2024-09-16T03:37:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206195463 2024-09-16T03:37:37.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921233+0000", 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5446999999999998e-05 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921288+0000", 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5847999999999998e-05 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:404 4.3 e45/32 4:cf22405f:::smithi15736696-5:head v 45'15, mlcod=42'13)", 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.309931+0000", 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.991522195000002, 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30105416800000001, 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309931+0000", 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309931+0000", 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309933+0000", 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0099999999999998e-06 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309943+0000", 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4050000000000006e-06 2024-09-16T03:37:37.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309959+0000", 2024-09-16T03:37:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6365e-05 2024-09-16T03:37:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.410036+0000", 2024-09-16T03:37:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100077477 2024-09-16T03:37:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610252+0000", 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20021509700000001 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610290+0000", 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8068999999999998e-05 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610947+0000", 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065726700000000001 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610985+0000", 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8478e-05 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:503 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.47966023, 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60521621599999997, 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.358 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 503 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:37:37.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821794+0000", 2024-09-16T03:37:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-09-16T03:37:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821798+0000", 2024-09-16T03:37:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2030000000000002e-06 2024-09-16T03:37:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821804+0000", 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821810+0000", 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4249999999999997e-06 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922212+0000", 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040228800000001 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922307+0000", 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5089999999999999e-05 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.126007+0000", 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20369931499999999 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.127605+0000", 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001598213 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.327263+0000", 2024-09-16T03:37:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199657905 2024-09-16T03:37:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426900+0000", 2024-09-16T03:37:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099637217 2024-09-16T03:37:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426963+0000", 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.427009+0000", 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6533000000000003e-05 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:848 4.a 4:503187b3:::smithi15736696-34:head [write 541192~628415] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.553032+0000", 2024-09-16T03:37:37.364 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.748421197999999, 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20855610899999999, 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 848 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.553032+0000", 2024-09-16T03:37:37.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.553032+0000", 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.553042+0000", 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0186999999999999e-05 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.553852+0000", 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000809211 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.553866+0000", 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4404e-05 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.553882+0000", 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5985999999999999e-05 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.654103+0000", 2024-09-16T03:37:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100221269 2024-09-16T03:37:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.654219+0000", 2024-09-16T03:37:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011623700000000001 2024-09-16T03:37:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.654411+0000", 2024-09-16T03:37:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019102999999999999 2024-09-16T03:37:37.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658655+0000", 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042448140000000004 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658665+0000", 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3910000000000004e-06 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.761400+0000", 2024-09-16T03:37:37.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10273526600000001 2024-09-16T03:37:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.761552+0000", 2024-09-16T03:37:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015201400000000001 2024-09-16T03:37:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.761588+0000", 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6300000000000001e-05 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:849 4.a 4:503187b3:::smithi15736696-34:head [write 1660521~553673] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.558487+0000", 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.742966571, 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203229874, 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.371 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 849 2024-09-16T03:37:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.558487+0000", 2024-09-16T03:37:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.558487+0000", 2024-09-16T03:37:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.558488+0000", 2024-09-16T03:37:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.367e-06 2024-09-16T03:37:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.559021+0000", 2024-09-16T03:37:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053312699999999999 2024-09-16T03:37:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.559033+0000", 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2128e-05 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.559039+0000", 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0340000000000002e-06 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.656443+0000", 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097403643999999998 2024-09-16T03:37:37.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.656499+0000", 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6388999999999998e-05 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.656606+0000", 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106708 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759296+0000", 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10268964899999999 2024-09-16T03:37:37.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759350+0000", 2024-09-16T03:37:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3767999999999997e-05 2024-09-16T03:37:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.761620+0000", 2024-09-16T03:37:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022707790000000001 2024-09-16T03:37:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.761694+0000", 2024-09-16T03:37:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3402e-05 2024-09-16T03:37:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.761717+0000", 2024-09-16T03:37:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2878999999999999e-05 2024-09-16T03:37:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:850 4.a 4:503187b3:::smithi15736696-34:head [write 2888511~52681] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:37.377 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.562268+0000", 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.739184924, 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29934466599999998, 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 850 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562268+0000", 2024-09-16T03:37:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562268+0000", 2024-09-16T03:37:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562274+0000", 2024-09-16T03:37:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3460000000000001e-06 2024-09-16T03:37:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562334+0000", 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9993e-05 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562346+0000", 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2048999999999999e-05 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562351+0000", 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6879999999999998e-06 2024-09-16T03:37:37.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.657867+0000", 2024-09-16T03:37:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095516648999999995 2024-09-16T03:37:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.657964+0000", 2024-09-16T03:37:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6646999999999998e-05 2024-09-16T03:37:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658121+0000", 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015688 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759327+0000", 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101206726 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759490+0000", 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016302600000000001 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861466+0000", 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101975808 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861543+0000", 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6316000000000006e-05 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861613+0000", 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0537999999999994e-05 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.383 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:851 4.a 4:503187b3:::smithi15736696-34:head [setxattr _header (56) in=7b,truncate 2941192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.562360+0000", 2024-09-16T03:37:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.739093795, 2024-09-16T03:37:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29963510700000001, 2024-09-16T03:37:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 851 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562360+0000", 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562360+0000", 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562360+0000", 2024-09-16T03:37:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-16T03:37:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562363+0000", 2024-09-16T03:37:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8100000000000002e-06 2024-09-16T03:37:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562367+0000", 2024-09-16T03:37:37.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.873e-06 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562370+0000", 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1820000000000002e-06 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658694+0000", 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096324009000000002 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658785+0000", 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0370000000000004e-05 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658928+0000", 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014321099999999999 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759398+0000", 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10046983900000001 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759493+0000", 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5639999999999999e-05 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861917+0000", 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102423575 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861953+0000", 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6004999999999998e-05 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861995+0000", 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1848000000000003e-05 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:852 4.a 4:503187b3:::smithi15736696-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.562374+0000", 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.739079137000001, 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29976180800000002, 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.390 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 852 2024-09-16T03:37:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562374+0000", 2024-09-16T03:37:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562374+0000", 2024-09-16T03:37:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562375+0000", 2024-09-16T03:37:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:37:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562376+0000", 2024-09-16T03:37:37.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4360000000000001e-06 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562379+0000", 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3610000000000001e-06 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.662476+0000", 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10009655100000001 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759421+0000", 2024-09-16T03:37:37.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096945552000000004 2024-09-16T03:37:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759482+0000", 2024-09-16T03:37:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0058e-05 2024-09-16T03:37:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.862023+0000", 2024-09-16T03:37:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10254131599999999 2024-09-16T03:37:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.862072+0000", 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9104999999999998e-05 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.862136+0000", 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4098999999999997e-05 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.395 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:843 4.e e47/32 4:773b934b:::smithi15736696-9:head v 47'21, mlcod=47'19)", 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.734480+0000", 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.566973476999999, 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.204094623, 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734480+0000", 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734480+0000", 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734483+0000", 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1149999999999998e-06 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734967+0000", 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048381800000000001 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734988+0000", 2024-09-16T03:37:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1101e-05 2024-09-16T03:37:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.835063+0000", 2024-09-16T03:37:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007540400000001 2024-09-16T03:37:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.935278+0000", 2024-09-16T03:37:37.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10021431 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.935326+0000", 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8186000000000002e-05 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.938544+0000", 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00321857 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.938575+0000", 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0119000000000001e-05 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1082 4.a 4:532814f2:::smithi15736696-19:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:32.215762+0000", 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.0856912129999996, 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.202931628, 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1082 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.215762+0000", 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.215762+0000", 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.215764+0000", 2024-09-16T03:37:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2929999999999999e-06 2024-09-16T03:37:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.215772+0000", 2024-09-16T03:37:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1690000000000003e-06 2024-09-16T03:37:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.215787+0000", 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5179e-05 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.215795+0000", 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9969999999999998e-06 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.315930+0000", 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10013505 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.316065+0000", 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013536399999999999 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.316284+0000", 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021895399999999999 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.417359+0000", 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10107530100000001 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.417383+0000", 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3680999999999999e-05 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.418540+0000", 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011570669999999999 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.418663+0000", 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012310199999999999 2024-09-16T03:37:37.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:32.418694+0000", 2024-09-16T03:37:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0471e-05 2024-09-16T03:37:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1193 4.a 4:503187b3:::smithi15736696-34:head [write 1515641~485017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:37:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.059561+0000", 2024-09-16T03:37:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.241892518, 2024-09-16T03:37:37.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20444264700000001, 2024-09-16T03:37:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1193 2024-09-16T03:37:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.059561+0000", 2024-09-16T03:37:37.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.059561+0000", 2024-09-16T03:37:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.059563+0000", 2024-09-16T03:37:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9319999999999999e-06 2024-09-16T03:37:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.060005+0000", 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044178199999999999 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.060013+0000", 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4689999999999997e-06 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.060018+0000", 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2530000000000004e-06 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.160377+0000", 2024-09-16T03:37:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100358237 2024-09-16T03:37:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.160444+0000", 2024-09-16T03:37:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7553999999999997e-05 2024-09-16T03:37:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.160551+0000", 2024-09-16T03:37:37.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107396 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.164463+0000", 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0039115560000000001 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.164810+0000", 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000346578 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.263889+0000", 2024-09-16T03:37:37.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099079162999999998 2024-09-16T03:37:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.263950+0000", 2024-09-16T03:37:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1388000000000002e-05 2024-09-16T03:37:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.264004+0000", 2024-09-16T03:37:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3338999999999999e-05 2024-09-16T03:37:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1194 4.a 4:503187b3:::smithi15736696-34:head [write 2736097~502305] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.066371+0000", 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.2350820979999999, 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30263757600000002, 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1194 2024-09-16T03:37:37.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066371+0000", 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066371+0000", 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066373+0000", 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.216e-06 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066794+0000", 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042050699999999998 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066812+0000", 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8462999999999999e-05 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166903+0000", 2024-09-16T03:37:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100090889 2024-09-16T03:37:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166970+0000", 2024-09-16T03:37:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6148000000000004e-05 2024-09-16T03:37:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.167075+0000", 2024-09-16T03:37:37.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010519200000000001 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.167251+0000", 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176405 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.169014+0000", 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001762516 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.264039+0000", 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095025709 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.368934+0000", 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104894738 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.368975+0000", 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0605999999999997e-05 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.369009+0000", 2024-09-16T03:37:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4187000000000003e-05 2024-09-16T03:37:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1195 4.a 4:503187b3:::smithi15736696-34:head [setxattr _header (56) in=7b,truncate 3378174] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:37:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.166919+0000", 2024-09-16T03:37:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.134534814, 2024-09-16T03:37:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.26348176800000001, 2024-09-16T03:37:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.419 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1195 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166919+0000", 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166919+0000", 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166920+0000", 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.66e-06 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166925+0000", 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0429999999999998e-06 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166932+0000", 2024-09-16T03:37:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1060000000000002e-06 2024-09-16T03:37:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.267067+0000", 2024-09-16T03:37:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100135059 2024-09-16T03:37:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.267184+0000", 2024-09-16T03:37:37.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011629399999999999 2024-09-16T03:37:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.267291+0000", 2024-09-16T03:37:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107719 2024-09-16T03:37:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.267463+0000", 2024-09-16T03:37:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171962 2024-09-16T03:37:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.268123+0000", 2024-09-16T03:37:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065973599999999996 2024-09-16T03:37:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.268655+0000", 2024-09-16T03:37:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053218799999999995 2024-09-16T03:37:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.430240+0000", 2024-09-16T03:37:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16158494900000001 2024-09-16T03:37:37.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.430311+0000", 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0706999999999999e-05 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.430400+0000", 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9345000000000006e-05 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1227 4.a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v18,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.625483966, 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.44160531400000003, 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1227 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:37.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:37:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:37:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675972+0000", 2024-09-16T03:37:37.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7269999999999999e-06 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675981+0000", 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.166e-06 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675996+0000", 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4901e-05 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:37.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676005+0000", 2024-09-16T03:37:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4579999999999993e-06 2024-09-16T03:37:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676081+0000", 2024-09-16T03:37:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6157999999999998e-05 2024-09-16T03:37:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676188+0000", 2024-09-16T03:37:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106871 2024-09-16T03:37:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.016594+0000", 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34040622300000001 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017089+0000", 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049523899999999997 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017779+0000", 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069032200000000005 2024-09-16T03:37:37.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117504+0000", 2024-09-16T03:37:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099724207999999995 2024-09-16T03:37:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117543+0000", 2024-09-16T03:37:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9208999999999997e-05 2024-09-16T03:37:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:37.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117575+0000", 2024-09-16T03:37:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1832000000000003e-05 2024-09-16T03:37:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:37.432 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:37.432 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:37.433 INFO:tasks.rados.rados.0.smithi157.stdout:760: expect (ObjNum 76 snap 0 seq_num 76) 2024-09-16T03:37:37.434 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:37.434 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:37.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:37:37.435 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:37.435 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:37.435 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:37.436 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:37.436 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:37:37.436 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:37.436 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:37.437 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:37.438 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:37.438 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:37.432+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:37.456 INFO:tasks.rados.rados.0.smithi157.stdout:762: expect (ObjNum 72 snap 0 seq_num 72) 2024-09-16T03:37:37.474 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:37.474 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:37.491 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:37.491 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:37.493 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:37.509 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:37.509 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:37.525 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:37.526 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:37.534 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:37.532+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:37.542 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:37.542 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:37.561 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:37.562 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:37.577 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:37.577 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:37.581 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:37.581 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:37.581 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:37:37.581 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:37.587 INFO:tasks.rados.rados.0.smithi157.stdout:764: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:37:37.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:37:37.593 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:37.594 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:37.608 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 649821 2024-09-16T03:37:37.609 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:37.610 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:37.627 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:37.627 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:37.634 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 792839 2024-09-16T03:37:37.635 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:37.633+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:37.644 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:37.644 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:37.652 INFO:tasks.rados.rados.0.smithi157.stdout:757: finishing set_chunk to oid 36 2024-09-16T03:37:37.652 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 18 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:37:37.653 INFO:tasks.rados.rados.0.smithi157.stdout:753: done (10 left) 2024-09-16T03:37:37.653 INFO:tasks.rados.rados.0.smithi157.stdout:755: done (9 left) 2024-09-16T03:37:37.653 INFO:tasks.rados.rados.0.smithi157.stdout:756: done (8 left) 2024-09-16T03:37:37.653 INFO:tasks.rados.rados.0.smithi157.stdout:757: done (7 left) 2024-09-16T03:37:37.653 INFO:tasks.rados.rados.0.smithi157.stdout:758: done (6 left) 2024-09-16T03:37:37.653 INFO:tasks.rados.rados.0.smithi157.stdout:760: done (5 left) 2024-09-16T03:37:37.653 INFO:tasks.rados.rados.0.smithi157.stdout:761: done (4 left) 2024-09-16T03:37:37.653 INFO:tasks.rados.rados.0.smithi157.stdout:762: done (3 left) 2024-09-16T03:37:37.653 INFO:tasks.rados.rados.0.smithi157.stdout:763: done (2 left) 2024-09-16T03:37:37.653 INFO:tasks.rados.rados.0.smithi157.stdout:764: done (1 left) 2024-09-16T03:37:37.653 INFO:tasks.rados.rados.0.smithi157.stdout:765: done (0 left) 2024-09-16T03:37:37.653 INFO:tasks.rados.rados.0.smithi157.stdout:766: read oid 19 snap -1 2024-09-16T03:37:37.653 INFO:tasks.rados.rados.0.smithi157.stdout:766: expect deleted 2024-09-16T03:37:37.654 INFO:tasks.rados.rados.0.smithi157.stdout:767: write (excl) oid 33 current snap is 0 2024-09-16T03:37:37.654 INFO:tasks.rados.rados.0.smithi157.stdout:767: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-09-16T03:37:37.660 INFO:tasks.rados.rados.0.smithi157.stdout:767: writing smithi15736696-33 from 695823 to 1472867 tid 1 2024-09-16T03:37:37.661 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:37.661 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:37.666 INFO:tasks.rados.rados.0.smithi157.stdout:767: writing smithi15736696-33 from 1925015 to 2564697 tid 2 2024-09-16T03:37:37.671 INFO:tasks.rados.rados.0.smithi157.stdout:767: writing smithi15736696-33 from 3328058 to 3495823 tid 3 2024-09-16T03:37:37.672 INFO:tasks.rados.rados.0.smithi157.stdout:768: read oid 6 snap -1 2024-09-16T03:37:37.672 INFO:tasks.rados.rados.0.smithi157.stdout:768: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T03:37:37.672 INFO:tasks.rados.rados.0.smithi157.stdout:769: write oid 5 current snap is 0 2024-09-16T03:37:37.672 INFO:tasks.rados.rados.0.smithi157.stdout:767: finishing write tid 1 to smithi15736696-33 2024-09-16T03:37:37.672 INFO:tasks.rados.rados.0.smithi157.stdout:769: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2024-09-16T03:37:37.677 INFO:tasks.rados.rados.0.smithi157.stdout:769: writing smithi15736696-5 from 712630 to 1164923 tid 1 2024-09-16T03:37:37.682 INFO:tasks.rados.rados.0.smithi157.stdout:769: writing smithi15736696-5 from 1583497 to 2083190 tid 2 2024-09-16T03:37:37.687 INFO:tasks.rados.rados.0.smithi157.stdout:769: writing smithi15736696-5 from 2555481 to 2980668 tid 3 2024-09-16T03:37:37.688 INFO:tasks.rados.rados.0.smithi157.stdout:770: read oid 23 snap -1 2024-09-16T03:37:37.688 INFO:tasks.rados.rados.0.smithi157.stdout:770: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T03:37:37.688 INFO:tasks.rados.rados.0.smithi157.stdout:771: read oid 22 snap -1 2024-09-16T03:37:37.688 INFO:tasks.rados.rados.0.smithi157.stdout:771: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T03:37:37.688 INFO:tasks.rados.rados.0.smithi157.stdout:772: set_chunk oid 16 target oid 2024-09-16T03:37:37.702 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:37.702 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:37.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:37:37.735 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:37.733+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:37.835 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:37.834+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:37.859 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:37.859 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:37.868 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:37:37.937 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:37.935+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:37.957 INFO:tasks.rados.rados.0.smithi157.stdout:772: set_chunk oid 16 offset: 307712 length: 511488 target oid 8852e8a626c09dbfc4fcc27d9fb2ee37af85f89f132d669d42d96bc361f177e9 offset: 0 snap -1 2024-09-16T03:37:37.957 INFO:tasks.rados.rados.0.smithi157.stdout:773: set_chunk oid 34 target oid 2024-09-16T03:37:37.980 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:37.981 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:37.990 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:37:38.038 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:38.036+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:38.093 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:38.094 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:38.102 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:37:38.138 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:38.137+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:38.239 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:38.237+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:38.255 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:38.255 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:38.264 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:37:38.268 INFO:tasks.rados.rados.0.smithi157.stdout:773: set_chunk oid 34 offset: 1299456 length: 1156096 target oid a1801e4470cc7cf071d1bf5d7defc27cebd47c6e2d66c961774cc9d5895cb734 offset: 0 snap -1 2024-09-16T03:37:38.268 INFO:tasks.rados.rados.0.smithi157.stdout:774: delete oid 37 current snap is 0 2024-09-16T03:37:38.268 INFO:tasks.rados.rados.0.smithi157.stdout:767: finishing write tid 2 to smithi15736696-33 2024-09-16T03:37:38.268 INFO:tasks.rados.rados.0.smithi157.stdout:767: finishing write tid 3 to smithi15736696-33 2024-09-16T03:37:38.268 INFO:tasks.rados.rados.0.smithi157.stdout:768: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T03:37:38.333 INFO:tasks.rados.rados.0.smithi157.stdout:771: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T03:37:38.340 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:38.338+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:38.343 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2230252 2024-09-16T03:37:38.345 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:38.367 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:38.367 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:38.371 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:38.376 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:37:38.398 INFO:tasks.rados.rados.0.smithi157.stdout:770: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T03:37:38.441 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:38.440+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:38.465 INFO:tasks.rados.rados.0.smithi157.stdout:772: finishing set_chunk to oid 16 2024-09-16T03:37:38.465 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 16 set_chunk returned error code -95 offset: 307712 length: 511488 2024-09-16T03:37:38.465 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 0 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T03:37:38.465 INFO:tasks.rados.rados.0.smithi157.stdout:767: finishing write tid 4 to smithi15736696-33 2024-09-16T03:37:38.465 INFO:tasks.rados.rados.0.smithi157.stdout:767: finishing write tid 5 to smithi15736696-33 2024-09-16T03:37:38.465 INFO:tasks.rados.rados.0.smithi157.stdout:767: oid 33 updating version 0 to 24 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:767: oid 33 version 24 is already newer than 23 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:767: oid 33 version 24 is already newer than 22 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:767: oid 33 updating version 24 to 25 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 25 (ObjNum 89 snap 0 seq_num 89) dirty exists 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:767: left oid 33 (ObjNum 89 snap 0 seq_num 89) 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:769: finishing write tid 1 to smithi15736696-5 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:769: finishing write tid 2 to smithi15736696-5 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:769: finishing write tid 3 to smithi15736696-5 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:769: finishing write tid 4 to smithi15736696-5 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:769: finishing write tid 5 to smithi15736696-5 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:769: oid 5 updating version 0 to 22 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:769: oid 5 updating version 22 to 25 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:769: oid 5 version 25 is already newer than 24 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:769: oid 5 version 25 is already newer than 23 2024-09-16T03:37:38.466 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 25 (ObjNum 90 snap 0 seq_num 90) dirty exists 2024-09-16T03:37:38.467 INFO:tasks.rados.rados.0.smithi157.stdout:769: left oid 5 (ObjNum 90 snap 0 seq_num 90) 2024-09-16T03:37:38.470 INFO:tasks.rados.rados.0.smithi157.stdout:766: done (8 left) 2024-09-16T03:37:38.470 INFO:tasks.rados.rados.0.smithi157.stdout:767: done (7 left) 2024-09-16T03:37:38.470 INFO:tasks.rados.rados.0.smithi157.stdout:768: done (6 left) 2024-09-16T03:37:38.470 INFO:tasks.rados.rados.0.smithi157.stdout:769: done (5 left) 2024-09-16T03:37:38.470 INFO:tasks.rados.rados.0.smithi157.stdout:770: done (4 left) 2024-09-16T03:37:38.470 INFO:tasks.rados.rados.0.smithi157.stdout:771: done (3 left) 2024-09-16T03:37:38.470 INFO:tasks.rados.rados.0.smithi157.stdout:772: done (2 left) 2024-09-16T03:37:38.471 INFO:tasks.rados.rados.0.smithi157.stdout:774: done (1 left) 2024-09-16T03:37:38.471 INFO:tasks.rados.rados.0.smithi157.stdout:775: read oid 25 snap -1 2024-09-16T03:37:38.471 INFO:tasks.rados.rados.0.smithi157.stdout:775: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-16T03:37:38.471 INFO:tasks.rados.rados.0.smithi157.stdout:776: write (excl) oid 41 current snap is 0 2024-09-16T03:37:38.471 INFO:tasks.rados.rados.0.smithi157.stdout:776: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-09-16T03:37:38.478 INFO:tasks.rados.rados.0.smithi157.stdout:776: writing smithi15736696-41 from 746244 to 1465388 tid 1 2024-09-16T03:37:38.479 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:38.479 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:38.484 INFO:tasks.rados.rados.0.smithi157.stdout:776: writing smithi15736696-41 from 2100461 to 2603823 tid 2 2024-09-16T03:37:38.488 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:37:38.489 INFO:tasks.rados.rados.0.smithi157.stdout:776: writing smithi15736696-41 from 3126680 to 3546244 tid 3 2024-09-16T03:37:38.489 INFO:tasks.rados.rados.0.smithi157.stdout:777: write oid 19 current snap is 0 2024-09-16T03:37:38.489 INFO:tasks.rados.rados.0.smithi157.stdout:777: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-09-16T03:37:38.497 INFO:tasks.rados.rados.0.smithi157.stdout:777: writing smithi15736696-19 from 763051 to 1557444 tid 1 2024-09-16T03:37:38.504 INFO:tasks.rados.rados.0.smithi157.stdout:777: writing smithi15736696-19 from 2158943 to 2805963 tid 2 2024-09-16T03:37:38.508 INFO:tasks.rados.rados.0.smithi157.stdout:777: writing smithi15736696-19 from 3437750 to 3563051 tid 3 2024-09-16T03:37:38.508 INFO:tasks.rados.rados.0.smithi157.stdout:778: set_chunk oid 38 target oid 2024-09-16T03:37:38.542 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:38.540+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:38.586 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:38.584+0000 7f3a1f71e640 -1 osd.6 55 handle_osd_map injecting map crc failure 2024-09-16T03:37:38.587 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:38.585+0000 7f66f3205640 -1 osd.15 55 handle_osd_map injecting map crc failure 2024-09-16T03:37:38.592 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:38.592 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:38.602 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:37:38.615 INFO:tasks.rados.rados.0.smithi157.stdout:778: set_chunk oid 38 offset: 1709056 length: 194048 target oid c10ecd4d51c9643148216a3c02f32da5a1e67252ff586b2c7a313ee5cbfdc431 offset: 0 snap -1 2024-09-16T03:37:38.615 INFO:tasks.rados.rados.0.smithi157.stdout:779: read oid 31 snap -1 2024-09-16T03:37:38.615 INFO:tasks.rados.rados.0.smithi157.stdout:779: expect deleted 2024-09-16T03:37:38.615 INFO:tasks.rados.rados.0.smithi157.stdout:780: delete oid 46 current snap is 0 2024-09-16T03:37:38.615 INFO:tasks.rados.rados.0.smithi157.stdout:775: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-16T03:37:38.615 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:38.641 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:38.650 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:38.648+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:38.694 INFO:tasks.rados.rados.0.smithi157.stdout:778: finishing set_chunk to oid 38 2024-09-16T03:37:38.694 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 38 set_chunk returned error code -95 offset: 1709056 length: 194048 2024-09-16T03:37:38.694 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 0 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T03:37:38.694 INFO:tasks.rados.rados.0.smithi157.stdout:776: finishing write tid 1 to smithi15736696-41 2024-09-16T03:37:38.707 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:38.707 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:38.716 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:37:38.744 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:38.742+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:38.788 INFO:tasks.rados.rados.0.smithi157.stdout:776: finishing write tid 2 to smithi15736696-41 2024-09-16T03:37:38.788 INFO:tasks.rados.rados.0.smithi157.stdout:776: finishing write tid 3 to smithi15736696-41 2024-09-16T03:37:38.788 INFO:tasks.rados.rados.0.smithi157.stdout:777: finishing write tid 1 to smithi15736696-19 2024-09-16T03:37:38.789 INFO:tasks.rados.rados.0.smithi157.stdout:776: finishing write tid 4 to smithi15736696-41 2024-09-16T03:37:38.790 INFO:tasks.rados.rados.0.smithi157.stdout:775: done (6 left) 2024-09-16T03:37:38.790 INFO:tasks.rados.rados.0.smithi157.stdout:778: done (5 left) 2024-09-16T03:37:38.790 INFO:tasks.rados.rados.0.smithi157.stdout:779: done (4 left) 2024-09-16T03:37:38.790 INFO:tasks.rados.rados.0.smithi157.stdout:780: done (3 left) 2024-09-16T03:37:38.790 INFO:tasks.rados.rados.0.smithi157.stdout:781: tier_promote oid 5 2024-09-16T03:37:38.790 INFO:tasks.rados.rados.0.smithi157.stdout:782: write (excl) oid 38 current snap is 0 2024-09-16T03:37:38.790 INFO:tasks.rados.rados.0.smithi157.stdout:782: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2024-09-16T03:37:38.797 INFO:tasks.rados.rados.0.smithi157.stdout:782: writing smithi15736696-38 from 796665 to 1341556 tid 1 2024-09-16T03:37:38.803 INFO:tasks.rados.rados.0.smithi157.stdout:782: writing smithi15736696-38 from 1759554 to 2410243 tid 2 2024-09-16T03:37:38.809 INFO:tasks.rados.rados.0.smithi157.stdout:782: writing smithi15736696-38 from 2976243 to 3596665 tid 3 2024-09-16T03:37:38.809 INFO:tasks.rados.rados.0.smithi157.stdout:783: read oid 43 snap -1 2024-09-16T03:37:38.809 INFO:tasks.rados.rados.0.smithi157.stdout:783: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T03:37:38.810 INFO:tasks.rados.rados.0.smithi157.stdout:784: read oid 7 snap -1 2024-09-16T03:37:38.810 INFO:tasks.rados.rados.0.smithi157.stdout:784: expect deleted 2024-09-16T03:37:38.810 INFO:tasks.rados.rados.0.smithi157.stdout:785: write oid 21 current snap is 0 2024-09-16T03:37:38.810 INFO:tasks.rados.rados.0.smithi157.stdout:785: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-09-16T03:37:38.814 INFO:tasks.rados.rados.0.smithi157.stdout:785: writing smithi15736696-21 from 413472 to 1033612 tid 1 2024-09-16T03:37:38.821 INFO:tasks.rados.rados.0.smithi157.stdout:785: writing smithi15736696-21 from 1818036 to 2328736 tid 2 2024-09-16T03:37:38.829 INFO:tasks.rados.rados.0.smithi157.stdout:785: writing smithi15736696-21 from 3120019 to 3613472 tid 3 2024-09-16T03:37:38.829 INFO:tasks.rados.rados.0.smithi157.stdout:786: tier_promote oid 27 2024-09-16T03:37:38.829 INFO:tasks.rados.rados.0.smithi157.stdout:787: write oid 46 current snap is 0 2024-09-16T03:37:38.829 INFO:tasks.rados.rados.0.smithi157.stdout:787: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-09-16T03:37:38.835 INFO:tasks.rados.rados.0.smithi157.stdout:787: writing smithi15736696-46 from 430279 to 1125668 tid 1 2024-09-16T03:37:38.842 INFO:tasks.rados.rados.0.smithi157.stdout:787: writing smithi15736696-46 from 1876518 to 2530876 tid 2 2024-09-16T03:37:38.845 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:38.843+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:38.849 INFO:tasks.rados.rados.0.smithi157.stdout:787: writing smithi15736696-46 from 3031089 to 3567886 tid 3 2024-09-16T03:37:38.850 INFO:tasks.rados.rados.0.smithi157.stdout:788: tier_flush oid 35 2024-09-16T03:37:38.850 INFO:tasks.rados.rados.0.smithi157.stdout:788: tier_flush oid 35 snap -1 2024-09-16T03:37:38.850 INFO:tasks.rados.rados.0.smithi157.stdout:789: read oid 3 snap -1 2024-09-16T03:37:38.850 INFO:tasks.rados.rados.0.smithi157.stdout:789: expect (ObjNum 84 snap 0 seq_num 84) 2024-09-16T03:37:38.850 INFO:tasks.rados.rados.0.smithi157.stdout:790: read oid 50 snap -1 2024-09-16T03:37:38.850 INFO:tasks.rados.rados.0.smithi157.stdout:790: expect (ObjNum 70 snap 0 seq_num 70) 2024-09-16T03:37:38.851 INFO:tasks.rados.rados.0.smithi157.stdout:791: write oid 30 current snap is 0 2024-09-16T03:37:38.851 INFO:tasks.rados.rados.0.smithi157.stdout:791: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-09-16T03:37:38.856 INFO:tasks.rados.rados.0.smithi157.stdout:791: writing smithi15736696-30 from 447086 to 1217724 tid 1 2024-09-16T03:37:38.861 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:38.861 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:38.864 INFO:tasks.rados.rados.0.smithi157.stdout:791: writing smithi15736696-30 from 1935000 to 2733016 tid 2 2024-09-16T03:37:38.869 INFO:tasks.rados.rados.0.smithi157.stdout:791: writing smithi15736696-30 from 3458512 to 3647086 tid 3 2024-09-16T03:37:38.869 INFO:tasks.rados.rados.0.smithi157.stdout:792: tier_flush oid 32 2024-09-16T03:37:38.869 INFO:tasks.rados.rados.0.smithi157.stdout:792: tier_flush oid 32 snap -1 2024-09-16T03:37:38.870 INFO:tasks.rados.rados.0.smithi157.stdout:793: write oid 6 current snap is 0 2024-09-16T03:37:38.870 INFO:tasks.rados.rados.0.smithi157.stdout:793: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-09-16T03:37:38.871 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:37:38.874 INFO:tasks.rados.rados.0.smithi157.stdout:793: writing smithi15736696-6 from 463893 to 1026133 tid 1 2024-09-16T03:37:38.881 INFO:tasks.rados.rados.0.smithi157.stdout:793: writing smithi15736696-6 from 1593482 to 2251509 tid 2 2024-09-16T03:37:38.887 INFO:tasks.rados.rados.0.smithi157.stdout:793: writing smithi15736696-6 from 2685935 to 3245276 tid 3 2024-09-16T03:37:38.887 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:38.891 INFO:tasks.rados.rados.0.smithi157.stdout:776: finishing write tid 5 to smithi15736696-41 2024-09-16T03:37:38.891 INFO:tasks.rados.rados.0.smithi157.stdout:776: oid 41 updating version 0 to 27 2024-09-16T03:37:38.891 INFO:tasks.rados.rados.0.smithi157.stdout:776: oid 41 version 27 is already newer than 24 2024-09-16T03:37:38.891 INFO:tasks.rados.rados.0.smithi157.stdout:776: oid 41 version 27 is already newer than 25 2024-09-16T03:37:38.892 INFO:tasks.rados.rados.0.smithi157.stdout:776: oid 41 version 27 is already newer than 26 2024-09-16T03:37:38.892 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 27 (ObjNum 92 snap 0 seq_num 92) dirty exists 2024-09-16T03:37:38.892 INFO:tasks.rados.rados.0.smithi157.stdout:776: left oid 41 (ObjNum 92 snap 0 seq_num 92) 2024-09-16T03:37:38.892 INFO:tasks.rados.rados.0.smithi157.stdout:776: done (15 left) 2024-09-16T03:37:38.892 INFO:tasks.rados.rados.0.smithi157.stdout:794: write (excl) oid 10 current snap is 0 2024-09-16T03:37:38.892 INFO:tasks.rados.rados.0.smithi157.stdout:794: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-09-16T03:37:38.896 INFO:tasks.rados.rados.0.smithi157.stdout:794: writing smithi15736696-10 from 480700 to 1118189 tid 1 2024-09-16T03:37:38.901 INFO:tasks.rados.rados.0.smithi157.stdout:794: writing smithi15736696-10 from 1651964 to 2053649 tid 2 2024-09-16T03:37:38.909 INFO:tasks.rados.rados.0.smithi157.stdout:794: writing smithi15736696-10 from 2713358 to 3483971 tid 3 2024-09-16T03:37:38.909 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:38.911 INFO:tasks.rados.rados.0.smithi157.stdout:783: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-16T03:37:38.946 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:38.945+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:38.978 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:38.979 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:38.990 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:37:39.002 INFO:tasks.rados.rados.0.smithi157.stdout:782: finishing write tid 1 to smithi15736696-38 2024-09-16T03:37:39.002 INFO:tasks.rados.rados.0.smithi157.stdout:790: expect (ObjNum 70 snap 0 seq_num 70) 2024-09-16T03:37:39.047 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:39.045+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:39.120 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:39.137 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:39.138 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:39.146 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:37:39.149 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:39.147+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:39.168 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:37:39.168 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:37:39.168 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:37:39.169 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-16T03:37:39.169 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T03:37:39.169 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T03:37:39.169 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2024-09-16T03:37:39.181 INFO:tasks.rados.rados.0.smithi157.stdout:777: finishing write tid 2 to smithi15736696-19 2024-09-16T03:37:39.181 INFO:tasks.rados.rados.0.smithi157.stdout:777: finishing write tid 3 to smithi15736696-19 2024-09-16T03:37:39.181 INFO:tasks.rados.rados.0.smithi157.stdout:777: finishing write tid 4 to smithi15736696-19 2024-09-16T03:37:39.181 INFO:tasks.rados.rados.0.smithi157.stdout:777: finishing write tid 5 to smithi15736696-19 2024-09-16T03:37:39.181 INFO:tasks.rados.rados.0.smithi157.stdout:777: oid 19 updating version 0 to 47 2024-09-16T03:37:39.181 INFO:tasks.rados.rados.0.smithi157.stdout:777: oid 19 updating version 47 to 48 2024-09-16T03:37:39.181 INFO:tasks.rados.rados.0.smithi157.stdout:777: oid 19 updating version 48 to 49 2024-09-16T03:37:39.181 INFO:tasks.rados.rados.0.smithi157.stdout:777: oid 19 updating version 49 to 50 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 50 (ObjNum 93 snap 0 seq_num 93) dirty exists 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:777: left oid 19 (ObjNum 93 snap 0 seq_num 93) 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:794: finishing write tid 1 to smithi15736696-10 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:777: done (15 left) 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:783: done (14 left) 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:790: done (13 left) 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:795: read oid 19 snap -1 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:795: expect (ObjNum 93 snap 0 seq_num 93) 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:796: write (excl) oid 43 current snap is 0 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:785: finishing write tid 1 to smithi15736696-21 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:773: finishing set_chunk to oid 34 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 45 (ObjNum 82 snap 0 seq_num 82) dirty exists 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:789: expect (ObjNum 84 snap 0 seq_num 84) 2024-09-16T03:37:39.182 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:39.249 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:39.247+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:39.254 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:39.254 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:39.263 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:37:39.300 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:39.300 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T03:37:39.300 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:39.312 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:39.350 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:39.348+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:39.370 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:39.370 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:39.372 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:39.370+0000 7f49419d0640 -1 osd.4 56 handle_osd_map injecting map crc failure 2024-09-16T03:37:39.380 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:37:39.386 INFO:tasks.rados.rados.0.smithi157.stdout:796: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-09-16T03:37:39.396 INFO:tasks.rados.rados.0.smithi157.stdout:796: writing smithi15736696-43 from 497507 to 1210245 tid 1 2024-09-16T03:37:39.399 INFO:tasks.rados.rados.0.smithi157.stdout:796: writing smithi15736696-43 from 1710446 to 2372142 tid 2 2024-09-16T03:37:39.406 INFO:tasks.rados.rados.0.smithi157.stdout:796: writing smithi15736696-43 from 3140781 to 3697507 tid 3 2024-09-16T03:37:39.406 INFO:tasks.rados.rados.0.smithi157.stdout:773: done (14 left) 2024-09-16T03:37:39.407 INFO:tasks.rados.rados.0.smithi157.stdout:789: done (13 left) 2024-09-16T03:37:39.407 INFO:tasks.rados.rados.0.smithi157.stdout:797: set_chunk oid 26 target oid 2024-09-16T03:37:39.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:37:39.450 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:39.448+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:39.488 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:39.488 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:39.497 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:37:39.551 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:39.549+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:39.641 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:39.641 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:39.651 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:37:39.652 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:39.651+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:39.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:37:39.753 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:39.752+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:39.756 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:39.756 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:39.765 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:37:39.800 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:39.800 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:39.817 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:39.818 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:39.834 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:39.835 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:39.851 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:39.851 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:39.854 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:39.853+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:39.867 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:39.868 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:39.869 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:39.869 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:39.878 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:37:39.884 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:39.884 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:39.901 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:39.901 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:39.919 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:39.919 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:39.934 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:39.935 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:39.952 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:39.952 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:39.955 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:39.954+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:39.969 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:39.969 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:39.981 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:39.981 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:39.982 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:39.982 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:39.982 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:39.986 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:39.986 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:39.990 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:37:40.001 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:40.001 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:40.018 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:40.018 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:40.035 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:40.035 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:40.051 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:40.051 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:40.057 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:40.054+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:40.098 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:40.098 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:40.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:40.098 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:40.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.098 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:53 4.7 e39/32 4:ed874e32:::smithi15736696-14:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:40.098 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:37:40.098 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.849858991000001, 2024-09-16T03:37:40.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20229554399999999, 2024-09-16T03:37:40.099 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.099 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.099 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.099 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:37:40.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:37:40.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:37:40.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9600000000000001e-07 2024-09-16T03:37:40.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243671+0000", 2024-09-16T03:37:40.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034761699999999999 2024-09-16T03:37:40.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243681+0000", 2024-09-16T03:37:40.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0505e-05 2024-09-16T03:37:40.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343786+0000", 2024-09-16T03:37:40.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10010472300000001 2024-09-16T03:37:40.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.443948+0000", 2024-09-16T03:37:40.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100161739 2024-09-16T03:37:40.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.443966+0000", 2024-09-16T03:37:40.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8045000000000001e-05 2024-09-16T03:37:40.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445575+0000", 2024-09-16T03:37:40.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001608841 2024-09-16T03:37:40.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445619+0000", 2024-09-16T03:37:40.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3878000000000002e-05 2024-09-16T03:37:40.104 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.104 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.104 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.104 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:54 4.7 e39/32 4:ed874e32:::smithi15736696-14:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:40.104 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.343820+0000", 2024-09-16T03:37:40.105 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.749361822000001, 2024-09-16T03:37:40.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10196166, 2024-09-16T03:37:40.105 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.105 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.105 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.105 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343820+0000", 2024-09-16T03:37:40.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343820+0000", 2024-09-16T03:37:40.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343821+0000", 2024-09-16T03:37:40.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.181e-06 2024-09-16T03:37:40.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343931+0000", 2024-09-16T03:37:40.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000109572 2024-09-16T03:37:40.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343939+0000", 2024-09-16T03:37:40.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3690000000000007e-06 2024-09-16T03:37:40.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343945+0000", 2024-09-16T03:37:40.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2549999999999997e-06 2024-09-16T03:37:40.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444867+0000", 2024-09-16T03:37:40.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.1009227 2024-09-16T03:37:40.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444885+0000", 2024-09-16T03:37:40.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7852000000000001e-05 2024-09-16T03:37:40.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445753+0000", 2024-09-16T03:37:40.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086790399999999996 2024-09-16T03:37:40.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445782+0000", 2024-09-16T03:37:40.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8827e-05 2024-09-16T03:37:40.110 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.110 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.110 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.110 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:55 4.7 e39/32 4:ed874e32:::smithi15736696-14:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:40.110 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.343952+0000", 2024-09-16T03:37:40.110 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.749229599, 2024-09-16T03:37:40.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10231132599999999, 2024-09-16T03:37:40.111 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.111 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.111 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.111 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343952+0000", 2024-09-16T03:37:40.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343952+0000", 2024-09-16T03:37:40.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343953+0000", 2024-09-16T03:37:40.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000002e-07 2024-09-16T03:37:40.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343955+0000", 2024-09-16T03:37:40.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4210000000000002e-06 2024-09-16T03:37:40.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343959+0000", 2024-09-16T03:37:40.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6389999999999998e-06 2024-09-16T03:37:40.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343962+0000", 2024-09-16T03:37:40.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.632e-06 2024-09-16T03:37:40.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445398+0000", 2024-09-16T03:37:40.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101436019 2024-09-16T03:37:40.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445418+0000", 2024-09-16T03:37:40.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0815e-05 2024-09-16T03:37:40.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446223+0000", 2024-09-16T03:37:40.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00080508100000000003 2024-09-16T03:37:40.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446264+0000", 2024-09-16T03:37:40.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0352999999999998e-05 2024-09-16T03:37:40.115 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.115 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.115 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.115 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:269 4.6 e40/32 4:64ce87bc:::smithi15736696-20:head v 40'15, mlcod=40'13)", 2024-09-16T03:37:40.115 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.196770+0000", 2024-09-16T03:37:40.116 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.896411908000001, 2024-09-16T03:37:40.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20089027500000001, 2024-09-16T03:37:40.116 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.116 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.116 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.116 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196770+0000", 2024-09-16T03:37:40.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196770+0000", 2024-09-16T03:37:40.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196771+0000", 2024-09-16T03:37:40.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.288e-06 2024-09-16T03:37:40.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196777+0000", 2024-09-16T03:37:40.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7169999999999996e-06 2024-09-16T03:37:40.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196787+0000", 2024-09-16T03:37:40.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.028e-05 2024-09-16T03:37:40.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.296854+0000", 2024-09-16T03:37:40.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100066848 2024-09-16T03:37:40.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.397050+0000", 2024-09-16T03:37:40.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100195387 2024-09-16T03:37:40.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.397082+0000", 2024-09-16T03:37:40.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1977000000000002e-05 2024-09-16T03:37:40.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.397619+0000", 2024-09-16T03:37:40.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000537734 2024-09-16T03:37:40.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.397660+0000", 2024-09-16T03:37:40.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1044000000000002e-05 2024-09-16T03:37:40.121 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.121 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.121 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.121 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.121 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:7 3.1 3:a961e47d:::smithi15736696-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:40.121 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.302964+0000", 2024-09-16T03:37:40.121 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.790217612999999, 2024-09-16T03:37:40.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19676334100000001, 2024-09-16T03:37:40.121 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.122 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.122 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.122 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302964+0000", 2024-09-16T03:37:40.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302964+0000", 2024-09-16T03:37:40.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302966+0000", 2024-09-16T03:37:40.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5889999999999999e-06 2024-09-16T03:37:40.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302972+0000", 2024-09-16T03:37:40.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.682e-06 2024-09-16T03:37:40.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302983+0000", 2024-09-16T03:37:40.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1147e-05 2024-09-16T03:37:40.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403076+0000", 2024-09-16T03:37:40.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100093178 2024-09-16T03:37:40.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403178+0000", 2024-09-16T03:37:40.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101525 2024-09-16T03:37:40.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403248+0000", 2024-09-16T03:37:40.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0548000000000003e-05 2024-09-16T03:37:40.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403370+0000", 2024-09-16T03:37:40.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012235100000000001 2024-09-16T03:37:40.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403908+0000", 2024-09-16T03:37:40.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053726999999999996 2024-09-16T03:37:40.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.404170+0000", 2024-09-16T03:37:40.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00026224899999999999 2024-09-16T03:37:40.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499671+0000", 2024-09-16T03:37:40.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.095501213000000001 2024-09-16T03:37:40.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499704+0000", 2024-09-16T03:37:40.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2641000000000001e-05 2024-09-16T03:37:40.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499728+0000", 2024-09-16T03:37:40.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3947999999999998e-05 2024-09-16T03:37:40.129 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.129 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.129 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.130 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.4.22:3 3.1 3:bffb5658:::smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:40.130 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.499594+0000", 2024-09-16T03:37:40.130 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.593588459999999, 2024-09-16T03:37:40.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.106172458, 2024-09-16T03:37:40.130 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.130 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.130 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.130 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499594+0000", 2024-09-16T03:37:40.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499594+0000", 2024-09-16T03:37:40.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499595+0000", 2024-09-16T03:37:40.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.361e-06 2024-09-16T03:37:40.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499600+0000", 2024-09-16T03:37:40.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.429e-06 2024-09-16T03:37:40.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499610+0000", 2024-09-16T03:37:40.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1460000000000002e-06 2024-09-16T03:37:40.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499617+0000", 2024-09-16T03:37:40.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7919999999999993e-06 2024-09-16T03:37:40.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.504406+0000", 2024-09-16T03:37:40.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0047886129999999997 2024-09-16T03:37:40.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.504474+0000", 2024-09-16T03:37:40.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7587000000000002e-05 2024-09-16T03:37:40.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.504600+0000", 2024-09-16T03:37:40.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012637799999999999 2024-09-16T03:37:40.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.505058+0000", 2024-09-16T03:37:40.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000457579 2024-09-16T03:37:40.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.605493+0000", 2024-09-16T03:37:40.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100435943 2024-09-16T03:37:40.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.605677+0000", 2024-09-16T03:37:40.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018323999999999999 2024-09-16T03:37:40.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.605725+0000", 2024-09-16T03:37:40.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8748999999999997e-05 2024-09-16T03:37:40.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.605766+0000", 2024-09-16T03:37:40.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0640999999999999e-05 2024-09-16T03:37:40.136 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.136 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.136 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.137 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:17 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:40.137 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.086208+0000", 2024-09-16T03:37:40.137 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.006973807000001, 2024-09-16T03:37:40.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20243159599999999, 2024-09-16T03:37:40.138 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.138 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.138 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.138 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086208+0000", 2024-09-16T03:37:40.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.139 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.139 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.139 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086208+0000", 2024-09-16T03:37:40.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.139 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.139 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.139 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086210+0000", 2024-09-16T03:37:40.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.058e-06 2024-09-16T03:37:40.139 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086216+0000", 2024-09-16T03:37:40.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1399999999999997e-06 2024-09-16T03:37:40.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086226+0000", 2024-09-16T03:37:40.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4110000000000002e-06 2024-09-16T03:37:40.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086235+0000", 2024-09-16T03:37:40.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4420000000000004e-06 2024-09-16T03:37:40.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.186482+0000", 2024-09-16T03:37:40.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10024678200000001 2024-09-16T03:37:40.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.186574+0000", 2024-09-16T03:37:40.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2035999999999998e-05 2024-09-16T03:37:40.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.186725+0000", 2024-09-16T03:37:40.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150629 2024-09-16T03:37:40.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.187384+0000", 2024-09-16T03:37:40.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065916599999999998 2024-09-16T03:37:40.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.187823+0000", 2024-09-16T03:37:40.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000439018 2024-09-16T03:37:40.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.288509+0000", 2024-09-16T03:37:40.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100685624 2024-09-16T03:37:40.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.288611+0000", 2024-09-16T03:37:40.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102155 2024-09-16T03:37:40.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.288640+0000", 2024-09-16T03:37:40.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9135000000000001e-05 2024-09-16T03:37:40.144 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.144 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.144 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.145 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:19 3.1 3:9aba8983:::smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:40.145 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:37:40.145 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 23.674750322000001, 2024-09-16T03:37:40.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30153978799999998, 2024-09-16T03:37:40.145 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.145 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.145 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.145 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:37:40.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:37:40.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418452+0000", 2024-09-16T03:37:40.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0363000000000001e-05 2024-09-16T03:37:40.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418470+0000", 2024-09-16T03:37:40.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7506000000000001e-05 2024-09-16T03:37:40.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418504+0000", 2024-09-16T03:37:40.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4542999999999998e-05 2024-09-16T03:37:40.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418522+0000", 2024-09-16T03:37:40.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8070000000000001e-05 2024-09-16T03:37:40.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418662+0000", 2024-09-16T03:37:40.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013946299999999999 2024-09-16T03:37:40.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418826+0000", 2024-09-16T03:37:40.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016469299999999999 2024-09-16T03:37:40.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418989+0000", 2024-09-16T03:37:40.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000162759 2024-09-16T03:37:40.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.419750+0000", 2024-09-16T03:37:40.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076104299999999998 2024-09-16T03:37:40.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.419865+0000", 2024-09-16T03:37:40.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114338 2024-09-16T03:37:40.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719876+0000", 2024-09-16T03:37:40.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30001127300000002 2024-09-16T03:37:40.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.151 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719920+0000", 2024-09-16T03:37:40.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4549999999999999e-05 2024-09-16T03:37:40.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.151 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719972+0000", 2024-09-16T03:37:40.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1187000000000003e-05 2024-09-16T03:37:40.152 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.152 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.152 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.152 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:10 3.1 3:8f38b759:::smithi15736696-6 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:40.152 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:37:40.152 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 23.585317684, 2024-09-16T03:37:40.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.21220708699999999, 2024-09-16T03:37:40.152 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.152 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.153 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.153 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:37:40.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:37:40.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507866+0000", 2024-09-16T03:37:40.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9999999999999999e-06 2024-09-16T03:37:40.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507873+0000", 2024-09-16T03:37:40.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7900000000000002e-06 2024-09-16T03:37:40.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507888+0000", 2024-09-16T03:37:40.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4554e-05 2024-09-16T03:37:40.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507896+0000", 2024-09-16T03:37:40.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1130000000000001e-06 2024-09-16T03:37:40.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507967+0000", 2024-09-16T03:37:40.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0808000000000004e-05 2024-09-16T03:37:40.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508028+0000", 2024-09-16T03:37:40.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1325e-05 2024-09-16T03:37:40.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508146+0000", 2024-09-16T03:37:40.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000117922 2024-09-16T03:37:40.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508722+0000", 2024-09-16T03:37:40.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000575779 2024-09-16T03:37:40.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.609567+0000", 2024-09-16T03:37:40.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100845448 2024-09-16T03:37:40.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720013+0000", 2024-09-16T03:37:40.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.110445644 2024-09-16T03:37:40.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720045+0000", 2024-09-16T03:37:40.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2186e-05 2024-09-16T03:37:40.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720071+0000", 2024-09-16T03:37:40.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6517999999999999e-05 2024-09-16T03:37:40.160 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.160 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.161 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.161 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:11 3.1 3:adc537a6:::smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:40.161 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:37:40.161 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 23.579974453999998, 2024-09-16T03:37:40.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20740488100000001, 2024-09-16T03:37:40.161 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.161 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.161 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.162 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:37:40.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:37:40.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513209+0000", 2024-09-16T03:37:40.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7439999999999999e-06 2024-09-16T03:37:40.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513216+0000", 2024-09-16T03:37:40.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9890000000000003e-06 2024-09-16T03:37:40.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513230+0000", 2024-09-16T03:37:40.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3263e-05 2024-09-16T03:37:40.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513236+0000", 2024-09-16T03:37:40.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7270000000000001e-06 2024-09-16T03:37:40.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513297+0000", 2024-09-16T03:37:40.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0767999999999998e-05 2024-09-16T03:37:40.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513375+0000", 2024-09-16T03:37:40.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8014000000000002e-05 2024-09-16T03:37:40.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513529+0000", 2024-09-16T03:37:40.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015342900000000001 2024-09-16T03:37:40.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.514258+0000", 2024-09-16T03:37:40.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072896799999999998 2024-09-16T03:37:40.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.709867+0000", 2024-09-16T03:37:40.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.195609016 2024-09-16T03:37:40.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720546+0000", 2024-09-16T03:37:40.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010679905999999999 2024-09-16T03:37:40.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720583+0000", 2024-09-16T03:37:40.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6659999999999998e-05 2024-09-16T03:37:40.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720613+0000", 2024-09-16T03:37:40.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9397e-05 2024-09-16T03:37:40.168 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.168 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.168 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.169 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:18 3.1 3:9f9f4ec5:::smithi15736696-22 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:40.169 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.104882+0000", 2024-09-16T03:37:40.169 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.988299691999998, 2024-09-16T03:37:40.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20218581599999999, 2024-09-16T03:37:40.169 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.169 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.169 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.169 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104882+0000", 2024-09-16T03:37:40.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104882+0000", 2024-09-16T03:37:40.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104885+0000", 2024-09-16T03:37:40.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7760000000000002e-06 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104895+0000", 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9909999999999997e-06 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104911+0000", 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5654999999999998e-05 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104927+0000", 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6620999999999999e-05 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104985+0000", 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.787e-05 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105043+0000", 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7720999999999998e-05 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105174+0000", 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013075599999999999 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105846+0000", 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067215899999999997 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206477+0000", 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10063103299999999 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307001+0000", 2024-09-16T03:37:40.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10052391200000001 2024-09-16T03:37:40.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307035+0000", 2024-09-16T03:37:40.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3674999999999999e-05 2024-09-16T03:37:40.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307068+0000", 2024-09-16T03:37:40.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3646999999999998e-05 2024-09-16T03:37:40.175 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.175 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.175 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.175 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:19 3.1 3:aca5d7ac:::smithi15736696-26 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:40.175 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:37:40.175 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.988117501000001, 2024-09-16T03:37:40.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264779099999999, 2024-09-16T03:37:40.175 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.175 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.176 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.176 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:37:40.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:37:40.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105068+0000", 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.751e-06 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105072+0000", 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5730000000000001e-06 2024-09-16T03:37:40.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205143+0000", 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100070665 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206515+0000", 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013715789999999999 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206570+0000", 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5049000000000001e-05 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206695+0000", 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012521599999999999 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307091+0000", 2024-09-16T03:37:40.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10039582 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307128+0000", 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7290999999999999e-05 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307656+0000", 2024-09-16T03:37:40.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00052813199999999995 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307689+0000", 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3056000000000003e-05 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307712+0000", 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3099999999999999e-05 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.181 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:22 3.1 3:9aba8983:::smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.207469+0000", 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.885713533000001, 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10211275, 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.207469+0000", 2024-09-16T03:37:40.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.207469+0000", 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.207470+0000", 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5069999999999999e-06 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.207475+0000", 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9849999999999997e-06 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.207484+0000", 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4509999999999998e-06 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.207490+0000", 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4870000000000002e-06 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307150+0000", 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099659532999999995 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307212+0000", 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2577999999999996e-05 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307329+0000", 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116841 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308257+0000", 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00092779699999999995 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308485+0000", 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000228013 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309524+0000", 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010396680000000001 2024-09-16T03:37:40.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309559+0000", 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4557999999999997e-05 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309581+0000", 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2331999999999999e-05 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.187 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.188 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:779 3.1 3:abdaa5c9:::2782d12a88f9842f7f754fca32c4222cf10fb780cd136830d32faaa700e687ba:head [write 0~45056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:40.188 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:37:40.188 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.606602005999999, 2024-09-16T03:37:40.188 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30397390600000002, 2024-09-16T03:37:40.188 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.188 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.188 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.188 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 779 2024-09-16T03:37:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:37:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:37:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486583+0000", 2024-09-16T03:37:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.762e-06 2024-09-16T03:37:40.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486649+0000", 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6143999999999996e-05 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486669+0000", 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9837e-05 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486681+0000", 2024-09-16T03:37:40.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2194999999999999e-05 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486746+0000", 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5480000000000003e-05 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486752+0000", 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4659999999999998e-06 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.192 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588316+0000", 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10156393399999999 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588539+0000", 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022308300000000001 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588647+0000", 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000108047 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.193 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.589516+0000", 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086926899999999999 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.789989+0000", 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20047296100000001 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790478+0000", 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048882099999999996 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790518+0000", 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0343000000000003e-05 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790554+0000", 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5564e-05 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.195 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:22 3.1 3:adc537a6:::smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.685615+0000", 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.407567001, 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10245631099999999, 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.685615+0000", 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.685615+0000", 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.685616+0000", 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8400000000000005e-07 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.685620+0000", 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5959999999999997e-06 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.685625+0000", 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4279999999999998e-06 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.685628+0000", 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.698e-06 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.686484+0000", 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00085608400000000001 2024-09-16T03:37:40.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.199 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.686571+0000", 2024-09-16T03:37:40.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6681999999999994e-05 2024-09-16T03:37:40.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.200 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.686670+0000", 2024-09-16T03:37:40.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8795000000000005e-05 2024-09-16T03:37:40.200 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.200 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.200 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.200 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.687862+0000", 2024-09-16T03:37:40.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00119217 2024-09-16T03:37:40.200 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.200 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.200 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.200 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.736408+0000", 2024-09-16T03:37:40.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048546266999999997 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.787986+0000", 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.051577286999999999 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.788036+0000", 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0182000000000002e-05 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.201 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.788071+0000", 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5638e-05 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:38 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.254774815999999, 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30216061300000002, 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.202 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838411+0000", 2024-09-16T03:37:40.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3780000000000001e-06 2024-09-16T03:37:40.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.204 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.204 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838422+0000", 2024-09-16T03:37:40.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1223000000000001e-05 2024-09-16T03:37:40.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.204 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.204 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838450+0000", 2024-09-16T03:37:40.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7878999999999999e-05 2024-09-16T03:37:40.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.204 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838464+0000", 2024-09-16T03:37:40.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3811e-05 2024-09-16T03:37:40.205 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838539+0000", 2024-09-16T03:37:40.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5685000000000005e-05 2024-09-16T03:37:40.205 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838604+0000", 2024-09-16T03:37:40.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4980999999999993e-05 2024-09-16T03:37:40.205 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838732+0000", 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012821100000000001 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839340+0000", 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060762399999999997 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840716+0000", 2024-09-16T03:37:40.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001375692 2024-09-16T03:37:40.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.207 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.207 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140491+0000", 2024-09-16T03:37:40.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29977497600000003 2024-09-16T03:37:40.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.207 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.207 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140534+0000", 2024-09-16T03:37:40.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3411999999999997e-05 2024-09-16T03:37:40.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140568+0000", 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3741000000000001e-05 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:39 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.253484598, 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40113680400000001, 2024-09-16T03:37:40.208 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.209 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.209 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.209 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.209 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:37:40.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.209 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.209 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:37:40.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.210 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839700+0000", 2024-09-16T03:37:40.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.571e-06 2024-09-16T03:37:40.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.210 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839704+0000", 2024-09-16T03:37:40.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8399999999999997e-06 2024-09-16T03:37:40.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.211 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839712+0000", 2024-09-16T03:37:40.211 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9990000000000008e-06 2024-09-16T03:37:40.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.211 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839720+0000", 2024-09-16T03:37:40.211 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0099999999999995e-06 2024-09-16T03:37:40.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.211 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839783+0000", 2024-09-16T03:37:40.212 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3014000000000003e-05 2024-09-16T03:37:40.212 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.212 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.212 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.212 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839870+0000", 2024-09-16T03:37:40.212 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7392999999999995e-05 2024-09-16T03:37:40.212 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.212 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.212 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.212 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840049+0000", 2024-09-16T03:37:40.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017825800000000001 2024-09-16T03:37:40.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.213 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840874+0000", 2024-09-16T03:37:40.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082532099999999995 2024-09-16T03:37:40.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.213 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.040285+0000", 2024-09-16T03:37:40.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19941088400000001 2024-09-16T03:37:40.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.214 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.214 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240748+0000", 2024-09-16T03:37:40.214 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20046322699999999 2024-09-16T03:37:40.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.214 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.214 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240786+0000", 2024-09-16T03:37:40.214 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8158000000000003e-05 2024-09-16T03:37:40.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.215 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.215 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240834+0000", 2024-09-16T03:37:40.215 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8128999999999999e-05 2024-09-16T03:37:40.215 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.215 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.215 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.215 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.215 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.215 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop_reply(osd.15.23:39 3.1 e47/31 ondisk, result = 0)", 2024-09-16T03:37:40.215 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:27.041370+0000", 2024-09-16T03:37:40.215 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.051812188, 2024-09-16T03:37:40.215 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19947226200000001, 2024-09-16T03:37:40.216 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.216 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.216 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:40.216 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.216 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.216 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.216 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.041370+0000", 2024-09-16T03:37:40.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.216 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.216 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.041370+0000", 2024-09-16T03:37:40.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.041371+0000", 2024-09-16T03:37:40.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:37:40.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.041375+0000", 2024-09-16T03:37:40.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9500000000000003e-06 2024-09-16T03:37:40.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.041387+0000", 2024-09-16T03:37:40.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1578e-05 2024-09-16T03:37:40.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.041393+0000", 2024-09-16T03:37:40.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.567e-06 2024-09-16T03:37:40.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.219 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.219 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.219 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240721+0000", 2024-09-16T03:37:40.219 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19932809100000001 2024-09-16T03:37:40.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.219 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.219 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.219 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240744+0000", 2024-09-16T03:37:40.220 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3000999999999999e-05 2024-09-16T03:37:40.220 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.220 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.220 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.220 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240842+0000", 2024-09-16T03:37:40.220 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7931999999999997e-05 2024-09-16T03:37:40.220 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.220 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.220 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.220 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.221 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.221 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:983 3.1 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head [write 0~2138112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:37:40.221 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:37:40.221 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.029581658, 2024-09-16T03:37:40.221 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30819799799999997, 2024-09-16T03:37:40.221 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.221 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.221 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.221 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.221 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.221 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.222 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 983 2024-09-16T03:37:40.222 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.222 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.222 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.222 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.222 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:37:40.222 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.222 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.222 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.222 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.222 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:37:40.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.223 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.223 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.223 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063610+0000", 2024-09-16T03:37:40.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2990000000000002e-06 2024-09-16T03:37:40.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.223 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.223 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.223 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065735+0000", 2024-09-16T03:37:40.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021257799999999999 2024-09-16T03:37:40.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.224 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.224 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.224 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065752+0000", 2024-09-16T03:37:40.224 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6895999999999999e-05 2024-09-16T03:37:40.224 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.224 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.224 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.224 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065762+0000", 2024-09-16T03:37:40.224 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6830000000000003e-06 2024-09-16T03:37:40.224 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.224 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.224 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.224 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065871+0000", 2024-09-16T03:37:40.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010904900000000001 2024-09-16T03:37:40.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.225 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.225 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.225 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066005+0000", 2024-09-16T03:37:40.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013369799999999999 2024-09-16T03:37:40.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.225 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.225 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.225 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066135+0000", 2024-09-16T03:37:40.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000130481 2024-09-16T03:37:40.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.226 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.226 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.226 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.069822+0000", 2024-09-16T03:37:40.226 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036863759999999999 2024-09-16T03:37:40.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.226 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.226 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.226 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272370+0000", 2024-09-16T03:37:40.226 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20254802599999999 2024-09-16T03:37:40.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.227 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371671+0000", 2024-09-16T03:37:40.227 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099300893000000001 2024-09-16T03:37:40.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.227 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371754+0000", 2024-09-16T03:37:40.227 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3075e-05 2024-09-16T03:37:40.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.228 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.228 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371798+0000", 2024-09-16T03:37:40.228 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4742e-05 2024-09-16T03:37:40.228 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.228 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.228 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.228 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.228 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.228 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1011 3.1 3:b99b36ea:::8de5e216df7365778a9df2a50922cc390a2a900c1854ce3c1d19b1da882722e1:head [write 0~193536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:40.228 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.107697+0000", 2024-09-16T03:37:40.228 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 8.9854849199999993, 2024-09-16T03:37:40.229 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102116056, 2024-09-16T03:37:40.229 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.229 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.229 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.229 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.229 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.229 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.229 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1011 2024-09-16T03:37:40.229 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.229 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.229 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.229 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.230 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.107697+0000", 2024-09-16T03:37:40.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.230 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.230 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.230 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.230 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.107697+0000", 2024-09-16T03:37:40.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.230 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.230 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.230 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.230 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.107699+0000", 2024-09-16T03:37:40.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.283e-06 2024-09-16T03:37:40.231 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.231 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.107969+0000", 2024-09-16T03:37:40.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000269845 2024-09-16T03:37:40.231 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.231 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.107985+0000", 2024-09-16T03:37:40.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5705e-05 2024-09-16T03:37:40.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.107993+0000", 2024-09-16T03:37:40.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1030000000000002e-06 2024-09-16T03:37:40.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.108079+0000", 2024-09-16T03:37:40.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5669999999999998e-05 2024-09-16T03:37:40.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.233 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.233 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.108205+0000", 2024-09-16T03:37:40.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012593599999999999 2024-09-16T03:37:40.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.233 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.233 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:40.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.108334+0000", 2024-09-16T03:37:40.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000129381 2024-09-16T03:37:40.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.234 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.109113+0000", 2024-09-16T03:37:40.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00077857499999999999 2024-09-16T03:37:40.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.234 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.209618+0000", 2024-09-16T03:37:40.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100505206 2024-09-16T03:37:40.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.234 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.209714+0000", 2024-09-16T03:37:40.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5969000000000002e-05 2024-09-16T03:37:40.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.235 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.235 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.235 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.209756+0000", 2024-09-16T03:37:40.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2274000000000002e-05 2024-09-16T03:37:40.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.235 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.235 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.235 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.209813+0000", 2024-09-16T03:37:40.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7108999999999998e-05 2024-09-16T03:37:40.235 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.236 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.236 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.236 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.236 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.236 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:40.239 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:37:40.239 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:40.235+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:40.242 INFO:tasks.rados.rados.0.smithi157.stdout:797: set_chunk oid 26 offset: 1059328 length: 695808 target oid 7caeb75b30ec898384742d54cecd2fd4bfd99c530bb097d7e542554b0db6f529 offset: 0 snap -1 2024-09-16T03:37:40.242 INFO:tasks.rados.rados.0.smithi157.stdout:798: tier_evict oid 48 2024-09-16T03:37:40.242 INFO:tasks.rados.rados.0.smithi157.stdout:798: tier_evict oid 48 snap -1 2024-09-16T03:37:40.242 INFO:tasks.rados.rados.0.smithi157.stdout:791: finishing write tid 1 to smithi15736696-30 2024-09-16T03:37:40.243 INFO:tasks.rados.rados.0.smithi157.stdout:794: finishing write tid 2 to smithi15736696-10 2024-09-16T03:37:40.243 INFO:tasks.rados.rados.0.smithi157.stdout:794: finishing write tid 3 to smithi15736696-10 2024-09-16T03:37:40.243 INFO:tasks.rados.rados.0.smithi157.stdout:782: finishing write tid 2 to smithi15736696-38 2024-09-16T03:37:40.243 INFO:tasks.rados.rados.0.smithi157.stdout:782: finishing write tid 3 to smithi15736696-38 2024-09-16T03:37:40.243 INFO:tasks.rados.rados.0.smithi157.stdout:791: finishing write tid 2 to smithi15736696-30 2024-09-16T03:37:40.243 INFO:tasks.rados.rados.0.smithi157.stdout:791: finishing write tid 3 to smithi15736696-30 2024-09-16T03:37:40.243 INFO:tasks.rados.rados.0.smithi157.stdout:791: finishing write tid 4 to smithi15736696-30 2024-09-16T03:37:40.243 INFO:tasks.rados.rados.0.smithi157.stdout:791: finishing write tid 5 to smithi15736696-30 2024-09-16T03:37:40.243 INFO:tasks.rados.rados.0.smithi157.stdout:791: oid 30 updating version 0 to 28 2024-09-16T03:37:40.243 INFO:tasks.rados.rados.0.smithi157.stdout:791: oid 30 updating version 28 to 29 2024-09-16T03:37:40.243 INFO:tasks.rados.rados.0.smithi157.stdout:791: oid 30 version 29 is already newer than 26 2024-09-16T03:37:40.244 INFO:tasks.rados.rados.0.smithi157.stdout:791: oid 30 version 29 is already newer than 27 2024-09-16T03:37:40.244 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 29 (ObjNum 98 snap 0 seq_num 98) dirty exists 2024-09-16T03:37:40.244 INFO:tasks.rados.rados.0.smithi157.stdout:791: left oid 30 (ObjNum 98 snap 0 seq_num 98) 2024-09-16T03:37:40.244 INFO:tasks.rados.rados.0.smithi157.stdout:784: done (14 left) 2024-09-16T03:37:40.244 INFO:tasks.rados.rados.0.smithi157.stdout:791: done (13 left) 2024-09-16T03:37:40.244 INFO:tasks.rados.rados.0.smithi157.stdout:799: read oid 12 snap -1 2024-09-16T03:37:40.244 INFO:tasks.rados.rados.0.smithi157.stdout:799: expect deleted 2024-09-16T03:37:40.244 INFO:tasks.rados.rados.0.smithi157.stdout:800: write (excl) oid 20 current snap is 0 2024-09-16T03:37:40.244 INFO:tasks.rados.rados.0.smithi157.stdout:794: finishing write tid 4 to smithi15736696-10 2024-09-16T03:37:40.244 INFO:tasks.rados.rados.0.smithi157.stdout:793: finishing write tid 1 to smithi15736696-6 2024-09-16T03:37:40.244 INFO:tasks.rados.rados.0.smithi157.stdout:785: finishing write tid 2 to smithi15736696-21 2024-09-16T03:37:40.244 INFO:tasks.rados.rados.0.smithi157.stdout:785: finishing write tid 3 to smithi15736696-21 2024-09-16T03:37:40.245 INFO:tasks.rados.rados.0.smithi157.stdout:800: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2024-09-16T03:37:40.245 INFO:tasks.rados.rados.0.smithi157.stdout:800: writing smithi15736696-20 from 514314 to 1302301 tid 1 2024-09-16T03:37:40.245 INFO:tasks.rados.rados.0.smithi157.stdout:800: writing smithi15736696-20 from 1768928 to 2174282 tid 2 2024-09-16T03:37:40.245 INFO:tasks.rados.rados.0.smithi157.stdout:800: writing smithi15736696-20 from 2768204 to 3277714 tid 3 2024-09-16T03:37:40.245 INFO:tasks.rados.rados.0.smithi157.stdout:801: write (excl) oid 39 current snap is 0 2024-09-16T03:37:40.245 INFO:tasks.rados.rados.0.smithi157.stdout:801: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-09-16T03:37:40.245 INFO:tasks.rados.rados.0.smithi157.stdout:801: writing smithi15736696-39 from 531121 to 994357 tid 1 2024-09-16T03:37:40.245 INFO:tasks.rados.rados.0.smithi157.stdout:801: writing smithi15736696-39 from 1711057 to 2376422 tid 2 2024-09-16T03:37:40.245 INFO:tasks.rados.rados.0.smithi157.stdout:801: writing smithi15736696-39 from 3079274 to 3731121 tid 3 2024-09-16T03:37:40.245 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:40.245 INFO:tasks.rados.rados.0.smithi157.stdout:781: got (0) Success 2024-09-16T03:37:40.245 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 25 (ObjNum 90 snap 0 seq_num 90) dirty exists 2024-09-16T03:37:40.246 INFO:tasks.rados.rados.0.smithi157.stdout:782: finishing write tid 4 to smithi15736696-38 2024-09-16T03:37:40.246 INFO:tasks.rados.rados.0.smithi157.stdout:793: finishing write tid 2 to smithi15736696-6 2024-09-16T03:37:40.246 INFO:tasks.rados.rados.0.smithi157.stdout:793: finishing write tid 3 to smithi15736696-6 2024-09-16T03:37:40.246 INFO:tasks.rados.rados.0.smithi157.stdout:781: done (15 left) 2024-09-16T03:37:40.246 INFO:tasks.rados.rados.0.smithi157.stdout:802: read oid 29 snap -1 2024-09-16T03:37:40.246 INFO:tasks.rados.rados.0.smithi157.stdout:802: expect (ObjNum 75 snap 0 seq_num 75) 2024-09-16T03:37:40.246 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:40.246 INFO:tasks.rados.rados.0.smithi157.stdout:787: finishing write tid 1 to smithi15736696-46 2024-09-16T03:37:40.246 INFO:tasks.rados.rados.0.smithi157.stdout:787: finishing write tid 2 to smithi15736696-46 2024-09-16T03:37:40.246 INFO:tasks.rados.rados.0.smithi157.stdout:787: finishing write tid 3 to smithi15736696-46 2024-09-16T03:37:40.246 INFO:tasks.rados.rados.0.smithi157.stdout:787: finishing write tid 4 to smithi15736696-46 2024-09-16T03:37:40.246 INFO:tasks.rados.rados.0.smithi157.stdout:787: finishing write tid 5 to smithi15736696-46 2024-09-16T03:37:40.247 INFO:tasks.rados.rados.0.smithi157.stdout:787: oid 46 updating version 0 to 43 2024-09-16T03:37:40.247 INFO:tasks.rados.rados.0.smithi157.stdout:787: oid 46 updating version 43 to 45 2024-09-16T03:37:40.247 INFO:tasks.rados.rados.0.smithi157.stdout:787: oid 46 version 45 is already newer than 42 2024-09-16T03:37:40.247 INFO:tasks.rados.rados.0.smithi157.stdout:787: oid 46 version 45 is already newer than 44 2024-09-16T03:37:40.247 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 45 (ObjNum 97 snap 0 seq_num 97) dirty exists 2024-09-16T03:37:40.247 INFO:tasks.rados.rados.0.smithi157.stdout:787: left oid 46 (ObjNum 97 snap 0 seq_num 97) 2024-09-16T03:37:40.247 INFO:tasks.rados.rados.0.smithi157.stdout:794: finishing write tid 5 to smithi15736696-10 2024-09-16T03:37:40.247 INFO:tasks.rados.rados.0.smithi157.stdout:794: oid 10 updating version 0 to 46 2024-09-16T03:37:40.247 INFO:tasks.rados.rados.0.smithi157.stdout:794: oid 10 updating version 46 to 48 2024-09-16T03:37:40.247 INFO:tasks.rados.rados.0.smithi157.stdout:794: oid 10 updating version 48 to 49 2024-09-16T03:37:40.247 INFO:tasks.rados.rados.0.smithi157.stdout:794: oid 10 version 49 is already newer than 47 2024-09-16T03:37:40.247 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 49 (ObjNum 100 snap 0 seq_num 100) dirty exists 2024-09-16T03:37:40.248 INFO:tasks.rados.rados.0.smithi157.stdout:794: left oid 10 (ObjNum 100 snap 0 seq_num 100) 2024-09-16T03:37:40.248 INFO:tasks.rados.rados.0.smithi157.stdout:785: finishing write tid 4 to smithi15736696-21 2024-09-16T03:37:40.248 INFO:tasks.rados.rados.0.smithi157.stdout:785: finishing write tid 5 to smithi15736696-21 2024-09-16T03:37:40.248 INFO:tasks.rados.rados.0.smithi157.stdout:785: oid 21 updating version 0 to 26 2024-09-16T03:37:40.248 INFO:tasks.rados.rados.0.smithi157.stdout:785: oid 21 updating version 26 to 28 2024-09-16T03:37:40.248 INFO:tasks.rados.rados.0.smithi157.stdout:785: oid 21 updating version 28 to 29 2024-09-16T03:37:40.248 INFO:tasks.rados.rados.0.smithi157.stdout:785: oid 21 version 29 is already newer than 27 2024-09-16T03:37:40.248 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 29 (ObjNum 96 snap 0 seq_num 96) dirty exists 2024-09-16T03:37:40.248 INFO:tasks.rados.rados.0.smithi157.stdout:785: left oid 21 (ObjNum 96 snap 0 seq_num 96) 2024-09-16T03:37:40.248 INFO:tasks.rados.rados.0.smithi157.stdout:782: finishing write tid 5 to smithi15736696-38 2024-09-16T03:37:40.248 INFO:tasks.rados.rados.0.smithi157.stdout:782: oid 38 updating version 0 to 21 2024-09-16T03:37:40.249 INFO:tasks.rados.rados.0.smithi157.stdout:782: oid 38 version 21 is already newer than 19 2024-09-16T03:37:40.249 INFO:tasks.rados.rados.0.smithi157.stdout:782: oid 38 version 21 is already newer than 18 2024-09-16T03:37:40.249 INFO:tasks.rados.rados.0.smithi157.stdout:782: oid 38 version 21 is already newer than 20 2024-09-16T03:37:40.249 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 21 (ObjNum 95 snap 0 seq_num 95) dirty exists 2024-09-16T03:37:40.249 INFO:tasks.rados.rados.0.smithi157.stdout:782: left oid 38 (ObjNum 95 snap 0 seq_num 95) 2024-09-16T03:37:40.249 INFO:tasks.rados.rados.0.smithi157.stdout:793: finishing write tid 4 to smithi15736696-6 2024-09-16T03:37:40.249 INFO:tasks.rados.rados.0.smithi157.stdout:793: finishing write tid 5 to smithi15736696-6 2024-09-16T03:37:40.249 INFO:tasks.rados.rados.0.smithi157.stdout:793: oid 6 updating version 0 to 38 2024-09-16T03:37:40.249 INFO:tasks.rados.rados.0.smithi157.stdout:793: oid 6 version 38 is already newer than 37 2024-09-16T03:37:40.249 INFO:tasks.rados.rados.0.smithi157.stdout:793: oid 6 version 38 is already newer than 35 2024-09-16T03:37:40.249 INFO:tasks.rados.rados.0.smithi157.stdout:793: oid 6 version 38 is already newer than 36 2024-09-16T03:37:40.249 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 38 (ObjNum 99 snap 0 seq_num 99) dirty exists 2024-09-16T03:37:40.250 INFO:tasks.rados.rados.0.smithi157.stdout:793: left oid 6 (ObjNum 99 snap 0 seq_num 99) 2024-09-16T03:37:40.250 INFO:tasks.rados.rados.0.smithi157.stdout:795: expect (ObjNum 93 snap 0 seq_num 93) 2024-09-16T03:37:40.250 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:40.286 INFO:tasks.rados.rados.0.smithi157.stdout:786: got (0) Success 2024-09-16T03:37:40.286 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:40.286 INFO:tasks.rados.rados.0.smithi157.stdout:792: got (0) Success 2024-09-16T03:37:40.286 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:37:40.286 INFO:tasks.rados.rados.0.smithi157.stdout:796: finishing write tid 1 to smithi15736696-43 2024-09-16T03:37:40.286 INFO:tasks.rados.rados.0.smithi157.stdout:796: finishing write tid 2 to smithi15736696-43 2024-09-16T03:37:40.286 INFO:tasks.rados.rados.0.smithi157.stdout:796: finishing write tid 3 to smithi15736696-43 2024-09-16T03:37:40.286 INFO:tasks.rados.rados.0.smithi157.stdout:796: finishing write tid 4 to smithi15736696-43 2024-09-16T03:37:40.286 INFO:tasks.rados.rados.0.smithi157.stdout:782: done (15 left) 2024-09-16T03:37:40.286 INFO:tasks.rados.rados.0.smithi157.stdout:785: done (14 left) 2024-09-16T03:37:40.286 INFO:tasks.rados.rados.0.smithi157.stdout:786: done (13 left) 2024-09-16T03:37:40.286 INFO:tasks.rados.rados.0.smithi157.stdout:787: done (12 left) 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:792: done (11 left) 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:793: done (10 left) 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:794: done (9 left) 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:795: done (8 left) 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:803: read oid 34 snap -1 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:803: expect (ObjNum 82 snap 0 seq_num 82) 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:804: read oid 18 snap -1 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:804: expect (ObjNum 74 snap 0 seq_num 74) 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:805: read oid 41 snap -1 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:805: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:806: write oid 50 current snap is 0 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:796: finishing write tid 5 to smithi15736696-43 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:796: oid 43 updating version 0 to 32 2024-09-16T03:37:40.287 INFO:tasks.rados.rados.0.smithi157.stdout:796: oid 43 updating version 32 to 33 2024-09-16T03:37:40.288 INFO:tasks.rados.rados.0.smithi157.stdout:796: oid 43 updating version 33 to 34 2024-09-16T03:37:40.288 INFO:tasks.rados.rados.0.smithi157.stdout:796: oid 43 version 34 is already newer than 31 2024-09-16T03:37:40.288 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 34 (ObjNum 101 snap 0 seq_num 101) dirty exists 2024-09-16T03:37:40.288 INFO:tasks.rados.rados.0.smithi157.stdout:796: left oid 43 (ObjNum 101 snap 0 seq_num 101) 2024-09-16T03:37:40.288 INFO:tasks.rados.rados.0.smithi157.stdout:806: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2024-09-16T03:37:40.291 INFO:tasks.rados.rados.0.smithi157.stdout:806: writing smithi15736696-50 from 547928 to 1086413 tid 1 2024-09-16T03:37:40.297 INFO:tasks.rados.rados.0.smithi157.stdout:806: writing smithi15736696-50 from 1769539 to 2178562 tid 2 2024-09-16T03:37:40.303 INFO:tasks.rados.rados.0.smithi157.stdout:806: writing smithi15736696-50 from 2706697 to 3238751 tid 3 2024-09-16T03:37:40.303 INFO:tasks.rados.rados.0.smithi157.stdout:796: done (11 left) 2024-09-16T03:37:40.303 INFO:tasks.rados.rados.0.smithi157.stdout:807: read oid 14 snap -1 2024-09-16T03:37:40.303 INFO:tasks.rados.rados.0.smithi157.stdout:807: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T03:37:40.303 INFO:tasks.rados.rados.0.smithi157.stdout:808: set_chunk oid 8 target oid 2024-09-16T03:37:40.338 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:40.336+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:40.377 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:37:40.384 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1377 4.17 4.b14506d7 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-16T03:37:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:40.284951+0000", 2024-09-16T03:37:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.098716806000000004, 2024-09-16T03:37:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098731839000000002, 2024-09-16T03:37:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "no flag points reached", 2024-09-16T03:37:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1377 2024-09-16T03:37:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.284951+0000", 2024-09-16T03:37:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.284951+0000", 2024-09-16T03:37:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.284951+0000", 2024-09-16T03:37:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.89e-07 2024-09-16T03:37:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.284953+0000", 2024-09-16T03:37:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4099999999999998e-06 2024-09-16T03:37:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.284957+0000", 2024-09-16T03:37:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.799e-06 2024-09-16T03:37:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2024-09-16T03:37:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 1 2024-09-16T03:37:40.388 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:40.394 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:37:40.439 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:40.437+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:40.499 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:40.500 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:40.500 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:40.500 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:40.500 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:40.509 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:37:40.540 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:40.538+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:40.616 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:40.616 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:40.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:40.616 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:40.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.616 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:31 4.f 4:f0d14c71:::smithi15736696-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:40.616 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.043076+0000", 2024-09-16T03:37:40.616 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.569976297, 2024-09-16T03:37:40.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50359721000000002, 2024-09-16T03:37:40.616 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.616 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 31 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043076+0000", 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043076+0000", 2024-09-16T03:37:40.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043078+0000", 2024-09-16T03:37:40.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.189e-06 2024-09-16T03:37:40.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043084+0000", 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.896e-06 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043091+0000", 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9770000000000003e-06 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143160+0000", 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10006909 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143222+0000", 2024-09-16T03:37:40.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2049000000000002e-05 2024-09-16T03:37:40.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143324+0000", 2024-09-16T03:37:40.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102737 2024-09-16T03:37:40.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:37:40.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143485+0000", 2024-09-16T03:37:40.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016040100000000001 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.147043+0000", 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0035581609999999998 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.244108+0000", 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.097064967000000002 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.546541+0000", 2024-09-16T03:37:40.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30243345199999999 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.546605+0000", 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4039000000000001e-05 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.546674+0000", 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8251999999999996e-05 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:66 4.0 4:023abada:::smithi15736696-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.200996+0000", 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.412056682999999, 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50798839299999998, 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 66 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.623 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.200996+0000", 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.200996+0000", 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.200998+0000", 2024-09-16T03:37:40.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5179999999999999e-06 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.201330+0000", 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000332672 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.201340+0000", 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8099999999999992e-06 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301446+0000", 2024-09-16T03:37:40.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100105877 2024-09-16T03:37:40.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.628 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301543+0000", 2024-09-16T03:37:40.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7058000000000005e-05 2024-09-16T03:37:40.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301628+0000", 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5078000000000001e-05 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301770+0000", 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014212400000000001 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.303247+0000", 2024-09-16T03:37:40.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001476595 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.606126+0000", 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30287937799999998 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.708902+0000", 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102775402 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.708938+0000", 2024-09-16T03:37:40.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5947e-05 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.708984+0000", 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6934000000000002e-05 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:69 4.0 4:0a55bbf2:::smithi15736696-18:head [write 685719~711939,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.301546+0000", 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.31150676, 2024-09-16T03:37:40.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50794553600000003, 2024-09-16T03:37:40.632 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.632 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.632 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.632 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.632 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.632 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.632 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 69 2024-09-16T03:37:40.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.632 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301546+0000", 2024-09-16T03:37:40.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301546+0000", 2024-09-16T03:37:40.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301547+0000", 2024-09-16T03:37:40.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4799999999999997e-07 2024-09-16T03:37:40.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302372+0000", 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000824828 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302381+0000", 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1759999999999999e-06 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302389+0000", 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4710000000000007e-06 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.401577+0000", 2024-09-16T03:37:40.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099188002999999997 2024-09-16T03:37:40.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.401681+0000", 2024-09-16T03:37:40.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000103618 2024-09-16T03:37:40.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:40.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.401797+0000", 2024-09-16T03:37:40.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116177 2024-09-16T03:37:40.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.403396+0000", 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001598857 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.606185+0000", 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202788996 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.809397+0000", 2024-09-16T03:37:40.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20321167700000001 2024-09-16T03:37:40.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.809449+0000", 2024-09-16T03:37:40.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2389000000000003e-05 2024-09-16T03:37:40.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.809492+0000", 2024-09-16T03:37:40.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2496000000000001e-05 2024-09-16T03:37:40.637 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.637 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.637 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.638 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:286 4.7 4:e2f8c24a:::smithi15736696-37:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:40.638 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.600823+0000", 2024-09-16T03:37:40.638 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 25.012229495, 2024-09-16T03:37:40.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50653485200000004, 2024-09-16T03:37:40.638 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.638 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.638 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.638 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.638 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.638 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.638 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 286 2024-09-16T03:37:40.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.639 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600823+0000", 2024-09-16T03:37:40.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600823+0000", 2024-09-16T03:37:40.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600825+0000", 2024-09-16T03:37:40.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.736e-06 2024-09-16T03:37:40.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600830+0000", 2024-09-16T03:37:40.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2480000000000004e-06 2024-09-16T03:37:40.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600842+0000", 2024-09-16T03:37:40.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1309e-05 2024-09-16T03:37:40.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.700982+0000", 2024-09-16T03:37:40.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100140487 2024-09-16T03:37:40.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701107+0000", 2024-09-16T03:37:40.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000124671 2024-09-16T03:37:40.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701165+0000", 2024-09-16T03:37:40.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7995000000000003e-05 2024-09-16T03:37:40.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:37:40.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903559+0000", 2024-09-16T03:37:40.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20239401100000001 2024-09-16T03:37:40.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.904281+0000", 2024-09-16T03:37:40.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072182699999999997 2024-09-16T03:37:40.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.904415+0000", 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013477300000000001 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.107259+0000", 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202843209 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.107321+0000", 2024-09-16T03:37:40.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2242000000000005e-05 2024-09-16T03:37:40.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.107358+0000", 2024-09-16T03:37:40.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7344e-05 2024-09-16T03:37:40.644 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.644 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.644 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.644 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:409 4.f 4:fa163101:::smithi15736696-10:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:40.644 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.306109+0000", 2024-09-16T03:37:40.644 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.306943390000001, 2024-09-16T03:37:40.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50402713799999999, 2024-09-16T03:37:40.645 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.645 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.645 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.645 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.645 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.645 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.645 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 409 2024-09-16T03:37:40.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.645 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306109+0000", 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306109+0000", 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306111+0000", 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.429e-06 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306116+0000", 2024-09-16T03:37:40.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1019999999999996e-06 2024-09-16T03:37:40.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.647 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306129+0000", 2024-09-16T03:37:40.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2840000000000001e-05 2024-09-16T03:37:40.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.647 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306137+0000", 2024-09-16T03:37:40.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1370000000000002e-06 2024-09-16T03:37:40.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.506224+0000", 2024-09-16T03:37:40.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20008714899999999 2024-09-16T03:37:40.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.506352+0000", 2024-09-16T03:37:40.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000128424 2024-09-16T03:37:40.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:37:40.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709180+0000", 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20282708299999999 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709780+0000", 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060068800000000005 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.710070+0000", 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000289683 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.809989+0000", 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099918689000000005 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.810070+0000", 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1135000000000002e-05 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.810136+0000", 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6779000000000006e-05 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.650 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:416 4.0 4:023abada:::smithi15736696-17:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.303811035999999, 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60401983699999995, 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.651 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 416 2024-09-16T03:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309243+0000", 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.178e-06 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309248+0000", 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309257+0000", 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4720000000000001e-06 2024-09-16T03:37:40.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309262+0000", 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0839999999999999e-06 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309297+0000", 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4997999999999999e-05 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309367+0000", 2024-09-16T03:37:40.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0124999999999997e-05 2024-09-16T03:37:40.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:40.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812055+0000", 2024-09-16T03:37:40.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50268796900000001 2024-09-16T03:37:40.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812614+0000", 2024-09-16T03:37:40.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000558309 2024-09-16T03:37:40.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.813040+0000", 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042652800000000002 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913189+0000", 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10014880399999999 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913220+0000", 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1353000000000003e-05 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913262+0000", 2024-09-16T03:37:40.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1284e-05 2024-09-16T03:37:40.657 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.657 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.657 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.657 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:494 4.f 4:f817def9:::smithi15736696-45:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:40.657 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:37:40.657 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 18.890871301000001, 2024-09-16T03:37:40.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60451760399999999, 2024-09-16T03:37:40.658 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.658 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.658 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.658 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.658 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.658 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.658 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 494 2024-09-16T03:37:40.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.658 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:37:40.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:37:40.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722183+0000", 2024-09-16T03:37:40.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:37:40.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722189+0000", 2024-09-16T03:37:40.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9429999999999997e-06 2024-09-16T03:37:40.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722203+0000", 2024-09-16T03:37:40.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4367e-05 2024-09-16T03:37:40.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722210+0000", 2024-09-16T03:37:40.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822367+0000", 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100156933 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822461+0000", 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3216999999999999e-05 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:37:40.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125593+0000", 2024-09-16T03:37:40.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30313253699999998 2024-09-16T03:37:40.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.662 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126307+0000", 2024-09-16T03:37:40.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071350000000000005 2024-09-16T03:37:40.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.662 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126825+0000", 2024-09-16T03:37:40.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051846100000000001 2024-09-16T03:37:40.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326612+0000", 2024-09-16T03:37:40.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19978721999999999 2024-09-16T03:37:40.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326663+0000", 2024-09-16T03:37:40.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0460000000000001e-05 2024-09-16T03:37:40.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326699+0000", 2024-09-16T03:37:40.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6340000000000001e-05 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:517 4.0 4:0a55bbf2:::smithi15736696-18:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.922514+0000", 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 18.690538670999999, 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50578298600000005, 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.664 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 517 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922514+0000", 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922514+0000", 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.666 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922515+0000", 2024-09-16T03:37:40.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.381e-06 2024-09-16T03:37:40.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.666 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922520+0000", 2024-09-16T03:37:40.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8590000000000004e-06 2024-09-16T03:37:40.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.666 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922531+0000", 2024-09-16T03:37:40.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0783e-05 2024-09-16T03:37:40.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922539+0000", 2024-09-16T03:37:40.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4220000000000003e-06 2024-09-16T03:37:40.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.923072+0000", 2024-09-16T03:37:40.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053305500000000005 2024-09-16T03:37:40.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.923145+0000", 2024-09-16T03:37:40.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3572000000000006e-05 2024-09-16T03:37:40.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:40.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.327214+0000", 2024-09-16T03:37:40.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.404069177 2024-09-16T03:37:40.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.328231+0000", 2024-09-16T03:37:40.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010163240000000001 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.330360+0000", 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021293029999999999 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.428103+0000", 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.097742869999999996 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.428244+0000", 2024-09-16T03:37:40.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014071999999999999 2024-09-16T03:37:40.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.670 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.670 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.428297+0000", 2024-09-16T03:37:40.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3520000000000003e-05 2024-09-16T03:37:40.670 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.670 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.670 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.670 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:784 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=63b,truncate 2840350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:40.671 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:37:40.671 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.806463425, 2024-09-16T03:37:40.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63358405200000001, 2024-09-16T03:37:40.671 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.671 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.671 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.671 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.671 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.671 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.671 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 784 2024-09-16T03:37:40.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.671 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806590+0000", 2024-09-16T03:37:40.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6000000000000003e-07 2024-09-16T03:37:40.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806593+0000", 2024-09-16T03:37:40.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6649999999999999e-06 2024-09-16T03:37:40.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806597+0000", 2024-09-16T03:37:40.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.245e-06 2024-09-16T03:37:40.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806600+0000", 2024-09-16T03:37:40.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.23e-06 2024-09-16T03:37:40.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099381+0000", 2024-09-16T03:37:40.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29278119400000002 2024-09-16T03:37:40.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099438+0000", 2024-09-16T03:37:40.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6198e-05 2024-09-16T03:37:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099527+0000", 2024-09-16T03:37:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9505000000000004e-05 2024-09-16T03:37:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200775+0000", 2024-09-16T03:37:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101247904 2024-09-16T03:37:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200889+0000", 2024-09-16T03:37:40.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114301 2024-09-16T03:37:40.676 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440066+0000", 2024-09-16T03:37:40.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.239176367 2024-09-16T03:37:40.676 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440109+0000", 2024-09-16T03:37:40.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2941000000000001e-05 2024-09-16T03:37:40.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440173+0000", 2024-09-16T03:37:40.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4842e-05 2024-09-16T03:37:40.677 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.677 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.677 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.677 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:785 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:40.677 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:37:40.677 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.806447500000001, 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63376373200000002, 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 785 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:37:40.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806606+0000", 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806607+0000", 2024-09-16T03:37:40.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.424e-06 2024-09-16T03:37:40.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806611+0000", 2024-09-16T03:37:40.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1080000000000003e-06 2024-09-16T03:37:40.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806614+0000", 2024-09-16T03:37:40.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.067e-06 2024-09-16T03:37:40.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099813+0000", 2024-09-16T03:37:40.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29319853699999998 2024-09-16T03:37:40.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:40.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099858+0000", 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5667999999999997e-05 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440236+0000", 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34037791499999998 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440297+0000", 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0988e-05 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440369+0000", 2024-09-16T03:37:40.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1682999999999998e-05 2024-09-16T03:37:40.683 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.683 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.683 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.683 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:801 4.0 4:07ad0560:::smithi15736696-25:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:40.683 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:37:40.683 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.606969601999999, 2024-09-16T03:37:40.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.534510445, 2024-09-16T03:37:40.683 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.683 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 801 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:37:40.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006084+0000", 2024-09-16T03:37:40.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.322e-06 2024-09-16T03:37:40.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006088+0000", 2024-09-16T03:37:40.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000001e-06 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006098+0000", 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0186999999999999e-05 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006104+0000", 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.271e-06 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.199952+0000", 2024-09-16T03:37:40.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.193847935 2024-09-16T03:37:40.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.687 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.687 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.687 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200027+0000", 2024-09-16T03:37:40.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4987000000000006e-05 2024-09-16T03:37:40.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.540594+0000", 2024-09-16T03:37:40.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34056708299999999 2024-09-16T03:37:40.688 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.688 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.688 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.688 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:992 4.f 4:f817def9:::smithi15736696-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:37:40.688 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.389682+0000", 2024-09-16T03:37:40.689 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.223371186, 2024-09-16T03:37:40.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50248604799999996, 2024-09-16T03:37:40.689 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.689 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.689 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:40.689 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.689 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.689 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.689 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 992 2024-09-16T03:37:40.689 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.689 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.389682+0000", 2024-09-16T03:37:40.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.389682+0000", 2024-09-16T03:37:40.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.389682+0000", 2024-09-16T03:37:40.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.72e-07 2024-09-16T03:37:40.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.691 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.389684+0000", 2024-09-16T03:37:40.691 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5859999999999999e-06 2024-09-16T03:37:40.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.691 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.389688+0000", 2024-09-16T03:37:40.691 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2490000000000003e-06 2024-09-16T03:37:40.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.389692+0000", 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3900000000000003e-06 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.485786+0000", 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.096093897999999997 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.485843+0000", 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7429999999999997e-05 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.692 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.892021+0000", 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40617789599999998 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.892086+0000", 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5013999999999998e-05 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.892168+0000", 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1212999999999996e-05 2024-09-16T03:37:40.693 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.694 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.694 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.694 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.694 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1022 4.0 4:06b6d050:::smithi15736696-50:head [write 776490~714607] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:40.694 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.310462+0000", 2024-09-16T03:37:40.694 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.3025902729999999, 2024-09-16T03:37:40.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50630361300000004, 2024-09-16T03:37:40.694 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.694 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.694 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.694 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1022 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.310462+0000", 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.310462+0000", 2024-09-16T03:37:40.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.696 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.310464+0000", 2024-09-16T03:37:40.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.567e-06 2024-09-16T03:37:40.696 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311198+0000", 2024-09-16T03:37:40.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000734265 2024-09-16T03:37:40.696 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311212+0000", 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4022000000000001e-05 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311219+0000", 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6259999999999997e-06 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.356651+0000", 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.045432019999999997 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.356728+0000", 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6823000000000006e-05 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.356841+0000", 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113606 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361972+0000", 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0051310319999999998 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362101+0000", 2024-09-16T03:37:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000128777 2024-09-16T03:37:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.816692+0000", 2024-09-16T03:37:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.45459126599999999 2024-09-16T03:37:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.816734+0000", 2024-09-16T03:37:40.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1276000000000003e-05 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.816766+0000", 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2332999999999997e-05 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1023 4.0 4:06b6d050:::smithi15736696-50:head [write 2246563~579356] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.3018205320000007, 2024-09-16T03:37:40.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60727534699999997, 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1023 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:37:40.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311233+0000", 2024-09-16T03:37:40.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1039999999999999e-06 2024-09-16T03:37:40.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311692+0000", 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045908800000000002 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311704+0000", 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1844e-05 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311713+0000", 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2500000000000006e-06 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359204+0000", 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.047491646999999998 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359258+0000", 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4082000000000002e-05 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359354+0000", 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5686999999999994e-05 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512245+0000", 2024-09-16T03:37:40.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15289135700000001 2024-09-16T03:37:40.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512956+0000", 2024-09-16T03:37:40.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071110300000000004 2024-09-16T03:37:40.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918361+0000", 2024-09-16T03:37:40.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40540489400000002 2024-09-16T03:37:40.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.706 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918455+0000", 2024-09-16T03:37:40.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3412999999999995e-05 2024-09-16T03:37:40.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.706 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918508+0000", 2024-09-16T03:37:40.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:37:40.706 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.706 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.706 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.707 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1024 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3176490] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:40.707 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:37:40.707 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.301327551, 2024-09-16T03:37:40.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60701227000000002, 2024-09-16T03:37:40.707 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.707 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.707 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.707 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.707 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.707 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.707 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1024 2024-09-16T03:37:40.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.708 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:37:40.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:37:40.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311726+0000", 2024-09-16T03:37:40.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.02e-07 2024-09-16T03:37:40.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311730+0000", 2024-09-16T03:37:40.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1099999999999996e-06 2024-09-16T03:37:40.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311735+0000", 2024-09-16T03:37:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4680000000000003e-06 2024-09-16T03:37:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311739+0000", 2024-09-16T03:37:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9380000000000002e-06 2024-09-16T03:37:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361468+0000", 2024-09-16T03:37:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.049729450000000001 2024-09-16T03:37:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361539+0000", 2024-09-16T03:37:40.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0896999999999996e-05 2024-09-16T03:37:40.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:40.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361661+0000", 2024-09-16T03:37:40.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000122404 2024-09-16T03:37:40.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512943+0000", 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.151281481 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512959+0000", 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918588+0000", 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40562815499999999 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.712 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918646+0000", 2024-09-16T03:37:40.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8566999999999997e-05 2024-09-16T03:37:40.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918737+0000", 2024-09-16T03:37:40.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1308000000000001e-05 2024-09-16T03:37:40.713 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.713 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.713 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.713 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1025 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.3013090530000007, 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60723649599999996, 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1025 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9400000000000001e-07 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311746+0000", 2024-09-16T03:37:40.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2670000000000001e-06 2024-09-16T03:37:40.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311752+0000", 2024-09-16T03:37:40.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6389999999999997e-06 2024-09-16T03:37:40.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311757+0000", 2024-09-16T03:37:40.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3639999999999998e-06 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362018+0000", 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.050260969000000003 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362089+0000", 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0735000000000006e-05 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918781+0000", 2024-09-16T03:37:40.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55669215400000005 2024-09-16T03:37:40.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918867+0000", 2024-09-16T03:37:40.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6019000000000004e-05 2024-09-16T03:37:40.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918980+0000", 2024-09-16T03:37:40.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011295500000000001 2024-09-16T03:37:40.718 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.718 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1054 4.10 4:0a55bbf2:::smithi15736696-18:head [write 2633066~610652] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.0998158450000002, 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51115411600000005, 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.719 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1054 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513238+0000", 2024-09-16T03:37:40.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1999999999999998e-07 2024-09-16T03:37:40.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513695+0000", 2024-09-16T03:37:40.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045695099999999998 2024-09-16T03:37:40.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513720+0000", 2024-09-16T03:37:40.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.567e-05 2024-09-16T03:37:40.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513725+0000", 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9899999999999997e-06 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613892+0000", 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166253 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613898+0000", 2024-09-16T03:37:40.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8710000000000002e-06 2024-09-16T03:37:40.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916870+0000", 2024-09-16T03:37:40.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30297253800000001 2024-09-16T03:37:40.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916934+0000", 2024-09-16T03:37:40.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3723e-05 2024-09-16T03:37:40.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.917033+0000", 2024-09-16T03:37:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8908999999999998e-05 2024-09-16T03:37:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921278+0000", 2024-09-16T03:37:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0042456999999999998 2024-09-16T03:37:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019617+0000", 2024-09-16T03:37:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098338910000000002 2024-09-16T03:37:40.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024269+0000", 2024-09-16T03:37:40.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046515519999999998 2024-09-16T03:37:40.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024358+0000", 2024-09-16T03:37:40.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9096000000000002e-05 2024-09-16T03:37:40.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024391+0000", 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3033e-05 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1055 4.10 4:0a55bbf2:::smithi15736696-18:head [setxattr _header (56) in=7b,truncate 3243718] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.0993163450000001, 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61098759599999997, 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:40.726 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1055 2024-09-16T03:37:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:37:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:37:40.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513737+0000", 2024-09-16T03:37:40.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9399999999999993e-07 2024-09-16T03:37:40.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513759+0000", 2024-09-16T03:37:40.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1903999999999999e-05 2024-09-16T03:37:40.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513773+0000", 2024-09-16T03:37:40.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3244000000000001e-05 2024-09-16T03:37:40.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513780+0000", 2024-09-16T03:37:40.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999999e-06 2024-09-16T03:37:40.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714016+0000", 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20023635000000001 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714024+0000", 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8359999999999992e-06 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918736+0000", 2024-09-16T03:37:40.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471236700000001 2024-09-16T03:37:40.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918830+0000", 2024-09-16T03:37:40.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3769000000000003e-05 2024-09-16T03:37:40.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:37:40.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919003+0000", 2024-09-16T03:37:40.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000173198 2024-09-16T03:37:40.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921589+0000", 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002586419 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019655+0000", 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098065801999999994 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124538+0000", 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10488272899999999 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:40.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124635+0000", 2024-09-16T03:37:40.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7125000000000003e-05 2024-09-16T03:37:40.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124724+0000", 2024-09-16T03:37:40.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8858999999999997e-05 2024-09-16T03:37:40.733 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.733 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.733 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.733 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1056 4.10 4:0a55bbf2:::smithi15736696-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.0992637740000006, 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61111632800000004, 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1056 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2499999999999995e-07 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513792+0000", 2024-09-16T03:37:40.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:37:40.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513797+0000", 2024-09-16T03:37:40.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4940000000000003e-06 2024-09-16T03:37:40.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513802+0000", 2024-09-16T03:37:40.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5390000000000001e-06 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919221+0000", 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40541913899999998 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919268+0000", 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7027000000000002e-05 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124766+0000", 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20549842099999999 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124828+0000", 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1803000000000004e-05 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124905+0000", 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7163000000000006e-05 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.738 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.739 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.739 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.739 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1219 4.f 4:f817def9:::smithi15736696-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:37:40.739 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.370389+0000", 2024-09-16T03:37:40.739 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.2426638250000002, 2024-09-16T03:37:40.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50632193400000003, 2024-09-16T03:37:40.739 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:40.739 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:40.739 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:40.739 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:40.739 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:40.739 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:40.739 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1219 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.370389+0000", 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.370389+0000", 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:40.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.370390+0000", 2024-09-16T03:37:40.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4000000000000001e-07 2024-09-16T03:37:40.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:40.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.370393+0000", 2024-09-16T03:37:40.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.45e-06 2024-09-16T03:37:40.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:40.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.370400+0000", 2024-09-16T03:37:40.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0840000000000003e-06 2024-09-16T03:37:40.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.370406+0000", 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0229999999999998e-06 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.775357+0000", 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40495086400000002 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.775448+0000", 2024-09-16T03:37:40.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0650999999999997e-05 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.876590+0000", 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10114192800000001 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.876636+0000", 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6607000000000003e-05 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:40.743 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.876711+0000", 2024-09-16T03:37:40.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4686999999999998e-05 2024-09-16T03:37:40.744 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.744 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.744 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.744 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:40.744 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:40.744 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:40.746 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:37:40.746 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:40.744+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:40.841 INFO:tasks.rados.rados.0.smithi157.stdout:808: set_chunk oid 8 offset: 718336 length: 522752 target oid 82b3c9b1e88165e9d738e7d957ec67a45e2266030d9cac33aacbe86f8360ecac offset: 0 snap -1 2024-09-16T03:37:40.841 INFO:tasks.rados.rados.0.smithi157.stdout:809: write (excl) oid 13 current snap is 0 2024-09-16T03:37:40.841 INFO:tasks.rados.rados.0.smithi157.stdout:788: got (0) Success 2024-09-16T03:37:40.842 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:37:40.842 INFO:tasks.rados.rados.0.smithi157.stdout:798: got (2) No such file or directory 2024-09-16T03:37:40.842 INFO:tasks.rados.rados.0.smithi157.stdout:798: got expected ENOENT (src dne) 2024-09-16T03:37:40.842 INFO:tasks.rados.rados.0.smithi157.stdout:797: finishing set_chunk to oid 26 2024-09-16T03:37:40.842 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 26 set_chunk returned error code -95 offset: 1059328 length: 695808 2024-09-16T03:37:40.842 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 0 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T03:37:40.842 INFO:tasks.rados.rados.0.smithi157.stdout:802: expect (ObjNum 75 snap 0 seq_num 75) 2024-09-16T03:37:40.842 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:40.847 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:40.844+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:40.853 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:40.853 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:40.853 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:37:40.853 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:40.861 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:37:40.947 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:40.946+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:40.972 INFO:tasks.rados.rados.0.smithi157.stdout:801: finishing write tid 1 to smithi15736696-39 2024-09-16T03:37:40.972 INFO:tasks.rados.rados.0.smithi157.stdout:801: finishing write tid 2 to smithi15736696-39 2024-09-16T03:37:40.972 INFO:tasks.rados.rados.0.smithi157.stdout:801: finishing write tid 3 to smithi15736696-39 2024-09-16T03:37:40.972 INFO:tasks.rados.rados.0.smithi157.stdout:801: finishing write tid 4 to smithi15736696-39 2024-09-16T03:37:40.972 INFO:tasks.rados.rados.0.smithi157.stdout:801: finishing write tid 5 to smithi15736696-39 2024-09-16T03:37:40.972 INFO:tasks.rados.rados.0.smithi157.stdout:801: oid 39 updating version 0 to 38 2024-09-16T03:37:40.972 INFO:tasks.rados.rados.0.smithi157.stdout:801: oid 39 version 38 is already newer than 36 2024-09-16T03:37:40.972 INFO:tasks.rados.rados.0.smithi157.stdout:801: oid 39 version 38 is already newer than 35 2024-09-16T03:37:40.972 INFO:tasks.rados.rados.0.smithi157.stdout:801: oid 39 version 38 is already newer than 37 2024-09-16T03:37:40.972 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 38 (ObjNum 103 snap 0 seq_num 103) dirty exists 2024-09-16T03:37:40.972 INFO:tasks.rados.rados.0.smithi157.stdout:801: left oid 39 (ObjNum 103 snap 0 seq_num 103) 2024-09-16T03:37:40.972 INFO:tasks.rados.rados.0.smithi157.stdout:809: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-09-16T03:37:40.977 INFO:tasks.rados.rados.0.smithi157.stdout:809: writing smithi15736696-13 from 564735 to 1178469 tid 1 2024-09-16T03:37:40.985 INFO:tasks.rados.rados.0.smithi157.stdout:809: writing smithi15736696-13 from 1828021 to 2497055 tid 2 2024-09-16T03:37:40.992 INFO:tasks.rados.rados.0.smithi157.stdout:809: writing smithi15736696-13 from 3134120 to 3764735 tid 3 2024-09-16T03:37:40.992 INFO:tasks.rados.rados.0.smithi157.stdout:788: done (13 left) 2024-09-16T03:37:40.992 INFO:tasks.rados.rados.0.smithi157.stdout:797: done (12 left) 2024-09-16T03:37:40.992 INFO:tasks.rados.rados.0.smithi157.stdout:798: done (11 left) 2024-09-16T03:37:40.992 INFO:tasks.rados.rados.0.smithi157.stdout:799: done (10 left) 2024-09-16T03:37:40.992 INFO:tasks.rados.rados.0.smithi157.stdout:801: done (9 left) 2024-09-16T03:37:40.993 INFO:tasks.rados.rados.0.smithi157.stdout:802: done (8 left) 2024-09-16T03:37:40.993 INFO:tasks.rados.rados.0.smithi157.stdout:810: write (excl) oid 29 current snap is 0 2024-09-16T03:37:40.993 INFO:tasks.rados.rados.0.smithi157.stdout:810: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-09-16T03:37:40.998 INFO:tasks.rados.rados.0.smithi157.stdout:810: writing smithi15736696-29 from 581542 to 1270525 tid 1 2024-09-16T03:37:41.004 INFO:tasks.rados.rados.0.smithi157.stdout:810: writing smithi15736696-29 from 1886503 to 2299195 tid 2 2024-09-16T03:37:41.009 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:41.010 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:41.010 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:41.010 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:41.010 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:41.011 INFO:tasks.rados.rados.0.smithi157.stdout:810: writing smithi15736696-29 from 2761543 to 3432494 tid 3 2024-09-16T03:37:41.011 INFO:tasks.rados.rados.0.smithi157.stdout:811: write oid 6 current snap is 0 2024-09-16T03:37:41.011 INFO:tasks.rados.rados.0.smithi157.stdout:811: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2024-09-16T03:37:41.016 INFO:tasks.rados.rados.0.smithi157.stdout:811: writing smithi15736696-6 from 598349 to 1078934 tid 1 2024-09-16T03:37:41.019 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:37:41.022 INFO:tasks.rados.rados.0.smithi157.stdout:811: writing smithi15736696-6 from 1544985 to 2217688 tid 2 2024-09-16T03:37:41.028 INFO:tasks.rados.rados.0.smithi157.stdout:811: writing smithi15736696-6 from 2905319 to 3387542 tid 3 2024-09-16T03:37:41.028 INFO:tasks.rados.rados.0.smithi157.stdout:812: read oid 45 snap -1 2024-09-16T03:37:41.028 INFO:tasks.rados.rados.0.smithi157.stdout:812: expect (ObjNum 85 snap 0 seq_num 85) 2024-09-16T03:37:41.028 INFO:tasks.rados.rados.0.smithi157.stdout:803: expect (ObjNum 82 snap 0 seq_num 82) 2024-09-16T03:37:41.029 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:41.048 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:41.046+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:41.126 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:41.126 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:41.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:41.126 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:41.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.126 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:41.126 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:37:41.126 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.979778824, 2024-09-16T03:37:41.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:37:41.126 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.127 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.127 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.127 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:37:41.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:37:41.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:37:41.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:37:41.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:37:41.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:37:41.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:37:41.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:37:41.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:37:41.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:37:41.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:37:41.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:37:41.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:37:41.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:37:41.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:37:41.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:37:41.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:37:41.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:37:41.131 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.131 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.131 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.132 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:41.132 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:37:41.132 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.979425463999998, 2024-09-16T03:37:41.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:37:41.132 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.132 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.132 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.132 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:37:41.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:37:41.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:37:41.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:37:41.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:37:41.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:37:41.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:37:41.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:37:41.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:37:41.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:37:41.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:37:41.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:37:41.136 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.136 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.136 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.136 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:37:41.137 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:37:41.137 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.978297666, 2024-09-16T03:37:41.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:37:41.137 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.137 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.137 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.137 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:37:41.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:37:41.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.139 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.139 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:37:41.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:37:41.139 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.139 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.139 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:37:41.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:37:41.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:37:41.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:37:41.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:37:41.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:37:41.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:37:41.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:37:41.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:37:41.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:37:41.142 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.142 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.143 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.143 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:37:41.143 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:37:41.143 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.977310225, 2024-09-16T03:37:41.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:37:41.144 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.144 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.144 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.144 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:37:41.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.145 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:37:41.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.145 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:37:41.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:37:41.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:37:41.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:37:41.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:37:41.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:37:41.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:37:41.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:37:41.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:37:41.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:37:41.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:37:41.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:37:41.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:37:41.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:37:41.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:37:41.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:37:41.150 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.150 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.150 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.151 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:37:41.151 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:37:41.151 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.976553481, 2024-09-16T03:37:41.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:37:41.151 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.151 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.151 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.151 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:37:41.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:37:41.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:37:41.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:37:41.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:37:41.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:37:41.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:37:41.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:37:41.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:37:41.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:37:41.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:37:41.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:37:41.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:37:41.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:37:41.156 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.156 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.156 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.156 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:86 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'8, mlcod=39'5)", 2024-09-16T03:37:41.157 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:37:41.157 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.515041244999999, 2024-09-16T03:37:41.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20388281899999999, 2024-09-16T03:37:41.157 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.157 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.157 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.157 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:37:41.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608472+0000", 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.227e-06 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608616+0000", 2024-09-16T03:37:41.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000144177 2024-09-16T03:37:41.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608638+0000", 2024-09-16T03:37:41.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1543e-05 2024-09-16T03:37:41.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608645+0000", 2024-09-16T03:37:41.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5839999999999997e-06 2024-09-16T03:37:41.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608740+0000", 2024-09-16T03:37:41.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4716999999999995e-05 2024-09-16T03:37:41.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608763+0000", 2024-09-16T03:37:41.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2671e-05 2024-09-16T03:37:41.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812264+0000", 2024-09-16T03:37:41.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203501194 2024-09-16T03:37:41.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812353+0000", 2024-09-16T03:37:41.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8706000000000001e-05 2024-09-16T03:37:41.161 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.161 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.161 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.161 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:87 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'9, mlcod=39'5)", 2024-09-16T03:37:41.161 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:37:41.161 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.514827871000001, 2024-09-16T03:37:41.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20372206400000001, 2024-09-16T03:37:41.162 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.162 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.162 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.162 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:37:41.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:37:41.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608684+0000", 2024-09-16T03:37:41.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6700000000000003e-07 2024-09-16T03:37:41.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608689+0000", 2024-09-16T03:37:41.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.412e-06 2024-09-16T03:37:41.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608704+0000", 2024-09-16T03:37:41.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4397e-05 2024-09-16T03:37:41.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608713+0000", 2024-09-16T03:37:41.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:37:41.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710025+0000", 2024-09-16T03:37:41.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101312032 2024-09-16T03:37:41.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710042+0000", 2024-09-16T03:37:41.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6889e-05 2024-09-16T03:37:41.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812361+0000", 2024-09-16T03:37:41.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10231889600000001 2024-09-16T03:37:41.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812405+0000", 2024-09-16T03:37:41.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4101999999999999e-05 2024-09-16T03:37:41.166 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.166 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.166 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.166 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:101 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'10, mlcod=39'5)", 2024-09-16T03:37:41.167 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:37:41.167 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.513814434, 2024-09-16T03:37:41.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20275586900000001, 2024-09-16T03:37:41.167 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.167 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.167 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.168 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:37:41.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:37:41.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609698+0000", 2024-09-16T03:37:41.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.24e-06 2024-09-16T03:37:41.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609892+0000", 2024-09-16T03:37:41.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019465899999999999 2024-09-16T03:37:41.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609904+0000", 2024-09-16T03:37:41.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2031000000000001e-05 2024-09-16T03:37:41.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609912+0000", 2024-09-16T03:37:41.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7160000000000003e-06 2024-09-16T03:37:41.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710228+0000", 2024-09-16T03:37:41.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100315682 2024-09-16T03:37:41.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710242+0000", 2024-09-16T03:37:41.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4201e-05 2024-09-16T03:37:41.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812411+0000", 2024-09-16T03:37:41.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10216881799999999 2024-09-16T03:37:41.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812452+0000", 2024-09-16T03:37:41.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1522e-05 2024-09-16T03:37:41.174 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.174 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.174 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.174 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:102 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'11, mlcod=39'5)", 2024-09-16T03:37:41.174 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:37:41.174 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.511798409000001, 2024-09-16T03:37:41.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20134195399999999, 2024-09-16T03:37:41.174 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.174 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.174 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611714+0000", 2024-09-16T03:37:41.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.269e-06 2024-09-16T03:37:41.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611926+0000", 2024-09-16T03:37:41.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021252700000000001 2024-09-16T03:37:41.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611938+0000", 2024-09-16T03:37:41.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1437000000000001e-05 2024-09-16T03:37:41.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611944+0000", 2024-09-16T03:37:41.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.993e-06 2024-09-16T03:37:41.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811266+0000", 2024-09-16T03:37:41.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.199322306 2024-09-16T03:37:41.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811291+0000", 2024-09-16T03:37:41.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4542999999999999e-05 2024-09-16T03:37:41.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813021+0000", 2024-09-16T03:37:41.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001730363 2024-09-16T03:37:41.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813054+0000", 2024-09-16T03:37:41.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3516000000000002e-05 2024-09-16T03:37:41.180 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.180 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.180 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.180 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:103 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'12, mlcod=39'5)", 2024-09-16T03:37:41.180 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:37:41.181 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.508591908, 2024-09-16T03:37:41.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19838012999999999, 2024-09-16T03:37:41.181 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.181 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.181 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.181 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.182 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.182 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:37:41.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.182 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.182 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:37:41.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614920+0000", 2024-09-16T03:37:41.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.192e-06 2024-09-16T03:37:41.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614927+0000", 2024-09-16T03:37:41.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4640000000000003e-06 2024-09-16T03:37:41.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614937+0000", 2024-09-16T03:37:41.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9599999999999995e-06 2024-09-16T03:37:41.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614942+0000", 2024-09-16T03:37:41.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.31e-06 2024-09-16T03:37:41.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812489+0000", 2024-09-16T03:37:41.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.197546629 2024-09-16T03:37:41.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812511+0000", 2024-09-16T03:37:41.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2385e-05 2024-09-16T03:37:41.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813279+0000", 2024-09-16T03:37:41.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076818100000000005 2024-09-16T03:37:41.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813299+0000", 2024-09-16T03:37:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0009000000000001e-05 2024-09-16T03:37:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:375 4.f e45/32 4:fcb75e43:::smithi15736696-26:head v 45'23, mlcod=41'20)", 2024-09-16T03:37:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:37:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.815470681000001, 2024-09-16T03:37:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200786568, 2024-09-16T03:37:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.187 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.188 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.188 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.188 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.188 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:37:41.188 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.188 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.188 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.188 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.188 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:37:41.188 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.188 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.188 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.188 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308041+0000", 2024-09-16T03:37:41.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8299999999999995e-07 2024-09-16T03:37:41.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.189 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308046+0000", 2024-09-16T03:37:41.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.904e-06 2024-09-16T03:37:41.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.189 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308056+0000", 2024-09-16T03:37:41.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9210000000000004e-06 2024-09-16T03:37:41.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.408122+0000", 2024-09-16T03:37:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100066302 2024-09-16T03:37:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508289+0000", 2024-09-16T03:37:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166531 2024-09-16T03:37:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508302+0000", 2024-09-16T03:37:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2641e-05 2024-09-16T03:37:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508794+0000", 2024-09-16T03:37:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00049272400000000005 2024-09-16T03:37:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508827+0000", 2024-09-16T03:37:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2561999999999998e-05 2024-09-16T03:37:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:459 4.f e45/32 4:fa163101:::smithi15736696-10:head v 45'25, mlcod=45'23)", 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.318000+0000", 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.805510776999999, 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10092035100000001, 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318000+0000", 2024-09-16T03:37:41.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.193 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.193 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318000+0000", 2024-09-16T03:37:41.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.193 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.193 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318002+0000", 2024-09-16T03:37:41.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.624e-06 2024-09-16T03:37:41.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318009+0000", 2024-09-16T03:37:41.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9310000000000002e-06 2024-09-16T03:37:41.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318018+0000", 2024-09-16T03:37:41.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4949999999999997e-06 2024-09-16T03:37:41.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318031+0000", 2024-09-16T03:37:41.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2869e-05 2024-09-16T03:37:41.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418228+0000", 2024-09-16T03:37:41.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10019649799999999 2024-09-16T03:37:41.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418286+0000", 2024-09-16T03:37:41.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8904999999999999e-05 2024-09-16T03:37:41.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.197 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.197 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418884+0000", 2024-09-16T03:37:41.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059763899999999996 2024-09-16T03:37:41.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418921+0000", 2024-09-16T03:37:41.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6390000000000002e-05 2024-09-16T03:37:41.198 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.199 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.199 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.199 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:509 4.f e45/32 4:fa163101:::smithi15736696-10:head v 45'26, mlcod=45'24)", 2024-09-16T03:37:41.200 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:22.024930+0000", 2024-09-16T03:37:41.200 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.098581246999998, 2024-09-16T03:37:41.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10084305, 2024-09-16T03:37:41.200 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.200 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.201 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.201 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.201 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024930+0000", 2024-09-16T03:37:41.201 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.202 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.202 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.202 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024930+0000", 2024-09-16T03:37:41.202 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024932+0000", 2024-09-16T03:37:41.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.863e-06 2024-09-16T03:37:41.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.204 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.204 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024942+0000", 2024-09-16T03:37:41.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9960000000000005e-06 2024-09-16T03:37:41.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024954+0000", 2024-09-16T03:37:41.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2128e-05 2024-09-16T03:37:41.205 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024962+0000", 2024-09-16T03:37:41.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4209999999999995e-06 2024-09-16T03:37:41.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125116+0000", 2024-09-16T03:37:41.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015399899999999 2024-09-16T03:37:41.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.207 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.207 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125160+0000", 2024-09-16T03:37:41.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4359e-05 2024-09-16T03:37:41.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.207 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.207 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125734+0000", 2024-09-16T03:37:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00057321599999999998 2024-09-16T03:37:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125773+0000", 2024-09-16T03:37:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9067999999999999e-05 2024-09-16T03:37:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:989 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'31, mlcod=46'28)", 2024-09-16T03:37:41.208 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:37:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.735702120999999, 2024-09-16T03:37:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40332266900000002, 2024-09-16T03:37:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:37:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387812+0000", 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388467+0000", 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065539500000000002 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388481+0000", 2024-09-16T03:37:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3905e-05 2024-09-16T03:37:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488552+0000", 2024-09-16T03:37:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10007120899999999 2024-09-16T03:37:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688818+0000", 2024-09-16T03:37:41.212 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20026627599999999 2024-09-16T03:37:41.212 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.212 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.212 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.212 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688845+0000", 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791091+0000", 2024-09-16T03:37:41.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10224588699999999 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791131+0000", 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0293000000000002e-05 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:990 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'32, mlcod=46'28)", 2024-09-16T03:37:41.214 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:37:41.215 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.634918449000001, 2024-09-16T03:37:41.215 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30256419200000001, 2024-09-16T03:37:41.215 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.215 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.215 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.215 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.215 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.215 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.215 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:37:41.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.216 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.216 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.216 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:37:41.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.216 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.216 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.216 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488594+0000", 2024-09-16T03:37:41.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:37:41.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.216 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488616+0000", 2024-09-16T03:37:41.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2419e-05 2024-09-16T03:37:41.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488627+0000", 2024-09-16T03:37:41.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1251e-05 2024-09-16T03:37:41.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588753+0000", 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012544399999999 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690679+0000", 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101926027 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690695+0000", 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6528000000000002e-05 2024-09-16T03:37:41.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.219 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.219 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.219 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791133+0000", 2024-09-16T03:37:41.219 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100438233 2024-09-16T03:37:41.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.219 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.219 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.219 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791157+0000", 2024-09-16T03:37:41.219 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-05 2024-09-16T03:37:41.219 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.220 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.220 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.220 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.220 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.220 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:991 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'33, mlcod=46'28)", 2024-09-16T03:37:41.220 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:37:41.220 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.534720952000001, 2024-09-16T03:37:41.221 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202786365, 2024-09-16T03:37:41.221 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.221 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.221 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.221 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.221 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.221 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.221 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:37:41.221 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.221 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.221 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.222 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.222 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:37:41.222 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.222 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.222 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588792+0000", 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0530000000000001e-06 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588807+0000", 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5257e-05 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588832+0000", 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5123e-05 2024-09-16T03:37:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.224 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.224 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.224 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588846+0000", 2024-09-16T03:37:41.224 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3312999999999999e-05 2024-09-16T03:37:41.224 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.225 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.225 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.225 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790810+0000", 2024-09-16T03:37:41.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20196378100000001 2024-09-16T03:37:41.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.225 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.225 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.225 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790866+0000", 2024-09-16T03:37:41.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6168999999999997e-05 2024-09-16T03:37:41.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.226 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.226 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.226 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791542+0000", 2024-09-16T03:37:41.226 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067583699999999999 2024-09-16T03:37:41.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.226 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.226 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.226 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791576+0000", 2024-09-16T03:37:41.226 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4832000000000001e-05 2024-09-16T03:37:41.226 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.226 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.226 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.227 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1072 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'35, mlcod=50'32)", 2024-09-16T03:37:41.227 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:37:41.227 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 8.8154791249999995, 2024-09-16T03:37:41.227 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.103370715, 2024-09-16T03:37:41.227 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.227 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.227 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.227 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.227 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:37:41.228 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.228 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.228 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.228 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.228 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:37:41.228 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308034+0000", 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4099999999999998e-06 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308377+0000", 2024-09-16T03:37:41.229 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034276400000000001 2024-09-16T03:37:41.230 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.230 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.230 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.230 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308415+0000", 2024-09-16T03:37:41.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7916000000000001e-05 2024-09-16T03:37:41.230 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.230 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.230 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.230 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408542+0000", 2024-09-16T03:37:41.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012673699999999 2024-09-16T03:37:41.230 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.230 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.230 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.231 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408694+0000", 2024-09-16T03:37:41.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015200399999999999 2024-09-16T03:37:41.231 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.231 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408760+0000", 2024-09-16T03:37:41.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6549000000000003e-05 2024-09-16T03:37:41.231 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.411366+0000", 2024-09-16T03:37:41.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002605324 2024-09-16T03:37:41.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.411403+0000", 2024-09-16T03:37:41.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7011000000000002e-05 2024-09-16T03:37:41.232 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.232 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.232 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.233 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.233 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1216 4.f e54/32 4:f817def9:::smithi15736696-45:head v 54'43, mlcod=54'40)", 2024-09-16T03:37:41.233 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.476715+0000", 2024-09-16T03:37:41.233 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.6467958390000002, 2024-09-16T03:37:41.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0055621179999999996, 2024-09-16T03:37:41.233 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.233 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.233 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.233 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.233 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.233 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.476715+0000", 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.476715+0000", 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.476719+0000", 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.45e-06 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.477214+0000", 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00049523199999999996 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.477231+0000", 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6883000000000001e-05 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.477239+0000", 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6090000000000001e-06 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.235 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.477306+0000", 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6709000000000001e-05 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.477328+0000", 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1942000000000001e-05 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.482233+0000", 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.004905034 2024-09-16T03:37:41.236 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.237 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.237 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.240 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.482277+0000", 2024-09-16T03:37:41.240 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4258999999999998e-05 2024-09-16T03:37:41.240 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.241 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.241 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.241 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.241 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.241 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1217 4.f e54/32 4:f817def9:::smithi15736696-45:head v 54'44, mlcod=54'40)", 2024-09-16T03:37:41.241 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.477261+0000", 2024-09-16T03:37:41.241 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.6462499590000004, 2024-09-16T03:37:41.241 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0055452590000000003, 2024-09-16T03:37:41.241 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.241 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.241 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.241 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.241 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.477261+0000", 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.477261+0000", 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.477262+0000", 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9999999999999996e-07 2024-09-16T03:37:41.242 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.477538+0000", 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027594099999999998 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.477547+0000", 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8959999999999992e-06 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.477554+0000", 2024-09-16T03:37:41.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6290000000000001e-06 2024-09-16T03:37:41.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.244 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.478783+0000", 2024-09-16T03:37:41.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001228266 2024-09-16T03:37:41.245 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.245 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.245 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.478801+0000", 2024-09-16T03:37:41.246 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8051e-05 2024-09-16T03:37:41.246 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.246 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.246 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.246 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.482773+0000", 2024-09-16T03:37:41.247 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0039726780000000003 2024-09-16T03:37:41.247 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.247 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.247 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.247 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.482806+0000", 2024-09-16T03:37:41.247 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2997999999999998e-05 2024-09-16T03:37:41.248 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.248 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.248 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.248 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.248 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.249 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1218 4.f e54/32 4:f817def9:::smithi15736696-45:head v 54'45, mlcod=54'41)", 2024-09-16T03:37:41.249 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.575624+0000", 2024-09-16T03:37:41.249 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.5478864760000004, 2024-09-16T03:37:41.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100734039, 2024-09-16T03:37:41.249 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:41.250 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:41.250 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.250 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:41.250 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:41.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.575624+0000", 2024-09-16T03:37:41.251 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.251 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:41.251 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.575624+0000", 2024-09-16T03:37:41.251 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:41.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.252 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:41.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.575626+0000", 2024-09-16T03:37:41.253 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7150000000000001e-06 2024-09-16T03:37:41.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:41.253 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.575634+0000", 2024-09-16T03:37:41.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0129999999999993e-06 2024-09-16T03:37:41.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.254 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.254 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:41.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.575646+0000", 2024-09-16T03:37:41.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1615e-05 2024-09-16T03:37:41.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.255 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.575654+0000", 2024-09-16T03:37:41.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2930000000000001e-06 2024-09-16T03:37:41.256 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.256 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.256 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:41.256 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.675781+0000", 2024-09-16T03:37:41.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100126907 2024-09-16T03:37:41.257 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:41.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.675797+0000", 2024-09-16T03:37:41.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6036e-05 2024-09-16T03:37:41.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.258 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.258 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:41.258 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.676320+0000", 2024-09-16T03:37:41.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00052342299999999997 2024-09-16T03:37:41.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:41.259 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:41.259 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:41.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.676358+0000", 2024-09-16T03:37:41.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8037000000000002e-05 2024-09-16T03:37:41.259 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.259 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.259 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.259 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:41.259 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:41.259 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:41.262 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:37:41.262 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:41.259+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:41.264 INFO:tasks.rados.rados.0.smithi157.stdout:800: finishing write tid 1 to smithi15736696-20 2024-09-16T03:37:41.264 INFO:tasks.rados.rados.0.smithi157.stdout:800: finishing write tid 2 to smithi15736696-20 2024-09-16T03:37:41.264 INFO:tasks.rados.rados.0.smithi157.stdout:813: read oid 35 snap -1 2024-09-16T03:37:41.264 INFO:tasks.rados.rados.0.smithi157.stdout:813: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:37:41.264 INFO:tasks.rados.rados.0.smithi157.stdout:803: done (11 left) 2024-09-16T03:37:41.264 INFO:tasks.rados.rados.0.smithi157.stdout:814: read oid 16 snap -1 2024-09-16T03:37:41.264 INFO:tasks.rados.rados.0.smithi157.stdout:814: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-16T03:37:41.264 INFO:tasks.rados.rados.0.smithi157.stdout:815: read oid 44 snap -1 2024-09-16T03:37:41.264 INFO:tasks.rados.rados.0.smithi157.stdout:815: expect (ObjNum 63 snap 0 seq_num 63) 2024-09-16T03:37:41.265 INFO:tasks.rados.rados.0.smithi157.stdout:816: write oid 21 current snap is 0 2024-09-16T03:37:41.265 INFO:tasks.rados.rados.0.smithi157.stdout:800: finishing write tid 3 to smithi15736696-20 2024-09-16T03:37:41.265 INFO:tasks.rados.rados.0.smithi157.stdout:800: finishing write tid 4 to smithi15736696-20 2024-09-16T03:37:41.265 INFO:tasks.rados.rados.0.smithi157.stdout:816: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2024-09-16T03:37:41.265 INFO:tasks.rados.rados.0.smithi157.stdout:816: writing smithi15736696-21 from 615156 to 1170990 tid 1 2024-09-16T03:37:41.265 INFO:tasks.rados.rados.0.smithi157.stdout:816: writing smithi15736696-21 from 1603467 to 2019828 tid 2 2024-09-16T03:37:41.265 INFO:tasks.rados.rados.0.smithi157.stdout:816: writing smithi15736696-21 from 2816389 to 3509884 tid 3 2024-09-16T03:37:41.265 INFO:tasks.rados.rados.0.smithi157.stdout:817: read oid 9 snap -1 2024-09-16T03:37:41.265 INFO:tasks.rados.rados.0.smithi157.stdout:817: expect (ObjNum 55 snap 0 seq_num 55) 2024-09-16T03:37:41.266 INFO:tasks.rados.rados.0.smithi157.stdout:818: delete oid 34 current snap is 0 2024-09-16T03:37:41.266 INFO:tasks.rados.rados.0.smithi157.stdout:800: finishing write tid 5 to smithi15736696-20 2024-09-16T03:37:41.266 INFO:tasks.rados.rados.0.smithi157.stdout:800: oid 20 updating version 0 to 30 2024-09-16T03:37:41.266 INFO:tasks.rados.rados.0.smithi157.stdout:800: oid 20 updating version 30 to 31 2024-09-16T03:37:41.266 INFO:tasks.rados.rados.0.smithi157.stdout:800: oid 20 version 31 is already newer than 29 2024-09-16T03:37:41.266 INFO:tasks.rados.rados.0.smithi157.stdout:800: oid 20 updating version 31 to 32 2024-09-16T03:37:41.266 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 32 (ObjNum 102 snap 0 seq_num 102) dirty exists 2024-09-16T03:37:41.266 INFO:tasks.rados.rados.0.smithi157.stdout:800: left oid 20 (ObjNum 102 snap 0 seq_num 102) 2024-09-16T03:37:41.267 INFO:tasks.rados.rados.0.smithi157.stdout:805: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-16T03:37:41.296 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2593977 2024-09-16T03:37:41.356 INFO:tasks.rados.rados.0.smithi157.stdout:807: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T03:37:41.362 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:41.360+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:41.373 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:37:41.373 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:41.382 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:37:41.388 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1236901 2024-09-16T03:37:41.404 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1394113 2024-09-16T03:37:41.419 INFO:tasks.rados.rados.0.smithi157.stdout:806: finishing write tid 1 to smithi15736696-50 2024-09-16T03:37:41.419 INFO:tasks.rados.rados.0.smithi157.stdout:806: finishing write tid 2 to smithi15736696-50 2024-09-16T03:37:41.419 INFO:tasks.rados.rados.0.smithi157.stdout:800: done (15 left) 2024-09-16T03:37:41.419 INFO:tasks.rados.rados.0.smithi157.stdout:805: done (14 left) 2024-09-16T03:37:41.419 INFO:tasks.rados.rados.0.smithi157.stdout:807: done (13 left) 2024-09-16T03:37:41.419 INFO:tasks.rados.rados.0.smithi157.stdout:818: done (12 left) 2024-09-16T03:37:41.420 INFO:tasks.rados.rados.0.smithi157.stdout:819: write (excl) oid 37 current snap is 0 2024-09-16T03:37:41.420 INFO:tasks.rados.rados.0.smithi157.stdout:806: finishing write tid 3 to smithi15736696-50 2024-09-16T03:37:41.420 INFO:tasks.rados.rados.0.smithi157.stdout:806: finishing write tid 4 to smithi15736696-50 2024-09-16T03:37:41.420 INFO:tasks.rados.rados.0.smithi157.stdout:819: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-09-16T03:37:41.426 INFO:tasks.rados.rados.0.smithi157.stdout:819: writing smithi15736696-37 from 648770 to 1355102 tid 1 2024-09-16T03:37:41.432 INFO:tasks.rados.rados.0.smithi157.stdout:819: writing smithi15736696-37 from 2120431 to 2540461 tid 2 2024-09-16T03:37:41.439 INFO:tasks.rados.rados.0.smithi157.stdout:819: writing smithi15736696-37 from 3271235 to 3703627 tid 3 2024-09-16T03:37:41.439 INFO:tasks.rados.rados.0.smithi157.stdout:820: set_chunk oid 19 target oid 2024-09-16T03:37:41.462 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:41.460+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:41.491 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:41.491 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:41.491 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:37:41.491 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:37:41.491 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:41.500 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:37:41.563 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:41.560+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:41.647 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:41.647 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:37:41.648 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:37:41.648 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:41.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.648 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:14 4.a e39/32 4:51338165:::smithi15736696-4:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:41.648 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:11.997261+0000", 2024-09-16T03:37:41.648 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.647641586999999, 2024-09-16T03:37:41.648 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185554500000001, 2024-09-16T03:37:41.648 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.648 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.648 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.648 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.648 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997261+0000", 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997261+0000", 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997263+0000", 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.373e-06 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.649 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.650 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.650 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997764+0000", 2024-09-16T03:37:41.650 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00050155299999999996 2024-09-16T03:37:41.650 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.650 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.650 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.650 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997775+0000", 2024-09-16T03:37:41.650 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1066999999999999e-05 2024-09-16T03:37:41.650 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.650 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097841+0000", 2024-09-16T03:37:41.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100066316 2024-09-16T03:37:41.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097882+0000", 2024-09-16T03:37:41.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0850999999999999e-05 2024-09-16T03:37:41.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097904+0000", 2024-09-16T03:37:41.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1347000000000001e-05 2024-09-16T03:37:41.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.099089+0000", 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001185195 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.099117+0000", 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7843000000000001e-05 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.4.22:3 3.1 e41/31 3:bffb5658:::smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'22, mlcod=41'20)", 2024-09-16T03:37:41.654 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:15.504663+0000", 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 26.140239937, 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100603585, 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.504663+0000", 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.504663+0000", 2024-09-16T03:37:41.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.504664+0000", 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1969999999999999e-06 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.504678+0000", 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3793e-05 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.504688+0000", 2024-09-16T03:37:41.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.01e-05 2024-09-16T03:37:41.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.604761+0000", 2024-09-16T03:37:41.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100072681 2024-09-16T03:37:41.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.604805+0000", 2024-09-16T03:37:41.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4035000000000001e-05 2024-09-16T03:37:41.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.604821+0000", 2024-09-16T03:37:41.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6095000000000001e-05 2024-09-16T03:37:41.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.605217+0000", 2024-09-16T03:37:41.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00039639200000000002 2024-09-16T03:37:41.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.605266+0000", 2024-09-16T03:37:41.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9292000000000002e-05 2024-09-16T03:37:41.659 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.659 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.659 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.659 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:276 4.a e41/32 4:50107449:::smithi15736696-27:head v 41'20, mlcod=39'18)", 2024-09-16T03:37:41.659 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:15.701824+0000", 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.943078267000001, 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10067820700000001, 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.701824+0000", 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.660 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.701824+0000", 2024-09-16T03:37:41.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.701826+0000", 2024-09-16T03:37:41.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.232e-06 2024-09-16T03:37:41.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.701832+0000", 2024-09-16T03:37:41.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9479999999999996e-06 2024-09-16T03:37:41.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.701843+0000", 2024-09-16T03:37:41.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0987e-05 2024-09-16T03:37:41.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.701848+0000", 2024-09-16T03:37:41.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2809999999999999e-06 2024-09-16T03:37:41.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.801996+0000", 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100148312 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.802032+0000", 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5642000000000002e-05 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.802470+0000", 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043790499999999999 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.802503+0000", 2024-09-16T03:37:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.29e-05 2024-09-16T03:37:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.15.23:17 3.1 e42/31 3:86477bf9:::smithi15736696-46 low_tier:head v 42'23, mlcod=41'21)", 2024-09-16T03:37:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.187034+0000", 2024-09-16T03:37:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.457868276999999, 2024-09-16T03:37:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10100469400000001, 2024-09-16T03:37:41.664 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187034+0000", 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187034+0000", 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187035+0000", 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5400000000000001e-07 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187042+0000", 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1729999999999998e-06 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187063+0000", 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1160000000000001e-05 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187081+0000", 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7935999999999999e-05 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.287313+0000", 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100231325 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.287345+0000", 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2119000000000002e-05 2024-09-16T03:37:41.667 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.668 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.668 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.668 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.288013+0000", 2024-09-16T03:37:41.668 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00066847800000000002 2024-09-16T03:37:41.668 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.668 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.668 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.668 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.288039+0000", 2024-09-16T03:37:41.668 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5948999999999998e-05 2024-09-16T03:37:41.668 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.668 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.668 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.11.23:19 3.1 e42/31 3:9aba8983:::smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'26, mlcod=42'24)", 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.419116+0000", 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.225787232999998, 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10061766900000001, 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.419116+0000", 2024-09-16T03:37:41.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.419116+0000", 2024-09-16T03:37:41.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.419117+0000", 2024-09-16T03:37:41.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0869999999999999e-06 2024-09-16T03:37:41.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.419130+0000", 2024-09-16T03:37:41.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3256e-05 2024-09-16T03:37:41.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.671 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.419140+0000", 2024-09-16T03:37:41.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0397000000000001e-05 2024-09-16T03:37:41.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.671 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519236+0000", 2024-09-16T03:37:41.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100095583 2024-09-16T03:37:41.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.672 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.672 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519277+0000", 2024-09-16T03:37:41.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1631000000000002e-05 2024-09-16T03:37:41.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.672 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.672 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519298+0000", 2024-09-16T03:37:41.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0472e-05 2024-09-16T03:37:41.673 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.673 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.673 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.673 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519706+0000", 2024-09-16T03:37:41.673 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000408 2024-09-16T03:37:41.673 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.673 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.673 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.673 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519733+0000", 2024-09-16T03:37:41.673 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7243e-05 2024-09-16T03:37:41.673 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.673 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.673 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.5.21:10 3.1 e42/31 3:8f38b759:::smithi15736696-6 low_tier:head v 42'27, mlcod=42'24)", 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.519254+0000", 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.125649205999999, 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100789212, 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519254+0000", 2024-09-16T03:37:41.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519254+0000", 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519254+0000", 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9999999999999997e-07 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519259+0000", 2024-09-16T03:37:41.676 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6489999999999999e-06 2024-09-16T03:37:41.676 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.676 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.676 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.676 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.519264+0000", 2024-09-16T03:37:41.676 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5289999999999999e-06 2024-09-16T03:37:41.676 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.676 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.676 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.676 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619388+0000", 2024-09-16T03:37:41.676 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100123484 2024-09-16T03:37:41.676 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.676 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.677 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.677 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619501+0000", 2024-09-16T03:37:41.677 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011266500000000001 2024-09-16T03:37:41.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.677 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.677 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.677 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619536+0000", 2024-09-16T03:37:41.677 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5352999999999998e-05 2024-09-16T03:37:41.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.677 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.677 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.677 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.620019+0000", 2024-09-16T03:37:41.677 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00048265200000000003 2024-09-16T03:37:41.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.678 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.678 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.620043+0000", 2024-09-16T03:37:41.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4179999999999999e-05 2024-09-16T03:37:41.678 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.678 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.678 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.678 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.5.21:11 3.1 e42/31 3:adc537a6:::smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'28, mlcod=42'24)", 2024-09-16T03:37:41.678 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.619416+0000", 2024-09-16T03:37:41.678 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.025486607000001, 2024-09-16T03:37:41.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100981345, 2024-09-16T03:37:41.679 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.679 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.679 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.679 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619416+0000", 2024-09-16T03:37:41.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619416+0000", 2024-09-16T03:37:41.680 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.680 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.680 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.680 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.680 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619418+0000", 2024-09-16T03:37:41.680 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.99e-06 2024-09-16T03:37:41.680 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.680 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.680 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.680 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619445+0000", 2024-09-16T03:37:41.680 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6755e-05 2024-09-16T03:37:41.680 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.680 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619465+0000", 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0097000000000001e-05 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719589+0000", 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10012367799999999 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719706+0000", 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117259 2024-09-16T03:37:41.681 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.682 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.682 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.682 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719784+0000", 2024-09-16T03:37:41.682 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7714999999999997e-05 2024-09-16T03:37:41.682 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.682 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.682 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.682 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.720366+0000", 2024-09-16T03:37:41.682 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058208999999999995 2024-09-16T03:37:41.682 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.682 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.682 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.682 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.720397+0000", 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1761000000000003e-05 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:333 4.a e45/32 4:503187b3:::smithi15736696-34:head v 45'29, mlcod=44'27)", 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.003000+0000", 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 21.641903109000001, 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10086444999999999, 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.683 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.684 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003000+0000", 2024-09-16T03:37:41.684 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.684 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003000+0000", 2024-09-16T03:37:41.684 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.684 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003001+0000", 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.35e-06 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003019+0000", 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8182999999999999e-05 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003033+0000", 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4307999999999999e-05 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.685 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.686 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003040+0000", 2024-09-16T03:37:41.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4359999999999999e-06 2024-09-16T03:37:41.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.686 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.686 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.103213+0000", 2024-09-16T03:37:41.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100172795 2024-09-16T03:37:41.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.686 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.686 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.103232+0000", 2024-09-16T03:37:41.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9746999999999999e-05 2024-09-16T03:37:41.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.103822+0000", 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058940000000000002 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.103864+0000", 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2231000000000003e-05 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.688 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.5.21:15 3.1 e45/31 3:a961e47d:::smithi15736696-30 low_tier:head v 45'37, mlcod=45'35)", 2024-09-16T03:37:41.688 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.922626+0000", 2024-09-16T03:37:41.688 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 19.722276404999999, 2024-09-16T03:37:41.688 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100747954, 2024-09-16T03:37:41.688 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.688 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.688 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.688 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.688 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.688 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.688 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.922626+0000", 2024-09-16T03:37:41.688 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.688 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.922626+0000", 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.922628+0000", 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0099999999999998e-06 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.922638+0000", 2024-09-16T03:37:41.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2709999999999998e-06 2024-09-16T03:37:41.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.922653+0000", 2024-09-16T03:37:41.690 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5338000000000001e-05 2024-09-16T03:37:41.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.022758+0000", 2024-09-16T03:37:41.690 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100104733 2024-09-16T03:37:41.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.691 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.022873+0000", 2024-09-16T03:37:41.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115466 2024-09-16T03:37:41.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.691 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.691 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.691 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.022895+0000", 2024-09-16T03:37:41.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.228e-05 2024-09-16T03:37:41.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.691 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.691 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.691 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.023337+0000", 2024-09-16T03:37:41.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00044163200000000002 2024-09-16T03:37:41.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.023374+0000", 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7224000000000001e-05 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:509 4.f e45/32 4:fa163101:::smithi15736696-10:head v 45'26, mlcod=45'24)", 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:22.024903+0000", 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 19.619999352000001, 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101058093, 2024-09-16T03:37:41.692 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.693 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.693 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.693 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024903+0000", 2024-09-16T03:37:41.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024903+0000", 2024-09-16T03:37:41.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024905+0000", 2024-09-16T03:37:41.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1659999999999999e-06 2024-09-16T03:37:41.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024913+0000", 2024-09-16T03:37:41.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9840000000000001e-06 2024-09-16T03:37:41.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024923+0000", 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0811e-05 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024929+0000", 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5509999999999999e-06 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.125071+0000", 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10014179300000001 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.125095+0000", 2024-09-16T03:37:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3944999999999999e-05 2024-09-16T03:37:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.125926+0000", 2024-09-16T03:37:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00083087799999999996 2024-09-16T03:37:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.125961+0000", 2024-09-16T03:37:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5964999999999999e-05 2024-09-16T03:37:41.696 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:503 4.a e45/32 4:51338165:::smithi15736696-4:head v 45'30, mlcod=45'28)", 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:22.126147+0000", 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 19.518755608999999, 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100843981, 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.126147+0000", 2024-09-16T03:37:41.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.698 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.126147+0000", 2024-09-16T03:37:41.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.698 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.126148+0000", 2024-09-16T03:37:41.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7800000000000002e-07 2024-09-16T03:37:41.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.126156+0000", 2024-09-16T03:37:41.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0439999999999996e-06 2024-09-16T03:37:41.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.126165+0000", 2024-09-16T03:37:41.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1370000000000008e-06 2024-09-16T03:37:41.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.126172+0000", 2024-09-16T03:37:41.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.878e-06 2024-09-16T03:37:41.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.226374+0000", 2024-09-16T03:37:41.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100201992 2024-09-16T03:37:41.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.226415+0000", 2024-09-16T03:37:41.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1196000000000003e-05 2024-09-16T03:37:41.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.701 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.701 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.226965+0000", 2024-09-16T03:37:41.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00055020199999999996 2024-09-16T03:37:41.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.701 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.701 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.226991+0000", 2024-09-16T03:37:41.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6054000000000002e-05 2024-09-16T03:37:41.701 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.701 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.701 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.702 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:779 3.1 e47/31 3:abdaa5c9:::2782d12a88f9842f7f754fca32c4222cf10fb780cd136830d32faaa700e687ba:head v 47'41, mlcod=46'39)", 2024-09-16T03:37:41.702 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:25.588789+0000", 2024-09-16T03:37:41.702 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.056113818, 2024-09-16T03:37:41.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100868377, 2024-09-16T03:37:41.702 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.702 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.702 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.702 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.702 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.588789+0000", 2024-09-16T03:37:41.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.588789+0000", 2024-09-16T03:37:41.703 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.588789+0000", 2024-09-16T03:37:41.703 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1800000000000002e-07 2024-09-16T03:37:41.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.588843+0000", 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3720000000000001e-05 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.588851+0000", 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2500000000000006e-06 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.588861+0000", 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4520000000000003e-06 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.688905+0000", 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100044786 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.688923+0000", 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7408999999999999e-05 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.689629+0000", 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00070633499999999999 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.689657+0000", 2024-09-16T03:37:41.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8107e-05 2024-09-16T03:37:41.706 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.706 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.706 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.706 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.5.21:22 3.1 e47/31 3:adc537a6:::smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 47'46, mlcod=47'42)", 2024-09-16T03:37:41.706 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.686705+0000", 2024-09-16T03:37:41.706 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.958198149999999, 2024-09-16T03:37:41.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10113362200000001, 2024-09-16T03:37:41.708 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.708 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.708 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.708 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.708 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.708 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686705+0000", 2024-09-16T03:37:41.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.708 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.709 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686705+0000", 2024-09-16T03:37:41.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.709 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.709 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.709 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686705+0000", 2024-09-16T03:37:41.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1699999999999998e-07 2024-09-16T03:37:41.709 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.709 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.709 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686709+0000", 2024-09-16T03:37:41.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6279999999999998e-06 2024-09-16T03:37:41.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686716+0000", 2024-09-16T03:37:41.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3730000000000004e-06 2024-09-16T03:37:41.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.786814+0000", 2024-09-16T03:37:41.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100097964 2024-09-16T03:37:41.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.786880+0000", 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5883999999999995e-05 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.786909+0000", 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8764000000000001e-05 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.787802+0000", 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00089355100000000002 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.712 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.787838+0000", 2024-09-16T03:37:41.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5840999999999998e-05 2024-09-16T03:37:41.712 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.712 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.712 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.712 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:983 3.1 e50/31 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head v 50'54, mlcod=49'52)", 2024-09-16T03:37:41.712 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:37:41.712 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 11.578585042, 2024-09-16T03:37:41.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.105768445, 2024-09-16T03:37:41.712 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.713 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.713 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.713 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.713 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.713 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:37:41.713 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.713 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.713 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:37:41.713 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:37:41.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7700000000000001e-07 2024-09-16T03:37:41.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.070158+0000", 2024-09-16T03:37:41.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0038403170000000002 2024-09-16T03:37:41.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.070168+0000", 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9019999999999994e-06 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.070178+0000", 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6860000000000001e-06 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.170328+0000", 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150196 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.170363+0000", 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4588999999999997e-05 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.172053+0000", 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016902900000000001 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.172086+0000", 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3287999999999997e-05 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.716 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:988 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'30, mlcod=46'28)", 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.381297+0000", 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 11.263605954999999, 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10422211300000001, 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.381297+0000", 2024-09-16T03:37:41.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.381297+0000", 2024-09-16T03:37:41.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.381298+0000", 2024-09-16T03:37:41.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5409999999999999e-06 2024-09-16T03:37:41.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.382086+0000", 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078808600000000002 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.382097+0000", 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0557000000000001e-05 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.482188+0000", 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100091478 2024-09-16T03:37:41.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.482261+0000", 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2992999999999997e-05 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.482287+0000", 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5211999999999998e-05 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.485483+0000", 2024-09-16T03:37:41.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031964939999999998 2024-09-16T03:37:41.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.485519+0000", 2024-09-16T03:37:41.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5751999999999999e-05 2024-09-16T03:37:41.721 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.721 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.721 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.722 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1071 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'34, mlcod=50'32)", 2024-09-16T03:37:41.722 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:32.306208+0000", 2024-09-16T03:37:41.722 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.3386948269999994, 2024-09-16T03:37:41.722 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103114758, 2024-09-16T03:37:41.722 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.722 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.722 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.722 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.306208+0000", 2024-09-16T03:37:41.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.306208+0000", 2024-09-16T03:37:41.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.306210+0000", 2024-09-16T03:37:41.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6619999999999999e-06 2024-09-16T03:37:41.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307136+0000", 2024-09-16T03:37:41.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092664400000000001 2024-09-16T03:37:41.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307149+0000", 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2615e-05 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307159+0000", 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0341e-05 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307221+0000", 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1871000000000004e-05 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.726 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307239+0000", 2024-09-16T03:37:41.726 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7876999999999998e-05 2024-09-16T03:37:41.726 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.726 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.726 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.726 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409282+0000", 2024-09-16T03:37:41.726 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102042801 2024-09-16T03:37:41.726 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.726 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.726 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.726 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409323+0000", 2024-09-16T03:37:41.726 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0946999999999999e-05 2024-09-16T03:37:41.726 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.727 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.727 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.727 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.727 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.727 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1072 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'35, mlcod=50'32)", 2024-09-16T03:37:41.727 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:37:41.727 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.3369591639999996, 2024-09-16T03:37:41.727 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10192738899999999, 2024-09-16T03:37:41.727 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.727 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.727 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.727 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.727 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9999999999999996e-07 2024-09-16T03:37:41.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.308383+0000", 2024-09-16T03:37:41.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000438794 2024-09-16T03:37:41.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.308391+0000", 2024-09-16T03:37:41.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6149999999999999e-06 2024-09-16T03:37:41.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.730 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.308396+0000", 2024-09-16T03:37:41.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1719999999999998e-06 2024-09-16T03:37:41.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.730 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.730 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.730 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.408530+0000", 2024-09-16T03:37:41.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013438700000001 2024-09-16T03:37:41.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.730 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.730 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.730 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.408547+0000", 2024-09-16T03:37:41.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6554000000000001e-05 2024-09-16T03:37:41.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409844+0000", 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001297166 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409871+0000", 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6900999999999999e-05 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.731 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1073 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'36, mlcod=50'32)", 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:32.311182+0000", 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.3337208070000006, 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101901307, 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311182+0000", 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.732 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311182+0000", 2024-09-16T03:37:41.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.733 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311183+0000", 2024-09-16T03:37:41.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:37:41.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.733 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311720+0000", 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053692400000000004 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311728+0000", 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7849999999999992e-06 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.411763+0000", 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100035393 2024-09-16T03:37:41.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.411838+0000", 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4658000000000002e-05 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.411853+0000", 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5268e-05 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.413058+0000", 2024-09-16T03:37:41.735 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001204751 2024-09-16T03:37:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.413083+0000", 2024-09-16T03:37:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5609000000000002e-05 2024-09-16T03:37:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1193 4.a e54/32 4:503187b3:::smithi15736696-34:head v 54'43, mlcod=53'41)", 2024-09-16T03:37:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:36.160789+0000", 2024-09-16T03:37:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.4841138389999999, 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10275106000000001, 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.160789+0000", 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.160789+0000", 2024-09-16T03:37:41.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.160791+0000", 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.638e-06 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.161609+0000", 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00081820899999999997 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.738 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.161635+0000", 2024-09-16T03:37:41.739 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5939999999999999e-05 2024-09-16T03:37:41.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.739 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.739 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.739 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.261770+0000", 2024-09-16T03:37:41.739 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100135416 2024-09-16T03:37:41.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.739 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.739 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.739 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.261913+0000", 2024-09-16T03:37:41.739 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000142695 2024-09-16T03:37:41.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.739 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.740 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.740 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.261950+0000", 2024-09-16T03:37:41.740 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7324999999999999e-05 2024-09-16T03:37:41.740 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.740 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.740 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.740 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.263493+0000", 2024-09-16T03:37:41.740 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015430800000000001 2024-09-16T03:37:41.740 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.740 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.740 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.740 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.263540+0000", 2024-09-16T03:37:41.740 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6757e-05 2024-09-16T03:37:41.741 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.741 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.741 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.741 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.741 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1227 4.a e54/32 4:5a5e9bc2:::smithi15736696-36:head v 54'46, mlcod=54'44)", 2024-09-16T03:37:41.741 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:37.016658+0000", 2024-09-16T03:37:41.741 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.6282444979999999, 2024-09-16T03:37:41.741 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100632174, 2024-09-16T03:37:41.741 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:41.741 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:41.741 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:41.741 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.016658+0000", 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.016658+0000", 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.016660+0000", 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4720000000000001e-06 2024-09-16T03:37:41.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.016670+0000", 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0773e-05 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.016682+0000", 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.113e-05 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.016690+0000", 2024-09-16T03:37:41.743 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7180000000000002e-06 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.116831+0000", 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10014052499999999 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.116845+0000", 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3874e-05 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:41.744 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.117247+0000", 2024-09-16T03:37:41.745 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040270099999999999 2024-09-16T03:37:41.745 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:41.745 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:41.745 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:41.745 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.117290+0000", 2024-09-16T03:37:41.745 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2981000000000001e-05 2024-09-16T03:37:41.745 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.745 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.745 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.745 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:41.745 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:41.745 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:41.747 INFO:tasks.rados.rados.0.smithi157.stdout:820: set_chunk oid 19 offset: 875520 length: 1824768 target oid 24c51d2d1b2fce93004db569321d70341b216a4934a8a8378d9721192b2537dc offset: 0 snap -1 2024-09-16T03:37:41.747 INFO:tasks.rados.rados.0.smithi157.stdout:821: read oid 2 snap -1 2024-09-16T03:37:41.747 INFO:tasks.rados.rados.0.smithi157.stdout:821: expect (ObjNum 64 snap 0 seq_num 64) 2024-09-16T03:37:41.747 INFO:tasks.rados.rados.0.smithi157.stdout:822: read oid 17 snap -1 2024-09-16T03:37:41.748 INFO:tasks.rados.rados.0.smithi157.stdout:822: expect (ObjNum 86 snap 0 seq_num 86) 2024-09-16T03:37:41.748 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:41.748 INFO:tasks.rados.rados.0.smithi157.stdout:806: finishing write tid 5 to smithi15736696-50 2024-09-16T03:37:41.748 INFO:tasks.rados.rados.0.smithi157.stdout:806: oid 50 updating version 0 to 56 2024-09-16T03:37:41.748 INFO:tasks.rados.rados.0.smithi157.stdout:806: oid 50 version 56 is already newer than 53 2024-09-16T03:37:41.748 INFO:tasks.rados.rados.0.smithi157.stdout:806: oid 50 version 56 is already newer than 55 2024-09-16T03:37:41.748 INFO:tasks.rados.rados.0.smithi157.stdout:806: oid 50 version 56 is already newer than 54 2024-09-16T03:37:41.748 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 56 (ObjNum 104 snap 0 seq_num 104) dirty exists 2024-09-16T03:37:41.748 INFO:tasks.rados.rados.0.smithi157.stdout:806: left oid 50 (ObjNum 104 snap 0 seq_num 104) 2024-09-16T03:37:41.748 INFO:tasks.rados.rados.0.smithi157.stdout:806: done (15 left) 2024-09-16T03:37:41.748 INFO:tasks.rados.rados.0.smithi157.stdout:823: write oid 5 current snap is 0 2024-09-16T03:37:41.748 INFO:tasks.rados.rados.0.smithi157.stdout:823: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2024-09-16T03:37:41.748 INFO:tasks.rados.rados.0.smithi157.stdout:823: writing smithi15736696-5 from 665577 to 1447158 tid 1 2024-09-16T03:37:41.749 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:37:41.750 INFO:tasks.rados.rados.0.smithi157.stdout:823: writing smithi15736696-5 from 2062560 to 2626248 tid 2 2024-09-16T03:37:41.750 INFO:tasks.rados.rados.0.smithi157.stdout:823: writing smithi15736696-5 from 3182305 to 3825969 tid 3 2024-09-16T03:37:41.750 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:41.750 INFO:tasks.rados.rados.0.smithi157.stdout:804: expect (ObjNum 74 snap 0 seq_num 74) 2024-09-16T03:37:41.750 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 15783 2024-09-16T03:37:41.751 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:41.744+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:41.805 INFO:tasks.rados.rados.0.smithi157.stdout:809: finishing write tid 1 to smithi15736696-13 2024-09-16T03:37:41.805 INFO:tasks.rados.rados.0.smithi157.stdout:804: done (15 left) 2024-09-16T03:37:41.805 INFO:tasks.rados.rados.0.smithi157.stdout:824: set_chunk oid 36 target oid 2024-09-16T03:37:41.847 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:41.846+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:41.856 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:41.856 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:41.856 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:37:41.857 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:41.865 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:37:41.909 INFO:tasks.rados.rados.0.smithi157.stdout:824: set_chunk oid 36 offset: 1922048 length: 320000 target oid 1fda83a16db131b06a5bf7beb910c0624eb0c0478a30a35e6b1e31b63f6e3ff4 offset: 0 snap -1 2024-09-16T03:37:41.910 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:41.910 INFO:tasks.rados.rados.0.smithi157.stdout:810: finishing write tid 1 to smithi15736696-29 2024-09-16T03:37:41.910 INFO:tasks.rados.rados.0.smithi157.stdout:809: finishing write tid 2 to smithi15736696-13 2024-09-16T03:37:41.910 INFO:tasks.rados.rados.0.smithi157.stdout:809: finishing write tid 3 to smithi15736696-13 2024-09-16T03:37:41.910 INFO:tasks.rados.rados.0.smithi157.stdout:811: finishing write tid 1 to smithi15736696-6 2024-09-16T03:37:41.910 INFO:tasks.rados.rados.0.smithi157.stdout:811: finishing write tid 2 to smithi15736696-6 2024-09-16T03:37:41.910 INFO:tasks.rados.rados.0.smithi157.stdout:811: finishing write tid 3 to smithi15736696-6 2024-09-16T03:37:41.910 INFO:tasks.rados.rados.0.smithi157.stdout:812: expect (ObjNum 85 snap 0 seq_num 85) 2024-09-16T03:37:41.910 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:41.948 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:41.946+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:42.011 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:37:42.011 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:37:42.011 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:42.020 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:37:42.055 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3405890 2024-09-16T03:37:42.056 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:42.054+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:42.064 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:37:42.142 INFO:tasks.rados.rados.0.smithi157.stdout:808: finishing set_chunk to oid 8 2024-09-16T03:37:42.142 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 37 (ObjNum 76 snap 0 seq_num 76) dirty exists 2024-09-16T03:37:42.142 INFO:tasks.rados.rados.0.smithi157.stdout:808: done (15 left) 2024-09-16T03:37:42.142 INFO:tasks.rados.rados.0.smithi157.stdout:812: done (14 left) 2024-09-16T03:37:42.142 INFO:tasks.rados.rados.0.smithi157.stdout:825: read oid 30 snap -1 2024-09-16T03:37:42.142 INFO:tasks.rados.rados.0.smithi157.stdout:825: expect (ObjNum 98 snap 0 seq_num 98) 2024-09-16T03:37:42.142 INFO:tasks.rados.rados.0.smithi157.stdout:826: write oid 41 current snap is 0 2024-09-16T03:37:42.143 INFO:tasks.rados.rados.0.smithi157.stdout:814: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-16T03:37:42.143 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 574640 2024-09-16T03:37:42.150 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:42.148+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:42.162 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:42.162 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:37:42.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:37:42.163 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:42.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.163 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:41 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:42.163 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.156949+0000", 2024-09-16T03:37:42.163 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.002947463999998, 2024-09-16T03:37:42.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021202600000000001, 2024-09-16T03:37:42.163 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.163 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.163 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.163 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156949+0000", 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156949+0000", 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156949+0000", 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9100000000000004e-07 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157324+0000", 2024-09-16T03:37:42.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037452500000000001 2024-09-16T03:37:42.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157340+0000", 2024-09-16T03:37:42.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6426000000000002e-05 2024-09-16T03:37:42.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157350+0000", 2024-09-16T03:37:42.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6150000000000003e-06 2024-09-16T03:37:42.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157415+0000", 2024-09-16T03:37:42.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5425999999999993e-05 2024-09-16T03:37:42.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157441+0000", 2024-09-16T03:37:42.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6177999999999999e-05 2024-09-16T03:37:42.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.159020+0000", 2024-09-16T03:37:42.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001578325 2024-09-16T03:37:42.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.159069+0000", 2024-09-16T03:37:42.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9274000000000003e-05 2024-09-16T03:37:42.167 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.167 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.167 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.168 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:42.168 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:37:42.168 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.973815754, 2024-09-16T03:37:42.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102630163, 2024-09-16T03:37:42.168 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.168 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.168 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.169 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:37:42.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:37:42.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186081+0000", 2024-09-16T03:37:42.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5600000000000005e-07 2024-09-16T03:37:42.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186970+0000", 2024-09-16T03:37:42.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00088889000000000004 2024-09-16T03:37:42.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186979+0000", 2024-09-16T03:37:42.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0620000000000007e-06 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186986+0000", 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287184+0000", 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100198036 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287199+0000", 2024-09-16T03:37:42.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5458999999999999e-05 2024-09-16T03:37:42.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288659+0000", 2024-09-16T03:37:42.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014601900000000001 2024-09-16T03:37:42.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288711+0000", 2024-09-16T03:37:42.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1088000000000003e-05 2024-09-16T03:37:42.172 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.173 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.173 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.173 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:42.173 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:37:42.173 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.972779321000001, 2024-09-16T03:37:42.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10205370900000001, 2024-09-16T03:37:42.173 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.173 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.173 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.173 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187118+0000", 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8599999999999998e-07 2024-09-16T03:37:42.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188062+0000", 2024-09-16T03:37:42.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094447300000000004 2024-09-16T03:37:42.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188070+0000", 2024-09-16T03:37:42.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.695e-06 2024-09-16T03:37:42.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188074+0000", 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0709999999999996e-06 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287842+0000", 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099768478999999993 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287864+0000", 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1679999999999999e-05 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289137+0000", 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012732430000000001 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289171+0000", 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3382e-05 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.177 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:42 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:42.178 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:37:42.178 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.902697412999999, 2024-09-16T03:37:42.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10172858, 2024-09-16T03:37:42.178 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.180 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.180 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.180 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257201+0000", 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257536+0000", 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033513899999999998 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257555+0000", 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8547999999999999e-05 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257566+0000", 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1379999999999999e-05 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357720+0000", 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100153532 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357734+0000", 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4491000000000001e-05 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358896+0000", 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011620599999999999 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358927+0000", 2024-09-16T03:37:42.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1158999999999998e-05 2024-09-16T03:37:42.184 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.184 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.184 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.184 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:43 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:42.184 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:37:42.184 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.900913501000002, 2024-09-16T03:37:42.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101764453, 2024-09-16T03:37:42.184 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.185 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.185 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.185 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:37:42.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:37:42.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258984+0000", 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5689999999999999e-06 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258994+0000", 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259008+0000", 2024-09-16T03:37:42.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4151999999999999e-05 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359073+0000", 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100065628 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359110+0000", 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7045999999999997e-05 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359124+0000", 2024-09-16T03:37:42.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.382e-05 2024-09-16T03:37:42.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360716+0000", 2024-09-16T03:37:42.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015919829999999999 2024-09-16T03:37:42.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360747+0000", 2024-09-16T03:37:42.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0874999999999998e-05 2024-09-16T03:37:42.188 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.188 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.188 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:61 4.5 e39/32 4:ad0b9852:::smithi15736696-16:head v 39'4, mlcod=0'0)", 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.359093+0000", 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.800803232, 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001688403, 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359093+0000", 2024-09-16T03:37:42.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359093+0000", 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359094+0000", 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0999999999999997e-07 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359205+0000", 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000111025 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359211+0000", 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8930000000000002e-06 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359215+0000", 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2300000000000002e-06 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359251+0000", 2024-09-16T03:37:42.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6585000000000003e-05 2024-09-16T03:37:42.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359270+0000", 2024-09-16T03:37:42.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8785000000000001e-05 2024-09-16T03:37:42.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360754+0000", 2024-09-16T03:37:42.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014841240000000001 2024-09-16T03:37:42.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360781+0000", 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6951e-05 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:62 4.5 e39/32 4:ad0b9852:::smithi15736696-16:head v 39'5, mlcod=0'0)", 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.359222+0000", 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.800674104999999, 2024-09-16T03:37:42.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020277289999999998, 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359222+0000", 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359222+0000", 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359222+0000", 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3999999999999997e-07 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359341+0000", 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011825899999999999 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359345+0000", 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1180000000000002e-06 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359349+0000", 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.941e-06 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359991+0000", 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00064214700000000003 2024-09-16T03:37:42.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360010+0000", 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9028000000000001e-05 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.361209+0000", 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011994460000000001 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.361250+0000", 2024-09-16T03:37:42.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0450000000000001e-05 2024-09-16T03:37:42.198 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.198 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.198 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.198 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:63 4.5 e39/32 4:ad0b9852:::smithi15736696-16:head v 39'6, mlcod=0'0)", 2024-09-16T03:37:42.198 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.359355+0000", 2024-09-16T03:37:42.198 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.800541468999999, 2024-09-16T03:37:42.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019271519999999999, 2024-09-16T03:37:42.198 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.198 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.198 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.198 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359355+0000", 2024-09-16T03:37:42.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359355+0000", 2024-09-16T03:37:42.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359355+0000", 2024-09-16T03:37:42.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.22e-07 2024-09-16T03:37:42.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359357+0000", 2024-09-16T03:37:42.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.187e-06 2024-09-16T03:37:42.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359360+0000", 2024-09-16T03:37:42.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.745e-06 2024-09-16T03:37:42.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359363+0000", 2024-09-16T03:37:42.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7039999999999999e-06 2024-09-16T03:37:42.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360603+0000", 2024-09-16T03:37:42.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012399850000000001 2024-09-16T03:37:42.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360614+0000", 2024-09-16T03:37:42.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1599e-05 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.361254+0000", 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063991999999999999 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.361282+0000", 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7690000000000001e-05 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.203 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:260 4.5 e40/32 4:aff4cc41:::smithi15736696-11:head v 40'13, mlcod=40'11)", 2024-09-16T03:37:42.203 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:15.196507+0000", 2024-09-16T03:37:42.203 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 26.963389554999999, 2024-09-16T03:37:42.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016026790000000001, 2024-09-16T03:37:42.203 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.203 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.203 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.203 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.196507+0000", 2024-09-16T03:37:42.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.196507+0000", 2024-09-16T03:37:42.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.196507+0000", 2024-09-16T03:37:42.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3200000000000001e-07 2024-09-16T03:37:42.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.196516+0000", 2024-09-16T03:37:42.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.115e-06 2024-09-16T03:37:42.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.196526+0000", 2024-09-16T03:37:42.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8439999999999993e-06 2024-09-16T03:37:42.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.196539+0000", 2024-09-16T03:37:42.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.342e-05 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.197579+0000", 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010397029999999999 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.197595+0000", 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5471000000000001e-05 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.198002+0000", 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000407315 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:15.198109+0000", 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010747900000000001 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:492 4.5 e45/32 4:a11e1d7c:::smithi15736696-43:head v 45'22, mlcod=45'20)", 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 20.336545214000001, 2024-09-16T03:37:42.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100797151, 2024-09-16T03:37:42.208 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.208 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.208 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.208 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:37:42.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:37:42.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823353+0000", 2024-09-16T03:37:42.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2450000000000001e-06 2024-09-16T03:37:42.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823363+0000", 2024-09-16T03:37:42.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0074999999999999e-05 2024-09-16T03:37:42.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823378+0000", 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4534e-05 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923461+0000", 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100082859 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923525+0000", 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4616999999999995e-05 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923549+0000", 2024-09-16T03:37:42.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.357e-05 2024-09-16T03:37:42.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924106+0000", 2024-09-16T03:37:42.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000556753 2024-09-16T03:37:42.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924148+0000", 2024-09-16T03:37:42.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2497999999999999e-05 2024-09-16T03:37:42.211 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.211 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.212 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.212 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:805 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'24, mlcod=45'22)", 2024-09-16T03:37:42.212 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.012699+0000", 2024-09-16T03:37:42.212 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.147197491, 2024-09-16T03:37:42.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021199320000000002, 2024-09-16T03:37:42.212 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.212 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.212 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.212 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.012699+0000", 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.012699+0000", 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.012700+0000", 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4300000000000001e-06 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.012997+0000", 2024-09-16T03:37:42.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029683299999999998 2024-09-16T03:37:42.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.013012+0000", 2024-09-16T03:37:42.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4636e-05 2024-09-16T03:37:42.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.013019+0000", 2024-09-16T03:37:42.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3730000000000004e-06 2024-09-16T03:37:42.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.013081+0000", 2024-09-16T03:37:42.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2349999999999998e-05 2024-09-16T03:37:42.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.013098+0000", 2024-09-16T03:37:42.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6694e-05 2024-09-16T03:37:42.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.014776+0000", 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016784770000000001 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.014819+0000", 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2138999999999998e-05 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:806 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'25, mlcod=45'22)", 2024-09-16T03:37:42.216 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.019370+0000", 2024-09-16T03:37:42.217 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.140525886999999, 2024-09-16T03:37:42.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048461379999999998, 2024-09-16T03:37:42.217 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.217 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.217 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.217 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019370+0000", 2024-09-16T03:37:42.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019370+0000", 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019372+0000", 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.424e-06 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019541+0000", 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016961999999999999 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019553+0000", 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1678000000000001e-05 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019561+0000", 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7579999999999992e-06 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022299+0000", 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002738061 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022326+0000", 2024-09-16T03:37:42.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7515999999999999e-05 2024-09-16T03:37:42.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024179+0000", 2024-09-16T03:37:42.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001853005 2024-09-16T03:37:42.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024216+0000", 2024-09-16T03:37:42.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7076000000000002e-05 2024-09-16T03:37:42.221 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.221 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.221 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.221 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:807 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'26, mlcod=45'22)", 2024-09-16T03:37:42.221 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.022119+0000", 2024-09-16T03:37:42.221 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.137777058000001, 2024-09-16T03:37:42.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023221940000000001, 2024-09-16T03:37:42.221 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.221 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022119+0000", 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022119+0000", 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022120+0000", 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3e-06 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022193+0000", 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2364e-05 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022210+0000", 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6773999999999999e-05 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022221+0000", 2024-09-16T03:37:42.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1087000000000001e-05 2024-09-16T03:37:42.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023441+0000", 2024-09-16T03:37:42.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001220524 2024-09-16T03:37:42.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023470+0000", 2024-09-16T03:37:42.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8541e-05 2024-09-16T03:37:42.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024418+0000", 2024-09-16T03:37:42.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094820699999999996 2024-09-16T03:37:42.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024441+0000", 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3397e-05 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:808 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'27, mlcod=45'22)", 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.136851939, 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.097081386000000006, 2024-09-16T03:37:42.226 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.227 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.227 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.227 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:37:42.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:37:42.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023046+0000", 2024-09-16T03:37:42.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1059999999999998e-06 2024-09-16T03:37:42.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023056+0000", 2024-09-16T03:37:42.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3719999999999994e-06 2024-09-16T03:37:42.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023067+0000", 2024-09-16T03:37:42.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1246999999999999e-05 2024-09-16T03:37:42.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023075+0000", 2024-09-16T03:37:42.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4710000000000007e-06 2024-09-16T03:37:42.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.119708+0000", 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096632179999999998 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.119725+0000", 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7754999999999999e-05 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.120092+0000", 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00036668099999999999 2024-09-16T03:37:42.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.120126+0000", 2024-09-16T03:37:42.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3574000000000001e-05 2024-09-16T03:37:42.231 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.231 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.231 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.231 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1247 4.11 e54/52 4:89839b7e:::smithi15736696-12:head v 54'8, mlcod=0'0)", 2024-09-16T03:37:42.231 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:37.056281+0000", 2024-09-16T03:37:42.231 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.103615209, 2024-09-16T03:37:42.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010846919999999999, 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.056281+0000", 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.056281+0000", 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.056281+0000", 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6999999999999999e-07 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.056286+0000", 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6179999999999997e-06 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.056315+0000", 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9271000000000001e-05 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.056324+0000", 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8689999999999993e-06 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.056392+0000", 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8102999999999995e-05 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.056421+0000", 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8847e-05 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.057330+0000", 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00090917199999999997 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:37.057366+0000", 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5642000000000002e-05 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.235 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1363 4.5 e57/50 4:a4fbca73:::smithi15736696-39:head v 57'35, mlcod=57'33)", 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.177785+0000", 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.982111594, 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018490550000000001, 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.177785+0000", 2024-09-16T03:37:42.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.177785+0000", 2024-09-16T03:37:42.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.177787+0000", 2024-09-16T03:37:42.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0250000000000001e-06 2024-09-16T03:37:42.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.178089+0000", 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030217999999999998 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.178102+0000", 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3025e-05 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.178111+0000", 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3060000000000003e-06 2024-09-16T03:37:42.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.178176+0000", 2024-09-16T03:37:42.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4598999999999996e-05 2024-09-16T03:37:42.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.178200+0000", 2024-09-16T03:37:42.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4190999999999999e-05 2024-09-16T03:37:42.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.179586+0000", 2024-09-16T03:37:42.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013864389999999999 2024-09-16T03:37:42.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.179634+0000", 2024-09-16T03:37:42.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7290000000000003e-05 2024-09-16T03:37:42.240 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.240 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.240 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.240 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1364 4.5 e57/50 4:a4fbca73:::smithi15736696-39:head v 57'36, mlcod=57'33)", 2024-09-16T03:37:42.240 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.279986+0000", 2024-09-16T03:37:42.241 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.879909684, 2024-09-16T03:37:42.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021206100000000002, 2024-09-16T03:37:42.241 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.241 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.241 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.241 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.279986+0000", 2024-09-16T03:37:42.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.279986+0000", 2024-09-16T03:37:42.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.279988+0000", 2024-09-16T03:37:42.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.531e-06 2024-09-16T03:37:42.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280208+0000", 2024-09-16T03:37:42.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0002195 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280218+0000", 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0722e-05 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280226+0000", 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8949999999999998e-06 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280280+0000", 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4132000000000003e-05 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280298+0000", 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7371999999999999e-05 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282080+0000", 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0017824099999999999 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282107+0000", 2024-09-16T03:37:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7047999999999999e-05 2024-09-16T03:37:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1365 4.5 e57/50 4:a4fbca73:::smithi15736696-39:head v 57'37, mlcod=57'33)", 2024-09-16T03:37:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.280249+0000", 2024-09-16T03:37:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.8796472120000001, 2024-09-16T03:37:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024448349999999998, 2024-09-16T03:37:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.246 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.246 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280249+0000", 2024-09-16T03:37:42.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280249+0000", 2024-09-16T03:37:42.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280250+0000", 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2600000000000002e-07 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280485+0000", 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023584999999999999 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280494+0000", 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2349999999999999e-06 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280510+0000", 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6626999999999999e-05 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.281163+0000", 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00065277800000000002 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.281177+0000", 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4164e-05 2024-09-16T03:37:42.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282658+0000", 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001480594 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282694+0000", 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5960999999999997e-05 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1366 4.5 e57/50 4:a4fbca73:::smithi15736696-39:head v 57'38, mlcod=57'33)", 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.280525+0000", 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.8793716329999999, 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022401679999999998, 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280525+0000", 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280525+0000", 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280525+0000", 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280529+0000", 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.42e-06 2024-09-16T03:37:42.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280536+0000", 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.241e-06 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280540+0000", 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.561e-06 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.281977+0000", 2024-09-16T03:37:42.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014370909999999999 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.281993+0000", 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5659e-05 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282730+0000", 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073740000000000003 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282765+0000", 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4335999999999998e-05 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:53'30,epoch:57/50,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.761193598, 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100230893, 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:42.254 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398705+0000", 2024-09-16T03:37:42.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5689999999999999e-06 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398722+0000", 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6558e-05 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398735+0000", 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3499999999999999e-05 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:42.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398746+0000", 2024-09-16T03:37:42.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1182999999999999e-05 2024-09-16T03:37:42.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:42.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498898+0000", 2024-09-16T03:37:42.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015112599999999 2024-09-16T03:37:42.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:42.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498906+0000", 2024-09-16T03:37:42.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3189999999999995e-06 2024-09-16T03:37:42.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.258 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:42.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498933+0000", 2024-09-16T03:37:42.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7637999999999999e-05 2024-09-16T03:37:42.258 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.258 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.258 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.258 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.258 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.258 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:42.259 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:42.260 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:42.258+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:42.260 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:42.261 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:42.261 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:42.262 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:37:42.262 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:42.262 INFO:tasks.rados.rados.0.smithi157.stdout:813: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:37:42.262 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:42.262 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:42.263 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:42.263 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:42.263 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:42.269 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:42.270 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:42.270 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:42.287 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:42.287 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:42.298 INFO:tasks.rados.rados.0.smithi157.stdout:811: finishing write tid 4 to smithi15736696-6 2024-09-16T03:37:42.298 INFO:tasks.rados.rados.0.smithi157.stdout:811: finishing write tid 5 to smithi15736696-6 2024-09-16T03:37:42.298 INFO:tasks.rados.rados.0.smithi157.stdout:811: oid 6 updating version 0 to 41 2024-09-16T03:37:42.298 INFO:tasks.rados.rados.0.smithi157.stdout:811: oid 6 updating version 41 to 42 2024-09-16T03:37:42.298 INFO:tasks.rados.rados.0.smithi157.stdout:811: oid 6 version 42 is already newer than 39 2024-09-16T03:37:42.299 INFO:tasks.rados.rados.0.smithi157.stdout:811: oid 6 version 42 is already newer than 40 2024-09-16T03:37:42.299 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 42 (ObjNum 107 snap 0 seq_num 107) dirty exists 2024-09-16T03:37:42.299 INFO:tasks.rados.rados.0.smithi157.stdout:811: left oid 6 (ObjNum 107 snap 0 seq_num 107) 2024-09-16T03:37:42.299 INFO:tasks.rados.rados.0.smithi157.stdout:810: finishing write tid 2 to smithi15736696-29 2024-09-16T03:37:42.299 INFO:tasks.rados.rados.0.smithi157.stdout:810: finishing write tid 3 to smithi15736696-29 2024-09-16T03:37:42.299 INFO:tasks.rados.rados.0.smithi157.stdout:826: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-09-16T03:37:42.304 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:42.304 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:42.304 INFO:tasks.rados.rados.0.smithi157.stdout:826: writing smithi15736696-41 from 682384 to 1139214 tid 1 2024-09-16T03:37:42.310 INFO:tasks.rados.rados.0.smithi157.stdout:826: writing smithi15736696-41 from 1721042 to 2144741 tid 2 2024-09-16T03:37:42.316 INFO:tasks.rados.rados.0.smithi157.stdout:826: writing smithi15736696-41 from 2809728 to 3264664 tid 3 2024-09-16T03:37:42.319 INFO:tasks.rados.rados.0.smithi157.stdout:826: writing smithi15736696-41 from 3731357 to 3882384 tid 4 2024-09-16T03:37:42.319 INFO:tasks.rados.rados.0.smithi157.stdout:811: done (15 left) 2024-09-16T03:37:42.319 INFO:tasks.rados.rados.0.smithi157.stdout:813: done (14 left) 2024-09-16T03:37:42.320 INFO:tasks.rados.rados.0.smithi157.stdout:814: done (13 left) 2024-09-16T03:37:42.320 INFO:tasks.rados.rados.0.smithi157.stdout:827: set_chunk oid 39 target oid 2024-09-16T03:37:42.321 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:42.321 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:42.337 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:42.337 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:42.354 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:42.354 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:42.360 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:42.359+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:42.368 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:42.368 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:42.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.368 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1453 4.17 e58/50)", 2024-09-16T03:37:42.369 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:42.310376+0000", 2024-09-16T03:37:42.369 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.057045616, 2024-09-16T03:37:42.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.057062624999999999, 2024-09-16T03:37:42.369 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:42.369 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:42.369 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "no flag points reached", 2024-09-16T03:37:42.369 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:42.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:42.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:42.310376+0000", 2024-09-16T03:37:42.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:42.310376+0000", 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:42.310377+0000", 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.508e-06 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:42.310774+0000", 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00039705399999999998 2024-09-16T03:37:42.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:42.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:42.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:42.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:42.310786+0000", 2024-09-16T03:37:42.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1977999999999999e-05 2024-09-16T03:37:42.371 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.371 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:42.371 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.371 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:42.371 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-16T03:37:42.371 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 1 2024-09-16T03:37:42.371 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:42.372 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:42.372 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:42.377 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:37:42.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:37:42.429 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:42.429 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:42.445 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:42.445 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:42.461 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:42.460+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:42.463 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:42.463 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:42.479 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:42.479 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:42.496 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:42.496 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:42.523 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:42.523 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:42.532 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:37:42.573 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:42.571+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:42.638 INFO:tasks.rados.rados.0.smithi157.stdout:827: set_chunk oid 39 offset: 588800 length: 2740224 target oid d48b9b4eb53cf5de58533f5f6fef302de5e53ec892df839e8b33af8eba0a3179 offset: 0 snap -1 2024-09-16T03:37:42.638 INFO:tasks.rados.rados.0.smithi157.stdout:809: finishing write tid 4 to smithi15736696-13 2024-09-16T03:37:42.638 INFO:tasks.rados.rados.0.smithi157.stdout:828: read oid 18 snap -1 2024-09-16T03:37:42.638 INFO:tasks.rados.rados.0.smithi157.stdout:828: expect (ObjNum 74 snap 0 seq_num 74) 2024-09-16T03:37:42.638 INFO:tasks.rados.rados.0.smithi157.stdout:829: write oid 47 current snap is 0 2024-09-16T03:37:42.638 INFO:tasks.rados.rados.0.smithi157.stdout:809: finishing write tid 5 to smithi15736696-13 2024-09-16T03:37:42.639 INFO:tasks.rados.rados.0.smithi157.stdout:809: oid 13 updating version 0 to 36 2024-09-16T03:37:42.639 INFO:tasks.rados.rados.0.smithi157.stdout:809: oid 13 version 36 is already newer than 34 2024-09-16T03:37:42.639 INFO:tasks.rados.rados.0.smithi157.stdout:809: oid 13 version 36 is already newer than 35 2024-09-16T03:37:42.639 INFO:tasks.rados.rados.0.smithi157.stdout:809: oid 13 version 36 is already newer than 33 2024-09-16T03:37:42.639 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 36 (ObjNum 105 snap 0 seq_num 105) dirty exists 2024-09-16T03:37:42.639 INFO:tasks.rados.rados.0.smithi157.stdout:809: left oid 13 (ObjNum 105 snap 0 seq_num 105) 2024-09-16T03:37:42.639 INFO:tasks.rados.rados.0.smithi157.stdout:815: expect (ObjNum 63 snap 0 seq_num 63) 2024-09-16T03:37:42.639 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:42.663 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:42.662+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:42.676 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:42.677 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:42.685 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:37:42.764 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:42.762+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:42.790 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:42.791 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:42.799 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:37:42.817 INFO:tasks.rados.rados.0.smithi157.stdout:810: finishing write tid 4 to smithi15736696-29 2024-09-16T03:37:42.818 INFO:tasks.rados.rados.0.smithi157.stdout:810: finishing write tid 5 to smithi15736696-29 2024-09-16T03:37:42.818 INFO:tasks.rados.rados.0.smithi157.stdout:810: oid 29 updating version 0 to 58 2024-09-16T03:37:42.818 INFO:tasks.rados.rados.0.smithi157.stdout:810: oid 29 updating version 58 to 59 2024-09-16T03:37:42.818 INFO:tasks.rados.rados.0.smithi157.stdout:810: oid 29 updating version 59 to 60 2024-09-16T03:37:42.818 INFO:tasks.rados.rados.0.smithi157.stdout:810: oid 29 version 60 is already newer than 57 2024-09-16T03:37:42.818 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 60 (ObjNum 106 snap 0 seq_num 106) dirty exists 2024-09-16T03:37:42.818 INFO:tasks.rados.rados.0.smithi157.stdout:810: left oid 29 (ObjNum 106 snap 0 seq_num 106) 2024-09-16T03:37:42.818 INFO:tasks.rados.rados.0.smithi157.stdout:817: expect (ObjNum 55 snap 0 seq_num 55) 2024-09-16T03:37:42.865 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:42.863+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:42.904 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:42.904 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:42.912 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:37:42.916 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2580472 2024-09-16T03:37:42.919 INFO:tasks.rados.rados.0.smithi157.stdout:819: finishing write tid 1 to smithi15736696-37 2024-09-16T03:37:42.920 INFO:tasks.rados.rados.0.smithi157.stdout:816: finishing write tid 1 to smithi15736696-21 2024-09-16T03:37:42.920 INFO:tasks.rados.rados.0.smithi157.stdout:816: finishing write tid 2 to smithi15736696-21 2024-09-16T03:37:42.920 INFO:tasks.rados.rados.0.smithi157.stdout:816: finishing write tid 3 to smithi15736696-21 2024-09-16T03:37:42.920 INFO:tasks.rados.rados.0.smithi157.stdout:819: finishing write tid 2 to smithi15736696-37 2024-09-16T03:37:42.920 INFO:tasks.rados.rados.0.smithi157.stdout:829: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-09-16T03:37:42.926 INFO:tasks.rados.rados.0.smithi157.stdout:829: writing smithi15736696-47 from 699191 to 1231270 tid 1 2024-09-16T03:37:42.932 INFO:tasks.rados.rados.0.smithi157.stdout:829: writing smithi15736696-47 from 1779524 to 2346881 tid 2 2024-09-16T03:37:42.939 INFO:tasks.rados.rados.0.smithi157.stdout:829: writing smithi15736696-47 from 2837151 to 3503359 tid 3 2024-09-16T03:37:42.939 INFO:tasks.rados.rados.0.smithi157.stdout:809: done (15 left) 2024-09-16T03:37:42.939 INFO:tasks.rados.rados.0.smithi157.stdout:810: done (14 left) 2024-09-16T03:37:42.939 INFO:tasks.rados.rados.0.smithi157.stdout:815: done (13 left) 2024-09-16T03:37:42.939 INFO:tasks.rados.rados.0.smithi157.stdout:817: done (12 left) 2024-09-16T03:37:42.939 INFO:tasks.rados.rados.0.smithi157.stdout:830: read oid 20 snap -1 2024-09-16T03:37:42.939 INFO:tasks.rados.rados.0.smithi157.stdout:830: expect (ObjNum 102 snap 0 seq_num 102) 2024-09-16T03:37:42.939 INFO:tasks.rados.rados.0.smithi157.stdout:831: tier_evict oid 15 2024-09-16T03:37:42.939 INFO:tasks.rados.rados.0.smithi157.stdout:831: tier_evict oid 15 snap -1 2024-09-16T03:37:42.940 INFO:tasks.rados.rados.0.smithi157.stdout:819: finishing write tid 3 to smithi15736696-37 2024-09-16T03:37:42.940 INFO:tasks.rados.rados.0.smithi157.stdout:822: expect (ObjNum 86 snap 0 seq_num 86) 2024-09-16T03:37:42.940 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:42.965 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:42.963+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:43.021 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:43.022 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:43.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:37:43.066 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:43.064+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:43.093 INFO:tasks.rados.rados.0.smithi157.stdout:816: finishing write tid 4 to smithi15736696-21 2024-09-16T03:37:43.094 INFO:tasks.rados.rados.0.smithi157.stdout:816: finishing write tid 5 to smithi15736696-21 2024-09-16T03:37:43.094 INFO:tasks.rados.rados.0.smithi157.stdout:816: oid 21 updating version 0 to 31 2024-09-16T03:37:43.094 INFO:tasks.rados.rados.0.smithi157.stdout:816: oid 21 updating version 31 to 32 2024-09-16T03:37:43.094 INFO:tasks.rados.rados.0.smithi157.stdout:816: oid 21 updating version 32 to 33 2024-09-16T03:37:43.094 INFO:tasks.rados.rados.0.smithi157.stdout:816: oid 21 version 33 is already newer than 30 2024-09-16T03:37:43.094 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 33 (ObjNum 108 snap 0 seq_num 108) dirty exists 2024-09-16T03:37:43.094 INFO:tasks.rados.rados.0.smithi157.stdout:816: left oid 21 (ObjNum 108 snap 0 seq_num 108) 2024-09-16T03:37:43.094 INFO:tasks.rados.rados.0.smithi157.stdout:816: done (13 left) 2024-09-16T03:37:43.094 INFO:tasks.rados.rados.0.smithi157.stdout:822: done (12 left) 2024-09-16T03:37:43.094 INFO:tasks.rados.rados.0.smithi157.stdout:832: read oid 4 snap -1 2024-09-16T03:37:43.094 INFO:tasks.rados.rados.0.smithi157.stdout:832: expect deleted 2024-09-16T03:37:43.094 INFO:tasks.rados.rados.0.smithi157.stdout:833: read oid 14 snap -1 2024-09-16T03:37:43.094 INFO:tasks.rados.rados.0.smithi157.stdout:833: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T03:37:43.095 INFO:tasks.rados.rados.0.smithi157.stdout:834: read oid 22 snap -1 2024-09-16T03:37:43.095 INFO:tasks.rados.rados.0.smithi157.stdout:834: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T03:37:43.095 INFO:tasks.rados.rados.0.smithi157.stdout:835: set_chunk oid 25 target oid 2024-09-16T03:37:43.141 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:43.141 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:43.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:37:43.166 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:43.165+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:43.257 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:43.292 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:43.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:37:43.295 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:43.292+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:43.297 INFO:tasks.rados.rados.0.smithi157.stdout:835: set_chunk oid 25 offset: 774144 length: 1232896 target oid c3ade2e0e011664dec5abcec0a540e939391e86e9e9e055243f2e5d9884305fd offset: 0 snap -1 2024-09-16T03:37:43.297 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:43.297 INFO:tasks.rados.rados.0.smithi157.stdout:819: finishing write tid 4 to smithi15736696-37 2024-09-16T03:37:43.298 INFO:tasks.rados.rados.0.smithi157.stdout:819: finishing write tid 5 to smithi15736696-37 2024-09-16T03:37:43.298 INFO:tasks.rados.rados.0.smithi157.stdout:819: oid 37 updating version 0 to 29 2024-09-16T03:37:43.298 INFO:tasks.rados.rados.0.smithi157.stdout:819: oid 37 updating version 29 to 32 2024-09-16T03:37:43.298 INFO:tasks.rados.rados.0.smithi157.stdout:819: oid 37 version 32 is already newer than 30 2024-09-16T03:37:43.298 INFO:tasks.rados.rados.0.smithi157.stdout:819: oid 37 version 32 is already newer than 31 2024-09-16T03:37:43.298 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 32 (ObjNum 110 snap 0 seq_num 110) dirty exists 2024-09-16T03:37:43.298 INFO:tasks.rados.rados.0.smithi157.stdout:819: left oid 37 (ObjNum 110 snap 0 seq_num 110) 2024-09-16T03:37:43.298 INFO:tasks.rados.rados.0.smithi157.stdout:820: finishing set_chunk to oid 19 2024-09-16T03:37:43.298 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 50 (ObjNum 93 snap 0 seq_num 93) dirty exists 2024-09-16T03:37:43.298 INFO:tasks.rados.rados.0.smithi157.stdout:821: expect (ObjNum 64 snap 0 seq_num 64) 2024-09-16T03:37:43.302 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1462852 2024-09-16T03:37:43.335 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:43.381 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 359676 2024-09-16T03:37:43.382 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:43.380+0000 7f928a16a640 -1 osd.14 58 handle_osd_map injecting map crc failure 2024-09-16T03:37:43.385 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:37:43.395 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:43.394+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:43.404 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:43.404 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:43.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:37:43.423 INFO:tasks.rados.rados.0.smithi157.stdout:823: finishing write tid 1 to smithi15736696-5 2024-09-16T03:37:43.423 INFO:tasks.rados.rados.0.smithi157.stdout:823: finishing write tid 2 to smithi15736696-5 2024-09-16T03:37:43.423 INFO:tasks.rados.rados.0.smithi157.stdout:823: finishing write tid 3 to smithi15736696-5 2024-09-16T03:37:43.424 INFO:tasks.rados.rados.0.smithi157.stdout:824: finishing set_chunk to oid 36 2024-09-16T03:37:43.424 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 36 set_chunk returned error code -95 offset: 1922048 length: 320000 2024-09-16T03:37:43.424 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T03:37:43.424 INFO:tasks.rados.rados.0.smithi157.stdout:823: finishing write tid 4 to smithi15736696-5 2024-09-16T03:37:43.424 INFO:tasks.rados.rados.0.smithi157.stdout:819: done (15 left) 2024-09-16T03:37:43.424 INFO:tasks.rados.rados.0.smithi157.stdout:820: done (14 left) 2024-09-16T03:37:43.424 INFO:tasks.rados.rados.0.smithi157.stdout:821: done (13 left) 2024-09-16T03:37:43.424 INFO:tasks.rados.rados.0.smithi157.stdout:824: done (12 left) 2024-09-16T03:37:43.424 INFO:tasks.rados.rados.0.smithi157.stdout:836: read oid 9 snap -1 2024-09-16T03:37:43.424 INFO:tasks.rados.rados.0.smithi157.stdout:836: expect (ObjNum 55 snap 0 seq_num 55) 2024-09-16T03:37:43.424 INFO:tasks.rados.rados.0.smithi157.stdout:837: write (excl) oid 43 current snap is 0 2024-09-16T03:37:43.424 INFO:tasks.rados.rados.0.smithi157.stdout:823: finishing write tid 5 to smithi15736696-5 2024-09-16T03:37:43.424 INFO:tasks.rados.rados.0.smithi157.stdout:823: oid 5 updating version 0 to 31 2024-09-16T03:37:43.425 INFO:tasks.rados.rados.0.smithi157.stdout:823: oid 5 version 31 is already newer than 29 2024-09-16T03:37:43.425 INFO:tasks.rados.rados.0.smithi157.stdout:823: oid 5 version 31 is already newer than 28 2024-09-16T03:37:43.425 INFO:tasks.rados.rados.0.smithi157.stdout:823: oid 5 version 31 is already newer than 30 2024-09-16T03:37:43.425 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 31 (ObjNum 111 snap 0 seq_num 111) dirty exists 2024-09-16T03:37:43.425 INFO:tasks.rados.rados.0.smithi157.stdout:823: left oid 5 (ObjNum 111 snap 0 seq_num 111) 2024-09-16T03:37:43.425 INFO:tasks.rados.rados.0.smithi157.stdout:825: expect (ObjNum 98 snap 0 seq_num 98) 2024-09-16T03:37:43.495 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2124447 2024-09-16T03:37:43.497 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:43.494+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:43.537 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2053670 2024-09-16T03:37:43.560 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:43.560 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:43.561 INFO:tasks.rados.rados.0.smithi157.stdout:826: finishing write tid 1 to smithi15736696-41 2024-09-16T03:37:43.561 INFO:tasks.rados.rados.0.smithi157.stdout:826: finishing write tid 2 to smithi15736696-41 2024-09-16T03:37:43.561 INFO:tasks.rados.rados.0.smithi157.stdout:826: finishing write tid 3 to smithi15736696-41 2024-09-16T03:37:43.561 INFO:tasks.rados.rados.0.smithi157.stdout:837: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-09-16T03:37:43.568 INFO:tasks.rados.rados.0.smithi157.stdout:837: writing smithi15736696-43 from 715998 to 1323326 tid 1 2024-09-16T03:37:43.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:37:43.573 INFO:tasks.rados.rados.0.smithi157.stdout:837: writing smithi15736696-43 from 1838006 to 2265374 tid 2 2024-09-16T03:37:43.579 INFO:tasks.rados.rados.0.smithi157.stdout:837: writing smithi15736696-43 from 2864574 to 3342054 tid 3 2024-09-16T03:37:43.582 INFO:tasks.rados.rados.0.smithi157.stdout:837: writing smithi15736696-43 from 3863835 to 3915998 tid 4 2024-09-16T03:37:43.583 INFO:tasks.rados.rados.0.smithi157.stdout:823: done (13 left) 2024-09-16T03:37:43.583 INFO:tasks.rados.rados.0.smithi157.stdout:825: done (12 left) 2024-09-16T03:37:43.583 INFO:tasks.rados.rados.0.smithi157.stdout:838: tier_flush oid 31 2024-09-16T03:37:43.583 INFO:tasks.rados.rados.0.smithi157.stdout:838: tier_flush oid 31 snap -1 2024-09-16T03:37:43.583 INFO:tasks.rados.rados.0.smithi157.stdout:826: finishing write tid 4 to smithi15736696-41 2024-09-16T03:37:43.583 INFO:tasks.rados.rados.0.smithi157.stdout:826: finishing write tid 5 to smithi15736696-41 2024-09-16T03:37:43.583 INFO:tasks.rados.rados.0.smithi157.stdout:826: finishing write tid 6 to smithi15736696-41 2024-09-16T03:37:43.583 INFO:tasks.rados.rados.0.smithi157.stdout:826: oid 41 updating version 0 to 32 2024-09-16T03:37:43.583 INFO:tasks.rados.rados.0.smithi157.stdout:826: oid 41 version 32 is already newer than 29 2024-09-16T03:37:43.583 INFO:tasks.rados.rados.0.smithi157.stdout:826: oid 41 version 32 is already newer than 28 2024-09-16T03:37:43.583 INFO:tasks.rados.rados.0.smithi157.stdout:826: oid 41 version 32 is already newer than 31 2024-09-16T03:37:43.584 INFO:tasks.rados.rados.0.smithi157.stdout:826: oid 41 version 32 is already newer than 30 2024-09-16T03:37:43.584 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 32 (ObjNum 112 snap 0 seq_num 112) dirty exists 2024-09-16T03:37:43.584 INFO:tasks.rados.rados.0.smithi157.stdout:826: left oid 41 (ObjNum 112 snap 0 seq_num 112) 2024-09-16T03:37:43.584 INFO:tasks.rados.rados.0.smithi157.stdout:827: finishing set_chunk to oid 39 2024-09-16T03:37:43.584 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 38 (ObjNum 103 snap 0 seq_num 103) dirty exists 2024-09-16T03:37:43.584 INFO:tasks.rados.rados.0.smithi157.stdout:826: done (12 left) 2024-09-16T03:37:43.584 INFO:tasks.rados.rados.0.smithi157.stdout:827: done (11 left) 2024-09-16T03:37:43.584 INFO:tasks.rados.rados.0.smithi157.stdout:839: tier_promote oid 30 2024-09-16T03:37:43.584 INFO:tasks.rados.rados.0.smithi157.stdout:830: expect (ObjNum 102 snap 0 seq_num 102) 2024-09-16T03:37:43.596 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:43.595+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:43.701 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:43.700+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:43.717 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:43.717 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:43.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:37:43.731 INFO:tasks.rados.rados.0.smithi157.stdout:828: expect (ObjNum 74 snap 0 seq_num 74) 2024-09-16T03:37:43.731 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:43.782 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:43.780+0000 7efeef6d5640 -1 osd.13 58 handle_osd_map injecting map crc failure 2024-09-16T03:37:43.802 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:43.801+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:43.822 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:43.832 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:43.833 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:43.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:37:43.867 INFO:tasks.rados.rados.0.smithi157.stdout:831: got (22) Invalid argument 2024-09-16T03:37:43.867 INFO:tasks.rados.rados.0.smithi157.stdout:828: done (11 left) 2024-09-16T03:37:43.867 INFO:tasks.rados.rados.0.smithi157.stdout:830: done (10 left) 2024-09-16T03:37:43.867 INFO:tasks.rados.rados.0.smithi157.stdout:831: done (9 left) 2024-09-16T03:37:43.867 INFO:tasks.rados.rados.0.smithi157.stdout:840: read oid 7 snap -1 2024-09-16T03:37:43.867 INFO:tasks.rados.rados.0.smithi157.stdout:840: expect deleted 2024-09-16T03:37:43.868 INFO:tasks.rados.rados.0.smithi157.stdout:841: tier_promote oid 38 2024-09-16T03:37:43.868 INFO:tasks.rados.rados.0.smithi157.stdout:833: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-16T03:37:43.873 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1236952 2024-09-16T03:37:43.903 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:43.901+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:43.929 INFO:tasks.rados.rados.0.smithi157.stdout:834: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T03:37:43.929 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:43.948 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:43.949 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:43.955 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:43.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:37:44.003 INFO:tasks.rados.rados.0.smithi157.stdout:829: finishing write tid 1 to smithi15736696-47 2024-09-16T03:37:44.003 INFO:tasks.rados.rados.0.smithi157.stdout:835: finishing set_chunk to oid 25 2024-09-16T03:37:44.003 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk returned error code -95 offset: 774144 length: 1232896 2024-09-16T03:37:44.003 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:44.003 INFO:tasks.rados.rados.0.smithi157.stdout:829: finishing write tid 2 to smithi15736696-47 2024-09-16T03:37:44.003 INFO:tasks.rados.rados.0.smithi157.stdout:829: finishing write tid 3 to smithi15736696-47 2024-09-16T03:37:44.003 INFO:tasks.rados.rados.0.smithi157.stdout:829: finishing write tid 4 to smithi15736696-47 2024-09-16T03:37:44.003 INFO:tasks.rados.rados.0.smithi157.stdout:836: expect (ObjNum 55 snap 0 seq_num 55) 2024-09-16T03:37:44.005 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:44.003+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:44.037 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:44.065 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:44.065 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:44.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:37:44.079 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:44.105 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:44.104+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:44.124 INFO:tasks.rados.rados.0.smithi157.stdout:829: finishing write tid 5 to smithi15736696-47 2024-09-16T03:37:44.124 INFO:tasks.rados.rados.0.smithi157.stdout:829: oid 47 updating version 0 to 47 2024-09-16T03:37:44.124 INFO:tasks.rados.rados.0.smithi157.stdout:829: oid 47 updating version 47 to 49 2024-09-16T03:37:44.124 INFO:tasks.rados.rados.0.smithi157.stdout:829: oid 47 version 49 is already newer than 48 2024-09-16T03:37:44.125 INFO:tasks.rados.rados.0.smithi157.stdout:829: oid 47 updating version 49 to 50 2024-09-16T03:37:44.125 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 50 (ObjNum 113 snap 0 seq_num 113) dirty exists 2024-09-16T03:37:44.128 INFO:tasks.rados.rados.0.smithi157.stdout:829: left oid 47 (ObjNum 113 snap 0 seq_num 113) 2024-09-16T03:37:44.128 INFO:tasks.rados.rados.0.smithi157.stdout:838: got (2) No such file or directory 2024-09-16T03:37:44.128 INFO:tasks.rados.rados.0.smithi157.stdout:838: got expected ENOENT (src dne) 2024-09-16T03:37:44.129 INFO:tasks.rados.rados.0.smithi157.stdout:839: got (0) Success 2024-09-16T03:37:44.129 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 29 (ObjNum 98 snap 0 seq_num 98) dirty exists 2024-09-16T03:37:44.129 INFO:tasks.rados.rados.0.smithi157.stdout:829: done (10 left) 2024-09-16T03:37:44.129 INFO:tasks.rados.rados.0.smithi157.stdout:832: done (9 left) 2024-09-16T03:37:44.129 INFO:tasks.rados.rados.0.smithi157.stdout:833: done (8 left) 2024-09-16T03:37:44.129 INFO:tasks.rados.rados.0.smithi157.stdout:834: done (7 left) 2024-09-16T03:37:44.130 INFO:tasks.rados.rados.0.smithi157.stdout:835: done (6 left) 2024-09-16T03:37:44.130 INFO:tasks.rados.rados.0.smithi157.stdout:836: done (5 left) 2024-09-16T03:37:44.130 INFO:tasks.rados.rados.0.smithi157.stdout:838: done (4 left) 2024-09-16T03:37:44.130 INFO:tasks.rados.rados.0.smithi157.stdout:839: done (3 left) 2024-09-16T03:37:44.130 INFO:tasks.rados.rados.0.smithi157.stdout:842: delete oid 8 current snap is 0 2024-09-16T03:37:44.131 INFO:tasks.rados.rados.0.smithi157.stdout:837: finishing write tid 1 to smithi15736696-43 2024-09-16T03:37:44.131 INFO:tasks.rados.rados.0.smithi157.stdout:837: finishing write tid 2 to smithi15736696-43 2024-09-16T03:37:44.131 INFO:tasks.rados.rados.0.smithi157.stdout:837: finishing write tid 3 to smithi15736696-43 2024-09-16T03:37:44.131 INFO:tasks.rados.rados.0.smithi157.stdout:837: finishing write tid 4 to smithi15736696-43 2024-09-16T03:37:44.131 INFO:tasks.rados.rados.0.smithi157.stdout:837: finishing write tid 5 to smithi15736696-43 2024-09-16T03:37:44.131 INFO:tasks.rados.rados.0.smithi157.stdout:837: finishing write tid 6 to smithi15736696-43 2024-09-16T03:37:44.132 INFO:tasks.rados.rados.0.smithi157.stdout:837: oid 43 updating version 0 to 44 2024-09-16T03:37:44.132 INFO:tasks.rados.rados.0.smithi157.stdout:837: oid 43 version 44 is already newer than 40 2024-09-16T03:37:44.132 INFO:tasks.rados.rados.0.smithi157.stdout:837: oid 43 version 44 is already newer than 41 2024-09-16T03:37:44.132 INFO:tasks.rados.rados.0.smithi157.stdout:837: oid 43 version 44 is already newer than 43 2024-09-16T03:37:44.132 INFO:tasks.rados.rados.0.smithi157.stdout:837: oid 43 version 44 is already newer than 42 2024-09-16T03:37:44.133 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 44 (ObjNum 114 snap 0 seq_num 114) dirty exists 2024-09-16T03:37:44.133 INFO:tasks.rados.rados.0.smithi157.stdout:837: left oid 43 (ObjNum 114 snap 0 seq_num 114) 2024-09-16T03:37:44.181 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:44.181 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:44.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:37:44.207 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:44.205+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:44.269 INFO:tasks.rados.rados.0.smithi157.stdout:841: got (0) Success 2024-09-16T03:37:44.269 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 21 (ObjNum 95 snap 0 seq_num 95) dirty exists 2024-09-16T03:37:44.298 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:44.298 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:44.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:37:44.308 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:44.306+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:44.310 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:37:44.310 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:37:44.310 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:37:44.310 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-16T03:37:44.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:37:44.408 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:44.407+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:44.414 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:44.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:37:44.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:37:44.510 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:44.508+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:44.534 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:44.535 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:44.544 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:37:44.610 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:44.609+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:44.625 INFO:tasks.rados.rados.0.smithi157.stdout:837: done (3 left) 2024-09-16T03:37:44.625 INFO:tasks.rados.rados.0.smithi157.stdout:840: done (2 left) 2024-09-16T03:37:44.626 INFO:tasks.rados.rados.0.smithi157.stdout:841: done (1 left) 2024-09-16T03:37:44.626 INFO:tasks.rados.rados.0.smithi157.stdout:842: done (0 left) 2024-09-16T03:37:44.626 INFO:tasks.rados.rados.0.smithi157.stdout:843: set_chunk oid 41 target oid 2024-09-16T03:37:44.636 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:44.636 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:44.653 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:44.653 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:44.657 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:37:44.657 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:37:44.661 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:44.661 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:44.661 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:44.661 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:44.669 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-09-16T03:37:43.686613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":208665162,"num_objects":137,"num_object_clones":0,"num_object_copies":411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":0,"num_read":1166,"num_read_kb":577261,"num_write":886,"num_write_kb":256133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":25,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106,"ondisk_log_size":106,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":797744,"kb_used_data":340000,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509151696,"statfs":{"total":1546188226560,"available":1545371336704,"internally_reserved":0,"allocated":348160000,"data_stored":357385920,"data_compressed":10612665,"data_compressed_allocated":11358208,"data_compressed_original":25292800,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6958292,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":160,"num_read_kb":101020,"num_write":104,"num_write_kb":21477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-8,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.320616"},"pg_stats":[{"pgid":"4.19","version":"0'0","reported_seq":53,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.787362+0000","last_change":"2024-09-16T03:37:34.047028+0000","last_active":"2024-09-16T03:37:38.787362+0000","last_peered":"2024-09-16T03:37:38.787362+0000","last_clean":"2024-09-16T03:37:38.787362+0000","last_became_active":"2024-09-16T03:37:31.348734+0000","last_became_peered":"2024-09-16T03:37:31.348734+0000","last_unstale":"2024-09-16T03:37:38.787362+0000","last_undegraded":"2024-09-16T03:37:38.787362+0000","last_fullsized":"2024-09-16T03:37:38.787362+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,8],"acting":[14,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"56'29","reported_seq":130,"reported_epoch":56,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:37:39.154262+0000","last_change":"2024-09-16T03:37:39.004558+0000","last_active":"2024-09-16T03:37:39.154262+0000","last_peered":"2024-09-16T03:37:39.154262+0000","last_clean":"2024-09-16T03:37:39.154262+0000","last_became_active":"2024-09-16T03:37:31.248952+0000","last_became_peered":"2024-09-16T03:37:31.248952+0000","last_unstale":"2024-09-16T03:37:39.154262+0000","last_undegraded":"2024-09-16T03:37:39.154262+0000","last_fullsized":"2024-09-16T03:37:39.154262+0000","mapping_epoch":50,"log_start":"47'25","ondisk_log_start":"47'25","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":22,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6126086,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":15205,"num_write":16,"num_write_kb":4222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"56'27","reported_seq":126,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.788731+0000","last_change":"2024-09-16T03:37:34.145032+0000","last_active":"2024-09-16T03:37:38.788731+0000","last_peered":"2024-09-16T03:37:38.788731+0000","last_clean":"2024-09-16T03:37:38.788731+0000","last_became_active":"2024-09-16T03:37:31.713115+0000","last_became_peered":"2024-09-16T03:37:31.713115+0000","last_unstale":"2024-09-16T03:37:38.788731+0000","last_undegraded":"2024-09-16T03:37:38.788731+0000","last_fullsized":"2024-09-16T03:37:38.788731+0000","mapping_epoch":50,"log_start":"55'24","ondisk_log_start":"55'24","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":23,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6025244,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":36579,"num_write":14,"num_write_kb":4745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.16","version":"52'32","reported_seq":135,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:39.009518+0000","last_change":"2024-09-16T03:37:35.189522+0000","last_active":"2024-09-16T03:37:39.009518+0000","last_peered":"2024-09-16T03:37:39.009518+0000","last_clean":"2024-09-16T03:37:39.009518+0000","last_became_active":"2024-09-16T03:37:31.775904+0000","last_became_peered":"2024-09-16T03:37:31.775904+0000","last_unstale":"2024-09-16T03:37:39.009518+0000","last_undegraded":"2024-09-16T03:37:39.009518+0000","last_fullsized":"2024-09-16T03:37:39.009518+0000","mapping_epoch":50,"log_start":"52'29","ondisk_log_start":"52'29","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'32","last_scrub_stamp":"2024-09-16T03:37:35.189364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.189364+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":27,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:50.008531+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3226911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,6],"acting":[5,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"53'30","reported_seq":138,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:41.220640+0000","last_change":"2024-09-16T03:37:41.220415+0000","last_active":"2024-09-16T03:37:41.220640+0000","last_peered":"2024-09-16T03:37:41.220640+0000","last_clean":"2024-09-16T03:37:41.220640+0000","last_became_active":"2024-09-16T03:37:31.349745+0000","last_became_peered":"2024-09-16T03:37:31.349745+0000","last_unstale":"2024-09-16T03:37:41.220640+0000","last_undegraded":"2024-09-16T03:37:41.220640+0000","last_fullsized":"2024-09-16T03:37:41.220640+0000","mapping_epoch":50,"log_start":"47'26","ondisk_log_start":"47'26","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":26,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:38:10.699682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5235315,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":26187,"num_write":12,"num_write_kb":3104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.14","version":"57'13","reported_seq":115,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:40.700732+0000","last_change":"2024-09-16T03:37:39.953719+0000","last_active":"2024-09-16T03:37:40.700732+0000","last_peered":"2024-09-16T03:37:40.700732+0000","last_clean":"2024-09-16T03:37:40.700732+0000","last_became_active":"2024-09-16T03:37:31.149621+0000","last_became_peered":"2024-09-16T03:37:31.149621+0000","last_unstale":"2024-09-16T03:37:40.700732+0000","last_undegraded":"2024-09-16T03:37:40.700732+0000","last_fullsized":"2024-09-16T03:37:40.700732+0000","mapping_epoch":50,"log_start":"44'9","ondisk_log_start":"44'9","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'13","last_scrub_stamp":"2024-09-16T03:37:39.953656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.953656+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":9,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:51.714008+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521017,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.13","version":"56'27","reported_seq":132,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:40.700697+0000","last_change":"2024-09-16T03:37:34.916816+0000","last_active":"2024-09-16T03:37:40.700697+0000","last_peered":"2024-09-16T03:37:40.700697+0000","last_clean":"2024-09-16T03:37:40.700697+0000","last_became_active":"2024-09-16T03:37:31.548418+0000","last_became_peered":"2024-09-16T03:37:31.548418+0000","last_unstale":"2024-09-16T03:37:40.700697+0000","last_undegraded":"2024-09-16T03:37:40.700697+0000","last_fullsized":"2024-09-16T03:37:40.700697+0000","mapping_epoch":50,"log_start":"55'24","ondisk_log_start":"55'24","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'21","last_scrub_stamp":"2024-09-16T03:37:34.916681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.916681+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":22,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:47.907409+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722734,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":15514,"num_write":6,"num_write_kb":1439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.12","version":"45'6","reported_seq":101,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:40.589084+0000","last_change":"2024-09-16T03:37:34.762056+0000","last_active":"2024-09-16T03:37:40.589084+0000","last_peered":"2024-09-16T03:37:40.589084+0000","last_clean":"2024-09-16T03:37:40.589084+0000","last_became_active":"2024-09-16T03:37:31.352486+0000","last_became_peered":"2024-09-16T03:37:31.352486+0000","last_unstale":"2024-09-16T03:37:40.589084+0000","last_undegraded":"2024-09-16T03:37:40.589084+0000","last_fullsized":"2024-09-16T03:37:40.589084+0000","mapping_epoch":50,"log_start":"39'3","ondisk_log_start":"39'3","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:36.977197+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2016807,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.11","version":"54'8","reported_seq":67,"reported_epoch":58,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:37:40.846248+0000","last_change":"2024-09-16T03:37:32.847550+0000","last_active":"2024-09-16T03:37:40.846248+0000","last_peered":"2024-09-16T03:37:40.846248+0000","last_clean":"2024-09-16T03:37:28.467886+0000","last_became_active":"2024-09-16T03:37:32.745785+0000","last_became_peered":"2024-09-16T03:37:32.745785+0000","last_unstale":"2024-09-16T03:37:40.846248+0000","last_undegraded":"2024-09-16T03:37:40.846248+0000","last_fullsized":"2024-09-16T03:37:40.846248+0000","mapping_epoch":52,"log_start":"48'5","ondisk_log_start":"48'5","created":50,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":5,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,12],"acting":[2,4,7],"avail_no_missing":["2","4","7","8","11","12"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"53'46","reported_seq":155,"reported_epoch":55,"state":"peering","last_fresh":"2024-09-16T03:37:37.356685+0000","last_change":"2024-09-16T03:37:38.585698+0000","last_active":"2024-09-16T03:37:38.585698+0000","last_peered":"2024-09-16T03:37:37.356685+0000","last_clean":"2024-09-16T03:37:37.356685+0000","last_became_active":"2024-09-16T03:37:31.714323+0000","last_became_peered":"2024-09-16T03:37:31.714323+0000","last_unstale":"2024-09-16T03:37:37.356685+0000","last_undegraded":"2024-09-16T03:37:37.356685+0000","last_fullsized":"2024-09-16T03:37:37.356685+0000","mapping_epoch":56,"log_start":"52'43","ondisk_log_start":"52'43","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243718,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,13],"acting":[10,3,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4,13,15],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.503806+0000","last_change":"2024-09-16T03:36:55.100546+0000","last_active":"2024-09-16T03:37:38.503806+0000","last_peered":"2024-09-16T03:37:38.503806+0000","last_clean":"2024-09-16T03:37:38.503806+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:37:38.503806+0000","last_undegraded":"2024-09-16T03:37:38.503806+0000","last_fullsized":"2024-09-16T03:37:38.503806+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.896892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":72,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.587182+0000","last_change":"2024-09-16T03:37:31.438151+0000","last_active":"2024-09-16T03:37:38.587182+0000","last_peered":"2024-09-16T03:37:38.587182+0000","last_clean":"2024-09-16T03:37:38.587182+0000","last_became_active":"2024-09-16T03:37:31.054119+0000","last_became_peered":"2024-09-16T03:37:31.054119+0000","last_unstale":"2024-09-16T03:37:38.587182+0000","last_undegraded":"2024-09-16T03:37:38.587182+0000","last_fullsized":"2024-09-16T03:37:38.587182+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1521017,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":77,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.887449+0000","last_change":"2024-09-16T03:36:55.883348+0000","last_active":"2024-09-16T03:37:38.887449+0000","last_peered":"2024-09-16T03:37:38.887449+0000","last_clean":"2024-09-16T03:37:38.887449+0000","last_became_active":"2024-09-16T03:36:53.672370+0000","last_became_peered":"2024-09-16T03:36:53.672370+0000","last_unstale":"2024-09-16T03:37:38.887449+0000","last_undegraded":"2024-09-16T03:37:38.887449+0000","last_fullsized":"2024-09-16T03:37:38.887449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:52.231720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.0","version":"54'52","reported_seq":194,"reported_epoch":56,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:37:39.125875+0000","last_change":"2024-09-16T03:37:39.125679+0000","last_active":"2024-09-16T03:37:39.125875+0000","last_peered":"2024-09-16T03:37:39.125875+0000","last_clean":"2024-09-16T03:37:39.125875+0000","last_became_active":"2024-09-16T03:37:31.151595+0000","last_became_peered":"2024-09-16T03:37:31.151595+0000","last_unstale":"2024-09-16T03:37:39.125875+0000","last_undegraded":"2024-09-16T03:37:39.125875+0000","last_fullsized":"2024-09-16T03:37:39.125875+0000","mapping_epoch":32,"log_start":"52'48","ondisk_log_start":"52'48","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9764768,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":114,"num_read_kb":90784,"num_write":29,"num_write_kb":9540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.690013+0000","last_change":"2024-09-16T03:36:55.605861+0000","last_active":"2024-09-16T03:37:38.690013+0000","last_peered":"2024-09-16T03:37:38.690013+0000","last_clean":"2024-09-16T03:37:38.690013+0000","last_became_active":"2024-09-16T03:36:53.379870+0000","last_became_peered":"2024-09-16T03:36:53.379870+0000","last_unstale":"2024-09-16T03:37:38.690013+0000","last_undegraded":"2024-09-16T03:37:38.690013+0000","last_fullsized":"2024-09-16T03:37:38.690013+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:04.628405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044368400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.3","version":"56'21","reported_seq":124,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:40.700813+0000","last_change":"2024-09-16T03:37:39.685967+0000","last_active":"2024-09-16T03:37:40.700813+0000","last_peered":"2024-09-16T03:37:40.700813+0000","last_clean":"2024-09-16T03:37:40.700813+0000","last_became_active":"2024-09-16T03:37:31.050969+0000","last_became_peered":"2024-09-16T03:37:31.050969+0000","last_unstale":"2024-09-16T03:37:40.700813+0000","last_undegraded":"2024-09-16T03:37:40.700813+0000","last_fullsized":"2024-09-16T03:37:40.700813+0000","mapping_epoch":32,"log_start":"46'17","ondisk_log_start":"46'17","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'21","last_scrub_stamp":"2024-09-16T03:37:39.685904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.685904+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":16,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:53.669073+0000","scrub_duration":949,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3596665,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":81,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:40.375278+0000","last_change":"2024-09-16T03:36:55.270467+0000","last_active":"2024-09-16T03:37:40.375278+0000","last_peered":"2024-09-16T03:37:40.375278+0000","last_clean":"2024-09-16T03:37:40.375278+0000","last_became_active":"2024-09-16T03:36:53.570778+0000","last_became_peered":"2024-09-16T03:36:53.570778+0000","last_unstale":"2024-09-16T03:37:40.375278+0000","last_undegraded":"2024-09-16T03:37:40.375278+0000","last_fullsized":"2024-09-16T03:37:40.375278+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:03.986104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":97,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:40.589116+0000","last_change":"2024-09-16T03:37:39.853712+0000","last_active":"2024-09-16T03:37:40.589116+0000","last_peered":"2024-09-16T03:37:40.589116+0000","last_clean":"2024-09-16T03:37:40.589116+0000","last_became_active":"2024-09-16T03:37:31.150931+0000","last_became_peered":"2024-09-16T03:37:31.150931+0000","last_unstale":"2024-09-16T03:37:40.589116+0000","last_undegraded":"2024-09-16T03:37:40.589116+0000","last_fullsized":"2024-09-16T03:37:40.589116+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:39.853620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.853620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:00.453676+0000","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"58'89","reported_seq":152,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:41.042294+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:37:41.042294+0000","last_peered":"2024-09-16T03:37:41.042294+0000","last_clean":"2024-09-16T03:37:41.042294+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:37:41.042294+0000","last_undegraded":"2024-09-16T03:37:41.042294+0000","last_fullsized":"2024-09-16T03:37:41.042294+0000","mapping_epoch":31,"log_start":"57'86","ondisk_log_start":"57'86","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":98,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:19.199339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14297240,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":69,"num_read_kb":45,"num_write":99,"num_write_kb":35198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":78,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.503918+0000","last_change":"2024-09-16T03:36:55.201161+0000","last_active":"2024-09-16T03:37:38.503918+0000","last_peered":"2024-09-16T03:37:38.503918+0000","last_clean":"2024-09-16T03:37:38.503918+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:37:38.503918+0000","last_undegraded":"2024-09-16T03:37:38.503918+0000","last_fullsized":"2024-09-16T03:37:38.503918+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:07.449762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.4","version":"48'12","reported_seq":112,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:37:40.884449+0000","last_change":"2024-09-16T03:37:40.784192+0000","last_active":"2024-09-16T03:37:40.884449+0000","last_peered":"2024-09-16T03:37:40.884449+0000","last_clean":"2024-09-16T03:37:40.884449+0000","last_became_active":"2024-09-16T03:37:31.249346+0000","last_became_peered":"2024-09-16T03:37:31.249346+0000","last_unstale":"2024-09-16T03:37:40.884449+0000","last_undegraded":"2024-09-16T03:37:40.884449+0000","last_fullsized":"2024-09-16T03:37:40.884449+0000","mapping_epoch":32,"log_start":"44'9","ondisk_log_start":"44'9","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":9,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2798332,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":26600,"num_write":6,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.0","version":"54'80","reported_seq":157,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.690013+0000","last_change":"2024-09-16T03:37:19.299850+0000","last_active":"2024-09-16T03:37:38.690013+0000","last_peered":"2024-09-16T03:37:38.690013+0000","last_clean":"2024-09-16T03:37:38.690013+0000","last_became_active":"2024-09-16T03:37:19.298876+0000","last_became_peered":"2024-09-16T03:37:19.298876+0000","last_unstale":"2024-09-16T03:37:38.690013+0000","last_undegraded":"2024-09-16T03:37:38.690013+0000","last_fullsized":"2024-09-16T03:37:38.690013+0000","mapping_epoch":44,"log_start":"54'77","ondisk_log_start":"54'77","created":31,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":98,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:13.639406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26685815,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":63,"num_read_kb":3105,"num_write":93,"num_write_kb":44801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,4],"acting":[9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":77,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:40.589074+0000","last_change":"2024-09-16T03:36:56.004079+0000","last_active":"2024-09-16T03:37:40.589074+0000","last_peered":"2024-09-16T03:37:40.589074+0000","last_clean":"2024-09-16T03:37:40.589074+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:37:40.589074+0000","last_undegraded":"2024-09-16T03:37:40.589074+0000","last_fullsized":"2024-09-16T03:37:40.589074+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:10.599963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.7","version":"55'28","reported_seq":115,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.687168+0000","last_change":"2024-09-16T03:37:32.515123+0000","last_active":"2024-09-16T03:37:38.687168+0000","last_peered":"2024-09-16T03:37:38.687168+0000","last_clean":"2024-09-16T03:37:38.687168+0000","last_became_active":"2024-09-16T03:37:31.153814+0000","last_became_peered":"2024-09-16T03:37:31.153814+0000","last_unstale":"2024-09-16T03:37:38.687168+0000","last_undegraded":"2024-09-16T03:37:38.687168+0000","last_fullsized":"2024-09-16T03:37:38.687168+0000","mapping_epoch":32,"log_start":"54'25","ondisk_log_start":"54'25","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":23,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2546227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":2111,"num_write":15,"num_write_kb":4722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1","version":"56'68","reported_seq":133,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:40.475386+0000","last_change":"2024-09-16T03:37:05.134238+0000","last_active":"2024-09-16T03:37:40.475386+0000","last_peered":"2024-09-16T03:37:40.475386+0000","last_clean":"2024-09-16T03:37:40.475386+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:37:40.475386+0000","last_undegraded":"2024-09-16T03:37:40.475386+0000","last_fullsized":"2024-09-16T03:37:40.475386+0000","mapping_epoch":31,"log_start":"56'63","ondisk_log_start":"56'63","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":74,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:13.684062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18959689,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":52,"num_read_kb":125,"num_write":75,"num_write_kb":32295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.503647+0000","last_change":"2024-09-16T03:36:55.200565+0000","last_active":"2024-09-16T03:37:38.503647+0000","last_peered":"2024-09-16T03:37:38.503647+0000","last_clean":"2024-09-16T03:37:38.503647+0000","last_became_active":"2024-09-16T03:36:53.182912+0000","last_became_peered":"2024-09-16T03:36:53.182912+0000","last_unstale":"2024-09-16T03:37:38.503647+0000","last_undegraded":"2024-09-16T03:37:38.503647+0000","last_fullsized":"2024-09-16T03:37:38.503647+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:08.335036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.6","version":"49'28","reported_seq":135,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:39.009389+0000","last_change":"2024-09-16T03:37:34.095212+0000","last_active":"2024-09-16T03:37:39.009389+0000","last_peered":"2024-09-16T03:37:39.009389+0000","last_clean":"2024-09-16T03:37:39.009389+0000","last_became_active":"2024-09-16T03:37:31.149856+0000","last_became_peered":"2024-09-16T03:37:31.149856+0000","last_unstale":"2024-09-16T03:37:39.009389+0000","last_undegraded":"2024-09-16T03:37:39.009389+0000","last_fullsized":"2024-09-16T03:37:39.009389+0000","mapping_epoch":32,"log_start":"48'25","ondisk_log_start":"48'25","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":25,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2991613,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":43935,"num_write":13,"num_write_kb":3228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,6],"acting":[5,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"57'100","reported_seq":175,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:40.700736+0000","last_change":"2024-09-16T03:37:04.942264+0000","last_active":"2024-09-16T03:37:40.700736+0000","last_peered":"2024-09-16T03:37:40.700736+0000","last_clean":"2024-09-16T03:37:40.700736+0000","last_became_active":"2024-09-16T03:37:04.941953+0000","last_became_peered":"2024-09-16T03:37:04.941953+0000","last_unstale":"2024-09-16T03:37:40.700736+0000","last_undegraded":"2024-09-16T03:37:40.700736+0000","last_fullsized":"2024-09-16T03:37:40.700736+0000","mapping_epoch":31,"log_start":"57'97","ondisk_log_start":"57'97","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":106,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:04.021943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19879217,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":94,"num_read_kb":5819,"num_write":119,"num_write_kb":38583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":82,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:40.800926+0000","last_change":"2024-09-16T03:36:55.504324+0000","last_active":"2024-09-16T03:37:40.800926+0000","last_peered":"2024-09-16T03:37:40.800926+0000","last_clean":"2024-09-16T03:37:40.800926+0000","last_became_active":"2024-09-16T03:36:53.284975+0000","last_became_peered":"2024-09-16T03:36:53.284975+0000","last_unstale":"2024-09-16T03:37:40.800926+0000","last_undegraded":"2024-09-16T03:37:40.800926+0000","last_fullsized":"2024-09-16T03:37:40.800926+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:55.990331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":216,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:40.700625+0000","last_change":"2024-09-16T03:36:51.858122+0000","last_active":"2024-09-16T03:37:40.700625+0000","last_peered":"2024-09-16T03:37:40.700625+0000","last_clean":"2024-09-16T03:37:40.700625+0000","last_became_active":"2024-09-16T03:36:51.855488+0000","last_became_peered":"2024-09-16T03:36:51.855488+0000","last_unstale":"2024-09-16T03:37:40.700625+0000","last_undegraded":"2024-09-16T03:37:40.700625+0000","last_fullsized":"2024-09-16T03:37:40.700625+0000","mapping_epoch":23,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:36:50.058111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:37:54.474463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.5","version":"57'38","reported_seq":143,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:37:41.388565+0000","last_change":"2024-09-16T03:37:41.388412+0000","last_active":"2024-09-16T03:37:41.388565+0000","last_peered":"2024-09-16T03:37:41.388565+0000","last_clean":"2024-09-16T03:37:41.388565+0000","last_became_active":"2024-09-16T03:37:31.248101+0000","last_became_peered":"2024-09-16T03:37:31.248101+0000","last_unstale":"2024-09-16T03:37:41.388565+0000","last_undegraded":"2024-09-16T03:37:41.388565+0000","last_fullsized":"2024-09-16T03:37:41.388565+0000","mapping_epoch":32,"log_start":"57'34","ondisk_log_start":"57'34","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":33,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7319382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":16513,"num_write":22,"num_write_kb":6732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.8","version":"54'34","reported_seq":142,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.850599+0000","last_change":"2024-09-16T03:37:34.345485+0000","last_active":"2024-09-16T03:37:38.850599+0000","last_peered":"2024-09-16T03:37:38.850599+0000","last_clean":"2024-09-16T03:37:38.850599+0000","last_became_active":"2024-09-16T03:37:31.349116+0000","last_became_peered":"2024-09-16T03:37:31.349116+0000","last_unstale":"2024-09-16T03:37:38.850599+0000","last_undegraded":"2024-09-16T03:37:38.850599+0000","last_fullsized":"2024-09-16T03:37:38.850599+0000","mapping_epoch":32,"log_start":"54'31","ondisk_log_start":"54'31","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":31,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5386577,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":29953,"num_write":20,"num_write_kb":5321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":53,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.787366+0000","last_change":"2024-09-16T03:37:32.516691+0000","last_active":"2024-09-16T03:37:38.787366+0000","last_peered":"2024-09-16T03:37:38.787366+0000","last_clean":"2024-09-16T03:37:38.787366+0000","last_became_active":"2024-09-16T03:37:31.348019+0000","last_became_peered":"2024-09-16T03:37:31.348019+0000","last_unstale":"2024-09-16T03:37:38.787366+0000","last_undegraded":"2024-09-16T03:37:38.787366+0000","last_fullsized":"2024-09-16T03:37:38.787366+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,8],"acting":[14,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.a","version":"56'51","reported_seq":169,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.988648+0000","last_change":"2024-09-16T03:37:06.386519+0000","last_active":"2024-09-16T03:37:38.988648+0000","last_peered":"2024-09-16T03:37:38.988648+0000","last_clean":"2024-09-16T03:37:38.988648+0000","last_became_active":"2024-09-16T03:37:06.286063+0000","last_became_peered":"2024-09-16T03:37:06.286063+0000","last_unstale":"2024-09-16T03:37:38.988648+0000","last_undegraded":"2024-09-16T03:37:38.988648+0000","last_fullsized":"2024-09-16T03:37:38.988648+0000","mapping_epoch":32,"log_start":"54'46","ondisk_log_start":"54'46","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":46,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:23.457175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14353013,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":75706,"num_write":45,"num_write_kb":11857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.b","version":"57'46","reported_seq":151,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T03:37:41.220880+0000","last_change":"2024-09-16T03:37:06.268430+0000","last_active":"2024-09-16T03:37:41.220880+0000","last_peered":"2024-09-16T03:37:41.220880+0000","last_clean":"2024-09-16T03:37:41.220880+0000","last_became_active":"2024-09-16T03:37:06.268115+0000","last_became_peered":"2024-09-16T03:37:06.268115+0000","last_unstale":"2024-09-16T03:37:41.220880+0000","last_undegraded":"2024-09-16T03:37:41.220880+0000","last_fullsized":"2024-09-16T03:37:41.220880+0000","mapping_epoch":32,"log_start":"56'41","ondisk_log_start":"56'41","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":41,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:27.734648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12369820,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":65677,"num_write":41,"num_write_kb":14776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.c","version":"56'29","reported_seq":104,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.369284+0000","last_change":"2024-09-16T03:37:05.939190+0000","last_active":"2024-09-16T03:37:38.369284+0000","last_peered":"2024-09-16T03:37:38.369284+0000","last_clean":"2024-09-16T03:37:38.369284+0000","last_became_active":"2024-09-16T03:37:05.938951+0000","last_became_peered":"2024-09-16T03:37:05.938951+0000","last_unstale":"2024-09-16T03:37:38.369284+0000","last_undegraded":"2024-09-16T03:37:38.369284+0000","last_fullsized":"2024-09-16T03:37:38.369284+0000","mapping_epoch":32,"log_start":"54'25","ondisk_log_start":"54'25","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":25,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:23.808993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7109295,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":4628,"num_write":29,"num_write_kb":7646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,1],"acting":[12,15,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.d","version":"47'5","reported_seq":74,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.369259+0000","last_change":"2024-09-16T03:37:06.033957+0000","last_active":"2024-09-16T03:37:38.369259+0000","last_peered":"2024-09-16T03:37:38.369259+0000","last_clean":"2024-09-16T03:37:38.369259+0000","last_became_active":"2024-09-16T03:37:06.033717+0000","last_became_peered":"2024-09-16T03:37:06.033717+0000","last_unstale":"2024-09-16T03:37:38.369259+0000","last_undegraded":"2024-09-16T03:37:38.369259+0000","last_fullsized":"2024-09-16T03:37:38.369259+0000","mapping_epoch":32,"log_start":"39'1","ondisk_log_start":"39'1","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":1,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:05.360000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":190,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,1,13],"acting":[12,1,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.e","version":"53'30","reported_seq":114,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:38.587229+0000","last_change":"2024-09-16T03:37:06.037443+0000","last_active":"2024-09-16T03:37:38.587229+0000","last_peered":"2024-09-16T03:37:38.587229+0000","last_clean":"2024-09-16T03:37:38.587229+0000","last_became_active":"2024-09-16T03:37:06.037035+0000","last_became_peered":"2024-09-16T03:37:06.037035+0000","last_unstale":"2024-09-16T03:37:38.587229+0000","last_undegraded":"2024-09-16T03:37:38.587229+0000","last_fullsized":"2024-09-16T03:37:38.587229+0000","mapping_epoch":32,"log_start":"48'26","ondisk_log_start":"48'26","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":26,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:09.704413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5983226,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":28445,"num_write":27,"num_write_kb":6900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"56'49","reported_seq":164,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T03:37:39.103511+0000","last_change":"2024-09-16T03:37:06.386966+0000","last_active":"2024-09-16T03:37:39.103511+0000","last_peered":"2024-09-16T03:37:39.103511+0000","last_clean":"2024-09-16T03:37:39.103511+0000","last_became_active":"2024-09-16T03:37:06.386648+0000","last_became_peered":"2024-09-16T03:37:06.386648+0000","last_unstale":"2024-09-16T03:37:39.103511+0000","last_undegraded":"2024-09-16T03:37:39.103511+0000","last_fullsized":"2024-09-16T03:37:39.103511+0000","mapping_epoch":32,"log_start":"54'45","ondisk_log_start":"54'45","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":45,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:07.955986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15159749,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":121,"num_read_kb":90065,"num_write":45,"num_write_kb":13390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":128252814,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":802,"num_read_kb":568094,"num_write":353,"num_write_kb":102582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":25,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216076288,"data_stored":215747859,"data_compressed":798465,"data_compressed_allocated":872448,"data_compressed_original":1757184,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":78,"acting":78,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":79821961,"num_objects":91,"num_object_clones":0,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":278,"num_read_kb":9094,"num_write":386,"num_write_kb":150877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":130969600,"data_stored":142171847,"data_compressed":9512289,"data_compressed_allocated":10145792,"data_compressed_original":21688320,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":1586784,"data_compressed":93744,"data_compressed_allocated":122880,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247820,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76728,"kb_used_data":45408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94295112,"statfs":{"total":96636764160,"available":96558194688,"internally_reserved":0,"allocated":46497792,"data_stored":46871164,"data_compressed":818911,"data_compressed_allocated":864256,"data_compressed_original":1638400,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247821,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58184,"kb_used_data":30064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94313656,"statfs":{"total":96636764160,"available":96577183744,"internally_reserved":0,"allocated":30785536,"data_stored":31956327,"data_compressed":914876,"data_compressed_allocated":991232,"data_compressed_original":2539520,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,4,5,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247820,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45328,"kb_used_data":16568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94326512,"statfs":{"total":96636764160,"available":96590348288,"internally_reserved":0,"allocated":16965632,"data_stored":17660930,"data_compressed":783867,"data_compressed_allocated":843776,"data_compressed_original":1785856,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,4,5,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280525,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47140,"kb_used_data":19496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324700,"statfs":{"total":96636764160,"available":96588492800,"internally_reserved":0,"allocated":19963904,"data_stored":21255585,"data_compressed":914925,"data_compressed_allocated":987136,"data_compressed_original":2543616,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280525,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28484,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343356,"statfs":{"total":96636764160,"available":96607596544,"internally_reserved":0,"allocated":1486848,"data_stored":1261027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280524,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49920,"kb_used_data":21840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94321920,"statfs":{"total":96636764160,"available":96585646080,"internally_reserved":0,"allocated":22364160,"data_stored":23272785,"data_compressed":854511,"data_compressed_allocated":913408,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,4,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313229,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34728,"kb_used_data":7700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337112,"statfs":{"total":96636764160,"available":96601202688,"internally_reserved":0,"allocated":7884800,"data_stored":7595215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,7,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280525,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54784,"kb_used_data":23592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94317056,"statfs":{"total":96636764160,"available":96580665344,"internally_reserved":0,"allocated":24158208,"data_stored":24719848,"data_compressed":802438,"data_compressed_allocated":860160,"data_compressed_original":1724416,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280524,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70468,"kb_used_data":43444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94301372,"statfs":{"total":96636764160,"available":96564604928,"internally_reserved":0,"allocated":44486656,"data_stored":46360433,"data_compressed":1804136,"data_compressed_allocated":1933312,"data_compressed_original":4153344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,5,7,8,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313229,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58488,"kb_used_data":28268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94313352,"statfs":{"total":96636764160,"available":96576872448,"internally_reserved":0,"allocated":28946432,"data_stored":30044604,"data_compressed":878977,"data_compressed_allocated":946176,"data_compressed_original":2412544,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280524,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35244,"kb_used_data":4052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336596,"statfs":{"total":96636764160,"available":96600674304,"internally_reserved":0,"allocated":4149248,"data_stored":3934342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280524,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56356,"kb_used_data":28544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94315484,"statfs":{"total":96636764160,"available":96579055616,"internally_reserved":0,"allocated":29229056,"data_stored":30186082,"data_compressed":1017808,"data_compressed_allocated":1081344,"data_compressed_original":2338816,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280524,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48516,"kb_used_data":20176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94323324,"statfs":{"total":96636764160,"available":96587083776,"internally_reserved":0,"allocated":20660224,"data_stored":21735439,"data_compressed":962517,"data_compressed_allocated":1032192,"data_compressed_original":2359296,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,4,6,7,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280525,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61848,"kb_used_data":32980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94309992,"statfs":{"total":96636764160,"available":96573431808,"internally_reserved":0,"allocated":33771520,"data_stored":33530722,"data_compressed":158149,"data_compressed_allocated":172032,"data_compressed_original":323584,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,6,7,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313228,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":3724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":3813376,"data_stored":3578182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,5,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247821,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39724,"kb_used_data":12692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332116,"statfs":{"total":96636764160,"available":96596086784,"internally_reserved":0,"allocated":12996608,"data_stored":13423235,"data_compressed":701550,"data_compressed_allocated":733184,"data_compressed_original":1376256,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12562711,"data_compressed":883677,"data_compressed_allocated":946176,"data_compressed_original":2031616,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13078528,"data_stored":14236934,"data_compressed":854511,"data_compressed_allocated":913408,"data_compressed_original":2097152,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9891840,"data_stored":10501589,"data_compressed":701550,"data_compressed_allocated":733184,"data_compressed_original":1376256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10022912,"data_stored":10921772,"data_compressed":808648,"data_compressed_allocated":868352,"data_compressed_original":1789952,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":22188032,"data_stored":24160384,"data_compressed":1556482,"data_compressed_allocated":1667072,"data_compressed_original":3690496,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12562711,"data_compressed":883677,"data_compressed_allocated":946176,"data_compressed_original":2031616,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":9109504,"data_stored":9923450,"data_compressed":701971,"data_compressed_allocated":753664,"data_compressed_original":1593344,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11522048,"data_stored":12512520,"data_compressed":830878,"data_compressed_allocated":880640,"data_compressed_original":1900544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":13168640,"data_stored":14323323,"data_compressed":854511,"data_compressed_allocated":913408,"data_compressed_original":2097152,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":9809920,"data_stored":10751296,"data_compressed":801732,"data_compressed_allocated":860160,"data_compressed_original":1835008,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":9166848,"data_stored":9715157,"data_compressed":634652,"data_compressed_allocated":663552,"data_compressed_original":1245184,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7880704,"data_stored":7842033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11612160,"data_stored":11533372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3141632,"data_stored":3124316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":15294464,"data_stored":15201579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20852736,"data_stored":20954308,"data_compressed":184259,"data_compressed_allocated":200704,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16900096,"data_stored":16804052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8187904,"data_stored":8251285,"data_compressed":81896,"data_compressed_allocated":90112,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3637248,"data_stored":3611667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":17428480,"data_stored":17319986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":6426624,"data_stored":6549431,"data_compressed":108006,"data_compressed_allocated":118784,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":33234944,"data_stored":33206283,"data_compressed":158149,"data_compressed_allocated":172032,"data_compressed_original":323584,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20369408,"data_stored":20341309,"data_compressed":81896,"data_compressed_allocated":90112,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4927488,"data_stored":4884376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":935448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":8413184,"data_stored":8361986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":36827136,"data_stored":36826428,"data_compressed":184259,"data_compressed_allocated":200704,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:37:44.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 26 2024-09-16T03:37:44.673 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:44.673 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:44.674 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:37:44.688 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:44.688 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:44.705 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:44.705 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:44.711 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:44.709+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:44.721 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:44.722 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:44.739 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:44.740 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:44.758 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:44.758 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:44.774 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:44.774 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:44.778 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:44.778 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:44.778 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:37:44.779 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:37:44.779 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:44.787 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:37:44.791 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:44.792 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:44.809 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:44.809 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:44.826 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:44.826 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:44.831 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:44.829+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:44.843 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:44.844 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:44.845 INFO:tasks.rados.rados.0.smithi157.stdout:843: set_chunk oid 41 offset: 206336 length: 2233344 target oid d2796490e6261f1e80454b988343176471943849340b4a42563bfb42f0f0f217 offset: 0 snap -1 2024-09-16T03:37:44.845 INFO:tasks.rados.rados.0.smithi157.stdout:844: read oid 17 snap -1 2024-09-16T03:37:44.845 INFO:tasks.rados.rados.0.smithi157.stdout:844: expect (ObjNum 86 snap 0 seq_num 86) 2024-09-16T03:37:44.845 INFO:tasks.rados.rados.0.smithi157.stdout:845: tier_evict oid 6 2024-09-16T03:37:44.845 INFO:tasks.rados.rados.0.smithi157.stdout:845: tier_evict oid 6 snap -1 2024-09-16T03:37:44.845 INFO:tasks.rados.rados.0.smithi157.stdout:846: read oid 8 snap -1 2024-09-16T03:37:44.845 INFO:tasks.rados.rados.0.smithi157.stdout:846: expect deleted 2024-09-16T03:37:44.845 INFO:tasks.rados.rados.0.smithi157.stdout:847: read oid 20 snap -1 2024-09-16T03:37:44.846 INFO:tasks.rados.rados.0.smithi157.stdout:847: expect (ObjNum 102 snap 0 seq_num 102) 2024-09-16T03:37:44.846 INFO:tasks.rados.rados.0.smithi157.stdout:848: set_chunk oid 24 target oid 2024-09-16T03:37:44.860 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:44.860 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:44.877 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:44.878 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:44.892 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:37:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:21 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'4, mlcod=0'0)", 2024-09-16T03:37:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:37:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 32.702642863000001, 2024-09-16T03:37:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10231100799999999, 2024-09-16T03:37:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:37:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:37:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187300+0000", 2024-09-16T03:37:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0360000000000001e-06 2024-09-16T03:37:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187905+0000", 2024-09-16T03:37:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060508100000000004 2024-09-16T03:37:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187920+0000", 2024-09-16T03:37:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4314000000000001e-05 2024-09-16T03:37:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187928+0000", 2024-09-16T03:37:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7150000000000004e-06 2024-09-16T03:37:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187984+0000", 2024-09-16T03:37:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5872999999999999e-05 2024-09-16T03:37:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188016+0000", 2024-09-16T03:37:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2045000000000002e-05 2024-09-16T03:37:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289567+0000", 2024-09-16T03:37:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101550875 2024-09-16T03:37:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289609+0000", 2024-09-16T03:37:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2069e-05 2024-09-16T03:37:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:22 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'5, mlcod=0'0)", 2024-09-16T03:37:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187945+0000", 2024-09-16T03:37:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 32.701996096999999, 2024-09-16T03:37:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101698394, 2024-09-16T03:37:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187945+0000", 2024-09-16T03:37:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187945+0000", 2024-09-16T03:37:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187946+0000", 2024-09-16T03:37:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9900000000000001e-07 2024-09-16T03:37:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187959+0000", 2024-09-16T03:37:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3242e-05 2024-09-16T03:37:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187963+0000", 2024-09-16T03:37:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8410000000000002e-06 2024-09-16T03:37:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187966+0000", 2024-09-16T03:37:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9390000000000002e-06 2024-09-16T03:37:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188816+0000", 2024-09-16T03:37:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085090000000000003 2024-09-16T03:37:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188836+0000", 2024-09-16T03:37:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9638000000000001e-05 2024-09-16T03:37:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289615+0000", 2024-09-16T03:37:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10077847299999999 2024-09-16T03:37:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289643+0000", 2024-09-16T03:37:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8861999999999999e-05 2024-09-16T03:37:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:23 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'6, mlcod=0'0)", 2024-09-16T03:37:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187987+0000", 2024-09-16T03:37:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 32.701953819000003, 2024-09-16T03:37:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102020717, 2024-09-16T03:37:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187987+0000", 2024-09-16T03:37:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187987+0000", 2024-09-16T03:37:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187988+0000", 2024-09-16T03:37:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9900000000000001e-07 2024-09-16T03:37:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187992+0000", 2024-09-16T03:37:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.707e-06 2024-09-16T03:37:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187998+0000", 2024-09-16T03:37:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7699999999999998e-06 2024-09-16T03:37:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188003+0000", 2024-09-16T03:37:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8459999999999999e-06 2024-09-16T03:37:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289285+0000", 2024-09-16T03:37:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101282107 2024-09-16T03:37:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289343+0000", 2024-09-16T03:37:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7852000000000001e-05 2024-09-16T03:37:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289968+0000", 2024-09-16T03:37:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062506400000000004 2024-09-16T03:37:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.290008+0000", 2024-09-16T03:37:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9872e-05 2024-09-16T03:37:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:81 4.c e39/32 4:3313e97c:::smithi15736696-21:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.398719+0000", 2024-09-16T03:37:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 32.491221842000002, 2024-09-16T03:37:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102114725, 2024-09-16T03:37:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.398719+0000", 2024-09-16T03:37:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.398719+0000", 2024-09-16T03:37:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.398722+0000", 2024-09-16T03:37:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7199999999999998e-06 2024-09-16T03:37:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.399189+0000", 2024-09-16T03:37:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046724000000000001 2024-09-16T03:37:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.399200+0000", 2024-09-16T03:37:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1068e-05 2024-09-16T03:37:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499313+0000", 2024-09-16T03:37:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100112249 2024-09-16T03:37:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499411+0000", 2024-09-16T03:37:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8930000000000003e-05 2024-09-16T03:37:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499444+0000", 2024-09-16T03:37:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2966000000000003e-05 2024-09-16T03:37:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.500783+0000", 2024-09-16T03:37:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013380880000000001 2024-09-16T03:37:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.500834+0000", 2024-09-16T03:37:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1464000000000001e-05 2024-09-16T03:37:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:279 4.8 e41/32 4:1f63731d:::smithi15736696-30:head v 41'15, mlcod=40'13)", 2024-09-16T03:37:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.500221+0000", 2024-09-16T03:37:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.389720276999999, 2024-09-16T03:37:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100673015, 2024-09-16T03:37:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500221+0000", 2024-09-16T03:37:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500221+0000", 2024-09-16T03:37:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500221+0000", 2024-09-16T03:37:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3500000000000002e-07 2024-09-16T03:37:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500229+0000", 2024-09-16T03:37:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7209999999999994e-06 2024-09-16T03:37:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500242+0000", 2024-09-16T03:37:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2699e-05 2024-09-16T03:37:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.600342+0000", 2024-09-16T03:37:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100100734 2024-09-16T03:37:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.600425+0000", 2024-09-16T03:37:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2952000000000001e-05 2024-09-16T03:37:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.600447+0000", 2024-09-16T03:37:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.181e-05 2024-09-16T03:37:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.600853+0000", 2024-09-16T03:37:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040597400000000003 2024-09-16T03:37:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.600894+0000", 2024-09-16T03:37:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0790000000000001e-05 2024-09-16T03:37:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:302 4.8 e42/32 4:13b03fc9:::smithi15736696-3:head v 42'16, mlcod=40'14)", 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.317007+0000", 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 28.572934318000001, 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100648134, 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.317007+0000", 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.317007+0000", 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.317007+0000", 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4499999999999997e-07 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.317012+0000", 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5730000000000001e-06 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.317024+0000", 2024-09-16T03:37:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2659e-05 2024-09-16T03:37:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.417120+0000", 2024-09-16T03:37:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100095954 2024-09-16T03:37:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.417163+0000", 2024-09-16T03:37:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2546999999999998e-05 2024-09-16T03:37:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.417177+0000", 2024-09-16T03:37:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3754000000000001e-05 2024-09-16T03:37:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.417620+0000", 2024-09-16T03:37:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00044358400000000001 2024-09-16T03:37:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.417655+0000", 2024-09-16T03:37:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4618e-05 2024-09-16T03:37:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.5.21:6 3.0 e43/42 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 43'21, mlcod=0'0)", 2024-09-16T03:37:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:17.284972+0000", 2024-09-16T03:37:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.604969252, 2024-09-16T03:37:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.1008151, 2024-09-16T03:37:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284972+0000", 2024-09-16T03:37:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284972+0000", 2024-09-16T03:37:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284973+0000", 2024-09-16T03:37:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8299999999999995e-07 2024-09-16T03:37:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284988+0000", 2024-09-16T03:37:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4946000000000001e-05 2024-09-16T03:37:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284997+0000", 2024-09-16T03:37:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3440000000000007e-06 2024-09-16T03:37:44.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385069+0000", 2024-09-16T03:37:44.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10007182000000001 2024-09-16T03:37:44.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385127+0000", 2024-09-16T03:37:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7676999999999997e-05 2024-09-16T03:37:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385152+0000", 2024-09-16T03:37:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5395999999999999e-05 2024-09-16T03:37:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385754+0000", 2024-09-16T03:37:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060247899999999995 2024-09-16T03:37:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385787+0000", 2024-09-16T03:37:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2555000000000003e-05 2024-09-16T03:37:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:26 3.0 e45/44 3:114c937e:::smithi15736696-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'36, mlcod=45'34)", 2024-09-16T03:37:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.106326+0000", 2024-09-16T03:37:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.783614712999999, 2024-09-16T03:37:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100770135, 2024-09-16T03:37:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.106326+0000", 2024-09-16T03:37:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.106326+0000", 2024-09-16T03:37:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.106328+0000", 2024-09-16T03:37:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7680000000000001e-06 2024-09-16T03:37:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.106341+0000", 2024-09-16T03:37:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2727e-05 2024-09-16T03:37:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.106354+0000", 2024-09-16T03:37:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3569e-05 2024-09-16T03:37:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.206423+0000", 2024-09-16T03:37:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100068303 2024-09-16T03:37:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.206486+0000", 2024-09-16T03:37:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3242000000000002e-05 2024-09-16T03:37:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.206511+0000", 2024-09-16T03:37:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5015999999999999e-05 2024-09-16T03:37:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.207066+0000", 2024-09-16T03:37:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00055467199999999998 2024-09-16T03:37:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.207097+0000", 2024-09-16T03:37:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0837999999999998e-05 2024-09-16T03:37:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.21:5 3.0 e45/44 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'39, mlcod=45'37)", 2024-09-16T03:37:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:37:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.975345828999998, 2024-09-16T03:37:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20119541799999999, 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914597+0000", 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.494e-06 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914603+0000", 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7349999999999999e-06 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914614+0000", 2024-09-16T03:37:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0995999999999999e-05 2024-09-16T03:37:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914623+0000", 2024-09-16T03:37:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3899999999999993e-06 2024-09-16T03:37:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.114941+0000", 2024-09-16T03:37:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20031812399999999 2024-09-16T03:37:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115001+0000", 2024-09-16T03:37:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0161999999999998e-05 2024-09-16T03:37:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115750+0000", 2024-09-16T03:37:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074851699999999998 2024-09-16T03:37:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115791+0000", 2024-09-16T03:37:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1e-05 2024-09-16T03:37:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:432 4.c e45/32 4:34492f38:::smithi15736696-33:head v 45'10, mlcod=43'8)", 2024-09-16T03:37:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.217233+0000", 2024-09-16T03:37:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.672708426, 2024-09-16T03:37:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100706, 2024-09-16T03:37:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217233+0000", 2024-09-16T03:37:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217233+0000", 2024-09-16T03:37:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217234+0000", 2024-09-16T03:37:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6750000000000001e-06 2024-09-16T03:37:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217243+0000", 2024-09-16T03:37:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4600000000000003e-06 2024-09-16T03:37:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217256+0000", 2024-09-16T03:37:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3417999999999999e-05 2024-09-16T03:37:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317325+0000", 2024-09-16T03:37:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100068803 2024-09-16T03:37:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317390+0000", 2024-09-16T03:37:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5381000000000003e-05 2024-09-16T03:37:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317415+0000", 2024-09-16T03:37:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4487000000000001e-05 2024-09-16T03:37:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317904+0000", 2024-09-16T03:37:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00048901100000000002 2024-09-16T03:37:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317939+0000", 2024-09-16T03:37:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4764999999999997e-05 2024-09-16T03:37:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:854 4.c e47/32 4:34492f38:::smithi15736696-33:head v 47'13, mlcod=45'10)", 2024-09-16T03:37:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.579183+0000", 2024-09-16T03:37:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.310758525000001, 2024-09-16T03:37:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10085426, 2024-09-16T03:37:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.579183+0000", 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.579183+0000", 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.579184+0000", 2024-09-16T03:37:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.838e-06 2024-09-16T03:37:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.579554+0000", 2024-09-16T03:37:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036928699999999999 2024-09-16T03:37:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.579569+0000", 2024-09-16T03:37:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5688e-05 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.579578+0000", 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1310000000000006e-06 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580977+0000", 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013990750000000001 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580995+0000", 2024-09-16T03:37:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8598e-05 2024-09-16T03:37:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.679994+0000", 2024-09-16T03:37:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098998728999999994 2024-09-16T03:37:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.680037+0000", 2024-09-16T03:37:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2914000000000003e-05 2024-09-16T03:37:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:855 4.c e47/32 4:34492f38:::smithi15736696-33:head v 47'14, mlcod=45'10)", 2024-09-16T03:37:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.580582+0000", 2024-09-16T03:37:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.309358709000001, 2024-09-16T03:37:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099491817999999996, 2024-09-16T03:37:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580582+0000", 2024-09-16T03:37:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580582+0000", 2024-09-16T03:37:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580584+0000", 2024-09-16T03:37:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.172e-06 2024-09-16T03:37:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580901+0000", 2024-09-16T03:37:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00031786999999999999 2024-09-16T03:37:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580913+0000", 2024-09-16T03:37:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1884e-05 2024-09-16T03:37:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.580920+0000", 2024-09-16T03:37:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9040000000000003e-06 2024-09-16T03:37:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.582092+0000", 2024-09-16T03:37:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011721469999999999 2024-09-16T03:37:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.582119+0000", 2024-09-16T03:37:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6437000000000001e-05 2024-09-16T03:37:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.680039+0000", 2024-09-16T03:37:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097920046999999996 2024-09-16T03:37:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.680074+0000", 2024-09-16T03:37:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5357e-05 2024-09-16T03:37:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:856 4.c e47/32 4:34492f38:::smithi15736696-33:head v 47'15, mlcod=45'10)", 2024-09-16T03:37:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.581380+0000", 2024-09-16T03:37:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.308561219000001, 2024-09-16T03:37:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099086726999999999, 2024-09-16T03:37:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.581380+0000", 2024-09-16T03:37:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.581380+0000", 2024-09-16T03:37:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.581381+0000", 2024-09-16T03:37:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.362e-06 2024-09-16T03:37:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.581395+0000", 2024-09-16T03:37:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.379e-05 2024-09-16T03:37:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.581405+0000", 2024-09-16T03:37:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7610000000000002e-06 2024-09-16T03:37:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.581412+0000", 2024-09-16T03:37:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6070000000000002e-06 2024-09-16T03:37:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.679721+0000", 2024-09-16T03:37:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098308393999999993 2024-09-16T03:37:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.679744+0000", 2024-09-16T03:37:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3495000000000001e-05 2024-09-16T03:37:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.680433+0000", 2024-09-16T03:37:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00068914900000000001 2024-09-16T03:37:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.680467+0000", 2024-09-16T03:37:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3169e-05 2024-09-16T03:37:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.8.22:16 3.0 e47/44 3:037ddf75:::smithi15736696-34 low_tier:head v 47'52, mlcod=47'50)", 2024-09-16T03:37:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.939321+0000", 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.950619812999999, 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10108816800000001, 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.939321+0000", 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.939321+0000", 2024-09-16T03:37:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.939323+0000", 2024-09-16T03:37:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.553e-06 2024-09-16T03:37:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.939330+0000", 2024-09-16T03:37:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4200000000000001e-06 2024-09-16T03:37:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.939347+0000", 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6481999999999999e-05 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.039479+0000", 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100131839 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.039641+0000", 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016204899999999999 2024-09-16T03:37:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.039705+0000", 2024-09-16T03:37:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4146999999999994e-05 2024-09-16T03:37:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.040350+0000", 2024-09-16T03:37:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064533099999999996 2024-09-16T03:37:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.040409+0000", 2024-09-16T03:37:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9346999999999999e-05 2024-09-16T03:37:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.8.22:18 3.0 e47/44 3:037ddf75:::smithi15736696-34 low_tier:head v 47'54, mlcod=47'50)", 2024-09-16T03:37:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:27.140611+0000", 2024-09-16T03:37:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.749329685999999, 2024-09-16T03:37:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101057298, 2024-09-16T03:37:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140611+0000", 2024-09-16T03:37:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140611+0000", 2024-09-16T03:37:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140614+0000", 2024-09-16T03:37:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2989999999999999e-06 2024-09-16T03:37:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140628+0000", 2024-09-16T03:37:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4571e-05 2024-09-16T03:37:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140645+0000", 2024-09-16T03:37:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6568000000000001e-05 2024-09-16T03:37:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140655+0000", 2024-09-16T03:37:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7620000000000007e-06 2024-09-16T03:37:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.240832+0000", 2024-09-16T03:37:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100177195 2024-09-16T03:37:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.240849+0000", 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7323e-05 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.241630+0000", 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000780545 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.241669+0000", 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9035000000000001e-05 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.5.0:54 4.8 e52/50 4:148e627d:::smithi15736696-23:head v 52'26, mlcod=0'0)", 2024-09-16T03:37:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.783077+0000", 2024-09-16T03:37:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.106863935, 2024-09-16T03:37:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10065676, 2024-09-16T03:37:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783077+0000", 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783077+0000", 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783077+0000", 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.91e-07 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783082+0000", 2024-09-16T03:37:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5279999999999997e-06 2024-09-16T03:37:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783093+0000", 2024-09-16T03:37:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1058e-05 2024-09-16T03:37:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783100+0000", 2024-09-16T03:37:44.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0890000000000002e-06 2024-09-16T03:37:44.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883217+0000", 2024-09-16T03:37:44.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100116934 2024-09-16T03:37:44.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883233+0000", 2024-09-16T03:37:44.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.575e-05 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883705+0000", 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047257799999999997 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883734+0000", 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8632e-05 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.975 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.5.21:36 3.0 e52/44 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 52'62, mlcod=51'60)", 2024-09-16T03:37:44.975 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:37:44.975 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.005315169999999, 2024-09-16T03:37:44.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20104840199999999, 2024-09-16T03:37:44.975 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.975 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.975 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.975 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:37:44.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:37:44.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884627+0000", 2024-09-16T03:37:44.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.322e-06 2024-09-16T03:37:44.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884634+0000", 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884643+0000", 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7680000000000003e-06 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984740+0000", 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100096066 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084931+0000", 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100191408 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084952+0000", 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.128e-05 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085645+0000", 2024-09-16T03:37:44.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00069309500000000002 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085674+0000", 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9025000000000001e-05 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:49 3.0 e54/44 3:114c937e:::smithi15736696-39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 54'80, mlcod=54'77)", 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:36.678885+0000", 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 8.2110565179999995, 2024-09-16T03:37:44.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100901088, 2024-09-16T03:37:44.980 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.980 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.980 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.980 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.678885+0000", 2024-09-16T03:37:44.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.678885+0000", 2024-09-16T03:37:44.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.678886+0000", 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1850000000000001e-06 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.678892+0000", 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3439999999999997e-06 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.678901+0000", 2024-09-16T03:37:44.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6680000000000007e-06 2024-09-16T03:37:44.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.779002+0000", 2024-09-16T03:37:44.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10010157 2024-09-16T03:37:44.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.779055+0000", 2024-09-16T03:37:44.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2437e-05 2024-09-16T03:37:44.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.779074+0000", 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9213999999999999e-05 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.779745+0000", 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00067053299999999996 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.779786+0000", 2024-09-16T03:37:44.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1137000000000003e-05 2024-09-16T03:37:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1401 4.8 e58/50 4:146f84f0:::smithi15736696-6:head v 58'39, mlcod=56'37)", 2024-09-16T03:37:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:41.096102+0000", 2024-09-16T03:37:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.7938387950000001, 2024-09-16T03:37:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.010000461, 2024-09-16T03:37:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.984 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.096102+0000", 2024-09-16T03:37:44.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.096102+0000", 2024-09-16T03:37:44.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.096103+0000", 2024-09-16T03:37:44.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6899999999999998e-07 2024-09-16T03:37:44.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.096445+0000", 2024-09-16T03:37:44.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00034268299999999999 2024-09-16T03:37:44.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.096462+0000", 2024-09-16T03:37:44.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6909e-05 2024-09-16T03:37:44.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.096477+0000", 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4925e-05 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.096550+0000", 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3248000000000001e-05 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.096580+0000", 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9405000000000001e-05 2024-09-16T03:37:44.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.106064+0000", 2024-09-16T03:37:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0094842299999999997 2024-09-16T03:37:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.106103+0000", 2024-09-16T03:37:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8692000000000002e-05 2024-09-16T03:37:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1402 4.8 e58/50 4:146f84f0:::smithi15736696-6:head v 58'40, mlcod=56'37)", 2024-09-16T03:37:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:41.098348+0000", 2024-09-16T03:37:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.7915934039999999, 2024-09-16T03:37:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0077751829999999997, 2024-09-16T03:37:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.098348+0000", 2024-09-16T03:37:44.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.098348+0000", 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.098349+0000", 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.592e-06 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:44.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.098760+0000", 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00041025300000000002 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.098773+0000", 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3427e-05 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.098781+0000", 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3180000000000007e-06 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.098841+0000", 2024-09-16T03:37:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9592000000000001e-05 2024-09-16T03:37:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.098858+0000", 2024-09-16T03:37:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.73e-05 2024-09-16T03:37:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.106107+0000", 2024-09-16T03:37:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0072486989999999999 2024-09-16T03:37:44.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:41.106123+0000", 2024-09-16T03:37:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6002e-05 2024-09-16T03:37:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:44.993 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:44.993 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:44.994 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:44.995 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:37:44.996 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:44.996 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:44.992+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:45.096 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:45.094+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:45.139 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:45.140 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:45.140 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:45.140 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:45.148 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:37:45.197 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:45.195+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:45.251 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:45.251 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:45.251 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:37:45.251 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:37:45.251 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:45.256 INFO:tasks.rados.rados.0.smithi157.stdout:848: set_chunk oid 24 offset: 1058816 length: 1094656 target oid 7249cb65b52cadf4fe8f4ec923c7e548d1de7d9628823acd2c8de19121f1c1ce offset: 0 snap -1 2024-09-16T03:37:45.256 INFO:tasks.rados.rados.0.smithi157.stdout:849: set_chunk oid 25 target oid 2024-09-16T03:37:45.259 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:37:45.300 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:45.298+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:45.364 INFO:tasks.rados.rados.0.smithi157.stdout:849: set_chunk oid 25 offset: 1787392 length: 224256 target oid e06092e68680905b5745d5057e93f6c7581518aed07ca5db2802546e9c29aeef offset: 0 snap -1 2024-09-16T03:37:45.364 INFO:tasks.rados.rados.0.smithi157.stdout:850: read oid 45 snap -1 2024-09-16T03:37:45.364 INFO:tasks.rados.rados.0.smithi157.stdout:850: expect (ObjNum 85 snap 0 seq_num 85) 2024-09-16T03:37:45.364 INFO:tasks.rados.rados.0.smithi157.stdout:851: tier_promote oid 27 2024-09-16T03:37:45.364 INFO:tasks.rados.rados.0.smithi157.stdout:845: got (22) Invalid argument 2024-09-16T03:37:45.364 INFO:tasks.rados.rados.0.smithi157.stdout:845: done (8 left) 2024-09-16T03:37:45.364 INFO:tasks.rados.rados.0.smithi157.stdout:852: read oid 47 snap -1 2024-09-16T03:37:45.364 INFO:tasks.rados.rados.0.smithi157.stdout:852: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-16T03:37:45.365 INFO:tasks.rados.rados.0.smithi157.stdout:853: set_chunk oid 28 target oid 2024-09-16T03:37:45.366 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:37:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:25 4.6 4:6ece7830:::smithi15736696-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:37:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.242310203999999, 2024-09-16T03:37:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40559452699999998, 2024-09-16T03:37:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 25 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119830+0000", 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9200000000000001e-07 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120013+0000", 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018367200000000001 2024-09-16T03:37:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120021+0000", 2024-09-16T03:37:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9179999999999994e-06 2024-09-16T03:37:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120027+0000", 2024-09-16T03:37:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6710000000000004e-06 2024-09-16T03:37:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220194+0000", 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10016689500000001 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220399+0000", 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020502899999999999 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220613+0000", 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021383900000000001 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224640+0000", 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040268099999999996 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423348+0000", 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19870853399999999 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525343+0000", 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101995081 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525381+0000", 2024-09-16T03:37:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7842000000000001e-05 2024-09-16T03:37:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525424+0000", 2024-09-16T03:37:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2644e-05 2024-09-16T03:37:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:26 4.6 4:6ece7830:::smithi15736696-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.242097977, 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550111300000002, 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 26 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:37:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:37:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120042+0000", 2024-09-16T03:37:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7999999999999995e-07 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120210+0000", 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016779499999999999 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120219+0000", 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6859999999999995e-06 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120223+0000", 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0330000000000002e-06 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221494+0000", 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101271812 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221572+0000", 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7733999999999998e-05 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221697+0000", 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012474900000000001 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224670+0000", 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029729589999999998 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423398+0000", 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19872784900000001 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525465+0000", 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102067557 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525494+0000", 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9113999999999999e-05 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525542+0000", 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8145000000000001e-05 2024-09-16T03:37:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:27 4.6 4:6ece7830:::smithi15736696-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.241910392000001, 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40541507399999999, 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27 2024-09-16T03:37:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:37:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:37:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120230+0000", 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5799999999999999e-07 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120233+0000", 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.055e-06 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120237+0000", 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7099999999999998e-06 2024-09-16T03:37:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120244+0000", 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4810000000000003e-06 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222660+0000", 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102416089 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222705+0000", 2024-09-16T03:37:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5368999999999999e-05 2024-09-16T03:37:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222782+0000", 2024-09-16T03:37:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6623000000000001e-05 2024-09-16T03:37:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224696+0000", 2024-09-16T03:37:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019143929999999999 2024-09-16T03:37:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423402+0000", 2024-09-16T03:37:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198705305 2024-09-16T03:37:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525582+0000", 2024-09-16T03:37:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102180178 2024-09-16T03:37:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525615+0000", 2024-09-16T03:37:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3303000000000003e-05 2024-09-16T03:37:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525644+0000", 2024-09-16T03:37:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8909999999999999e-05 2024-09-16T03:37:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:28 4.6 4:6ece7830:::smithi15736696-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:37:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.241889643999997, 2024-09-16T03:37:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550787799999999, 2024-09-16T03:37:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:37:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 28 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7999999999999996e-07 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120252+0000", 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.249e-06 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120257+0000", 2024-09-16T03:37:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1969999999999998e-06 2024-09-16T03:37:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120261+0000", 2024-09-16T03:37:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1130000000000003e-06 2024-09-16T03:37:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222976+0000", 2024-09-16T03:37:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102715397 2024-09-16T03:37:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223049+0000", 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2846e-05 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525666+0000", 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30261668800000002 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525704+0000", 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8025999999999998e-05 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525758+0000", 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3881999999999997e-05 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:49 4.6 4:6df3bbbe:::smithi15736696-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.241874580000001, 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40556222800000002, 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8700000000000002e-07 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120454+0000", 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018909299999999999 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120465+0000", 2024-09-16T03:37:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.083e-05 2024-09-16T03:37:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120468+0000", 2024-09-16T03:37:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3869999999999999e-06 2024-09-16T03:37:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223082+0000", 2024-09-16T03:37:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102613379 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223189+0000", 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107728 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223368+0000", 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179011 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.226087+0000", 2024-09-16T03:37:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027183390000000002 2024-09-16T03:37:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423408+0000", 2024-09-16T03:37:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19732106499999999 2024-09-16T03:37:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525779+0000", 2024-09-16T03:37:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10237133 2024-09-16T03:37:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525805+0000", 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5967000000000001e-05 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525827+0000", 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1812000000000001e-05 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:50 4.6 4:6df3bbbe:::smithi15736696-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.241666271, 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50526462999999999, 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 50 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120474+0000", 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4600000000000005e-07 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120901+0000", 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00042690000000000002 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120907+0000", 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7710000000000001e-06 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120911+0000", 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0500000000000002e-06 2024-09-16T03:37:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224717+0000", 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103806175 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224763+0000", 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5629e-05 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224838+0000", 2024-09-16T03:37:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4548999999999994e-05 2024-09-16T03:37:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423390+0000", 2024-09-16T03:37:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19855242400000001 2024-09-16T03:37:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423411+0000", 2024-09-16T03:37:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0888000000000001e-05 2024-09-16T03:37:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625680+0000", 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20226923099999999 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625709+0000", 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9232000000000001e-05 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625738+0000", 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8235e-05 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:51 4.6 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.241221689, 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50490003699999997, 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 51 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8699999999999995e-07 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120920+0000", 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0820000000000001e-06 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120924+0000", 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120927+0000", 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5809999999999999e-06 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225783+0000", 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10485588699999999 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225832+0000", 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9750000000000003e-05 2024-09-16T03:37:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225910+0000", 2024-09-16T03:37:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7084999999999998e-05 2024-09-16T03:37:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423414+0000", 2024-09-16T03:37:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19750438200000001 2024-09-16T03:37:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423456+0000", 2024-09-16T03:37:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2386999999999999e-05 2024-09-16T03:37:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625766+0000", 2024-09-16T03:37:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20231011300000001 2024-09-16T03:37:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625790+0000", 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3672999999999998e-05 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625818+0000", 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7576000000000002e-05 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:52 4.6 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.241209492999999, 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50502012900000004, 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 52 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:37:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:37:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:37:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:37:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120932+0000", 2024-09-16T03:37:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.181e-06 2024-09-16T03:37:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120935+0000", 2024-09-16T03:37:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7690000000000001e-06 2024-09-16T03:37:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120938+0000", 2024-09-16T03:37:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6639999999999998e-06 2024-09-16T03:37:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423188+0000", 2024-09-16T03:37:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30224989200000002 2024-09-16T03:37:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423318+0000", 2024-09-16T03:37:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013045999999999999 2024-09-16T03:37:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625844+0000", 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20252547600000001 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625895+0000", 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1242000000000002e-05 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625950+0000", 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5164999999999998e-05 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:279 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:37:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.165257705999998, 2024-09-16T03:37:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50446570800000001, 2024-09-16T03:37:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 279 2024-09-16T03:37:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:37:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:37:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196883+0000", 2024-09-16T03:37:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.48e-06 2024-09-16T03:37:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196889+0000", 2024-09-16T03:37:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6239999999999999e-06 2024-09-16T03:37:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196902+0000", 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2804e-05 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196909+0000", 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9379999999999992e-06 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196963+0000", 2024-09-16T03:37:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3437999999999999e-05 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op must wait for map", 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196969+0000", 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1709999999999999e-06 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302538+0000", 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10556889899999999 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302659+0000", 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121441 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500040+0000", 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19738082900000001 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500589+0000", 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00054827000000000001 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701118+0000", 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20052917100000001 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701205+0000", 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7120999999999995e-05 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701289+0000", 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4457999999999997e-05 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701347+0000", 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8063999999999998e-05 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:306 4.6 4:6ece7830:::smithi15736696-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 28.849255391, 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40927214699999998, 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 306 2024-09-16T03:37:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:37:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:37:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512885+0000", 2024-09-16T03:37:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.277e-06 2024-09-16T03:37:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512890+0000", 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6689999999999997e-06 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512900+0000", 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0393e-05 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512906+0000", 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4569999999999996e-06 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512951+0000", 2024-09-16T03:37:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4891000000000001e-05 2024-09-16T03:37:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.513000+0000", 2024-09-16T03:37:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9187999999999997e-05 2024-09-16T03:37:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.720758+0000", 2024-09-16T03:37:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20775775699999999 2024-09-16T03:37:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.721960+0000", 2024-09-16T03:37:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001202107 2024-09-16T03:37:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922010+0000", 2024-09-16T03:37:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20004993900000001 2024-09-16T03:37:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922055+0000", 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4981000000000002e-05 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922103+0000", 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8097999999999999e-05 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922156+0000", 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3390000000000002e-05 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:419 4.6 4:64ce87bc:::smithi15736696-20:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:37:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.448455777, 2024-09-16T03:37:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40437787200000003, 2024-09-16T03:37:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 419 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913687+0000", 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3299999999999999e-06 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913700+0000", 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2830000000000001e-05 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913733+0000", 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2815999999999999e-05 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913750+0000", 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7073000000000001e-05 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913920+0000", 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000170015 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913981+0000", 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1631999999999995e-05 2024-09-16T03:37:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216580+0000", 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30259888200000001 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217205+0000", 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062461099999999996 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317915+0000", 2024-09-16T03:37:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10071053000000001 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317996+0000", 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0440999999999998e-05 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318032+0000", 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5944e-05 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318061+0000", 2024-09-16T03:37:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9768e-05 2024-09-16T03:37:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:479 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:37:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 23.640599631000001, 2024-09-16T03:37:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.504011706, 2024-09-16T03:37:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 479 2024-09-16T03:37:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:37:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:37:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721542+0000", 2024-09-16T03:37:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3199999999999998e-06 2024-09-16T03:37:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721553+0000", 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1413000000000001e-05 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721570+0000", 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.685e-05 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721596+0000", 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5647999999999999e-05 2024-09-16T03:37:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.921950+0000", 2024-09-16T03:37:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20035402899999999 2024-09-16T03:37:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.922009+0000", 2024-09-16T03:37:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9046999999999999e-05 2024-09-16T03:37:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:37:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.023850+0000", 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10184130199999999 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.024503+0000", 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065257399999999995 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.125105+0000", 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100602046 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225461+0000", 2024-09-16T03:37:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100355835 2024-09-16T03:37:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225512+0000", 2024-09-16T03:37:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0862999999999997e-05 2024-09-16T03:37:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225551+0000", 2024-09-16T03:37:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9779e-05 2024-09-16T03:37:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:996 4.8 4:148e627d:::smithi15736696-23:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.314796724000001, 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.73588397999999999, 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 996 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047344+0000", 2024-09-16T03:37:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.846e-06 2024-09-16T03:37:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047351+0000", 2024-09-16T03:37:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.973e-06 2024-09-16T03:37:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047365+0000", 2024-09-16T03:37:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3556e-05 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047382+0000", 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6994e-05 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047453+0000", 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1458000000000006e-05 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047465+0000", 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1119e-05 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449411+0000", 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40194680399999999 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449535+0000", 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124043 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783227+0000", 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.333691187 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1048 4.16 4:6df3bbbe:::smithi15736696-13:head [write 1622009~726683] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.786352607, 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40682847599999999, 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1048 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576941+0000", 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011532230000000001 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576949+0000", 2024-09-16T03:37:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1089999999999998e-06 2024-09-16T03:37:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576956+0000", 2024-09-16T03:37:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000001e-06 2024-09-16T03:37:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878710+0000", 2024-09-16T03:37:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30175434299999998 2024-09-16T03:37:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878765+0000", 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4988000000000003e-05 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878852+0000", 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7201999999999996e-05 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881535+0000", 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026828310000000001 2024-09-16T03:37:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883968+0000", 2024-09-16T03:37:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024330659999999998 2024-09-16T03:37:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982541+0000", 2024-09-16T03:37:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098572876000000004 2024-09-16T03:37:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982580+0000", 2024-09-16T03:37:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8714000000000002e-05 2024-09-16T03:37:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982615+0000", 2024-09-16T03:37:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5438000000000002e-05 2024-09-16T03:37:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1049 4.16 4:6df3bbbe:::smithi15736696-13:head [write 3121996~104915] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:37:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.785174458, 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40579890600000001, 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1049 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576966+0000", 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8199999999999999e-07 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577069+0000", 2024-09-16T03:37:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103583 2024-09-16T03:37:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577076+0000", 2024-09-16T03:37:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8190000000000003e-06 2024-09-16T03:37:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577082+0000", 2024-09-16T03:37:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5130000000000004e-06 2024-09-16T03:37:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880130+0000", 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.303048127 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880183+0000", 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3127999999999997e-05 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880282+0000", 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8881000000000004e-05 2024-09-16T03:37:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883101+0000", 2024-09-16T03:37:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028194180000000002 2024-09-16T03:37:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.979716+0000", 2024-09-16T03:37:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.096614899000000004 2024-09-16T03:37:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982707+0000", 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029914249999999998 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982739+0000", 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1250000000000001e-05 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982764+0000", 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5080999999999999e-05 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1050 4.16 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 3226911] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:37:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:37:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.785049862999999, 2024-09-16T03:37:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40784258099999998, 2024-09-16T03:37:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1050 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577090+0000", 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000003e-07 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577094+0000", 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4340000000000001e-06 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577099+0000", 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8599999999999998e-06 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577104+0000", 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0010000000000001e-06 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881242+0000", 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30413779000000002 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881290+0000", 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7298e-05 2024-09-16T03:37:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881367+0000", 2024-09-16T03:37:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7750000000000006e-05 2024-09-16T03:37:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883117+0000", 2024-09-16T03:37:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001750096 2024-09-16T03:37:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.980218+0000", 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097100724999999999 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984859+0000", 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046412950000000001 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984900+0000", 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0300999999999999e-05 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984932+0000", 2024-09-16T03:37:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2332999999999997e-05 2024-09-16T03:37:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1051 4.16 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:37:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:37:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.785027583, 2024-09-16T03:37:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50807376999999998, 2024-09-16T03:37:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:37:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1051 2024-09-16T03:37:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:37:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577115+0000", 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3980000000000002e-06 2024-09-16T03:37:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577120+0000", 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5589999999999996e-06 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577125+0000", 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9910000000000002e-06 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882780+0000", 2024-09-16T03:37:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30565438499999997 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882825+0000", 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5739999999999999e-05 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085039+0000", 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20221376499999999 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085096+0000", 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6666000000000003e-05 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085186+0000", 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9716000000000006e-05 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1359 4.6 4:64ce87bc:::smithi15736696-20:head [write 1768928~405354] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:37:45.472 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.1960044449999998, 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40239613499999999, 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1359 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:37:45.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166136+0000", 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166923+0000", 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00078683500000000005 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166933+0000", 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0445000000000001e-05 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166938+0000", 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0119999999999996e-06 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264901+0000", 2024-09-16T03:37:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097962831 2024-09-16T03:37:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264963+0000", 2024-09-16T03:37:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1812999999999999e-05 2024-09-16T03:37:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.265085+0000", 2024-09-16T03:37:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121914 2024-09-16T03:37:45.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.269430+0000", 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0043455000000000004 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.372259+0000", 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102828645 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568289+0000", 2024-09-16T03:37:45.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19603034999999999 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568475+0000", 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018613599999999999 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568531+0000", 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5729000000000001e-05 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1360 4.6 4:64ce87bc:::smithi15736696-20:head [write 2768204~509510] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T03:37:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:40.171977+0000", 2024-09-16T03:37:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.1901618620000001, 2024-09-16T03:37:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.39669463500000002, 2024-09-16T03:37:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1360 2024-09-16T03:37:45.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.171977+0000", 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.171977+0000", 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.171983+0000", 2024-09-16T03:37:45.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8880000000000002e-06 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.172828+0000", 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084511500000000002 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.172841+0000", 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2841e-05 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.172848+0000", 2024-09-16T03:37:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2820000000000002e-06 2024-09-16T03:37:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.266308+0000", 2024-09-16T03:37:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.093460809000000006 2024-09-16T03:37:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.266386+0000", 2024-09-16T03:37:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7816000000000002e-05 2024-09-16T03:37:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:37:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.266543+0000", 2024-09-16T03:37:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000156949 2024-09-16T03:37:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.372096+0000", 2024-09-16T03:37:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.105552426 2024-09-16T03:37:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.372262+0000", 2024-09-16T03:37:45.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000166157 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568600+0000", 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19633862399999999 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568640+0000", 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0022999999999999e-05 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568672+0000", 2024-09-16T03:37:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1705000000000002e-05 2024-09-16T03:37:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1362 4.6 4:64ce87bc:::smithi15736696-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2024-09-16T03:37:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:40.172879+0000", 2024-09-16T03:37:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.1892603619999997, 2024-09-16T03:37:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.39616614999999999, 2024-09-16T03:37:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.485 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:37:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1362 2024-09-16T03:37:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.172879+0000", 2024-09-16T03:37:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.172879+0000", 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.172879+0000", 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1899999999999998e-07 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.172899+0000", 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9565e-05 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.172903+0000", 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.092e-06 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.172907+0000", 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.467e-06 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.267975+0000", 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.095068738 2024-09-16T03:37:45.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.268023+0000", 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7919e-05 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568832+0000", 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30080834699999998 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568928+0000", 2024-09-16T03:37:45.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6984000000000005e-05 2024-09-16T03:37:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.569045+0000", 2024-09-16T03:37:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116619 2024-09-16T03:37:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.490 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.490 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:45.491 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:37:45.492 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:45.490+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:45.493 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 4 pgp_num to 26 2024-09-16T03:37:45.493 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:37:45.592 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:45.590+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:45.596 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:45.596 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:45.596 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:45.597 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:45.605 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:37:45.608 INFO:tasks.rados.rados.0.smithi157.stdout:853: set_chunk oid 28 offset: 129024 length: 1693696 target oid 8c9392b2bb5dcd9aeb358fd53aab4bf8d7b7b07146d5e0c68d69fa179cda2b1c offset: 0 snap -1 2024-09-16T03:37:45.609 INFO:tasks.rados.rados.0.smithi157.stdout:854: read oid 19 snap -1 2024-09-16T03:37:45.609 INFO:tasks.rados.rados.0.smithi157.stdout:854: expect (ObjNum 93 snap 0 seq_num 93) 2024-09-16T03:37:45.609 INFO:tasks.rados.rados.0.smithi157.stdout:855: read oid 5 snap -1 2024-09-16T03:37:45.609 INFO:tasks.rados.rados.0.smithi157.stdout:855: expect (ObjNum 111 snap 0 seq_num 111) 2024-09-16T03:37:45.609 INFO:tasks.rados.rados.0.smithi157.stdout:856: write (excl) oid 33 current snap is 0 2024-09-16T03:37:45.609 INFO:tasks.rados.rados.0.smithi157.stdout:856: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-09-16T03:37:45.615 INFO:tasks.rados.rados.0.smithi157.stdout:856: writing smithi15736696-33 from 749612 to 1223791 tid 1 2024-09-16T03:37:45.621 INFO:tasks.rados.rados.0.smithi157.stdout:856: writing smithi15736696-33 from 1954970 to 2386007 tid 2 2024-09-16T03:37:45.628 INFO:tasks.rados.rados.0.smithi157.stdout:856: writing smithi15736696-33 from 2919420 to 3535797 tid 3 2024-09-16T03:37:45.628 INFO:tasks.rados.rados.0.smithi157.stdout:857: write (excl) oid 40 current snap is 0 2024-09-16T03:37:45.628 INFO:tasks.rados.rados.0.smithi157.stdout:857: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-09-16T03:37:45.634 INFO:tasks.rados.rados.0.smithi157.stdout:857: writing smithi15736696-40 from 766419 to 1315847 tid 1 2024-09-16T03:37:45.641 INFO:tasks.rados.rados.0.smithi157.stdout:857: writing smithi15736696-40 from 2013452 to 2588147 tid 2 2024-09-16T03:37:45.648 INFO:tasks.rados.rados.0.smithi157.stdout:857: writing smithi15736696-40 from 3346843 to 3774492 tid 3 2024-09-16T03:37:45.648 INFO:tasks.rados.rados.0.smithi157.stdout:858: write oid 42 current snap is 0 2024-09-16T03:37:45.648 INFO:tasks.rados.rados.0.smithi157.stdout:858: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2024-09-16T03:37:45.655 INFO:tasks.rados.rados.0.smithi157.stdout:858: writing smithi15736696-42 from 783226 to 1407903 tid 1 2024-09-16T03:37:45.661 INFO:tasks.rados.rados.0.smithi157.stdout:858: writing smithi15736696-42 from 2071934 to 2506640 tid 2 2024-09-16T03:37:45.667 INFO:tasks.rados.rados.0.smithi157.stdout:858: writing smithi15736696-42 from 2974266 to 3613187 tid 3 2024-09-16T03:37:45.667 INFO:tasks.rados.rados.0.smithi157.stdout:859: read oid 46 snap -1 2024-09-16T03:37:45.667 INFO:tasks.rados.rados.0.smithi157.stdout:859: expect (ObjNum 97 snap 0 seq_num 97) 2024-09-16T03:37:45.667 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:45.667 INFO:tasks.rados.rados.0.smithi157.stdout:847: expect (ObjNum 102 snap 0 seq_num 102) 2024-09-16T03:37:45.667 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:45.693 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:45.691+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:45.709 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:45.709 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:45.709 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:37:45.709 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:37:45.709 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:45.718 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:37:45.743 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2152517 2024-09-16T03:37:45.787 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1917335 2024-09-16T03:37:45.793 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:45.790+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:45.815 INFO:tasks.rados.rados.0.smithi157.stdout:844: expect (ObjNum 86 snap 0 seq_num 86) 2024-09-16T03:37:45.824 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:37:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:3 3.0 3.f934d18 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:37:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.421813584999999, 2024-09-16T03:37:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78381636700000001, 2024-09-16T03:37:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:37:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:37:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399429+0000", 2024-09-16T03:37:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:37:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399434+0000", 2024-09-16T03:37:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1050000000000003e-06 2024-09-16T03:37:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399442+0000", 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0160000000000008e-06 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399448+0000", 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.879e-06 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399489+0000", 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0815999999999997e-05 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399495+0000", 2024-09-16T03:37:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3439999999999997e-06 2024-09-16T03:37:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183226+0000", 2024-09-16T03:37:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78373077400000002 2024-09-16T03:37:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183244+0000", 2024-09-16T03:37:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8128999999999999e-05 2024-09-16T03:37:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:5 3.0 3.aefbbec0 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.322046123, 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68447923499999996, 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:37:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:37:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499196+0000", 2024-09-16T03:37:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3480000000000001e-06 2024-09-16T03:37:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499202+0000", 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4219999999999999e-06 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499212+0000", 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499218+0000", 2024-09-16T03:37:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4920000000000001e-06 2024-09-16T03:37:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499283+0000", 2024-09-16T03:37:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4906000000000005e-05 2024-09-16T03:37:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499293+0000", 2024-09-16T03:37:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0513000000000001e-05 2024-09-16T03:37:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183665+0000", 2024-09-16T03:37:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68437196600000005 2024-09-16T03:37:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183674+0000", 2024-09-16T03:37:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9700000000000005e-06 2024-09-16T03:37:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:6 3.0 3.3bcda120 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:37:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.321985514000001, 2024-09-16T03:37:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78474037699999999, 2024-09-16T03:37:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:37:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0500000000000002e-07 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499260+0000", 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6669999999999998e-06 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499266+0000", 2024-09-16T03:37:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:37:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499274+0000", 2024-09-16T03:37:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2250000000000003e-06 2024-09-16T03:37:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499316+0000", 2024-09-16T03:37:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2670000000000003e-05 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499323+0000", 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.37e-06 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283987+0000", 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78466433700000005 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283996+0000", 2024-09-16T03:37:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8650000000000007e-06 2024-09-16T03:37:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:15 3.0 3.7ec93288 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:37:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.212793504, 2024-09-16T03:37:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57518172599999995, 2024-09-16T03:37:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:37:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608449+0000", 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.184e-06 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608459+0000", 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0553999999999999e-05 2024-09-16T03:37:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608470+0000", 2024-09-16T03:37:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0524e-05 2024-09-16T03:37:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608493+0000", 2024-09-16T03:37:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2980999999999999e-05 2024-09-16T03:37:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608537+0000", 2024-09-16T03:37:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3850000000000002e-05 2024-09-16T03:37:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608542+0000", 2024-09-16T03:37:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:37:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183623+0000", 2024-09-16T03:37:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.575080649 2024-09-16T03:37:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183629+0000", 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6950000000000003e-06 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:11 3.0 3.77005160 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.119293194000001, 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.48153733399999998, 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:37:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:37:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701949+0000", 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.314e-06 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701954+0000", 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701964+0000", 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9550000000000004e-06 2024-09-16T03:37:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701986+0000", 2024-09-16T03:37:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1931999999999999e-05 2024-09-16T03:37:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802129+0000", 2024-09-16T03:37:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100142986 2024-09-16T03:37:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802135+0000", 2024-09-16T03:37:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6350000000000003e-06 2024-09-16T03:37:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183475+0000", 2024-09-16T03:37:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38133958099999998 2024-09-16T03:37:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183485+0000", 2024-09-16T03:37:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0642000000000001e-05 2024-09-16T03:37:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:6 3.0 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:37:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 28.838389281000001, 2024-09-16T03:37:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60469085300000003, 2024-09-16T03:37:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:37:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:37:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982853+0000", 2024-09-16T03:37:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.393e-06 2024-09-16T03:37:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982858+0000", 2024-09-16T03:37:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.062e-06 2024-09-16T03:37:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982867+0000", 2024-09-16T03:37:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:37:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982879+0000", 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2241e-05 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982996+0000", 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116996 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983007+0000", 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0694e-05 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284499+0000", 2024-09-16T03:37:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30149171400000002 2024-09-16T03:37:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284690+0000", 2024-09-16T03:37:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000191032 2024-09-16T03:37:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284908+0000", 2024-09-16T03:37:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021827399999999999 2024-09-16T03:37:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487092+0000", 2024-09-16T03:37:45.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.202184156 2024-09-16T03:37:45.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487110+0000", 2024-09-16T03:37:45.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7856999999999999e-05 2024-09-16T03:37:45.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587335+0000", 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100224802 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587410+0000", 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5343e-05 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587543+0000", 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000132492 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.854 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:2 3.0 3:16550cfd:::smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:37:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 28.838317089, 2024-09-16T03:37:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40289696000000003, 2024-09-16T03:37:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982925+0000", 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982928+0000", 2024-09-16T03:37:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4130000000000002e-06 2024-09-16T03:37:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982936+0000", 2024-09-16T03:37:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.875e-06 2024-09-16T03:37:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982942+0000", 2024-09-16T03:37:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:37:45.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983093+0000", 2024-09-16T03:37:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015144400000000001 2024-09-16T03:37:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983098+0000", 2024-09-16T03:37:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4909999999999999e-06 2024-09-16T03:37:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385695+0000", 2024-09-16T03:37:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402597539 2024-09-16T03:37:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385821+0000", 2024-09-16T03:37:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125418 2024-09-16T03:37:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:9 3.0 3:2c9b9e7c:::smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:37:45.859 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:37:45.860 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 28.8383009, 2024-09-16T03:37:45.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60615655800000001, 2024-09-16T03:37:45.860 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.860 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.860 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.860 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:37:45.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982941+0000", 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0860000000000001e-06 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982946+0000", 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982954+0000", 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8520000000000004e-06 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982958+0000", 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4870000000000004e-06 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983116+0000", 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015774699999999999 2024-09-16T03:37:45.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983120+0000", 2024-09-16T03:37:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:37:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486023+0000", 2024-09-16T03:37:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50290281400000003 2024-09-16T03:37:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486423+0000", 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040079299999999998 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486707+0000", 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028336600000000001 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588861+0000", 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102154756 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588952+0000", 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0630000000000005e-05 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588984+0000", 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2116999999999998e-05 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589022+0000", 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7910000000000001e-05 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589097+0000", 2024-09-16T03:37:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4536999999999995e-05 2024-09-16T03:37:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:12 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:37:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 28.838225552000001, 2024-09-16T03:37:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60616102900000002, 2024-09-16T03:37:45.866 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:37:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:37:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983017+0000", 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.483e-06 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983022+0000", 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983032+0000", 2024-09-16T03:37:45.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0036e-05 2024-09-16T03:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983038+0000", 2024-09-16T03:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6800000000000004e-06 2024-09-16T03:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983155+0000", 2024-09-16T03:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116401 2024-09-16T03:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983163+0000", 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6219999999999998e-06 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589120+0000", 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60595653199999999 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589176+0000", 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6542000000000002e-05 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.870 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:5 3.0 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:37:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.907036623, 2024-09-16T03:37:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30268081699999999, 2024-09-16T03:37:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:37:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:37:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914207+0000", 2024-09-16T03:37:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1500000000000002e-06 2024-09-16T03:37:45.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914215+0000", 2024-09-16T03:37:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7530000000000007e-06 2024-09-16T03:37:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914232+0000", 2024-09-16T03:37:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.628e-05 2024-09-16T03:37:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914242+0000", 2024-09-16T03:37:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.058e-05 2024-09-16T03:37:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914305+0000", 2024-09-16T03:37:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3294999999999996e-05 2024-09-16T03:37:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914377+0000", 2024-09-16T03:37:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1698000000000004e-05 2024-09-16T03:37:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914517+0000", 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000139443 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.015450+0000", 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100933668 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.015480+0000", 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9467000000000001e-05 2024-09-16T03:37:45.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216766+0000", 2024-09-16T03:37:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20128665300000001 2024-09-16T03:37:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216831+0000", 2024-09-16T03:37:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4220999999999994e-05 2024-09-16T03:37:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216885+0000", 2024-09-16T03:37:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4608999999999998e-05 2024-09-16T03:37:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:24 3.0 3:2dd0f25f:::smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:37:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.806702226999999, 2024-09-16T03:37:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30270049300000001, 2024-09-16T03:37:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.877 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:37:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:37:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014541+0000", 2024-09-16T03:37:45.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2479999999999999e-06 2024-09-16T03:37:45.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014550+0000", 2024-09-16T03:37:45.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.115e-06 2024-09-16T03:37:45.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014567+0000", 2024-09-16T03:37:45.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7292999999999999e-05 2024-09-16T03:37:45.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014577+0000", 2024-09-16T03:37:45.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8020000000000003e-06 2024-09-16T03:37:45.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115566+0000", 2024-09-16T03:37:45.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10098905900000001 2024-09-16T03:37:45.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115702+0000", 2024-09-16T03:37:45.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013602500000000001 2024-09-16T03:37:45.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:45.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115893+0000", 2024-09-16T03:37:45.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019095700000000001 2024-09-16T03:37:45.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216936+0000", 2024-09-16T03:37:45.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101042992 2024-09-16T03:37:45.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317125+0000", 2024-09-16T03:37:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100188757 2024-09-16T03:37:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317155+0000", 2024-09-16T03:37:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0009999999999999e-05 2024-09-16T03:37:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317210+0000", 2024-09-16T03:37:45.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4589999999999997e-05 2024-09-16T03:37:45.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317239+0000", 2024-09-16T03:37:45.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9645000000000002e-05 2024-09-16T03:37:45.883 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.883 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.883 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.883 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:30 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:45.883 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:37:45.883 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.604907046000001, 2024-09-16T03:37:45.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40312788999999999, 2024-09-16T03:37:45.884 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.884 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.884 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.884 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:37:45.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:37:45.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216338+0000", 2024-09-16T03:37:45.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0799999999999999e-06 2024-09-16T03:37:45.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216354+0000", 2024-09-16T03:37:45.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6439e-05 2024-09-16T03:37:45.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216392+0000", 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7162000000000001e-05 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216444+0000", 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2082e-05 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417447+0000", 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20100358400000001 2024-09-16T03:37:45.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417583+0000", 2024-09-16T03:37:45.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135537 2024-09-16T03:37:45.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:45.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417777+0000", 2024-09-16T03:37:45.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019431200000000001 2024-09-16T03:37:45.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418479+0000", 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00070148499999999996 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418755+0000", 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027627100000000003 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619285+0000", 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200530138 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619370+0000", 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4567999999999994e-05 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619462+0000", 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2231999999999994e-05 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:18 3.0 3:037ddf75:::smithi15736696-34 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:45.889 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:37:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 18.885312774999999, 2024-09-16T03:37:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30645336299999998, 2024-09-16T03:37:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:37:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935929+0000", 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4600000000000003e-07 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935936+0000", 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3199999999999996e-06 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935943+0000", 2024-09-16T03:37:45.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8900000000000007e-06 2024-09-16T03:37:45.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935954+0000", 2024-09-16T03:37:45.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0246e-05 2024-09-16T03:37:45.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140278+0000", 2024-09-16T03:37:45.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20432396999999999 2024-09-16T03:37:45.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140342+0000", 2024-09-16T03:37:45.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4442000000000004e-05 2024-09-16T03:37:45.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:45.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140468+0000", 2024-09-16T03:37:45.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125718 2024-09-16T03:37:45.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.141460+0000", 2024-09-16T03:37:45.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00099208400000000006 2024-09-16T03:37:45.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242240+0000", 2024-09-16T03:37:45.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100779733 2024-09-16T03:37:45.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242316+0000", 2024-09-16T03:37:45.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6038999999999995e-05 2024-09-16T03:37:45.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242351+0000", 2024-09-16T03:37:45.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4928999999999997e-05 2024-09-16T03:37:45.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242382+0000", 2024-09-16T03:37:45.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1046000000000001e-05 2024-09-16T03:37:45.895 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.895 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.896 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:28 3.0 3:26d46705:::076d7e878f580d0ea556d11054cb0027df80eadb9845d4d121514c92db92c9b8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:45.896 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.473539+0000", 2024-09-16T03:37:45.896 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.347702009000001, 2024-09-16T03:37:45.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.207364408, 2024-09-16T03:37:45.896 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.896 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.896 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.897 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473539+0000", 2024-09-16T03:37:45.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473539+0000", 2024-09-16T03:37:45.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473542+0000", 2024-09-16T03:37:45.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9210000000000001e-06 2024-09-16T03:37:45.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473828+0000", 2024-09-16T03:37:45.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000286085 2024-09-16T03:37:45.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473847+0000", 2024-09-16T03:37:45.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.946e-05 2024-09-16T03:37:45.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473858+0000", 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0285999999999999e-05 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474128+0000", 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000269886 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474242+0000", 2024-09-16T03:37:45.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000114714 2024-09-16T03:37:45.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:45.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474404+0000", 2024-09-16T03:37:45.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000161595 2024-09-16T03:37:45.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.480364+0000", 2024-09-16T03:37:45.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0059603629999999998 2024-09-16T03:37:45.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680752+0000", 2024-09-16T03:37:45.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20038762800000001 2024-09-16T03:37:45.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680811+0000", 2024-09-16T03:37:45.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9502000000000001e-05 2024-09-16T03:37:45.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680850+0000", 2024-09-16T03:37:45.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8717000000000002e-05 2024-09-16T03:37:45.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680903+0000", 2024-09-16T03:37:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3251000000000002e-05 2024-09-16T03:37:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:29 3.0 3:363c19ae:::d7c46b815608541b41d2b012df2d45d94bbc9173d36826cf8a21a48f304023a1:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.473885+0000", 2024-09-16T03:37:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.347355989, 2024-09-16T03:37:45.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20712613699999999, 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473885+0000", 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473885+0000", 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.473886+0000", 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.64e-07 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474063+0000", 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000177441 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474074+0000", 2024-09-16T03:37:45.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.059e-05 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474082+0000", 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0909999999999993e-06 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.475039+0000", 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000956886 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.475123+0000", 2024-09-16T03:37:45.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4462000000000005e-05 2024-09-16T03:37:45.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:45.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.475298+0000", 2024-09-16T03:37:45.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000174742 2024-09-16T03:37:45.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.580527+0000", 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.105228435 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680758+0000", 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100231606 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680953+0000", 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019436 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680986+0000", 2024-09-16T03:37:45.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3760999999999997e-05 2024-09-16T03:37:45.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681011+0000", 2024-09-16T03:37:45.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4799000000000001e-05 2024-09-16T03:37:45.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.908 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.909 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:30 3.0 3:1e508a18:::6088440c44690292c78eb2d574d796d457a957c4bf203df90b3df0f9d0ca33b8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:45.909 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474098+0000", 2024-09-16T03:37:45.909 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.347143162, 2024-09-16T03:37:45.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.207015378, 2024-09-16T03:37:45.909 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.909 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.909 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.909 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474098+0000", 2024-09-16T03:37:45.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474098+0000", 2024-09-16T03:37:45.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474099+0000", 2024-09-16T03:37:45.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8700000000000005e-07 2024-09-16T03:37:45.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474147+0000", 2024-09-16T03:37:45.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8300999999999997e-05 2024-09-16T03:37:45.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474156+0000", 2024-09-16T03:37:45.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5740000000000002e-06 2024-09-16T03:37:45.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474168+0000", 2024-09-16T03:37:45.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1294999999999999e-05 2024-09-16T03:37:45.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.476209+0000", 2024-09-16T03:37:45.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020416240000000001 2024-09-16T03:37:45.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.476369+0000", 2024-09-16T03:37:45.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015955200000000001 2024-09-16T03:37:45.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:45.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.476591+0000", 2024-09-16T03:37:45.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000222065 2024-09-16T03:37:45.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680697+0000", 2024-09-16T03:37:45.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20410637000000001 2024-09-16T03:37:45.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680763+0000", 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6118999999999995e-05 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681056+0000", 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00029263599999999998 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681088+0000", 2024-09-16T03:37:45.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1520999999999999e-05 2024-09-16T03:37:45.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681113+0000", 2024-09-16T03:37:45.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5633999999999999e-05 2024-09-16T03:37:45.915 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.915 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.915 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.915 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:33 3.0 3:1e59f26f:::f36fb4795b66b358920b15346e41df644c7e2d9386f5f260380eec916b18bcae:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:45.915 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474176+0000", 2024-09-16T03:37:45.915 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.347065106000001, 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20702289600000001, 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474176+0000", 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474176+0000", 2024-09-16T03:37:45.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474177+0000", 2024-09-16T03:37:45.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8999999999999995e-07 2024-09-16T03:37:45.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474740+0000", 2024-09-16T03:37:45.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056329899999999996 2024-09-16T03:37:45.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474750+0000", 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5599999999999999e-06 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474758+0000", 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7779999999999996e-06 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.477151+0000", 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023921900000000002 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.477265+0000", 2024-09-16T03:37:45.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011442000000000001 2024-09-16T03:37:45.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:45.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.477487+0000", 2024-09-16T03:37:45.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022229500000000001 2024-09-16T03:37:45.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680740+0000", 2024-09-16T03:37:45.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20325288599999999 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.680769+0000", 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8694999999999999e-05 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681151+0000", 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000382622 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681177+0000", 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5581000000000001e-05 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681199+0000", 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1780000000000002e-05 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:34 3.0 3:08307d25:::2b3cb797ae2d3abc24b2cae4a012f26325debe2fc38ccb08bd4503e949d305ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:37:45.921 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.346450988999999, 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30798127600000003, 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:37:45.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474791+0000", 2024-09-16T03:37:45.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6000000000000003e-07 2024-09-16T03:37:45.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474865+0000", 2024-09-16T03:37:45.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4504000000000004e-05 2024-09-16T03:37:45.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474882+0000", 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6765999999999998e-05 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474890+0000", 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.14e-06 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681242+0000", 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20635221200000001 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681355+0000", 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000112211 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681505+0000", 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015040100000000001 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682352+0000", 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084686400000000002 2024-09-16T03:37:45.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682989+0000", 2024-09-16T03:37:45.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063749800000000001 2024-09-16T03:37:45.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782695+0000", 2024-09-16T03:37:45.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099706063999999997 2024-09-16T03:37:45.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782735+0000", 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9304000000000002e-05 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782771+0000", 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6551999999999998e-05 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:36 3.0 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.884285+0000", 2024-09-16T03:37:45.927 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.936956195, 2024-09-16T03:37:45.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30167871800000001, 2024-09-16T03:37:45.928 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.928 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.928 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.928 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884285+0000", 2024-09-16T03:37:45.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884285+0000", 2024-09-16T03:37:45.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884287+0000", 2024-09-16T03:37:45.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2400000000000002e-06 2024-09-16T03:37:45.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884295+0000", 2024-09-16T03:37:45.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7940000000000003e-06 2024-09-16T03:37:45.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884309+0000", 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.413e-05 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884317+0000", 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8499999999999994e-06 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884364+0000", 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7259999999999998e-05 2024-09-16T03:37:45.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884468+0000", 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010361099999999999 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884570+0000", 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010202800000000001 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.885223+0000", 2024-09-16T03:37:45.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065319100000000001 2024-09-16T03:37:45.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.885453+0000", 2024-09-16T03:37:45.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00023042200000000001 2024-09-16T03:37:45.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.185867+0000", 2024-09-16T03:37:45.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30041376800000003 2024-09-16T03:37:45.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.185930+0000", 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2675000000000006e-05 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.185964+0000", 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3748999999999998e-05 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1203 3.0 3:3cdebe83:::28f3dbd02a2494b004e7cef3f0db24ce043047807c298214f7b10042c9638a72:head [write 0~778240] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:37:45.933 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:36.087090+0000", 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.7341509500000001, 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20700696900000001, 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1203 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:45.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.087090+0000", 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.087090+0000", 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.087093+0000", 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4660000000000002e-06 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.088045+0000", 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000952121 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.088060+0000", 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4884e-05 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.188208+0000", 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100148458 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.288465+0000", 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100257376 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.288687+0000", 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000221764 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.288799+0000", 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000111984 2024-09-16T03:37:45.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.290814+0000", 2024-09-16T03:37:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002014428 2024-09-16T03:37:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.291275+0000", 2024-09-16T03:37:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046160400000000001 2024-09-16T03:37:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.294015+0000", 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027394939999999999 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.294055+0000", 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9869000000000001e-05 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:36.294097+0000", 2024-09-16T03:37:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2521000000000002e-05 2024-09-16T03:37:45.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.940 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:45.940 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:45.940 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:45.941 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2446107 2024-09-16T03:37:45.941 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:45.942 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:37:45.942 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:45.939+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:45.948 INFO:tasks.rados.rados.0.smithi157.stdout:843: finishing set_chunk to oid 41 2024-09-16T03:37:45.948 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 32 (ObjNum 112 snap 0 seq_num 112) dirty exists 2024-09-16T03:37:45.948 INFO:tasks.rados.rados.0.smithi157.stdout:848: finishing set_chunk to oid 24 2024-09-16T03:37:45.948 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 1058816 length: 1094656 2024-09-16T03:37:45.948 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T03:37:45.948 INFO:tasks.rados.rados.0.smithi157.stdout:843: done (15 left) 2024-09-16T03:37:45.948 INFO:tasks.rados.rados.0.smithi157.stdout:844: done (14 left) 2024-09-16T03:37:45.948 INFO:tasks.rados.rados.0.smithi157.stdout:846: done (13 left) 2024-09-16T03:37:45.948 INFO:tasks.rados.rados.0.smithi157.stdout:847: done (12 left) 2024-09-16T03:37:45.948 INFO:tasks.rados.rados.0.smithi157.stdout:848: done (11 left) 2024-09-16T03:37:45.948 INFO:tasks.rados.rados.0.smithi157.stdout:860: write (excl) oid 1 current snap is 0 2024-09-16T03:37:45.948 INFO:tasks.rados.rados.0.smithi157.stdout:860: seq_num 119 ranges {400033=699926,1614063=385970} 2024-09-16T03:37:45.956 INFO:tasks.rados.rados.0.smithi157.stdout:860: writing smithi15736696-1 from 400033 to 1099959 tid 1 2024-09-16T03:37:45.958 INFO:tasks.rados.rados.0.smithi157.stdout:860: writing smithi15736696-1 from 1614063 to 2000033 tid 2 2024-09-16T03:37:45.958 INFO:tasks.rados.rados.0.smithi157.stdout:861: set_chunk oid 13 target oid 2024-09-16T03:37:46.041 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:46.040+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:46.046 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:46.046 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:46.047 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:46.047 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:46.055 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:37:46.067 INFO:tasks.rados.rados.0.smithi157.stdout:861: set_chunk oid 13 offset: 363520 length: 2086400 target oid 578eb536425c2a88a8f9a268127a0413570a39e75f5242f04a0c75b88096df49 offset: 0 snap -1 2024-09-16T03:37:46.067 INFO:tasks.rados.rados.0.smithi157.stdout:862: write oid 2 current snap is 0 2024-09-16T03:37:46.067 INFO:tasks.rados.rados.0.smithi157.stdout:849: finishing set_chunk to oid 25 2024-09-16T03:37:46.068 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk returned error code -95 offset: 1787392 length: 224256 2024-09-16T03:37:46.068 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:46.068 INFO:tasks.rados.rados.0.smithi157.stdout:862: seq_num 120 ranges {416840=775175,1672545=344295} 2024-09-16T03:37:46.073 INFO:tasks.rados.rados.0.smithi157.stdout:862: writing smithi15736696-2 from 416840 to 1192015 tid 1 2024-09-16T03:37:46.078 INFO:tasks.rados.rados.0.smithi157.stdout:862: writing smithi15736696-2 from 1672545 to 2016840 tid 2 2024-09-16T03:37:46.078 INFO:tasks.rados.rados.0.smithi157.stdout:849: done (13 left) 2024-09-16T03:37:46.078 INFO:tasks.rados.rados.0.smithi157.stdout:863: write (excl) oid 38 current snap is 0 2024-09-16T03:37:46.078 INFO:tasks.rados.rados.0.smithi157.stdout:863: seq_num 121 ranges {433647=450424,1331027=582033} 2024-09-16T03:37:46.083 INFO:tasks.rados.rados.0.smithi157.stdout:863: writing smithi15736696-38 from 433647 to 884071 tid 1 2024-09-16T03:37:46.089 INFO:tasks.rados.rados.0.smithi157.stdout:863: writing smithi15736696-38 from 1331027 to 1913060 tid 2 2024-09-16T03:37:46.089 INFO:tasks.rados.rados.0.smithi157.stdout:864: set_chunk oid 31 target oid 2024-09-16T03:37:46.089 INFO:tasks.rados.rados.0.smithi157.stdout:864: set_chunk oid 31 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:37:46.089 INFO:tasks.rados.rados.0.smithi157.stdout:865: write oid 16 current snap is 0 2024-09-16T03:37:46.089 INFO:tasks.rados.rados.0.smithi157.stdout:865: seq_num 122 ranges {450454=642026,1505862=544592} 2024-09-16T03:37:46.094 INFO:tasks.rados.rados.0.smithi157.stdout:865: writing smithi15736696-16 from 450454 to 1092480 tid 1 2024-09-16T03:37:46.100 INFO:tasks.rados.rados.0.smithi157.stdout:865: writing smithi15736696-16 from 1505862 to 2050454 tid 2 2024-09-16T03:37:46.100 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:46.100 INFO:tasks.rados.rados.0.smithi157.stdout:852: expect (ObjNum 113 snap 0 seq_num 113) 2024-09-16T03:37:46.100 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:46.142 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:46.141+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:46.200 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:37:46.200 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:37:46.200 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:46.209 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:37:46.237 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1845831 2024-09-16T03:37:46.244 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:46.242+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:46.270 INFO:tasks.rados.rados.0.smithi157.stdout:850: expect (ObjNum 85 snap 0 seq_num 85) 2024-09-16T03:37:46.270 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:46.345 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:46.343+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:46.352 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:46.352 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:46.352 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:46.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:46.352 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:37:46.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:57 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.225940168999998, 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20372090900000001, 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:46.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:37:46.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:46.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:46.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125313+0000", 2024-09-16T03:37:46.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7180000000000001e-06 2024-09-16T03:37:46.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:46.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126730+0000", 2024-09-16T03:37:46.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00141731 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126746+0000", 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6455000000000001e-05 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126756+0000", 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:46.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227068+0000", 2024-09-16T03:37:46.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100312308 2024-09-16T03:37:46.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:46.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227085+0000", 2024-09-16T03:37:46.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6898e-05 2024-09-16T03:37:46.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:46.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328995+0000", 2024-09-16T03:37:46.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101909903 2024-09-16T03:37:46.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329031+0000", 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5717999999999999e-05 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:58 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.222607039000003, 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084524500000001, 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:46.357 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:46.358 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.358 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:46.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:46.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:37:46.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:46.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:46.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:37:46.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:46.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128644+0000", 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.251e-06 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128913+0000", 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026856200000000002 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128925+0000", 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2109999999999999e-05 2024-09-16T03:37:46.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128933+0000", 2024-09-16T03:37:46.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6260000000000003e-06 2024-09-16T03:37:46.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:46.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328196+0000", 2024-09-16T03:37:46.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19926303200000001 2024-09-16T03:37:46.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:46.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328253+0000", 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7262999999999997e-05 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329458+0000", 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012051939999999999 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329488+0000", 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0207000000000001e-05 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:46.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:59 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.222111605999999, 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036954900000001, 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:37:46.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129139+0000", 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6099999999999999e-07 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:46.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129158+0000", 2024-09-16T03:37:46.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088e-05 2024-09-16T03:37:46.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:46.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129168+0000", 2024-09-16T03:37:46.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.048e-05 2024-09-16T03:37:46.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.229233+0000", 2024-09-16T03:37:46.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100065168 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328815+0000", 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099581694999999998 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328845+0000", 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0193e-05 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:46.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329492+0000", 2024-09-16T03:37:46.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064671800000000001 2024-09-16T03:37:46.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:46.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329508+0000", 2024-09-16T03:37:46.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6246e-05 2024-09-16T03:37:46.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:46.366 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:46.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:46.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.366 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.0:15 4.d e47/32 4:b3fe05b2:::smithi15736696-15:head v 47'5, mlcod=39'3)", 2024-09-16T03:37:46.366 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:37:46.367 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.04881688, 2024-09-16T03:37:46.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100920751, 2024-09-16T03:37:46.367 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:46.367 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:46.367 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.367 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:46.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:46.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:37:46.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:46.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:46.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:37:46.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:46.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:46.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302435+0000", 2024-09-16T03:37:46.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2340000000000001e-06 2024-09-16T03:37:46.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:46.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302446+0000", 2024-09-16T03:37:46.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0596e-05 2024-09-16T03:37:46.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302461+0000", 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5082000000000001e-05 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402592+0000", 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100130488 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402712+0000", 2024-09-16T03:37:46.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012013100000000001 2024-09-16T03:37:46.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:46.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402741+0000", 2024-09-16T03:37:46.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9717999999999999e-05 2024-09-16T03:37:46.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:46.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403309+0000", 2024-09-16T03:37:46.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056725700000000005 2024-09-16T03:37:46.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:46.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:46.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:46.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403354+0000", 2024-09-16T03:37:46.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5244999999999998e-05 2024-09-16T03:37:46.371 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:46.371 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:46.371 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:46.371 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:46.371 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:46.371 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:46.372 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:37:46.408 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:37:46.432 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:46.445 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:46.444+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:46.486 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:46.486 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:46.487 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:37:46.487 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:46.495 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:37:46.509 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:46.507+0000 7f49419d0640 -1 osd.4 60 handle_osd_map injecting map crc failure 2024-09-16T03:37:46.513 INFO:tasks.rados.rados.0.smithi157.stdout:853: finishing set_chunk to oid 28 2024-09-16T03:37:46.513 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 28 set_chunk returned error code -95 offset: 129024 length: 1693696 2024-09-16T03:37:46.513 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 0 (ObjNum 83 snap 0 seq_num 83) dirty exists 2024-09-16T03:37:46.513 INFO:tasks.rados.rados.0.smithi157.stdout:855: expect (ObjNum 111 snap 0 seq_num 111) 2024-09-16T03:37:46.537 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:46.536+0000 7f12e2a1a640 -1 osd.5 60 handle_osd_map injecting map crc failure 2024-09-16T03:37:46.546 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:46.545+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:46.599 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:46.599 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:46.599 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:46.599 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:46.599 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:46.608 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:37:46.647 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:46.645+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:46.657 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2831897 2024-09-16T03:37:46.690 INFO:tasks.rados.rados.0.smithi157.stdout:854: expect (ObjNum 93 snap 0 seq_num 93) 2024-09-16T03:37:46.690 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:46.714 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:46.714 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:46.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:46.714 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:46.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.714 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:5 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'1, mlcod=0'0)", 2024-09-16T03:37:46.714 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:37:46.714 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.743056922999997, 2024-09-16T03:37:46.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.105742802, 2024-09-16T03:37:46.714 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.714 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.714 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.715 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:37:46.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:37:46.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968147+0000", 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4e-07 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968752+0000", 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060437900000000005 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968762+0000", 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0284e-05 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968778+0000", 2024-09-16T03:37:46.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6124e-05 2024-09-16T03:37:46.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069797+0000", 2024-09-16T03:37:46.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10101895700000001 2024-09-16T03:37:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069833+0000", 2024-09-16T03:37:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6013000000000002e-05 2024-09-16T03:37:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073838+0000", 2024-09-16T03:37:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040049049999999996 2024-09-16T03:37:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073889+0000", 2024-09-16T03:37:46.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.13e-05 2024-09-16T03:37:46.719 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.719 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.719 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.719 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:6 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'2, mlcod=0'0)", 2024-09-16T03:37:46.719 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.973669+0000", 2024-09-16T03:37:46.719 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.737534193000002, 2024-09-16T03:37:46.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100614443, 2024-09-16T03:37:46.720 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.720 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.720 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.720 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.973669+0000", 2024-09-16T03:37:46.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.973669+0000", 2024-09-16T03:37:46.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.973671+0000", 2024-09-16T03:37:46.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.463e-06 2024-09-16T03:37:46.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.974070+0000", 2024-09-16T03:37:46.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000399334 2024-09-16T03:37:46.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.974082+0000", 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1523e-05 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.974087+0000", 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8069999999999996e-06 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.071099+0000", 2024-09-16T03:37:46.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.097011981999999997 2024-09-16T03:37:46.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.071116+0000", 2024-09-16T03:37:46.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6566e-05 2024-09-16T03:37:46.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.074255+0000", 2024-09-16T03:37:46.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031387540000000001 2024-09-16T03:37:46.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.074284+0000", 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9014e-05 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:7 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'3, mlcod=0'0)", 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.736164424000002, 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20061241399999999, 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.724 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975042+0000", 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5289999999999998e-06 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975079+0000", 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7626999999999997e-05 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975104+0000", 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4958999999999999e-05 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.075211+0000", 2024-09-16T03:37:46.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100106814 2024-09-16T03:37:46.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080573+0000", 2024-09-16T03:37:46.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0053618650000000004 2024-09-16T03:37:46.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080592+0000", 2024-09-16T03:37:46.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9079000000000001e-05 2024-09-16T03:37:46.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175497+0000", 2024-09-16T03:37:46.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.094904832999999994 2024-09-16T03:37:46.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175651+0000", 2024-09-16T03:37:46.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015470800000000001 2024-09-16T03:37:46.728 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.728 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.728 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.729 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:46.729 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:37:46.729 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.525211251999998, 2024-09-16T03:37:46.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10311670100000001, 2024-09-16T03:37:46.729 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.729 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.729 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.729 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:37:46.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186761+0000", 2024-09-16T03:37:46.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076880999999999998 2024-09-16T03:37:46.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186772+0000", 2024-09-16T03:37:46.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0986e-05 2024-09-16T03:37:46.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186799+0000", 2024-09-16T03:37:46.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6254e-05 2024-09-16T03:37:46.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186876+0000", 2024-09-16T03:37:46.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7523999999999998e-05 2024-09-16T03:37:46.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186896+0000", 2024-09-16T03:37:46.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0302999999999999e-05 2024-09-16T03:37:46.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289077+0000", 2024-09-16T03:37:46.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10218086899999999 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289109+0000", 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1625000000000003e-05 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36)", 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.524005463000002, 2024-09-16T03:37:46.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102408122, 2024-09-16T03:37:46.734 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.734 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.734 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.734 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:37:46.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:37:46.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187199+0000", 2024-09-16T03:37:46.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6199999999999996e-07 2024-09-16T03:37:46.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188184+0000", 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000985442 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188194+0000", 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5659999999999996e-06 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188210+0000", 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5858999999999998e-05 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288363+0000", 2024-09-16T03:37:46.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100152872 2024-09-16T03:37:46.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288390+0000", 2024-09-16T03:37:46.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7081999999999999e-05 2024-09-16T03:37:46.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289577+0000", 2024-09-16T03:37:46.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011875740000000001 2024-09-16T03:37:46.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289606+0000", 2024-09-16T03:37:46.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8864999999999999e-05 2024-09-16T03:37:46.738 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.738 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.738 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.738 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:47 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.288361+0000", 2024-09-16T03:37:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.422841921, 2024-09-16T03:37:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100659055, 2024-09-16T03:37:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288361+0000", 2024-09-16T03:37:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.739 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288361+0000", 2024-09-16T03:37:46.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288364+0000", 2024-09-16T03:37:46.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2570000000000002e-06 2024-09-16T03:37:46.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288376+0000", 2024-09-16T03:37:46.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2444e-05 2024-09-16T03:37:46.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288387+0000", 2024-09-16T03:37:46.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0706999999999999e-05 2024-09-16T03:37:46.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288395+0000", 2024-09-16T03:37:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1419999999999993e-06 2024-09-16T03:37:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.388498+0000", 2024-09-16T03:37:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100103236 2024-09-16T03:37:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.388520+0000", 2024-09-16T03:37:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1815e-05 2024-09-16T03:37:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.388985+0000", 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00046473100000000002 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.389021+0000", 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5722999999999997e-05 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.743 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:10 3.3 e41/31 3:c069f14c:::smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'16, mlcod=41'14)", 2024-09-16T03:37:46.744 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.701871+0000", 2024-09-16T03:37:46.744 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.009331979999999, 2024-09-16T03:37:46.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100943475, 2024-09-16T03:37:46.744 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.744 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.744 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.744 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.744 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.744 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701871+0000", 2024-09-16T03:37:46.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.745 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701871+0000", 2024-09-16T03:37:46.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.745 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.745 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701873+0000", 2024-09-16T03:37:46.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1659999999999999e-06 2024-09-16T03:37:46.745 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.745 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701879+0000", 2024-09-16T03:37:46.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6540000000000002e-06 2024-09-16T03:37:46.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701891+0000", 2024-09-16T03:37:46.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.182e-05 2024-09-16T03:37:46.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701898+0000", 2024-09-16T03:37:46.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3270000000000003e-06 2024-09-16T03:37:46.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802103+0000", 2024-09-16T03:37:46.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10020419899999999 2024-09-16T03:37:46.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802141+0000", 2024-09-16T03:37:46.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8411999999999997e-05 2024-09-16T03:37:46.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802782+0000", 2024-09-16T03:37:46.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00064137500000000002 2024-09-16T03:37:46.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.748 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802815+0000", 2024-09-16T03:37:46.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2521999999999998e-05 2024-09-16T03:37:46.748 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.748 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.748 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.748 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:24 3.3 e45/31 3:dc284e8d:::smithi15736696-28 low_tier:head v 45'23, mlcod=42'21)", 2024-09-16T03:37:46.748 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.205792+0000", 2024-09-16T03:37:46.749 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 26.505410978, 2024-09-16T03:37:46.749 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100620102, 2024-09-16T03:37:46.749 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.749 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.749 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.749 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.749 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.749 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.749 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205792+0000", 2024-09-16T03:37:46.749 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.749 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.749 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.750 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205792+0000", 2024-09-16T03:37:46.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.750 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.750 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.750 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205793+0000", 2024-09-16T03:37:46.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-07 2024-09-16T03:37:46.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.750 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.750 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.750 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205799+0000", 2024-09-16T03:37:46.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3409999999999999e-06 2024-09-16T03:37:46.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.751 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.751 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.751 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205808+0000", 2024-09-16T03:37:46.751 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7800000000000006e-06 2024-09-16T03:37:46.751 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.751 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.751 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.751 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205814+0000", 2024-09-16T03:37:46.751 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2330000000000003e-06 2024-09-16T03:37:46.751 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.751 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.751 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.305921+0000", 2024-09-16T03:37:46.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100107423 2024-09-16T03:37:46.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.752 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.752 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.305935+0000", 2024-09-16T03:37:46.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3529e-05 2024-09-16T03:37:46.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.752 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.752 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306385+0000", 2024-09-16T03:37:46.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045013900000000001 2024-09-16T03:37:46.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306413+0000", 2024-09-16T03:37:46.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7424999999999999e-05 2024-09-16T03:37:46.753 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.753 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.753 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.753 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.4.22:6 3.3 e45/31 3:deac805c:::smithi15736696-44 low_tier:head v 45'24, mlcod=42'21)", 2024-09-16T03:37:46.753 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.206475+0000", 2024-09-16T03:37:46.753 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 26.504728366999998, 2024-09-16T03:37:46.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10029721, 2024-09-16T03:37:46.754 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.754 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.754 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.754 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206475+0000", 2024-09-16T03:37:46.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206475+0000", 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206476+0000", 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.032e-06 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206484+0000", 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7659999999999998e-06 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206493+0000", 2024-09-16T03:37:46.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0010000000000007e-06 2024-09-16T03:37:46.756 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.756 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206498+0000", 2024-09-16T03:37:46.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7620000000000003e-06 2024-09-16T03:37:46.756 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.756 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306126+0000", 2024-09-16T03:37:46.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099628128999999996 2024-09-16T03:37:46.756 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306137+0000", 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0815999999999999e-05 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306750+0000", 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061334499999999995 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306772+0000", 2024-09-16T03:37:46.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2359000000000001e-05 2024-09-16T03:37:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.5.21:14 3.3 e45/31 3:fb369346:::smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'29, mlcod=45'27)", 2024-09-16T03:37:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.924074+0000", 2024-09-16T03:37:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.787129081, 2024-09-16T03:37:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100324169, 2024-09-16T03:37:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.758 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.759 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924074+0000", 2024-09-16T03:37:46.759 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.759 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.759 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924074+0000", 2024-09-16T03:37:46.759 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.759 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.760 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924076+0000", 2024-09-16T03:37:46.760 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2780000000000001e-06 2024-09-16T03:37:46.760 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.760 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.760 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.760 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924081+0000", 2024-09-16T03:37:46.760 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4190000000000001e-06 2024-09-16T03:37:46.760 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.760 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.760 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.760 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924090+0000", 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4700000000000008e-06 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924096+0000", 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2689999999999999e-06 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924929+0000", 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00083369399999999995 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.762 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.924955+0000", 2024-09-16T03:37:46.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5893000000000001e-05 2024-09-16T03:37:46.762 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.762 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.762 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.762 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024359+0000", 2024-09-16T03:37:46.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099403434999999998 2024-09-16T03:37:46.762 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.762 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.762 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.762 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024398+0000", 2024-09-16T03:37:46.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9711e-05 2024-09-16T03:37:46.763 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.763 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.763 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.763 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.763 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.763 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.5.21:19 3.3 e47/31 3:fb369346:::smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 47'44, mlcod=47'41)", 2024-09-16T03:37:46.763 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.339975+0000", 2024-09-16T03:37:46.763 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.371228747, 2024-09-16T03:37:46.763 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10201615, 2024-09-16T03:37:46.763 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.763 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.763 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.764 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.764 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.764 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339975+0000", 2024-09-16T03:37:46.764 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.764 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.764 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.764 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339975+0000", 2024-09-16T03:37:46.764 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.764 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.765 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339976+0000", 2024-09-16T03:37:46.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.204e-06 2024-09-16T03:37:46.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.765 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.765 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339982+0000", 2024-09-16T03:37:46.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:37:46.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.765 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.765 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339992+0000", 2024-09-16T03:37:46.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0196e-05 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339998+0000", 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.463e-06 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.438719+0000", 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098721712000000003 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.766 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.438747+0000", 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7475e-05 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.441935+0000", 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031878480000000001 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.441991+0000", 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6100999999999997e-05 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.767 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.768 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.768 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:44 3.3 e49/31 3:dc284e8d:::smithi15736696-28 low_tier:head v 49'55, mlcod=49'53)", 2024-09-16T03:37:46.768 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:37:46.768 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.588240028000001, 2024-09-16T03:37:46.768 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.149341897, 2024-09-16T03:37:46.768 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.768 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.768 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.768 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.768 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122964+0000", 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.91e-07 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.770 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.770 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122968+0000", 2024-09-16T03:37:46.770 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1470000000000003e-06 2024-09-16T03:37:46.770 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.770 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.770 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.770 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122979+0000", 2024-09-16T03:37:46.770 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1090000000000001e-05 2024-09-16T03:37:46.770 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.770 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.770 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.770 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.223110+0000", 2024-09-16T03:37:46.770 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100131357 2024-09-16T03:37:46.771 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.771 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271688+0000", 2024-09-16T03:37:46.771 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048578192999999999 2024-09-16T03:37:46.771 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.771 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271712+0000", 2024-09-16T03:37:46.771 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4066e-05 2024-09-16T03:37:46.771 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272260+0000", 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054737000000000004 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272305+0000", 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5482999999999998e-05 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1007 3.3 e50/31 3:fda66a0d:::920f29fe85d0745d4cd24a887714ac53f1b9a978ec2208ff473c99287fdf8feb:head v 50'57, mlcod=49'55)", 2024-09-16T03:37:46.772 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:37:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.116613474000001, 2024-09-16T03:37:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20236485200000001, 2024-09-16T03:37:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:37:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.773 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:37:46.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594592+0000", 2024-09-16T03:37:46.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0700000000000001e-06 2024-09-16T03:37:46.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595508+0000", 2024-09-16T03:37:46.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00091599499999999996 2024-09-16T03:37:46.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.775 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.775 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595521+0000", 2024-09-16T03:37:46.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:37:46.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.775 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.775 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595531+0000", 2024-09-16T03:37:46.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2499999999999995e-06 2024-09-16T03:37:46.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.776 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.776 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.776 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795762+0000", 2024-09-16T03:37:46.776 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200230977 2024-09-16T03:37:46.776 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.776 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.776 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.776 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795778+0000", 2024-09-16T03:37:46.776 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796912+0000", 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011338520000000001 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796955+0000", 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2768000000000001e-05 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:57 3.3 e54/31 3:c069f14c:::smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 54'65, mlcod=53'63)", 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:34.964218+0000", 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.746985765, 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10111977699999999, 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964218+0000", 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.778 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.779 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964218+0000", 2024-09-16T03:37:46.779 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.779 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.779 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964218+0000", 2024-09-16T03:37:46.779 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8900000000000001e-07 2024-09-16T03:37:46.779 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964222+0000", 2024-09-16T03:37:46.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8929999999999998e-06 2024-09-16T03:37:46.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.780 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.780 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964231+0000", 2024-09-16T03:37:46.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1449999999999997e-06 2024-09-16T03:37:46.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.780 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.780 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964239+0000", 2024-09-16T03:37:46.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2369999999999992e-06 2024-09-16T03:37:46.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.781 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.781 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.781 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.064498+0000", 2024-09-16T03:37:46.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100259205 2024-09-16T03:37:46.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.781 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.781 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.781 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.064537+0000", 2024-09-16T03:37:46.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8486999999999999e-05 2024-09-16T03:37:46.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.781 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.781 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.781 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.065291+0000", 2024-09-16T03:37:46.782 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00075431700000000001 2024-09-16T03:37:46.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.782 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.782 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.065337+0000", 2024-09-16T03:37:46.782 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6304000000000003e-05 2024-09-16T03:37:46.782 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.782 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.782 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.782 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:58 3.3 e54/31 3:e9b7d9c2:::f15a4bc1b12b17a23021c0e3457eb377158b231decb6e67343dc236bab942ef9:head v 54'66, mlcod=53'63)", 2024-09-16T03:37:46.782 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:34.964431+0000", 2024-09-16T03:37:46.783 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.746772713, 2024-09-16T03:37:46.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101289616, 2024-09-16T03:37:46.783 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.783 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.783 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.783 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.783 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.783 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964431+0000", 2024-09-16T03:37:46.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.783 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.783 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.784 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.784 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964431+0000", 2024-09-16T03:37:46.784 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.784 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.784 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.784 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.784 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964431+0000", 2024-09-16T03:37:46.784 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9999999999999997e-07 2024-09-16T03:37:46.784 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.784 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.784 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.784 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964445+0000", 2024-09-16T03:37:46.784 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3375e-05 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964453+0000", 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0029999999999994e-06 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964462+0000", 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1919999999999994e-06 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.785 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.064937+0000", 2024-09-16T03:37:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10047542600000001 2024-09-16T03:37:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.064952+0000", 2024-09-16T03:37:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4623999999999999e-05 2024-09-16T03:37:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.065678+0000", 2024-09-16T03:37:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072640599999999997 2024-09-16T03:37:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.787 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.787 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.065720+0000", 2024-09-16T03:37:46.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1990000000000003e-05 2024-09-16T03:37:46.787 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.787 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.787 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.787 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.787 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1247 4.11 4:89839b7e:::smithi15736696-12:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-16T03:37:46.787 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:37:46.787 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.7556262930000006, 2024-09-16T03:37:46.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202294525, 2024-09-16T03:37:46.787 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.788 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.788 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.788 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:46.788 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:46.788 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:46.788 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1247 2024-09-16T03:37:46.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.788 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.788 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:37:46.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955579+0000", 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3240000000000001e-06 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955587+0000", 2024-09-16T03:37:46.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:37:46.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955600+0000", 2024-09-16T03:37:46.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2723e-05 2024-09-16T03:37:46.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955612+0000", 2024-09-16T03:37:46.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2422999999999999e-05 2024-09-16T03:37:46.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055837+0000", 2024-09-16T03:37:46.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10022539699999999 2024-09-16T03:37:46.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055974+0000", 2024-09-16T03:37:46.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000136995 2024-09-16T03:37:46.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,7,8,11,12", 2024-09-16T03:37:46.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.056092+0000", 2024-09-16T03:37:46.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011738 2024-09-16T03:37:46.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:46.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057209+0000", 2024-09-16T03:37:46.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011170329999999999 2024-09-16T03:37:46.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:46.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057262+0000", 2024-09-16T03:37:46.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2859000000000003e-05 2024-09-16T03:37:46.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:46.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157594+0000", 2024-09-16T03:37:46.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10033241699999999 2024-09-16T03:37:46.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:46.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157648+0000", 2024-09-16T03:37:46.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3863999999999998e-05 2024-09-16T03:37:46.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:46.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157692+0000", 2024-09-16T03:37:46.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4406999999999997e-05 2024-09-16T03:37:46.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:46.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157732+0000", 2024-09-16T03:37:46.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9156000000000003e-05 2024-09-16T03:37:46.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157780+0000", 2024-09-16T03:37:46.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8801999999999998e-05 2024-09-16T03:37:46.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157872+0000", 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1311999999999996e-05 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop_reply(client.4628.0:1247 4.11 e54/52 ondisk, result = 0)", 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:37.057448+0000", 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.6537557310000004, 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100154046, 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.795 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057448+0000", 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057448+0000", 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.796 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057449+0000", 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5910000000000001e-06 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057454+0000", 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0100000000000003e-06 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057466+0000", 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1627000000000001e-05 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057473+0000", 2024-09-16T03:37:46.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6409999999999996e-06 2024-09-16T03:37:46.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157584+0000", 2024-09-16T03:37:46.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100110977 2024-09-16T03:37:46.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157590+0000", 2024-09-16T03:37:46.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4409999999999998e-06 2024-09-16T03:37:46.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157602+0000", 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1759e-05 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:61 3.3 e57/31 3:e44244d1:::smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 57'85, mlcod=57'82)", 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:39.853906+0000", 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.8572972769999998, 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100871035, 2024-09-16T03:37:46.799 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.800 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.800 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.800 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.853906+0000", 2024-09-16T03:37:46.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.853906+0000", 2024-09-16T03:37:46.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.853907+0000", 2024-09-16T03:37:46.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7199999999999997e-07 2024-09-16T03:37:46.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.853913+0000", 2024-09-16T03:37:46.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9220000000000002e-06 2024-09-16T03:37:46.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.853924+0000", 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1212e-05 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.953994+0000", 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100069273 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.954038+0000", 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4628e-05 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.803 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.954056+0000", 2024-09-16T03:37:46.803 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7651000000000001e-05 2024-09-16T03:37:46.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.803 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.954734+0000", 2024-09-16T03:37:46.803 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067852899999999998 2024-09-16T03:37:46.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.804 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.954777+0000", 2024-09-16T03:37:46.804 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2848e-05 2024-09-16T03:37:46.804 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.804 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.804 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.804 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.804 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.804 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1447 3.3 e58/31 3:d2eec433:::1fda83a16db131b06a5bf7beb910c0624eb0c0478a30a35e6b1e31b63f6e3ff4:head v 58'90, mlcod=58'88)", 2024-09-16T03:37:46.804 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:41.806613+0000", 2024-09-16T03:37:46.804 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.9045905760000004, 2024-09-16T03:37:46.804 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101561763, 2024-09-16T03:37:46.804 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.805 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.805 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.805 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.805 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.805 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.805 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.806613+0000", 2024-09-16T03:37:46.805 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.805 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.805 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.805 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.805 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.806613+0000", 2024-09-16T03:37:46.805 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.805 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.806616+0000", 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.456e-06 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.807133+0000", 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000516787 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.807144+0000", 2024-09-16T03:37:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0448000000000001e-05 2024-09-16T03:37:46.807 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.807 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.807 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.807 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.807150+0000", 2024-09-16T03:37:46.807 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6989999999999997e-06 2024-09-16T03:37:46.807 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.807 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.807 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.807 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.907320+0000", 2024-09-16T03:37:46.807 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10016939900000001 2024-09-16T03:37:46.807 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.807 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.807 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.907334+0000", 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4241e-05 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.908143+0000", 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00080902999999999997 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.908175+0000", 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1702999999999998e-05 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.808 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.809 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.809 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.809 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.809 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1518 3.3 e59/31 3:df1be05b:::e06092e68680905b5745d5057e93f6c7581518aed07ca5db2802546e9c29aeef:head v 59'95, mlcod=58'93)", 2024-09-16T03:37:46.809 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:37:46.809 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.45318262, 2024-09-16T03:37:46.809 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104523454, 2024-09-16T03:37:46.809 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:46.809 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:46.809 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:46.809 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:46.809 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.809 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258022+0000", 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.372e-06 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.810 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258462+0000", 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00043970100000000002 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258476+0000", 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.376e-05 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258483+0000", 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.294e-06 2024-09-16T03:37:46.811 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.812 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.812 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:46.812 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358679+0000", 2024-09-16T03:37:46.812 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100196177 2024-09-16T03:37:46.812 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.812 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.812 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:46.812 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358718+0000", 2024-09-16T03:37:46.812 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9295999999999998e-05 2024-09-16T03:37:46.812 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.812 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.812 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:46.813 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362496+0000", 2024-09-16T03:37:46.813 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037772069999999999 2024-09-16T03:37:46.813 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:46.813 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:46.813 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:46.813 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362544+0000", 2024-09-16T03:37:46.813 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8646999999999997e-05 2024-09-16T03:37:46.813 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.813 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.813 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.813 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:46.813 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:46.813 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:46.814 INFO:tasks.rados.rados.0.smithi157.stdout:856: finishing write tid 1 to smithi15736696-33 2024-09-16T03:37:46.815 INFO:tasks.rados.rados.0.smithi157.stdout:859: expect (ObjNum 97 snap 0 seq_num 97) 2024-09-16T03:37:46.815 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:46.816 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:37:46.816 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:46.812+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:46.822 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 142213 2024-09-16T03:37:46.863 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 655757 2024-09-16T03:37:46.897 INFO:tasks.rados.rados.0.smithi157.stdout:851: got (0) Success 2024-09-16T03:37:46.897 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:46.897 INFO:tasks.rados.rados.0.smithi157.stdout:857: finishing write tid 1 to smithi15736696-40 2024-09-16T03:37:46.897 INFO:tasks.rados.rados.0.smithi157.stdout:857: finishing write tid 2 to smithi15736696-40 2024-09-16T03:37:46.897 INFO:tasks.rados.rados.0.smithi157.stdout:850: done (15 left) 2024-09-16T03:37:46.897 INFO:tasks.rados.rados.0.smithi157.stdout:851: done (14 left) 2024-09-16T03:37:46.897 INFO:tasks.rados.rados.0.smithi157.stdout:852: done (13 left) 2024-09-16T03:37:46.898 INFO:tasks.rados.rados.0.smithi157.stdout:853: done (12 left) 2024-09-16T03:37:46.898 INFO:tasks.rados.rados.0.smithi157.stdout:854: done (11 left) 2024-09-16T03:37:46.898 INFO:tasks.rados.rados.0.smithi157.stdout:855: done (10 left) 2024-09-16T03:37:46.898 INFO:tasks.rados.rados.0.smithi157.stdout:859: done (9 left) 2024-09-16T03:37:46.898 INFO:tasks.rados.rados.0.smithi157.stdout:866: read oid 48 snap -1 2024-09-16T03:37:46.898 INFO:tasks.rados.rados.0.smithi157.stdout:866: expect deleted 2024-09-16T03:37:46.898 INFO:tasks.rados.rados.0.smithi157.stdout:857: finishing write tid 3 to smithi15736696-40 2024-09-16T03:37:46.898 INFO:tasks.rados.rados.0.smithi157.stdout:856: finishing write tid 2 to smithi15736696-33 2024-09-16T03:37:46.898 INFO:tasks.rados.rados.0.smithi157.stdout:856: finishing write tid 3 to smithi15736696-33 2024-09-16T03:37:46.898 INFO:tasks.rados.rados.0.smithi157.stdout:867: read oid 37 snap -1 2024-09-16T03:37:46.898 INFO:tasks.rados.rados.0.smithi157.stdout:867: expect (ObjNum 110 snap 0 seq_num 110) 2024-09-16T03:37:46.899 INFO:tasks.rados.rados.0.smithi157.stdout:868: write oid 10 current snap is 0 2024-09-16T03:37:46.899 INFO:tasks.rados.rados.0.smithi157.stdout:856: finishing write tid 4 to smithi15736696-33 2024-09-16T03:37:46.899 INFO:tasks.rados.rados.0.smithi157.stdout:868: seq_num 123 ranges {467261=717275,1847991=219270} 2024-09-16T03:37:46.902 INFO:tasks.rados.rados.0.smithi157.stdout:868: writing smithi15736696-10 from 467261 to 1184536 tid 1 2024-09-16T03:37:46.908 INFO:tasks.rados.rados.0.smithi157.stdout:868: writing smithi15736696-10 from 1847991 to 2067261 tid 2 2024-09-16T03:37:46.908 INFO:tasks.rados.rados.0.smithi157.stdout:869: read oid 20 snap -1 2024-09-16T03:37:46.908 INFO:tasks.rados.rados.0.smithi157.stdout:869: expect (ObjNum 102 snap 0 seq_num 102) 2024-09-16T03:37:46.908 INFO:tasks.rados.rados.0.smithi157.stdout:870: set_chunk oid 9 target oid 2024-09-16T03:37:46.915 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:46.913+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:46.960 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:46.961 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:46.961 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:37:46.961 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:46.969 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:37:46.997 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:37:47.016 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:47.014+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:47.073 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:47.073 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:47.073 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:47.073 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:47.073 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:47.082 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:37:47.117 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:47.115+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:47.127 INFO:tasks.rados.rados.0.smithi157.stdout:870: set_chunk oid 9 offset: 241664 length: 2061312 target oid c9c74db4cc3b05b40ab52a9ed9c665f10e27723063858665cdba49de2698a01f offset: 0 snap -1 2024-09-16T03:37:47.127 INFO:tasks.rados.rados.0.smithi157.stdout:871: read oid 19 snap -1 2024-09-16T03:37:47.127 INFO:tasks.rados.rados.0.smithi157.stdout:871: expect (ObjNum 93 snap 0 seq_num 93) 2024-09-16T03:37:47.127 INFO:tasks.rados.rados.0.smithi157.stdout:872: read oid 32 snap -1 2024-09-16T03:37:47.127 INFO:tasks.rados.rados.0.smithi157.stdout:872: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-16T03:37:47.127 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:47.127 INFO:tasks.rados.rados.0.smithi157.stdout:856: finishing write tid 5 to smithi15736696-33 2024-09-16T03:37:47.127 INFO:tasks.rados.rados.0.smithi157.stdout:856: oid 33 updating version 0 to 35 2024-09-16T03:37:47.127 INFO:tasks.rados.rados.0.smithi157.stdout:856: oid 33 updating version 35 to 37 2024-09-16T03:37:47.127 INFO:tasks.rados.rados.0.smithi157.stdout:856: oid 33 version 37 is already newer than 34 2024-09-16T03:37:47.128 INFO:tasks.rados.rados.0.smithi157.stdout:856: oid 33 version 37 is already newer than 36 2024-09-16T03:37:47.128 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 37 (ObjNum 116 snap 0 seq_num 116) dirty exists 2024-09-16T03:37:47.128 INFO:tasks.rados.rados.0.smithi157.stdout:856: left oid 33 (ObjNum 116 snap 0 seq_num 116) 2024-09-16T03:37:47.128 INFO:tasks.rados.rados.0.smithi157.stdout:856: done (15 left) 2024-09-16T03:37:47.128 INFO:tasks.rados.rados.0.smithi157.stdout:873: delete oid 12 current snap is 0 2024-09-16T03:37:47.128 INFO:tasks.rados.rados.0.smithi157.stdout:857: finishing write tid 4 to smithi15736696-40 2024-09-16T03:37:47.128 INFO:tasks.rados.rados.0.smithi157.stdout:857: finishing write tid 5 to smithi15736696-40 2024-09-16T03:37:47.128 INFO:tasks.rados.rados.0.smithi157.stdout:857: oid 40 updating version 0 to 61 2024-09-16T03:37:47.128 INFO:tasks.rados.rados.0.smithi157.stdout:857: oid 40 updating version 61 to 62 2024-09-16T03:37:47.128 INFO:tasks.rados.rados.0.smithi157.stdout:857: oid 40 updating version 62 to 64 2024-09-16T03:37:47.128 INFO:tasks.rados.rados.0.smithi157.stdout:857: oid 40 version 64 is already newer than 63 2024-09-16T03:37:47.129 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 64 (ObjNum 117 snap 0 seq_num 117) dirty exists 2024-09-16T03:37:47.129 INFO:tasks.rados.rados.0.smithi157.stdout:857: left oid 40 (ObjNum 117 snap 0 seq_num 117) 2024-09-16T03:37:47.129 INFO:tasks.rados.rados.0.smithi157.stdout:858: finishing write tid 1 to smithi15736696-42 2024-09-16T03:37:47.129 INFO:tasks.rados.rados.0.smithi157.stdout:860: finishing write tid 1 to smithi15736696-1 2024-09-16T03:37:47.129 INFO:tasks.rados.rados.0.smithi157.stdout:864: finishing set_chunk to oid 31 2024-09-16T03:37:47.129 INFO:tasks.rados.rados.0.smithi157.stdout:864: got expected ENOENT (src dne) 2024-09-16T03:37:47.129 INFO:tasks.rados.rados.0.smithi157.stdout:858: finishing write tid 2 to smithi15736696-42 2024-09-16T03:37:47.129 INFO:tasks.rados.rados.0.smithi157.stdout:858: finishing write tid 3 to smithi15736696-42 2024-09-16T03:37:47.129 INFO:tasks.rados.rados.0.smithi157.stdout:862: finishing write tid 1 to smithi15736696-2 2024-09-16T03:37:47.129 INFO:tasks.rados.rados.0.smithi157.stdout:862: finishing write tid 2 to smithi15736696-2 2024-09-16T03:37:47.129 INFO:tasks.rados.rados.0.smithi157.stdout:858: finishing write tid 4 to smithi15736696-42 2024-09-16T03:37:47.129 INFO:tasks.rados.rados.0.smithi157.stdout:860: finishing write tid 2 to smithi15736696-1 2024-09-16T03:37:47.130 INFO:tasks.rados.rados.0.smithi157.stdout:865: finishing write tid 1 to smithi15736696-16 2024-09-16T03:37:47.130 INFO:tasks.rados.rados.0.smithi157.stdout:865: finishing write tid 2 to smithi15736696-16 2024-09-16T03:37:47.130 INFO:tasks.rados.rados.0.smithi157.stdout:865: finishing write tid 3 to smithi15736696-16 2024-09-16T03:37:47.130 INFO:tasks.rados.rados.0.smithi157.stdout:865: finishing write tid 4 to smithi15736696-16 2024-09-16T03:37:47.130 INFO:tasks.rados.rados.0.smithi157.stdout:865: oid 16 updating version 0 to 33 2024-09-16T03:37:47.130 INFO:tasks.rados.rados.0.smithi157.stdout:865: oid 16 version 33 is already newer than 31 2024-09-16T03:37:47.130 INFO:tasks.rados.rados.0.smithi157.stdout:865: oid 16 version 33 is already newer than 32 2024-09-16T03:37:47.130 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 33 (ObjNum 122 snap 0 seq_num 122) dirty exists 2024-09-16T03:37:47.130 INFO:tasks.rados.rados.0.smithi157.stdout:865: left oid 16 (ObjNum 122 snap 0 seq_num 122) 2024-09-16T03:37:47.130 INFO:tasks.rados.rados.0.smithi157.stdout:862: finishing write tid 3 to smithi15736696-2 2024-09-16T03:37:47.131 INFO:tasks.rados.rados.0.smithi157.stdout:862: finishing write tid 4 to smithi15736696-2 2024-09-16T03:37:47.131 INFO:tasks.rados.rados.0.smithi157.stdout:862: oid 2 updating version 0 to 14 2024-09-16T03:37:47.131 INFO:tasks.rados.rados.0.smithi157.stdout:862: oid 2 version 14 is already newer than 13 2024-09-16T03:37:47.131 INFO:tasks.rados.rados.0.smithi157.stdout:862: oid 2 updating version 14 to 15 2024-09-16T03:37:47.131 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 15 (ObjNum 120 snap 0 seq_num 120) dirty exists 2024-09-16T03:37:47.131 INFO:tasks.rados.rados.0.smithi157.stdout:862: left oid 2 (ObjNum 120 snap 0 seq_num 120) 2024-09-16T03:37:47.131 INFO:tasks.rados.rados.0.smithi157.stdout:863: finishing write tid 1 to smithi15736696-38 2024-09-16T03:37:47.131 INFO:tasks.rados.rados.0.smithi157.stdout:863: finishing write tid 2 to smithi15736696-38 2024-09-16T03:37:47.131 INFO:tasks.rados.rados.0.smithi157.stdout:863: finishing write tid 3 to smithi15736696-38 2024-09-16T03:37:47.131 INFO:tasks.rados.rados.0.smithi157.stdout:858: finishing write tid 5 to smithi15736696-42 2024-09-16T03:37:47.131 INFO:tasks.rados.rados.0.smithi157.stdout:858: oid 42 updating version 0 to 35 2024-09-16T03:37:47.131 INFO:tasks.rados.rados.0.smithi157.stdout:858: oid 42 version 35 is already newer than 32 2024-09-16T03:37:47.132 INFO:tasks.rados.rados.0.smithi157.stdout:858: oid 42 version 35 is already newer than 34 2024-09-16T03:37:47.132 INFO:tasks.rados.rados.0.smithi157.stdout:858: oid 42 version 35 is already newer than 33 2024-09-16T03:37:47.132 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 35 (ObjNum 118 snap 0 seq_num 118) dirty exists 2024-09-16T03:37:47.132 INFO:tasks.rados.rados.0.smithi157.stdout:858: left oid 42 (ObjNum 118 snap 0 seq_num 118) 2024-09-16T03:37:47.132 INFO:tasks.rados.rados.0.smithi157.stdout:863: finishing write tid 4 to smithi15736696-38 2024-09-16T03:37:47.132 INFO:tasks.rados.rados.0.smithi157.stdout:863: oid 38 updating version 0 to 25 2024-09-16T03:37:47.132 INFO:tasks.rados.rados.0.smithi157.stdout:863: oid 38 version 25 is already newer than 24 2024-09-16T03:37:47.132 INFO:tasks.rados.rados.0.smithi157.stdout:863: oid 38 version 25 is already newer than 23 2024-09-16T03:37:47.132 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 25 (ObjNum 121 snap 0 seq_num 121) dirty exists 2024-09-16T03:37:47.132 INFO:tasks.rados.rados.0.smithi157.stdout:863: left oid 38 (ObjNum 121 snap 0 seq_num 121) 2024-09-16T03:37:47.132 INFO:tasks.rados.rados.0.smithi157.stdout:860: finishing write tid 3 to smithi15736696-1 2024-09-16T03:37:47.133 INFO:tasks.rados.rados.0.smithi157.stdout:860: finishing write tid 4 to smithi15736696-1 2024-09-16T03:37:47.133 INFO:tasks.rados.rados.0.smithi157.stdout:860: oid 1 updating version 0 to 9 2024-09-16T03:37:47.133 INFO:tasks.rados.rados.0.smithi157.stdout:860: oid 1 version 9 is already newer than 7 2024-09-16T03:37:47.133 INFO:tasks.rados.rados.0.smithi157.stdout:860: oid 1 version 9 is already newer than 8 2024-09-16T03:37:47.133 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 9 (ObjNum 119 snap 0 seq_num 119) dirty exists 2024-09-16T03:37:47.133 INFO:tasks.rados.rados.0.smithi157.stdout:860: left oid 1 (ObjNum 119 snap 0 seq_num 119) 2024-09-16T03:37:47.133 INFO:tasks.rados.rados.0.smithi157.stdout:861: finishing set_chunk to oid 13 2024-09-16T03:37:47.133 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 36 (ObjNum 105 snap 0 seq_num 105) dirty exists 2024-09-16T03:37:47.133 INFO:tasks.rados.rados.0.smithi157.stdout:867: expect (ObjNum 110 snap 0 seq_num 110) 2024-09-16T03:37:47.136 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:47.136 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:47.154 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:47.154 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:47.163 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:47.170 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:47.170 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:47.188 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:47.188 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:47.203 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:47.203 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:47.207 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:47.218 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:47.216+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:47.220 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:47.220 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:47.229 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:47.230 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:47.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:47.230 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:47.230 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.230 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:53 4.7 e39/32 4:ed874e32:::smithi15736696-14:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:47.230 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:37:47.230 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.983929990999997, 2024-09-16T03:37:47.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20229554399999999, 2024-09-16T03:37:47.230 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.230 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.230 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.230 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:37:47.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9600000000000001e-07 2024-09-16T03:37:47.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243671+0000", 2024-09-16T03:37:47.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034761699999999999 2024-09-16T03:37:47.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243681+0000", 2024-09-16T03:37:47.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0505e-05 2024-09-16T03:37:47.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343786+0000", 2024-09-16T03:37:47.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10010472300000001 2024-09-16T03:37:47.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.233 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.233 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.443948+0000", 2024-09-16T03:37:47.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100161739 2024-09-16T03:37:47.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.233 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.233 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.443966+0000", 2024-09-16T03:37:47.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8045000000000001e-05 2024-09-16T03:37:47.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445575+0000", 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001608841 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445619+0000", 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3878000000000002e-05 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.235 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:269 4.6 e40/32 4:64ce87bc:::smithi15736696-20:head v 40'15, mlcod=40'13)", 2024-09-16T03:37:47.235 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.196770+0000", 2024-09-16T03:37:47.235 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 32.030482908000003, 2024-09-16T03:37:47.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20089027500000001, 2024-09-16T03:37:47.235 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.235 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.235 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.235 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.235 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.235 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.235 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196770+0000", 2024-09-16T03:37:47.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.236 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.236 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.236 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196770+0000", 2024-09-16T03:37:47.236 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.236 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.236 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.236 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.236 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196771+0000", 2024-09-16T03:37:47.236 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.288e-06 2024-09-16T03:37:47.236 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.236 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.237 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.237 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196777+0000", 2024-09-16T03:37:47.237 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7169999999999996e-06 2024-09-16T03:37:47.237 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.237 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.237 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.237 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.196787+0000", 2024-09-16T03:37:47.237 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.028e-05 2024-09-16T03:37:47.237 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.237 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.237 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.237 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.296854+0000", 2024-09-16T03:37:47.238 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100066848 2024-09-16T03:37:47.238 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.238 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.238 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.238 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.397050+0000", 2024-09-16T03:37:47.238 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100195387 2024-09-16T03:37:47.238 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.238 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.238 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.238 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.397082+0000", 2024-09-16T03:37:47.238 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1977000000000002e-05 2024-09-16T03:37:47.238 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.238 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.239 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.239 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.397619+0000", 2024-09-16T03:37:47.239 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000537734 2024-09-16T03:37:47.239 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.239 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.239 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.239 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.397660+0000", 2024-09-16T03:37:47.239 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1044000000000002e-05 2024-09-16T03:37:47.239 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.239 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.239 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.239 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.240 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.240 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:7 3.1 3:a961e47d:::smithi15736696-30 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:47.240 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.302964+0000", 2024-09-16T03:37:47.240 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.924288613000002, 2024-09-16T03:37:47.240 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19676334100000001, 2024-09-16T03:37:47.240 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.240 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.240 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.240 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.240 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.240 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.240 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302964+0000", 2024-09-16T03:37:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302964+0000", 2024-09-16T03:37:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302966+0000", 2024-09-16T03:37:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5889999999999999e-06 2024-09-16T03:37:47.241 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302972+0000", 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.682e-06 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.302983+0000", 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1147e-05 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403076+0000", 2024-09-16T03:37:47.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100093178 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403178+0000", 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101525 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403248+0000", 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0548000000000003e-05 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403370+0000", 2024-09-16T03:37:47.244 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012235100000000001 2024-09-16T03:37:47.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.244 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.403908+0000", 2024-09-16T03:37:47.244 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053726999999999996 2024-09-16T03:37:47.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.244 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.404170+0000", 2024-09-16T03:37:47.244 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00026224899999999999 2024-09-16T03:37:47.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.245 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499671+0000", 2024-09-16T03:37:47.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.095501213000000001 2024-09-16T03:37:47.245 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.245 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.245 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499704+0000", 2024-09-16T03:37:47.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2641000000000001e-05 2024-09-16T03:37:47.245 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.245 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.245 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.246 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499728+0000", 2024-09-16T03:37:47.246 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3947999999999998e-05 2024-09-16T03:37:47.246 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.246 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.246 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.246 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.246 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.246 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.4.22:3 3.1 3:bffb5658:::smithi15736696-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:47.246 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.499594+0000", 2024-09-16T03:37:47.247 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.727659460000002, 2024-09-16T03:37:47.247 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.106172458, 2024-09-16T03:37:47.247 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.247 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.247 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.247 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.247 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.247 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.247 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499594+0000", 2024-09-16T03:37:47.247 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.247 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.248 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.248 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.248 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499594+0000", 2024-09-16T03:37:47.248 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.248 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.248 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.248 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.248 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499595+0000", 2024-09-16T03:37:47.248 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.361e-06 2024-09-16T03:37:47.248 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.248 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.248 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.248 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499600+0000", 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.429e-06 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499610+0000", 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1460000000000002e-06 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.499617+0000", 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7919999999999993e-06 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.249 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.504406+0000", 2024-09-16T03:37:47.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0047886129999999997 2024-09-16T03:37:47.250 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.250 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.504474+0000", 2024-09-16T03:37:47.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7587000000000002e-05 2024-09-16T03:37:47.250 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.250 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.504600+0000", 2024-09-16T03:37:47.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012637799999999999 2024-09-16T03:37:47.250 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.251 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.251 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.505058+0000", 2024-09-16T03:37:47.251 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000457579 2024-09-16T03:37:47.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.251 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.251 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.605493+0000", 2024-09-16T03:37:47.251 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100435943 2024-09-16T03:37:47.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.251 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.251 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.605677+0000", 2024-09-16T03:37:47.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018323999999999999 2024-09-16T03:37:47.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.252 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.605725+0000", 2024-09-16T03:37:47.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8748999999999997e-05 2024-09-16T03:37:47.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.252 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.605766+0000", 2024-09-16T03:37:47.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0640999999999999e-05 2024-09-16T03:37:47.252 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.252 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.253 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.253 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:17 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:47.253 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.086208+0000", 2024-09-16T03:37:47.253 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.141044807, 2024-09-16T03:37:47.253 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20243159599999999, 2024-09-16T03:37:47.253 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.253 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.253 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.253 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086208+0000", 2024-09-16T03:37:47.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.254 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.254 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086208+0000", 2024-09-16T03:37:47.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.254 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.254 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086210+0000", 2024-09-16T03:37:47.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.058e-06 2024-09-16T03:37:47.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.255 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086216+0000", 2024-09-16T03:37:47.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1399999999999997e-06 2024-09-16T03:37:47.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.256 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086226+0000", 2024-09-16T03:37:47.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4110000000000002e-06 2024-09-16T03:37:47.256 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.256 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.256 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.256 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086235+0000", 2024-09-16T03:37:47.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4420000000000004e-06 2024-09-16T03:37:47.256 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.256 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.256 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.256 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.186482+0000", 2024-09-16T03:37:47.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10024678200000001 2024-09-16T03:37:47.257 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.186574+0000", 2024-09-16T03:37:47.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2035999999999998e-05 2024-09-16T03:37:47.257 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.186725+0000", 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150629 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.187384+0000", 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065916599999999998 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.187823+0000", 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000439018 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.258 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.288509+0000", 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100685624 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.288611+0000", 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102155 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.288640+0000", 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9135000000000001e-05 2024-09-16T03:37:47.259 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.260 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.260 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.260 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.260 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.260 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:19 3.1 3:9aba8983:::smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:47.260 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:37:47.260 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.808821322, 2024-09-16T03:37:47.260 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30153978799999998, 2024-09-16T03:37:47.260 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.260 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.260 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.260 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418452+0000", 2024-09-16T03:37:47.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0363000000000001e-05 2024-09-16T03:37:47.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418470+0000", 2024-09-16T03:37:47.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7506000000000001e-05 2024-09-16T03:37:47.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418504+0000", 2024-09-16T03:37:47.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4542999999999998e-05 2024-09-16T03:37:47.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.263 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.263 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418522+0000", 2024-09-16T03:37:47.263 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8070000000000001e-05 2024-09-16T03:37:47.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.263 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.263 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418662+0000", 2024-09-16T03:37:47.263 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013946299999999999 2024-09-16T03:37:47.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.263 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.263 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418826+0000", 2024-09-16T03:37:47.263 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016469299999999999 2024-09-16T03:37:47.264 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.264 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.264 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.264 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418989+0000", 2024-09-16T03:37:47.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000162759 2024-09-16T03:37:47.264 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.264 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.264 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.264 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.419750+0000", 2024-09-16T03:37:47.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076104299999999998 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.419865+0000", 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114338 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719876+0000", 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30001127300000002 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719920+0000", 2024-09-16T03:37:47.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4549999999999999e-05 2024-09-16T03:37:47.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.266 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719972+0000", 2024-09-16T03:37:47.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1187000000000003e-05 2024-09-16T03:37:47.266 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.266 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.266 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.267 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:10 3.1 3:8f38b759:::smithi15736696-6 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:47.267 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:37:47.267 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.719388683999998, 2024-09-16T03:37:47.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.21220708699999999, 2024-09-16T03:37:47.267 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.267 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.267 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.267 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:37:47.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:37:47.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507866+0000", 2024-09-16T03:37:47.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9999999999999999e-06 2024-09-16T03:37:47.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507873+0000", 2024-09-16T03:37:47.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7900000000000002e-06 2024-09-16T03:37:47.269 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.269 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507888+0000", 2024-09-16T03:37:47.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4554e-05 2024-09-16T03:37:47.269 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.269 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507896+0000", 2024-09-16T03:37:47.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1130000000000001e-06 2024-09-16T03:37:47.269 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.270 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.270 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507967+0000", 2024-09-16T03:37:47.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0808000000000004e-05 2024-09-16T03:37:47.270 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.270 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.270 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508028+0000", 2024-09-16T03:37:47.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1325e-05 2024-09-16T03:37:47.270 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.270 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.270 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508146+0000", 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000117922 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508722+0000", 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000575779 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.609567+0000", 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100845448 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720013+0000", 2024-09-16T03:37:47.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.110445644 2024-09-16T03:37:47.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720045+0000", 2024-09-16T03:37:47.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2186e-05 2024-09-16T03:37:47.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720071+0000", 2024-09-16T03:37:47.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6517999999999999e-05 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:11 3.1 3:adc537a6:::smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.714045454000001, 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20740488100000001, 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.274 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.274 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:37:47.274 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.274 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.274 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.274 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.274 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:37:47.274 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.274 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.274 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.274 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.274 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513209+0000", 2024-09-16T03:37:47.274 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7439999999999999e-06 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513216+0000", 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9890000000000003e-06 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513230+0000", 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3263e-05 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.275 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513236+0000", 2024-09-16T03:37:47.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7270000000000001e-06 2024-09-16T03:37:47.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513297+0000", 2024-09-16T03:37:47.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0767999999999998e-05 2024-09-16T03:37:47.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513375+0000", 2024-09-16T03:37:47.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8014000000000002e-05 2024-09-16T03:37:47.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513529+0000", 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015342900000000001 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.514258+0000", 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072896799999999998 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.709867+0000", 2024-09-16T03:37:47.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.195609016 2024-09-16T03:37:47.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.278 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720546+0000", 2024-09-16T03:37:47.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010679905999999999 2024-09-16T03:37:47.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.278 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720583+0000", 2024-09-16T03:37:47.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6659999999999998e-05 2024-09-16T03:37:47.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.279 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.279 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720613+0000", 2024-09-16T03:37:47.279 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9397e-05 2024-09-16T03:37:47.279 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.279 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.279 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.279 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.279 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.279 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:18 3.1 3:9f9f4ec5:::smithi15736696-22 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:47.279 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.104882+0000", 2024-09-16T03:37:47.279 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.122370692000001, 2024-09-16T03:37:47.279 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20218581599999999, 2024-09-16T03:37:47.279 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.280 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.280 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.280 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.280 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.280 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.280 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104882+0000", 2024-09-16T03:37:47.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.280 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.280 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.280 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.280 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104882+0000", 2024-09-16T03:37:47.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.280 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.281 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.281 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.281 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104885+0000", 2024-09-16T03:37:47.281 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7760000000000002e-06 2024-09-16T03:37:47.281 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.281 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.281 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.281 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104895+0000", 2024-09-16T03:37:47.281 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9909999999999997e-06 2024-09-16T03:37:47.281 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.281 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.281 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.281 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104911+0000", 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5654999999999998e-05 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104927+0000", 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6620999999999999e-05 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104985+0000", 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.787e-05 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105043+0000", 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7720999999999998e-05 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105174+0000", 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013075599999999999 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105846+0000", 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067215899999999997 2024-09-16T03:37:47.283 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206477+0000", 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10063103299999999 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307001+0000", 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10052391200000001 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307035+0000", 2024-09-16T03:37:47.284 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3674999999999999e-05 2024-09-16T03:37:47.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307068+0000", 2024-09-16T03:37:47.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3646999999999998e-05 2024-09-16T03:37:47.285 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.285 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.285 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.285 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:19 3.1 3:aca5d7ac:::smithi15736696-26 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:47.285 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:37:47.285 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.122188501, 2024-09-16T03:37:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264779099999999, 2024-09-16T03:37:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:37:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:37:47.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:37:47.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:37:47.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105068+0000", 2024-09-16T03:37:47.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.751e-06 2024-09-16T03:37:47.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105072+0000", 2024-09-16T03:37:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5730000000000001e-06 2024-09-16T03:37:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205143+0000", 2024-09-16T03:37:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100070665 2024-09-16T03:37:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206515+0000", 2024-09-16T03:37:47.288 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013715789999999999 2024-09-16T03:37:47.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206570+0000", 2024-09-16T03:37:47.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5049000000000001e-05 2024-09-16T03:37:47.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206695+0000", 2024-09-16T03:37:47.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012521599999999999 2024-09-16T03:37:47.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307091+0000", 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10039582 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307128+0000", 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7290999999999999e-05 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307656+0000", 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00052813199999999995 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307689+0000", 2024-09-16T03:37:47.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3056000000000003e-05 2024-09-16T03:37:47.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307712+0000", 2024-09-16T03:37:47.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3099999999999999e-05 2024-09-16T03:37:47.291 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.291 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.291 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.292 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:779 3.1 3:abdaa5c9:::2782d12a88f9842f7f754fca32c4222cf10fb780cd136830d32faaa700e687ba:head [write 0~45056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:47.292 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:37:47.292 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.740673006000002, 2024-09-16T03:37:47.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30397390600000002, 2024-09-16T03:37:47.292 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.292 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.292 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.292 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:47.292 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:47.292 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:47.292 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 779 2024-09-16T03:37:47.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.293 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:37:47.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:37:47.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.294 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486583+0000", 2024-09-16T03:37:47.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.762e-06 2024-09-16T03:37:47.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.294 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486649+0000", 2024-09-16T03:37:47.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6143999999999996e-05 2024-09-16T03:37:47.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486669+0000", 2024-09-16T03:37:47.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9837e-05 2024-09-16T03:37:47.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486681+0000", 2024-09-16T03:37:47.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2194999999999999e-05 2024-09-16T03:37:47.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486746+0000", 2024-09-16T03:37:47.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5480000000000003e-05 2024-09-16T03:37:47.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-09-16T03:37:47.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486752+0000", 2024-09-16T03:37:47.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4659999999999998e-06 2024-09-16T03:37:47.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588316+0000", 2024-09-16T03:37:47.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10156393399999999 2024-09-16T03:37:47.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588539+0000", 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022308300000000001 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588647+0000", 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000108047 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.589516+0000", 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086926899999999999 2024-09-16T03:37:47.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.298 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.298 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.789989+0000", 2024-09-16T03:37:47.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20047296100000001 2024-09-16T03:37:47.298 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.298 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.298 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790478+0000", 2024-09-16T03:37:47.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048882099999999996 2024-09-16T03:37:47.298 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.298 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790518+0000", 2024-09-16T03:37:47.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0343000000000003e-05 2024-09-16T03:37:47.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790554+0000", 2024-09-16T03:37:47.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5564e-05 2024-09-16T03:37:47.299 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.299 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.299 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.299 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:38 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:47.300 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:37:47.300 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.388845816, 2024-09-16T03:37:47.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30216061300000002, 2024-09-16T03:37:47.300 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.300 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.300 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.300 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.300 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:37:47.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:37:47.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838411+0000", 2024-09-16T03:37:47.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3780000000000001e-06 2024-09-16T03:37:47.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838422+0000", 2024-09-16T03:37:47.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1223000000000001e-05 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838450+0000", 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7878999999999999e-05 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838464+0000", 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3811e-05 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838539+0000", 2024-09-16T03:37:47.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5685000000000005e-05 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838604+0000", 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4980999999999993e-05 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838732+0000", 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012821100000000001 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839340+0000", 2024-09-16T03:37:47.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060762399999999997 2024-09-16T03:37:47.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840716+0000", 2024-09-16T03:37:47.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001375692 2024-09-16T03:37:47.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140491+0000", 2024-09-16T03:37:47.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29977497600000003 2024-09-16T03:37:47.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140534+0000", 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3411999999999997e-05 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140568+0000", 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3741000000000001e-05 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.305 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:39 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:47.306 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:37:47.306 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.387555597999999, 2024-09-16T03:37:47.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40113680400000001, 2024-09-16T03:37:47.306 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.306 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.306 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.306 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.306 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:37:47.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:37:47.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839700+0000", 2024-09-16T03:37:47.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.571e-06 2024-09-16T03:37:47.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839704+0000", 2024-09-16T03:37:47.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8399999999999997e-06 2024-09-16T03:37:47.308 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839712+0000", 2024-09-16T03:37:47.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9990000000000008e-06 2024-09-16T03:37:47.308 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839720+0000", 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0099999999999995e-06 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839783+0000", 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3014000000000003e-05 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839870+0000", 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7392999999999995e-05 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840049+0000", 2024-09-16T03:37:47.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017825800000000001 2024-09-16T03:37:47.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840874+0000", 2024-09-16T03:37:47.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082532099999999995 2024-09-16T03:37:47.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.040285+0000", 2024-09-16T03:37:47.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19941088400000001 2024-09-16T03:37:47.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.311 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.311 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240748+0000", 2024-09-16T03:37:47.311 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20046322699999999 2024-09-16T03:37:47.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.311 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.311 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240786+0000", 2024-09-16T03:37:47.311 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8158000000000003e-05 2024-09-16T03:37:47.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.311 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240834+0000", 2024-09-16T03:37:47.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8128999999999999e-05 2024-09-16T03:37:47.312 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.312 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.312 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.312 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop_reply(osd.15.23:39 3.1 e47/31 ondisk, result = 0)", 2024-09-16T03:37:47.312 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:27.041370+0000", 2024-09-16T03:37:47.312 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.185883187999998, 2024-09-16T03:37:47.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19947226200000001, 2024-09-16T03:37:47.312 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.313 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.313 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:37:47.313 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.041370+0000", 2024-09-16T03:37:47.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.041370+0000", 2024-09-16T03:37:47.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.041371+0000", 2024-09-16T03:37:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:37:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.041375+0000", 2024-09-16T03:37:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9500000000000003e-06 2024-09-16T03:37:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.041387+0000", 2024-09-16T03:37:47.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1578e-05 2024-09-16T03:37:47.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.041393+0000", 2024-09-16T03:37:47.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.567e-06 2024-09-16T03:37:47.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240721+0000", 2024-09-16T03:37:47.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19932809100000001 2024-09-16T03:37:47.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240744+0000", 2024-09-16T03:37:47.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3000999999999999e-05 2024-09-16T03:37:47.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240842+0000", 2024-09-16T03:37:47.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7931999999999997e-05 2024-09-16T03:37:47.316 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:983 3.1 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head [write 0~2138112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:37:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:37:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 17.163652658, 2024-09-16T03:37:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30819799799999997, 2024-09-16T03:37:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:47.317 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:47.318 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:47.318 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 983 2024-09-16T03:37:47.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.318 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:37:47.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:37:47.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063610+0000", 2024-09-16T03:37:47.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2990000000000002e-06 2024-09-16T03:37:47.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065735+0000", 2024-09-16T03:37:47.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021257799999999999 2024-09-16T03:37:47.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.320 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.320 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065752+0000", 2024-09-16T03:37:47.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6895999999999999e-05 2024-09-16T03:37:47.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.320 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.320 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065762+0000", 2024-09-16T03:37:47.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6830000000000003e-06 2024-09-16T03:37:47.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065871+0000", 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010904900000000001 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066005+0000", 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013369799999999999 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066135+0000", 2024-09-16T03:37:47.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000130481 2024-09-16T03:37:47.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.069822+0000", 2024-09-16T03:37:47.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036863759999999999 2024-09-16T03:37:47.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272370+0000", 2024-09-16T03:37:47.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20254802599999999 2024-09-16T03:37:47.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371671+0000", 2024-09-16T03:37:47.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099300893000000001 2024-09-16T03:37:47.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371754+0000", 2024-09-16T03:37:47.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3075e-05 2024-09-16T03:37:47.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371798+0000", 2024-09-16T03:37:47.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4742e-05 2024-09-16T03:37:47.323 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.324 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.324 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.324 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1358 4.6 e57/50 4:64ce87bc:::smithi15736696-20:head v 57'29, mlcod=0'0)", 2024-09-16T03:37:47.324 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:40.263194+0000", 2024-09-16T03:37:47.324 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.964058595, 2024-09-16T03:37:47.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.103924304, 2024-09-16T03:37:47.324 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.325 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.325 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.325 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.263194+0000", 2024-09-16T03:37:47.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.263194+0000", 2024-09-16T03:37:47.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.263195+0000", 2024-09-16T03:37:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.54e-07 2024-09-16T03:37:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.264248+0000", 2024-09-16T03:37:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010531270000000001 2024-09-16T03:37:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.264258+0000", 2024-09-16T03:37:47.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0287e-05 2024-09-16T03:37:47.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.264266+0000", 2024-09-16T03:37:47.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5700000000000004e-06 2024-09-16T03:37:47.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.264309+0000", 2024-09-16T03:37:47.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3133999999999997e-05 2024-09-16T03:37:47.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.264324+0000", 2024-09-16T03:37:47.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.539e-05 2024-09-16T03:37:47.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.367071+0000", 2024-09-16T03:37:47.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102746751 2024-09-16T03:37:47.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.367119+0000", 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7890999999999999e-05 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1392 4.16 e58/50 4:6df3bbbe:::smithi15736696-13:head v 58'34, mlcod=52'32)", 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:40.985641+0000", 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.2416124149999996, 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.109940655, 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.329 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.330 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.330 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.985641+0000", 2024-09-16T03:37:47.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.330 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.985641+0000", 2024-09-16T03:37:47.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.985642+0000", 2024-09-16T03:37:47.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.327e-06 2024-09-16T03:37:47.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.331 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.331 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.986714+0000", 2024-09-16T03:37:47.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001072164 2024-09-16T03:37:47.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.331 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.331 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.986726+0000", 2024-09-16T03:37:47.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2038999999999999e-05 2024-09-16T03:37:47.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.332 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.332 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:40.986734+0000", 2024-09-16T03:37:47.332 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6729999999999992e-06 2024-09-16T03:37:47.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.332 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.332 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.093497+0000", 2024-09-16T03:37:47.332 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.106763424 2024-09-16T03:37:47.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.332 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.333 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.093516+0000", 2024-09-16T03:37:47.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.91e-05 2024-09-16T03:37:47.333 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.333 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.333 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.095530+0000", 2024-09-16T03:37:47.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020139440000000002 2024-09-16T03:37:47.333 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.333 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.333 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.095581+0000", 2024-09-16T03:37:47.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0983999999999998e-05 2024-09-16T03:37:47.333 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.334 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.334 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.334 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1428 4.7 e58/50 4:e2f8c24a:::smithi15736696-37:head v 58'30, mlcod=54'27)", 2024-09-16T03:37:47.334 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:41.432965+0000", 2024-09-16T03:37:47.334 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.7942884299999999, 2024-09-16T03:37:47.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104066941, 2024-09-16T03:37:47.334 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.334 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.334 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.334 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.432965+0000", 2024-09-16T03:37:47.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.432965+0000", 2024-09-16T03:37:47.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.432966+0000", 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.184e-06 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.433146+0000", 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017997400000000001 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.433155+0000", 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6719999999999999e-06 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.433162+0000", 2024-09-16T03:37:47.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4819999999999999e-06 2024-09-16T03:37:47.337 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.533301+0000", 2024-09-16T03:37:47.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100138801 2024-09-16T03:37:47.337 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.533335+0000", 2024-09-16T03:37:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4437000000000002e-05 2024-09-16T03:37:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.537004+0000", 2024-09-16T03:37:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036689000000000001 2024-09-16T03:37:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.537032+0000", 2024-09-16T03:37:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7491000000000002e-05 2024-09-16T03:37:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.338 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1515 3.1 3:9c9e970c:::7249cb65b52cadf4fe8f4ec923c7e548d1de7d9628823acd2c8de19121f1c1ce:head [write 0~1094656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 2.2775269209999998, 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.305958065, 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1515 2024-09-16T03:37:47.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949729+0000", 2024-09-16T03:37:47.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.785e-06 2024-09-16T03:37:47.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.341 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.341 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950933+0000", 2024-09-16T03:37:47.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001204341 2024-09-16T03:37:47.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.341 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.341 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950950+0000", 2024-09-16T03:37:47.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6838e-05 2024-09-16T03:37:47.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950960+0000", 2024-09-16T03:37:47.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0316000000000001e-05 2024-09-16T03:37:47.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951065+0000", 2024-09-16T03:37:47.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104128 2024-09-16T03:37:47.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951202+0000", 2024-09-16T03:37:47.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013706600000000001 2024-09-16T03:37:47.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951350+0000", 2024-09-16T03:37:47.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000148197 2024-09-16T03:37:47.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.953857+0000", 2024-09-16T03:37:47.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0025067150000000001 2024-09-16T03:37:47.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.958049+0000", 2024-09-16T03:37:47.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0041924650000000003 2024-09-16T03:37:47.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255547+0000", 2024-09-16T03:37:47.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29749841900000001 2024-09-16T03:37:47.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255649+0000", 2024-09-16T03:37:47.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101513 2024-09-16T03:37:47.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255684+0000", 2024-09-16T03:37:47.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5281999999999999e-05 2024-09-16T03:37:47.345 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.345 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.345 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.345 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1528 3.1 3:8299fd0d:::8c9392b2bb5dcd9aeb358fd53aab4bf8d7b7b07146d5e0c68d69fa179cda2b1c:head [write 0~1693696] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:37:47.345 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:45.471734+0000", 2024-09-16T03:37:47.345 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.755519525, 2024-09-16T03:37:47.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.13557891399999999, 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1528 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.471734+0000", 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.471734+0000", 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.471737+0000", 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3960000000000002e-06 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.473381+0000", 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016437089999999999 2024-09-16T03:37:47.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:47.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.473404+0000", 2024-09-16T03:37:47.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3745000000000001e-05 2024-09-16T03:37:47.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:47.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.473419+0000", 2024-09-16T03:37:47.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4168000000000001e-05 2024-09-16T03:37:47.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.500937+0000", 2024-09-16T03:37:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.027517957999999999 2024-09-16T03:37:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.501045+0000", 2024-09-16T03:37:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000108867 2024-09-16T03:37:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:37:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.501159+0000", 2024-09-16T03:37:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113183 2024-09-16T03:37:47.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.504480+0000", 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0033214199999999998 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.505893+0000", 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001412815 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.607147+0000", 2024-09-16T03:37:47.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101254016 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.607242+0000", 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5361999999999999e-05 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.607312+0000", 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0275000000000001e-05 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:47.351 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:47.352 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:47.353 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:47.354 INFO:tasks.rados.rados.0.smithi157.stdout:869: expect (ObjNum 102 snap 0 seq_num 102) 2024-09-16T03:37:47.354 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:47.354 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 119467 2024-09-16T03:37:47.354 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:47.354 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:47.355 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:47.355 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:37:47.355 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:47.350+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:47.356 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:47.356 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:47.356 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:47.357 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:47.357 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:47.357 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:47.358 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:47.358 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:47.358 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:47.358 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:47.359 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:47.359 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:47.368 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:47.368 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:47.384 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:47.385 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:47.407 INFO:tasks.rados.rados.0.smithi157.stdout:870: finishing set_chunk to oid 9 2024-09-16T03:37:47.407 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 9 set_chunk returned error code -95 offset: 241664 length: 2061312 2024-09-16T03:37:47.407 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 0 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-09-16T03:37:47.407 INFO:tasks.rados.rados.0.smithi157.stdout:872: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-16T03:37:47.417 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1853428 2024-09-16T03:37:47.454 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:47.452+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:47.469 INFO:tasks.rados.rados.0.smithi157.stdout:871: expect (ObjNum 93 snap 0 seq_num 93) 2024-09-16T03:37:47.469 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:47.500 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:47.501 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:47.509 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:37:47.510 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:47.555 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:47.553+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:47.583 INFO:tasks.rados.rados.0.smithi157.stdout:868: finishing write tid 1 to smithi15736696-10 2024-09-16T03:37:47.583 INFO:tasks.rados.rados.0.smithi157.stdout:868: finishing write tid 2 to smithi15736696-10 2024-09-16T03:37:47.583 INFO:tasks.rados.rados.0.smithi157.stdout:868: finishing write tid 3 to smithi15736696-10 2024-09-16T03:37:47.584 INFO:tasks.rados.rados.0.smithi157.stdout:868: finishing write tid 4 to smithi15736696-10 2024-09-16T03:37:47.584 INFO:tasks.rados.rados.0.smithi157.stdout:868: oid 10 updating version 0 to 54 2024-09-16T03:37:47.584 INFO:tasks.rados.rados.0.smithi157.stdout:868: oid 10 version 54 is already newer than 52 2024-09-16T03:37:47.584 INFO:tasks.rados.rados.0.smithi157.stdout:868: oid 10 version 54 is already newer than 53 2024-09-16T03:37:47.584 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 54 (ObjNum 123 snap 0 seq_num 123) dirty exists 2024-09-16T03:37:47.584 INFO:tasks.rados.rados.0.smithi157.stdout:868: left oid 10 (ObjNum 123 snap 0 seq_num 123) 2024-09-16T03:37:47.655 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:47.655 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:47.656 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:47.654+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:47.663 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:37:47.757 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:47.755+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:47.768 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:47.768 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:47.776 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:37:47.822 INFO:tasks.rados.rados.0.smithi157.stdout:857: done (15 left) 2024-09-16T03:37:47.822 INFO:tasks.rados.rados.0.smithi157.stdout:858: done (14 left) 2024-09-16T03:37:47.822 INFO:tasks.rados.rados.0.smithi157.stdout:860: done (13 left) 2024-09-16T03:37:47.822 INFO:tasks.rados.rados.0.smithi157.stdout:861: done (12 left) 2024-09-16T03:37:47.822 INFO:tasks.rados.rados.0.smithi157.stdout:862: done (11 left) 2024-09-16T03:37:47.822 INFO:tasks.rados.rados.0.smithi157.stdout:863: done (10 left) 2024-09-16T03:37:47.822 INFO:tasks.rados.rados.0.smithi157.stdout:864: done (9 left) 2024-09-16T03:37:47.822 INFO:tasks.rados.rados.0.smithi157.stdout:865: done (8 left) 2024-09-16T03:37:47.822 INFO:tasks.rados.rados.0.smithi157.stdout:866: done (7 left) 2024-09-16T03:37:47.823 INFO:tasks.rados.rados.0.smithi157.stdout:867: done (6 left) 2024-09-16T03:37:47.823 INFO:tasks.rados.rados.0.smithi157.stdout:868: done (5 left) 2024-09-16T03:37:47.823 INFO:tasks.rados.rados.0.smithi157.stdout:869: done (4 left) 2024-09-16T03:37:47.823 INFO:tasks.rados.rados.0.smithi157.stdout:870: done (3 left) 2024-09-16T03:37:47.823 INFO:tasks.rados.rados.0.smithi157.stdout:871: done (2 left) 2024-09-16T03:37:47.823 INFO:tasks.rados.rados.0.smithi157.stdout:872: done (1 left) 2024-09-16T03:37:47.823 INFO:tasks.rados.rados.0.smithi157.stdout:873: done (0 left) 2024-09-16T03:37:47.823 INFO:tasks.rados.rados.0.smithi157.stdout:874: write (excl) oid 47 current snap is 0 2024-09-16T03:37:47.823 INFO:tasks.rados.rados.0.smithi157.stdout:874: seq_num 125 ranges {500875=467773,1564955=535920} 2024-09-16T03:37:47.827 INFO:tasks.rados.rados.0.smithi157.stdout:874: writing smithi15736696-47 from 500875 to 968648 tid 1 2024-09-16T03:37:47.834 INFO:tasks.rados.rados.0.smithi157.stdout:874: writing smithi15736696-47 from 1564955 to 2100875 tid 2 2024-09-16T03:37:47.834 INFO:tasks.rados.rados.0.smithi157.stdout:875: read oid 13 snap -1 2024-09-16T03:37:47.834 INFO:tasks.rados.rados.0.smithi157.stdout:875: expect (ObjNum 105 snap 0 seq_num 105) 2024-09-16T03:37:47.834 INFO:tasks.rados.rados.0.smithi157.stdout:876: read oid 10 snap -1 2024-09-16T03:37:47.834 INFO:tasks.rados.rados.0.smithi157.stdout:876: expect (ObjNum 123 snap 0 seq_num 123) 2024-09-16T03:37:47.834 INFO:tasks.rados.rados.0.smithi157.stdout:877: write (excl) oid 34 current snap is 0 2024-09-16T03:37:47.834 INFO:tasks.rados.rados.0.smithi157.stdout:877: seq_num 126 ranges {517682=543022,1623437=494245} 2024-09-16T03:37:47.839 INFO:tasks.rados.rados.0.smithi157.stdout:877: writing smithi15736696-34 from 517682 to 1060704 tid 1 2024-09-16T03:37:47.845 INFO:tasks.rados.rados.0.smithi157.stdout:877: writing smithi15736696-34 from 1623437 to 2117682 tid 2 2024-09-16T03:37:47.845 INFO:tasks.rados.rados.0.smithi157.stdout:878: write (excl) oid 33 current snap is 0 2024-09-16T03:37:47.845 INFO:tasks.rados.rados.0.smithi157.stdout:878: seq_num 127 ranges {534489=618271,1565566=568923} 2024-09-16T03:37:47.851 INFO:tasks.rados.rados.0.smithi157.stdout:878: writing smithi15736696-33 from 534489 to 1152760 tid 1 2024-09-16T03:37:47.857 INFO:tasks.rados.rados.0.smithi157.stdout:878: writing smithi15736696-33 from 1565566 to 2134489 tid 2 2024-09-16T03:37:47.857 INFO:tasks.rados.rados.0.smithi157.stdout:879: read oid 22 snap -1 2024-09-16T03:37:47.857 INFO:tasks.rados.rados.0.smithi157.stdout:879: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T03:37:47.857 INFO:tasks.rados.rados.0.smithi157.stdout:880: write oid 2 current snap is 0 2024-09-16T03:37:47.857 INFO:tasks.rados.rados.0.smithi157.stdout:880: seq_num 128 ranges {551296=693520,2024048=127248} 2024-09-16T03:37:47.860 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:47.858+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:47.863 INFO:tasks.rados.rados.0.smithi157.stdout:880: writing smithi15736696-2 from 551296 to 1244816 tid 1 2024-09-16T03:37:47.868 INFO:tasks.rados.rados.0.smithi157.stdout:880: writing smithi15736696-2 from 2024048 to 2151296 tid 2 2024-09-16T03:37:47.868 INFO:tasks.rados.rados.0.smithi157.stdout:881: read oid 27 snap -1 2024-09-16T03:37:47.868 INFO:tasks.rados.rados.0.smithi157.stdout:881: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:47.868 INFO:tasks.rados.rados.0.smithi157.stdout:882: write (excl) oid 17 current snap is 0 2024-09-16T03:37:47.868 INFO:tasks.rados.rados.0.smithi157.stdout:882: seq_num 129 ranges {568103=768769,2082530=85573} 2024-09-16T03:37:47.875 INFO:tasks.rados.rados.0.smithi157.stdout:882: writing smithi15736696-17 from 568103 to 1336872 tid 1 2024-09-16T03:37:47.880 INFO:tasks.rados.rados.0.smithi157.stdout:882: writing smithi15736696-17 from 2082530 to 2168103 tid 2 2024-09-16T03:37:47.880 INFO:tasks.rados.rados.0.smithi157.stdout:883: write (excl) oid 4 current snap is 0 2024-09-16T03:37:47.880 INFO:tasks.rados.rados.0.smithi157.stdout:883: seq_num 130 ranges {584910=560371,1857365=327545} 2024-09-16T03:37:47.886 INFO:tasks.rados.rados.0.smithi157.stdout:883: writing smithi15736696-4 from 584910 to 1145281 tid 1 2024-09-16T03:37:47.891 INFO:tasks.rados.rados.0.smithi157.stdout:883: writing smithi15736696-4 from 1857365 to 2184910 tid 2 2024-09-16T03:37:47.892 INFO:tasks.rados.rados.0.smithi157.stdout:884: set_chunk oid 15 target oid 2024-09-16T03:37:47.923 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:47.924 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:47.932 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:37:47.960 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:47.958+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:48.060 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:48.059+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:48.076 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:48.077 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:48.085 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:37:48.161 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:48.159+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:48.190 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:48.190 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:48.199 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:37:48.259 INFO:tasks.rados.rados.0.smithi157.stdout:884: set_chunk oid 15 offset: 316416 length: 605184 target oid 6dbc4a48b50301ae2e8a1e7d273a83ada0d4fccff49c6973e0d41da568607c7b offset: 0 snap -1 2024-09-16T03:37:48.259 INFO:tasks.rados.rados.0.smithi157.stdout:885: read oid 24 snap -1 2024-09-16T03:37:48.259 INFO:tasks.rados.rados.0.smithi157.stdout:885: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:37:48.259 INFO:tasks.rados.rados.0.smithi157.stdout:876: expect (ObjNum 123 snap 0 seq_num 123) 2024-09-16T03:37:48.259 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:48.262 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:48.261+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:48.315 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:48.315 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:48.325 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:37:48.346 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:48.364 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:48.361+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:48.386 INFO:tasks.rados.rados.0.smithi157.stdout:874: finishing write tid 1 to smithi15736696-47 2024-09-16T03:37:48.386 INFO:tasks.rados.rados.0.smithi157.stdout:874: finishing write tid 2 to smithi15736696-47 2024-09-16T03:37:48.386 INFO:tasks.rados.rados.0.smithi157.stdout:886: read oid 8 snap -1 2024-09-16T03:37:48.386 INFO:tasks.rados.rados.0.smithi157.stdout:886: expect deleted 2024-09-16T03:37:48.386 INFO:tasks.rados.rados.0.smithi157.stdout:876: done (12 left) 2024-09-16T03:37:48.387 INFO:tasks.rados.rados.0.smithi157.stdout:887: write oid 14 current snap is 0 2024-09-16T03:37:48.387 INFO:tasks.rados.rados.0.smithi157.stdout:881: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:48.390 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 802635 2024-09-16T03:37:48.421 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:48.420+0000 7f8be56cd640 -1 osd.7 62 handle_osd_map injecting map crc failure 2024-09-16T03:37:48.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:37:48.423 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:48.422+0000 7f8be76d1640 -1 osd.7 62 handle_osd_map injecting map crc failure 2024-09-16T03:37:48.430 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:48.430 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:48.439 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:37:48.461 INFO:tasks.rados.rados.0.smithi157.stdout:879: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-16T03:37:48.466 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:48.465+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:48.493 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1611004 2024-09-16T03:37:48.533 INFO:tasks.rados.rados.0.smithi157.stdout:877: finishing write tid 1 to smithi15736696-34 2024-09-16T03:37:48.533 INFO:tasks.rados.rados.0.smithi157.stdout:880: finishing write tid 1 to smithi15736696-2 2024-09-16T03:37:48.533 INFO:tasks.rados.rados.0.smithi157.stdout:880: finishing write tid 2 to smithi15736696-2 2024-09-16T03:37:48.533 INFO:tasks.rados.rados.0.smithi157.stdout:880: finishing write tid 3 to smithi15736696-2 2024-09-16T03:37:48.533 INFO:tasks.rados.rados.0.smithi157.stdout:880: finishing write tid 4 to smithi15736696-2 2024-09-16T03:37:48.533 INFO:tasks.rados.rados.0.smithi157.stdout:880: oid 2 updating version 0 to 17 2024-09-16T03:37:48.534 INFO:tasks.rados.rados.0.smithi157.stdout:880: oid 2 version 17 is already newer than 16 2024-09-16T03:37:48.534 INFO:tasks.rados.rados.0.smithi157.stdout:880: oid 2 updating version 17 to 18 2024-09-16T03:37:48.534 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 18 (ObjNum 128 snap 0 seq_num 128) dirty exists 2024-09-16T03:37:48.534 INFO:tasks.rados.rados.0.smithi157.stdout:880: left oid 2 (ObjNum 128 snap 0 seq_num 128) 2024-09-16T03:37:48.534 INFO:tasks.rados.rados.0.smithi157.stdout:882: finishing write tid 1 to smithi15736696-17 2024-09-16T03:37:48.534 INFO:tasks.rados.rados.0.smithi157.stdout:887: seq_num 131 ranges {601717=635620,1799494=402223} 2024-09-16T03:37:48.539 INFO:tasks.rados.rados.0.smithi157.stdout:887: writing smithi15736696-14 from 601717 to 1237337 tid 1 2024-09-16T03:37:48.545 INFO:tasks.rados.rados.0.smithi157.stdout:887: writing smithi15736696-14 from 1799494 to 2201717 tid 2 2024-09-16T03:37:48.545 INFO:tasks.rados.rados.0.smithi157.stdout:879: done (12 left) 2024-09-16T03:37:48.545 INFO:tasks.rados.rados.0.smithi157.stdout:880: done (11 left) 2024-09-16T03:37:48.546 INFO:tasks.rados.rados.0.smithi157.stdout:881: done (10 left) 2024-09-16T03:37:48.546 INFO:tasks.rados.rados.0.smithi157.stdout:888: write oid 20 current snap is 0 2024-09-16T03:37:48.546 INFO:tasks.rados.rados.0.smithi157.stdout:888: seq_num 132 ranges {618524=710869,1857976=360548} 2024-09-16T03:37:48.552 INFO:tasks.rados.rados.0.smithi157.stdout:888: writing smithi15736696-20 from 618524 to 1329393 tid 1 2024-09-16T03:37:48.557 INFO:tasks.rados.rados.0.smithi157.stdout:888: writing smithi15736696-20 from 1857976 to 2218524 tid 2 2024-09-16T03:37:48.557 INFO:tasks.rados.rados.0.smithi157.stdout:889: write (excl) oid 40 current snap is 0 2024-09-16T03:37:48.557 INFO:tasks.rados.rados.0.smithi157.stdout:875: expect (ObjNum 105 snap 0 seq_num 105) 2024-09-16T03:37:48.565 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:48.562+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:48.567 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:48.567 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:48.576 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:37:48.636 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:48.665 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:48.664+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:48.721 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:48.721 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:48.722 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:48.729 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:37:48.770 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:48.769+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:48.810 INFO:tasks.rados.rados.0.smithi157.stdout:874: finishing write tid 3 to smithi15736696-47 2024-09-16T03:37:48.810 INFO:tasks.rados.rados.0.smithi157.stdout:874: finishing write tid 4 to smithi15736696-47 2024-09-16T03:37:48.810 INFO:tasks.rados.rados.0.smithi157.stdout:874: oid 47 updating version 0 to 53 2024-09-16T03:37:48.810 INFO:tasks.rados.rados.0.smithi157.stdout:874: oid 47 version 53 is already newer than 51 2024-09-16T03:37:48.810 INFO:tasks.rados.rados.0.smithi157.stdout:874: oid 47 version 53 is already newer than 52 2024-09-16T03:37:48.810 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 53 (ObjNum 125 snap 0 seq_num 125) dirty exists 2024-09-16T03:37:48.810 INFO:tasks.rados.rados.0.smithi157.stdout:874: left oid 47 (ObjNum 125 snap 0 seq_num 125) 2024-09-16T03:37:48.810 INFO:tasks.rados.rados.0.smithi157.stdout:882: finishing write tid 2 to smithi15736696-17 2024-09-16T03:37:48.810 INFO:tasks.rados.rados.0.smithi157.stdout:882: finishing write tid 3 to smithi15736696-17 2024-09-16T03:37:48.810 INFO:tasks.rados.rados.0.smithi157.stdout:882: finishing write tid 4 to smithi15736696-17 2024-09-16T03:37:48.810 INFO:tasks.rados.rados.0.smithi157.stdout:882: oid 17 updating version 0 to 67 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:882: oid 17 version 67 is already newer than 65 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:882: oid 17 version 67 is already newer than 66 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 67 (ObjNum 129 snap 0 seq_num 129) dirty exists 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:882: left oid 17 (ObjNum 129 snap 0 seq_num 129) 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:878: finishing write tid 1 to smithi15736696-33 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:877: finishing write tid 2 to smithi15736696-34 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:877: finishing write tid 3 to smithi15736696-34 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:883: finishing write tid 1 to smithi15736696-4 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:877: finishing write tid 4 to smithi15736696-34 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:877: oid 34 updating version 0 to 59 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:877: oid 34 updating version 59 to 60 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:877: oid 34 version 60 is already newer than 58 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 60 (ObjNum 126 snap 0 seq_num 126) dirty exists 2024-09-16T03:37:48.811 INFO:tasks.rados.rados.0.smithi157.stdout:877: left oid 34 (ObjNum 126 snap 0 seq_num 126) 2024-09-16T03:37:48.812 INFO:tasks.rados.rados.0.smithi157.stdout:878: finishing write tid 2 to smithi15736696-33 2024-09-16T03:37:48.812 INFO:tasks.rados.rados.0.smithi157.stdout:878: finishing write tid 3 to smithi15736696-33 2024-09-16T03:37:48.812 INFO:tasks.rados.rados.0.smithi157.stdout:878: finishing write tid 4 to smithi15736696-33 2024-09-16T03:37:48.812 INFO:tasks.rados.rados.0.smithi157.stdout:878: oid 33 updating version 0 to 40 2024-09-16T03:37:48.812 INFO:tasks.rados.rados.0.smithi157.stdout:878: oid 33 version 40 is already newer than 39 2024-09-16T03:37:48.812 INFO:tasks.rados.rados.0.smithi157.stdout:878: oid 33 version 40 is already newer than 38 2024-09-16T03:37:48.812 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 40 (ObjNum 127 snap 0 seq_num 127) dirty exists 2024-09-16T03:37:48.812 INFO:tasks.rados.rados.0.smithi157.stdout:878: left oid 33 (ObjNum 127 snap 0 seq_num 127) 2024-09-16T03:37:48.812 INFO:tasks.rados.rados.0.smithi157.stdout:884: finishing set_chunk to oid 15 2024-09-16T03:37:48.812 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 15 set_chunk returned error code -95 offset: 316416 length: 605184 2024-09-16T03:37:48.812 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T03:37:48.812 INFO:tasks.rados.rados.0.smithi157.stdout:889: seq_num 133 ranges {635331=786118,1916458=318873} 2024-09-16T03:37:48.817 INFO:tasks.rados.rados.0.smithi157.stdout:889: writing smithi15736696-40 from 635331 to 1421449 tid 1 2024-09-16T03:37:48.825 INFO:tasks.rados.rados.0.smithi157.stdout:889: writing smithi15736696-40 from 1916458 to 2235331 tid 2 2024-09-16T03:37:48.825 INFO:tasks.rados.rados.0.smithi157.stdout:874: done (11 left) 2024-09-16T03:37:48.825 INFO:tasks.rados.rados.0.smithi157.stdout:875: done (10 left) 2024-09-16T03:37:48.825 INFO:tasks.rados.rados.0.smithi157.stdout:877: done (9 left) 2024-09-16T03:37:48.825 INFO:tasks.rados.rados.0.smithi157.stdout:878: done (8 left) 2024-09-16T03:37:48.825 INFO:tasks.rados.rados.0.smithi157.stdout:882: done (7 left) 2024-09-16T03:37:48.825 INFO:tasks.rados.rados.0.smithi157.stdout:884: done (6 left) 2024-09-16T03:37:48.825 INFO:tasks.rados.rados.0.smithi157.stdout:890: write (excl) oid 1 current snap is 0 2024-09-16T03:37:48.825 INFO:tasks.rados.rados.0.smithi157.stdout:890: seq_num 134 ranges {652138=461367,1574940=677198} 2024-09-16T03:37:48.828 INFO:tasks.rados.rados.0.smithi157.stdout:890: writing smithi15736696-1 from 652138 to 1113505 tid 1 2024-09-16T03:37:48.834 INFO:tasks.rados.rados.0.smithi157.stdout:890: writing smithi15736696-1 from 1574940 to 2252138 tid 2 2024-09-16T03:37:48.834 INFO:tasks.rados.rados.0.smithi157.stdout:891: set_chunk oid 5 target oid 2024-09-16T03:37:48.866 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:48.865+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:48.875 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:48.875 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:48.883 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:37:48.967 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:48.965+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:49.028 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:49.028 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:49.037 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:37:49.068 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:49.067+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:49.141 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:49.141 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:49.150 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:37:49.169 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:49.168+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:49.248 INFO:tasks.rados.rados.0.smithi157.stdout:891: set_chunk oid 5 offset: 2044416 length: 1597440 target oid bc57696c09e41dbb1fcdea0abea7c89f109e92d410d0cf40128955f18cd3a41f offset: 0 snap -1 2024-09-16T03:37:49.248 INFO:tasks.rados.rados.0.smithi157.stdout:892: set_chunk oid 44 target oid 2024-09-16T03:37:49.253 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:49.253 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:49.262 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:37:49.305 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:49.304+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:49.366 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:49.366 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:49.370 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:49.369+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:49.374 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:37:49.396 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:37:49.424 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:37:49.471 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:49.470+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:49.520 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:49.520 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:49.528 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:37:49.529 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:49.529 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:49.546 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:49.546 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:49.563 INFO:tasks.rados.rados.0.smithi157.stdout:892: set_chunk oid 44 offset: 230912 length: 1561088 target oid b04d32d7e3b705225e6aa5d315fafdcc195f8aa26efa9eedb6d38e940750c537 offset: 0 snap -1 2024-09-16T03:37:49.563 INFO:tasks.rados.rados.0.smithi157.stdout:893: read oid 49 snap -1 2024-09-16T03:37:49.564 INFO:tasks.rados.rados.0.smithi157.stdout:893: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-16T03:37:49.564 INFO:tasks.rados.rados.0.smithi157.stdout:894: read oid 11 snap -1 2024-09-16T03:37:49.564 INFO:tasks.rados.rados.0.smithi157.stdout:894: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:37:49.564 INFO:tasks.rados.rados.0.smithi157.stdout:895: set_chunk oid 25 target oid 2024-09-16T03:37:49.564 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:49.565 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:49.572 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:49.569+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:49.667 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:49.667 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:49.671 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:49.671 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:49.672 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:49.670+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:49.676 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:37:49.687 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:49.688 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:49.705 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:49.705 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:49.722 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:49.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:49.739 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:49.739 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:49.755 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:49.756 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:49.771 INFO:tasks.rados.rados.0.smithi157.stdout:895: set_chunk oid 25 offset: 2174464 length: 84992 target oid 29ca2d3618350021661430bbafaffc79902cc1dcc9db2addf3e0249e36dddd03 offset: 0 snap -1 2024-09-16T03:37:49.771 INFO:tasks.rados.rados.0.smithi157.stdout:896: tier_evict oid 16 2024-09-16T03:37:49.771 INFO:tasks.rados.rados.0.smithi157.stdout:896: tier_evict oid 16 snap -1 2024-09-16T03:37:49.771 INFO:tasks.rados.rados.0.smithi157.stdout:897: set_chunk oid 30 target oid 2024-09-16T03:37:49.773 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:49.773 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:49.774 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:49.771+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:49.781 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:49.782 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:49.790 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:49.790 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:49.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:37:49.844 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:49.844 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:49.860 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:49.860 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:49.874 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:49.873+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:49.878 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:49.878 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:49.894 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:49.895 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:49.900 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:49.900 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:49.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:49.900 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1675 3.2 e64/31)", 2024-09-16T03:37:49.900 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:49.826099+0000", 2024-09-16T03:37:49.900 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.073152253, 2024-09-16T03:37:49.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073169863000000002, 2024-09-16T03:37:49.900 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:49.900 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:49.901 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:49.901 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:49.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:49.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:49.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.826099+0000", 2024-09-16T03:37:49.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:49.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:49.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:49.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:49.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.826099+0000", 2024-09-16T03:37:49.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:49.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:49.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:49.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:49.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.826099+0000", 2024-09-16T03:37:49.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4499999999999997e-07 2024-09-16T03:37:49.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:49.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:49.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:49.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.827531+0000", 2024-09-16T03:37:49.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001431924 2024-09-16T03:37:49.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:49.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:49.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:49.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.827543+0000", 2024-09-16T03:37:49.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2422999999999999e-05 2024-09-16T03:37:49.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:49.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:49.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:49.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.827553+0000", 2024-09-16T03:37:49.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1679999999999993e-06 2024-09-16T03:37:49.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:49.903 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:49.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:49.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:49.903 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:37:49.903 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-09-16T03:37:49.903 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:49.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:37:49.912 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:49.912 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:49.975 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:49.974+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:50.018 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:50.018 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:50.018 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:50.018 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:50.019 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:50.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:37:50.076 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:50.074+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:50.130 INFO:tasks.rados.rados.0.smithi157.stdout:897: set_chunk oid 30 offset: 1918976 length: 673792 target oid c8d16f2878ffeb81e08803c9f8fc7bb94a2024298e0df1583bb3f4c93dc25bc4 offset: 0 snap -1 2024-09-16T03:37:50.131 INFO:tasks.rados.rados.0.smithi157.stdout:898: write (excl) oid 42 current snap is 0 2024-09-16T03:37:50.131 INFO:tasks.rados.rados.0.smithi157.stdout:898: seq_num 135 ranges {668945=536616,1633422=491363} 2024-09-16T03:37:50.137 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:17 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'7, mlcod=25'5)", 2024-09-16T03:37:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:52.877437+0000", 2024-09-16T03:37:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.2570427, 2024-09-16T03:37:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201252123, 2024-09-16T03:37:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877437+0000", 2024-09-16T03:37:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877437+0000", 2024-09-16T03:37:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877439+0000", 2024-09-16T03:37:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:37:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877446+0000", 2024-09-16T03:37:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:37:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877459+0000", 2024-09-16T03:37:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3124e-05 2024-09-16T03:37:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977610+0000", 2024-09-16T03:37:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100150878 2024-09-16T03:37:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977758+0000", 2024-09-16T03:37:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147644 2024-09-16T03:37:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977819+0000", 2024-09-16T03:37:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0798999999999998e-05 2024-09-16T03:37:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.078644+0000", 2024-09-16T03:37:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10082574399999999 2024-09-16T03:37:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.078689+0000", 2024-09-16T03:37:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5016999999999999e-05 2024-09-16T03:37:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:22 1.0 e26/23 1:30306672:devicehealth::main.db.0000000000000000:head v 26'12, mlcod=25'8)", 2024-09-16T03:37:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:53.483871+0000", 2024-09-16T03:37:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.650609277000001, 2024-09-16T03:37:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101867204, 2024-09-16T03:37:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483871+0000", 2024-09-16T03:37:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483871+0000", 2024-09-16T03:37:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483892+0000", 2024-09-16T03:37:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0962999999999999e-05 2024-09-16T03:37:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483977+0000", 2024-09-16T03:37:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4975000000000005e-05 2024-09-16T03:37:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.483986+0000", 2024-09-16T03:37:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1339999999999993e-06 2024-09-16T03:37:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.584127+0000", 2024-09-16T03:37:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10014146 2024-09-16T03:37:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.584272+0000", 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014464100000000001 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.584327+0000", 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5012000000000001e-05 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.585688+0000", 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013607890000000001 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.585738+0000", 2024-09-16T03:37:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0229999999999998e-05 2024-09-16T03:37:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:39 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'21, mlcod=26'19)", 2024-09-16T03:37:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:37:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.539161976000003, 2024-09-16T03:37:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30197182500000003, 2024-09-16T03:37:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:37:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:37:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595320+0000", 2024-09-16T03:37:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.962e-06 2024-09-16T03:37:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595336+0000", 2024-09-16T03:37:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5568000000000001e-05 2024-09-16T03:37:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595349+0000", 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3203e-05 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695424+0000", 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007542899999999 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695495+0000", 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0450999999999993e-05 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695538+0000", 2024-09-16T03:37:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3433000000000002e-05 2024-09-16T03:37:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897256+0000", 2024-09-16T03:37:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20171840699999999 2024-09-16T03:37:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897290+0000", 2024-09-16T03:37:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3371999999999998e-05 2024-09-16T03:37:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:40 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'22, mlcod=26'19)", 2024-09-16T03:37:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695446+0000", 2024-09-16T03:37:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.439033926999997, 2024-09-16T03:37:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20236575600000001, 2024-09-16T03:37:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695446+0000", 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695446+0000", 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695447+0000", 2024-09-16T03:37:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695461+0000", 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3915e-05 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695470+0000", 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6959999999999994e-06 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695478+0000", 2024-09-16T03:37:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2050000000000002e-06 2024-09-16T03:37:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695933+0000", 2024-09-16T03:37:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00045501500000000002 2024-09-16T03:37:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695958+0000", 2024-09-16T03:37:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4578000000000001e-05 2024-09-16T03:37:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897782+0000", 2024-09-16T03:37:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20182425700000001 2024-09-16T03:37:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897812+0000", 2024-09-16T03:37:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9947e-05 2024-09-16T03:37:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:41 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'23, mlcod=26'19)", 2024-09-16T03:37:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695515+0000", 2024-09-16T03:37:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.438965359000001, 2024-09-16T03:37:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20231417400000001, 2024-09-16T03:37:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695515+0000", 2024-09-16T03:37:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695515+0000", 2024-09-16T03:37:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695516+0000", 2024-09-16T03:37:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7499999999999998e-07 2024-09-16T03:37:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695590+0000", 2024-09-16T03:37:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3924999999999994e-05 2024-09-16T03:37:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695601+0000", 2024-09-16T03:37:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1253999999999999e-05 2024-09-16T03:37:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695611+0000", 2024-09-16T03:37:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9059999999999997e-06 2024-09-16T03:37:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696250+0000", 2024-09-16T03:37:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063903099999999997 2024-09-16T03:37:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696263+0000", 2024-09-16T03:37:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.276e-05 2024-09-16T03:37:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897815+0000", 2024-09-16T03:37:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201552074 2024-09-16T03:37:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897829+0000", 2024-09-16T03:37:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4249e-05 2024-09-16T03:37:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:42 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'24, mlcod=26'19)", 2024-09-16T03:37:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:37:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.438857450999997, 2024-09-16T03:37:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.202223289, 2024-09-16T03:37:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:37:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8599999999999997e-07 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695638+0000", 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4844000000000001e-05 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695645+0000", 2024-09-16T03:37:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8510000000000001e-06 2024-09-16T03:37:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695651+0000", 2024-09-16T03:37:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5600000000000001e-06 2024-09-16T03:37:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696535+0000", 2024-09-16T03:37:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088443500000000004 2024-09-16T03:37:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696550+0000", 2024-09-16T03:37:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4788e-05 2024-09-16T03:37:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897831+0000", 2024-09-16T03:37:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201281234 2024-09-16T03:37:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897846+0000", 2024-09-16T03:37:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4691e-05 2024-09-16T03:37:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:43 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'25, mlcod=26'19)", 2024-09-16T03:37:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695660+0000", 2024-09-16T03:37:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.438820081000003, 2024-09-16T03:37:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.202209369, 2024-09-16T03:37:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695660+0000", 2024-09-16T03:37:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695660+0000", 2024-09-16T03:37:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695661+0000", 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695669+0000", 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.32e-06 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695676+0000", 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3610000000000003e-06 2024-09-16T03:37:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695681+0000", 2024-09-16T03:37:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0889999999999999e-06 2024-09-16T03:37:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696770+0000", 2024-09-16T03:37:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001088852 2024-09-16T03:37:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696784+0000", 2024-09-16T03:37:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4178999999999999e-05 2024-09-16T03:37:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897849+0000", 2024-09-16T03:37:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20106433700000001 2024-09-16T03:37:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897869+0000", 2024-09-16T03:37:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0652000000000001e-05 2024-09-16T03:37:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:44 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'26, mlcod=26'19)", 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.438789413999999, 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20224351600000001, 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:37:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:37:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:37:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1099999999999996e-07 2024-09-16T03:37:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695765+0000", 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3973000000000005e-05 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695775+0000", 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5400000000000001e-06 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695781+0000", 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8369999999999999e-06 2024-09-16T03:37:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698245+0000", 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024633480000000002 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698260+0000", 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4859e-05 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897880+0000", 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199619933 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897934+0000", 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4515000000000002e-05 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:45 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'27, mlcod=26'19)", 2024-09-16T03:37:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.698152+0000", 2024-09-16T03:37:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.436328238000002, 2024-09-16T03:37:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200415062, 2024-09-16T03:37:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698152+0000", 2024-09-16T03:37:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698152+0000", 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698153+0000", 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.525e-06 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698165+0000", 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1861e-05 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698178+0000", 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3275000000000001e-05 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698186+0000", 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4139999999999997e-06 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897000+0000", 2024-09-16T03:37:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.198814149 2024-09-16T03:37:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897035+0000", 2024-09-16T03:37:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5039000000000001e-05 2024-09-16T03:37:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.898524+0000", 2024-09-16T03:37:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001488952 2024-09-16T03:37:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.898567+0000", 2024-09-16T03:37:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2846999999999998e-05 2024-09-16T03:37:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:46 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'28, mlcod=26'19)", 2024-09-16T03:37:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.698626+0000", 2024-09-16T03:37:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.435853971, 2024-09-16T03:37:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20067976400000001, 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698626+0000", 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698626+0000", 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698627+0000", 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.395e-06 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698641+0000", 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698653+0000", 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1846e-05 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.798792+0000", 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100139397 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897344+0000", 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098552085999999997 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897361+0000", 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6716000000000001e-05 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.899266+0000", 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019056520000000001 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.899306+0000", 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9322000000000001e-05 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:79 1.0 e28/23 1:30306672:devicehealth::main.db.0000000000000000:head v 28'57, mlcod=28'53)", 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:57.288914+0000", 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.845566331999997, 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10187072799999999, 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.288914+0000", 2024-09-16T03:37:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.288914+0000", 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.288917+0000", 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5049999999999999e-06 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.288998+0000", 2024-09-16T03:37:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1298000000000007e-05 2024-09-16T03:37:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.289019+0000", 2024-09-16T03:37:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0234999999999999e-05 2024-09-16T03:37:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.389117+0000", 2024-09-16T03:37:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10009857 2024-09-16T03:37:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.389340+0000", 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022286100000000001 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.389372+0000", 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2057000000000002e-05 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.390733+0000", 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013608769999999999 2024-09-16T03:37:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:57.390784+0000", 2024-09-16T03:37:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1325000000000001e-05 2024-09-16T03:37:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:99 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'73, mlcod=28'66)", 2024-09-16T03:37:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:37:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.043188172999997, 2024-09-16T03:37:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201204829, 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9800000000000004e-07 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091329+0000", 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7039000000000002e-05 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091336+0000", 2024-09-16T03:37:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:37:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191408+0000", 2024-09-16T03:37:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100072733 2024-09-16T03:37:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191470+0000", 2024-09-16T03:37:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2087000000000004e-05 2024-09-16T03:37:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191504+0000", 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3453e-05 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292441+0000", 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100937374 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292497+0000", 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5365999999999998e-05 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:9 4.8 e38/32 4:13b03fc9:::smithi15736696-3:head v 38'1, mlcod=0'0)", 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.980745+0000", 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.153734806000003, 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102593847, 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.980745+0000", 2024-09-16T03:37:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.980745+0000", 2024-09-16T03:37:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.980746+0000", 2024-09-16T03:37:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9099999999999999e-07 2024-09-16T03:37:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.981323+0000", 2024-09-16T03:37:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057713399999999998 2024-09-16T03:37:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.981335+0000", 2024-09-16T03:37:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2257e-05 2024-09-16T03:37:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.981345+0000", 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0399e-05 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.081502+0000", 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015674400000001 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.081527+0000", 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5106e-05 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.083241+0000", 2024-09-16T03:37:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017139410000000001 2024-09-16T03:37:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.083339+0000", 2024-09-16T03:37:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7874999999999994e-05 2024-09-16T03:37:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.5.21:12 3.2 e45/31 3:43460c1e:::smithi15736696-20 low_tier:head v 45'36, mlcod=45'34)", 2024-09-16T03:37:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.014759+0000", 2024-09-16T03:37:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.119720940000001, 2024-09-16T03:37:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20057897499999999, 2024-09-16T03:37:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014759+0000", 2024-09-16T03:37:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014759+0000", 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014762+0000", 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7429999999999998e-06 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014774+0000", 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1790999999999999e-05 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014790+0000", 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6409000000000002e-05 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.114865+0000", 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100074861 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.114929+0000", 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3769000000000006e-05 2024-09-16T03:37:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.114947+0000", 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8085999999999999e-05 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.215300+0000", 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100353422 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.215338+0000", 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7894e-05 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1187 3.2 e54/31 3:4b00b361:::fb2abb5d8594ae081bd7a522ca068bd21ee8449fbf7bef92e8b363b8b3f628bf:head v 54'81, mlcod=53'79)", 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:35.100308+0000", 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.034171926999999, 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10508909499999999, 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.203 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.100308+0000", 2024-09-16T03:37:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.100308+0000", 2024-09-16T03:37:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.100310+0000", 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.48e-06 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.103823+0000", 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003512987 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.103835+0000", 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2763e-05 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.203922+0000", 2024-09-16T03:37:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100086403 2024-09-16T03:37:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.203987+0000", 2024-09-16T03:37:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5300000000000002e-05 2024-09-16T03:37:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.204005+0000", 2024-09-16T03:37:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7631000000000001e-05 2024-09-16T03:37:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.205362+0000", 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013569470000000001 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:35.205397+0000", 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5583999999999997e-05 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.207 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1211 4.8 e54/50 4:13b03fc9:::smithi15736696-3:head v 54'33, mlcod=53'30)", 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.314987+0000", 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.819492666, 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102508168, 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.314987+0000", 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.314987+0000", 2024-09-16T03:37:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.314988+0000", 2024-09-16T03:37:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1149999999999999e-06 2024-09-16T03:37:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.315477+0000", 2024-09-16T03:37:50.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00048868499999999999 2024-09-16T03:37:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.315487+0000", 2024-09-16T03:37:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-06 2024-09-16T03:37:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.315492+0000", 2024-09-16T03:37:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9400000000000001e-06 2024-09-16T03:37:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.415627+0000", 2024-09-16T03:37:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100134853 2024-09-16T03:37:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.415654+0000", 2024-09-16T03:37:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7841999999999999e-05 2024-09-16T03:37:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.417458+0000", 2024-09-16T03:37:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001803708 2024-09-16T03:37:50.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.417496+0000", 2024-09-16T03:37:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7394999999999997e-05 2024-09-16T03:37:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1226 3.2 e54/31 3:6f0527c9:::7f01c73e0f253c35d7776d11c46f7e5d9edb0e9d6a760d7305f052eba477b6c2:head v 54'82, mlcod=54'80)", 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.472805+0000", 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.661674929, 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20253328100000001, 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.472805+0000", 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.472805+0000", 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.472807+0000", 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9489999999999999e-06 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.473596+0000", 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078904500000000005 2024-09-16T03:37:50.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.473614+0000", 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7832000000000001e-05 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.573713+0000", 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100098627 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.673879+0000", 2024-09-16T03:37:50.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100166088 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.673918+0000", 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9035999999999997e-05 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675283+0000", 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001365668 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675338+0000", 2024-09-16T03:37:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5036e-05 2024-09-16T03:37:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.8.22:29 3.2 e54/31 3:6f0527c9:::7f01c73e0f253c35d7776d11c46f7e5d9edb0e9d6a760d7305f052eba477b6c2:head v 54'83, mlcod=54'81)", 2024-09-16T03:37:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.876931+0000", 2024-09-16T03:37:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.257549314, 2024-09-16T03:37:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.13927160799999999, 2024-09-16T03:37:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.217 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.876931+0000", 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.876931+0000", 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.876932+0000", 2024-09-16T03:37:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4529999999999999e-06 2024-09-16T03:37:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.876939+0000", 2024-09-16T03:37:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9759999999999998e-06 2024-09-16T03:37:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.876951+0000", 2024-09-16T03:37:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2150999999999999e-05 2024-09-16T03:37:50.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.977028+0000", 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100076822 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.015635+0000", 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038607173000000002 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.015656+0000", 2024-09-16T03:37:50.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0492e-05 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.016172+0000", 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00051640599999999996 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.016202+0000", 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0134999999999999e-05 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1588 3.2 e61/31 3:57b21e08:::c9c74db4cc3b05b40ab52a9ed9c665f10e27723063858665cdba49de2698a01f:head v 61'105, mlcod=59'103)", 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:46.917498+0000", 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.2169815970000002, 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206346578, 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.917498+0000", 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.917498+0000", 2024-09-16T03:37:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.917502+0000", 2024-09-16T03:37:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0230000000000003e-06 2024-09-16T03:37:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.921109+0000", 2024-09-16T03:37:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036070149999999999 2024-09-16T03:37:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.921129+0000", 2024-09-16T03:37:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9741999999999998e-05 2024-09-16T03:37:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.921139+0000", 2024-09-16T03:37:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5079999999999994e-06 2024-09-16T03:37:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.121366+0000", 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20022730599999999 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.121386+0000", 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9856000000000001e-05 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.123808+0000", 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024219979999999999 2024-09-16T03:37:50.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.123845+0000", 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7129999999999999e-05 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1675 3.2 e64/31)", 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:49.826099+0000", 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.30838137700000001, 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102889625, 2024-09-16T03:37:50.226 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.826099+0000", 2024-09-16T03:37:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.826099+0000", 2024-09-16T03:37:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.826099+0000", 2024-09-16T03:37:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4499999999999997e-07 2024-09-16T03:37:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.827531+0000", 2024-09-16T03:37:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001431924 2024-09-16T03:37:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.827543+0000", 2024-09-16T03:37:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2422999999999999e-05 2024-09-16T03:37:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.827553+0000", 2024-09-16T03:37:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1679999999999993e-06 2024-09-16T03:37:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.927656+0000", 2024-09-16T03:37:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10010374599999999 2024-09-16T03:37:50.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.927676+0000", 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9613e-05 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.928950+0000", 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012737709999999999 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:49.928988+0000", 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8535000000000003e-05 2024-09-16T03:37:50.230 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.231 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:50.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:37:50.234 INFO:tasks.rados.rados.0.smithi157.stdout:898: writing smithi15736696-42 from 668945 to 1205561 tid 1 2024-09-16T03:37:50.234 INFO:tasks.rados.rados.0.smithi157.stdout:898: writing smithi15736696-42 from 1633422 to 2124785 tid 2 2024-09-16T03:37:50.234 INFO:tasks.rados.rados.0.smithi157.stdout:899: write (excl) oid 15 current snap is 0 2024-09-16T03:37:50.234 INFO:tasks.rados.rados.0.smithi157.stdout:899: seq_num 136 ranges {685752=611865,1975551=310201} 2024-09-16T03:37:50.234 INFO:tasks.rados.rados.0.smithi157.stdout:899: writing smithi15736696-15 from 685752 to 1297617 tid 1 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout:899: writing smithi15736696-15 from 1975551 to 2285752 tid 2 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout:883: finishing write tid 2 to smithi15736696-4 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout:883: finishing write tid 3 to smithi15736696-4 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout:883: finishing write tid 4 to smithi15736696-4 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout:883: oid 4 updating version 0 to 62 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout:883: oid 4 updating version 62 to 63 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout:883: oid 4 version 63 is already newer than 61 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 63 (ObjNum 130 snap 0 seq_num 130) dirty exists 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout:883: left oid 4 (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout:885: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2104169 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout:887: finishing write tid 1 to smithi15736696-14 2024-09-16T03:37:50.235 INFO:tasks.rados.rados.0.smithi157.stdout:887: finishing write tid 2 to smithi15736696-14 2024-09-16T03:37:50.236 INFO:tasks.rados.rados.0.smithi157.stdout:889: finishing write tid 1 to smithi15736696-40 2024-09-16T03:37:50.236 INFO:tasks.rados.rados.0.smithi157.stdout:888: finishing write tid 1 to smithi15736696-20 2024-09-16T03:37:50.236 INFO:tasks.rados.rados.0.smithi157.stdout:887: finishing write tid 3 to smithi15736696-14 2024-09-16T03:37:50.236 INFO:tasks.rados.rados.0.smithi157.stdout:883: done (15 left) 2024-09-16T03:37:50.236 INFO:tasks.rados.rados.0.smithi157.stdout:885: done (14 left) 2024-09-16T03:37:50.236 INFO:tasks.rados.rados.0.smithi157.stdout:886: done (13 left) 2024-09-16T03:37:50.236 INFO:tasks.rados.rados.0.smithi157.stdout:900: read oid 27 snap -1 2024-09-16T03:37:50.236 INFO:tasks.rados.rados.0.smithi157.stdout:900: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:50.236 INFO:tasks.rados.rados.0.smithi157.stdout:901: set_chunk oid 4 target oid 2024-09-16T03:37:50.236 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:50.230+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:50.332 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:50.331+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:50.381 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:50.381 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:50.381 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:37:50.381 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:50.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:37:50.433 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:50.431+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:50.451 INFO:tasks.rados.rados.0.smithi157.stdout:901: set_chunk oid 4 offset: 1190912 length: 574464 target oid a2b7d44d389d3ac8752762fa1d216869a8025d7e5ae15e8811d7e7f2543095f6 offset: 0 snap -1 2024-09-16T03:37:50.451 INFO:tasks.rados.rados.0.smithi157.stdout:902: write (excl) oid 37 current snap is 0 2024-09-16T03:37:50.451 INFO:tasks.rados.rados.0.smithi157.stdout:902: seq_num 137 ranges {702559=403467,1750386=495032} 2024-09-16T03:37:50.456 INFO:tasks.rados.rados.0.smithi157.stdout:902: writing smithi15736696-37 from 702559 to 1106026 tid 1 2024-09-16T03:37:50.464 INFO:tasks.rados.rados.0.smithi157.stdout:902: writing smithi15736696-37 from 1750386 to 2245418 tid 2 2024-09-16T03:37:50.464 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:50.464 INFO:tasks.rados.rados.0.smithi157.stdout:887: finishing write tid 4 to smithi15736696-14 2024-09-16T03:37:50.464 INFO:tasks.rados.rados.0.smithi157.stdout:887: oid 14 updating version 0 to 34 2024-09-16T03:37:50.465 INFO:tasks.rados.rados.0.smithi157.stdout:887: oid 14 updating version 34 to 35 2024-09-16T03:37:50.465 INFO:tasks.rados.rados.0.smithi157.stdout:887: oid 14 updating version 35 to 36 2024-09-16T03:37:50.465 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 36 (ObjNum 131 snap 0 seq_num 131) dirty exists 2024-09-16T03:37:50.465 INFO:tasks.rados.rados.0.smithi157.stdout:887: left oid 14 (ObjNum 131 snap 0 seq_num 131) 2024-09-16T03:37:50.465 INFO:tasks.rados.rados.0.smithi157.stdout:889: finishing write tid 2 to smithi15736696-40 2024-09-16T03:37:50.465 INFO:tasks.rados.rados.0.smithi157.stdout:887: done (15 left) 2024-09-16T03:37:50.465 INFO:tasks.rados.rados.0.smithi157.stdout:903: delete oid 36 current snap is 0 2024-09-16T03:37:50.465 INFO:tasks.rados.rados.0.smithi157.stdout:889: finishing write tid 3 to smithi15736696-40 2024-09-16T03:37:50.465 INFO:tasks.rados.rados.0.smithi157.stdout:889: finishing write tid 4 to smithi15736696-40 2024-09-16T03:37:50.465 INFO:tasks.rados.rados.0.smithi157.stdout:889: oid 40 updating version 0 to 69 2024-09-16T03:37:50.465 INFO:tasks.rados.rados.0.smithi157.stdout:889: oid 40 updating version 69 to 70 2024-09-16T03:37:50.465 INFO:tasks.rados.rados.0.smithi157.stdout:889: oid 40 version 70 is already newer than 68 2024-09-16T03:37:50.465 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 70 (ObjNum 133 snap 0 seq_num 133) dirty exists 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:889: left oid 40 (ObjNum 133 snap 0 seq_num 133) 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:888: finishing write tid 2 to smithi15736696-20 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:888: finishing write tid 3 to smithi15736696-20 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:888: finishing write tid 4 to smithi15736696-20 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:888: oid 20 updating version 0 to 34 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:888: oid 20 updating version 34 to 35 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:888: oid 20 version 35 is already newer than 33 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 35 (ObjNum 132 snap 0 seq_num 132) dirty exists 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:888: left oid 20 (ObjNum 132 snap 0 seq_num 132) 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:892: finishing set_chunk to oid 44 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 44 set_chunk returned error code -95 offset: 230912 length: 1561088 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 0 (ObjNum 63 snap 0 seq_num 63) dirty exists 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:894: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:37:50.466 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:50.494 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:37:50.494 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:37:50.494 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:37:50.494 INFO:tasks.thrashosds.thrasher:Checking the number of in OSDs in host ubuntu@smithi112.front.sepia.ceph.com 2024-09-16T03:37:50.494 INFO:tasks.thrashosds.thrasher:Removing all OSDs in host ubuntu@smithi112.front.sepia.ceph.com 2024-09-16T03:37:50.494 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-16T03:37:50.495 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-09-16T03:37:50.507 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:50.533 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:50.531+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:50.539 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:50.539 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:50.539 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:50.549 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:37:50.553 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:50.552+0000 7f49419d0640 -1 osd.4 64 handle_osd_map injecting map crc failure 2024-09-16T03:37:50.594 INFO:tasks.rados.rados.0.smithi157.stdout:896: got (22) Invalid argument 2024-09-16T03:37:50.594 INFO:tasks.rados.rados.0.smithi157.stdout:893: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-16T03:37:50.594 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:50.625 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:50.634 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:50.633+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:50.656 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:50.662 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:33 4.e 4:773b934b:::smithi15736696-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.050039+0000", 2024-09-16T03:37:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.608985849, 2024-09-16T03:37:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40763112099999999, 2024-09-16T03:37:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-09-16T03:37:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050039+0000", 2024-09-16T03:37:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050039+0000", 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050042+0000", 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5009999999999999e-06 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050806+0000", 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076374199999999998 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050819+0000", 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3692999999999999e-05 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050829+0000", 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8630000000000003e-06 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050917+0000", 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8196999999999996e-05 2024-09-16T03:37:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050922+0000", 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.831e-06 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153032+0000", 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102109992 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153123+0000", 2024-09-16T03:37:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1037000000000003e-05 2024-09-16T03:37:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153218+0000", 2024-09-16T03:37:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5047e-05 2024-09-16T03:37:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156107+0000", 2024-09-16T03:37:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028893299999999998 2024-09-16T03:37:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255606+0000", 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099498438999999994 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457591+0000", 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201984996 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457631+0000", 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0110000000000001e-05 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457670+0000", 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9342999999999999e-05 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:34 4.e 4:773b934b:::smithi15736696-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.604613831000002, 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50346527299999999, 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:37:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054413+0000", 2024-09-16T03:37:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0669999999999999e-06 2024-09-16T03:37:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054525+0000", 2024-09-16T03:37:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111462 2024-09-16T03:37:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054538+0000", 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3341e-05 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054545+0000", 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.584e-06 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054608+0000", 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2953999999999994e-05 2024-09-16T03:37:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054613+0000", 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6110000000000001e-06 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154852+0000", 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10023838 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154915+0000", 2024-09-16T03:37:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3138000000000004e-05 2024-09-16T03:37:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155001+0000", 2024-09-16T03:37:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5730999999999996e-05 2024-09-16T03:37:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156166+0000", 2024-09-16T03:37:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001165485 2024-09-16T03:37:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255909+0000", 2024-09-16T03:37:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099742558999999995 2024-09-16T03:37:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557800+0000", 2024-09-16T03:37:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30189130600000003 2024-09-16T03:37:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557836+0000", 2024-09-16T03:37:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6566000000000002e-05 2024-09-16T03:37:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557877+0000", 2024-09-16T03:37:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0089000000000002e-05 2024-09-16T03:37:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:35 4.e 4:773b934b:::smithi15736696-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:37:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.604465500000003, 2024-09-16T03:37:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50344320499999995, 2024-09-16T03:37:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-16T03:37:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054564+0000", 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5710000000000002e-06 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054569+0000", 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0490000000000003e-06 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054578+0000", 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054639+0000", 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1061000000000004e-05 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054644+0000", 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999996e-06 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155607+0000", 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100963372 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155652+0000", 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155722+0000", 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0130999999999997e-05 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.157921+0000", 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021990159999999998 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255940+0000", 2024-09-16T03:37:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098018569999999999 2024-09-16T03:37:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557930+0000", 2024-09-16T03:37:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.301990333 2024-09-16T03:37:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557963+0000", 2024-09-16T03:37:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2348000000000003e-05 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558003+0000", 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:36 4.e 4:773b934b:::smithi15736696-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.604426811000003, 2024-09-16T03:37:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50353811800000003, 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054599+0000", 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054601+0000", 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4360000000000001e-06 2024-09-16T03:37:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054606+0000", 2024-09-16T03:37:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1019999999999996e-06 2024-09-16T03:37:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054610+0000", 2024-09-16T03:37:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.681e-06 2024-09-16T03:37:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054662+0000", 2024-09-16T03:37:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1545000000000002e-05 2024-09-16T03:37:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:37:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054664+0000", 2024-09-16T03:37:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1919999999999999e-06 2024-09-16T03:37:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155903+0000", 2024-09-16T03:37:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101239068 2024-09-16T03:37:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155970+0000", 2024-09-16T03:37:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6934999999999995e-05 2024-09-16T03:37:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558029+0000", 2024-09-16T03:37:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40205888899999997 2024-09-16T03:37:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558078+0000", 2024-09-16T03:37:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9842999999999997e-05 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558136+0000", 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7985000000000001e-05 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:280 4.e 4:70297eed:::smithi15736696-31:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.298531+0000", 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.360494406000001, 2024-09-16T03:37:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30954304300000002, 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 280 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298531+0000", 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298531+0000", 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298534+0000", 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5769999999999998e-06 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298548+0000", 2024-09-16T03:37:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.377e-05 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.298576+0000", 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8181999999999999e-05 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.398649+0000", 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007253200000001 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.498772+0000", 2024-09-16T03:37:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100123273 2024-09-16T03:37:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.498838+0000", 2024-09-16T03:37:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5927999999999996e-05 2024-09-16T03:37:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.605923+0000", 2024-09-16T03:37:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10708543399999999 2024-09-16T03:37:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.606636+0000", 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000712836 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.607909+0000", 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00127289 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.608017+0000", 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108004 2024-09-16T03:37:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.608048+0000", 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.048e-05 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.608074+0000", 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6137000000000001e-05 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:293 4.e 4:765e9179:::smithi15736696-44:head [assert-version v9,set-chunk in=88b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:15.902028+0000", 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.756997302999999, 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40392601299999997, 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 293 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902028+0000", 2024-09-16T03:37:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902028+0000", 2024-09-16T03:37:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902031+0000", 2024-09-16T03:37:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9440000000000001e-06 2024-09-16T03:37:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902041+0000", 2024-09-16T03:37:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0477e-05 2024-09-16T03:37:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902058+0000", 2024-09-16T03:37:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7227e-05 2024-09-16T03:37:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902069+0000", 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0165e-05 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902130+0000", 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1043000000000005e-05 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:15.902202+0000", 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1884000000000005e-05 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.204680+0000", 2024-09-16T03:37:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30247867099999998 2024-09-16T03:37:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.205381+0000", 2024-09-16T03:37:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000700675 2024-09-16T03:37:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.305797+0000", 2024-09-16T03:37:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10041573099999999 2024-09-16T03:37:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.305865+0000", 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8712000000000002e-05 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.305913+0000", 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7669e-05 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.305954+0000", 2024-09-16T03:37:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0815000000000001e-05 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:308 4.e 4:773b934b:::smithi15736696-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.721774+0000", 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.937250984000002, 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30397478300000003, 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 308 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.721774+0000", 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.721774+0000", 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.721777+0000", 2024-09-16T03:37:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4159999999999998e-06 2024-09-16T03:37:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.721787+0000", 2024-09-16T03:37:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0162e-05 2024-09-16T03:37:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.721802+0000", 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4810000000000001e-05 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.721810+0000", 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4489999999999999e-06 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.822017+0000", 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10020680999999999 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.822182+0000", 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016551000000000001 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.824437+0000", 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00225435 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.824912+0000", 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047550700000000001 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.825108+0000", 2024-09-16T03:37:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019587399999999999 2024-09-16T03:37:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:17.025669+0000", 2024-09-16T03:37:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20056116500000001 2024-09-16T03:37:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:17.025713+0000", 2024-09-16T03:37:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4079000000000003e-05 2024-09-16T03:37:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:17.025749+0000", 2024-09-16T03:37:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5651000000000001e-05 2024-09-16T03:37:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:393 4.e 4:765e9179:::smithi15736696-44:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.204958+0000", 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.454067601999999, 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40651974299999999, 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 393 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.204958+0000", 2024-09-16T03:37:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.204958+0000", 2024-09-16T03:37:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.204960+0000", 2024-09-16T03:37:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0660000000000002e-06 2024-09-16T03:37:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.204970+0000", 2024-09-16T03:37:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0752e-05 2024-09-16T03:37:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.204981+0000", 2024-09-16T03:37:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-16T03:37:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.204989+0000", 2024-09-16T03:37:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4549999999999998e-06 2024-09-16T03:37:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.205292+0000", 2024-09-16T03:37:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030287400000000001 2024-09-16T03:37:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.205363+0000", 2024-09-16T03:37:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1194999999999999e-05 2024-09-16T03:37:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309936+0000", 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10457277 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.310501+0000", 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056559799999999995 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.310890+0000", 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038853899999999997 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.611406+0000", 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30051645900000001 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.611442+0000", 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6077e-05 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.611477+0000", 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5011000000000001e-05 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:480 4.e 4:70297eed:::smithi15736696-31:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.621490+0000", 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.037535286000001, 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40240205499999998, 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 480 2024-09-16T03:37:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621490+0000", 2024-09-16T03:37:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621490+0000", 2024-09-16T03:37:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621493+0000", 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5099999999999999e-06 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621508+0000", 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.489e-05 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621536+0000", 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8232e-05 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621560+0000", 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3757e-05 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621675+0000", 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114401 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.621814+0000", 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013906500000000001 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822507+0000", 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20069381 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.825430+0000", 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029227799999999998 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.825754+0000", 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032421700000000002 2024-09-16T03:37:50.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.023790+0000", 2024-09-16T03:37:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.198035397 2024-09-16T03:37:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.023842+0000", 2024-09-16T03:37:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1958e-05 2024-09-16T03:37:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.023892+0000", 2024-09-16T03:37:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0037999999999998e-05 2024-09-16T03:37:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:508 4.e 4:773b934b:::smithi15736696-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.822186+0000", 2024-09-16T03:37:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.836838788000001, 2024-09-16T03:37:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41573648800000002, 2024-09-16T03:37:50.722 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 508 2024-09-16T03:37:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822186+0000", 2024-09-16T03:37:50.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822186+0000", 2024-09-16T03:37:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822189+0000", 2024-09-16T03:37:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6010000000000002e-06 2024-09-16T03:37:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822198+0000", 2024-09-16T03:37:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0990000000000005e-06 2024-09-16T03:37:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822213+0000", 2024-09-16T03:37:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4481000000000001e-05 2024-09-16T03:37:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922294+0000", 2024-09-16T03:37:50.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10008130799999999 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922365+0000", 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0973e-05 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922457+0000", 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2393999999999997e-05 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.025017+0000", 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102560259 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.025625+0000", 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060773999999999995 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.025985+0000", 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036012000000000001 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.237775+0000", 2024-09-16T03:37:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.211790115 2024-09-16T03:37:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.237859+0000", 2024-09-16T03:37:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3387000000000006e-05 2024-09-16T03:37:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.237923+0000", 2024-09-16T03:37:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4010999999999994e-05 2024-09-16T03:37:50.728 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:797 4.e 4:765e9179:::smithi15736696-44:head [write 457157~535817,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.999730+0000", 2024-09-16T03:37:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.659295257, 2024-09-16T03:37:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40739979599999998, 2024-09-16T03:37:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.729 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 797 2024-09-16T03:37:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.999730+0000", 2024-09-16T03:37:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.999730+0000", 2024-09-16T03:37:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.999732+0000", 2024-09-16T03:37:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6479999999999999e-06 2024-09-16T03:37:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000380+0000", 2024-09-16T03:37:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064818899999999997 2024-09-16T03:37:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000392+0000", 2024-09-16T03:37:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2757e-05 2024-09-16T03:37:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000399+0000", 2024-09-16T03:37:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.511e-06 2024-09-16T03:37:50.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000462+0000", 2024-09-16T03:37:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2874999999999997e-05 2024-09-16T03:37:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000564+0000", 2024-09-16T03:37:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010215100000000001 2024-09-16T03:37:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000746+0000", 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018165600000000001 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.103052+0000", 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102305858 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205188+0000", 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10213634100000001 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.406992+0000", 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20180416300000001 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.407098+0000", 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010646599999999999 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.407130+0000", 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1180999999999998e-05 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.735 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:798 4.e 4:765e9179:::smithi15736696-44:head [write 1768111~402677] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006162+0000", 2024-09-16T03:37:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.652863334999999, 2024-09-16T03:37:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40110245500000002, 2024-09-16T03:37:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.736 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 798 2024-09-16T03:37:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006162+0000", 2024-09-16T03:37:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006162+0000", 2024-09-16T03:37:50.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006163+0000", 2024-09-16T03:37:50.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4789999999999999e-06 2024-09-16T03:37:50.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006421+0000", 2024-09-16T03:37:50.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025733100000000002 2024-09-16T03:37:50.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006434+0000", 2024-09-16T03:37:50.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2982e-05 2024-09-16T03:37:50.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006440+0000", 2024-09-16T03:37:50.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0839999999999997e-06 2024-09-16T03:37:50.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.103103+0000", 2024-09-16T03:37:50.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096663157999999999 2024-09-16T03:37:50.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.103173+0000", 2024-09-16T03:37:50.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9968999999999994e-05 2024-09-16T03:37:50.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.103271+0000", 2024-09-16T03:37:50.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8428e-05 2024-09-16T03:37:50.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205344+0000", 2024-09-16T03:37:50.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102072367 2024-09-16T03:37:50.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205357+0000", 2024-09-16T03:37:50.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3351e-05 2024-09-16T03:37:50.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.407194+0000", 2024-09-16T03:37:50.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201836714 2024-09-16T03:37:50.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.407231+0000", 2024-09-16T03:37:50.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6931999999999998e-05 2024-09-16T03:37:50.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.407264+0000", 2024-09-16T03:37:50.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.366e-05 2024-09-16T03:37:50.742 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.742 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:799 4.e 4:765e9179:::smithi15736696-44:head [setxattr _header (56) in=7b,truncate 2857157] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.652574584, 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50104159800000003, 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 799 2024-09-16T03:37:50.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:37:50.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:37:50.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006454+0000", 2024-09-16T03:37:50.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5739999999999998e-06 2024-09-16T03:37:50.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006458+0000", 2024-09-16T03:37:50.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6179999999999997e-06 2024-09-16T03:37:50.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006462+0000", 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4999999999999999e-06 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204504+0000", 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19804184999999999 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204702+0000", 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019868199999999999 2024-09-16T03:37:50.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204888+0000", 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185701 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.206549+0000", 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016612790000000001 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507371+0000", 2024-09-16T03:37:50.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30082157399999998 2024-09-16T03:37:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507414+0000", 2024-09-16T03:37:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3053999999999998e-05 2024-09-16T03:37:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507451+0000", 2024-09-16T03:37:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6847999999999997e-05 2024-09-16T03:37:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507492+0000", 2024-09-16T03:37:50.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1267999999999999e-05 2024-09-16T03:37:50.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.749 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.749 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.749 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:800 4.e 4:765e9179:::smithi15736696-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:50.749 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:37:50.749 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.652551248999998, 2024-09-16T03:37:50.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50119258099999997, 2024-09-16T03:37:50.749 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.749 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.750 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:50.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.750 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.750 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 800 2024-09-16T03:37:50.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.750 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:37:50.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1899999999999998e-07 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006478+0000", 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006484+0000", 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6620000000000002e-06 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006488+0000", 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8700000000000002e-06 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205228+0000", 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19873985899999999 2024-09-16T03:37:50.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:50.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205283+0000", 2024-09-16T03:37:50.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5878999999999998e-05 2024-09-16T03:37:50.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507527+0000", 2024-09-16T03:37:50.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302243924 2024-09-16T03:37:50.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507593+0000", 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6061000000000003e-05 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507666+0000", 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3059000000000006e-05 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:843 4.e 4:773b934b:::smithi15736696-9:head [write 524385~553166] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.630592+0000", 2024-09-16T03:37:50.754 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.028433235000001, 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30844299400000003, 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 843 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.630592+0000", 2024-09-16T03:37:50.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.630592+0000", 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.630593+0000", 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0100000000000003e-07 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.630993+0000", 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040056299999999999 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631015+0000", 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1614999999999999e-05 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631025+0000", 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6840000000000008e-06 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734154+0000", 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10312974900000001 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734219+0000", 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4425999999999996e-05 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.734352+0000", 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013289700000000001 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739572+0000", 2024-09-16T03:37:50.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005220616 2024-09-16T03:37:50.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.839754+0000", 2024-09-16T03:37:50.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10018216100000001 2024-09-16T03:37:50.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.938748+0000", 2024-09-16T03:37:50.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098993780000000003 2024-09-16T03:37:50.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.938981+0000", 2024-09-16T03:37:50.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023294799999999999 2024-09-16T03:37:50.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.939035+0000", 2024-09-16T03:37:50.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3653999999999998e-05 2024-09-16T03:37:50.760 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.760 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:844 4.e 4:773b934b:::smithi15736696-9:head [write 1602039~410015] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.631035+0000", 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.027989695999999, 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30812540599999999, 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 844 2024-09-16T03:37:50.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.762 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631035+0000", 2024-09-16T03:37:50.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631035+0000", 2024-09-16T03:37:50.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631036+0000", 2024-09-16T03:37:50.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6300000000000004e-07 2024-09-16T03:37:50.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631729+0000", 2024-09-16T03:37:50.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069264200000000004 2024-09-16T03:37:50.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631740+0000", 2024-09-16T03:37:50.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0713999999999999e-05 2024-09-16T03:37:50.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631745+0000", 2024-09-16T03:37:50.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7729999999999996e-06 2024-09-16T03:37:50.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.736236+0000", 2024-09-16T03:37:50.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104490641 2024-09-16T03:37:50.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.736332+0000", 2024-09-16T03:37:50.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5878000000000006e-05 2024-09-16T03:37:50.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.736536+0000", 2024-09-16T03:37:50.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020407800000000001 2024-09-16T03:37:50.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739577+0000", 2024-09-16T03:37:50.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0030408219999999999 2024-09-16T03:37:50.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.839797+0000", 2024-09-16T03:37:50.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100220029 2024-09-16T03:37:50.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.939096+0000", 2024-09-16T03:37:50.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099299540000000006 2024-09-16T03:37:50.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.939131+0000", 2024-09-16T03:37:50.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4588999999999997e-05 2024-09-16T03:37:50.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.939161+0000", 2024-09-16T03:37:50.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9836999999999999e-05 2024-09-16T03:37:50.767 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.767 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.767 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.767 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:845 4.e 4:773b934b:::smithi15736696-9:head [write 2577441~346944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:50.767 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.631766+0000", 2024-09-16T03:37:50.768 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.027259306000001, 2024-09-16T03:37:50.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40847320799999998, 2024-09-16T03:37:50.768 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.768 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.768 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.768 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.768 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 845 2024-09-16T03:37:50.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.768 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631766+0000", 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631766+0000", 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631767+0000", 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2699999999999999e-07 2024-09-16T03:37:50.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733448+0000", 2024-09-16T03:37:50.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101681228 2024-09-16T03:37:50.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733458+0000", 2024-09-16T03:37:50.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0573e-05 2024-09-16T03:37:50.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733473+0000", 2024-09-16T03:37:50.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4953999999999999e-05 2024-09-16T03:37:50.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.737776+0000", 2024-09-16T03:37:50.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043023940000000002 2024-09-16T03:37:50.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.737868+0000", 2024-09-16T03:37:50.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2223000000000001e-05 2024-09-16T03:37:50.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.738066+0000", 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000197951 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.740101+0000", 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020355379999999999 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.939218+0000", 2024-09-16T03:37:50.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19911701800000001 2024-09-16T03:37:50.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040070+0000", 2024-09-16T03:37:50.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100851658 2024-09-16T03:37:50.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040179+0000", 2024-09-16T03:37:50.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109085 2024-09-16T03:37:50.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040239+0000", 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9858999999999997e-05 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:846 4.e 4:773b934b:::smithi15736696-9:head [setxattr _header (55) in=7b,truncate 2924385] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.733487+0000", 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.925537984999998, 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.306915575, 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.774 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 846 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733487+0000", 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733487+0000", 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733489+0000", 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.491e-06 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733492+0000", 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3850000000000002e-06 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733497+0000", 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2610000000000001e-06 2024-09-16T03:37:50.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733503+0000", 2024-09-16T03:37:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0990000000000004e-06 2024-09-16T03:37:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739098+0000", 2024-09-16T03:37:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005594646 2024-09-16T03:37:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739185+0000", 2024-09-16T03:37:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6910000000000006e-05 2024-09-16T03:37:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:37:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739306+0000", 2024-09-16T03:37:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121348 2024-09-16T03:37:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.740904+0000", 2024-09-16T03:37:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015982170000000001 2024-09-16T03:37:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.939432+0000", 2024-09-16T03:37:50.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.198527603 2024-09-16T03:37:50.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040299+0000", 2024-09-16T03:37:50.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100866576 2024-09-16T03:37:50.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040343+0000", 2024-09-16T03:37:50.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3937999999999998e-05 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040403+0000", 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0100999999999999e-05 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:847 4.e 4:773b934b:::smithi15736696-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.733507+0000", 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.925517687999999, 2024-09-16T03:37:50.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30716211100000002, 2024-09-16T03:37:50.781 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.781 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.781 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:50.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.781 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 847 2024-09-16T03:37:50.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.781 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733507+0000", 2024-09-16T03:37:50.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733507+0000", 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733508+0000", 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733511+0000", 2024-09-16T03:37:50.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.757e-06 2024-09-16T03:37:50.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733516+0000", 2024-09-16T03:37:50.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0780000000000003e-06 2024-09-16T03:37:50.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733519+0000", 2024-09-16T03:37:50.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6160000000000002e-06 2024-09-16T03:37:50.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739510+0000", 2024-09-16T03:37:50.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00599068 2024-09-16T03:37:50.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:50.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.739564+0000", 2024-09-16T03:37:50.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3844000000000002e-05 2024-09-16T03:37:50.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040449+0000", 2024-09-16T03:37:50.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30088547399999999 2024-09-16T03:37:50.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040562+0000", 2024-09-16T03:37:50.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112389 2024-09-16T03:37:50.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040670+0000", 2024-09-16T03:37:50.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010792499999999999 2024-09-16T03:37:50.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.785 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.786 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:983 3.1 e50/31 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head v 50'54, mlcod=49'52)", 2024-09-16T03:37:50.786 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:30.066426+0000", 2024-09-16T03:37:50.786 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.592598962, 2024-09-16T03:37:50.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30487072599999998, 2024-09-16T03:37:50.786 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.786 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.786 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:50.786 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.066426+0000", 2024-09-16T03:37:50.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.066426+0000", 2024-09-16T03:37:50.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.066426+0000", 2024-09-16T03:37:50.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6600000000000003e-07 2024-09-16T03:37:50.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.070698+0000", 2024-09-16T03:37:50.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042713880000000001 2024-09-16T03:37:50.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.070707+0000", 2024-09-16T03:37:50.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3279999999999995e-06 2024-09-16T03:37:50.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.070715+0000", 2024-09-16T03:37:50.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1920000000000005e-06 2024-09-16T03:37:50.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.369583+0000", 2024-09-16T03:37:50.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29886805599999999 2024-09-16T03:37:50.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.369601+0000", 2024-09-16T03:37:50.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7176999999999998e-05 2024-09-16T03:37:50.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.371260+0000", 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001659352 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:30.371297+0000", 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6967e-05 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.790 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:50.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:37:50.793 INFO:tasks.rados.rados.0.smithi157.stdout:895: finishing set_chunk to oid 25 2024-09-16T03:37:50.793 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk returned error code -95 offset: 2174464 length: 84992 2024-09-16T03:37:50.793 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T03:37:50.793 INFO:tasks.rados.rados.0.smithi157.stdout:891: finishing set_chunk to oid 5 2024-09-16T03:37:50.794 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 31 (ObjNum 111 snap 0 seq_num 111) dirty exists 2024-09-16T03:37:50.794 INFO:tasks.rados.rados.0.smithi157.stdout:888: done (15 left) 2024-09-16T03:37:50.794 INFO:tasks.rados.rados.0.smithi157.stdout:889: done (14 left) 2024-09-16T03:37:50.794 INFO:tasks.rados.rados.0.smithi157.stdout:891: done (13 left) 2024-09-16T03:37:50.794 INFO:tasks.rados.rados.0.smithi157.stdout:892: done (12 left) 2024-09-16T03:37:50.794 INFO:tasks.rados.rados.0.smithi157.stdout:893: done (11 left) 2024-09-16T03:37:50.794 INFO:tasks.rados.rados.0.smithi157.stdout:894: done (10 left) 2024-09-16T03:37:50.794 INFO:tasks.rados.rados.0.smithi157.stdout:895: done (9 left) 2024-09-16T03:37:50.794 INFO:tasks.rados.rados.0.smithi157.stdout:896: done (8 left) 2024-09-16T03:37:50.794 INFO:tasks.rados.rados.0.smithi157.stdout:903: done (7 left) 2024-09-16T03:37:50.794 INFO:tasks.rados.rados.0.smithi157.stdout:904: read oid 28 snap -1 2024-09-16T03:37:50.794 INFO:tasks.rados.rados.0.smithi157.stdout:904: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-16T03:37:50.794 INFO:tasks.rados.rados.0.smithi157.stdout:905: read oid 23 snap -1 2024-09-16T03:37:50.795 INFO:tasks.rados.rados.0.smithi157.stdout:905: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T03:37:50.795 INFO:tasks.rados.rados.0.smithi157.stdout:906: tier_flush oid 18 2024-09-16T03:37:50.795 INFO:tasks.rados.rados.0.smithi157.stdout:906: tier_flush oid 18 snap -1 2024-09-16T03:37:50.795 INFO:tasks.rados.rados.0.smithi157.stdout:900: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:50.795 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:50.795 INFO:tasks.rados.rados.0.smithi157.stdout:898: finishing write tid 1 to smithi15736696-42 2024-09-16T03:37:50.795 INFO:tasks.rados.rados.0.smithi157.stdout:898: finishing write tid 2 to smithi15736696-42 2024-09-16T03:37:50.795 INFO:tasks.rados.rados.0.smithi157.stdout:900: done (9 left) 2024-09-16T03:37:50.795 INFO:tasks.rados.rados.0.smithi157.stdout:907: set_chunk oid 41 target oid 2024-09-16T03:37:50.795 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:50.789+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:50.815 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-09-16T03:37:50.813+0000 7fdb04aef640 -1 mon.c@2(leader).osd e65 definitely_dead 0 2024-09-16T03:37:50.892 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:50.891+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:50.943 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:50.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1655 4.12 e65/64)", 2024-09-16T03:37:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.867993+0000", 2024-09-16T03:37:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.073949666999999997, 2024-09-16T03:37:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.073965744, 2024-09-16T03:37:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.867993+0000", 2024-09-16T03:37:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.867993+0000", 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.867994+0000", 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.13e-06 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.868001+0000", 2024-09-16T03:37:50.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1470000000000004e-06 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.868011+0000", 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9150000000000008e-06 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.868016+0000", 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.728e-06 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1655 4.12 e65/64)", 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.905229+0000", 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.036713689000000001, 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036754263000000002, 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.905229+0000", 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.905229+0000", 2024-09-16T03:37:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.905229+0000", 2024-09-16T03:37:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4700000000000001e-07 2024-09-16T03:37:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.905233+0000", 2024-09-16T03:37:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.332e-06 2024-09-16T03:37:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.905239+0000", 2024-09-16T03:37:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1580000000000002e-06 2024-09-16T03:37:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.905245+0000", 2024-09-16T03:37:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.993e-06 2024-09-16T03:37:50.949 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1690 4.a 4:51338165:::smithi15736696-4:head [assert-version v63,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.49165879099999998, 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.491729105, 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.950 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1690 2024-09-16T03:37:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:37:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:37:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450286+0000", 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450293+0000", 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5089999999999996e-06 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450308+0000", 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4955e-05 2024-09-16T03:37:50.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450316+0000", 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7079999999999997e-06 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450386+0000", 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9715999999999995e-05 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450478+0000", 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2610999999999998e-05 2024-09-16T03:37:50.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.665802+0000", 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21532342700000001 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668724+0000", 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029220600000000002 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668758+0000", 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.434e-05 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.954 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1654 4.12 4:4a9d235d:::smithi15736696-1:head [write 652138~461367 in=461367b,stat out=16b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.39990530000000002, 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40002163099999999, 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1654 2024-09-16T03:37:50.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542039+0000", 2024-09-16T03:37:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0250000000000001e-06 2024-09-16T03:37:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542682+0000", 2024-09-16T03:37:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064305099999999995 2024-09-16T03:37:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542698+0000", 2024-09-16T03:37:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6469000000000001e-05 2024-09-16T03:37:50.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542710+0000", 2024-09-16T03:37:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1173e-05 2024-09-16T03:37:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542767+0000", 2024-09-16T03:37:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7584000000000003e-05 2024-09-16T03:37:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:37:50.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542775+0000", 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6149999999999999e-06 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666597+0000", 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123821815 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666745+0000", 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148675 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:37:50.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666935+0000", 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018963899999999999 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1655 4.12 4:4a9d235d:::smithi15736696-1:head [write 1574940~677198 in=677198b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.399213182, 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39936960599999999, 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1655 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542730+0000", 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.085e-06 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543338+0000", 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060826100000000002 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543351+0000", 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2789000000000001e-05 2024-09-16T03:37:50.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543360+0000", 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1970000000000002e-06 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543420+0000", 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9562999999999999e-05 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543431+0000", 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0866999999999999e-05 2024-09-16T03:37:50.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.768963+0000", 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.225532487 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769029+0000", 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5881000000000002e-05 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769134+0000", 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104923 2024-09-16T03:37:50.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1656 4.12 4.bac4b952 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.39856711, 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39876037199999997, 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for peered", 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.965 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1656 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543376+0000", 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4200000000000004e-07 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543383+0000", 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6290000000000004e-06 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543390+0000", 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543396+0000", 2024-09-16T03:37:50.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.02e-06 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543461+0000", 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4876999999999996e-05 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543465+0000", 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2139999999999998e-06 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1657 4.12 4.bac4b952 (undecoded) ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.39853512400000002, 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39875756400000001, 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for peered", 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1657 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543408+0000", 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.13e-07 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543410+0000", 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7030000000000002e-06 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543417+0000", 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.567e-06 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543423+0000", 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0619999999999997e-06 2024-09-16T03:37:50.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543489+0000", 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6315999999999993e-05 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543498+0000", 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7239999999999998e-06 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.972 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1654 4.12 e65/64)", 2024-09-16T03:37:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.669695+0000", 2024-09-16T03:37:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.272247409, 2024-09-16T03:37:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.27249867, 2024-09-16T03:37:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669695+0000", 2024-09-16T03:37:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669695+0000", 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669696+0000", 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.437e-06 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669702+0000", 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.733e-06 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669713+0000", 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0550999999999999e-05 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669719+0000", 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1140000000000002e-06 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1654 4.12 e65/64)", 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.669734+0000", 2024-09-16T03:37:50.975 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.27220783700000001, 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.27247852299999997, 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669734+0000", 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669734+0000", 2024-09-16T03:37:50.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669735+0000", 2024-09-16T03:37:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1030000000000001e-06 2024-09-16T03:37:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669739+0000", 2024-09-16T03:37:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9210000000000002e-06 2024-09-16T03:37:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669750+0000", 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0883999999999999e-05 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669756+0000", 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0159999999999997e-06 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1654 4.12 e65/64)", 2024-09-16T03:37:50.978 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.670075+0000", 2024-09-16T03:37:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.27186672000000001, 2024-09-16T03:37:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.27215672200000002, 2024-09-16T03:37:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.670075+0000", 2024-09-16T03:37:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.670075+0000", 2024-09-16T03:37:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.670076+0000", 2024-09-16T03:37:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7599999999999996e-07 2024-09-16T03:37:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.670079+0000", 2024-09-16T03:37:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5809999999999999e-06 2024-09-16T03:37:50.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.670086+0000", 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0720000000000002e-06 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.670089+0000", 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.478e-06 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1654 4.12 e65/64)", 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.736214+0000", 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.205728563, 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20603785699999999, 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.736214+0000", 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.736214+0000", 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.736215+0000", 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.736218+0000", 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7309999999999999e-06 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.736227+0000", 2024-09-16T03:37:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.772e-06 2024-09-16T03:37:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.736233+0000", 2024-09-16T03:37:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2729999999999999e-06 2024-09-16T03:37:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1690 4.a e65/32)", 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.766756+0000", 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.17518579300000001, 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.175514683, 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.766756+0000", 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.766756+0000", 2024-09-16T03:37:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.766758+0000", 2024-09-16T03:37:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.108e-06 2024-09-16T03:37:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.766761+0000", 2024-09-16T03:37:50.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8890000000000004e-06 2024-09-16T03:37:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.766770+0000", 2024-09-16T03:37:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.081e-06 2024-09-16T03:37:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.866864+0000", 2024-09-16T03:37:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100093917 2024-09-16T03:37:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1654 4.12 e65/64)", 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.767907+0000", 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.174035466, 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.174383863, 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.767907+0000", 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.767907+0000", 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.767908+0000", 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5199999999999995e-07 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.767911+0000", 2024-09-16T03:37:50.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4769999999999999e-06 2024-09-16T03:37:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.767920+0000", 2024-09-16T03:37:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-16T03:37:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.767925+0000", 2024-09-16T03:37:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.87e-06 2024-09-16T03:37:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1655 4.12 e65/64)", 2024-09-16T03:37:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.769992+0000", 2024-09-16T03:37:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.17195060600000001, 2024-09-16T03:37:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.172318374, 2024-09-16T03:37:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769992+0000", 2024-09-16T03:37:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769992+0000", 2024-09-16T03:37:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769993+0000", 2024-09-16T03:37:50.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9600000000000008e-07 2024-09-16T03:37:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769996+0000", 2024-09-16T03:37:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8709999999999999e-06 2024-09-16T03:37:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.770005+0000", 2024-09-16T03:37:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5539999999999998e-06 2024-09-16T03:37:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.770010+0000", 2024-09-16T03:37:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1630000000000004e-06 2024-09-16T03:37:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1655 4.12 e65/64)", 2024-09-16T03:37:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.772067+0000", 2024-09-16T03:37:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.169875159, 2024-09-16T03:37:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.170262304, 2024-09-16T03:37:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.994 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.772067+0000", 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.772067+0000", 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.772068+0000", 2024-09-16T03:37:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9600000000000008e-07 2024-09-16T03:37:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.772071+0000", 2024-09-16T03:37:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3510000000000002e-06 2024-09-16T03:37:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.772080+0000", 2024-09-16T03:37:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9220000000000003e-06 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.772085+0000", 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5480000000000003e-06 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1655 4.12 e65/64)", 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.772931+0000", 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.16901108300000001, 2024-09-16T03:37:50.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.169417452, 2024-09-16T03:37:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:37:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.772931+0000", 2024-09-16T03:37:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.772931+0000", 2024-09-16T03:37:50.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.772932+0000", 2024-09-16T03:37:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8100000000000002e-07 2024-09-16T03:37:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.772935+0000", 2024-09-16T03:37:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9380000000000001e-06 2024-09-16T03:37:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:50.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.772941+0000", 2024-09-16T03:37:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5969999999999997e-06 2024-09-16T03:37:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.772945+0000", 2024-09-16T03:37:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7409999999999998e-06 2024-09-16T03:37:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.000 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:37:51.001 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 16 2024-09-16T03:37:51.001 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:51.002 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:51.000+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:51.002 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:37:51.103 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:51.101+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:51.110 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:51.110 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:51.110 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:51.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:37:51.197 INFO:tasks.rados.rados.0.smithi157.stdout:907: set_chunk oid 41 offset: 2745856 length: 453120 target oid f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99 offset: 0 snap -1 2024-09-16T03:37:51.197 INFO:tasks.rados.rados.0.smithi157.stdout:908: write (excl) oid 47 current snap is 0 2024-09-16T03:37:51.198 INFO:tasks.rados.rados.0.smithi157.stdout:908: seq_num 139 ranges {736173=553965,1867350=468823} 2024-09-16T03:37:51.204 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:51.202+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:51.204 INFO:tasks.rados.rados.0.smithi157.stdout:908: writing smithi15736696-47 from 736173 to 1290138 tid 1 2024-09-16T03:37:51.211 INFO:tasks.rados.rados.0.smithi157.stdout:908: writing smithi15736696-47 from 1867350 to 2336173 tid 2 2024-09-16T03:37:51.211 INFO:tasks.rados.rados.0.smithi157.stdout:909: read oid 3 snap -1 2024-09-16T03:37:51.212 INFO:tasks.rados.rados.0.smithi157.stdout:909: expect (ObjNum 84 snap 0 seq_num 84) 2024-09-16T03:37:51.212 INFO:tasks.rados.rados.0.smithi157.stdout:910: read oid 36 snap -1 2024-09-16T03:37:51.212 INFO:tasks.rados.rados.0.smithi157.stdout:910: expect deleted 2024-09-16T03:37:51.212 INFO:tasks.rados.rados.0.smithi157.stdout:911: delete oid 38 current snap is 0 2024-09-16T03:37:51.212 INFO:tasks.rados.rados.0.smithi157.stdout:899: finishing write tid 1 to smithi15736696-15 2024-09-16T03:37:51.212 INFO:tasks.rados.rados.0.smithi157.stdout:899: finishing write tid 2 to smithi15736696-15 2024-09-16T03:37:51.212 INFO:tasks.rados.rados.0.smithi157.stdout:899: finishing write tid 3 to smithi15736696-15 2024-09-16T03:37:51.212 INFO:tasks.rados.rados.0.smithi157.stdout:898: finishing write tid 3 to smithi15736696-42 2024-09-16T03:37:51.212 INFO:tasks.rados.rados.0.smithi157.stdout:898: finishing write tid 4 to smithi15736696-42 2024-09-16T03:37:51.212 INFO:tasks.rados.rados.0.smithi157.stdout:898: oid 42 updating version 0 to 38 2024-09-16T03:37:51.212 INFO:tasks.rados.rados.0.smithi157.stdout:898: oid 42 version 38 is already newer than 37 2024-09-16T03:37:51.212 INFO:tasks.rados.rados.0.smithi157.stdout:898: oid 42 updating version 38 to 39 2024-09-16T03:37:51.213 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 39 (ObjNum 135 snap 0 seq_num 135) dirty exists 2024-09-16T03:37:51.213 INFO:tasks.rados.rados.0.smithi157.stdout:898: left oid 42 (ObjNum 135 snap 0 seq_num 135) 2024-09-16T03:37:51.213 INFO:tasks.rados.rados.0.smithi157.stdout:899: finishing write tid 4 to smithi15736696-15 2024-09-16T03:37:51.213 INFO:tasks.rados.rados.0.smithi157.stdout:899: oid 15 updating version 0 to 8 2024-09-16T03:37:51.213 INFO:tasks.rados.rados.0.smithi157.stdout:899: oid 15 version 8 is already newer than 7 2024-09-16T03:37:51.213 INFO:tasks.rados.rados.0.smithi157.stdout:899: oid 15 version 8 is already newer than 6 2024-09-16T03:37:51.213 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 8 (ObjNum 136 snap 0 seq_num 136) dirty exists 2024-09-16T03:37:51.213 INFO:tasks.rados.rados.0.smithi157.stdout:899: left oid 15 (ObjNum 136 snap 0 seq_num 136) 2024-09-16T03:37:51.213 INFO:tasks.rados.rados.0.smithi157.stdout:902: finishing write tid 1 to smithi15736696-37 2024-09-16T03:37:51.213 INFO:tasks.rados.rados.0.smithi157.stdout:902: finishing write tid 2 to smithi15736696-37 2024-09-16T03:37:51.213 INFO:tasks.rados.rados.0.smithi157.stdout:897: finishing set_chunk to oid 30 2024-09-16T03:37:51.213 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 29 (ObjNum 98 snap 0 seq_num 98) dirty exists 2024-09-16T03:37:51.213 INFO:tasks.rados.rados.0.smithi157.stdout:902: finishing write tid 3 to smithi15736696-37 2024-09-16T03:37:51.214 INFO:tasks.rados.rados.0.smithi157.stdout:902: finishing write tid 4 to smithi15736696-37 2024-09-16T03:37:51.214 INFO:tasks.rados.rados.0.smithi157.stdout:902: oid 37 updating version 0 to 35 2024-09-16T03:37:51.214 INFO:tasks.rados.rados.0.smithi157.stdout:902: oid 37 version 35 is already newer than 33 2024-09-16T03:37:51.214 INFO:tasks.rados.rados.0.smithi157.stdout:902: oid 37 version 35 is already newer than 34 2024-09-16T03:37:51.214 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 35 (ObjNum 137 snap 0 seq_num 137) dirty exists 2024-09-16T03:37:51.214 INFO:tasks.rados.rados.0.smithi157.stdout:902: left oid 37 (ObjNum 137 snap 0 seq_num 137) 2024-09-16T03:37:51.214 INFO:tasks.rados.rados.0.smithi157.stdout:904: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-16T03:37:51.268 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:51.268 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4338.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.168535632999998, 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30502727499999999, 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4338", 2024-09-16T03:37:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3073758685", 2024-09-16T03:37:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-16T03:37:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:37:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:37:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097101+0000", 2024-09-16T03:37:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.915e-06 2024-09-16T03:37:51.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097111+0000", 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0589e-05 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097130+0000", 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8958999999999999e-05 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097141+0000", 2024-09-16T03:37:51.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0616e-05 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097227+0000", 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6322000000000005e-05 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097232+0000", 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2059999999999998e-06 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200235+0000", 2024-09-16T03:37:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103002945 2024-09-16T03:37:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200318+0000", 2024-09-16T03:37:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2515999999999994e-05 2024-09-16T03:37:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,10", 2024-09-16T03:37:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200429+0000", 2024-09-16T03:37:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011158999999999999 2024-09-16T03:37:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201478+0000", 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00104893 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201942+0000", 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046354700000000003 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402048+0000", 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20010616000000001 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402089+0000", 2024-09-16T03:37:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0762e-05 2024-09-16T03:37:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402125+0000", 2024-09-16T03:37:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6217999999999998e-05 2024-09-16T03:37:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:15 4.a 4:51338165:::smithi15736696-4:head [setxattr _header (55) in=7b,truncate 2050421] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:37:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.268663699999998, 2024-09-16T03:37:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30116835400000003, 2024-09-16T03:37:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 15 2024-09-16T03:37:51.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.278 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:37:51.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:37:51.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996971+0000", 2024-09-16T03:37:51.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.066e-06 2024-09-16T03:37:51.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996984+0000", 2024-09-16T03:37:51.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3246e-05 2024-09-16T03:37:51.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996994+0000", 2024-09-16T03:37:51.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.713e-06 2024-09-16T03:37:51.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097070+0000", 2024-09-16T03:37:51.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100076313 2024-09-16T03:37:51.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097141+0000", 2024-09-16T03:37:51.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1298999999999996e-05 2024-09-16T03:37:51.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097205+0000", 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3907999999999999e-05 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097300+0000", 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5209000000000003e-05 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197645+0000", 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10034422899999999 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197837+0000", 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000192454 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298049+0000", 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100211868 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298082+0000", 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3467000000000003e-05 2024-09-16T03:37:51.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298138+0000", 2024-09-16T03:37:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5581999999999997e-05 2024-09-16T03:37:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:73 4.a 4:532814f2:::smithi15736696-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:37:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.954373496999999, 2024-09-16T03:37:51.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.362642662, 2024-09-16T03:37:51.284 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.284 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.284 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.284 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.284 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.284 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.284 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73 2024-09-16T03:37:51.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.284 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:37:51.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:37:51.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311261+0000", 2024-09-16T03:37:51.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.562e-06 2024-09-16T03:37:51.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311925+0000", 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066385499999999998 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311937+0000", 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1881e-05 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311943+0000", 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6999999999999996e-06 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311988+0000", 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5358000000000002e-05 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312070+0000", 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1735000000000003e-05 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312167+0000", 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6685e-05 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.313661+0000", 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001494324 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.414759+0000", 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10109850300000001 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673821+0000", 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.25906127400000001 2024-09-16T03:37:51.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673858+0000", 2024-09-16T03:37:51.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7620999999999998e-05 2024-09-16T03:37:51.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673902+0000", 2024-09-16T03:37:51.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4163999999999999e-05 2024-09-16T03:37:51.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.293 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:303 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:51.293 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:37:51.293 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.853914666999998, 2024-09-16T03:37:51.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50956986000000004, 2024-09-16T03:37:51.293 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.293 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 303 2024-09-16T03:37:51.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:37:51.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:37:51.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411721+0000", 2024-09-16T03:37:51.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0200000000000001e-06 2024-09-16T03:37:51.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411729+0000", 2024-09-16T03:37:51.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1240000000000005e-06 2024-09-16T03:37:51.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411744+0000", 2024-09-16T03:37:51.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099000000000001e-05 2024-09-16T03:37:51.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411751+0000", 2024-09-16T03:37:51.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.029e-06 2024-09-16T03:37:51.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511892+0000", 2024-09-16T03:37:51.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100140879 2024-09-16T03:37:51.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511961+0000", 2024-09-16T03:37:51.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9379000000000001e-05 2024-09-16T03:37:51.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:51.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.613647+0000", 2024-09-16T03:37:51.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101685994 2024-09-16T03:37:51.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.614294+0000", 2024-09-16T03:37:51.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064692799999999996 2024-09-16T03:37:51.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.714962+0000", 2024-09-16T03:37:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066765 2024-09-16T03:37:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921157+0000", 2024-09-16T03:37:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206195463 2024-09-16T03:37:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921233+0000", 2024-09-16T03:37:51.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5446999999999998e-05 2024-09-16T03:37:51.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921288+0000", 2024-09-16T03:37:51.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5847999999999998e-05 2024-09-16T03:37:51.299 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.299 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.299 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.299 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:404 4.3 e45/32 4:cf22405f:::smithi15736696-5:head v 45'15, mlcod=42'13)", 2024-09-16T03:37:51.299 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.309931+0000", 2024-09-16T03:37:51.300 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.955702080000002, 2024-09-16T03:37:51.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30105416800000001, 2024-09-16T03:37:51.300 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.300 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.300 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.300 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309931+0000", 2024-09-16T03:37:51.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309931+0000", 2024-09-16T03:37:51.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309933+0000", 2024-09-16T03:37:51.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0099999999999998e-06 2024-09-16T03:37:51.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309943+0000", 2024-09-16T03:37:51.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4050000000000006e-06 2024-09-16T03:37:51.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309959+0000", 2024-09-16T03:37:51.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6365e-05 2024-09-16T03:37:51.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.410036+0000", 2024-09-16T03:37:51.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100077477 2024-09-16T03:37:51.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610252+0000", 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20021509700000001 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610290+0000", 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8068999999999998e-05 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610947+0000", 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065726700000000001 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610985+0000", 2024-09-16T03:37:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8478e-05 2024-09-16T03:37:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:503 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:37:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.443840115, 2024-09-16T03:37:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60521621599999997, 2024-09-16T03:37:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.304 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.305 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.305 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.305 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.305 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.305 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 503 2024-09-16T03:37:51.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.305 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:37:51.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:37:51.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821794+0000", 2024-09-16T03:37:51.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-09-16T03:37:51.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821798+0000", 2024-09-16T03:37:51.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2030000000000002e-06 2024-09-16T03:37:51.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821804+0000", 2024-09-16T03:37:51.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:37:51.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821810+0000", 2024-09-16T03:37:51.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4249999999999997e-06 2024-09-16T03:37:51.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922212+0000", 2024-09-16T03:37:51.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040228800000001 2024-09-16T03:37:51.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922307+0000", 2024-09-16T03:37:51.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5089999999999999e-05 2024-09-16T03:37:51.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:51.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.126007+0000", 2024-09-16T03:37:51.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20369931499999999 2024-09-16T03:37:51.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.127605+0000", 2024-09-16T03:37:51.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001598213 2024-09-16T03:37:51.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.327263+0000", 2024-09-16T03:37:51.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199657905 2024-09-16T03:37:51.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426900+0000", 2024-09-16T03:37:51.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099637217 2024-09-16T03:37:51.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426963+0000", 2024-09-16T03:37:51.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:37:51.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.427009+0000", 2024-09-16T03:37:51.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6533000000000003e-05 2024-09-16T03:37:51.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.311 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.312 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.312 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:850 4.a 4:503187b3:::smithi15736696-34:head [write 2888511~52681] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:51.312 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.562268+0000", 2024-09-16T03:37:51.312 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.703364809, 2024-09-16T03:37:51.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29934466599999998, 2024-09-16T03:37:51.312 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.312 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.312 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.312 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.313 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.313 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.313 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 850 2024-09-16T03:37:51.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.313 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562268+0000", 2024-09-16T03:37:51.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562268+0000", 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562274+0000", 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3460000000000001e-06 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562334+0000", 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9993e-05 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562346+0000", 2024-09-16T03:37:51.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2048999999999999e-05 2024-09-16T03:37:51.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562351+0000", 2024-09-16T03:37:51.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6879999999999998e-06 2024-09-16T03:37:51.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.657867+0000", 2024-09-16T03:37:51.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095516648999999995 2024-09-16T03:37:51.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.657964+0000", 2024-09-16T03:37:51.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6646999999999998e-05 2024-09-16T03:37:51.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:51.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658121+0000", 2024-09-16T03:37:51.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015688 2024-09-16T03:37:51.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759327+0000", 2024-09-16T03:37:51.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101206726 2024-09-16T03:37:51.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759490+0000", 2024-09-16T03:37:51.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016302600000000001 2024-09-16T03:37:51.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861466+0000", 2024-09-16T03:37:51.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101975808 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861543+0000", 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6316000000000006e-05 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861613+0000", 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0537999999999994e-05 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:851 4.a 4:503187b3:::smithi15736696-34:head [setxattr _header (56) in=7b,truncate 2941192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.562360+0000", 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.703273679999999, 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29963510700000001, 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 851 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.319 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562360+0000", 2024-09-16T03:37:51.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562360+0000", 2024-09-16T03:37:51.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562360+0000", 2024-09-16T03:37:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-16T03:37:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562363+0000", 2024-09-16T03:37:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8100000000000002e-06 2024-09-16T03:37:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562367+0000", 2024-09-16T03:37:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.873e-06 2024-09-16T03:37:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562370+0000", 2024-09-16T03:37:51.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1820000000000002e-06 2024-09-16T03:37:51.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658694+0000", 2024-09-16T03:37:51.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096324009000000002 2024-09-16T03:37:51.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658785+0000", 2024-09-16T03:37:51.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0370000000000004e-05 2024-09-16T03:37:51.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:51.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.658928+0000", 2024-09-16T03:37:51.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014321099999999999 2024-09-16T03:37:51.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759398+0000", 2024-09-16T03:37:51.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10046983900000001 2024-09-16T03:37:51.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759493+0000", 2024-09-16T03:37:51.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5639999999999999e-05 2024-09-16T03:37:51.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861917+0000", 2024-09-16T03:37:51.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102423575 2024-09-16T03:37:51.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861953+0000", 2024-09-16T03:37:51.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6004999999999998e-05 2024-09-16T03:37:51.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.861995+0000", 2024-09-16T03:37:51.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1848000000000003e-05 2024-09-16T03:37:51.326 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.326 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.326 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.326 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:852 4.a 4:503187b3:::smithi15736696-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:51.326 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.562374+0000", 2024-09-16T03:37:51.326 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.703259022000001, 2024-09-16T03:37:51.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29976180800000002, 2024-09-16T03:37:51.327 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.327 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.327 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:51.327 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.327 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.327 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.327 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 852 2024-09-16T03:37:51.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.327 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562374+0000", 2024-09-16T03:37:51.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562374+0000", 2024-09-16T03:37:51.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562375+0000", 2024-09-16T03:37:51.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:37:51.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562376+0000", 2024-09-16T03:37:51.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4360000000000001e-06 2024-09-16T03:37:51.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562379+0000", 2024-09-16T03:37:51.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3610000000000001e-06 2024-09-16T03:37:51.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.662476+0000", 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10009655100000001 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759421+0000", 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096945552000000004 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759482+0000", 2024-09-16T03:37:51.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0058e-05 2024-09-16T03:37:51.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.862023+0000", 2024-09-16T03:37:51.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10254131599999999 2024-09-16T03:37:51.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.862072+0000", 2024-09-16T03:37:51.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9104999999999998e-05 2024-09-16T03:37:51.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.862136+0000", 2024-09-16T03:37:51.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4098999999999997e-05 2024-09-16T03:37:51.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.332 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.332 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.333 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1194 4.a 4:503187b3:::smithi15736696-34:head [write 2736097~502305] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:37:51.333 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.066371+0000", 2024-09-16T03:37:51.333 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.199261983, 2024-09-16T03:37:51.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30263757600000002, 2024-09-16T03:37:51.333 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.333 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.333 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.334 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1194 2024-09-16T03:37:51.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.334 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066371+0000", 2024-09-16T03:37:51.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066371+0000", 2024-09-16T03:37:51.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066373+0000", 2024-09-16T03:37:51.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.216e-06 2024-09-16T03:37:51.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066794+0000", 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042050699999999998 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066812+0000", 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8462999999999999e-05 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166903+0000", 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100090889 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166970+0000", 2024-09-16T03:37:51.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6148000000000004e-05 2024-09-16T03:37:51.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.167075+0000", 2024-09-16T03:37:51.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010519200000000001 2024-09-16T03:37:51.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:51.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.167251+0000", 2024-09-16T03:37:51.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176405 2024-09-16T03:37:51.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.169014+0000", 2024-09-16T03:37:51.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001762516 2024-09-16T03:37:51.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.264039+0000", 2024-09-16T03:37:51.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095025709 2024-09-16T03:37:51.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.368934+0000", 2024-09-16T03:37:51.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104894738 2024-09-16T03:37:51.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.368975+0000", 2024-09-16T03:37:51.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0605999999999997e-05 2024-09-16T03:37:51.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.369009+0000", 2024-09-16T03:37:51.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4187000000000003e-05 2024-09-16T03:37:51.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.340 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.340 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.340 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1195 4.a 4:503187b3:::smithi15736696-34:head [setxattr _header (56) in=7b,truncate 3378174] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:37:51.340 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.166919+0000", 2024-09-16T03:37:51.340 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.098714699, 2024-09-16T03:37:51.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.26348176800000001, 2024-09-16T03:37:51.340 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.340 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.340 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.341 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.341 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1195 2024-09-16T03:37:51.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.341 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166919+0000", 2024-09-16T03:37:51.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166919+0000", 2024-09-16T03:37:51.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166920+0000", 2024-09-16T03:37:51.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.66e-06 2024-09-16T03:37:51.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166925+0000", 2024-09-16T03:37:51.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0429999999999998e-06 2024-09-16T03:37:51.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166932+0000", 2024-09-16T03:37:51.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1060000000000002e-06 2024-09-16T03:37:51.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.267067+0000", 2024-09-16T03:37:51.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100135059 2024-09-16T03:37:51.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.267184+0000", 2024-09-16T03:37:51.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011629399999999999 2024-09-16T03:37:51.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.267291+0000", 2024-09-16T03:37:51.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107719 2024-09-16T03:37:51.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:51.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.267463+0000", 2024-09-16T03:37:51.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000171962 2024-09-16T03:37:51.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.268123+0000", 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065973599999999996 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.268655+0000", 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00053218799999999995 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.430240+0000", 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.16158494900000001 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.430311+0000", 2024-09-16T03:37:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0706999999999999e-05 2024-09-16T03:37:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.430400+0000", 2024-09-16T03:37:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9345000000000006e-05 2024-09-16T03:37:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1227 4.a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v18,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.589663850999999, 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.44160531400000003, 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1227 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:37:51.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:37:51.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675972+0000", 2024-09-16T03:37:51.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7269999999999999e-06 2024-09-16T03:37:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675981+0000", 2024-09-16T03:37:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.166e-06 2024-09-16T03:37:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675996+0000", 2024-09-16T03:37:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4901e-05 2024-09-16T03:37:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676005+0000", 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4579999999999993e-06 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676081+0000", 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6157999999999998e-05 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676188+0000", 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106871 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.016594+0000", 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34040622300000001 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017089+0000", 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049523899999999997 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017779+0000", 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069032200000000005 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117504+0000", 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099724207999999995 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117543+0000", 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9208999999999997e-05 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117575+0000", 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1832000000000003e-05 2024-09-16T03:37:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1523 4.a 4:50107449:::smithi15736696-27:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.902622525, 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.410522742, 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1523 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363012+0000", 2024-09-16T03:37:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.269e-06 2024-09-16T03:37:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363016+0000", 2024-09-16T03:37:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8920000000000002e-06 2024-09-16T03:37:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363027+0000", 2024-09-16T03:37:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1079e-05 2024-09-16T03:37:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463160+0000", 2024-09-16T03:37:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10013267100000001 2024-09-16T03:37:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463297+0000", 2024-09-16T03:37:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013717699999999999 2024-09-16T03:37:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463381+0000", 2024-09-16T03:37:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3701000000000004e-05 2024-09-16T03:37:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.773533+0000", 2024-09-16T03:37:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31015295300000001 2024-09-16T03:37:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1677 4.13 e64/50 4:c8492bfc:::smithi15736696-42:head v 64'37, mlcod=60'35)", 2024-09-16T03:37:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.136995+0000", 2024-09-16T03:37:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.1286382340000001, 2024-09-16T03:37:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30289115700000002, 2024-09-16T03:37:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.136995+0000", 2024-09-16T03:37:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.136995+0000", 2024-09-16T03:37:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.136997+0000", 2024-09-16T03:37:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2450000000000001e-06 2024-09-16T03:37:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.137887+0000", 2024-09-16T03:37:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088958899999999998 2024-09-16T03:37:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.137904+0000", 2024-09-16T03:37:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6843000000000001e-05 2024-09-16T03:37:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.237997+0000", 2024-09-16T03:37:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100093631 2024-09-16T03:37:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.338117+0000", 2024-09-16T03:37:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100120158 2024-09-16T03:37:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.338133+0000", 2024-09-16T03:37:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5624000000000002e-05 2024-09-16T03:37:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.439797+0000", 2024-09-16T03:37:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101664168 2024-09-16T03:37:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.439886+0000", 2024-09-16T03:37:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8899000000000004e-05 2024-09-16T03:37:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1690 4.a 4:51338165:::smithi15736696-4:head [assert-version v63,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:37:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:37:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.81534985800000004, 2024-09-16T03:37:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52002349699999995, 2024-09-16T03:37:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1690 2024-09-16T03:37:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:37:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:37:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450286+0000", 2024-09-16T03:37:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-16T03:37:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450293+0000", 2024-09-16T03:37:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5089999999999996e-06 2024-09-16T03:37:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450308+0000", 2024-09-16T03:37:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4955e-05 2024-09-16T03:37:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450316+0000", 2024-09-16T03:37:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7079999999999997e-06 2024-09-16T03:37:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450386+0000", 2024-09-16T03:37:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9715999999999995e-05 2024-09-16T03:37:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450478+0000", 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2610999999999998e-05 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.665802+0000", 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21532342700000001 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668724+0000", 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029220600000000002 2024-09-16T03:37:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668758+0000", 2024-09-16T03:37:51.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.434e-05 2024-09-16T03:37:51.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970243+0000", 2024-09-16T03:37:51.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30148501 2024-09-16T03:37:51.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970278+0000", 2024-09-16T03:37:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4965000000000002e-05 2024-09-16T03:37:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970307+0000", 2024-09-16T03:37:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8968000000000001e-05 2024-09-16T03:37:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1654 4.12 4:4a9d235d:::smithi15736696-1:head [write 652138~461367 in=461367b,stat out=16b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:37:51.369 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.72359636699999996, 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42790198699999998, 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1654 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:37:51.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:37:51.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542039+0000", 2024-09-16T03:37:51.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0250000000000001e-06 2024-09-16T03:37:51.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542682+0000", 2024-09-16T03:37:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064305099999999995 2024-09-16T03:37:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542698+0000", 2024-09-16T03:37:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6469000000000001e-05 2024-09-16T03:37:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542710+0000", 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1173e-05 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542767+0000", 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7584000000000003e-05 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542775+0000", 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6149999999999999e-06 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666597+0000", 2024-09-16T03:37:51.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123821815 2024-09-16T03:37:51.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666745+0000", 2024-09-16T03:37:51.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148675 2024-09-16T03:37:51.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:37:51.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666935+0000", 2024-09-16T03:37:51.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018963899999999999 2024-09-16T03:37:51.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969207+0000", 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302272031 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969374+0000", 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016686699999999999 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969533+0000", 2024-09-16T03:37:51.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015909299999999999 2024-09-16T03:37:51.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969560+0000", 2024-09-16T03:37:51.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7148000000000002e-05 2024-09-16T03:37:51.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969631+0000", 2024-09-16T03:37:51.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1174000000000007e-05 2024-09-16T03:37:51.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969685+0000", 2024-09-16T03:37:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3613000000000003e-05 2024-09-16T03:37:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969764+0000", 2024-09-16T03:37:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9281000000000002e-05 2024-09-16T03:37:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969939+0000", 2024-09-16T03:37:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473400000000001 2024-09-16T03:37:51.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.378 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.378 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1655 4.12 4:4a9d235d:::smithi15736696-1:head [write 1574940~677198 in=677198b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:37:51.378 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:37:51.378 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.72290424900000005, 2024-09-16T03:37:51.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42773147700000003, 2024-09-16T03:37:51.378 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.378 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.378 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.379 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1655 2024-09-16T03:37:51.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.379 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:37:51.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:37:51.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542730+0000", 2024-09-16T03:37:51.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.085e-06 2024-09-16T03:37:51.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543338+0000", 2024-09-16T03:37:51.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060826100000000002 2024-09-16T03:37:51.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543351+0000", 2024-09-16T03:37:51.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2789000000000001e-05 2024-09-16T03:37:51.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543360+0000", 2024-09-16T03:37:51.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1970000000000002e-06 2024-09-16T03:37:51.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543420+0000", 2024-09-16T03:37:51.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9562999999999999e-05 2024-09-16T03:37:51.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:37:51.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543431+0000", 2024-09-16T03:37:51.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0866999999999999e-05 2024-09-16T03:37:51.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.768963+0000", 2024-09-16T03:37:51.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.225532487 2024-09-16T03:37:51.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769029+0000", 2024-09-16T03:37:51.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5881000000000002e-05 2024-09-16T03:37:51.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:37:51.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769134+0000", 2024-09-16T03:37:51.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104923 2024-09-16T03:37:51.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969639+0000", 2024-09-16T03:37:51.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20050517900000001 2024-09-16T03:37:51.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969978+0000", 2024-09-16T03:37:51.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033848599999999998 2024-09-16T03:37:51.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970206+0000", 2024-09-16T03:37:51.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022867300000000001 2024-09-16T03:37:51.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970248+0000", 2024-09-16T03:37:51.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1433999999999997e-05 2024-09-16T03:37:51.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970299+0000", 2024-09-16T03:37:51.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0677999999999998e-05 2024-09-16T03:37:51.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970338+0000", 2024-09-16T03:37:51.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9646e-05 2024-09-16T03:37:51.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970377+0000", 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8439999999999998e-05 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970460+0000", 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3887999999999994e-05 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.386 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1654 4.12 e65/64)", 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.669695+0000", 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.59593847600000005, 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29955032799999998, 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669695+0000", 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669695+0000", 2024-09-16T03:37:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669696+0000", 2024-09-16T03:37:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.437e-06 2024-09-16T03:37:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669702+0000", 2024-09-16T03:37:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.733e-06 2024-09-16T03:37:51.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669713+0000", 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0550999999999999e-05 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669719+0000", 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1140000000000002e-06 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969147+0000", 2024-09-16T03:37:51.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.299428372 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969182+0000", 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4839000000000003e-05 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969245+0000", 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3281999999999995e-05 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1654 4.12 e65/64)", 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.669734+0000", 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.59589890400000001, 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29967232300000002, 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669734+0000", 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669734+0000", 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669735+0000", 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1030000000000001e-06 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669739+0000", 2024-09-16T03:37:51.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9210000000000002e-06 2024-09-16T03:37:51.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669750+0000", 2024-09-16T03:37:51.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0883999999999999e-05 2024-09-16T03:37:51.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.669756+0000", 2024-09-16T03:37:51.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0159999999999997e-06 2024-09-16T03:37:51.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969334+0000", 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29957776400000002 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969356+0000", 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2229e-05 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969407+0000", 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0405999999999998e-05 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.394 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.395 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1654 4.12 e65/64)", 2024-09-16T03:37:51.395 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.670075+0000", 2024-09-16T03:37:51.395 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.59555778699999995, 2024-09-16T03:37:51.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.299486262, 2024-09-16T03:37:51.395 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.395 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.395 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:51.395 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.670075+0000", 2024-09-16T03:37:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.670075+0000", 2024-09-16T03:37:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.670076+0000", 2024-09-16T03:37:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7599999999999996e-07 2024-09-16T03:37:51.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.670079+0000", 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5809999999999999e-06 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.670086+0000", 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0720000000000002e-06 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.670089+0000", 2024-09-16T03:37:51.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.478e-06 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969504+0000", 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.299414501 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969522+0000", 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7966e-05 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969562+0000", 2024-09-16T03:37:51.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9888000000000002e-05 2024-09-16T03:37:51.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.399 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.399 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.399 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:51.400 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2682432 2024-09-16T03:37:51.401 INFO:tasks.rados.rados.0.smithi157.stdout:905: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T03:37:51.401 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:51.398+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:51.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:37:51.407 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1800327 2024-09-16T03:37:51.423 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1959160 2024-09-16T03:37:51.434 INFO:tasks.rados.rados.0.smithi157.stdout:890: finishing write tid 1 to smithi15736696-1 2024-09-16T03:37:51.434 INFO:tasks.rados.rados.0.smithi157.stdout:901: finishing set_chunk to oid 4 2024-09-16T03:37:51.434 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 63 (ObjNum 130 snap 0 seq_num 130) dirty exists 2024-09-16T03:37:51.434 INFO:tasks.rados.rados.0.smithi157.stdout:890: finishing write tid 2 to smithi15736696-1 2024-09-16T03:37:51.434 INFO:tasks.rados.rados.0.smithi157.stdout:906: got (0) Success 2024-09-16T03:37:51.434 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 0 (ObjNum 74 snap 0 seq_num 74) dirty exists 2024-09-16T03:37:51.435 INFO:tasks.rados.rados.0.smithi157.stdout:897: done (13 left) 2024-09-16T03:37:51.435 INFO:tasks.rados.rados.0.smithi157.stdout:898: done (12 left) 2024-09-16T03:37:51.435 INFO:tasks.rados.rados.0.smithi157.stdout:899: done (11 left) 2024-09-16T03:37:51.435 INFO:tasks.rados.rados.0.smithi157.stdout:901: done (10 left) 2024-09-16T03:37:51.435 INFO:tasks.rados.rados.0.smithi157.stdout:902: done (9 left) 2024-09-16T03:37:51.435 INFO:tasks.rados.rados.0.smithi157.stdout:904: done (8 left) 2024-09-16T03:37:51.435 INFO:tasks.rados.rados.0.smithi157.stdout:905: done (7 left) 2024-09-16T03:37:51.435 INFO:tasks.rados.rados.0.smithi157.stdout:906: done (6 left) 2024-09-16T03:37:51.435 INFO:tasks.rados.rados.0.smithi157.stdout:911: done (5 left) 2024-09-16T03:37:51.435 INFO:tasks.rados.rados.0.smithi157.stdout:912: read oid 45 snap -1 2024-09-16T03:37:51.435 INFO:tasks.rados.rados.0.smithi157.stdout:912: expect (ObjNum 85 snap 0 seq_num 85) 2024-09-16T03:37:51.435 INFO:tasks.rados.rados.0.smithi157.stdout:908: finishing write tid 1 to smithi15736696-47 2024-09-16T03:37:51.435 INFO:tasks.rados.rados.0.smithi157.stdout:909: expect (ObjNum 84 snap 0 seq_num 84) 2024-09-16T03:37:51.440 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:37:51.440 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:51.438+0000 7ff26836a640 -1 osd.11 65 handle_osd_map injecting map crc failure 2024-09-16T03:37:51.442 INFO:teuthology.orchestra.run.smithi064.stderr:marked out osd.1. 2024-09-16T03:37:51.456 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-16T03:37:51.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-09-16T03:37:51.500 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:51.501 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:51.499+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:51.542 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:51.540+0000 7efeef6d5640 -1 osd.13 65 handle_osd_map injecting map crc failure 2024-09-16T03:37:51.557 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:51.557 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:51.557 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:37:51.557 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:51.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:37:51.602 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:51.600+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:51.640 INFO:tasks.rados.rados.0.smithi157.stdout:890: finishing write tid 3 to smithi15736696-1 2024-09-16T03:37:51.640 INFO:tasks.rados.rados.0.smithi157.stdout:890: finishing write tid 4 to smithi15736696-1 2024-09-16T03:37:51.640 INFO:tasks.rados.rados.0.smithi157.stdout:890: oid 1 updating version 0 to 12 2024-09-16T03:37:51.640 INFO:tasks.rados.rados.0.smithi157.stdout:890: oid 1 version 12 is already newer than 10 2024-09-16T03:37:51.640 INFO:tasks.rados.rados.0.smithi157.stdout:890: oid 1 version 12 is already newer than 11 2024-09-16T03:37:51.640 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 12 (ObjNum 134 snap 0 seq_num 134) dirty exists 2024-09-16T03:37:51.640 INFO:tasks.rados.rados.0.smithi157.stdout:890: left oid 1 (ObjNum 134 snap 0 seq_num 134) 2024-09-16T03:37:51.640 INFO:tasks.rados.rados.0.smithi157.stdout:908: finishing write tid 2 to smithi15736696-47 2024-09-16T03:37:51.640 INFO:tasks.rados.rados.0.smithi157.stdout:913: read oid 28 snap -1 2024-09-16T03:37:51.640 INFO:tasks.rados.rados.0.smithi157.stdout:913: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-16T03:37:51.641 INFO:tasks.rados.rados.0.smithi157.stdout:890: done (6 left) 2024-09-16T03:37:51.641 INFO:tasks.rados.rados.0.smithi157.stdout:909: done (5 left) 2024-09-16T03:37:51.641 INFO:tasks.rados.rados.0.smithi157.stdout:910: done (4 left) 2024-09-16T03:37:51.641 INFO:tasks.rados.rados.0.smithi157.stdout:914: write (excl) oid 18 current snap is 0 2024-09-16T03:37:51.641 INFO:tasks.rados.rados.0.smithi157.stdout:914: seq_num 141 ranges {769787=704463,2267961=101826} 2024-09-16T03:37:51.647 INFO:tasks.rados.rados.0.smithi157.stdout:914: writing smithi15736696-18 from 769787 to 1474250 tid 1 2024-09-16T03:37:51.653 INFO:tasks.rados.rados.0.smithi157.stdout:914: writing smithi15736696-18 from 2267961 to 2369787 tid 2 2024-09-16T03:37:51.653 INFO:tasks.rados.rados.0.smithi157.stdout:915: read oid 50 snap -1 2024-09-16T03:37:51.653 INFO:tasks.rados.rados.0.smithi157.stdout:915: expect (ObjNum 104 snap 0 seq_num 104) 2024-09-16T03:37:51.654 INFO:tasks.rados.rados.0.smithi157.stdout:916: write (excl) oid 42 current snap is 0 2024-09-16T03:37:51.654 INFO:tasks.rados.rados.0.smithi157.stdout:908: finishing write tid 3 to smithi15736696-47 2024-09-16T03:37:51.654 INFO:tasks.rados.rados.0.smithi157.stdout:908: finishing write tid 4 to smithi15736696-47 2024-09-16T03:37:51.654 INFO:tasks.rados.rados.0.smithi157.stdout:908: oid 47 updating version 0 to 54 2024-09-16T03:37:51.654 INFO:tasks.rados.rados.0.smithi157.stdout:908: oid 47 updating version 54 to 56 2024-09-16T03:37:51.654 INFO:tasks.rados.rados.0.smithi157.stdout:908: oid 47 version 56 is already newer than 55 2024-09-16T03:37:51.654 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 56 (ObjNum 139 snap 0 seq_num 139) dirty exists 2024-09-16T03:37:51.654 INFO:tasks.rados.rados.0.smithi157.stdout:908: left oid 47 (ObjNum 139 snap 0 seq_num 139) 2024-09-16T03:37:51.654 INFO:tasks.rados.rados.0.smithi157.stdout:912: expect (ObjNum 85 snap 0 seq_num 85) 2024-09-16T03:37:51.654 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:51.675 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:37:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:37:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:37:51.676 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:51.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:37:51.703 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:51.700+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:51.796 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:37:51.796 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:37:51.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:37:51.796 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:37:51.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:57 4.d 4:b3fe05b2:::smithi15736696-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:37:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.668540213, 2024-09-16T03:37:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.205477148, 2024-09-16T03:37:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 57 2024-09-16T03:37:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.797 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:37:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:37:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124125+0000", 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.604e-06 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124908+0000", 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000783086 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124919+0000", 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0591000000000001e-05 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124925+0000", 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.55e-06 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124985+0000", 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9817e-05 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.125113+0000", 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127429 2024-09-16T03:37:51.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:37:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.125229+0000", 2024-09-16T03:37:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116784 2024-09-16T03:37:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.127169+0000", 2024-09-16T03:37:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001939187 2024-09-16T03:37:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128052+0000", 2024-09-16T03:37:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088291800000000001 2024-09-16T03:37:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329495+0000", 2024-09-16T03:37:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201443328 2024-09-16T03:37:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329535+0000", 2024-09-16T03:37:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9823999999999997e-05 2024-09-16T03:37:51.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329601+0000", 2024-09-16T03:37:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000003e-05 2024-09-16T03:37:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:60 4.d 4:b3fe05b2:::smithi15736696-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:37:51.803 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.664277509000001, 2024-09-16T03:37:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30164545100000001, 2024-09-16T03:37:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 60 2024-09-16T03:37:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:37:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:37:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128387+0000", 2024-09-16T03:37:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3599999999999999e-07 2024-09-16T03:37:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128388+0000", 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7659999999999999e-06 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128394+0000", 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0679999999999996e-06 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128398+0000", 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4220000000000002e-06 2024-09-16T03:37:51.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129762+0000", 2024-09-16T03:37:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001364285 2024-09-16T03:37:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129804+0000", 2024-09-16T03:37:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2058000000000003e-05 2024-09-16T03:37:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.429903+0000", 2024-09-16T03:37:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30009841900000001 2024-09-16T03:37:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.429952+0000", 2024-09-16T03:37:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9066000000000001e-05 2024-09-16T03:37:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.430032+0000", 2024-09-16T03:37:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9931000000000004e-05 2024-09-16T03:37:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.808 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:81 4.c 4:3313e97c:::smithi15736696-21:head [write 736140~537686,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.395037485000003, 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203525015, 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.809 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397629+0000", 2024-09-16T03:37:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1889999999999999e-06 2024-09-16T03:37:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398085+0000", 2024-09-16T03:37:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00045514200000000001 2024-09-16T03:37:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398100+0000", 2024-09-16T03:37:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5637999999999998e-05 2024-09-16T03:37:51.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398110+0000", 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6549999999999999e-06 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398190+0000", 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0272999999999995e-05 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398339+0000", 2024-09-16T03:37:51.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014924200000000001 2024-09-16T03:37:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398504+0000", 2024-09-16T03:37:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016471500000000001 2024-09-16T03:37:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.400267+0000", 2024-09-16T03:37:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017625169999999999 2024-09-16T03:37:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.401104+0000", 2024-09-16T03:37:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083686399999999999 2024-09-16T03:37:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601084+0000", 2024-09-16T03:37:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199980397 2024-09-16T03:37:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601114+0000", 2024-09-16T03:37:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0484000000000002e-05 2024-09-16T03:37:51.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601151+0000", 2024-09-16T03:37:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6899e-05 2024-09-16T03:37:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:83 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 2336140] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:37:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.388842689999997, 2024-09-16T03:37:51.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29747773399999999, 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:37:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403822+0000", 2024-09-16T03:37:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-07 2024-09-16T03:37:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403825+0000", 2024-09-16T03:37:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000001e-06 2024-09-16T03:37:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403831+0000", 2024-09-16T03:37:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9399999999999999e-06 2024-09-16T03:37:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403834+0000", 2024-09-16T03:37:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1939999999999998e-06 2024-09-16T03:37:51.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.404994+0000", 2024-09-16T03:37:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011601249999999999 2024-09-16T03:37:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405083+0000", 2024-09-16T03:37:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8809999999999998e-05 2024-09-16T03:37:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:51.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405217+0000", 2024-09-16T03:37:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013401900000000001 2024-09-16T03:37:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.503958+0000", 2024-09-16T03:37:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098741342999999995 2024-09-16T03:37:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.506033+0000", 2024-09-16T03:37:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020749660000000001 2024-09-16T03:37:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701227+0000", 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.195194171 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701262+0000", 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4768999999999999e-05 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701299+0000", 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6884000000000001e-05 2024-09-16T03:37:51.821 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:84 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:37:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.388810964999998, 2024-09-16T03:37:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29760038700000002, 2024-09-16T03:37:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.822 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.58e-07 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403861+0000", 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7689999999999996e-06 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403866+0000", 2024-09-16T03:37:51.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6330000000000004e-06 2024-09-16T03:37:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403869+0000", 2024-09-16T03:37:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7390000000000001e-06 2024-09-16T03:37:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405790+0000", 2024-09-16T03:37:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001920351 2024-09-16T03:37:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405865+0000", 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5234999999999994e-05 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701328+0000", 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29546253300000003 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701382+0000", 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4481999999999998e-05 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701453+0000", 2024-09-16T03:37:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1186999999999995e-05 2024-09-16T03:37:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:432 4.c 4:34492f38:::smithi15736696-33:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:37:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.878831725000001, 2024-09-16T03:37:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.404460861, 2024-09-16T03:37:51.827 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 432 2024-09-16T03:37:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:37:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:37:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913833+0000", 2024-09-16T03:37:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0610000000000001e-06 2024-09-16T03:37:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913839+0000", 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9479999999999996e-06 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913848+0000", 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5629999999999992e-06 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913857+0000", 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0399999999999998e-06 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914022+0000", 2024-09-16T03:37:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000165422 2024-09-16T03:37:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914101+0000", 2024-09-16T03:37:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8462999999999997e-05 2024-09-16T03:37:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217171+0000", 2024-09-16T03:37:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30307040000000002 2024-09-16T03:37:51.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217854+0000", 2024-09-16T03:37:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068278700000000002 2024-09-16T03:37:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.218098+0000", 2024-09-16T03:37:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243825 2024-09-16T03:37:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318164+0000", 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100066451 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318232+0000", 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8113000000000003e-05 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318293+0000", 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0788000000000001e-05 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:790 4.d 4:b3fe05b2:::smithi15736696-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:37:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.904691651, 2024-09-16T03:37:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41456747799999999, 2024-09-16T03:37:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.834 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 790 2024-09-16T03:37:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:37:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:37:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887975+0000", 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0369999999999999e-06 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887987+0000", 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1606e-05 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888016+0000", 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.917e-05 2024-09-16T03:37:51.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888032+0000", 2024-09-16T03:37:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5616000000000001e-05 2024-09-16T03:37:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888178+0000", 2024-09-16T03:37:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612100000000001 2024-09-16T03:37:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888307+0000", 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012953899999999999 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.302540+0000", 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41423238899999998 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1418 4.c 4:3313e97c:::smithi15736696-21:head [write 615156~555834] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:37:51.838 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.566580479000001, 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40891108799999998, 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1418 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:37:51.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:37:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226087+0000", 2024-09-16T03:37:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8890000000000004e-06 2024-09-16T03:37:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226827+0000", 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000739298 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226848+0000", 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1418000000000001e-05 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226858+0000", 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0434e-05 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327036+0000", 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10017769 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327098+0000", 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1760999999999994e-05 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327230+0000", 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131839 2024-09-16T03:37:51.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434369+0000", 2024-09-16T03:37:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107139788 2024-09-16T03:37:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434444+0000", 2024-09-16T03:37:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4171999999999994e-05 2024-09-16T03:37:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634830+0000", 2024-09-16T03:37:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20038660599999999 2024-09-16T03:37:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634937+0000", 2024-09-16T03:37:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010716499999999999 2024-09-16T03:37:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634994+0000", 2024-09-16T03:37:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7027999999999997e-05 2024-09-16T03:37:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.844 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1419 4.c 4:3313e97c:::smithi15736696-21:head [write 1603467~416361] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.561915089999999, 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40430432100000002, 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.845 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1419 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230752+0000", 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.354e-06 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231164+0000", 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041208599999999998 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231179+0000", 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4885999999999999e-05 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231186+0000", 2024-09-16T03:37:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8759999999999999e-06 2024-09-16T03:37:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328726+0000", 2024-09-16T03:37:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097539855999999994 2024-09-16T03:37:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328786+0000", 2024-09-16T03:37:51.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0059000000000002e-05 2024-09-16T03:37:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328890+0000", 2024-09-16T03:37:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104338 2024-09-16T03:37:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434475+0000", 2024-09-16T03:37:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.105584709 2024-09-16T03:37:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.534594+0000", 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10011914099999999 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635003+0000", 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100408834 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635029+0000", 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6551999999999999e-05 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635053+0000", 2024-09-16T03:37:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3629999999999999e-05 2024-09-16T03:37:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1420 4.c 4:3313e97c:::smithi15736696-21:head [write 2816389~693495] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:37:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:37:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.554000779000001, 2024-09-16T03:37:51.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39653502600000001, 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1420 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238666+0000", 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.163e-06 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239395+0000", 2024-09-16T03:37:51.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072868599999999998 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239410+0000", 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5330999999999999e-05 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239419+0000", 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7360000000000007e-06 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332338+0000", 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092919165999999997 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332435+0000", 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6972999999999995e-05 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332544+0000", 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010898700000000001 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634808+0000", 2024-09-16T03:37:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30226407700000002 2024-09-16T03:37:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635058+0000", 2024-09-16T03:37:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025009300000000001 2024-09-16T03:37:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635103+0000", 2024-09-16T03:37:51.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4539000000000002e-05 2024-09-16T03:37:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635145+0000", 2024-09-16T03:37:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1938999999999999e-05 2024-09-16T03:37:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635198+0000", 2024-09-16T03:37:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3335999999999999e-05 2024-09-16T03:37:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.857 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1421 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:37:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:37:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.55323233, 2024-09-16T03:37:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.497492088, 2024-09-16T03:37:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1421 2024-09-16T03:37:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:37:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:37:51.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239432+0000", 2024-09-16T03:37:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5999999999999991e-07 2024-09-16T03:37:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239436+0000", 2024-09-16T03:37:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:37:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239443+0000", 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.968e-06 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239447+0000", 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6969999999999999e-06 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635225+0000", 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39577807399999998 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635290+0000", 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4583000000000002e-05 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635410+0000", 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120684 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636180+0000", 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076997899999999995 2024-09-16T03:37:51.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636799+0000", 2024-09-16T03:37:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061906600000000004 2024-09-16T03:37:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736742+0000", 2024-09-16T03:37:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099943041999999996 2024-09-16T03:37:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736843+0000", 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010076799999999999 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736924+0000", 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0419000000000004e-05 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1422 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:37:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.55321105, 2024-09-16T03:37:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49764818700000002, 2024-09-16T03:37:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1422 2024-09-16T03:37:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239455+0000", 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9860000000000001e-06 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239461+0000", 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239464+0000", 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2619999999999999e-06 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635736+0000", 2024-09-16T03:37:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.396272287 2024-09-16T03:37:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635790+0000", 2024-09-16T03:37:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4252000000000002e-05 2024-09-16T03:37:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736964+0000", 2024-09-16T03:37:51.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101173319 2024-09-16T03:37:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737024+0000", 2024-09-16T03:37:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9975000000000001e-05 2024-09-16T03:37:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737101+0000", 2024-09-16T03:37:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7303000000000002e-05 2024-09-16T03:37:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1420 4.c e58/32 ondisk, result = 0)", 2024-09-16T03:37:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.335674+0000", 2024-09-16T03:37:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.456990235999999, 2024-09-16T03:37:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29938561899999999, 2024-09-16T03:37:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335674+0000", 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335674+0000", 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335675+0000", 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5510000000000001e-06 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335684+0000", 2024-09-16T03:37:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6519999999999995e-06 2024-09-16T03:37:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335696+0000", 2024-09-16T03:37:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2662e-05 2024-09-16T03:37:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335703+0000", 2024-09-16T03:37:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2079999999999997e-06 2024-09-16T03:37:51.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634784+0000", 2024-09-16T03:37:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29908135299999999 2024-09-16T03:37:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634797+0000", 2024-09-16T03:37:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3393999999999999e-05 2024-09-16T03:37:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635059+0000", 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026179900000000001 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1420 4.c e58/32 ondisk, result = 0)", 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.336136+0000", 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.456527479, 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29907022, 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:51.874 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336136+0000", 2024-09-16T03:37:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336136+0000", 2024-09-16T03:37:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336138+0000", 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4279999999999999e-06 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336147+0000", 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0210000000000005e-06 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336159+0000", 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2285e-05 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336165+0000", 2024-09-16T03:37:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7799999999999997e-06 2024-09-16T03:37:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635091+0000", 2024-09-16T03:37:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29892637599999999 2024-09-16T03:37:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635098+0000", 2024-09-16T03:37:51.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5019999999999998e-06 2024-09-16T03:37:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635207+0000", 2024-09-16T03:37:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108828 2024-09-16T03:37:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1537 4.c 4:34492f38:::smithi15736696-33:head [write 1954970~431037] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:37:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.719891+0000", 2024-09-16T03:37:51.878 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.072773132, 2024-09-16T03:37:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20396983899999999, 2024-09-16T03:37:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1537 2024-09-16T03:37:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.719891+0000", 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.719891+0000", 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.719893+0000", 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.962e-06 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720069+0000", 2024-09-16T03:37:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176298 2024-09-16T03:37:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720080+0000", 2024-09-16T03:37:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1002000000000001e-05 2024-09-16T03:37:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720088+0000", 2024-09-16T03:37:51.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7440000000000007e-06 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720132+0000", 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4471999999999997e-05 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720196+0000", 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3946000000000001e-05 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:51.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720286+0000", 2024-09-16T03:37:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9423000000000001e-05 2024-09-16T03:37:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.723040+0000", 2024-09-16T03:37:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027547169999999998 2024-09-16T03:37:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923715+0000", 2024-09-16T03:37:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20067458299999999 2024-09-16T03:37:51.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923777+0000", 2024-09-16T03:37:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2135000000000001e-05 2024-09-16T03:37:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923816+0000", 2024-09-16T03:37:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8563999999999999e-05 2024-09-16T03:37:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923861+0000", 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4993000000000001e-05 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1538 4.c 4:34492f38:::smithi15736696-33:head [write 2919420~616377] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.720103+0000", 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.0725605959999998, 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20383674299999999, 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1538 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720103+0000", 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720103+0000", 2024-09-16T03:37:51.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720104+0000", 2024-09-16T03:37:51.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9400000000000004e-07 2024-09-16T03:37:51.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721193+0000", 2024-09-16T03:37:51.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001089268 2024-09-16T03:37:51.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721203+0000", 2024-09-16T03:37:51.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9799999999999993e-06 2024-09-16T03:37:51.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721212+0000", 2024-09-16T03:37:51.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4430000000000003e-06 2024-09-16T03:37:51.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721350+0000", 2024-09-16T03:37:51.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013873600000000001 2024-09-16T03:37:51.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721407+0000", 2024-09-16T03:37:51.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6328e-05 2024-09-16T03:37:51.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:37:51.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721501+0000", 2024-09-16T03:37:51.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3868000000000003e-05 2024-09-16T03:37:51.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.723066+0000", 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001565461 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923739+0000", 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20067338300000001 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923894+0000", 2024-09-16T03:37:51.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015406600000000001 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923921+0000", 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7440000000000002e-05 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923940+0000", 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8975999999999999e-05 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.892 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1540 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:37:51.892 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.721248+0000", 2024-09-16T03:37:51.892 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.0714161569999998, 2024-09-16T03:37:51.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20319308899999999, 2024-09-16T03:37:51.892 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.892 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.892 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:51.892 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.892 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.892 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.892 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1540 2024-09-16T03:37:51.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721248+0000", 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721248+0000", 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721248+0000", 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5599999999999995e-07 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721251+0000", 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.633e-06 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721257+0000", 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8329999999999999e-06 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721262+0000", 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0429999999999998e-06 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.722980+0000", 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001717847 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.723032+0000", 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2911999999999998e-05 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.924330+0000", 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201297272 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.924383+0000", 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3007999999999998e-05 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.924441+0000", 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7985000000000001e-05 2024-09-16T03:37:51.896 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.897 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1681 4.d 4:b3fe05b2:::smithi15736696-15:head [write 685752~611865,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:37:51.897 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.148763+0000", 2024-09-16T03:37:51.897 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.6439006840000001, 2024-09-16T03:37:51.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20615159899999999, 2024-09-16T03:37:51.897 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.897 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.897 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.897 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.898 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1681 2024-09-16T03:37:51.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.898 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.148763+0000", 2024-09-16T03:37:51.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.148763+0000", 2024-09-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.148768+0000", 2024-09-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.527e-06 2024-09-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149421+0000", 2024-09-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065293199999999995 2024-09-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149439+0000", 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8210999999999999e-05 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149448+0000", 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2480000000000002e-06 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149507+0000", 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8980000000000001e-05 2024-09-16T03:37:51.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149593+0000", 2024-09-16T03:37:51.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6113999999999996e-05 2024-09-16T03:37:51.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:37:51.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149730+0000", 2024-09-16T03:37:51.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013668 2024-09-16T03:37:51.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.151623+0000", 2024-09-16T03:37:51.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018936350000000001 2024-09-16T03:37:51.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354505+0000", 2024-09-16T03:37:51.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20288149799999999 2024-09-16T03:37:51.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354562+0000", 2024-09-16T03:37:51.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7255e-05 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354854+0000", 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029133000000000002 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354915+0000", 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1189e-05 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.904 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1683 4.d 4:b3fe05b2:::smithi15736696-15:head [setxattr _header (56) in=7b,truncate 2285752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:37:51.904 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.154316+0000", 2024-09-16T03:37:51.904 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.638348191, 2024-09-16T03:37:51.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29800856100000001, 2024-09-16T03:37:51.904 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.904 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.904 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:51.904 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.904 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.904 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.904 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1683 2024-09-16T03:37:51.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154316+0000", 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154316+0000", 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154316+0000", 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1500000000000003e-07 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154320+0000", 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154332+0000", 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2254e-05 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154340+0000", 2024-09-16T03:37:51.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5340000000000002e-06 2024-09-16T03:37:51.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.155080+0000", 2024-09-16T03:37:51.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073960699999999998 2024-09-16T03:37:51.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.155143+0000", 2024-09-16T03:37:51.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3859e-05 2024-09-16T03:37:51.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:37:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.155223+0000", 2024-09-16T03:37:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9331000000000003e-05 2024-09-16T03:37:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:37:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354955+0000", 2024-09-16T03:37:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19973200099999999 2024-09-16T03:37:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.355860+0000", 2024-09-16T03:37:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090486199999999998 2024-09-16T03:37:51.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:51.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.452226+0000", 2024-09-16T03:37:51.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096366118000000001 2024-09-16T03:37:51.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:37:51.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.452263+0000", 2024-09-16T03:37:51.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7422000000000002e-05 2024-09-16T03:37:51.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.452324+0000", 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1048000000000002e-05 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1684 4.d 4:b3fe05b2:::smithi15736696-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.638315934, 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39825815199999998, 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:37:51.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:37:51.913 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:37:51.913 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:51.913 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1684 2024-09-16T03:37:51.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.913 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:37:51.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:37:51.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:37:51.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:37:51.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:37:51.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:37:51.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:37:51.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:37:51.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9999999999999998e-07 2024-09-16T03:37:51.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:37:51.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154351+0000", 2024-09-16T03:37:51.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.249e-06 2024-09-16T03:37:51.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154356+0000", 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7690000000000002e-06 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154361+0000", 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4209999999999997e-06 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.253488+0000", 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099126895000000007 2024-09-16T03:37:51.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.253585+0000", 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7204e-05 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552451+0000", 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.298866358 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552521+0000", 2024-09-16T03:37:51.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9873e-05 2024-09-16T03:37:51.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:37:51.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:37:51.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:37:51.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552606+0000", 2024-09-16T03:37:51.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4882999999999994e-05 2024-09-16T03:37:51.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.917 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:37:51.917 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:37:51.917 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:37:51.918 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-09-16T03:37:51.873+0000 7fdb04aef640 -1 mon.c@2(leader).osd e66 definitely_dead 0 2024-09-16T03:37:51.920 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:37:51.921 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:51.921 INFO:tasks.rados.rados.0.smithi157.stdout:907: finishing set_chunk to oid 41 2024-09-16T03:37:51.921 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 32 (ObjNum 112 snap 0 seq_num 112) dirty exists 2024-09-16T03:37:51.921 INFO:tasks.rados.rados.0.smithi157.stdout:916: seq_num 142 ranges {786594=779712,2326443=60151} 2024-09-16T03:37:51.921 INFO:tasks.rados.rados.0.smithi157.stdout:916: writing smithi15736696-42 from 786594 to 1566306 tid 1 2024-09-16T03:37:51.921 INFO:tasks.rados.rados.0.smithi157.stdout:916: writing smithi15736696-42 from 2326443 to 2386594 tid 2 2024-09-16T03:37:51.921 INFO:tasks.rados.rados.0.smithi157.stdout:907: done (6 left) 2024-09-16T03:37:51.922 INFO:tasks.rados.rados.0.smithi157.stdout:908: done (5 left) 2024-09-16T03:37:51.922 INFO:tasks.rados.rados.0.smithi157.stdout:912: done (4 left) 2024-09-16T03:37:51.922 INFO:tasks.rados.rados.0.smithi157.stdout:917: read oid 19 snap -1 2024-09-16T03:37:51.922 INFO:tasks.rados.rados.0.smithi157.stdout:917: expect (ObjNum 93 snap 0 seq_num 93) 2024-09-16T03:37:51.922 INFO:tasks.rados.rados.0.smithi157.stdout:918: tier_evict oid 9 2024-09-16T03:37:51.922 INFO:tasks.rados.rados.0.smithi157.stdout:918: tier_evict oid 9 snap -1 2024-09-16T03:37:51.922 INFO:tasks.rados.rados.0.smithi157.stdout:914: finishing write tid 1 to smithi15736696-18 2024-09-16T03:37:51.922 INFO:tasks.rados.rados.0.smithi157.stdout:915: expect (ObjNum 104 snap 0 seq_num 104) 2024-09-16T03:37:51.922 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:51.923 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:51.917+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:51.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:37:51.987 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 197653 2024-09-16T03:37:52.019 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:52.018+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:52.036 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:52.036 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:52.036 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:52.036 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:52.045 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:37:52.047 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:52.047 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:52.065 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:52.065 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:52.082 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:52.082 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:52.088 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2079265 2024-09-16T03:37:52.098 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:52.098 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:52.114 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:52.115 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:52.120 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:52.118+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:52.125 INFO:tasks.rados.rados.0.smithi157.stdout:913: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-16T03:37:52.132 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 780373 2024-09-16T03:37:52.132 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:52.132 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:52.149 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:52.149 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:52.158 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:37:52.173 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:52.173 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:52.187 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:52.189 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:52.190 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:52.205 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:52.206 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:52.221 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:52.219+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:52.224 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:52.224 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:52.240 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:52.241 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:52.257 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:52.257 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:52.257 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:52.262 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:52.262 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:52.270 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:37:52.273 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:52.273 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:52.289 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:52.289 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:52.306 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:52.306 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:52.320 INFO:tasks.rados.rados.0.smithi157.stdout:918: got (22) Invalid argument 2024-09-16T03:37:52.320 INFO:tasks.rados.rados.0.smithi157.stdout:917: expect (ObjNum 93 snap 0 seq_num 93) 2024-09-16T03:37:52.320 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:52.321 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:52.319+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:52.322 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:52.323 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:52.365 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 824498 2024-09-16T03:37:52.413 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:52.414 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:52.422 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:37:52.423 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:52.420+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:52.433 INFO:tasks.rados.rados.0.smithi157.stdout:916: finishing write tid 1 to smithi15736696-42 2024-09-16T03:37:52.434 INFO:tasks.rados.rados.0.smithi157.stdout:916: finishing write tid 2 to smithi15736696-42 2024-09-16T03:37:52.434 INFO:tasks.rados.rados.0.smithi157.stdout:916: finishing write tid 3 to smithi15736696-42 2024-09-16T03:37:52.434 INFO:tasks.rados.rados.0.smithi157.stdout:916: finishing write tid 4 to smithi15736696-42 2024-09-16T03:37:52.435 INFO:tasks.rados.rados.0.smithi157.stdout:916: oid 42 updating version 0 to 42 2024-09-16T03:37:52.435 INFO:tasks.rados.rados.0.smithi157.stdout:916: oid 42 version 42 is already newer than 40 2024-09-16T03:37:52.435 INFO:tasks.rados.rados.0.smithi157.stdout:916: oid 42 version 42 is already newer than 41 2024-09-16T03:37:52.435 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 42 (ObjNum 142 snap 0 seq_num 142) dirty exists 2024-09-16T03:37:52.435 INFO:tasks.rados.rados.0.smithi157.stdout:916: left oid 42 (ObjNum 142 snap 0 seq_num 142) 2024-09-16T03:37:52.435 INFO:tasks.rados.rados.0.smithi157.stdout:914: finishing write tid 2 to smithi15736696-18 2024-09-16T03:37:52.435 INFO:tasks.rados.rados.0.smithi157.stdout:913: done (5 left) 2024-09-16T03:37:52.435 INFO:tasks.rados.rados.0.smithi157.stdout:915: done (4 left) 2024-09-16T03:37:52.435 INFO:tasks.rados.rados.0.smithi157.stdout:916: done (3 left) 2024-09-16T03:37:52.435 INFO:tasks.rados.rados.0.smithi157.stdout:917: done (2 left) 2024-09-16T03:37:52.436 INFO:tasks.rados.rados.0.smithi157.stdout:918: done (1 left) 2024-09-16T03:37:52.436 INFO:tasks.rados.rados.0.smithi157.stdout:919: read oid 11 snap -1 2024-09-16T03:37:52.436 INFO:tasks.rados.rados.0.smithi157.stdout:919: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:37:52.436 INFO:tasks.rados.rados.0.smithi157.stdout:920: set_chunk oid 27 target oid 2024-09-16T03:37:52.450 INFO:teuthology.orchestra.run.smithi064.stderr:marked out osd.5. 2024-09-16T03:37:52.450 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:37:52.452 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:52.450+0000 7efeef6d5640 -1 osd.13 66 handle_osd_map injecting map crc failure 2024-09-16T03:37:52.466 INFO:tasks.thrashosds.thrasher:Removing osd 9, in_osds are: [0, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-16T03:37:52.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 9 2024-09-16T03:37:52.523 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:52.521+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:52.528 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:52.529 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:52.537 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:37:52.624 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:52.622+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:52.683 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:52.683 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:52.692 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:37:52.725 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:52.723+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:52.817 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-09-16T03:37:52.815+0000 7fdb04aef640 -1 mon.c@2(leader).osd e67 definitely_dead 0 2024-09-16T03:37:52.825 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:52.824+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:52.835 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:52.835 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:52.844 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:37:52.929 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:52.924+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:52.946 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:52.946 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:52.954 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:37:53.026 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:53.025+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:53.057 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:53.057 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:53.066 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:37:53.127 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:53.126+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:53.170 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:53.170 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:53.178 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:37:53.263 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:53.261+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:53.321 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:53.321 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:53.329 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:53.327+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:53.330 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:37:53.430 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:53.427+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:53.453 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:53.450+0000 7fedb773d640 -1 osd.9 67 handle_osd_map injecting map crc failure 2024-09-16T03:37:53.453 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:53.451+0000 7fa38a6ea640 -1 osd.10 67 handle_osd_map injecting map crc failure 2024-09-16T03:37:53.455 INFO:teuthology.orchestra.run.smithi064.stderr:marked out osd.9. 2024-09-16T03:37:53.457 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:53.455+0000 7eff7884b640 -1 osd.3 67 handle_osd_map injecting map crc failure 2024-09-16T03:37:53.473 INFO:tasks.thrashosds.thrasher:Removing osd 13, in_osds are: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 13, 14, 15] 2024-09-16T03:37:53.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 13 2024-09-16T03:37:53.476 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:53.476 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:53.485 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:37:53.531 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:53.530+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:53.600 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:53.600 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:53.609 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:37:53.632 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:53.631+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:53.712 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:53.712 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:53.721 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:37:53.766 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:53.764+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:53.825 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:53.825 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:53.833 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-09-16T03:37:53.830+0000 7fdb04aef640 -1 mon.c@2(leader).osd e68 definitely_dead 0 2024-09-16T03:37:53.834 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:37:53.834 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:53.832+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:53.934 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:53.933+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:53.978 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:53.978 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:53.987 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:37:54.035 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:54.034+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:37:54.091 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:54.091 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:54.100 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:37:54.144 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:54.142+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:54.203 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:54.204 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:54.212 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:37:54.236 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:54.235+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:54.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:37:54.337 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:54.335+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:54.358 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:54.358 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:54.366 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:37:54.438 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:54.436+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:54.457 INFO:teuthology.orchestra.run.smithi064.stderr:marked out osd.13. 2024-09-16T03:37:54.467 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:54.467 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:54.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:37:54.485 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:54.486 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:54.489 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:37:54.487+0000 7f8be56cd640 -1 osd.7 68 handle_osd_map injecting map crc failure 2024-09-16T03:37:54.503 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:54.503 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:54.515 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:54.515 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:54.520 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:54.520 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:54.524 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:37:54.536 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:54.537 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:54.539 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:54.538+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:54.554 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:54.554 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:54.572 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:54.572 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:54.598 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:54.599 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:54.630 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:54.630 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:54.630 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:54.630 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:54.639 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:37:54.640 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:54.639+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:54.662 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:54.662 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:54.679 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:54.679 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:54.696 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:54.697 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:54.714 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:54.714 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:54.731 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:54.731 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:54.742 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:54.740+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:54.744 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:54.744 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:54.747 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:54.747 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:54.753 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:37:54.764 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:54.765 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:54.843 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:54.841+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:54.856 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:37:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:37:54.857 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:37:54.857 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:54.866 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:37:54.944 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:54.942+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:54.972 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:37:54.972 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:37:54.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:37:54.972 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:37:54.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.972 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:54.972 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:37:54.972 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.825602117000003, 2024-09-16T03:37:54.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:37:54.972 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:54.973 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:54.973 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:54.973 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:54.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.973 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:54.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:37:54.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:54.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.973 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:54.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:37:54.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:37:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:37:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:37:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:37:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:37:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:37:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:37:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:37:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.975 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:37:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:37:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:37:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:37:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:37:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:37:54.976 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:37:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:37:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:37:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.825248756999997, 2024-09-16T03:37:54.977 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:37:54.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:37:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:37:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:37:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:37:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:37:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:37:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:37:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:37:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.980 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:37:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:37:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:37:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:37:54.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:37:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:37:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:37:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:37:54.982 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.824120958999998, 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:37:54.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:37:54.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:37:54.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.987 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.987 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:54.987 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:37:54.987 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:37:54.987 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:54.987 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:54.987 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:54.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.987 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.987 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:37:54.987 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:37:54.987 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.823133517999999, 2024-09-16T03:37:54.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:37:54.988 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:54.988 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:54.988 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:54.988 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:37:54.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:54.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:37:54.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:37:54.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:54.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:37:54.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:37:54.992 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:54.992 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:54.992 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:54.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.993 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.993 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:37:54.993 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:37:54.993 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.822376773999999, 2024-09-16T03:37:54.993 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:37:54.993 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:54.993 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:54.993 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:54.993 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:54.993 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.993 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:54.993 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:37:54.993 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:54.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:54.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:37:54.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:37:54.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:54.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:37:54.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:37:54.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:54.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:37:54.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:37:54.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.997 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:54.997 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:37:54.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:37:54.997 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:54.997 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:54.997 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:54.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.997 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.997 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:86 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'8, mlcod=39'5)", 2024-09-16T03:37:54.997 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:37:54.997 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.360864538000001, 2024-09-16T03:37:54.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20388281899999999, 2024-09-16T03:37:54.998 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:54.998 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:54.998 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:54.998 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:54.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.998 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:54.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:37:54.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:54.998 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.998 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:54.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:37:54.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608472+0000", 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.227e-06 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608616+0000", 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000144177 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:54.999 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608638+0000", 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1543e-05 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608645+0000", 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5839999999999997e-06 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608740+0000", 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4716999999999995e-05 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.000 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.001 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608763+0000", 2024-09-16T03:37:55.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2671e-05 2024-09-16T03:37:55.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.001 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.001 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812264+0000", 2024-09-16T03:37:55.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203501194 2024-09-16T03:37:55.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.001 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.001 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812353+0000", 2024-09-16T03:37:55.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8706000000000001e-05 2024-09-16T03:37:55.001 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.002 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.002 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.002 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.002 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.002 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:87 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'9, mlcod=39'5)", 2024-09-16T03:37:55.002 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:37:55.002 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.360651163999997, 2024-09-16T03:37:55.002 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20372206400000001, 2024-09-16T03:37:55.002 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608684+0000", 2024-09-16T03:37:55.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6700000000000003e-07 2024-09-16T03:37:55.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608689+0000", 2024-09-16T03:37:55.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.412e-06 2024-09-16T03:37:55.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608704+0000", 2024-09-16T03:37:55.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4397e-05 2024-09-16T03:37:55.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.005 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.005 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.005 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608713+0000", 2024-09-16T03:37:55.005 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:37:55.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.005 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.005 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.005 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710025+0000", 2024-09-16T03:37:55.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101312032 2024-09-16T03:37:55.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710042+0000", 2024-09-16T03:37:55.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6889e-05 2024-09-16T03:37:55.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812361+0000", 2024-09-16T03:37:55.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10231889600000001 2024-09-16T03:37:55.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812405+0000", 2024-09-16T03:37:55.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4101999999999999e-05 2024-09-16T03:37:55.007 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.007 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.007 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.007 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:101 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'10, mlcod=39'5)", 2024-09-16T03:37:55.008 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:37:55.008 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.359637726999999, 2024-09-16T03:37:55.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20275586900000001, 2024-09-16T03:37:55.008 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.008 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.008 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.008 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:37:55.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:37:55.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609698+0000", 2024-09-16T03:37:55.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.24e-06 2024-09-16T03:37:55.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609892+0000", 2024-09-16T03:37:55.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019465899999999999 2024-09-16T03:37:55.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609904+0000", 2024-09-16T03:37:55.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2031000000000001e-05 2024-09-16T03:37:55.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609912+0000", 2024-09-16T03:37:55.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7160000000000003e-06 2024-09-16T03:37:55.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710228+0000", 2024-09-16T03:37:55.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100315682 2024-09-16T03:37:55.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710242+0000", 2024-09-16T03:37:55.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4201e-05 2024-09-16T03:37:55.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812411+0000", 2024-09-16T03:37:55.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10216881799999999 2024-09-16T03:37:55.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812452+0000", 2024-09-16T03:37:55.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1522e-05 2024-09-16T03:37:55.012 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.012 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.012 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.013 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:102 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'11, mlcod=39'5)", 2024-09-16T03:37:55.013 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:37:55.013 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.357621702000003, 2024-09-16T03:37:55.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20134195399999999, 2024-09-16T03:37:55.013 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.013 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.013 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.013 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:37:55.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611714+0000", 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.269e-06 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611926+0000", 2024-09-16T03:37:55.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021252700000000001 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611938+0000", 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1437000000000001e-05 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611944+0000", 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.993e-06 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811266+0000", 2024-09-16T03:37:55.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.199322306 2024-09-16T03:37:55.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811291+0000", 2024-09-16T03:37:55.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4542999999999999e-05 2024-09-16T03:37:55.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813021+0000", 2024-09-16T03:37:55.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001730363 2024-09-16T03:37:55.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813054+0000", 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3516000000000002e-05 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:103 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'12, mlcod=39'5)", 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.354415201000002, 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19838012999999999, 2024-09-16T03:37:55.017 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614920+0000", 2024-09-16T03:37:55.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.192e-06 2024-09-16T03:37:55.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614927+0000", 2024-09-16T03:37:55.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4640000000000003e-06 2024-09-16T03:37:55.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614937+0000", 2024-09-16T03:37:55.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9599999999999995e-06 2024-09-16T03:37:55.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614942+0000", 2024-09-16T03:37:55.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.31e-06 2024-09-16T03:37:55.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812489+0000", 2024-09-16T03:37:55.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.197546629 2024-09-16T03:37:55.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812511+0000", 2024-09-16T03:37:55.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2385e-05 2024-09-16T03:37:55.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813279+0000", 2024-09-16T03:37:55.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076818100000000005 2024-09-16T03:37:55.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813299+0000", 2024-09-16T03:37:55.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0009000000000001e-05 2024-09-16T03:37:55.022 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.022 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.022 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.023 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:375 4.f e45/32 4:fcb75e43:::smithi15736696-26:head v 45'23, mlcod=41'20)", 2024-09-16T03:37:55.023 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:37:55.023 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.661293974000003, 2024-09-16T03:37:55.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200786568, 2024-09-16T03:37:55.023 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.023 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.023 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.023 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:37:55.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:37:55.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308041+0000", 2024-09-16T03:37:55.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8299999999999995e-07 2024-09-16T03:37:55.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308046+0000", 2024-09-16T03:37:55.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.904e-06 2024-09-16T03:37:55.025 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308056+0000", 2024-09-16T03:37:55.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9210000000000004e-06 2024-09-16T03:37:55.025 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.408122+0000", 2024-09-16T03:37:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100066302 2024-09-16T03:37:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508289+0000", 2024-09-16T03:37:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166531 2024-09-16T03:37:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508302+0000", 2024-09-16T03:37:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2641e-05 2024-09-16T03:37:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508794+0000", 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00049272400000000005 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508827+0000", 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2561999999999998e-05 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.027 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:459 4.f e45/32 4:fa163101:::smithi15736696-10:head v 45'25, mlcod=45'23)", 2024-09-16T03:37:55.028 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.318000+0000", 2024-09-16T03:37:55.028 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 33.651334069999997, 2024-09-16T03:37:55.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10092035100000001, 2024-09-16T03:37:55.028 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.028 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.028 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.028 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318000+0000", 2024-09-16T03:37:55.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318000+0000", 2024-09-16T03:37:55.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318002+0000", 2024-09-16T03:37:55.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.624e-06 2024-09-16T03:37:55.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318009+0000", 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9310000000000002e-06 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318018+0000", 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4949999999999997e-06 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318031+0000", 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2869e-05 2024-09-16T03:37:55.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418228+0000", 2024-09-16T03:37:55.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10019649799999999 2024-09-16T03:37:55.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418286+0000", 2024-09-16T03:37:55.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8904999999999999e-05 2024-09-16T03:37:55.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418884+0000", 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059763899999999996 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418921+0000", 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6390000000000002e-05 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:509 4.f e45/32 4:fa163101:::smithi15736696-10:head v 45'26, mlcod=45'24)", 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:22.024930+0000", 2024-09-16T03:37:55.032 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 32.944404540000001, 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10084305, 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024930+0000", 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024930+0000", 2024-09-16T03:37:55.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024932+0000", 2024-09-16T03:37:55.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.863e-06 2024-09-16T03:37:55.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024942+0000", 2024-09-16T03:37:55.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9960000000000005e-06 2024-09-16T03:37:55.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024954+0000", 2024-09-16T03:37:55.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2128e-05 2024-09-16T03:37:55.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.024962+0000", 2024-09-16T03:37:55.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4209999999999995e-06 2024-09-16T03:37:55.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125116+0000", 2024-09-16T03:37:55.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015399899999999 2024-09-16T03:37:55.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125160+0000", 2024-09-16T03:37:55.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4359e-05 2024-09-16T03:37:55.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125734+0000", 2024-09-16T03:37:55.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00057321599999999998 2024-09-16T03:37:55.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125773+0000", 2024-09-16T03:37:55.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9067999999999999e-05 2024-09-16T03:37:55.037 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.037 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.037 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.037 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:989 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'31, mlcod=46'28)", 2024-09-16T03:37:55.037 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:37:55.037 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.581525414000001, 2024-09-16T03:37:55.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40332266900000002, 2024-09-16T03:37:55.038 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.038 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.038 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.038 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:37:55.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:37:55.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387812+0000", 2024-09-16T03:37:55.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:37:55.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388467+0000", 2024-09-16T03:37:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065539500000000002 2024-09-16T03:37:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388481+0000", 2024-09-16T03:37:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3905e-05 2024-09-16T03:37:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488552+0000", 2024-09-16T03:37:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10007120899999999 2024-09-16T03:37:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688818+0000", 2024-09-16T03:37:55.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20026627599999999 2024-09-16T03:37:55.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688845+0000", 2024-09-16T03:37:55.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:37:55.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791091+0000", 2024-09-16T03:37:55.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10224588699999999 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791131+0000", 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0293000000000002e-05 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:990 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'32, mlcod=46'28)", 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.480741741999999, 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30256419200000001, 2024-09-16T03:37:55.042 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.043 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.043 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.043 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:37:55.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:37:55.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488594+0000", 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488616+0000", 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2419e-05 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488627+0000", 2024-09-16T03:37:55.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1251e-05 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588753+0000", 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012544399999999 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690679+0000", 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101926027 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690695+0000", 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6528000000000002e-05 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791133+0000", 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100438233 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791157+0000", 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-05 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:991 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'33, mlcod=46'28)", 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.380544244999999, 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202786365, 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:37:55.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588792+0000", 2024-09-16T03:37:55.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0530000000000001e-06 2024-09-16T03:37:55.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588807+0000", 2024-09-16T03:37:55.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5257e-05 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588832+0000", 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5123e-05 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588846+0000", 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3312999999999999e-05 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790810+0000", 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20196378100000001 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790866+0000", 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6168999999999997e-05 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791542+0000", 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067583699999999999 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791576+0000", 2024-09-16T03:37:55.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4832000000000001e-05 2024-09-16T03:37:55.051 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.051 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.051 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.051 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1072 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'35, mlcod=50'32)", 2024-09-16T03:37:55.051 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:37:55.051 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.661302417999998, 2024-09-16T03:37:55.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.103370715, 2024-09-16T03:37:55.051 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.051 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.051 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308034+0000", 2024-09-16T03:37:55.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4099999999999998e-06 2024-09-16T03:37:55.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.053 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308377+0000", 2024-09-16T03:37:55.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034276400000000001 2024-09-16T03:37:55.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.053 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308415+0000", 2024-09-16T03:37:55.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7916000000000001e-05 2024-09-16T03:37:55.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408542+0000", 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012673699999999 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408694+0000", 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015200399999999999 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408760+0000", 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6549000000000003e-05 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.411366+0000", 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002605324 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.411403+0000", 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7011000000000002e-05 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1580 4.f e61/32 4:fa163101:::smithi15736696-10:head v 61'52, mlcod=58'50)", 2024-09-16T03:37:55.055 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:37:55.056 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 7.8637127190000005, 2024-09-16T03:37:55.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.201787367, 2024-09-16T03:37:55.056 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.056 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.056 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.056 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.056 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:37:55.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.056 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.056 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2600000000000001e-07 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106083+0000", 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00046163900000000002 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106093+0000", 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9320000000000008e-06 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206190+0000", 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100096716 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206286+0000", 2024-09-16T03:37:55.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5982000000000004e-05 2024-09-16T03:37:55.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.059 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.059 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206312+0000", 2024-09-16T03:37:55.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6279999999999999e-05 2024-09-16T03:37:55.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.059 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.059 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307345+0000", 2024-09-16T03:37:55.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10103240199999999 2024-09-16T03:37:55.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.059 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307409+0000", 2024-09-16T03:37:55.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4189999999999994e-05 2024-09-16T03:37:55.060 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.060 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.060 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.060 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1581 4.f e61/32 4:fa163101:::smithi15736696-10:head v 61'53, mlcod=58'50)", 2024-09-16T03:37:55.060 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:47.206244+0000", 2024-09-16T03:37:55.060 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 7.7630901879999996, 2024-09-16T03:37:55.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101636226, 2024-09-16T03:37:55.060 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.060 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.061 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.061 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.061 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206244+0000", 2024-09-16T03:37:55.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.061 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206244+0000", 2024-09-16T03:37:55.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206246+0000", 2024-09-16T03:37:55.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7549999999999999e-06 2024-09-16T03:37:55.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206367+0000", 2024-09-16T03:37:55.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012166 2024-09-16T03:37:55.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206375+0000", 2024-09-16T03:37:55.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.571e-06 2024-09-16T03:37:55.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206380+0000", 2024-09-16T03:37:55.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5210000000000004e-06 2024-09-16T03:37:55.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.306514+0000", 2024-09-16T03:37:55.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10013456 2024-09-16T03:37:55.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.306531+0000", 2024-09-16T03:37:55.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.702e-05 2024-09-16T03:37:55.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307860+0000", 2024-09-16T03:37:55.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013284080000000001 2024-09-16T03:37:55.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307880+0000", 2024-09-16T03:37:55.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0730999999999999e-05 2024-09-16T03:37:55.064 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1582 4.f e61/32 4:fa163101:::smithi15736696-10:head v 61'54, mlcod=58'50)", 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:47.210745+0000", 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 7.7585889989999997, 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101062938, 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.065 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210745+0000", 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210745+0000", 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210747+0000", 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.494e-06 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210754+0000", 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1069999999999999e-06 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210767+0000", 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3054000000000001e-05 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210774+0000", 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4959999999999999e-06 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:37:55.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.310881+0000", 2024-09-16T03:37:55.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100106371 2024-09-16T03:37:55.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:37:55.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.310900+0000", 2024-09-16T03:37:55.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9666000000000001e-05 2024-09-16T03:37:55.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:37:55.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.311778+0000", 2024-09-16T03:37:55.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00087738400000000004 2024-09-16T03:37:55.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:37:55.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:37:55.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:37:55.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.311808+0000", 2024-09-16T03:37:55.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0366e-05 2024-09-16T03:37:55.069 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.069 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.069 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.069 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:37:55.069 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:37:55.069 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:37:55.071 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:37:55.071 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:55.068+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:55.171 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:55.169+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:55.186 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:55.187 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:55.187 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:37:55.187 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:55.197 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:37:55.272 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:55.271+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:55.301 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:55.302 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:55.302 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:37:55.302 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:37:55.302 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:55.310 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:37:55.373 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:55.371+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:55.455 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:55.455 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:37:55.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:37:55.455 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:55.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.456 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:14 4.a e39/32 4:51338165:::smithi15736696-4:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:55.456 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:11.997261+0000", 2024-09-16T03:37:55.456 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.455455202000003, 2024-09-16T03:37:55.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185554500000001, 2024-09-16T03:37:55.456 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.456 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.456 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.456 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997261+0000", 2024-09-16T03:37:55.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.457 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.457 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997261+0000", 2024-09-16T03:37:55.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.457 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.457 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.457 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997263+0000", 2024-09-16T03:37:55.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.373e-06 2024-09-16T03:37:55.457 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.457 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997764+0000", 2024-09-16T03:37:55.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00050155299999999996 2024-09-16T03:37:55.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997775+0000", 2024-09-16T03:37:55.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1066999999999999e-05 2024-09-16T03:37:55.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097841+0000", 2024-09-16T03:37:55.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100066316 2024-09-16T03:37:55.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097882+0000", 2024-09-16T03:37:55.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0850999999999999e-05 2024-09-16T03:37:55.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097904+0000", 2024-09-16T03:37:55.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1347000000000001e-05 2024-09-16T03:37:55.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.099089+0000", 2024-09-16T03:37:55.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001185195 2024-09-16T03:37:55.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.099117+0000", 2024-09-16T03:37:55.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7843000000000001e-05 2024-09-16T03:37:55.460 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.460 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.460 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.15.23:17 3.1 e42/31 3:86477bf9:::smithi15736696-46 low_tier:head v 42'23, mlcod=41'21)", 2024-09-16T03:37:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.187034+0000", 2024-09-16T03:37:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 39.265681892000003, 2024-09-16T03:37:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10100469400000001, 2024-09-16T03:37:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187034+0000", 2024-09-16T03:37:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187034+0000", 2024-09-16T03:37:55.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187035+0000", 2024-09-16T03:37:55.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5400000000000001e-07 2024-09-16T03:37:55.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187042+0000", 2024-09-16T03:37:55.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1729999999999998e-06 2024-09-16T03:37:55.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187063+0000", 2024-09-16T03:37:55.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1160000000000001e-05 2024-09-16T03:37:55.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.187081+0000", 2024-09-16T03:37:55.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7935999999999999e-05 2024-09-16T03:37:55.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.287313+0000", 2024-09-16T03:37:55.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100231325 2024-09-16T03:37:55.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.287345+0000", 2024-09-16T03:37:55.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2119000000000002e-05 2024-09-16T03:37:55.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.288013+0000", 2024-09-16T03:37:55.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00066847800000000002 2024-09-16T03:37:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.288039+0000", 2024-09-16T03:37:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5948999999999998e-05 2024-09-16T03:37:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.5.21:11 3.1 e42/31 3:adc537a6:::smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'28, mlcod=42'24)", 2024-09-16T03:37:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.619416+0000", 2024-09-16T03:37:55.465 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 38.833300221999998, 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100981345, 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619416+0000", 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619416+0000", 2024-09-16T03:37:55.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619418+0000", 2024-09-16T03:37:55.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.99e-06 2024-09-16T03:37:55.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619445+0000", 2024-09-16T03:37:55.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6755e-05 2024-09-16T03:37:55.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.619465+0000", 2024-09-16T03:37:55.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0097000000000001e-05 2024-09-16T03:37:55.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719589+0000", 2024-09-16T03:37:55.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10012367799999999 2024-09-16T03:37:55.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719706+0000", 2024-09-16T03:37:55.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117259 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.719784+0000", 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7714999999999997e-05 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.720366+0000", 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058208999999999995 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.720397+0000", 2024-09-16T03:37:55.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1761000000000003e-05 2024-09-16T03:37:55.470 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.470 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.470 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.470 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:333 4.a e45/32 4:503187b3:::smithi15736696-34:head v 45'29, mlcod=44'27)", 2024-09-16T03:37:55.470 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.003000+0000", 2024-09-16T03:37:55.470 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.449716723999998, 2024-09-16T03:37:55.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10086444999999999, 2024-09-16T03:37:55.470 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.470 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.470 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.470 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003000+0000", 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003000+0000", 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003001+0000", 2024-09-16T03:37:55.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.35e-06 2024-09-16T03:37:55.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003019+0000", 2024-09-16T03:37:55.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8182999999999999e-05 2024-09-16T03:37:55.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003033+0000", 2024-09-16T03:37:55.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4307999999999999e-05 2024-09-16T03:37:55.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.003040+0000", 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4359999999999999e-06 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.103213+0000", 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100172795 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.103232+0000", 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9746999999999999e-05 2024-09-16T03:37:55.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.103822+0000", 2024-09-16T03:37:55.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058940000000000002 2024-09-16T03:37:55.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.103864+0000", 2024-09-16T03:37:55.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2231000000000003e-05 2024-09-16T03:37:55.474 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.474 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.474 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:509 4.f e45/32 4:fa163101:::smithi15736696-10:head v 45'26, mlcod=45'24)", 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:22.024903+0000", 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 33.427812967000001, 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101058093, 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024903+0000", 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024903+0000", 2024-09-16T03:37:55.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024905+0000", 2024-09-16T03:37:55.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1659999999999999e-06 2024-09-16T03:37:55.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024913+0000", 2024-09-16T03:37:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9840000000000001e-06 2024-09-16T03:37:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024923+0000", 2024-09-16T03:37:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0811e-05 2024-09-16T03:37:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024929+0000", 2024-09-16T03:37:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5509999999999999e-06 2024-09-16T03:37:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.125071+0000", 2024-09-16T03:37:55.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10014179300000001 2024-09-16T03:37:55.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.125095+0000", 2024-09-16T03:37:55.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3944999999999999e-05 2024-09-16T03:37:55.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.125926+0000", 2024-09-16T03:37:55.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00083087799999999996 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.125961+0000", 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5964999999999999e-05 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:503 4.a e45/32 4:51338165:::smithi15736696-4:head v 45'30, mlcod=45'28)", 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:22.126147+0000", 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 33.326569223999996, 2024-09-16T03:37:55.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100843981, 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.126147+0000", 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.126147+0000", 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.126148+0000", 2024-09-16T03:37:55.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7800000000000002e-07 2024-09-16T03:37:55.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.126156+0000", 2024-09-16T03:37:55.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0439999999999996e-06 2024-09-16T03:37:55.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.126165+0000", 2024-09-16T03:37:55.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1370000000000008e-06 2024-09-16T03:37:55.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.126172+0000", 2024-09-16T03:37:55.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.878e-06 2024-09-16T03:37:55.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.226374+0000", 2024-09-16T03:37:55.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100201992 2024-09-16T03:37:55.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.226415+0000", 2024-09-16T03:37:55.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1196000000000003e-05 2024-09-16T03:37:55.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.226965+0000", 2024-09-16T03:37:55.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00055020199999999996 2024-09-16T03:37:55.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.226991+0000", 2024-09-16T03:37:55.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6054000000000002e-05 2024-09-16T03:37:55.484 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.484 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.484 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.484 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:779 3.1 e47/31 3:abdaa5c9:::2782d12a88f9842f7f754fca32c4222cf10fb780cd136830d32faaa700e687ba:head v 47'41, mlcod=46'39)", 2024-09-16T03:37:55.484 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:25.588789+0000", 2024-09-16T03:37:55.485 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.863927433000001, 2024-09-16T03:37:55.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100868377, 2024-09-16T03:37:55.485 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.485 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.485 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.485 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.588789+0000", 2024-09-16T03:37:55.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.588789+0000", 2024-09-16T03:37:55.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.588789+0000", 2024-09-16T03:37:55.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1800000000000002e-07 2024-09-16T03:37:55.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.588843+0000", 2024-09-16T03:37:55.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3720000000000001e-05 2024-09-16T03:37:55.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.588851+0000", 2024-09-16T03:37:55.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2500000000000006e-06 2024-09-16T03:37:55.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.588861+0000", 2024-09-16T03:37:55.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4520000000000003e-06 2024-09-16T03:37:55.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.688905+0000", 2024-09-16T03:37:55.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100044786 2024-09-16T03:37:55.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.688923+0000", 2024-09-16T03:37:55.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7408999999999999e-05 2024-09-16T03:37:55.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.689629+0000", 2024-09-16T03:37:55.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00070633499999999999 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:25.689657+0000", 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8107e-05 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.5.21:22 3.1 e47/31 3:adc537a6:::smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 47'46, mlcod=47'42)", 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.686705+0000", 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 28.766011765000002, 2024-09-16T03:37:55.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10113362200000001, 2024-09-16T03:37:55.490 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.490 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.490 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.490 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686705+0000", 2024-09-16T03:37:55.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686705+0000", 2024-09-16T03:37:55.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686705+0000", 2024-09-16T03:37:55.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1699999999999998e-07 2024-09-16T03:37:55.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686709+0000", 2024-09-16T03:37:55.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6279999999999998e-06 2024-09-16T03:37:55.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686716+0000", 2024-09-16T03:37:55.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3730000000000004e-06 2024-09-16T03:37:55.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.786814+0000", 2024-09-16T03:37:55.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100097964 2024-09-16T03:37:55.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.786880+0000", 2024-09-16T03:37:55.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5883999999999995e-05 2024-09-16T03:37:55.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.786909+0000", 2024-09-16T03:37:55.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8764000000000001e-05 2024-09-16T03:37:55.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.787802+0000", 2024-09-16T03:37:55.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00089355100000000002 2024-09-16T03:37:55.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.787838+0000", 2024-09-16T03:37:55.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5840999999999998e-05 2024-09-16T03:37:55.494 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.494 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.495 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.496 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:983 3.1 e50/31 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head v 50'54, mlcod=49'52)", 2024-09-16T03:37:55.496 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:37:55.496 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.386398657000001, 2024-09-16T03:37:55.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.105768445, 2024-09-16T03:37:55.496 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.496 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.496 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.496 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:37:55.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:37:55.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7700000000000001e-07 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.070158+0000", 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0038403170000000002 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.070168+0000", 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9019999999999994e-06 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.070178+0000", 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6860000000000001e-06 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.170328+0000", 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150196 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.170363+0000", 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4588999999999997e-05 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.172053+0000", 2024-09-16T03:37:55.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016902900000000001 2024-09-16T03:37:55.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.172086+0000", 2024-09-16T03:37:55.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3287999999999997e-05 2024-09-16T03:37:55.500 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.500 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.500 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.501 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:988 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'30, mlcod=46'28)", 2024-09-16T03:37:55.501 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.381297+0000", 2024-09-16T03:37:55.501 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.07141957, 2024-09-16T03:37:55.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10422211300000001, 2024-09-16T03:37:55.501 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.501 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.501 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.501 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.381297+0000", 2024-09-16T03:37:55.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.381297+0000", 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.381298+0000", 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5409999999999999e-06 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.382086+0000", 2024-09-16T03:37:55.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078808600000000002 2024-09-16T03:37:55.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.382097+0000", 2024-09-16T03:37:55.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0557000000000001e-05 2024-09-16T03:37:55.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.482188+0000", 2024-09-16T03:37:55.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100091478 2024-09-16T03:37:55.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.482261+0000", 2024-09-16T03:37:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2992999999999997e-05 2024-09-16T03:37:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.482287+0000", 2024-09-16T03:37:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5211999999999998e-05 2024-09-16T03:37:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.485483+0000", 2024-09-16T03:37:55.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031964939999999998 2024-09-16T03:37:55.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.485519+0000", 2024-09-16T03:37:55.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5751999999999999e-05 2024-09-16T03:37:55.505 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.505 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.505 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.506 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1071 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'34, mlcod=50'32)", 2024-09-16T03:37:55.506 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:32.306208+0000", 2024-09-16T03:37:55.506 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.146508441999998, 2024-09-16T03:37:55.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103114758, 2024-09-16T03:37:55.506 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.506 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.506 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.506 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.306208+0000", 2024-09-16T03:37:55.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.306208+0000", 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.306210+0000", 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6619999999999999e-06 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307136+0000", 2024-09-16T03:37:55.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092664400000000001 2024-09-16T03:37:55.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307149+0000", 2024-09-16T03:37:55.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2615e-05 2024-09-16T03:37:55.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307159+0000", 2024-09-16T03:37:55.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0341e-05 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307221+0000", 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1871000000000004e-05 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307239+0000", 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7876999999999998e-05 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409282+0000", 2024-09-16T03:37:55.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102042801 2024-09-16T03:37:55.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409323+0000", 2024-09-16T03:37:55.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0946999999999999e-05 2024-09-16T03:37:55.510 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.510 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.510 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.510 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1072 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'35, mlcod=50'32)", 2024-09-16T03:37:55.510 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.144772779, 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10192738899999999, 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:37:55.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:37:55.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9999999999999996e-07 2024-09-16T03:37:55.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.308383+0000", 2024-09-16T03:37:55.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000438794 2024-09-16T03:37:55.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.308391+0000", 2024-09-16T03:37:55.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6149999999999999e-06 2024-09-16T03:37:55.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.308396+0000", 2024-09-16T03:37:55.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1719999999999998e-06 2024-09-16T03:37:55.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.408530+0000", 2024-09-16T03:37:55.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013438700000001 2024-09-16T03:37:55.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.408547+0000", 2024-09-16T03:37:55.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6554000000000001e-05 2024-09-16T03:37:55.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409844+0000", 2024-09-16T03:37:55.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001297166 2024-09-16T03:37:55.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409871+0000", 2024-09-16T03:37:55.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6900999999999999e-05 2024-09-16T03:37:55.515 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.515 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.515 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.515 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1073 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'36, mlcod=50'32)", 2024-09-16T03:37:55.515 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:32.311182+0000", 2024-09-16T03:37:55.515 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.141534421999999, 2024-09-16T03:37:55.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101901307, 2024-09-16T03:37:55.515 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.515 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.516 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.516 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311182+0000", 2024-09-16T03:37:55.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311182+0000", 2024-09-16T03:37:55.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311183+0000", 2024-09-16T03:37:55.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:37:55.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311720+0000", 2024-09-16T03:37:55.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053692400000000004 2024-09-16T03:37:55.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311728+0000", 2024-09-16T03:37:55.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7849999999999992e-06 2024-09-16T03:37:55.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.411763+0000", 2024-09-16T03:37:55.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100035393 2024-09-16T03:37:55.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.411838+0000", 2024-09-16T03:37:55.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4658000000000002e-05 2024-09-16T03:37:55.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.411853+0000", 2024-09-16T03:37:55.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5268e-05 2024-09-16T03:37:55.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.413058+0000", 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001204751 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.413083+0000", 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5609000000000002e-05 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1193 4.a e54/32 4:503187b3:::smithi15736696-34:head v 54'43, mlcod=53'41)", 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:36.160789+0000", 2024-09-16T03:37:55.523 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 19.291927454, 2024-09-16T03:37:55.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10275106000000001, 2024-09-16T03:37:55.524 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.524 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.524 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.524 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.160789+0000", 2024-09-16T03:37:55.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.160789+0000", 2024-09-16T03:37:55.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.160791+0000", 2024-09-16T03:37:55.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.638e-06 2024-09-16T03:37:55.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.161609+0000", 2024-09-16T03:37:55.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00081820899999999997 2024-09-16T03:37:55.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.161635+0000", 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5939999999999999e-05 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.261770+0000", 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100135416 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.261913+0000", 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000142695 2024-09-16T03:37:55.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.261950+0000", 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7324999999999999e-05 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.263493+0000", 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015430800000000001 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.263540+0000", 2024-09-16T03:37:55.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6757e-05 2024-09-16T03:37:55.528 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.528 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.528 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.528 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1515 3.1 e59/31 3:9c9e970c:::7249cb65b52cadf4fe8f4ec923c7e548d1de7d9628823acd2c8de19121f1c1ce:head v 59'69, mlcod=56'67)", 2024-09-16T03:37:55.528 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:37:55.528 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.501065949000001, 2024-09-16T03:37:55.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.203376156, 2024-09-16T03:37:55.528 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.528 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.528 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.528 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951651+0000", 2024-09-16T03:37:55.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8299999999999996e-07 2024-09-16T03:37:55.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953610+0000", 2024-09-16T03:37:55.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019588769999999999 2024-09-16T03:37:55.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953622+0000", 2024-09-16T03:37:55.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1846e-05 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953632+0000", 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0202e-05 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.053808+0000", 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100175615 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.053846+0000", 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8383000000000002e-05 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.155006+0000", 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101160131 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.155027+0000", 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0319e-05 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.532 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.15.23:70 3.1 e60/31 3:9b343f17:::smithi15736696-1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 60'73, mlcod=60'70)", 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.257925+0000", 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.1947912919999997, 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100795284, 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.257925+0000", 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.257925+0000", 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.257926+0000", 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3600000000000003e-07 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.257932+0000", 2024-09-16T03:37:55.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4069999999999998e-06 2024-09-16T03:37:55.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.257939+0000", 2024-09-16T03:37:55.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.562e-06 2024-09-16T03:37:55.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.257944+0000", 2024-09-16T03:37:55.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2190000000000003e-06 2024-09-16T03:37:55.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.358060+0000", 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100115863 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.358076+0000", 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6099999999999998e-05 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.358682+0000", 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000606297 2024-09-16T03:37:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.358720+0000", 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8099999999999998e-05 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1704 3.1 e65/31 3:b6a94245:::f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99:head v 65'77, mlcod=64'75)", 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.6589306749999997, 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30251757499999998, 2024-09-16T03:37:55.537 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.538 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.538 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.538 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:37:55.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:37:55.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793788+0000", 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794391+0000", 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060354199999999999 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794406+0000", 2024-09-16T03:37:55.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5045000000000001e-05 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794432+0000", 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5996999999999999e-05 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.994704+0000", 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20027177700000001 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.994719+0000", 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5021000000000001e-05 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096277+0000", 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101557437 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096303+0000", 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6732000000000001e-05 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.541 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.542 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:84 3.1 e65/31 3:b24f289d:::e84532c28f467d2f2a6e2cf44904fa0d4a555e1304bbe1b25db645af42152bd7:head v 65'78, mlcod=64'75)", 2024-09-16T03:37:55.542 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:37:55.542 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.6499155119999998, 2024-09-16T03:37:55.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29353782099999998, 2024-09-16T03:37:55.542 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.542 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.542 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.542 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802805+0000", 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1200000000000004e-06 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803910+0000", 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011051660000000001 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803928+0000", 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7880000000000002e-05 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803938+0000", 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6690000000000001e-06 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095473+0000", 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29153552399999999 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095486+0000", 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2455e-05 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096306+0000", 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00082031900000000002 2024-09-16T03:37:55.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096339+0000", 2024-09-16T03:37:55.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2688000000000003e-05 2024-09-16T03:37:55.546 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.546 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.546 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.546 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:85 3.1 e65/31 3:b380f269:::e65351e8a65a0b1a517e3b1a85e04b84c0ab7cfa6655f11fb0da31f56b4dd9af:head v 65'79, mlcod=64'75)", 2024-09-16T03:37:55.546 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:37:55.546 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.6486349340000004, 2024-09-16T03:37:55.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29260815400000001, 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804083+0000", 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804266+0000", 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018268099999999999 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804275+0000", 2024-09-16T03:37:55.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4439999999999997e-06 2024-09-16T03:37:55.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904371+0000", 2024-09-16T03:37:55.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100095276 2024-09-16T03:37:55.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095883+0000", 2024-09-16T03:37:55.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.191512444 2024-09-16T03:37:55.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095901+0000", 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8462999999999999e-05 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096658+0000", 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00075622000000000005 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096690+0000", 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1919999999999999e-05 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.550 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:86 3.1 e65/31 3:b3acb53e:::e951eb69ce5151a3f54107fa22c70f80729eff845da1613447a91536a24046e8:head v 65'80, mlcod=64'75)", 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 4.5483315180000004, 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29244589199999999, 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:37:55.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:37:55.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904386+0000", 2024-09-16T03:37:55.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3e-07 2024-09-16T03:37:55.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904537+0000", 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015122700000000001 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904544+0000", 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.024e-06 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904547+0000", 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6540000000000001e-06 2024-09-16T03:37:55.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196016+0000", 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29146812599999999 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196027+0000", 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1172000000000001e-05 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196791+0000", 2024-09-16T03:37:55.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00076418700000000005 2024-09-16T03:37:55.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196831+0000", 2024-09-16T03:37:55.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9771999999999998e-05 2024-09-16T03:37:55.555 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.555 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.555 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.555 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.555 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.555 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:55.557 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:37:55.558 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:37:55.558 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:55.554+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:55.658 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:55.656+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:55.663 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:55.663 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:55.663 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:37:55.663 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:55.672 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:37:55.758 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:55.757+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:37:55.817 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:55.817 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:55.817 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:37:55.817 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:37:55.817 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:55.825 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:37:55.868 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:55.867+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:55.930 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:55.930 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:37:55.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:37:55.931 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:55.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.931 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:41 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:55.931 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.156949+0000", 2024-09-16T03:37:55.931 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.771297285999999, 2024-09-16T03:37:55.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021202600000000001, 2024-09-16T03:37:55.931 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.931 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.932 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.932 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.932 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156949+0000", 2024-09-16T03:37:55.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156949+0000", 2024-09-16T03:37:55.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156949+0000", 2024-09-16T03:37:55.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9100000000000004e-07 2024-09-16T03:37:55.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157324+0000", 2024-09-16T03:37:55.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037452500000000001 2024-09-16T03:37:55.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157340+0000", 2024-09-16T03:37:55.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6426000000000002e-05 2024-09-16T03:37:55.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157350+0000", 2024-09-16T03:37:55.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6150000000000003e-06 2024-09-16T03:37:55.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157415+0000", 2024-09-16T03:37:55.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5425999999999993e-05 2024-09-16T03:37:55.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157441+0000", 2024-09-16T03:37:55.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6177999999999999e-05 2024-09-16T03:37:55.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.159020+0000", 2024-09-16T03:37:55.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001578325 2024-09-16T03:37:55.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.159069+0000", 2024-09-16T03:37:55.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9274000000000003e-05 2024-09-16T03:37:55.936 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.936 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.936 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.937 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:37:55.937 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:37:55.937 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.742165575999998, 2024-09-16T03:37:55.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102630163, 2024-09-16T03:37:55.937 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.937 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.937 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.937 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:37:55.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:37:55.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186081+0000", 2024-09-16T03:37:55.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5600000000000005e-07 2024-09-16T03:37:55.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186970+0000", 2024-09-16T03:37:55.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00088889000000000004 2024-09-16T03:37:55.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186979+0000", 2024-09-16T03:37:55.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0620000000000007e-06 2024-09-16T03:37:55.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186986+0000", 2024-09-16T03:37:55.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:37:55.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287184+0000", 2024-09-16T03:37:55.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100198036 2024-09-16T03:37:55.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287199+0000", 2024-09-16T03:37:55.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5458999999999999e-05 2024-09-16T03:37:55.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288659+0000", 2024-09-16T03:37:55.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014601900000000001 2024-09-16T03:37:55.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288711+0000", 2024-09-16T03:37:55.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1088000000000003e-05 2024-09-16T03:37:55.942 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.942 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.942 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.942 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:55.942 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:37:55.942 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.741129143000002, 2024-09-16T03:37:55.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10205370900000001, 2024-09-16T03:37:55.943 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.943 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.943 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.943 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:37:55.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:37:55.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187118+0000", 2024-09-16T03:37:55.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8599999999999998e-07 2024-09-16T03:37:55.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188062+0000", 2024-09-16T03:37:55.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094447300000000004 2024-09-16T03:37:55.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188070+0000", 2024-09-16T03:37:55.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.695e-06 2024-09-16T03:37:55.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188074+0000", 2024-09-16T03:37:55.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0709999999999996e-06 2024-09-16T03:37:55.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287842+0000", 2024-09-16T03:37:55.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099768478999999993 2024-09-16T03:37:55.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287864+0000", 2024-09-16T03:37:55.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1679999999999999e-05 2024-09-16T03:37:55.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289137+0000", 2024-09-16T03:37:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012732430000000001 2024-09-16T03:37:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289171+0000", 2024-09-16T03:37:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3382e-05 2024-09-16T03:37:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:42 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'2, mlcod=0'0)", 2024-09-16T03:37:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:37:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.671047235000003, 2024-09-16T03:37:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10172858, 2024-09-16T03:37:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:37:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:37:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257201+0000", 2024-09-16T03:37:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:37:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257536+0000", 2024-09-16T03:37:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033513899999999998 2024-09-16T03:37:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257555+0000", 2024-09-16T03:37:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8547999999999999e-05 2024-09-16T03:37:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257566+0000", 2024-09-16T03:37:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1379999999999999e-05 2024-09-16T03:37:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357720+0000", 2024-09-16T03:37:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100153532 2024-09-16T03:37:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357734+0000", 2024-09-16T03:37:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4491000000000001e-05 2024-09-16T03:37:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358896+0000", 2024-09-16T03:37:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011620599999999999 2024-09-16T03:37:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358927+0000", 2024-09-16T03:37:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1158999999999998e-05 2024-09-16T03:37:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:43 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'3, mlcod=0'0)", 2024-09-16T03:37:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:37:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.669263323000003, 2024-09-16T03:37:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101764453, 2024-09-16T03:37:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:37:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:37:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258984+0000", 2024-09-16T03:37:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5689999999999999e-06 2024-09-16T03:37:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258994+0000", 2024-09-16T03:37:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259008+0000", 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4151999999999999e-05 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359073+0000", 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100065628 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359110+0000", 2024-09-16T03:37:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7045999999999997e-05 2024-09-16T03:37:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359124+0000", 2024-09-16T03:37:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.382e-05 2024-09-16T03:37:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360716+0000", 2024-09-16T03:37:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015919829999999999 2024-09-16T03:37:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360747+0000", 2024-09-16T03:37:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0874999999999998e-05 2024-09-16T03:37:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:62 4.5 e39/32 4:ad0b9852:::smithi15736696-16:head v 39'5, mlcod=0'0)", 2024-09-16T03:37:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.359222+0000", 2024-09-16T03:37:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.569023927000003, 2024-09-16T03:37:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020277289999999998, 2024-09-16T03:37:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359222+0000", 2024-09-16T03:37:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359222+0000", 2024-09-16T03:37:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359222+0000", 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3999999999999997e-07 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359341+0000", 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011825899999999999 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359345+0000", 2024-09-16T03:37:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1180000000000002e-06 2024-09-16T03:37:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359349+0000", 2024-09-16T03:37:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.941e-06 2024-09-16T03:37:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359991+0000", 2024-09-16T03:37:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00064214700000000003 2024-09-16T03:37:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360010+0000", 2024-09-16T03:37:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9028000000000001e-05 2024-09-16T03:37:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.361209+0000", 2024-09-16T03:37:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011994460000000001 2024-09-16T03:37:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.361250+0000", 2024-09-16T03:37:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0450000000000001e-05 2024-09-16T03:37:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:63 4.5 e39/32 4:ad0b9852:::smithi15736696-16:head v 39'6, mlcod=0'0)", 2024-09-16T03:37:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.359355+0000", 2024-09-16T03:37:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.568891291, 2024-09-16T03:37:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019271519999999999, 2024-09-16T03:37:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359355+0000", 2024-09-16T03:37:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359355+0000", 2024-09-16T03:37:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359355+0000", 2024-09-16T03:37:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.22e-07 2024-09-16T03:37:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359357+0000", 2024-09-16T03:37:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.187e-06 2024-09-16T03:37:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359360+0000", 2024-09-16T03:37:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.745e-06 2024-09-16T03:37:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359363+0000", 2024-09-16T03:37:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7039999999999999e-06 2024-09-16T03:37:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360603+0000", 2024-09-16T03:37:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012399850000000001 2024-09-16T03:37:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360614+0000", 2024-09-16T03:37:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1599e-05 2024-09-16T03:37:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.361254+0000", 2024-09-16T03:37:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063991999999999999 2024-09-16T03:37:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.361282+0000", 2024-09-16T03:37:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7690000000000001e-05 2024-09-16T03:37:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:492 4.5 e45/32 4:a11e1d7c:::smithi15736696-43:head v 45'22, mlcod=45'20)", 2024-09-16T03:37:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:37:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.104895036000002, 2024-09-16T03:37:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100797151, 2024-09-16T03:37:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:37:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:37:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823353+0000", 2024-09-16T03:37:55.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2450000000000001e-06 2024-09-16T03:37:55.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823363+0000", 2024-09-16T03:37:55.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0074999999999999e-05 2024-09-16T03:37:55.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823378+0000", 2024-09-16T03:37:55.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4534e-05 2024-09-16T03:37:55.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923461+0000", 2024-09-16T03:37:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100082859 2024-09-16T03:37:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923525+0000", 2024-09-16T03:37:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4616999999999995e-05 2024-09-16T03:37:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923549+0000", 2024-09-16T03:37:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.357e-05 2024-09-16T03:37:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924106+0000", 2024-09-16T03:37:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000556753 2024-09-16T03:37:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924148+0000", 2024-09-16T03:37:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2497999999999999e-05 2024-09-16T03:37:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:805 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'24, mlcod=45'22)", 2024-09-16T03:37:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.012699+0000", 2024-09-16T03:37:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.915547313000001, 2024-09-16T03:37:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021199320000000002, 2024-09-16T03:37:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.012699+0000", 2024-09-16T03:37:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.012699+0000", 2024-09-16T03:37:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.012700+0000", 2024-09-16T03:37:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4300000000000001e-06 2024-09-16T03:37:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.012997+0000", 2024-09-16T03:37:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029683299999999998 2024-09-16T03:37:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.013012+0000", 2024-09-16T03:37:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4636e-05 2024-09-16T03:37:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.013019+0000", 2024-09-16T03:37:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3730000000000004e-06 2024-09-16T03:37:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.013081+0000", 2024-09-16T03:37:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2349999999999998e-05 2024-09-16T03:37:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.013098+0000", 2024-09-16T03:37:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6694e-05 2024-09-16T03:37:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.014776+0000", 2024-09-16T03:37:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016784770000000001 2024-09-16T03:37:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.014819+0000", 2024-09-16T03:37:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2138999999999998e-05 2024-09-16T03:37:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:806 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'25, mlcod=45'22)", 2024-09-16T03:37:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.019370+0000", 2024-09-16T03:37:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.908875709, 2024-09-16T03:37:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048461379999999998, 2024-09-16T03:37:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019370+0000", 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019370+0000", 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019372+0000", 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.424e-06 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019541+0000", 2024-09-16T03:37:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016961999999999999 2024-09-16T03:37:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019553+0000", 2024-09-16T03:37:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1678000000000001e-05 2024-09-16T03:37:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019561+0000", 2024-09-16T03:37:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7579999999999992e-06 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022299+0000", 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002738061 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022326+0000", 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7515999999999999e-05 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024179+0000", 2024-09-16T03:37:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001853005 2024-09-16T03:37:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024216+0000", 2024-09-16T03:37:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7076000000000002e-05 2024-09-16T03:37:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:807 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'26, mlcod=45'22)", 2024-09-16T03:37:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.022119+0000", 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.906126879999999, 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023221940000000001, 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022119+0000", 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022119+0000", 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022120+0000", 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3e-06 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022193+0000", 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2364e-05 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022210+0000", 2024-09-16T03:37:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6773999999999999e-05 2024-09-16T03:37:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022221+0000", 2024-09-16T03:37:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1087000000000001e-05 2024-09-16T03:37:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023441+0000", 2024-09-16T03:37:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001220524 2024-09-16T03:37:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023470+0000", 2024-09-16T03:37:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8541e-05 2024-09-16T03:37:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024418+0000", 2024-09-16T03:37:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094820699999999996 2024-09-16T03:37:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024441+0000", 2024-09-16T03:37:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3397e-05 2024-09-16T03:37:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:808 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'27, mlcod=45'22)", 2024-09-16T03:37:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:37:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.905201761000001, 2024-09-16T03:37:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.097081386000000006, 2024-09-16T03:37:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:37:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:37:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023046+0000", 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1059999999999998e-06 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023056+0000", 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3719999999999994e-06 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023067+0000", 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1246999999999999e-05 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023075+0000", 2024-09-16T03:37:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4710000000000007e-06 2024-09-16T03:37:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.119708+0000", 2024-09-16T03:37:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096632179999999998 2024-09-16T03:37:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.119725+0000", 2024-09-16T03:37:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7754999999999999e-05 2024-09-16T03:37:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.120092+0000", 2024-09-16T03:37:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00036668099999999999 2024-09-16T03:37:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.120126+0000", 2024-09-16T03:37:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3574000000000001e-05 2024-09-16T03:37:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1364 4.5 e57/50 4:a4fbca73:::smithi15736696-39:head v 57'36, mlcod=57'33)", 2024-09-16T03:37:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.279986+0000", 2024-09-16T03:37:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.648259506, 2024-09-16T03:37:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021206100000000002, 2024-09-16T03:37:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.279986+0000", 2024-09-16T03:37:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.279986+0000", 2024-09-16T03:37:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.279988+0000", 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.531e-06 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280208+0000", 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0002195 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280218+0000", 2024-09-16T03:37:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0722e-05 2024-09-16T03:37:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280226+0000", 2024-09-16T03:37:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8949999999999998e-06 2024-09-16T03:37:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280280+0000", 2024-09-16T03:37:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4132000000000003e-05 2024-09-16T03:37:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280298+0000", 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7371999999999999e-05 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282080+0000", 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0017824099999999999 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282107+0000", 2024-09-16T03:37:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7047999999999999e-05 2024-09-16T03:37:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1365 4.5 e57/50 4:a4fbca73:::smithi15736696-39:head v 57'37, mlcod=57'33)", 2024-09-16T03:37:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.280249+0000", 2024-09-16T03:37:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.647997033999999, 2024-09-16T03:37:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024448349999999998, 2024-09-16T03:37:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280249+0000", 2024-09-16T03:37:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280249+0000", 2024-09-16T03:37:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280250+0000", 2024-09-16T03:37:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2600000000000002e-07 2024-09-16T03:37:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280485+0000", 2024-09-16T03:37:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023584999999999999 2024-09-16T03:37:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280494+0000", 2024-09-16T03:37:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2349999999999999e-06 2024-09-16T03:37:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280510+0000", 2024-09-16T03:37:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6626999999999999e-05 2024-09-16T03:37:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.281163+0000", 2024-09-16T03:37:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00065277800000000002 2024-09-16T03:37:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.281177+0000", 2024-09-16T03:37:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4164e-05 2024-09-16T03:37:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282658+0000", 2024-09-16T03:37:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001480594 2024-09-16T03:37:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282694+0000", 2024-09-16T03:37:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5960999999999997e-05 2024-09-16T03:37:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1366 4.5 e57/50 4:a4fbca73:::smithi15736696-39:head v 57'38, mlcod=57'33)", 2024-09-16T03:37:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.280525+0000", 2024-09-16T03:37:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.647721454999999, 2024-09-16T03:37:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022401679999999998, 2024-09-16T03:37:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280525+0000", 2024-09-16T03:37:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280525+0000", 2024-09-16T03:37:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280525+0000", 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280529+0000", 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.42e-06 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280536+0000", 2024-09-16T03:37:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.241e-06 2024-09-16T03:37:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280540+0000", 2024-09-16T03:37:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.561e-06 2024-09-16T03:37:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.281977+0000", 2024-09-16T03:37:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014370909999999999 2024-09-16T03:37:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.281993+0000", 2024-09-16T03:37:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5659e-05 2024-09-16T03:37:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282730+0000", 2024-09-16T03:37:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073740000000000003 2024-09-16T03:37:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282765+0000", 2024-09-16T03:37:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4335999999999998e-05 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:53'30,epoch:57/50,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.52954342, 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100230893, 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:37:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:37:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398705+0000", 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5689999999999999e-06 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398722+0000", 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6558e-05 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398735+0000", 2024-09-16T03:37:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3499999999999999e-05 2024-09-16T03:37:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398746+0000", 2024-09-16T03:37:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1182999999999999e-05 2024-09-16T03:37:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498898+0000", 2024-09-16T03:37:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015112599999999 2024-09-16T03:37:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498906+0000", 2024-09-16T03:37:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3189999999999995e-06 2024-09-16T03:37:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498933+0000", 2024-09-16T03:37:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7637999999999999e-05 2024-09-16T03:37:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1570 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'31, mlcod=48'29)", 2024-09-16T03:37:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:37:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7539364249999991, 2024-09-16T03:37:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10253683500000001, 2024-09-16T03:37:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:37:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:37:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:37:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8500000000000002e-07 2024-09-16T03:37:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174689+0000", 2024-09-16T03:37:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037958600000000001 2024-09-16T03:37:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174699+0000", 2024-09-16T03:37:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0015e-05 2024-09-16T03:37:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174706+0000", 2024-09-16T03:37:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6250000000000001e-06 2024-09-16T03:37:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274865+0000", 2024-09-16T03:37:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100159157 2024-09-16T03:37:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274879+0000", 2024-09-16T03:37:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3983e-05 2024-09-16T03:37:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276820+0000", 2024-09-16T03:37:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001941248 2024-09-16T03:37:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276846+0000", 2024-09-16T03:37:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5936e-05 2024-09-16T03:37:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1571 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'32, mlcod=48'29)", 2024-09-16T03:37:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:37:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.753485671, 2024-09-16T03:37:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10249543, 2024-09-16T03:37:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:37:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:37:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174761+0000", 2024-09-16T03:37:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0800000000000004e-07 2024-09-16T03:37:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175053+0000", 2024-09-16T03:37:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029151600000000002 2024-09-16T03:37:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175063+0000", 2024-09-16T03:37:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0538e-05 2024-09-16T03:37:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175070+0000", 2024-09-16T03:37:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7819999999999996e-06 2024-09-16T03:37:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275895+0000", 2024-09-16T03:37:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10082516 2024-09-16T03:37:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275909+0000", 2024-09-16T03:37:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4197e-05 2024-09-16T03:37:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277232+0000", 2024-09-16T03:37:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013231009999999999 2024-09-16T03:37:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277256+0000", 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3428e-05 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1572 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'33, mlcod=48'29)", 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7531443709999994, 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10216886, 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:37:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8400000000000004e-07 2024-09-16T03:37:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175107+0000", 2024-09-16T03:37:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.578e-06 2024-09-16T03:37:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175114+0000", 2024-09-16T03:37:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.888e-06 2024-09-16T03:37:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175119+0000", 2024-09-16T03:37:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.074e-06 2024-09-16T03:37:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276662+0000", 2024-09-16T03:37:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101542753 2024-09-16T03:37:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276674+0000", 2024-09-16T03:37:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1960000000000001e-05 2024-09-16T03:37:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277258+0000", 2024-09-16T03:37:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058431900000000003 2024-09-16T03:37:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277270+0000", 2024-09-16T03:37:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2604e-05 2024-09-16T03:37:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:60'33,epoch:61/50,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:37:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:37:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.2512231170000003, 2024-09-16T03:37:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10023048699999999, 2024-09-16T03:37:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:37:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:37:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:37:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8599999999999999e-07 2024-09-16T03:37:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677027+0000", 2024-09-16T03:37:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7359999999999999e-06 2024-09-16T03:37:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677042+0000", 2024-09-16T03:37:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4888e-05 2024-09-16T03:37:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677052+0000", 2024-09-16T03:37:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7750000000000004e-06 2024-09-16T03:37:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777182+0000", 2024-09-16T03:37:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100130394 2024-09-16T03:37:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777195+0000", 2024-09-16T03:37:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3373999999999999e-05 2024-09-16T03:37:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777253+0000", 2024-09-16T03:37:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7933999999999998e-05 2024-09-16T03:37:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.053 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:56.055 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:37:56.056 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:56.053+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:56.156 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:56.154+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:56.202 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:56.202 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:56.202 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:37:56.203 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:56.211 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:37:56.257 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:56.256+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:37:56.262 INFO:tasks.rados.rados.0.smithi157.stdout:920: set_chunk oid 27 offset: 220672 length: 1659392 target oid 5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6 offset: 0 snap -1 2024-09-16T03:37:56.262 INFO:tasks.rados.rados.0.smithi157.stdout:921: read oid 50 snap -1 2024-09-16T03:37:56.262 INFO:tasks.rados.rados.0.smithi157.stdout:921: expect (ObjNum 104 snap 0 seq_num 104) 2024-09-16T03:37:56.262 INFO:tasks.rados.rados.0.smithi157.stdout:922: set_chunk oid 33 target oid 2024-09-16T03:37:56.315 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:56.315 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:56.316 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:37:56.316 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:37:56.316 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:56.324 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:37:56.339 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:56.337+0000 7efef16d9640 -1 osd.13 70 handle_osd_map injecting map crc failure 2024-09-16T03:37:56.367 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:56.366+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:56.369 INFO:tasks.rados.rados.0.smithi157.stdout:922: set_chunk oid 33 offset: 1819648 length: 122880 target oid 6b81d68e6e96bdaff988780d892ba62d7a8785341fee4052637efd4986c93f11 offset: 0 snap -1 2024-09-16T03:37:56.369 INFO:tasks.rados.rados.0.smithi157.stdout:923: write oid 19 current snap is 0 2024-09-16T03:37:56.369 INFO:tasks.rados.rados.0.smithi157.stdout:914: finishing write tid 3 to smithi15736696-18 2024-09-16T03:37:56.369 INFO:tasks.rados.rados.0.smithi157.stdout:914: finishing write tid 4 to smithi15736696-18 2024-09-16T03:37:56.370 INFO:tasks.rados.rados.0.smithi157.stdout:914: oid 18 updating version 0 to 49 2024-09-16T03:37:56.370 INFO:tasks.rados.rados.0.smithi157.stdout:914: oid 18 updating version 49 to 50 2024-09-16T03:37:56.370 INFO:tasks.rados.rados.0.smithi157.stdout:914: oid 18 version 50 is already newer than 48 2024-09-16T03:37:56.370 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 50 (ObjNum 141 snap 0 seq_num 141) dirty exists 2024-09-16T03:37:56.370 INFO:tasks.rados.rados.0.smithi157.stdout:914: left oid 18 (ObjNum 141 snap 0 seq_num 141) 2024-09-16T03:37:56.370 INFO:tasks.rados.rados.0.smithi157.stdout:923: seq_num 143 ranges {403401=454961,1584925=506039} 2024-09-16T03:37:56.373 INFO:tasks.rados.rados.0.smithi157.stdout:923: writing smithi15736696-19 from 403401 to 858362 tid 1 2024-09-16T03:37:56.381 INFO:tasks.rados.rados.0.smithi157.stdout:923: writing smithi15736696-19 from 1584925 to 2090964 tid 2 2024-09-16T03:37:56.381 INFO:tasks.rados.rados.0.smithi157.stdout:914: done (5 left) 2024-09-16T03:37:56.381 INFO:tasks.rados.rados.0.smithi157.stdout:924: read oid 24 snap -1 2024-09-16T03:37:56.381 INFO:tasks.rados.rados.0.smithi157.stdout:924: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:37:56.381 INFO:tasks.rados.rados.0.smithi157.stdout:925: read oid 46 snap -1 2024-09-16T03:37:56.381 INFO:tasks.rados.rados.0.smithi157.stdout:925: expect (ObjNum 97 snap 0 seq_num 97) 2024-09-16T03:37:56.381 INFO:tasks.rados.rados.0.smithi157.stdout:926: delete oid 22 current snap is 0 2024-09-16T03:37:56.382 INFO:tasks.rados.rados.0.smithi157.stdout:919: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:37:56.382 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:56.433 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:56.433 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:37:56.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:37:56.433 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:56.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.433 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T03:37:56.433 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:37:56.434 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 65.353168756000002, 2024-09-16T03:37:56.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.88203620400000005, 2024-09-16T03:37:56.434 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.434 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.434 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.434 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.434 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.434 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.434 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:37:56.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.434 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:37:56.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:37:56.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076767+0000", 2024-09-16T03:37:56.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9300000000000002e-06 2024-09-16T03:37:56.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076774+0000", 2024-09-16T03:37:56.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6630000000000004e-06 2024-09-16T03:37:56.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076785+0000", 2024-09-16T03:37:56.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1221e-05 2024-09-16T03:37:56.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076796+0000", 2024-09-16T03:37:56.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0428000000000001e-05 2024-09-16T03:37:56.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076856+0000", 2024-09-16T03:37:56.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:37:56.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:37:56.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076863+0000", 2024-09-16T03:37:56.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9840000000000004e-06 2024-09-16T03:37:56.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855770+0000", 2024-09-16T03:37:56.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.77890736999999999 2024-09-16T03:37:56.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855902+0000", 2024-09-16T03:37:56.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013207 2024-09-16T03:37:56.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.856054+0000", 2024-09-16T03:37:56.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015220899999999999 2024-09-16T03:37:56.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.858407+0000", 2024-09-16T03:37:56.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023530069999999998 2024-09-16T03:37:56.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958598+0000", 2024-09-16T03:37:56.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100190194 2024-09-16T03:37:56.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958716+0000", 2024-09-16T03:37:56.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118589 2024-09-16T03:37:56.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958761+0000", 2024-09-16T03:37:56.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5120000000000002e-05 2024-09-16T03:37:56.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958802+0000", 2024-09-16T03:37:56.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0383999999999998e-05 2024-09-16T03:37:56.441 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.441 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.441 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.441 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:37:56.441 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:37:56.441 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 61.834752579000003, 2024-09-16T03:37:56.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70508991300000001, 2024-09-16T03:37:56.441 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.441 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.441 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 52 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6300000000000005e-07 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595187+0000", 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9790000000000001e-06 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595194+0000", 2024-09-16T03:37:56.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6490000000000002e-06 2024-09-16T03:37:56.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595208+0000", 2024-09-16T03:37:56.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4139e-05 2024-09-16T03:37:56.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801931+0000", 2024-09-16T03:37:56.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.206723083 2024-09-16T03:37:56.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801990+0000", 2024-09-16T03:37:56.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8625999999999997e-05 2024-09-16T03:37:56.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802066+0000", 2024-09-16T03:37:56.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6553999999999999e-05 2024-09-16T03:37:56.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802504+0000", 2024-09-16T03:37:56.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043775200000000002 2024-09-16T03:37:56.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.199998+0000", 2024-09-16T03:37:56.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39749365599999997 2024-09-16T03:37:56.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300151+0000", 2024-09-16T03:37:56.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015310099999999 2024-09-16T03:37:56.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300210+0000", 2024-09-16T03:37:56.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9376999999999997e-05 2024-09-16T03:37:56.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300271+0000", 2024-09-16T03:37:56.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1334000000000006e-05 2024-09-16T03:37:56.447 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.447 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.447 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.448 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:69 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.448 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.369956+0000", 2024-09-16T03:37:56.448 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.059978264999998, 2024-09-16T03:37:56.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60610467099999998, 2024-09-16T03:37:56.448 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.448 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.448 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.448 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.448 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.448 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.448 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 69 2024-09-16T03:37:56.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.448 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.369956+0000", 2024-09-16T03:37:56.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.369956+0000", 2024-09-16T03:37:56.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.369957+0000", 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3099999999999997e-07 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370028+0000", 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1404999999999998e-05 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370036+0000", 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.79e-06 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370042+0000", 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.494e-06 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572455+0000", 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20241295100000001 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572512+0000", 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7071000000000003e-05 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572602+0000", 2024-09-16T03:37:56.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9275000000000002e-05 2024-09-16T03:37:56.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673919+0000", 2024-09-16T03:37:56.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10131736099999999 2024-09-16T03:37:56.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674651+0000", 2024-09-16T03:37:56.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073190500000000003 2024-09-16T03:37:56.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975976+0000", 2024-09-16T03:37:56.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.301325391 2024-09-16T03:37:56.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976014+0000", 2024-09-16T03:37:56.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8226000000000003e-05 2024-09-16T03:37:56.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976061+0000", 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6071e-05 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.059882807000001, 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60616296300000005, 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.454 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.455 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.455 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.455 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.455 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 70 2024-09-16T03:37:56.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.455 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:37:56.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:37:56.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370052+0000", 2024-09-16T03:37:56.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8899999999999999e-07 2024-09-16T03:37:56.457 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.457 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.457 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370055+0000", 2024-09-16T03:37:56.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7709999999999998e-06 2024-09-16T03:37:56.457 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.457 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.457 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370060+0000", 2024-09-16T03:37:56.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6910000000000002e-06 2024-09-16T03:37:56.457 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.457 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370064+0000", 2024-09-16T03:37:56.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.619e-06 2024-09-16T03:37:56.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572988+0000", 2024-09-16T03:37:56.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20292439300000001 2024-09-16T03:37:56.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.573049+0000", 2024-09-16T03:37:56.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0418999999999999e-05 2024-09-16T03:37:56.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.573133+0000", 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4324000000000001e-05 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674789+0000", 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101655658 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.875723+0000", 2024-09-16T03:37:56.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20093406699999999 2024-09-16T03:37:56.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976140+0000", 2024-09-16T03:37:56.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10041718199999999 2024-09-16T03:37:56.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976172+0000", 2024-09-16T03:37:56.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2169000000000003e-05 2024-09-16T03:37:56.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976214+0000", 2024-09-16T03:37:56.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2080999999999999e-05 2024-09-16T03:37:56.461 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.461 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.461 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.461 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.461 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:37:56.462 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.059861595999998, 2024-09-16T03:37:56.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60623559900000001, 2024-09-16T03:37:56.462 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.462 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.462 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.462 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.462 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.462 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.462 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 71 2024-09-16T03:37:56.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.463 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:37:56.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:37:56.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:37:56.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:37:56.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370077+0000", 2024-09-16T03:37:56.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9940000000000002e-06 2024-09-16T03:37:56.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370081+0000", 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5850000000000001e-06 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370087+0000", 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673522+0000", 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.303434969 2024-09-16T03:37:56.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673567+0000", 2024-09-16T03:37:56.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5080000000000002e-05 2024-09-16T03:37:56.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673651+0000", 2024-09-16T03:37:56.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3652999999999993e-05 2024-09-16T03:37:56.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775115+0000", 2024-09-16T03:37:56.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101464178 2024-09-16T03:37:56.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975918+0000", 2024-09-16T03:37:56.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20080318699999999 2024-09-16T03:37:56.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976254+0000", 2024-09-16T03:37:56.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033619200000000002 2024-09-16T03:37:56.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976285+0000", 2024-09-16T03:37:56.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0679000000000002e-05 2024-09-16T03:37:56.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976308+0000", 2024-09-16T03:37:56.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3051e-05 2024-09-16T03:37:56.469 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.469 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.469 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.469 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.469 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:37:56.469 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.059838235999997, 2024-09-16T03:37:56.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60632925100000001, 2024-09-16T03:37:56.469 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.469 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.469 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.469 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.470 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.470 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.470 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 72 2024-09-16T03:37:56.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.470 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.470 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.470 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:37:56.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.470 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.470 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:37:56.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:37:56.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:37:56.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370150+0000", 2024-09-16T03:37:56.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3906999999999997e-05 2024-09-16T03:37:56.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370157+0000", 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370162+0000", 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7249999999999997e-06 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673969+0000", 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30380713199999998 2024-09-16T03:37:56.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674030+0000", 2024-09-16T03:37:56.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1141000000000003e-05 2024-09-16T03:37:56.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674124+0000", 2024-09-16T03:37:56.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3859999999999999e-05 2024-09-16T03:37:56.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775687+0000", 2024-09-16T03:37:56.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10156293299999999 2024-09-16T03:37:56.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976103+0000", 2024-09-16T03:37:56.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20041631400000001 2024-09-16T03:37:56.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976355+0000", 2024-09-16T03:37:56.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000251611 2024-09-16T03:37:56.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976385+0000", 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0883999999999998e-05 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976425+0000", 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9666000000000003e-05 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.059765751999997, 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60763126999999995, 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 73 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:37:56.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:37:56.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370169+0000", 2024-09-16T03:37:56.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6199999999999998e-07 2024-09-16T03:37:56.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370171+0000", 2024-09-16T03:37:56.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:37:56.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370175+0000", 2024-09-16T03:37:56.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.596e-06 2024-09-16T03:37:56.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470295+0000", 2024-09-16T03:37:56.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100119967 2024-09-16T03:37:56.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775148+0000", 2024-09-16T03:37:56.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30485311799999998 2024-09-16T03:37:56.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775203+0000", 2024-09-16T03:37:56.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4592999999999997e-05 2024-09-16T03:37:56.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775293+0000", 2024-09-16T03:37:56.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0859999999999994e-05 2024-09-16T03:37:56.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976223+0000", 2024-09-16T03:37:56.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20092996599999999 2024-09-16T03:37:56.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976463+0000", 2024-09-16T03:37:56.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000239695 2024-09-16T03:37:56.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977730+0000", 2024-09-16T03:37:56.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012667690000000001 2024-09-16T03:37:56.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977773+0000", 2024-09-16T03:37:56.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2604000000000001e-05 2024-09-16T03:37:56.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977800+0000", 2024-09-16T03:37:56.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7137000000000001e-05 2024-09-16T03:37:56.482 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.482 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.482 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.482 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.482 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:37:56.482 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.959626897, 2024-09-16T03:37:56.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60633810099999996, 2024-09-16T03:37:56.482 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.483 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.483 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.483 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.483 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.483 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.483 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 74 2024-09-16T03:37:56.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.483 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:37:56.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:37:56.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470308+0000", 2024-09-16T03:37:56.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0950000000000001e-06 2024-09-16T03:37:56.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470313+0000", 2024-09-16T03:37:56.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:37:56.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470320+0000", 2024-09-16T03:37:56.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5089999999999999e-06 2024-09-16T03:37:56.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470327+0000", 2024-09-16T03:37:56.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:37:56.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976483+0000", 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50615579499999996 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976533+0000", 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0096000000000003e-05 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976643+0000", 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109481 2024-09-16T03:37:56.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977808+0000", 2024-09-16T03:37:56.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011649869999999999 2024-09-16T03:37:56.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.981699+0000", 2024-09-16T03:37:56.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003891579 2024-09-16T03:37:56.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076496+0000", 2024-09-16T03:37:56.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.094797333999999997 2024-09-16T03:37:56.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076583+0000", 2024-09-16T03:37:56.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6571999999999997e-05 2024-09-16T03:37:56.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076645+0000", 2024-09-16T03:37:56.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2331999999999999e-05 2024-09-16T03:37:56.488 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.488 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.489 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.489 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.489 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:37:56.489 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.959596761999997, 2024-09-16T03:37:56.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71251765099999997, 2024-09-16T03:37:56.489 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.489 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.489 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.489 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.489 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.489 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 75 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470338+0000", 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2500000000000005e-07 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470389+0000", 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.134e-05 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470397+0000", 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8140000000000001e-06 2024-09-16T03:37:56.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570537+0000", 2024-09-16T03:37:56.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100139425 2024-09-16T03:37:56.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977018+0000", 2024-09-16T03:37:56.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40648102899999999 2024-09-16T03:37:56.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977063+0000", 2024-09-16T03:37:56.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5247000000000002e-05 2024-09-16T03:37:56.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977143+0000", 2024-09-16T03:37:56.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0054000000000003e-05 2024-09-16T03:37:56.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.978132+0000", 2024-09-16T03:37:56.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00098896399999999995 2024-09-16T03:37:56.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182725+0000", 2024-09-16T03:37:56.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.204592626 2024-09-16T03:37:56.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182783+0000", 2024-09-16T03:37:56.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8285000000000002e-05 2024-09-16T03:37:56.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182825+0000", 2024-09-16T03:37:56.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2398999999999998e-05 2024-09-16T03:37:56.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182855+0000", 2024-09-16T03:37:56.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9842999999999999e-05 2024-09-16T03:37:56.495 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.495 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.495 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.495 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.495 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:37:56.495 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.859370194999997, 2024-09-16T03:37:56.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71263271800000005, 2024-09-16T03:37:56.496 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.496 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.496 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.496 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.496 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.496 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.496 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 76 2024-09-16T03:37:56.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.496 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:37:56.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:37:56.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570566+0000", 2024-09-16T03:37:56.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:37:56.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570577+0000", 2024-09-16T03:37:56.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0672999999999999e-05 2024-09-16T03:37:56.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570593+0000", 2024-09-16T03:37:56.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5979999999999999e-05 2024-09-16T03:37:56.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570604+0000", 2024-09-16T03:37:56.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.077e-05 2024-09-16T03:37:56.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979890+0000", 2024-09-16T03:37:56.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40928624200000002 2024-09-16T03:37:56.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979945+0000", 2024-09-16T03:37:56.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5651e-05 2024-09-16T03:37:56.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980038+0000", 2024-09-16T03:37:56.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2603999999999996e-05 2024-09-16T03:37:56.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980669+0000", 2024-09-16T03:37:56.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063121900000000003 2024-09-16T03:37:56.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283050+0000", 2024-09-16T03:37:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30238120800000001 2024-09-16T03:37:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283132+0000", 2024-09-16T03:37:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1841999999999993e-05 2024-09-16T03:37:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283168+0000", 2024-09-16T03:37:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5438999999999997e-05 2024-09-16T03:37:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283197+0000", 2024-09-16T03:37:56.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8960000000000001e-05 2024-09-16T03:37:56.502 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.502 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.502 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.502 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.502 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:37:56.503 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.044835096, 2024-09-16T03:37:56.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70799990400000001, 2024-09-16T03:37:56.503 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.503 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.503 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.503 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.503 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.503 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.503 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 94 2024-09-16T03:37:56.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.503 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:37:56.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:37:56.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385100+0000", 2024-09-16T03:37:56.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9599999999999998e-07 2024-09-16T03:37:56.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385104+0000", 2024-09-16T03:37:56.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7579999999999998e-06 2024-09-16T03:37:56.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385128+0000", 2024-09-16T03:37:56.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4700999999999999e-05 2024-09-16T03:37:56.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385139+0000", 2024-09-16T03:37:56.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0587e-05 2024-09-16T03:37:56.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787499+0000", 2024-09-16T03:37:56.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40235976099999998 2024-09-16T03:37:56.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787545+0000", 2024-09-16T03:37:56.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6418999999999997e-05 2024-09-16T03:37:56.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787623+0000", 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7897000000000003e-05 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888605+0000", 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100982321 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990721+0000", 2024-09-16T03:37:56.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102115922 2024-09-16T03:37:56.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093019+0000", 2024-09-16T03:37:56.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10229788500000001 2024-09-16T03:37:56.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093051+0000", 2024-09-16T03:37:56.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2255999999999997e-05 2024-09-16T03:37:56.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093099+0000", 2024-09-16T03:37:56.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7500999999999998e-05 2024-09-16T03:37:56.509 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.509 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.509 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.509 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.509 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:37:56.509 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.044783772999999, 2024-09-16T03:37:56.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80404594699999998, 2024-09-16T03:37:56.509 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.509 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.510 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.510 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.510 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.510 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.510 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 95 2024-09-16T03:37:56.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.510 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:37:56.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:37:56.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385151+0000", 2024-09-16T03:37:56.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:37:56.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385165+0000", 2024-09-16T03:37:56.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4175999999999999e-05 2024-09-16T03:37:56.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385175+0000", 2024-09-16T03:37:56.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:37:56.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385183+0000", 2024-09-16T03:37:56.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:37:56.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888643+0000", 2024-09-16T03:37:56.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50345988200000003 2024-09-16T03:37:56.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888695+0000", 2024-09-16T03:37:56.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2834999999999998e-05 2024-09-16T03:37:56.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888802+0000", 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010697899999999999 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990658+0000", 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185585699999999 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990917+0000", 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025853600000000002 2024-09-16T03:37:56.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189054+0000", 2024-09-16T03:37:56.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198136812 2024-09-16T03:37:56.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189135+0000", 2024-09-16T03:37:56.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1482000000000004e-05 2024-09-16T03:37:56.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189196+0000", 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1175999999999998e-05 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.044743146000002, 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80412488400000004, 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.517 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.518 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.518 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.518 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 96 2024-09-16T03:37:56.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.518 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:37:56.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:37:56.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385192+0000", 2024-09-16T03:37:56.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:37:56.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385267+0000", 2024-09-16T03:37:56.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5535000000000001e-05 2024-09-16T03:37:56.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385277+0000", 2024-09-16T03:37:56.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6239999999999997e-06 2024-09-16T03:37:56.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385284+0000", 2024-09-16T03:37:56.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4800000000000004e-06 2024-09-16T03:37:56.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989283+0000", 2024-09-16T03:37:56.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60399849500000002 2024-09-16T03:37:56.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989410+0000", 2024-09-16T03:37:56.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127298 2024-09-16T03:37:56.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989615+0000", 2024-09-16T03:37:56.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000205045 2024-09-16T03:37:56.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991036+0000", 2024-09-16T03:37:56.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001421042 2024-09-16T03:37:56.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189228+0000", 2024-09-16T03:37:56.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19819188500000001 2024-09-16T03:37:56.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189249+0000", 2024-09-16T03:37:56.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0996999999999999e-05 2024-09-16T03:37:56.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189293+0000", 2024-09-16T03:37:56.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3912000000000002e-05 2024-09-16T03:37:56.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189316+0000", 2024-09-16T03:37:56.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2739e-05 2024-09-16T03:37:56.523 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.523 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.523 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.044639414999999, 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80642264100000005, 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 97 2024-09-16T03:37:56.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.525 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:37:56.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:37:56.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385296+0000", 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.006e-06 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385301+0000", 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2630000000000003e-06 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385308+0000", 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6030000000000001e-06 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485459+0000", 2024-09-16T03:37:56.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150929 2024-09-16T03:37:56.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990238+0000", 2024-09-16T03:37:56.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50477992000000005 2024-09-16T03:37:56.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990287+0000", 2024-09-16T03:37:56.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8863999999999998e-05 2024-09-16T03:37:56.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990377+0000", 2024-09-16T03:37:56.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9900000000000003e-05 2024-09-16T03:37:56.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093109+0000", 2024-09-16T03:37:56.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102732055 2024-09-16T03:37:56.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189351+0000", 2024-09-16T03:37:56.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096241766000000006 2024-09-16T03:37:56.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191570+0000", 2024-09-16T03:37:56.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00221915 2024-09-16T03:37:56.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191653+0000", 2024-09-16T03:37:56.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3046000000000004e-05 2024-09-16T03:37:56.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191717+0000", 2024-09-16T03:37:56.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4139000000000004e-05 2024-09-16T03:37:56.530 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.530 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.530 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.530 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.530 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:37:56.530 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 57.944448958000002, 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90688294999999997, 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 98 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:37:56.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:37:56.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485487+0000", 2024-09-16T03:37:56.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1799999999999999e-06 2024-09-16T03:37:56.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485498+0000", 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0648999999999999e-05 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485514+0000", 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5943999999999998e-05 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485525+0000", 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.065e-05 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991102+0000", 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50557748400000002 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991161+0000", 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8801999999999997e-05 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991255+0000", 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3936000000000003e-05 2024-09-16T03:37:56.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093112+0000", 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185709499999999 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291911+0000", 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19879897699999999 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392167+0000", 2024-09-16T03:37:56.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10025634999999999 2024-09-16T03:37:56.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392249+0000", 2024-09-16T03:37:56.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2175000000000005e-05 2024-09-16T03:37:56.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392368+0000", 2024-09-16T03:37:56.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118708 2024-09-16T03:37:56.536 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.536 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.536 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.537 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.537 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:37:56.537 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 57.944395335000003, 2024-09-16T03:37:56.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90707203199999997, 2024-09-16T03:37:56.537 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.537 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.537 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.537 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.537 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.537 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.537 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 99 2024-09-16T03:37:56.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.538 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:37:56.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:37:56.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485540+0000", 2024-09-16T03:37:56.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-06 2024-09-16T03:37:56.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485661+0000", 2024-09-16T03:37:56.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120835 2024-09-16T03:37:56.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485680+0000", 2024-09-16T03:37:56.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8885999999999998e-05 2024-09-16T03:37:56.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485693+0000", 2024-09-16T03:37:56.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3297e-05 2024-09-16T03:37:56.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991546+0000", 2024-09-16T03:37:56.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.505853098 2024-09-16T03:37:56.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991598+0000", 2024-09-16T03:37:56.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1921e-05 2024-09-16T03:37:56.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991681+0000", 2024-09-16T03:37:56.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2502000000000004e-05 2024-09-16T03:37:56.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093115+0000", 2024-09-16T03:37:56.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10143424299999999 2024-09-16T03:37:56.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291977+0000", 2024-09-16T03:37:56.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198862172 2024-09-16T03:37:56.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392541+0000", 2024-09-16T03:37:56.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100563758 2024-09-16T03:37:56.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392579+0000", 2024-09-16T03:37:56.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7796999999999997e-05 2024-09-16T03:37:56.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392611+0000", 2024-09-16T03:37:56.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2073999999999998e-05 2024-09-16T03:37:56.552 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.552 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.552 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.553 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.553 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:37:56.553 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 57.944216441999998, 2024-09-16T03:37:56.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0073149939999999, 2024-09-16T03:37:56.553 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.553 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.553 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.553 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.553 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.554 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.554 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 100 2024-09-16T03:37:56.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.554 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:37:56.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:37:56.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:37:56.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7400000000000003e-07 2024-09-16T03:37:56.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485721+0000", 2024-09-16T03:37:56.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4530000000000001e-06 2024-09-16T03:37:56.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485726+0000", 2024-09-16T03:37:56.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0590000000000002e-06 2024-09-16T03:37:56.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485729+0000", 2024-09-16T03:37:56.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:37:56.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092342+0000", 2024-09-16T03:37:56.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60661343700000003 2024-09-16T03:37:56.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092469+0000", 2024-09-16T03:37:56.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012658799999999999 2024-09-16T03:37:56.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092671+0000", 2024-09-16T03:37:56.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020206 2024-09-16T03:37:56.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392452+0000", 2024-09-16T03:37:56.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29978163699999999 2024-09-16T03:37:56.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392486+0000", 2024-09-16T03:37:56.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3736999999999999e-05 2024-09-16T03:37:56.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492812+0000", 2024-09-16T03:37:56.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100325621 2024-09-16T03:37:56.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492901+0000", 2024-09-16T03:37:56.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8702000000000006e-05 2024-09-16T03:37:56.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.493033+0000", 2024-09-16T03:37:56.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000132185 2024-09-16T03:37:56.561 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.561 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.561 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.562 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.562 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:37:56.562 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.769065920999999, 2024-09-16T03:37:56.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60874581500000002, 2024-09-16T03:37:56.562 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.562 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.562 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.562 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.562 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 123 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:37:56.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660869+0000", 2024-09-16T03:37:56.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9400000000000005e-07 2024-09-16T03:37:56.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660921+0000", 2024-09-16T03:37:56.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2312000000000003e-05 2024-09-16T03:37:56.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660927+0000", 2024-09-16T03:37:56.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9329999999999998e-06 2024-09-16T03:37:56.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660931+0000", 2024-09-16T03:37:56.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:37:56.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.967991+0000", 2024-09-16T03:37:56.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30706032700000002 2024-09-16T03:37:56.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968039+0000", 2024-09-16T03:37:56.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7477e-05 2024-09-16T03:37:56.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968117+0000", 2024-09-16T03:37:56.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8589000000000003e-05 2024-09-16T03:37:56.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067678+0000", 2024-09-16T03:37:56.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099560406000000004 2024-09-16T03:37:56.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170175+0000", 2024-09-16T03:37:56.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102496927 2024-09-16T03:37:56.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269491+0000", 2024-09-16T03:37:56.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099316205000000005 2024-09-16T03:37:56.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269574+0000", 2024-09-16T03:37:56.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2903000000000002e-05 2024-09-16T03:37:56.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269614+0000", 2024-09-16T03:37:56.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0188000000000002e-05 2024-09-16T03:37:56.570 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.570 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.570 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.570 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:37:56.571 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:37:56.571 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.768997223, 2024-09-16T03:37:56.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60879245500000001, 2024-09-16T03:37:56.571 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.571 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.571 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.571 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.571 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:37:56.571 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:37:56.571 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 124 2024-09-16T03:37:56.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.572 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:37:56.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:37:56.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660938+0000", 2024-09-16T03:37:56.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:37:56.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660940+0000", 2024-09-16T03:37:56.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2529999999999999e-06 2024-09-16T03:37:56.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660943+0000", 2024-09-16T03:37:56.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.01e-06 2024-09-16T03:37:56.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660946+0000", 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8159999999999998e-06 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012417+0000", 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.35147120599999998 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012497+0000", 2024-09-16T03:37:56.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9963000000000007e-05 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012585+0000", 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067680+0000", 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055095203000000002 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170243+0000", 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10256290799999999 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269662+0000", 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099418896000000007 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269701+0000", 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8829999999999999e-05 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269729+0000", 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8569e-05 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1550 4.13 4:c8492bfc:::smithi15736696-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.763269083000001, 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62706974999999998, 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:56.578 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.579 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:56.579 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:56.579 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1550 2024-09-16T03:37:56.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.579 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:37:56.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0699999999999998e-07 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666669+0000", 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.393e-06 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666673+0000", 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666682+0000", 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6510000000000007e-06 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.913940+0000", 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.24725763000000001 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:56.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.914034+0000", 2024-09-16T03:37:56.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3923000000000002e-05 2024-09-16T03:37:56.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293650+0000", 2024-09-16T03:37:56.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.37961662200000001 2024-09-16T03:37:56.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293688+0000", 2024-09-16T03:37:56.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8096000000000003e-05 2024-09-16T03:37:56.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293735+0000", 2024-09-16T03:37:56.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6451e-05 2024-09-16T03:37:56.583 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.583 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.583 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.583 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.583 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.583 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:56.585 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:37:56.586 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:56.586 INFO:tasks.rados.rados.0.smithi157.stdout:920: finishing set_chunk to oid 27 2024-09-16T03:37:56.586 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 220672 length: 1659392 2024-09-16T03:37:56.586 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:37:56.586 INFO:tasks.rados.rados.0.smithi157.stdout:923: finishing write tid 1 to smithi15736696-19 2024-09-16T03:37:56.586 INFO:tasks.rados.rados.0.smithi157.stdout:923: finishing write tid 2 to smithi15736696-19 2024-09-16T03:37:56.586 INFO:tasks.rados.rados.0.smithi157.stdout:923: finishing write tid 3 to smithi15736696-19 2024-09-16T03:37:56.586 INFO:tasks.rados.rados.0.smithi157.stdout:923: finishing write tid 4 to smithi15736696-19 2024-09-16T03:37:56.586 INFO:tasks.rados.rados.0.smithi157.stdout:923: oid 19 updating version 0 to 67 2024-09-16T03:37:56.586 INFO:tasks.rados.rados.0.smithi157.stdout:923: oid 19 version 67 is already newer than 66 2024-09-16T03:37:56.586 INFO:tasks.rados.rados.0.smithi157.stdout:923: oid 19 updating version 67 to 68 2024-09-16T03:37:56.586 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 68 (ObjNum 143 snap 0 seq_num 143) dirty exists 2024-09-16T03:37:56.586 INFO:tasks.rados.rados.0.smithi157.stdout:923: left oid 19 (ObjNum 143 snap 0 seq_num 143) 2024-09-16T03:37:56.587 INFO:tasks.rados.rados.0.smithi157.stdout:924: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T03:37:56.587 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1669999 2024-09-16T03:37:56.587 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:56.583+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:56.616 INFO:tasks.rados.rados.0.smithi157.stdout:921: expect (ObjNum 104 snap 0 seq_num 104) 2024-09-16T03:37:56.686 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:56.683+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:56.720 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:56.721 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:56.721 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:37:56.721 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:56.729 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:37:56.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:37:56.787 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:56.785+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:56.834 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:56.834 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:37:56.834 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:37:56.834 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:37:56.834 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:56.835 INFO:tasks.rados.rados.0.smithi157.stdout:925: expect (ObjNum 97 snap 0 seq_num 97) 2024-09-16T03:37:56.843 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:37:56.889 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:56.887+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:56.889 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2267260 2024-09-16T03:37:56.906 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1413553 2024-09-16T03:37:56.916 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:56.916 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:56.934 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:56.934 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:56.936 INFO:tasks.rados.rados.0.smithi157.stdout:922: finishing set_chunk to oid 33 2024-09-16T03:37:56.936 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 40 (ObjNum 127 snap 0 seq_num 127) dirty exists 2024-09-16T03:37:56.936 INFO:tasks.rados.rados.0.smithi157.stdout:919: done (7 left) 2024-09-16T03:37:56.936 INFO:tasks.rados.rados.0.smithi157.stdout:920: done (6 left) 2024-09-16T03:37:56.936 INFO:tasks.rados.rados.0.smithi157.stdout:921: done (5 left) 2024-09-16T03:37:56.936 INFO:tasks.rados.rados.0.smithi157.stdout:922: done (4 left) 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:923: done (3 left) 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:924: done (2 left) 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:925: done (1 left) 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:926: done (0 left) 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:927: read oid 27 snap -1 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:927: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:928: read oid 40 snap -1 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:928: expect (ObjNum 133 snap 0 seq_num 133) 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:929: read oid 12 snap -1 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:929: expect deleted 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:930: tier_flush oid 5 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:930: tier_flush oid 5 snap -1 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:931: read oid 11 snap -1 2024-09-16T03:37:56.937 INFO:tasks.rados.rados.0.smithi157.stdout:931: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:37:56.938 INFO:tasks.rados.rados.0.smithi157.stdout:932: read oid 35 snap -1 2024-09-16T03:37:56.938 INFO:tasks.rados.rados.0.smithi157.stdout:932: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:37:56.938 INFO:tasks.rados.rados.0.smithi157.stdout:933: read oid 30 snap -1 2024-09-16T03:37:56.938 INFO:tasks.rados.rados.0.smithi157.stdout:933: expect (ObjNum 98 snap 0 seq_num 98) 2024-09-16T03:37:56.938 INFO:tasks.rados.rados.0.smithi157.stdout:934: read oid 19 snap -1 2024-09-16T03:37:56.938 INFO:tasks.rados.rados.0.smithi157.stdout:934: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-16T03:37:56.938 INFO:tasks.rados.rados.0.smithi157.stdout:935: read oid 2 snap -1 2024-09-16T03:37:56.938 INFO:tasks.rados.rados.0.smithi157.stdout:935: expect (ObjNum 128 snap 0 seq_num 128) 2024-09-16T03:37:56.938 INFO:tasks.rados.rados.0.smithi157.stdout:936: write oid 13 current snap is 0 2024-09-16T03:37:56.938 INFO:tasks.rados.rados.0.smithi157.stdout:936: seq_num 145 ranges {437015=721812,1701889=509708} 2024-09-16T03:37:56.942 INFO:tasks.rados.rados.0.smithi157.stdout:936: writing smithi15736696-13 from 437015 to 1158827 tid 1 2024-09-16T03:37:56.948 INFO:tasks.rados.rados.0.smithi157.stdout:936: writing smithi15736696-13 from 1701889 to 2211597 tid 2 2024-09-16T03:37:56.948 INFO:tasks.rados.rados.0.smithi157.stdout:937: delete oid 47 current snap is 0 2024-09-16T03:37:56.949 INFO:tasks.rados.rados.0.smithi157.stdout:931: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:37:56.949 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:56.950 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:37:56.951 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:37:56.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:37:56.951 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:37:56.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.951 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1 4.2 4:4a9d235d:::smithi15736696-1:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T03:37:56.951 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:11.955475+0000", 2024-09-16T03:37:56.951 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.992358596999999, 2024-09-16T03:37:56.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.405292714, 2024-09-16T03:37:56.951 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.951 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.951 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.952 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.952 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:56.952 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:56.952 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:37:56.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.952 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955475+0000", 2024-09-16T03:37:56.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955475+0000", 2024-09-16T03:37:56.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955477+0000", 2024-09-16T03:37:56.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5040000000000001e-06 2024-09-16T03:37:56.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955870+0000", 2024-09-16T03:37:56.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000393082 2024-09-16T03:37:56.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.955883+0000", 2024-09-16T03:37:56.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3173000000000001e-05 2024-09-16T03:37:56.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.055959+0000", 2024-09-16T03:37:56.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10007564300000001 2024-09-16T03:37:56.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156811+0000", 2024-09-16T03:37:56.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100852073 2024-09-16T03:37:56.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156907+0000", 2024-09-16T03:37:56.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5945000000000004e-05 2024-09-16T03:37:56.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 8,11", 2024-09-16T03:37:56.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.157029+0000", 2024-09-16T03:37:56.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012207099999999999 2024-09-16T03:37:56.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.160049+0000", 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0030194409999999999 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260277+0000", 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10022834 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360663+0000", 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10038625599999999 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360710+0000", 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7165e-05 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360767+0000", 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7021000000000002e-05 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.958 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2 4.2 4:4a9d235d:::smithi15736696-1:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.056000+0000", 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.891832993000001, 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30489393999999997, 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2 2024-09-16T03:37:56.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.960 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056000+0000", 2024-09-16T03:37:56.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056000+0000", 2024-09-16T03:37:56.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056003+0000", 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3250000000000002e-06 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056130+0000", 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127059 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.056139+0000", 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1419999999999999e-06 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156244+0000", 2024-09-16T03:37:56.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10010532799999999 2024-09-16T03:37:56.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.158582+0000", 2024-09-16T03:37:56.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023374759999999998 2024-09-16T03:37:56.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.158698+0000", 2024-09-16T03:37:56.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000116051 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 8,11", 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.158807+0000", 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010915400000000001 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260247+0000", 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10144022900000001 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360554+0000", 2024-09-16T03:37:56.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100307113 2024-09-16T03:37:56.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360813+0000", 2024-09-16T03:37:56.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025846499999999999 2024-09-16T03:37:56.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360857+0000", 2024-09-16T03:37:56.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3954e-05 2024-09-16T03:37:56.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360894+0000", 2024-09-16T03:37:56.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7644000000000001e-05 2024-09-16T03:37:56.965 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4 4.2 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T03:37:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.156281+0000", 2024-09-16T03:37:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.791552154000001, 2024-09-16T03:37:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30482965400000001, 2024-09-16T03:37:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.966 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:56.967 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:56.967 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4 2024-09-16T03:37:56.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.967 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156281+0000", 2024-09-16T03:37:56.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156281+0000", 2024-09-16T03:37:56.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156282+0000", 2024-09-16T03:37:56.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.08e-07 2024-09-16T03:37:56.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156284+0000", 2024-09-16T03:37:56.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7159999999999999e-06 2024-09-16T03:37:56.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156288+0000", 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8139999999999998e-06 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.156291+0000", 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.971e-06 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260143+0000", 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103851517 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:56.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.260223+0000", 2024-09-16T03:37:56.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0439999999999996e-05 2024-09-16T03:37:56.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461007+0000", 2024-09-16T03:37:56.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20078405799999999 2024-09-16T03:37:56.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461053+0000", 2024-09-16T03:37:56.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5667000000000002e-05 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.461111+0000", 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8063000000000003e-05 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:64 4.5 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.690492982999999, 2024-09-16T03:37:56.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50484290399999998, 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 64 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:37:56.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257341+0000", 2024-09-16T03:37:56.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:37:56.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257343+0000", 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3889999999999999e-06 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257346+0000", 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257349+0000", 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8559999999999998e-06 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357057+0000", 2024-09-16T03:37:56.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099708121999999996 2024-09-16T03:37:56.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:56.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357156+0000", 2024-09-16T03:37:56.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9162999999999999e-05 2024-09-16T03:37:56.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762060+0000", 2024-09-16T03:37:56.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40490364400000001 2024-09-16T03:37:56.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762112+0000", 2024-09-16T03:37:56.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1882000000000002e-05 2024-09-16T03:37:56.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762183+0000", 2024-09-16T03:37:56.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1297000000000005e-05 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:295 4.b 4:dbb349e8:::smithi15736696-46:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:16.085151+0000", 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 40.862682053, 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.42198747199999997, 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.977 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:56.978 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:56.978 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 295 2024-09-16T03:37:56.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.978 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085151+0000", 2024-09-16T03:37:56.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085151+0000", 2024-09-16T03:37:56.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085153+0000", 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.421e-06 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085158+0000", 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0239999999999997e-06 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085166+0000", 2024-09-16T03:37:56.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6400000000000003e-06 2024-09-16T03:37:56.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085172+0000", 2024-09-16T03:37:56.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2499999999999997e-06 2024-09-16T03:37:56.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085293+0000", 2024-09-16T03:37:56.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000121853 2024-09-16T03:37:56.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.085366+0000", 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2872000000000003e-05 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,9", 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.288783+0000", 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20341624799999999 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.289205+0000", 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042208099999999999 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.289914+0000", 2024-09-16T03:37:56.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00070928600000000005 2024-09-16T03:37:56.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.507063+0000", 2024-09-16T03:37:56.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.21714887299999999 2024-09-16T03:37:56.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.507095+0000", 2024-09-16T03:37:56.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2570000000000002e-05 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:16.507139+0000", 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3353999999999999e-05 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:400 4.2 4:4a9d235d:::smithi15736696-1:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.205970+0000", 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.741862900000001, 2024-09-16T03:37:56.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40465369000000001, 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 400 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205970+0000", 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205970+0000", 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205972+0000", 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.841e-06 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205979+0000", 2024-09-16T03:37:56.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9090000000000003e-06 2024-09-16T03:37:56.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.205991+0000", 2024-09-16T03:37:56.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.165e-05 2024-09-16T03:37:56.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.206013+0000", 2024-09-16T03:37:56.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2079e-05 2024-09-16T03:37:56.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.306137+0000", 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10012456 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.306223+0000", 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5181999999999999e-05 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 8,11", 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309852+0000", 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036292189999999999 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.310473+0000", 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00062141500000000005 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.608696+0000", 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29822300400000001 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.610553+0000", 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018567989999999999 2024-09-16T03:37:56.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.610587+0000", 2024-09-16T03:37:56.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3995999999999998e-05 2024-09-16T03:37:56.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.610624+0000", 2024-09-16T03:37:56.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7036000000000003e-05 2024-09-16T03:37:56.989 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.989 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.989 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.10.22:22 3.3 3:f84e13a5:::smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.638345747999999, 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50224271099999995, 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309489+0000", 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4309999999999999e-06 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309496+0000", 2024-09-16T03:37:56.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5810000000000002e-06 2024-09-16T03:37:56.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309505+0000", 2024-09-16T03:37:56.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6889999999999999e-06 2024-09-16T03:37:56.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309513+0000", 2024-09-16T03:37:56.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3749999999999997e-06 2024-09-16T03:37:56.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.406932+0000", 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.097419631000000007 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407009+0000", 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7220999999999998e-05 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407143+0000", 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013323799999999999 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407732+0000", 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058938399999999998 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.408133+0000", 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040090500000000001 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811599+0000", 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40346633399999998 2024-09-16T03:37:56.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811682+0000", 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2463e-05 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811730+0000", 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8458999999999998e-05 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.996 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:465 4.5 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:56.996 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:37:56.996 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.932128210000002, 2024-09-16T03:37:56.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60529649299999999, 2024-09-16T03:37:56.996 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:56.996 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:56.996 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:56.996 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:56.996 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:56.996 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:56.996 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 465 2024-09-16T03:37:56.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.996 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015706+0000", 2024-09-16T03:37:56.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.091e-06 2024-09-16T03:37:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015712+0000", 2024-09-16T03:37:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-16T03:37:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015724+0000", 2024-09-16T03:37:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1408e-05 2024-09-16T03:37:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015731+0000", 2024-09-16T03:37:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:37:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216049+0000", 2024-09-16T03:37:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200317895 2024-09-16T03:37:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216178+0000", 2024-09-16T03:37:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012848800000000001 2024-09-16T03:37:56.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:37:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620010+0000", 2024-09-16T03:37:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40383207199999999 2024-09-16T03:37:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620710+0000", 2024-09-16T03:37:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069988400000000001 2024-09-16T03:37:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620902+0000", 2024-09-16T03:37:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019197800000000001 2024-09-16T03:37:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620933+0000", 2024-09-16T03:37:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1513999999999997e-05 2024-09-16T03:37:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620968+0000", 2024-09-16T03:37:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4524999999999999e-05 2024-09-16T03:37:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.621002+0000", 2024-09-16T03:37:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3955000000000003e-05 2024-09-16T03:37:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:10 3.3 3:deac805c:::smithi15736696-44 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:37:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.540633234000001, 2024-09-16T03:37:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53191748100000003, 2024-09-16T03:37:57.002 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407202+0000", 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.855e-06 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407208+0000", 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9819999999999996e-06 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407220+0000", 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2191999999999999e-05 2024-09-16T03:37:57.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407236+0000", 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5698000000000001e-05 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538167+0000", 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.13093111499999999 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538229+0000", 2024-09-16T03:37:57.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2441999999999996e-05 2024-09-16T03:37:57.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:37:57.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538346+0000", 2024-09-16T03:37:57.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011716400000000001 2024-09-16T03:37:57.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:57.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540757+0000", 2024-09-16T03:37:57.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024105089999999999 2024-09-16T03:37:57.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.639501+0000", 2024-09-16T03:37:57.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098743545000000002 2024-09-16T03:37:57.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939038+0000", 2024-09-16T03:37:57.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29953765700000001 2024-09-16T03:37:57.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:57.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939075+0000", 2024-09-16T03:37:57.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6600999999999997e-05 2024-09-16T03:37:57.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939118+0000", 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2721e-05 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:826 4.b 4:dbb349e8:::smithi15736696-46:head [write 507578~477917,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.519716+0000", 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.428117431, 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.31863338600000002, 2024-09-16T03:37:57.008 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:57.009 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:57.009 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:57.009 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:57.009 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:57.009 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:57.009 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 826 2024-09-16T03:37:57.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.009 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:57.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:57.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.519716+0000", 2024-09-16T03:37:57.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.519716+0000", 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.519718+0000", 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2069999999999998e-06 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:57.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.519844+0000", 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012567899999999999 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.519856+0000", 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2649e-05 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.519863+0000", 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6039999999999998e-06 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538663+0000", 2024-09-16T03:37:57.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.018799607999999999 2024-09-16T03:37:57.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:57.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538712+0000", 2024-09-16T03:37:57.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9579000000000001e-05 2024-09-16T03:37:57.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,9", 2024-09-16T03:37:57.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538814+0000", 2024-09-16T03:37:57.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000101813 2024-09-16T03:37:57.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:57.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.541185+0000", 2024-09-16T03:37:57.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023706790000000001 2024-09-16T03:37:57.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.543732+0000", 2024-09-16T03:37:57.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0025473420000000002 2024-09-16T03:37:57.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.838214+0000", 2024-09-16T03:37:57.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29448223099999998 2024-09-16T03:37:57.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:57.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.838314+0000", 2024-09-16T03:37:57.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9327999999999995e-05 2024-09-16T03:37:57.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:57.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.838349+0000", 2024-09-16T03:37:57.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5667000000000003e-05 2024-09-16T03:37:57.014 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.014 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:827 4.b 4:dbb349e8:::smithi15736696-46:head [write 1543557~550004] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.525668+0000", 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.422165524, 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.31401412899999998, 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:57.015 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 827 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.525668+0000", 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.525668+0000", 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:57.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.525669+0000", 2024-09-16T03:37:57.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1489999999999999e-06 2024-09-16T03:37:57.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:57.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.525806+0000", 2024-09-16T03:37:57.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000136802 2024-09-16T03:37:57.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:57.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.525816+0000", 2024-09-16T03:37:57.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0725e-05 2024-09-16T03:37:57.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.525822+0000", 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9959999999999999e-06 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.539804+0000", 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.013981650999999999 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.539860+0000", 2024-09-16T03:37:57.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6131999999999997e-05 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,9", 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.539953+0000", 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2998999999999995e-05 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.541560+0000", 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001606569 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.544223+0000", 2024-09-16T03:37:57.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0026632719999999999 2024-09-16T03:37:57.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839587+0000", 2024-09-16T03:37:57.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29536388699999999 2024-09-16T03:37:57.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:57.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839657+0000", 2024-09-16T03:37:57.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9630999999999998e-05 2024-09-16T03:37:57.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839682+0000", 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5316e-05 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:828 4.b 4:dbb349e8:::smithi15736696-46:head [write 2833665~73913] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.529956+0000", 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.417877128000001, 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30987494399999999, 2024-09-16T03:37:57.021 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 828 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.529956+0000", 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:57.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.529956+0000", 2024-09-16T03:37:57.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:57.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.529957+0000", 2024-09-16T03:37:57.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2699999999999999e-06 2024-09-16T03:37:57.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:57.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.529980+0000", 2024-09-16T03:37:57.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2515999999999999e-05 2024-09-16T03:37:57.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:57.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.529990+0000", 2024-09-16T03:37:57.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0356999999999999e-05 2024-09-16T03:37:57.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:57.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.529996+0000", 2024-09-16T03:37:57.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4439999999999999e-06 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540781+0000", 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.010784782 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540826+0000", 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5901e-05 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,9", 2024-09-16T03:37:57.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540903+0000", 2024-09-16T03:37:57.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6629000000000001e-05 2024-09-16T03:37:57.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:57.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.541745+0000", 2024-09-16T03:37:57.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00084188500000000001 2024-09-16T03:37:57.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.732006+0000", 2024-09-16T03:37:57.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19026100600000001 2024-09-16T03:37:57.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839735+0000", 2024-09-16T03:37:57.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.107728796 2024-09-16T03:37:57.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:57.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839790+0000", 2024-09-16T03:37:57.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4766999999999999e-05 2024-09-16T03:37:57.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:57.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839831+0000", 2024-09-16T03:37:57.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1591000000000002e-05 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:829 4.b 4:dbb349e8:::smithi15736696-46:head [setxattr _header (56) in=7b,truncate 2907578] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.530013+0000", 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.417820559999999, 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30994171300000001, 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:57.028 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:57.029 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:57.029 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 829 2024-09-16T03:37:57.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.029 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:57.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:57.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.530013+0000", 2024-09-16T03:37:57.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:57.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.530013+0000", 2024-09-16T03:37:57.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.530013+0000", 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8800000000000002e-07 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.530016+0000", 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6599999999999999e-06 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:57.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.530021+0000", 2024-09-16T03:37:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.271e-06 2024-09-16T03:37:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.630115+0000", 2024-09-16T03:37:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100094003 2024-09-16T03:37:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.630214+0000", 2024-09-16T03:37:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9085000000000004e-05 2024-09-16T03:37:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:57.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.630326+0000", 2024-09-16T03:37:57.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011141099999999999 2024-09-16T03:37:57.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,9", 2024-09-16T03:37:57.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.630486+0000", 2024-09-16T03:37:57.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016000500000000001 2024-09-16T03:37:57.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:57.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.633653+0000", 2024-09-16T03:37:57.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031677480000000002 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.732280+0000", 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098626948000000006 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839882+0000", 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10760173200000001 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:57.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839909+0000", 2024-09-16T03:37:57.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6891000000000001e-05 2024-09-16T03:37:57.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:57.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.839954+0000", 2024-09-16T03:37:57.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5371000000000003e-05 2024-09-16T03:37:57.034 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.034 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.034 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.034 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:984 4.15 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:37:57.034 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 26.575713306000001, 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.97836159599999994, 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 984 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:57.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372122+0000", 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.934e-06 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372129+0000", 2024-09-16T03:37:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8029999999999999e-06 2024-09-16T03:37:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372141+0000", 2024-09-16T03:37:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2006e-05 2024-09-16T03:37:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472234+0000", 2024-09-16T03:37:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100092985 2024-09-16T03:37:57.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472318+0000", 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4257000000000003e-05 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472326+0000", 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8960000000000003e-06 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350293+0000", 2024-09-16T03:37:57.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.87796699899999997 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350433+0000", 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013999600000000001 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350482+0000", 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8720000000000001e-05 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.040 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1000 4.15 4:aff4cc41:::smithi15736696-11:head [read 0~2873964,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:37:57.040 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:37:57.040 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 26.357274221000001, 2024-09-16T03:37:57.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.761707841, 2024-09-16T03:37:57.040 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:57.040 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:57.040 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:57.040 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:57.040 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:57.040 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:57.040 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1000 2024-09-16T03:37:57.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.041 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:57.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:57.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:37:57.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:57.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:37:57.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:57.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590560+0000", 2024-09-16T03:37:57.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:37:57.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:57.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590562+0000", 2024-09-16T03:37:57.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.17e-06 2024-09-16T03:37:57.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:57.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590568+0000", 2024-09-16T03:37:57.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7939999999999997e-06 2024-09-16T03:37:57.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:57.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590572+0000", 2024-09-16T03:37:57.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1300000000000001e-06 2024-09-16T03:37:57.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590619+0000", 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7148999999999999e-05 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590625+0000", 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5679999999999996e-06 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350858+0000", 2024-09-16T03:37:57.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76023295000000002 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350973+0000", 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115023 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352267+0000", 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012935970000000001 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.047 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1001 4.15 4:aff4cc41:::smithi15736696-11:head [read 1254674~1619290,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:37:57.047 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:37:57.047 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 26.357255636000001, 2024-09-16T03:37:57.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76275827799999996, 2024-09-16T03:37:57.047 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:57.047 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:57.047 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:57.047 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:57.047 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:57.047 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:57.047 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1001 2024-09-16T03:37:57.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:57.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:37:57.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:37:57.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:57.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590580+0000", 2024-09-16T03:37:57.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:37:57.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:57.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590584+0000", 2024-09-16T03:37:57.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8800000000000001e-06 2024-09-16T03:37:57.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590587+0000", 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6630000000000002e-06 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590650+0000", 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3950999999999998e-05 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590654+0000", 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.89e-06 2024-09-16T03:37:57.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352309+0000", 2024-09-16T03:37:57.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76165495699999997 2024-09-16T03:37:57.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:57.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352407+0000", 2024-09-16T03:37:57.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7188999999999994e-05 2024-09-16T03:37:57.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.353336+0000", 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092939400000000005 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1002 4.15 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~1527025] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 26.357226382, 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85963523600000002, 2024-09-16T03:37:57.052 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:57.053 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:57.053 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:57.053 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:57.053 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:57.053 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:57.053 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1002 2024-09-16T03:37:57.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.053 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:57.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:57.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:37:57.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.15e-07 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:57.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590609+0000", 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.606e-06 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590615+0000", 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3040000000000001e-06 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590618+0000", 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.309e-06 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590669+0000", 2024-09-16T03:37:57.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1048000000000003e-05 2024-09-16T03:37:57.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:37:57.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590673+0000", 2024-09-16T03:37:57.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.061e-06 2024-09-16T03:37:57.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.449932+0000", 2024-09-16T03:37:57.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85925979699999999 2024-09-16T03:37:57.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450032+0000", 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9858000000000005e-05 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450242+0000", 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020993800000000001 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1038 4.12 4:4a9d235d:::smithi15736696-1:head [sparse-read 1073257~474678] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:31.241887+0000", 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.705946197999999, 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.31208923199999999, 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1038 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:57.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:57.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.241887+0000", 2024-09-16T03:37:57.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:57.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.241887+0000", 2024-09-16T03:37:57.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:57.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.241887+0000", 2024-09-16T03:37:57.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.15e-07 2024-09-16T03:37:57.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:57.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.241889+0000", 2024-09-16T03:37:57.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9640000000000002e-06 2024-09-16T03:37:57.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:57.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.241894+0000", 2024-09-16T03:37:57.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9960000000000001e-06 2024-09-16T03:37:57.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:57.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.241899+0000", 2024-09-16T03:37:57.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.155e-06 2024-09-16T03:37:57.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.351769+0000", 2024-09-16T03:37:57.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.109870366 2024-09-16T03:37:57.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:37:57.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.351779+0000", 2024-09-16T03:37:57.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0069e-05 2024-09-16T03:37:57.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.553770+0000", 2024-09-16T03:37:57.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.201990593 2024-09-16T03:37:57.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:57.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.553847+0000", 2024-09-16T03:37:57.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7294999999999998e-05 2024-09-16T03:37:57.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:57.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.553976+0000", 2024-09-16T03:37:57.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000129379 2024-09-16T03:37:57.063 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.063 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.063 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.063 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1556 4.12 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:37:57.063 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.056215+0000", 2024-09-16T03:37:57.063 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.891617813, 2024-09-16T03:37:57.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.305698362, 2024-09-16T03:37:57.063 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:57.063 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:57.063 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:57.064 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:57.064 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:57.064 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:57.064 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1556 2024-09-16T03:37:57.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.064 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:57.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:57.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.056215+0000", 2024-09-16T03:37:57.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.056215+0000", 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.056216+0000", 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.06e-07 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.056218+0000", 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4600000000000002e-06 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.056222+0000", 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.337e-06 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.056225+0000", 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.92e-06 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.057393+0000", 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011687729999999999 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.057469+0000", 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5913000000000003e-05 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 8,11", 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.057603+0000", 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000133504 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.256490+0000", 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19888693700000001 2024-09-16T03:37:57.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:37:57.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.256647+0000", 2024-09-16T03:37:57.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015728199999999999 2024-09-16T03:37:57.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:57.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.361845+0000", 2024-09-16T03:37:57.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10519753599999999 2024-09-16T03:37:57.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:37:57.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.361877+0000", 2024-09-16T03:37:57.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2184999999999998e-05 2024-09-16T03:37:57.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:57.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.361914+0000", 2024-09-16T03:37:57.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7008999999999998e-05 2024-09-16T03:37:57.069 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.069 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.069 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.069 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1557 4.12 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:37:57.069 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.056228+0000", 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.891605140999999, 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30582585200000001, 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1557 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:37:57.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.056228+0000", 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.056228+0000", 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.056228+0000", 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.03e-07 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:37:57.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.056230+0000", 2024-09-16T03:37:57.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3769999999999999e-06 2024-09-16T03:37:57.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:37:57.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.056234+0000", 2024-09-16T03:37:57.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7869999999999999e-06 2024-09-16T03:37:57.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:37:57.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.056237+0000", 2024-09-16T03:37:57.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.994e-06 2024-09-16T03:37:57.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.059723+0000", 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034866839999999999 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.059795+0000", 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1777000000000001e-05 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.361952+0000", 2024-09-16T03:37:57.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30215688299999999 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.361997+0000", 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5028999999999999e-05 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.362054+0000", 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7018000000000002e-05 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.074 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:37:57.075 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:37:57.075 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:37:57.076 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:57.076 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:57.077 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:57.077 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:57.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:37:57.078 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2486912 2024-09-16T03:37:57.078 INFO:tasks.rados.rados.0.smithi157.stdout:928: expect (ObjNum 133 snap 0 seq_num 133) 2024-09-16T03:37:57.078 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:57.078 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:57.079 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:57.079 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:57.079 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:57.080 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:57.080 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:57.080 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:57.080 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:57.081 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:57.074+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:57.091 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:57.116 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 142969 2024-09-16T03:37:57.140 INFO:tasks.rados.rados.0.smithi157.stdout:933: expect (ObjNum 98 snap 0 seq_num 98) 2024-09-16T03:37:57.140 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:57.177 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:57.176+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:57.189 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:57.189 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:57.197 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:37:57.269 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:57.270 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:57.278 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:57.276+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:57.299 INFO:tasks.rados.rados.0.smithi157.stdout:935: expect (ObjNum 128 snap 0 seq_num 128) 2024-09-16T03:37:57.299 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 205955 2024-09-16T03:37:57.302 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:57.303 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:57.335 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:57.335 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:57.345 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:57.345 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:57.346 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:37:57.344+0000 7f928a16a640 -1 osd.14 71 handle_osd_map injecting map crc failure 2024-09-16T03:37:57.346 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:57.344+0000 7ff26836a640 -1 osd.11 71 handle_osd_map injecting map crc failure 2024-09-16T03:37:57.346 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:57.345+0000 7f6528e68640 -1 osd.1 71 handle_osd_map injecting map crc failure 2024-09-16T03:37:57.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:37:57.362 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:57.363 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:57.379 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:57.379 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:57.380 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:57.378+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:57.396 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:57.397 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:57.412 INFO:tasks.rados.rados.0.smithi157.stdout:932: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:37:57.414 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:57.414 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:57.431 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:37:57.431 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:37:57.443 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1769802 2024-09-16T03:37:57.458 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:57.471 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:57.471 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:57.481 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:37:57.481 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:57.479+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:57.487 INFO:tasks.rados.rados.0.smithi157.stdout:928: done (10 left) 2024-09-16T03:37:57.487 INFO:tasks.rados.rados.0.smithi157.stdout:929: done (9 left) 2024-09-16T03:37:57.487 INFO:tasks.rados.rados.0.smithi157.stdout:931: done (8 left) 2024-09-16T03:37:57.487 INFO:tasks.rados.rados.0.smithi157.stdout:932: done (7 left) 2024-09-16T03:37:57.487 INFO:tasks.rados.rados.0.smithi157.stdout:933: done (6 left) 2024-09-16T03:37:57.487 INFO:tasks.rados.rados.0.smithi157.stdout:935: done (5 left) 2024-09-16T03:37:57.488 INFO:tasks.rados.rados.0.smithi157.stdout:937: done (4 left) 2024-09-16T03:37:57.488 INFO:tasks.rados.rados.0.smithi157.stdout:938: delete oid 45 current snap is 0 2024-09-16T03:37:57.488 INFO:tasks.rados.rados.0.smithi157.stdout:927: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:57.534 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:57.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:37:57.560 INFO:tasks.rados.rados.0.smithi157.stdout:934: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-16T03:37:57.581 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:57.579+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:57.630 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:57.630 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:57.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:37:57.683 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:57.681+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:57.689 INFO:tasks.rados.rados.0.smithi157.stdout:927: done (4 left) 2024-09-16T03:37:57.689 INFO:tasks.rados.rados.0.smithi157.stdout:934: done (3 left) 2024-09-16T03:37:57.689 INFO:tasks.rados.rados.0.smithi157.stdout:938: done (2 left) 2024-09-16T03:37:57.689 INFO:tasks.rados.rados.0.smithi157.stdout:939: write (excl) oid 12 current snap is 0 2024-09-16T03:37:57.689 INFO:tasks.rados.rados.0.smithi157.stdout:939: seq_num 148 ranges {487436=547559,1760982=657035} 2024-09-16T03:37:57.694 INFO:tasks.rados.rados.0.smithi157.stdout:939: writing smithi15736696-12 from 487436 to 1034995 tid 1 2024-09-16T03:37:57.702 INFO:tasks.rados.rados.0.smithi157.stdout:939: writing smithi15736696-12 from 1760982 to 2418017 tid 2 2024-09-16T03:37:57.702 INFO:tasks.rados.rados.0.smithi157.stdout:940: write (excl) oid 43 current snap is 0 2024-09-16T03:37:57.702 INFO:tasks.rados.rados.0.smithi157.stdout:940: seq_num 149 ranges {504243=622808,1819464=517046} 2024-09-16T03:37:57.708 INFO:tasks.rados.rados.0.smithi157.stdout:940: writing smithi15736696-43 from 504243 to 1127051 tid 1 2024-09-16T03:37:57.715 INFO:tasks.rados.rados.0.smithi157.stdout:940: writing smithi15736696-43 from 1819464 to 2336510 tid 2 2024-09-16T03:37:57.715 INFO:tasks.rados.rados.0.smithi157.stdout:941: tier_evict oid 10 2024-09-16T03:37:57.715 INFO:tasks.rados.rados.0.smithi157.stdout:941: tier_evict oid 10 snap -1 2024-09-16T03:37:57.715 INFO:tasks.rados.rados.0.smithi157.stdout:942: read oid 36 snap -1 2024-09-16T03:37:57.715 INFO:tasks.rados.rados.0.smithi157.stdout:942: expect deleted 2024-09-16T03:37:57.715 INFO:tasks.rados.rados.0.smithi157.stdout:943: set_chunk oid 14 target oid 2024-09-16T03:37:57.752 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:57.752 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:57.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:37:57.783 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:57.781+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:57.884 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:57.882+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:57.912 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:57.912 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:57.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:37:57.984 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:57.983+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:58.069 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:58.070 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:58.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:37:58.085 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:37:58.083+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:37:58.161 INFO:tasks.rados.rados.0.smithi157.stdout:943: set_chunk oid 14 offset: 1278976 length: 323584 target oid 030809787146be52785a28ada48c88d20b6479ecb45d1331e5aa4ee718a393dc offset: 0 snap -1 2024-09-16T03:37:58.161 INFO:tasks.rados.rados.0.smithi157.stdout:944: read oid 44 snap -1 2024-09-16T03:37:58.161 INFO:tasks.rados.rados.0.smithi157.stdout:944: expect (ObjNum 63 snap 0 seq_num 63) 2024-09-16T03:37:58.161 INFO:tasks.rados.rados.0.smithi157.stdout:945: read oid 15 snap -1 2024-09-16T03:37:58.161 INFO:tasks.rados.rados.0.smithi157.stdout:945: expect (ObjNum 136 snap 0 seq_num 136) 2024-09-16T03:37:58.161 INFO:tasks.rados.rados.0.smithi157.stdout:946: write (excl) oid 39 current snap is 0 2024-09-16T03:37:58.161 INFO:tasks.rados.rados.0.smithi157.stdout:946: seq_num 150 ranges {521050=698057,1877946=643104} 2024-09-16T03:37:58.167 INFO:tasks.rados.rados.0.smithi157.stdout:946: writing smithi15736696-39 from 521050 to 1219107 tid 1 2024-09-16T03:37:58.175 INFO:tasks.rados.rados.0.smithi157.stdout:946: writing smithi15736696-39 from 1877946 to 2521050 tid 2 2024-09-16T03:37:58.175 INFO:tasks.rados.rados.0.smithi157.stdout:947: read oid 49 snap -1 2024-09-16T03:37:58.175 INFO:tasks.rados.rados.0.smithi157.stdout:947: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-16T03:37:58.175 INFO:tasks.rados.rados.0.smithi157.stdout:948: read oid 35 snap -1 2024-09-16T03:37:58.175 INFO:tasks.rados.rados.0.smithi157.stdout:948: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:37:58.175 INFO:tasks.rados.rados.0.smithi157.stdout:949: delete oid 24 current snap is 0 2024-09-16T03:37:58.175 INFO:tasks.rados.rados.0.smithi157.stdout:941: got (22) Invalid argument 2024-09-16T03:37:58.175 INFO:tasks.rados.rados.0.smithi157.stdout:940: finishing write tid 1 to smithi15736696-43 2024-09-16T03:37:58.175 INFO:tasks.rados.rados.0.smithi157.stdout:936: finishing write tid 1 to smithi15736696-13 2024-09-16T03:37:58.175 INFO:tasks.rados.rados.0.smithi157.stdout:939: finishing write tid 1 to smithi15736696-12 2024-09-16T03:37:58.175 INFO:tasks.rados.rados.0.smithi157.stdout:936: finishing write tid 2 to smithi15736696-13 2024-09-16T03:37:58.175 INFO:tasks.rados.rados.0.smithi157.stdout:936: finishing write tid 3 to smithi15736696-13 2024-09-16T03:37:58.175 INFO:tasks.rados.rados.0.smithi157.stdout:939: finishing write tid 2 to smithi15736696-12 2024-09-16T03:37:58.176 INFO:tasks.rados.rados.0.smithi157.stdout:939: finishing write tid 3 to smithi15736696-12 2024-09-16T03:37:58.176 INFO:tasks.rados.rados.0.smithi157.stdout:939: finishing write tid 4 to smithi15736696-12 2024-09-16T03:37:58.176 INFO:tasks.rados.rados.0.smithi157.stdout:939: oid 12 updating version 0 to 10 2024-09-16T03:37:58.176 INFO:tasks.rados.rados.0.smithi157.stdout:939: oid 12 updating version 10 to 12 2024-09-16T03:37:58.176 INFO:tasks.rados.rados.0.smithi157.stdout:939: oid 12 version 12 is already newer than 11 2024-09-16T03:37:58.176 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 12 (ObjNum 148 snap 0 seq_num 148) dirty exists 2024-09-16T03:37:58.176 INFO:tasks.rados.rados.0.smithi157.stdout:939: left oid 12 (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:37:58.176 INFO:tasks.rados.rados.0.smithi157.stdout:936: finishing write tid 4 to smithi15736696-13 2024-09-16T03:37:58.176 INFO:tasks.rados.rados.0.smithi157.stdout:936: oid 13 updating version 0 to 40 2024-09-16T03:37:58.176 INFO:tasks.rados.rados.0.smithi157.stdout:936: oid 13 version 40 is already newer than 38 2024-09-16T03:37:58.176 INFO:tasks.rados.rados.0.smithi157.stdout:936: oid 13 version 40 is already newer than 39 2024-09-16T03:37:58.176 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 40 (ObjNum 145 snap 0 seq_num 145) dirty exists 2024-09-16T03:37:58.176 INFO:tasks.rados.rados.0.smithi157.stdout:936: left oid 13 (ObjNum 145 snap 0 seq_num 145) 2024-09-16T03:37:58.177 INFO:tasks.rados.rados.0.smithi157.stdout:940: finishing write tid 2 to smithi15736696-43 2024-09-16T03:37:58.177 INFO:tasks.rados.rados.0.smithi157.stdout:940: finishing write tid 3 to smithi15736696-43 2024-09-16T03:37:58.177 INFO:tasks.rados.rados.0.smithi157.stdout:940: finishing write tid 4 to smithi15736696-43 2024-09-16T03:37:58.177 INFO:tasks.rados.rados.0.smithi157.stdout:940: oid 43 updating version 0 to 45 2024-09-16T03:37:58.177 INFO:tasks.rados.rados.0.smithi157.stdout:940: oid 43 updating version 45 to 47 2024-09-16T03:37:58.177 INFO:tasks.rados.rados.0.smithi157.stdout:940: oid 43 version 47 is already newer than 46 2024-09-16T03:37:58.177 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 47 (ObjNum 149 snap 0 seq_num 149) dirty exists 2024-09-16T03:37:58.177 INFO:tasks.rados.rados.0.smithi157.stdout:940: left oid 43 (ObjNum 149 snap 0 seq_num 149) 2024-09-16T03:37:58.177 INFO:tasks.rados.rados.0.smithi157.stdout:944: expect (ObjNum 63 snap 0 seq_num 63) 2024-09-16T03:37:58.185 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:58.183+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:58.186 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:58.186 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:58.195 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:37:58.286 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:37:58.284+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:37:58.302 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:58.302 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:58.311 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:37:58.329 INFO:tasks.rados.rados.0.smithi157.stdout:947: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-16T03:37:58.329 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 36904 2024-09-16T03:37:58.344 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:37:58.342+0000 7f4214d04640 -1 osd.8 72 handle_osd_map injecting map crc failure 2024-09-16T03:37:58.346 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:37:58.347 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:58.344+0000 7eff7884b640 -1 osd.3 72 handle_osd_map injecting map crc failure 2024-09-16T03:37:58.361 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:58.387 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:58.384+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:58.422 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:58.422 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:58.431 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:37:58.459 INFO:tasks.rados.rados.0.smithi157.stdout:945: expect (ObjNum 136 snap 0 seq_num 136) 2024-09-16T03:37:58.459 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:58.488 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:37:58.487+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:37:58.492 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:58.542 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:58.544 INFO:tasks.rados.rados.0.smithi157.stdout:948: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-16T03:37:58.547 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 643551 2024-09-16T03:37:58.551 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:37:58.589 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:37:58.587+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:37:58.614 INFO:tasks.rados.rados.0.smithi157.stdout:946: finishing write tid 1 to smithi15736696-39 2024-09-16T03:37:58.614 INFO:tasks.rados.rados.0.smithi157.stdout:930: got (0) Success 2024-09-16T03:37:58.614 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 0 (ObjNum 111 snap 0 seq_num 111) dirty exists 2024-09-16T03:37:58.614 INFO:tasks.rados.rados.0.smithi157.stdout:930: done (12 left) 2024-09-16T03:37:58.614 INFO:tasks.rados.rados.0.smithi157.stdout:936: done (11 left) 2024-09-16T03:37:58.614 INFO:tasks.rados.rados.0.smithi157.stdout:939: done (10 left) 2024-09-16T03:37:58.614 INFO:tasks.rados.rados.0.smithi157.stdout:940: done (9 left) 2024-09-16T03:37:58.614 INFO:tasks.rados.rados.0.smithi157.stdout:941: done (8 left) 2024-09-16T03:37:58.614 INFO:tasks.rados.rados.0.smithi157.stdout:942: done (7 left) 2024-09-16T03:37:58.614 INFO:tasks.rados.rados.0.smithi157.stdout:944: done (6 left) 2024-09-16T03:37:58.614 INFO:tasks.rados.rados.0.smithi157.stdout:945: done (5 left) 2024-09-16T03:37:58.615 INFO:tasks.rados.rados.0.smithi157.stdout:947: done (4 left) 2024-09-16T03:37:58.615 INFO:tasks.rados.rados.0.smithi157.stdout:948: done (3 left) 2024-09-16T03:37:58.615 INFO:tasks.rados.rados.0.smithi157.stdout:949: done (2 left) 2024-09-16T03:37:58.615 INFO:tasks.rados.rados.0.smithi157.stdout:950: read oid 41 snap -1 2024-09-16T03:37:58.615 INFO:tasks.rados.rados.0.smithi157.stdout:950: expect (ObjNum 112 snap 0 seq_num 112) 2024-09-16T03:37:58.615 INFO:tasks.rados.rados.0.smithi157.stdout:951: set_chunk oid 38 target oid 2024-09-16T03:37:58.615 INFO:tasks.rados.rados.0.smithi157.stdout:951: set_chunk oid 38 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:37:58.615 INFO:tasks.rados.rados.0.smithi157.stdout:952: tier_evict oid 40 2024-09-16T03:37:58.615 INFO:tasks.rados.rados.0.smithi157.stdout:952: tier_evict oid 40 snap -1 2024-09-16T03:37:58.615 INFO:tasks.rados.rados.0.smithi157.stdout:953: set_chunk oid 35 target oid 2024-09-16T03:37:58.658 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:58.659 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:58.667 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:37:58.672 INFO:tasks.rados.rados.0.smithi157.stdout:953: set_chunk oid 35 offset: 982016 length: 1069056 target oid ce29c1230032357166cd45d2f751fac48c51f77a78a480dc3f78b8fd0ff53e70 offset: 0 snap -1 2024-09-16T03:37:58.672 INFO:tasks.rados.rados.0.smithi157.stdout:954: write oid 22 current snap is 0 2024-09-16T03:37:58.672 INFO:tasks.rados.rados.0.smithi157.stdout:951: finishing set_chunk to oid 38 2024-09-16T03:37:58.672 INFO:tasks.rados.rados.0.smithi157.stdout:951: got expected ENOENT (src dne) 2024-09-16T03:37:58.672 INFO:tasks.rados.rados.0.smithi157.stdout:954: seq_num 152 ranges {554664=448555,1478557=664373} 2024-09-16T03:37:58.677 INFO:tasks.rados.rados.0.smithi157.stdout:954: writing smithi15736696-22 from 554664 to 1003219 tid 1 2024-09-16T03:37:58.684 INFO:tasks.rados.rados.0.smithi157.stdout:954: writing smithi15736696-22 from 1478557 to 2142930 tid 2 2024-09-16T03:37:58.684 INFO:tasks.rados.rados.0.smithi157.stdout:951: done (6 left) 2024-09-16T03:37:58.684 INFO:tasks.rados.rados.0.smithi157.stdout:955: read oid 45 snap -1 2024-09-16T03:37:58.684 INFO:tasks.rados.rados.0.smithi157.stdout:955: expect deleted 2024-09-16T03:37:58.684 INFO:tasks.rados.rados.0.smithi157.stdout:956: read oid 16 snap -1 2024-09-16T03:37:58.684 INFO:tasks.rados.rados.0.smithi157.stdout:956: expect (ObjNum 122 snap 0 seq_num 122) 2024-09-16T03:37:58.684 INFO:tasks.rados.rados.0.smithi157.stdout:957: tier_promote oid 5 2024-09-16T03:37:58.684 INFO:tasks.rados.rados.0.smithi157.stdout:946: finishing write tid 2 to smithi15736696-39 2024-09-16T03:37:58.685 INFO:tasks.rados.rados.0.smithi157.stdout:958: read oid 31 snap -1 2024-09-16T03:37:58.685 INFO:tasks.rados.rados.0.smithi157.stdout:958: expect deleted 2024-09-16T03:37:58.685 INFO:tasks.rados.rados.0.smithi157.stdout:953: finishing set_chunk to oid 35 2024-09-16T03:37:58.685 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 35 set_chunk returned error code -95 offset: 982016 length: 1069056 2024-09-16T03:37:58.685 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T03:37:58.685 INFO:tasks.rados.rados.0.smithi157.stdout:959: read oid 4 snap -1 2024-09-16T03:37:58.685 INFO:tasks.rados.rados.0.smithi157.stdout:959: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:37:58.685 INFO:tasks.rados.rados.0.smithi157.stdout:953: done (10 left) 2024-09-16T03:37:58.685 INFO:tasks.rados.rados.0.smithi157.stdout:960: read oid 19 snap -1 2024-09-16T03:37:58.685 INFO:tasks.rados.rados.0.smithi157.stdout:960: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-16T03:37:58.685 INFO:tasks.rados.rados.0.smithi157.stdout:961: write (excl) oid 3 current snap is 0 2024-09-16T03:37:58.685 INFO:tasks.rados.rados.0.smithi157.stdout:961: seq_num 153 ranges {571471=640157,1653392=524384} 2024-09-16T03:37:58.689 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:58.687+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:58.690 INFO:tasks.rados.rados.0.smithi157.stdout:961: writing smithi15736696-3 from 571471 to 1211628 tid 1 2024-09-16T03:37:58.695 INFO:tasks.rados.rados.0.smithi157.stdout:961: writing smithi15736696-3 from 1653392 to 2177776 tid 2 2024-09-16T03:37:58.695 INFO:tasks.rados.rados.0.smithi157.stdout:962: read oid 42 snap -1 2024-09-16T03:37:58.696 INFO:tasks.rados.rados.0.smithi157.stdout:962: expect (ObjNum 142 snap 0 seq_num 142) 2024-09-16T03:37:58.696 INFO:tasks.rados.rados.0.smithi157.stdout:963: read oid 25 snap -1 2024-09-16T03:37:58.696 INFO:tasks.rados.rados.0.smithi157.stdout:963: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-16T03:37:58.696 INFO:tasks.rados.rados.0.smithi157.stdout:964: read oid 15 snap -1 2024-09-16T03:37:58.696 INFO:tasks.rados.rados.0.smithi157.stdout:964: expect (ObjNum 136 snap 0 seq_num 136) 2024-09-16T03:37:58.696 INFO:tasks.rados.rados.0.smithi157.stdout:958: done (14 left) 2024-09-16T03:37:58.696 INFO:tasks.rados.rados.0.smithi157.stdout:965: write (excl) oid 36 current snap is 0 2024-09-16T03:37:58.696 INFO:tasks.rados.rados.0.smithi157.stdout:965: seq_num 154 ranges {588278=715406,1711874=668042} 2024-09-16T03:37:58.702 INFO:tasks.rados.rados.0.smithi157.stdout:965: writing smithi15736696-36 from 588278 to 1303684 tid 1 2024-09-16T03:37:58.708 INFO:tasks.rados.rados.0.smithi157.stdout:965: writing smithi15736696-36 from 1711874 to 2379916 tid 2 2024-09-16T03:37:58.708 INFO:tasks.rados.rados.0.smithi157.stdout:966: read oid 28 snap -1 2024-09-16T03:37:58.708 INFO:tasks.rados.rados.0.smithi157.stdout:966: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-16T03:37:58.708 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:58.783 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:58.784 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:58.786 INFO:tasks.rados.rados.0.smithi157.stdout:962: expect (ObjNum 142 snap 0 seq_num 142) 2024-09-16T03:37:58.790 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:37:58.789+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:37:58.792 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:37:58.809 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1791659 2024-09-16T03:37:58.891 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:37:58.889+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:37:58.896 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:58.896 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:58.905 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:37:58.912 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1871721 2024-09-16T03:37:58.937 INFO:tasks.rados.rados.0.smithi157.stdout:964: expect (ObjNum 136 snap 0 seq_num 136) 2024-09-16T03:37:58.937 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:58.992 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:58.989+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:59.023 INFO:tasks.rados.rados.0.smithi157.stdout:961: finishing write tid 1 to smithi15736696-3 2024-09-16T03:37:59.023 INFO:tasks.rados.rados.0.smithi157.stdout:946: finishing write tid 3 to smithi15736696-39 2024-09-16T03:37:59.023 INFO:tasks.rados.rados.0.smithi157.stdout:946: finishing write tid 4 to smithi15736696-39 2024-09-16T03:37:59.023 INFO:tasks.rados.rados.0.smithi157.stdout:946: oid 39 updating version 0 to 50 2024-09-16T03:37:59.023 INFO:tasks.rados.rados.0.smithi157.stdout:946: oid 39 version 50 is already newer than 48 2024-09-16T03:37:59.023 INFO:tasks.rados.rados.0.smithi157.stdout:946: oid 39 version 50 is already newer than 49 2024-09-16T03:37:59.023 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 50 (ObjNum 150 snap 0 seq_num 150) dirty exists 2024-09-16T03:37:59.023 INFO:tasks.rados.rados.0.smithi157.stdout:946: left oid 39 (ObjNum 150 snap 0 seq_num 150) 2024-09-16T03:37:59.024 INFO:tasks.rados.rados.0.smithi157.stdout:946: done (15 left) 2024-09-16T03:37:59.024 INFO:tasks.rados.rados.0.smithi157.stdout:962: done (14 left) 2024-09-16T03:37:59.024 INFO:tasks.rados.rados.0.smithi157.stdout:964: done (13 left) 2024-09-16T03:37:59.024 INFO:tasks.rados.rados.0.smithi157.stdout:967: read oid 31 snap -1 2024-09-16T03:37:59.024 INFO:tasks.rados.rados.0.smithi157.stdout:967: expect deleted 2024-09-16T03:37:59.024 INFO:tasks.rados.rados.0.smithi157.stdout:968: write (excl) oid 46 current snap is 0 2024-09-16T03:37:59.024 INFO:tasks.rados.rados.0.smithi157.stdout:956: expect (ObjNum 122 snap 0 seq_num 122) 2024-09-16T03:37:59.049 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:59.049 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:59.057 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:37:59.064 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1876952 2024-09-16T03:37:59.066 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 77440 2024-09-16T03:37:59.093 INFO:tasks.rados.rados.0.smithi157.stdout:959: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:37:59.093 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:37:59.092+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:37:59.117 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:59.160 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:59.160 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:59.161 INFO:tasks.rados.rados.0.smithi157.stdout:961: finishing write tid 2 to smithi15736696-3 2024-09-16T03:37:59.161 INFO:tasks.rados.rados.0.smithi157.stdout:961: finishing write tid 3 to smithi15736696-3 2024-09-16T03:37:59.162 INFO:tasks.rados.rados.0.smithi157.stdout:961: finishing write tid 4 to smithi15736696-3 2024-09-16T03:37:59.162 INFO:tasks.rados.rados.0.smithi157.stdout:961: oid 3 updating version 0 to 43 2024-09-16T03:37:59.162 INFO:tasks.rados.rados.0.smithi157.stdout:961: oid 3 updating version 43 to 44 2024-09-16T03:37:59.162 INFO:tasks.rados.rados.0.smithi157.stdout:961: oid 3 updating version 44 to 45 2024-09-16T03:37:59.162 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 45 (ObjNum 153 snap 0 seq_num 153) dirty exists 2024-09-16T03:37:59.162 INFO:tasks.rados.rados.0.smithi157.stdout:961: left oid 3 (ObjNum 153 snap 0 seq_num 153) 2024-09-16T03:37:59.162 INFO:tasks.rados.rados.0.smithi157.stdout:960: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-16T03:37:59.170 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:37:59.187 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:59.194 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:37:59.191+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:37:59.224 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:59.259 INFO:tasks.rados.rados.0.smithi157.stdout:943: finishing set_chunk to oid 14 2024-09-16T03:37:59.260 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 36 (ObjNum 131 snap 0 seq_num 131) dirty exists 2024-09-16T03:37:59.260 INFO:tasks.rados.rados.0.smithi157.stdout:952: got (22) Invalid argument 2024-09-16T03:37:59.260 INFO:tasks.rados.rados.0.smithi157.stdout:968: seq_num 155 ranges {605085=790655,2170356=434729} 2024-09-16T03:37:59.267 INFO:tasks.rados.rados.0.smithi157.stdout:968: writing smithi15736696-46 from 605085 to 1395740 tid 1 2024-09-16T03:37:59.273 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:59.273 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:59.274 INFO:tasks.rados.rados.0.smithi157.stdout:968: writing smithi15736696-46 from 2170356 to 2605085 tid 2 2024-09-16T03:37:59.274 INFO:tasks.rados.rados.0.smithi157.stdout:943: done (14 left) 2024-09-16T03:37:59.274 INFO:tasks.rados.rados.0.smithi157.stdout:952: done (13 left) 2024-09-16T03:37:59.274 INFO:tasks.rados.rados.0.smithi157.stdout:956: done (12 left) 2024-09-16T03:37:59.274 INFO:tasks.rados.rados.0.smithi157.stdout:959: done (11 left) 2024-09-16T03:37:59.274 INFO:tasks.rados.rados.0.smithi157.stdout:960: done (10 left) 2024-09-16T03:37:59.274 INFO:tasks.rados.rados.0.smithi157.stdout:961: done (9 left) 2024-09-16T03:37:59.274 INFO:tasks.rados.rados.0.smithi157.stdout:969: write oid 26 current snap is 0 2024-09-16T03:37:59.274 INFO:tasks.rados.rados.0.smithi157.stdout:965: finishing write tid 1 to smithi15736696-36 2024-09-16T03:37:59.274 INFO:tasks.rados.rados.0.smithi157.stdout:969: seq_num 156 ranges {621892=465904,1712485=671711} 2024-09-16T03:37:59.279 INFO:tasks.rados.rados.0.smithi157.stdout:969: writing smithi15736696-26 from 621892 to 1087796 tid 1 2024-09-16T03:37:59.283 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:37:59.290 INFO:tasks.rados.rados.0.smithi157.stdout:969: writing smithi15736696-26 from 1712485 to 2384196 tid 2 2024-09-16T03:37:59.290 INFO:tasks.rados.rados.0.smithi157.stdout:970: set_chunk oid 13 target oid 2024-09-16T03:37:59.295 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:59.293+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:59.387 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:59.387 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:59.396 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:37:59.396 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:37:59.394+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:37:59.397 INFO:tasks.rados.rados.0.smithi157.stdout:970: set_chunk oid 13 offset: 1068544 length: 1033728 target oid ccf59f4130696ef67b0ed908ebe15d5d92f6b26e4e268953c0f3425294e6d6ea offset: 0 snap -1 2024-09-16T03:37:59.397 INFO:tasks.rados.rados.0.smithi157.stdout:971: write oid 38 current snap is 0 2024-09-16T03:37:59.397 INFO:tasks.rados.rados.0.smithi157.stdout:966: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-16T03:37:59.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:37:59.475 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:37:59.475 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:37:59.475 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:37:59.475 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-16T03:37:59.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 3.0 2024-09-16T03:37:59.496 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:37:59.494+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:37:59.500 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:59.500 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:59.509 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:37:59.515 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 60344 2024-09-16T03:37:59.570 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:37:59.571 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:37:59.580 INFO:tasks.rados.rados.0.smithi157.stdout:950: expect (ObjNum 112 snap 0 seq_num 112) 2024-09-16T03:37:59.588 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:37:59.588 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:37:59.597 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:37:59.596+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:37:59.607 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:37:59.607 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:37:59.613 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:37:59.613 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:37:59.622 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:37:59.624 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:37:59.625 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:37:59.640 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:37:59.641 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:37:59.657 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:37:59.657 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:37:59.674 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:37:59.674 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:37:59.691 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:37:59.691 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:37:59.701 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:59.700+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:59.710 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:37:59.710 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:37:59.726 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:37:59.726 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:37:59.728 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:37:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:37:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:37:59.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:3 3.0 3.f934d18 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:37:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 44.326270719999997, 2024-09-16T03:37:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78381636700000001, 2024-09-16T03:37:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:37:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399429+0000", 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399434+0000", 2024-09-16T03:37:59.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1050000000000003e-06 2024-09-16T03:37:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399442+0000", 2024-09-16T03:37:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0160000000000008e-06 2024-09-16T03:37:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399448+0000", 2024-09-16T03:37:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.879e-06 2024-09-16T03:37:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399489+0000", 2024-09-16T03:37:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0815999999999997e-05 2024-09-16T03:37:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399495+0000", 2024-09-16T03:37:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3439999999999997e-06 2024-09-16T03:37:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183226+0000", 2024-09-16T03:37:59.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78373077400000002 2024-09-16T03:37:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183244+0000", 2024-09-16T03:37:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8128999999999999e-05 2024-09-16T03:37:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.733 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:5 3.0 3.aefbbec0 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:37:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 44.226503258000001, 2024-09-16T03:37:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68447923499999996, 2024-09-16T03:37:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:37:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:37:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499196+0000", 2024-09-16T03:37:59.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3480000000000001e-06 2024-09-16T03:37:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499202+0000", 2024-09-16T03:37:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4219999999999999e-06 2024-09-16T03:37:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499212+0000", 2024-09-16T03:37:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:37:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499218+0000", 2024-09-16T03:37:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4920000000000001e-06 2024-09-16T03:37:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499283+0000", 2024-09-16T03:37:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4906000000000005e-05 2024-09-16T03:37:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499293+0000", 2024-09-16T03:37:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0513000000000001e-05 2024-09-16T03:37:59.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183665+0000", 2024-09-16T03:37:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68437196600000005 2024-09-16T03:37:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183674+0000", 2024-09-16T03:37:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9700000000000005e-06 2024-09-16T03:37:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:6 3.0 3.3bcda120 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 44.226442648999999, 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78474037699999999, 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:37:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:37:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0500000000000002e-07 2024-09-16T03:37:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499260+0000", 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6669999999999998e-06 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499266+0000", 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499274+0000", 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2250000000000003e-06 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499316+0000", 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2670000000000003e-05 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499323+0000", 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.37e-06 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283987+0000", 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78466433700000005 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283996+0000", 2024-09-16T03:37:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8650000000000007e-06 2024-09-16T03:37:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:15 3.0 3.7ec93288 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:59.743 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:37:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 44.117250638999998, 2024-09-16T03:37:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57518172599999995, 2024-09-16T03:37:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:37:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:37:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608449+0000", 2024-09-16T03:37:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.184e-06 2024-09-16T03:37:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608459+0000", 2024-09-16T03:37:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0553999999999999e-05 2024-09-16T03:37:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608470+0000", 2024-09-16T03:37:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0524e-05 2024-09-16T03:37:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608493+0000", 2024-09-16T03:37:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2980999999999999e-05 2024-09-16T03:37:59.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608537+0000", 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3850000000000002e-05 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608542+0000", 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183623+0000", 2024-09-16T03:37:59.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.575080649 2024-09-16T03:37:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183629+0000", 2024-09-16T03:37:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6950000000000003e-06 2024-09-16T03:37:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:11 3.0 3.77005160 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:37:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:37:59.748 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 44.023750329000002, 2024-09-16T03:37:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.48153733399999998, 2024-09-16T03:37:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:37:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.749 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701949+0000", 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.314e-06 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701954+0000", 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701964+0000", 2024-09-16T03:37:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9550000000000004e-06 2024-09-16T03:37:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701986+0000", 2024-09-16T03:37:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1931999999999999e-05 2024-09-16T03:37:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802129+0000", 2024-09-16T03:37:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100142986 2024-09-16T03:37:59.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802135+0000", 2024-09-16T03:37:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6350000000000003e-06 2024-09-16T03:37:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183475+0000", 2024-09-16T03:37:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38133958099999998 2024-09-16T03:37:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183485+0000", 2024-09-16T03:37:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0642000000000001e-05 2024-09-16T03:37:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:6 3.0 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:37:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.742846415999999, 2024-09-16T03:37:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60469085300000003, 2024-09-16T03:37:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.753 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:37:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:37:59.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982853+0000", 2024-09-16T03:37:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.393e-06 2024-09-16T03:37:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982858+0000", 2024-09-16T03:37:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.062e-06 2024-09-16T03:37:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982867+0000", 2024-09-16T03:37:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:37:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982879+0000", 2024-09-16T03:37:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2241e-05 2024-09-16T03:37:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982996+0000", 2024-09-16T03:37:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116996 2024-09-16T03:37:59.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983007+0000", 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0694e-05 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284499+0000", 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30149171400000002 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284690+0000", 2024-09-16T03:37:59.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000191032 2024-09-16T03:37:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284908+0000", 2024-09-16T03:37:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021827399999999999 2024-09-16T03:37:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487092+0000", 2024-09-16T03:37:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.202184156 2024-09-16T03:37:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487110+0000", 2024-09-16T03:37:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7856999999999999e-05 2024-09-16T03:37:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587335+0000", 2024-09-16T03:37:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100224802 2024-09-16T03:37:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587410+0000", 2024-09-16T03:37:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5343e-05 2024-09-16T03:37:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587543+0000", 2024-09-16T03:37:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000132492 2024-09-16T03:37:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:2 3.0 3:16550cfd:::smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:37:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.742774224000001, 2024-09-16T03:37:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40289696000000003, 2024-09-16T03:37:59.761 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982925+0000", 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982928+0000", 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4130000000000002e-06 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982936+0000", 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.875e-06 2024-09-16T03:37:59.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982942+0000", 2024-09-16T03:37:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:37:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983093+0000", 2024-09-16T03:37:59.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015144400000000001 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983098+0000", 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4909999999999999e-06 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385695+0000", 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402597539 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385821+0000", 2024-09-16T03:37:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125418 2024-09-16T03:37:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:9 3.0 3:2c9b9e7c:::smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:37:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:37:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.742758035000001, 2024-09-16T03:37:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60615655800000001, 2024-09-16T03:37:59.766 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:37:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:37:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982941+0000", 2024-09-16T03:37:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0860000000000001e-06 2024-09-16T03:37:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982946+0000", 2024-09-16T03:37:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:37:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982954+0000", 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8520000000000004e-06 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982958+0000", 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4870000000000004e-06 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983116+0000", 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015774699999999999 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983120+0000", 2024-09-16T03:37:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:37:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486023+0000", 2024-09-16T03:37:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50290281400000003 2024-09-16T03:37:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:59.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486423+0000", 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040079299999999998 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486707+0000", 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028336600000000001 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588861+0000", 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102154756 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588952+0000", 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0630000000000005e-05 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588984+0000", 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2116999999999998e-05 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589022+0000", 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7910000000000001e-05 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589097+0000", 2024-09-16T03:37:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4536999999999995e-05 2024-09-16T03:37:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:12 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:37:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:37:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 42.742682686999999, 2024-09-16T03:37:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60616102900000002, 2024-09-16T03:37:59.773 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:37:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:37:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:37:59.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983017+0000", 2024-09-16T03:37:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.483e-06 2024-09-16T03:37:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983022+0000", 2024-09-16T03:37:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:37:59.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983032+0000", 2024-09-16T03:37:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0036e-05 2024-09-16T03:37:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983038+0000", 2024-09-16T03:37:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6800000000000004e-06 2024-09-16T03:37:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983155+0000", 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116401 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983163+0000", 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6219999999999998e-06 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589120+0000", 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60595653199999999 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589176+0000", 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6542000000000002e-05 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:5 3.0 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.811493757999997, 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30268081699999999, 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.778 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914207+0000", 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1500000000000002e-06 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914215+0000", 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7530000000000007e-06 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914232+0000", 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.628e-05 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914242+0000", 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.058e-05 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914305+0000", 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3294999999999996e-05 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914377+0000", 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1698000000000004e-05 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914517+0000", 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000139443 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.015450+0000", 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100933668 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.015480+0000", 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9467000000000001e-05 2024-09-16T03:37:59.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216766+0000", 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20128665300000001 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216831+0000", 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4220999999999994e-05 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216885+0000", 2024-09-16T03:37:59.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4608999999999998e-05 2024-09-16T03:37:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:24 3.0 3:2dd0f25f:::smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:37:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.711159361999997, 2024-09-16T03:37:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30270049300000001, 2024-09-16T03:37:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:59.784 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:37:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:37:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014541+0000", 2024-09-16T03:37:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2479999999999999e-06 2024-09-16T03:37:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014550+0000", 2024-09-16T03:37:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.115e-06 2024-09-16T03:37:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014567+0000", 2024-09-16T03:37:59.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7292999999999999e-05 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014577+0000", 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8020000000000003e-06 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115566+0000", 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10098905900000001 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:59.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115702+0000", 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013602500000000001 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115893+0000", 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019095700000000001 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216936+0000", 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101042992 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317125+0000", 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100188757 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317155+0000", 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0009999999999999e-05 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317210+0000", 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4589999999999997e-05 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317239+0000", 2024-09-16T03:37:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9645000000000002e-05 2024-09-16T03:37:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:30 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:37:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:37:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.509364181000002, 2024-09-16T03:37:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40312788999999999, 2024-09-16T03:37:59.790 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:37:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:37:59.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216338+0000", 2024-09-16T03:37:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0799999999999999e-06 2024-09-16T03:37:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216354+0000", 2024-09-16T03:37:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6439e-05 2024-09-16T03:37:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216392+0000", 2024-09-16T03:37:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7162000000000001e-05 2024-09-16T03:37:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216444+0000", 2024-09-16T03:37:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2082e-05 2024-09-16T03:37:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417447+0000", 2024-09-16T03:37:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20100358400000001 2024-09-16T03:37:59.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417583+0000", 2024-09-16T03:37:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135537 2024-09-16T03:37:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417777+0000", 2024-09-16T03:37:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019431200000000001 2024-09-16T03:37:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:59.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418479+0000", 2024-09-16T03:37:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00070148499999999996 2024-09-16T03:37:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418755+0000", 2024-09-16T03:37:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027627100000000003 2024-09-16T03:37:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619285+0000", 2024-09-16T03:37:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200530138 2024-09-16T03:37:59.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619370+0000", 2024-09-16T03:37:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4567999999999994e-05 2024-09-16T03:37:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619462+0000", 2024-09-16T03:37:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2231999999999994e-05 2024-09-16T03:37:59.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:18 3.0 3:037ddf75:::smithi15736696-34 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 32.789769909999997, 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30645336299999998, 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935929+0000", 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4600000000000003e-07 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935936+0000", 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3199999999999996e-06 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935943+0000", 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8900000000000007e-06 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935954+0000", 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0246e-05 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140278+0000", 2024-09-16T03:37:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20432396999999999 2024-09-16T03:37:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140342+0000", 2024-09-16T03:37:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4442000000000004e-05 2024-09-16T03:37:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140468+0000", 2024-09-16T03:37:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125718 2024-09-16T03:37:59.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.141460+0000", 2024-09-16T03:37:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00099208400000000006 2024-09-16T03:37:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242240+0000", 2024-09-16T03:37:59.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100779733 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242316+0000", 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6038999999999995e-05 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242351+0000", 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4928999999999997e-05 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242382+0000", 2024-09-16T03:37:59.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1046000000000001e-05 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:34 3.0 3:08307d25:::2b3cb797ae2d3abc24b2cae4a012f26325debe2fc38ccb08bd4503e949d305ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 28.250908123999999, 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30798127600000003, 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474791+0000", 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6000000000000003e-07 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474865+0000", 2024-09-16T03:37:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4504000000000004e-05 2024-09-16T03:37:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474882+0000", 2024-09-16T03:37:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6765999999999998e-05 2024-09-16T03:37:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474890+0000", 2024-09-16T03:37:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.14e-06 2024-09-16T03:37:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681242+0000", 2024-09-16T03:37:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20635221200000001 2024-09-16T03:37:59.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681355+0000", 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000112211 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681505+0000", 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015040100000000001 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682352+0000", 2024-09-16T03:37:59.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084686400000000002 2024-09-16T03:37:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682989+0000", 2024-09-16T03:37:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063749800000000001 2024-09-16T03:37:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782695+0000", 2024-09-16T03:37:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099706063999999997 2024-09-16T03:37:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782735+0000", 2024-09-16T03:37:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9304000000000002e-05 2024-09-16T03:37:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782771+0000", 2024-09-16T03:37:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6551999999999998e-05 2024-09-16T03:37:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.809 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:73 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:46.277582+0000", 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.448116076, 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30283920399999997, 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277582+0000", 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277582+0000", 2024-09-16T03:37:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277584+0000", 2024-09-16T03:37:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.708e-06 2024-09-16T03:37:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277586+0000", 2024-09-16T03:37:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.526e-06 2024-09-16T03:37:59.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277592+0000", 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3190000000000002e-06 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277595+0000", 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6279999999999998e-06 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.378846+0000", 2024-09-16T03:37:59.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10125093 2024-09-16T03:37:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379007+0000", 2024-09-16T03:37:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000161046 2024-09-16T03:37:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379230+0000", 2024-09-16T03:37:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022235000000000001 2024-09-16T03:37:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.480276+0000", 2024-09-16T03:37:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101046812 2024-09-16T03:37:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.480479+0000", 2024-09-16T03:37:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020296100000000001 2024-09-16T03:37:59.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580317+0000", 2024-09-16T03:37:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099837767999999993 2024-09-16T03:37:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:59.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580373+0000", 2024-09-16T03:37:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6249999999999998e-05 2024-09-16T03:37:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580421+0000", 2024-09-16T03:37:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7905999999999998e-05 2024-09-16T03:37:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.817 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:74 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:37:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:46.277614+0000", 2024-09-16T03:37:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.448084428, 2024-09-16T03:37:59.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.302949411, 2024-09-16T03:37:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277614+0000", 2024-09-16T03:37:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277614+0000", 2024-09-16T03:37:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277614+0000", 2024-09-16T03:37:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4300000000000003e-07 2024-09-16T03:37:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277618+0000", 2024-09-16T03:37:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0369999999999996e-06 2024-09-16T03:37:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277622+0000", 2024-09-16T03:37:59.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9790000000000004e-06 2024-09-16T03:37:59.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277626+0000", 2024-09-16T03:37:59.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2629999999999999e-06 2024-09-16T03:37:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379529+0000", 2024-09-16T03:37:59.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101903627 2024-09-16T03:37:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379609+0000", 2024-09-16T03:37:59.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9721000000000005e-05 2024-09-16T03:37:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379690+0000", 2024-09-16T03:37:59.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1267000000000006e-05 2024-09-16T03:37:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.480491+0000", 2024-09-16T03:37:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100801053 2024-09-16T03:37:59.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.480528+0000", 2024-09-16T03:37:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6871e-05 2024-09-16T03:37:59.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580470+0000", 2024-09-16T03:37:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099941422000000002 2024-09-16T03:37:59.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580525+0000", 2024-09-16T03:37:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5197e-05 2024-09-16T03:37:59.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580563+0000", 2024-09-16T03:37:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8430999999999998e-05 2024-09-16T03:37:59.830 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.831 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.831 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.831 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:77 3.0 3:1a18dfd0:::9d9929bebc2b17b3e19b9b502dd3b0e63f1b7b4c1cf7796e8161d7773d261a75:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:37:59.832 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.802529+0000", 2024-09-16T03:37:59.832 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 8.9231692500000008, 2024-09-16T03:37:59.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30442216599999999, 2024-09-16T03:37:59.832 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.832 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.833 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:59.833 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.802529+0000", 2024-09-16T03:37:59.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.802529+0000", 2024-09-16T03:37:59.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.802530+0000", 2024-09-16T03:37:59.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.311e-06 2024-09-16T03:37:59.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803091+0000", 2024-09-16T03:37:59.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056044899999999997 2024-09-16T03:37:59.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803101+0000", 2024-09-16T03:37:59.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9189999999999994e-06 2024-09-16T03:37:59.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803106+0000", 2024-09-16T03:37:59.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.626e-06 2024-09-16T03:37:59.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.903907+0000", 2024-09-16T03:37:59.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100800951 2024-09-16T03:37:59.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:59.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.904063+0000", 2024-09-16T03:37:59.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015626 2024-09-16T03:37:59.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:59.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.904369+0000", 2024-09-16T03:37:59.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030599300000000001 2024-09-16T03:37:59.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:59.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907929+0000", 2024-09-16T03:37:59.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035599080000000001 2024-09-16T03:37:59.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.006526+0000", 2024-09-16T03:37:59.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098596345000000002 2024-09-16T03:37:59.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.106808+0000", 2024-09-16T03:37:59.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100282198 2024-09-16T03:37:59.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:59.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.106917+0000", 2024-09-16T03:37:59.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010936 2024-09-16T03:37:59.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.106951+0000", 2024-09-16T03:37:59.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3846000000000001e-05 2024-09-16T03:37:59.846 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.847 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.847 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:78 3.0 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:37:59.847 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803117+0000", 2024-09-16T03:37:59.848 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 8.9225807330000002, 2024-09-16T03:37:59.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30401020200000001, 2024-09-16T03:37:59.848 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.848 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.848 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:59.849 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803117+0000", 2024-09-16T03:37:59.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803117+0000", 2024-09-16T03:37:59.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803118+0000", 2024-09-16T03:37:59.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0400000000000005e-07 2024-09-16T03:37:59.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803503+0000", 2024-09-16T03:37:59.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038435999999999999 2024-09-16T03:37:59.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803515+0000", 2024-09-16T03:37:59.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2279999999999999e-05 2024-09-16T03:37:59.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803521+0000", 2024-09-16T03:37:59.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.054e-06 2024-09-16T03:37:59.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.905093+0000", 2024-09-16T03:37:59.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101572302 2024-09-16T03:37:59.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:59.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.905196+0000", 2024-09-16T03:37:59.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000102876 2024-09-16T03:37:59.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:59.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.905387+0000", 2024-09-16T03:37:59.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019066499999999999 2024-09-16T03:37:59.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:59.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907935+0000", 2024-09-16T03:37:59.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025477949999999998 2024-09-16T03:37:59.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.106742+0000", 2024-09-16T03:37:59.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19880740599999999 2024-09-16T03:37:59.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107021+0000", 2024-09-16T03:37:59.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027904200000000002 2024-09-16T03:37:59.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:59.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107083+0000", 2024-09-16T03:37:59.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2149000000000004e-05 2024-09-16T03:37:59.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107128+0000", 2024-09-16T03:37:59.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4468999999999997e-05 2024-09-16T03:37:59.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.862 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.862 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:79 3.0 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:37:59.863 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803530+0000", 2024-09-16T03:37:59.863 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 8.9221686229999992, 2024-09-16T03:37:59.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30378224199999998, 2024-09-16T03:37:59.863 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.863 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.863 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:59.864 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803530+0000", 2024-09-16T03:37:59.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803530+0000", 2024-09-16T03:37:59.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803530+0000", 2024-09-16T03:37:59.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8000000000000005e-07 2024-09-16T03:37:59.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803942+0000", 2024-09-16T03:37:59.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00041188299999999999 2024-09-16T03:37:59.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803952+0000", 2024-09-16T03:37:59.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.027e-05 2024-09-16T03:37:59.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803957+0000", 2024-09-16T03:37:59.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0159999999999999e-06 2024-09-16T03:37:59.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.906129+0000", 2024-09-16T03:37:59.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102171915 2024-09-16T03:37:59.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:59.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.906280+0000", 2024-09-16T03:37:59.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000150447 2024-09-16T03:37:59.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:59.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.906551+0000", 2024-09-16T03:37:59.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027134700000000002 2024-09-16T03:37:59.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:59.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907944+0000", 2024-09-16T03:37:59.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013923410000000001 2024-09-16T03:37:59.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107181+0000", 2024-09-16T03:37:59.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19923718800000001 2024-09-16T03:37:59.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107248+0000", 2024-09-16T03:37:59.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7341999999999993e-05 2024-09-16T03:37:59.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:59.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107283+0000", 2024-09-16T03:37:59.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4711e-05 2024-09-16T03:37:59.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107312+0000", 2024-09-16T03:37:59.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9002000000000001e-05 2024-09-16T03:37:59.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.877 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.878 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:87 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:37:59.878 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:37:59.878 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 8.9217133999999998, 2024-09-16T03:37:59.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402368742, 2024-09-16T03:37:59.879 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:37:59.879 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:37:59.879 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:37:59.879 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:37:59.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:37:59.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:37:59.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803986+0000", 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9400000000000004e-07 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804061+0000", 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5230999999999999e-05 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:37:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804069+0000", 2024-09-16T03:37:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7530000000000001e-06 2024-09-16T03:37:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:37:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804073+0000", 2024-09-16T03:37:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9660000000000004e-06 2024-09-16T03:37:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:37:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907236+0000", 2024-09-16T03:37:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103162566 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907324+0000", 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7894999999999998e-05 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907474+0000", 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014983999999999999 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:37:59.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.908330+0000", 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085600100000000001 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107220+0000", 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198889812 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206269+0000", 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099049611999999995 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:37:59.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206305+0000", 2024-09-16T03:37:59.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5679000000000002e-05 2024-09-16T03:37:59.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:37:59.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:37:59.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:37:59.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206353+0000", 2024-09-16T03:37:59.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8593e-05 2024-09-16T03:37:59.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.884 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:37:59.884 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:37:59.884 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:37:59.886 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:37:59.886 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:37:59.887 INFO:teuthology.orchestra.run.smithi064.stdout:moved 0 / 117 (0%) 2024-09-16T03:37:59.887 INFO:teuthology.orchestra.run.smithi064.stdout:avg 9.75 2024-09-16T03:37:59.887 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 3.85411 -> 3.85411 (expected baseline 2.98957) 2024-09-16T03:37:59.887 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.12 with 4 -> 4 pgs (0.410256 -> 0.410256 * mean) 2024-09-16T03:37:59.887 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.10 with 15 -> 15 pgs (1.53846 -> 1.53846 * mean) 2024-09-16T03:37:59.887 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:37:59.887 INFO:teuthology.orchestra.run.smithi064.stdout:oload 1000 2024-09-16T03:37:59.887 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 3 2024-09-16T03:37:59.887 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-09-16T03:37:59.887 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0006 2024-09-16T03:37:59.887 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0056 2024-09-16T03:37:59.888 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:37:59.890 INFO:teuthology.orchestra.run.smithi064.stderr:no change 2024-09-16T03:37:59.890 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-16T03:37:59.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 3.0 2024-09-16T03:37:59.890 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:37:59.891 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:37:59.891 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:37:59.892 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:37:59.892 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:37:59.893 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:37:59.893 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:37:59.894 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:37:59.894 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:37:59.884+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:37:59.896 INFO:tasks.rados.rados.0.smithi157.stdout:963: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-16T03:37:59.896 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:59.896 INFO:tasks.rados.rados.0.smithi157.stdout:965: finishing write tid 2 to smithi15736696-36 2024-09-16T03:37:59.896 INFO:tasks.rados.rados.0.smithi157.stdout:965: finishing write tid 3 to smithi15736696-36 2024-09-16T03:37:59.896 INFO:tasks.rados.rados.0.smithi157.stdout:965: finishing write tid 4 to smithi15736696-36 2024-09-16T03:37:59.896 INFO:tasks.rados.rados.0.smithi157.stdout:965: oid 36 updating version 0 to 71 2024-09-16T03:37:59.896 INFO:tasks.rados.rados.0.smithi157.stdout:965: oid 36 version 71 is already newer than 70 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:965: oid 36 updating version 71 to 72 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 72 (ObjNum 154 snap 0 seq_num 154) dirty exists 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:965: left oid 36 (ObjNum 154 snap 0 seq_num 154) 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:954: finishing write tid 1 to smithi15736696-22 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:957: got (0) Success 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 31 (ObjNum 111 snap 0 seq_num 111) dirty exists 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:971: seq_num 157 ranges {638699=541153,1770967=531722} 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:971: writing smithi15736696-38 from 638699 to 1179852 tid 1 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:971: writing smithi15736696-38 from 1770967 to 2302689 tid 2 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:950: done (11 left) 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:955: done (10 left) 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:957: done (9 left) 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:963: done (8 left) 2024-09-16T03:37:59.897 INFO:tasks.rados.rados.0.smithi157.stdout:965: done (7 left) 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:966: done (6 left) 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:972: read oid 3 snap -1 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:972: expect (ObjNum 153 snap 0 seq_num 153) 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:973: read oid 16 snap -1 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:973: expect (ObjNum 122 snap 0 seq_num 122) 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:974: write (excl) oid 37 current snap is 0 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:974: seq_num 158 ranges {655506=616402,1829449=675380} 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:974: writing smithi15736696-37 from 655506 to 1271908 tid 1 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:974: writing smithi15736696-37 from 1829449 to 2504829 tid 2 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:975: write oid 18 current snap is 0 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:954: finishing write tid 2 to smithi15736696-22 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:954: finishing write tid 3 to smithi15736696-22 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:954: finishing write tid 4 to smithi15736696-22 2024-09-16T03:37:59.898 INFO:tasks.rados.rados.0.smithi157.stdout:954: oid 22 updating version 0 to 57 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:954: oid 22 updating version 57 to 59 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:954: oid 22 version 59 is already newer than 58 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 59 (ObjNum 152 snap 0 seq_num 152) dirty exists 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:954: left oid 22 (ObjNum 152 snap 0 seq_num 152) 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:969: finishing write tid 1 to smithi15736696-26 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:975: seq_num 159 ranges {672313=691651,1887931=419038} 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:975: writing smithi15736696-18 from 672313 to 1363964 tid 1 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:975: writing smithi15736696-18 from 1887931 to 2306969 tid 2 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:954: done (9 left) 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:967: done (8 left) 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:976: read oid 27 snap -1 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:976: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:977: read oid 43 snap -1 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:977: expect (ObjNum 149 snap 0 seq_num 149) 2024-09-16T03:37:59.899 INFO:tasks.rados.rados.0.smithi157.stdout:978: read oid 49 snap -1 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:978: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:979: read oid 22 snap -1 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:979: expect (ObjNum 152 snap 0 seq_num 152) 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:980: read oid 19 snap -1 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:980: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:981: read oid 45 snap -1 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:981: expect deleted 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:982: tier_promote oid 42 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:983: write oid 2 current snap is 0 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:968: finishing write tid 1 to smithi15736696-46 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:968: finishing write tid 2 to smithi15736696-46 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:983: seq_num 160 ranges {689120=483253,1946413=679049} 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:983: writing smithi15736696-2 from 689120 to 1172373 tid 1 2024-09-16T03:37:59.900 INFO:tasks.rados.rados.0.smithi157.stdout:983: writing smithi15736696-2 from 1946413 to 2625462 tid 2 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout:969: finishing write tid 2 to smithi15736696-26 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout:969: finishing write tid 3 to smithi15736696-26 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout:969: finishing write tid 4 to smithi15736696-26 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout:969: oid 26 updating version 0 to 60 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout:969: oid 26 updating version 60 to 61 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout:969: oid 26 updating version 61 to 62 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 62 (ObjNum 156 snap 0 seq_num 156) dirty exists 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout:969: left oid 26 (ObjNum 156 snap 0 seq_num 156) 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout:968: finishing write tid 3 to smithi15736696-46 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout:969: done (15 left) 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout:984: read oid 23 snap -1 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout:984: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:59.901 INFO:tasks.rados.rados.0.smithi157.stdout:970: finishing set_chunk to oid 13 2024-09-16T03:37:59.902 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 40 (ObjNum 145 snap 0 seq_num 145) dirty exists 2024-09-16T03:37:59.902 INFO:tasks.rados.rados.0.smithi157.stdout:968: finishing write tid 4 to smithi15736696-46 2024-09-16T03:37:59.902 INFO:tasks.rados.rados.0.smithi157.stdout:968: oid 46 updating version 0 to 58 2024-09-16T03:37:59.902 INFO:tasks.rados.rados.0.smithi157.stdout:968: oid 46 updating version 58 to 60 2024-09-16T03:37:59.902 INFO:tasks.rados.rados.0.smithi157.stdout:968: oid 46 version 60 is already newer than 59 2024-09-16T03:37:59.902 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 60 (ObjNum 155 snap 0 seq_num 155) dirty exists 2024-09-16T03:37:59.902 INFO:tasks.rados.rados.0.smithi157.stdout:968: left oid 46 (ObjNum 155 snap 0 seq_num 155) 2024-09-16T03:37:59.902 INFO:tasks.rados.rados.0.smithi157.stdout:968: done (15 left) 2024-09-16T03:37:59.902 INFO:tasks.rados.rados.0.smithi157.stdout:970: done (14 left) 2024-09-16T03:37:59.902 INFO:tasks.rados.rados.0.smithi157.stdout:985: write oid 35 current snap is 0 2024-09-16T03:37:59.902 INFO:tasks.rados.rados.0.smithi157.stdout:985: seq_num 161 ranges {705927=558502,2004895=422707} 2024-09-16T03:37:59.902 INFO:tasks.rados.rados.0.smithi157.stdout:985: writing smithi15736696-35 from 705927 to 1264429 tid 1 2024-09-16T03:37:59.904 INFO:tasks.rados.rados.0.smithi157.stdout:985: writing smithi15736696-35 from 2004895 to 2427602 tid 2 2024-09-16T03:37:59.904 INFO:tasks.rados.rados.0.smithi157.stdout:986: tier_evict oid 48 2024-09-16T03:37:59.904 INFO:tasks.rados.rados.0.smithi157.stdout:986: tier_evict oid 48 snap -1 2024-09-16T03:37:59.904 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:37:59.904 INFO:tasks.rados.rados.0.smithi157.stdout:973: expect (ObjNum 122 snap 0 seq_num 122) 2024-09-16T03:37:59.904 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 122360 2024-09-16T03:37:59.928 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:59.957 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:37:59.987 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:37:59.984+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:37:59.989 INFO:tasks.rados.rados.0.smithi157.stdout:972: expect (ObjNum 153 snap 0 seq_num 153) 2024-09-16T03:37:59.998 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:37:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:37:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:37:59.999 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:00.007 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:38:00.013 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:00.013 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:00.050 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:00.087 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:00.086+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:00.110 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:00.110 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:00.111 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:00.111 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:00.111 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:00.120 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:38:00.171 INFO:tasks.rados.rados.0.smithi157.stdout:971: finishing write tid 1 to smithi15736696-38 2024-09-16T03:38:00.172 INFO:tasks.rados.rados.0.smithi157.stdout:974: finishing write tid 1 to smithi15736696-37 2024-09-16T03:38:00.172 INFO:tasks.rados.rados.0.smithi157.stdout:977: expect (ObjNum 149 snap 0 seq_num 149) 2024-09-16T03:38:00.172 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 291942 2024-09-16T03:38:00.188 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:00.185+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:00.223 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.223 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:57 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 48.096388355999999, 2024-09-16T03:38:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20372090900000001, 2024-09-16T03:38:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125313+0000", 2024-09-16T03:38:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7180000000000001e-06 2024-09-16T03:38:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:00.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126730+0000", 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00141731 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126746+0000", 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6455000000000001e-05 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126756+0000", 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227068+0000", 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100312308 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227085+0000", 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6898e-05 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328995+0000", 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101909903 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329031+0000", 2024-09-16T03:38:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5717999999999999e-05 2024-09-16T03:38:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:58 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 48.093055225999997, 2024-09-16T03:38:00.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084524500000001, 2024-09-16T03:38:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:00.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128644+0000", 2024-09-16T03:38:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.251e-06 2024-09-16T03:38:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:00.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128913+0000", 2024-09-16T03:38:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026856200000000002 2024-09-16T03:38:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128925+0000", 2024-09-16T03:38:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2109999999999999e-05 2024-09-16T03:38:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128933+0000", 2024-09-16T03:38:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6260000000000003e-06 2024-09-16T03:38:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328196+0000", 2024-09-16T03:38:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19926303200000001 2024-09-16T03:38:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328253+0000", 2024-09-16T03:38:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7262999999999997e-05 2024-09-16T03:38:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:00.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329458+0000", 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012051939999999999 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329488+0000", 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0207000000000001e-05 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:59 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'3, mlcod=0'0)", 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:00.233 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 48.092559793, 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036954900000001, 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:00.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129139+0000", 2024-09-16T03:38:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6099999999999999e-07 2024-09-16T03:38:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129158+0000", 2024-09-16T03:38:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088e-05 2024-09-16T03:38:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129168+0000", 2024-09-16T03:38:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.048e-05 2024-09-16T03:38:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.229233+0000", 2024-09-16T03:38:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100065168 2024-09-16T03:38:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328815+0000", 2024-09-16T03:38:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099581694999999998 2024-09-16T03:38:00.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328845+0000", 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0193e-05 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329492+0000", 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064671800000000001 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329508+0000", 2024-09-16T03:38:00.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6246e-05 2024-09-16T03:38:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.0:15 4.d e47/32 4:b3fe05b2:::smithi15736696-15:head v 47'5, mlcod=39'3)", 2024-09-16T03:38:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.919265066999998, 2024-09-16T03:38:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100920751, 2024-09-16T03:38:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.238 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302435+0000", 2024-09-16T03:38:00.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2340000000000001e-06 2024-09-16T03:38:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302446+0000", 2024-09-16T03:38:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0596e-05 2024-09-16T03:38:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302461+0000", 2024-09-16T03:38:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5082000000000001e-05 2024-09-16T03:38:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402592+0000", 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100130488 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402712+0000", 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012013100000000001 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402741+0000", 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9717999999999999e-05 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403309+0000", 2024-09-16T03:38:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056725700000000005 2024-09-16T03:38:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403354+0000", 2024-09-16T03:38:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5244999999999998e-05 2024-09-16T03:38:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:38:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.071766358, 2024-09-16T03:38:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032593800000000001, 2024-09-16T03:38:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.243 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149943+0000", 2024-09-16T03:38:00.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0817e-05 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151026+0000", 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010835350000000001 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151040+0000", 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3884e-05 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151049+0000", 2024-09-16T03:38:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:38:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151107+0000", 2024-09-16T03:38:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8199000000000003e-05 2024-09-16T03:38:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151124+0000", 2024-09-16T03:38:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7756000000000001e-05 2024-09-16T03:38:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153142+0000", 2024-09-16T03:38:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020177569999999998 2024-09-16T03:38:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153191+0000", 2024-09-16T03:38:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9051999999999997e-05 2024-09-16T03:38:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1682 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'7, mlcod=40'4)", 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.9681691679999993, 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10169502499999999, 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253540+0000", 2024-09-16T03:38:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0888e-05 2024-09-16T03:38:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254015+0000", 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047485499999999999 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254030+0000", 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4899000000000001e-05 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254039+0000", 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5000000000000005e-06 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254107+0000", 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7298000000000005e-05 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254122+0000", 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5928999999999999e-05 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355184+0000", 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101061703 2024-09-16T03:38:00.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355224+0000", 2024-09-16T03:38:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9953000000000002e-05 2024-09-16T03:38:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1683 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'8, mlcod=40'4)", 2024-09-16T03:38:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.9676316200000006, 2024-09-16T03:38:00.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101630885, 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254068+0000", 2024-09-16T03:38:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5700000000000001e-07 2024-09-16T03:38:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254075+0000", 2024-09-16T03:38:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2710000000000003e-06 2024-09-16T03:38:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:00.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254083+0000", 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0220000000000004e-06 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254089+0000", 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8699999999999997e-06 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354968+0000", 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100879625 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354992+0000", 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355665+0000", 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00067371100000000001 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355698+0000", 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2291e-05 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.256 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.0:177 4.10 e65/57 4:0a55bbf2:::smithi15736696-18:head v 65'47, mlcod=0'0)", 2024-09-16T03:38:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.0146850319999992, 2024-09-16T03:38:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074461099999999995, 2024-09-16T03:38:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.257 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5300000000000001e-07 2024-09-16T03:38:00.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207018+0000", 2024-09-16T03:38:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7210000000000002e-06 2024-09-16T03:38:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207031+0000", 2024-09-16T03:38:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2767e-05 2024-09-16T03:38:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207040+0000", 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5400000000000001e-06 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207107+0000", 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6625999999999995e-05 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207131+0000", 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4128e-05 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207712+0000", 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00058124700000000001 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207758+0000", 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5429000000000002e-05 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.261 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1723 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'48, mlcod=53'46)", 2024-09-16T03:38:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 8.5726178510000004, 2024-09-16T03:38:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00088284599999999996, 2024-09-16T03:38:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649081+0000", 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8299999999999995e-07 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649101+0000", 2024-09-16T03:38:00.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0126e-05 2024-09-16T03:38:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649141+0000", 2024-09-16T03:38:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9609e-05 2024-09-16T03:38:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649161+0000", 2024-09-16T03:38:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9599999999999999e-05 2024-09-16T03:38:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649299+0000", 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013822199999999999 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649318+0000", 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9273999999999999e-05 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649919+0000", 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060074100000000001 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649963+0000", 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4391000000000002e-05 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1724 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'49, mlcod=53'46)", 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 8.5689528859999999, 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064536300000000003, 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:00.266 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652747+0000", 2024-09-16T03:38:00.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.291e-06 2024-09-16T03:38:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652753+0000", 2024-09-16T03:38:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4450000000000001e-06 2024-09-16T03:38:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652782+0000", 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9094999999999998e-05 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652797+0000", 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5021999999999999e-05 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652854+0000", 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6586999999999999e-05 2024-09-16T03:38:00.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652885+0000", 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.146e-05 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653350+0000", 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046443599999999998 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:00.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653391+0000", 2024-09-16T03:38:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1026999999999999e-05 2024-09-16T03:38:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1725 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'50, mlcod=53'46)", 2024-09-16T03:38:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 8.5681600630000005, 2024-09-16T03:38:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056324100000000002, 2024-09-16T03:38:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:00.271 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.272 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653539+0000", 2024-09-16T03:38:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9699999999999994e-07 2024-09-16T03:38:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653545+0000", 2024-09-16T03:38:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9309999999999996e-06 2024-09-16T03:38:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653557+0000", 2024-09-16T03:38:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2340999999999999e-05 2024-09-16T03:38:00.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653564+0000", 2024-09-16T03:38:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2779999999999999e-06 2024-09-16T03:38:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653605+0000", 2024-09-16T03:38:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1301000000000003e-05 2024-09-16T03:38:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:00.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653621+0000", 2024-09-16T03:38:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5951e-05 2024-09-16T03:38:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654064+0000", 2024-09-16T03:38:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000442843 2024-09-16T03:38:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654101+0000", 2024-09-16T03:38:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7598999999999997e-05 2024-09-16T03:38:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.275 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:00.276 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:00.276 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:00.277 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-09-16T03:38:00.250+0000 7f5019b9e640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-16T03:38:00.277 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:00.277 INFO:teuthology.orchestra.run.smithi064.stdout:moved 14 / 117 (11.9658%) 2024-09-16T03:38:00.277 INFO:teuthology.orchestra.run.smithi064.stdout:avg 9.75 2024-09-16T03:38:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 3.85411 -> 3.05846 (expected baseline 2.98957) 2024-09-16T03:38:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.12 with 4 -> 6 pgs (0.410256 -> 0.615385 * mean) 2024-09-16T03:38:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.10 with 15 -> 14 pgs (1.53846 -> 1.4359 * mean) 2024-09-16T03:38:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:38:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:oload 110 2024-09-16T03:38:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 3 2024-09-16T03:38:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-09-16T03:38:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0006 2024-09-16T03:38:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0006 2024-09-16T03:38:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 weight 1.0000 -> 0.6409 2024-09-16T03:38:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15 weight 1.0000 -> 0.7130 2024-09-16T03:38:00.278 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11 weight 1.0000 -> 0.8510 2024-09-16T03:38:00.279 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10 weight 1.0000 -> 0.8967 2024-09-16T03:38:00.279 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:38:00.280 INFO:teuthology.orchestra.run.smithi064.stderr:no change 2024-09-16T03:38:00.280 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-16T03:38:00.280 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2024-09-16T03:38:00.289 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:00.288+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:00.344 INFO:tasks.rados.rados.0.smithi157.stdout:976: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:38:00.361 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:38:00.369 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:00.393 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:00.394 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:00.394 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:00.394 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:00.396 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:00.402 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:38:00.405 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:00.404+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:00.422 INFO:tasks.rados.rados.0.smithi157.stdout:986: got (2) No such file or directory 2024-09-16T03:38:00.422 INFO:tasks.rados.rados.0.smithi157.stdout:986: got expected ENOENT (src dne) 2024-09-16T03:38:00.422 INFO:tasks.rados.rados.0.smithi157.stdout:978: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-16T03:38:00.422 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:00.453 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:00.490 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:00.489+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:00.509 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:00.509 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:00.514 INFO:tasks.rados.rados.0.smithi157.stdout:979: expect (ObjNum 152 snap 0 seq_num 152) 2024-09-16T03:38:00.517 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:38:00.524 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1706120 2024-09-16T03:38:00.556 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:00.554+0000 7f12e2a1a640 -1 osd.5 73 handle_osd_map injecting map crc failure 2024-09-16T03:38:00.570 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1876744 2024-09-16T03:38:00.578 INFO:tasks.rados.rados.0.smithi157.stdout:984: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T03:38:00.591 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:00.589+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:00.593 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:00.591+0000 7f0ae754b640 -1 osd.0 73 handle_osd_map injecting map crc failure 2024-09-16T03:38:00.594 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-09-16T03:38:00.592+0000 7f5019b9e640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-16T03:38:00.594 INFO:teuthology.orchestra.run.smithi064.stdout:moved 14 / 117 (11.9658%) 2024-09-16T03:38:00.594 INFO:teuthology.orchestra.run.smithi064.stdout:avg 9.75 2024-09-16T03:38:00.594 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 3.85411 -> 3.05846 (expected baseline 2.98957) 2024-09-16T03:38:00.594 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.12 with 4 -> 6 pgs (0.410256 -> 0.615385 * mean) 2024-09-16T03:38:00.594 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.10 with 15 -> 14 pgs (1.53846 -> 1.4359 * mean) 2024-09-16T03:38:00.594 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:38:00.594 INFO:teuthology.orchestra.run.smithi064.stdout:oload 110 2024-09-16T03:38:00.594 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 1 2024-09-16T03:38:00.595 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-09-16T03:38:00.595 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0006 2024-09-16T03:38:00.595 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0006 2024-09-16T03:38:00.595 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 weight 1.0000 -> 0.6409 2024-09-16T03:38:00.595 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15 weight 1.0000 -> 0.7130 2024-09-16T03:38:00.595 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11 weight 1.0000 -> 0.8510 2024-09-16T03:38:00.595 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10 weight 1.0000 -> 0.8967 2024-09-16T03:38:00.595 INFO:teuthology.orchestra.run.smithi064.stderr:no change 2024-09-16T03:38:00.607 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-16T03:38:00.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2024-09-16T03:38:00.613 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2166946 2024-09-16T03:38:00.615 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 337055 2024-09-16T03:38:00.624 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:00.624 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:00.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:00.625 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:00.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.625 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:5 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'1, mlcod=0'0)", 2024-09-16T03:38:00.625 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:38:00.625 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 48.653781918, 2024-09-16T03:38:00.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.105742802, 2024-09-16T03:38:00.625 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.625 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.625 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.625 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:38:00.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:38:00.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968147+0000", 2024-09-16T03:38:00.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4e-07 2024-09-16T03:38:00.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968752+0000", 2024-09-16T03:38:00.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060437900000000005 2024-09-16T03:38:00.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968762+0000", 2024-09-16T03:38:00.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0284e-05 2024-09-16T03:38:00.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.628 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968778+0000", 2024-09-16T03:38:00.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6124e-05 2024-09-16T03:38:00.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.628 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069797+0000", 2024-09-16T03:38:00.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10101895700000001 2024-09-16T03:38:00.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069833+0000", 2024-09-16T03:38:00.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6013000000000002e-05 2024-09-16T03:38:00.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073838+0000", 2024-09-16T03:38:00.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040049049999999996 2024-09-16T03:38:00.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073889+0000", 2024-09-16T03:38:00.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.13e-05 2024-09-16T03:38:00.630 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.630 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.630 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.630 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:7 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'3, mlcod=0'0)", 2024-09-16T03:38:00.630 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:38:00.630 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 48.646889418999997, 2024-09-16T03:38:00.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20061241399999999, 2024-09-16T03:38:00.631 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.631 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.631 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.631 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:38:00.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:38:00.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975042+0000", 2024-09-16T03:38:00.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5289999999999998e-06 2024-09-16T03:38:00.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975079+0000", 2024-09-16T03:38:00.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7626999999999997e-05 2024-09-16T03:38:00.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975104+0000", 2024-09-16T03:38:00.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4958999999999999e-05 2024-09-16T03:38:00.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.075211+0000", 2024-09-16T03:38:00.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100106814 2024-09-16T03:38:00.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080573+0000", 2024-09-16T03:38:00.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0053618650000000004 2024-09-16T03:38:00.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080592+0000", 2024-09-16T03:38:00.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9079000000000001e-05 2024-09-16T03:38:00.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175497+0000", 2024-09-16T03:38:00.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.094904832999999994 2024-09-16T03:38:00.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175651+0000", 2024-09-16T03:38:00.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015470800000000001 2024-09-16T03:38:00.636 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.636 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.636 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.636 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:00.636 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:00.636 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 48.435936247000001, 2024-09-16T03:38:00.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10311670100000001, 2024-09-16T03:38:00.637 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.637 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.637 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.637 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:00.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:00.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:00.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:38:00.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186761+0000", 2024-09-16T03:38:00.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076880999999999998 2024-09-16T03:38:00.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186772+0000", 2024-09-16T03:38:00.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0986e-05 2024-09-16T03:38:00.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186799+0000", 2024-09-16T03:38:00.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6254e-05 2024-09-16T03:38:00.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186876+0000", 2024-09-16T03:38:00.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7523999999999998e-05 2024-09-16T03:38:00.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186896+0000", 2024-09-16T03:38:00.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0302999999999999e-05 2024-09-16T03:38:00.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289077+0000", 2024-09-16T03:38:00.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10218086899999999 2024-09-16T03:38:00.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289109+0000", 2024-09-16T03:38:00.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1625000000000003e-05 2024-09-16T03:38:00.641 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.641 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.641 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.642 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36)", 2024-09-16T03:38:00.642 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:38:00.642 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 48.434730457999997, 2024-09-16T03:38:00.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102408122, 2024-09-16T03:38:00.642 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.642 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.642 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.642 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:38:00.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:38:00.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187199+0000", 2024-09-16T03:38:00.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6199999999999996e-07 2024-09-16T03:38:00.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188184+0000", 2024-09-16T03:38:00.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000985442 2024-09-16T03:38:00.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188194+0000", 2024-09-16T03:38:00.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5659999999999996e-06 2024-09-16T03:38:00.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188210+0000", 2024-09-16T03:38:00.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5858999999999998e-05 2024-09-16T03:38:00.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288363+0000", 2024-09-16T03:38:00.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100152872 2024-09-16T03:38:00.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288390+0000", 2024-09-16T03:38:00.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7081999999999999e-05 2024-09-16T03:38:00.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289577+0000", 2024-09-16T03:38:00.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011875740000000001 2024-09-16T03:38:00.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289606+0000", 2024-09-16T03:38:00.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8864999999999999e-05 2024-09-16T03:38:00.647 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.647 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.647 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.647 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:10 3.3 e41/31 3:c069f14c:::smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 41'16, mlcod=41'14)", 2024-09-16T03:38:00.647 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.701871+0000", 2024-09-16T03:38:00.647 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 44.920056975000001, 2024-09-16T03:38:00.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100943475, 2024-09-16T03:38:00.647 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.648 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.648 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.648 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701871+0000", 2024-09-16T03:38:00.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701871+0000", 2024-09-16T03:38:00.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701873+0000", 2024-09-16T03:38:00.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1659999999999999e-06 2024-09-16T03:38:00.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701879+0000", 2024-09-16T03:38:00.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6540000000000002e-06 2024-09-16T03:38:00.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701891+0000", 2024-09-16T03:38:00.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.182e-05 2024-09-16T03:38:00.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701898+0000", 2024-09-16T03:38:00.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3270000000000003e-06 2024-09-16T03:38:00.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802103+0000", 2024-09-16T03:38:00.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10020419899999999 2024-09-16T03:38:00.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802141+0000", 2024-09-16T03:38:00.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8411999999999997e-05 2024-09-16T03:38:00.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802782+0000", 2024-09-16T03:38:00.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00064137500000000002 2024-09-16T03:38:00.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.802815+0000", 2024-09-16T03:38:00.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2521999999999998e-05 2024-09-16T03:38:00.652 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.652 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.652 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.653 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.5.21:19 3.3 e47/31 3:fb369346:::smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 47'44, mlcod=47'41)", 2024-09-16T03:38:00.653 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.339975+0000", 2024-09-16T03:38:00.653 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.281953741999999, 2024-09-16T03:38:00.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10201615, 2024-09-16T03:38:00.653 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.653 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.653 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.653 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339975+0000", 2024-09-16T03:38:00.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339975+0000", 2024-09-16T03:38:00.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339976+0000", 2024-09-16T03:38:00.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.204e-06 2024-09-16T03:38:00.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339982+0000", 2024-09-16T03:38:00.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:38:00.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339992+0000", 2024-09-16T03:38:00.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0196e-05 2024-09-16T03:38:00.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339998+0000", 2024-09-16T03:38:00.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.463e-06 2024-09-16T03:38:00.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.438719+0000", 2024-09-16T03:38:00.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098721712000000003 2024-09-16T03:38:00.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.438747+0000", 2024-09-16T03:38:00.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7475e-05 2024-09-16T03:38:00.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.441935+0000", 2024-09-16T03:38:00.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031878480000000001 2024-09-16T03:38:00.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.441991+0000", 2024-09-16T03:38:00.658 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6100999999999997e-05 2024-09-16T03:38:00.658 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.658 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.658 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.658 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:44 3.3 e49/31 3:dc284e8d:::smithi15736696-28 low_tier:head v 49'55, mlcod=49'53)", 2024-09-16T03:38:00.658 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:38:00.658 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.498965023, 2024-09-16T03:38:00.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.149341897, 2024-09-16T03:38:00.659 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.659 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.659 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.659 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:38:00.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:38:00.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122964+0000", 2024-09-16T03:38:00.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.91e-07 2024-09-16T03:38:00.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122968+0000", 2024-09-16T03:38:00.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1470000000000003e-06 2024-09-16T03:38:00.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.661 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122979+0000", 2024-09-16T03:38:00.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1090000000000001e-05 2024-09-16T03:38:00.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.661 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.223110+0000", 2024-09-16T03:38:00.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100131357 2024-09-16T03:38:00.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.662 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271688+0000", 2024-09-16T03:38:00.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048578192999999999 2024-09-16T03:38:00.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.662 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271712+0000", 2024-09-16T03:38:00.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4066e-05 2024-09-16T03:38:00.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272260+0000", 2024-09-16T03:38:00.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054737000000000004 2024-09-16T03:38:00.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272305+0000", 2024-09-16T03:38:00.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5482999999999998e-05 2024-09-16T03:38:00.663 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.663 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.664 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.664 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1007 3.3 e50/31 3:fda66a0d:::920f29fe85d0745d4cd24a887714ac53f1b9a978ec2208ff473c99287fdf8feb:head v 50'57, mlcod=49'55)", 2024-09-16T03:38:00.664 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:38:00.664 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.027338469, 2024-09-16T03:38:00.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20236485200000001, 2024-09-16T03:38:00.664 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.664 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.664 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.665 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:38:00.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:38:00.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.666 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594592+0000", 2024-09-16T03:38:00.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0700000000000001e-06 2024-09-16T03:38:00.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.666 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595508+0000", 2024-09-16T03:38:00.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00091599499999999996 2024-09-16T03:38:00.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595521+0000", 2024-09-16T03:38:00.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:38:00.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595531+0000", 2024-09-16T03:38:00.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2499999999999995e-06 2024-09-16T03:38:00.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795762+0000", 2024-09-16T03:38:00.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200230977 2024-09-16T03:38:00.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795778+0000", 2024-09-16T03:38:00.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:38:00.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796912+0000", 2024-09-16T03:38:00.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011338520000000001 2024-09-16T03:38:00.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796955+0000", 2024-09-16T03:38:00.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2768000000000001e-05 2024-09-16T03:38:00.669 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.669 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.669 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.669 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:57 3.3 e54/31 3:c069f14c:::smithi15736696-37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 54'65, mlcod=53'63)", 2024-09-16T03:38:00.669 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:34.964218+0000", 2024-09-16T03:38:00.670 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 25.657710760000001, 2024-09-16T03:38:00.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10111977699999999, 2024-09-16T03:38:00.670 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.670 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.670 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.670 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.670 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.670 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964218+0000", 2024-09-16T03:38:00.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964218+0000", 2024-09-16T03:38:00.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964218+0000", 2024-09-16T03:38:00.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8900000000000001e-07 2024-09-16T03:38:00.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964222+0000", 2024-09-16T03:38:00.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8929999999999998e-06 2024-09-16T03:38:00.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964231+0000", 2024-09-16T03:38:00.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1449999999999997e-06 2024-09-16T03:38:00.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964239+0000", 2024-09-16T03:38:00.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2369999999999992e-06 2024-09-16T03:38:00.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.064498+0000", 2024-09-16T03:38:00.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100259205 2024-09-16T03:38:00.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.064537+0000", 2024-09-16T03:38:00.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8486999999999999e-05 2024-09-16T03:38:00.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.065291+0000", 2024-09-16T03:38:00.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00075431700000000001 2024-09-16T03:38:00.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.065337+0000", 2024-09-16T03:38:00.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6304000000000003e-05 2024-09-16T03:38:00.675 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.675 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.675 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.675 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:58 3.3 e54/31 3:e9b7d9c2:::f15a4bc1b12b17a23021c0e3457eb377158b231decb6e67343dc236bab942ef9:head v 54'66, mlcod=53'63)", 2024-09-16T03:38:00.675 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:34.964431+0000", 2024-09-16T03:38:00.675 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 25.657497708000001, 2024-09-16T03:38:00.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101289616, 2024-09-16T03:38:00.675 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.675 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.676 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.676 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964431+0000", 2024-09-16T03:38:00.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.676 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964431+0000", 2024-09-16T03:38:00.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964431+0000", 2024-09-16T03:38:00.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9999999999999997e-07 2024-09-16T03:38:00.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964445+0000", 2024-09-16T03:38:00.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3375e-05 2024-09-16T03:38:00.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.678 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964453+0000", 2024-09-16T03:38:00.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0029999999999994e-06 2024-09-16T03:38:00.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.678 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:34.964462+0000", 2024-09-16T03:38:00.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1919999999999994e-06 2024-09-16T03:38:00.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.064937+0000", 2024-09-16T03:38:00.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10047542600000001 2024-09-16T03:38:00.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.064952+0000", 2024-09-16T03:38:00.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4623999999999999e-05 2024-09-16T03:38:00.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.065678+0000", 2024-09-16T03:38:00.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072640599999999997 2024-09-16T03:38:00.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:35.065720+0000", 2024-09-16T03:38:00.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1990000000000003e-05 2024-09-16T03:38:00.680 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.680 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.680 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.680 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1247 4.11 4:89839b7e:::smithi15736696-12:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-16T03:38:00.681 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:38:00.681 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 23.666351288000001, 2024-09-16T03:38:00.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202294525, 2024-09-16T03:38:00.681 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.681 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.681 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.681 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:00.681 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:00.681 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:00.681 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1247 2024-09-16T03:38:00.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.682 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:38:00.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:38:00.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955579+0000", 2024-09-16T03:38:00.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3240000000000001e-06 2024-09-16T03:38:00.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955587+0000", 2024-09-16T03:38:00.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:38:00.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955600+0000", 2024-09-16T03:38:00.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2723e-05 2024-09-16T03:38:00.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955612+0000", 2024-09-16T03:38:00.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2422999999999999e-05 2024-09-16T03:38:00.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055837+0000", 2024-09-16T03:38:00.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10022539699999999 2024-09-16T03:38:00.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055974+0000", 2024-09-16T03:38:00.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000136995 2024-09-16T03:38:00.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,7,8,11,12", 2024-09-16T03:38:00.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.056092+0000", 2024-09-16T03:38:00.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011738 2024-09-16T03:38:00.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:00.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057209+0000", 2024-09-16T03:38:00.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011170329999999999 2024-09-16T03:38:00.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:00.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057262+0000", 2024-09-16T03:38:00.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2859000000000003e-05 2024-09-16T03:38:00.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:00.687 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157594+0000", 2024-09-16T03:38:00.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10033241699999999 2024-09-16T03:38:00.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.687 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.687 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:00.687 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157648+0000", 2024-09-16T03:38:00.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3863999999999998e-05 2024-09-16T03:38:00.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.687 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.687 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:00.687 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157692+0000", 2024-09-16T03:38:00.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4406999999999997e-05 2024-09-16T03:38:00.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:00.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157732+0000", 2024-09-16T03:38:00.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9156000000000003e-05 2024-09-16T03:38:00.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157780+0000", 2024-09-16T03:38:00.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8801999999999998e-05 2024-09-16T03:38:00.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.689 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157872+0000", 2024-09-16T03:38:00.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1311999999999996e-05 2024-09-16T03:38:00.689 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.689 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.689 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.689 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.689 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:61 3.3 e57/31 3:e44244d1:::smithi15736696-43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 57'85, mlcod=57'82)", 2024-09-16T03:38:00.690 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:39.853906+0000", 2024-09-16T03:38:00.690 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.768022272, 2024-09-16T03:38:00.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100871035, 2024-09-16T03:38:00.690 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.690 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.690 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.690 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.853906+0000", 2024-09-16T03:38:00.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.691 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.853906+0000", 2024-09-16T03:38:00.691 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.691 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.853907+0000", 2024-09-16T03:38:00.691 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7199999999999997e-07 2024-09-16T03:38:00.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.692 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.853913+0000", 2024-09-16T03:38:00.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9220000000000002e-06 2024-09-16T03:38:00.692 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.692 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.692 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.692 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.853924+0000", 2024-09-16T03:38:00.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1212e-05 2024-09-16T03:38:00.692 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.692 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.692 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.692 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.953994+0000", 2024-09-16T03:38:00.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100069273 2024-09-16T03:38:00.693 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.693 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.693 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.954038+0000", 2024-09-16T03:38:00.693 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4628e-05 2024-09-16T03:38:00.693 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.693 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.693 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.954056+0000", 2024-09-16T03:38:00.693 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7651000000000001e-05 2024-09-16T03:38:00.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.694 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.694 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.694 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.954734+0000", 2024-09-16T03:38:00.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067852899999999998 2024-09-16T03:38:00.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.694 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.694 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.694 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:39.954777+0000", 2024-09-16T03:38:00.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2848e-05 2024-09-16T03:38:00.694 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.695 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.695 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.695 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.695 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1447 3.3 e58/31 3:d2eec433:::1fda83a16db131b06a5bf7beb910c0624eb0c0478a30a35e6b1e31b63f6e3ff4:head v 58'90, mlcod=58'88)", 2024-09-16T03:38:00.695 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:41.806613+0000", 2024-09-16T03:38:00.695 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 18.815315570999999, 2024-09-16T03:38:00.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101561763, 2024-09-16T03:38:00.695 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.695 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.696 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.696 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.806613+0000", 2024-09-16T03:38:00.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.696 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.806613+0000", 2024-09-16T03:38:00.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.806616+0000", 2024-09-16T03:38:00.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.456e-06 2024-09-16T03:38:00.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.807133+0000", 2024-09-16T03:38:00.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000516787 2024-09-16T03:38:00.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.807144+0000", 2024-09-16T03:38:00.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0448000000000001e-05 2024-09-16T03:38:00.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.807150+0000", 2024-09-16T03:38:00.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6989999999999997e-06 2024-09-16T03:38:00.699 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.699 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.907320+0000", 2024-09-16T03:38:00.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10016939900000001 2024-09-16T03:38:00.699 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.699 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.907334+0000", 2024-09-16T03:38:00.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4241e-05 2024-09-16T03:38:00.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.700 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.700 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.908143+0000", 2024-09-16T03:38:00.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00080902999999999997 2024-09-16T03:38:00.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.700 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.700 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.908175+0000", 2024-09-16T03:38:00.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1702999999999998e-05 2024-09-16T03:38:00.701 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.701 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.701 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.701 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1518 3.3 e59/31 3:df1be05b:::e06092e68680905b5745d5057e93f6c7581518aed07ca5db2802546e9c29aeef:head v 59'95, mlcod=58'93)", 2024-09-16T03:38:00.701 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:38:00.701 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.363907615, 2024-09-16T03:38:00.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104523454, 2024-09-16T03:38:00.701 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.701 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.702 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.702 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:38:00.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:38:00.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258022+0000", 2024-09-16T03:38:00.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.372e-06 2024-09-16T03:38:00.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258462+0000", 2024-09-16T03:38:00.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00043970100000000002 2024-09-16T03:38:00.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258476+0000", 2024-09-16T03:38:00.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.376e-05 2024-09-16T03:38:00.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258483+0000", 2024-09-16T03:38:00.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.294e-06 2024-09-16T03:38:00.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358679+0000", 2024-09-16T03:38:00.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100196177 2024-09-16T03:38:00.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358718+0000", 2024-09-16T03:38:00.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9295999999999998e-05 2024-09-16T03:38:00.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.706 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362496+0000", 2024-09-16T03:38:00.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037772069999999999 2024-09-16T03:38:00.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.706 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362544+0000", 2024-09-16T03:38:00.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8646999999999997e-05 2024-09-16T03:38:00.706 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.707 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.707 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.707 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDPGPush(4.12 69/68 [PushOp(4:4a9d235d:::smithi15736696-1:head, version: 65'12, data_included: [1877233~374905], data_size: 374905, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:4a9d235d:::smithi15736696-1:head@65'12, size: 2252138, copy_subset: [(0, 2252138)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2252138, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1877233, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T03:38:00.707 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:55.168485+0000", 2024-09-16T03:38:00.707 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.4534433629999999, 2024-09-16T03:38:00.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10125331999999999, 2024-09-16T03:38:00.707 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.707 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.707 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:00.708 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.168485+0000", 2024-09-16T03:38:00.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.168485+0000", 2024-09-16T03:38:00.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.168487+0000", 2024-09-16T03:38:00.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.601e-06 2024-09-16T03:38:00.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.168842+0000", 2024-09-16T03:38:00.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035501899999999998 2024-09-16T03:38:00.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.168857+0000", 2024-09-16T03:38:00.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5551999999999999e-05 2024-09-16T03:38:00.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.168872+0000", 2024-09-16T03:38:00.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5248e-05 2024-09-16T03:38:00.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.269031+0000", 2024-09-16T03:38:00.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100158916 2024-09-16T03:38:00.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.269040+0000", 2024-09-16T03:38:00.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4489999999999999e-06 2024-09-16T03:38:00.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.269738+0000", 2024-09-16T03:38:00.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00069853500000000002 2024-09-16T03:38:00.712 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.712 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.712 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.712 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1923 4.4 e73/50 4:239dbd85:::smithi15736696-2:head v 73'19, mlcod=62'17)", 2024-09-16T03:38:00.712 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:38:00.712 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.65804679300000002, 2024-09-16T03:38:00.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104144848, 2024-09-16T03:38:00.712 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.712 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.713 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.713 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:38:00.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:38:00.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963884+0000", 2024-09-16T03:38:00.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:38:00.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964597+0000", 2024-09-16T03:38:00.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071314700000000002 2024-09-16T03:38:00.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964615+0000", 2024-09-16T03:38:00.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8209000000000001e-05 2024-09-16T03:38:00.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964626+0000", 2024-09-16T03:38:00.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0511e-05 2024-09-16T03:38:00.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064779+0000", 2024-09-16T03:38:00.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100153495 2024-09-16T03:38:00.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064794+0000", 2024-09-16T03:38:00.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4946000000000001e-05 2024-09-16T03:38:00.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.067999+0000", 2024-09-16T03:38:00.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032050709999999999 2024-09-16T03:38:00.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068026+0000", 2024-09-16T03:38:00.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7066000000000001e-05 2024-09-16T03:38:00.717 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.717 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.717 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.718 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1924 4.4 e73/50 4:239dbd85:::smithi15736696-2:head v 73'20, mlcod=62'17)", 2024-09-16T03:38:00.718 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:59.965625+0000", 2024-09-16T03:38:00.718 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.65630337400000005, 2024-09-16T03:38:00.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102804671, 2024-09-16T03:38:00.718 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.718 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.718 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.718 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.965625+0000", 2024-09-16T03:38:00.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.965625+0000", 2024-09-16T03:38:00.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.965627+0000", 2024-09-16T03:38:00.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5179999999999999e-06 2024-09-16T03:38:00.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.966256+0000", 2024-09-16T03:38:00.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00062937300000000002 2024-09-16T03:38:00.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.966274+0000", 2024-09-16T03:38:00.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8257000000000002e-05 2024-09-16T03:38:00.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.966284+0000", 2024-09-16T03:38:00.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4299999999999995e-06 2024-09-16T03:38:00.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.066012+0000", 2024-09-16T03:38:00.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099728368999999997 2024-09-16T03:38:00.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.066028+0000", 2024-09-16T03:38:00.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6512e-05 2024-09-16T03:38:00.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068405+0000", 2024-09-16T03:38:00.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023766740000000001 2024-09-16T03:38:00.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068430+0000", 2024-09-16T03:38:00.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4538000000000001e-05 2024-09-16T03:38:00.723 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.723 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.723 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.723 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1925 4.4 e73/50 4:239dbd85:::smithi15736696-2:head v 73'21, mlcod=62'17)", 2024-09-16T03:38:00.723 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:59.967211+0000", 2024-09-16T03:38:00.723 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.65471747599999996, 2024-09-16T03:38:00.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101242033, 2024-09-16T03:38:00.724 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.724 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.724 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.724 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.967211+0000", 2024-09-16T03:38:00.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.967211+0000", 2024-09-16T03:38:00.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.967212+0000", 2024-09-16T03:38:00.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.573e-06 2024-09-16T03:38:00.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.967224+0000", 2024-09-16T03:38:00.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2e-05 2024-09-16T03:38:00.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.967239+0000", 2024-09-16T03:38:00.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4301000000000001e-05 2024-09-16T03:38:00.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.967265+0000", 2024-09-16T03:38:00.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5783999999999999e-05 2024-09-16T03:38:00.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.067874+0000", 2024-09-16T03:38:00.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100609092 2024-09-16T03:38:00.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.067888+0000", 2024-09-16T03:38:00.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4134999999999999e-05 2024-09-16T03:38:00.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068431+0000", 2024-09-16T03:38:00.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054355799999999998 2024-09-16T03:38:00.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068453+0000", 2024-09-16T03:38:00.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1590000000000002e-05 2024-09-16T03:38:00.731 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.731 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.731 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.731 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:79 3.3 e73/31 3:ef816e51:::c911d1f913ee06648afe1626a5dd8763148c66f2ecb1585da932e966636faf1a:head v 73'118, mlcod=73'116)", 2024-09-16T03:38:00.731 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:00.176974+0000", 2024-09-16T03:38:00.731 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.44495401299999998, 2024-09-16T03:38:00.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100838249, 2024-09-16T03:38:00.731 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.731 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.731 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.732 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.176974+0000", 2024-09-16T03:38:00.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.176974+0000", 2024-09-16T03:38:00.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.176976+0000", 2024-09-16T03:38:00.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.815e-06 2024-09-16T03:38:00.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.176983+0000", 2024-09-16T03:38:00.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2030000000000003e-06 2024-09-16T03:38:00.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.176995+0000", 2024-09-16T03:38:00.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1994000000000001e-05 2024-09-16T03:38:00.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.277127+0000", 2024-09-16T03:38:00.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100131653 2024-09-16T03:38:00.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.277209+0000", 2024-09-16T03:38:00.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1577999999999997e-05 2024-09-16T03:38:00.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.277232+0000", 2024-09-16T03:38:00.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3187999999999999e-05 2024-09-16T03:38:00.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.277754+0000", 2024-09-16T03:38:00.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00052237300000000002 2024-09-16T03:38:00.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.277813+0000", 2024-09-16T03:38:00.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8445e-05 2024-09-16T03:38:00.736 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.736 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.736 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.736 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:106 3.3 e73/31 3:dbb0d6f3:::b417679517d63919e63cf420e20217e40570d74b9322ef8b960ef314107040f0:head v 73'120, mlcod=73'116)", 2024-09-16T03:38:00.736 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:00.736 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.34382586700000001, 2024-09-16T03:38:00.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.1035437, 2024-09-16T03:38:00.737 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:00.737 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:00.737 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:00.737 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:00.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:00.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:00.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:00.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:00.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:00.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:00.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:00.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4900000000000004e-07 2024-09-16T03:38:00.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:00.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278120+0000", 2024-09-16T03:38:00.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6214000000000001e-05 2024-09-16T03:38:00.739 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.739 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.739 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:00.739 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278129+0000", 2024-09-16T03:38:00.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.115e-06 2024-09-16T03:38:00.739 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.739 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.739 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:00.739 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278138+0000", 2024-09-16T03:38:00.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.329e-06 2024-09-16T03:38:00.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:00.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.378298+0000", 2024-09-16T03:38:00.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100159945 2024-09-16T03:38:00.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:00.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.378320+0000", 2024-09-16T03:38:00.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1810999999999998e-05 2024-09-16T03:38:00.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:00.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.381607+0000", 2024-09-16T03:38:00.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032868210000000001 2024-09-16T03:38:00.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:00.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:00.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:00.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.381646+0000", 2024-09-16T03:38:00.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9515999999999999e-05 2024-09-16T03:38:00.741 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.741 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.741 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.741 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:00.742 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:00.742 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:00.743 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:00.638+0000 7f3d40708640 -1 osd.12 73 handle_osd_map injecting map crc failure 2024-09-16T03:38:00.745 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:38:00.747 INFO:tasks.rados.rados.0.smithi157.stdout:971: finishing write tid 2 to smithi15736696-38 2024-09-16T03:38:00.747 INFO:tasks.rados.rados.0.smithi157.stdout:971: finishing write tid 3 to smithi15736696-38 2024-09-16T03:38:00.747 INFO:tasks.rados.rados.0.smithi157.stdout:982: got (0) Success 2024-09-16T03:38:00.747 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 42 (ObjNum 142 snap 0 seq_num 142) dirty exists 2024-09-16T03:38:00.747 INFO:tasks.rados.rados.0.smithi157.stdout:971: finishing write tid 4 to smithi15736696-38 2024-09-16T03:38:00.747 INFO:tasks.rados.rados.0.smithi157.stdout:971: oid 38 updating version 0 to 27 2024-09-16T03:38:00.747 INFO:tasks.rados.rados.0.smithi157.stdout:971: oid 38 updating version 27 to 29 2024-09-16T03:38:00.747 INFO:tasks.rados.rados.0.smithi157.stdout:971: oid 38 version 29 is already newer than 28 2024-09-16T03:38:00.748 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 29 (ObjNum 157 snap 0 seq_num 157) dirty exists 2024-09-16T03:38:00.748 INFO:tasks.rados.rados.0.smithi157.stdout:971: left oid 38 (ObjNum 157 snap 0 seq_num 157) 2024-09-16T03:38:00.748 INFO:tasks.rados.rados.0.smithi157.stdout:974: finishing write tid 2 to smithi15736696-37 2024-09-16T03:38:00.748 INFO:tasks.rados.rados.0.smithi157.stdout:971: done (15 left) 2024-09-16T03:38:00.748 INFO:tasks.rados.rados.0.smithi157.stdout:972: done (14 left) 2024-09-16T03:38:00.748 INFO:tasks.rados.rados.0.smithi157.stdout:973: done (13 left) 2024-09-16T03:38:00.748 INFO:tasks.rados.rados.0.smithi157.stdout:976: done (12 left) 2024-09-16T03:38:00.748 INFO:tasks.rados.rados.0.smithi157.stdout:977: done (11 left) 2024-09-16T03:38:00.748 INFO:tasks.rados.rados.0.smithi157.stdout:978: done (10 left) 2024-09-16T03:38:00.748 INFO:tasks.rados.rados.0.smithi157.stdout:979: done (9 left) 2024-09-16T03:38:00.748 INFO:tasks.rados.rados.0.smithi157.stdout:981: done (8 left) 2024-09-16T03:38:00.749 INFO:tasks.rados.rados.0.smithi157.stdout:982: done (7 left) 2024-09-16T03:38:00.749 INFO:tasks.rados.rados.0.smithi157.stdout:984: done (6 left) 2024-09-16T03:38:00.749 INFO:tasks.rados.rados.0.smithi157.stdout:986: done (5 left) 2024-09-16T03:38:00.749 INFO:tasks.rados.rados.0.smithi157.stdout:987: delete oid 21 current snap is 0 2024-09-16T03:38:00.749 INFO:tasks.rados.rados.0.smithi157.stdout:983: finishing write tid 1 to smithi15736696-2 2024-09-16T03:38:00.749 INFO:tasks.rados.rados.0.smithi157.stdout:983: finishing write tid 2 to smithi15736696-2 2024-09-16T03:38:00.749 INFO:tasks.rados.rados.0.smithi157.stdout:983: finishing write tid 3 to smithi15736696-2 2024-09-16T03:38:00.749 INFO:tasks.rados.rados.0.smithi157.stdout:983: finishing write tid 4 to smithi15736696-2 2024-09-16T03:38:00.749 INFO:tasks.rados.rados.0.smithi157.stdout:983: oid 2 updating version 0 to 19 2024-09-16T03:38:00.749 INFO:tasks.rados.rados.0.smithi157.stdout:983: oid 2 updating version 19 to 20 2024-09-16T03:38:00.750 INFO:tasks.rados.rados.0.smithi157.stdout:983: oid 2 updating version 20 to 21 2024-09-16T03:38:00.750 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 21 (ObjNum 160 snap 0 seq_num 160) dirty exists 2024-09-16T03:38:00.750 INFO:tasks.rados.rados.0.smithi157.stdout:983: left oid 2 (ObjNum 160 snap 0 seq_num 160) 2024-09-16T03:38:00.750 INFO:tasks.rados.rados.0.smithi157.stdout:985: finishing write tid 1 to smithi15736696-35 2024-09-16T03:38:00.750 INFO:tasks.rados.rados.0.smithi157.stdout:985: finishing write tid 2 to smithi15736696-35 2024-09-16T03:38:00.750 INFO:tasks.rados.rados.0.smithi157.stdout:974: finishing write tid 3 to smithi15736696-37 2024-09-16T03:38:00.750 INFO:tasks.rados.rados.0.smithi157.stdout:975: finishing write tid 1 to smithi15736696-18 2024-09-16T03:38:00.750 INFO:tasks.rados.rados.0.smithi157.stdout:974: finishing write tid 4 to smithi15736696-37 2024-09-16T03:38:00.750 INFO:tasks.rados.rados.0.smithi157.stdout:974: oid 37 updating version 0 to 37 2024-09-16T03:38:00.750 INFO:tasks.rados.rados.0.smithi157.stdout:974: oid 37 updating version 37 to 38 2024-09-16T03:38:00.750 INFO:tasks.rados.rados.0.smithi157.stdout:974: oid 37 version 38 is already newer than 36 2024-09-16T03:38:00.750 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 38 (ObjNum 158 snap 0 seq_num 158) dirty exists 2024-09-16T03:38:00.751 INFO:tasks.rados.rados.0.smithi157.stdout:974: left oid 37 (ObjNum 158 snap 0 seq_num 158) 2024-09-16T03:38:00.751 INFO:tasks.rados.rados.0.smithi157.stdout:985: finishing write tid 3 to smithi15736696-35 2024-09-16T03:38:00.751 INFO:tasks.rados.rados.0.smithi157.stdout:985: finishing write tid 4 to smithi15736696-35 2024-09-16T03:38:00.751 INFO:tasks.rados.rados.0.smithi157.stdout:985: oid 35 updating version 0 to 61 2024-09-16T03:38:00.751 INFO:tasks.rados.rados.0.smithi157.stdout:985: oid 35 updating version 61 to 63 2024-09-16T03:38:00.751 INFO:tasks.rados.rados.0.smithi157.stdout:985: oid 35 version 63 is already newer than 62 2024-09-16T03:38:00.751 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 63 (ObjNum 161 snap 0 seq_num 161) dirty exists 2024-09-16T03:38:00.751 INFO:tasks.rados.rados.0.smithi157.stdout:985: left oid 35 (ObjNum 161 snap 0 seq_num 161) 2024-09-16T03:38:00.751 INFO:tasks.rados.rados.0.smithi157.stdout:980: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-16T03:38:00.751 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1200056 2024-09-16T03:38:00.751 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:00.752 INFO:tasks.rados.rados.0.smithi157.stdout:975: finishing write tid 2 to smithi15736696-18 2024-09-16T03:38:00.752 INFO:tasks.rados.rados.0.smithi157.stdout:975: finishing write tid 3 to smithi15736696-18 2024-09-16T03:38:00.752 INFO:tasks.rados.rados.0.smithi157.stdout:975: finishing write tid 4 to smithi15736696-18 2024-09-16T03:38:00.752 INFO:tasks.rados.rados.0.smithi157.stdout:975: oid 18 updating version 0 to 52 2024-09-16T03:38:00.752 INFO:tasks.rados.rados.0.smithi157.stdout:975: oid 18 version 52 is already newer than 51 2024-09-16T03:38:00.752 INFO:tasks.rados.rados.0.smithi157.stdout:975: oid 18 updating version 52 to 53 2024-09-16T03:38:00.752 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 53 (ObjNum 159 snap 0 seq_num 159) dirty exists 2024-09-16T03:38:00.752 INFO:tasks.rados.rados.0.smithi157.stdout:975: left oid 18 (ObjNum 159 snap 0 seq_num 159) 2024-09-16T03:38:00.752 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:00.740+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:00.793 INFO:tasks.rados.rados.0.smithi157.stdout:974: done (5 left) 2024-09-16T03:38:00.794 INFO:tasks.rados.rados.0.smithi157.stdout:975: done (4 left) 2024-09-16T03:38:00.794 INFO:tasks.rados.rados.0.smithi157.stdout:980: done (3 left) 2024-09-16T03:38:00.794 INFO:tasks.rados.rados.0.smithi157.stdout:983: done (2 left) 2024-09-16T03:38:00.794 INFO:tasks.rados.rados.0.smithi157.stdout:985: done (1 left) 2024-09-16T03:38:00.794 INFO:tasks.rados.rados.0.smithi157.stdout:987: done (0 left) 2024-09-16T03:38:00.794 INFO:tasks.rados.rados.0.smithi157.stdout:988: write oid 46 current snap is 0 2024-09-16T03:38:00.794 INFO:tasks.rados.rados.0.smithi157.stdout:988: seq_num 163 ranges {739541=709000,2121859=426376} 2024-09-16T03:38:00.801 INFO:tasks.rados.rados.0.smithi157.stdout:988: writing smithi15736696-46 from 739541 to 1448541 tid 1 2024-09-16T03:38:00.807 INFO:tasks.rados.rados.0.smithi157.stdout:988: writing smithi15736696-46 from 2121859 to 2548235 tid 2 2024-09-16T03:38:00.808 INFO:tasks.rados.rados.0.smithi157.stdout:989: read oid 7 snap -1 2024-09-16T03:38:00.808 INFO:tasks.rados.rados.0.smithi157.stdout:989: expect deleted 2024-09-16T03:38:00.808 INFO:tasks.rados.rados.0.smithi157.stdout:990: set_chunk oid 16 target oid 2024-09-16T03:38:00.844 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:00.842+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:00.857 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:00.857 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:00.857 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:00.857 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:00.866 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:38:00.945 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:00.943+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:00.970 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:00.970 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:00.978 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:38:01.044 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-09-16T03:38:01.042+0000 7f5019b9e640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-16T03:38:01.045 INFO:teuthology.orchestra.run.smithi064.stdout:moved 14 / 117 (11.9658%) 2024-09-16T03:38:01.045 INFO:teuthology.orchestra.run.smithi064.stdout:avg 9.75 2024-09-16T03:38:01.045 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 3.85411 -> 3.05846 (expected baseline 2.98957) 2024-09-16T03:38:01.045 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.12 with 4 -> 6 pgs (0.410256 -> 0.615385 * mean) 2024-09-16T03:38:01.045 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.10 with 15 -> 14 pgs (1.53846 -> 1.4359 * mean) 2024-09-16T03:38:01.045 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:38:01.045 INFO:teuthology.orchestra.run.smithi064.stdout:oload 110 2024-09-16T03:38:01.045 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 1 2024-09-16T03:38:01.045 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-09-16T03:38:01.045 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0006 2024-09-16T03:38:01.045 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0006 2024-09-16T03:38:01.046 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 weight 1.0000 -> 0.6409 2024-09-16T03:38:01.046 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15 weight 1.0000 -> 0.7130 2024-09-16T03:38:01.046 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11 weight 1.0000 -> 0.8510 2024-09-16T03:38:01.046 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10 weight 1.0000 -> 0.8967 2024-09-16T03:38:01.046 INFO:teuthology.orchestra.run.smithi064.stderr:no change 2024-09-16T03:38:01.047 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:01.046+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:01.057 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-16T03:38:01.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 1.0 2024-09-16T03:38:01.085 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:01.085 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:01.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:01.085 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:01.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.085 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:53 4.7 e39/32 4:ed874e32:::smithi15736696-14:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:01.085 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:38:01.085 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 48.838583941000003, 2024-09-16T03:38:01.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20229554399999999, 2024-09-16T03:38:01.086 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.086 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.086 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.086 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:38:01.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:38:01.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243323+0000", 2024-09-16T03:38:01.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9600000000000001e-07 2024-09-16T03:38:01.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243671+0000", 2024-09-16T03:38:01.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034761699999999999 2024-09-16T03:38:01.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.243681+0000", 2024-09-16T03:38:01.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0505e-05 2024-09-16T03:38:01.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.343786+0000", 2024-09-16T03:38:01.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10010472300000001 2024-09-16T03:38:01.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.443948+0000", 2024-09-16T03:38:01.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100161739 2024-09-16T03:38:01.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.443966+0000", 2024-09-16T03:38:01.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8045000000000001e-05 2024-09-16T03:38:01.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445575+0000", 2024-09-16T03:38:01.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001608841 2024-09-16T03:38:01.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445619+0000", 2024-09-16T03:38:01.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3878000000000002e-05 2024-09-16T03:38:01.090 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.090 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.090 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.091 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:17 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:01.091 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.086208+0000", 2024-09-16T03:38:01.091 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 44.995698757, 2024-09-16T03:38:01.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20243159599999999, 2024-09-16T03:38:01.091 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.091 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.091 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.091 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086208+0000", 2024-09-16T03:38:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086208+0000", 2024-09-16T03:38:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086210+0000", 2024-09-16T03:38:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.058e-06 2024-09-16T03:38:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086216+0000", 2024-09-16T03:38:01.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1399999999999997e-06 2024-09-16T03:38:01.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086226+0000", 2024-09-16T03:38:01.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4110000000000002e-06 2024-09-16T03:38:01.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.086235+0000", 2024-09-16T03:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4420000000000004e-06 2024-09-16T03:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.186482+0000", 2024-09-16T03:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10024678200000001 2024-09-16T03:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.186574+0000", 2024-09-16T03:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2035999999999998e-05 2024-09-16T03:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.186725+0000", 2024-09-16T03:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150629 2024-09-16T03:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.187384+0000", 2024-09-16T03:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065916599999999998 2024-09-16T03:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.187823+0000", 2024-09-16T03:38:01.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000439018 2024-09-16T03:38:01.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.288509+0000", 2024-09-16T03:38:01.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100685624 2024-09-16T03:38:01.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.288611+0000", 2024-09-16T03:38:01.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102155 2024-09-16T03:38:01.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.288640+0000", 2024-09-16T03:38:01.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9135000000000001e-05 2024-09-16T03:38:01.097 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.097 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.097 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.097 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:19 3.1 3:9aba8983:::smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:01.097 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:38:01.097 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 44.663475271999999, 2024-09-16T03:38:01.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30153978799999998, 2024-09-16T03:38:01.097 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.098 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.098 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.098 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:38:01.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:38:01.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418452+0000", 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0363000000000001e-05 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418470+0000", 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7506000000000001e-05 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418504+0000", 2024-09-16T03:38:01.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4542999999999998e-05 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418522+0000", 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8070000000000001e-05 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418662+0000", 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013946299999999999 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418826+0000", 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016469299999999999 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418989+0000", 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000162759 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.419750+0000", 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076104299999999998 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.419865+0000", 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114338 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719876+0000", 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30001127300000002 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719920+0000", 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4549999999999999e-05 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719972+0000", 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1187000000000003e-05 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:10 3.1 3:8f38b759:::smithi15736696-6 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 44.574042634000001, 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.21220708699999999, 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.103 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507866+0000", 2024-09-16T03:38:01.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9999999999999999e-06 2024-09-16T03:38:01.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507873+0000", 2024-09-16T03:38:01.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7900000000000002e-06 2024-09-16T03:38:01.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507888+0000", 2024-09-16T03:38:01.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4554e-05 2024-09-16T03:38:01.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507896+0000", 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1130000000000001e-06 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507967+0000", 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0808000000000004e-05 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508028+0000", 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1325e-05 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508146+0000", 2024-09-16T03:38:01.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000117922 2024-09-16T03:38:01.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508722+0000", 2024-09-16T03:38:01.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000575779 2024-09-16T03:38:01.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.609567+0000", 2024-09-16T03:38:01.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100845448 2024-09-16T03:38:01.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720013+0000", 2024-09-16T03:38:01.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.110445644 2024-09-16T03:38:01.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720045+0000", 2024-09-16T03:38:01.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2186e-05 2024-09-16T03:38:01.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720071+0000", 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6517999999999999e-05 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:11 3.1 3:adc537a6:::smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 44.568699404, 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20740488100000001, 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.109 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513209+0000", 2024-09-16T03:38:01.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7439999999999999e-06 2024-09-16T03:38:01.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513216+0000", 2024-09-16T03:38:01.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9890000000000003e-06 2024-09-16T03:38:01.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513230+0000", 2024-09-16T03:38:01.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3263e-05 2024-09-16T03:38:01.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513236+0000", 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7270000000000001e-06 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513297+0000", 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0767999999999998e-05 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513375+0000", 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8014000000000002e-05 2024-09-16T03:38:01.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513529+0000", 2024-09-16T03:38:01.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015342900000000001 2024-09-16T03:38:01.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.514258+0000", 2024-09-16T03:38:01.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072896799999999998 2024-09-16T03:38:01.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.709867+0000", 2024-09-16T03:38:01.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.195609016 2024-09-16T03:38:01.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720546+0000", 2024-09-16T03:38:01.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010679905999999999 2024-09-16T03:38:01.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720583+0000", 2024-09-16T03:38:01.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6659999999999998e-05 2024-09-16T03:38:01.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720613+0000", 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9397e-05 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:18 3.1 3:9f9f4ec5:::smithi15736696-22 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.104882+0000", 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 40.977024642000003, 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20218581599999999, 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.115 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104882+0000", 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104882+0000", 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104885+0000", 2024-09-16T03:38:01.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7760000000000002e-06 2024-09-16T03:38:01.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104895+0000", 2024-09-16T03:38:01.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9909999999999997e-06 2024-09-16T03:38:01.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104911+0000", 2024-09-16T03:38:01.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5654999999999998e-05 2024-09-16T03:38:01.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104927+0000", 2024-09-16T03:38:01.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6620999999999999e-05 2024-09-16T03:38:01.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.104985+0000", 2024-09-16T03:38:01.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.787e-05 2024-09-16T03:38:01.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105043+0000", 2024-09-16T03:38:01.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7720999999999998e-05 2024-09-16T03:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105174+0000", 2024-09-16T03:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013075599999999999 2024-09-16T03:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105846+0000", 2024-09-16T03:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067215899999999997 2024-09-16T03:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206477+0000", 2024-09-16T03:38:01.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10063103299999999 2024-09-16T03:38:01.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307001+0000", 2024-09-16T03:38:01.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10052391200000001 2024-09-16T03:38:01.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307035+0000", 2024-09-16T03:38:01.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3674999999999999e-05 2024-09-16T03:38:01.121 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.121 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307068+0000", 2024-09-16T03:38:01.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3646999999999998e-05 2024-09-16T03:38:01.121 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.121 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.121 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.121 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.122 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:19 3.1 3:aca5d7ac:::smithi15736696-26 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:01.122 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:38:01.122 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 40.976842451000003, 2024-09-16T03:38:01.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264779099999999, 2024-09-16T03:38:01.122 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.122 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.122 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.122 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:38:01.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:38:01.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105065+0000", 2024-09-16T03:38:01.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:38:01.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105068+0000", 2024-09-16T03:38:01.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.751e-06 2024-09-16T03:38:01.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.105072+0000", 2024-09-16T03:38:01.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5730000000000001e-06 2024-09-16T03:38:01.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.205143+0000", 2024-09-16T03:38:01.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100070665 2024-09-16T03:38:01.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206515+0000", 2024-09-16T03:38:01.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013715789999999999 2024-09-16T03:38:01.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206570+0000", 2024-09-16T03:38:01.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5049000000000001e-05 2024-09-16T03:38:01.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.206695+0000", 2024-09-16T03:38:01.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012521599999999999 2024-09-16T03:38:01.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307091+0000", 2024-09-16T03:38:01.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10039582 2024-09-16T03:38:01.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307128+0000", 2024-09-16T03:38:01.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7290999999999999e-05 2024-09-16T03:38:01.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307656+0000", 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00052813199999999995 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307689+0000", 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3056000000000003e-05 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.307712+0000", 2024-09-16T03:38:01.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3099999999999999e-05 2024-09-16T03:38:01.128 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.128 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.128 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.128 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:779 3.1 3:abdaa5c9:::2782d12a88f9842f7f754fca32c4222cf10fb780cd136830d32faaa700e687ba:head [write 0~45056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:01.128 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:38:01.128 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.595326956000001, 2024-09-16T03:38:01.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30397390600000002, 2024-09-16T03:38:01.128 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.128 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.128 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.128 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.129 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.129 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.129 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 779 2024-09-16T03:38:01.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.129 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:38:01.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486583+0000", 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.762e-06 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486649+0000", 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6143999999999996e-05 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486669+0000", 2024-09-16T03:38:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9837e-05 2024-09-16T03:38:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486681+0000", 2024-09-16T03:38:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2194999999999999e-05 2024-09-16T03:38:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486746+0000", 2024-09-16T03:38:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5480000000000003e-05 2024-09-16T03:38:01.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-09-16T03:38:01.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486752+0000", 2024-09-16T03:38:01.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4659999999999998e-06 2024-09-16T03:38:01.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588316+0000", 2024-09-16T03:38:01.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10156393399999999 2024-09-16T03:38:01.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588539+0000", 2024-09-16T03:38:01.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022308300000000001 2024-09-16T03:38:01.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588647+0000", 2024-09-16T03:38:01.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000108047 2024-09-16T03:38:01.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.589516+0000", 2024-09-16T03:38:01.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086926899999999999 2024-09-16T03:38:01.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.789989+0000", 2024-09-16T03:38:01.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20047296100000001 2024-09-16T03:38:01.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790478+0000", 2024-09-16T03:38:01.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048882099999999996 2024-09-16T03:38:01.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790518+0000", 2024-09-16T03:38:01.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0343000000000003e-05 2024-09-16T03:38:01.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790554+0000", 2024-09-16T03:38:01.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5564e-05 2024-09-16T03:38:01.135 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.135 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.135 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:38 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.243499765999999, 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30216061300000002, 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:38:01.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838411+0000", 2024-09-16T03:38:01.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3780000000000001e-06 2024-09-16T03:38:01.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838422+0000", 2024-09-16T03:38:01.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1223000000000001e-05 2024-09-16T03:38:01.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838450+0000", 2024-09-16T03:38:01.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7878999999999999e-05 2024-09-16T03:38:01.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838464+0000", 2024-09-16T03:38:01.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3811e-05 2024-09-16T03:38:01.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838539+0000", 2024-09-16T03:38:01.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5685000000000005e-05 2024-09-16T03:38:01.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838604+0000", 2024-09-16T03:38:01.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4980999999999993e-05 2024-09-16T03:38:01.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838732+0000", 2024-09-16T03:38:01.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012821100000000001 2024-09-16T03:38:01.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839340+0000", 2024-09-16T03:38:01.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060762399999999997 2024-09-16T03:38:01.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840716+0000", 2024-09-16T03:38:01.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001375692 2024-09-16T03:38:01.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140491+0000", 2024-09-16T03:38:01.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29977497600000003 2024-09-16T03:38:01.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140534+0000", 2024-09-16T03:38:01.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3411999999999997e-05 2024-09-16T03:38:01.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140568+0000", 2024-09-16T03:38:01.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3741000000000001e-05 2024-09-16T03:38:01.144 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.144 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.145 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.145 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.145 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:39 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:01.145 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:38:01.145 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.242209547999998, 2024-09-16T03:38:01.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40113680400000001, 2024-09-16T03:38:01.145 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.145 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.145 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.145 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839700+0000", 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.571e-06 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839704+0000", 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8399999999999997e-06 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839712+0000", 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9990000000000008e-06 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839720+0000", 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0099999999999995e-06 2024-09-16T03:38:01.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839783+0000", 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3014000000000003e-05 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839870+0000", 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7392999999999995e-05 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840049+0000", 2024-09-16T03:38:01.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017825800000000001 2024-09-16T03:38:01.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840874+0000", 2024-09-16T03:38:01.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082532099999999995 2024-09-16T03:38:01.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.040285+0000", 2024-09-16T03:38:01.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19941088400000001 2024-09-16T03:38:01.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240748+0000", 2024-09-16T03:38:01.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20046322699999999 2024-09-16T03:38:01.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240786+0000", 2024-09-16T03:38:01.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8158000000000003e-05 2024-09-16T03:38:01.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240834+0000", 2024-09-16T03:38:01.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8128999999999999e-05 2024-09-16T03:38:01.151 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.151 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.151 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.151 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:983 3.1 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head [write 0~2138112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:38:01.151 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:38:01.151 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.018306608, 2024-09-16T03:38:01.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30819799799999997, 2024-09-16T03:38:01.151 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.151 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 983 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063610+0000", 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2990000000000002e-06 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065735+0000", 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021257799999999999 2024-09-16T03:38:01.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065752+0000", 2024-09-16T03:38:01.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6895999999999999e-05 2024-09-16T03:38:01.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065762+0000", 2024-09-16T03:38:01.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6830000000000003e-06 2024-09-16T03:38:01.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065871+0000", 2024-09-16T03:38:01.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010904900000000001 2024-09-16T03:38:01.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066005+0000", 2024-09-16T03:38:01.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013369799999999999 2024-09-16T03:38:01.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066135+0000", 2024-09-16T03:38:01.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000130481 2024-09-16T03:38:01.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.069822+0000", 2024-09-16T03:38:01.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036863759999999999 2024-09-16T03:38:01.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272370+0000", 2024-09-16T03:38:01.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20254802599999999 2024-09-16T03:38:01.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371671+0000", 2024-09-16T03:38:01.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099300893000000001 2024-09-16T03:38:01.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371754+0000", 2024-09-16T03:38:01.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3075e-05 2024-09-16T03:38:01.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371798+0000", 2024-09-16T03:38:01.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4742e-05 2024-09-16T03:38:01.157 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.157 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1515 3.1 3:9c9e970c:::7249cb65b52cadf4fe8f4ec923c7e548d1de7d9628823acd2c8de19121f1c1ce:head [write 0~1094656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.132180870999999, 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.305958065, 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.158 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1515 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949729+0000", 2024-09-16T03:38:01.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.785e-06 2024-09-16T03:38:01.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950933+0000", 2024-09-16T03:38:01.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001204341 2024-09-16T03:38:01.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950950+0000", 2024-09-16T03:38:01.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6838e-05 2024-09-16T03:38:01.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950960+0000", 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0316000000000001e-05 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951065+0000", 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104128 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951202+0000", 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013706600000000001 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951350+0000", 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000148197 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.953857+0000", 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0025067150000000001 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.958049+0000", 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0041924650000000003 2024-09-16T03:38:01.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255547+0000", 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29749841900000001 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255649+0000", 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101513 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255684+0000", 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5281999999999999e-05 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.163 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.164 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.164 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1704 3.1 3:b6a94245:::f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99:head [write 0~453120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:01.164 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:38:01.164 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.289069516, 2024-09-16T03:38:01.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40403313699999999, 2024-09-16T03:38:01.164 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.164 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.164 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1704 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:38:01.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792841+0000", 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.433e-06 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793329+0000", 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048839200000000004 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793345+0000", 2024-09-16T03:38:01.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6079e-05 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793354+0000", 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.32e-06 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793418+0000", 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3944999999999998e-05 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793549+0000", 2024-09-16T03:38:01.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013159900000000001 2024-09-16T03:38:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793666+0000", 2024-09-16T03:38:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116532 2024-09-16T03:38:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.797707+0000", 2024-09-16T03:38:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040409629999999998 2024-09-16T03:38:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897288+0000", 2024-09-16T03:38:01.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099581449000000002 2024-09-16T03:38:01.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196775+0000", 2024-09-16T03:38:01.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948696800000002 2024-09-16T03:38:01.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196814+0000", 2024-09-16T03:38:01.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8336e-05 2024-09-16T03:38:01.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196871+0000", 2024-09-16T03:38:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7120999999999997e-05 2024-09-16T03:38:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:84 3.1 3:b24f289d:::e84532c28f467d2f2a6e2cf44904fa0d4a555e1304bbe1b25db645af42152bd7:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:38:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.280297577000001, 2024-09-16T03:38:01.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.395455581, 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801613+0000", 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9559999999999999e-06 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802117+0000", 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050401600000000003 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802141+0000", 2024-09-16T03:38:01.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3618e-05 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802155+0000", 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4358e-05 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802229+0000", 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3930999999999994e-05 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802355+0000", 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012527299999999999 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802567+0000", 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021250199999999999 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806218+0000", 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036512189999999998 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897323+0000", 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091104892000000007 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196959+0000", 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299635971 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197021+0000", 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1938000000000002e-05 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197065+0000", 2024-09-16T03:38:01.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3906999999999998e-05 2024-09-16T03:38:01.176 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.176 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.176 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.176 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:85 3.1 3:b380f269:::e65351e8a65a0b1a517e3b1a85e04b84c0ab7cfa6655f11fb0da31f56b4dd9af:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:01.176 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:38:01.176 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.27971868, 2024-09-16T03:38:01.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39502727300000001, 2024-09-16T03:38:01.176 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.176 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.176 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802190+0000", 2024-09-16T03:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2419999999999999e-06 2024-09-16T03:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802464+0000", 2024-09-16T03:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027463699999999999 2024-09-16T03:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802478+0000", 2024-09-16T03:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3375e-05 2024-09-16T03:38:01.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802484+0000", 2024-09-16T03:38:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0990000000000004e-06 2024-09-16T03:38:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803475+0000", 2024-09-16T03:38:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00099113099999999991 2024-09-16T03:38:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803651+0000", 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017654300000000001 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803923+0000", 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000271842 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806228+0000", 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023049590000000001 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897563+0000", 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091334724000000006 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197123+0000", 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299559992 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197177+0000", 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4447999999999998e-05 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197216+0000", 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8281000000000002e-05 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:86 3.1 3:b3acb53e:::e951eb69ce5151a3f54107fa22c70f80729eff845da1613447a91536a24046e8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.279411585, 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.394851071, 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:38:01.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:38:01.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802496+0000", 2024-09-16T03:38:01.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802782+0000", 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00028523600000000002 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802796+0000", 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4594e-05 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802804+0000", 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3790000000000006e-06 2024-09-16T03:38:01.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804787+0000", 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019825060000000002 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804895+0000", 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010813499999999999 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.805046+0000", 2024-09-16T03:38:01.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150972 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806681+0000", 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016353839999999999 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.005712+0000", 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19903093199999999 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197278+0000", 2024-09-16T03:38:01.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.191565295 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197315+0000", 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6946000000000002e-05 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197347+0000", 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1863000000000003e-05 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.187 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.188 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.188 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDPGPush(4.13 69/68 [PushOp(4:c8492bfc:::smithi15736696-42:head, version: 66'42, data_included: [410104~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:c8492bfc:::smithi15736696-42:head@66'42, size: 2386594, copy_subset: [(0, 2386594)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1458680, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T03:38:01.188 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:55.062041+0000", 2024-09-16T03:38:01.188 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.0198663809999999, 2024-09-16T03:38:01.188 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20286673599999999, 2024-09-16T03:38:01.188 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.188 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.188 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:01.188 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.188 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.188 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.188 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.062041+0000", 2024-09-16T03:38:01.188 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.062041+0000", 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.062043+0000", 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0229999999999999e-06 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.063203+0000", 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011607970000000001 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.063220+0000", 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6821999999999999e-05 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.163295+0000", 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100075075 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.263476+0000", 2024-09-16T03:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100180994 2024-09-16T03:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.263491+0000", 2024-09-16T03:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4639e-05 2024-09-16T03:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:55.264907+0000", 2024-09-16T03:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014163859999999999 2024-09-16T03:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.191 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:97 3.1 3:9f9f4ec5:::smithi15736696-22 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e71)", 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:56.682453+0000", 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.3994536340000003, 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20249146200000001, 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.192 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682453+0000", 2024-09-16T03:38:01.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.193 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.193 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682453+0000", 2024-09-16T03:38:01.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.193 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.193 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682457+0000", 2024-09-16T03:38:01.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3100000000000001e-06 2024-09-16T03:38:01.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682469+0000", 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2359e-05 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682488+0000", 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9341999999999999e-05 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682501+0000", 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2473999999999999e-05 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.782708+0000", 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100206798 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.782780+0000", 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2191000000000001e-05 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.782931+0000", 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015064899999999999 2024-09-16T03:38:01.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.783603+0000", 2024-09-16T03:38:01.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067266299999999999 2024-09-16T03:38:01.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.784521+0000", 2024-09-16T03:38:01.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00091793199999999999 2024-09-16T03:38:01.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.884833+0000", 2024-09-16T03:38:01.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100311415 2024-09-16T03:38:01.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.197 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.197 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.884883+0000", 2024-09-16T03:38:01.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0692000000000002e-05 2024-09-16T03:38:01.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.197 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.197 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.884945+0000", 2024-09-16T03:38:01.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1637000000000006e-05 2024-09-16T03:38:01.197 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.197 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:98 3.1 3:9f9f4ec5:::smithi15736696-22 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e71)", 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:56.682533+0000", 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.3993737140000002, 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20254659699999999, 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682533+0000", 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682533+0000", 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682535+0000", 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.187e-06 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682540+0000", 2024-09-16T03:38:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5809999999999996e-06 2024-09-16T03:38:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682548+0000", 2024-09-16T03:38:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5120000000000002e-06 2024-09-16T03:38:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682557+0000", 2024-09-16T03:38:01.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2199999999999998e-06 2024-09-16T03:38:01.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.201 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.783198+0000", 2024-09-16T03:38:01.201 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100641324 2024-09-16T03:38:01.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.201 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.783257+0000", 2024-09-16T03:38:01.201 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9176999999999999e-05 2024-09-16T03:38:01.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.783380+0000", 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012254799999999999 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.784557+0000", 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001177612 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.784590+0000", 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2146e-05 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.202 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.885002+0000", 2024-09-16T03:38:01.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10041255 2024-09-16T03:38:01.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.885044+0000", 2024-09-16T03:38:01.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1579000000000003e-05 2024-09-16T03:38:01.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.885080+0000", 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6161000000000002e-05 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:99 3.1 3:9f9f4ec5:::smithi15736696-22 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e71)", 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:56.682569+0000", 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.3993377870000003, 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30291003799999999, 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.204 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.205 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682569+0000", 2024-09-16T03:38:01.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.205 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682569+0000", 2024-09-16T03:38:01.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.205 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682570+0000", 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7000000000000004e-07 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682574+0000", 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3660000000000002e-06 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682589+0000", 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.49e-05 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.207 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682596+0000", 2024-09-16T03:38:01.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0589999999999997e-06 2024-09-16T03:38:01.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.207 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.207 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.783627+0000", 2024-09-16T03:38:01.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101030406 2024-09-16T03:38:01.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.207 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.207 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.783680+0000", 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3562e-05 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.783780+0000", 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100028 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.884772+0000", 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10099123 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.885096+0000", 2024-09-16T03:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00032439900000000001 2024-09-16T03:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.985281+0000", 2024-09-16T03:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100184936 2024-09-16T03:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.985395+0000", 2024-09-16T03:38:01.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114166 2024-09-16T03:38:01.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.210 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.985479+0000", 2024-09-16T03:38:01.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4215999999999995e-05 2024-09-16T03:38:01.210 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.210 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.210 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.210 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.210 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.210 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:01.212 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:01.210+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:01.213 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:38:01.215 INFO:tasks.rados.rados.0.smithi157.stdout:990: set_chunk oid 16 offset: 1296384 length: 339968 target oid 3c9a721811383c366589257800994279d46e77bd52566574d66a70d9b0737bcc offset: 0 snap -1 2024-09-16T03:38:01.215 INFO:tasks.rados.rados.0.smithi157.stdout:991: write oid 6 current snap is 0 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:988: finishing write tid 1 to smithi15736696-46 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:991: seq_num 164 ranges {756348=784249,2063988=686387} 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:991: writing smithi15736696-6 from 756348 to 1540597 tid 1 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:991: writing smithi15736696-6 from 2063988 to 2750375 tid 2 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:992: read oid 22 snap -1 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:992: expect (ObjNum 152 snap 0 seq_num 152) 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:993: read oid 27 snap -1 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:993: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:994: tier_evict oid 2 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:994: tier_evict oid 2 snap -1 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:991: finishing write tid 1 to smithi15736696-6 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:989: done (6 left) 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:995: read oid 4 snap -1 2024-09-16T03:38:01.216 INFO:tasks.rados.rados.0.smithi157.stdout:995: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:996: read oid 39 snap -1 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:996: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:997: write (excl) oid 5 current snap is 0 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:997: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:997: writing smithi15736696-5 from 773155 to 1232653 tid 1 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:997: writing smithi15736696-5 from 1722470 to 2152515 tid 2 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:997: writing smithi15736696-5 from 2648676 to 2773155 tid 3 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:998: read oid 33 snap -1 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:998: expect (ObjNum 127 snap 0 seq_num 127) 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:999: read oid 44 snap -1 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:999: expect (ObjNum 63 snap 0 seq_num 63) 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:1000: tier_promote oid 8 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:994: got (22) Invalid argument 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:994: done (11 left) 2024-09-16T03:38:01.217 INFO:tasks.rados.rados.0.smithi157.stdout:1001: tier_evict oid 30 2024-09-16T03:38:01.218 INFO:tasks.rados.rados.0.smithi157.stdout:1001: tier_evict oid 30 snap -1 2024-09-16T03:38:01.218 INFO:tasks.rados.rados.0.smithi157.stdout:993: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-16T03:38:01.218 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:01.218 INFO:tasks.rados.rados.0.smithi157.stdout:992: expect (ObjNum 152 snap 0 seq_num 152) 2024-09-16T03:38:01.218 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 354981 2024-09-16T03:38:01.233 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:01.265 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1236138 2024-09-16T03:38:01.284 INFO:tasks.rados.rados.0.smithi157.stdout:995: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:01.309 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:01.312 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:01.311+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:01.359 INFO:tasks.rados.rados.0.smithi157.stdout:1001: got (22) Invalid argument 2024-09-16T03:38:01.359 INFO:tasks.rados.rados.0.smithi157.stdout:999: expect (ObjNum 63 snap 0 seq_num 63) 2024-09-16T03:38:01.360 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:01.360 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:01.360 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:01.360 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:01.369 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:38:01.381 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2277734 2024-09-16T03:38:01.408 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:01.454 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:01.452+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:01.489 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1883873 2024-09-16T03:38:01.495 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:01.493+0000 7f49439d4640 -1 osd.4 74 handle_osd_map injecting map crc failure 2024-09-16T03:38:01.496 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:38:01.497 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:01.496+0000 7f0ae754b640 -1 osd.0 74 handle_osd_map injecting map crc failure 2024-09-16T03:38:01.513 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:01.512+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:01.515 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:01.515 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:01.515 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:01.515 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:01.515 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:01.515 INFO:teuthology.orchestra.run.smithi064.stdout:moved 0 / 117 (0%) 2024-09-16T03:38:01.516 INFO:teuthology.orchestra.run.smithi064.stdout:avg 9.75 2024-09-16T03:38:01.516 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 3.85411 -> 3.85411 (expected baseline 2.98957) 2024-09-16T03:38:01.516 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.12 with 4 -> 4 pgs (0.410256 -> 0.410256 * mean) 2024-09-16T03:38:01.516 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.10 with 15 -> 15 pgs (1.53846 -> 1.53846 * mean) 2024-09-16T03:38:01.516 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:38:01.516 INFO:teuthology.orchestra.run.smithi064.stdout:oload 1000 2024-09-16T03:38:01.516 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 1 2024-09-16T03:38:01.516 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-09-16T03:38:01.516 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0006 2024-09-16T03:38:01.516 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0057 2024-09-16T03:38:01.516 INFO:teuthology.orchestra.run.smithi064.stderr:no change 2024-09-16T03:38:01.521 INFO:tasks.rados.rados.0.smithi157.stdout:998: expect (ObjNum 127 snap 0 seq_num 127) 2024-09-16T03:38:01.523 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:38:01.558 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:01.593 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:01.614 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:01.612+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:01.630 INFO:tasks.rados.rados.0.smithi157.stdout:988: finishing write tid 2 to smithi15736696-46 2024-09-16T03:38:01.630 INFO:tasks.rados.rados.0.smithi157.stdout:1000: got (2) No such file or directory 2024-09-16T03:38:01.630 INFO:tasks.rados.rados.0.smithi157.stdout:1000: got expected ENOENT (src dne) 2024-09-16T03:38:01.630 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 62 (ObjNum 65 snap 0 seq_num 0) clean dne 2024-09-16T03:38:01.630 INFO:tasks.rados.rados.0.smithi157.stdout:992: done (11 left) 2024-09-16T03:38:01.630 INFO:tasks.rados.rados.0.smithi157.stdout:993: done (10 left) 2024-09-16T03:38:01.630 INFO:tasks.rados.rados.0.smithi157.stdout:995: done (9 left) 2024-09-16T03:38:01.630 INFO:tasks.rados.rados.0.smithi157.stdout:998: done (8 left) 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:999: done (7 left) 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:1000: done (6 left) 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:1001: done (5 left) 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:1002: read oid 48 snap -1 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:1002: expect deleted 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:1003: write oid 30 current snap is 0 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:988: finishing write tid 3 to smithi15736696-46 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:988: finishing write tid 4 to smithi15736696-46 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:988: oid 46 updating version 0 to 65 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:988: oid 46 version 65 is already newer than 64 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:988: oid 46 updating version 65 to 66 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 66 (ObjNum 163 snap 0 seq_num 163) dirty exists 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:988: left oid 46 (ObjNum 163 snap 0 seq_num 163) 2024-09-16T03:38:01.631 INFO:tasks.rados.rados.0.smithi157.stdout:991: finishing write tid 2 to smithi15736696-6 2024-09-16T03:38:01.632 INFO:tasks.rados.rados.0.smithi157.stdout:997: finishing write tid 1 to smithi15736696-5 2024-09-16T03:38:01.632 INFO:tasks.rados.rados.0.smithi157.stdout:1003: seq_num 166 ranges {789962=534747,1780952=690056} 2024-09-16T03:38:01.636 INFO:tasks.rados.rados.0.smithi157.stdout:1003: writing smithi15736696-30 from 789962 to 1324709 tid 1 2024-09-16T03:38:01.643 INFO:tasks.rados.rados.0.smithi157.stdout:1003: writing smithi15736696-30 from 1780952 to 2471008 tid 2 2024-09-16T03:38:01.643 INFO:tasks.rados.rados.0.smithi157.stdout:988: done (6 left) 2024-09-16T03:38:01.643 INFO:tasks.rados.rados.0.smithi157.stdout:1004: read oid 17 snap -1 2024-09-16T03:38:01.643 INFO:tasks.rados.rados.0.smithi157.stdout:1004: expect (ObjNum 129 snap 0 seq_num 129) 2024-09-16T03:38:01.643 INFO:tasks.rados.rados.0.smithi157.stdout:1005: read oid 47 snap -1 2024-09-16T03:38:01.643 INFO:tasks.rados.rados.0.smithi157.stdout:1005: expect deleted 2024-09-16T03:38:01.643 INFO:tasks.rados.rados.0.smithi157.stdout:1006: delete oid 25 current snap is 0 2024-09-16T03:38:01.643 INFO:tasks.rados.rados.0.smithi157.stdout:997: finishing write tid 2 to smithi15736696-5 2024-09-16T03:38:01.643 INFO:tasks.rados.rados.0.smithi157.stdout:997: finishing write tid 3 to smithi15736696-5 2024-09-16T03:38:01.643 INFO:tasks.rados.rados.0.smithi157.stdout:996: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-16T03:38:01.644 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:01.670 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:01.670 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:01.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:01.671 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:01.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.671 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:31 4.f 4:f0d14c71:::smithi15736696-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:01.671 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.043076+0000", 2024-09-16T03:38:01.671 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.624759582000003, 2024-09-16T03:38:01.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50359721000000002, 2024-09-16T03:38:01.671 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.671 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.671 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.671 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.671 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 31 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043076+0000", 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043076+0000", 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043078+0000", 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.189e-06 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043084+0000", 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.896e-06 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.043091+0000", 2024-09-16T03:38:01.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9770000000000003e-06 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143160+0000", 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10006909 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143222+0000", 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2049000000000002e-05 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143324+0000", 2024-09-16T03:38:01.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102737 2024-09-16T03:38:01.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:38:01.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143485+0000", 2024-09-16T03:38:01.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016040100000000001 2024-09-16T03:38:01.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.147043+0000", 2024-09-16T03:38:01.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0035581609999999998 2024-09-16T03:38:01.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.244108+0000", 2024-09-16T03:38:01.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.097064967000000002 2024-09-16T03:38:01.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.546541+0000", 2024-09-16T03:38:01.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30243345199999999 2024-09-16T03:38:01.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.546605+0000", 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4039000000000001e-05 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.546674+0000", 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8251999999999996e-05 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:66 4.0 4:023abada:::smithi15736696-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:01.678 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.200996+0000", 2024-09-16T03:38:01.679 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.466839968000002, 2024-09-16T03:38:01.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50798839299999998, 2024-09-16T03:38:01.679 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.679 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.679 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.679 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.679 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.679 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.679 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 66 2024-09-16T03:38:01.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.679 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.200996+0000", 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.200996+0000", 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.200998+0000", 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5179999999999999e-06 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.201330+0000", 2024-09-16T03:38:01.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000332672 2024-09-16T03:38:01.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.201340+0000", 2024-09-16T03:38:01.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8099999999999992e-06 2024-09-16T03:38:01.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301446+0000", 2024-09-16T03:38:01.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100105877 2024-09-16T03:38:01.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301543+0000", 2024-09-16T03:38:01.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7058000000000005e-05 2024-09-16T03:38:01.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301628+0000", 2024-09-16T03:38:01.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5078000000000001e-05 2024-09-16T03:38:01.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301770+0000", 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014212400000000001 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.303247+0000", 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001476595 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.606126+0000", 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30287937799999998 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.708902+0000", 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102775402 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.708938+0000", 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5947e-05 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.708984+0000", 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6934000000000002e-05 2024-09-16T03:38:01.685 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:69 4.0 4:0a55bbf2:::smithi15736696-18:head [write 685719~711939,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.301546+0000", 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.366290045, 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50794553600000003, 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.686 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 69 2024-09-16T03:38:01.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.687 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.687 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.687 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.687 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301546+0000", 2024-09-16T03:38:01.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.687 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.687 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.687 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301546+0000", 2024-09-16T03:38:01.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.687 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301547+0000", 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4799999999999997e-07 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302372+0000", 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000824828 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302381+0000", 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1759999999999999e-06 2024-09-16T03:38:01.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302389+0000", 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4710000000000007e-06 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.401577+0000", 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099188002999999997 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.401681+0000", 2024-09-16T03:38:01.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000103618 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.401797+0000", 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116177 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.403396+0000", 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001598857 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.606185+0000", 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202788996 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.809397+0000", 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20321167700000001 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.809449+0000", 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2389000000000003e-05 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.809492+0000", 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2496000000000001e-05 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:286 4.7 4:e2f8c24a:::smithi15736696-37:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:15.600823+0000", 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 46.067012779999999, 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50653485200000004, 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.692 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 286 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600823+0000", 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.693 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600823+0000", 2024-09-16T03:38:01.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.694 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.694 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.694 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600825+0000", 2024-09-16T03:38:01.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.736e-06 2024-09-16T03:38:01.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.694 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.694 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.694 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600830+0000", 2024-09-16T03:38:01.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2480000000000004e-06 2024-09-16T03:38:01.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.694 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.600842+0000", 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1309e-05 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.700982+0000", 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100140487 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701107+0000", 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000124671 2024-09-16T03:38:01.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.701165+0000", 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7995000000000003e-05 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.903559+0000", 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20239401100000001 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.904281+0000", 2024-09-16T03:38:01.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072182699999999997 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:15.904415+0000", 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013477300000000001 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.107259+0000", 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202843209 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.107321+0000", 2024-09-16T03:38:01.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2242000000000005e-05 2024-09-16T03:38:01.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.107358+0000", 2024-09-16T03:38:01.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7344e-05 2024-09-16T03:38:01.698 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.698 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.698 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.698 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:409 4.f 4:fa163101:::smithi15736696-10:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:01.698 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.306109+0000", 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 41.361726675, 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50402713799999999, 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 409 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306109+0000", 2024-09-16T03:38:01.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.700 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.700 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306109+0000", 2024-09-16T03:38:01.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.700 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.700 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306111+0000", 2024-09-16T03:38:01.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.429e-06 2024-09-16T03:38:01.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.701 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306116+0000", 2024-09-16T03:38:01.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1019999999999996e-06 2024-09-16T03:38:01.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.701 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306129+0000", 2024-09-16T03:38:01.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2840000000000001e-05 2024-09-16T03:38:01.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.306137+0000", 2024-09-16T03:38:01.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1370000000000002e-06 2024-09-16T03:38:01.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.506224+0000", 2024-09-16T03:38:01.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20008714899999999 2024-09-16T03:38:01.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.506352+0000", 2024-09-16T03:38:01.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000128424 2024-09-16T03:38:01.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:38:01.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709180+0000", 2024-09-16T03:38:01.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20282708299999999 2024-09-16T03:38:01.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.709780+0000", 2024-09-16T03:38:01.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060068800000000005 2024-09-16T03:38:01.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.710070+0000", 2024-09-16T03:38:01.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000289683 2024-09-16T03:38:01.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.809989+0000", 2024-09-16T03:38:01.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099918689000000005 2024-09-16T03:38:01.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.810070+0000", 2024-09-16T03:38:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1135000000000002e-05 2024-09-16T03:38:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.810136+0000", 2024-09-16T03:38:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6779000000000006e-05 2024-09-16T03:38:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.705 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:416 4.0 4:023abada:::smithi15736696-17:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:38:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 41.358594320999998, 2024-09-16T03:38:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60401983699999995, 2024-09-16T03:38:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 416 2024-09-16T03:38:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.707 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:38:01.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.707 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:38:01.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.707 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309243+0000", 2024-09-16T03:38:01.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.178e-06 2024-09-16T03:38:01.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309248+0000", 2024-09-16T03:38:01.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:38:01.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309257+0000", 2024-09-16T03:38:01.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4720000000000001e-06 2024-09-16T03:38:01.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309262+0000", 2024-09-16T03:38:01.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0839999999999999e-06 2024-09-16T03:38:01.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309297+0000", 2024-09-16T03:38:01.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4997999999999999e-05 2024-09-16T03:38:01.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309367+0000", 2024-09-16T03:38:01.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0124999999999997e-05 2024-09-16T03:38:01.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812055+0000", 2024-09-16T03:38:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50268796900000001 2024-09-16T03:38:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812614+0000", 2024-09-16T03:38:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000558309 2024-09-16T03:38:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.813040+0000", 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042652800000000002 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913189+0000", 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10014880399999999 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913220+0000", 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1353000000000003e-05 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913262+0000", 2024-09-16T03:38:01.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1284e-05 2024-09-16T03:38:01.712 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.712 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.712 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.712 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:494 4.f 4:f817def9:::smithi15736696-45:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:01.712 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:38:01.712 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 39.945654586000003, 2024-09-16T03:38:01.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60451760399999999, 2024-09-16T03:38:01.712 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.712 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.713 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.713 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.713 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.713 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.713 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 494 2024-09-16T03:38:01.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.713 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:38:01.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:38:01.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722183+0000", 2024-09-16T03:38:01.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:38:01.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722189+0000", 2024-09-16T03:38:01.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9429999999999997e-06 2024-09-16T03:38:01.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722203+0000", 2024-09-16T03:38:01.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4367e-05 2024-09-16T03:38:01.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722210+0000", 2024-09-16T03:38:01.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:38:01.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822367+0000", 2024-09-16T03:38:01.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100156933 2024-09-16T03:38:01.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822461+0000", 2024-09-16T03:38:01.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3216999999999999e-05 2024-09-16T03:38:01.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:38:01.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125593+0000", 2024-09-16T03:38:01.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30313253699999998 2024-09-16T03:38:01.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126307+0000", 2024-09-16T03:38:01.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071350000000000005 2024-09-16T03:38:01.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126825+0000", 2024-09-16T03:38:01.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051846100000000001 2024-09-16T03:38:01.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326612+0000", 2024-09-16T03:38:01.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19978721999999999 2024-09-16T03:38:01.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326663+0000", 2024-09-16T03:38:01.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0460000000000001e-05 2024-09-16T03:38:01.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326699+0000", 2024-09-16T03:38:01.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6340000000000001e-05 2024-09-16T03:38:01.719 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.719 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.719 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.719 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:517 4.0 4:0a55bbf2:::smithi15736696-18:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:01.719 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.922514+0000", 2024-09-16T03:38:01.719 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 39.745321955999998, 2024-09-16T03:38:01.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50578298600000005, 2024-09-16T03:38:01.719 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.719 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.719 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 517 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922514+0000", 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922514+0000", 2024-09-16T03:38:01.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922515+0000", 2024-09-16T03:38:01.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.381e-06 2024-09-16T03:38:01.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922520+0000", 2024-09-16T03:38:01.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8590000000000004e-06 2024-09-16T03:38:01.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922531+0000", 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0783e-05 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.922539+0000", 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4220000000000003e-06 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.923072+0000", 2024-09-16T03:38:01.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053305500000000005 2024-09-16T03:38:01.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.923145+0000", 2024-09-16T03:38:01.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3572000000000006e-05 2024-09-16T03:38:01.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:01.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.327214+0000", 2024-09-16T03:38:01.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.404069177 2024-09-16T03:38:01.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.328231+0000", 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010163240000000001 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.330360+0000", 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021293029999999999 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.428103+0000", 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.097742869999999996 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.428244+0000", 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014071999999999999 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.428297+0000", 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3520000000000003e-05 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.725 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:784 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=63b,truncate 2840350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:01.726 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:38:01.726 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.861246710000003, 2024-09-16T03:38:01.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63358405200000001, 2024-09-16T03:38:01.726 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.726 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.726 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.726 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.726 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.726 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.726 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 784 2024-09-16T03:38:01.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.726 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:38:01.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:38:01.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806590+0000", 2024-09-16T03:38:01.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6000000000000003e-07 2024-09-16T03:38:01.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806593+0000", 2024-09-16T03:38:01.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6649999999999999e-06 2024-09-16T03:38:01.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806597+0000", 2024-09-16T03:38:01.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.245e-06 2024-09-16T03:38:01.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806600+0000", 2024-09-16T03:38:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.23e-06 2024-09-16T03:38:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099381+0000", 2024-09-16T03:38:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29278119400000002 2024-09-16T03:38:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099438+0000", 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6198e-05 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099527+0000", 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9505000000000004e-05 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200775+0000", 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101247904 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200889+0000", 2024-09-16T03:38:01.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114301 2024-09-16T03:38:01.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440066+0000", 2024-09-16T03:38:01.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.239176367 2024-09-16T03:38:01.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440109+0000", 2024-09-16T03:38:01.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2941000000000001e-05 2024-09-16T03:38:01.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440173+0000", 2024-09-16T03:38:01.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4842e-05 2024-09-16T03:38:01.732 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.732 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.732 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.732 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:785 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:01.732 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:38:01.732 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.861230785000004, 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63376373200000002, 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 785 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:38:01.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:38:01.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806606+0000", 2024-09-16T03:38:01.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:38:01.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806607+0000", 2024-09-16T03:38:01.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.424e-06 2024-09-16T03:38:01.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806611+0000", 2024-09-16T03:38:01.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1080000000000003e-06 2024-09-16T03:38:01.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806614+0000", 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.067e-06 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099813+0000", 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29319853699999998 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099858+0000", 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5667999999999997e-05 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440236+0000", 2024-09-16T03:38:01.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34037791499999998 2024-09-16T03:38:01.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440297+0000", 2024-09-16T03:38:01.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0988e-05 2024-09-16T03:38:01.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440369+0000", 2024-09-16T03:38:01.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1682999999999998e-05 2024-09-16T03:38:01.738 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.738 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.738 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.738 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:801 4.0 4:07ad0560:::smithi15736696-25:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:01.738 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:38:01.738 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.661752886999999, 2024-09-16T03:38:01.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.534510445, 2024-09-16T03:38:01.738 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.738 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.738 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:01.738 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.739 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.739 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.739 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 801 2024-09-16T03:38:01.739 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.739 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.739 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.739 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.739 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:38:01.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.739 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.739 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.739 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.739 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:38:01.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006084+0000", 2024-09-16T03:38:01.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.322e-06 2024-09-16T03:38:01.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006088+0000", 2024-09-16T03:38:01.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000001e-06 2024-09-16T03:38:01.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006098+0000", 2024-09-16T03:38:01.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0186999999999999e-05 2024-09-16T03:38:01.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006104+0000", 2024-09-16T03:38:01.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.271e-06 2024-09-16T03:38:01.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.199952+0000", 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.193847935 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200027+0000", 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4987000000000006e-05 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.540594+0000", 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34056708299999999 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.742 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.743 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1022 4.0 4:06b6d050:::smithi15736696-50:head [write 776490~714607] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:01.743 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.310462+0000", 2024-09-16T03:38:01.743 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.357373557999999, 2024-09-16T03:38:01.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50630361300000004, 2024-09-16T03:38:01.743 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.743 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.743 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.743 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.743 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.743 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.743 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1022 2024-09-16T03:38:01.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.744 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.744 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.744 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.310462+0000", 2024-09-16T03:38:01.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.744 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.744 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.310462+0000", 2024-09-16T03:38:01.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.310464+0000", 2024-09-16T03:38:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.567e-06 2024-09-16T03:38:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311198+0000", 2024-09-16T03:38:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000734265 2024-09-16T03:38:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311212+0000", 2024-09-16T03:38:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4022000000000001e-05 2024-09-16T03:38:01.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311219+0000", 2024-09-16T03:38:01.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6259999999999997e-06 2024-09-16T03:38:01.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.356651+0000", 2024-09-16T03:38:01.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.045432019999999997 2024-09-16T03:38:01.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.356728+0000", 2024-09-16T03:38:01.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6823000000000006e-05 2024-09-16T03:38:01.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:01.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.356841+0000", 2024-09-16T03:38:01.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113606 2024-09-16T03:38:01.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361972+0000", 2024-09-16T03:38:01.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0051310319999999998 2024-09-16T03:38:01.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.748 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362101+0000", 2024-09-16T03:38:01.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000128777 2024-09-16T03:38:01.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.748 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.816692+0000", 2024-09-16T03:38:01.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.45459126599999999 2024-09-16T03:38:01.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.749 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.749 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.816734+0000", 2024-09-16T03:38:01.749 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1276000000000003e-05 2024-09-16T03:38:01.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.749 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.749 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.749 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.816766+0000", 2024-09-16T03:38:01.749 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2332999999999997e-05 2024-09-16T03:38:01.749 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.749 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.749 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.749 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.750 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1023 4.0 4:06b6d050:::smithi15736696-50:head [write 2246563~579356] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:01.750 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:38:01.750 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.356603817, 2024-09-16T03:38:01.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60727534699999997, 2024-09-16T03:38:01.750 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.752 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.752 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.752 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1023 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:38:01.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311233+0000", 2024-09-16T03:38:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1039999999999999e-06 2024-09-16T03:38:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311692+0000", 2024-09-16T03:38:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045908800000000002 2024-09-16T03:38:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311704+0000", 2024-09-16T03:38:01.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1844e-05 2024-09-16T03:38:01.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311713+0000", 2024-09-16T03:38:01.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2500000000000006e-06 2024-09-16T03:38:01.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359204+0000", 2024-09-16T03:38:01.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.047491646999999998 2024-09-16T03:38:01.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.756 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359258+0000", 2024-09-16T03:38:01.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4082000000000002e-05 2024-09-16T03:38:01.756 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.756 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:01.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359354+0000", 2024-09-16T03:38:01.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5686999999999994e-05 2024-09-16T03:38:01.756 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.756 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512245+0000", 2024-09-16T03:38:01.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15289135700000001 2024-09-16T03:38:01.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512956+0000", 2024-09-16T03:38:01.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071110300000000004 2024-09-16T03:38:01.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918361+0000", 2024-09-16T03:38:01.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40540489400000002 2024-09-16T03:38:01.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918455+0000", 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3412999999999995e-05 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918508+0000", 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.758 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1024 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3176490] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.356110835999999, 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60701227000000002, 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1024 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311726+0000", 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.02e-07 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.760 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311730+0000", 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1099999999999996e-06 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311735+0000", 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4680000000000003e-06 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311739+0000", 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9380000000000002e-06 2024-09-16T03:38:01.761 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.762 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.762 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.762 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361468+0000", 2024-09-16T03:38:01.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.049729450000000001 2024-09-16T03:38:01.762 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.762 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.762 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.762 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361539+0000", 2024-09-16T03:38:01.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0896999999999996e-05 2024-09-16T03:38:01.762 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.762 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.762 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:01.762 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361661+0000", 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000122404 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512943+0000", 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.151281481 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512959+0000", 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.763 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.764 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918588+0000", 2024-09-16T03:38:01.764 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40562815499999999 2024-09-16T03:38:01.764 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.764 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.764 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918646+0000", 2024-09-16T03:38:01.764 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8566999999999997e-05 2024-09-16T03:38:01.764 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.764 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.764 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918737+0000", 2024-09-16T03:38:01.764 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1308000000000001e-05 2024-09-16T03:38:01.764 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1025 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.356092338, 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60723649599999996, 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.765 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1025 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:01.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9400000000000001e-07 2024-09-16T03:38:01.767 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.767 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311746+0000", 2024-09-16T03:38:01.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2670000000000001e-06 2024-09-16T03:38:01.767 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.767 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311752+0000", 2024-09-16T03:38:01.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6389999999999997e-06 2024-09-16T03:38:01.768 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.768 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.768 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311757+0000", 2024-09-16T03:38:01.768 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3639999999999998e-06 2024-09-16T03:38:01.768 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.768 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.768 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362018+0000", 2024-09-16T03:38:01.768 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.050260969000000003 2024-09-16T03:38:01.768 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.768 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362089+0000", 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0735000000000006e-05 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918781+0000", 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55669215400000005 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918867+0000", 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6019000000000004e-05 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.770 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.770 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918980+0000", 2024-09-16T03:38:01.770 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011295500000000001 2024-09-16T03:38:01.770 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.770 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.770 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.770 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.770 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.770 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1054 4.10 4:0a55bbf2:::smithi15736696-18:head [write 2633066~610652] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:01.770 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:38:01.770 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.154599130000001, 2024-09-16T03:38:01.770 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51115411600000005, 2024-09-16T03:38:01.771 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.771 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.771 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.771 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.771 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.771 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.771 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1054 2024-09-16T03:38:01.771 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.771 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.771 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:38:01.771 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513238+0000", 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1999999999999998e-07 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513695+0000", 2024-09-16T03:38:01.773 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045695099999999998 2024-09-16T03:38:01.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.773 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.773 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513720+0000", 2024-09-16T03:38:01.773 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.567e-05 2024-09-16T03:38:01.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.773 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.773 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513725+0000", 2024-09-16T03:38:01.773 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9899999999999997e-06 2024-09-16T03:38:01.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613892+0000", 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166253 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613898+0000", 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8710000000000002e-06 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916870+0000", 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30297253800000001 2024-09-16T03:38:01.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.775 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.775 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916934+0000", 2024-09-16T03:38:01.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3723e-05 2024-09-16T03:38:01.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.775 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.775 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:01.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.917033+0000", 2024-09-16T03:38:01.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8908999999999998e-05 2024-09-16T03:38:01.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.775 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.775 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921278+0000", 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0042456999999999998 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019617+0000", 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098338910000000002 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024269+0000", 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046515519999999998 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.776 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.777 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.777 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024358+0000", 2024-09-16T03:38:01.777 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9096000000000002e-05 2024-09-16T03:38:01.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.777 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.777 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.777 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024391+0000", 2024-09-16T03:38:01.777 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3033e-05 2024-09-16T03:38:01.777 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.777 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.777 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1055 4.10 4:0a55bbf2:::smithi15736696-18:head [setxattr _header (56) in=7b,truncate 3243718] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:38:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.154099630000001, 2024-09-16T03:38:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61098759599999997, 2024-09-16T03:38:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.778 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1055 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513737+0000", 2024-09-16T03:38:01.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9399999999999993e-07 2024-09-16T03:38:01.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.780 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.780 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513759+0000", 2024-09-16T03:38:01.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1903999999999999e-05 2024-09-16T03:38:01.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.780 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.780 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513773+0000", 2024-09-16T03:38:01.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3244000000000001e-05 2024-09-16T03:38:01.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.781 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.781 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.781 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513780+0000", 2024-09-16T03:38:01.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999999e-06 2024-09-16T03:38:01.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.781 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.781 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.781 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714016+0000", 2024-09-16T03:38:01.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20023635000000001 2024-09-16T03:38:01.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.781 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.781 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:38:01.781 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714024+0000", 2024-09-16T03:38:01.782 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8359999999999992e-06 2024-09-16T03:38:01.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.782 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.782 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918736+0000", 2024-09-16T03:38:01.782 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471236700000001 2024-09-16T03:38:01.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.782 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.782 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918830+0000", 2024-09-16T03:38:01.782 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3769000000000003e-05 2024-09-16T03:38:01.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.783 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:01.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919003+0000", 2024-09-16T03:38:01.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000173198 2024-09-16T03:38:01.783 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.783 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.783 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:01.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921589+0000", 2024-09-16T03:38:01.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002586419 2024-09-16T03:38:01.783 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.783 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.783 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019655+0000", 2024-09-16T03:38:01.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098065801999999994 2024-09-16T03:38:01.784 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.784 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.784 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:01.784 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124538+0000", 2024-09-16T03:38:01.784 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10488272899999999 2024-09-16T03:38:01.784 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.784 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.784 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:01.784 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124635+0000", 2024-09-16T03:38:01.784 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7125000000000003e-05 2024-09-16T03:38:01.784 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.784 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.784 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124724+0000", 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8858999999999997e-05 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1056 4.10 4:0a55bbf2:::smithi15736696-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.154047059, 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61111632800000004, 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.785 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1056 2024-09-16T03:38:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2499999999999995e-07 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513792+0000", 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513797+0000", 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4940000000000003e-06 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513802+0000", 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5390000000000001e-06 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919221+0000", 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40541913899999998 2024-09-16T03:38:01.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:01.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919268+0000", 2024-09-16T03:38:01.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7027000000000002e-05 2024-09-16T03:38:01.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124766+0000", 2024-09-16T03:38:01.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20549842099999999 2024-09-16T03:38:01.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124828+0000", 2024-09-16T03:38:01.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1803000000000004e-05 2024-09-16T03:38:01.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124905+0000", 2024-09-16T03:38:01.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7163000000000006e-05 2024-09-16T03:38:01.790 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.790 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.790 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.791 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1219 4.f 4:f817def9:::smithi15736696-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:38:01.791 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.370389+0000", 2024-09-16T03:38:01.791 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 25.29744711, 2024-09-16T03:38:01.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50632193400000003, 2024-09-16T03:38:01.791 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.791 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.791 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:01.791 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.791 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.791 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.791 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1219 2024-09-16T03:38:01.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.792 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.370389+0000", 2024-09-16T03:38:01.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.370389+0000", 2024-09-16T03:38:01.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.370390+0000", 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4000000000000001e-07 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.370393+0000", 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.45e-06 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.370400+0000", 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0840000000000003e-06 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.370406+0000", 2024-09-16T03:38:01.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0229999999999998e-06 2024-09-16T03:38:01.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.775357+0000", 2024-09-16T03:38:01.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40495086400000002 2024-09-16T03:38:01.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:01.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.775448+0000", 2024-09-16T03:38:01.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0650999999999997e-05 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.876590+0000", 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10114192800000001 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.876636+0000", 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6607000000000003e-05 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.876711+0000", 2024-09-16T03:38:01.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4686999999999998e-05 2024-09-16T03:38:01.796 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.796 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.796 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.796 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1702 4.10 4:0a55bbf2:::smithi15736696-18:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:01.796 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:38:01.796 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 10.978692069000001, 2024-09-16T03:38:01.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51800317600000001, 2024-09-16T03:38:01.796 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:01.796 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:01.796 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:01.797 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:01.797 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:01.797 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:01.797 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1702 2024-09-16T03:38:01.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.797 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:01.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:01.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:38:01.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689147+0000", 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689158+0000", 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0426e-05 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689185+0000", 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7299000000000001e-05 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789328+0000", 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100143102 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789532+0000", 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000204317 2024-09-16T03:38:01.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:01.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789671+0000", 2024-09-16T03:38:01.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013852600000000001 2024-09-16T03:38:01.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:01.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:01.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:01.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.207147+0000", 2024-09-16T03:38:01.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.41747625599999999 2024-09-16T03:38:01.800 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.800 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.800 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.801 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:01.801 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:01.801 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:01.802 INFO:tasks.rados.rados.0.smithi157.stdout:990: finishing set_chunk to oid 16 2024-09-16T03:38:01.802 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 33 (ObjNum 122 snap 0 seq_num 122) dirty exists 2024-09-16T03:38:01.802 INFO:tasks.rados.rados.0.smithi157.stdout:997: finishing write tid 4 to smithi15736696-5 2024-09-16T03:38:01.802 INFO:tasks.rados.rados.0.smithi157.stdout:997: finishing write tid 5 to smithi15736696-5 2024-09-16T03:38:01.802 INFO:tasks.rados.rados.0.smithi157.stdout:997: oid 5 updating version 0 to 47 2024-09-16T03:38:01.802 INFO:tasks.rados.rados.0.smithi157.stdout:997: oid 5 updating version 47 to 50 2024-09-16T03:38:01.802 INFO:tasks.rados.rados.0.smithi157.stdout:997: oid 5 version 50 is already newer than 49 2024-09-16T03:38:01.802 INFO:tasks.rados.rados.0.smithi157.stdout:997: oid 5 version 50 is already newer than 48 2024-09-16T03:38:01.803 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 50 (ObjNum 165 snap 0 seq_num 165) dirty exists 2024-09-16T03:38:01.803 INFO:tasks.rados.rados.0.smithi157.stdout:997: left oid 5 (ObjNum 165 snap 0 seq_num 165) 2024-09-16T03:38:01.803 INFO:tasks.rados.rados.0.smithi157.stdout:991: finishing write tid 3 to smithi15736696-6 2024-09-16T03:38:01.803 INFO:tasks.rados.rados.0.smithi157.stdout:991: finishing write tid 4 to smithi15736696-6 2024-09-16T03:38:01.803 INFO:tasks.rados.rados.0.smithi157.stdout:991: oid 6 updating version 0 to 48 2024-09-16T03:38:01.803 INFO:tasks.rados.rados.0.smithi157.stdout:991: oid 6 version 48 is already newer than 47 2024-09-16T03:38:01.803 INFO:tasks.rados.rados.0.smithi157.stdout:991: oid 6 version 48 is already newer than 46 2024-09-16T03:38:01.803 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 48 (ObjNum 164 snap 0 seq_num 164) dirty exists 2024-09-16T03:38:01.803 INFO:tasks.rados.rados.0.smithi157.stdout:991: left oid 6 (ObjNum 164 snap 0 seq_num 164) 2024-09-16T03:38:01.804 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:01.800+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:01.804 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:38:01.903 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:01.902+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:01.908 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:01.909 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:01.909 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:01.909 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:01.917 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:38:01.933 INFO:tasks.rados.rados.0.smithi157.stdout:1004: expect (ObjNum 129 snap 0 seq_num 129) 2024-09-16T03:38:01.969 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 917400 2024-09-16T03:38:02.004 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:02.001+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:02.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:38:02.044 INFO:tasks.rados.rados.0.smithi157.stdout:990: done (8 left) 2024-09-16T03:38:02.044 INFO:tasks.rados.rados.0.smithi157.stdout:991: done (7 left) 2024-09-16T03:38:02.044 INFO:tasks.rados.rados.0.smithi157.stdout:996: done (6 left) 2024-09-16T03:38:02.044 INFO:tasks.rados.rados.0.smithi157.stdout:997: done (5 left) 2024-09-16T03:38:02.044 INFO:tasks.rados.rados.0.smithi157.stdout:1002: done (4 left) 2024-09-16T03:38:02.045 INFO:tasks.rados.rados.0.smithi157.stdout:1004: done (3 left) 2024-09-16T03:38:02.045 INFO:tasks.rados.rados.0.smithi157.stdout:1005: done (2 left) 2024-09-16T03:38:02.045 INFO:tasks.rados.rados.0.smithi157.stdout:1006: done (1 left) 2024-09-16T03:38:02.045 INFO:tasks.rados.rados.0.smithi157.stdout:1007: read oid 5 snap -1 2024-09-16T03:38:02.045 INFO:tasks.rados.rados.0.smithi157.stdout:1007: expect (ObjNum 165 snap 0 seq_num 165) 2024-09-16T03:38:02.045 INFO:tasks.rados.rados.0.smithi157.stdout:1008: read oid 28 snap -1 2024-09-16T03:38:02.045 INFO:tasks.rados.rados.0.smithi157.stdout:1008: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-16T03:38:02.045 INFO:tasks.rados.rados.0.smithi157.stdout:1009: read oid 35 snap -1 2024-09-16T03:38:02.045 INFO:tasks.rados.rados.0.smithi157.stdout:1009: expect (ObjNum 161 snap 0 seq_num 161) 2024-09-16T03:38:02.045 INFO:tasks.rados.rados.0.smithi157.stdout:1010: delete oid 26 current snap is 0 2024-09-16T03:38:02.048 INFO:tasks.rados.rados.0.smithi157.stdout:1008: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-16T03:38:02.060 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:02.060 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:02.060 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:02.060 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:02.061 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:02.071 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:38:02.079 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2701105 2024-09-16T03:38:02.101 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:02.104 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:02.102+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:02.156 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:02.157 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:02.175 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:02.175 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:02.176 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:02.176 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:02.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:02.176 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:02.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.177 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:02.177 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:02.177 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.030018445000003, 2024-09-16T03:38:02.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:38:02.177 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.177 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.177 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.177 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:02.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:02.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:38:02.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:38:02.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:38:02.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:38:02.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:38:02.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:38:02.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:38:02.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:38:02.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:38:02.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:38:02.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:38:02.181 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.182 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.182 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.182 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:38:02.182 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:02.182 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.029665084999998, 2024-09-16T03:38:02.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:38:02.182 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.182 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.182 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.182 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:02.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:02.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:38:02.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:38:02.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:38:02.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:38:02.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.186 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.187 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.187 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.187 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.187 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:38:02.187 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:02.187 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.028537286999999, 2024-09-16T03:38:02.187 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:38:02.187 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.187 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.188 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:38:02.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:38:02.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.189 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:38:02.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:38:02.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.189 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:38:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:38:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:38:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:38:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:38:02.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:38:02.192 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.192 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.192 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.192 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.192 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.192 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:38:02.192 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:02.192 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.027549845999999, 2024-09-16T03:38:02.192 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:38:02.192 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.192 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.192 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.193 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.193 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.193 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:02.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.193 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.193 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:02.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:38:02.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:38:02.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:38:02.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:38:02.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:38:02.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:38:02.196 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.197 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.197 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.197 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:38:02.197 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:02.197 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.026793101999999, 2024-09-16T03:38:02.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:38:02.197 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.197 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.197 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.197 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:02.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.200 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.200 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:38:02.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:38:02.200 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.200 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.200 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.200 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:38:02.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:38:02.200 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.200 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.200 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.200 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:38:02.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.201 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:38:02.202 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:38:02.202 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.202 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.202 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.202 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.202 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:86 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'8, mlcod=39'5)", 2024-09-16T03:38:02.202 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:02.202 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.565280866000002, 2024-09-16T03:38:02.202 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20388281899999999, 2024-09-16T03:38:02.202 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.202 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.202 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.203 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:02.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608472+0000", 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.227e-06 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608616+0000", 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000144177 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608638+0000", 2024-09-16T03:38:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1543e-05 2024-09-16T03:38:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608645+0000", 2024-09-16T03:38:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5839999999999997e-06 2024-09-16T03:38:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608740+0000", 2024-09-16T03:38:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4716999999999995e-05 2024-09-16T03:38:02.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608763+0000", 2024-09-16T03:38:02.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2671e-05 2024-09-16T03:38:02.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812264+0000", 2024-09-16T03:38:02.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203501194 2024-09-16T03:38:02.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812353+0000", 2024-09-16T03:38:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8706000000000001e-05 2024-09-16T03:38:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:87 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'9, mlcod=39'5)", 2024-09-16T03:38:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.565067491999997, 2024-09-16T03:38:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20372206400000001, 2024-09-16T03:38:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.208 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608684+0000", 2024-09-16T03:38:02.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6700000000000003e-07 2024-09-16T03:38:02.209 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.209 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608689+0000", 2024-09-16T03:38:02.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.412e-06 2024-09-16T03:38:02.209 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.209 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608704+0000", 2024-09-16T03:38:02.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4397e-05 2024-09-16T03:38:02.209 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.210 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608713+0000", 2024-09-16T03:38:02.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:38:02.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.210 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710025+0000", 2024-09-16T03:38:02.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101312032 2024-09-16T03:38:02.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.210 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710042+0000", 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6889e-05 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812361+0000", 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10231889600000001 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812405+0000", 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4101999999999999e-05 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.211 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:101 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'10, mlcod=39'5)", 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.564054055, 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20275586900000001, 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:02.212 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609698+0000", 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.24e-06 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609892+0000", 2024-09-16T03:38:02.214 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019465899999999999 2024-09-16T03:38:02.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.214 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.214 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609904+0000", 2024-09-16T03:38:02.214 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2031000000000001e-05 2024-09-16T03:38:02.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.214 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.214 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609912+0000", 2024-09-16T03:38:02.214 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7160000000000003e-06 2024-09-16T03:38:02.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710228+0000", 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100315682 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710242+0000", 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4201e-05 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812411+0000", 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10216881799999999 2024-09-16T03:38:02.215 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.216 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.216 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.216 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812452+0000", 2024-09-16T03:38:02.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1522e-05 2024-09-16T03:38:02.216 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.216 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.216 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.216 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.216 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:102 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'11, mlcod=39'5)", 2024-09-16T03:38:02.216 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:02.216 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.562038029999997, 2024-09-16T03:38:02.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20134195399999999, 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611714+0000", 2024-09-16T03:38:02.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.269e-06 2024-09-16T03:38:02.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611926+0000", 2024-09-16T03:38:02.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021252700000000001 2024-09-16T03:38:02.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611938+0000", 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1437000000000001e-05 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611944+0000", 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.993e-06 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811266+0000", 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.199322306 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.219 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.220 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811291+0000", 2024-09-16T03:38:02.220 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4542999999999999e-05 2024-09-16T03:38:02.220 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.220 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.220 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.220 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813021+0000", 2024-09-16T03:38:02.220 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001730363 2024-09-16T03:38:02.220 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.220 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.220 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.220 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813054+0000", 2024-09-16T03:38:02.220 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3516000000000002e-05 2024-09-16T03:38:02.221 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.221 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.221 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.221 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.221 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.221 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:103 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'12, mlcod=39'5)", 2024-09-16T03:38:02.221 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:38:02.221 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.558831529000003, 2024-09-16T03:38:02.221 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19838012999999999, 2024-09-16T03:38:02.221 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.222 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.222 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.222 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.222 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.222 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.222 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:38:02.222 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.222 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.222 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.222 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.222 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:38:02.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.223 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.223 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.223 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614920+0000", 2024-09-16T03:38:02.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.192e-06 2024-09-16T03:38:02.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.223 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.223 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.223 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614927+0000", 2024-09-16T03:38:02.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4640000000000003e-06 2024-09-16T03:38:02.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.223 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614937+0000", 2024-09-16T03:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9599999999999995e-06 2024-09-16T03:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614942+0000", 2024-09-16T03:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.31e-06 2024-09-16T03:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812489+0000", 2024-09-16T03:38:02.224 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.197546629 2024-09-16T03:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812511+0000", 2024-09-16T03:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2385e-05 2024-09-16T03:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813279+0000", 2024-09-16T03:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076818100000000005 2024-09-16T03:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.225 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813299+0000", 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0009000000000001e-05 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:375 4.f e45/32 4:fcb75e43:::smithi15736696-26:head v 45'23, mlcod=41'20)", 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 41.865710301999997, 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200786568, 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.226 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.227 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308041+0000", 2024-09-16T03:38:02.228 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8299999999999995e-07 2024-09-16T03:38:02.228 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.228 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.228 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.228 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308046+0000", 2024-09-16T03:38:02.228 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.904e-06 2024-09-16T03:38:02.228 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.228 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.228 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.228 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308056+0000", 2024-09-16T03:38:02.228 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9210000000000004e-06 2024-09-16T03:38:02.228 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.228 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.229 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.229 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.408122+0000", 2024-09-16T03:38:02.229 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100066302 2024-09-16T03:38:02.229 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.229 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.229 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.229 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508289+0000", 2024-09-16T03:38:02.229 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166531 2024-09-16T03:38:02.229 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.229 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.229 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.229 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508302+0000", 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2641e-05 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508794+0000", 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00049272400000000005 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508827+0000", 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2561999999999998e-05 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.230 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.231 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.231 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.231 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:459 4.f e45/32 4:fa163101:::smithi15736696-10:head v 45'25, mlcod=45'23)", 2024-09-16T03:38:02.231 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.318000+0000", 2024-09-16T03:38:02.231 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 40.855750397999998, 2024-09-16T03:38:02.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10092035100000001, 2024-09-16T03:38:02.231 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.231 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.231 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.231 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318000+0000", 2024-09-16T03:38:02.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318000+0000", 2024-09-16T03:38:02.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318002+0000", 2024-09-16T03:38:02.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.624e-06 2024-09-16T03:38:02.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.233 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.233 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318009+0000", 2024-09-16T03:38:02.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9310000000000002e-06 2024-09-16T03:38:02.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.233 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.233 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318018+0000", 2024-09-16T03:38:02.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4949999999999997e-06 2024-09-16T03:38:02.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.233 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.233 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.318031+0000", 2024-09-16T03:38:02.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2869e-05 2024-09-16T03:38:02.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.234 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418228+0000", 2024-09-16T03:38:02.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10019649799999999 2024-09-16T03:38:02.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.234 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418286+0000", 2024-09-16T03:38:02.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8904999999999999e-05 2024-09-16T03:38:02.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.235 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.235 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418884+0000", 2024-09-16T03:38:02.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059763899999999996 2024-09-16T03:38:02.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.235 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.235 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.235 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.418921+0000", 2024-09-16T03:38:02.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6390000000000002e-05 2024-09-16T03:38:02.235 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.235 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.236 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.236 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.236 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.236 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:989 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'31, mlcod=46'28)", 2024-09-16T03:38:02.236 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:02.236 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.785941741999999, 2024-09-16T03:38:02.236 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40332266900000002, 2024-09-16T03:38:02.236 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.236 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.236 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.236 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.236 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.236 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.237 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:02.237 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.237 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.237 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.237 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.237 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:02.237 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.237 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.237 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.237 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.237 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387812+0000", 2024-09-16T03:38:02.237 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388467+0000", 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065539500000000002 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388481+0000", 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3905e-05 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.238 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488552+0000", 2024-09-16T03:38:02.239 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10007120899999999 2024-09-16T03:38:02.239 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.239 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.239 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.239 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688818+0000", 2024-09-16T03:38:02.239 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20026627599999999 2024-09-16T03:38:02.239 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.239 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.239 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.239 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688845+0000", 2024-09-16T03:38:02.239 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:38:02.239 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.239 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.240 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.240 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791091+0000", 2024-09-16T03:38:02.240 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10224588699999999 2024-09-16T03:38:02.240 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.240 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.240 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.240 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791131+0000", 2024-09-16T03:38:02.240 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0293000000000002e-05 2024-09-16T03:38:02.240 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.240 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.240 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.240 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.240 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:990 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'32, mlcod=46'28)", 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.68515807, 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30256419200000001, 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.241 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.242 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.242 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:02.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.242 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.242 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.242 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.242 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488594+0000", 2024-09-16T03:38:02.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:38:02.242 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.242 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.242 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.242 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488616+0000", 2024-09-16T03:38:02.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2419e-05 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488627+0000", 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1251e-05 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588753+0000", 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012544399999999 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690679+0000", 2024-09-16T03:38:02.244 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101926027 2024-09-16T03:38:02.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.244 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690695+0000", 2024-09-16T03:38:02.244 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6528000000000002e-05 2024-09-16T03:38:02.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.244 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791133+0000", 2024-09-16T03:38:02.244 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100438233 2024-09-16T03:38:02.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791157+0000", 2024-09-16T03:38:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-05 2024-09-16T03:38:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:991 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'33, mlcod=46'28)", 2024-09-16T03:38:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.584960573, 2024-09-16T03:38:02.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202786365, 2024-09-16T03:38:02.246 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.246 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.246 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.246 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.246 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.246 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.246 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:02.246 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.246 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.246 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.246 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.246 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:02.246 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.247 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.247 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.247 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.247 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588792+0000", 2024-09-16T03:38:02.247 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0530000000000001e-06 2024-09-16T03:38:02.247 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.247 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.247 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.247 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588807+0000", 2024-09-16T03:38:02.247 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5257e-05 2024-09-16T03:38:02.247 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.247 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.247 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.248 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588832+0000", 2024-09-16T03:38:02.248 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5123e-05 2024-09-16T03:38:02.248 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.248 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.248 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.248 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588846+0000", 2024-09-16T03:38:02.248 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3312999999999999e-05 2024-09-16T03:38:02.248 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.248 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.248 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790810+0000", 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20196378100000001 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790866+0000", 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6168999999999997e-05 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791542+0000", 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067583699999999999 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791576+0000", 2024-09-16T03:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4832000000000001e-05 2024-09-16T03:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1072 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'35, mlcod=50'32)", 2024-09-16T03:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 29.865718745999999, 2024-09-16T03:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.103370715, 2024-09-16T03:38:02.250 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308034+0000", 2024-09-16T03:38:02.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4099999999999998e-06 2024-09-16T03:38:02.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.252 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308377+0000", 2024-09-16T03:38:02.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034276400000000001 2024-09-16T03:38:02.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.252 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308415+0000", 2024-09-16T03:38:02.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7916000000000001e-05 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408542+0000", 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012673699999999 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408694+0000", 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015200399999999999 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408760+0000", 2024-09-16T03:38:02.253 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6549000000000003e-05 2024-09-16T03:38:02.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.254 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.254 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.411366+0000", 2024-09-16T03:38:02.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002605324 2024-09-16T03:38:02.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.254 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.254 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.411403+0000", 2024-09-16T03:38:02.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7011000000000002e-05 2024-09-16T03:38:02.254 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.254 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1580 4.f e61/32 4:fa163101:::smithi15736696-10:head v 61'52, mlcod=58'50)", 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.068129046999999, 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.201787367, 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.255 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:02.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.256 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.256 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.256 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.256 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:02.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.256 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.256 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.256 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.256 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:02.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2600000000000001e-07 2024-09-16T03:38:02.256 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.256 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106083+0000", 2024-09-16T03:38:02.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00046163900000000002 2024-09-16T03:38:02.257 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106093+0000", 2024-09-16T03:38:02.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9320000000000008e-06 2024-09-16T03:38:02.257 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206190+0000", 2024-09-16T03:38:02.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100096716 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206286+0000", 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5982000000000004e-05 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206312+0000", 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6279999999999999e-05 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.258 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307345+0000", 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10103240199999999 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307409+0000", 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4189999999999994e-05 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1581 4.f e61/32 4:fa163101:::smithi15736696-10:head v 61'53, mlcod=58'50)", 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:47.206244+0000", 2024-09-16T03:38:02.259 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.967506516, 2024-09-16T03:38:02.260 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101636226, 2024-09-16T03:38:02.260 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.260 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.260 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.260 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.260 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.260 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.260 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206244+0000", 2024-09-16T03:38:02.260 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.260 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.260 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.260 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.260 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206244+0000", 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206246+0000", 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7549999999999999e-06 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206367+0000", 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012166 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206375+0000", 2024-09-16T03:38:02.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.571e-06 2024-09-16T03:38:02.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206380+0000", 2024-09-16T03:38:02.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5210000000000004e-06 2024-09-16T03:38:02.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.306514+0000", 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10013456 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.306531+0000", 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.702e-05 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307860+0000", 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013284080000000001 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.263 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.264 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307880+0000", 2024-09-16T03:38:02.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0730999999999999e-05 2024-09-16T03:38:02.264 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.264 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.264 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.264 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.264 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.264 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1582 4.f e61/32 4:fa163101:::smithi15736696-10:head v 61'54, mlcod=58'50)", 2024-09-16T03:38:02.264 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:47.210745+0000", 2024-09-16T03:38:02.264 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.963005326999999, 2024-09-16T03:38:02.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101062938, 2024-09-16T03:38:02.264 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210745+0000", 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210745+0000", 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.266 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210747+0000", 2024-09-16T03:38:02.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.494e-06 2024-09-16T03:38:02.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.266 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210754+0000", 2024-09-16T03:38:02.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1069999999999999e-06 2024-09-16T03:38:02.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.266 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210767+0000", 2024-09-16T03:38:02.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3054000000000001e-05 2024-09-16T03:38:02.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210774+0000", 2024-09-16T03:38:02.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4959999999999999e-06 2024-09-16T03:38:02.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.310881+0000", 2024-09-16T03:38:02.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100106371 2024-09-16T03:38:02.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.310900+0000", 2024-09-16T03:38:02.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9666000000000001e-05 2024-09-16T03:38:02.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.311778+0000", 2024-09-16T03:38:02.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00087738400000000004 2024-09-16T03:38:02.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.311808+0000", 2024-09-16T03:38:02.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0366e-05 2024-09-16T03:38:02.268 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.269 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.269 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.269 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.269 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1760 4.f e71/32 4:f101f67d:::smithi15736696-22:head v 71'55, mlcod=61'53)", 2024-09-16T03:38:02.269 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:02.269 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.7929250809999999, 2024-09-16T03:38:02.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20118894600000001, 2024-09-16T03:38:02.269 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:02.269 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:02.269 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:02.269 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:02.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380826+0000", 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1899999999999998e-07 2024-09-16T03:38:02.270 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.271 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:02.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380831+0000", 2024-09-16T03:38:02.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6049999999999997e-06 2024-09-16T03:38:02.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.271 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:02.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380851+0000", 2024-09-16T03:38:02.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0021e-05 2024-09-16T03:38:02.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.271 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:02.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380862+0000", 2024-09-16T03:38:02.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0458e-05 2024-09-16T03:38:02.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:02.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581184+0000", 2024-09-16T03:38:02.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20032164299999999 2024-09-16T03:38:02.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:02.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581202+0000", 2024-09-16T03:38:02.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8410000000000002e-05 2024-09-16T03:38:02.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581965+0000", 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076291200000000003 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.582014+0000", 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9577999999999999e-05 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:02.273 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:02.274 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:02.275 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:02.275 INFO:tasks.rados.rados.0.smithi157.stdout:1003: finishing write tid 1 to smithi15736696-30 2024-09-16T03:38:02.275 INFO:tasks.rados.rados.0.smithi157.stdout:1003: finishing write tid 2 to smithi15736696-30 2024-09-16T03:38:02.275 INFO:tasks.rados.rados.0.smithi157.stdout:1009: expect (ObjNum 161 snap 0 seq_num 161) 2024-09-16T03:38:02.276 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:02.276 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:02.276 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:02.277 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:02.277 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:38:02.277 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:02.278 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:02.278 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:02.278 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:02.279 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:02.273+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:02.337 INFO:tasks.rados.rados.0.smithi157.stdout:1007: expect (ObjNum 165 snap 0 seq_num 165) 2024-09-16T03:38:02.349 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:02.349 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:02.350 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1090531 2024-09-16T03:38:02.366 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:02.366 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:02.375 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:02.374+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:02.381 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:02.382 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:02.384 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:02.384 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:02.393 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:38:02.399 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:02.399 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:02.416 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:02.416 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:02.433 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:02.433 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:02.449 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:02.449 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:02.456 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1141461 2024-09-16T03:38:02.476 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:02.475+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:02.497 INFO:tasks.rados.rados.0.smithi157.stdout:1003: finishing write tid 3 to smithi15736696-30 2024-09-16T03:38:02.497 INFO:tasks.rados.rados.0.smithi157.stdout:1003: finishing write tid 4 to smithi15736696-30 2024-09-16T03:38:02.497 INFO:tasks.rados.rados.0.smithi157.stdout:1003: oid 30 updating version 0 to 34 2024-09-16T03:38:02.497 INFO:tasks.rados.rados.0.smithi157.stdout:1003: oid 30 version 34 is already newer than 33 2024-09-16T03:38:02.497 INFO:tasks.rados.rados.0.smithi157.stdout:1003: oid 30 version 34 is already newer than 32 2024-09-16T03:38:02.497 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 34 (ObjNum 166 snap 0 seq_num 166) dirty exists 2024-09-16T03:38:02.497 INFO:tasks.rados.rados.0.smithi157.stdout:1003: left oid 30 (ObjNum 166 snap 0 seq_num 166) 2024-09-16T03:38:02.497 INFO:tasks.rados.rados.0.smithi157.stdout:1003: done (4 left) 2024-09-16T03:38:02.498 INFO:tasks.rados.rados.0.smithi157.stdout:1007: done (3 left) 2024-09-16T03:38:02.498 INFO:tasks.rados.rados.0.smithi157.stdout:1008: done (2 left) 2024-09-16T03:38:02.498 INFO:tasks.rados.rados.0.smithi157.stdout:1009: done (1 left) 2024-09-16T03:38:02.498 INFO:tasks.rados.rados.0.smithi157.stdout:1010: done (0 left) 2024-09-16T03:38:02.498 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:02.498 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:02.499 INFO:tasks.rados.rados.0.smithi157.stdout:1011: write (excl) oid 43 current snap is 0 2024-09-16T03:38:02.499 INFO:tasks.rados.rados.0.smithi157.stdout:1011: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-09-16T03:38:02.501 INFO:tasks.rados.rados.0.smithi157.stdout:1011: writing smithi15736696-43 from 440383 to 917230 tid 1 2024-09-16T03:38:02.502 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:02.502 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:02.505 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:38:02.508 INFO:tasks.rados.rados.0.smithi157.stdout:1011: writing smithi15736696-43 from 1556398 to 1993781 tid 2 2024-09-16T03:38:02.512 INFO:tasks.rados.rados.0.smithi157.stdout:1011: writing smithi15736696-43 from 2474721 to 2840383 tid 3 2024-09-16T03:38:02.512 INFO:tasks.rados.rados.0.smithi157.stdout:1012: delete oid 7 current snap is 0 2024-09-16T03:38:02.514 INFO:tasks.rados.rados.0.smithi157.stdout:1012: done (1 left) 2024-09-16T03:38:02.514 INFO:tasks.rados.rados.0.smithi157.stdout:1013: read oid 35 snap -1 2024-09-16T03:38:02.514 INFO:tasks.rados.rados.0.smithi157.stdout:1013: expect (ObjNum 161 snap 0 seq_num 161) 2024-09-16T03:38:02.514 INFO:tasks.rados.rados.0.smithi157.stdout:1014: read oid 20 snap -1 2024-09-16T03:38:02.515 INFO:tasks.rados.rados.0.smithi157.stdout:1014: expect (ObjNum 132 snap 0 seq_num 132) 2024-09-16T03:38:02.515 INFO:tasks.rados.rados.0.smithi157.stdout:1015: read oid 13 snap -1 2024-09-16T03:38:02.515 INFO:tasks.rados.rados.0.smithi157.stdout:1015: expect (ObjNum 145 snap 0 seq_num 145) 2024-09-16T03:38:02.515 INFO:tasks.rados.rados.0.smithi157.stdout:1016: read oid 34 snap -1 2024-09-16T03:38:02.515 INFO:tasks.rados.rados.0.smithi157.stdout:1016: expect (ObjNum 126 snap 0 seq_num 126) 2024-09-16T03:38:02.515 INFO:tasks.rados.rados.0.smithi157.stdout:1017: read oid 50 snap -1 2024-09-16T03:38:02.515 INFO:tasks.rados.rados.0.smithi157.stdout:1017: expect (ObjNum 104 snap 0 seq_num 104) 2024-09-16T03:38:02.515 INFO:tasks.rados.rados.0.smithi157.stdout:1018: delete oid 3 current snap is 0 2024-09-16T03:38:02.519 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:02.519 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:02.523 INFO:tasks.rados.rados.0.smithi157.stdout:1017: expect (ObjNum 104 snap 0 seq_num 104) 2024-09-16T03:38:02.577 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:02.576+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:02.614 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1355887 2024-09-16T03:38:02.614 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:02.615 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:02.624 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:38:02.678 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:02.677+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:02.727 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:02.727 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:02.739 INFO:tasks.rados.rados.0.smithi157.stdout:1014: expect (ObjNum 132 snap 0 seq_num 132) 2024-09-16T03:38:02.746 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 746060 2024-09-16T03:38:02.770 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:02.771 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:02.781 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:38:02.782 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:02.779+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:02.803 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:02.838 INFO:tasks.rados.rados.0.smithi157.stdout:1015: expect (ObjNum 145 snap 0 seq_num 145) 2024-09-16T03:38:02.838 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:02.882 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:02.879+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:02.886 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:02.886 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:02.895 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:38:02.982 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:02.981+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:02.998 INFO:tasks.rados.rados.0.smithi157.stdout:1016: expect (ObjNum 126 snap 0 seq_num 126) 2024-09-16T03:38:03.036 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:03.039 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:03.039 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:03.048 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:38:03.060 INFO:tasks.rados.rados.0.smithi157.stdout:1011: finishing write tid 1 to smithi15736696-43 2024-09-16T03:38:03.061 INFO:tasks.rados.rados.0.smithi157.stdout:1013: expect (ObjNum 161 snap 0 seq_num 161) 2024-09-16T03:38:03.083 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:03.082+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:03.152 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:03.152 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:03.161 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:38:03.183 INFO:tasks.rados.rados.0.smithi157.stdout:1011: finishing write tid 2 to smithi15736696-43 2024-09-16T03:38:03.184 INFO:tasks.rados.rados.0.smithi157.stdout:1013: done (6 left) 2024-09-16T03:38:03.184 INFO:tasks.rados.rados.0.smithi157.stdout:1014: done (5 left) 2024-09-16T03:38:03.184 INFO:tasks.rados.rados.0.smithi157.stdout:1015: done (4 left) 2024-09-16T03:38:03.184 INFO:tasks.rados.rados.0.smithi157.stdout:1016: done (3 left) 2024-09-16T03:38:03.184 INFO:tasks.rados.rados.0.smithi157.stdout:1017: done (2 left) 2024-09-16T03:38:03.184 INFO:tasks.rados.rados.0.smithi157.stdout:1018: done (1 left) 2024-09-16T03:38:03.184 INFO:tasks.rados.rados.0.smithi157.stdout:1019: tier_flush oid 48 2024-09-16T03:38:03.184 INFO:tasks.rados.rados.0.smithi157.stdout:1019: tier_flush oid 48 snap -1 2024-09-16T03:38:03.184 INFO:tasks.rados.rados.0.smithi157.stdout:1011: finishing write tid 3 to smithi15736696-43 2024-09-16T03:38:03.184 INFO:tasks.rados.rados.0.smithi157.stdout:1011: finishing write tid 4 to smithi15736696-43 2024-09-16T03:38:03.184 INFO:tasks.rados.rados.0.smithi157.stdout:1011: finishing write tid 5 to smithi15736696-43 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:1011: oid 43 updating version 0 to 54 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:1011: oid 43 version 54 is already newer than 51 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:1011: oid 43 version 54 is already newer than 53 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:1011: oid 43 version 54 is already newer than 52 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 54 (ObjNum 169 snap 0 seq_num 169) dirty exists 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:1011: left oid 43 (ObjNum 169 snap 0 seq_num 169) 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:1011: done (1 left) 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:1020: read oid 44 snap -1 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:1020: expect (ObjNum 63 snap 0 seq_num 63) 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:1021: tier_evict oid 38 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:1021: tier_evict oid 38 snap -1 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:1022: read oid 11 snap -1 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:1022: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:38:03.185 INFO:tasks.rados.rados.0.smithi157.stdout:1023: read oid 12 snap -1 2024-09-16T03:38:03.186 INFO:tasks.rados.rados.0.smithi157.stdout:1023: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:03.186 INFO:tasks.rados.rados.0.smithi157.stdout:1024: read oid 50 snap -1 2024-09-16T03:38:03.186 INFO:tasks.rados.rados.0.smithi157.stdout:1024: expect (ObjNum 104 snap 0 seq_num 104) 2024-09-16T03:38:03.186 INFO:tasks.rados.rados.0.smithi157.stdout:1025: read oid 47 snap -1 2024-09-16T03:38:03.186 INFO:tasks.rados.rados.0.smithi157.stdout:1025: expect deleted 2024-09-16T03:38:03.186 INFO:tasks.rados.rados.0.smithi157.stdout:1026: delete oid 40 current snap is 0 2024-09-16T03:38:03.186 INFO:tasks.rados.rados.0.smithi157.stdout:1022: expect (ObjNum 52 snap 0 seq_num 52) 2024-09-16T03:38:03.187 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:03.186+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:03.288 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:03.288 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:03.288 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:03.286+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:03.297 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:38:03.297 INFO:tasks.rados.rados.0.smithi157.stdout:1021: got (22) Invalid argument 2024-09-16T03:38:03.298 INFO:tasks.rados.rados.0.smithi157.stdout:1020: expect (ObjNum 63 snap 0 seq_num 63) 2024-09-16T03:38:03.298 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:03.389 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:03.387+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:03.401 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:03.401 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:03.407 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:03.410 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:38:03.471 INFO:tasks.rados.rados.0.smithi157.stdout:1019: got (2) No such file or directory 2024-09-16T03:38:03.471 INFO:tasks.rados.rados.0.smithi157.stdout:1019: got expected ENOENT (src dne) 2024-09-16T03:38:03.471 INFO:tasks.rados.rados.0.smithi157.stdout:1023: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:03.471 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 17538 2024-09-16T03:38:03.489 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:03.488+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:03.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:38:03.515 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:03.515 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:03.516 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:03.523 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:38:03.562 INFO:tasks.rados.rados.0.smithi157.stdout:1024: expect (ObjNum 104 snap 0 seq_num 104) 2024-09-16T03:38:03.590 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:03.587+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:03.628 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:03.629 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:03.637 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:38:03.640 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:03.690 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:03.689+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:03.744 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:03.744 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:03.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:38:03.792 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:03.790+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:03.796 INFO:tasks.rados.rados.0.smithi157.stdout:1019: done (7 left) 2024-09-16T03:38:03.796 INFO:tasks.rados.rados.0.smithi157.stdout:1020: done (6 left) 2024-09-16T03:38:03.796 INFO:tasks.rados.rados.0.smithi157.stdout:1021: done (5 left) 2024-09-16T03:38:03.796 INFO:tasks.rados.rados.0.smithi157.stdout:1022: done (4 left) 2024-09-16T03:38:03.796 INFO:tasks.rados.rados.0.smithi157.stdout:1023: done (3 left) 2024-09-16T03:38:03.796 INFO:tasks.rados.rados.0.smithi157.stdout:1024: done (2 left) 2024-09-16T03:38:03.796 INFO:tasks.rados.rados.0.smithi157.stdout:1025: done (1 left) 2024-09-16T03:38:03.796 INFO:tasks.rados.rados.0.smithi157.stdout:1026: done (0 left) 2024-09-16T03:38:03.796 INFO:tasks.rados.rados.0.smithi157.stdout:1027: write (excl) oid 44 current snap is 0 2024-09-16T03:38:03.796 INFO:tasks.rados.rados.0.smithi157.stdout:1027: seq_num 173 ranges {507611=777843,2073973=444721} 2024-09-16T03:38:03.802 INFO:tasks.rados.rados.0.smithi157.stdout:1027: writing smithi15736696-44 from 507611 to 1285454 tid 1 2024-09-16T03:38:03.809 INFO:tasks.rados.rados.0.smithi157.stdout:1027: writing smithi15736696-44 from 2073973 to 2518694 tid 2 2024-09-16T03:38:03.809 INFO:tasks.rados.rados.0.smithi157.stdout:1028: write oid 20 current snap is 0 2024-09-16T03:38:03.809 INFO:tasks.rados.rados.0.smithi157.stdout:1028: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2024-09-16T03:38:03.814 INFO:tasks.rados.rados.0.smithi157.stdout:1028: writing smithi15736696-20 from 524418 to 977510 tid 1 2024-09-16T03:38:03.821 INFO:tasks.rados.rados.0.smithi157.stdout:1028: writing smithi15736696-20 from 1732455 to 2320834 tid 2 2024-09-16T03:38:03.825 INFO:tasks.rados.rados.0.smithi157.stdout:1028: writing smithi15736696-20 from 2779130 to 2924418 tid 3 2024-09-16T03:38:03.825 INFO:tasks.rados.rados.0.smithi157.stdout:1029: set_chunk oid 40 target oid 2024-09-16T03:38:03.825 INFO:tasks.rados.rados.0.smithi157.stdout:1029: set_chunk oid 40 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:03.825 INFO:tasks.rados.rados.0.smithi157.stdout:1030: read oid 6 snap -1 2024-09-16T03:38:03.825 INFO:tasks.rados.rados.0.smithi157.stdout:1030: expect (ObjNum 164 snap 0 seq_num 164) 2024-09-16T03:38:03.825 INFO:tasks.rados.rados.0.smithi157.stdout:1031: tier_evict oid 18 2024-09-16T03:38:03.826 INFO:tasks.rados.rados.0.smithi157.stdout:1031: tier_evict oid 18 snap -1 2024-09-16T03:38:03.826 INFO:tasks.rados.rados.0.smithi157.stdout:1032: read oid 3 snap -1 2024-09-16T03:38:03.826 INFO:tasks.rados.rados.0.smithi157.stdout:1032: expect deleted 2024-09-16T03:38:03.826 INFO:tasks.rados.rados.0.smithi157.stdout:1033: tier_promote oid 43 2024-09-16T03:38:03.826 INFO:tasks.rados.rados.0.smithi157.stdout:1034: read oid 31 snap -1 2024-09-16T03:38:03.826 INFO:tasks.rados.rados.0.smithi157.stdout:1034: expect deleted 2024-09-16T03:38:03.826 INFO:tasks.rados.rados.0.smithi157.stdout:1035: tier_flush oid 9 2024-09-16T03:38:03.826 INFO:tasks.rados.rados.0.smithi157.stdout:1035: tier_flush oid 9 snap -1 2024-09-16T03:38:03.826 INFO:tasks.rados.rados.0.smithi157.stdout:1036: read oid 5 snap -1 2024-09-16T03:38:03.826 INFO:tasks.rados.rados.0.smithi157.stdout:1036: expect (ObjNum 165 snap 0 seq_num 165) 2024-09-16T03:38:03.826 INFO:tasks.rados.rados.0.smithi157.stdout:1037: read oid 14 snap -1 2024-09-16T03:38:03.826 INFO:tasks.rados.rados.0.smithi157.stdout:1037: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-16T03:38:03.826 INFO:tasks.rados.rados.0.smithi157.stdout:1038: write oid 48 current snap is 0 2024-09-16T03:38:03.827 INFO:tasks.rados.rados.0.smithi157.stdout:1038: seq_num 175 ranges {541225=644694,1907290=448390} 2024-09-16T03:38:03.831 INFO:tasks.rados.rados.0.smithi157.stdout:1038: writing smithi15736696-48 from 541225 to 1185919 tid 1 2024-09-16T03:38:03.838 INFO:tasks.rados.rados.0.smithi157.stdout:1038: writing smithi15736696-48 from 1907290 to 2355680 tid 2 2024-09-16T03:38:03.838 INFO:tasks.rados.rados.0.smithi157.stdout:1039: write oid 36 current snap is 0 2024-09-16T03:38:03.838 INFO:tasks.rados.rados.0.smithi157.stdout:1031: got (22) Invalid argument 2024-09-16T03:38:03.838 INFO:tasks.rados.rados.0.smithi157.stdout:1033: got (0) Success 2024-09-16T03:38:03.838 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 54 (ObjNum 169 snap 0 seq_num 169) dirty exists 2024-09-16T03:38:03.838 INFO:tasks.rados.rados.0.smithi157.stdout:1037: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-16T03:38:03.862 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:03.862 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:03.870 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:38:03.893 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:03.892+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:03.932 INFO:tasks.rados.rados.0.smithi157.stdout:1036: expect (ObjNum 165 snap 0 seq_num 165) 2024-09-16T03:38:03.981 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:03.981 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:03.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:38:04.034 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:04.032+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:04.094 INFO:tasks.rados.rados.0.smithi157.stdout:1029: finishing set_chunk to oid 40 2024-09-16T03:38:04.094 INFO:tasks.rados.rados.0.smithi157.stdout:1029: got expected ENOENT (src dne) 2024-09-16T03:38:04.094 INFO:tasks.rados.rados.0.smithi157.stdout:1030: expect (ObjNum 164 snap 0 seq_num 164) 2024-09-16T03:38:04.094 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:04.093+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:04.099 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:04.099 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:04.108 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:38:04.195 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:04.193+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:04.226 INFO:tasks.rados.rados.0.smithi157.stdout:1027: finishing write tid 1 to smithi15736696-44 2024-09-16T03:38:04.226 INFO:tasks.rados.rados.0.smithi157.stdout:1028: finishing write tid 1 to smithi15736696-20 2024-09-16T03:38:04.226 INFO:tasks.rados.rados.0.smithi157.stdout:1039: seq_num 176 ranges {558032=719943,1849419=592048} 2024-09-16T03:38:04.233 INFO:tasks.rados.rados.0.smithi157.stdout:1039: writing smithi15736696-36 from 558032 to 1277975 tid 1 2024-09-16T03:38:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1039: writing smithi15736696-36 from 1849419 to 2441467 tid 2 2024-09-16T03:38:04.240 INFO:tasks.rados.rados.0.smithi157.stdout:1029: done (12 left) 2024-09-16T03:38:04.240 INFO:tasks.rados.rados.0.smithi157.stdout:1030: done (11 left) 2024-09-16T03:38:04.240 INFO:tasks.rados.rados.0.smithi157.stdout:1031: done (10 left) 2024-09-16T03:38:04.240 INFO:tasks.rados.rados.0.smithi157.stdout:1032: done (9 left) 2024-09-16T03:38:04.240 INFO:tasks.rados.rados.0.smithi157.stdout:1033: done (8 left) 2024-09-16T03:38:04.240 INFO:tasks.rados.rados.0.smithi157.stdout:1034: done (7 left) 2024-09-16T03:38:04.240 INFO:tasks.rados.rados.0.smithi157.stdout:1036: done (6 left) 2024-09-16T03:38:04.240 INFO:tasks.rados.rados.0.smithi157.stdout:1037: done (5 left) 2024-09-16T03:38:04.240 INFO:tasks.rados.rados.0.smithi157.stdout:1040: write (excl) oid 19 current snap is 0 2024-09-16T03:38:04.240 INFO:tasks.rados.rados.0.smithi157.stdout:1040: seq_num 177 ranges {574839=795192,1907901=452059} 2024-09-16T03:38:04.246 INFO:tasks.rados.rados.0.smithi157.stdout:1040: writing smithi15736696-19 from 574839 to 1370031 tid 1 2024-09-16T03:38:04.252 INFO:tasks.rados.rados.0.smithi157.stdout:1040: writing smithi15736696-19 from 1907901 to 2359960 tid 2 2024-09-16T03:38:04.252 INFO:tasks.rados.rados.0.smithi157.stdout:1041: tier_promote oid 13 2024-09-16T03:38:04.252 INFO:tasks.rados.rados.0.smithi157.stdout:1042: read oid 42 snap -1 2024-09-16T03:38:04.253 INFO:tasks.rados.rados.0.smithi157.stdout:1042: expect (ObjNum 142 snap 0 seq_num 142) 2024-09-16T03:38:04.253 INFO:tasks.rados.rados.0.smithi157.stdout:1043: read oid 46 snap -1 2024-09-16T03:38:04.253 INFO:tasks.rados.rados.0.smithi157.stdout:1043: expect (ObjNum 163 snap 0 seq_num 163) 2024-09-16T03:38:04.253 INFO:tasks.rados.rados.0.smithi157.stdout:1044: read oid 39 snap -1 2024-09-16T03:38:04.253 INFO:tasks.rados.rados.0.smithi157.stdout:1044: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-16T03:38:04.253 INFO:tasks.rados.rados.0.smithi157.stdout:1045: read oid 49 snap -1 2024-09-16T03:38:04.253 INFO:tasks.rados.rados.0.smithi157.stdout:1045: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-16T03:38:04.253 INFO:tasks.rados.rados.0.smithi157.stdout:1046: set_chunk oid 34 target oid 2024-09-16T03:38:04.259 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:04.259 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:04.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:38:04.296 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:04.294+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:04.397 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:04.396+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:04.418 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:04.418 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:04.427 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:38:04.498 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:04.497+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:04.537 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:04.537 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:04.546 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:38:04.599 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:04.597+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:04.653 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:04.654 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:04.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:38:04.700 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:04.699+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:04.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:38:04.747 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:04.746+0000 7f3a21722640 -1 osd.6 75 handle_osd_map injecting map crc failure 2024-09-16T03:38:04.750 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:04.748+0000 7f12e4a1e640 -1 osd.5 75 handle_osd_map injecting map crc failure 2024-09-16T03:38:04.751 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:38:04.801 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:04.799+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:04.814 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:04.814 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:04.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:38:04.874 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:04.874 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:04.892 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:04.892 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:04.903 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:04.901+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:04.910 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:04.910 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:04.927 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:04.927 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:04.932 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:04.932 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:04.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:38:04.944 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:04.944 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:04.963 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:04.964 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:04.981 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:04.981 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:04.998 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:04.998 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:05.003 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:05.002+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:05.014 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:05.014 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:05.031 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:05.032 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:05.033 INFO:tasks.rados.rados.0.smithi157.stdout:1046: set_chunk oid 34 offset: 350208 length: 1056256 target oid 483c6c0fa4e07a5bc50ecabc5f1110026b894ed62b3444194db3b3716fde6e42 offset: 0 snap -1 2024-09-16T03:38:05.033 INFO:tasks.rados.rados.0.smithi157.stdout:1047: set_chunk oid 21 target oid 2024-09-16T03:38:05.033 INFO:tasks.rados.rados.0.smithi157.stdout:1047: set_chunk oid 21 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:05.033 INFO:tasks.rados.rados.0.smithi157.stdout:1048: write (excl) oid 37 current snap is 0 2024-09-16T03:38:05.033 INFO:tasks.rados.rados.0.smithi157.stdout:1039: finishing write tid 1 to smithi15736696-36 2024-09-16T03:38:05.034 INFO:tasks.rados.rados.0.smithi157.stdout:1048: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-09-16T03:38:05.041 INFO:tasks.rados.rados.0.smithi157.stdout:1048: writing smithi15736696-37 from 591646 to 1062087 tid 1 2024-09-16T03:38:05.044 INFO:tasks.rados.rados.0.smithi157.stdout:1048: writing smithi15736696-37 from 1566383 to 2162100 tid 2 2024-09-16T03:38:05.048 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:05.049 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:05.050 INFO:tasks.rados.rados.0.smithi157.stdout:1048: writing smithi15736696-37 from 2888822 to 2991646 tid 3 2024-09-16T03:38:05.050 INFO:tasks.rados.rados.0.smithi157.stdout:1049: read oid 18 snap -1 2024-09-16T03:38:05.050 INFO:tasks.rados.rados.0.smithi157.stdout:1049: expect (ObjNum 159 snap 0 seq_num 159) 2024-09-16T03:38:05.050 INFO:tasks.rados.rados.0.smithi157.stdout:1050: read oid 4 snap -1 2024-09-16T03:38:05.050 INFO:tasks.rados.rados.0.smithi157.stdout:1050: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:05.050 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:05.050 INFO:tasks.rados.rados.0.smithi157.stdout:1045: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-16T03:38:05.050 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 351141 2024-09-16T03:38:05.051 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:05.051 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:05.059 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:38:05.065 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:05.065 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:05.082 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:05.082 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:05.099 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:05.099 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:05.104 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:05.102+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:05.116 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:05.116 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:05.132 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:05.133 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:05.139 INFO:tasks.rados.rados.0.smithi157.stdout:1028: finishing write tid 2 to smithi15736696-20 2024-09-16T03:38:05.139 INFO:tasks.rados.rados.0.smithi157.stdout:1038: finishing write tid 1 to smithi15736696-48 2024-09-16T03:38:05.139 INFO:tasks.rados.rados.0.smithi157.stdout:1028: finishing write tid 3 to smithi15736696-20 2024-09-16T03:38:05.139 INFO:tasks.rados.rados.0.smithi157.stdout:1027: finishing write tid 2 to smithi15736696-44 2024-09-16T03:38:05.139 INFO:tasks.rados.rados.0.smithi157.stdout:1038: finishing write tid 2 to smithi15736696-48 2024-09-16T03:38:05.139 INFO:tasks.rados.rados.0.smithi157.stdout:1045: done (15 left) 2024-09-16T03:38:05.139 INFO:tasks.rados.rados.0.smithi157.stdout:1051: set_chunk oid 25 target oid 2024-09-16T03:38:05.139 INFO:tasks.rados.rados.0.smithi157.stdout:1051: set_chunk oid 25 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:05.139 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:05.140 INFO:tasks.rados.rados.0.smithi157.stdout:1043: expect (ObjNum 163 snap 0 seq_num 163) 2024-09-16T03:38:05.140 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:05.173 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:38:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:38:05.173 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:05.182 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:38:05.205 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:05.204+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:05.256 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1041781 2024-09-16T03:38:05.293 INFO:tasks.rados.rados.0.smithi157.stdout:1028: finishing write tid 4 to smithi15736696-20 2024-09-16T03:38:05.293 INFO:tasks.rados.rados.0.smithi157.stdout:1028: finishing write tid 5 to smithi15736696-20 2024-09-16T03:38:05.293 INFO:tasks.rados.rados.0.smithi157.stdout:1028: oid 20 updating version 0 to 36 2024-09-16T03:38:05.294 INFO:tasks.rados.rados.0.smithi157.stdout:1028: oid 20 updating version 36 to 37 2024-09-16T03:38:05.294 INFO:tasks.rados.rados.0.smithi157.stdout:1028: oid 20 updating version 37 to 38 2024-09-16T03:38:05.294 INFO:tasks.rados.rados.0.smithi157.stdout:1028: oid 20 updating version 38 to 39 2024-09-16T03:38:05.294 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 39 (ObjNum 174 snap 0 seq_num 174) dirty exists 2024-09-16T03:38:05.294 INFO:tasks.rados.rados.0.smithi157.stdout:1028: left oid 20 (ObjNum 174 snap 0 seq_num 174) 2024-09-16T03:38:05.294 INFO:tasks.rados.rados.0.smithi157.stdout:1038: finishing write tid 3 to smithi15736696-48 2024-09-16T03:38:05.294 INFO:tasks.rados.rados.0.smithi157.stdout:1044: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-16T03:38:05.294 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:05.305 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:05.303+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:05.332 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:38:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:38:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:05.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:57 4.d 4:b3fe05b2:::smithi15736696-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:38:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.205356905999999, 2024-09-16T03:38:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.205477148, 2024-09-16T03:38:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 57 2024-09-16T03:38:05.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124125+0000", 2024-09-16T03:38:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.604e-06 2024-09-16T03:38:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124908+0000", 2024-09-16T03:38:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000783086 2024-09-16T03:38:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124919+0000", 2024-09-16T03:38:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0591000000000001e-05 2024-09-16T03:38:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124925+0000", 2024-09-16T03:38:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.55e-06 2024-09-16T03:38:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124985+0000", 2024-09-16T03:38:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9817e-05 2024-09-16T03:38:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.125113+0000", 2024-09-16T03:38:05.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127429 2024-09-16T03:38:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:38:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.125229+0000", 2024-09-16T03:38:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116784 2024-09-16T03:38:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.127169+0000", 2024-09-16T03:38:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001939187 2024-09-16T03:38:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128052+0000", 2024-09-16T03:38:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088291800000000001 2024-09-16T03:38:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329495+0000", 2024-09-16T03:38:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201443328 2024-09-16T03:38:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329535+0000", 2024-09-16T03:38:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9823999999999997e-05 2024-09-16T03:38:05.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329601+0000", 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000003e-05 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:60 4.d 4:b3fe05b2:::smithi15736696-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.201094202, 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30164545100000001, 2024-09-16T03:38:05.339 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 60 2024-09-16T03:38:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:38:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:38:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128387+0000", 2024-09-16T03:38:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3599999999999999e-07 2024-09-16T03:38:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128388+0000", 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7659999999999999e-06 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128394+0000", 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0679999999999996e-06 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128398+0000", 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4220000000000002e-06 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129762+0000", 2024-09-16T03:38:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001364285 2024-09-16T03:38:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129804+0000", 2024-09-16T03:38:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2058000000000003e-05 2024-09-16T03:38:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.429903+0000", 2024-09-16T03:38:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30009841900000001 2024-09-16T03:38:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.429952+0000", 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9066000000000001e-05 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.430032+0000", 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9931000000000004e-05 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.344 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:81 4.c 4:3313e97c:::smithi15736696-21:head [write 736140~537686,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.931854178000002, 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203525015, 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397629+0000", 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1889999999999999e-06 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398085+0000", 2024-09-16T03:38:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00045514200000000001 2024-09-16T03:38:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398100+0000", 2024-09-16T03:38:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5637999999999998e-05 2024-09-16T03:38:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398110+0000", 2024-09-16T03:38:05.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6549999999999999e-06 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398190+0000", 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0272999999999995e-05 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398339+0000", 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014924200000000001 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:05.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398504+0000", 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016471500000000001 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.400267+0000", 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017625169999999999 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.401104+0000", 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083686399999999999 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601084+0000", 2024-09-16T03:38:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199980397 2024-09-16T03:38:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601114+0000", 2024-09-16T03:38:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0484000000000002e-05 2024-09-16T03:38:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601151+0000", 2024-09-16T03:38:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6899e-05 2024-09-16T03:38:05.350 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:83 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 2336140] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.925659383000003, 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29747773399999999, 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.351 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-09-16T03:38:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:38:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:38:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403822+0000", 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-07 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403825+0000", 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000001e-06 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403831+0000", 2024-09-16T03:38:05.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9399999999999999e-06 2024-09-16T03:38:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403834+0000", 2024-09-16T03:38:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1939999999999998e-06 2024-09-16T03:38:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.404994+0000", 2024-09-16T03:38:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011601249999999999 2024-09-16T03:38:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405083+0000", 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8809999999999998e-05 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405217+0000", 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013401900000000001 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.503958+0000", 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098741342999999995 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.506033+0000", 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020749660000000001 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701227+0000", 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.195194171 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701262+0000", 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4768999999999999e-05 2024-09-16T03:38:05.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701299+0000", 2024-09-16T03:38:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6884000000000001e-05 2024-09-16T03:38:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:84 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:38:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.925627658000003, 2024-09-16T03:38:05.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29760038700000002, 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:38:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:38:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.58e-07 2024-09-16T03:38:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403861+0000", 2024-09-16T03:38:05.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7689999999999996e-06 2024-09-16T03:38:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403866+0000", 2024-09-16T03:38:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6330000000000004e-06 2024-09-16T03:38:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403869+0000", 2024-09-16T03:38:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7390000000000001e-06 2024-09-16T03:38:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405790+0000", 2024-09-16T03:38:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001920351 2024-09-16T03:38:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405865+0000", 2024-09-16T03:38:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5234999999999994e-05 2024-09-16T03:38:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701328+0000", 2024-09-16T03:38:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29546253300000003 2024-09-16T03:38:05.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701382+0000", 2024-09-16T03:38:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4481999999999998e-05 2024-09-16T03:38:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701453+0000", 2024-09-16T03:38:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1186999999999995e-05 2024-09-16T03:38:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:432 4.c 4:34492f38:::smithi15736696-33:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.415648418000004, 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.404460861, 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 432 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.363 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:38:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:38:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913833+0000", 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0610000000000001e-06 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913839+0000", 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9479999999999996e-06 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913848+0000", 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5629999999999992e-06 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913857+0000", 2024-09-16T03:38:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0399999999999998e-06 2024-09-16T03:38:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914022+0000", 2024-09-16T03:38:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000165422 2024-09-16T03:38:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914101+0000", 2024-09-16T03:38:05.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8462999999999997e-05 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217171+0000", 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30307040000000002 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217854+0000", 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068278700000000002 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.218098+0000", 2024-09-16T03:38:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243825 2024-09-16T03:38:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318164+0000", 2024-09-16T03:38:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100066451 2024-09-16T03:38:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318232+0000", 2024-09-16T03:38:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8113000000000003e-05 2024-09-16T03:38:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318293+0000", 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0788000000000001e-05 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:790 4.d 4:b3fe05b2:::smithi15736696-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.441508343999999, 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41456747799999999, 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 790 2024-09-16T03:38:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:38:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887975+0000", 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0369999999999999e-06 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887987+0000", 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1606e-05 2024-09-16T03:38:05.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888016+0000", 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.917e-05 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888032+0000", 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5616000000000001e-05 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888178+0000", 2024-09-16T03:38:05.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612100000000001 2024-09-16T03:38:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888307+0000", 2024-09-16T03:38:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012953899999999999 2024-09-16T03:38:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.302540+0000", 2024-09-16T03:38:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41423238899999998 2024-09-16T03:38:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.373 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1418 4.c 4:3313e97c:::smithi15736696-21:head [write 615156~555834] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.103397172000001, 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40891108799999998, 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1418 2024-09-16T03:38:05.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:38:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:38:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226087+0000", 2024-09-16T03:38:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8890000000000004e-06 2024-09-16T03:38:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226827+0000", 2024-09-16T03:38:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000739298 2024-09-16T03:38:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226848+0000", 2024-09-16T03:38:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1418000000000001e-05 2024-09-16T03:38:05.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226858+0000", 2024-09-16T03:38:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0434e-05 2024-09-16T03:38:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327036+0000", 2024-09-16T03:38:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10017769 2024-09-16T03:38:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327098+0000", 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1760999999999994e-05 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327230+0000", 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131839 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434369+0000", 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107139788 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434444+0000", 2024-09-16T03:38:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4171999999999994e-05 2024-09-16T03:38:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634830+0000", 2024-09-16T03:38:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20038660599999999 2024-09-16T03:38:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634937+0000", 2024-09-16T03:38:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010716499999999999 2024-09-16T03:38:05.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634994+0000", 2024-09-16T03:38:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7027999999999997e-05 2024-09-16T03:38:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1419 4.c 4:3313e97c:::smithi15736696-21:head [write 1603467~416361] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:38:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.098731783000002, 2024-09-16T03:38:05.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40430432100000002, 2024-09-16T03:38:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1419 2024-09-16T03:38:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:38:05.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230752+0000", 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.354e-06 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231164+0000", 2024-09-16T03:38:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041208599999999998 2024-09-16T03:38:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231179+0000", 2024-09-16T03:38:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4885999999999999e-05 2024-09-16T03:38:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231186+0000", 2024-09-16T03:38:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8759999999999999e-06 2024-09-16T03:38:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328726+0000", 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097539855999999994 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328786+0000", 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0059000000000002e-05 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328890+0000", 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104338 2024-09-16T03:38:05.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434475+0000", 2024-09-16T03:38:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.105584709 2024-09-16T03:38:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.534594+0000", 2024-09-16T03:38:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10011914099999999 2024-09-16T03:38:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:05.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635003+0000", 2024-09-16T03:38:05.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100408834 2024-09-16T03:38:05.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:05.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635029+0000", 2024-09-16T03:38:05.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6551999999999999e-05 2024-09-16T03:38:05.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635053+0000", 2024-09-16T03:38:05.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3629999999999999e-05 2024-09-16T03:38:05.386 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.386 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1420 4.c 4:3313e97c:::smithi15736696-21:head [write 2816389~693495] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.090817472000001, 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39653502600000001, 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.387 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1420 2024-09-16T03:38:05.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.388 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:38:05.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:38:05.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238666+0000", 2024-09-16T03:38:05.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.163e-06 2024-09-16T03:38:05.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239395+0000", 2024-09-16T03:38:05.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072868599999999998 2024-09-16T03:38:05.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239410+0000", 2024-09-16T03:38:05.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5330999999999999e-05 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239419+0000", 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7360000000000007e-06 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332338+0000", 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092919165999999997 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332435+0000", 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6972999999999995e-05 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332544+0000", 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010898700000000001 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634808+0000", 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30226407700000002 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:05.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635058+0000", 2024-09-16T03:38:05.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025009300000000001 2024-09-16T03:38:05.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635103+0000", 2024-09-16T03:38:05.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4539000000000002e-05 2024-09-16T03:38:05.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:05.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635145+0000", 2024-09-16T03:38:05.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1938999999999999e-05 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635198+0000", 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3335999999999999e-05 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1421 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.090049022999999, 2024-09-16T03:38:05.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.497492088, 2024-09-16T03:38:05.394 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.394 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.394 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.394 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1421 2024-09-16T03:38:05.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.394 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:38:05.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:38:05.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239432+0000", 2024-09-16T03:38:05.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5999999999999991e-07 2024-09-16T03:38:05.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239436+0000", 2024-09-16T03:38:05.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:38:05.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239443+0000", 2024-09-16T03:38:05.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.968e-06 2024-09-16T03:38:05.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239447+0000", 2024-09-16T03:38:05.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6969999999999999e-06 2024-09-16T03:38:05.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635225+0000", 2024-09-16T03:38:05.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39577807399999998 2024-09-16T03:38:05.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635290+0000", 2024-09-16T03:38:05.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4583000000000002e-05 2024-09-16T03:38:05.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:05.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635410+0000", 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120684 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636180+0000", 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076997899999999995 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636799+0000", 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061906600000000004 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736742+0000", 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099943041999999996 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736843+0000", 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010076799999999999 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736924+0000", 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0419000000000004e-05 2024-09-16T03:38:05.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1422 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.090027743, 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49764818700000002, 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1422 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:05.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:05.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:38:05.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239455+0000", 2024-09-16T03:38:05.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9860000000000001e-06 2024-09-16T03:38:05.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239461+0000", 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239464+0000", 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2619999999999999e-06 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635736+0000", 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.396272287 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:05.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635790+0000", 2024-09-16T03:38:05.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4252000000000002e-05 2024-09-16T03:38:05.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736964+0000", 2024-09-16T03:38:05.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101173319 2024-09-16T03:38:05.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737024+0000", 2024-09-16T03:38:05.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9975000000000001e-05 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737101+0000", 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7303000000000002e-05 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1420 4.c e58/32 ondisk, result = 0)", 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.335674+0000", 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.993806929000002, 2024-09-16T03:38:05.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29938561899999999, 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335674+0000", 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335674+0000", 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335675+0000", 2024-09-16T03:38:05.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5510000000000001e-06 2024-09-16T03:38:05.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335684+0000", 2024-09-16T03:38:05.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6519999999999995e-06 2024-09-16T03:38:05.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335696+0000", 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2662e-05 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335703+0000", 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2079999999999997e-06 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634784+0000", 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29908135299999999 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634797+0000", 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3393999999999999e-05 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635059+0000", 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026179900000000001 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1420 4.c e58/32 ondisk, result = 0)", 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.336136+0000", 2024-09-16T03:38:05.409 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.993344172, 2024-09-16T03:38:05.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29907022, 2024-09-16T03:38:05.410 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.410 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:05.410 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336136+0000", 2024-09-16T03:38:05.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336136+0000", 2024-09-16T03:38:05.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336138+0000", 2024-09-16T03:38:05.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4279999999999999e-06 2024-09-16T03:38:05.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336147+0000", 2024-09-16T03:38:05.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0210000000000005e-06 2024-09-16T03:38:05.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336159+0000", 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2285e-05 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336165+0000", 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7799999999999997e-06 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635091+0000", 2024-09-16T03:38:05.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29892637599999999 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635098+0000", 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5019999999999998e-06 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635207+0000", 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108828 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1537 4.c 4:34492f38:::smithi15736696-33:head [write 1954970~431037] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:38:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.719891+0000", 2024-09-16T03:38:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.609589825, 2024-09-16T03:38:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20396983899999999, 2024-09-16T03:38:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1537 2024-09-16T03:38:05.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.415 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.719891+0000", 2024-09-16T03:38:05.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.719891+0000", 2024-09-16T03:38:05.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.719893+0000", 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.962e-06 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720069+0000", 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176298 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720080+0000", 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1002000000000001e-05 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720088+0000", 2024-09-16T03:38:05.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7440000000000007e-06 2024-09-16T03:38:05.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720132+0000", 2024-09-16T03:38:05.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4471999999999997e-05 2024-09-16T03:38:05.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720196+0000", 2024-09-16T03:38:05.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3946000000000001e-05 2024-09-16T03:38:05.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:05.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720286+0000", 2024-09-16T03:38:05.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9423000000000001e-05 2024-09-16T03:38:05.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:05.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.723040+0000", 2024-09-16T03:38:05.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027547169999999998 2024-09-16T03:38:05.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923715+0000", 2024-09-16T03:38:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20067458299999999 2024-09-16T03:38:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923777+0000", 2024-09-16T03:38:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2135000000000001e-05 2024-09-16T03:38:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923816+0000", 2024-09-16T03:38:05.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8563999999999999e-05 2024-09-16T03:38:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923861+0000", 2024-09-16T03:38:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4993000000000001e-05 2024-09-16T03:38:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1538 4.c 4:34492f38:::smithi15736696-33:head [write 2919420~616377] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:38:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.720103+0000", 2024-09-16T03:38:05.420 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.609377289000001, 2024-09-16T03:38:05.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20383674299999999, 2024-09-16T03:38:05.421 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.421 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.421 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.421 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1538 2024-09-16T03:38:05.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.421 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720103+0000", 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720103+0000", 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720104+0000", 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9400000000000004e-07 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721193+0000", 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001089268 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721203+0000", 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9799999999999993e-06 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721212+0000", 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4430000000000003e-06 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721350+0000", 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013873600000000001 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721407+0000", 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6328e-05 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721501+0000", 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3868000000000003e-05 2024-09-16T03:38:05.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.723066+0000", 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001565461 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923739+0000", 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20067338300000001 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923894+0000", 2024-09-16T03:38:05.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015406600000000001 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923921+0000", 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7440000000000002e-05 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923940+0000", 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8975999999999999e-05 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.427 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1681 4.d 4:b3fe05b2:::smithi15736696-15:head [write 685752~611865,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:05.427 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.148763+0000", 2024-09-16T03:38:05.427 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.180717377000001, 2024-09-16T03:38:05.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20615159899999999, 2024-09-16T03:38:05.427 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.427 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.427 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.427 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.427 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.427 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.427 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1681 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.148763+0000", 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.148763+0000", 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.148768+0000", 2024-09-16T03:38:05.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.527e-06 2024-09-16T03:38:05.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149421+0000", 2024-09-16T03:38:05.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065293199999999995 2024-09-16T03:38:05.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149439+0000", 2024-09-16T03:38:05.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8210999999999999e-05 2024-09-16T03:38:05.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149448+0000", 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2480000000000002e-06 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149507+0000", 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8980000000000001e-05 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149593+0000", 2024-09-16T03:38:05.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6113999999999996e-05 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149730+0000", 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013668 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.151623+0000", 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018936350000000001 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354505+0000", 2024-09-16T03:38:05.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20288149799999999 2024-09-16T03:38:05.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354562+0000", 2024-09-16T03:38:05.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7255e-05 2024-09-16T03:38:05.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:05.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354854+0000", 2024-09-16T03:38:05.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029133000000000002 2024-09-16T03:38:05.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354915+0000", 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1189e-05 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1683 4.d 4:b3fe05b2:::smithi15736696-15:head [setxattr _header (56) in=7b,truncate 2285752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.154316+0000", 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.175164884000001, 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29800856100000001, 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.433 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1683 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154316+0000", 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154316+0000", 2024-09-16T03:38:05.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154316+0000", 2024-09-16T03:38:05.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1500000000000003e-07 2024-09-16T03:38:05.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154320+0000", 2024-09-16T03:38:05.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:38:05.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154332+0000", 2024-09-16T03:38:05.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2254e-05 2024-09-16T03:38:05.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154340+0000", 2024-09-16T03:38:05.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5340000000000002e-06 2024-09-16T03:38:05.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.155080+0000", 2024-09-16T03:38:05.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073960699999999998 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.155143+0000", 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3859e-05 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.155223+0000", 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9331000000000003e-05 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:05.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354955+0000", 2024-09-16T03:38:05.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19973200099999999 2024-09-16T03:38:05.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.355860+0000", 2024-09-16T03:38:05.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090486199999999998 2024-09-16T03:38:05.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.452226+0000", 2024-09-16T03:38:05.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096366118000000001 2024-09-16T03:38:05.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:05.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.452263+0000", 2024-09-16T03:38:05.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7422000000000002e-05 2024-09-16T03:38:05.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.452324+0000", 2024-09-16T03:38:05.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1048000000000002e-05 2024-09-16T03:38:05.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.439 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.440 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1684 4.d 4:b3fe05b2:::smithi15736696-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:05.440 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:05.440 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.175132627, 2024-09-16T03:38:05.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39825815199999998, 2024-09-16T03:38:05.440 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.440 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.440 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:05.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.440 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.440 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1684 2024-09-16T03:38:05.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.440 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:05.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9999999999999998e-07 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154351+0000", 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.249e-06 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154356+0000", 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7690000000000002e-06 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154361+0000", 2024-09-16T03:38:05.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4209999999999997e-06 2024-09-16T03:38:05.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.253488+0000", 2024-09-16T03:38:05.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099126895000000007 2024-09-16T03:38:05.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:05.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.253585+0000", 2024-09-16T03:38:05.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7204e-05 2024-09-16T03:38:05.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552451+0000", 2024-09-16T03:38:05.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.298866358 2024-09-16T03:38:05.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552521+0000", 2024-09-16T03:38:05.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9873e-05 2024-09-16T03:38:05.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552606+0000", 2024-09-16T03:38:05.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4882999999999994e-05 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1749 4.c 4:34492f38:::smithi15736696-33:head [assert-version v40,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:56.368656+0000", 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9608243469999991, 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30391657799999999, 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:05.445 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1749 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368656+0000", 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368656+0000", 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:05.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:05.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368660+0000", 2024-09-16T03:38:05.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.693e-06 2024-09-16T03:38:05.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:05.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368673+0000", 2024-09-16T03:38:05.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2944e-05 2024-09-16T03:38:05.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:05.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368702+0000", 2024-09-16T03:38:05.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9365000000000001e-05 2024-09-16T03:38:05.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368719+0000", 2024-09-16T03:38:05.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6756999999999999e-05 2024-09-16T03:38:05.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:05.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368857+0000", 2024-09-16T03:38:05.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000138166 2024-09-16T03:38:05.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368950+0000", 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2843000000000005e-05 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.471081+0000", 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102130647 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.471807+0000", 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072618800000000001 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.547167+0000", 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075359801000000004 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.572606+0000", 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025439215000000001 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.672443+0000", 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099837710999999996 2024-09-16T03:38:05.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:05.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.672493+0000", 2024-09-16T03:38:05.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9645999999999999e-05 2024-09-16T03:38:05.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:05.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:05.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:05.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.672573+0000", 2024-09-16T03:38:05.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9602000000000001e-05 2024-09-16T03:38:05.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.453 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:05.453 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:05.453 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:05.455 INFO:tasks.rados.rados.0.smithi157.stdout:1027: finishing write tid 3 to smithi15736696-44 2024-09-16T03:38:05.455 INFO:tasks.rados.rados.0.smithi157.stdout:1027: finishing write tid 4 to smithi15736696-44 2024-09-16T03:38:05.455 INFO:tasks.rados.rados.0.smithi157.stdout:1027: oid 44 updating version 0 to 32 2024-09-16T03:38:05.455 INFO:tasks.rados.rados.0.smithi157.stdout:1027: oid 44 version 32 is already newer than 31 2024-09-16T03:38:05.455 INFO:tasks.rados.rados.0.smithi157.stdout:1027: oid 44 updating version 32 to 33 2024-09-16T03:38:05.455 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 33 (ObjNum 173 snap 0 seq_num 173) dirty exists 2024-09-16T03:38:05.455 INFO:tasks.rados.rados.0.smithi157.stdout:1027: left oid 44 (ObjNum 173 snap 0 seq_num 173) 2024-09-16T03:38:05.456 INFO:tasks.rados.rados.0.smithi157.stdout:1035: got (0) Success 2024-09-16T03:38:05.456 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 0 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-09-16T03:38:05.456 INFO:tasks.rados.rados.0.smithi157.stdout:1042: expect (ObjNum 142 snap 0 seq_num 142) 2024-09-16T03:38:05.456 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1370313 2024-09-16T03:38:05.456 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:05.452+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:05.456 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:38:05.487 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:05.543 INFO:tasks.rados.rados.0.smithi157.stdout:1038: finishing write tid 4 to smithi15736696-48 2024-09-16T03:38:05.544 INFO:tasks.rados.rados.0.smithi157.stdout:1038: oid 48 updating version 0 to 44 2024-09-16T03:38:05.544 INFO:tasks.rados.rados.0.smithi157.stdout:1038: oid 48 version 44 is already newer than 43 2024-09-16T03:38:05.544 INFO:tasks.rados.rados.0.smithi157.stdout:1038: oid 48 updating version 44 to 45 2024-09-16T03:38:05.544 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 45 (ObjNum 175 snap 0 seq_num 175) dirty exists 2024-09-16T03:38:05.544 INFO:tasks.rados.rados.0.smithi157.stdout:1038: left oid 48 (ObjNum 175 snap 0 seq_num 175) 2024-09-16T03:38:05.544 INFO:tasks.rados.rados.0.smithi157.stdout:1039: finishing write tid 2 to smithi15736696-36 2024-09-16T03:38:05.544 INFO:tasks.rados.rados.0.smithi157.stdout:1039: finishing write tid 3 to smithi15736696-36 2024-09-16T03:38:05.544 INFO:tasks.rados.rados.0.smithi157.stdout:1039: finishing write tid 4 to smithi15736696-36 2024-09-16T03:38:05.544 INFO:tasks.rados.rados.0.smithi157.stdout:1039: oid 36 updating version 0 to 73 2024-09-16T03:38:05.544 INFO:tasks.rados.rados.0.smithi157.stdout:1039: oid 36 updating version 73 to 74 2024-09-16T03:38:05.544 INFO:tasks.rados.rados.0.smithi157.stdout:1039: oid 36 updating version 74 to 75 2024-09-16T03:38:05.544 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 75 (ObjNum 176 snap 0 seq_num 176) dirty exists 2024-09-16T03:38:05.544 INFO:tasks.rados.rados.0.smithi157.stdout:1039: left oid 36 (ObjNum 176 snap 0 seq_num 176) 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1040: finishing write tid 1 to smithi15736696-19 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1040: finishing write tid 2 to smithi15736696-19 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1040: finishing write tid 3 to smithi15736696-19 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1027: done (15 left) 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1028: done (14 left) 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1035: done (13 left) 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1038: done (12 left) 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1039: done (11 left) 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1042: done (10 left) 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1043: done (9 left) 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1044: done (8 left) 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1052: read oid 48 snap -1 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1052: expect (ObjNum 175 snap 0 seq_num 175) 2024-09-16T03:38:05.545 INFO:tasks.rados.rados.0.smithi157.stdout:1053: read oid 1 snap -1 2024-09-16T03:38:05.546 INFO:tasks.rados.rados.0.smithi157.stdout:1053: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-16T03:38:05.546 INFO:tasks.rados.rados.0.smithi157.stdout:1054: set_chunk oid 31 target oid 2024-09-16T03:38:05.546 INFO:tasks.rados.rados.0.smithi157.stdout:1054: set_chunk oid 31 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:05.546 INFO:tasks.rados.rados.0.smithi157.stdout:1055: set_chunk oid 27 target oid 2024-09-16T03:38:05.555 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:05.553+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:05.571 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:05.572 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:05.572 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:05.572 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:05.581 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:38:05.656 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:05.654+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:05.685 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:05.686 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:05.695 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:38:05.756 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:05.755+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:05.758 INFO:tasks.rados.rados.0.smithi157.stdout:1055: set_chunk oid 27 offset: 757248 length: 1381888 target oid 593314a3a723c40dec2b25351adb282089b02c0264cb0891ee3f17edde05dfe9 offset: 0 snap -1 2024-09-16T03:38:05.758 INFO:tasks.rados.rados.0.smithi157.stdout:1056: read oid 40 snap -1 2024-09-16T03:38:05.758 INFO:tasks.rados.rados.0.smithi157.stdout:1056: expect deleted 2024-09-16T03:38:05.758 INFO:tasks.rados.rados.0.smithi157.stdout:1057: read oid 23 snap -1 2024-09-16T03:38:05.758 INFO:tasks.rados.rados.0.smithi157.stdout:1057: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T03:38:05.758 INFO:tasks.rados.rados.0.smithi157.stdout:1058: tier_promote oid 11 2024-09-16T03:38:05.758 INFO:tasks.rados.rados.0.smithi157.stdout:1041: got (0) Success 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 40 (ObjNum 145 snap 0 seq_num 145) dirty exists 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:1040: finishing write tid 4 to smithi15736696-19 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:1040: oid 19 updating version 0 to 77 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:1040: oid 19 version 77 is already newer than 76 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:1040: oid 19 updating version 77 to 78 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 78 (ObjNum 177 snap 0 seq_num 177) dirty exists 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:1040: left oid 19 (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:1040: done (14 left) 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:1041: done (13 left) 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:1059: read oid 47 snap -1 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:1059: expect deleted 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:1060: read oid 41 snap -1 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:1060: expect (ObjNum 112 snap 0 seq_num 112) 2024-09-16T03:38:05.759 INFO:tasks.rados.rados.0.smithi157.stdout:1061: set_chunk oid 7 target oid 2024-09-16T03:38:05.760 INFO:tasks.rados.rados.0.smithi157.stdout:1061: set_chunk oid 7 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:05.760 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:05.760 INFO:tasks.rados.rados.0.smithi157.stdout:1047: finishing set_chunk to oid 21 2024-09-16T03:38:05.760 INFO:tasks.rados.rados.0.smithi157.stdout:1047: got expected ENOENT (src dne) 2024-09-16T03:38:05.760 INFO:tasks.rados.rados.0.smithi157.stdout:1051: finishing set_chunk to oid 25 2024-09-16T03:38:05.760 INFO:tasks.rados.rados.0.smithi157.stdout:1051: got expected ENOENT (src dne) 2024-09-16T03:38:05.760 INFO:tasks.rados.rados.0.smithi157.stdout:1049: expect (ObjNum 159 snap 0 seq_num 159) 2024-09-16T03:38:05.802 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.802 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:21 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'4, mlcod=0'0)", 2024-09-16T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.612643233, 2024-09-16T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10231100799999999, 2024-09-16T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187300+0000", 2024-09-16T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0360000000000001e-06 2024-09-16T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187905+0000", 2024-09-16T03:38:05.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060508100000000004 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187920+0000", 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4314000000000001e-05 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187928+0000", 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7150000000000004e-06 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187984+0000", 2024-09-16T03:38:05.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5872999999999999e-05 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188016+0000", 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2045000000000002e-05 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289567+0000", 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101550875 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289609+0000", 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2069e-05 2024-09-16T03:38:05.806 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:22 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'5, mlcod=0'0)", 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187945+0000", 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.611996466999997, 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101698394, 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187945+0000", 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187945+0000", 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187946+0000", 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9900000000000001e-07 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187959+0000", 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3242e-05 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187963+0000", 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8410000000000002e-06 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187966+0000", 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9390000000000002e-06 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188816+0000", 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085090000000000003 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188836+0000", 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9638000000000001e-05 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289615+0000", 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10077847299999999 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289643+0000", 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8861999999999999e-05 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:23 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'6, mlcod=0'0)", 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187987+0000", 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.611954189000002, 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102020717, 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.811 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187987+0000", 2024-09-16T03:38:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187987+0000", 2024-09-16T03:38:05.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187988+0000", 2024-09-16T03:38:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9900000000000001e-07 2024-09-16T03:38:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187992+0000", 2024-09-16T03:38:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.707e-06 2024-09-16T03:38:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187998+0000", 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7699999999999998e-06 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188003+0000", 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8459999999999999e-06 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289285+0000", 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101282107 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289343+0000", 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7852000000000001e-05 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289968+0000", 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062506400000000004 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.290008+0000", 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9872e-05 2024-09-16T03:38:05.815 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:81 4.c e39/32 4:3313e97c:::smithi15736696-21:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.398719+0000", 2024-09-16T03:38:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.401222212, 2024-09-16T03:38:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102114725, 2024-09-16T03:38:05.816 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.398719+0000", 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.398719+0000", 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.398722+0000", 2024-09-16T03:38:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7199999999999998e-06 2024-09-16T03:38:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.399189+0000", 2024-09-16T03:38:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046724000000000001 2024-09-16T03:38:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.399200+0000", 2024-09-16T03:38:05.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1068e-05 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499313+0000", 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100112249 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499411+0000", 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8930000000000003e-05 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499444+0000", 2024-09-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2966000000000003e-05 2024-09-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.500783+0000", 2024-09-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013380880000000001 2024-09-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.500834+0000", 2024-09-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1464000000000001e-05 2024-09-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.820 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.21:5 3.0 e45/44 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'39, mlcod=45'37)", 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 44.885346198999997, 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20119541799999999, 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:38:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:38:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914597+0000", 2024-09-16T03:38:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.494e-06 2024-09-16T03:38:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914603+0000", 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7349999999999999e-06 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914614+0000", 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0995999999999999e-05 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914623+0000", 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3899999999999993e-06 2024-09-16T03:38:05.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.114941+0000", 2024-09-16T03:38:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20031812399999999 2024-09-16T03:38:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115001+0000", 2024-09-16T03:38:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0161999999999998e-05 2024-09-16T03:38:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115750+0000", 2024-09-16T03:38:05.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074851699999999998 2024-09-16T03:38:05.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115791+0000", 2024-09-16T03:38:05.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1e-05 2024-09-16T03:38:05.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.825 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.825 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.5.21:36 3.0 e52/44 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 52'62, mlcod=51'60)", 2024-09-16T03:38:05.825 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.915315540000002, 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20104840199999999, 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884627+0000", 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.322e-06 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884634+0000", 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884643+0000", 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7680000000000003e-06 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984740+0000", 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100096066 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084931+0000", 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100191408 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084952+0000", 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.128e-05 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085645+0000", 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00069309500000000002 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085674+0000", 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9025000000000001e-05 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.830 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1537 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'35, mlcod=58'33)", 2024-09-16T03:38:05.830 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:38:05.830 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.079562245999998, 2024-09-16T03:38:05.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102969106, 2024-09-16T03:38:05.830 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.830 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.830 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.830 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720381+0000", 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.767e-06 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721134+0000", 2024-09-16T03:38:05.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00075281999999999996 2024-09-16T03:38:05.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721148+0000", 2024-09-16T03:38:05.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4534e-05 2024-09-16T03:38:05.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721160+0000", 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1595e-05 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721256+0000", 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6136000000000002e-05 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721274+0000", 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8006e-05 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823312+0000", 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102038064 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823348+0000", 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6183999999999998e-05 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1538 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'36, mlcod=58'33)", 2024-09-16T03:38:05.834 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.078407908999999, 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102397932, 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:05.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:05.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5199999999999996e-07 2024-09-16T03:38:05.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722162+0000", 2024-09-16T03:38:05.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062758599999999996 2024-09-16T03:38:05.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722171+0000", 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7139999999999999e-06 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722177+0000", 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.866e-06 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822283+0000", 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10010508899999999 2024-09-16T03:38:05.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822298+0000", 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5497000000000001e-05 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823907+0000", 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001609166 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823931+0000", 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4261999999999999e-05 2024-09-16T03:38:05.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.839 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.839 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.839 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1539 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'37, mlcod=58'33)", 2024-09-16T03:38:05.839 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:38:05.839 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.077207848, 2024-09-16T03:38:05.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103545683, 2024-09-16T03:38:05.839 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.839 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.839 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.839 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722735+0000", 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8400000000000002e-07 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722740+0000", 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3129999999999998e-06 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722747+0000", 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0600000000000002e-06 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822841+0000", 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100094477 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823376+0000", 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00053497199999999999 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823399+0000", 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2680999999999998e-05 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.826259+0000", 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028598500000000002 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.826279+0000", 2024-09-16T03:38:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0346000000000001e-05 2024-09-16T03:38:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:38:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:38:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.650016943000001, 2024-09-16T03:38:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20169899299999999, 2024-09-16T03:38:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.843 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:38:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:38:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149927+0000", 2024-09-16T03:38:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9299999999999999e-06 2024-09-16T03:38:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151090+0000", 2024-09-16T03:38:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011624459999999999 2024-09-16T03:38:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151105+0000", 2024-09-16T03:38:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5238999999999999e-05 2024-09-16T03:38:05.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251193+0000", 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100088025 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251556+0000", 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036245599999999999 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251578+0000", 2024-09-16T03:38:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2651e-05 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351582+0000", 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100003811 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351624+0000", 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1434999999999999e-05 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1689 3.0 e64/44 3:16d0c248:::a2b7d44d389d3ac8752762fa1d216869a8025d7e5ae15e8811d7e7f2543095f6:head v 64'95, mlcod=63'93)", 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.348219+0000", 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.451722806999999, 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101440108, 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.348219+0000", 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.348219+0000", 2024-09-16T03:38:05.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.348219+0000", 2024-09-16T03:38:05.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7700000000000001e-07 2024-09-16T03:38:05.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.348597+0000", 2024-09-16T03:38:05.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00037810200000000001 2024-09-16T03:38:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.348611+0000", 2024-09-16T03:38:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3983e-05 2024-09-16T03:38:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.348623+0000", 2024-09-16T03:38:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2275e-05 2024-09-16T03:38:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.448809+0000", 2024-09-16T03:38:05.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100185356 2024-09-16T03:38:05.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.448828+0000", 2024-09-16T03:38:05.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9417999999999999e-05 2024-09-16T03:38:05.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.449616+0000", 2024-09-16T03:38:05.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00078828999999999998 2024-09-16T03:38:05.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.449659+0000", 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2506999999999998e-05 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.8.22:40 3.0 e65/44 3:0485b3dc:::smithi15736696-36 low_tier:head v 65'97, mlcod=63'94)", 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.564835+0000", 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.23510681, 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10132529699999999, 2024-09-16T03:38:05.852 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.853 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.853 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.853 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.564835+0000", 2024-09-16T03:38:05.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.564835+0000", 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.564836+0000", 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.319e-06 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.564842+0000", 2024-09-16T03:38:05.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8470000000000001e-06 2024-09-16T03:38:05.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.564854+0000", 2024-09-16T03:38:05.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1694e-05 2024-09-16T03:38:05.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.665016+0000", 2024-09-16T03:38:05.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10016246500000001 2024-09-16T03:38:05.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.665165+0000", 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000148667 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.665224+0000", 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9787999999999997e-05 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.666124+0000", 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000899554 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.666160+0000", 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5963000000000001e-05 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:90 3.0 e70/69 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head v 70'103, mlcod=0'0)", 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.02417866, 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102185333, 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.860 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775769+0000", 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7379999999999999e-06 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775782+0000", 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2989e-05 2024-09-16T03:38:05.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775791+0000", 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5369999999999997e-06 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876295+0000", 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100504817 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876316+0000", 2024-09-16T03:38:05.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0409999999999999e-05 2024-09-16T03:38:05.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877916+0000", 2024-09-16T03:38:05.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016003790000000001 2024-09-16T03:38:05.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877948+0000", 2024-09-16T03:38:05.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1949999999999997e-05 2024-09-16T03:38:05.864 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.864 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.864 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.865 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1742 3.0 e70/69 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head v 70'104, mlcod=0'0)", 2024-09-16T03:38:05.865 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:05.865 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.024141964, 2024-09-16T03:38:05.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20271830699999999, 2024-09-16T03:38:05.865 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.865 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.865 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.865 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777455+0000", 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001655524 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777468+0000", 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2486999999999999e-05 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777474+0000", 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876577+0000", 2024-09-16T03:38:05.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099103360000000001 2024-09-16T03:38:05.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876594+0000", 2024-09-16T03:38:05.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6449000000000001e-05 2024-09-16T03:38:05.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978469+0000", 2024-09-16T03:38:05.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101875521 2024-09-16T03:38:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978518+0000", 2024-09-16T03:38:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8488000000000001e-05 2024-09-16T03:38:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:91 3.0 e70/69 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head v 70'105, mlcod=0'0)", 2024-09-16T03:38:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:38:05.869 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.9379300950000005, 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.116532025, 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:38:05.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862015+0000", 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3569999999999998e-06 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862029+0000", 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4082e-05 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862055+0000", 2024-09-16T03:38:05.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6570000000000001e-05 2024-09-16T03:38:05.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862070+0000", 2024-09-16T03:38:05.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4473e-05 2024-09-16T03:38:05.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877631+0000", 2024-09-16T03:38:05.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.015560744 2024-09-16T03:38:05.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877674+0000", 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3615999999999997e-05 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978525+0000", 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100850696 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978543+0000", 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8487000000000001e-05 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.873 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:94 3.0 e70/69 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head v 70'106, mlcod=0'0)", 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.9373938759999998, 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11627603, 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862549+0000", 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1650000000000001e-06 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862556+0000", 2024-09-16T03:38:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8709999999999999e-06 2024-09-16T03:38:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862566+0000", 2024-09-16T03:38:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0271e-05 2024-09-16T03:38:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862573+0000", 2024-09-16T03:38:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.861e-06 2024-09-16T03:38:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978123+0000", 2024-09-16T03:38:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11554993199999999 2024-09-16T03:38:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978163+0000", 2024-09-16T03:38:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0201999999999999e-05 2024-09-16T03:38:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978796+0000", 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063316700000000002 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978824+0000", 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7560999999999999e-05 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1947 4.8 e74/68 4:146f84f0:::smithi15736696-6:head v 74'47, mlcod=73'45)", 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:38:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.6747777379999995, 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103866244, 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:38:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125165+0000", 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.513e-06 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126855+0000", 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016901010000000001 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126870+0000", 2024-09-16T03:38:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4487999999999999e-05 2024-09-16T03:38:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227009+0000", 2024-09-16T03:38:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100138888 2024-09-16T03:38:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227117+0000", 2024-09-16T03:38:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107984 2024-09-16T03:38:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227136+0000", 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9247999999999999e-05 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229002+0000", 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001866143 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229030+0000", 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7878999999999999e-05 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.882 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1979 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'32, mlcod=0'0)", 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.962203191, 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20429768500000001, 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:38:05.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:38:05.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837740+0000", 2024-09-16T03:38:05.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.779e-06 2024-09-16T03:38:05.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838356+0000", 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061634500000000002 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838371+0000", 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5007e-05 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838380+0000", 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1009999999999992e-06 2024-09-16T03:38:05.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038674+0000", 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20029450500000001 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038710+0000", 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6100999999999999e-05 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042001+0000", 2024-09-16T03:38:05.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032909430000000002 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042036+0000", 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4904000000000003e-05 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1980 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'33, mlcod=0'0)", 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.9603265949999997, 2024-09-16T03:38:05.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20296795100000001, 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839616+0000", 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3099999999999996e-07 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840706+0000", 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010900980000000001 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840720+0000", 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.378e-05 2024-09-16T03:38:05.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840727+0000", 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.525e-06 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039940+0000", 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19921313199999999 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039967+0000", 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6393999999999998e-05 2024-09-16T03:38:05.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042552+0000", 2024-09-16T03:38:05.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025850999999999999 2024-09-16T03:38:05.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042583+0000", 2024-09-16T03:38:05.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0991000000000002e-05 2024-09-16T03:38:05.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.891 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.892 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.892 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1981 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'34, mlcod=0'0)", 2024-09-16T03:38:05.892 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:38:05.892 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 3.9583042399999999, 2024-09-16T03:38:05.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201270175, 2024-09-16T03:38:05.892 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:05.892 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:05.892 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:05.892 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:05.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:05.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:38:05.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:05.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:38:05.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:05.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:05.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841638+0000", 2024-09-16T03:38:05.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-16T03:38:05.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:05.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841644+0000", 2024-09-16T03:38:05.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8050000000000003e-06 2024-09-16T03:38:05.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:05.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841653+0000", 2024-09-16T03:38:05.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6580000000000008e-06 2024-09-16T03:38:05.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841660+0000", 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9290000000000001e-06 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041673+0000", 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20001311499999999 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041696+0000", 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3365000000000001e-05 2024-09-16T03:38:05.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042878+0000", 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011814499999999999 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042907+0000", 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9655999999999999e-05 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:05.896 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:05.897 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:05.899 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:38:05.900 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 710896 2024-09-16T03:38:05.900 INFO:tasks.rados.rados.0.smithi157.stdout:1046: finishing set_chunk to oid 34 2024-09-16T03:38:05.900 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 60 (ObjNum 126 snap 0 seq_num 126) dirty exists 2024-09-16T03:38:05.900 INFO:tasks.rados.rados.0.smithi157.stdout:1048: finishing write tid 1 to smithi15736696-37 2024-09-16T03:38:05.900 INFO:tasks.rados.rados.0.smithi157.stdout:1048: finishing write tid 2 to smithi15736696-37 2024-09-16T03:38:05.900 INFO:tasks.rados.rados.0.smithi157.stdout:1046: done (15 left) 2024-09-16T03:38:05.900 INFO:tasks.rados.rados.0.smithi157.stdout:1047: done (14 left) 2024-09-16T03:38:05.900 INFO:tasks.rados.rados.0.smithi157.stdout:1049: done (13 left) 2024-09-16T03:38:05.900 INFO:tasks.rados.rados.0.smithi157.stdout:1051: done (12 left) 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:1062: read oid 25 snap -1 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:1062: expect deleted 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:1063: read oid 24 snap -1 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:1063: expect deleted 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:1048: finishing write tid 3 to smithi15736696-37 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:1048: finishing write tid 4 to smithi15736696-37 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:1048: finishing write tid 5 to smithi15736696-37 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:1048: oid 37 updating version 0 to 42 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:1048: oid 37 version 42 is already newer than 41 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:1048: oid 37 version 42 is already newer than 40 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:1048: oid 37 version 42 is already newer than 39 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 42 (ObjNum 178 snap 0 seq_num 178) dirty exists 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:1048: left oid 37 (ObjNum 178 snap 0 seq_num 178) 2024-09-16T03:38:05.901 INFO:tasks.rados.rados.0.smithi157.stdout:1064: read oid 29 snap -1 2024-09-16T03:38:05.902 INFO:tasks.rados.rados.0.smithi157.stdout:1064: expect (ObjNum 106 snap 0 seq_num 106) 2024-09-16T03:38:05.902 INFO:tasks.rados.rados.0.smithi157.stdout:1048: done (14 left) 2024-09-16T03:38:05.902 INFO:tasks.rados.rados.0.smithi157.stdout:1065: read oid 28 snap -1 2024-09-16T03:38:05.902 INFO:tasks.rados.rados.0.smithi157.stdout:1065: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-16T03:38:05.902 INFO:tasks.rados.rados.0.smithi157.stdout:1066: write (excl) oid 2 current snap is 0 2024-09-16T03:38:05.902 INFO:tasks.rados.rados.0.smithi157.stdout:1050: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:05.902 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:05.895+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:05.934 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1698310 2024-09-16T03:38:05.945 INFO:tasks.rados.rados.0.smithi157.stdout:1054: finishing set_chunk to oid 31 2024-09-16T03:38:05.946 INFO:tasks.rados.rados.0.smithi157.stdout:1054: got expected ENOENT (src dne) 2024-09-16T03:38:05.946 INFO:tasks.rados.rados.0.smithi157.stdout:1053: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-16T03:38:05.998 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:05.996+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:06.005 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:06.005 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:06.006 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:06.006 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:06.014 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:38:06.015 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:06.050 INFO:tasks.rados.rados.0.smithi157.stdout:1052: expect (ObjNum 175 snap 0 seq_num 175) 2024-09-16T03:38:06.050 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:06.094 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2494444 2024-09-16T03:38:06.099 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:06.098+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:06.103 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:06.137 INFO:tasks.rados.rados.0.smithi157.stdout:1057: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-16T03:38:06.156 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:06.156 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:06.156 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:06.156 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:06.156 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:06.157 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:06.161 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:06.159+0000 7fa3886e6640 -1 osd.10 76 handle_osd_map injecting map crc failure 2024-09-16T03:38:06.165 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:38:06.184 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:06.200 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:06.198+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:06.210 INFO:tasks.rados.rados.0.smithi157.stdout:1065: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-16T03:38:06.301 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:06.300+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:06.310 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:06.310 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:06.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:06.310 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:06.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.312 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:25 4.6 4:6ece7830:::smithi15736696-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:06.312 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:38:06.312 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 54.186872145000002, 2024-09-16T03:38:06.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40559452699999998, 2024-09-16T03:38:06.312 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.312 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.312 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.312 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.312 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.312 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.313 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 25 2024-09-16T03:38:06.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.313 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:38:06.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:38:06.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119830+0000", 2024-09-16T03:38:06.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9200000000000001e-07 2024-09-16T03:38:06.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120013+0000", 2024-09-16T03:38:06.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018367200000000001 2024-09-16T03:38:06.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120021+0000", 2024-09-16T03:38:06.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9179999999999994e-06 2024-09-16T03:38:06.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120027+0000", 2024-09-16T03:38:06.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6710000000000004e-06 2024-09-16T03:38:06.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220194+0000", 2024-09-16T03:38:06.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10016689500000001 2024-09-16T03:38:06.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220399+0000", 2024-09-16T03:38:06.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020502899999999999 2024-09-16T03:38:06.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:06.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220613+0000", 2024-09-16T03:38:06.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021383900000000001 2024-09-16T03:38:06.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224640+0000", 2024-09-16T03:38:06.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040268099999999996 2024-09-16T03:38:06.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423348+0000", 2024-09-16T03:38:06.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19870853399999999 2024-09-16T03:38:06.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525343+0000", 2024-09-16T03:38:06.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101995081 2024-09-16T03:38:06.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525381+0000", 2024-09-16T03:38:06.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7842000000000001e-05 2024-09-16T03:38:06.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525424+0000", 2024-09-16T03:38:06.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2644e-05 2024-09-16T03:38:06.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.325 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.325 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.325 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:26 4.6 4:6ece7830:::smithi15736696-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:06.326 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:38:06.326 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 54.186659917999997, 2024-09-16T03:38:06.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550111300000002, 2024-09-16T03:38:06.326 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.326 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.327 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.327 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.327 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.327 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.327 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 26 2024-09-16T03:38:06.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.327 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:38:06.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:38:06.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120042+0000", 2024-09-16T03:38:06.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7999999999999995e-07 2024-09-16T03:38:06.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120210+0000", 2024-09-16T03:38:06.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016779499999999999 2024-09-16T03:38:06.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120219+0000", 2024-09-16T03:38:06.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6859999999999995e-06 2024-09-16T03:38:06.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120223+0000", 2024-09-16T03:38:06.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0330000000000002e-06 2024-09-16T03:38:06.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221494+0000", 2024-09-16T03:38:06.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101271812 2024-09-16T03:38:06.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221572+0000", 2024-09-16T03:38:06.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7733999999999998e-05 2024-09-16T03:38:06.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:06.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221697+0000", 2024-09-16T03:38:06.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012474900000000001 2024-09-16T03:38:06.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224670+0000", 2024-09-16T03:38:06.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029729589999999998 2024-09-16T03:38:06.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423398+0000", 2024-09-16T03:38:06.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19872784900000001 2024-09-16T03:38:06.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525465+0000", 2024-09-16T03:38:06.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102067557 2024-09-16T03:38:06.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525494+0000", 2024-09-16T03:38:06.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9113999999999999e-05 2024-09-16T03:38:06.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525542+0000", 2024-09-16T03:38:06.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8145000000000001e-05 2024-09-16T03:38:06.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.338 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.339 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:27 4.6 4:6ece7830:::smithi15736696-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:06.339 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:38:06.339 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 54.186472332999998, 2024-09-16T03:38:06.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40541507399999999, 2024-09-16T03:38:06.339 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.339 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.339 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.339 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.339 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.339 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.340 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27 2024-09-16T03:38:06.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.340 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:38:06.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:38:06.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120230+0000", 2024-09-16T03:38:06.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5799999999999999e-07 2024-09-16T03:38:06.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120233+0000", 2024-09-16T03:38:06.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.055e-06 2024-09-16T03:38:06.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120237+0000", 2024-09-16T03:38:06.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7099999999999998e-06 2024-09-16T03:38:06.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120244+0000", 2024-09-16T03:38:06.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4810000000000003e-06 2024-09-16T03:38:06.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222660+0000", 2024-09-16T03:38:06.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102416089 2024-09-16T03:38:06.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222705+0000", 2024-09-16T03:38:06.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5368999999999999e-05 2024-09-16T03:38:06.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:06.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222782+0000", 2024-09-16T03:38:06.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6623000000000001e-05 2024-09-16T03:38:06.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224696+0000", 2024-09-16T03:38:06.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019143929999999999 2024-09-16T03:38:06.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423402+0000", 2024-09-16T03:38:06.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198705305 2024-09-16T03:38:06.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525582+0000", 2024-09-16T03:38:06.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102180178 2024-09-16T03:38:06.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525615+0000", 2024-09-16T03:38:06.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3303000000000003e-05 2024-09-16T03:38:06.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525644+0000", 2024-09-16T03:38:06.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8909999999999999e-05 2024-09-16T03:38:06.347 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.347 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.348 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.348 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:28 4.6 4:6ece7830:::smithi15736696-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:06.348 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:06.348 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 54.186451585, 2024-09-16T03:38:06.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550787799999999, 2024-09-16T03:38:06.349 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.349 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.349 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:06.349 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.349 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.349 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.350 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 28 2024-09-16T03:38:06.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.351 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:06.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:06.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:06.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7999999999999996e-07 2024-09-16T03:38:06.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120252+0000", 2024-09-16T03:38:06.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.249e-06 2024-09-16T03:38:06.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120257+0000", 2024-09-16T03:38:06.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1969999999999998e-06 2024-09-16T03:38:06.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120261+0000", 2024-09-16T03:38:06.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1130000000000003e-06 2024-09-16T03:38:06.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222976+0000", 2024-09-16T03:38:06.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102715397 2024-09-16T03:38:06.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:06.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223049+0000", 2024-09-16T03:38:06.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2846e-05 2024-09-16T03:38:06.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525666+0000", 2024-09-16T03:38:06.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30261668800000002 2024-09-16T03:38:06.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525704+0000", 2024-09-16T03:38:06.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8025999999999998e-05 2024-09-16T03:38:06.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525758+0000", 2024-09-16T03:38:06.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3881999999999997e-05 2024-09-16T03:38:06.360 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.361 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.362 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:49 4.6 4:6df3bbbe:::smithi15736696-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:06.362 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:06.362 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 54.186436520999997, 2024-09-16T03:38:06.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40556222800000002, 2024-09-16T03:38:06.363 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.363 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.363 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.363 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.363 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.363 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.364 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49 2024-09-16T03:38:06.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.364 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:06.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:06.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:06.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8700000000000002e-07 2024-09-16T03:38:06.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120454+0000", 2024-09-16T03:38:06.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018909299999999999 2024-09-16T03:38:06.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120465+0000", 2024-09-16T03:38:06.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.083e-05 2024-09-16T03:38:06.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120468+0000", 2024-09-16T03:38:06.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3869999999999999e-06 2024-09-16T03:38:06.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223082+0000", 2024-09-16T03:38:06.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102613379 2024-09-16T03:38:06.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223189+0000", 2024-09-16T03:38:06.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107728 2024-09-16T03:38:06.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:06.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223368+0000", 2024-09-16T03:38:06.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179011 2024-09-16T03:38:06.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.226087+0000", 2024-09-16T03:38:06.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027183390000000002 2024-09-16T03:38:06.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423408+0000", 2024-09-16T03:38:06.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19732106499999999 2024-09-16T03:38:06.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525779+0000", 2024-09-16T03:38:06.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10237133 2024-09-16T03:38:06.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525805+0000", 2024-09-16T03:38:06.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5967000000000001e-05 2024-09-16T03:38:06.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525827+0000", 2024-09-16T03:38:06.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1812000000000001e-05 2024-09-16T03:38:06.375 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.376 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.376 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.376 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:50 4.6 4:6df3bbbe:::smithi15736696-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:06.376 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:38:06.376 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 54.186228212000003, 2024-09-16T03:38:06.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50526462999999999, 2024-09-16T03:38:06.376 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.376 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.376 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.376 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.377 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.377 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.377 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 50 2024-09-16T03:38:06.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.377 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:38:06.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:38:06.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120474+0000", 2024-09-16T03:38:06.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4600000000000005e-07 2024-09-16T03:38:06.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120901+0000", 2024-09-16T03:38:06.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00042690000000000002 2024-09-16T03:38:06.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120907+0000", 2024-09-16T03:38:06.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7710000000000001e-06 2024-09-16T03:38:06.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120911+0000", 2024-09-16T03:38:06.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0500000000000002e-06 2024-09-16T03:38:06.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224717+0000", 2024-09-16T03:38:06.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103806175 2024-09-16T03:38:06.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224763+0000", 2024-09-16T03:38:06.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5629e-05 2024-09-16T03:38:06.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:06.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224838+0000", 2024-09-16T03:38:06.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4548999999999994e-05 2024-09-16T03:38:06.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423390+0000", 2024-09-16T03:38:06.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19855242400000001 2024-09-16T03:38:06.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423411+0000", 2024-09-16T03:38:06.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0888000000000001e-05 2024-09-16T03:38:06.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625680+0000", 2024-09-16T03:38:06.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20226923099999999 2024-09-16T03:38:06.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625709+0000", 2024-09-16T03:38:06.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9232000000000001e-05 2024-09-16T03:38:06.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625738+0000", 2024-09-16T03:38:06.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8235e-05 2024-09-16T03:38:06.382 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.382 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:51 4.6 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 54.185783630000003, 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50490003699999997, 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.383 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 51 2024-09-16T03:38:06.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.384 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:06.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:06.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8699999999999995e-07 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120920+0000", 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0820000000000001e-06 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120924+0000", 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:38:06.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120927+0000", 2024-09-16T03:38:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5809999999999999e-06 2024-09-16T03:38:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225783+0000", 2024-09-16T03:38:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10485588699999999 2024-09-16T03:38:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225832+0000", 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9750000000000003e-05 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225910+0000", 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7084999999999998e-05 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423414+0000", 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19750438200000001 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423456+0000", 2024-09-16T03:38:06.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2386999999999999e-05 2024-09-16T03:38:06.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625766+0000", 2024-09-16T03:38:06.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20231011300000001 2024-09-16T03:38:06.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625790+0000", 2024-09-16T03:38:06.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3672999999999998e-05 2024-09-16T03:38:06.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625818+0000", 2024-09-16T03:38:06.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7576000000000002e-05 2024-09-16T03:38:06.389 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.389 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.389 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.389 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:52 4.6 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:06.389 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:06.389 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 54.185771434000003, 2024-09-16T03:38:06.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50502012900000004, 2024-09-16T03:38:06.390 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.390 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.390 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:06.390 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.390 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.390 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.390 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 52 2024-09-16T03:38:06.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.390 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120932+0000", 2024-09-16T03:38:06.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.181e-06 2024-09-16T03:38:06.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120935+0000", 2024-09-16T03:38:06.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7690000000000001e-06 2024-09-16T03:38:06.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120938+0000", 2024-09-16T03:38:06.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6639999999999998e-06 2024-09-16T03:38:06.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423188+0000", 2024-09-16T03:38:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30224989200000002 2024-09-16T03:38:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423318+0000", 2024-09-16T03:38:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013045999999999999 2024-09-16T03:38:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625844+0000", 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20252547600000001 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625895+0000", 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1242000000000002e-05 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625950+0000", 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5164999999999998e-05 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:279 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:38:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.109819647000002, 2024-09-16T03:38:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50446570800000001, 2024-09-16T03:38:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.395 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 279 2024-09-16T03:38:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:38:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:38:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196883+0000", 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.48e-06 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196889+0000", 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6239999999999999e-06 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196902+0000", 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2804e-05 2024-09-16T03:38:06.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196909+0000", 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9379999999999992e-06 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196963+0000", 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3437999999999999e-05 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op must wait for map", 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196969+0000", 2024-09-16T03:38:06.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1709999999999999e-06 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302538+0000", 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10556889899999999 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302659+0000", 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121441 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:38:06.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500040+0000", 2024-09-16T03:38:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19738082900000001 2024-09-16T03:38:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500589+0000", 2024-09-16T03:38:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00054827000000000001 2024-09-16T03:38:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701118+0000", 2024-09-16T03:38:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20052917100000001 2024-09-16T03:38:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701205+0000", 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7120999999999995e-05 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701289+0000", 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4457999999999997e-05 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701347+0000", 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8063999999999998e-05 2024-09-16T03:38:06.401 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:306 4.6 4:6ece7830:::smithi15736696-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:38:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 49.793817332000003, 2024-09-16T03:38:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40927214699999998, 2024-09-16T03:38:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.402 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 306 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512885+0000", 2024-09-16T03:38:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.277e-06 2024-09-16T03:38:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512890+0000", 2024-09-16T03:38:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6689999999999997e-06 2024-09-16T03:38:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512900+0000", 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0393e-05 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512906+0000", 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4569999999999996e-06 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512951+0000", 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4891000000000001e-05 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.513000+0000", 2024-09-16T03:38:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9187999999999997e-05 2024-09-16T03:38:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.720758+0000", 2024-09-16T03:38:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20775775699999999 2024-09-16T03:38:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.721960+0000", 2024-09-16T03:38:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001202107 2024-09-16T03:38:06.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922010+0000", 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20004993900000001 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922055+0000", 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4981000000000002e-05 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922103+0000", 2024-09-16T03:38:06.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8097999999999999e-05 2024-09-16T03:38:06.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922156+0000", 2024-09-16T03:38:06.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3390000000000002e-05 2024-09-16T03:38:06.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.408 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.408 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:419 4.6 4:64ce87bc:::smithi15736696-20:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:06.408 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:38:06.408 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 45.393017718000003, 2024-09-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40437787200000003, 2024-09-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 419 2024-09-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:38:06.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:38:06.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913687+0000", 2024-09-16T03:38:06.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3299999999999999e-06 2024-09-16T03:38:06.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913700+0000", 2024-09-16T03:38:06.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2830000000000001e-05 2024-09-16T03:38:06.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913733+0000", 2024-09-16T03:38:06.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2815999999999999e-05 2024-09-16T03:38:06.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913750+0000", 2024-09-16T03:38:06.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7073000000000001e-05 2024-09-16T03:38:06.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913920+0000", 2024-09-16T03:38:06.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000170015 2024-09-16T03:38:06.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913981+0000", 2024-09-16T03:38:06.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1631999999999995e-05 2024-09-16T03:38:06.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:06.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216580+0000", 2024-09-16T03:38:06.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30259888200000001 2024-09-16T03:38:06.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217205+0000", 2024-09-16T03:38:06.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062461099999999996 2024-09-16T03:38:06.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317915+0000", 2024-09-16T03:38:06.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10071053000000001 2024-09-16T03:38:06.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317996+0000", 2024-09-16T03:38:06.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0440999999999998e-05 2024-09-16T03:38:06.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318032+0000", 2024-09-16T03:38:06.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5944e-05 2024-09-16T03:38:06.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318061+0000", 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9768e-05 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:479 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 44.585161571999997, 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.504011706, 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.415 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.416 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.416 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.416 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 479 2024-09-16T03:38:06.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.416 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:38:06.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:38:06.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721542+0000", 2024-09-16T03:38:06.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3199999999999998e-06 2024-09-16T03:38:06.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721553+0000", 2024-09-16T03:38:06.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1413000000000001e-05 2024-09-16T03:38:06.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721570+0000", 2024-09-16T03:38:06.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.685e-05 2024-09-16T03:38:06.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721596+0000", 2024-09-16T03:38:06.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5647999999999999e-05 2024-09-16T03:38:06.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.921950+0000", 2024-09-16T03:38:06.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20035402899999999 2024-09-16T03:38:06.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.922009+0000", 2024-09-16T03:38:06.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9046999999999999e-05 2024-09-16T03:38:06.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:38:06.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.023850+0000", 2024-09-16T03:38:06.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10184130199999999 2024-09-16T03:38:06.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.024503+0000", 2024-09-16T03:38:06.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065257399999999995 2024-09-16T03:38:06.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.125105+0000", 2024-09-16T03:38:06.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100602046 2024-09-16T03:38:06.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225461+0000", 2024-09-16T03:38:06.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100355835 2024-09-16T03:38:06.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225512+0000", 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0862999999999997e-05 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225551+0000", 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9779e-05 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.422 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:996 4.8 4:148e627d:::smithi15736696-23:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:06.422 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:38:06.422 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 35.259358665000001, 2024-09-16T03:38:06.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.73588397999999999, 2024-09-16T03:38:06.422 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.422 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.422 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:06.422 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.422 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.422 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.422 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 996 2024-09-16T03:38:06.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.422 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:38:06.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:38:06.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047344+0000", 2024-09-16T03:38:06.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.846e-06 2024-09-16T03:38:06.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047351+0000", 2024-09-16T03:38:06.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.973e-06 2024-09-16T03:38:06.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047365+0000", 2024-09-16T03:38:06.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3556e-05 2024-09-16T03:38:06.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047382+0000", 2024-09-16T03:38:06.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6994e-05 2024-09-16T03:38:06.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047453+0000", 2024-09-16T03:38:06.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1458000000000006e-05 2024-09-16T03:38:06.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:06.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047465+0000", 2024-09-16T03:38:06.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1119e-05 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449411+0000", 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40194680399999999 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449535+0000", 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124043 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783227+0000", 2024-09-16T03:38:06.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.333691187 2024-09-16T03:38:06.427 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.427 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.427 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.427 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1048 4.16 4:6df3bbbe:::smithi15736696-13:head [write 1622009~726683] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:06.427 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:06.427 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.730914548000001, 2024-09-16T03:38:06.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40682847599999999, 2024-09-16T03:38:06.427 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.427 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.427 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1048 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576941+0000", 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011532230000000001 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576949+0000", 2024-09-16T03:38:06.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1089999999999998e-06 2024-09-16T03:38:06.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576956+0000", 2024-09-16T03:38:06.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000001e-06 2024-09-16T03:38:06.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878710+0000", 2024-09-16T03:38:06.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30175434299999998 2024-09-16T03:38:06.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878765+0000", 2024-09-16T03:38:06.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4988000000000003e-05 2024-09-16T03:38:06.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:06.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878852+0000", 2024-09-16T03:38:06.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7201999999999996e-05 2024-09-16T03:38:06.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881535+0000", 2024-09-16T03:38:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026828310000000001 2024-09-16T03:38:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883968+0000", 2024-09-16T03:38:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024330659999999998 2024-09-16T03:38:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982541+0000", 2024-09-16T03:38:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098572876000000004 2024-09-16T03:38:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982580+0000", 2024-09-16T03:38:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8714000000000002e-05 2024-09-16T03:38:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982615+0000", 2024-09-16T03:38:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5438000000000002e-05 2024-09-16T03:38:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1049 4.16 4:6df3bbbe:::smithi15736696-13:head [write 3121996~104915] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:38:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.729736398999997, 2024-09-16T03:38:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40579890600000001, 2024-09-16T03:38:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1049 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576966+0000", 2024-09-16T03:38:06.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8199999999999999e-07 2024-09-16T03:38:06.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577069+0000", 2024-09-16T03:38:06.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103583 2024-09-16T03:38:06.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577076+0000", 2024-09-16T03:38:06.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8190000000000003e-06 2024-09-16T03:38:06.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577082+0000", 2024-09-16T03:38:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5130000000000004e-06 2024-09-16T03:38:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880130+0000", 2024-09-16T03:38:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.303048127 2024-09-16T03:38:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880183+0000", 2024-09-16T03:38:06.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3127999999999997e-05 2024-09-16T03:38:06.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:06.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880282+0000", 2024-09-16T03:38:06.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8881000000000004e-05 2024-09-16T03:38:06.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883101+0000", 2024-09-16T03:38:06.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028194180000000002 2024-09-16T03:38:06.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.979716+0000", 2024-09-16T03:38:06.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.096614899000000004 2024-09-16T03:38:06.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982707+0000", 2024-09-16T03:38:06.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029914249999999998 2024-09-16T03:38:06.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982739+0000", 2024-09-16T03:38:06.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1250000000000001e-05 2024-09-16T03:38:06.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982764+0000", 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5080999999999999e-05 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1050 4.16 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 3226911] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.729611804000001, 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40784258099999998, 2024-09-16T03:38:06.440 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.441 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.441 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.441 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.441 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.441 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.441 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1050 2024-09-16T03:38:06.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.441 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:38:06.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:38:06.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577090+0000", 2024-09-16T03:38:06.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000003e-07 2024-09-16T03:38:06.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577094+0000", 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4340000000000001e-06 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577099+0000", 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8599999999999998e-06 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577104+0000", 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0010000000000001e-06 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881242+0000", 2024-09-16T03:38:06.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30413779000000002 2024-09-16T03:38:06.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881290+0000", 2024-09-16T03:38:06.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7298e-05 2024-09-16T03:38:06.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:06.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881367+0000", 2024-09-16T03:38:06.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7750000000000006e-05 2024-09-16T03:38:06.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883117+0000", 2024-09-16T03:38:06.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001750096 2024-09-16T03:38:06.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.980218+0000", 2024-09-16T03:38:06.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097100724999999999 2024-09-16T03:38:06.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984859+0000", 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046412950000000001 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984900+0000", 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0300999999999999e-05 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984932+0000", 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2332999999999997e-05 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.446 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.447 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1051 4.16 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:06.447 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:06.447 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 34.729589523999998, 2024-09-16T03:38:06.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50807376999999998, 2024-09-16T03:38:06.447 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.447 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.447 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:06.447 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.447 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.447 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1051 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577115+0000", 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3980000000000002e-06 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577120+0000", 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5589999999999996e-06 2024-09-16T03:38:06.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577125+0000", 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9910000000000002e-06 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882780+0000", 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30565438499999997 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882825+0000", 2024-09-16T03:38:06.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5739999999999999e-05 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085039+0000", 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20221376499999999 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085096+0000", 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6666000000000003e-05 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085186+0000", 2024-09-16T03:38:06.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9716000000000006e-05 2024-09-16T03:38:06.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.452 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.452 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1359 4.6 4:64ce87bc:::smithi15736696-20:head [write 1768928~405354] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T03:38:06.452 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:38:06.452 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 26.140566386, 2024-09-16T03:38:06.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40239613499999999, 2024-09-16T03:38:06.452 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.452 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.452 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.453 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.453 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.453 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.453 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1359 2024-09-16T03:38:06.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.453 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:38:06.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166136+0000", 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166923+0000", 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00078683500000000005 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166933+0000", 2024-09-16T03:38:06.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0445000000000001e-05 2024-09-16T03:38:06.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166938+0000", 2024-09-16T03:38:06.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0119999999999996e-06 2024-09-16T03:38:06.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264901+0000", 2024-09-16T03:38:06.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097962831 2024-09-16T03:38:06.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264963+0000", 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1812999999999999e-05 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.265085+0000", 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121914 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.269430+0000", 2024-09-16T03:38:06.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0043455000000000004 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.372259+0000", 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102828645 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568289+0000", 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19603034999999999 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568475+0000", 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018613599999999999 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568531+0000", 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5729000000000001e-05 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1676 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v29,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:38:06.458 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.176737523, 2024-09-16T03:38:06.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50424139300000004, 2024-09-16T03:38:06.459 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.459 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.459 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.459 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:06.459 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:06.459 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:06.459 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1676 2024-09-16T03:38:06.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.459 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:38:06.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:38:06.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129968+0000", 2024-09-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8430000000000003e-06 2024-09-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129982+0000", 2024-09-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4514e-05 2024-09-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130013+0000", 2024-09-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1081999999999998e-05 2024-09-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130029+0000", 2024-09-16T03:38:06.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5849999999999999e-05 2024-09-16T03:38:06.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130169+0000", 2024-09-16T03:38:06.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000139973 2024-09-16T03:38:06.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130240+0000", 2024-09-16T03:38:06.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0638999999999999e-05 2024-09-16T03:38:06.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:38:06.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.532892+0000", 2024-09-16T03:38:06.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402652654 2024-09-16T03:38:06.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533546+0000", 2024-09-16T03:38:06.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065390700000000001 2024-09-16T03:38:06.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533943+0000", 2024-09-16T03:38:06.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039703999999999998 2024-09-16T03:38:06.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634089+0000", 2024-09-16T03:38:06.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100146098 2024-09-16T03:38:06.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634153+0000", 2024-09-16T03:38:06.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3581e-05 2024-09-16T03:38:06.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634205+0000", 2024-09-16T03:38:06.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2212000000000001e-05 2024-09-16T03:38:06.465 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.465 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.465 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.465 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:77 3.2 e73/68 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head v 73'117, mlcod=66'115)", 2024-09-16T03:38:06.465 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:38:06.465 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.1276992449999996, 2024-09-16T03:38:06.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.401393467, 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179005+0000", 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.553e-06 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179029+0000", 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4039000000000001e-05 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179046+0000", 2024-09-16T03:38:06.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7139999999999999e-05 2024-09-16T03:38:06.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.279144+0000", 2024-09-16T03:38:06.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100098245 2024-09-16T03:38:06.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379323+0000", 2024-09-16T03:38:06.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100178621 2024-09-16T03:38:06.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379348+0000", 2024-09-16T03:38:06.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5315000000000001e-05 2024-09-16T03:38:06.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580315+0000", 2024-09-16T03:38:06.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20096734599999999 2024-09-16T03:38:06.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580395+0000", 2024-09-16T03:38:06.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0208000000000002e-05 2024-09-16T03:38:06.470 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.470 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.470 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.470 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.470 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.470 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:06.472 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:38:06.473 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1044820 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:1055: finishing set_chunk to oid 27 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 757248 length: 1381888 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:1058: got (0) Success 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 27 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:1066: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:1066: writing smithi15736696-2 from 608453 to 1154143 tid 1 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:1066: writing smithi15736696-2 from 1624865 to 2080593 tid 2 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:1066: writing smithi15736696-2 from 2632598 to 3008453 tid 3 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:1050: done (15 left) 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:1052: done (14 left) 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:1053: done (13 left) 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:1054: done (12 left) 2024-09-16T03:38:06.474 INFO:tasks.rados.rados.0.smithi157.stdout:1055: done (11 left) 2024-09-16T03:38:06.475 INFO:tasks.rados.rados.0.smithi157.stdout:1057: done (10 left) 2024-09-16T03:38:06.475 INFO:tasks.rados.rados.0.smithi157.stdout:1058: done (9 left) 2024-09-16T03:38:06.475 INFO:tasks.rados.rados.0.smithi157.stdout:1059: done (8 left) 2024-09-16T03:38:06.475 INFO:tasks.rados.rados.0.smithi157.stdout:1063: done (7 left) 2024-09-16T03:38:06.475 INFO:tasks.rados.rados.0.smithi157.stdout:1065: done (6 left) 2024-09-16T03:38:06.475 INFO:tasks.rados.rados.0.smithi157.stdout:1067: read oid 12 snap -1 2024-09-16T03:38:06.475 INFO:tasks.rados.rados.0.smithi157.stdout:1067: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:06.475 INFO:tasks.rados.rados.0.smithi157.stdout:1068: read oid 16 snap -1 2024-09-16T03:38:06.475 INFO:tasks.rados.rados.0.smithi157.stdout:1068: expect (ObjNum 122 snap 0 seq_num 122) 2024-09-16T03:38:06.475 INFO:tasks.rados.rados.0.smithi157.stdout:1069: tier_flush oid 32 2024-09-16T03:38:06.475 INFO:tasks.rados.rados.0.smithi157.stdout:1069: tier_flush oid 32 snap -1 2024-09-16T03:38:06.475 INFO:tasks.rados.rados.0.smithi157.stdout:1060: expect (ObjNum 112 snap 0 seq_num 112) 2024-09-16T03:38:06.475 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:06.476 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:06.468+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:06.528 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:38:06.528 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:38:06.528 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:38:06.528 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T03:38:06.529 INFO:tasks.thrashosds.thrasher:Setting osd 11 primary_affinity to 0.295095 2024-09-16T03:38:06.529 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 0.29509498764516673 2024-09-16T03:38:06.572 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:06.571+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:06.577 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:06.577 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:06.577 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:06.577 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:06.586 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:38:06.607 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3218660 2024-09-16T03:38:06.623 INFO:tasks.rados.rados.0.smithi157.stdout:1061: finishing set_chunk to oid 7 2024-09-16T03:38:06.623 INFO:tasks.rados.rados.0.smithi157.stdout:1061: got expected ENOENT (src dne) 2024-09-16T03:38:06.623 INFO:tasks.rados.rados.0.smithi157.stdout:1056: done (8 left) 2024-09-16T03:38:06.623 INFO:tasks.rados.rados.0.smithi157.stdout:1060: done (7 left) 2024-09-16T03:38:06.623 INFO:tasks.rados.rados.0.smithi157.stdout:1061: done (6 left) 2024-09-16T03:38:06.623 INFO:tasks.rados.rados.0.smithi157.stdout:1070: set_chunk oid 47 target oid 2024-09-16T03:38:06.623 INFO:tasks.rados.rados.0.smithi157.stdout:1070: set_chunk oid 47 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:06.623 INFO:tasks.rados.rados.0.smithi157.stdout:1071: set_chunk oid 14 target oid 2024-09-16T03:38:06.624 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:06.622+0000 7fa38a6ea640 -1 osd.10 77 handle_osd_map injecting map crc failure 2024-09-16T03:38:06.673 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:06.672+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:06.676 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:06.675+0000 7f49439d4640 -1 osd.4 76 handle_osd_map injecting map crc failure 2024-09-16T03:38:06.689 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:06.690 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:06.690 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:06.690 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:06.690 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:06.698 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:38:06.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:38:06.774 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:06.772+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:06.804 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:06.804 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:06.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:06.804 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:06.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.804 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:3 3.0 3.f934d18 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:06.804 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:38:06.804 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.401763293999998, 2024-09-16T03:38:06.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78381636700000001, 2024-09-16T03:38:06.804 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399429+0000", 2024-09-16T03:38:06.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:38:06.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399434+0000", 2024-09-16T03:38:06.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1050000000000003e-06 2024-09-16T03:38:06.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399442+0000", 2024-09-16T03:38:06.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0160000000000008e-06 2024-09-16T03:38:06.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399448+0000", 2024-09-16T03:38:06.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.879e-06 2024-09-16T03:38:06.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399489+0000", 2024-09-16T03:38:06.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0815999999999997e-05 2024-09-16T03:38:06.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:06.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399495+0000", 2024-09-16T03:38:06.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3439999999999997e-06 2024-09-16T03:38:06.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183226+0000", 2024-09-16T03:38:06.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78373077400000002 2024-09-16T03:38:06.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183244+0000", 2024-09-16T03:38:06.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8128999999999999e-05 2024-09-16T03:38:06.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.809 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.809 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:5 3.0 3.aefbbec0 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:06.809 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:38:06.809 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.301995832000003, 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68447923499999996, 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:38:06.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499196+0000", 2024-09-16T03:38:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3480000000000001e-06 2024-09-16T03:38:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499202+0000", 2024-09-16T03:38:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4219999999999999e-06 2024-09-16T03:38:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499212+0000", 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499218+0000", 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4920000000000001e-06 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499283+0000", 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4906000000000005e-05 2024-09-16T03:38:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499293+0000", 2024-09-16T03:38:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0513000000000001e-05 2024-09-16T03:38:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183665+0000", 2024-09-16T03:38:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68437196600000005 2024-09-16T03:38:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183674+0000", 2024-09-16T03:38:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9700000000000005e-06 2024-09-16T03:38:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:6 3.0 3.3bcda120 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.301935223000001, 2024-09-16T03:38:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78474037699999999, 2024-09-16T03:38:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0500000000000002e-07 2024-09-16T03:38:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499260+0000", 2024-09-16T03:38:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6669999999999998e-06 2024-09-16T03:38:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499266+0000", 2024-09-16T03:38:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:38:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499274+0000", 2024-09-16T03:38:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2250000000000003e-06 2024-09-16T03:38:06.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499316+0000", 2024-09-16T03:38:06.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2670000000000003e-05 2024-09-16T03:38:06.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:06.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499323+0000", 2024-09-16T03:38:06.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.37e-06 2024-09-16T03:38:06.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283987+0000", 2024-09-16T03:38:06.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78466433700000005 2024-09-16T03:38:06.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283996+0000", 2024-09-16T03:38:06.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8650000000000007e-06 2024-09-16T03:38:06.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.820 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.820 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:15 3.0 3.7ec93288 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:06.821 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:38:06.821 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.192743213, 2024-09-16T03:38:06.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57518172599999995, 2024-09-16T03:38:06.821 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.821 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.821 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:06.821 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608449+0000", 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.184e-06 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608459+0000", 2024-09-16T03:38:06.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0553999999999999e-05 2024-09-16T03:38:06.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608470+0000", 2024-09-16T03:38:06.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0524e-05 2024-09-16T03:38:06.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608493+0000", 2024-09-16T03:38:06.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2980999999999999e-05 2024-09-16T03:38:06.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608537+0000", 2024-09-16T03:38:06.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3850000000000002e-05 2024-09-16T03:38:06.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:06.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608542+0000", 2024-09-16T03:38:06.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:38:06.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183623+0000", 2024-09-16T03:38:06.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.575080649 2024-09-16T03:38:06.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183629+0000", 2024-09-16T03:38:06.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6950000000000003e-06 2024-09-16T03:38:06.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.825 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.826 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:11 3.0 3.77005160 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:06.826 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:38:06.826 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.099242902999997, 2024-09-16T03:38:06.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.48153733399999998, 2024-09-16T03:38:06.826 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.826 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.826 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:06.826 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:38:06.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:38:06.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701949+0000", 2024-09-16T03:38:06.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.314e-06 2024-09-16T03:38:06.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701954+0000", 2024-09-16T03:38:06.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:38:06.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701964+0000", 2024-09-16T03:38:06.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9550000000000004e-06 2024-09-16T03:38:06.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701986+0000", 2024-09-16T03:38:06.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1931999999999999e-05 2024-09-16T03:38:06.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802129+0000", 2024-09-16T03:38:06.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100142986 2024-09-16T03:38:06.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:06.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802135+0000", 2024-09-16T03:38:06.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6350000000000003e-06 2024-09-16T03:38:06.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183475+0000", 2024-09-16T03:38:06.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38133958099999998 2024-09-16T03:38:06.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183485+0000", 2024-09-16T03:38:06.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0642000000000001e-05 2024-09-16T03:38:06.831 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.831 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.831 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.831 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:6 3.0 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:06.831 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:38:06.831 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 49.818338990000001, 2024-09-16T03:38:06.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60469085300000003, 2024-09-16T03:38:06.832 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.832 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.832 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.832 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:38:06.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:38:06.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982853+0000", 2024-09-16T03:38:06.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.393e-06 2024-09-16T03:38:06.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982858+0000", 2024-09-16T03:38:06.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.062e-06 2024-09-16T03:38:06.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982867+0000", 2024-09-16T03:38:06.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:38:06.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982879+0000", 2024-09-16T03:38:06.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2241e-05 2024-09-16T03:38:06.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982996+0000", 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116996 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983007+0000", 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0694e-05 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284499+0000", 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30149171400000002 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284690+0000", 2024-09-16T03:38:06.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000191032 2024-09-16T03:38:06.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:06.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284908+0000", 2024-09-16T03:38:06.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021827399999999999 2024-09-16T03:38:06.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487092+0000", 2024-09-16T03:38:06.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.202184156 2024-09-16T03:38:06.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487110+0000", 2024-09-16T03:38:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7856999999999999e-05 2024-09-16T03:38:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587335+0000", 2024-09-16T03:38:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100224802 2024-09-16T03:38:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587410+0000", 2024-09-16T03:38:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5343e-05 2024-09-16T03:38:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587543+0000", 2024-09-16T03:38:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000132492 2024-09-16T03:38:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:2 3.0 3:16550cfd:::smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:38:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 49.818266798000003, 2024-09-16T03:38:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40289696000000003, 2024-09-16T03:38:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:38:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:38:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982925+0000", 2024-09-16T03:38:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982928+0000", 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4130000000000002e-06 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982936+0000", 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.875e-06 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982942+0000", 2024-09-16T03:38:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:38:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983093+0000", 2024-09-16T03:38:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015144400000000001 2024-09-16T03:38:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983098+0000", 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4909999999999999e-06 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385695+0000", 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402597539 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385821+0000", 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125418 2024-09-16T03:38:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.844 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.844 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.844 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:9 3.0 3:2c9b9e7c:::smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:38:06.844 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:38:06.844 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 49.818250609000003, 2024-09-16T03:38:06.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60615655800000001, 2024-09-16T03:38:06.844 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.844 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.844 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.844 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982941+0000", 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0860000000000001e-06 2024-09-16T03:38:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982946+0000", 2024-09-16T03:38:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:38:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982954+0000", 2024-09-16T03:38:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8520000000000004e-06 2024-09-16T03:38:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982958+0000", 2024-09-16T03:38:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4870000000000004e-06 2024-09-16T03:38:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983116+0000", 2024-09-16T03:38:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015774699999999999 2024-09-16T03:38:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983120+0000", 2024-09-16T03:38:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:38:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486023+0000", 2024-09-16T03:38:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50290281400000003 2024-09-16T03:38:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486423+0000", 2024-09-16T03:38:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040079299999999998 2024-09-16T03:38:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486707+0000", 2024-09-16T03:38:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028336600000000001 2024-09-16T03:38:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588861+0000", 2024-09-16T03:38:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102154756 2024-09-16T03:38:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588952+0000", 2024-09-16T03:38:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0630000000000005e-05 2024-09-16T03:38:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588984+0000", 2024-09-16T03:38:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2116999999999998e-05 2024-09-16T03:38:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589022+0000", 2024-09-16T03:38:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7910000000000001e-05 2024-09-16T03:38:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589097+0000", 2024-09-16T03:38:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4536999999999995e-05 2024-09-16T03:38:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.852 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:12 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:06.852 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:38:06.852 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 49.818175261, 2024-09-16T03:38:06.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60616102900000002, 2024-09-16T03:38:06.852 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.852 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.852 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:06.852 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:38:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:38:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983017+0000", 2024-09-16T03:38:06.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.483e-06 2024-09-16T03:38:06.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983022+0000", 2024-09-16T03:38:06.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:38:06.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983032+0000", 2024-09-16T03:38:06.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0036e-05 2024-09-16T03:38:06.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983038+0000", 2024-09-16T03:38:06.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6800000000000004e-06 2024-09-16T03:38:06.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983155+0000", 2024-09-16T03:38:06.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116401 2024-09-16T03:38:06.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:06.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983163+0000", 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6219999999999998e-06 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589120+0000", 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60595653199999999 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589176+0000", 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6542000000000002e-05 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.858 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:5 3.0 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:06.858 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:38:06.858 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 45.886986331999999, 2024-09-16T03:38:06.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30268081699999999, 2024-09-16T03:38:06.858 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.858 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.858 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.858 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:38:06.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:38:06.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914207+0000", 2024-09-16T03:38:06.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1500000000000002e-06 2024-09-16T03:38:06.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914215+0000", 2024-09-16T03:38:06.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7530000000000007e-06 2024-09-16T03:38:06.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914232+0000", 2024-09-16T03:38:06.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.628e-05 2024-09-16T03:38:06.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914242+0000", 2024-09-16T03:38:06.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.058e-05 2024-09-16T03:38:06.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914305+0000", 2024-09-16T03:38:06.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3294999999999996e-05 2024-09-16T03:38:06.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914377+0000", 2024-09-16T03:38:06.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1698000000000004e-05 2024-09-16T03:38:06.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:06.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914517+0000", 2024-09-16T03:38:06.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000139443 2024-09-16T03:38:06.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.015450+0000", 2024-09-16T03:38:06.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100933668 2024-09-16T03:38:06.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.015480+0000", 2024-09-16T03:38:06.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9467000000000001e-05 2024-09-16T03:38:06.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216766+0000", 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20128665300000001 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216831+0000", 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4220999999999994e-05 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216885+0000", 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4608999999999998e-05 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.864 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:24 3.0 3:2dd0f25f:::smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 45.786651935999998, 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30270049300000001, 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014541+0000", 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2479999999999999e-06 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014550+0000", 2024-09-16T03:38:06.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.115e-06 2024-09-16T03:38:06.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014567+0000", 2024-09-16T03:38:06.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7292999999999999e-05 2024-09-16T03:38:06.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014577+0000", 2024-09-16T03:38:06.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8020000000000003e-06 2024-09-16T03:38:06.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115566+0000", 2024-09-16T03:38:06.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10098905900000001 2024-09-16T03:38:06.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115702+0000", 2024-09-16T03:38:06.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013602500000000001 2024-09-16T03:38:06.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:06.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115893+0000", 2024-09-16T03:38:06.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019095700000000001 2024-09-16T03:38:06.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216936+0000", 2024-09-16T03:38:06.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101042992 2024-09-16T03:38:06.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317125+0000", 2024-09-16T03:38:06.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100188757 2024-09-16T03:38:06.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317155+0000", 2024-09-16T03:38:06.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0009999999999999e-05 2024-09-16T03:38:06.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317210+0000", 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4589999999999997e-05 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317239+0000", 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9645000000000002e-05 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.871 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:30 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 45.584856754999997, 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40312788999999999, 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:38:06.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216338+0000", 2024-09-16T03:38:06.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0799999999999999e-06 2024-09-16T03:38:06.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216354+0000", 2024-09-16T03:38:06.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6439e-05 2024-09-16T03:38:06.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216392+0000", 2024-09-16T03:38:06.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7162000000000001e-05 2024-09-16T03:38:06.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216444+0000", 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2082e-05 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417447+0000", 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20100358400000001 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417583+0000", 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135537 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417777+0000", 2024-09-16T03:38:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019431200000000001 2024-09-16T03:38:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418479+0000", 2024-09-16T03:38:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00070148499999999996 2024-09-16T03:38:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418755+0000", 2024-09-16T03:38:06.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027627100000000003 2024-09-16T03:38:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619285+0000", 2024-09-16T03:38:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200530138 2024-09-16T03:38:06.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619370+0000", 2024-09-16T03:38:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4567999999999994e-05 2024-09-16T03:38:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619462+0000", 2024-09-16T03:38:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2231999999999994e-05 2024-09-16T03:38:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:18 3.0 3:037ddf75:::smithi15736696-34 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:38:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 39.865262483999999, 2024-09-16T03:38:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30645336299999998, 2024-09-16T03:38:06.882 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935929+0000", 2024-09-16T03:38:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4600000000000003e-07 2024-09-16T03:38:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935936+0000", 2024-09-16T03:38:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3199999999999996e-06 2024-09-16T03:38:06.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935943+0000", 2024-09-16T03:38:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8900000000000007e-06 2024-09-16T03:38:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935954+0000", 2024-09-16T03:38:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0246e-05 2024-09-16T03:38:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140278+0000", 2024-09-16T03:38:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20432396999999999 2024-09-16T03:38:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140342+0000", 2024-09-16T03:38:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4442000000000004e-05 2024-09-16T03:38:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140468+0000", 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125718 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.141460+0000", 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00099208400000000006 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242240+0000", 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100779733 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242316+0000", 2024-09-16T03:38:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6038999999999995e-05 2024-09-16T03:38:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242351+0000", 2024-09-16T03:38:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4928999999999997e-05 2024-09-16T03:38:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242382+0000", 2024-09-16T03:38:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1046000000000001e-05 2024-09-16T03:38:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:34 3.0 3:08307d25:::2b3cb797ae2d3abc24b2cae4a012f26325debe2fc38ccb08bd4503e949d305ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:38:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 35.326400698, 2024-09-16T03:38:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30798127600000003, 2024-09-16T03:38:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.890 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:38:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:38:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474791+0000", 2024-09-16T03:38:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6000000000000003e-07 2024-09-16T03:38:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474865+0000", 2024-09-16T03:38:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4504000000000004e-05 2024-09-16T03:38:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474882+0000", 2024-09-16T03:38:06.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6765999999999998e-05 2024-09-16T03:38:06.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474890+0000", 2024-09-16T03:38:06.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.14e-06 2024-09-16T03:38:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681242+0000", 2024-09-16T03:38:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20635221200000001 2024-09-16T03:38:06.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681355+0000", 2024-09-16T03:38:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000112211 2024-09-16T03:38:06.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681505+0000", 2024-09-16T03:38:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015040100000000001 2024-09-16T03:38:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682352+0000", 2024-09-16T03:38:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084686400000000002 2024-09-16T03:38:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682989+0000", 2024-09-16T03:38:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063749800000000001 2024-09-16T03:38:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782695+0000", 2024-09-16T03:38:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099706063999999997 2024-09-16T03:38:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782735+0000", 2024-09-16T03:38:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9304000000000002e-05 2024-09-16T03:38:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782771+0000", 2024-09-16T03:38:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6551999999999998e-05 2024-09-16T03:38:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:73 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:38:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:46.277582+0000", 2024-09-16T03:38:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.52360865, 2024-09-16T03:38:06.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30283920399999997, 2024-09-16T03:38:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277582+0000", 2024-09-16T03:38:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277582+0000", 2024-09-16T03:38:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277584+0000", 2024-09-16T03:38:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.708e-06 2024-09-16T03:38:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277586+0000", 2024-09-16T03:38:06.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.526e-06 2024-09-16T03:38:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277592+0000", 2024-09-16T03:38:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3190000000000002e-06 2024-09-16T03:38:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277595+0000", 2024-09-16T03:38:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6279999999999998e-06 2024-09-16T03:38:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.378846+0000", 2024-09-16T03:38:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10125093 2024-09-16T03:38:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379007+0000", 2024-09-16T03:38:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000161046 2024-09-16T03:38:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:06.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379230+0000", 2024-09-16T03:38:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022235000000000001 2024-09-16T03:38:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.480276+0000", 2024-09-16T03:38:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101046812 2024-09-16T03:38:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.480479+0000", 2024-09-16T03:38:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020296100000000001 2024-09-16T03:38:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580317+0000", 2024-09-16T03:38:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099837767999999993 2024-09-16T03:38:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580373+0000", 2024-09-16T03:38:06.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6249999999999998e-05 2024-09-16T03:38:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580421+0000", 2024-09-16T03:38:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7905999999999998e-05 2024-09-16T03:38:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:74 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:38:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:46.277614+0000", 2024-09-16T03:38:06.910 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.523577002, 2024-09-16T03:38:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.302949411, 2024-09-16T03:38:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277614+0000", 2024-09-16T03:38:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277614+0000", 2024-09-16T03:38:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277614+0000", 2024-09-16T03:38:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4300000000000003e-07 2024-09-16T03:38:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277618+0000", 2024-09-16T03:38:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0369999999999996e-06 2024-09-16T03:38:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277622+0000", 2024-09-16T03:38:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9790000000000004e-06 2024-09-16T03:38:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277626+0000", 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2629999999999999e-06 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379529+0000", 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101903627 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379609+0000", 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9721000000000005e-05 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379690+0000", 2024-09-16T03:38:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1267000000000006e-05 2024-09-16T03:38:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.480491+0000", 2024-09-16T03:38:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100801053 2024-09-16T03:38:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.480528+0000", 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6871e-05 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580470+0000", 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099941422000000002 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580525+0000", 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5197e-05 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580563+0000", 2024-09-16T03:38:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8430999999999998e-05 2024-09-16T03:38:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:77 3.0 3:1a18dfd0:::9d9929bebc2b17b3e19b9b502dd3b0e63f1b7b4c1cf7796e8161d7773d261a75:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.802529+0000", 2024-09-16T03:38:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.998661823999999, 2024-09-16T03:38:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30442216599999999, 2024-09-16T03:38:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.917 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.802529+0000", 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.802529+0000", 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.802530+0000", 2024-09-16T03:38:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.311e-06 2024-09-16T03:38:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803091+0000", 2024-09-16T03:38:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056044899999999997 2024-09-16T03:38:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803101+0000", 2024-09-16T03:38:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9189999999999994e-06 2024-09-16T03:38:06.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803106+0000", 2024-09-16T03:38:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.626e-06 2024-09-16T03:38:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.903907+0000", 2024-09-16T03:38:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100800951 2024-09-16T03:38:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.904063+0000", 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015626 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.904369+0000", 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030599300000000001 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907929+0000", 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035599080000000001 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.006526+0000", 2024-09-16T03:38:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098596345000000002 2024-09-16T03:38:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.106808+0000", 2024-09-16T03:38:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100282198 2024-09-16T03:38:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.106917+0000", 2024-09-16T03:38:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010936 2024-09-16T03:38:06.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.106951+0000", 2024-09-16T03:38:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3846000000000001e-05 2024-09-16T03:38:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:78 3.0 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803117+0000", 2024-09-16T03:38:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.998073307, 2024-09-16T03:38:06.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30401020200000001, 2024-09-16T03:38:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803117+0000", 2024-09-16T03:38:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803117+0000", 2024-09-16T03:38:06.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803118+0000", 2024-09-16T03:38:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0400000000000005e-07 2024-09-16T03:38:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803503+0000", 2024-09-16T03:38:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038435999999999999 2024-09-16T03:38:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803515+0000", 2024-09-16T03:38:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2279999999999999e-05 2024-09-16T03:38:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803521+0000", 2024-09-16T03:38:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.054e-06 2024-09-16T03:38:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.905093+0000", 2024-09-16T03:38:06.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101572302 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.905196+0000", 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000102876 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.905387+0000", 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019066499999999999 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907935+0000", 2024-09-16T03:38:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025477949999999998 2024-09-16T03:38:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.106742+0000", 2024-09-16T03:38:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19880740599999999 2024-09-16T03:38:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107021+0000", 2024-09-16T03:38:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027904200000000002 2024-09-16T03:38:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107083+0000", 2024-09-16T03:38:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2149000000000004e-05 2024-09-16T03:38:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107128+0000", 2024-09-16T03:38:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4468999999999997e-05 2024-09-16T03:38:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:79 3.0 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803530+0000", 2024-09-16T03:38:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.997661196999999, 2024-09-16T03:38:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30378224199999998, 2024-09-16T03:38:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803530+0000", 2024-09-16T03:38:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803530+0000", 2024-09-16T03:38:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803530+0000", 2024-09-16T03:38:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8000000000000005e-07 2024-09-16T03:38:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803942+0000", 2024-09-16T03:38:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00041188299999999999 2024-09-16T03:38:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803952+0000", 2024-09-16T03:38:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.027e-05 2024-09-16T03:38:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803957+0000", 2024-09-16T03:38:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0159999999999999e-06 2024-09-16T03:38:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.906129+0000", 2024-09-16T03:38:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102171915 2024-09-16T03:38:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.906280+0000", 2024-09-16T03:38:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000150447 2024-09-16T03:38:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.906551+0000", 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027134700000000002 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907944+0000", 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013923410000000001 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107181+0000", 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19923718800000001 2024-09-16T03:38:06.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107248+0000", 2024-09-16T03:38:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7341999999999993e-05 2024-09-16T03:38:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107283+0000", 2024-09-16T03:38:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4711e-05 2024-09-16T03:38:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107312+0000", 2024-09-16T03:38:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9002000000000001e-05 2024-09-16T03:38:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:87 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:38:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.997205974, 2024-09-16T03:38:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402368742, 2024-09-16T03:38:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:06.936 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803986+0000", 2024-09-16T03:38:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9400000000000004e-07 2024-09-16T03:38:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804061+0000", 2024-09-16T03:38:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5230999999999999e-05 2024-09-16T03:38:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804069+0000", 2024-09-16T03:38:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7530000000000001e-06 2024-09-16T03:38:06.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804073+0000", 2024-09-16T03:38:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9660000000000004e-06 2024-09-16T03:38:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907236+0000", 2024-09-16T03:38:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103162566 2024-09-16T03:38:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:06.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907324+0000", 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7894999999999998e-05 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907474+0000", 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014983999999999999 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.908330+0000", 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085600100000000001 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107220+0000", 2024-09-16T03:38:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198889812 2024-09-16T03:38:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206269+0000", 2024-09-16T03:38:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099049611999999995 2024-09-16T03:38:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206305+0000", 2024-09-16T03:38:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5679000000000002e-05 2024-09-16T03:38:06.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206353+0000", 2024-09-16T03:38:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8593e-05 2024-09-16T03:38:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:06.942 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:06.942 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:06.944 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:38:06.945 INFO:tasks.rados.rados.0.smithi157.stdout:1071: set_chunk oid 14 offset: 904192 length: 229376 target oid dbc7eb371ee5d6ba31245beb0b68f6cc4d0554f9871496d89f6818e28d8062b4 offset: 0 snap -1 2024-09-16T03:38:06.945 INFO:tasks.rados.rados.0.smithi157.stdout:1072: read oid 33 snap -1 2024-09-16T03:38:06.945 INFO:tasks.rados.rados.0.smithi157.stdout:1072: expect (ObjNum 127 snap 0 seq_num 127) 2024-09-16T03:38:06.945 INFO:tasks.rados.rados.0.smithi157.stdout:1073: write oid 9 current snap is 0 2024-09-16T03:38:06.945 INFO:tasks.rados.rados.0.smithi157.stdout:1073: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2024-09-16T03:38:06.945 INFO:tasks.rados.rados.0.smithi157.stdout:1073: writing smithi15736696-9 from 625260 to 1246199 tid 1 2024-09-16T03:38:06.945 INFO:tasks.rados.rados.0.smithi157.stdout:1073: writing smithi15736696-9 from 1683347 to 2282733 tid 2 2024-09-16T03:38:06.945 INFO:tasks.rados.rados.0.smithi157.stdout:1073: writing smithi15736696-9 from 2943668 to 3025260 tid 3 2024-09-16T03:38:06.946 INFO:tasks.rados.rados.0.smithi157.stdout:1074: set_chunk oid 6 target oid 2024-09-16T03:38:06.946 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:06.942+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:07.044 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:07.042+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:07.049 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:07.049 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:07.049 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:07.049 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:07.057 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:38:07.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:38:07.146 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:07.144+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:07.153 INFO:tasks.rados.rados.0.smithi157.stdout:1074: set_chunk oid 6 offset: 513536 length: 146944 target oid 711c26f4edafd93e496c38a9563bdc869f5455fd63dbafc69ccf64f4233eb58b offset: 0 snap -1 2024-09-16T03:38:07.153 INFO:tasks.rados.rados.0.smithi157.stdout:1075: write oid 24 current snap is 0 2024-09-16T03:38:07.153 INFO:tasks.rados.rados.0.smithi157.stdout:1075: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-09-16T03:38:07.160 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:07.160 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:07.160 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:07.160 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:07.160 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:07.160 INFO:tasks.rados.rados.0.smithi157.stdout:1075: writing smithi15736696-24 from 642067 to 1338255 tid 1 2024-09-16T03:38:07.166 INFO:tasks.rados.rados.0.smithi157.stdout:1075: writing smithi15736696-24 from 2025476 to 2484873 tid 2 2024-09-16T03:38:07.168 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:38:07.170 INFO:tasks.rados.rados.0.smithi157.stdout:1075: writing smithi15736696-24 from 2971091 to 3042067 tid 3 2024-09-16T03:38:07.170 INFO:tasks.rados.rados.0.smithi157.stdout:1062: done (11 left) 2024-09-16T03:38:07.170 INFO:tasks.rados.rados.0.smithi157.stdout:1076: read oid 46 snap -1 2024-09-16T03:38:07.170 INFO:tasks.rados.rados.0.smithi157.stdout:1076: expect (ObjNum 163 snap 0 seq_num 163) 2024-09-16T03:38:07.170 INFO:tasks.rados.rados.0.smithi157.stdout:1077: write oid 23 current snap is 0 2024-09-16T03:38:07.170 INFO:tasks.rados.rados.0.smithi157.stdout:1077: seq_num 182 ranges {658874=771437,2083958=603055} 2024-09-16T03:38:07.178 INFO:tasks.rados.rados.0.smithi157.stdout:1077: writing smithi15736696-23 from 658874 to 1430311 tid 1 2024-09-16T03:38:07.184 INFO:tasks.rados.rados.0.smithi157.stdout:1077: writing smithi15736696-23 from 2083958 to 2687013 tid 2 2024-09-16T03:38:07.184 INFO:tasks.rados.rados.0.smithi157.stdout:1078: write oid 31 current snap is 0 2024-09-16T03:38:07.184 INFO:tasks.rados.rados.0.smithi157.stdout:1078: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-09-16T03:38:07.190 INFO:tasks.rados.rados.0.smithi157.stdout:1078: writing smithi15736696-31 from 675681 to 1238720 tid 1 2024-09-16T03:38:07.198 INFO:tasks.rados.rados.0.smithi157.stdout:1078: writing smithi15736696-31 from 1858793 to 2605506 tid 2 2024-09-16T03:38:07.201 INFO:tasks.rados.rados.0.smithi157.stdout:1078: writing smithi15736696-31 from 3025937 to 3075681 tid 3 2024-09-16T03:38:07.201 INFO:tasks.rados.rados.0.smithi157.stdout:1079: read oid 7 snap -1 2024-09-16T03:38:07.201 INFO:tasks.rados.rados.0.smithi157.stdout:1079: expect deleted 2024-09-16T03:38:07.201 INFO:tasks.rados.rados.0.smithi157.stdout:1080: set_chunk oid 27 target oid 2024-09-16T03:38:07.246 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:07.244+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:07.272 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:07.272 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:07.272 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:07.272 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:07.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.273 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:57 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:07.273 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:07.273 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.145467826000001, 2024-09-16T03:38:07.273 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20372090900000001, 2024-09-16T03:38:07.273 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:07.273 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:07.273 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:07.273 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:07.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.273 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:07.273 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125313+0000", 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7180000000000001e-06 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.274 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:07.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126730+0000", 2024-09-16T03:38:07.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00141731 2024-09-16T03:38:07.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:07.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126746+0000", 2024-09-16T03:38:07.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6455000000000001e-05 2024-09-16T03:38:07.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:07.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126756+0000", 2024-09-16T03:38:07.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:38:07.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:07.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227068+0000", 2024-09-16T03:38:07.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100312308 2024-09-16T03:38:07.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:07.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227085+0000", 2024-09-16T03:38:07.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6898e-05 2024-09-16T03:38:07.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:07.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328995+0000", 2024-09-16T03:38:07.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101909903 2024-09-16T03:38:07.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:07.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329031+0000", 2024-09-16T03:38:07.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5717999999999999e-05 2024-09-16T03:38:07.277 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.277 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:07.277 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.277 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:58 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:07.277 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:07.278 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.142134695999999, 2024-09-16T03:38:07.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084524500000001, 2024-09-16T03:38:07.278 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:07.278 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:07.278 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:07.278 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:07.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:07.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:07.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128644+0000", 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.251e-06 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128913+0000", 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026856200000000002 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:07.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128925+0000", 2024-09-16T03:38:07.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2109999999999999e-05 2024-09-16T03:38:07.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:07.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128933+0000", 2024-09-16T03:38:07.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6260000000000003e-06 2024-09-16T03:38:07.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:07.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328196+0000", 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19926303200000001 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328253+0000", 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7262999999999997e-05 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329458+0000", 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012051939999999999 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329488+0000", 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0207000000000001e-05 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:59 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'3, mlcod=0'0)", 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.141639263000002, 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036954900000001, 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:07.282 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:07.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129139+0000", 2024-09-16T03:38:07.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6099999999999999e-07 2024-09-16T03:38:07.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:07.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129158+0000", 2024-09-16T03:38:07.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088e-05 2024-09-16T03:38:07.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:07.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129168+0000", 2024-09-16T03:38:07.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.048e-05 2024-09-16T03:38:07.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.229233+0000", 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100065168 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328815+0000", 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099581694999999998 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328845+0000", 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0193e-05 2024-09-16T03:38:07.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329492+0000", 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064671800000000001 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329508+0000", 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6246e-05 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.287 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.0:15 4.d e47/32 4:b3fe05b2:::smithi15736696-15:head v 47'5, mlcod=39'3)", 2024-09-16T03:38:07.287 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:07.287 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 40.968344537, 2024-09-16T03:38:07.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100920751, 2024-09-16T03:38:07.287 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:07.287 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:07.287 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:07.287 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:07.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:07.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:07.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302435+0000", 2024-09-16T03:38:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2340000000000001e-06 2024-09-16T03:38:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:07.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302446+0000", 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0596e-05 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302461+0000", 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5082000000000001e-05 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402592+0000", 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100130488 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402712+0000", 2024-09-16T03:38:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012013100000000001 2024-09-16T03:38:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402741+0000", 2024-09-16T03:38:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9717999999999999e-05 2024-09-16T03:38:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403309+0000", 2024-09-16T03:38:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056725700000000005 2024-09-16T03:38:07.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403354+0000", 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5244999999999998e-05 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.120845828, 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032593800000000001, 2024-09-16T03:38:07.291 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149943+0000", 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0817e-05 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151026+0000", 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010835350000000001 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151040+0000", 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3884e-05 2024-09-16T03:38:07.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151049+0000", 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151107+0000", 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8199000000000003e-05 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151124+0000", 2024-09-16T03:38:07.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7756000000000001e-05 2024-09-16T03:38:07.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:07.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153142+0000", 2024-09-16T03:38:07.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020177569999999998 2024-09-16T03:38:07.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:07.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153191+0000", 2024-09-16T03:38:07.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9051999999999997e-05 2024-09-16T03:38:07.295 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.295 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:07.295 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1682 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'7, mlcod=40'4)", 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.017248638000002, 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10169502499999999, 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:07.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253540+0000", 2024-09-16T03:38:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0888e-05 2024-09-16T03:38:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254015+0000", 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047485499999999999 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254030+0000", 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4899000000000001e-05 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254039+0000", 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5000000000000005e-06 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254107+0000", 2024-09-16T03:38:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7298000000000005e-05 2024-09-16T03:38:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254122+0000", 2024-09-16T03:38:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5928999999999999e-05 2024-09-16T03:38:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355184+0000", 2024-09-16T03:38:07.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101061703 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355224+0000", 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9953000000000002e-05 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1683 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'8, mlcod=40'4)", 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 17.016711090000001, 2024-09-16T03:38:07.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101630885, 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254068+0000", 2024-09-16T03:38:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5700000000000001e-07 2024-09-16T03:38:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254075+0000", 2024-09-16T03:38:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2710000000000003e-06 2024-09-16T03:38:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254083+0000", 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0220000000000004e-06 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254089+0000", 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8699999999999997e-06 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354968+0000", 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100879625 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354992+0000", 2024-09-16T03:38:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-16T03:38:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355665+0000", 2024-09-16T03:38:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00067371100000000001 2024-09-16T03:38:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355698+0000", 2024-09-16T03:38:07.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2291e-05 2024-09-16T03:38:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.0:177 4.10 e65/57 4:0a55bbf2:::smithi15736696-18:head v 65'47, mlcod=0'0)", 2024-09-16T03:38:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.063764502000001, 2024-09-16T03:38:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074461099999999995, 2024-09-16T03:38:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:07.305 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:07.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5300000000000001e-07 2024-09-16T03:38:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207018+0000", 2024-09-16T03:38:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7210000000000002e-06 2024-09-16T03:38:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207031+0000", 2024-09-16T03:38:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2767e-05 2024-09-16T03:38:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207040+0000", 2024-09-16T03:38:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5400000000000001e-06 2024-09-16T03:38:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207107+0000", 2024-09-16T03:38:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6625999999999995e-05 2024-09-16T03:38:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207131+0000", 2024-09-16T03:38:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4128e-05 2024-09-16T03:38:07.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:07.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207712+0000", 2024-09-16T03:38:07.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00058124700000000001 2024-09-16T03:38:07.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:07.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207758+0000", 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5429000000000002e-05 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1723 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'48, mlcod=53'46)", 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.621697320999999, 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00088284599999999996, 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:07.310 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:07.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649081+0000", 2024-09-16T03:38:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8299999999999995e-07 2024-09-16T03:38:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649101+0000", 2024-09-16T03:38:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0126e-05 2024-09-16T03:38:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649141+0000", 2024-09-16T03:38:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9609e-05 2024-09-16T03:38:07.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649161+0000", 2024-09-16T03:38:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9599999999999999e-05 2024-09-16T03:38:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649299+0000", 2024-09-16T03:38:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013822199999999999 2024-09-16T03:38:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:07.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649318+0000", 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9273999999999999e-05 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649919+0000", 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060074100000000001 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649963+0000", 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4391000000000002e-05 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:07.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1724 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'49, mlcod=53'46)", 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.618032356000001, 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064536300000000003, 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652747+0000", 2024-09-16T03:38:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.291e-06 2024-09-16T03:38:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:07.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652753+0000", 2024-09-16T03:38:07.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4450000000000001e-06 2024-09-16T03:38:07.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:07.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652782+0000", 2024-09-16T03:38:07.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9094999999999998e-05 2024-09-16T03:38:07.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652797+0000", 2024-09-16T03:38:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5021999999999999e-05 2024-09-16T03:38:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652854+0000", 2024-09-16T03:38:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6586999999999999e-05 2024-09-16T03:38:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652885+0000", 2024-09-16T03:38:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.146e-05 2024-09-16T03:38:07.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653350+0000", 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046443599999999998 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653391+0000", 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1026999999999999e-05 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1725 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'50, mlcod=53'46)", 2024-09-16T03:38:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 15.617239532999999, 2024-09-16T03:38:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056324100000000002, 2024-09-16T03:38:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653539+0000", 2024-09-16T03:38:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9699999999999994e-07 2024-09-16T03:38:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:07.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653545+0000", 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9309999999999996e-06 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653557+0000", 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2340999999999999e-05 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653564+0000", 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2779999999999999e-06 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653605+0000", 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1301000000000003e-05 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653621+0000", 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5951e-05 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654064+0000", 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000442843 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:07.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654101+0000", 2024-09-16T03:38:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7598999999999997e-05 2024-09-16T03:38:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:07.324 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:07.324 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:07.325 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:38:07.370 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:07.369+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:07.389 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:07.390 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:07.441 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:07.441 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:07.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:07.441 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.4.22:64 3.3 e78/31)", 2024-09-16T03:38:07.442 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:07.340349+0000", 2024-09-16T03:38:07.442 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.099978206999999999, 2024-09-16T03:38:07.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099992869999999998, 2024-09-16T03:38:07.442 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:07.442 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:07.442 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "no flag points reached", 2024-09-16T03:38:07.442 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:07.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:07.442 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:07.442 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.340349+0000", 2024-09-16T03:38:07.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:07.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:07.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.340349+0000", 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.340351+0000", 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5210000000000001e-06 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.340363+0000", 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2201000000000001e-05 2024-09-16T03:38:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.340380+0000", 2024-09-16T03:38:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6569999999999999e-05 2024-09-16T03:38:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2024-09-16T03:38:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 1 2024-09-16T03:38:07.444 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:07.449 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:07.447+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:07.451 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:38:07.508 INFO:tasks.rados.rados.0.smithi157.stdout:1080: set_chunk oid 27 offset: 287744 length: 1006080 target oid 18305b639b612b29c3c6bc00d70c23c0110ecea6ec450baae2823d65caf0ffa2 offset: 0 snap -1 2024-09-16T03:38:07.508 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:07.509 INFO:tasks.rados.rados.0.smithi157.stdout:1064: expect (ObjNum 106 snap 0 seq_num 106) 2024-09-16T03:38:07.550 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:07.547+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:07.579 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:07.596 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:07.597 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:07.597 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:07.597 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:07.597 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:07.603 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:07.603 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:07.605 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:38:07.620 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:07.621 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:07.638 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:07.638 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:07.651 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:07.649+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:07.705 INFO:tasks.rados.rados.0.smithi157.stdout:1069: got (0) Success 2024-09-16T03:38:07.705 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T03:38:07.705 INFO:tasks.rados.rados.0.smithi157.stdout:1067: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:07.705 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:07.709 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:07.710 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:07.718 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:38:07.732 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:07.759 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:07.761 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:07.760+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:07.761 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:07.762 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:07.767 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.11 primary-affinity to 0.295095 (8193392) 2024-09-16T03:38:07.767 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:38:07.779 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:07.779 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:07.786 INFO:tasks.rados.rados.0.smithi157.stdout:1066: finishing write tid 1 to smithi15736696-2 2024-09-16T03:38:07.786 INFO:tasks.rados.rados.0.smithi157.stdout:1066: finishing write tid 2 to smithi15736696-2 2024-09-16T03:38:07.786 INFO:tasks.rados.rados.0.smithi157.stdout:1066: finishing write tid 3 to smithi15736696-2 2024-09-16T03:38:07.786 INFO:tasks.rados.rados.0.smithi157.stdout:1066: finishing write tid 4 to smithi15736696-2 2024-09-16T03:38:07.786 INFO:tasks.rados.rados.0.smithi157.stdout:1066: finishing write tid 5 to smithi15736696-2 2024-09-16T03:38:07.786 INFO:tasks.rados.rados.0.smithi157.stdout:1066: oid 2 updating version 0 to 25 2024-09-16T03:38:07.786 INFO:tasks.rados.rados.0.smithi157.stdout:1066: oid 2 version 25 is already newer than 22 2024-09-16T03:38:07.786 INFO:tasks.rados.rados.0.smithi157.stdout:1066: oid 2 version 25 is already newer than 24 2024-09-16T03:38:07.786 INFO:tasks.rados.rados.0.smithi157.stdout:1066: oid 2 version 25 is already newer than 23 2024-09-16T03:38:07.787 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 25 (ObjNum 179 snap 0 seq_num 179) dirty exists 2024-09-16T03:38:07.787 INFO:tasks.rados.rados.0.smithi157.stdout:1066: left oid 2 (ObjNum 179 snap 0 seq_num 179) 2024-09-16T03:38:07.787 INFO:tasks.rados.rados.0.smithi157.stdout:1068: expect (ObjNum 122 snap 0 seq_num 122) 2024-09-16T03:38:07.797 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:07.797 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:07.814 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:07.814 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:07.817 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1196112 2024-09-16T03:38:07.823 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:07.823 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:07.830 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:07.830 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:07.833 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:38:07.838 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1280255 2024-09-16T03:38:07.847 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:07.847 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:07.850 INFO:tasks.rados.rados.0.smithi157.stdout:1070: finishing set_chunk to oid 47 2024-09-16T03:38:07.850 INFO:tasks.rados.rados.0.smithi157.stdout:1070: got expected ENOENT (src dne) 2024-09-16T03:38:07.850 INFO:tasks.rados.rados.0.smithi157.stdout:1072: expect (ObjNum 127 snap 0 seq_num 127) 2024-09-16T03:38:07.852 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:07.849+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:07.864 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:07.864 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:07.881 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:07.881 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:07.885 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:07.896 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:07.897 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:07.913 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:07.914 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:07.914 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:07.912+0000 7fedb5739640 -1 osd.9 78 handle_osd_map injecting map crc failure 2024-09-16T03:38:07.925 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 740590 2024-09-16T03:38:07.931 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:07.931 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:07.952 INFO:tasks.rados.rados.0.smithi157.stdout:1073: finishing write tid 1 to smithi15736696-9 2024-09-16T03:38:07.953 INFO:tasks.rados.rados.0.smithi157.stdout:1073: finishing write tid 2 to smithi15736696-9 2024-09-16T03:38:07.953 INFO:tasks.rados.rados.0.smithi157.stdout:1075: finishing write tid 1 to smithi15736696-24 2024-09-16T03:38:07.953 INFO:tasks.rados.rados.0.smithi157.stdout:1076: expect (ObjNum 163 snap 0 seq_num 163) 2024-09-16T03:38:07.953 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:07.950+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:07.969 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1886664 2024-09-16T03:38:07.978 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:07.978 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:07.987 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:38:07.990 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:07.987+0000 7f8be56cd640 -1 osd.7 78 handle_osd_map injecting map crc failure 2024-09-16T03:38:08.032 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:08.032 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:08.077 INFO:tasks.rados.rados.0.smithi157.stdout:1075: finishing write tid 2 to smithi15736696-24 2024-09-16T03:38:08.078 INFO:tasks.rados.rados.0.smithi157.stdout:1077: finishing write tid 1 to smithi15736696-23 2024-09-16T03:38:08.078 INFO:tasks.rados.rados.0.smithi157.stdout:1075: finishing write tid 3 to smithi15736696-24 2024-09-16T03:38:08.078 INFO:tasks.rados.rados.0.smithi157.stdout:1075: finishing write tid 4 to smithi15736696-24 2024-09-16T03:38:08.078 INFO:tasks.rados.rados.0.smithi157.stdout:1075: finishing write tid 5 to smithi15736696-24 2024-09-16T03:38:08.078 INFO:tasks.rados.rados.0.smithi157.stdout:1075: oid 24 updating version 0 to 80 2024-09-16T03:38:08.078 INFO:tasks.rados.rados.0.smithi157.stdout:1075: oid 24 updating version 80 to 81 2024-09-16T03:38:08.078 INFO:tasks.rados.rados.0.smithi157.stdout:1075: oid 24 updating version 81 to 82 2024-09-16T03:38:08.078 INFO:tasks.rados.rados.0.smithi157.stdout:1075: oid 24 updating version 82 to 83 2024-09-16T03:38:08.078 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 83 (ObjNum 181 snap 0 seq_num 181) dirty exists 2024-09-16T03:38:08.079 INFO:tasks.rados.rados.0.smithi157.stdout:1075: left oid 24 (ObjNum 181 snap 0 seq_num 181) 2024-09-16T03:38:08.079 INFO:tasks.rados.rados.0.smithi157.stdout:1073: finishing write tid 3 to smithi15736696-9 2024-09-16T03:38:08.079 INFO:tasks.rados.rados.0.smithi157.stdout:1073: finishing write tid 4 to smithi15736696-9 2024-09-16T03:38:08.079 INFO:tasks.rados.rados.0.smithi157.stdout:1073: finishing write tid 5 to smithi15736696-9 2024-09-16T03:38:08.079 INFO:tasks.rados.rados.0.smithi157.stdout:1073: oid 9 updating version 0 to 35 2024-09-16T03:38:08.079 INFO:tasks.rados.rados.0.smithi157.stdout:1073: oid 9 updating version 35 to 38 2024-09-16T03:38:08.079 INFO:tasks.rados.rados.0.smithi157.stdout:1073: oid 9 version 38 is already newer than 36 2024-09-16T03:38:08.079 INFO:tasks.rados.rados.0.smithi157.stdout:1073: oid 9 version 38 is already newer than 37 2024-09-16T03:38:08.079 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 38 (ObjNum 180 snap 0 seq_num 180) dirty exists 2024-09-16T03:38:08.079 INFO:tasks.rados.rados.0.smithi157.stdout:1073: left oid 9 (ObjNum 180 snap 0 seq_num 180) 2024-09-16T03:38:08.079 INFO:tasks.rados.rados.0.smithi157.stdout:1071: finishing set_chunk to oid 14 2024-09-16T03:38:08.080 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 36 (ObjNum 131 snap 0 seq_num 131) dirty exists 2024-09-16T03:38:08.080 INFO:tasks.rados.rados.0.smithi157.stdout:1074: finishing set_chunk to oid 6 2024-09-16T03:38:08.080 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 48 (ObjNum 164 snap 0 seq_num 164) dirty exists 2024-09-16T03:38:08.080 INFO:tasks.rados.rados.0.smithi157.stdout:1077: finishing write tid 2 to smithi15736696-23 2024-09-16T03:38:08.080 INFO:tasks.rados.rados.0.smithi157.stdout:1080: finishing set_chunk to oid 27 2024-09-16T03:38:08.080 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 287744 length: 1006080 2024-09-16T03:38:08.080 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:38:08.080 INFO:tasks.rados.rados.0.smithi157.stdout:1078: finishing write tid 1 to smithi15736696-31 2024-09-16T03:38:08.080 INFO:tasks.rados.rados.0.smithi157.stdout:1064: done (15 left) 2024-09-16T03:38:08.080 INFO:tasks.rados.rados.0.smithi157.stdout:1066: done (14 left) 2024-09-16T03:38:08.080 INFO:tasks.rados.rados.0.smithi157.stdout:1067: done (13 left) 2024-09-16T03:38:08.081 INFO:tasks.rados.rados.0.smithi157.stdout:1068: done (12 left) 2024-09-16T03:38:08.081 INFO:tasks.rados.rados.0.smithi157.stdout:1069: done (11 left) 2024-09-16T03:38:08.081 INFO:tasks.rados.rados.0.smithi157.stdout:1070: done (10 left) 2024-09-16T03:38:08.081 INFO:tasks.rados.rados.0.smithi157.stdout:1071: done (9 left) 2024-09-16T03:38:08.081 INFO:tasks.rados.rados.0.smithi157.stdout:1072: done (8 left) 2024-09-16T03:38:08.081 INFO:tasks.rados.rados.0.smithi157.stdout:1073: done (7 left) 2024-09-16T03:38:08.081 INFO:tasks.rados.rados.0.smithi157.stdout:1074: done (6 left) 2024-09-16T03:38:08.081 INFO:tasks.rados.rados.0.smithi157.stdout:1075: done (5 left) 2024-09-16T03:38:08.081 INFO:tasks.rados.rados.0.smithi157.stdout:1076: done (4 left) 2024-09-16T03:38:08.081 INFO:tasks.rados.rados.0.smithi157.stdout:1079: done (3 left) 2024-09-16T03:38:08.081 INFO:tasks.rados.rados.0.smithi157.stdout:1080: done (2 left) 2024-09-16T03:38:08.081 INFO:tasks.rados.rados.0.smithi157.stdout:1081: write oid 11 current snap is 0 2024-09-16T03:38:08.082 INFO:tasks.rados.rados.0.smithi157.stdout:1081: seq_num 184 ranges {692488=638288,1917275=606724} 2024-09-16T03:38:08.083 INFO:tasks.rados.rados.0.smithi157.stdout:1081: writing smithi15736696-11 from 692488 to 1330776 tid 1 2024-09-16T03:38:08.084 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:08.083+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:08.090 INFO:tasks.rados.rados.0.smithi157.stdout:1081: writing smithi15736696-11 from 1917275 to 2523999 tid 2 2024-09-16T03:38:08.090 INFO:tasks.rados.rados.0.smithi157.stdout:1082: read oid 46 snap -1 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1082: expect (ObjNum 163 snap 0 seq_num 163) 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1083: read oid 29 snap -1 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1083: expect (ObjNum 106 snap 0 seq_num 106) 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1078: finishing write tid 2 to smithi15736696-31 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1078: finishing write tid 3 to smithi15736696-31 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1084: read oid 20 snap -1 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1084: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1085: delete oid 7 current snap is 0 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1078: finishing write tid 4 to smithi15736696-31 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1078: finishing write tid 5 to smithi15736696-31 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1078: oid 31 updating version 0 to 41 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1078: oid 31 version 41 is already newer than 39 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1078: oid 31 updating version 41 to 42 2024-09-16T03:38:08.091 INFO:tasks.rados.rados.0.smithi157.stdout:1078: oid 31 version 42 is already newer than 40 2024-09-16T03:38:08.092 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 42 (ObjNum 183 snap 0 seq_num 183) dirty exists 2024-09-16T03:38:08.092 INFO:tasks.rados.rados.0.smithi157.stdout:1078: left oid 31 (ObjNum 183 snap 0 seq_num 183) 2024-09-16T03:38:08.092 INFO:tasks.rados.rados.0.smithi157.stdout:1077: finishing write tid 3 to smithi15736696-23 2024-09-16T03:38:08.092 INFO:tasks.rados.rados.0.smithi157.stdout:1077: finishing write tid 4 to smithi15736696-23 2024-09-16T03:38:08.092 INFO:tasks.rados.rados.0.smithi157.stdout:1077: oid 23 updating version 0 to 52 2024-09-16T03:38:08.092 INFO:tasks.rados.rados.0.smithi157.stdout:1077: oid 23 updating version 52 to 53 2024-09-16T03:38:08.092 INFO:tasks.rados.rados.0.smithi157.stdout:1077: oid 23 version 53 is already newer than 50 2024-09-16T03:38:08.092 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 53 (ObjNum 182 snap 0 seq_num 182) dirty exists 2024-09-16T03:38:08.092 INFO:tasks.rados.rados.0.smithi157.stdout:1077: left oid 23 (ObjNum 182 snap 0 seq_num 182) 2024-09-16T03:38:08.093 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:08.093 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:08.093 INFO:tasks.rados.rados.0.smithi157.stdout:1082: expect (ObjNum 163 snap 0 seq_num 163) 2024-09-16T03:38:08.099 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:38:08.155 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:08.153+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:08.180 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:08.206 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:08.206 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:08.215 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:38:08.235 INFO:tasks.rados.rados.0.smithi157.stdout:1084: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-16T03:38:08.256 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:08.254+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:08.281 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:08.320 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:08.320 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:08.329 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:38:08.357 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:08.355+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:08.374 INFO:tasks.rados.rados.0.smithi157.stdout:1081: finishing write tid 1 to smithi15736696-11 2024-09-16T03:38:08.374 INFO:tasks.rados.rados.0.smithi157.stdout:1081: finishing write tid 2 to smithi15736696-11 2024-09-16T03:38:08.374 INFO:tasks.rados.rados.0.smithi157.stdout:1081: finishing write tid 3 to smithi15736696-11 2024-09-16T03:38:08.374 INFO:tasks.rados.rados.0.smithi157.stdout:1081: finishing write tid 4 to smithi15736696-11 2024-09-16T03:38:08.374 INFO:tasks.rados.rados.0.smithi157.stdout:1081: oid 11 updating version 0 to 37 2024-09-16T03:38:08.374 INFO:tasks.rados.rados.0.smithi157.stdout:1081: oid 11 updating version 37 to 39 2024-09-16T03:38:08.374 INFO:tasks.rados.rados.0.smithi157.stdout:1081: oid 11 version 39 is already newer than 38 2024-09-16T03:38:08.374 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 39 (ObjNum 184 snap 0 seq_num 184) dirty exists 2024-09-16T03:38:08.374 INFO:tasks.rados.rados.0.smithi157.stdout:1081: left oid 11 (ObjNum 184 snap 0 seq_num 184) 2024-09-16T03:38:08.374 INFO:tasks.rados.rados.0.smithi157.stdout:1083: expect (ObjNum 106 snap 0 seq_num 106) 2024-09-16T03:38:08.420 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3726327 2024-09-16T03:38:08.425 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:08.433 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:08.434 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:08.443 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:38:08.457 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:08.456+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:08.501 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 606389 2024-09-16T03:38:08.545 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:08.546 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:08.554 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:38:08.561 INFO:tasks.rados.rados.0.smithi157.stdout:1077: done (6 left) 2024-09-16T03:38:08.561 INFO:tasks.rados.rados.0.smithi157.stdout:1078: done (5 left) 2024-09-16T03:38:08.562 INFO:tasks.rados.rados.0.smithi157.stdout:1081: done (4 left) 2024-09-16T03:38:08.562 INFO:tasks.rados.rados.0.smithi157.stdout:1082: done (3 left) 2024-09-16T03:38:08.562 INFO:tasks.rados.rados.0.smithi157.stdout:1083: done (2 left) 2024-09-16T03:38:08.562 INFO:tasks.rados.rados.0.smithi157.stdout:1084: done (1 left) 2024-09-16T03:38:08.562 INFO:tasks.rados.rados.0.smithi157.stdout:1085: done (0 left) 2024-09-16T03:38:08.562 INFO:tasks.rados.rados.0.smithi157.stdout:1086: tier_flush oid 33 2024-09-16T03:38:08.562 INFO:tasks.rados.rados.0.smithi157.stdout:1086: tier_flush oid 33 snap -1 2024-09-16T03:38:08.562 INFO:tasks.rados.rados.0.smithi157.stdout:1087: write (excl) oid 28 current snap is 0 2024-09-16T03:38:08.562 INFO:tasks.rados.rados.0.smithi157.stdout:1087: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-09-16T03:38:08.569 INFO:tasks.rados.rados.0.smithi157.stdout:1087: writing smithi15736696-28 from 726102 to 1514888 tid 1 2024-09-16T03:38:08.575 INFO:tasks.rados.rados.0.smithi157.stdout:1087: writing smithi15736696-28 from 1917886 to 2528279 tid 2 2024-09-16T03:38:08.578 INFO:tasks.rados.rados.0.smithi157.stdout:1087: writing smithi15736696-28 from 3108206 to 3126102 tid 3 2024-09-16T03:38:08.578 INFO:tasks.rados.rados.0.smithi157.stdout:1088: tier_flush oid 18 2024-09-16T03:38:08.578 INFO:tasks.rados.rados.0.smithi157.stdout:1088: tier_flush oid 18 snap -1 2024-09-16T03:38:08.578 INFO:tasks.rados.rados.0.smithi157.stdout:1089: write (excl) oid 47 current snap is 0 2024-09-16T03:38:08.579 INFO:tasks.rados.rados.0.smithi157.stdout:1089: seq_num 187 ranges {742909=464035,1976368=754051} 2024-09-16T03:38:08.584 INFO:tasks.rados.rados.0.smithi157.stdout:1089: writing smithi15736696-47 from 742909 to 1206944 tid 1 2024-09-16T03:38:08.593 INFO:tasks.rados.rados.0.smithi157.stdout:1089: writing smithi15736696-47 from 1976368 to 2730419 tid 2 2024-09-16T03:38:08.593 INFO:tasks.rados.rados.0.smithi157.stdout:1090: read oid 48 snap -1 2024-09-16T03:38:08.594 INFO:tasks.rados.rados.0.smithi157.stdout:1090: expect (ObjNum 175 snap 0 seq_num 175) 2024-09-16T03:38:08.594 INFO:tasks.rados.rados.0.smithi157.stdout:1091: tier_promote oid 30 2024-09-16T03:38:08.594 INFO:tasks.rados.rados.0.smithi157.stdout:1092: read oid 26 snap -1 2024-09-16T03:38:08.594 INFO:tasks.rados.rados.0.smithi157.stdout:1092: expect deleted 2024-09-16T03:38:08.594 INFO:tasks.rados.rados.0.smithi157.stdout:1093: set_chunk oid 40 target oid 2024-09-16T03:38:08.594 INFO:tasks.rados.rados.0.smithi157.stdout:1093: set_chunk oid 40 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:08.594 INFO:tasks.rados.rados.0.smithi157.stdout:1094: write oid 32 current snap is 0 2024-09-16T03:38:08.594 INFO:tasks.rados.rados.0.smithi157.stdout:1094: seq_num 188 ranges {759716=539284,2034850=614062} 2024-09-16T03:38:08.598 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:08.597+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:08.600 INFO:tasks.rados.rados.0.smithi157.stdout:1094: writing smithi15736696-32 from 759716 to 1299000 tid 1 2024-09-16T03:38:08.607 INFO:tasks.rados.rados.0.smithi157.stdout:1094: writing smithi15736696-32 from 2034850 to 2648912 tid 2 2024-09-16T03:38:08.607 INFO:tasks.rados.rados.0.smithi157.stdout:1095: read oid 24 snap -1 2024-09-16T03:38:08.607 INFO:tasks.rados.rados.0.smithi157.stdout:1095: expect (ObjNum 181 snap 0 seq_num 181) 2024-09-16T03:38:08.607 INFO:tasks.rados.rados.0.smithi157.stdout:1096: set_chunk oid 43 target oid 2024-09-16T03:38:08.660 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:08.658+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:08.660 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:08.660 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:08.669 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:38:08.761 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:08.759+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:08.814 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:08.814 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:08.823 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:38:08.867 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:08.865+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:08.915 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:08.913+0000 7fa3886e6640 -1 osd.10 79 handle_osd_map injecting map crc failure 2024-09-16T03:38:08.928 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:08.928 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:08.936 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:38:08.962 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:08.960+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:09.062 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:09.060+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:09.082 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:09.082 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:09.091 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:38:09.164 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:09.162+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:09.235 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:09.235 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:09.245 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:38:09.264 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:09.263+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:09.350 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:09.351 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:09.359 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:38:09.365 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:09.363+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:09.462 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:09.462 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:09.466 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:09.464+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:09.470 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:38:09.567 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:09.565+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:09.573 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:09.573 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:09.581 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:38:09.667 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:09.666+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:09.685 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:09.685 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:09.694 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:38:09.768 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:09.767+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:09.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:38:09.798 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:09.798 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:09.806 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:38:09.869 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:09.867+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:09.885 INFO:tasks.rados.rados.0.smithi157.stdout:1096: set_chunk oid 43 offset: 1463808 length: 458752 target oid 31b35e765994002a8c9d674ea28e0de695c3578a4ad8f1c7df2f28730bc3f7b4 offset: 0 snap -1 2024-09-16T03:38:09.885 INFO:tasks.rados.rados.0.smithi157.stdout:1097: read oid 41 snap -1 2024-09-16T03:38:09.885 INFO:tasks.rados.rados.0.smithi157.stdout:1097: expect (ObjNum 112 snap 0 seq_num 112) 2024-09-16T03:38:09.885 INFO:tasks.rados.rados.0.smithi157.stdout:1098: write (excl) oid 23 current snap is 0 2024-09-16T03:38:09.886 INFO:tasks.rados.rados.0.smithi157.stdout:1098: seq_num 189 ranges {776523=614533,1976979=757720} 2024-09-16T03:38:09.893 INFO:tasks.rados.rados.0.smithi157.stdout:1098: writing smithi15736696-23 from 776523 to 1391056 tid 1 2024-09-16T03:38:09.901 INFO:tasks.rados.rados.0.smithi157.stdout:1098: writing smithi15736696-23 from 1976979 to 2734699 tid 2 2024-09-16T03:38:09.901 INFO:tasks.rados.rados.0.smithi157.stdout:1099: read oid 1 snap -1 2024-09-16T03:38:09.901 INFO:tasks.rados.rados.0.smithi157.stdout:1099: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-16T03:38:09.901 INFO:tasks.rados.rados.0.smithi157.stdout:1100: write (excl) oid 35 current snap is 0 2024-09-16T03:38:09.901 INFO:tasks.rados.rados.0.smithi157.stdout:1095: expect (ObjNum 181 snap 0 seq_num 181) 2024-09-16T03:38:09.904 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1892237 2024-09-16T03:38:09.911 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:09.911 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:09.921 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:38:09.928 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1496751 2024-09-16T03:38:09.969 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:09.968+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:09.973 INFO:tasks.rados.rados.0.smithi157.stdout:1090: expect (ObjNum 175 snap 0 seq_num 175) 2024-09-16T03:38:09.973 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 293543 2024-09-16T03:38:10.006 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:10.027 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:10.027 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:10.036 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:38:10.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:38:10.068 INFO:tasks.rados.rados.0.smithi157.stdout:1093: finishing set_chunk to oid 40 2024-09-16T03:38:10.068 INFO:tasks.rados.rados.0.smithi157.stdout:1093: got expected ENOENT (src dne) 2024-09-16T03:38:10.068 INFO:tasks.rados.rados.0.smithi157.stdout:1094: finishing write tid 1 to smithi15736696-32 2024-09-16T03:38:10.068 INFO:tasks.rados.rados.0.smithi157.stdout:1087: finishing write tid 1 to smithi15736696-28 2024-09-16T03:38:10.068 INFO:tasks.rados.rados.0.smithi157.stdout:1087: finishing write tid 2 to smithi15736696-28 2024-09-16T03:38:10.068 INFO:tasks.rados.rados.0.smithi157.stdout:1094: finishing write tid 2 to smithi15736696-32 2024-09-16T03:38:10.068 INFO:tasks.rados.rados.0.smithi157.stdout:1094: finishing write tid 3 to smithi15736696-32 2024-09-16T03:38:10.068 INFO:tasks.rados.rados.0.smithi157.stdout:1094: finishing write tid 4 to smithi15736696-32 2024-09-16T03:38:10.068 INFO:tasks.rados.rados.0.smithi157.stdout:1094: oid 32 updating version 0 to 15 2024-09-16T03:38:10.068 INFO:tasks.rados.rados.0.smithi157.stdout:1094: oid 32 updating version 15 to 16 2024-09-16T03:38:10.068 INFO:tasks.rados.rados.0.smithi157.stdout:1094: oid 32 version 16 is already newer than 14 2024-09-16T03:38:10.068 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 16 (ObjNum 188 snap 0 seq_num 188) dirty exists 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:1094: left oid 32 (ObjNum 188 snap 0 seq_num 188) 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:1087: finishing write tid 3 to smithi15736696-28 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:1087: finishing write tid 4 to smithi15736696-28 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:1091: got (0) Success 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 34 (ObjNum 166 snap 0 seq_num 166) dirty exists 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:1087: finishing write tid 5 to smithi15736696-28 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:1087: oid 28 updating version 0 to 69 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:1087: oid 28 updating version 69 to 70 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:1087: oid 28 version 70 is already newer than 67 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:1087: oid 28 version 70 is already newer than 68 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 70 (ObjNum 186 snap 0 seq_num 186) dirty exists 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:1087: left oid 28 (ObjNum 186 snap 0 seq_num 186) 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:1089: finishing write tid 1 to smithi15736696-47 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:1089: finishing write tid 2 to smithi15736696-47 2024-09-16T03:38:10.069 INFO:tasks.rados.rados.0.smithi157.stdout:1100: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2024-09-16T03:38:10.071 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:10.070+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:10.075 INFO:tasks.rados.rados.0.smithi157.stdout:1100: writing smithi15736696-35 from 793330 to 1483112 tid 1 2024-09-16T03:38:10.081 INFO:tasks.rados.rados.0.smithi157.stdout:1100: writing smithi15736696-35 from 2035461 to 2653192 tid 2 2024-09-16T03:38:10.085 INFO:tasks.rados.rados.0.smithi157.stdout:1100: writing smithi15736696-35 from 3101545 to 3193330 tid 3 2024-09-16T03:38:10.085 INFO:tasks.rados.rados.0.smithi157.stdout:1087: done (14 left) 2024-09-16T03:38:10.085 INFO:tasks.rados.rados.0.smithi157.stdout:1090: done (13 left) 2024-09-16T03:38:10.085 INFO:tasks.rados.rados.0.smithi157.stdout:1091: done (12 left) 2024-09-16T03:38:10.085 INFO:tasks.rados.rados.0.smithi157.stdout:1092: done (11 left) 2024-09-16T03:38:10.085 INFO:tasks.rados.rados.0.smithi157.stdout:1093: done (10 left) 2024-09-16T03:38:10.085 INFO:tasks.rados.rados.0.smithi157.stdout:1094: done (9 left) 2024-09-16T03:38:10.085 INFO:tasks.rados.rados.0.smithi157.stdout:1095: done (8 left) 2024-09-16T03:38:10.085 INFO:tasks.rados.rados.0.smithi157.stdout:1101: read oid 39 snap -1 2024-09-16T03:38:10.086 INFO:tasks.rados.rados.0.smithi157.stdout:1101: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-16T03:38:10.086 INFO:tasks.rados.rados.0.smithi157.stdout:1102: write (excl) oid 49 current snap is 0 2024-09-16T03:38:10.086 INFO:tasks.rados.rados.0.smithi157.stdout:1089: finishing write tid 3 to smithi15736696-47 2024-09-16T03:38:10.086 INFO:tasks.rados.rados.0.smithi157.stdout:1089: finishing write tid 4 to smithi15736696-47 2024-09-16T03:38:10.086 INFO:tasks.rados.rados.0.smithi157.stdout:1089: oid 47 updating version 0 to 71 2024-09-16T03:38:10.086 INFO:tasks.rados.rados.0.smithi157.stdout:1089: oid 47 updating version 71 to 73 2024-09-16T03:38:10.086 INFO:tasks.rados.rados.0.smithi157.stdout:1089: oid 47 version 73 is already newer than 72 2024-09-16T03:38:10.086 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 73 (ObjNum 187 snap 0 seq_num 187) dirty exists 2024-09-16T03:38:10.086 INFO:tasks.rados.rados.0.smithi157.stdout:1089: left oid 47 (ObjNum 187 snap 0 seq_num 187) 2024-09-16T03:38:10.086 INFO:tasks.rados.rados.0.smithi157.stdout:1088: got (0) Success 2024-09-16T03:38:10.086 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 0 (ObjNum 159 snap 0 seq_num 159) dirty exists 2024-09-16T03:38:10.086 INFO:tasks.rados.rados.0.smithi157.stdout:1102: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-09-16T03:38:10.089 INFO:tasks.rados.rados.0.smithi157.stdout:1102: writing smithi15736696-49 from 410137 to 891521 tid 1 2024-09-16T03:38:10.097 INFO:tasks.rados.rados.0.smithi157.stdout:1102: writing smithi15736696-49 from 1410296 to 2171685 tid 2 2024-09-16T03:38:10.103 INFO:tasks.rados.rados.0.smithi157.stdout:1102: writing smithi15736696-49 from 2728968 to 3210137 tid 3 2024-09-16T03:38:10.103 INFO:tasks.rados.rados.0.smithi157.stdout:1088: done (9 left) 2024-09-16T03:38:10.103 INFO:tasks.rados.rados.0.smithi157.stdout:1089: done (8 left) 2024-09-16T03:38:10.103 INFO:tasks.rados.rados.0.smithi157.stdout:1103: set_chunk oid 34 target oid 2024-09-16T03:38:10.168 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:10.168 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:10.172 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:10.170+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:10.172 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:10.172 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:10.182 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:38:10.185 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:10.186 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:10.204 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:10.204 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:10.222 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:10.222 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:10.239 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:10.239 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:10.255 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:10.255 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:10.273 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:10.272+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:10.308 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:10.308 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:10.324 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:10.324 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:10.327 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:10.327 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:10.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:38:10.340 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:10.357 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:10.357 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:10.374 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:10.372+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:10.374 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:10.375 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:10.382 INFO:tasks.rados.rados.0.smithi157.stdout:1103: set_chunk oid 34 offset: 496128 length: 1322496 target oid 5d4614979cfda19659bd99d1cf0da5c74da1295053f56a96592416f5227d06fb offset: 0 snap -1 2024-09-16T03:38:10.382 INFO:tasks.rados.rados.0.smithi157.stdout:1086: got (0) Success 2024-09-16T03:38:10.382 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 0 (ObjNum 127 snap 0 seq_num 127) dirty exists 2024-09-16T03:38:10.383 INFO:tasks.rados.rados.0.smithi157.stdout:1104: read oid 12 snap -1 2024-09-16T03:38:10.383 INFO:tasks.rados.rados.0.smithi157.stdout:1104: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:10.383 INFO:tasks.rados.rados.0.smithi157.stdout:1086: done (9 left) 2024-09-16T03:38:10.383 INFO:tasks.rados.rados.0.smithi157.stdout:1105: write oid 46 current snap is 0 2024-09-16T03:38:10.383 INFO:tasks.rados.rados.0.smithi157.stdout:1105: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-09-16T03:38:10.387 INFO:tasks.rados.rados.0.smithi157.stdout:1105: writing smithi15736696-46 from 426944 to 983577 tid 1 2024-09-16T03:38:10.392 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:10.393 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:10.393 INFO:tasks.rados.rados.0.smithi157.stdout:1105: writing smithi15736696-46 from 1468778 to 2090178 tid 2 2024-09-16T03:38:10.400 INFO:tasks.rados.rados.0.smithi157.stdout:1105: writing smithi15736696-46 from 2872744 to 3226944 tid 3 2024-09-16T03:38:10.400 INFO:tasks.rados.rados.0.smithi157.stdout:1106: write oid 47 current snap is 0 2024-09-16T03:38:10.400 INFO:tasks.rados.rados.0.smithi157.stdout:1106: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-09-16T03:38:10.405 INFO:tasks.rados.rados.0.smithi157.stdout:1106: writing smithi15736696-47 from 443751 to 1075633 tid 1 2024-09-16T03:38:10.409 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:10.409 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:10.413 INFO:tasks.rados.rados.0.smithi157.stdout:1106: writing smithi15736696-47 from 1810907 to 2575965 tid 2 2024-09-16T03:38:10.417 INFO:tasks.rados.rados.0.smithi157.stdout:1106: writing smithi15736696-47 from 3067461 to 3243751 tid 3 2024-09-16T03:38:10.417 INFO:tasks.rados.rados.0.smithi157.stdout:1107: write oid 30 current snap is 0 2024-09-16T03:38:10.417 INFO:tasks.rados.rados.0.smithi157.stdout:1107: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-09-16T03:38:10.423 INFO:tasks.rados.rados.0.smithi157.stdout:1107: writing smithi15736696-30 from 460558 to 1167689 tid 1 2024-09-16T03:38:10.426 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:10.426 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:10.429 INFO:tasks.rados.rados.0.smithi157.stdout:1107: writing smithi15736696-30 from 1869389 to 2378105 tid 2 2024-09-16T03:38:10.434 INFO:tasks.rados.rados.0.smithi157.stdout:1107: writing smithi15736696-30 from 3094884 to 3260558 tid 3 2024-09-16T03:38:10.435 INFO:tasks.rados.rados.0.smithi157.stdout:1108: read oid 22 snap -1 2024-09-16T03:38:10.435 INFO:tasks.rados.rados.0.smithi157.stdout:1108: expect (ObjNum 152 snap 0 seq_num 152) 2024-09-16T03:38:10.435 INFO:tasks.rados.rados.0.smithi157.stdout:1109: read oid 6 snap -1 2024-09-16T03:38:10.435 INFO:tasks.rados.rados.0.smithi157.stdout:1109: expect (ObjNum 164 snap 0 seq_num 164) 2024-09-16T03:38:10.435 INFO:tasks.rados.rados.0.smithi157.stdout:1097: expect (ObjNum 112 snap 0 seq_num 112) 2024-09-16T03:38:10.443 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:10.444 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:10.446 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2277 4.18 4.b8cec6f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:38:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.022862059000000001, 2024-09-16T03:38:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022886239999999999, 2024-09-16T03:38:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2277 2024-09-16T03:38:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:38:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:38:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422785+0000", 2024-09-16T03:38:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9809999999999999e-06 2024-09-16T03:38:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423586+0000", 2024-09-16T03:38:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080133600000000004 2024-09-16T03:38:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423607+0000", 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1039e-05 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423616+0000", 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0189999999999995e-06 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2278 4.18 4.b8cec6f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:38:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.016293057999999999, 2024-09-16T03:38:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016379205000000001, 2024-09-16T03:38:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2278 2024-09-16T03:38:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:38:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:38:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429352+0000", 2024-09-16T03:38:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.604e-06 2024-09-16T03:38:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429895+0000", 2024-09-16T03:38:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054308899999999996 2024-09-16T03:38:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429909+0000", 2024-09-16T03:38:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3518e-05 2024-09-16T03:38:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429917+0000", 2024-09-16T03:38:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-09-16T03:38:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2279 4.18 4.b8cec6f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:38:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.011604263, 2024-09-16T03:38:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011742358, 2024-09-16T03:38:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2279 2024-09-16T03:38:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:38:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:38:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434041+0000", 2024-09-16T03:38:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.195e-06 2024-09-16T03:38:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434191+0000", 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000150488 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434202+0000", 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0978000000000001e-05 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434208+0000", 2024-09-16T03:38:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2280 4.18 4.b8cec6f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.011425981999999999, 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011615136999999999, 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2280 2024-09-16T03:38:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:38:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:38:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:38:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-09-16T03:38:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434221+0000", 2024-09-16T03:38:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8329999999999998e-06 2024-09-16T03:38:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434225+0000", 2024-09-16T03:38:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7589999999999999e-06 2024-09-16T03:38:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434228+0000", 2024-09-16T03:38:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.971e-06 2024-09-16T03:38:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2281 4.18 4.b8cec6f8 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:38:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.011411687, 2024-09-16T03:38:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011660782, 2024-09-16T03:38:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2281 2024-09-16T03:38:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:38:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:38:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:38:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.96e-07 2024-09-16T03:38:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434234+0000", 2024-09-16T03:38:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4050000000000001e-06 2024-09-16T03:38:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434237+0000", 2024-09-16T03:38:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1939999999999998e-06 2024-09-16T03:38:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434240+0000", 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.74e-06 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2285 4.8 4.f21f628 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434244+0000", 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.011400185, 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011701425, 2024-09-16T03:38:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "no flag points reached", 2024-09-16T03:38:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2285 2024-09-16T03:38:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434244+0000", 2024-09-16T03:38:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434244+0000", 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434244+0000", 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.91e-07 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434246+0000", 2024-09-16T03:38:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0420000000000001e-06 2024-09-16T03:38:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434249+0000", 2024-09-16T03:38:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.999e-06 2024-09-16T03:38:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:38:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 6 2024-09-16T03:38:10.468 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:10.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:38:10.470 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:10.470 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:10.475 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:10.473+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:10.576 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:10.574+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:10.615 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2745407 2024-09-16T03:38:10.620 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:10.620 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:10.620 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:38:10.620 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:38:10.620 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:10.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:38:10.639 INFO:tasks.rados.rados.0.smithi157.stdout:1101: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-16T03:38:10.639 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 181846 2024-09-16T03:38:10.677 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:10.676+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:10.710 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:10.746 INFO:tasks.rados.rados.0.smithi157.stdout:1098: finishing write tid 1 to smithi15736696-23 2024-09-16T03:38:10.746 INFO:tasks.rados.rados.0.smithi157.stdout:1099: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-16T03:38:10.748 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 458642 2024-09-16T03:38:10.777 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:10.778 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:10.775+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:10.779 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:10.779 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:38:10.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:38:10.779 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:10.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.779 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:17 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'7, mlcod=25'5)", 2024-09-16T03:38:10.779 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:52.877437+0000", 2024-09-16T03:38:10.780 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.899118444999999, 2024-09-16T03:38:10.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201252123, 2024-09-16T03:38:10.780 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.780 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.780 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.780 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877437+0000", 2024-09-16T03:38:10.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877437+0000", 2024-09-16T03:38:10.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877439+0000", 2024-09-16T03:38:10.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:38:10.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877446+0000", 2024-09-16T03:38:10.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:38:10.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.877459+0000", 2024-09-16T03:38:10.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3124e-05 2024-09-16T03:38:10.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977610+0000", 2024-09-16T03:38:10.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100150878 2024-09-16T03:38:10.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977758+0000", 2024-09-16T03:38:10.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147644 2024-09-16T03:38:10.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:52.977819+0000", 2024-09-16T03:38:10.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0798999999999998e-05 2024-09-16T03:38:10.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.078644+0000", 2024-09-16T03:38:10.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10082574399999999 2024-09-16T03:38:10.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:53.078689+0000", 2024-09-16T03:38:10.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5016999999999999e-05 2024-09-16T03:38:10.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.785 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.785 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.785 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:39 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'21, mlcod=26'19)", 2024-09-16T03:38:10.785 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:38:10.785 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.181237721000002, 2024-09-16T03:38:10.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30197182500000003, 2024-09-16T03:38:10.786 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.786 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.786 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.786 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:38:10.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:38:10.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595320+0000", 2024-09-16T03:38:10.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.962e-06 2024-09-16T03:38:10.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595336+0000", 2024-09-16T03:38:10.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5568000000000001e-05 2024-09-16T03:38:10.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595349+0000", 2024-09-16T03:38:10.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3203e-05 2024-09-16T03:38:10.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695424+0000", 2024-09-16T03:38:10.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007542899999999 2024-09-16T03:38:10.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695495+0000", 2024-09-16T03:38:10.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0450999999999993e-05 2024-09-16T03:38:10.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695538+0000", 2024-09-16T03:38:10.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3433000000000002e-05 2024-09-16T03:38:10.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897256+0000", 2024-09-16T03:38:10.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20171840699999999 2024-09-16T03:38:10.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897290+0000", 2024-09-16T03:38:10.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3371999999999998e-05 2024-09-16T03:38:10.790 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.790 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.790 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.791 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:40 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'22, mlcod=26'19)", 2024-09-16T03:38:10.791 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695446+0000", 2024-09-16T03:38:10.791 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.081109671999997, 2024-09-16T03:38:10.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20236575600000001, 2024-09-16T03:38:10.791 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.791 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.791 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.791 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695446+0000", 2024-09-16T03:38:10.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695446+0000", 2024-09-16T03:38:10.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695447+0000", 2024-09-16T03:38:10.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:38:10.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695461+0000", 2024-09-16T03:38:10.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3915e-05 2024-09-16T03:38:10.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695470+0000", 2024-09-16T03:38:10.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6959999999999994e-06 2024-09-16T03:38:10.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695478+0000", 2024-09-16T03:38:10.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2050000000000002e-06 2024-09-16T03:38:10.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695933+0000", 2024-09-16T03:38:10.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00045501500000000002 2024-09-16T03:38:10.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695958+0000", 2024-09-16T03:38:10.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4578000000000001e-05 2024-09-16T03:38:10.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897782+0000", 2024-09-16T03:38:10.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20182425700000001 2024-09-16T03:38:10.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897812+0000", 2024-09-16T03:38:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9947e-05 2024-09-16T03:38:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:41 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'23, mlcod=26'19)", 2024-09-16T03:38:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695515+0000", 2024-09-16T03:38:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.081041103999993, 2024-09-16T03:38:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20231417400000001, 2024-09-16T03:38:10.796 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.797 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.797 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.797 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695515+0000", 2024-09-16T03:38:10.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695515+0000", 2024-09-16T03:38:10.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695516+0000", 2024-09-16T03:38:10.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7499999999999998e-07 2024-09-16T03:38:10.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695590+0000", 2024-09-16T03:38:10.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3924999999999994e-05 2024-09-16T03:38:10.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695601+0000", 2024-09-16T03:38:10.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1253999999999999e-05 2024-09-16T03:38:10.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695611+0000", 2024-09-16T03:38:10.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9059999999999997e-06 2024-09-16T03:38:10.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696250+0000", 2024-09-16T03:38:10.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00063903099999999997 2024-09-16T03:38:10.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696263+0000", 2024-09-16T03:38:10.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.276e-05 2024-09-16T03:38:10.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897815+0000", 2024-09-16T03:38:10.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201552074 2024-09-16T03:38:10.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897829+0000", 2024-09-16T03:38:10.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4249e-05 2024-09-16T03:38:10.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.802 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.802 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.802 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:42 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'24, mlcod=26'19)", 2024-09-16T03:38:10.802 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:38:10.802 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.080933196000004, 2024-09-16T03:38:10.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.202223289, 2024-09-16T03:38:10.802 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.803 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.803 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.803 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:38:10.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:38:10.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695623+0000", 2024-09-16T03:38:10.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8599999999999997e-07 2024-09-16T03:38:10.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695638+0000", 2024-09-16T03:38:10.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4844000000000001e-05 2024-09-16T03:38:10.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695645+0000", 2024-09-16T03:38:10.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8510000000000001e-06 2024-09-16T03:38:10.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695651+0000", 2024-09-16T03:38:10.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5600000000000001e-06 2024-09-16T03:38:10.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696535+0000", 2024-09-16T03:38:10.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088443500000000004 2024-09-16T03:38:10.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696550+0000", 2024-09-16T03:38:10.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4788e-05 2024-09-16T03:38:10.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897831+0000", 2024-09-16T03:38:10.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201281234 2024-09-16T03:38:10.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897846+0000", 2024-09-16T03:38:10.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4691e-05 2024-09-16T03:38:10.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.807 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.808 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:43 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'25, mlcod=26'19)", 2024-09-16T03:38:10.808 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695660+0000", 2024-09-16T03:38:10.808 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.080895826000003, 2024-09-16T03:38:10.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.202209369, 2024-09-16T03:38:10.808 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.808 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.808 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.808 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695660+0000", 2024-09-16T03:38:10.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695660+0000", 2024-09-16T03:38:10.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695661+0000", 2024-09-16T03:38:10.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:38:10.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695669+0000", 2024-09-16T03:38:10.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.32e-06 2024-09-16T03:38:10.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695676+0000", 2024-09-16T03:38:10.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3610000000000003e-06 2024-09-16T03:38:10.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695681+0000", 2024-09-16T03:38:10.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0889999999999999e-06 2024-09-16T03:38:10.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696770+0000", 2024-09-16T03:38:10.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001088852 2024-09-16T03:38:10.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.696784+0000", 2024-09-16T03:38:10.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4178999999999999e-05 2024-09-16T03:38:10.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897849+0000", 2024-09-16T03:38:10.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20106433700000001 2024-09-16T03:38:10.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897869+0000", 2024-09-16T03:38:10.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0652000000000001e-05 2024-09-16T03:38:10.813 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.813 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.813 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.813 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:44 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'26, mlcod=26'19)", 2024-09-16T03:38:10.813 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:38:10.814 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.080865158999998, 2024-09-16T03:38:10.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20224351600000001, 2024-09-16T03:38:10.814 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.814 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.814 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.814 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:38:10.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:38:10.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695691+0000", 2024-09-16T03:38:10.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1099999999999996e-07 2024-09-16T03:38:10.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695765+0000", 2024-09-16T03:38:10.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3973000000000005e-05 2024-09-16T03:38:10.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695775+0000", 2024-09-16T03:38:10.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5400000000000001e-06 2024-09-16T03:38:10.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695781+0000", 2024-09-16T03:38:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8369999999999999e-06 2024-09-16T03:38:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698245+0000", 2024-09-16T03:38:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024633480000000002 2024-09-16T03:38:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698260+0000", 2024-09-16T03:38:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4859e-05 2024-09-16T03:38:10.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897880+0000", 2024-09-16T03:38:10.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199619933 2024-09-16T03:38:10.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897934+0000", 2024-09-16T03:38:10.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4515000000000002e-05 2024-09-16T03:38:10.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.819 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.819 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.819 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:45 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'27, mlcod=26'19)", 2024-09-16T03:38:10.819 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.698152+0000", 2024-09-16T03:38:10.819 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.078403983000001, 2024-09-16T03:38:10.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200415062, 2024-09-16T03:38:10.819 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.819 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.819 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.820 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698152+0000", 2024-09-16T03:38:10.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698152+0000", 2024-09-16T03:38:10.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698153+0000", 2024-09-16T03:38:10.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.525e-06 2024-09-16T03:38:10.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698165+0000", 2024-09-16T03:38:10.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1861e-05 2024-09-16T03:38:10.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698178+0000", 2024-09-16T03:38:10.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3275000000000001e-05 2024-09-16T03:38:10.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698186+0000", 2024-09-16T03:38:10.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4139999999999997e-06 2024-09-16T03:38:10.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897000+0000", 2024-09-16T03:38:10.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.198814149 2024-09-16T03:38:10.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897035+0000", 2024-09-16T03:38:10.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5039000000000001e-05 2024-09-16T03:38:10.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.898524+0000", 2024-09-16T03:38:10.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001488952 2024-09-16T03:38:10.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.898567+0000", 2024-09-16T03:38:10.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2846999999999998e-05 2024-09-16T03:38:10.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.824 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.824 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.825 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:46 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'28, mlcod=26'19)", 2024-09-16T03:38:10.825 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.698626+0000", 2024-09-16T03:38:10.825 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.077929716, 2024-09-16T03:38:10.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20067976400000001, 2024-09-16T03:38:10.825 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.825 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.825 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.825 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698626+0000", 2024-09-16T03:38:10.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698626+0000", 2024-09-16T03:38:10.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698627+0000", 2024-09-16T03:38:10.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.395e-06 2024-09-16T03:38:10.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698641+0000", 2024-09-16T03:38:10.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:38:10.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.698653+0000", 2024-09-16T03:38:10.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1846e-05 2024-09-16T03:38:10.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.798792+0000", 2024-09-16T03:38:10.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100139397 2024-09-16T03:38:10.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897344+0000", 2024-09-16T03:38:10.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098552085999999997 2024-09-16T03:38:10.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897361+0000", 2024-09-16T03:38:10.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6716000000000001e-05 2024-09-16T03:38:10.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.899266+0000", 2024-09-16T03:38:10.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019056520000000001 2024-09-16T03:38:10.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.899306+0000", 2024-09-16T03:38:10.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9322000000000001e-05 2024-09-16T03:38:10.830 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.830 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.830 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.830 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:99 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'73, mlcod=28'66)", 2024-09-16T03:38:10.831 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:38:10.831 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 71.685263918000004, 2024-09-16T03:38:10.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201204829, 2024-09-16T03:38:10.831 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.831 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.831 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.831 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:38:10.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:38:10.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091292+0000", 2024-09-16T03:38:10.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9800000000000004e-07 2024-09-16T03:38:10.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091329+0000", 2024-09-16T03:38:10.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7039000000000002e-05 2024-09-16T03:38:10.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.091336+0000", 2024-09-16T03:38:10.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:38:10.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191408+0000", 2024-09-16T03:38:10.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100072733 2024-09-16T03:38:10.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191470+0000", 2024-09-16T03:38:10.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2087000000000004e-05 2024-09-16T03:38:10.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.191504+0000", 2024-09-16T03:38:10.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3453e-05 2024-09-16T03:38:10.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292441+0000", 2024-09-16T03:38:10.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100937374 2024-09-16T03:38:10.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:59.292497+0000", 2024-09-16T03:38:10.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5365999999999998e-05 2024-09-16T03:38:10.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.836 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.836 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.5.21:12 3.2 e45/31 3:43460c1e:::smithi15736696-20 low_tier:head v 45'36, mlcod=45'34)", 2024-09-16T03:38:10.836 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.014759+0000", 2024-09-16T03:38:10.836 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.761796685, 2024-09-16T03:38:10.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20057897499999999, 2024-09-16T03:38:10.837 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.837 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.837 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.837 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014759+0000", 2024-09-16T03:38:10.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014759+0000", 2024-09-16T03:38:10.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014762+0000", 2024-09-16T03:38:10.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7429999999999998e-06 2024-09-16T03:38:10.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014774+0000", 2024-09-16T03:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1790999999999999e-05 2024-09-16T03:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.014790+0000", 2024-09-16T03:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6409000000000002e-05 2024-09-16T03:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.114865+0000", 2024-09-16T03:38:10.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100074861 2024-09-16T03:38:10.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.114929+0000", 2024-09-16T03:38:10.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3769000000000006e-05 2024-09-16T03:38:10.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.114947+0000", 2024-09-16T03:38:10.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8085999999999999e-05 2024-09-16T03:38:10.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.215300+0000", 2024-09-16T03:38:10.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100353422 2024-09-16T03:38:10.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.215338+0000", 2024-09-16T03:38:10.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7894e-05 2024-09-16T03:38:10.841 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.842 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.842 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1226 3.2 e54/31 3:6f0527c9:::7f01c73e0f253c35d7776d11c46f7e5d9edb0e9d6a760d7305f052eba477b6c2:head v 54'82, mlcod=54'80)", 2024-09-16T03:38:10.842 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.472805+0000", 2024-09-16T03:38:10.842 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.303750674, 2024-09-16T03:38:10.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20253328100000001, 2024-09-16T03:38:10.842 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.842 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.842 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.842 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.472805+0000", 2024-09-16T03:38:10.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.472805+0000", 2024-09-16T03:38:10.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.472807+0000", 2024-09-16T03:38:10.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9489999999999999e-06 2024-09-16T03:38:10.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.473596+0000", 2024-09-16T03:38:10.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078904500000000005 2024-09-16T03:38:10.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.473614+0000", 2024-09-16T03:38:10.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7832000000000001e-05 2024-09-16T03:38:10.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.573713+0000", 2024-09-16T03:38:10.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100098627 2024-09-16T03:38:10.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.673879+0000", 2024-09-16T03:38:10.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100166088 2024-09-16T03:38:10.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.673918+0000", 2024-09-16T03:38:10.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9035999999999997e-05 2024-09-16T03:38:10.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675283+0000", 2024-09-16T03:38:10.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001365668 2024-09-16T03:38:10.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675338+0000", 2024-09-16T03:38:10.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5036e-05 2024-09-16T03:38:10.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.847 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.847 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1588 3.2 e61/31 3:57b21e08:::c9c74db4cc3b05b40ab52a9ed9c665f10e27723063858665cdba49de2698a01f:head v 61'105, mlcod=59'103)", 2024-09-16T03:38:10.847 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:46.917498+0000", 2024-09-16T03:38:10.847 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.859057342, 2024-09-16T03:38:10.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206346578, 2024-09-16T03:38:10.848 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.848 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.848 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.848 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.917498+0000", 2024-09-16T03:38:10.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.917498+0000", 2024-09-16T03:38:10.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.917502+0000", 2024-09-16T03:38:10.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0230000000000003e-06 2024-09-16T03:38:10.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.921109+0000", 2024-09-16T03:38:10.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036070149999999999 2024-09-16T03:38:10.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.921129+0000", 2024-09-16T03:38:10.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9741999999999998e-05 2024-09-16T03:38:10.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.921139+0000", 2024-09-16T03:38:10.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5079999999999994e-06 2024-09-16T03:38:10.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.121366+0000", 2024-09-16T03:38:10.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20022730599999999 2024-09-16T03:38:10.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.121386+0000", 2024-09-16T03:38:10.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9856000000000001e-05 2024-09-16T03:38:10.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.123808+0000", 2024-09-16T03:38:10.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024219979999999999 2024-09-16T03:38:10.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.123845+0000", 2024-09-16T03:38:10.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7129999999999999e-05 2024-09-16T03:38:10.852 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.852 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.853 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1947 4.8 4:146f84f0:::smithi15736696-6:head [write 2063988~686387] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:10.853 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124065+0000", 2024-09-16T03:38:10.853 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.652490362, 2024-09-16T03:38:10.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203590252, 2024-09-16T03:38:10.853 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.853 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.853 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.854 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:10.854 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:10.854 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:10.854 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1947 2024-09-16T03:38:10.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.854 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124065+0000", 2024-09-16T03:38:10.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124065+0000", 2024-09-16T03:38:10.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124067+0000", 2024-09-16T03:38:10.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.66e-06 2024-09-16T03:38:10.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124770+0000", 2024-09-16T03:38:10.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070269099999999997 2024-09-16T03:38:10.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124783+0000", 2024-09-16T03:38:10.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2989999999999999e-05 2024-09-16T03:38:10.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124791+0000", 2024-09-16T03:38:10.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5159999999999994e-06 2024-09-16T03:38:10.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124848+0000", 2024-09-16T03:38:10.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6787999999999999e-05 2024-09-16T03:38:10.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124933+0000", 2024-09-16T03:38:10.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4753999999999995e-05 2024-09-16T03:38:10.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:10.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.125040+0000", 2024-09-16T03:38:10.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106827 2024-09-16T03:38:10.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127507+0000", 2024-09-16T03:38:10.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024672119999999999 2024-09-16T03:38:10.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:10.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127880+0000", 2024-09-16T03:38:10.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000373237 2024-09-16T03:38:10.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.131974+0000", 2024-09-16T03:38:10.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040936230000000002 2024-09-16T03:38:10.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.226110+0000", 2024-09-16T03:38:10.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094136475999999997 2024-09-16T03:38:10.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.327424+0000", 2024-09-16T03:38:10.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101313954 2024-09-16T03:38:10.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.327503+0000", 2024-09-16T03:38:10.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8850000000000006e-05 2024-09-16T03:38:10.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.327656+0000", 2024-09-16T03:38:10.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152674 2024-09-16T03:38:10.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.862 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.862 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1948 4.8 4:146f84f0:::smithi15736696-6:head [setxattr _header (55) in=7b,truncate 2756348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:10.862 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:38:10.862 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.6517489150000007, 2024-09-16T03:38:10.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40326984799999999, 2024-09-16T03:38:10.862 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.862 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.863 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:10.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:10.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:10.863 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1948 2024-09-16T03:38:10.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.863 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:38:10.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:38:10.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124808+0000", 2024-09-16T03:38:10.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9099999999999991e-07 2024-09-16T03:38:10.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124813+0000", 2024-09-16T03:38:10.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9599999999999999e-06 2024-09-16T03:38:10.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124821+0000", 2024-09-16T03:38:10.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0490000000000004e-06 2024-09-16T03:38:10.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124827+0000", 2024-09-16T03:38:10.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3339999999999998e-06 2024-09-16T03:38:10.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126739+0000", 2024-09-16T03:38:10.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00191147 2024-09-16T03:38:10.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126830+0000", 2024-09-16T03:38:10.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1751999999999998e-05 2024-09-16T03:38:10.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:10.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127005+0000", 2024-09-16T03:38:10.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174227 2024-09-16T03:38:10.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127981+0000", 2024-09-16T03:38:10.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097622799999999999 2024-09-16T03:38:10.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:10.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.128193+0000", 2024-09-16T03:38:10.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021176999999999999 2024-09-16T03:38:10.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.135281+0000", 2024-09-16T03:38:10.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070882150000000001 2024-09-16T03:38:10.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.427785+0000", 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29250390599999998 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527932+0000", 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10014759500000001 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527980+0000", 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7725000000000001e-05 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528077+0000", 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6626000000000006e-05 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1949 4.8 4:146f84f0:::smithi15736696-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.6517181710000006, 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40346921899999999, 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.870 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:10.871 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:10.871 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:10.871 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:10.871 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1949 2024-09-16T03:38:10.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.871 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:38:10.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124841+0000", 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5890000000000001e-06 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124847+0000", 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0820000000000004e-06 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124852+0000", 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1070000000000004e-06 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127536+0000", 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026844619999999999 2024-09-16T03:38:10.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127585+0000", 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8915999999999997e-05 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528126+0000", 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40054061899999999 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528193+0000", 2024-09-16T03:38:10.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6717999999999994e-05 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528307+0000", 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011413600000000001 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1979 4.18 4:1f63731d:::smithi15736696-30:head [write 789962~534747] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1407108590000004, 2024-09-16T03:38:10.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.506754182, 2024-09-16T03:38:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1979 2024-09-16T03:38:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635850+0000", 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6670000000000004e-06 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636429+0000", 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000579745 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636441+0000", 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1965e-05 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736594+0000", 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100152291 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837326+0000", 2024-09-16T03:38:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10073201399999999 2024-09-16T03:38:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837437+0000", 2024-09-16T03:38:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111914 2024-09-16T03:38:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837625+0000", 2024-09-16T03:38:10.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000187391 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939449+0000", 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10182371599999999 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939476+0000", 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7117000000000001e-05 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142297+0000", 2024-09-16T03:38:10.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20282091499999999 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142347+0000", 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0800000000000002e-05 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142485+0000", 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013799999999999999 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142526+0000", 2024-09-16T03:38:10.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0269999999999999e-05 2024-09-16T03:38:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142599+0000", 2024-09-16T03:38:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3376999999999999e-05 2024-09-16T03:38:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1980 4.18 4:1f63731d:::smithi15736696-30:head [write 1780952~690056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:38:10.882 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0399096980000007, 2024-09-16T03:38:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40615011400000001, 2024-09-16T03:38:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1980 2024-09-16T03:38:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:38:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:38:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736656+0000", 2024-09-16T03:38:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0339999999999999e-05 2024-09-16T03:38:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737092+0000", 2024-09-16T03:38:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435957 2024-09-16T03:38:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737102+0000", 2024-09-16T03:38:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7860000000000008e-06 2024-09-16T03:38:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737107+0000", 2024-09-16T03:38:10.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839282+0000", 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10217523100000001 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839389+0000", 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010619399999999999 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:10.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839567+0000", 2024-09-16T03:38:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017886200000000001 2024-09-16T03:38:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939485+0000", 2024-09-16T03:38:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099917168000000001 2024-09-16T03:38:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943655+0000", 2024-09-16T03:38:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004170287 2024-09-16T03:38:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142373+0000", 2024-09-16T03:38:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19871839599999999 2024-09-16T03:38:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142431+0000", 2024-09-16T03:38:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7880999999999997e-05 2024-09-16T03:38:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142627+0000", 2024-09-16T03:38:10.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019580500000000001 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142723+0000", 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6209e-05 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142796+0000", 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2978000000000005e-05 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1981 4.18 e75/68 ondisk, result = 0)", 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.843824+0000", 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9327318239999993, 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.198132161, 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.843824+0000", 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.843824+0000", 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.843825+0000", 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7700000000000004e-07 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.843828+0000", 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.608e-06 2024-09-16T03:38:10.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.843834+0000", 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7130000000000002e-06 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943923+0000", 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100088977 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041938+0000", 2024-09-16T03:38:10.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098014991999999995 2024-09-16T03:38:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041947+0000", 2024-09-16T03:38:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9819999999999997e-06 2024-09-16T03:38:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041956+0000", 2024-09-16T03:38:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.312e-06 2024-09-16T03:38:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1979 4.18 e75/68 ondisk, result = 0)", 2024-09-16T03:38:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.943958+0000", 2024-09-16T03:38:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8325976829999995, 2024-09-16T03:38:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19864527000000001, 2024-09-16T03:38:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:10.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943958+0000", 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943958+0000", 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943959+0000", 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0860000000000001e-06 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:10.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943963+0000", 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.014e-06 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943971+0000", 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0520000000000002e-06 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.044051+0000", 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10008026 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142479+0000", 2024-09-16T03:38:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098427833000000006 2024-09-16T03:38:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142483+0000", 2024-09-16T03:38:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.45e-06 2024-09-16T03:38:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142603+0000", 2024-09-16T03:38:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120575 2024-09-16T03:38:10.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:10.898 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:10.898 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:10.899 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:38:10.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:38:10.901 INFO:tasks.rados.rados.0.smithi157.stdout:1098: finishing write tid 2 to smithi15736696-23 2024-09-16T03:38:10.901 INFO:tasks.rados.rados.0.smithi157.stdout:1096: finishing set_chunk to oid 43 2024-09-16T03:38:10.902 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 54 (ObjNum 169 snap 0 seq_num 169) dirty exists 2024-09-16T03:38:10.902 INFO:tasks.rados.rados.0.smithi157.stdout:1110: read oid 45 snap -1 2024-09-16T03:38:10.902 INFO:tasks.rados.rados.0.smithi157.stdout:1110: expect deleted 2024-09-16T03:38:10.902 INFO:tasks.rados.rados.0.smithi157.stdout:1096: done (14 left) 2024-09-16T03:38:10.902 INFO:tasks.rados.rados.0.smithi157.stdout:1097: done (13 left) 2024-09-16T03:38:10.902 INFO:tasks.rados.rados.0.smithi157.stdout:1099: done (12 left) 2024-09-16T03:38:10.902 INFO:tasks.rados.rados.0.smithi157.stdout:1101: done (11 left) 2024-09-16T03:38:10.902 INFO:tasks.rados.rados.0.smithi157.stdout:1111: delete oid 16 current snap is 0 2024-09-16T03:38:10.902 INFO:tasks.rados.rados.0.smithi157.stdout:1102: finishing write tid 1 to smithi15736696-49 2024-09-16T03:38:10.902 INFO:tasks.rados.rados.0.smithi157.stdout:1102: finishing write tid 2 to smithi15736696-49 2024-09-16T03:38:10.902 INFO:tasks.rados.rados.0.smithi157.stdout:1102: finishing write tid 3 to smithi15736696-49 2024-09-16T03:38:10.902 INFO:tasks.rados.rados.0.smithi157.stdout:1102: finishing write tid 4 to smithi15736696-49 2024-09-16T03:38:10.902 INFO:tasks.rados.rados.0.smithi157.stdout:1104: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:10.903 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:10.896+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:10.913 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1896206 2024-09-16T03:38:10.919 INFO:tasks.rados.rados.0.smithi157.stdout:1098: finishing write tid 3 to smithi15736696-23 2024-09-16T03:38:10.919 INFO:tasks.rados.rados.0.smithi157.stdout:1098: finishing write tid 4 to smithi15736696-23 2024-09-16T03:38:10.919 INFO:tasks.rados.rados.0.smithi157.stdout:1098: oid 23 updating version 0 to 54 2024-09-16T03:38:10.919 INFO:tasks.rados.rados.0.smithi157.stdout:1098: oid 23 updating version 54 to 56 2024-09-16T03:38:10.919 INFO:tasks.rados.rados.0.smithi157.stdout:1098: oid 23 version 56 is already newer than 55 2024-09-16T03:38:10.919 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 56 (ObjNum 189 snap 0 seq_num 189) dirty exists 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:1098: left oid 23 (ObjNum 189 snap 0 seq_num 189) 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:1100: finishing write tid 1 to smithi15736696-35 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:1100: finishing write tid 2 to smithi15736696-35 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:1102: finishing write tid 5 to smithi15736696-49 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:1102: oid 49 updating version 0 to 45 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:1102: oid 49 updating version 45 to 46 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:1102: oid 49 version 46 is already newer than 44 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:1102: oid 49 version 46 is already newer than 43 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 46 (ObjNum 191 snap 0 seq_num 191) dirty exists 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:1102: left oid 49 (ObjNum 191 snap 0 seq_num 191) 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:1103: finishing set_chunk to oid 34 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 34 set_chunk returned error code -95 offset: 496128 length: 1322496 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 0 (ObjNum 126 snap 0 seq_num 126) dirty exists 2024-09-16T03:38:10.920 INFO:tasks.rados.rados.0.smithi157.stdout:1108: expect (ObjNum 152 snap 0 seq_num 152) 2024-09-16T03:38:10.921 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:10.947 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:10.994 INFO:tasks.rados.rados.0.smithi157.stdout:1100: finishing write tid 3 to smithi15736696-35 2024-09-16T03:38:10.994 INFO:tasks.rados.rados.0.smithi157.stdout:1100: finishing write tid 4 to smithi15736696-35 2024-09-16T03:38:10.994 INFO:tasks.rados.rados.0.smithi157.stdout:1100: finishing write tid 5 to smithi15736696-35 2024-09-16T03:38:10.994 INFO:tasks.rados.rados.0.smithi157.stdout:1100: oid 35 updating version 0 to 75 2024-09-16T03:38:10.995 INFO:tasks.rados.rados.0.smithi157.stdout:1100: oid 35 updating version 75 to 76 2024-09-16T03:38:10.995 INFO:tasks.rados.rados.0.smithi157.stdout:1100: oid 35 updating version 76 to 77 2024-09-16T03:38:10.995 INFO:tasks.rados.rados.0.smithi157.stdout:1100: oid 35 version 77 is already newer than 74 2024-09-16T03:38:10.995 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 77 (ObjNum 190 snap 0 seq_num 190) dirty exists 2024-09-16T03:38:10.995 INFO:tasks.rados.rados.0.smithi157.stdout:1100: left oid 35 (ObjNum 190 snap 0 seq_num 190) 2024-09-16T03:38:10.995 INFO:tasks.rados.rados.0.smithi157.stdout:1098: done (11 left) 2024-09-16T03:38:10.995 INFO:tasks.rados.rados.0.smithi157.stdout:1100: done (10 left) 2024-09-16T03:38:10.995 INFO:tasks.rados.rados.0.smithi157.stdout:1102: done (9 left) 2024-09-16T03:38:10.995 INFO:tasks.rados.rados.0.smithi157.stdout:1103: done (8 left) 2024-09-16T03:38:10.995 INFO:tasks.rados.rados.0.smithi157.stdout:1104: done (7 left) 2024-09-16T03:38:10.995 INFO:tasks.rados.rados.0.smithi157.stdout:1108: done (6 left) 2024-09-16T03:38:10.995 INFO:tasks.rados.rados.0.smithi157.stdout:1111: done (5 left) 2024-09-16T03:38:10.995 INFO:tasks.rados.rados.0.smithi157.stdout:1112: write (excl) oid 25 current snap is 0 2024-09-16T03:38:10.996 INFO:tasks.rados.rados.0.smithi157.stdout:1109: expect (ObjNum 164 snap 0 seq_num 164) 2024-09-16T03:38:11.000 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:10.998+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:11.050 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:38:11.051 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:11.060 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:38:11.069 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2588139 2024-09-16T03:38:11.101 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:11.099+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:11.129 INFO:tasks.rados.rados.0.smithi157.stdout:1105: finishing write tid 1 to smithi15736696-46 2024-09-16T03:38:11.129 INFO:tasks.rados.rados.0.smithi157.stdout:1105: finishing write tid 2 to smithi15736696-46 2024-09-16T03:38:11.129 INFO:tasks.rados.rados.0.smithi157.stdout:1105: finishing write tid 3 to smithi15736696-46 2024-09-16T03:38:11.129 INFO:tasks.rados.rados.0.smithi157.stdout:1105: finishing write tid 4 to smithi15736696-46 2024-09-16T03:38:11.129 INFO:tasks.rados.rados.0.smithi157.stdout:1105: finishing write tid 5 to smithi15736696-46 2024-09-16T03:38:11.129 INFO:tasks.rados.rados.0.smithi157.stdout:1105: oid 46 updating version 0 to 79 2024-09-16T03:38:11.129 INFO:tasks.rados.rados.0.smithi157.stdout:1105: oid 46 updating version 79 to 80 2024-09-16T03:38:11.129 INFO:tasks.rados.rados.0.smithi157.stdout:1105: oid 46 version 80 is already newer than 78 2024-09-16T03:38:11.130 INFO:tasks.rados.rados.0.smithi157.stdout:1105: oid 46 updating version 80 to 81 2024-09-16T03:38:11.130 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 81 (ObjNum 192 snap 0 seq_num 192) dirty exists 2024-09-16T03:38:11.130 INFO:tasks.rados.rados.0.smithi157.stdout:1105: left oid 46 (ObjNum 192 snap 0 seq_num 192) 2024-09-16T03:38:11.130 INFO:tasks.rados.rados.0.smithi157.stdout:1106: finishing write tid 1 to smithi15736696-47 2024-09-16T03:38:11.130 INFO:tasks.rados.rados.0.smithi157.stdout:1112: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-09-16T03:38:11.134 INFO:tasks.rados.rados.0.smithi157.stdout:1112: writing smithi15736696-25 from 494172 to 951801 tid 1 2024-09-16T03:38:11.140 INFO:tasks.rados.rados.0.smithi157.stdout:1112: writing smithi15736696-25 from 1586353 to 2098738 tid 2 2024-09-16T03:38:11.147 INFO:tasks.rados.rados.0.smithi157.stdout:1112: writing smithi15736696-25 from 2749730 to 3294172 tid 3 2024-09-16T03:38:11.147 INFO:tasks.rados.rados.0.smithi157.stdout:1105: done (5 left) 2024-09-16T03:38:11.147 INFO:tasks.rados.rados.0.smithi157.stdout:1109: done (4 left) 2024-09-16T03:38:11.147 INFO:tasks.rados.rados.0.smithi157.stdout:1113: set_chunk oid 43 target oid 2024-09-16T03:38:11.201 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:11.199+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:11.207 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:11.207 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:11.207 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:38:11.207 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:38:11.207 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:11.216 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:38:11.303 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:11.302+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:11.344 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:11.342+0000 7f12e4a1e640 -1 osd.5 81 handle_osd_map injecting map crc failure 2024-09-16T03:38:11.369 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:38:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:38:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:33 4.e 4:773b934b:::smithi15736696-9:head [write 534456~718345,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.050039+0000", 2024-09-16T03:38:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.315781579000003, 2024-09-16T03:38:11.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40763112099999999, 2024-09-16T03:38:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-09-16T03:38:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050039+0000", 2024-09-16T03:38:11.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050039+0000", 2024-09-16T03:38:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050042+0000", 2024-09-16T03:38:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5009999999999999e-06 2024-09-16T03:38:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050806+0000", 2024-09-16T03:38:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076374199999999998 2024-09-16T03:38:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050819+0000", 2024-09-16T03:38:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3692999999999999e-05 2024-09-16T03:38:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050829+0000", 2024-09-16T03:38:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8630000000000003e-06 2024-09-16T03:38:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050917+0000", 2024-09-16T03:38:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8196999999999996e-05 2024-09-16T03:38:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.050922+0000", 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.831e-06 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153032+0000", 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102109992 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153123+0000", 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1037000000000003e-05 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.153218+0000", 2024-09-16T03:38:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5047e-05 2024-09-16T03:38:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156107+0000", 2024-09-16T03:38:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028893299999999998 2024-09-16T03:38:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255606+0000", 2024-09-16T03:38:11.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099498438999999994 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457591+0000", 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201984996 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457631+0000", 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0110000000000001e-05 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.457670+0000", 2024-09-16T03:38:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9342999999999999e-05 2024-09-16T03:38:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:34 4.e 4:773b934b:::smithi15736696-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:38:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.311409560999998, 2024-09-16T03:38:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50346527299999999, 2024-09-16T03:38:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.377 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054413+0000", 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0669999999999999e-06 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054525+0000", 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111462 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054538+0000", 2024-09-16T03:38:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3341e-05 2024-09-16T03:38:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054545+0000", 2024-09-16T03:38:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.584e-06 2024-09-16T03:38:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054608+0000", 2024-09-16T03:38:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2953999999999994e-05 2024-09-16T03:38:11.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054613+0000", 2024-09-16T03:38:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6110000000000001e-06 2024-09-16T03:38:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154852+0000", 2024-09-16T03:38:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10023838 2024-09-16T03:38:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154915+0000", 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3138000000000004e-05 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155001+0000", 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5730999999999996e-05 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156166+0000", 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001165485 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255909+0000", 2024-09-16T03:38:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099742558999999995 2024-09-16T03:38:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557800+0000", 2024-09-16T03:38:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30189130600000003 2024-09-16T03:38:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557836+0000", 2024-09-16T03:38:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6566000000000002e-05 2024-09-16T03:38:11.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557877+0000", 2024-09-16T03:38:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0089000000000002e-05 2024-09-16T03:38:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:35 4.e 4:773b934b:::smithi15736696-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:38:11.384 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.31126123, 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50344320499999995, 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:38:11.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:38:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:38:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:38:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054564+0000", 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5710000000000002e-06 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054569+0000", 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0490000000000003e-06 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054578+0000", 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054639+0000", 2024-09-16T03:38:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1061000000000004e-05 2024-09-16T03:38:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054644+0000", 2024-09-16T03:38:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999996e-06 2024-09-16T03:38:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155607+0000", 2024-09-16T03:38:11.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100963372 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155652+0000", 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155722+0000", 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0130999999999997e-05 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.157921+0000", 2024-09-16T03:38:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021990159999999998 2024-09-16T03:38:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255940+0000", 2024-09-16T03:38:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098018569999999999 2024-09-16T03:38:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557930+0000", 2024-09-16T03:38:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.301990333 2024-09-16T03:38:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557963+0000", 2024-09-16T03:38:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2348000000000003e-05 2024-09-16T03:38:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558003+0000", 2024-09-16T03:38:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:38:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:36 4.e 4:773b934b:::smithi15736696-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:38:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.311222540999999, 2024-09-16T03:38:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50353811800000003, 2024-09-16T03:38:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-16T03:38:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.392 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054599+0000", 2024-09-16T03:38:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:38:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054601+0000", 2024-09-16T03:38:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4360000000000001e-06 2024-09-16T03:38:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054606+0000", 2024-09-16T03:38:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1019999999999996e-06 2024-09-16T03:38:11.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054610+0000", 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.681e-06 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054662+0000", 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1545000000000002e-05 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054664+0000", 2024-09-16T03:38:11.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1919999999999999e-06 2024-09-16T03:38:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155903+0000", 2024-09-16T03:38:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101239068 2024-09-16T03:38:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155970+0000", 2024-09-16T03:38:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6934999999999995e-05 2024-09-16T03:38:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558029+0000", 2024-09-16T03:38:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40205888899999997 2024-09-16T03:38:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558078+0000", 2024-09-16T03:38:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9842999999999997e-05 2024-09-16T03:38:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558136+0000", 2024-09-16T03:38:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7985000000000001e-05 2024-09-16T03:38:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:508 4.e 4:773b934b:::smithi15736696-9:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.822186+0000", 2024-09-16T03:38:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.543634517999998, 2024-09-16T03:38:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41573648800000002, 2024-09-16T03:38:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 508 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822186+0000", 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822186+0000", 2024-09-16T03:38:11.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822189+0000", 2024-09-16T03:38:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6010000000000002e-06 2024-09-16T03:38:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822198+0000", 2024-09-16T03:38:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0990000000000005e-06 2024-09-16T03:38:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.822213+0000", 2024-09-16T03:38:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4481000000000001e-05 2024-09-16T03:38:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922294+0000", 2024-09-16T03:38:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10008130799999999 2024-09-16T03:38:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922365+0000", 2024-09-16T03:38:11.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0973e-05 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922457+0000", 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2393999999999997e-05 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.025017+0000", 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102560259 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.025625+0000", 2024-09-16T03:38:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060773999999999995 2024-09-16T03:38:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.025985+0000", 2024-09-16T03:38:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036012000000000001 2024-09-16T03:38:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.237775+0000", 2024-09-16T03:38:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.211790115 2024-09-16T03:38:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.237859+0000", 2024-09-16T03:38:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3387000000000006e-05 2024-09-16T03:38:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.237923+0000", 2024-09-16T03:38:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4010999999999994e-05 2024-09-16T03:38:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:797 4.e 4:765e9179:::smithi15736696-44:head [write 457157~535817,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.999730+0000", 2024-09-16T03:38:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.366090987, 2024-09-16T03:38:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40739979599999998, 2024-09-16T03:38:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.405 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 797 2024-09-16T03:38:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.999730+0000", 2024-09-16T03:38:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.999730+0000", 2024-09-16T03:38:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.999732+0000", 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6479999999999999e-06 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000380+0000", 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064818899999999997 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000392+0000", 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2757e-05 2024-09-16T03:38:11.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000399+0000", 2024-09-16T03:38:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.511e-06 2024-09-16T03:38:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000462+0000", 2024-09-16T03:38:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2874999999999997e-05 2024-09-16T03:38:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000564+0000", 2024-09-16T03:38:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010215100000000001 2024-09-16T03:38:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.000746+0000", 2024-09-16T03:38:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018165600000000001 2024-09-16T03:38:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.103052+0000", 2024-09-16T03:38:11.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102305858 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205188+0000", 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10213634100000001 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.406992+0000", 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20180416300000001 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.407098+0000", 2024-09-16T03:38:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010646599999999999 2024-09-16T03:38:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.407130+0000", 2024-09-16T03:38:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1180999999999998e-05 2024-09-16T03:38:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:799 4.e 4:765e9179:::smithi15736696-44:head [setxattr _header (56) in=7b,truncate 2857157] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:11.411 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:38:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.359370314000003, 2024-09-16T03:38:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50104159800000003, 2024-09-16T03:38:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 799 2024-09-16T03:38:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:38:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:38:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:38:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:38:11.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006454+0000", 2024-09-16T03:38:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5739999999999998e-06 2024-09-16T03:38:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006458+0000", 2024-09-16T03:38:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6179999999999997e-06 2024-09-16T03:38:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006462+0000", 2024-09-16T03:38:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4999999999999999e-06 2024-09-16T03:38:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204504+0000", 2024-09-16T03:38:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19804184999999999 2024-09-16T03:38:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204702+0000", 2024-09-16T03:38:11.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019868199999999999 2024-09-16T03:38:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204888+0000", 2024-09-16T03:38:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185701 2024-09-16T03:38:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.206549+0000", 2024-09-16T03:38:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016612790000000001 2024-09-16T03:38:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507371+0000", 2024-09-16T03:38:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30082157399999998 2024-09-16T03:38:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507414+0000", 2024-09-16T03:38:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3053999999999998e-05 2024-09-16T03:38:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507451+0000", 2024-09-16T03:38:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6847999999999997e-05 2024-09-16T03:38:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507492+0000", 2024-09-16T03:38:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1267999999999999e-05 2024-09-16T03:38:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:800 4.e 4:765e9179:::smithi15736696-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:38:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.359346979000001, 2024-09-16T03:38:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50119258099999997, 2024-09-16T03:38:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 800 2024-09-16T03:38:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.419 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:38:11.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1899999999999998e-07 2024-09-16T03:38:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006478+0000", 2024-09-16T03:38:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:38:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006484+0000", 2024-09-16T03:38:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6620000000000002e-06 2024-09-16T03:38:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006488+0000", 2024-09-16T03:38:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8700000000000002e-06 2024-09-16T03:38:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205228+0000", 2024-09-16T03:38:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19873985899999999 2024-09-16T03:38:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205283+0000", 2024-09-16T03:38:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5878999999999998e-05 2024-09-16T03:38:11.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507527+0000", 2024-09-16T03:38:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302243924 2024-09-16T03:38:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507593+0000", 2024-09-16T03:38:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6061000000000003e-05 2024-09-16T03:38:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507666+0000", 2024-09-16T03:38:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3059000000000006e-05 2024-09-16T03:38:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:845 4.e 4:773b934b:::smithi15736696-9:head [write 2577441~346944] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.631766+0000", 2024-09-16T03:38:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.734055036000001, 2024-09-16T03:38:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40847320799999998, 2024-09-16T03:38:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.424 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 845 2024-09-16T03:38:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631766+0000", 2024-09-16T03:38:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631766+0000", 2024-09-16T03:38:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.631767+0000", 2024-09-16T03:38:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2699999999999999e-07 2024-09-16T03:38:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733448+0000", 2024-09-16T03:38:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101681228 2024-09-16T03:38:11.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733458+0000", 2024-09-16T03:38:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0573e-05 2024-09-16T03:38:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.733473+0000", 2024-09-16T03:38:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4953999999999999e-05 2024-09-16T03:38:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.737776+0000", 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0043023940000000002 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.737868+0000", 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2223000000000001e-05 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.738066+0000", 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000197951 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.740101+0000", 2024-09-16T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020355379999999999 2024-09-16T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.939218+0000", 2024-09-16T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19911701800000001 2024-09-16T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040070+0000", 2024-09-16T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100851658 2024-09-16T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040179+0000", 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109085 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:27.040239+0000", 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9858999999999997e-05 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:90 3.0 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:38:11.430 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.809209372, 2024-09-16T03:38:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62288137399999999, 2024-09-16T03:38:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:38:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556614+0000", 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3439999999999999e-06 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556621+0000", 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7440000000000001e-06 2024-09-16T03:38:11.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556632+0000", 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1725e-05 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556640+0000", 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0239999999999997e-06 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556709+0000", 2024-09-16T03:38:11.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8787000000000004e-05 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556718+0000", 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6920000000000008e-06 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758133+0000", 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201414977 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758273+0000", 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140684 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758479+0000", 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020508599999999999 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862766+0000", 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104287952 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.878363+0000", 2024-09-16T03:38:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015596601999999999 2024-09-16T03:38:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960000+0000", 2024-09-16T03:38:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081636630000000002 2024-09-16T03:38:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960162+0000", 2024-09-16T03:38:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000162597 2024-09-16T03:38:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179366+0000", 2024-09-16T03:38:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21920357100000001 2024-09-16T03:38:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179411+0000", 2024-09-16T03:38:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5461999999999999e-05 2024-09-16T03:38:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179493+0000", 2024-09-16T03:38:11.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1496999999999996e-05 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:91 3.0 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.809164414, 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70460932600000004, 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556657+0000", 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556663+0000", 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6579999999999999e-06 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556670+0000", 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3189999999999997e-06 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556675+0000", 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000002e-06 2024-09-16T03:38:11.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656929+0000", 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10025371299999999 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656944+0000", 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5460000000000001e-05 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861436+0000", 2024-09-16T03:38:11.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.204491969 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861676+0000", 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024051899999999999 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861828+0000", 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015110199999999999 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862775+0000", 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094718600000000003 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079130+0000", 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21635549200000001 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179557+0000", 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10042638399999999 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179972+0000", 2024-09-16T03:38:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041574 2024-09-16T03:38:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261034+0000", 2024-09-16T03:38:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081061604999999995 2024-09-16T03:38:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261142+0000", 2024-09-16T03:38:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107688 2024-09-16T03:38:11.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261266+0000", 2024-09-16T03:38:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124082 2024-09-16T03:38:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:94 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:38:11.445 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.809138289, 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70479513699999996, 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556685+0000", 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4279999999999999e-06 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556689+0000", 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.709e-06 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556693+0000", 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2939999999999999e-06 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657036+0000", 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10034209600000001 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657054+0000", 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8162999999999999e-05 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862279+0000", 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20522560200000001 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862380+0000", 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101178 2024-09-16T03:38:11.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862509+0000", 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128909 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.863063+0000", 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055327600000000001 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079172+0000", 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21610910899999999 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179932+0000", 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100760575 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179997+0000", 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5018999999999995e-05 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261318+0000", 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081320838000000006 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261385+0000", 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7115999999999999e-05 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261478+0000", 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2360000000000003e-05 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.452 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1742 3.0 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head [write 0~1659392] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T03:38:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:38:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.809109889, 2024-09-16T03:38:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62317617400000003, 2024-09-16T03:38:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1742 2024-09-16T03:38:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556713+0000", 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7209999999999999e-06 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558531+0000", 2024-09-16T03:38:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018184480000000001 2024-09-16T03:38:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558539+0000", 2024-09-16T03:38:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0830000000000004e-06 2024-09-16T03:38:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558546+0000", 2024-09-16T03:38:11.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3369999999999996e-06 2024-09-16T03:38:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759148+0000", 2024-09-16T03:38:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20060235500000001 2024-09-16T03:38:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759230+0000", 2024-09-16T03:38:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2391000000000004e-05 2024-09-16T03:38:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759329+0000", 2024-09-16T03:38:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9147000000000005e-05 2024-09-16T03:38:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862771+0000", 2024-09-16T03:38:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103441085 2024-09-16T03:38:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960102+0000", 2024-09-16T03:38:11.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097331259000000003 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960218+0000", 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011581100000000001 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079074+0000", 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118856436 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179758+0000", 2024-09-16T03:38:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100683884 2024-09-16T03:38:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179800+0000", 2024-09-16T03:38:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1928000000000002e-05 2024-09-16T03:38:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179887+0000", 2024-09-16T03:38:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7288999999999997e-05 2024-09-16T03:38:11.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:89 3.0 3:1c07e18b:::d3cef63b99e391058a88d444a56984ba6fc25b22efbc98e24ba274246c542d3c:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.029707393000001, 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50388158999999999, 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:38:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:38:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336115+0000", 2024-09-16T03:38:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.866e-06 2024-09-16T03:38:11.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336123+0000", 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6990000000000003e-06 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336136+0000", 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3342e-05 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336143+0000", 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2099999999999998e-06 2024-09-16T03:38:11.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336214+0000", 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1538000000000005e-05 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336342+0000", 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127408 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336593+0000", 2024-09-16T03:38:11.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025150500000000001 2024-09-16T03:38:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337284+0000", 2024-09-16T03:38:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069060900000000001 2024-09-16T03:38:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338068+0000", 2024-09-16T03:38:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078438900000000003 2024-09-16T03:38:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338107+0000", 2024-09-16T03:38:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8858e-05 2024-09-16T03:38:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538164+0000", 2024-09-16T03:38:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200056971 2024-09-16T03:38:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839843+0000", 2024-09-16T03:38:11.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30167921599999997 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839905+0000", 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1542000000000001e-05 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839995+0000", 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0437000000000002e-05 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:98 3.0 3:04dfaaf9:::3de53e1d584ced9b0f970855867dbc9a14b3fc90962aa20a81c1ed25ff4b961f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.028379491999999, 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70310345299999999, 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:38:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337442+0000", 2024-09-16T03:38:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0729999999999999e-06 2024-09-16T03:38:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337447+0000", 2024-09-16T03:38:11.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3259999999999997e-06 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337457+0000", 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337462+0000", 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0780000000000003e-06 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337511+0000", 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337574+0000", 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3114999999999995e-05 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337723+0000", 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014882299999999999 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.339039+0000", 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001315524 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538225+0000", 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19918607199999999 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638838+0000", 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100613466 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.739726+0000", 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100887854 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040319+0000", 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30059265699999999 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040398+0000", 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8852999999999999e-05 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040545+0000", 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147236 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:101 3.0 3:090a0a71:::0ecec3e4e81d718f460a9826fb4cef2398bf4c951ee653be84827f33946eef5f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9292087460000005, 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60435370600000005, 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.473 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436614+0000", 2024-09-16T03:38:11.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.844e-06 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436623+0000", 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0739999999999999e-06 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436636+0000", 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2629999999999999e-05 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436643+0000", 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7489999999999998e-06 2024-09-16T03:38:11.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536906+0000", 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10026262599999999 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536979+0000", 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3311000000000003e-05 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537109+0000", 2024-09-16T03:38:11.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013015800000000001 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538241+0000", 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001131726 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638904+0000", 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066240799999999 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.940200+0000", 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30129597299999999 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040604+0000", 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040478999999999 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040759+0000", 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015445500000000001 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040824+0000", 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5633e-05 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040966+0000", 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014132899999999999 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:102 3.0 3:17742417:::632266e231c4c385ace473a8f4464285cf620fbc8799f68869353e2aa4660567:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:11.479 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9291619200000003, 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.604540823, 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:38:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436660+0000", 2024-09-16T03:38:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:38:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436663+0000", 2024-09-16T03:38:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1599999999999998e-06 2024-09-16T03:38:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436670+0000", 2024-09-16T03:38:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.742e-06 2024-09-16T03:38:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436675+0000", 2024-09-16T03:38:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3839999999999996e-06 2024-09-16T03:38:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537499+0000", 2024-09-16T03:38:11.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100823987 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537602+0000", 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102837 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537793+0000", 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019123499999999999 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638962+0000", 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101168944 2024-09-16T03:38:11.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.840035+0000", 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201073481 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040706+0000", 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200670606 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041009+0000", 2024-09-16T03:38:11.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030271600000000002 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041087+0000", 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8310000000000001e-05 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041127+0000", 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041200+0000", 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2381999999999999e-05 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:103 3.0 3:28447854:::c811732c201b76a96e1532c4d9a18d4f4f2a8ddec3bf79945689a1a62c8a5a25:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9291166709999992, 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60467526100000002, 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.486 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436705+0000", 2024-09-16T03:38:11.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.23e-07 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436719+0000", 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3957999999999999e-05 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436725+0000", 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6100000000000002e-06 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436730+0000", 2024-09-16T03:38:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0609999999999997e-06 2024-09-16T03:38:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538257+0000", 2024-09-16T03:38:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10152739500000001 2024-09-16T03:38:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538314+0000", 2024-09-16T03:38:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7355000000000002e-05 2024-09-16T03:38:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538435+0000", 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120967 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638968+0000", 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100532994 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040662+0000", 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40169386299999998 2024-09-16T03:38:11.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041059+0000", 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039723900000000002 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041247+0000", 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018756900000000001 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041286+0000", 2024-09-16T03:38:11.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8971999999999999e-05 2024-09-16T03:38:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041318+0000", 2024-09-16T03:38:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1628999999999998e-05 2024-09-16T03:38:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041379+0000", 2024-09-16T03:38:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1926000000000003e-05 2024-09-16T03:38:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.492 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(osd.11.23:98 3.0 e74/69 ondisk, result = 0)", 2024-09-16T03:38:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:38:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8273003029999995, 2024-09-16T03:38:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50203460700000002, 2024-09-16T03:38:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:38:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:38:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:38:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0899999999999997e-07 2024-09-16T03:38:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538523+0000", 2024-09-16T03:38:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-09-16T03:38:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538527+0000", 2024-09-16T03:38:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0539999999999996e-06 2024-09-16T03:38:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538529+0000", 2024-09-16T03:38:11.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7420000000000002e-06 2024-09-16T03:38:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040301+0000", 2024-09-16T03:38:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50177146399999994 2024-09-16T03:38:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040312+0000", 2024-09-16T03:38:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1039999999999999e-05 2024-09-16T03:38:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040555+0000", 2024-09-16T03:38:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243212 2024-09-16T03:38:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(osd.11.23:103 3.0 e75/69 ondisk, result = 0)", 2024-09-16T03:38:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.540430+0000", 2024-09-16T03:38:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8253908350000003, 2024-09-16T03:38:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50024338400000001, 2024-09-16T03:38:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.497 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540430+0000", 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540430+0000", 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540431+0000", 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3600000000000002e-07 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540434+0000", 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5149999999999998e-06 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540442+0000", 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6340000000000001e-06 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540448+0000", 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.285e-06 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040646+0000", 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50019782800000001 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040655+0000", 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3429999999999996e-06 2024-09-16T03:38:11.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040673+0000", 2024-09-16T03:38:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8842999999999999e-05 2024-09-16T03:38:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.501 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.501 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:11.503 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:38:11.504 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:11.501+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:11.504 INFO:tasks.rados.rados.0.smithi157.stdout:1113: set_chunk oid 43 offset: 140288 length: 2537984 target oid bdcecef1995ea82ddc9b3b1d470e745bea2c1808ed30897ed7b0d48669adb55d offset: 0 snap -1 2024-09-16T03:38:11.504 INFO:tasks.rados.rados.0.smithi157.stdout:1114: read oid 28 snap -1 2024-09-16T03:38:11.504 INFO:tasks.rados.rados.0.smithi157.stdout:1114: expect (ObjNum 186 snap 0 seq_num 186) 2024-09-16T03:38:11.504 INFO:tasks.rados.rados.0.smithi157.stdout:1115: read oid 38 snap -1 2024-09-16T03:38:11.504 INFO:tasks.rados.rados.0.smithi157.stdout:1115: expect (ObjNum 157 snap 0 seq_num 157) 2024-09-16T03:38:11.504 INFO:tasks.rados.rados.0.smithi157.stdout:1116: tier_promote oid 7 2024-09-16T03:38:11.505 INFO:tasks.rados.rados.0.smithi157.stdout:1117: read oid 4 snap -1 2024-09-16T03:38:11.505 INFO:tasks.rados.rados.0.smithi157.stdout:1117: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:11.505 INFO:tasks.rados.rados.0.smithi157.stdout:1118: read oid 42 snap -1 2024-09-16T03:38:11.505 INFO:tasks.rados.rados.0.smithi157.stdout:1118: expect (ObjNum 142 snap 0 seq_num 142) 2024-09-16T03:38:11.505 INFO:tasks.rados.rados.0.smithi157.stdout:1119: set_chunk oid 23 target oid 2024-09-16T03:38:11.603 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:11.602+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:11.652 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:38:11.652 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:11.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:38:11.704 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:11.702+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:11.771 INFO:tasks.rados.rados.0.smithi157.stdout:1119: set_chunk oid 23 offset: 1476608 length: 419328 target oid e0df31149537e9cc545635066892bf2c5a436b27ccfe01e8edbc5ee94468f6fc offset: 0 snap -1 2024-09-16T03:38:11.771 INFO:tasks.rados.rados.0.smithi157.stdout:1120: read oid 12 snap -1 2024-09-16T03:38:11.771 INFO:tasks.rados.rados.0.smithi157.stdout:1120: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:11.771 INFO:tasks.rados.rados.0.smithi157.stdout:1121: tier_evict oid 49 2024-09-16T03:38:11.772 INFO:tasks.rados.rados.0.smithi157.stdout:1121: tier_evict oid 49 snap -1 2024-09-16T03:38:11.772 INFO:tasks.rados.rados.0.smithi157.stdout:1107: finishing write tid 1 to smithi15736696-30 2024-09-16T03:38:11.772 INFO:tasks.rados.rados.0.smithi157.stdout:1122: read oid 22 snap -1 2024-09-16T03:38:11.772 INFO:tasks.rados.rados.0.smithi157.stdout:1122: expect (ObjNum 152 snap 0 seq_num 152) 2024-09-16T03:38:11.772 INFO:tasks.rados.rados.0.smithi157.stdout:1123: read oid 34 snap -1 2024-09-16T03:38:11.772 INFO:tasks.rados.rados.0.smithi157.stdout:1123: expect (ObjNum 126 snap 0 seq_num 126) 2024-09-16T03:38:11.772 INFO:tasks.rados.rados.0.smithi157.stdout:1124: set_chunk oid 50 target oid 2024-09-16T03:38:11.804 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:11.803+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:11.810 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:11.810 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:11.811 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:38:11.811 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:38:11.811 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:11.819 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:38:11.906 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:11.904+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:11.929 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:11.929 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:38:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:38:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4338.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:38:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.829465579000001, 2024-09-16T03:38:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30502727499999999, 2024-09-16T03:38:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4338", 2024-09-16T03:38:11.930 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3073758685", 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097101+0000", 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.915e-06 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097111+0000", 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0589e-05 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097130+0000", 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8958999999999999e-05 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097141+0000", 2024-09-16T03:38:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0616e-05 2024-09-16T03:38:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097227+0000", 2024-09-16T03:38:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6322000000000005e-05 2024-09-16T03:38:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:38:11.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097232+0000", 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2059999999999998e-06 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200235+0000", 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103002945 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200318+0000", 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2515999999999994e-05 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,10", 2024-09-16T03:38:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200429+0000", 2024-09-16T03:38:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011158999999999999 2024-09-16T03:38:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201478+0000", 2024-09-16T03:38:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00104893 2024-09-16T03:38:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201942+0000", 2024-09-16T03:38:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046354700000000003 2024-09-16T03:38:11.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402048+0000", 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20010616000000001 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402089+0000", 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0762e-05 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402125+0000", 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6217999999999998e-05 2024-09-16T03:38:11.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:15 4.a 4:51338165:::smithi15736696-4:head [setxattr _header (55) in=7b,truncate 2050421] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:38:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.929593646000001, 2024-09-16T03:38:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30116835400000003, 2024-09-16T03:38:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.937 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 15 2024-09-16T03:38:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:38:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996970+0000", 2024-09-16T03:38:11.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996971+0000", 2024-09-16T03:38:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.066e-06 2024-09-16T03:38:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996984+0000", 2024-09-16T03:38:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3246e-05 2024-09-16T03:38:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:11.996994+0000", 2024-09-16T03:38:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.713e-06 2024-09-16T03:38:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097070+0000", 2024-09-16T03:38:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100076313 2024-09-16T03:38:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097141+0000", 2024-09-16T03:38:11.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1298999999999996e-05 2024-09-16T03:38:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097205+0000", 2024-09-16T03:38:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3907999999999999e-05 2024-09-16T03:38:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.097300+0000", 2024-09-16T03:38:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5209000000000003e-05 2024-09-16T03:38:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197645+0000", 2024-09-16T03:38:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10034422899999999 2024-09-16T03:38:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.197837+0000", 2024-09-16T03:38:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000192454 2024-09-16T03:38:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298049+0000", 2024-09-16T03:38:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100211868 2024-09-16T03:38:11.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298082+0000", 2024-09-16T03:38:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3467000000000003e-05 2024-09-16T03:38:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.298138+0000", 2024-09-16T03:38:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5581999999999997e-05 2024-09-16T03:38:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:73 4.a 4:532814f2:::smithi15736696-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:38:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.615303443000002, 2024-09-16T03:38:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.362642662, 2024-09-16T03:38:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73 2024-09-16T03:38:11.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311261+0000", 2024-09-16T03:38:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.562e-06 2024-09-16T03:38:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311925+0000", 2024-09-16T03:38:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066385499999999998 2024-09-16T03:38:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311937+0000", 2024-09-16T03:38:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1881e-05 2024-09-16T03:38:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311943+0000", 2024-09-16T03:38:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6999999999999996e-06 2024-09-16T03:38:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311988+0000", 2024-09-16T03:38:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5358000000000002e-05 2024-09-16T03:38:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312070+0000", 2024-09-16T03:38:11.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1735000000000003e-05 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312167+0000", 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6685e-05 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.313661+0000", 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001494324 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.414759+0000", 2024-09-16T03:38:11.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10109850300000001 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673821+0000", 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.25906127400000001 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673858+0000", 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7620999999999998e-05 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673902+0000", 2024-09-16T03:38:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4163999999999999e-05 2024-09-16T03:38:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:303 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:38:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.514844613000001, 2024-09-16T03:38:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50956986000000004, 2024-09-16T03:38:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.950 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 303 2024-09-16T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:38:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411721+0000", 2024-09-16T03:38:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0200000000000001e-06 2024-09-16T03:38:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411729+0000", 2024-09-16T03:38:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1240000000000005e-06 2024-09-16T03:38:11.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411744+0000", 2024-09-16T03:38:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099000000000001e-05 2024-09-16T03:38:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411751+0000", 2024-09-16T03:38:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.029e-06 2024-09-16T03:38:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511892+0000", 2024-09-16T03:38:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100140879 2024-09-16T03:38:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511961+0000", 2024-09-16T03:38:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9379000000000001e-05 2024-09-16T03:38:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.613647+0000", 2024-09-16T03:38:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101685994 2024-09-16T03:38:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.614294+0000", 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064692799999999996 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.714962+0000", 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066765 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921157+0000", 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206195463 2024-09-16T03:38:11.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921233+0000", 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5446999999999998e-05 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921288+0000", 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5847999999999998e-05 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:404 4.3 e45/32 4:cf22405f:::smithi15736696-5:head v 45'15, mlcod=42'13)", 2024-09-16T03:38:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.309931+0000", 2024-09-16T03:38:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.616632025999998, 2024-09-16T03:38:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30105416800000001, 2024-09-16T03:38:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309931+0000", 2024-09-16T03:38:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309931+0000", 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309933+0000", 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0099999999999998e-06 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309943+0000", 2024-09-16T03:38:11.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4050000000000006e-06 2024-09-16T03:38:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.309959+0000", 2024-09-16T03:38:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6365e-05 2024-09-16T03:38:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.410036+0000", 2024-09-16T03:38:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100077477 2024-09-16T03:38:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610252+0000", 2024-09-16T03:38:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20021509700000001 2024-09-16T03:38:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610290+0000", 2024-09-16T03:38:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8068999999999998e-05 2024-09-16T03:38:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610947+0000", 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065726700000000001 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.610985+0000", 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8478e-05 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:503 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:11.961 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.104770061000004, 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60521621599999997, 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 503 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821794+0000", 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821798+0000", 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2030000000000002e-06 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821804+0000", 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821810+0000", 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4249999999999997e-06 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922212+0000", 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040228800000001 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922307+0000", 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5089999999999999e-05 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.126007+0000", 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20369931499999999 2024-09-16T03:38:11.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.127605+0000", 2024-09-16T03:38:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001598213 2024-09-16T03:38:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.327263+0000", 2024-09-16T03:38:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199657905 2024-09-16T03:38:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426900+0000", 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099637217 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426963+0000", 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.427009+0000", 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6533000000000003e-05 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:852 4.a 4:503187b3:::smithi15736696-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.562374+0000", 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.364188968000001, 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29976180800000002, 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 852 2024-09-16T03:38:11.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562374+0000", 2024-09-16T03:38:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562374+0000", 2024-09-16T03:38:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562375+0000", 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562376+0000", 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4360000000000001e-06 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.562379+0000", 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3610000000000001e-06 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.662476+0000", 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10009655100000001 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759421+0000", 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096945552000000004 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.759482+0000", 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0058e-05 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.862023+0000", 2024-09-16T03:38:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10254131599999999 2024-09-16T03:38:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.862072+0000", 2024-09-16T03:38:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9104999999999998e-05 2024-09-16T03:38:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.862136+0000", 2024-09-16T03:38:11.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4098999999999997e-05 2024-09-16T03:38:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1194 4.a 4:503187b3:::smithi15736696-34:head [write 2736097~502305] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:38:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.066371+0000", 2024-09-16T03:38:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.860191929000003, 2024-09-16T03:38:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30263757600000002, 2024-09-16T03:38:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.973 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1194 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066371+0000", 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066371+0000", 2024-09-16T03:38:11.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066373+0000", 2024-09-16T03:38:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.216e-06 2024-09-16T03:38:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066794+0000", 2024-09-16T03:38:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042050699999999998 2024-09-16T03:38:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.066812+0000", 2024-09-16T03:38:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8462999999999999e-05 2024-09-16T03:38:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166903+0000", 2024-09-16T03:38:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100090889 2024-09-16T03:38:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.166970+0000", 2024-09-16T03:38:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6148000000000004e-05 2024-09-16T03:38:11.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.167075+0000", 2024-09-16T03:38:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010519200000000001 2024-09-16T03:38:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.167251+0000", 2024-09-16T03:38:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176405 2024-09-16T03:38:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.169014+0000", 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001762516 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.264039+0000", 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095025709 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.368934+0000", 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104894738 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.368975+0000", 2024-09-16T03:38:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0605999999999997e-05 2024-09-16T03:38:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.369009+0000", 2024-09-16T03:38:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4187000000000003e-05 2024-09-16T03:38:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.979 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1227 4.a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v18,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:38:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:38:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.250593797000001, 2024-09-16T03:38:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.44160531400000003, 2024-09-16T03:38:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1227 2024-09-16T03:38:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:38:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:38:11.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675972+0000", 2024-09-16T03:38:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7269999999999999e-06 2024-09-16T03:38:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675981+0000", 2024-09-16T03:38:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.166e-06 2024-09-16T03:38:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675996+0000", 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4901e-05 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676005+0000", 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4579999999999993e-06 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676081+0000", 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6157999999999998e-05 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676188+0000", 2024-09-16T03:38:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106871 2024-09-16T03:38:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.016594+0000", 2024-09-16T03:38:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34040622300000001 2024-09-16T03:38:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017089+0000", 2024-09-16T03:38:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049523899999999997 2024-09-16T03:38:11.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017779+0000", 2024-09-16T03:38:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069032200000000005 2024-09-16T03:38:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117504+0000", 2024-09-16T03:38:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099724207999999995 2024-09-16T03:38:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117543+0000", 2024-09-16T03:38:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9208999999999997e-05 2024-09-16T03:38:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117575+0000", 2024-09-16T03:38:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1832000000000003e-05 2024-09-16T03:38:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.986 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1523 4.a 4:50107449:::smithi15736696-27:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.563552471000001, 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.410522742, 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1523 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:38:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:38:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363012+0000", 2024-09-16T03:38:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.269e-06 2024-09-16T03:38:11.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363016+0000", 2024-09-16T03:38:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8920000000000002e-06 2024-09-16T03:38:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363027+0000", 2024-09-16T03:38:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1079e-05 2024-09-16T03:38:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463160+0000", 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10013267100000001 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463297+0000", 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013717699999999999 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463381+0000", 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3701000000000004e-05 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.773533+0000", 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31015295300000001 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1677 4.13 e64/50 4:c8492bfc:::smithi15736696-42:head v 64'37, mlcod=60'35)", 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.136995+0000", 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.78956818, 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30289115700000002, 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.991 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.136995+0000", 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.136995+0000", 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.136997+0000", 2024-09-16T03:38:11.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2450000000000001e-06 2024-09-16T03:38:11.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.137887+0000", 2024-09-16T03:38:11.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088958899999999998 2024-09-16T03:38:11.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.137904+0000", 2024-09-16T03:38:11.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6843000000000001e-05 2024-09-16T03:38:11.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.237997+0000", 2024-09-16T03:38:11.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100093631 2024-09-16T03:38:11.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.338117+0000", 2024-09-16T03:38:11.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100120158 2024-09-16T03:38:11.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:11.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.338133+0000", 2024-09-16T03:38:11.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5624000000000002e-05 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.439797+0000", 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101664168 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.439886+0000", 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8899000000000004e-05 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:11.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1690 4.a 4:51338165:::smithi15736696-4:head [assert-version v63,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:38:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.476279804000001, 2024-09-16T03:38:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52002349699999995, 2024-09-16T03:38:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:11.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1690 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:38:11.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450286+0000", 2024-09-16T03:38:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-16T03:38:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450293+0000", 2024-09-16T03:38:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5089999999999996e-06 2024-09-16T03:38:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450308+0000", 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4955e-05 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450316+0000", 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7079999999999997e-06 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450386+0000", 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9715999999999995e-05 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:11.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450478+0000", 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2610999999999998e-05 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.665802+0000", 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21532342700000001 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668724+0000", 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029220600000000002 2024-09-16T03:38:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668758+0000", 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.434e-05 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970243+0000", 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30148501 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970278+0000", 2024-09-16T03:38:12.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4965000000000002e-05 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970307+0000", 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8968000000000001e-05 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1654 4.12 4:4a9d235d:::smithi15736696-1:head [write 652138~461367 in=461367b,stat out=16b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.384526312999999, 2024-09-16T03:38:12.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42790198699999998, 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1654 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:38:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542039+0000", 2024-09-16T03:38:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0250000000000001e-06 2024-09-16T03:38:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542682+0000", 2024-09-16T03:38:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064305099999999995 2024-09-16T03:38:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542698+0000", 2024-09-16T03:38:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6469000000000001e-05 2024-09-16T03:38:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542710+0000", 2024-09-16T03:38:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1173e-05 2024-09-16T03:38:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542767+0000", 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7584000000000003e-05 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542775+0000", 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6149999999999999e-06 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666597+0000", 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123821815 2024-09-16T03:38:12.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666745+0000", 2024-09-16T03:38:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148675 2024-09-16T03:38:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:38:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666935+0000", 2024-09-16T03:38:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018963899999999999 2024-09-16T03:38:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969207+0000", 2024-09-16T03:38:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302272031 2024-09-16T03:38:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969374+0000", 2024-09-16T03:38:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016686699999999999 2024-09-16T03:38:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969533+0000", 2024-09-16T03:38:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015909299999999999 2024-09-16T03:38:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969560+0000", 2024-09-16T03:38:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7148000000000002e-05 2024-09-16T03:38:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969631+0000", 2024-09-16T03:38:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1174000000000007e-05 2024-09-16T03:38:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969685+0000", 2024-09-16T03:38:12.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3613000000000003e-05 2024-09-16T03:38:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969764+0000", 2024-09-16T03:38:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9281000000000002e-05 2024-09-16T03:38:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969939+0000", 2024-09-16T03:38:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473400000000001 2024-09-16T03:38:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1655 4.12 4:4a9d235d:::smithi15736696-1:head [write 1574940~677198 in=677198b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:38:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.383834194999999, 2024-09-16T03:38:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42773147700000003, 2024-09-16T03:38:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.011 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1655 2024-09-16T03:38:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:38:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:38:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542730+0000", 2024-09-16T03:38:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.085e-06 2024-09-16T03:38:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543338+0000", 2024-09-16T03:38:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060826100000000002 2024-09-16T03:38:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543351+0000", 2024-09-16T03:38:12.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2789000000000001e-05 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543360+0000", 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1970000000000002e-06 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543420+0000", 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9562999999999999e-05 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543431+0000", 2024-09-16T03:38:12.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0866999999999999e-05 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.768963+0000", 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.225532487 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769029+0000", 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5881000000000002e-05 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:38:12.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769134+0000", 2024-09-16T03:38:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104923 2024-09-16T03:38:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969639+0000", 2024-09-16T03:38:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20050517900000001 2024-09-16T03:38:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969978+0000", 2024-09-16T03:38:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033848599999999998 2024-09-16T03:38:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970206+0000", 2024-09-16T03:38:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022867300000000001 2024-09-16T03:38:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970248+0000", 2024-09-16T03:38:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1433999999999997e-05 2024-09-16T03:38:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970299+0000", 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0677999999999998e-05 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970338+0000", 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9646e-05 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970377+0000", 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8439999999999998e-05 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970460+0000", 2024-09-16T03:38:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3887999999999994e-05 2024-09-16T03:38:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1656 4.12 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2252138] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:38:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.383188123, 2024-09-16T03:38:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72904596899999996, 2024-09-16T03:38:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.019 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1656 2024-09-16T03:38:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:38:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:38:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543376+0000", 2024-09-16T03:38:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4200000000000004e-07 2024-09-16T03:38:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543383+0000", 2024-09-16T03:38:12.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6290000000000004e-06 2024-09-16T03:38:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543390+0000", 2024-09-16T03:38:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:38:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543396+0000", 2024-09-16T03:38:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.02e-06 2024-09-16T03:38:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543461+0000", 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4876999999999996e-05 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543465+0000", 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2139999999999998e-06 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970490+0000", 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42702478300000002 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970556+0000", 2024-09-16T03:38:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6137999999999996e-05 2024-09-16T03:38:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:38:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970670+0000", 2024-09-16T03:38:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114416 2024-09-16T03:38:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971375+0000", 2024-09-16T03:38:12.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070496000000000003 2024-09-16T03:38:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071773+0000", 2024-09-16T03:38:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100397945 2024-09-16T03:38:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071823+0000", 2024-09-16T03:38:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:38:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.072067+0000", 2024-09-16T03:38:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243856 2024-09-16T03:38:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.171991+0000", 2024-09-16T03:38:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099924052999999999 2024-09-16T03:38:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272176+0000", 2024-09-16T03:38:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100184963 2024-09-16T03:38:12.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272251+0000", 2024-09-16T03:38:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5902000000000006e-05 2024-09-16T03:38:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272421+0000", 2024-09-16T03:38:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016962800000000001 2024-09-16T03:38:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1657 4.12 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:38:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.383156137, 2024-09-16T03:38:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72934181899999995, 2024-09-16T03:38:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1657 2024-09-16T03:38:12.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543408+0000", 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.13e-07 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543410+0000", 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7030000000000002e-06 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543417+0000", 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.567e-06 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543423+0000", 2024-09-16T03:38:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0619999999999997e-06 2024-09-16T03:38:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543489+0000", 2024-09-16T03:38:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6315999999999993e-05 2024-09-16T03:38:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543498+0000", 2024-09-16T03:38:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7239999999999998e-06 2024-09-16T03:38:12.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970952+0000", 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42745424399999998 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971012+0000", 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9771000000000001e-05 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272463+0000", 2024-09-16T03:38:12.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30145090800000002 2024-09-16T03:38:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272558+0000", 2024-09-16T03:38:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4863000000000003e-05 2024-09-16T03:38:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272749+0000", 2024-09-16T03:38:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191048 2024-09-16T03:38:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2095 4.a 4:503187b3:::smithi15736696-34:head [assert-version v60,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.032567+0000", 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.8939963520000003, 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30423177099999998, 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2095 2024-09-16T03:38:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032567+0000", 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032567+0000", 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032571+0000", 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9609999999999999e-06 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032585+0000", 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3752e-05 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032608+0000", 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3683999999999999e-05 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032621+0000", 2024-09-16T03:38:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3087e-05 2024-09-16T03:38:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032707+0000", 2024-09-16T03:38:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5498e-05 2024-09-16T03:38:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032814+0000", 2024-09-16T03:38:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107268 2024-09-16T03:38:12.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.235147+0000", 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20233309899999999 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.236234+0000", 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010869969999999999 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.236347+0000", 2024-09-16T03:38:12.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112335 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.336668+0000", 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10032134500000001 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.336740+0000", 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2047999999999999e-05 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.336799+0000", 2024-09-16T03:38:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8696999999999997e-05 2024-09-16T03:38:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2105 4.a 4:51338165:::smithi15736696-4:head [sparse-read 0~887916,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:38:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:38:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.8776919430000003, 2024-09-16T03:38:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40244934799999998, 2024-09-16T03:38:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2105 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:38:12.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048881+0000", 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6080000000000002e-06 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048889+0000", 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7209999999999994e-06 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048903+0000", 2024-09-16T03:38:12.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4538e-05 2024-09-16T03:38:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048912+0000", 2024-09-16T03:38:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2390000000000008e-06 2024-09-16T03:38:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149039+0000", 2024-09-16T03:38:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100126563 2024-09-16T03:38:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149044+0000", 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0340000000000004e-06 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451087+0000", 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30204318000000002 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451159+0000", 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1627999999999999e-05 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451321+0000", 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016183699999999999 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2106 4.a 4:51338165:::smithi15736696-4:head [sparse-read 0~887916] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.8776311840000002, 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40254453200000001, 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.045 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2106 2024-09-16T03:38:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:38:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:38:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048933+0000", 2024-09-16T03:38:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:38:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048936+0000", 2024-09-16T03:38:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3890000000000001e-06 2024-09-16T03:38:12.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048943+0000", 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1250000000000004e-06 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048953+0000", 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8190000000000003e-06 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149113+0000", 2024-09-16T03:38:12.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100160007 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149118+0000", 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7650000000000001e-06 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451341+0000", 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30222344699999998 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451381+0000", 2024-09-16T03:38:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9517000000000001e-05 2024-09-16T03:38:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451477+0000", 2024-09-16T03:38:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5656999999999996e-05 2024-09-16T03:38:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2107 4.a 4:51338165:::smithi15736696-4:head [read 1698310~486600,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:38:12.050 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.8776011209999997, 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40288642400000002, 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2107 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:38:12.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:38:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048963+0000", 2024-09-16T03:38:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:38:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048967+0000", 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9060000000000004e-06 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048972+0000", 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5910000000000003e-06 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048977+0000", 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9629999999999997e-06 2024-09-16T03:38:12.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149135+0000", 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015828 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149138+0000", 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.779e-06 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451506+0000", 2024-09-16T03:38:12.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302367891 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451558+0000", 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2374999999999999e-05 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451849+0000", 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029009899999999999 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.056 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.056 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:12.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:38:12.058 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:12.055+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:12.078 INFO:tasks.rados.rados.0.smithi157.stdout:1124: set_chunk oid 50 offset: 2653696 length: 838144 target oid fa73f0ab3a7395fcfc19219899ff8c0a4a3c96caf8327f9da9656e2450435b27 offset: 0 snap -1 2024-09-16T03:38:12.078 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:12.078 INFO:tasks.rados.rados.0.smithi157.stdout:1107: finishing write tid 2 to smithi15736696-30 2024-09-16T03:38:12.078 INFO:tasks.rados.rados.0.smithi157.stdout:1107: finishing write tid 3 to smithi15736696-30 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:1110: done (15 left) 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:1125: read oid 11 snap -1 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:1125: expect (ObjNum 184 snap 0 seq_num 184) 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:1107: finishing write tid 4 to smithi15736696-30 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:1107: finishing write tid 5 to smithi15736696-30 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:1107: oid 30 updating version 0 to 36 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:1107: oid 30 updating version 36 to 39 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:1107: oid 30 version 39 is already newer than 38 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:1107: oid 30 version 39 is already newer than 37 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 39 (ObjNum 194 snap 0 seq_num 194) dirty exists 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:1107: left oid 30 (ObjNum 194 snap 0 seq_num 194) 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:1106: finishing write tid 2 to smithi15736696-47 2024-09-16T03:38:12.079 INFO:tasks.rados.rados.0.smithi157.stdout:1106: finishing write tid 3 to smithi15736696-47 2024-09-16T03:38:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:1106: finishing write tid 4 to smithi15736696-47 2024-09-16T03:38:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:1106: finishing write tid 5 to smithi15736696-47 2024-09-16T03:38:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:1106: oid 47 updating version 0 to 85 2024-09-16T03:38:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:1106: oid 47 version 85 is already newer than 83 2024-09-16T03:38:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:1106: oid 47 version 85 is already newer than 82 2024-09-16T03:38:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:1106: oid 47 version 85 is already newer than 84 2024-09-16T03:38:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 85 (ObjNum 193 snap 0 seq_num 193) dirty exists 2024-09-16T03:38:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:1106: left oid 47 (ObjNum 193 snap 0 seq_num 193) 2024-09-16T03:38:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:1112: finishing write tid 1 to smithi15736696-25 2024-09-16T03:38:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:1112: finishing write tid 2 to smithi15736696-25 2024-09-16T03:38:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:1106: done (15 left) 2024-09-16T03:38:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:1107: done (14 left) 2024-09-16T03:38:12.080 INFO:tasks.rados.rados.0.smithi157.stdout:1126: write oid 33 current snap is 0 2024-09-16T03:38:12.081 INFO:tasks.rados.rados.0.smithi157.stdout:1112: finishing write tid 3 to smithi15736696-25 2024-09-16T03:38:12.081 INFO:tasks.rados.rados.0.smithi157.stdout:1126: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-09-16T03:38:12.083 INFO:tasks.rados.rados.0.smithi157.stdout:1126: writing smithi15736696-33 from 510979 to 1043857 tid 1 2024-09-16T03:38:12.089 INFO:tasks.rados.rados.0.smithi157.stdout:1126: writing smithi15736696-33 from 1528482 to 2300878 tid 2 2024-09-16T03:38:12.095 INFO:tasks.rados.rados.0.smithi157.stdout:1126: writing smithi15736696-33 from 3060800 to 3310979 tid 3 2024-09-16T03:38:12.095 INFO:tasks.rados.rados.0.smithi157.stdout:1127: read oid 47 snap -1 2024-09-16T03:38:12.095 INFO:tasks.rados.rados.0.smithi157.stdout:1127: expect (ObjNum 193 snap 0 seq_num 193) 2024-09-16T03:38:12.095 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:12.096 INFO:tasks.rados.rados.0.smithi157.stdout:1112: finishing write tid 4 to smithi15736696-25 2024-09-16T03:38:12.096 INFO:tasks.rados.rados.0.smithi157.stdout:1112: finishing write tid 5 to smithi15736696-25 2024-09-16T03:38:12.096 INFO:tasks.rados.rados.0.smithi157.stdout:1112: oid 25 updating version 0 to 75 2024-09-16T03:38:12.096 INFO:tasks.rados.rados.0.smithi157.stdout:1112: oid 25 version 75 is already newer than 73 2024-09-16T03:38:12.096 INFO:tasks.rados.rados.0.smithi157.stdout:1112: oid 25 version 75 is already newer than 74 2024-09-16T03:38:12.096 INFO:tasks.rados.rados.0.smithi157.stdout:1112: oid 25 updating version 75 to 76 2024-09-16T03:38:12.096 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 76 (ObjNum 196 snap 0 seq_num 196) dirty exists 2024-09-16T03:38:12.096 INFO:tasks.rados.rados.0.smithi157.stdout:1112: left oid 25 (ObjNum 196 snap 0 seq_num 196) 2024-09-16T03:38:12.096 INFO:tasks.rados.rados.0.smithi157.stdout:1117: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:12.158 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:12.156+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:12.159 INFO:tasks.rados.rados.0.smithi157.stdout:1115: expect (ObjNum 157 snap 0 seq_num 157) 2024-09-16T03:38:12.165 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2332 4.c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.082795+0000", 2024-09-16T03:38:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.081890642999999999, 2024-09-16T03:38:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081910462000000003, 2024-09-16T03:38:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2332 2024-09-16T03:38:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.082795+0000", 2024-09-16T03:38:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.082795+0000", 2024-09-16T03:38:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.082798+0000", 2024-09-16T03:38:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5959999999999998e-06 2024-09-16T03:38:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.083509+0000", 2024-09-16T03:38:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071178500000000004 2024-09-16T03:38:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.083524+0000", 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5162e-05 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.083534+0000", 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4930000000000004e-06 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2333 4.c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.089332+0000", 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.075354028000000003, 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075406984999999996, 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2333 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.089332+0000", 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.089332+0000", 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.089333+0000", 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.61e-06 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.090029+0000", 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069538 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.090042+0000", 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3574000000000001e-05 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.090048+0000", 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0129999999999999e-06 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2334 4.c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.094561+0000", 2024-09-16T03:38:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.070124680999999994, 2024-09-16T03:38:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.070202957999999996, 2024-09-16T03:38:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2334 2024-09-16T03:38:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.094561+0000", 2024-09-16T03:38:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.094561+0000", 2024-09-16T03:38:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.094562+0000", 2024-09-16T03:38:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.026e-06 2024-09-16T03:38:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095003+0000", 2024-09-16T03:38:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000441366 2024-09-16T03:38:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095011+0000", 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1380000000000007e-06 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095016+0000", 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2459999999999997e-06 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2335 4.c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.095024+0000", 2024-09-16T03:38:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.069661503, 2024-09-16T03:38:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.069764191000000003, 2024-09-16T03:38:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2335 2024-09-16T03:38:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095024+0000", 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095024+0000", 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095025+0000", 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8100000000000003e-07 2024-09-16T03:38:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095027+0000", 2024-09-16T03:38:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5579999999999999e-06 2024-09-16T03:38:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095031+0000", 2024-09-16T03:38:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.585e-06 2024-09-16T03:38:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095034+0000", 2024-09-16T03:38:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8710000000000001e-06 2024-09-16T03:38:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2336 4.c 4.1cf4922c (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:38:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.069645704000000003, 2024-09-16T03:38:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.069772697999999994, 2024-09-16T03:38:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "no flag points reached", 2024-09-16T03:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2336 2024-09-16T03:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:38:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:38:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2399999999999999e-07 2024-09-16T03:38:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095042+0000", 2024-09-16T03:38:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.567e-06 2024-09-16T03:38:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095046+0000", 2024-09-16T03:38:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.639e-06 2024-09-16T03:38:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:38:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 5 2024-09-16T03:38:12.183 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:12.184 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:38:12.220 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:12.250 INFO:tasks.rados.rados.0.smithi157.stdout:1113: finishing set_chunk to oid 43 2024-09-16T03:38:12.250 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 43 set_chunk returned error code -95 offset: 140288 length: 2537984 2024-09-16T03:38:12.250 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 0 (ObjNum 169 snap 0 seq_num 169) dirty exists 2024-09-16T03:38:12.250 INFO:tasks.rados.rados.0.smithi157.stdout:1114: expect (ObjNum 186 snap 0 seq_num 186) 2024-09-16T03:38:12.250 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:12.258 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:12.257+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:12.293 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:12.293 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:38:12.294 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:38:12.294 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:12.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:38:12.336 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 921770 2024-09-16T03:38:12.359 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:12.358+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:12.412 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:38:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:38:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:57 4.d 4:b3fe05b2:::smithi15736696-15:head [write 635298~769839,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:12.412 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:38:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.285378895000001, 2024-09-16T03:38:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.205477148, 2024-09-16T03:38:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 57 2024-09-16T03:38:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124124+0000", 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124125+0000", 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.604e-06 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124908+0000", 2024-09-16T03:38:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000783086 2024-09-16T03:38:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124919+0000", 2024-09-16T03:38:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0591000000000001e-05 2024-09-16T03:38:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124925+0000", 2024-09-16T03:38:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.55e-06 2024-09-16T03:38:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.124985+0000", 2024-09-16T03:38:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9817e-05 2024-09-16T03:38:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.125113+0000", 2024-09-16T03:38:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127429 2024-09-16T03:38:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:38:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.125229+0000", 2024-09-16T03:38:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116784 2024-09-16T03:38:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.127169+0000", 2024-09-16T03:38:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001939187 2024-09-16T03:38:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128052+0000", 2024-09-16T03:38:12.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088291800000000001 2024-09-16T03:38:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329495+0000", 2024-09-16T03:38:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201443328 2024-09-16T03:38:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329535+0000", 2024-09-16T03:38:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9823999999999997e-05 2024-09-16T03:38:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.329601+0000", 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6030000000000003e-05 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:60 4.d 4:b3fe05b2:::smithi15736696-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.281116191000002, 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30164545100000001, 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.419 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 60 2024-09-16T03:38:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:38:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128386+0000", 2024-09-16T03:38:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128387+0000", 2024-09-16T03:38:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3599999999999999e-07 2024-09-16T03:38:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128388+0000", 2024-09-16T03:38:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7659999999999999e-06 2024-09-16T03:38:12.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128394+0000", 2024-09-16T03:38:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0679999999999996e-06 2024-09-16T03:38:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.128398+0000", 2024-09-16T03:38:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4220000000000002e-06 2024-09-16T03:38:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129762+0000", 2024-09-16T03:38:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001364285 2024-09-16T03:38:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.129804+0000", 2024-09-16T03:38:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2058000000000003e-05 2024-09-16T03:38:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.429903+0000", 2024-09-16T03:38:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30009841900000001 2024-09-16T03:38:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.429952+0000", 2024-09-16T03:38:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9066000000000001e-05 2024-09-16T03:38:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.430032+0000", 2024-09-16T03:38:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9931000000000004e-05 2024-09-16T03:38:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:81 4.c 4:3313e97c:::smithi15736696-21:head [write 736140~537686,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.011876166999997, 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203525015, 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 81 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:38:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397626+0000", 2024-09-16T03:38:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.397629+0000", 2024-09-16T03:38:12.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1889999999999999e-06 2024-09-16T03:38:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398085+0000", 2024-09-16T03:38:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00045514200000000001 2024-09-16T03:38:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398100+0000", 2024-09-16T03:38:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5637999999999998e-05 2024-09-16T03:38:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398110+0000", 2024-09-16T03:38:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6549999999999999e-06 2024-09-16T03:38:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398190+0000", 2024-09-16T03:38:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0272999999999995e-05 2024-09-16T03:38:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398339+0000", 2024-09-16T03:38:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014924200000000001 2024-09-16T03:38:12.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.398504+0000", 2024-09-16T03:38:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016471500000000001 2024-09-16T03:38:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.400267+0000", 2024-09-16T03:38:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017625169999999999 2024-09-16T03:38:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.401104+0000", 2024-09-16T03:38:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083686399999999999 2024-09-16T03:38:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601084+0000", 2024-09-16T03:38:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199980397 2024-09-16T03:38:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601114+0000", 2024-09-16T03:38:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0484000000000002e-05 2024-09-16T03:38:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.601151+0000", 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6899e-05 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:83 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 2336140] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.005681371999998, 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29747773399999999, 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.431 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 83 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403821+0000", 2024-09-16T03:38:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403822+0000", 2024-09-16T03:38:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4300000000000003e-07 2024-09-16T03:38:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403825+0000", 2024-09-16T03:38:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8700000000000001e-06 2024-09-16T03:38:12.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403831+0000", 2024-09-16T03:38:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9399999999999999e-06 2024-09-16T03:38:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403834+0000", 2024-09-16T03:38:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1939999999999998e-06 2024-09-16T03:38:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.404994+0000", 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011601249999999999 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405083+0000", 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8809999999999998e-05 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405217+0000", 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013401900000000001 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.503958+0000", 2024-09-16T03:38:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098741342999999995 2024-09-16T03:38:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.506033+0000", 2024-09-16T03:38:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0020749660000000001 2024-09-16T03:38:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701227+0000", 2024-09-16T03:38:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.195194171 2024-09-16T03:38:12.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701262+0000", 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4768999999999999e-05 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701299+0000", 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6884000000000001e-05 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:84 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:38:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.005649646999998, 2024-09-16T03:38:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29760038700000002, 2024-09-16T03:38:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 84 2024-09-16T03:38:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.438 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403853+0000", 2024-09-16T03:38:12.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.58e-07 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403861+0000", 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7689999999999996e-06 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403866+0000", 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6330000000000004e-06 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.403869+0000", 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7390000000000001e-06 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405790+0000", 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001920351 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.405865+0000", 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5234999999999994e-05 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701328+0000", 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29546253300000003 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701382+0000", 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4481999999999998e-05 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.701453+0000", 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1186999999999995e-05 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.442 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:432 4.c 4:34492f38:::smithi15736696-33:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.495670406999999, 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.404460861, 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 432 2024-09-16T03:38:12.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:38:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:38:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913833+0000", 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0610000000000001e-06 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913839+0000", 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9479999999999996e-06 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913848+0000", 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5629999999999992e-06 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913857+0000", 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0399999999999998e-06 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914022+0000", 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000165422 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914101+0000", 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8462999999999997e-05 2024-09-16T03:38:12.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217171+0000", 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30307040000000002 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217854+0000", 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068278700000000002 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.218098+0000", 2024-09-16T03:38:12.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243825 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318164+0000", 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100066451 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318232+0000", 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8113000000000003e-05 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318293+0000", 2024-09-16T03:38:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0788000000000001e-05 2024-09-16T03:38:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:790 4.d 4:b3fe05b2:::smithi15736696-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:38:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.521530333000001, 2024-09-16T03:38:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41456747799999999, 2024-09-16T03:38:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 790 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887975+0000", 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0369999999999999e-06 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887987+0000", 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1606e-05 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888016+0000", 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.917e-05 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888032+0000", 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5616000000000001e-05 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888178+0000", 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612100000000001 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888307+0000", 2024-09-16T03:38:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012953899999999999 2024-09-16T03:38:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.302540+0000", 2024-09-16T03:38:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41423238899999998 2024-09-16T03:38:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1418 4.c 4:3313e97c:::smithi15736696-21:head [write 615156~555834] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.183419161, 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40891108799999998, 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1418 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226087+0000", 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8890000000000004e-06 2024-09-16T03:38:12.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226827+0000", 2024-09-16T03:38:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000739298 2024-09-16T03:38:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226848+0000", 2024-09-16T03:38:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1418000000000001e-05 2024-09-16T03:38:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226858+0000", 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0434e-05 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327036+0000", 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10017769 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327098+0000", 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1760999999999994e-05 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327230+0000", 2024-09-16T03:38:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131839 2024-09-16T03:38:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434369+0000", 2024-09-16T03:38:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107139788 2024-09-16T03:38:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434444+0000", 2024-09-16T03:38:12.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4171999999999994e-05 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634830+0000", 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20038660599999999 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634937+0000", 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010716499999999999 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634994+0000", 2024-09-16T03:38:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7027999999999997e-05 2024-09-16T03:38:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1419 4.c 4:3313e97c:::smithi15736696-21:head [write 1603467~416361] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:38:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.178753772, 2024-09-16T03:38:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40430432100000002, 2024-09-16T03:38:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.460 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1419 2024-09-16T03:38:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:38:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:38:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230752+0000", 2024-09-16T03:38:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.354e-06 2024-09-16T03:38:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231164+0000", 2024-09-16T03:38:12.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041208599999999998 2024-09-16T03:38:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231179+0000", 2024-09-16T03:38:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4885999999999999e-05 2024-09-16T03:38:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231186+0000", 2024-09-16T03:38:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8759999999999999e-06 2024-09-16T03:38:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328726+0000", 2024-09-16T03:38:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097539855999999994 2024-09-16T03:38:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328786+0000", 2024-09-16T03:38:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0059000000000002e-05 2024-09-16T03:38:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328890+0000", 2024-09-16T03:38:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104338 2024-09-16T03:38:12.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434475+0000", 2024-09-16T03:38:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.105584709 2024-09-16T03:38:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.534594+0000", 2024-09-16T03:38:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10011914099999999 2024-09-16T03:38:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635003+0000", 2024-09-16T03:38:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100408834 2024-09-16T03:38:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635029+0000", 2024-09-16T03:38:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6551999999999999e-05 2024-09-16T03:38:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635053+0000", 2024-09-16T03:38:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3629999999999999e-05 2024-09-16T03:38:12.466 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1420 4.c 4:3313e97c:::smithi15736696-21:head [write 2816389~693495] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:38:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.170839461, 2024-09-16T03:38:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39653502600000001, 2024-09-16T03:38:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.467 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1420 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238666+0000", 2024-09-16T03:38:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.163e-06 2024-09-16T03:38:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239395+0000", 2024-09-16T03:38:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072868599999999998 2024-09-16T03:38:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239410+0000", 2024-09-16T03:38:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5330999999999999e-05 2024-09-16T03:38:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239419+0000", 2024-09-16T03:38:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7360000000000007e-06 2024-09-16T03:38:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332338+0000", 2024-09-16T03:38:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092919165999999997 2024-09-16T03:38:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332435+0000", 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6972999999999995e-05 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332544+0000", 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010898700000000001 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634808+0000", 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30226407700000002 2024-09-16T03:38:12.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635058+0000", 2024-09-16T03:38:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025009300000000001 2024-09-16T03:38:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635103+0000", 2024-09-16T03:38:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4539000000000002e-05 2024-09-16T03:38:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635145+0000", 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1938999999999999e-05 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635198+0000", 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3335999999999999e-05 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1421 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:38:12.473 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.170071012000001, 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.497492088, 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1421 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:38:12.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239432+0000", 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5999999999999991e-07 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239436+0000", 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239443+0000", 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.968e-06 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239447+0000", 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6969999999999999e-06 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635225+0000", 2024-09-16T03:38:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39577807399999998 2024-09-16T03:38:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635290+0000", 2024-09-16T03:38:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4583000000000002e-05 2024-09-16T03:38:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635410+0000", 2024-09-16T03:38:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120684 2024-09-16T03:38:12.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636180+0000", 2024-09-16T03:38:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076997899999999995 2024-09-16T03:38:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636799+0000", 2024-09-16T03:38:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061906600000000004 2024-09-16T03:38:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736742+0000", 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099943041999999996 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736843+0000", 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010076799999999999 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736924+0000", 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0419000000000004e-05 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1422 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.170049731999999, 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49764818700000002, 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1422 2024-09-16T03:38:12.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239455+0000", 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9860000000000001e-06 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239461+0000", 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239464+0000", 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2619999999999999e-06 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635736+0000", 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.396272287 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635790+0000", 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4252000000000002e-05 2024-09-16T03:38:12.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736964+0000", 2024-09-16T03:38:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101173319 2024-09-16T03:38:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737024+0000", 2024-09-16T03:38:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9975000000000001e-05 2024-09-16T03:38:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737101+0000", 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7303000000000002e-05 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1420 4.c e58/32 ondisk, result = 0)", 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.335674+0000", 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.073828918, 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29938561899999999, 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:12.485 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335674+0000", 2024-09-16T03:38:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335674+0000", 2024-09-16T03:38:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335675+0000", 2024-09-16T03:38:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5510000000000001e-06 2024-09-16T03:38:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335684+0000", 2024-09-16T03:38:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6519999999999995e-06 2024-09-16T03:38:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335696+0000", 2024-09-16T03:38:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2662e-05 2024-09-16T03:38:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.335703+0000", 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2079999999999997e-06 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634784+0000", 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29908135299999999 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634797+0000", 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3393999999999999e-05 2024-09-16T03:38:12.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635059+0000", 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026179900000000001 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:1420 4.c e58/32 ondisk, result = 0)", 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.336136+0000", 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.073366160999999, 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29907022, 2024-09-16T03:38:12.489 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336136+0000", 2024-09-16T03:38:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336136+0000", 2024-09-16T03:38:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336138+0000", 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4279999999999999e-06 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336147+0000", 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0210000000000005e-06 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336159+0000", 2024-09-16T03:38:12.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2285e-05 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.336165+0000", 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7799999999999997e-06 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635091+0000", 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29892637599999999 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635098+0000", 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5019999999999998e-06 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635207+0000", 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108828 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1537 4.c 4:34492f38:::smithi15736696-33:head [write 1954970~431037] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.719891+0000", 2024-09-16T03:38:12.493 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.689611813999999, 2024-09-16T03:38:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20396983899999999, 2024-09-16T03:38:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1537 2024-09-16T03:38:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.719891+0000", 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.719891+0000", 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.719893+0000", 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.962e-06 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720069+0000", 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000176298 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720080+0000", 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1002000000000001e-05 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720088+0000", 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7440000000000007e-06 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720132+0000", 2024-09-16T03:38:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4471999999999997e-05 2024-09-16T03:38:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720196+0000", 2024-09-16T03:38:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3946000000000001e-05 2024-09-16T03:38:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720286+0000", 2024-09-16T03:38:12.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9423000000000001e-05 2024-09-16T03:38:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.723040+0000", 2024-09-16T03:38:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027547169999999998 2024-09-16T03:38:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923715+0000", 2024-09-16T03:38:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20067458299999999 2024-09-16T03:38:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923777+0000", 2024-09-16T03:38:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2135000000000001e-05 2024-09-16T03:38:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923816+0000", 2024-09-16T03:38:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8563999999999999e-05 2024-09-16T03:38:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923861+0000", 2024-09-16T03:38:12.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4993000000000001e-05 2024-09-16T03:38:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1538 4.c 4:34492f38:::smithi15736696-33:head [write 2919420~616377] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:38:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.720103+0000", 2024-09-16T03:38:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.689399278, 2024-09-16T03:38:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20383674299999999, 2024-09-16T03:38:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.500 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1538 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720103+0000", 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720103+0000", 2024-09-16T03:38:12.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.720104+0000", 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9400000000000004e-07 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721193+0000", 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001089268 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721203+0000", 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9799999999999993e-06 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721212+0000", 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4430000000000003e-06 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721350+0000", 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013873600000000001 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721407+0000", 2024-09-16T03:38:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6328e-05 2024-09-16T03:38:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.721501+0000", 2024-09-16T03:38:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3868000000000003e-05 2024-09-16T03:38:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.723066+0000", 2024-09-16T03:38:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001565461 2024-09-16T03:38:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923739+0000", 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20067338300000001 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923894+0000", 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015406600000000001 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923921+0000", 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7440000000000002e-05 2024-09-16T03:38:12.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.923940+0000", 2024-09-16T03:38:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8975999999999999e-05 2024-09-16T03:38:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1681 4.d 4:b3fe05b2:::smithi15736696-15:head [write 685752~611865,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.148763+0000", 2024-09-16T03:38:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.260739365999999, 2024-09-16T03:38:12.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20615159899999999, 2024-09-16T03:38:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1681 2024-09-16T03:38:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.507 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.148763+0000", 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.148763+0000", 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.148768+0000", 2024-09-16T03:38:12.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.527e-06 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149421+0000", 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065293199999999995 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149439+0000", 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8210999999999999e-05 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149448+0000", 2024-09-16T03:38:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2480000000000002e-06 2024-09-16T03:38:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149507+0000", 2024-09-16T03:38:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8980000000000001e-05 2024-09-16T03:38:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149593+0000", 2024-09-16T03:38:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6113999999999996e-05 2024-09-16T03:38:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.149730+0000", 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013668 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.151623+0000", 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018936350000000001 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354505+0000", 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20288149799999999 2024-09-16T03:38:12.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354562+0000", 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7255e-05 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354854+0000", 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029133000000000002 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354915+0000", 2024-09-16T03:38:12.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1189e-05 2024-09-16T03:38:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1683 4.d 4:b3fe05b2:::smithi15736696-15:head [setxattr _header (56) in=7b,truncate 2285752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.154316+0000", 2024-09-16T03:38:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.255186873, 2024-09-16T03:38:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29800856100000001, 2024-09-16T03:38:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1683 2024-09-16T03:38:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154316+0000", 2024-09-16T03:38:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154316+0000", 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154316+0000", 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1500000000000003e-07 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154320+0000", 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154332+0000", 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2254e-05 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154340+0000", 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5340000000000002e-06 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.155080+0000", 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073960699999999998 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.155143+0000", 2024-09-16T03:38:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3859e-05 2024-09-16T03:38:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,13", 2024-09-16T03:38:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.155223+0000", 2024-09-16T03:38:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9331000000000003e-05 2024-09-16T03:38:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.354955+0000", 2024-09-16T03:38:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19973200099999999 2024-09-16T03:38:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.355860+0000", 2024-09-16T03:38:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090486199999999998 2024-09-16T03:38:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.452226+0000", 2024-09-16T03:38:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096366118000000001 2024-09-16T03:38:12.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.452263+0000", 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7422000000000002e-05 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.452324+0000", 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1048000000000002e-05 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1684 4.d 4:b3fe05b2:::smithi15736696-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.255154615999999, 2024-09-16T03:38:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39825815199999998, 2024-09-16T03:38:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1684 2024-09-16T03:38:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9999999999999998e-07 2024-09-16T03:38:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154351+0000", 2024-09-16T03:38:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.249e-06 2024-09-16T03:38:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154356+0000", 2024-09-16T03:38:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7690000000000002e-06 2024-09-16T03:38:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154361+0000", 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4209999999999997e-06 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.253488+0000", 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099126895000000007 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.253585+0000", 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7204e-05 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552451+0000", 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.298866358 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552521+0000", 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9873e-05 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552606+0000", 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4882999999999994e-05 2024-09-16T03:38:12.524 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1749 4.c 4:34492f38:::smithi15736696-33:head [assert-version v40,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:56.368656+0000", 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.040846336000001, 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30391657799999999, 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:12.525 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1749 2024-09-16T03:38:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368656+0000", 2024-09-16T03:38:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368656+0000", 2024-09-16T03:38:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:12.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368660+0000", 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.693e-06 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368673+0000", 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2944e-05 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368702+0000", 2024-09-16T03:38:12.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9365000000000001e-05 2024-09-16T03:38:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368719+0000", 2024-09-16T03:38:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6756999999999999e-05 2024-09-16T03:38:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368857+0000", 2024-09-16T03:38:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000138166 2024-09-16T03:38:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368950+0000", 2024-09-16T03:38:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2843000000000005e-05 2024-09-16T03:38:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.471081+0000", 2024-09-16T03:38:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102130647 2024-09-16T03:38:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.471807+0000", 2024-09-16T03:38:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072618800000000001 2024-09-16T03:38:12.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.547167+0000", 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075359801000000004 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.572606+0000", 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025439215000000001 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.672443+0000", 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099837710999999996 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.672493+0000", 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9645999999999999e-05 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.672573+0000", 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9602000000000001e-05 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:12.531 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:12.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:38:12.533 INFO:tasks.rados.rados.0.smithi157.stdout:1118: expect (ObjNum 142 snap 0 seq_num 142) 2024-09-16T03:38:12.533 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:12.533 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:12.534 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:38:12.535 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:12.531+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:12.602 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1797312 2024-09-16T03:38:12.627 INFO:tasks.rados.rados.0.smithi157.stdout:1116: got (2) No such file or directory 2024-09-16T03:38:12.627 INFO:tasks.rados.rados.0.smithi157.stdout:1116: got expected ENOENT (src dne) 2024-09-16T03:38:12.627 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 45 (ObjNum 113 snap 0 seq_num 2312176330) clean dne 2024-09-16T03:38:12.628 INFO:tasks.rados.rados.0.smithi157.stdout:1121: got (22) Invalid argument 2024-09-16T03:38:12.628 INFO:tasks.rados.rados.0.smithi157.stdout:1119: finishing set_chunk to oid 23 2024-09-16T03:38:12.628 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 56 (ObjNum 189 snap 0 seq_num 189) dirty exists 2024-09-16T03:38:12.628 INFO:tasks.rados.rados.0.smithi157.stdout:1122: expect (ObjNum 152 snap 0 seq_num 152) 2024-09-16T03:38:12.633 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:12.631+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:12.649 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:12.649 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:12.649 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:12.649 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:12.655 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:12.658 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:38:12.698 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:12.698 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:12.707 INFO:tasks.rados.rados.0.smithi157.stdout:1120: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:12.707 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:12.716 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:12.716 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:12.734 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:12.734 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:12.734 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:12.733+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:12.751 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:12.751 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:12.767 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:12.767 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:12.783 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:12.784 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:12.785 INFO:tasks.rados.rados.0.smithi157.stdout:1123: expect (ObjNum 126 snap 0 seq_num 126) 2024-09-16T03:38:12.789 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:38:12.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T03:38:12.801 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:12.801 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:12.802 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 885869 2024-09-16T03:38:12.804 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:12.804 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:12.813 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:38:12.818 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:12.826 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1994060 2024-09-16T03:38:12.827 INFO:tasks.rados.rados.0.smithi157.stdout:1127: expect (ObjNum 193 snap 0 seq_num 193) 2024-09-16T03:38:12.827 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:12.834 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:12.836 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:12.836 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:12.833+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:12.851 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:12.851 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:12.868 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:12.868 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:12.885 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:12.885 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:12.902 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:12.902 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:12.904 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:38:12.909 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:12.916 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:12.916 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:12.921 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:12.922 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:12.925 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:38:12.935 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:12.933+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:12.939 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:12.939 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:12.956 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:12.956 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:12.958 INFO:tasks.rados.rados.0.smithi157.stdout:1125: expect (ObjNum 184 snap 0 seq_num 184) 2024-09-16T03:38:12.996 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2942449 2024-09-16T03:38:13.032 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:13.032 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:13.035 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:13.034+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:13.041 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:38:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:1126: finishing write tid 1 to smithi15736696-33 2024-09-16T03:38:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:1126: finishing write tid 2 to smithi15736696-33 2024-09-16T03:38:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:1124: finishing set_chunk to oid 50 2024-09-16T03:38:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 56 (ObjNum 104 snap 0 seq_num 104) dirty exists 2024-09-16T03:38:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:1126: finishing write tid 3 to smithi15736696-33 2024-09-16T03:38:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:1126: finishing write tid 4 to smithi15736696-33 2024-09-16T03:38:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:1126: finishing write tid 5 to smithi15736696-33 2024-09-16T03:38:13.117 INFO:tasks.rados.rados.0.smithi157.stdout:1126: oid 33 updating version 0 to 45 2024-09-16T03:38:13.118 INFO:tasks.rados.rados.0.smithi157.stdout:1126: oid 33 updating version 45 to 46 2024-09-16T03:38:13.118 INFO:tasks.rados.rados.0.smithi157.stdout:1126: oid 33 updating version 46 to 47 2024-09-16T03:38:13.118 INFO:tasks.rados.rados.0.smithi157.stdout:1126: oid 33 version 47 is already newer than 44 2024-09-16T03:38:13.118 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 47 (ObjNum 197 snap 0 seq_num 197) dirty exists 2024-09-16T03:38:13.118 INFO:tasks.rados.rados.0.smithi157.stdout:1126: left oid 33 (ObjNum 197 snap 0 seq_num 197) 2024-09-16T03:38:13.118 INFO:tasks.rados.rados.0.smithi157.stdout:1112: done (15 left) 2024-09-16T03:38:13.118 INFO:tasks.rados.rados.0.smithi157.stdout:1113: done (14 left) 2024-09-16T03:38:13.118 INFO:tasks.rados.rados.0.smithi157.stdout:1114: done (13 left) 2024-09-16T03:38:13.118 INFO:tasks.rados.rados.0.smithi157.stdout:1115: done (12 left) 2024-09-16T03:38:13.118 INFO:tasks.rados.rados.0.smithi157.stdout:1116: done (11 left) 2024-09-16T03:38:13.118 INFO:tasks.rados.rados.0.smithi157.stdout:1117: done (10 left) 2024-09-16T03:38:13.118 INFO:tasks.rados.rados.0.smithi157.stdout:1118: done (9 left) 2024-09-16T03:38:13.118 INFO:tasks.rados.rados.0.smithi157.stdout:1119: done (8 left) 2024-09-16T03:38:13.119 INFO:tasks.rados.rados.0.smithi157.stdout:1120: done (7 left) 2024-09-16T03:38:13.119 INFO:tasks.rados.rados.0.smithi157.stdout:1121: done (6 left) 2024-09-16T03:38:13.119 INFO:tasks.rados.rados.0.smithi157.stdout:1122: done (5 left) 2024-09-16T03:38:13.119 INFO:tasks.rados.rados.0.smithi157.stdout:1123: done (4 left) 2024-09-16T03:38:13.119 INFO:tasks.rados.rados.0.smithi157.stdout:1124: done (3 left) 2024-09-16T03:38:13.119 INFO:tasks.rados.rados.0.smithi157.stdout:1125: done (2 left) 2024-09-16T03:38:13.119 INFO:tasks.rados.rados.0.smithi157.stdout:1126: done (1 left) 2024-09-16T03:38:13.119 INFO:tasks.rados.rados.0.smithi157.stdout:1127: done (0 left) 2024-09-16T03:38:13.119 INFO:tasks.rados.rados.0.smithi157.stdout:1128: read oid 36 snap -1 2024-09-16T03:38:13.119 INFO:tasks.rados.rados.0.smithi157.stdout:1128: expect (ObjNum 176 snap 0 seq_num 176) 2024-09-16T03:38:13.119 INFO:tasks.rados.rados.0.smithi157.stdout:1129: write (excl) oid 40 current snap is 0 2024-09-16T03:38:13.119 INFO:tasks.rados.rados.0.smithi157.stdout:1129: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2024-09-16T03:38:13.122 INFO:tasks.rados.rados.0.smithi157.stdout:1129: writing smithi15736696-40 from 527786 to 1252266 tid 1 2024-09-16T03:38:13.127 INFO:tasks.rados.rados.0.smithi157.stdout:1129: writing smithi15736696-40 from 1703317 to 2219371 tid 2 2024-09-16T03:38:13.133 INFO:tasks.rados.rados.0.smithi157.stdout:1129: writing smithi15736696-40 from 2804576 to 3239611 tid 3 2024-09-16T03:38:13.133 INFO:tasks.rados.rados.0.smithi157.stdout:1130: write (excl) oid 48 current snap is 0 2024-09-16T03:38:13.133 INFO:tasks.rados.rados.0.smithi157.stdout:1130: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-09-16T03:38:13.136 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:13.135+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:13.140 INFO:tasks.rados.rados.0.smithi157.stdout:1130: writing smithi15736696-48 from 544593 to 1344322 tid 1 2024-09-16T03:38:13.147 INFO:tasks.rados.rados.0.smithi157.stdout:1130: writing smithi15736696-48 from 1761799 to 2537864 tid 2 2024-09-16T03:38:13.151 INFO:tasks.rados.rados.0.smithi157.stdout:1130: writing smithi15736696-48 from 2948352 to 3344593 tid 3 2024-09-16T03:38:13.151 INFO:tasks.rados.rados.0.smithi157.stdout:1131: read oid 49 snap -1 2024-09-16T03:38:13.151 INFO:tasks.rados.rados.0.smithi157.stdout:1131: expect (ObjNum 191 snap 0 seq_num 191) 2024-09-16T03:38:13.151 INFO:tasks.rados.rados.0.smithi157.stdout:1132: set_chunk oid 9 target oid 2024-09-16T03:38:13.183 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:13.184 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:13.192 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:38:13.238 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:13.236+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:13.294 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:13.295 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:13.303 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:38:13.339 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:13.337+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:13.359 INFO:tasks.rados.rados.0.smithi157.stdout:1132: set_chunk oid 9 offset: 158720 length: 584704 target oid 499e9805be1f68ebe2a78a38a8107a3af67f39a5f107591b4c95db82a1f52a97 offset: 0 snap -1 2024-09-16T03:38:13.359 INFO:tasks.rados.rados.0.smithi157.stdout:1133: set_chunk oid 1 target oid 2024-09-16T03:38:13.407 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:13.407 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:13.416 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:38:13.440 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:13.438+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:13.469 INFO:tasks.rados.rados.0.smithi157.stdout:1133: set_chunk oid 1 offset: 564224 length: 872448 target oid eff7e829b2b666a7c6953813167ed766563a4d3660fead317c69a42204b4e0b4 offset: 0 snap -1 2024-09-16T03:38:13.470 INFO:tasks.rados.rados.0.smithi157.stdout:1134: set_chunk oid 23 target oid 2024-09-16T03:38:13.519 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:13.519 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:13.528 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:38:13.540 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:13.539+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:13.582 INFO:tasks.rados.rados.0.smithi157.stdout:1134: set_chunk oid 23 offset: 1050624 length: 1335808 target oid eb071235b302503f75b3b160d75638a5bbed69a0d700fbeaa1acece0b74f9bc2 offset: 0 snap -1 2024-09-16T03:38:13.582 INFO:tasks.rados.rados.0.smithi157.stdout:1135: tier_evict oid 18 2024-09-16T03:38:13.583 INFO:tasks.rados.rados.0.smithi157.stdout:1135: tier_evict oid 18 snap -1 2024-09-16T03:38:13.583 INFO:tasks.rados.rados.0.smithi157.stdout:1128: expect (ObjNum 176 snap 0 seq_num 176) 2024-09-16T03:38:13.631 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:13.631 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:13.640 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:38:13.641 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:13.639+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:13.648 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2258200 2024-09-16T03:38:13.661 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:13.707 INFO:tasks.rados.rados.0.smithi157.stdout:1131: expect (ObjNum 191 snap 0 seq_num 191) 2024-09-16T03:38:13.742 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:13.741+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:13.744 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:13.744 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:13.753 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:38:13.794 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T03:38:13.795 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:38:13.809 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T03:38:13.829 INFO:tasks.rados.rados.0.smithi157.stdout:1129: finishing write tid 1 to smithi15736696-40 2024-09-16T03:38:13.829 INFO:tasks.rados.rados.0.smithi157.stdout:1129: finishing write tid 2 to smithi15736696-40 2024-09-16T03:38:13.829 INFO:tasks.rados.rados.0.smithi157.stdout:1130: finishing write tid 1 to smithi15736696-48 2024-09-16T03:38:13.829 INFO:tasks.rados.rados.0.smithi157.stdout:1129: finishing write tid 3 to smithi15736696-40 2024-09-16T03:38:13.829 INFO:tasks.rados.rados.0.smithi157.stdout:1132: finishing set_chunk to oid 9 2024-09-16T03:38:13.829 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 9 set_chunk returned error code -95 offset: 158720 length: 584704 2024-09-16T03:38:13.829 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 0 (ObjNum 180 snap 0 seq_num 180) dirty exists 2024-09-16T03:38:13.829 INFO:tasks.rados.rados.0.smithi157.stdout:1129: finishing write tid 4 to smithi15736696-40 2024-09-16T03:38:13.829 INFO:tasks.rados.rados.0.smithi157.stdout:1129: finishing write tid 5 to smithi15736696-40 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:1129: oid 40 updating version 0 to 78 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:1129: oid 40 updating version 78 to 81 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:1129: oid 40 version 81 is already newer than 80 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:1129: oid 40 version 81 is already newer than 79 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 81 (ObjNum 198 snap 0 seq_num 198) dirty exists 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:1129: left oid 40 (ObjNum 198 snap 0 seq_num 198) 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:1130: finishing write tid 2 to smithi15736696-48 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:1128: done (7 left) 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:1129: done (6 left) 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:1131: done (5 left) 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:1132: done (4 left) 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:1136: tier_promote oid 14 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:1130: finishing write tid 3 to smithi15736696-48 2024-09-16T03:38:13.830 INFO:tasks.rados.rados.0.smithi157.stdout:1130: finishing write tid 4 to smithi15736696-48 2024-09-16T03:38:13.831 INFO:tasks.rados.rados.0.smithi157.stdout:1134: finishing set_chunk to oid 23 2024-09-16T03:38:13.831 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 23 set_chunk returned error code -95 offset: 1050624 length: 1335808 2024-09-16T03:38:13.831 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 0 (ObjNum 189 snap 0 seq_num 189) dirty exists 2024-09-16T03:38:13.831 INFO:tasks.rados.rados.0.smithi157.stdout:1135: got (22) Invalid argument 2024-09-16T03:38:13.831 INFO:tasks.rados.rados.0.smithi157.stdout:1130: finishing write tid 5 to smithi15736696-48 2024-09-16T03:38:13.831 INFO:tasks.rados.rados.0.smithi157.stdout:1130: oid 48 updating version 0 to 51 2024-09-16T03:38:13.831 INFO:tasks.rados.rados.0.smithi157.stdout:1130: oid 48 version 51 is already newer than 50 2024-09-16T03:38:13.831 INFO:tasks.rados.rados.0.smithi157.stdout:1130: oid 48 updating version 51 to 52 2024-09-16T03:38:13.831 INFO:tasks.rados.rados.0.smithi157.stdout:1130: oid 48 updating version 52 to 53 2024-09-16T03:38:13.831 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 53 (ObjNum 199 snap 0 seq_num 199) dirty exists 2024-09-16T03:38:13.831 INFO:tasks.rados.rados.0.smithi157.stdout:1130: left oid 48 (ObjNum 199 snap 0 seq_num 199) 2024-09-16T03:38:13.831 INFO:tasks.rados.rados.0.smithi157.stdout:1133: finishing set_chunk to oid 1 2024-09-16T03:38:13.831 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 12 (ObjNum 134 snap 0 seq_num 134) dirty exists 2024-09-16T03:38:13.832 INFO:tasks.rados.rados.0.smithi157.stdout:1130: done (4 left) 2024-09-16T03:38:13.832 INFO:tasks.rados.rados.0.smithi157.stdout:1133: done (3 left) 2024-09-16T03:38:13.832 INFO:tasks.rados.rados.0.smithi157.stdout:1134: done (2 left) 2024-09-16T03:38:13.832 INFO:tasks.rados.rados.0.smithi157.stdout:1135: done (1 left) 2024-09-16T03:38:13.832 INFO:tasks.rados.rados.0.smithi157.stdout:1137: read oid 39 snap -1 2024-09-16T03:38:13.832 INFO:tasks.rados.rados.0.smithi157.stdout:1137: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-16T03:38:13.832 INFO:tasks.rados.rados.0.smithi157.stdout:1138: set_chunk oid 3 target oid 2024-09-16T03:38:13.832 INFO:tasks.rados.rados.0.smithi157.stdout:1138: set_chunk oid 3 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:13.832 INFO:tasks.rados.rados.0.smithi157.stdout:1139: tier_evict oid 4 2024-09-16T03:38:13.832 INFO:tasks.rados.rados.0.smithi157.stdout:1139: tier_evict oid 4 snap -1 2024-09-16T03:38:13.832 INFO:tasks.rados.rados.0.smithi157.stdout:1140: write oid 42 current snap is 0 2024-09-16T03:38:13.832 INFO:tasks.rados.rados.0.smithi157.stdout:1140: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-09-16T03:38:13.833 INFO:tasks.rados.rados.0.smithi157.stdout:1140: writing smithi15736696-42 from 561400 to 1036378 tid 1 2024-09-16T03:38:13.841 INFO:tasks.rados.rados.0.smithi157.stdout:1140: writing smithi15736696-42 from 1820281 to 2340004 tid 2 2024-09-16T03:38:13.843 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:13.842+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:13.847 INFO:tasks.rados.rados.0.smithi157.stdout:1140: writing smithi15736696-42 from 2859422 to 3317001 tid 3 2024-09-16T03:38:13.847 INFO:tasks.rados.rados.0.smithi157.stdout:1141: tier_evict oid 19 2024-09-16T03:38:13.847 INFO:tasks.rados.rados.0.smithi157.stdout:1141: tier_evict oid 19 snap -1 2024-09-16T03:38:13.847 INFO:tasks.rados.rados.0.smithi157.stdout:1142: read oid 32 snap -1 2024-09-16T03:38:13.847 INFO:tasks.rados.rados.0.smithi157.stdout:1142: expect (ObjNum 188 snap 0 seq_num 188) 2024-09-16T03:38:13.847 INFO:tasks.rados.rados.0.smithi157.stdout:1143: set_chunk oid 21 target oid 2024-09-16T03:38:13.847 INFO:tasks.rados.rados.0.smithi157.stdout:1143: set_chunk oid 21 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:13.847 INFO:tasks.rados.rados.0.smithi157.stdout:1144: read oid 45 snap -1 2024-09-16T03:38:13.847 INFO:tasks.rados.rados.0.smithi157.stdout:1144: expect deleted 2024-09-16T03:38:13.847 INFO:tasks.rados.rados.0.smithi157.stdout:1145: write (excl) oid 2 current snap is 0 2024-09-16T03:38:13.848 INFO:tasks.rados.rados.0.smithi157.stdout:1145: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-09-16T03:38:13.852 INFO:tasks.rados.rados.0.smithi157.stdout:1145: writing smithi15736696-2 from 578207 to 1128434 tid 1 2024-09-16T03:38:13.857 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:13.857 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:13.859 INFO:tasks.rados.rados.0.smithi157.stdout:1145: writing smithi15736696-2 from 1762410 to 2542144 tid 2 2024-09-16T03:38:13.864 INFO:tasks.rados.rados.0.smithi157.stdout:1145: writing smithi15736696-2 from 3286845 to 3378207 tid 3 2024-09-16T03:38:13.864 INFO:tasks.rados.rados.0.smithi157.stdout:1146: set_chunk oid 48 target oid 2024-09-16T03:38:13.866 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:38:13.944 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:13.942+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:13.970 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:14.064 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:14.066 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:14.027+0000 7f49439d4640 -1 osd.4 82 handle_osd_map injecting map crc failure 2024-09-16T03:38:14.067 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:14.064+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:14.067 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:38:14.140 INFO:tasks.rados.rados.0.smithi157.stdout:1146: set_chunk oid 48 offset: 1792000 length: 1040384 target oid bb6d585cb84cc91b72d06614c374dcce090e9dd3a7d0c036f49ff9614070b91d offset: 0 snap -1 2024-09-16T03:38:14.140 INFO:tasks.rados.rados.0.smithi157.stdout:1147: read oid 11 snap -1 2024-09-16T03:38:14.140 INFO:tasks.rados.rados.0.smithi157.stdout:1147: expect (ObjNum 184 snap 0 seq_num 184) 2024-09-16T03:38:14.140 INFO:tasks.rados.rados.0.smithi157.stdout:1148: write oid 5 current snap is 0 2024-09-16T03:38:14.140 INFO:tasks.rados.rados.0.smithi157.stdout:1139: got (22) Invalid argument 2024-09-16T03:38:14.140 INFO:tasks.rados.rados.0.smithi157.stdout:1148: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-09-16T03:38:14.146 INFO:tasks.rados.rados.0.smithi157.stdout:1148: writing smithi15736696-5 from 595014 to 1220490 tid 1 2024-09-16T03:38:14.152 INFO:tasks.rados.rados.0.smithi157.stdout:1148: writing smithi15736696-5 from 1820892 to 2344284 tid 2 2024-09-16T03:38:14.158 INFO:tasks.rados.rados.0.smithi157.stdout:1148: writing smithi15736696-5 from 2797915 to 3394391 tid 3 2024-09-16T03:38:14.158 INFO:tasks.rados.rados.0.smithi157.stdout:1139: done (12 left) 2024-09-16T03:38:14.158 INFO:tasks.rados.rados.0.smithi157.stdout:1149: read oid 40 snap -1 2024-09-16T03:38:14.159 INFO:tasks.rados.rados.0.smithi157.stdout:1149: expect (ObjNum 198 snap 0 seq_num 198) 2024-09-16T03:38:14.159 INFO:tasks.rados.rados.0.smithi157.stdout:1150: read oid 25 snap -1 2024-09-16T03:38:14.159 INFO:tasks.rados.rados.0.smithi157.stdout:1150: expect (ObjNum 196 snap 0 seq_num 196) 2024-09-16T03:38:14.159 INFO:tasks.rados.rados.0.smithi157.stdout:1151: read oid 35 snap -1 2024-09-16T03:38:14.159 INFO:tasks.rados.rados.0.smithi157.stdout:1151: expect (ObjNum 190 snap 0 seq_num 190) 2024-09-16T03:38:14.159 INFO:tasks.rados.rados.0.smithi157.stdout:1152: write (excl) oid 37 current snap is 0 2024-09-16T03:38:14.159 INFO:tasks.rados.rados.0.smithi157.stdout:1138: finishing set_chunk to oid 3 2024-09-16T03:38:14.159 INFO:tasks.rados.rados.0.smithi157.stdout:1138: got expected ENOENT (src dne) 2024-09-16T03:38:14.159 INFO:tasks.rados.rados.0.smithi157.stdout:1141: got (22) Invalid argument 2024-09-16T03:38:14.159 INFO:tasks.rados.rados.0.smithi157.stdout:1137: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-16T03:38:14.159 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:14.166 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:14.165+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:14.184 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:14.184 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:14.193 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:38:14.197 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1577148 2024-09-16T03:38:14.223 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1312821 2024-09-16T03:38:14.251 INFO:tasks.rados.rados.0.smithi157.stdout:1143: finishing set_chunk to oid 21 2024-09-16T03:38:14.251 INFO:tasks.rados.rados.0.smithi157.stdout:1143: got expected ENOENT (src dne) 2024-09-16T03:38:14.251 INFO:tasks.rados.rados.0.smithi157.stdout:1140: finishing write tid 1 to smithi15736696-42 2024-09-16T03:38:14.252 INFO:tasks.rados.rados.0.smithi157.stdout:1152: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-09-16T03:38:14.259 INFO:tasks.rados.rados.0.smithi157.stdout:1152: writing smithi15736696-37 from 611821 to 1312546 tid 1 2024-09-16T03:38:14.266 INFO:tasks.rados.rados.0.smithi157.stdout:1152: writing smithi15736696-37 from 1879374 to 2662777 tid 2 2024-09-16T03:38:14.268 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:14.266+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:14.270 INFO:tasks.rados.rados.0.smithi157.stdout:1152: writing smithi15736696-37 from 3341691 to 3411821 tid 3 2024-09-16T03:38:14.270 INFO:tasks.rados.rados.0.smithi157.stdout:1137: done (15 left) 2024-09-16T03:38:14.270 INFO:tasks.rados.rados.0.smithi157.stdout:1138: done (14 left) 2024-09-16T03:38:14.270 INFO:tasks.rados.rados.0.smithi157.stdout:1141: done (13 left) 2024-09-16T03:38:14.270 INFO:tasks.rados.rados.0.smithi157.stdout:1143: done (12 left) 2024-09-16T03:38:14.270 INFO:tasks.rados.rados.0.smithi157.stdout:1144: done (11 left) 2024-09-16T03:38:14.270 INFO:tasks.rados.rados.0.smithi157.stdout:1153: tier_promote oid 39 2024-09-16T03:38:14.270 INFO:tasks.rados.rados.0.smithi157.stdout:1142: expect (ObjNum 188 snap 0 seq_num 188) 2024-09-16T03:38:14.296 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3054248 2024-09-16T03:38:14.301 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:14.302 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:14.310 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:38:14.370 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:14.368+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:14.392 INFO:tasks.rados.rados.0.smithi157.stdout:1149: expect (ObjNum 198 snap 0 seq_num 198) 2024-09-16T03:38:14.413 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:14.413 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:14.422 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:38:14.423 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:14.470 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:14.469+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:14.476 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2909549 2024-09-16T03:38:14.480 INFO:tasks.rados.rados.0.smithi157.stdout:1145: finishing write tid 1 to smithi15736696-2 2024-09-16T03:38:14.480 INFO:tasks.rados.rados.0.smithi157.stdout:1140: finishing write tid 2 to smithi15736696-42 2024-09-16T03:38:14.480 INFO:tasks.rados.rados.0.smithi157.stdout:1140: finishing write tid 3 to smithi15736696-42 2024-09-16T03:38:14.480 INFO:tasks.rados.rados.0.smithi157.stdout:1140: finishing write tid 4 to smithi15736696-42 2024-09-16T03:38:14.480 INFO:tasks.rados.rados.0.smithi157.stdout:1150: expect (ObjNum 196 snap 0 seq_num 196) 2024-09-16T03:38:14.484 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 663906 2024-09-16T03:38:14.527 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:14.528 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:14.536 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:38:14.577 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:14.576+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:14.586 INFO:tasks.rados.rados.0.smithi157.stdout:1140: finishing write tid 5 to smithi15736696-42 2024-09-16T03:38:14.586 INFO:tasks.rados.rados.0.smithi157.stdout:1140: oid 42 updating version 0 to 54 2024-09-16T03:38:14.586 INFO:tasks.rados.rados.0.smithi157.stdout:1140: oid 42 version 54 is already newer than 53 2024-09-16T03:38:14.586 INFO:tasks.rados.rados.0.smithi157.stdout:1140: oid 42 version 54 is already newer than 52 2024-09-16T03:38:14.586 INFO:tasks.rados.rados.0.smithi157.stdout:1140: oid 42 version 54 is already newer than 51 2024-09-16T03:38:14.586 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 54 (ObjNum 200 snap 0 seq_num 200) dirty exists 2024-09-16T03:38:14.586 INFO:tasks.rados.rados.0.smithi157.stdout:1140: left oid 42 (ObjNum 200 snap 0 seq_num 200) 2024-09-16T03:38:14.586 INFO:tasks.rados.rados.0.smithi157.stdout:1148: finishing write tid 1 to smithi15736696-5 2024-09-16T03:38:14.586 INFO:tasks.rados.rados.0.smithi157.stdout:1148: finishing write tid 2 to smithi15736696-5 2024-09-16T03:38:14.586 INFO:tasks.rados.rados.0.smithi157.stdout:1140: done (11 left) 2024-09-16T03:38:14.586 INFO:tasks.rados.rados.0.smithi157.stdout:1142: done (10 left) 2024-09-16T03:38:14.587 INFO:tasks.rados.rados.0.smithi157.stdout:1149: done (9 left) 2024-09-16T03:38:14.587 INFO:tasks.rados.rados.0.smithi157.stdout:1150: done (8 left) 2024-09-16T03:38:14.587 INFO:tasks.rados.rados.0.smithi157.stdout:1154: read oid 7 snap -1 2024-09-16T03:38:14.587 INFO:tasks.rados.rados.0.smithi157.stdout:1154: expect deleted 2024-09-16T03:38:14.587 INFO:tasks.rados.rados.0.smithi157.stdout:1155: write oid 40 current snap is 0 2024-09-16T03:38:14.587 INFO:tasks.rados.rados.0.smithi157.stdout:1136: got (0) Success 2024-09-16T03:38:14.587 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 36 (ObjNum 131 snap 0 seq_num 131) dirty exists 2024-09-16T03:38:14.587 INFO:tasks.rados.rados.0.smithi157.stdout:1147: expect (ObjNum 184 snap 0 seq_num 184) 2024-09-16T03:38:14.587 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:14.672 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:14.671+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:14.681 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:14.681 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:14.687 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:38:14.761 INFO:tasks.rados.rados.0.smithi157.stdout:1151: expect (ObjNum 190 snap 0 seq_num 190) 2024-09-16T03:38:14.761 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:14.774 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:14.772+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:14.791 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:14.791 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:14.800 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:38:14.805 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T03:38:14.820 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T03:38:14.823 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:14.843 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:14.842+0000 7eff7884b640 -1 osd.3 83 handle_osd_map injecting map crc failure 2024-09-16T03:38:14.903 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:14.902+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:14.904 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:14.905 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:14.913 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:38:14.946 INFO:tasks.rados.rados.0.smithi157.stdout:1145: finishing write tid 2 to smithi15736696-2 2024-09-16T03:38:14.946 INFO:tasks.rados.rados.0.smithi157.stdout:1145: finishing write tid 3 to smithi15736696-2 2024-09-16T03:38:14.946 INFO:tasks.rados.rados.0.smithi157.stdout:1148: finishing write tid 3 to smithi15736696-5 2024-09-16T03:38:14.946 INFO:tasks.rados.rados.0.smithi157.stdout:1152: finishing write tid 1 to smithi15736696-37 2024-09-16T03:38:14.946 INFO:tasks.rados.rados.0.smithi157.stdout:1152: finishing write tid 2 to smithi15736696-37 2024-09-16T03:38:14.946 INFO:tasks.rados.rados.0.smithi157.stdout:1152: finishing write tid 3 to smithi15736696-37 2024-09-16T03:38:14.947 INFO:tasks.rados.rados.0.smithi157.stdout:1145: finishing write tid 4 to smithi15736696-2 2024-09-16T03:38:14.947 INFO:tasks.rados.rados.0.smithi157.stdout:1145: finishing write tid 5 to smithi15736696-2 2024-09-16T03:38:14.947 INFO:tasks.rados.rados.0.smithi157.stdout:1145: oid 2 updating version 0 to 26 2024-09-16T03:38:14.947 INFO:tasks.rados.rados.0.smithi157.stdout:1145: oid 2 updating version 26 to 27 2024-09-16T03:38:14.947 INFO:tasks.rados.rados.0.smithi157.stdout:1145: oid 2 updating version 27 to 29 2024-09-16T03:38:14.947 INFO:tasks.rados.rados.0.smithi157.stdout:1145: oid 2 version 29 is already newer than 28 2024-09-16T03:38:14.947 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 29 (ObjNum 201 snap 0 seq_num 201) dirty exists 2024-09-16T03:38:14.947 INFO:tasks.rados.rados.0.smithi157.stdout:1145: left oid 2 (ObjNum 201 snap 0 seq_num 201) 2024-09-16T03:38:14.947 INFO:tasks.rados.rados.0.smithi157.stdout:1152: finishing write tid 4 to smithi15736696-37 2024-09-16T03:38:14.947 INFO:tasks.rados.rados.0.smithi157.stdout:1155: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-09-16T03:38:14.953 INFO:tasks.rados.rados.0.smithi157.stdout:1155: writing smithi15736696-40 from 628628 to 1404602 tid 1 2024-09-16T03:38:14.959 INFO:tasks.rados.rados.0.smithi157.stdout:1155: writing smithi15736696-40 from 1937856 to 2464917 tid 2 2024-09-16T03:38:14.965 INFO:tasks.rados.rados.0.smithi157.stdout:1155: writing smithi15736696-40 from 3252761 to 3428628 tid 3 2024-09-16T03:38:14.965 INFO:tasks.rados.rados.0.smithi157.stdout:1136: done (9 left) 2024-09-16T03:38:14.965 INFO:tasks.rados.rados.0.smithi157.stdout:1145: done (8 left) 2024-09-16T03:38:14.965 INFO:tasks.rados.rados.0.smithi157.stdout:1147: done (7 left) 2024-09-16T03:38:14.965 INFO:tasks.rados.rados.0.smithi157.stdout:1151: done (6 left) 2024-09-16T03:38:14.965 INFO:tasks.rados.rados.0.smithi157.stdout:1156: write oid 34 current snap is 0 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:1152: finishing write tid 5 to smithi15736696-37 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:1152: oid 37 updating version 0 to 47 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:1152: oid 37 updating version 47 to 49 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:1152: oid 37 updating version 49 to 50 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:1152: oid 37 version 50 is already newer than 48 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 50 (ObjNum 203 snap 0 seq_num 203) dirty exists 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:1152: left oid 37 (ObjNum 203 snap 0 seq_num 203) 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:1153: got (0) Success 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 50 (ObjNum 150 snap 0 seq_num 150) dirty exists 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:1148: finishing write tid 4 to smithi15736696-5 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:1148: finishing write tid 5 to smithi15736696-5 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:1148: oid 5 updating version 0 to 55 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:1148: oid 5 updating version 55 to 57 2024-09-16T03:38:14.966 INFO:tasks.rados.rados.0.smithi157.stdout:1148: oid 5 version 57 is already newer than 56 2024-09-16T03:38:14.967 INFO:tasks.rados.rados.0.smithi157.stdout:1148: oid 5 updating version 57 to 58 2024-09-16T03:38:14.967 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 58 (ObjNum 202 snap 0 seq_num 202) dirty exists 2024-09-16T03:38:14.967 INFO:tasks.rados.rados.0.smithi157.stdout:1148: left oid 5 (ObjNum 202 snap 0 seq_num 202) 2024-09-16T03:38:14.967 INFO:tasks.rados.rados.0.smithi157.stdout:1156: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-09-16T03:38:14.968 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:38:14.970 INFO:tasks.rados.rados.0.smithi157.stdout:1156: writing smithi15736696-34 from 645435 to 1096658 tid 1 2024-09-16T03:38:14.976 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:14.975+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:14.978 INFO:tasks.rados.rados.0.smithi157.stdout:1156: writing smithi15736696-34 from 1879985 to 2667057 tid 2 2024-09-16T03:38:14.983 INFO:tasks.rados.rados.0.smithi157.stdout:1156: writing smithi15736696-34 from 3280184 to 3445435 tid 3 2024-09-16T03:38:14.983 INFO:tasks.rados.rados.0.smithi157.stdout:1148: done (6 left) 2024-09-16T03:38:14.983 INFO:tasks.rados.rados.0.smithi157.stdout:1152: done (5 left) 2024-09-16T03:38:14.983 INFO:tasks.rados.rados.0.smithi157.stdout:1153: done (4 left) 2024-09-16T03:38:14.983 INFO:tasks.rados.rados.0.smithi157.stdout:1154: done (3 left) 2024-09-16T03:38:14.983 INFO:tasks.rados.rados.0.smithi157.stdout:1157: read oid 43 snap -1 2024-09-16T03:38:14.983 INFO:tasks.rados.rados.0.smithi157.stdout:1157: expect (ObjNum 169 snap 0 seq_num 169) 2024-09-16T03:38:14.984 INFO:tasks.rados.rados.0.smithi157.stdout:1158: read oid 25 snap -1 2024-09-16T03:38:14.984 INFO:tasks.rados.rados.0.smithi157.stdout:1158: expect (ObjNum 196 snap 0 seq_num 196) 2024-09-16T03:38:14.984 INFO:tasks.rados.rados.0.smithi157.stdout:1159: read oid 6 snap -1 2024-09-16T03:38:14.984 INFO:tasks.rados.rados.0.smithi157.stdout:1159: expect (ObjNum 164 snap 0 seq_num 164) 2024-09-16T03:38:14.984 INFO:tasks.rados.rados.0.smithi157.stdout:1160: read oid 4 snap -1 2024-09-16T03:38:14.984 INFO:tasks.rados.rados.0.smithi157.stdout:1160: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:14.984 INFO:tasks.rados.rados.0.smithi157.stdout:1161: delete oid 5 current snap is 0 2024-09-16T03:38:15.018 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:15.018 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:15.027 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:38:15.077 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:15.075+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:15.089 INFO:tasks.rados.rados.0.smithi157.stdout:1157: expect (ObjNum 169 snap 0 seq_num 169) 2024-09-16T03:38:15.089 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 479480 2024-09-16T03:38:15.101 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:15.101 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:15.119 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:15.119 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:15.135 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:15.135 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:15.153 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:15.153 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:15.156 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:15.169 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:15.170 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:15.174 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:15.174 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:15.178 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:15.177+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:15.183 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:38:15.188 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:15.188 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:15.205 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:15.205 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:15.221 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:15.221 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:15.237 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:15.237 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:15.254 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:15.254 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:15.273 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:15.273 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:15.279 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:15.278+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:15.289 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:15.289 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:15.292 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:15.292 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:15.297 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:38:15.302 INFO:tasks.rados.rados.0.smithi157.stdout:1146: finishing set_chunk to oid 48 2024-09-16T03:38:15.302 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 53 (ObjNum 199 snap 0 seq_num 199) dirty exists 2024-09-16T03:38:15.303 INFO:tasks.rados.rados.0.smithi157.stdout:1159: expect (ObjNum 164 snap 0 seq_num 164) 2024-09-16T03:38:15.307 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:15.307 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:15.314 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1116750 2024-09-16T03:38:15.324 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:15.324 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:15.340 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:15.340 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:15.348 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:15.373 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:15.373 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:15.380 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:15.378+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:15.402 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:15.403 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:15.403 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:15.403 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:15.403 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:15.411 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:38:15.417 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1786681 2024-09-16T03:38:15.445 INFO:tasks.rados.rados.0.smithi157.stdout:1156: finishing write tid 1 to smithi15736696-34 2024-09-16T03:38:15.445 INFO:tasks.rados.rados.0.smithi157.stdout:1156: finishing write tid 2 to smithi15736696-34 2024-09-16T03:38:15.445 INFO:tasks.rados.rados.0.smithi157.stdout:1155: finishing write tid 1 to smithi15736696-40 2024-09-16T03:38:15.446 INFO:tasks.rados.rados.0.smithi157.stdout:1155: finishing write tid 2 to smithi15736696-40 2024-09-16T03:38:15.446 INFO:tasks.rados.rados.0.smithi157.stdout:1146: done (7 left) 2024-09-16T03:38:15.446 INFO:tasks.rados.rados.0.smithi157.stdout:1157: done (6 left) 2024-09-16T03:38:15.446 INFO:tasks.rados.rados.0.smithi157.stdout:1159: done (5 left) 2024-09-16T03:38:15.446 INFO:tasks.rados.rados.0.smithi157.stdout:1161: done (4 left) 2024-09-16T03:38:15.446 INFO:tasks.rados.rados.0.smithi157.stdout:1162: read oid 29 snap -1 2024-09-16T03:38:15.446 INFO:tasks.rados.rados.0.smithi157.stdout:1162: expect (ObjNum 106 snap 0 seq_num 106) 2024-09-16T03:38:15.446 INFO:tasks.rados.rados.0.smithi157.stdout:1163: read oid 12 snap -1 2024-09-16T03:38:15.446 INFO:tasks.rados.rados.0.smithi157.stdout:1163: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:15.446 INFO:tasks.rados.rados.0.smithi157.stdout:1164: set_chunk oid 8 target oid 2024-09-16T03:38:15.446 INFO:tasks.rados.rados.0.smithi157.stdout:1164: set_chunk oid 8 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:15.446 INFO:tasks.rados.rados.0.smithi157.stdout:1165: write oid 33 current snap is 0 2024-09-16T03:38:15.447 INFO:tasks.rados.rados.0.smithi157.stdout:1165: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-09-16T03:38:15.452 INFO:tasks.rados.rados.0.smithi157.stdout:1165: writing smithi15736696-33 from 679049 to 1397123 tid 1 2024-09-16T03:38:15.459 INFO:tasks.rados.rados.0.smithi157.stdout:1165: writing smithi15736696-33 from 2113302 to 2787690 tid 2 2024-09-16T03:38:15.463 INFO:tasks.rados.rados.0.smithi157.stdout:1165: writing smithi15736696-33 from 3335030 to 3479049 tid 3 2024-09-16T03:38:15.463 INFO:tasks.rados.rados.0.smithi157.stdout:1166: delete oid 11 current snap is 0 2024-09-16T03:38:15.463 INFO:tasks.rados.rados.0.smithi157.stdout:1156: finishing write tid 3 to smithi15736696-34 2024-09-16T03:38:15.464 INFO:tasks.rados.rados.0.smithi157.stdout:1165: finishing write tid 1 to smithi15736696-33 2024-09-16T03:38:15.481 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:15.480+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:15.516 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:15.516 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:15.516 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:15.517 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:15.517 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.517 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:15.517 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:15.517 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 63.370304681999997, 2024-09-16T03:38:15.517 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:38:15.517 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.517 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.517 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.517 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.517 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.517 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.518 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:15.518 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.518 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.518 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.518 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:15.518 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.518 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.518 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.518 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:38:15.518 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:38:15.519 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.519 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.519 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.519 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:38:15.519 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:38:15.519 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.519 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.519 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.519 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:38:15.519 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:38:15.519 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.519 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.519 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.520 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:38:15.520 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:38:15.520 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.520 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.520 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.520 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:38:15.520 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:38:15.520 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.520 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.520 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.520 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:38:15.520 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:38:15.520 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:38:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:38:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:38:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:38:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.522 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.522 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:38:15.522 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:15.522 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 63.369951321999999, 2024-09-16T03:38:15.522 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:38:15.522 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.522 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.522 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.522 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.522 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.522 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.522 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:15.523 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.523 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.523 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.523 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.523 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:15.523 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.523 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.523 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.523 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.523 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:38:15.523 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:38:15.523 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.523 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.524 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.524 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:38:15.524 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:38:15.524 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.524 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.524 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.524 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:38:15.524 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:38:15.524 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.524 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.524 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.524 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:38:15.524 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:38:15.525 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.525 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.525 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.525 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:38:15.525 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:38:15.525 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.525 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.525 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.525 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:38:15.525 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:38:15.525 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.525 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.525 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.526 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:38:15.526 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:38:15.526 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.526 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.526 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.526 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:38:15.526 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:38:15.526 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.526 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.526 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.526 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.526 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.526 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:38:15.527 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:15.527 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 63.368823524, 2024-09-16T03:38:15.527 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:38:15.527 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.527 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.527 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.527 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.527 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.527 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.527 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:15.527 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.527 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.527 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.528 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.528 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:15.528 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.528 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.528 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.528 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.528 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:38:15.528 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:38:15.528 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.528 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.528 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.528 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:38:15.528 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:38:15.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.529 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.529 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.529 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:38:15.529 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:38:15.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.529 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.529 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.529 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:38:15.529 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:38:15.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.529 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.529 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.530 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:38:15.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:38:15.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.530 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.530 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.530 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:38:15.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:38:15.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.530 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.530 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.530 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:38:15.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:38:15.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.531 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.531 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:38:15.531 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:38:15.531 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.531 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.531 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.531 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.531 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:38:15.531 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:15.531 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 63.367836083, 2024-09-16T03:38:15.531 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:38:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:15.532 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.533 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.533 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.533 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.533 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:38:15.533 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:38:15.533 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.533 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.533 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.533 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:38:15.533 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:38:15.533 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.533 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.533 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.534 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:38:15.534 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:38:15.534 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.534 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.534 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.534 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:38:15.534 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:38:15.534 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.534 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.534 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.534 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:38:15.534 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:38:15.534 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:38:15.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 63.367079339, 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.536 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.537 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.537 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:15.537 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.537 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.537 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.537 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.537 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:15.537 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.537 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.537 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.537 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.538 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:38:15.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:38:15.538 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.538 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.538 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.538 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:38:15.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:38:15.538 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.538 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.538 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.538 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:38:15.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:38:15.538 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:38:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:38:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:38:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:38:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.539 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:38:15.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:38:15.540 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.540 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.540 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.540 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:38:15.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:38:15.540 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.540 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.540 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.540 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:38:15.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:38:15.540 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.540 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:86 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'8, mlcod=39'5)", 2024-09-16T03:38:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.905567103000003, 2024-09-16T03:38:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20388281899999999, 2024-09-16T03:38:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.541 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608472+0000", 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.227e-06 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.542 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.543 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608616+0000", 2024-09-16T03:38:15.543 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000144177 2024-09-16T03:38:15.543 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.543 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.543 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.544 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608638+0000", 2024-09-16T03:38:15.544 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1543e-05 2024-09-16T03:38:15.544 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.544 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.544 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.544 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608645+0000", 2024-09-16T03:38:15.544 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5839999999999997e-06 2024-09-16T03:38:15.544 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.544 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.544 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.544 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608740+0000", 2024-09-16T03:38:15.544 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4716999999999995e-05 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608763+0000", 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2671e-05 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812264+0000", 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203501194 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812353+0000", 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8706000000000001e-05 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:87 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'9, mlcod=39'5)", 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.905353728999998, 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20372206400000001, 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.546 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608684+0000", 2024-09-16T03:38:15.547 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6700000000000003e-07 2024-09-16T03:38:15.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.548 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.548 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608689+0000", 2024-09-16T03:38:15.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.412e-06 2024-09-16T03:38:15.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.548 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.548 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608704+0000", 2024-09-16T03:38:15.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4397e-05 2024-09-16T03:38:15.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.548 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.549 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608713+0000", 2024-09-16T03:38:15.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:38:15.549 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.549 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.549 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710025+0000", 2024-09-16T03:38:15.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101312032 2024-09-16T03:38:15.549 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.549 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.549 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710042+0000", 2024-09-16T03:38:15.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6889e-05 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812361+0000", 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10231889600000001 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812405+0000", 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4101999999999999e-05 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.550 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.551 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.551 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:101 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'10, mlcod=39'5)", 2024-09-16T03:38:15.551 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:15.551 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.904340292000001, 2024-09-16T03:38:15.551 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20275586900000001, 2024-09-16T03:38:15.551 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.551 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.551 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.551 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.551 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.551 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.551 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:15.551 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.552 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.552 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.552 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:15.552 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.552 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.552 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.552 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609698+0000", 2024-09-16T03:38:15.552 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.24e-06 2024-09-16T03:38:15.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.553 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.553 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.553 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609892+0000", 2024-09-16T03:38:15.553 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019465899999999999 2024-09-16T03:38:15.553 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.553 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.553 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.553 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609904+0000", 2024-09-16T03:38:15.553 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2031000000000001e-05 2024-09-16T03:38:15.553 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.553 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.553 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.553 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609912+0000", 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7160000000000003e-06 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710228+0000", 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100315682 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710242+0000", 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4201e-05 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.554 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812411+0000", 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10216881799999999 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812452+0000", 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1522e-05 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:102 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'11, mlcod=39'5)", 2024-09-16T03:38:15.555 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.902324266999997, 2024-09-16T03:38:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20134195399999999, 2024-09-16T03:38:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:15.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611714+0000", 2024-09-16T03:38:15.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.269e-06 2024-09-16T03:38:15.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611926+0000", 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021252700000000001 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611938+0000", 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1437000000000001e-05 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611944+0000", 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.993e-06 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.558 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811266+0000", 2024-09-16T03:38:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.199322306 2024-09-16T03:38:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811291+0000", 2024-09-16T03:38:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4542999999999999e-05 2024-09-16T03:38:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813021+0000", 2024-09-16T03:38:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001730363 2024-09-16T03:38:15.559 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.560 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.560 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813054+0000", 2024-09-16T03:38:15.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3516000000000002e-05 2024-09-16T03:38:15.560 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.560 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.560 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.560 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.560 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:103 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'12, mlcod=39'5)", 2024-09-16T03:38:15.560 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:38:15.560 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.899117766000003, 2024-09-16T03:38:15.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19838012999999999, 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.561 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614920+0000", 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.192e-06 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614927+0000", 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4640000000000003e-06 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614937+0000", 2024-09-16T03:38:15.562 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9599999999999995e-06 2024-09-16T03:38:15.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.563 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.563 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614942+0000", 2024-09-16T03:38:15.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.31e-06 2024-09-16T03:38:15.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.563 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.563 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812489+0000", 2024-09-16T03:38:15.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.197546629 2024-09-16T03:38:15.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.563 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812511+0000", 2024-09-16T03:38:15.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2385e-05 2024-09-16T03:38:15.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.564 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.564 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813279+0000", 2024-09-16T03:38:15.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076818100000000005 2024-09-16T03:38:15.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.564 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.564 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813299+0000", 2024-09-16T03:38:15.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0009000000000001e-05 2024-09-16T03:38:15.564 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.565 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.565 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.565 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:375 4.f e45/32 4:fcb75e43:::smithi15736696-26:head v 45'23, mlcod=41'20)", 2024-09-16T03:38:15.565 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:15.565 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 55.205996538999997, 2024-09-16T03:38:15.565 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200786568, 2024-09-16T03:38:15.565 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.565 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.565 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.565 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:15.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:15.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308041+0000", 2024-09-16T03:38:15.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8299999999999995e-07 2024-09-16T03:38:15.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308046+0000", 2024-09-16T03:38:15.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.904e-06 2024-09-16T03:38:15.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308056+0000", 2024-09-16T03:38:15.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9210000000000004e-06 2024-09-16T03:38:15.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.568 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.408122+0000", 2024-09-16T03:38:15.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100066302 2024-09-16T03:38:15.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.568 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.568 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.568 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508289+0000", 2024-09-16T03:38:15.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166531 2024-09-16T03:38:15.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.568 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.568 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.568 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508302+0000", 2024-09-16T03:38:15.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2641e-05 2024-09-16T03:38:15.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.569 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.569 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508794+0000", 2024-09-16T03:38:15.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00049272400000000005 2024-09-16T03:38:15.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.569 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.569 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508827+0000", 2024-09-16T03:38:15.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2561999999999998e-05 2024-09-16T03:38:15.569 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.569 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.569 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:989 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'31, mlcod=46'28)", 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 45.126227978999999, 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40332266900000002, 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:15.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.571 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:15.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.571 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387812+0000", 2024-09-16T03:38:15.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:38:15.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388467+0000", 2024-09-16T03:38:15.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065539500000000002 2024-09-16T03:38:15.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.572 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388481+0000", 2024-09-16T03:38:15.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3905e-05 2024-09-16T03:38:15.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.572 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488552+0000", 2024-09-16T03:38:15.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10007120899999999 2024-09-16T03:38:15.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688818+0000", 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20026627599999999 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688845+0000", 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791091+0000", 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10224588699999999 2024-09-16T03:38:15.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791131+0000", 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0293000000000002e-05 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:990 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'32, mlcod=46'28)", 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 45.025444307000001, 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30256419200000001, 2024-09-16T03:38:15.574 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.576 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488594+0000", 2024-09-16T03:38:15.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:38:15.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.576 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488616+0000", 2024-09-16T03:38:15.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2419e-05 2024-09-16T03:38:15.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.576 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488627+0000", 2024-09-16T03:38:15.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1251e-05 2024-09-16T03:38:15.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588753+0000", 2024-09-16T03:38:15.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012544399999999 2024-09-16T03:38:15.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690679+0000", 2024-09-16T03:38:15.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101926027 2024-09-16T03:38:15.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690695+0000", 2024-09-16T03:38:15.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6528000000000002e-05 2024-09-16T03:38:15.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.578 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791133+0000", 2024-09-16T03:38:15.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100438233 2024-09-16T03:38:15.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.578 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791157+0000", 2024-09-16T03:38:15.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-05 2024-09-16T03:38:15.579 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.579 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.579 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.579 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.579 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.579 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:991 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'33, mlcod=46'28)", 2024-09-16T03:38:15.579 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:15.579 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 44.925246809999997, 2024-09-16T03:38:15.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202786365, 2024-09-16T03:38:15.579 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.579 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.579 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.580 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.580 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:15.580 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.580 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:15.580 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.581 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588792+0000", 2024-09-16T03:38:15.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0530000000000001e-06 2024-09-16T03:38:15.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.581 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.581 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588807+0000", 2024-09-16T03:38:15.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5257e-05 2024-09-16T03:38:15.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.581 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.581 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588832+0000", 2024-09-16T03:38:15.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5123e-05 2024-09-16T03:38:15.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588846+0000", 2024-09-16T03:38:15.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3312999999999999e-05 2024-09-16T03:38:15.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790810+0000", 2024-09-16T03:38:15.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20196378100000001 2024-09-16T03:38:15.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790866+0000", 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6168999999999997e-05 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791542+0000", 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067583699999999999 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791576+0000", 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4832000000000001e-05 2024-09-16T03:38:15.583 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.584 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.584 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.584 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1072 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'35, mlcod=50'32)", 2024-09-16T03:38:15.584 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:38:15.584 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 43.206004983, 2024-09-16T03:38:15.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.103370715, 2024-09-16T03:38:15.584 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.584 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.584 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.584 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:38:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:38:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308034+0000", 2024-09-16T03:38:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4099999999999998e-06 2024-09-16T03:38:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308377+0000", 2024-09-16T03:38:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034276400000000001 2024-09-16T03:38:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308415+0000", 2024-09-16T03:38:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7916000000000001e-05 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408542+0000", 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012673699999999 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408694+0000", 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015200399999999999 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408760+0000", 2024-09-16T03:38:15.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6549000000000003e-05 2024-09-16T03:38:15.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.588 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.588 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.411366+0000", 2024-09-16T03:38:15.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002605324 2024-09-16T03:38:15.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.588 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.588 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.411403+0000", 2024-09-16T03:38:15.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7011000000000002e-05 2024-09-16T03:38:15.588 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.589 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.589 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.589 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1580 4.f e61/32 4:fa163101:::smithi15736696-10:head v 61'52, mlcod=58'50)", 2024-09-16T03:38:15.589 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:15.589 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.408415284, 2024-09-16T03:38:15.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.201787367, 2024-09-16T03:38:15.589 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.589 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.589 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.589 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2600000000000001e-07 2024-09-16T03:38:15.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106083+0000", 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00046163900000000002 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106093+0000", 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9320000000000008e-06 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206190+0000", 2024-09-16T03:38:15.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100096716 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206286+0000", 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5982000000000004e-05 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206312+0000", 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6279999999999999e-05 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307345+0000", 2024-09-16T03:38:15.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10103240199999999 2024-09-16T03:38:15.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307409+0000", 2024-09-16T03:38:15.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4189999999999994e-05 2024-09-16T03:38:15.593 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.593 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.593 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.593 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1581 4.f e61/32 4:fa163101:::smithi15736696-10:head v 61'53, mlcod=58'50)", 2024-09-16T03:38:15.593 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:47.206244+0000", 2024-09-16T03:38:15.593 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.307792753000001, 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101636226, 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206244+0000", 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206244+0000", 2024-09-16T03:38:15.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206246+0000", 2024-09-16T03:38:15.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7549999999999999e-06 2024-09-16T03:38:15.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206367+0000", 2024-09-16T03:38:15.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012166 2024-09-16T03:38:15.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206375+0000", 2024-09-16T03:38:15.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.571e-06 2024-09-16T03:38:15.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206380+0000", 2024-09-16T03:38:15.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5210000000000004e-06 2024-09-16T03:38:15.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.306514+0000", 2024-09-16T03:38:15.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10013456 2024-09-16T03:38:15.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.306531+0000", 2024-09-16T03:38:15.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.702e-05 2024-09-16T03:38:15.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307860+0000", 2024-09-16T03:38:15.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013284080000000001 2024-09-16T03:38:15.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307880+0000", 2024-09-16T03:38:15.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0730999999999999e-05 2024-09-16T03:38:15.598 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.598 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.598 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1582 4.f e61/32 4:fa163101:::smithi15736696-10:head v 61'54, mlcod=58'50)", 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:47.210745+0000", 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.303291563999998, 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101062938, 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210745+0000", 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210745+0000", 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210747+0000", 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.494e-06 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210754+0000", 2024-09-16T03:38:15.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1069999999999999e-06 2024-09-16T03:38:15.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210767+0000", 2024-09-16T03:38:15.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3054000000000001e-05 2024-09-16T03:38:15.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.210774+0000", 2024-09-16T03:38:15.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4959999999999999e-06 2024-09-16T03:38:15.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.310881+0000", 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100106371 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.310900+0000", 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9666000000000001e-05 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.311778+0000", 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00087738400000000004 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.311808+0000", 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0366e-05 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1760 4.f e71/32 4:f101f67d:::smithi15736696-22:head v 71'55, mlcod=61'53)", 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.133211318000001, 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20118894600000001, 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.603 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380826+0000", 2024-09-16T03:38:15.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1899999999999998e-07 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380831+0000", 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6049999999999997e-06 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380851+0000", 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0021e-05 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380862+0000", 2024-09-16T03:38:15.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0458e-05 2024-09-16T03:38:15.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581184+0000", 2024-09-16T03:38:15.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20032164299999999 2024-09-16T03:38:15.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581202+0000", 2024-09-16T03:38:15.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8410000000000002e-05 2024-09-16T03:38:15.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581965+0000", 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076291200000000003 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.582014+0000", 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9577999999999999e-05 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.608 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2112 4.12 4:4a9d235d:::smithi15736696-1:head [sparse-read 0~1423481,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:38:15.608 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:05.542212+0000", 2024-09-16T03:38:15.608 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.9718252009999997, 2024-09-16T03:38:15.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10240218600000001, 2024-09-16T03:38:15.608 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:15.608 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:15.608 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:15.608 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:15.608 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:15.608 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:15.608 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2112 2024-09-16T03:38:15.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.609 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:15.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:15.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.542212+0000", 2024-09-16T03:38:15.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:15.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.542212+0000", 2024-09-16T03:38:15.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:15.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:15.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.542215+0000", 2024-09-16T03:38:15.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6430000000000001e-06 2024-09-16T03:38:15.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:15.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.542225+0000", 2024-09-16T03:38:15.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0010000000000001e-05 2024-09-16T03:38:15.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:15.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.542243+0000", 2024-09-16T03:38:15.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7861000000000001e-05 2024-09-16T03:38:15.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.642406+0000", 2024-09-16T03:38:15.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100162559 2024-09-16T03:38:15.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:15.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.642547+0000", 2024-09-16T03:38:15.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014096099999999999 2024-09-16T03:38:15.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:15.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.642638+0000", 2024-09-16T03:38:15.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1836999999999995e-05 2024-09-16T03:38:15.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:15.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:15.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:15.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.644614+0000", 2024-09-16T03:38:15.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019753150000000001 2024-09-16T03:38:15.612 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.612 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.612 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.612 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:15.612 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:15.612 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:15.614 INFO:tasks.rados.rados.0.smithi157.stdout:1160: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:15.614 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:15.614 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1915717 2024-09-16T03:38:15.614 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:15.612+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:15.615 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:38:15.616 INFO:tasks.rados.rados.0.smithi157.stdout:1156: finishing write tid 4 to smithi15736696-34 2024-09-16T03:38:15.616 INFO:tasks.rados.rados.0.smithi157.stdout:1156: finishing write tid 5 to smithi15736696-34 2024-09-16T03:38:15.616 INFO:tasks.rados.rados.0.smithi157.stdout:1156: oid 34 updating version 0 to 84 2024-09-16T03:38:15.616 INFO:tasks.rados.rados.0.smithi157.stdout:1156: oid 34 updating version 84 to 87 2024-09-16T03:38:15.616 INFO:tasks.rados.rados.0.smithi157.stdout:1156: oid 34 version 87 is already newer than 85 2024-09-16T03:38:15.616 INFO:tasks.rados.rados.0.smithi157.stdout:1156: oid 34 version 87 is already newer than 86 2024-09-16T03:38:15.616 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 87 (ObjNum 205 snap 0 seq_num 205) dirty exists 2024-09-16T03:38:15.616 INFO:tasks.rados.rados.0.smithi157.stdout:1156: left oid 34 (ObjNum 205 snap 0 seq_num 205) 2024-09-16T03:38:15.616 INFO:tasks.rados.rados.0.smithi157.stdout:1158: expect (ObjNum 196 snap 0 seq_num 196) 2024-09-16T03:38:15.662 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3256665 2024-09-16T03:38:15.695 INFO:tasks.rados.rados.0.smithi157.stdout:1162: expect (ObjNum 106 snap 0 seq_num 106) 2024-09-16T03:38:15.695 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:15.715 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:15.713+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:15.721 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:15.722 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:15.722 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:38:15.722 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:15.730 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:38:15.768 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:15.811 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:38:15.812 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T03:38:15.826 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T03:38:15.827 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:15.824+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:15.873 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:15.873 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:15.873 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:38:15.873 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:38:15.874 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:15.882 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:38:15.898 INFO:tasks.rados.rados.0.smithi157.stdout:1164: finishing set_chunk to oid 8 2024-09-16T03:38:15.899 INFO:tasks.rados.rados.0.smithi157.stdout:1164: got expected ENOENT (src dne) 2024-09-16T03:38:15.899 INFO:tasks.rados.rados.0.smithi157.stdout:1155: finishing write tid 3 to smithi15736696-40 2024-09-16T03:38:15.899 INFO:tasks.rados.rados.0.smithi157.stdout:1155: finishing write tid 4 to smithi15736696-40 2024-09-16T03:38:15.899 INFO:tasks.rados.rados.0.smithi157.stdout:1155: finishing write tid 5 to smithi15736696-40 2024-09-16T03:38:15.899 INFO:tasks.rados.rados.0.smithi157.stdout:1155: oid 40 updating version 0 to 84 2024-09-16T03:38:15.899 INFO:tasks.rados.rados.0.smithi157.stdout:1155: oid 40 updating version 84 to 85 2024-09-16T03:38:15.899 INFO:tasks.rados.rados.0.smithi157.stdout:1155: oid 40 version 85 is already newer than 83 2024-09-16T03:38:15.899 INFO:tasks.rados.rados.0.smithi157.stdout:1155: oid 40 version 85 is already newer than 82 2024-09-16T03:38:15.899 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 85 (ObjNum 204 snap 0 seq_num 204) dirty exists 2024-09-16T03:38:15.899 INFO:tasks.rados.rados.0.smithi157.stdout:1155: left oid 40 (ObjNum 204 snap 0 seq_num 204) 2024-09-16T03:38:15.899 INFO:tasks.rados.rados.0.smithi157.stdout:1155: done (8 left) 2024-09-16T03:38:15.900 INFO:tasks.rados.rados.0.smithi157.stdout:1156: done (7 left) 2024-09-16T03:38:15.900 INFO:tasks.rados.rados.0.smithi157.stdout:1158: done (6 left) 2024-09-16T03:38:15.900 INFO:tasks.rados.rados.0.smithi157.stdout:1160: done (5 left) 2024-09-16T03:38:15.900 INFO:tasks.rados.rados.0.smithi157.stdout:1162: done (4 left) 2024-09-16T03:38:15.900 INFO:tasks.rados.rados.0.smithi157.stdout:1164: done (3 left) 2024-09-16T03:38:15.900 INFO:tasks.rados.rados.0.smithi157.stdout:1166: done (2 left) 2024-09-16T03:38:15.900 INFO:tasks.rados.rados.0.smithi157.stdout:1167: read oid 5 snap -1 2024-09-16T03:38:15.900 INFO:tasks.rados.rados.0.smithi157.stdout:1167: expect deleted 2024-09-16T03:38:15.900 INFO:tasks.rados.rados.0.smithi157.stdout:1168: write (excl) oid 13 current snap is 0 2024-09-16T03:38:15.900 INFO:tasks.rados.rados.0.smithi157.stdout:1163: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:15.916 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:15.915+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:15.931 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:15.960 INFO:tasks.rados.rados.0.smithi157.stdout:1165: finishing write tid 2 to smithi15736696-33 2024-09-16T03:38:15.960 INFO:tasks.rados.rados.0.smithi157.stdout:1165: finishing write tid 3 to smithi15736696-33 2024-09-16T03:38:15.960 INFO:tasks.rados.rados.0.smithi157.stdout:1165: finishing write tid 4 to smithi15736696-33 2024-09-16T03:38:15.960 INFO:tasks.rados.rados.0.smithi157.stdout:1165: finishing write tid 5 to smithi15736696-33 2024-09-16T03:38:15.960 INFO:tasks.rados.rados.0.smithi157.stdout:1165: oid 33 updating version 0 to 50 2024-09-16T03:38:15.960 INFO:tasks.rados.rados.0.smithi157.stdout:1165: oid 33 updating version 50 to 51 2024-09-16T03:38:15.960 INFO:tasks.rados.rados.0.smithi157.stdout:1165: oid 33 version 51 is already newer than 48 2024-09-16T03:38:15.960 INFO:tasks.rados.rados.0.smithi157.stdout:1165: oid 33 version 51 is already newer than 49 2024-09-16T03:38:15.960 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 51 (ObjNum 207 snap 0 seq_num 207) dirty exists 2024-09-16T03:38:15.960 INFO:tasks.rados.rados.0.smithi157.stdout:1165: left oid 33 (ObjNum 207 snap 0 seq_num 207) 2024-09-16T03:38:15.960 INFO:tasks.rados.rados.0.smithi157.stdout:1168: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-09-16T03:38:15.966 INFO:tasks.rados.rados.0.smithi157.stdout:1168: writing smithi15736696-13 from 712663 to 1181235 tid 1 2024-09-16T03:38:15.973 INFO:tasks.rados.rados.0.smithi157.stdout:1168: writing smithi15736696-13 from 1713913 to 2391970 tid 2 2024-09-16T03:38:15.979 INFO:tasks.rados.rados.0.smithi157.stdout:1168: writing smithi15736696-13 from 2873523 to 3465256 tid 3 2024-09-16T03:38:15.979 INFO:tasks.rados.rados.0.smithi157.stdout:1163: done (3 left) 2024-09-16T03:38:15.979 INFO:tasks.rados.rados.0.smithi157.stdout:1165: done (2 left) 2024-09-16T03:38:15.979 INFO:tasks.rados.rados.0.smithi157.stdout:1169: read oid 2 snap -1 2024-09-16T03:38:15.979 INFO:tasks.rados.rados.0.smithi157.stdout:1169: expect (ObjNum 201 snap 0 seq_num 201) 2024-09-16T03:38:15.979 INFO:tasks.rados.rados.0.smithi157.stdout:1170: write (excl) oid 43 current snap is 0 2024-09-16T03:38:15.979 INFO:tasks.rados.rados.0.smithi157.stdout:1170: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2024-09-16T03:38:15.985 INFO:tasks.rados.rados.0.smithi157.stdout:1170: writing smithi15736696-43 from 729470 to 1273291 tid 1 2024-09-16T03:38:15.991 INFO:tasks.rados.rados.0.smithi157.stdout:1170: writing smithi15736696-43 from 1772395 to 2310463 tid 2 2024-09-16T03:38:15.992 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:15.993 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:38:15.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:38:15.993 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:15.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.993 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:14 4.a e39/32 4:51338165:::smithi15736696-4:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:15.993 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:11.997261+0000", 2024-09-16T03:38:15.993 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.992601155000003, 2024-09-16T03:38:15.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185554500000001, 2024-09-16T03:38:15.993 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:15.993 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:15.993 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.993 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997261+0000", 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997261+0000", 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997263+0000", 2024-09-16T03:38:15.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.373e-06 2024-09-16T03:38:15.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:15.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:15.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997764+0000", 2024-09-16T03:38:15.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00050155299999999996 2024-09-16T03:38:15.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:15.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:15.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:11.997775+0000", 2024-09-16T03:38:15.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1066999999999999e-05 2024-09-16T03:38:15.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:15.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:15.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097841+0000", 2024-09-16T03:38:15.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100066316 2024-09-16T03:38:15.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:15.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:15.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097882+0000", 2024-09-16T03:38:15.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0850999999999999e-05 2024-09-16T03:38:15.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:15.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:15.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.097904+0000", 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1347000000000001e-05 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.099089+0000", 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001185195 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.099117+0000", 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7843000000000001e-05 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:15.997 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:15.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:15.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.998 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:509 4.f e45/32 4:fa163101:::smithi15736696-10:head v 45'26, mlcod=45'24)", 2024-09-16T03:38:15.998 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:22.024903+0000", 2024-09-16T03:38:15.998 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 53.964958920000001, 2024-09-16T03:38:15.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101058093, 2024-09-16T03:38:15.998 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:15.998 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:15.998 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:15.998 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:15.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:15.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024903+0000", 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024903+0000", 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024905+0000", 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1659999999999999e-06 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:15.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024913+0000", 2024-09-16T03:38:16.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9840000000000001e-06 2024-09-16T03:38:16.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024923+0000", 2024-09-16T03:38:16.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0811e-05 2024-09-16T03:38:16.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.024929+0000", 2024-09-16T03:38:16.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5509999999999999e-06 2024-09-16T03:38:16.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.125071+0000", 2024-09-16T03:38:16.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10014179300000001 2024-09-16T03:38:16.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.125095+0000", 2024-09-16T03:38:16.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3944999999999999e-05 2024-09-16T03:38:16.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.125926+0000", 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00083087799999999996 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:22.125961+0000", 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5964999999999999e-05 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.5.21:22 3.1 e47/31 3:adc537a6:::smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 47'46, mlcod=47'42)", 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.686705+0000", 2024-09-16T03:38:16.002 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 49.303157718000001, 2024-09-16T03:38:16.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10113362200000001, 2024-09-16T03:38:16.003 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.003 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.003 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.003 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686705+0000", 2024-09-16T03:38:16.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686705+0000", 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686705+0000", 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1699999999999998e-07 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686709+0000", 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6279999999999998e-06 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.686716+0000", 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3730000000000004e-06 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.786814+0000", 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100097964 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.786880+0000", 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5883999999999995e-05 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.786909+0000", 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8764000000000001e-05 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.787802+0000", 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00089355100000000002 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.787838+0000", 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5840999999999998e-05 2024-09-16T03:38:16.006 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.007 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.007 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.007 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:983 3.1 e50/31 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head v 50'54, mlcod=49'52)", 2024-09-16T03:38:16.007 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:38:16.007 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 45.92354461, 2024-09-16T03:38:16.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.105768445, 2024-09-16T03:38:16.007 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.007 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.007 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.007 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7700000000000001e-07 2024-09-16T03:38:16.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.070158+0000", 2024-09-16T03:38:16.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0038403170000000002 2024-09-16T03:38:16.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.070168+0000", 2024-09-16T03:38:16.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9019999999999994e-06 2024-09-16T03:38:16.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.070178+0000", 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6860000000000001e-06 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.170328+0000", 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150196 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.170363+0000", 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4588999999999997e-05 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.172053+0000", 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016902900000000001 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.172086+0000", 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3287999999999997e-05 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:988 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'30, mlcod=46'28)", 2024-09-16T03:38:16.011 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.381297+0000", 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 45.608565523000003, 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10422211300000001, 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.381297+0000", 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.381297+0000", 2024-09-16T03:38:16.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.381298+0000", 2024-09-16T03:38:16.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5409999999999999e-06 2024-09-16T03:38:16.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.382086+0000", 2024-09-16T03:38:16.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078808600000000002 2024-09-16T03:38:16.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.382097+0000", 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0557000000000001e-05 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.482188+0000", 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100091478 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.482261+0000", 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2992999999999997e-05 2024-09-16T03:38:16.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.482287+0000", 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5211999999999998e-05 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.485483+0000", 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031964939999999998 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.485519+0000", 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5751999999999999e-05 2024-09-16T03:38:16.015 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1071 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'34, mlcod=50'32)", 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:32.306208+0000", 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.683654394999998, 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103114758, 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.306208+0000", 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.306208+0000", 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.306210+0000", 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6619999999999999e-06 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307136+0000", 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092664400000000001 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307149+0000", 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2615e-05 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307159+0000", 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0341e-05 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307221+0000", 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1871000000000004e-05 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307239+0000", 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7876999999999998e-05 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409282+0000", 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102042801 2024-09-16T03:38:16.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409323+0000", 2024-09-16T03:38:16.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0946999999999999e-05 2024-09-16T03:38:16.020 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.020 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.020 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.020 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1072 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'35, mlcod=50'32)", 2024-09-16T03:38:16.020 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:38:16.020 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.681918732, 2024-09-16T03:38:16.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10192738899999999, 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9999999999999996e-07 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.308383+0000", 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000438794 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.308391+0000", 2024-09-16T03:38:16.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6149999999999999e-06 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.308396+0000", 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1719999999999998e-06 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.408530+0000", 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013438700000001 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.408547+0000", 2024-09-16T03:38:16.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6554000000000001e-05 2024-09-16T03:38:16.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409844+0000", 2024-09-16T03:38:16.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001297166 2024-09-16T03:38:16.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409871+0000", 2024-09-16T03:38:16.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6900999999999999e-05 2024-09-16T03:38:16.024 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1073 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'36, mlcod=50'32)", 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:32.311182+0000", 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.678680374999999, 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101901307, 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311182+0000", 2024-09-16T03:38:16.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311182+0000", 2024-09-16T03:38:16.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311183+0000", 2024-09-16T03:38:16.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:38:16.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311720+0000", 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053692400000000004 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311728+0000", 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7849999999999992e-06 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.411763+0000", 2024-09-16T03:38:16.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100035393 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.411838+0000", 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4658000000000002e-05 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.411853+0000", 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5268e-05 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.413058+0000", 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001204751 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.413083+0000", 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5609000000000002e-05 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1193 4.a e54/32 4:503187b3:::smithi15736696-34:head v 54'43, mlcod=53'41)", 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:36.160789+0000", 2024-09-16T03:38:16.029 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 39.829073407000003, 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10275106000000001, 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.160789+0000", 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.160789+0000", 2024-09-16T03:38:16.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.160791+0000", 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.638e-06 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.161609+0000", 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00081820899999999997 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.161635+0000", 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5939999999999999e-05 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.261770+0000", 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100135416 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.261913+0000", 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000142695 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.261950+0000", 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7324999999999999e-05 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.263493+0000", 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015430800000000001 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.263540+0000", 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6757e-05 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.033 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1515 3.1 e59/31 3:9c9e970c:::7249cb65b52cadf4fe8f4ec923c7e548d1de7d9628823acd2c8de19121f1c1ce:head v 59'69, mlcod=56'67)", 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 31.038211902, 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.203376156, 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:38:16.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951651+0000", 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8299999999999996e-07 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953610+0000", 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019588769999999999 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953622+0000", 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1846e-05 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953632+0000", 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0202e-05 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.053808+0000", 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100175615 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.053846+0000", 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8383000000000002e-05 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.155006+0000", 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101160131 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.155027+0000", 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0319e-05 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1704 3.1 e65/31 3:b6a94245:::f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99:head v 65'77, mlcod=64'75)", 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.196076628, 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30251757499999998, 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.038 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793788+0000", 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794391+0000", 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060354199999999999 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794406+0000", 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5045000000000001e-05 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794432+0000", 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5996999999999999e-05 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.994704+0000", 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20027177700000001 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.994719+0000", 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5021000000000001e-05 2024-09-16T03:38:16.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096277+0000", 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101557437 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096303+0000", 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6732000000000001e-05 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.042 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:84 3.1 e65/31 3:b24f289d:::e84532c28f467d2f2a6e2cf44904fa0d4a555e1304bbe1b25db645af42152bd7:head v 65'78, mlcod=64'75)", 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.187061464999999, 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29353782099999998, 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802805+0000", 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1200000000000004e-06 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803910+0000", 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011051660000000001 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803928+0000", 2024-09-16T03:38:16.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7880000000000002e-05 2024-09-16T03:38:16.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803938+0000", 2024-09-16T03:38:16.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6690000000000001e-06 2024-09-16T03:38:16.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095473+0000", 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29153552399999999 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095486+0000", 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2455e-05 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096306+0000", 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00082031900000000002 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096339+0000", 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2688000000000003e-05 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:85 3.1 e65/31 3:b380f269:::e65351e8a65a0b1a517e3b1a85e04b84c0ab7cfa6655f11fb0da31f56b4dd9af:head v 65'79, mlcod=64'75)", 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.185780887, 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29260815400000001, 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.047 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:38:16.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:38:16.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804083+0000", 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804266+0000", 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018268099999999999 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804275+0000", 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4439999999999997e-06 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904371+0000", 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100095276 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095883+0000", 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.191512444 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095901+0000", 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8462999999999999e-05 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096658+0000", 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00075622000000000005 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096690+0000", 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1919999999999999e-05 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:86 3.1 e65/31 3:b3acb53e:::e951eb69ce5151a3f54107fa22c70f80729eff845da1613447a91536a24046e8:head v 65'80, mlcod=64'75)", 2024-09-16T03:38:16.051 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.085477471000001, 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29244589199999999, 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:38:16.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904386+0000", 2024-09-16T03:38:16.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3e-07 2024-09-16T03:38:16.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904537+0000", 2024-09-16T03:38:16.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015122700000000001 2024-09-16T03:38:16.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904544+0000", 2024-09-16T03:38:16.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.024e-06 2024-09-16T03:38:16.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904547+0000", 2024-09-16T03:38:16.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6540000000000001e-06 2024-09-16T03:38:16.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196016+0000", 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29146812599999999 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196027+0000", 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1172000000000001e-05 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196791+0000", 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00076418700000000005 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196831+0000", 2024-09-16T03:38:16.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9771999999999998e-05 2024-09-16T03:38:16.056 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.056 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.056 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.056 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:91 3.0 e70/69 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head v 70'105, mlcod=0'0)", 2024-09-16T03:38:16.056 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:55.960027+0000", 2024-09-16T03:38:16.056 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 20.029834962999999, 2024-09-16T03:38:16.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200596892, 2024-09-16T03:38:16.056 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.056 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960027+0000", 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960027+0000", 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960030+0000", 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9909999999999998e-06 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960043+0000", 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2428e-05 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960076+0000", 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3494999999999997e-05 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960102+0000", 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5692000000000001e-05 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960181+0000", 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8827999999999998e-05 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960200+0000", 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9463e-05 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.160597+0000", 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200396611 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.160624+0000", 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7384000000000001e-05 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.060 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:94 3.0 e70/69 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head v 70'106, mlcod=0'0)", 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:55.960145+0000", 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 20.029717776999998, 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20090359099999999, 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960145+0000", 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960145+0000", 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960145+0000", 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7300000000000002e-07 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960147+0000", 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.509e-06 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960151+0000", 2024-09-16T03:38:16.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:38:16.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.060262+0000", 2024-09-16T03:38:16.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100110847 2024-09-16T03:38:16.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.160465+0000", 2024-09-16T03:38:16.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100203264 2024-09-16T03:38:16.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.160483+0000", 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.753e-05 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.161026+0000", 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00054326199999999998 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.161048+0000", 2024-09-16T03:38:16.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1951999999999999e-05 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.11.23:98 3.0 e74/69 3:04dfaaf9:::3de53e1d584ced9b0f970855867dbc9a14b3fc90962aa20a81c1ed25ff4b961f:head v 74'124, mlcod=73'121)", 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:01.337812+0000", 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.652050009, 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20064802300000001, 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.337812+0000", 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.337812+0000", 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.337813+0000", 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2699999999999998e-07 2024-09-16T03:38:16.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.337818+0000", 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4440000000000001e-06 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.337825+0000", 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0269999999999998e-06 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.437892+0000", 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100067131 2024-09-16T03:38:16.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.538006+0000", 2024-09-16T03:38:16.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100114093 2024-09-16T03:38:16.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.538030+0000", 2024-09-16T03:38:16.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3944e-05 2024-09-16T03:38:16.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.538424+0000", 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00039373000000000001 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.538460+0000", 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6727000000000003e-05 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.11.23:103 3.0 e74/69 3:28447854:::c811732c201b76a96e1532c4d9a18d4f4f2a8ddec3bf79945689a1a62c8a5a25:head v 74'127, mlcod=73'121)", 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:01.538557+0000", 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.451305066, 2024-09-16T03:38:16.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101549632, 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.538557+0000", 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.538557+0000", 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.538558+0000", 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8000000000000001e-07 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.538562+0000", 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0799999999999999e-06 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.538566+0000", 2024-09-16T03:38:16.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0790000000000002e-06 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.638640+0000", 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10007418899999999 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.639585+0000", 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094462099999999998 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.639609+0000", 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4219e-05 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.639994+0000", 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038509299999999998 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.640107+0000", 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112971 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.073 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.12.21:43 3.0 e82/69 3:380dcb5f:::c4b22a462d325aa5e31abdf141f18fa7703aa0c14d08fb31b9a2baff15827a7a:head v 82'160, mlcod=80'158)", 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:12.201421+0000", 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 3.7884413690000001, 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101587159, 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:12.201421+0000", 2024-09-16T03:38:16.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:12.201421+0000", 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:12.201421+0000", 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4400000000000001e-07 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:12.201425+0000", 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5480000000000002e-06 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:12.201435+0000", 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0319000000000001e-05 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:12.201443+0000", 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1589999999999993e-06 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:12.302298+0000", 2024-09-16T03:38:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10085458999999999 2024-09-16T03:38:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:12.302320+0000", 2024-09-16T03:38:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2405e-05 2024-09-16T03:38:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:12.302982+0000", 2024-09-16T03:38:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00066213199999999995 2024-09-16T03:38:16.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:12.303008+0000", 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5761999999999998e-05 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2426 4.a e84/32 4:503187b3:::smithi15736696-34:head v 84'86, mlcod=78'82)", 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.249275+0000", 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.740587204, 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.1011635, 2024-09-16T03:38:16.078 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.249275+0000", 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.249275+0000", 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.249277+0000", 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5540000000000001e-06 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.249412+0000", 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013506600000000001 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.249422+0000", 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0268999999999999e-05 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.349509+0000", 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10008711100000001 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.349581+0000", 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1709000000000001e-05 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.349604+0000", 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2748e-05 2024-09-16T03:38:16.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.350390+0000", 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000786467 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.350439+0000", 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8575999999999997e-05 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.082 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.083 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:16.084 INFO:tasks.rados.rados.0.smithi157.stdout:1170: writing smithi15736696-43 from 2900946 to 3303951 tid 3 2024-09-16T03:38:16.084 INFO:tasks.rados.rados.0.smithi157.stdout:1171: set_chunk oid 21 target oid 2024-09-16T03:38:16.084 INFO:tasks.rados.rados.0.smithi157.stdout:1171: set_chunk oid 21 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:16.084 INFO:tasks.rados.rados.0.smithi157.stdout:1172: write (excl) oid 44 current snap is 0 2024-09-16T03:38:16.084 INFO:tasks.rados.rados.0.smithi157.stdout:1172: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-09-16T03:38:16.085 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:38:16.086 INFO:tasks.rados.rados.0.smithi157.stdout:1172: writing smithi15736696-44 from 746277 to 1365347 tid 1 2024-09-16T03:38:16.086 INFO:tasks.rados.rados.0.smithi157.stdout:1172: writing smithi15736696-44 from 1830877 to 2512603 tid 2 2024-09-16T03:38:16.086 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:16.010+0000 7eff7884b640 -1 osd.3 84 handle_osd_map injecting map crc failure 2024-09-16T03:38:16.087 INFO:tasks.rados.rados.0.smithi157.stdout:1172: writing smithi15736696-44 from 2928369 to 3546277 tid 3 2024-09-16T03:38:16.087 INFO:tasks.rados.rados.0.smithi157.stdout:1173: tier_evict oid 34 2024-09-16T03:38:16.087 INFO:tasks.rados.rados.0.smithi157.stdout:1173: tier_evict oid 34 snap -1 2024-09-16T03:38:16.087 INFO:tasks.rados.rados.0.smithi157.stdout:1168: finishing write tid 1 to smithi15736696-13 2024-09-16T03:38:16.087 INFO:tasks.rados.rados.0.smithi157.stdout:1171: finishing set_chunk to oid 21 2024-09-16T03:38:16.087 INFO:tasks.rados.rados.0.smithi157.stdout:1171: got expected ENOENT (src dne) 2024-09-16T03:38:16.087 INFO:tasks.rados.rados.0.smithi157.stdout:1167: done (6 left) 2024-09-16T03:38:16.088 INFO:tasks.rados.rados.0.smithi157.stdout:1171: done (5 left) 2024-09-16T03:38:16.088 INFO:tasks.rados.rados.0.smithi157.stdout:1174: read oid 45 snap -1 2024-09-16T03:38:16.088 INFO:tasks.rados.rados.0.smithi157.stdout:1174: expect deleted 2024-09-16T03:38:16.088 INFO:tasks.rados.rados.0.smithi157.stdout:1175: tier_promote oid 48 2024-09-16T03:38:16.088 INFO:tasks.rados.rados.0.smithi157.stdout:1176: read oid 14 snap -1 2024-09-16T03:38:16.088 INFO:tasks.rados.rados.0.smithi157.stdout:1176: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-16T03:38:16.088 INFO:tasks.rados.rados.0.smithi157.stdout:1177: delete oid 18 current snap is 0 2024-09-16T03:38:16.088 INFO:tasks.rados.rados.0.smithi157.stdout:1169: expect (ObjNum 201 snap 0 seq_num 201) 2024-09-16T03:38:16.088 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:16.082+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:16.185 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:16.183+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:16.187 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:16.185+0000 7efeef6d5640 -1 osd.13 84 handle_osd_map injecting map crc failure 2024-09-16T03:38:16.230 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3321394 2024-09-16T03:38:16.233 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:16.233 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:16.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.233 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2467 4.5 e85/50)", 2024-09-16T03:38:16.233 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:38:16.233 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.081023917000000001, 2024-09-16T03:38:16.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.081039871999999999, 2024-09-16T03:38:16.234 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.234 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.234 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:16.234 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:38:16.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:38:16.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151134+0000", 2024-09-16T03:38:16.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2030000000000001e-06 2024-09-16T03:38:16.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151480+0000", 2024-09-16T03:38:16.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034613999999999999 2024-09-16T03:38:16.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151500+0000", 2024-09-16T03:38:16.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9582999999999999e-05 2024-09-16T03:38:16.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151513+0000", 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2468 4.5 e85/50)", 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.079555388000000005, 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.079595007999999995, 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:16.237 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152601+0000", 2024-09-16T03:38:16.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6869999999999999e-06 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152719+0000", 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117508 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152732+0000", 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152740+0000", 2024-09-16T03:38:16.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7090000000000002e-06 2024-09-16T03:38:16.240 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.240 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.240 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.240 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.240 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-16T03:38:16.240 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 2 2024-09-16T03:38:16.240 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:16.241 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:38:16.286 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:16.284+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:16.314 INFO:tasks.rados.rados.0.smithi157.stdout:1169: done (8 left) 2024-09-16T03:38:16.314 INFO:tasks.rados.rados.0.smithi157.stdout:1177: done (7 left) 2024-09-16T03:38:16.314 INFO:tasks.rados.rados.0.smithi157.stdout:1178: write (excl) oid 16 current snap is 0 2024-09-16T03:38:16.314 INFO:tasks.rados.rados.0.smithi157.stdout:1168: finishing write tid 2 to smithi15736696-13 2024-09-16T03:38:16.314 INFO:tasks.rados.rados.0.smithi157.stdout:1176: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-16T03:38:16.324 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1798807 2024-09-16T03:38:16.334 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:16.345 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T03:38:16.347 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:16.347 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:38:16.348 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:38:16.348 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:16.349 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:38:16.349 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:16.346+0000 7efef16d9640 -1 osd.13 85 handle_osd_map injecting map crc failure 2024-09-16T03:38:16.357 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:38:16.360 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T03:38:16.366 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:16.387 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:16.386+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:16.399 INFO:tasks.rados.rados.0.smithi157.stdout:1168: finishing write tid 3 to smithi15736696-13 2024-09-16T03:38:16.400 INFO:tasks.rados.rados.0.smithi157.stdout:1168: finishing write tid 4 to smithi15736696-13 2024-09-16T03:38:16.400 INFO:tasks.rados.rados.0.smithi157.stdout:1168: finishing write tid 5 to smithi15736696-13 2024-09-16T03:38:16.400 INFO:tasks.rados.rados.0.smithi157.stdout:1168: oid 13 updating version 0 to 55 2024-09-16T03:38:16.400 INFO:tasks.rados.rados.0.smithi157.stdout:1168: oid 13 updating version 55 to 56 2024-09-16T03:38:16.400 INFO:tasks.rados.rados.0.smithi157.stdout:1168: oid 13 updating version 56 to 58 2024-09-16T03:38:16.400 INFO:tasks.rados.rados.0.smithi157.stdout:1168: oid 13 version 58 is already newer than 57 2024-09-16T03:38:16.400 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 58 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-09-16T03:38:16.400 INFO:tasks.rados.rados.0.smithi157.stdout:1168: left oid 13 (ObjNum 209 snap 0 seq_num 209) 2024-09-16T03:38:16.400 INFO:tasks.rados.rados.0.smithi157.stdout:1173: got (22) Invalid argument 2024-09-16T03:38:16.400 INFO:tasks.rados.rados.0.smithi157.stdout:1170: finishing write tid 1 to smithi15736696-43 2024-09-16T03:38:16.400 INFO:tasks.rados.rados.0.smithi157.stdout:1178: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-09-16T03:38:16.407 INFO:tasks.rados.rados.0.smithi157.stdout:1178: writing smithi15736696-16 from 779891 to 1265812 tid 1 2024-09-16T03:38:16.416 INFO:tasks.rados.rados.0.smithi157.stdout:1178: writing smithi15736696-16 from 1947841 to 2633236 tid 2 2024-09-16T03:38:16.423 INFO:tasks.rados.rados.0.smithi157.stdout:1178: writing smithi15736696-16 from 3383215 to 3579891 tid 3 2024-09-16T03:38:16.423 INFO:tasks.rados.rados.0.smithi157.stdout:1168: done (7 left) 2024-09-16T03:38:16.423 INFO:tasks.rados.rados.0.smithi157.stdout:1173: done (6 left) 2024-09-16T03:38:16.423 INFO:tasks.rados.rados.0.smithi157.stdout:1174: done (5 left) 2024-09-16T03:38:16.423 INFO:tasks.rados.rados.0.smithi157.stdout:1176: done (4 left) 2024-09-16T03:38:16.423 INFO:tasks.rados.rados.0.smithi157.stdout:1179: tier_promote oid 13 2024-09-16T03:38:16.423 INFO:tasks.rados.rados.0.smithi157.stdout:1172: finishing write tid 1 to smithi15736696-44 2024-09-16T03:38:16.423 INFO:tasks.rados.rados.0.smithi157.stdout:1180: read oid 39 snap -1 2024-09-16T03:38:16.423 INFO:tasks.rados.rados.0.smithi157.stdout:1180: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-16T03:38:16.424 INFO:tasks.rados.rados.0.smithi157.stdout:1181: read oid 4 snap -1 2024-09-16T03:38:16.424 INFO:tasks.rados.rados.0.smithi157.stdout:1181: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:16.424 INFO:tasks.rados.rados.0.smithi157.stdout:1182: write oid 47 current snap is 0 2024-09-16T03:38:16.424 INFO:tasks.rados.rados.0.smithi157.stdout:1182: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-09-16T03:38:16.431 INFO:tasks.rados.rados.0.smithi157.stdout:1182: writing smithi15736696-47 from 796698 to 1357868 tid 1 2024-09-16T03:38:16.438 INFO:tasks.rados.rados.0.smithi157.stdout:1182: writing smithi15736696-47 from 2006323 to 2551729 tid 2 2024-09-16T03:38:16.445 INFO:tasks.rados.rados.0.smithi157.stdout:1182: writing smithi15736696-47 from 3126991 to 3596698 tid 3 2024-09-16T03:38:16.446 INFO:tasks.rados.rados.0.smithi157.stdout:1183: read oid 31 snap -1 2024-09-16T03:38:16.446 INFO:tasks.rados.rados.0.smithi157.stdout:1183: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-16T03:38:16.446 INFO:tasks.rados.rados.0.smithi157.stdout:1184: read oid 46 snap -1 2024-09-16T03:38:16.446 INFO:tasks.rados.rados.0.smithi157.stdout:1184: expect (ObjNum 192 snap 0 seq_num 192) 2024-09-16T03:38:16.446 INFO:tasks.rados.rados.0.smithi157.stdout:1175: got (0) Success 2024-09-16T03:38:16.446 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 53 (ObjNum 199 snap 0 seq_num 199) dirty exists 2024-09-16T03:38:16.446 INFO:tasks.rados.rados.0.smithi157.stdout:1181: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:16.479 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:38:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:38:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.479 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:38:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.290928656000006, 2024-09-16T03:38:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102630163, 2024-09-16T03:38:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:38:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:38:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186081+0000", 2024-09-16T03:38:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5600000000000005e-07 2024-09-16T03:38:16.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186970+0000", 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00088889000000000004 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186979+0000", 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0620000000000007e-06 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186986+0000", 2024-09-16T03:38:16.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:38:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287184+0000", 2024-09-16T03:38:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100198036 2024-09-16T03:38:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287199+0000", 2024-09-16T03:38:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5458999999999999e-05 2024-09-16T03:38:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288659+0000", 2024-09-16T03:38:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014601900000000001 2024-09-16T03:38:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288711+0000", 2024-09-16T03:38:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1088000000000003e-05 2024-09-16T03:38:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:38:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.289892222999995, 2024-09-16T03:38:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10205370900000001, 2024-09-16T03:38:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:38:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:38:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187118+0000", 2024-09-16T03:38:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8599999999999998e-07 2024-09-16T03:38:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188062+0000", 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094447300000000004 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188070+0000", 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.695e-06 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188074+0000", 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0709999999999996e-06 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287842+0000", 2024-09-16T03:38:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099768478999999993 2024-09-16T03:38:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287864+0000", 2024-09-16T03:38:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1679999999999999e-05 2024-09-16T03:38:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289137+0000", 2024-09-16T03:38:16.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012732430000000001 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289171+0000", 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3382e-05 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:42 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.219810315000004, 2024-09-16T03:38:16.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10172858, 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257201+0000", 2024-09-16T03:38:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:38:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257536+0000", 2024-09-16T03:38:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033513899999999998 2024-09-16T03:38:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257555+0000", 2024-09-16T03:38:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8547999999999999e-05 2024-09-16T03:38:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257566+0000", 2024-09-16T03:38:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1379999999999999e-05 2024-09-16T03:38:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357720+0000", 2024-09-16T03:38:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100153532 2024-09-16T03:38:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357734+0000", 2024-09-16T03:38:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4491000000000001e-05 2024-09-16T03:38:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358896+0000", 2024-09-16T03:38:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011620599999999999 2024-09-16T03:38:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358927+0000", 2024-09-16T03:38:16.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1158999999999998e-05 2024-09-16T03:38:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:43 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'3, mlcod=0'0)", 2024-09-16T03:38:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:38:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.218026402999996, 2024-09-16T03:38:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101764453, 2024-09-16T03:38:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.494 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:38:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:38:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258984+0000", 2024-09-16T03:38:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5689999999999999e-06 2024-09-16T03:38:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258994+0000", 2024-09-16T03:38:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-16T03:38:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259008+0000", 2024-09-16T03:38:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4151999999999999e-05 2024-09-16T03:38:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359073+0000", 2024-09-16T03:38:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100065628 2024-09-16T03:38:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359110+0000", 2024-09-16T03:38:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7045999999999997e-05 2024-09-16T03:38:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359124+0000", 2024-09-16T03:38:16.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.382e-05 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360716+0000", 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015919829999999999 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360747+0000", 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0874999999999998e-05 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:492 4.5 e45/32 4:a11e1d7c:::smithi15736696-43:head v 45'22, mlcod=45'20)", 2024-09-16T03:38:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:38:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 54.653658116000003, 2024-09-16T03:38:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100797151, 2024-09-16T03:38:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:38:16.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823353+0000", 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2450000000000001e-06 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823363+0000", 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0074999999999999e-05 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823378+0000", 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4534e-05 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923461+0000", 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100082859 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923525+0000", 2024-09-16T03:38:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4616999999999995e-05 2024-09-16T03:38:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923549+0000", 2024-09-16T03:38:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.357e-05 2024-09-16T03:38:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924106+0000", 2024-09-16T03:38:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000556753 2024-09-16T03:38:16.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924148+0000", 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2497999999999999e-05 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:806 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'25, mlcod=45'22)", 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.019370+0000", 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.457638789000001, 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048461379999999998, 2024-09-16T03:38:16.503 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019370+0000", 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019370+0000", 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019372+0000", 2024-09-16T03:38:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.424e-06 2024-09-16T03:38:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019541+0000", 2024-09-16T03:38:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016961999999999999 2024-09-16T03:38:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019553+0000", 2024-09-16T03:38:16.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1678000000000001e-05 2024-09-16T03:38:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019561+0000", 2024-09-16T03:38:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7579999999999992e-06 2024-09-16T03:38:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022299+0000", 2024-09-16T03:38:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002738061 2024-09-16T03:38:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022326+0000", 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7515999999999999e-05 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024179+0000", 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001853005 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024216+0000", 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7076000000000002e-05 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.507 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:807 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'26, mlcod=45'22)", 2024-09-16T03:38:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.022119+0000", 2024-09-16T03:38:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.454889960000003, 2024-09-16T03:38:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023221940000000001, 2024-09-16T03:38:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022119+0000", 2024-09-16T03:38:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022119+0000", 2024-09-16T03:38:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022120+0000", 2024-09-16T03:38:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3e-06 2024-09-16T03:38:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022193+0000", 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2364e-05 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022210+0000", 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6773999999999999e-05 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022221+0000", 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1087000000000001e-05 2024-09-16T03:38:16.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023441+0000", 2024-09-16T03:38:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001220524 2024-09-16T03:38:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023470+0000", 2024-09-16T03:38:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8541e-05 2024-09-16T03:38:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024418+0000", 2024-09-16T03:38:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094820699999999996 2024-09-16T03:38:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024441+0000", 2024-09-16T03:38:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3397e-05 2024-09-16T03:38:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:808 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'27, mlcod=45'22)", 2024-09-16T03:38:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:38:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.453964841000001, 2024-09-16T03:38:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.097081386000000006, 2024-09-16T03:38:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:38:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:38:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023046+0000", 2024-09-16T03:38:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1059999999999998e-06 2024-09-16T03:38:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023056+0000", 2024-09-16T03:38:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3719999999999994e-06 2024-09-16T03:38:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023067+0000", 2024-09-16T03:38:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1246999999999999e-05 2024-09-16T03:38:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023075+0000", 2024-09-16T03:38:16.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4710000000000007e-06 2024-09-16T03:38:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.119708+0000", 2024-09-16T03:38:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096632179999999998 2024-09-16T03:38:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.119725+0000", 2024-09-16T03:38:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7754999999999999e-05 2024-09-16T03:38:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.120092+0000", 2024-09-16T03:38:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00036668099999999999 2024-09-16T03:38:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.120126+0000", 2024-09-16T03:38:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3574000000000001e-05 2024-09-16T03:38:16.519 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1365 4.5 e57/50 4:a4fbca73:::smithi15736696-39:head v 57'37, mlcod=57'33)", 2024-09-16T03:38:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.280249+0000", 2024-09-16T03:38:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.196760114, 2024-09-16T03:38:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024448349999999998, 2024-09-16T03:38:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.520 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280249+0000", 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280249+0000", 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280250+0000", 2024-09-16T03:38:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2600000000000002e-07 2024-09-16T03:38:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280485+0000", 2024-09-16T03:38:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023584999999999999 2024-09-16T03:38:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280494+0000", 2024-09-16T03:38:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2349999999999999e-06 2024-09-16T03:38:16.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280510+0000", 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6626999999999999e-05 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.281163+0000", 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00065277800000000002 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.281177+0000", 2024-09-16T03:38:16.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4164e-05 2024-09-16T03:38:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282658+0000", 2024-09-16T03:38:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001480594 2024-09-16T03:38:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282694+0000", 2024-09-16T03:38:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5960999999999997e-05 2024-09-16T03:38:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.524 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1366 4.5 e57/50 4:a4fbca73:::smithi15736696-39:head v 57'38, mlcod=57'33)", 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.280525+0000", 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.196484535000003, 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0022401679999999998, 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280525+0000", 2024-09-16T03:38:16.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280525+0000", 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280525+0000", 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280529+0000", 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.42e-06 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280536+0000", 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.241e-06 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.280540+0000", 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.561e-06 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.281977+0000", 2024-09-16T03:38:16.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014370909999999999 2024-09-16T03:38:16.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.281993+0000", 2024-09-16T03:38:16.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5659e-05 2024-09-16T03:38:16.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282730+0000", 2024-09-16T03:38:16.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073740000000000003 2024-09-16T03:38:16.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.282765+0000", 2024-09-16T03:38:16.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4335999999999998e-05 2024-09-16T03:38:16.529 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.529 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.529 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.529 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:53'30,epoch:57/50,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:38:16.530 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:38:16.530 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.078306499999997, 2024-09-16T03:38:16.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100230893, 2024-09-16T03:38:16.530 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.530 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.530 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:16.531 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:38:16.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:38:16.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398705+0000", 2024-09-16T03:38:16.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5689999999999999e-06 2024-09-16T03:38:16.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398722+0000", 2024-09-16T03:38:16.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6558e-05 2024-09-16T03:38:16.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398735+0000", 2024-09-16T03:38:16.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3499999999999999e-05 2024-09-16T03:38:16.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398746+0000", 2024-09-16T03:38:16.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1182999999999999e-05 2024-09-16T03:38:16.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498898+0000", 2024-09-16T03:38:16.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015112599999999 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498906+0000", 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3189999999999995e-06 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498933+0000", 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7637999999999999e-05 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.535 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1570 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'31, mlcod=48'29)", 2024-09-16T03:38:16.535 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:38:16.535 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.302699505, 2024-09-16T03:38:16.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10253683500000001, 2024-09-16T03:38:16.535 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.535 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.535 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.535 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:38:16.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:38:16.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:38:16.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8500000000000002e-07 2024-09-16T03:38:16.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174689+0000", 2024-09-16T03:38:16.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037958600000000001 2024-09-16T03:38:16.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174699+0000", 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0015e-05 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174706+0000", 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6250000000000001e-06 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274865+0000", 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100159157 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274879+0000", 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3983e-05 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276820+0000", 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001941248 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276846+0000", 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5936e-05 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.539 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1571 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'32, mlcod=48'29)", 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.302248751, 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10249543, 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174761+0000", 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0800000000000004e-07 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175053+0000", 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029151600000000002 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175063+0000", 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0538e-05 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175070+0000", 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7819999999999996e-06 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275895+0000", 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10082516 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275909+0000", 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4197e-05 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277232+0000", 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013231009999999999 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277256+0000", 2024-09-16T03:38:16.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3428e-05 2024-09-16T03:38:16.544 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.544 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.544 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.544 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1572 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'33, mlcod=48'29)", 2024-09-16T03:38:16.544 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:38:16.544 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.301907451000002, 2024-09-16T03:38:16.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10216886, 2024-09-16T03:38:16.544 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8400000000000004e-07 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175107+0000", 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.578e-06 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175114+0000", 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.888e-06 2024-09-16T03:38:16.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175119+0000", 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.074e-06 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276662+0000", 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101542753 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276674+0000", 2024-09-16T03:38:16.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1960000000000001e-05 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277258+0000", 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058431900000000003 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277270+0000", 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2604e-05 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:60'33,epoch:61/50,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.799986196999999, 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10023048699999999, 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8599999999999999e-07 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677027+0000", 2024-09-16T03:38:16.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7359999999999999e-06 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677042+0000", 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4888e-05 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677052+0000", 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7750000000000004e-06 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777182+0000", 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100130394 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777195+0000", 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3373999999999999e-05 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777253+0000", 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7933999999999998e-05 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.552 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.15.0:272 4.15 e76/50 4:aff4cc41:::smithi15736696-11:head v 76'35, mlcod=60'33)", 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:05.957516+0000", 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.51949273, 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023910730000000001, 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:05.957516+0000", 2024-09-16T03:38:16.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:05.957516+0000", 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:05.957517+0000", 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9800000000000004e-07 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:05.957717+0000", 2024-09-16T03:38:16.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000200313 2024-09-16T03:38:16.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:05.957729+0000", 2024-09-16T03:38:16.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2228e-05 2024-09-16T03:38:16.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:05.957741+0000", 2024-09-16T03:38:16.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1234e-05 2024-09-16T03:38:16.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:05.957873+0000", 2024-09-16T03:38:16.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000131889 2024-09-16T03:38:16.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:05.957897+0000", 2024-09-16T03:38:16.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4269000000000001e-05 2024-09-16T03:38:16.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:05.959870+0000", 2024-09-16T03:38:16.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019734930000000002 2024-09-16T03:38:16.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:05.959907+0000", 2024-09-16T03:38:16.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7149e-05 2024-09-16T03:38:16.557 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.557 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.557 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2454 4.15 e84/50 4:aff4cc41:::smithi15736696-11:head v 84'41, mlcod=79'39)", 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.0137700890000001, 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101244889, 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2899999999999999e-07 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463249+0000", 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8039999999999996e-06 2024-09-16T03:38:16.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463267+0000", 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7955e-05 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563340+0000", 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100072999 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563396+0000", 2024-09-16T03:38:16.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6141999999999999e-05 2024-09-16T03:38:16.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563414+0000", 2024-09-16T03:38:16.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8088999999999999e-05 2024-09-16T03:38:16.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.564427+0000", 2024-09-16T03:38:16.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010127230000000001 2024-09-16T03:38:16.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.564484+0000", 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6747999999999999e-05 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2466 4.5 e85/50 4:a11e1d7c:::smithi15736696-43:head v 85'58, mlcod=80'56)", 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.986084+0000", 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.49092461500000001, 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024500669999999998, 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.562 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.986084+0000", 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.986084+0000", 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.986085+0000", 2024-09-16T03:38:16.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4900000000000001e-07 2024-09-16T03:38:16.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.986225+0000", 2024-09-16T03:38:16.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013970200000000001 2024-09-16T03:38:16.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.986238+0000", 2024-09-16T03:38:16.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3376e-05 2024-09-16T03:38:16.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.986246+0000", 2024-09-16T03:38:16.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2020000000000004e-06 2024-09-16T03:38:16.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.986331+0000", 2024-09-16T03:38:16.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5006000000000006e-05 2024-09-16T03:38:16.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.986348+0000", 2024-09-16T03:38:16.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6637e-05 2024-09-16T03:38:16.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.988490+0000", 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021422709999999998 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.988535+0000", 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4524000000000003e-05 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2467 4.5 e85/50)", 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.32587796299999999, 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102439455, 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151134+0000", 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2030000000000001e-06 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151480+0000", 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034613999999999999 2024-09-16T03:38:16.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151500+0000", 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9582999999999999e-05 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151513+0000", 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251653+0000", 2024-09-16T03:38:16.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10014007900000001 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251679+0000", 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5721e-05 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253532+0000", 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001852701 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253571+0000", 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8677999999999998e-05 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2468 4.5 e85/50)", 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.324409434, 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101505239, 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.571 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152601+0000", 2024-09-16T03:38:16.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6869999999999999e-06 2024-09-16T03:38:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152719+0000", 2024-09-16T03:38:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117508 2024-09-16T03:38:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152732+0000", 2024-09-16T03:38:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:38:16.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152740+0000", 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7090000000000002e-06 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252704+0000", 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099964259999999999 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252727+0000", 2024-09-16T03:38:16.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3601e-05 2024-09-16T03:38:16.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:16.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254075+0000", 2024-09-16T03:38:16.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00134772 2024-09-16T03:38:16.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:16.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254105+0000", 2024-09-16T03:38:16.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9802e-05 2024-09-16T03:38:16.575 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.575 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.575 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.576 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.576 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.576 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:16.577 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:38:16.580 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:16.580 INFO:tasks.rados.rados.0.smithi157.stdout:1172: finishing write tid 2 to smithi15736696-44 2024-09-16T03:38:16.580 INFO:tasks.rados.rados.0.smithi157.stdout:1172: finishing write tid 3 to smithi15736696-44 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1172: finishing write tid 4 to smithi15736696-44 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1172: finishing write tid 5 to smithi15736696-44 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1172: oid 44 updating version 0 to 45 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1172: oid 44 updating version 45 to 46 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1172: oid 44 version 46 is already newer than 43 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1172: oid 44 version 46 is already newer than 44 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 46 (ObjNum 211 snap 0 seq_num 211) dirty exists 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1172: left oid 44 (ObjNum 211 snap 0 seq_num 211) 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1172: done (9 left) 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1175: done (8 left) 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1181: done (7 left) 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1185: read oid 48 snap -1 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1185: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1186: write (excl) oid 38 current snap is 0 2024-09-16T03:38:16.581 INFO:tasks.rados.rados.0.smithi157.stdout:1186: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1186: writing smithi15736696-38 from 413505 to 1049924 tid 1 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1186: writing smithi15736696-38 from 1664805 to 2353869 tid 2 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1186: writing smithi15736696-38 from 3038061 to 3530132 tid 3 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1187: read oid 7 snap -1 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1187: expect deleted 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1188: write (excl) oid 25 current snap is 0 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1188: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1188: writing smithi15736696-25 from 430312 to 1141980 tid 1 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1188: writing smithi15736696-25 from 1723287 to 2272362 tid 2 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1188: writing smithi15736696-25 from 2781837 to 3485180 tid 3 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1189: write (excl) oid 37 current snap is 0 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1189: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1189: writing smithi15736696-37 from 447119 to 1234036 tid 1 2024-09-16T03:38:16.582 INFO:tasks.rados.rados.0.smithi157.stdout:1189: writing smithi15736696-37 from 1665416 to 2358149 tid 2 2024-09-16T03:38:16.583 INFO:tasks.rados.rados.0.smithi157.stdout:1189: writing smithi15736696-37 from 2976554 to 3491169 tid 3 2024-09-16T03:38:16.583 INFO:tasks.rados.rados.0.smithi157.stdout:1190: write (excl) oid 50 current snap is 0 2024-09-16T03:38:16.583 INFO:tasks.rados.rados.0.smithi157.stdout:1190: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-09-16T03:38:16.583 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:16.575+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:16.583 INFO:tasks.rados.rados.0.smithi157.stdout:1190: writing smithi15736696-50 from 463926 to 926092 tid 1 2024-09-16T03:38:16.588 INFO:tasks.rados.rados.0.smithi157.stdout:1190: writing smithi15736696-50 from 1723898 to 2160289 tid 2 2024-09-16T03:38:16.594 INFO:tasks.rados.rados.0.smithi157.stdout:1190: writing smithi15736696-50 from 2603977 to 3329864 tid 3 2024-09-16T03:38:16.594 INFO:tasks.rados.rados.0.smithi157.stdout:1191: set_chunk oid 34 target oid 2024-09-16T03:38:16.678 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:16.676+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:16.691 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:16.691 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2512 4.3 4:c512d873:::smithi15736696-38:head [write 413505~636419 in=636419b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.524000+0000", 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.16629797800000001, 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.16632084799999999, 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2512 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.692 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.524000+0000", 2024-09-16T03:38:16.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.524000+0000", 2024-09-16T03:38:16.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.524003+0000", 2024-09-16T03:38:16.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6309999999999999e-06 2024-09-16T03:38:16.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.524674+0000", 2024-09-16T03:38:16.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00067108100000000002 2024-09-16T03:38:16.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.524692+0000", 2024-09-16T03:38:16.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8247999999999999e-05 2024-09-16T03:38:16.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.524702+0000", 2024-09-16T03:38:16.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9539999999999999e-06 2024-09-16T03:38:16.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.524794+0000", 2024-09-16T03:38:16.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1706000000000006e-05 2024-09-16T03:38:16.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op must wait for map", 2024-09-16T03:38:16.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.524802+0000", 2024-09-16T03:38:16.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9610000000000005e-06 2024-09-16T03:38:16.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.526745+0000", 2024-09-16T03:38:16.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019432659999999999 2024-09-16T03:38:16.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.698 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.526809+0000", 2024-09-16T03:38:16.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3767000000000001e-05 2024-09-16T03:38:16.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5,6,8", 2024-09-16T03:38:16.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.526913+0000", 2024-09-16T03:38:16.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010459 2024-09-16T03:38:16.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:16.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.528474+0000", 2024-09-16T03:38:16.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015600939999999999 2024-09-16T03:38:16.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:16.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.530816+0000", 2024-09-16T03:38:16.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002342317 2024-09-16T03:38:16.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:16.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.535791+0000", 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.004974755 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2513 4.3 4:c512d873:::smithi15736696-38:head [write 1664805~689064 in=689064b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.531447+0000", 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.15885142599999999, 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15892610500000001, 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:16.701 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:16.702 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:16.702 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:16.702 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2513 2024-09-16T03:38:16.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.702 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.702 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.531447+0000", 2024-09-16T03:38:16.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.531447+0000", 2024-09-16T03:38:16.703 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.531449+0000", 2024-09-16T03:38:16.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.278e-06 2024-09-16T03:38:16.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.531579+0000", 2024-09-16T03:38:16.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000129961 2024-09-16T03:38:16.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.531590+0000", 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1396999999999999e-05 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.531597+0000", 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1090000000000003e-06 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.531634+0000", 2024-09-16T03:38:16.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7539000000000001e-05 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.531686+0000", 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2258e-05 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5,6,8", 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.531791+0000", 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010443299999999999 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:16.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.533300+0000", 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015091219999999999 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.536996+0000", 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036964649999999999 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2514 4.3 4:c512d873:::smithi15736696-38:head [write 3038061~492071 in=492071b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.538320+0000", 2024-09-16T03:38:16.707 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.15197857000000001, 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15209008199999999, 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2514 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538320+0000", 2024-09-16T03:38:16.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538320+0000", 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538322+0000", 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.04e-06 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538477+0000", 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000155372 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538490+0000", 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2789000000000001e-05 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538497+0000", 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1339999999999998e-06 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538544+0000", 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7275999999999999e-05 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538637+0000", 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2487999999999999e-05 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5,6,8", 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538800+0000", 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016305700000000001 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.540158+0000", 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001358117 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2515 4.3 4:c512d873:::smithi15736696-38:head [setxattr _header (56) in=63b,truncate 3613505] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.538510+0000", 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.15178839999999999, 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15193655, 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.712 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2515 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538510+0000", 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.713 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538510+0000", 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538511+0000", 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0999999999999998e-07 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538514+0000", 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.856e-06 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.538520+0000", 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2499999999999997e-06 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.638596+0000", 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100076104 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.638663+0000", 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6865000000000004e-05 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.638764+0000", 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010131700000000001 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 5,6,8", 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.638889+0000", 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000124642 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2516 4.3 4.ce1b48a3 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.638620+0000", 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.051678475000000001, 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051856975, 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2516 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.638620+0000", 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.638620+0000", 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.638622+0000", 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3319999999999999e-06 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.638631+0000", 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5720000000000003e-06 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.638641+0000", 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.079e-05 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.638650+0000", 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2979999999999992e-06 2024-09-16T03:38:16.719 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2526 4.7 e86/50)", 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.639613+0000", 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.050685579000000001, 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.050885791, 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "no flag points reached", 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.639613+0000", 2024-09-16T03:38:16.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.639613+0000", 2024-09-16T03:38:16.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.639613+0000", 2024-09-16T03:38:16.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0699999999999996e-07 2024-09-16T03:38:16.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.640177+0000", 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00056336199999999998 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.640184+0000", 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7739999999999999e-06 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.722 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop_reply(client.4628.0:2513 4.3 e86/68)", 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.641837+0000", 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.048461276999999997, 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.048678089000000001, 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.641837+0000", 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.641837+0000", 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.641838+0000", 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.373e-06 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.641843+0000", 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0379999999999999e-06 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.641855+0000", 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1211e-05 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.641863+0000", 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4079999999999998e-06 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.725 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop_reply(client.4628.0:2514 4.3 e86/68)", 2024-09-16T03:38:16.726 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.642893+0000", 2024-09-16T03:38:16.726 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.047404991, 2024-09-16T03:38:16.726 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.047641199000000002, 2024-09-16T03:38:16.726 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.726 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.726 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "no flag points reached", 2024-09-16T03:38:16.726 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.726 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.727 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.642893+0000", 2024-09-16T03:38:16.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.642893+0000", 2024-09-16T03:38:16.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.642894+0000", 2024-09-16T03:38:16.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1200000000000001e-06 2024-09-16T03:38:16.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.642898+0000", 2024-09-16T03:38:16.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9809999999999997e-06 2024-09-16T03:38:16.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.642910+0000", 2024-09-16T03:38:16.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1216999999999999e-05 2024-09-16T03:38:16.730 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.730 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:16.730 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.730 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:16.730 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-16T03:38:16.730 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 8 2024-09-16T03:38:16.730 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:16.731 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:38:16.779 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:16.777+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:16.877 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:16.877 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:16.877 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:38:16.877 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:38:16.877 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:16.880 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:16.878+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:16.886 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:38:16.981 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:16.979+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:16.993 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:16.993 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:38:16.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:38:16.993 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:16.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.993 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T03:38:16.993 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:38:16.993 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 85.913618241999998, 2024-09-16T03:38:16.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.88203620400000005, 2024-09-16T03:38:16.994 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:16.994 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:16.994 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:16.994 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:16.994 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:16.994 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:16.994 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:38:16.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.994 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:16.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:16.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:38:16.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:16.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:38:16.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:16.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:16.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076767+0000", 2024-09-16T03:38:16.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9300000000000002e-06 2024-09-16T03:38:16.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:16.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076774+0000", 2024-09-16T03:38:16.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6630000000000004e-06 2024-09-16T03:38:16.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:16.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076785+0000", 2024-09-16T03:38:16.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1221e-05 2024-09-16T03:38:16.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:16.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076796+0000", 2024-09-16T03:38:16.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0428000000000001e-05 2024-09-16T03:38:16.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076856+0000", 2024-09-16T03:38:16.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:38:16.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:16.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076863+0000", 2024-09-16T03:38:16.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9840000000000004e-06 2024-09-16T03:38:16.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:16.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855770+0000", 2024-09-16T03:38:16.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.77890736999999999 2024-09-16T03:38:16.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:16.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855902+0000", 2024-09-16T03:38:16.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013207 2024-09-16T03:38:16.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.856054+0000", 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015220899999999999 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.858407+0000", 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023530069999999998 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958598+0000", 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100190194 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:16.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958716+0000", 2024-09-16T03:38:17.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118589 2024-09-16T03:38:17.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958761+0000", 2024-09-16T03:38:17.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5120000000000002e-05 2024-09-16T03:38:17.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958802+0000", 2024-09-16T03:38:17.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0383999999999998e-05 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 82.395202065000007, 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70508991300000001, 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.001 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 52 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6300000000000005e-07 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595187+0000", 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9790000000000001e-06 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595194+0000", 2024-09-16T03:38:17.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6490000000000002e-06 2024-09-16T03:38:17.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595208+0000", 2024-09-16T03:38:17.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4139e-05 2024-09-16T03:38:17.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801931+0000", 2024-09-16T03:38:17.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.206723083 2024-09-16T03:38:17.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801990+0000", 2024-09-16T03:38:17.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8625999999999997e-05 2024-09-16T03:38:17.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802066+0000", 2024-09-16T03:38:17.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6553999999999999e-05 2024-09-16T03:38:17.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802504+0000", 2024-09-16T03:38:17.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043775200000000002 2024-09-16T03:38:17.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.199998+0000", 2024-09-16T03:38:17.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39749365599999997 2024-09-16T03:38:17.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300151+0000", 2024-09-16T03:38:17.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015310099999999 2024-09-16T03:38:17.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300210+0000", 2024-09-16T03:38:17.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9376999999999997e-05 2024-09-16T03:38:17.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300271+0000", 2024-09-16T03:38:17.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1334000000000006e-05 2024-09-16T03:38:17.007 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.007 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.007 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.007 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.007 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:38:17.008 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 80.620332293000004, 2024-09-16T03:38:17.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60616296300000005, 2024-09-16T03:38:17.008 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.008 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.008 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.008 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.008 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.008 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.008 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 70 2024-09-16T03:38:17.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.008 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:38:17.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:38:17.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370052+0000", 2024-09-16T03:38:17.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8899999999999999e-07 2024-09-16T03:38:17.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370055+0000", 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7709999999999998e-06 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370060+0000", 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6910000000000002e-06 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370064+0000", 2024-09-16T03:38:17.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.619e-06 2024-09-16T03:38:17.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572988+0000", 2024-09-16T03:38:17.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20292439300000001 2024-09-16T03:38:17.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.573049+0000", 2024-09-16T03:38:17.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0418999999999999e-05 2024-09-16T03:38:17.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.573133+0000", 2024-09-16T03:38:17.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4324000000000001e-05 2024-09-16T03:38:17.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674789+0000", 2024-09-16T03:38:17.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101655658 2024-09-16T03:38:17.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.875723+0000", 2024-09-16T03:38:17.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20093406699999999 2024-09-16T03:38:17.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976140+0000", 2024-09-16T03:38:17.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10041718199999999 2024-09-16T03:38:17.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976172+0000", 2024-09-16T03:38:17.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2169000000000003e-05 2024-09-16T03:38:17.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976214+0000", 2024-09-16T03:38:17.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2080999999999999e-05 2024-09-16T03:38:17.014 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.014 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.014 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.014 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.014 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:38:17.014 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 80.620311082000001, 2024-09-16T03:38:17.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60623559900000001, 2024-09-16T03:38:17.014 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.015 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.015 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.015 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.015 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.015 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.015 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 71 2024-09-16T03:38:17.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.015 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:38:17.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370077+0000", 2024-09-16T03:38:17.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9940000000000002e-06 2024-09-16T03:38:17.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370081+0000", 2024-09-16T03:38:17.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5850000000000001e-06 2024-09-16T03:38:17.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370087+0000", 2024-09-16T03:38:17.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:38:17.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673522+0000", 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.303434969 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673567+0000", 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5080000000000002e-05 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673651+0000", 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3652999999999993e-05 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775115+0000", 2024-09-16T03:38:17.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101464178 2024-09-16T03:38:17.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975918+0000", 2024-09-16T03:38:17.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20080318699999999 2024-09-16T03:38:17.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976254+0000", 2024-09-16T03:38:17.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033619200000000002 2024-09-16T03:38:17.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976285+0000", 2024-09-16T03:38:17.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0679000000000002e-05 2024-09-16T03:38:17.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976308+0000", 2024-09-16T03:38:17.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3051e-05 2024-09-16T03:38:17.020 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.020 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.020 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 80.620287722, 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60632925100000001, 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 72 2024-09-16T03:38:17.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:38:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:38:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370150+0000", 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3906999999999997e-05 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370157+0000", 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370162+0000", 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7249999999999997e-06 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673969+0000", 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30380713199999998 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674030+0000", 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1141000000000003e-05 2024-09-16T03:38:17.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674124+0000", 2024-09-16T03:38:17.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3859999999999999e-05 2024-09-16T03:38:17.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775687+0000", 2024-09-16T03:38:17.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10156293299999999 2024-09-16T03:38:17.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976103+0000", 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20041631400000001 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976355+0000", 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000251611 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976385+0000", 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0883999999999998e-05 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976425+0000", 2024-09-16T03:38:17.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9666000000000003e-05 2024-09-16T03:38:17.027 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.027 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.027 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.027 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.027 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:38:17.027 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 80.620215238, 2024-09-16T03:38:17.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60763126999999995, 2024-09-16T03:38:17.027 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.028 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.028 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.028 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.028 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.028 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.028 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 73 2024-09-16T03:38:17.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.028 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:38:17.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370169+0000", 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6199999999999998e-07 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370171+0000", 2024-09-16T03:38:17.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:38:17.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370175+0000", 2024-09-16T03:38:17.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.596e-06 2024-09-16T03:38:17.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470295+0000", 2024-09-16T03:38:17.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100119967 2024-09-16T03:38:17.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775148+0000", 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30485311799999998 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775203+0000", 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4592999999999997e-05 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775293+0000", 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0859999999999994e-05 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976223+0000", 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20092996599999999 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976463+0000", 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000239695 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977730+0000", 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012667690000000001 2024-09-16T03:38:17.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977773+0000", 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2604000000000001e-05 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977800+0000", 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7137000000000001e-05 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 80.520076383000003, 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60633810099999996, 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 74 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:38:17.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:38:17.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470308+0000", 2024-09-16T03:38:17.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0950000000000001e-06 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470313+0000", 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470320+0000", 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5089999999999999e-06 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470327+0000", 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976483+0000", 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50615579499999996 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976533+0000", 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0096000000000003e-05 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976643+0000", 2024-09-16T03:38:17.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109481 2024-09-16T03:38:17.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977808+0000", 2024-09-16T03:38:17.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011649869999999999 2024-09-16T03:38:17.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.981699+0000", 2024-09-16T03:38:17.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003891579 2024-09-16T03:38:17.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076496+0000", 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.094797333999999997 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076583+0000", 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6571999999999997e-05 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076645+0000", 2024-09-16T03:38:17.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2331999999999999e-05 2024-09-16T03:38:17.040 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.040 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.040 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.040 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.040 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:38:17.040 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 80.520046248, 2024-09-16T03:38:17.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71251765099999997, 2024-09-16T03:38:17.040 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.040 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.040 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.040 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 75 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:38:17.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470338+0000", 2024-09-16T03:38:17.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2500000000000005e-07 2024-09-16T03:38:17.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470389+0000", 2024-09-16T03:38:17.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.134e-05 2024-09-16T03:38:17.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470397+0000", 2024-09-16T03:38:17.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8140000000000001e-06 2024-09-16T03:38:17.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570537+0000", 2024-09-16T03:38:17.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100139425 2024-09-16T03:38:17.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977018+0000", 2024-09-16T03:38:17.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40648102899999999 2024-09-16T03:38:17.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977063+0000", 2024-09-16T03:38:17.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5247000000000002e-05 2024-09-16T03:38:17.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977143+0000", 2024-09-16T03:38:17.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0054000000000003e-05 2024-09-16T03:38:17.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.978132+0000", 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00098896399999999995 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182725+0000", 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.204592626 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182783+0000", 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8285000000000002e-05 2024-09-16T03:38:17.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182825+0000", 2024-09-16T03:38:17.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2398999999999998e-05 2024-09-16T03:38:17.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182855+0000", 2024-09-16T03:38:17.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9842999999999999e-05 2024-09-16T03:38:17.046 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.046 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.047 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.047 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.047 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:38:17.047 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 80.419819681000007, 2024-09-16T03:38:17.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71263271800000005, 2024-09-16T03:38:17.047 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.047 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.047 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.047 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.047 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.047 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 76 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570566+0000", 2024-09-16T03:38:17.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:38:17.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570577+0000", 2024-09-16T03:38:17.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0672999999999999e-05 2024-09-16T03:38:17.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570593+0000", 2024-09-16T03:38:17.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5979999999999999e-05 2024-09-16T03:38:17.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570604+0000", 2024-09-16T03:38:17.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.077e-05 2024-09-16T03:38:17.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979890+0000", 2024-09-16T03:38:17.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40928624200000002 2024-09-16T03:38:17.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979945+0000", 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5651e-05 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980038+0000", 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2603999999999996e-05 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980669+0000", 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063121900000000003 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283050+0000", 2024-09-16T03:38:17.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30238120800000001 2024-09-16T03:38:17.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283132+0000", 2024-09-16T03:38:17.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1841999999999993e-05 2024-09-16T03:38:17.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283168+0000", 2024-09-16T03:38:17.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5438999999999997e-05 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283197+0000", 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8960000000000001e-05 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.605284581999996, 2024-09-16T03:38:17.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70799990400000001, 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 94 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385100+0000", 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9599999999999998e-07 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385104+0000", 2024-09-16T03:38:17.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7579999999999998e-06 2024-09-16T03:38:17.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385128+0000", 2024-09-16T03:38:17.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4700999999999999e-05 2024-09-16T03:38:17.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385139+0000", 2024-09-16T03:38:17.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0587e-05 2024-09-16T03:38:17.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787499+0000", 2024-09-16T03:38:17.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40235976099999998 2024-09-16T03:38:17.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787545+0000", 2024-09-16T03:38:17.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6418999999999997e-05 2024-09-16T03:38:17.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787623+0000", 2024-09-16T03:38:17.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7897000000000003e-05 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888605+0000", 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100982321 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990721+0000", 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102115922 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093019+0000", 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10229788500000001 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093051+0000", 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2255999999999997e-05 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093099+0000", 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7500999999999998e-05 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.059 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.060 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.060 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:38:17.060 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.605233259000002, 2024-09-16T03:38:17.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80404594699999998, 2024-09-16T03:38:17.060 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.060 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.060 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.060 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.060 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.060 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.061 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 95 2024-09-16T03:38:17.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.061 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:38:17.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:38:17.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385151+0000", 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385165+0000", 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4175999999999999e-05 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385175+0000", 2024-09-16T03:38:17.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385183+0000", 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888643+0000", 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50345988200000003 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888695+0000", 2024-09-16T03:38:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2834999999999998e-05 2024-09-16T03:38:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888802+0000", 2024-09-16T03:38:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010697899999999999 2024-09-16T03:38:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990658+0000", 2024-09-16T03:38:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185585699999999 2024-09-16T03:38:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990917+0000", 2024-09-16T03:38:17.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025853600000000002 2024-09-16T03:38:17.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189054+0000", 2024-09-16T03:38:17.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198136812 2024-09-16T03:38:17.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189135+0000", 2024-09-16T03:38:17.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1482000000000004e-05 2024-09-16T03:38:17.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189196+0000", 2024-09-16T03:38:17.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1175999999999998e-05 2024-09-16T03:38:17.066 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.066 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.066 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.066 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.066 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.605192631999998, 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80412488400000004, 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 96 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:38:17.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:38:17.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385192+0000", 2024-09-16T03:38:17.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:38:17.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385267+0000", 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5535000000000001e-05 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385277+0000", 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6239999999999997e-06 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385284+0000", 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4800000000000004e-06 2024-09-16T03:38:17.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989283+0000", 2024-09-16T03:38:17.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60399849500000002 2024-09-16T03:38:17.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989410+0000", 2024-09-16T03:38:17.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127298 2024-09-16T03:38:17.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989615+0000", 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000205045 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991036+0000", 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001421042 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189228+0000", 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19819188500000001 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189249+0000", 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0996999999999999e-05 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189293+0000", 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3912000000000002e-05 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189316+0000", 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2739e-05 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.072 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.073 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.073 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.073 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:38:17.073 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.605088901000002, 2024-09-16T03:38:17.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80642264100000005, 2024-09-16T03:38:17.073 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.073 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.073 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 97 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385296+0000", 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.006e-06 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385301+0000", 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2630000000000003e-06 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385308+0000", 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6030000000000001e-06 2024-09-16T03:38:17.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485459+0000", 2024-09-16T03:38:17.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150929 2024-09-16T03:38:17.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990238+0000", 2024-09-16T03:38:17.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50477992000000005 2024-09-16T03:38:17.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990287+0000", 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8863999999999998e-05 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990377+0000", 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9900000000000003e-05 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093109+0000", 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102732055 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189351+0000", 2024-09-16T03:38:17.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096241766000000006 2024-09-16T03:38:17.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191570+0000", 2024-09-16T03:38:17.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00221915 2024-09-16T03:38:17.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191653+0000", 2024-09-16T03:38:17.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3046000000000004e-05 2024-09-16T03:38:17.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191717+0000", 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4139000000000004e-05 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.504898444000005, 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90688294999999997, 2024-09-16T03:38:17.079 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.080 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.080 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.080 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.080 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.080 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.080 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 98 2024-09-16T03:38:17.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.080 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:38:17.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485487+0000", 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1799999999999999e-06 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485498+0000", 2024-09-16T03:38:17.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0648999999999999e-05 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485514+0000", 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5943999999999998e-05 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485525+0000", 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.065e-05 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991102+0000", 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50557748400000002 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991161+0000", 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8801999999999997e-05 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991255+0000", 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3936000000000003e-05 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093112+0000", 2024-09-16T03:38:17.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185709499999999 2024-09-16T03:38:17.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291911+0000", 2024-09-16T03:38:17.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19879897699999999 2024-09-16T03:38:17.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392167+0000", 2024-09-16T03:38:17.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10025634999999999 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392249+0000", 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2175000000000005e-05 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392368+0000", 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118708 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.086 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.086 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:38:17.086 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.504844821000006, 2024-09-16T03:38:17.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90707203199999997, 2024-09-16T03:38:17.086 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.086 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.086 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.086 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.086 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.086 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.086 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 99 2024-09-16T03:38:17.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.086 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485540+0000", 2024-09-16T03:38:17.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-06 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485661+0000", 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120835 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485680+0000", 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8885999999999998e-05 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485693+0000", 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3297e-05 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991546+0000", 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.505853098 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991598+0000", 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1921e-05 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991681+0000", 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2502000000000004e-05 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093115+0000", 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10143424299999999 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291977+0000", 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198862172 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392541+0000", 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100563758 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392579+0000", 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7796999999999997e-05 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392611+0000", 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2073999999999998e-05 2024-09-16T03:38:17.091 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.092 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.092 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.092 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.092 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:38:17.092 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.504665927999994, 2024-09-16T03:38:17.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0073149939999999, 2024-09-16T03:38:17.092 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.092 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.092 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.092 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.092 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 100 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7400000000000003e-07 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485721+0000", 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4530000000000001e-06 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485726+0000", 2024-09-16T03:38:17.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0590000000000002e-06 2024-09-16T03:38:17.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485729+0000", 2024-09-16T03:38:17.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:38:17.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092342+0000", 2024-09-16T03:38:17.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60661343700000003 2024-09-16T03:38:17.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092469+0000", 2024-09-16T03:38:17.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012658799999999999 2024-09-16T03:38:17.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092671+0000", 2024-09-16T03:38:17.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020206 2024-09-16T03:38:17.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392452+0000", 2024-09-16T03:38:17.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29978163699999999 2024-09-16T03:38:17.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392486+0000", 2024-09-16T03:38:17.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3736999999999999e-05 2024-09-16T03:38:17.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492812+0000", 2024-09-16T03:38:17.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100325621 2024-09-16T03:38:17.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492901+0000", 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8702000000000006e-05 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.493033+0000", 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000132185 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.098 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:38:17.099 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 76.329515407000002, 2024-09-16T03:38:17.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60874581500000002, 2024-09-16T03:38:17.099 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.099 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.099 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.099 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.099 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.099 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.099 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 123 2024-09-16T03:38:17.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.099 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660869+0000", 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9400000000000005e-07 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660921+0000", 2024-09-16T03:38:17.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2312000000000003e-05 2024-09-16T03:38:17.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660927+0000", 2024-09-16T03:38:17.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9329999999999998e-06 2024-09-16T03:38:17.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660931+0000", 2024-09-16T03:38:17.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.967991+0000", 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30706032700000002 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968039+0000", 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7477e-05 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968117+0000", 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8589000000000003e-05 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067678+0000", 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099560406000000004 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170175+0000", 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102496927 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269491+0000", 2024-09-16T03:38:17.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099316205000000005 2024-09-16T03:38:17.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269574+0000", 2024-09-16T03:38:17.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2903000000000002e-05 2024-09-16T03:38:17.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269614+0000", 2024-09-16T03:38:17.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0188000000000002e-05 2024-09-16T03:38:17.104 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 76.329446708999996, 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60879245500000001, 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:17.105 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 124 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660938+0000", 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660940+0000", 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2529999999999999e-06 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660943+0000", 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.01e-06 2024-09-16T03:38:17.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660946+0000", 2024-09-16T03:38:17.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8159999999999998e-06 2024-09-16T03:38:17.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012417+0000", 2024-09-16T03:38:17.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.35147120599999998 2024-09-16T03:38:17.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012497+0000", 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9963000000000007e-05 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012585+0000", 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067680+0000", 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055095203000000002 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170243+0000", 2024-09-16T03:38:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10256290799999999 2024-09-16T03:38:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269662+0000", 2024-09-16T03:38:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099418896000000007 2024-09-16T03:38:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269701+0000", 2024-09-16T03:38:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8829999999999999e-05 2024-09-16T03:38:17.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269729+0000", 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8569e-05 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1550 4.13 4:c8492bfc:::smithi15736696-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 31.323718569, 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62706974999999998, 2024-09-16T03:38:17.111 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1550 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:38:17.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:38:17.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0699999999999998e-07 2024-09-16T03:38:17.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666669+0000", 2024-09-16T03:38:17.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.393e-06 2024-09-16T03:38:17.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666673+0000", 2024-09-16T03:38:17.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:38:17.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666682+0000", 2024-09-16T03:38:17.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6510000000000007e-06 2024-09-16T03:38:17.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.913940+0000", 2024-09-16T03:38:17.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.24725763000000001 2024-09-16T03:38:17.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:17.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.914034+0000", 2024-09-16T03:38:17.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3923000000000002e-05 2024-09-16T03:38:17.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293650+0000", 2024-09-16T03:38:17.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.37961662200000001 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293688+0000", 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8096000000000003e-05 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293735+0000", 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6451e-05 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.117 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:77 3.2 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:38:17.117 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:38:17.117 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.911841947999999, 2024-09-16T03:38:17.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.61452436499999996, 2024-09-16T03:38:17.117 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.117 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.117 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:38:17.117 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:38:17.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078543+0000", 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.553e-06 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078551+0000", 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0790000000000001e-06 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078564+0000", 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2943e-05 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178635+0000", 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10007044599999999 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178684+0000", 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9592999999999998e-05 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178747+0000", 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2286000000000006e-05 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4,5", 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178857+0000", 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110807 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280364+0000", 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101506518 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280373+0000", 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6899999999999998e-06 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.369249+0000", 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.088876172000000003 2024-09-16T03:38:17.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693044+0000", 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.32379486499999999 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693066+0000", 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2413e-05 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.122 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.123 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:17.124 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:38:17.126 INFO:tasks.rados.rados.0.smithi157.stdout:1191: set_chunk oid 34 offset: 1233408 length: 2098688 target oid 13dc2fab1f287170fabf96291b912bf80e0b081f291f2c6d41d494566f2d1f8c offset: 0 snap -1 2024-09-16T03:38:17.126 INFO:tasks.rados.rados.0.smithi157.stdout:1192: write oid 44 current snap is 0 2024-09-16T03:38:17.126 INFO:tasks.rados.rados.0.smithi157.stdout:1192: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-09-16T03:38:17.126 INFO:tasks.rados.rados.0.smithi157.stdout:1192: writing smithi15736696-44 from 480733 to 1018148 tid 1 2024-09-16T03:38:17.126 INFO:tasks.rados.rados.0.smithi157.stdout:1192: writing smithi15736696-44 from 1782380 to 2478782 tid 2 2024-09-16T03:38:17.126 INFO:tasks.rados.rados.0.smithi157.stdout:1192: writing smithi15736696-44 from 3031400 to 3568559 tid 3 2024-09-16T03:38:17.126 INFO:tasks.rados.rados.0.smithi157.stdout:1193: write oid 24 current snap is 0 2024-09-16T03:38:17.126 INFO:tasks.rados.rados.0.smithi157.stdout:1193: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-09-16T03:38:17.126 INFO:tasks.rados.rados.0.smithi157.stdout:1193: writing smithi15736696-24 from 497540 to 1110204 tid 1 2024-09-16T03:38:17.126 INFO:tasks.rados.rados.0.smithi157.stdout:1193: writing smithi15736696-24 from 1840862 to 2280922 tid 2 2024-09-16T03:38:17.126 INFO:tasks.rados.rados.0.smithi157.stdout:1193: writing smithi15736696-24 from 3058823 to 3523607 tid 3 2024-09-16T03:38:17.127 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:17.127 INFO:tasks.rados.rados.0.smithi157.stdout:1179: got (0) Success 2024-09-16T03:38:17.127 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 58 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-09-16T03:38:17.127 INFO:tasks.rados.rados.0.smithi157.stdout:1183: expect (ObjNum 183 snap 0 seq_num 183) 2024-09-16T03:38:17.127 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:17.127 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:17.122+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:17.130 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2917355 2024-09-16T03:38:17.138 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:17.174 INFO:tasks.rados.rados.0.smithi157.stdout:1170: finishing write tid 2 to smithi15736696-43 2024-09-16T03:38:17.175 INFO:tasks.rados.rados.0.smithi157.stdout:1170: finishing write tid 3 to smithi15736696-43 2024-09-16T03:38:17.175 INFO:tasks.rados.rados.0.smithi157.stdout:1182: finishing write tid 1 to smithi15736696-47 2024-09-16T03:38:17.175 INFO:tasks.rados.rados.0.smithi157.stdout:1182: finishing write tid 2 to smithi15736696-47 2024-09-16T03:38:17.175 INFO:tasks.rados.rados.0.smithi157.stdout:1184: expect (ObjNum 192 snap 0 seq_num 192) 2024-09-16T03:38:17.225 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:17.223+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:17.232 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:17.233 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:17.233 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:38:17.233 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:17.241 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:38:17.322 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:17.321+0000 7f0ae954f640 -1 osd.0 85 handle_osd_map injecting map crc failure 2024-09-16T03:38:17.325 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:17.324+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:17.329 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1098894 2024-09-16T03:38:17.347 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:17.345+0000 7efef16d9640 -1 osd.13 86 handle_osd_map injecting map crc failure 2024-09-16T03:38:17.349 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T03:38:17.363 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T03:38:17.384 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:38:17.387 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:17.387 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:17.387 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:38:17.387 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:38:17.387 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:17.389 INFO:tasks.rados.rados.0.smithi157.stdout:1182: finishing write tid 3 to smithi15736696-47 2024-09-16T03:38:17.389 INFO:tasks.rados.rados.0.smithi157.stdout:1170: finishing write tid 4 to smithi15736696-43 2024-09-16T03:38:17.389 INFO:tasks.rados.rados.0.smithi157.stdout:1178: finishing write tid 1 to smithi15736696-16 2024-09-16T03:38:17.389 INFO:tasks.rados.rados.0.smithi157.stdout:1170: finishing write tid 5 to smithi15736696-43 2024-09-16T03:38:17.389 INFO:tasks.rados.rados.0.smithi157.stdout:1170: oid 43 updating version 0 to 60 2024-09-16T03:38:17.389 INFO:tasks.rados.rados.0.smithi157.stdout:1170: oid 43 version 60 is already newer than 59 2024-09-16T03:38:17.389 INFO:tasks.rados.rados.0.smithi157.stdout:1170: oid 43 updating version 60 to 61 2024-09-16T03:38:17.389 INFO:tasks.rados.rados.0.smithi157.stdout:1170: oid 43 version 61 is already newer than 58 2024-09-16T03:38:17.389 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 61 (ObjNum 210 snap 0 seq_num 210) dirty exists 2024-09-16T03:38:17.390 INFO:tasks.rados.rados.0.smithi157.stdout:1170: left oid 43 (ObjNum 210 snap 0 seq_num 210) 2024-09-16T03:38:17.390 INFO:tasks.rados.rados.0.smithi157.stdout:1185: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-16T03:38:17.396 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:38:17.446 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:17.445+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:17.465 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3139190 2024-09-16T03:38:17.502 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:17.502 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:38:17.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:38:17.502 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:17.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.502 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:64 4.5 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:17.502 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:38:17.502 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 65.241934084999997, 2024-09-16T03:38:17.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50484290399999998, 2024-09-16T03:38:17.502 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.502 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 64 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:38:17.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257341+0000", 2024-09-16T03:38:17.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:38:17.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257343+0000", 2024-09-16T03:38:17.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3889999999999999e-06 2024-09-16T03:38:17.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257346+0000", 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257349+0000", 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8559999999999998e-06 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357057+0000", 2024-09-16T03:38:17.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099708121999999996 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357156+0000", 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9162999999999999e-05 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762060+0000", 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40490364400000001 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762112+0000", 2024-09-16T03:38:17.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1882000000000002e-05 2024-09-16T03:38:17.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762183+0000", 2024-09-16T03:38:17.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1297000000000005e-05 2024-09-16T03:38:17.507 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.507 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.507 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.507 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.10.22:22 3.3 3:f84e13a5:::smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:17.508 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:38:17.508 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 57.189786849999997, 2024-09-16T03:38:17.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50224271099999995, 2024-09-16T03:38:17.508 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.508 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.508 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.508 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:38:17.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:38:17.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309489+0000", 2024-09-16T03:38:17.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4309999999999999e-06 2024-09-16T03:38:17.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309496+0000", 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5810000000000002e-06 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309505+0000", 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6889999999999999e-06 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309513+0000", 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3749999999999997e-06 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.406932+0000", 2024-09-16T03:38:17.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.097419631000000007 2024-09-16T03:38:17.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407009+0000", 2024-09-16T03:38:17.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7220999999999998e-05 2024-09-16T03:38:17.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:38:17.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407143+0000", 2024-09-16T03:38:17.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013323799999999999 2024-09-16T03:38:17.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407732+0000", 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058938399999999998 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.408133+0000", 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040090500000000001 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811599+0000", 2024-09-16T03:38:17.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40346633399999998 2024-09-16T03:38:17.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811682+0000", 2024-09-16T03:38:17.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2463e-05 2024-09-16T03:38:17.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811730+0000", 2024-09-16T03:38:17.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8458999999999998e-05 2024-09-16T03:38:17.513 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.513 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.513 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:465 4.5 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 56.483569312, 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60529649299999999, 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 465 2024-09-16T03:38:17.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.515 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:38:17.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:38:17.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015706+0000", 2024-09-16T03:38:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.091e-06 2024-09-16T03:38:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015712+0000", 2024-09-16T03:38:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-16T03:38:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015724+0000", 2024-09-16T03:38:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1408e-05 2024-09-16T03:38:17.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015731+0000", 2024-09-16T03:38:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:38:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216049+0000", 2024-09-16T03:38:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200317895 2024-09-16T03:38:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216178+0000", 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012848800000000001 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620010+0000", 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40383207199999999 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620710+0000", 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069988400000000001 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620902+0000", 2024-09-16T03:38:17.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019197800000000001 2024-09-16T03:38:17.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620933+0000", 2024-09-16T03:38:17.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1513999999999997e-05 2024-09-16T03:38:17.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620968+0000", 2024-09-16T03:38:17.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4524999999999999e-05 2024-09-16T03:38:17.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.621002+0000", 2024-09-16T03:38:17.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3955000000000003e-05 2024-09-16T03:38:17.520 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.520 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.520 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.520 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:10 3.3 3:deac805c:::smithi15736696-44 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:17.520 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:38:17.520 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.092074336000003, 2024-09-16T03:38:17.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53191748100000003, 2024-09-16T03:38:17.521 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.521 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.521 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.521 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:38:17.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:38:17.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407202+0000", 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.855e-06 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407208+0000", 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9819999999999996e-06 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407220+0000", 2024-09-16T03:38:17.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2191999999999999e-05 2024-09-16T03:38:17.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407236+0000", 2024-09-16T03:38:17.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5698000000000001e-05 2024-09-16T03:38:17.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538167+0000", 2024-09-16T03:38:17.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.13093111499999999 2024-09-16T03:38:17.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538229+0000", 2024-09-16T03:38:17.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2441999999999996e-05 2024-09-16T03:38:17.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:38:17.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538346+0000", 2024-09-16T03:38:17.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011716400000000001 2024-09-16T03:38:17.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540757+0000", 2024-09-16T03:38:17.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024105089999999999 2024-09-16T03:38:17.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.639501+0000", 2024-09-16T03:38:17.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098743545000000002 2024-09-16T03:38:17.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939038+0000", 2024-09-16T03:38:17.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29953765700000001 2024-09-16T03:38:17.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939075+0000", 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6600999999999997e-05 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939118+0000", 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2721e-05 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:984 4.15 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:17.526 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 47.127154408000003, 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.97836159599999994, 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 984 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372122+0000", 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.934e-06 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372129+0000", 2024-09-16T03:38:17.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8029999999999999e-06 2024-09-16T03:38:17.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372141+0000", 2024-09-16T03:38:17.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2006e-05 2024-09-16T03:38:17.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472234+0000", 2024-09-16T03:38:17.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100092985 2024-09-16T03:38:17.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472318+0000", 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4257000000000003e-05 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472326+0000", 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8960000000000003e-06 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350293+0000", 2024-09-16T03:38:17.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.87796699899999997 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350433+0000", 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013999600000000001 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350482+0000", 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8720000000000001e-05 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.532 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1000 4.15 4:aff4cc41:::smithi15736696-11:head [read 0~2873964,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:17.532 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:38:17.532 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 46.908715323000003, 2024-09-16T03:38:17.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.761707841, 2024-09-16T03:38:17.532 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.532 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.532 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:17.532 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.532 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.532 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.532 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1000 2024-09-16T03:38:17.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.532 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590560+0000", 2024-09-16T03:38:17.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590562+0000", 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.17e-06 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590568+0000", 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7939999999999997e-06 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590572+0000", 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1300000000000001e-06 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590619+0000", 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7148999999999999e-05 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590625+0000", 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5679999999999996e-06 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350858+0000", 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76023295000000002 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350973+0000", 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115023 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352267+0000", 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012935970000000001 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.536 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.537 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.537 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1001 4.15 4:aff4cc41:::smithi15736696-11:head [read 1254674~1619290,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:17.537 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:38:17.537 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 46.908696738000003, 2024-09-16T03:38:17.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76275827799999996, 2024-09-16T03:38:17.537 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.537 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.537 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:17.537 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.537 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1001 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:38:17.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:38:17.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590580+0000", 2024-09-16T03:38:17.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:38:17.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.539 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590584+0000", 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8800000000000001e-06 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590587+0000", 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6630000000000002e-06 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590650+0000", 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3950999999999998e-05 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590654+0000", 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.89e-06 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352309+0000", 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76165495699999997 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352407+0000", 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7188999999999994e-05 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.353336+0000", 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092939400000000005 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1002 4.15 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~1527025] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 46.908667483999999, 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85963523600000002, 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.542 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1002 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.15e-07 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590609+0000", 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.606e-06 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590615+0000", 2024-09-16T03:38:17.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3040000000000001e-06 2024-09-16T03:38:17.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590618+0000", 2024-09-16T03:38:17.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.309e-06 2024-09-16T03:38:17.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590669+0000", 2024-09-16T03:38:17.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1048000000000003e-05 2024-09-16T03:38:17.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:17.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590673+0000", 2024-09-16T03:38:17.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.061e-06 2024-09-16T03:38:17.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.449932+0000", 2024-09-16T03:38:17.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85925979699999999 2024-09-16T03:38:17.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450032+0000", 2024-09-16T03:38:17.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9858000000000005e-05 2024-09-16T03:38:17.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450242+0000", 2024-09-16T03:38:17.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020993800000000001 2024-09-16T03:38:17.548 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.548 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.548 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.549 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1939 4.b 4:dbb349e8:::smithi15736696-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:17.549 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:00.807113+0000", 2024-09-16T03:38:17.549 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.692161721000002, 2024-09-16T03:38:17.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50053880399999995, 2024-09-16T03:38:17.549 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.549 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.549 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:17.549 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.549 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.549 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.549 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1939 2024-09-16T03:38:17.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.549 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.807113+0000", 2024-09-16T03:38:17.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.807113+0000", 2024-09-16T03:38:17.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.807113+0000", 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.79e-07 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.807114+0000", 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.463e-06 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.807119+0000", 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1609999999999996e-06 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.807122+0000", 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3129999999999999e-06 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.005907+0000", 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198785089 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.005956+0000", 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9023999999999997e-05 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.307488+0000", 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30153222000000002 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.307546+0000", 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7531999999999997e-05 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.307651+0000", 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105723 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.553 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2467 4.5 4:a11e1d7c:::smithi15736696-43:head [write 1772395~538068] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.5079911130000001, 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.56288137400000005, 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2467 2024-09-16T03:38:17.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.555 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:38:17.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:38:17.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991285+0000", 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0990000000000001e-06 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991447+0000", 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016170099999999999 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991458+0000", 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.128e-05 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991464+0000", 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3800000000000002e-06 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150823+0000", 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15935943799999999 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150899+0000", 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5359999999999997e-05 2024-09-16T03:38:17.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:17.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151007+0000", 2024-09-16T03:38:17.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108317 2024-09-16T03:38:17.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253992+0000", 2024-09-16T03:38:17.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102985249 2024-09-16T03:38:17.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.353797+0000", 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099805039999999998 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554039+0000", 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20024175899999999 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554129+0000", 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0067000000000004e-05 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554165+0000", 2024-09-16T03:38:17.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5683999999999999e-05 2024-09-16T03:38:17.560 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.560 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.560 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.560 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2468 4.5 4:a11e1d7c:::smithi15736696-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:17.560 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:38:17.560 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.5025703340000001, 2024-09-16T03:38:17.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.55765803899999999, 2024-09-16T03:38:17.560 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.560 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.561 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.561 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2468 2024-09-16T03:38:17.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.561 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:38:17.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996708+0000", 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5149999999999998e-06 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996921+0000", 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000213692 2024-09-16T03:38:17.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996937+0000", 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5699999999999999e-05 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996946+0000", 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7159999999999992e-06 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152362+0000", 2024-09-16T03:38:17.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15541682800000001 2024-09-16T03:38:17.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152421+0000", 2024-09-16T03:38:17.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-16T03:38:17.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:17.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152515+0000", 2024-09-16T03:38:17.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3659999999999994e-05 2024-09-16T03:38:17.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253995+0000", 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10148022499999999 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.453908+0000", 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19991299700000001 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554321+0000", 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100412977 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554344+0000", 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.262e-05 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554362+0000", 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8539999999999999e-05 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.566 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2469 4.5 4:a11e1d7c:::smithi15736696-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:17.567 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:38:17.567 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.5023149710000001, 2024-09-16T03:38:17.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.75785074799999996, 2024-09-16T03:38:17.567 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.567 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.567 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.567 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.567 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.567 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.567 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2469 2024-09-16T03:38:17.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.567 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:38:17.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:38:17.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996960+0000", 2024-09-16T03:38:17.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3000000000000001e-07 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996963+0000", 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8679999999999999e-06 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996967+0000", 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.622e-06 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996971+0000", 2024-09-16T03:38:17.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1429999999999998e-06 2024-09-16T03:38:17.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253536+0000", 2024-09-16T03:38:17.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25656582900000002 2024-09-16T03:38:17.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253632+0000", 2024-09-16T03:38:17.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6067e-05 2024-09-16T03:38:17.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253745+0000", 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011256699999999999 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254218+0000", 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00047321400000000001 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554397+0000", 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30017854399999999 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754557+0000", 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200159848 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754646+0000", 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9370999999999995e-05 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754810+0000", 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016414500000000001 2024-09-16T03:38:17.572 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.573 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.573 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.573 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2470 4.5 4:a11e1d7c:::smithi15736696-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:17.573 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:38:17.573 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.502298642, 2024-09-16T03:38:17.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76022711799999998, 2024-09-16T03:38:17.573 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.573 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.573 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:17.573 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.573 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2470 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:38:17.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.65e-07 2024-09-16T03:38:17.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996977+0000", 2024-09-16T03:38:17.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4079999999999999e-06 2024-09-16T03:38:17.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996981+0000", 2024-09-16T03:38:17.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-16T03:38:17.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996988+0000", 2024-09-16T03:38:17.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0890000000000002e-06 2024-09-16T03:38:17.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253947+0000", 2024-09-16T03:38:17.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25695931100000002 2024-09-16T03:38:17.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:17.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253985+0000", 2024-09-16T03:38:17.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7939999999999999e-05 2024-09-16T03:38:17.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757090+0000", 2024-09-16T03:38:17.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50310477499999995 2024-09-16T03:38:17.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757140+0000", 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9567999999999997e-05 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757203+0000", 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3215999999999999e-05 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2487 4.15 4:ad0b9852:::smithi15736696-16:head [write 1947841~685395] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:17.578 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:38:17.579 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.083254052, 2024-09-16T03:38:17.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.74140762199999999, 2024-09-16T03:38:17.579 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.579 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.579 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.579 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.579 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.579 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.579 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2487 2024-09-16T03:38:17.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.579 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416023+0000", 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:38:17.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416341+0000", 2024-09-16T03:38:17.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031881699999999999 2024-09-16T03:38:17.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416353+0000", 2024-09-16T03:38:17.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.189e-05 2024-09-16T03:38:17.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416360+0000", 2024-09-16T03:38:17.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5719999999999999e-06 2024-09-16T03:38:17.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755578+0000", 2024-09-16T03:38:17.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33921859199999999 2024-09-16T03:38:17.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755639+0000", 2024-09-16T03:38:17.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0615000000000002e-05 2024-09-16T03:38:17.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:17.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755735+0000", 2024-09-16T03:38:17.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5798999999999995e-05 2024-09-16T03:38:17.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858633+0000", 2024-09-16T03:38:17.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10289859599999999 2024-09-16T03:38:17.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858695+0000", 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2000000000000003e-05 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157315+0000", 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29861931600000002 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157375+0000", 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0268e-05 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157428+0000", 2024-09-16T03:38:17.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2939999999999998e-05 2024-09-16T03:38:17.585 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.585 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.585 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.585 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2488 4.15 4:ad0b9852:::smithi15736696-16:head [write 3383215~196676] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:17.585 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:38:17.585 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.0775814020000001, 2024-09-16T03:38:17.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73589615900000005, 2024-09-16T03:38:17.585 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.585 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2488 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421695+0000", 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1469999999999999e-06 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421975+0000", 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028008299999999998 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421987+0000", 2024-09-16T03:38:17.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1294000000000001e-05 2024-09-16T03:38:17.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421992+0000", 2024-09-16T03:38:17.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9349999999999999e-06 2024-09-16T03:38:17.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757205+0000", 2024-09-16T03:38:17.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33521242800000001 2024-09-16T03:38:17.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757269+0000", 2024-09-16T03:38:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3761000000000002e-05 2024-09-16T03:38:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757376+0000", 2024-09-16T03:38:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010706599999999999 2024-09-16T03:38:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858684+0000", 2024-09-16T03:38:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101308255 2024-09-16T03:38:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858700+0000", 2024-09-16T03:38:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6138999999999999e-05 2024-09-16T03:38:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157491+0000", 2024-09-16T03:38:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29879056999999998 2024-09-16T03:38:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157546+0000", 2024-09-16T03:38:17.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5668999999999998e-05 2024-09-16T03:38:17.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157589+0000", 2024-09-16T03:38:17.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2812000000000003e-05 2024-09-16T03:38:17.591 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.591 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.591 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.592 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2489 4.15 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=7b,truncate 3579891] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:17.592 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:38:17.592 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.07727258, 2024-09-16T03:38:17.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73575981700000004, 2024-09-16T03:38:17.592 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.592 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.592 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:17.592 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.592 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.592 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.592 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2489 2024-09-16T03:38:17.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.593 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:38:17.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:38:17.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:38:17.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0200000000000002e-07 2024-09-16T03:38:17.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422005+0000", 2024-09-16T03:38:17.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7379999999999999e-06 2024-09-16T03:38:17.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422009+0000", 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6219999999999998e-06 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422016+0000", 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5789999999999998e-06 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858071+0000", 2024-09-16T03:38:17.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.436054302 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858130+0000", 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9493000000000001e-05 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858226+0000", 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6174000000000004e-05 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10106483199999999 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959639+0000", 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034802999999999999 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157640+0000", 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19800045299999999 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:17.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157702+0000", 2024-09-16T03:38:17.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2218000000000006e-05 2024-09-16T03:38:17.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157762+0000", 2024-09-16T03:38:17.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9774000000000001e-05 2024-09-16T03:38:17.598 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.598 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.598 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.598 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2490 4.15 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:17.599 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:38:17.599 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.0772539880000001, 2024-09-16T03:38:17.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73597824999999994, 2024-09-16T03:38:17.599 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.599 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.599 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:17.599 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.599 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.599 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.599 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2490 2024-09-16T03:38:17.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.599 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:38:17.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:38:17.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422021+0000", 2024-09-16T03:38:17.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8099999999999999e-07 2024-09-16T03:38:17.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422022+0000", 2024-09-16T03:38:17.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.573e-06 2024-09-16T03:38:17.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422027+0000", 2024-09-16T03:38:17.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3250000000000001e-06 2024-09-16T03:38:17.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422030+0000", 2024-09-16T03:38:17.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.948e-06 2024-09-16T03:38:17.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858510+0000", 2024-09-16T03:38:17.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.43648032599999997 2024-09-16T03:38:17.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:17.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858588+0000", 2024-09-16T03:38:17.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7888999999999999e-05 2024-09-16T03:38:17.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157803+0000", 2024-09-16T03:38:17.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.299215642 2024-09-16T03:38:17.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157893+0000", 2024-09-16T03:38:17.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9565000000000001e-05 2024-09-16T03:38:17.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157999+0000", 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105701 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2492 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.077241479, 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53721099400000005, 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:17.604 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2492 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:38:17.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422036+0000", 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5900000000000002e-06 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422043+0000", 2024-09-16T03:38:17.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3069999999999997e-06 2024-09-16T03:38:17.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422048+0000", 2024-09-16T03:38:17.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6619999999999996e-06 2024-09-16T03:38:17.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958774+0000", 2024-09-16T03:38:17.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53672593499999999 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958868+0000", 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4408999999999997e-05 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959244+0000", 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037581099999999998 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.609 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2493 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:17.609 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:38:17.609 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.0772216999999999, 2024-09-16T03:38:17.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53757571299999996, 2024-09-16T03:38:17.609 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.609 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.609 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:17.609 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.609 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.609 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.609 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2493 2024-09-16T03:38:17.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.610 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:38:17.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:38:17.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3600000000000004e-07 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422055+0000", 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7460000000000001e-06 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422058+0000", 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422061+0000", 2024-09-16T03:38:17.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6759999999999999e-06 2024-09-16T03:38:17.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:38:17.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53723057900000004 2024-09-16T03:38:17.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959367+0000", 2024-09-16T03:38:17.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5158000000000001e-05 2024-09-16T03:38:17.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959628+0000", 2024-09-16T03:38:17.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026175800000000001 2024-09-16T03:38:17.613 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.613 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.613 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.613 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2494 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 1738052~782998] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:17.613 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:38:17.613 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.0772106699999999, 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53787640199999998, 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2494 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:38:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:38:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6100000000000002e-07 2024-09-16T03:38:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:17.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422065+0000", 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2839999999999999e-06 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422069+0000", 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7239999999999998e-06 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422072+0000", 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6560000000000001e-06 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:17.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959661+0000", 2024-09-16T03:38:17.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53758982600000005 2024-09-16T03:38:17.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:17.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959739+0000", 2024-09-16T03:38:17.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7581000000000001e-05 2024-09-16T03:38:17.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:17.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:17.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:17.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959940+0000", 2024-09-16T03:38:17.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020107 2024-09-16T03:38:17.618 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.618 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.618 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.618 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:17.618 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:17.618 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:17.619 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:17.620 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:17.618+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:17.620 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:17.620 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:17.621 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:17.621 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:17.517+0000 7f8be56cd640 -1 osd.7 86 handle_osd_map injecting map crc failure 2024-09-16T03:38:17.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:38:17.622 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:17.623 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:17.623 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:17.624 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:17.624 INFO:tasks.rados.rados.0.smithi157.stdout:1188: finishing write tid 1 to smithi15736696-25 2024-09-16T03:38:17.624 INFO:tasks.rados.rados.0.smithi157.stdout:1188: finishing write tid 2 to smithi15736696-25 2024-09-16T03:38:17.624 INFO:tasks.rados.rados.0.smithi157.stdout:1188: finishing write tid 3 to smithi15736696-25 2024-09-16T03:38:17.624 INFO:tasks.rados.rados.0.smithi157.stdout:1188: finishing write tid 4 to smithi15736696-25 2024-09-16T03:38:17.624 INFO:tasks.rados.rados.0.smithi157.stdout:1182: finishing write tid 4 to smithi15736696-47 2024-09-16T03:38:17.624 INFO:tasks.rados.rados.0.smithi157.stdout:1182: finishing write tid 5 to smithi15736696-47 2024-09-16T03:38:17.624 INFO:tasks.rados.rados.0.smithi157.stdout:1182: oid 47 updating version 0 to 86 2024-09-16T03:38:17.624 INFO:tasks.rados.rados.0.smithi157.stdout:1182: oid 47 updating version 86 to 87 2024-09-16T03:38:17.624 INFO:tasks.rados.rados.0.smithi157.stdout:1182: oid 47 updating version 87 to 88 2024-09-16T03:38:17.625 INFO:tasks.rados.rados.0.smithi157.stdout:1182: oid 47 updating version 88 to 89 2024-09-16T03:38:17.625 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 89 (ObjNum 214 snap 0 seq_num 214) dirty exists 2024-09-16T03:38:17.625 INFO:tasks.rados.rados.0.smithi157.stdout:1182: left oid 47 (ObjNum 214 snap 0 seq_num 214) 2024-09-16T03:38:17.625 INFO:tasks.rados.rados.0.smithi157.stdout:1186: finishing write tid 1 to smithi15736696-38 2024-09-16T03:38:17.625 INFO:tasks.rados.rados.0.smithi157.stdout:1186: finishing write tid 2 to smithi15736696-38 2024-09-16T03:38:17.625 INFO:tasks.rados.rados.0.smithi157.stdout:1188: finishing write tid 5 to smithi15736696-25 2024-09-16T03:38:17.625 INFO:tasks.rados.rados.0.smithi157.stdout:1188: oid 25 updating version 0 to 88 2024-09-16T03:38:17.625 INFO:tasks.rados.rados.0.smithi157.stdout:1188: oid 25 version 88 is already newer than 86 2024-09-16T03:38:17.625 INFO:tasks.rados.rados.0.smithi157.stdout:1188: oid 25 updating version 88 to 89 2024-09-16T03:38:17.625 INFO:tasks.rados.rados.0.smithi157.stdout:1188: oid 25 version 89 is already newer than 87 2024-09-16T03:38:17.625 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 89 (ObjNum 216 snap 0 seq_num 216) dirty exists 2024-09-16T03:38:17.625 INFO:tasks.rados.rados.0.smithi157.stdout:1188: left oid 25 (ObjNum 216 snap 0 seq_num 216) 2024-09-16T03:38:17.625 INFO:tasks.rados.rados.0.smithi157.stdout:1189: finishing write tid 1 to smithi15736696-37 2024-09-16T03:38:17.626 INFO:tasks.rados.rados.0.smithi157.stdout:1189: finishing write tid 2 to smithi15736696-37 2024-09-16T03:38:17.626 INFO:tasks.rados.rados.0.smithi157.stdout:1186: finishing write tid 3 to smithi15736696-38 2024-09-16T03:38:17.626 INFO:tasks.rados.rados.0.smithi157.stdout:1170: done (15 left) 2024-09-16T03:38:17.626 INFO:tasks.rados.rados.0.smithi157.stdout:1179: done (14 left) 2024-09-16T03:38:17.626 INFO:tasks.rados.rados.0.smithi157.stdout:1182: done (13 left) 2024-09-16T03:38:17.626 INFO:tasks.rados.rados.0.smithi157.stdout:1183: done (12 left) 2024-09-16T03:38:17.626 INFO:tasks.rados.rados.0.smithi157.stdout:1184: done (11 left) 2024-09-16T03:38:17.626 INFO:tasks.rados.rados.0.smithi157.stdout:1185: done (10 left) 2024-09-16T03:38:17.626 INFO:tasks.rados.rados.0.smithi157.stdout:1187: done (9 left) 2024-09-16T03:38:17.626 INFO:tasks.rados.rados.0.smithi157.stdout:1188: done (8 left) 2024-09-16T03:38:17.626 INFO:tasks.rados.rados.0.smithi157.stdout:1194: set_chunk oid 15 target oid 2024-09-16T03:38:17.626 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:17.627 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:17.627 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:17.627 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:17.635 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:17.635 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:17.652 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:17.652 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:17.669 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:17.670 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:17.687 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:17.687 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:17.721 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:17.719+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:17.722 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:17.722 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:17.736 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:17.736 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:17.739 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:17.739 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:17.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:38:17.759 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:17.759 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:17.776 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:17.776 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:17.793 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:17.793 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:17.809 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:17.810 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:17.821 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:17.819+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:17.868 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:17.868 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:17.876 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:38:17.921 INFO:tasks.rados.rados.0.smithi157.stdout:1194: set_chunk oid 15 offset: 2182144 length: 51200 target oid 99f27be929be4c05675733ba19cdedb708037f8717a9420ebcf98bb5a44a0c84 offset: 0 snap -1 2024-09-16T03:38:17.921 INFO:tasks.rados.rados.0.smithi157.stdout:1195: read oid 22 snap -1 2024-09-16T03:38:17.921 INFO:tasks.rados.rados.0.smithi157.stdout:1195: expect (ObjNum 152 snap 0 seq_num 152) 2024-09-16T03:38:17.921 INFO:tasks.rados.rados.0.smithi157.stdout:1196: tier_promote oid 26 2024-09-16T03:38:17.921 INFO:tasks.rados.rados.0.smithi157.stdout:1180: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-16T03:38:17.922 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:17.920+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:17.984 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:17.985 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:17.993 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:38:18.020 INFO:tasks.rados.rados.0.smithi157.stdout:1190: finishing write tid 1 to smithi15736696-50 2024-09-16T03:38:18.020 INFO:tasks.rados.rados.0.smithi157.stdout:1190: finishing write tid 2 to smithi15736696-50 2024-09-16T03:38:18.020 INFO:tasks.rados.rados.0.smithi157.stdout:1189: finishing write tid 3 to smithi15736696-37 2024-09-16T03:38:18.020 INFO:tasks.rados.rados.0.smithi157.stdout:1189: finishing write tid 4 to smithi15736696-37 2024-09-16T03:38:18.020 INFO:tasks.rados.rados.0.smithi157.stdout:1189: finishing write tid 5 to smithi15736696-37 2024-09-16T03:38:18.020 INFO:tasks.rados.rados.0.smithi157.stdout:1189: oid 37 updating version 0 to 53 2024-09-16T03:38:18.020 INFO:tasks.rados.rados.0.smithi157.stdout:1189: oid 37 version 53 is already newer than 51 2024-09-16T03:38:18.020 INFO:tasks.rados.rados.0.smithi157.stdout:1189: oid 37 version 53 is already newer than 52 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:1189: oid 37 updating version 53 to 54 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 54 (ObjNum 217 snap 0 seq_num 217) dirty exists 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:1189: left oid 37 (ObjNum 217 snap 0 seq_num 217) 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:1180: done (10 left) 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:1189: done (9 left) 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:1197: read oid 33 snap -1 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:1197: expect (ObjNum 207 snap 0 seq_num 207) 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:1198: write (excl) oid 31 current snap is 0 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:1186: finishing write tid 4 to smithi15736696-38 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:1186: finishing write tid 5 to smithi15736696-38 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:1186: oid 38 updating version 0 to 33 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:1186: oid 38 version 33 is already newer than 31 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:1186: oid 38 version 33 is already newer than 30 2024-09-16T03:38:18.021 INFO:tasks.rados.rados.0.smithi157.stdout:1186: oid 38 version 33 is already newer than 32 2024-09-16T03:38:18.022 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 33 (ObjNum 215 snap 0 seq_num 215) dirty exists 2024-09-16T03:38:18.022 INFO:tasks.rados.rados.0.smithi157.stdout:1186: left oid 38 (ObjNum 215 snap 0 seq_num 215) 2024-09-16T03:38:18.022 INFO:tasks.rados.rados.0.smithi157.stdout:1198: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-09-16T03:38:18.023 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:18.022+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:18.024 INFO:tasks.rados.rados.0.smithi157.stdout:1198: writing smithi15736696-31 from 514347 to 918613 tid 1 2024-09-16T03:38:18.031 INFO:tasks.rados.rados.0.smithi157.stdout:1198: writing smithi15736696-31 from 1499344 to 2199415 tid 2 2024-09-16T03:38:18.038 INFO:tasks.rados.rados.0.smithi157.stdout:1198: writing smithi15736696-31 from 2686246 to 3362302 tid 3 2024-09-16T03:38:18.038 INFO:tasks.rados.rados.0.smithi157.stdout:1186: done (10 left) 2024-09-16T03:38:18.038 INFO:tasks.rados.rados.0.smithi157.stdout:1199: set_chunk oid 40 target oid 2024-09-16T03:38:18.124 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:18.123+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:18.141 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:18.141 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:18.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:38:18.225 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:18.224+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:18.258 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:18.258 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:18.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:38:18.327 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:18.325+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:18.351 INFO:tasks.rados.rados.0.smithi157.stdout:1199: set_chunk oid 40 offset: 798720 length: 1205760 target oid 48e749166a40c19f9649810f41d8d2bf30089c670a8ffede39503e6ccf9262e0 offset: 0 snap -1 2024-09-16T03:38:18.351 INFO:tasks.rados.rados.0.smithi157.stdout:1200: read oid 10 snap -1 2024-09-16T03:38:18.351 INFO:tasks.rados.rados.0.smithi157.stdout:1200: expect (ObjNum 123 snap 0 seq_num 123) 2024-09-16T03:38:18.351 INFO:tasks.rados.rados.0.smithi157.stdout:1201: read oid 19 snap -1 2024-09-16T03:38:18.351 INFO:tasks.rados.rados.0.smithi157.stdout:1201: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:18.351 INFO:tasks.rados.rados.0.smithi157.stdout:1202: read oid 20 snap -1 2024-09-16T03:38:18.351 INFO:tasks.rados.rados.0.smithi157.stdout:1202: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-16T03:38:18.351 INFO:tasks.rados.rados.0.smithi157.stdout:1203: write oid 36 current snap is 0 2024-09-16T03:38:18.351 INFO:tasks.rados.rados.0.smithi157.stdout:1178: finishing write tid 2 to smithi15736696-16 2024-09-16T03:38:18.352 INFO:tasks.rados.rados.0.smithi157.stdout:1178: finishing write tid 3 to smithi15736696-16 2024-09-16T03:38:18.352 INFO:tasks.rados.rados.0.smithi157.stdout:1178: finishing write tid 4 to smithi15736696-16 2024-09-16T03:38:18.352 INFO:tasks.rados.rados.0.smithi157.stdout:1178: finishing write tid 5 to smithi15736696-16 2024-09-16T03:38:18.352 INFO:tasks.rados.rados.0.smithi157.stdout:1178: oid 16 updating version 0 to 45 2024-09-16T03:38:18.352 INFO:tasks.rados.rados.0.smithi157.stdout:1178: oid 16 version 45 is already newer than 43 2024-09-16T03:38:18.352 INFO:tasks.rados.rados.0.smithi157.stdout:1178: oid 16 version 45 is already newer than 44 2024-09-16T03:38:18.352 INFO:tasks.rados.rados.0.smithi157.stdout:1178: oid 16 version 45 is already newer than 42 2024-09-16T03:38:18.352 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 45 (ObjNum 213 snap 0 seq_num 213) dirty exists 2024-09-16T03:38:18.352 INFO:tasks.rados.rados.0.smithi157.stdout:1178: left oid 16 (ObjNum 213 snap 0 seq_num 213) 2024-09-16T03:38:18.352 INFO:tasks.rados.rados.0.smithi157.stdout:1192: finishing write tid 1 to smithi15736696-44 2024-09-16T03:38:18.352 INFO:tasks.rados.rados.0.smithi157.stdout:1190: finishing write tid 3 to smithi15736696-50 2024-09-16T03:38:18.352 INFO:tasks.rados.rados.0.smithi157.stdout:1203: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-09-16T03:38:18.353 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T03:38:18.355 INFO:tasks.rados.rados.0.smithi157.stdout:1203: writing smithi15736696-36 from 531154 to 1010669 tid 1 2024-09-16T03:38:18.361 INFO:tasks.rados.rados.0.smithi157.stdout:1203: writing smithi15736696-36 from 1557826 to 2001555 tid 2 2024-09-16T03:38:18.363 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:38:18.367 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-16T03:38:18.368 INFO:tasks.rados.rados.0.smithi157.stdout:1203: writing smithi15736696-36 from 2713669 to 3200997 tid 3 2024-09-16T03:38:18.371 INFO:tasks.rados.rados.0.smithi157.stdout:1203: writing smithi15736696-36 from 3679295 to 3731154 tid 4 2024-09-16T03:38:18.371 INFO:tasks.rados.rados.0.smithi157.stdout:1178: done (14 left) 2024-09-16T03:38:18.372 INFO:tasks.rados.rados.0.smithi157.stdout:1204: delete oid 2 current snap is 0 2024-09-16T03:38:18.372 INFO:tasks.rados.rados.0.smithi157.stdout:1190: finishing write tid 4 to smithi15736696-50 2024-09-16T03:38:18.372 INFO:tasks.rados.rados.0.smithi157.stdout:1190: finishing write tid 5 to smithi15736696-50 2024-09-16T03:38:18.372 INFO:tasks.rados.rados.0.smithi157.stdout:1190: oid 50 updating version 0 to 93 2024-09-16T03:38:18.372 INFO:tasks.rados.rados.0.smithi157.stdout:1190: oid 50 version 93 is already newer than 91 2024-09-16T03:38:18.372 INFO:tasks.rados.rados.0.smithi157.stdout:1190: oid 50 version 93 is already newer than 90 2024-09-16T03:38:18.372 INFO:tasks.rados.rados.0.smithi157.stdout:1190: oid 50 version 93 is already newer than 92 2024-09-16T03:38:18.372 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 93 (ObjNum 218 snap 0 seq_num 218) dirty exists 2024-09-16T03:38:18.372 INFO:tasks.rados.rados.0.smithi157.stdout:1190: left oid 50 (ObjNum 218 snap 0 seq_num 218) 2024-09-16T03:38:18.372 INFO:tasks.rados.rados.0.smithi157.stdout:1192: finishing write tid 2 to smithi15736696-44 2024-09-16T03:38:18.372 INFO:tasks.rados.rados.0.smithi157.stdout:1192: finishing write tid 3 to smithi15736696-44 2024-09-16T03:38:18.372 INFO:tasks.rados.rados.0.smithi157.stdout:1192: finishing write tid 4 to smithi15736696-44 2024-09-16T03:38:18.372 INFO:tasks.rados.rados.0.smithi157.stdout:1192: finishing write tid 5 to smithi15736696-44 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:1192: oid 44 updating version 0 to 49 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:1192: oid 44 version 49 is already newer than 47 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:1192: oid 44 updating version 49 to 50 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:1192: oid 44 version 50 is already newer than 48 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 50 (ObjNum 219 snap 0 seq_num 219) dirty exists 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:1192: left oid 44 (ObjNum 219 snap 0 seq_num 219) 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:1193: finishing write tid 1 to smithi15736696-24 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:1193: finishing write tid 2 to smithi15736696-24 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:1193: finishing write tid 3 to smithi15736696-24 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:1193: finishing write tid 4 to smithi15736696-24 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:1193: finishing write tid 5 to smithi15736696-24 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:1193: oid 24 updating version 0 to 89 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:1193: oid 24 updating version 89 to 90 2024-09-16T03:38:18.373 INFO:tasks.rados.rados.0.smithi157.stdout:1193: oid 24 version 90 is already newer than 88 2024-09-16T03:38:18.374 INFO:tasks.rados.rados.0.smithi157.stdout:1193: oid 24 updating version 90 to 91 2024-09-16T03:38:18.374 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 91 (ObjNum 220 snap 0 seq_num 220) dirty exists 2024-09-16T03:38:18.374 INFO:tasks.rados.rados.0.smithi157.stdout:1193: left oid 24 (ObjNum 220 snap 0 seq_num 220) 2024-09-16T03:38:18.374 INFO:tasks.rados.rados.0.smithi157.stdout:1191: finishing set_chunk to oid 34 2024-09-16T03:38:18.374 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 87 (ObjNum 205 snap 0 seq_num 205) dirty exists 2024-09-16T03:38:18.374 INFO:tasks.rados.rados.0.smithi157.stdout:1195: expect (ObjNum 152 snap 0 seq_num 152) 2024-09-16T03:38:18.374 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:18.380 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:18.381 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:18.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:38:18.432 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:18.430+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:18.439 INFO:tasks.rados.rados.0.smithi157.stdout:1196: got (2) No such file or directory 2024-09-16T03:38:18.439 INFO:tasks.rados.rados.0.smithi157.stdout:1196: got expected ENOENT (src dne) 2024-09-16T03:38:18.439 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 64 (ObjNum -801727072 snap 21913 seq_num 304) clean dne 2024-09-16T03:38:18.439 INFO:tasks.rados.rados.0.smithi157.stdout:1197: expect (ObjNum 207 snap 0 seq_num 207) 2024-09-16T03:38:18.452 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1514841 2024-09-16T03:38:18.519 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:18.518+0000 7fd101e6f640 -1 osd.2 87 handle_osd_map injecting map crc failure 2024-09-16T03:38:18.529 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:18.527+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:18.545 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:18.546 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:18.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:38:18.629 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:18.628+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:18.637 INFO:tasks.rados.rados.0.smithi157.stdout:1194: finishing set_chunk to oid 15 2024-09-16T03:38:18.637 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 8 (ObjNum 136 snap 0 seq_num 136) dirty exists 2024-09-16T03:38:18.638 INFO:tasks.rados.rados.0.smithi157.stdout:1198: finishing write tid 1 to smithi15736696-31 2024-09-16T03:38:18.638 INFO:tasks.rados.rados.0.smithi157.stdout:1198: finishing write tid 2 to smithi15736696-31 2024-09-16T03:38:18.638 INFO:tasks.rados.rados.0.smithi157.stdout:1198: finishing write tid 3 to smithi15736696-31 2024-09-16T03:38:18.638 INFO:tasks.rados.rados.0.smithi157.stdout:1198: finishing write tid 4 to smithi15736696-31 2024-09-16T03:38:18.638 INFO:tasks.rados.rados.0.smithi157.stdout:1198: finishing write tid 5 to smithi15736696-31 2024-09-16T03:38:18.638 INFO:tasks.rados.rados.0.smithi157.stdout:1198: oid 31 updating version 0 to 52 2024-09-16T03:38:18.638 INFO:tasks.rados.rados.0.smithi157.stdout:1198: oid 31 updating version 52 to 53 2024-09-16T03:38:18.638 INFO:tasks.rados.rados.0.smithi157.stdout:1198: oid 31 updating version 53 to 54 2024-09-16T03:38:18.638 INFO:tasks.rados.rados.0.smithi157.stdout:1198: oid 31 version 54 is already newer than 51 2024-09-16T03:38:18.638 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 54 (ObjNum 221 snap 0 seq_num 221) dirty exists 2024-09-16T03:38:18.638 INFO:tasks.rados.rados.0.smithi157.stdout:1198: left oid 31 (ObjNum 221 snap 0 seq_num 221) 2024-09-16T03:38:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1190: done (14 left) 2024-09-16T03:38:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1191: done (13 left) 2024-09-16T03:38:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1192: done (12 left) 2024-09-16T03:38:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1193: done (11 left) 2024-09-16T03:38:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1194: done (10 left) 2024-09-16T03:38:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1195: done (9 left) 2024-09-16T03:38:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1196: done (8 left) 2024-09-16T03:38:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1197: done (7 left) 2024-09-16T03:38:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1198: done (6 left) 2024-09-16T03:38:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1204: done (5 left) 2024-09-16T03:38:18.640 INFO:tasks.rados.rados.0.smithi157.stdout:1205: write oid 16 current snap is 0 2024-09-16T03:38:18.640 INFO:tasks.rados.rados.0.smithi157.stdout:1201: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:18.663 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:18.663 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:18.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:38:18.702 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1389378 2024-09-16T03:38:18.730 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:18.728+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:18.746 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1024899 2024-09-16T03:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:18.780 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:18.788 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:38:18.790 INFO:tasks.rados.rados.0.smithi157.stdout:1203: finishing write tid 1 to smithi15736696-36 2024-09-16T03:38:18.790 INFO:tasks.rados.rados.0.smithi157.stdout:1200: expect (ObjNum 123 snap 0 seq_num 123) 2024-09-16T03:38:18.790 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:18.831 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:18.829+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:18.865 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:18.897 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:18.897 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:18.903 INFO:tasks.rados.rados.0.smithi157.stdout:1202: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-16T03:38:18.903 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 70652 2024-09-16T03:38:18.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:38:18.932 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:18.930+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:18.944 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:18.942+0000 7efeef6d5640 -1 osd.13 87 handle_osd_map injecting map crc failure 2024-09-16T03:38:19.014 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:19.014 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:19.022 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:38:19.027 INFO:tasks.rados.rados.0.smithi157.stdout:1203: finishing write tid 2 to smithi15736696-36 2024-09-16T03:38:19.028 INFO:tasks.rados.rados.0.smithi157.stdout:1199: finishing set_chunk to oid 40 2024-09-16T03:38:19.028 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 85 (ObjNum 204 snap 0 seq_num 204) dirty exists 2024-09-16T03:38:19.028 INFO:tasks.rados.rados.0.smithi157.stdout:1203: finishing write tid 3 to smithi15736696-36 2024-09-16T03:38:19.028 INFO:tasks.rados.rados.0.smithi157.stdout:1203: finishing write tid 4 to smithi15736696-36 2024-09-16T03:38:19.028 INFO:tasks.rados.rados.0.smithi157.stdout:1205: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-09-16T03:38:19.033 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:19.031+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:19.034 INFO:tasks.rados.rados.0.smithi157.stdout:1205: writing smithi15736696-16 from 564768 to 1194781 tid 1 2024-09-16T03:38:19.039 INFO:tasks.rados.rados.0.smithi157.stdout:1205: writing smithi15736696-16 from 1674790 to 2122188 tid 2 2024-09-16T03:38:19.047 INFO:tasks.rados.rados.0.smithi157.stdout:1205: writing smithi15736696-16 from 2768515 to 3394740 tid 3 2024-09-16T03:38:19.047 INFO:tasks.rados.rados.0.smithi157.stdout:1199: done (5 left) 2024-09-16T03:38:19.047 INFO:tasks.rados.rados.0.smithi157.stdout:1200: done (4 left) 2024-09-16T03:38:19.047 INFO:tasks.rados.rados.0.smithi157.stdout:1201: done (3 left) 2024-09-16T03:38:19.047 INFO:tasks.rados.rados.0.smithi157.stdout:1202: done (2 left) 2024-09-16T03:38:19.047 INFO:tasks.rados.rados.0.smithi157.stdout:1206: set_chunk oid 11 target oid 2024-09-16T03:38:19.047 INFO:tasks.rados.rados.0.smithi157.stdout:1206: set_chunk oid 11 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:19.047 INFO:tasks.rados.rados.0.smithi157.stdout:1207: set_chunk oid 45 target oid 2024-09-16T03:38:19.048 INFO:tasks.rados.rados.0.smithi157.stdout:1207: set_chunk oid 45 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:19.048 INFO:tasks.rados.rados.0.smithi157.stdout:1208: read oid 28 snap -1 2024-09-16T03:38:19.048 INFO:tasks.rados.rados.0.smithi157.stdout:1208: expect (ObjNum 186 snap 0 seq_num 186) 2024-09-16T03:38:19.048 INFO:tasks.rados.rados.0.smithi157.stdout:1209: set_chunk oid 49 target oid 2024-09-16T03:38:19.048 INFO:tasks.rados.rados.0.smithi157.stdout:1203: finishing write tid 5 to smithi15736696-36 2024-09-16T03:38:19.048 INFO:tasks.rados.rados.0.smithi157.stdout:1203: finishing write tid 6 to smithi15736696-36 2024-09-16T03:38:19.048 INFO:tasks.rados.rados.0.smithi157.stdout:1203: oid 36 updating version 0 to 94 2024-09-16T03:38:19.048 INFO:tasks.rados.rados.0.smithi157.stdout:1203: oid 36 updating version 94 to 97 2024-09-16T03:38:19.048 INFO:tasks.rados.rados.0.smithi157.stdout:1203: oid 36 version 97 is already newer than 95 2024-09-16T03:38:19.048 INFO:tasks.rados.rados.0.smithi157.stdout:1203: oid 36 version 97 is already newer than 93 2024-09-16T03:38:19.048 INFO:tasks.rados.rados.0.smithi157.stdout:1203: oid 36 version 97 is already newer than 96 2024-09-16T03:38:19.048 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 97 (ObjNum 222 snap 0 seq_num 222) dirty exists 2024-09-16T03:38:19.048 INFO:tasks.rados.rados.0.smithi157.stdout:1203: left oid 36 (ObjNum 222 snap 0 seq_num 222) 2024-09-16T03:38:19.129 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:19.129 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:19.133 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:19.132+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:19.138 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:38:19.234 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:19.233+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:19.253 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:19.253 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:19.263 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:38:19.268 INFO:tasks.rados.rados.0.smithi157.stdout:1209: set_chunk oid 49 offset: 950784 length: 2193408 target oid 3a10b02be1d1ae054c57db2234f12fc8f86216da578c93c08f38f7cfe11326b4 offset: 0 snap -1 2024-09-16T03:38:19.268 INFO:tasks.rados.rados.0.smithi157.stdout:1203: done (5 left) 2024-09-16T03:38:19.268 INFO:tasks.rados.rados.0.smithi157.stdout:1210: delete oid 22 current snap is 0 2024-09-16T03:38:19.268 INFO:tasks.rados.rados.0.smithi157.stdout:1207: finishing set_chunk to oid 45 2024-09-16T03:38:19.269 INFO:tasks.rados.rados.0.smithi157.stdout:1207: got expected ENOENT (src dne) 2024-09-16T03:38:19.340 INFO:tasks.rados.rados.0.smithi157.stdout:1205: finishing write tid 1 to smithi15736696-16 2024-09-16T03:38:19.340 INFO:tasks.rados.rados.0.smithi157.stdout:1206: finishing set_chunk to oid 11 2024-09-16T03:38:19.340 INFO:tasks.rados.rados.0.smithi157.stdout:1206: got expected ENOENT (src dne) 2024-09-16T03:38:19.343 INFO:tasks.rados.rados.0.smithi157.stdout:1205: finishing write tid 2 to smithi15736696-16 2024-09-16T03:38:19.343 INFO:tasks.rados.rados.0.smithi157.stdout:1205: finishing write tid 3 to smithi15736696-16 2024-09-16T03:38:19.344 INFO:tasks.rados.rados.0.smithi157.stdout:1205: finishing write tid 4 to smithi15736696-16 2024-09-16T03:38:19.344 INFO:tasks.rados.rados.0.smithi157.stdout:1205: finishing write tid 5 to smithi15736696-16 2024-09-16T03:38:19.344 INFO:tasks.rados.rados.0.smithi157.stdout:1205: oid 16 updating version 0 to 49 2024-09-16T03:38:19.344 INFO:tasks.rados.rados.0.smithi157.stdout:1205: oid 16 version 49 is already newer than 47 2024-09-16T03:38:19.344 INFO:tasks.rados.rados.0.smithi157.stdout:1205: oid 16 version 49 is already newer than 48 2024-09-16T03:38:19.344 INFO:tasks.rados.rados.0.smithi157.stdout:1205: oid 16 version 49 is already newer than 46 2024-09-16T03:38:19.344 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 49 (ObjNum 224 snap 0 seq_num 224) dirty exists 2024-09-16T03:38:19.344 INFO:tasks.rados.rados.0.smithi157.stdout:1205: left oid 16 (ObjNum 224 snap 0 seq_num 224) 2024-09-16T03:38:19.346 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:19.345+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:19.357 INFO:tasks.rados.rados.0.smithi157.stdout:1205: done (5 left) 2024-09-16T03:38:19.357 INFO:tasks.rados.rados.0.smithi157.stdout:1206: done (4 left) 2024-09-16T03:38:19.357 INFO:tasks.rados.rados.0.smithi157.stdout:1207: done (3 left) 2024-09-16T03:38:19.357 INFO:tasks.rados.rados.0.smithi157.stdout:1210: done (2 left) 2024-09-16T03:38:19.357 INFO:tasks.rados.rados.0.smithi157.stdout:1211: read oid 4 snap -1 2024-09-16T03:38:19.358 INFO:tasks.rados.rados.0.smithi157.stdout:1211: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:19.358 INFO:tasks.rados.rados.0.smithi157.stdout:1212: read oid 19 snap -1 2024-09-16T03:38:19.358 INFO:tasks.rados.rados.0.smithi157.stdout:1212: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:19.358 INFO:tasks.rados.rados.0.smithi157.stdout:1213: read oid 1 snap -1 2024-09-16T03:38:19.358 INFO:tasks.rados.rados.0.smithi157.stdout:1213: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-16T03:38:19.358 INFO:tasks.rados.rados.0.smithi157.stdout:1214: set_chunk oid 15 target oid 2024-09-16T03:38:19.359 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:38:19.359 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T03:38:19.373 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-16T03:38:19.407 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:19.407 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:19.416 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:38:19.436 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:19.434+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:19.537 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:19.535+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:19.559 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:19.559 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:19.568 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:38:19.638 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:19.636+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:19.671 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:19.672 INFO:tasks.rados.rados.0.smithi157.stdout:1214: set_chunk oid 15 offset: 342016 length: 1395200 target oid b64b9f80e9917cdb6830a4e7d915e8191506cec036d9c6272ff0006f2b7d6167 offset: 0 snap -1 2024-09-16T03:38:19.672 INFO:tasks.rados.rados.0.smithi157.stdout:1215: write oid 31 current snap is 0 2024-09-16T03:38:19.672 INFO:tasks.rados.rados.0.smithi157.stdout:1209: finishing set_chunk to oid 49 2024-09-16T03:38:19.672 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 46 (ObjNum 191 snap 0 seq_num 191) dirty exists 2024-09-16T03:38:19.672 INFO:tasks.rados.rados.0.smithi157.stdout:1215: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-09-16T03:38:19.679 INFO:tasks.rados.rados.0.smithi157.stdout:1215: writing smithi15736696-31 from 598382 to 1378893 tid 1 2024-09-16T03:38:19.680 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:38:19.685 INFO:tasks.rados.rados.0.smithi157.stdout:1215: writing smithi15736696-31 from 2075401 to 2526468 tid 2 2024-09-16T03:38:19.692 INFO:tasks.rados.rados.0.smithi157.stdout:1215: writing smithi15736696-31 from 3107008 to 3755777 tid 3 2024-09-16T03:38:19.692 INFO:tasks.rados.rados.0.smithi157.stdout:1209: done (6 left) 2024-09-16T03:38:19.692 INFO:tasks.rados.rados.0.smithi157.stdout:1216: read oid 2 snap -1 2024-09-16T03:38:19.693 INFO:tasks.rados.rados.0.smithi157.stdout:1216: expect deleted 2024-09-16T03:38:19.693 INFO:tasks.rados.rados.0.smithi157.stdout:1217: set_chunk oid 18 target oid 2024-09-16T03:38:19.693 INFO:tasks.rados.rados.0.smithi157.stdout:1217: set_chunk oid 18 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:19.693 INFO:tasks.rados.rados.0.smithi157.stdout:1218: tier_evict oid 25 2024-09-16T03:38:19.693 INFO:tasks.rados.rados.0.smithi157.stdout:1218: tier_evict oid 25 snap -1 2024-09-16T03:38:19.694 INFO:tasks.rados.rados.0.smithi157.stdout:1208: expect (ObjNum 186 snap 0 seq_num 186) 2024-09-16T03:38:19.739 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:19.737+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:19.782 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:19.782 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:19.791 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:38:19.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:38:19.839 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:19.837+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:19.896 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:19.896 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:19.905 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:38:19.911 INFO:tasks.rados.rados.0.smithi157.stdout:1211: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:19.914 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1661112 2024-09-16T03:38:19.940 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:19.938+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:19.943 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:19.949 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:19.950 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:19.965 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:19.966 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:19.968 INFO:tasks.rados.rados.0.smithi157.stdout:1213: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-16T03:38:19.973 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1230430 2024-09-16T03:38:19.983 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:19.983 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:19.995 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:20.000 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:20.000 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:20.015 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:20.015 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:20.016 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:20.014+0000 7f652ae6c640 -1 osd.1 88 handle_osd_map injecting map crc failure 2024-09-16T03:38:20.031 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:20.032 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:20.041 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:20.039+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:20.048 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:20.049 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:20.049 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:20.049 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:20.056 INFO:tasks.rados.rados.0.smithi157.stdout:1212: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:20.057 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:20.058 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:38:20.065 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:20.065 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:20.081 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:20.081 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:20.097 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:20.097 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:20.114 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:20.115 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:20.131 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:20.131 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:20.142 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:20.141+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:20.146 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:20.147 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:20.160 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:20.160 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:20.160 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:20.160 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:20.160 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:20.163 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:20.163 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:20.168 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:20.169 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:38:20.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:20.180 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:20.196 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:20.197 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:20.224 INFO:tasks.rados.rados.0.smithi157.stdout:1217: finishing set_chunk to oid 18 2024-09-16T03:38:20.224 INFO:tasks.rados.rados.0.smithi157.stdout:1217: got expected ENOENT (src dne) 2024-09-16T03:38:20.224 INFO:tasks.rados.rados.0.smithi157.stdout:1218: got (22) Invalid argument 2024-09-16T03:38:20.224 INFO:tasks.rados.rados.0.smithi157.stdout:1214: finishing set_chunk to oid 15 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 15 set_chunk returned error code -95 offset: 342016 length: 1395200 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 136 snap 0 seq_num 136) dirty exists 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:1215: finishing write tid 1 to smithi15736696-31 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:1215: finishing write tid 2 to smithi15736696-31 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:1208: done (8 left) 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:1211: done (7 left) 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:1212: done (6 left) 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:1213: done (5 left) 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:1214: done (4 left) 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:1216: done (3 left) 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:1217: done (2 left) 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:1218: done (1 left) 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:1219: write oid 10 current snap is 0 2024-09-16T03:38:20.225 INFO:tasks.rados.rados.0.smithi157.stdout:1219: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-09-16T03:38:20.229 INFO:tasks.rados.rados.0.smithi157.stdout:1219: writing smithi15736696-10 from 615189 to 1070949 tid 1 2024-09-16T03:38:20.236 INFO:tasks.rados.rados.0.smithi157.stdout:1219: writing smithi15736696-10 from 1733883 to 2444961 tid 2 2024-09-16T03:38:20.243 INFO:tasks.rados.rados.0.smithi157.stdout:1219: writing smithi15736696-10 from 3134431 to 3594472 tid 3 2024-09-16T03:38:20.243 INFO:tasks.rados.rados.0.smithi157.stdout:1220: write oid 39 current snap is 0 2024-09-16T03:38:20.243 INFO:tasks.rados.rados.0.smithi157.stdout:1220: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-09-16T03:38:20.248 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:20.247+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:20.249 INFO:tasks.rados.rados.0.smithi157.stdout:1220: writing smithi15736696-39 from 631996 to 1163005 tid 1 2024-09-16T03:38:20.254 INFO:tasks.rados.rados.0.smithi157.stdout:1220: writing smithi15736696-39 from 1792365 to 2247101 tid 2 2024-09-16T03:38:20.261 INFO:tasks.rados.rados.0.smithi157.stdout:1220: writing smithi15736696-39 from 2761854 to 3433167 tid 3 2024-09-16T03:38:20.261 INFO:tasks.rados.rados.0.smithi157.stdout:1221: set_chunk oid 50 target oid 2024-09-16T03:38:20.273 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:3 3.0 3.f934d18 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:38:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.871631162, 2024-09-16T03:38:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78381636700000001, 2024-09-16T03:38:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:38:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:38:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399429+0000", 2024-09-16T03:38:20.275 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:38:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399434+0000", 2024-09-16T03:38:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1050000000000003e-06 2024-09-16T03:38:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399442+0000", 2024-09-16T03:38:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0160000000000008e-06 2024-09-16T03:38:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.276 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399448+0000", 2024-09-16T03:38:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.879e-06 2024-09-16T03:38:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399489+0000", 2024-09-16T03:38:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0815999999999997e-05 2024-09-16T03:38:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399495+0000", 2024-09-16T03:38:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3439999999999997e-06 2024-09-16T03:38:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183226+0000", 2024-09-16T03:38:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78373077400000002 2024-09-16T03:38:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183244+0000", 2024-09-16T03:38:20.278 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8128999999999999e-05 2024-09-16T03:38:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:5 3.0 3.aefbbec0 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:38:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.771863699999997, 2024-09-16T03:38:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68447923499999996, 2024-09-16T03:38:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:20.279 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:38:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:38:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.280 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499196+0000", 2024-09-16T03:38:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3480000000000001e-06 2024-09-16T03:38:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499202+0000", 2024-09-16T03:38:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4219999999999999e-06 2024-09-16T03:38:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499212+0000", 2024-09-16T03:38:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:38:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499218+0000", 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4920000000000001e-06 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499283+0000", 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4906000000000005e-05 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499293+0000", 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0513000000000001e-05 2024-09-16T03:38:20.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183665+0000", 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68437196600000005 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183674+0000", 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9700000000000005e-06 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:6 3.0 3.3bcda120 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.771803090999995, 2024-09-16T03:38:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78474037699999999, 2024-09-16T03:38:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0500000000000002e-07 2024-09-16T03:38:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499260+0000", 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6669999999999998e-06 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499266+0000", 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499274+0000", 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2250000000000003e-06 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499316+0000", 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2670000000000003e-05 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499323+0000", 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.37e-06 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283987+0000", 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78466433700000005 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283996+0000", 2024-09-16T03:38:20.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8650000000000007e-06 2024-09-16T03:38:20.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.288 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.288 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:15 3.0 3.7ec93288 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:20.288 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:38:20.288 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.662611080999994, 2024-09-16T03:38:20.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57518172599999995, 2024-09-16T03:38:20.288 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.288 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608449+0000", 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.184e-06 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608459+0000", 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0553999999999999e-05 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608470+0000", 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0524e-05 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608493+0000", 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2980999999999999e-05 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608537+0000", 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3850000000000002e-05 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608542+0000", 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:38:20.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183623+0000", 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.575080649 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183629+0000", 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6950000000000003e-06 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:11 3.0 3.77005160 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.569110770999998, 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.48153733399999998, 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701949+0000", 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.314e-06 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701954+0000", 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701964+0000", 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9550000000000004e-06 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701986+0000", 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1931999999999999e-05 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802129+0000", 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100142986 2024-09-16T03:38:20.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802135+0000", 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6350000000000003e-06 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183475+0000", 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38133958099999998 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183485+0000", 2024-09-16T03:38:20.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0642000000000001e-05 2024-09-16T03:38:20.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.297 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.297 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:6 3.0 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:20.297 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:38:20.297 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 63.288206858000002, 2024-09-16T03:38:20.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60469085300000003, 2024-09-16T03:38:20.297 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.297 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.297 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.297 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982853+0000", 2024-09-16T03:38:20.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.393e-06 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982858+0000", 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.062e-06 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982867+0000", 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982879+0000", 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2241e-05 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982996+0000", 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116996 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983007+0000", 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0694e-05 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284499+0000", 2024-09-16T03:38:20.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30149171400000002 2024-09-16T03:38:20.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284690+0000", 2024-09-16T03:38:20.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000191032 2024-09-16T03:38:20.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:20.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284908+0000", 2024-09-16T03:38:20.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021827399999999999 2024-09-16T03:38:20.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.302 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487092+0000", 2024-09-16T03:38:20.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.202184156 2024-09-16T03:38:20.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:20.302 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487110+0000", 2024-09-16T03:38:20.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7856999999999999e-05 2024-09-16T03:38:20.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587335+0000", 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100224802 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587410+0000", 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5343e-05 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587543+0000", 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000132492 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.303 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.304 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.304 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:2 3.0 3:16550cfd:::smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:20.304 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:38:20.304 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 63.288134665999998, 2024-09-16T03:38:20.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40289696000000003, 2024-09-16T03:38:20.304 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.304 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.304 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:20.304 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982925+0000", 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982928+0000", 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4130000000000002e-06 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982936+0000", 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.875e-06 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982942+0000", 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983093+0000", 2024-09-16T03:38:20.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015144400000000001 2024-09-16T03:38:20.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:20.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983098+0000", 2024-09-16T03:38:20.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4909999999999999e-06 2024-09-16T03:38:20.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385695+0000", 2024-09-16T03:38:20.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402597539 2024-09-16T03:38:20.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385821+0000", 2024-09-16T03:38:20.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125418 2024-09-16T03:38:20.308 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.308 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.308 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.308 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:9 3.0 3:2c9b9e7c:::smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:38:20.308 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:38:20.308 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 63.288118476999998, 2024-09-16T03:38:20.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60615655800000001, 2024-09-16T03:38:20.309 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.309 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.309 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.309 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:38:20.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982941+0000", 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0860000000000001e-06 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982946+0000", 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982954+0000", 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8520000000000004e-06 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982958+0000", 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4870000000000004e-06 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983116+0000", 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015774699999999999 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:20.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983120+0000", 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486023+0000", 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50290281400000003 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486423+0000", 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040079299999999998 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:20.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486707+0000", 2024-09-16T03:38:20.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028336600000000001 2024-09-16T03:38:20.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:20.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588861+0000", 2024-09-16T03:38:20.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102154756 2024-09-16T03:38:20.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588952+0000", 2024-09-16T03:38:20.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0630000000000005e-05 2024-09-16T03:38:20.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588984+0000", 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2116999999999998e-05 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589022+0000", 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7910000000000001e-05 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589097+0000", 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4536999999999995e-05 2024-09-16T03:38:20.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:12 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 63.288043129000002, 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60616102900000002, 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983017+0000", 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.483e-06 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983022+0000", 2024-09-16T03:38:20.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:38:20.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983032+0000", 2024-09-16T03:38:20.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0036e-05 2024-09-16T03:38:20.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983038+0000", 2024-09-16T03:38:20.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6800000000000004e-06 2024-09-16T03:38:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983155+0000", 2024-09-16T03:38:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116401 2024-09-16T03:38:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983163+0000", 2024-09-16T03:38:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6219999999999998e-06 2024-09-16T03:38:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589120+0000", 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60595653199999999 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589176+0000", 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6542000000000002e-05 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:5 3.0 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 59.356854200000001, 2024-09-16T03:38:20.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30268081699999999, 2024-09-16T03:38:20.320 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.320 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.320 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.320 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:38:20.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914204+0000", 2024-09-16T03:38:20.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914207+0000", 2024-09-16T03:38:20.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1500000000000002e-06 2024-09-16T03:38:20.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914215+0000", 2024-09-16T03:38:20.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7530000000000007e-06 2024-09-16T03:38:20.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914232+0000", 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.628e-05 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914242+0000", 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.058e-05 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914305+0000", 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3294999999999996e-05 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914377+0000", 2024-09-16T03:38:20.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1698000000000004e-05 2024-09-16T03:38:20.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:20.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914517+0000", 2024-09-16T03:38:20.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000139443 2024-09-16T03:38:20.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.015450+0000", 2024-09-16T03:38:20.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100933668 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.015480+0000", 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9467000000000001e-05 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216766+0000", 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20128665300000001 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216831+0000", 2024-09-16T03:38:20.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4220999999999994e-05 2024-09-16T03:38:20.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216885+0000", 2024-09-16T03:38:20.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4608999999999998e-05 2024-09-16T03:38:20.325 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.325 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.325 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.325 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:24 3.0 3:2dd0f25f:::smithi15736696-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:20.325 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 59.256519804, 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30270049300000001, 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014539+0000", 2024-09-16T03:38:20.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014541+0000", 2024-09-16T03:38:20.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2479999999999999e-06 2024-09-16T03:38:20.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014550+0000", 2024-09-16T03:38:20.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.115e-06 2024-09-16T03:38:20.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014567+0000", 2024-09-16T03:38:20.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7292999999999999e-05 2024-09-16T03:38:20.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.014577+0000", 2024-09-16T03:38:20.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8020000000000003e-06 2024-09-16T03:38:20.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115566+0000", 2024-09-16T03:38:20.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10098905900000001 2024-09-16T03:38:20.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115702+0000", 2024-09-16T03:38:20.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013602500000000001 2024-09-16T03:38:20.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:20.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115893+0000", 2024-09-16T03:38:20.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019095700000000001 2024-09-16T03:38:20.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216936+0000", 2024-09-16T03:38:20.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101042992 2024-09-16T03:38:20.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317125+0000", 2024-09-16T03:38:20.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100188757 2024-09-16T03:38:20.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:20.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317155+0000", 2024-09-16T03:38:20.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0009999999999999e-05 2024-09-16T03:38:20.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317210+0000", 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4589999999999997e-05 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317239+0000", 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9645000000000002e-05 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:30 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 59.054724622999998, 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40312788999999999, 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216338+0000", 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0799999999999999e-06 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216354+0000", 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6439e-05 2024-09-16T03:38:20.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216392+0000", 2024-09-16T03:38:20.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7162000000000001e-05 2024-09-16T03:38:20.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216444+0000", 2024-09-16T03:38:20.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2082e-05 2024-09-16T03:38:20.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417447+0000", 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20100358400000001 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417583+0000", 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135537 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417777+0000", 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019431200000000001 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418479+0000", 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00070148499999999996 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418755+0000", 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027627100000000003 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619285+0000", 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200530138 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619370+0000", 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4567999999999994e-05 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619462+0000", 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2231999999999994e-05 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.337 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:18 3.0 3:037ddf75:::smithi15736696-34 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.335130352, 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30645336299999998, 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935929+0000", 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4600000000000003e-07 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935936+0000", 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3199999999999996e-06 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935943+0000", 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8900000000000007e-06 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935954+0000", 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0246e-05 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140278+0000", 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20432396999999999 2024-09-16T03:38:20.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140342+0000", 2024-09-16T03:38:20.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4442000000000004e-05 2024-09-16T03:38:20.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:20.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140468+0000", 2024-09-16T03:38:20.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125718 2024-09-16T03:38:20.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:20.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.141460+0000", 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00099208400000000006 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242240+0000", 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100779733 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242316+0000", 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6038999999999995e-05 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242351+0000", 2024-09-16T03:38:20.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4928999999999997e-05 2024-09-16T03:38:20.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242382+0000", 2024-09-16T03:38:20.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1046000000000001e-05 2024-09-16T03:38:20.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.343 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.343 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:34 3.0 3:08307d25:::2b3cb797ae2d3abc24b2cae4a012f26325debe2fc38ccb08bd4503e949d305ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:20.343 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 48.796268566000002, 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30798127600000003, 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474791+0000", 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6000000000000003e-07 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474865+0000", 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4504000000000004e-05 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474882+0000", 2024-09-16T03:38:20.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6765999999999998e-05 2024-09-16T03:38:20.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474890+0000", 2024-09-16T03:38:20.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.14e-06 2024-09-16T03:38:20.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681242+0000", 2024-09-16T03:38:20.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20635221200000001 2024-09-16T03:38:20.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681355+0000", 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000112211 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681505+0000", 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015040100000000001 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682352+0000", 2024-09-16T03:38:20.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084686400000000002 2024-09-16T03:38:20.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682989+0000", 2024-09-16T03:38:20.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063749800000000001 2024-09-16T03:38:20.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782695+0000", 2024-09-16T03:38:20.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099706063999999997 2024-09-16T03:38:20.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782735+0000", 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9304000000000002e-05 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782771+0000", 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6551999999999998e-05 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:73 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:38:20.349 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:46.277582+0000", 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.993476518000001, 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30283920399999997, 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277582+0000", 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277582+0000", 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277584+0000", 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.708e-06 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277586+0000", 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.526e-06 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277592+0000", 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3190000000000002e-06 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277595+0000", 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6279999999999998e-06 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.378846+0000", 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10125093 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379007+0000", 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000161046 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379230+0000", 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022235000000000001 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.480276+0000", 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101046812 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.480479+0000", 2024-09-16T03:38:20.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020296100000000001 2024-09-16T03:38:20.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580317+0000", 2024-09-16T03:38:20.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099837767999999993 2024-09-16T03:38:20.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580373+0000", 2024-09-16T03:38:20.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6249999999999998e-05 2024-09-16T03:38:20.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580421+0000", 2024-09-16T03:38:20.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7905999999999998e-05 2024-09-16T03:38:20.355 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.355 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.355 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.355 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:74 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:38:20.356 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:46.277614+0000", 2024-09-16T03:38:20.356 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.993444869999998, 2024-09-16T03:38:20.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.302949411, 2024-09-16T03:38:20.356 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.356 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.356 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.356 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277614+0000", 2024-09-16T03:38:20.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277614+0000", 2024-09-16T03:38:20.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277614+0000", 2024-09-16T03:38:20.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4300000000000003e-07 2024-09-16T03:38:20.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277618+0000", 2024-09-16T03:38:20.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0369999999999996e-06 2024-09-16T03:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277622+0000", 2024-09-16T03:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9790000000000004e-06 2024-09-16T03:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.277626+0000", 2024-09-16T03:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2629999999999999e-06 2024-09-16T03:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379529+0000", 2024-09-16T03:38:20.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101903627 2024-09-16T03:38:20.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379609+0000", 2024-09-16T03:38:20.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9721000000000005e-05 2024-09-16T03:38:20.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:20.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.379690+0000", 2024-09-16T03:38:20.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1267000000000006e-05 2024-09-16T03:38:20.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:20.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.480491+0000", 2024-09-16T03:38:20.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100801053 2024-09-16T03:38:20.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.480528+0000", 2024-09-16T03:38:20.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6871e-05 2024-09-16T03:38:20.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580470+0000", 2024-09-16T03:38:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099941422000000002 2024-09-16T03:38:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580525+0000", 2024-09-16T03:38:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5197e-05 2024-09-16T03:38:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:46.580563+0000", 2024-09-16T03:38:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8430999999999998e-05 2024-09-16T03:38:20.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:77 3.0 3:1a18dfd0:::9d9929bebc2b17b3e19b9b502dd3b0e63f1b7b4c1cf7796e8161d7773d261a75:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.802529+0000", 2024-09-16T03:38:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.468529692000001, 2024-09-16T03:38:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30442216599999999, 2024-09-16T03:38:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.802529+0000", 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.802529+0000", 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.802530+0000", 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.311e-06 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803091+0000", 2024-09-16T03:38:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056044899999999997 2024-09-16T03:38:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803101+0000", 2024-09-16T03:38:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9189999999999994e-06 2024-09-16T03:38:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803106+0000", 2024-09-16T03:38:20.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.626e-06 2024-09-16T03:38:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.903907+0000", 2024-09-16T03:38:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100800951 2024-09-16T03:38:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.904063+0000", 2024-09-16T03:38:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015626 2024-09-16T03:38:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.904369+0000", 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00030599300000000001 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907929+0000", 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0035599080000000001 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.006526+0000", 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098596345000000002 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.106808+0000", 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100282198 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.106917+0000", 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00010936 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.106951+0000", 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3846000000000001e-05 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.367 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.368 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.368 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:78 3.0 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:20.368 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803117+0000", 2024-09-16T03:38:20.368 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.467941175, 2024-09-16T03:38:20.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30401020200000001, 2024-09-16T03:38:20.368 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.368 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.368 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.368 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803117+0000", 2024-09-16T03:38:20.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803117+0000", 2024-09-16T03:38:20.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803118+0000", 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0400000000000005e-07 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803503+0000", 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038435999999999999 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803515+0000", 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2279999999999999e-05 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803521+0000", 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.054e-06 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.905093+0000", 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101572302 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.905196+0000", 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000102876 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:20.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.905387+0000", 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019066499999999999 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907935+0000", 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025477949999999998 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.106742+0000", 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19880740599999999 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107021+0000", 2024-09-16T03:38:20.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027904200000000002 2024-09-16T03:38:20.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107083+0000", 2024-09-16T03:38:20.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2149000000000004e-05 2024-09-16T03:38:20.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107128+0000", 2024-09-16T03:38:20.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4468999999999997e-05 2024-09-16T03:38:20.373 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.373 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:79 3.0 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803530+0000", 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.467529065000001, 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30378224199999998, 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803530+0000", 2024-09-16T03:38:20.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803530+0000", 2024-09-16T03:38:20.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803530+0000", 2024-09-16T03:38:20.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8000000000000005e-07 2024-09-16T03:38:20.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803942+0000", 2024-09-16T03:38:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00041188299999999999 2024-09-16T03:38:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803952+0000", 2024-09-16T03:38:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.027e-05 2024-09-16T03:38:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803957+0000", 2024-09-16T03:38:20.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0159999999999999e-06 2024-09-16T03:38:20.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.906129+0000", 2024-09-16T03:38:20.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102171915 2024-09-16T03:38:20.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.906280+0000", 2024-09-16T03:38:20.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000150447 2024-09-16T03:38:20.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:20.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.906551+0000", 2024-09-16T03:38:20.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027134700000000002 2024-09-16T03:38:20.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:20.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907944+0000", 2024-09-16T03:38:20.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013923410000000001 2024-09-16T03:38:20.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107181+0000", 2024-09-16T03:38:20.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19923718800000001 2024-09-16T03:38:20.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107248+0000", 2024-09-16T03:38:20.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7341999999999993e-05 2024-09-16T03:38:20.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107283+0000", 2024-09-16T03:38:20.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4711e-05 2024-09-16T03:38:20.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107312+0000", 2024-09-16T03:38:20.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9002000000000001e-05 2024-09-16T03:38:20.380 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.380 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.380 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.380 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:87 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:20.380 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:38:20.380 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.467073842000001, 2024-09-16T03:38:20.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402368742, 2024-09-16T03:38:20.380 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.380 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803986+0000", 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9400000000000004e-07 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804061+0000", 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5230999999999999e-05 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804069+0000", 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7530000000000001e-06 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804073+0000", 2024-09-16T03:38:20.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9660000000000004e-06 2024-09-16T03:38:20.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907236+0000", 2024-09-16T03:38:20.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103162566 2024-09-16T03:38:20.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907324+0000", 2024-09-16T03:38:20.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7894999999999998e-05 2024-09-16T03:38:20.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:20.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907474+0000", 2024-09-16T03:38:20.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014983999999999999 2024-09-16T03:38:20.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:20.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.908330+0000", 2024-09-16T03:38:20.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085600100000000001 2024-09-16T03:38:20.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107220+0000", 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198889812 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206269+0000", 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099049611999999995 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206305+0000", 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5679000000000002e-05 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206353+0000", 2024-09-16T03:38:20.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8593e-05 2024-09-16T03:38:20.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.386 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.386 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.386 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:20.388 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T03:38:20.389 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:38:20.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-16T03:38:20.390 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:20.385+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:20.489 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:20.486+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:20.491 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:20.491 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:20.491 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:20.492 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:20.500 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:38:20.534 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:20.532+0000 7fa3886e6640 -1 osd.10 89 handle_osd_map injecting map crc failure 2024-09-16T03:38:20.562 INFO:tasks.rados.rados.0.smithi157.stdout:1221: set_chunk oid 50 offset: 696320 length: 2496512 target oid e447860b11e9f9fde68b75cb1d2f204a74952889ec51a2417ef18c1427ff9a70 offset: 0 snap -1 2024-09-16T03:38:20.562 INFO:tasks.rados.rados.0.smithi157.stdout:1222: read oid 23 snap -1 2024-09-16T03:38:20.562 INFO:tasks.rados.rados.0.smithi157.stdout:1222: expect (ObjNum 189 snap 0 seq_num 189) 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:1223: read oid 1 snap -1 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:1223: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:1224: tier_promote oid 40 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:1215: finishing write tid 3 to smithi15736696-31 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:1215: finishing write tid 4 to smithi15736696-31 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:1225: delete oid 24 current snap is 0 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:1215: finishing write tid 5 to smithi15736696-31 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:1215: oid 31 updating version 0 to 56 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:1215: oid 31 version 56 is already newer than 55 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:1215: oid 31 updating version 56 to 57 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:1215: oid 31 updating version 57 to 58 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 58 (ObjNum 226 snap 0 seq_num 226) dirty exists 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:1215: left oid 31 (ObjNum 226 snap 0 seq_num 226) 2024-09-16T03:38:20.563 INFO:tasks.rados.rados.0.smithi157.stdout:1219: finishing write tid 1 to smithi15736696-10 2024-09-16T03:38:20.567 INFO:tasks.rados.rados.0.smithi157.stdout:1223: expect (ObjNum 134 snap 0 seq_num 134) 2024-09-16T03:38:20.589 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:20.588+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:20.603 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:20.604 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:20.604 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:20.604 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:20.604 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:20.604 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:20.613 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:38:20.663 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:20.662+0000 7eff7884b640 -1 osd.3 89 handle_osd_map injecting map crc failure 2024-09-16T03:38:20.674 INFO:tasks.rados.rados.0.smithi157.stdout:1222: expect (ObjNum 189 snap 0 seq_num 189) 2024-09-16T03:38:20.674 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:20.690 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:20.688+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:20.716 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:20.716 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:20.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:20.716 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:20.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:57 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 68.589430684999996, 2024-09-16T03:38:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20372090900000001, 2024-09-16T03:38:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125313+0000", 2024-09-16T03:38:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7180000000000001e-06 2024-09-16T03:38:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126730+0000", 2024-09-16T03:38:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00141731 2024-09-16T03:38:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126746+0000", 2024-09-16T03:38:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6455000000000001e-05 2024-09-16T03:38:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126756+0000", 2024-09-16T03:38:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:38:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227068+0000", 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100312308 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227085+0000", 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6898e-05 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328995+0000", 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101909903 2024-09-16T03:38:20.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329031+0000", 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5717999999999999e-05 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:58 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 68.586097554999995, 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084524500000001, 2024-09-16T03:38:20.721 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.722 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.722 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.722 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:20.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128644+0000", 2024-09-16T03:38:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.251e-06 2024-09-16T03:38:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128913+0000", 2024-09-16T03:38:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026856200000000002 2024-09-16T03:38:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128925+0000", 2024-09-16T03:38:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2109999999999999e-05 2024-09-16T03:38:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128933+0000", 2024-09-16T03:38:20.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6260000000000003e-06 2024-09-16T03:38:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328196+0000", 2024-09-16T03:38:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19926303200000001 2024-09-16T03:38:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328253+0000", 2024-09-16T03:38:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7262999999999997e-05 2024-09-16T03:38:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329458+0000", 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012051939999999999 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329488+0000", 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0207000000000001e-05 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:59 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'3, mlcod=0'0)", 2024-09-16T03:38:20.726 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 68.585602121999997, 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036954900000001, 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129139+0000", 2024-09-16T03:38:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6099999999999999e-07 2024-09-16T03:38:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129158+0000", 2024-09-16T03:38:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088e-05 2024-09-16T03:38:20.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129168+0000", 2024-09-16T03:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.048e-05 2024-09-16T03:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.229233+0000", 2024-09-16T03:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100065168 2024-09-16T03:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328815+0000", 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099581694999999998 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328845+0000", 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0193e-05 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329492+0000", 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064671800000000001 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329508+0000", 2024-09-16T03:38:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6246e-05 2024-09-16T03:38:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.0:15 4.d e47/32 4:b3fe05b2:::smithi15736696-15:head v 47'5, mlcod=39'3)", 2024-09-16T03:38:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 54.412307396000003, 2024-09-16T03:38:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100920751, 2024-09-16T03:38:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.731 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.732 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.732 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:20.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:20.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302435+0000", 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2340000000000001e-06 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302446+0000", 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0596e-05 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302461+0000", 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5082000000000001e-05 2024-09-16T03:38:20.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402592+0000", 2024-09-16T03:38:20.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100130488 2024-09-16T03:38:20.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402712+0000", 2024-09-16T03:38:20.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012013100000000001 2024-09-16T03:38:20.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402741+0000", 2024-09-16T03:38:20.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9717999999999999e-05 2024-09-16T03:38:20.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403309+0000", 2024-09-16T03:38:20.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056725700000000005 2024-09-16T03:38:20.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403354+0000", 2024-09-16T03:38:20.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5244999999999998e-05 2024-09-16T03:38:20.735 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.735 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.564808686999999, 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032593800000000001, 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:20.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149943+0000", 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0817e-05 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151026+0000", 2024-09-16T03:38:20.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010835350000000001 2024-09-16T03:38:20.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151040+0000", 2024-09-16T03:38:20.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3884e-05 2024-09-16T03:38:20.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151049+0000", 2024-09-16T03:38:20.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:38:20.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151107+0000", 2024-09-16T03:38:20.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8199000000000003e-05 2024-09-16T03:38:20.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151124+0000", 2024-09-16T03:38:20.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7756000000000001e-05 2024-09-16T03:38:20.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153142+0000", 2024-09-16T03:38:20.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020177569999999998 2024-09-16T03:38:20.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153191+0000", 2024-09-16T03:38:20.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9051999999999997e-05 2024-09-16T03:38:20.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.740 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.740 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1682 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'7, mlcod=40'4)", 2024-09-16T03:38:20.740 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.461211497000001, 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10169502499999999, 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253540+0000", 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0888e-05 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254015+0000", 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047485499999999999 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254030+0000", 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4899000000000001e-05 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254039+0000", 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5000000000000005e-06 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254107+0000", 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7298000000000005e-05 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254122+0000", 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5928999999999999e-05 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355184+0000", 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101061703 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355224+0000", 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9953000000000002e-05 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.744 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1683 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'8, mlcod=40'4)", 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.460673949, 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101630885, 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254068+0000", 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5700000000000001e-07 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254075+0000", 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2710000000000003e-06 2024-09-16T03:38:20.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254083+0000", 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0220000000000004e-06 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254089+0000", 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8699999999999997e-06 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354968+0000", 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100879625 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354992+0000", 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355665+0000", 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00067371100000000001 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355698+0000", 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2291e-05 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.748 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.749 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.749 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.0:177 4.10 e65/57 4:0a55bbf2:::smithi15736696-18:head v 65'47, mlcod=0'0)", 2024-09-16T03:38:20.749 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:20.749 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.507727361000001, 2024-09-16T03:38:20.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074461099999999995, 2024-09-16T03:38:20.749 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.749 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.749 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.749 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.749 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5300000000000001e-07 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207018+0000", 2024-09-16T03:38:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7210000000000002e-06 2024-09-16T03:38:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207031+0000", 2024-09-16T03:38:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2767e-05 2024-09-16T03:38:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207040+0000", 2024-09-16T03:38:20.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5400000000000001e-06 2024-09-16T03:38:20.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207107+0000", 2024-09-16T03:38:20.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6625999999999995e-05 2024-09-16T03:38:20.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207131+0000", 2024-09-16T03:38:20.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4128e-05 2024-09-16T03:38:20.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207712+0000", 2024-09-16T03:38:20.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00058124700000000001 2024-09-16T03:38:20.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207758+0000", 2024-09-16T03:38:20.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5429000000000002e-05 2024-09-16T03:38:20.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.753 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.754 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1723 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'48, mlcod=53'46)", 2024-09-16T03:38:20.754 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:20.754 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.065660179999998, 2024-09-16T03:38:20.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00088284599999999996, 2024-09-16T03:38:20.754 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.754 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.754 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.754 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.754 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:20.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:20.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649081+0000", 2024-09-16T03:38:20.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8299999999999995e-07 2024-09-16T03:38:20.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649101+0000", 2024-09-16T03:38:20.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0126e-05 2024-09-16T03:38:20.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649141+0000", 2024-09-16T03:38:20.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9609e-05 2024-09-16T03:38:20.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.756 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649161+0000", 2024-09-16T03:38:20.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9599999999999999e-05 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649299+0000", 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013822199999999999 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649318+0000", 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9273999999999999e-05 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649919+0000", 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060074100000000001 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649963+0000", 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4391000000000002e-05 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1724 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'49, mlcod=53'46)", 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:20.758 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.061995215, 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064536300000000003, 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:20.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652747+0000", 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.291e-06 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652753+0000", 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4450000000000001e-06 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652782+0000", 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9094999999999998e-05 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652797+0000", 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5021999999999999e-05 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652854+0000", 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6586999999999999e-05 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.761 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652885+0000", 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.146e-05 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653350+0000", 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046443599999999998 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653391+0000", 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1026999999999999e-05 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.762 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.763 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1725 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'50, mlcod=53'46)", 2024-09-16T03:38:20.763 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:20.763 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 29.061202391999998, 2024-09-16T03:38:20.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056324100000000002, 2024-09-16T03:38:20.763 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:20.763 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:20.763 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:20.763 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:20.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:20.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:20.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:20.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:20.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:20.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:20.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653539+0000", 2024-09-16T03:38:20.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9699999999999994e-07 2024-09-16T03:38:20.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653545+0000", 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9309999999999996e-06 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653557+0000", 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2340999999999999e-05 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653564+0000", 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2779999999999999e-06 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:20.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653605+0000", 2024-09-16T03:38:20.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1301000000000003e-05 2024-09-16T03:38:20.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:20.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653621+0000", 2024-09-16T03:38:20.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5951e-05 2024-09-16T03:38:20.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:20.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654064+0000", 2024-09-16T03:38:20.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000442843 2024-09-16T03:38:20.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:20.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:20.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:20.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654101+0000", 2024-09-16T03:38:20.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7598999999999997e-05 2024-09-16T03:38:20.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.767 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:20.767 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:20.767 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:20.768 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:20.769 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:38:20.791 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:20.789+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:20.861 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:20.859+0000 7fedb5739640 -1 osd.9 89 handle_osd_map injecting map crc failure 2024-09-16T03:38:20.869 INFO:tasks.rados.rados.0.smithi157.stdout:1219: finishing write tid 2 to smithi15736696-10 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:1219: finishing write tid 3 to smithi15736696-10 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:1220: finishing write tid 1 to smithi15736696-39 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:1220: finishing write tid 2 to smithi15736696-39 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:1220: finishing write tid 3 to smithi15736696-39 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:1220: finishing write tid 4 to smithi15736696-39 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:1220: finishing write tid 5 to smithi15736696-39 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:1220: oid 39 updating version 0 to 65 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:1220: oid 39 version 65 is already newer than 64 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:1220: oid 39 version 65 is already newer than 62 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:1220: oid 39 version 65 is already newer than 63 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 65 (ObjNum 228 snap 0 seq_num 228) dirty exists 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:1220: left oid 39 (ObjNum 228 snap 0 seq_num 228) 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:1224: got (0) Success 2024-09-16T03:38:20.870 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 85 (ObjNum 204 snap 0 seq_num 204) dirty exists 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1219: finishing write tid 4 to smithi15736696-10 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1219: finishing write tid 5 to smithi15736696-10 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1219: oid 10 updating version 0 to 69 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1219: oid 10 updating version 69 to 70 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1219: oid 10 version 70 is already newer than 67 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1219: oid 10 version 70 is already newer than 68 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 70 (ObjNum 227 snap 0 seq_num 227) dirty exists 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1219: left oid 10 (ObjNum 227 snap 0 seq_num 227) 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1215: done (7 left) 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1219: done (6 left) 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1220: done (5 left) 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1222: done (4 left) 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1223: done (3 left) 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1224: done (2 left) 2024-09-16T03:38:20.871 INFO:tasks.rados.rados.0.smithi157.stdout:1225: done (1 left) 2024-09-16T03:38:20.872 INFO:tasks.rados.rados.0.smithi157.stdout:1226: read oid 19 snap -1 2024-09-16T03:38:20.872 INFO:tasks.rados.rados.0.smithi157.stdout:1226: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:20.872 INFO:tasks.rados.rados.0.smithi157.stdout:1227: write (excl) oid 33 current snap is 0 2024-09-16T03:38:20.872 INFO:tasks.rados.rados.0.smithi157.stdout:1227: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-09-16T03:38:20.876 INFO:tasks.rados.rados.0.smithi157.stdout:1227: writing smithi15736696-33 from 665610 to 1463470 tid 1 2024-09-16T03:38:20.881 INFO:tasks.rados.rados.0.smithi157.stdout:1227: writing smithi15736696-33 from 1909329 to 2367734 tid 2 2024-09-16T03:38:20.885 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:20.885 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:20.885 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:20.886 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:20.886 INFO:tasks.rados.rados.0.smithi157.stdout:1227: writing smithi15736696-33 from 2816700 to 3226910 tid 3 2024-09-16T03:38:20.887 INFO:tasks.rados.rados.0.smithi157.stdout:1228: read oid 21 snap -1 2024-09-16T03:38:20.887 INFO:tasks.rados.rados.0.smithi157.stdout:1228: expect deleted 2024-09-16T03:38:20.887 INFO:tasks.rados.rados.0.smithi157.stdout:1229: read oid 13 snap -1 2024-09-16T03:38:20.887 INFO:tasks.rados.rados.0.smithi157.stdout:1229: expect (ObjNum 209 snap 0 seq_num 209) 2024-09-16T03:38:20.887 INFO:tasks.rados.rados.0.smithi157.stdout:1230: delete oid 39 current snap is 0 2024-09-16T03:38:20.887 INFO:tasks.rados.rados.0.smithi157.stdout:1221: finishing set_chunk to oid 50 2024-09-16T03:38:20.887 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 93 (ObjNum 218 snap 0 seq_num 218) dirty exists 2024-09-16T03:38:20.891 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:20.889+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:20.895 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:38:20.975 INFO:tasks.rados.rados.0.smithi157.stdout:1226: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:20.992 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:20.990+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:21.039 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1121937 2024-09-16T03:38:21.040 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:21.040 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:21.040 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:21.040 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:21.040 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:21.048 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:38:21.081 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:21.094 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:21.092+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:21.137 INFO:tasks.rados.rados.0.smithi157.stdout:1229: expect (ObjNum 209 snap 0 seq_num 209) 2024-09-16T03:38:21.137 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:21.194 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:21.194 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:21.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:21.194 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:21.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.194 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:5 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'1, mlcod=0'0)", 2024-09-16T03:38:21.194 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:38:21.194 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.223149996999993, 2024-09-16T03:38:21.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.105742802, 2024-09-16T03:38:21.195 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.195 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.195 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.195 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:38:21.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968147+0000", 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4e-07 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968752+0000", 2024-09-16T03:38:21.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060437900000000005 2024-09-16T03:38:21.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.197 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.197 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968762+0000", 2024-09-16T03:38:21.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0284e-05 2024-09-16T03:38:21.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.197 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.197 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968778+0000", 2024-09-16T03:38:21.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6124e-05 2024-09-16T03:38:21.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069797+0000", 2024-09-16T03:38:21.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10101895700000001 2024-09-16T03:38:21.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069833+0000", 2024-09-16T03:38:21.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6013000000000002e-05 2024-09-16T03:38:21.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.199 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073838+0000", 2024-09-16T03:38:21.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040049049999999996 2024-09-16T03:38:21.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.199 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073889+0000", 2024-09-16T03:38:21.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.13e-05 2024-09-16T03:38:21.199 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.199 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.199 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:7 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'3, mlcod=0'0)", 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.216257498000004, 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20061241399999999, 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.200 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:38:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975042+0000", 2024-09-16T03:38:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5289999999999998e-06 2024-09-16T03:38:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975079+0000", 2024-09-16T03:38:21.201 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7626999999999997e-05 2024-09-16T03:38:21.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.202 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.202 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.202 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975104+0000", 2024-09-16T03:38:21.202 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4958999999999999e-05 2024-09-16T03:38:21.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.202 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.202 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.202 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.075211+0000", 2024-09-16T03:38:21.202 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100106814 2024-09-16T03:38:21.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080573+0000", 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0053618650000000004 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080592+0000", 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9079000000000001e-05 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175497+0000", 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.094904832999999994 2024-09-16T03:38:21.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175651+0000", 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015470800000000001 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.005304326000001, 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10311670100000001, 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.204 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186761+0000", 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076880999999999998 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186772+0000", 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0986e-05 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186799+0000", 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6254e-05 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186876+0000", 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7523999999999998e-05 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186896+0000", 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0302999999999999e-05 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.207 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.208 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.208 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289077+0000", 2024-09-16T03:38:21.208 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10218086899999999 2024-09-16T03:38:21.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.208 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.208 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289109+0000", 2024-09-16T03:38:21.208 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1625000000000003e-05 2024-09-16T03:38:21.208 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.208 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.208 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.209 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36)", 2024-09-16T03:38:21.209 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:38:21.209 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.004098537000004, 2024-09-16T03:38:21.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102408122, 2024-09-16T03:38:21.209 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.209 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.209 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.209 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.209 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:38:21.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.210 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:38:21.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.211 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187199+0000", 2024-09-16T03:38:21.211 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6199999999999996e-07 2024-09-16T03:38:21.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.212 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188184+0000", 2024-09-16T03:38:21.212 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000985442 2024-09-16T03:38:21.212 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.212 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.212 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.212 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188194+0000", 2024-09-16T03:38:21.212 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5659999999999996e-06 2024-09-16T03:38:21.212 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.213 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188210+0000", 2024-09-16T03:38:21.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5858999999999998e-05 2024-09-16T03:38:21.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.213 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288363+0000", 2024-09-16T03:38:21.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100152872 2024-09-16T03:38:21.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.213 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.214 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288390+0000", 2024-09-16T03:38:21.214 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7081999999999999e-05 2024-09-16T03:38:21.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.214 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.214 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289577+0000", 2024-09-16T03:38:21.214 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011875740000000001 2024-09-16T03:38:21.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.215 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.215 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289606+0000", 2024-09-16T03:38:21.215 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8864999999999999e-05 2024-09-16T03:38:21.215 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.215 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.215 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.215 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.215 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.215 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.5.21:19 3.3 e47/31 3:fb369346:::smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 47'44, mlcod=47'41)", 2024-09-16T03:38:21.216 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.339975+0000", 2024-09-16T03:38:21.216 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.851321820999999, 2024-09-16T03:38:21.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10201615, 2024-09-16T03:38:21.216 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.216 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.216 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.216 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.216 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.216 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.216 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339975+0000", 2024-09-16T03:38:21.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.216 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339975+0000", 2024-09-16T03:38:21.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339976+0000", 2024-09-16T03:38:21.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.204e-06 2024-09-16T03:38:21.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339982+0000", 2024-09-16T03:38:21.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:38:21.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339992+0000", 2024-09-16T03:38:21.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0196e-05 2024-09-16T03:38:21.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339998+0000", 2024-09-16T03:38:21.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.463e-06 2024-09-16T03:38:21.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.438719+0000", 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098721712000000003 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.438747+0000", 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7475e-05 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.441935+0000", 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031878480000000001 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.219 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.220 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.441991+0000", 2024-09-16T03:38:21.220 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6100999999999997e-05 2024-09-16T03:38:21.220 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.220 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.220 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.220 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.220 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.220 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:44 3.3 e49/31 3:dc284e8d:::smithi15736696-28 low_tier:head v 49'55, mlcod=49'53)", 2024-09-16T03:38:21.220 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:38:21.220 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 51.068333101999997, 2024-09-16T03:38:21.220 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.149341897, 2024-09-16T03:38:21.220 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.220 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.221 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.221 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.221 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.221 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.221 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:38:21.221 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.221 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.221 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.221 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.221 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:38:21.221 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.221 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.221 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.222 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.222 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122964+0000", 2024-09-16T03:38:21.222 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.91e-07 2024-09-16T03:38:21.222 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.222 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.222 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.222 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122968+0000", 2024-09-16T03:38:21.222 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1470000000000003e-06 2024-09-16T03:38:21.222 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.222 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.222 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.222 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122979+0000", 2024-09-16T03:38:21.222 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1090000000000001e-05 2024-09-16T03:38:21.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.223 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.223 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.223 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.223110+0000", 2024-09-16T03:38:21.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100131357 2024-09-16T03:38:21.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.223 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.223 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.223 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271688+0000", 2024-09-16T03:38:21.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048578192999999999 2024-09-16T03:38:21.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.223 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271712+0000", 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4066e-05 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272260+0000", 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054737000000000004 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272305+0000", 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5482999999999998e-05 2024-09-16T03:38:21.224 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.225 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.225 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.225 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.225 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1007 3.3 e50/31 3:fda66a0d:::920f29fe85d0745d4cd24a887714ac53f1b9a978ec2208ff473c99287fdf8feb:head v 50'57, mlcod=49'55)", 2024-09-16T03:38:21.225 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:38:21.225 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.596706548, 2024-09-16T03:38:21.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20236485200000001, 2024-09-16T03:38:21.225 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.225 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.225 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.225 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.225 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.226 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.226 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:38:21.226 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.226 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.226 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.226 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:38:21.226 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.226 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.226 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.226 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594592+0000", 2024-09-16T03:38:21.227 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0700000000000001e-06 2024-09-16T03:38:21.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.227 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595508+0000", 2024-09-16T03:38:21.227 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00091599499999999996 2024-09-16T03:38:21.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.227 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595521+0000", 2024-09-16T03:38:21.227 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:38:21.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.228 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.228 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595531+0000", 2024-09-16T03:38:21.228 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2499999999999995e-06 2024-09-16T03:38:21.228 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.228 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.228 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.228 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795762+0000", 2024-09-16T03:38:21.228 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200230977 2024-09-16T03:38:21.228 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.228 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.228 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.228 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795778+0000", 2024-09-16T03:38:21.228 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:38:21.229 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.229 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.229 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.229 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796912+0000", 2024-09-16T03:38:21.229 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011338520000000001 2024-09-16T03:38:21.229 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.229 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.229 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.229 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796955+0000", 2024-09-16T03:38:21.229 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2768000000000001e-05 2024-09-16T03:38:21.229 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.229 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.229 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.230 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.230 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.230 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1247 4.11 4:89839b7e:::smithi15736696-12:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-16T03:38:21.230 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:38:21.230 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 44.235719367000002, 2024-09-16T03:38:21.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202294525, 2024-09-16T03:38:21.230 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.230 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.230 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.230 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.230 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.230 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.230 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1247 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955579+0000", 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3240000000000001e-06 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955587+0000", 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955600+0000", 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2723e-05 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955612+0000", 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2422999999999999e-05 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055837+0000", 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10022539699999999 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055974+0000", 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000136995 2024-09-16T03:38:21.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,7,8,11,12", 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.056092+0000", 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011738 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057209+0000", 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011170329999999999 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057262+0000", 2024-09-16T03:38:21.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2859000000000003e-05 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157594+0000", 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10033241699999999 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157648+0000", 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3863999999999998e-05 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157692+0000", 2024-09-16T03:38:21.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4406999999999997e-05 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157732+0000", 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9156000000000003e-05 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157780+0000", 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8801999999999998e-05 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.236 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157872+0000", 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1311999999999996e-05 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1447 3.3 e58/31 3:d2eec433:::1fda83a16db131b06a5bf7beb910c0624eb0c0478a30a35e6b1e31b63f6e3ff4:head v 58'90, mlcod=58'88)", 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:41.806613+0000", 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 39.384683649999999, 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101561763, 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.237 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.806613+0000", 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.806613+0000", 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.806616+0000", 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.456e-06 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.238 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.807133+0000", 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000516787 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.807144+0000", 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0448000000000001e-05 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.807150+0000", 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6989999999999997e-06 2024-09-16T03:38:21.239 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.240 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.240 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.240 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.907320+0000", 2024-09-16T03:38:21.240 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10016939900000001 2024-09-16T03:38:21.240 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.240 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.240 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.240 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.907334+0000", 2024-09-16T03:38:21.240 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4241e-05 2024-09-16T03:38:21.240 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.240 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.240 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.240 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.908143+0000", 2024-09-16T03:38:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00080902999999999997 2024-09-16T03:38:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:41.908175+0000", 2024-09-16T03:38:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1702999999999998e-05 2024-09-16T03:38:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1518 3.3 e59/31 3:df1be05b:::e06092e68680905b5745d5057e93f6c7581518aed07ca5db2802546e9c29aeef:head v 59'95, mlcod=58'93)", 2024-09-16T03:38:21.241 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:38:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.933275694000002, 2024-09-16T03:38:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104523454, 2024-09-16T03:38:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:38:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.242 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258022+0000", 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.372e-06 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258462+0000", 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00043970100000000002 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.244 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258476+0000", 2024-09-16T03:38:21.244 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.376e-05 2024-09-16T03:38:21.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.244 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258483+0000", 2024-09-16T03:38:21.244 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.294e-06 2024-09-16T03:38:21.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.244 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358679+0000", 2024-09-16T03:38:21.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100196177 2024-09-16T03:38:21.245 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.245 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.245 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358718+0000", 2024-09-16T03:38:21.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9295999999999998e-05 2024-09-16T03:38:21.245 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.245 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.245 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362496+0000", 2024-09-16T03:38:21.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037772069999999999 2024-09-16T03:38:21.245 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.245 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.246 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.246 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362544+0000", 2024-09-16T03:38:21.246 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8646999999999997e-05 2024-09-16T03:38:21.246 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.246 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.246 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.246 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.246 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.246 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1923 4.4 e73/50 4:239dbd85:::smithi15736696-2:head v 73'19, mlcod=62'17)", 2024-09-16T03:38:21.246 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:38:21.246 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.227414872000001, 2024-09-16T03:38:21.246 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104144848, 2024-09-16T03:38:21.246 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.247 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963884+0000", 2024-09-16T03:38:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:38:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964597+0000", 2024-09-16T03:38:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071314700000000002 2024-09-16T03:38:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964615+0000", 2024-09-16T03:38:21.248 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8209000000000001e-05 2024-09-16T03:38:21.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.249 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.249 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964626+0000", 2024-09-16T03:38:21.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0511e-05 2024-09-16T03:38:21.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.249 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.249 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064779+0000", 2024-09-16T03:38:21.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100153495 2024-09-16T03:38:21.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.249 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064794+0000", 2024-09-16T03:38:21.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4946000000000001e-05 2024-09-16T03:38:21.250 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.250 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.067999+0000", 2024-09-16T03:38:21.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032050709999999999 2024-09-16T03:38:21.250 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.250 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068026+0000", 2024-09-16T03:38:21.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7066000000000001e-05 2024-09-16T03:38:21.250 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.251 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.251 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.251 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1924 4.4 e73/50 4:239dbd85:::smithi15736696-2:head v 73'20, mlcod=62'17)", 2024-09-16T03:38:21.251 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:59.965625+0000", 2024-09-16T03:38:21.251 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 21.225671453, 2024-09-16T03:38:21.251 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102804671, 2024-09-16T03:38:21.251 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.251 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.251 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.251 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.965625+0000", 2024-09-16T03:38:21.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.252 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.965625+0000", 2024-09-16T03:38:21.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.252 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.965627+0000", 2024-09-16T03:38:21.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5179999999999999e-06 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.966256+0000", 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00062937300000000002 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.966274+0000", 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8257000000000002e-05 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.253 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.966284+0000", 2024-09-16T03:38:21.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4299999999999995e-06 2024-09-16T03:38:21.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.254 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.254 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.066012+0000", 2024-09-16T03:38:21.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099728368999999997 2024-09-16T03:38:21.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.254 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.255 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.066028+0000", 2024-09-16T03:38:21.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6512e-05 2024-09-16T03:38:21.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.255 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068405+0000", 2024-09-16T03:38:21.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023766740000000001 2024-09-16T03:38:21.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.255 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068430+0000", 2024-09-16T03:38:21.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4538000000000001e-05 2024-09-16T03:38:21.256 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.256 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.256 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.256 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.256 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.256 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:106 3.3 e73/31 3:dbb0d6f3:::b417679517d63919e63cf420e20217e40570d74b9322ef8b960ef314107040f0:head v 73'120, mlcod=73'116)", 2024-09-16T03:38:21.256 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:21.256 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.913193946, 2024-09-16T03:38:21.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.1035437, 2024-09-16T03:38:21.256 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.256 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.256 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.256 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:21.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.257 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:21.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.257 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4900000000000004e-07 2024-09-16T03:38:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278120+0000", 2024-09-16T03:38:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6214000000000001e-05 2024-09-16T03:38:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278129+0000", 2024-09-16T03:38:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.115e-06 2024-09-16T03:38:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.258 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278138+0000", 2024-09-16T03:38:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.329e-06 2024-09-16T03:38:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.378298+0000", 2024-09-16T03:38:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100159945 2024-09-16T03:38:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.378320+0000", 2024-09-16T03:38:21.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1810999999999998e-05 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.381607+0000", 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032868210000000001 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.381646+0000", 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9515999999999999e-05 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.260 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2145 4.4 e77/50 4:239dbd85:::smithi15736696-2:head v 77'22, mlcod=73'20)", 2024-09-16T03:38:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.410542+0000", 2024-09-16T03:38:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.780754149, 2024-09-16T03:38:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10336585099999999, 2024-09-16T03:38:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.410542+0000", 2024-09-16T03:38:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.410542+0000", 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.410543+0000", 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0599999999999999e-07 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.411160+0000", 2024-09-16T03:38:21.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061733000000000003 2024-09-16T03:38:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.411172+0000", 2024-09-16T03:38:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2153e-05 2024-09-16T03:38:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511308+0000", 2024-09-16T03:38:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10013572900000001 2024-09-16T03:38:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.263 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511607+0000", 2024-09-16T03:38:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029879100000000002 2024-09-16T03:38:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511647+0000", 2024-09-16T03:38:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0241999999999999e-05 2024-09-16T03:38:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.513870+0000", 2024-09-16T03:38:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022230499999999999 2024-09-16T03:38:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.513908+0000", 2024-09-16T03:38:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8349999999999997e-05 2024-09-16T03:38:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2151 4.11 4:89839b7e:::smithi15736696-12:head [read 0~2487436,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:38:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.511486+0000", 2024-09-16T03:38:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.679810641, 2024-09-16T03:38:21.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101722046, 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2151 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511486+0000", 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511486+0000", 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511487+0000", 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6559999999999999e-06 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511500+0000", 2024-09-16T03:38:21.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2777e-05 2024-09-16T03:38:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511533+0000", 2024-09-16T03:38:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3166000000000001e-05 2024-09-16T03:38:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511550+0000", 2024-09-16T03:38:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6348999999999999e-05 2024-09-16T03:38:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.611701+0000", 2024-09-16T03:38:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015094400000001 2024-09-16T03:38:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.611833+0000", 2024-09-16T03:38:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013260999999999999 2024-09-16T03:38:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613208+0000", 2024-09-16T03:38:21.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013745439999999999 2024-09-16T03:38:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2152 4.11 4:89839b7e:::smithi15736696-12:head [read 0~2487436,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:38:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:38:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.679722465999999, 2024-09-16T03:38:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10362692900000001, 2024-09-16T03:38:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:21.270 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.271 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.271 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.271 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2152 2024-09-16T03:38:21.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.271 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.271 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:38:21.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.271 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511576+0000", 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511584+0000", 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5939999999999994e-06 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511612+0000", 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7855e-05 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.611754+0000", 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100141641 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613255+0000", 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001500744 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613366+0000", 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000111088 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.615201+0000", 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0018353009999999999 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2146 4.4 e77/50 4:239dbd85:::smithi15736696-2:head v 77'23, mlcod=73'20)", 2024-09-16T03:38:21.274 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.612746+0000", 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.578550707, 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101439637, 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.612746+0000", 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.612746+0000", 2024-09-16T03:38:21.275 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.612748+0000", 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.159e-06 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613080+0000", 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00033167500000000003 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613095+0000", 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5350999999999999e-05 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613129+0000", 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3603999999999999e-05 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613317+0000", 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018836100000000001 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.277 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613336+0000", 2024-09-16T03:38:21.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8765000000000001e-05 2024-09-16T03:38:21.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.278 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.714151+0000", 2024-09-16T03:38:21.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100815473 2024-09-16T03:38:21.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.278 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.714185+0000", 2024-09-16T03:38:21.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4248999999999997e-05 2024-09-16T03:38:21.278 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2316 4.11 4:89839b7e:::smithi15736696-12:head [read 0~2487436,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:11.770262+0000", 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.4210341619999998, 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10250772399999999, 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.279 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2316 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770262+0000", 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770262+0000", 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.280 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.281 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770265+0000", 2024-09-16T03:38:21.281 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8930000000000001e-06 2024-09-16T03:38:21.281 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.281 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.281 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.281 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770275+0000", 2024-09-16T03:38:21.281 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3810000000000005e-06 2024-09-16T03:38:21.281 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.281 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.281 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.281 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770287+0000", 2024-09-16T03:38:21.281 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2889e-05 2024-09-16T03:38:21.281 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.870382+0000", 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100094409 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.870584+0000", 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020166199999999999 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.870806+0000", 2024-09-16T03:38:21.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022220200000000001 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.872770+0000", 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019642879999999998 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2445 4.11 4:89839b7e:::smithi15736696-12:head [sparse-read 1818507~599510,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.7471330439999999, 2024-09-16T03:38:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200613392, 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2445 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.284 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444167+0000", 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6650000000000001e-06 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444182+0000", 2024-09-16T03:38:21.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4678000000000001e-05 2024-09-16T03:38:21.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444207+0000", 2024-09-16T03:38:21.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4768000000000001e-05 2024-09-16T03:38:21.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.544359+0000", 2024-09-16T03:38:21.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100152804 2024-09-16T03:38:21.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644561+0000", 2024-09-16T03:38:21.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10020214500000001 2024-09-16T03:38:21.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644624+0000", 2024-09-16T03:38:21.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2444e-05 2024-09-16T03:38:21.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644777+0000", 2024-09-16T03:38:21.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000152888 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2629 3.3 e89/31 3:e23364e2:::e447860b11e9f9fde68b75cb1d2f204a74952889ec51a2417ef18c1427ff9a70:head v 89'173, mlcod=87'171)", 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.73939035099999995, 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10848095100000001, 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.288 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:38:21.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:38:21.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451907+0000", 2024-09-16T03:38:21.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6599999999999996e-07 2024-09-16T03:38:21.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456443+0000", 2024-09-16T03:38:21.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0045365509999999998 2024-09-16T03:38:21.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456470+0000", 2024-09-16T03:38:21.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6897e-05 2024-09-16T03:38:21.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456496+0000", 2024-09-16T03:38:21.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.618e-05 2024-09-16T03:38:21.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556666+0000", 2024-09-16T03:38:21.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100169678 2024-09-16T03:38:21.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556686+0000", 2024-09-16T03:38:21.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9740000000000001e-05 2024-09-16T03:38:21.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560345+0000", 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003658821 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560387+0000", 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2518000000000002e-05 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.292 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.293 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:21.294 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:38:21.295 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:21.292+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:21.368 INFO:tasks.rados.rados.0.smithi157.stdout:1221: done (5 left) 2024-09-16T03:38:21.368 INFO:tasks.rados.rados.0.smithi157.stdout:1226: done (4 left) 2024-09-16T03:38:21.368 INFO:tasks.rados.rados.0.smithi157.stdout:1229: done (3 left) 2024-09-16T03:38:21.368 INFO:tasks.rados.rados.0.smithi157.stdout:1230: done (2 left) 2024-09-16T03:38:21.368 INFO:tasks.rados.rados.0.smithi157.stdout:1231: write oid 23 current snap is 0 2024-09-16T03:38:21.368 INFO:tasks.rados.rados.0.smithi157.stdout:1231: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-09-16T03:38:21.370 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:21.368+0000 7f49439d4640 -1 osd.4 90 handle_osd_map injecting map crc failure 2024-09-16T03:38:21.372 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T03:38:21.374 INFO:tasks.rados.rados.0.smithi157.stdout:1231: writing smithi15736696-23 from 699224 to 1247582 tid 1 2024-09-16T03:38:21.374 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:38:21.381 INFO:tasks.rados.rados.0.smithi157.stdout:1231: writing smithi15736696-23 from 2026293 to 2488367 tid 2 2024-09-16T03:38:21.387 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-16T03:38:21.388 INFO:tasks.rados.rados.0.smithi157.stdout:1231: writing smithi15736696-23 from 3271546 to 3704300 tid 3 2024-09-16T03:38:21.388 INFO:tasks.rados.rados.0.smithi157.stdout:1232: write oid 11 current snap is 0 2024-09-16T03:38:21.388 INFO:tasks.rados.rados.0.smithi157.stdout:1232: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2024-09-16T03:38:21.394 INFO:tasks.rados.rados.0.smithi157.stdout:1232: writing smithi15736696-11 from 716031 to 1339638 tid 1 2024-09-16T03:38:21.402 INFO:tasks.rados.rados.0.smithi157.stdout:1232: writing smithi15736696-11 from 2084775 to 2690507 tid 2 2024-09-16T03:38:21.403 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:21.404 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:21.404 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:21.404 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:21.409 INFO:tasks.rados.rados.0.smithi157.stdout:1232: writing smithi15736696-11 from 3298969 to 3916031 tid 3 2024-09-16T03:38:21.410 INFO:tasks.rados.rados.0.smithi157.stdout:1233: write oid 45 current snap is 0 2024-09-16T03:38:21.410 INFO:tasks.rados.rados.0.smithi157.stdout:1233: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-09-16T03:38:21.413 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:38:21.417 INFO:tasks.rados.rados.0.smithi157.stdout:1233: writing smithi15736696-45 from 732838 to 1431694 tid 1 2024-09-16T03:38:21.422 INFO:tasks.rados.rados.0.smithi157.stdout:1233: writing smithi15736696-45 from 2026904 to 2492647 tid 2 2024-09-16T03:38:21.428 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:21.426+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:21.430 INFO:tasks.rados.rados.0.smithi157.stdout:1233: writing smithi15736696-45 from 3210039 to 3781690 tid 3 2024-09-16T03:38:21.430 INFO:tasks.rados.rados.0.smithi157.stdout:1234: read oid 28 snap -1 2024-09-16T03:38:21.430 INFO:tasks.rados.rados.0.smithi157.stdout:1234: expect (ObjNum 186 snap 0 seq_num 186) 2024-09-16T03:38:21.430 INFO:tasks.rados.rados.0.smithi157.stdout:1235: read oid 37 snap -1 2024-09-16T03:38:21.430 INFO:tasks.rados.rados.0.smithi157.stdout:1235: expect (ObjNum 217 snap 0 seq_num 217) 2024-09-16T03:38:21.430 INFO:tasks.rados.rados.0.smithi157.stdout:1236: write (excl) oid 15 current snap is 0 2024-09-16T03:38:21.431 INFO:tasks.rados.rados.0.smithi157.stdout:1227: finishing write tid 1 to smithi15736696-33 2024-09-16T03:38:21.431 INFO:tasks.rados.rados.0.smithi157.stdout:1227: finishing write tid 2 to smithi15736696-33 2024-09-16T03:38:21.431 INFO:tasks.rados.rados.0.smithi157.stdout:1227: finishing write tid 3 to smithi15736696-33 2024-09-16T03:38:21.431 INFO:tasks.rados.rados.0.smithi157.stdout:1236: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2024-09-16T03:38:21.438 INFO:tasks.rados.rados.0.smithi157.stdout:1236: writing smithi15736696-15 from 749645 to 1523750 tid 1 2024-09-16T03:38:21.444 INFO:tasks.rados.rados.0.smithi157.stdout:1236: writing smithi15736696-15 from 2085386 to 2694787 tid 2 2024-09-16T03:38:21.452 INFO:tasks.rados.rados.0.smithi157.stdout:1236: writing smithi15736696-15 from 3237462 to 3949645 tid 3 2024-09-16T03:38:21.452 INFO:tasks.rados.rados.0.smithi157.stdout:1228: done (7 left) 2024-09-16T03:38:21.453 INFO:tasks.rados.rados.0.smithi157.stdout:1237: read oid 44 snap -1 2024-09-16T03:38:21.453 INFO:tasks.rados.rados.0.smithi157.stdout:1237: expect (ObjNum 219 snap 0 seq_num 219) 2024-09-16T03:38:21.453 INFO:tasks.rados.rados.0.smithi157.stdout:1238: tier_flush oid 50 2024-09-16T03:38:21.453 INFO:tasks.rados.rados.0.smithi157.stdout:1238: tier_flush oid 50 snap -1 2024-09-16T03:38:21.453 INFO:tasks.rados.rados.0.smithi157.stdout:1234: expect (ObjNum 186 snap 0 seq_num 186) 2024-09-16T03:38:21.495 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:21.493+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:21.518 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:21.518 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:21.519 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:21.519 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:21.519 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:21.527 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:38:21.597 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:21.595+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:21.675 INFO:tasks.rados.rados.0.smithi157.stdout:1227: finishing write tid 4 to smithi15736696-33 2024-09-16T03:38:21.675 INFO:tasks.rados.rados.0.smithi157.stdout:1227: finishing write tid 5 to smithi15736696-33 2024-09-16T03:38:21.675 INFO:tasks.rados.rados.0.smithi157.stdout:1227: oid 33 updating version 0 to 55 2024-09-16T03:38:21.675 INFO:tasks.rados.rados.0.smithi157.stdout:1227: oid 33 version 55 is already newer than 53 2024-09-16T03:38:21.675 INFO:tasks.rados.rados.0.smithi157.stdout:1227: oid 33 version 55 is already newer than 52 2024-09-16T03:38:21.675 INFO:tasks.rados.rados.0.smithi157.stdout:1227: oid 33 version 55 is already newer than 54 2024-09-16T03:38:21.676 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 55 (ObjNum 230 snap 0 seq_num 230) dirty exists 2024-09-16T03:38:21.676 INFO:tasks.rados.rados.0.smithi157.stdout:1227: left oid 33 (ObjNum 230 snap 0 seq_num 230) 2024-09-16T03:38:21.676 INFO:tasks.rados.rados.0.smithi157.stdout:1236: finishing write tid 1 to smithi15736696-15 2024-09-16T03:38:21.676 INFO:tasks.rados.rados.0.smithi157.stdout:1227: done (8 left) 2024-09-16T03:38:21.676 INFO:tasks.rados.rados.0.smithi157.stdout:1234: done (7 left) 2024-09-16T03:38:21.676 INFO:tasks.rados.rados.0.smithi157.stdout:1239: set_chunk oid 27 target oid 2024-09-16T03:38:21.678 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:21.678 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:21.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:21.678 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:21.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.678 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:19 3.1 3:9aba8983:::smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:21.678 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:38:21.678 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 65.256696568999999, 2024-09-16T03:38:21.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30153978799999998, 2024-09-16T03:38:21.678 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.679 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.679 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.679 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:38:21.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:38:21.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418452+0000", 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0363000000000001e-05 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418470+0000", 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7506000000000001e-05 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418504+0000", 2024-09-16T03:38:21.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4542999999999998e-05 2024-09-16T03:38:21.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418522+0000", 2024-09-16T03:38:21.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8070000000000001e-05 2024-09-16T03:38:21.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418662+0000", 2024-09-16T03:38:21.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013946299999999999 2024-09-16T03:38:21.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418826+0000", 2024-09-16T03:38:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016469299999999999 2024-09-16T03:38:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418989+0000", 2024-09-16T03:38:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000162759 2024-09-16T03:38:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.419750+0000", 2024-09-16T03:38:21.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076104299999999998 2024-09-16T03:38:21.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.419865+0000", 2024-09-16T03:38:21.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114338 2024-09-16T03:38:21.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719876+0000", 2024-09-16T03:38:21.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30001127300000002 2024-09-16T03:38:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719920+0000", 2024-09-16T03:38:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4549999999999999e-05 2024-09-16T03:38:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719972+0000", 2024-09-16T03:38:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1187000000000003e-05 2024-09-16T03:38:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.684 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:10 3.1 3:8f38b759:::smithi15736696-6 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 65.167263930999994, 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.21220708699999999, 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:38:21.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507864+0000", 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507866+0000", 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9999999999999999e-06 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507873+0000", 2024-09-16T03:38:21.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7900000000000002e-06 2024-09-16T03:38:21.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.687 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.687 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.687 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507888+0000", 2024-09-16T03:38:21.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4554e-05 2024-09-16T03:38:21.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.687 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.687 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.687 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507896+0000", 2024-09-16T03:38:21.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1130000000000001e-06 2024-09-16T03:38:21.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.687 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.507967+0000", 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0808000000000004e-05 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508028+0000", 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1325e-05 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508146+0000", 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000117922 2024-09-16T03:38:21.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.689 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.508722+0000", 2024-09-16T03:38:21.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000575779 2024-09-16T03:38:21.689 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.689 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.609567+0000", 2024-09-16T03:38:21.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100845448 2024-09-16T03:38:21.689 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.689 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720013+0000", 2024-09-16T03:38:21.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.110445644 2024-09-16T03:38:21.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720045+0000", 2024-09-16T03:38:21.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2186e-05 2024-09-16T03:38:21.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.691 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720071+0000", 2024-09-16T03:38:21.691 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6517999999999999e-05 2024-09-16T03:38:21.691 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.691 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.691 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.692 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.5.21:11 3.1 3:adc537a6:::smithi15736696-7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:21.692 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:38:21.692 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 65.161920701, 2024-09-16T03:38:21.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20740488100000001, 2024-09-16T03:38:21.692 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.693 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.693 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.693 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.693 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.693 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:38:21.693 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.693 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.693 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.694 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513208+0000", 2024-09-16T03:38:21.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.694 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.694 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.694 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513209+0000", 2024-09-16T03:38:21.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7439999999999999e-06 2024-09-16T03:38:21.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.694 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.694 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.694 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513216+0000", 2024-09-16T03:38:21.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9890000000000003e-06 2024-09-16T03:38:21.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513230+0000", 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3263e-05 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513236+0000", 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7270000000000001e-06 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513297+0000", 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0767999999999998e-05 2024-09-16T03:38:21.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513375+0000", 2024-09-16T03:38:21.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8014000000000002e-05 2024-09-16T03:38:21.696 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.513529+0000", 2024-09-16T03:38:21.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015342900000000001 2024-09-16T03:38:21.696 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.514258+0000", 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00072896799999999998 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.709867+0000", 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.195609016 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720546+0000", 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.010679905999999999 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720583+0000", 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6659999999999998e-05 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.720613+0000", 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9397e-05 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:779 3.1 3:abdaa5c9:::2782d12a88f9842f7f754fca32c4222cf10fb780cd136830d32faaa700e687ba:head [write 0~45056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:38:21.698 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 56.188548253, 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30397390600000002, 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 779 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:38:21.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.700 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.700 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:38:21.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.700 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.700 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486583+0000", 2024-09-16T03:38:21.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.762e-06 2024-09-16T03:38:21.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.700 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.700 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486649+0000", 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6143999999999996e-05 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486669+0000", 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9837e-05 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486681+0000", 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2194999999999999e-05 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486746+0000", 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5480000000000003e-05 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486752+0000", 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4659999999999998e-06 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588316+0000", 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10156393399999999 2024-09-16T03:38:21.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588539+0000", 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022308300000000001 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588647+0000", 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000108047 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.589516+0000", 2024-09-16T03:38:21.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086926899999999999 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.789989+0000", 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20047296100000001 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790478+0000", 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048882099999999996 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790518+0000", 2024-09-16T03:38:21.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0343000000000003e-05 2024-09-16T03:38:21.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790554+0000", 2024-09-16T03:38:21.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5564e-05 2024-09-16T03:38:21.705 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.705 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.705 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.705 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:38 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.836721062999999, 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30216061300000002, 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.706 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:38:21.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.707 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838411+0000", 2024-09-16T03:38:21.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3780000000000001e-06 2024-09-16T03:38:21.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.707 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838422+0000", 2024-09-16T03:38:21.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1223000000000001e-05 2024-09-16T03:38:21.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838450+0000", 2024-09-16T03:38:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7878999999999999e-05 2024-09-16T03:38:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838464+0000", 2024-09-16T03:38:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3811e-05 2024-09-16T03:38:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838539+0000", 2024-09-16T03:38:21.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5685000000000005e-05 2024-09-16T03:38:21.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838604+0000", 2024-09-16T03:38:21.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4980999999999993e-05 2024-09-16T03:38:21.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838732+0000", 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012821100000000001 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839340+0000", 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060762399999999997 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840716+0000", 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001375692 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140491+0000", 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29977497600000003 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140534+0000", 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3411999999999997e-05 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140568+0000", 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3741000000000001e-05 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.711 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.712 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:39 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:21.712 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:38:21.712 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.835430844999998, 2024-09-16T03:38:21.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40113680400000001, 2024-09-16T03:38:21.712 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.712 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.712 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.712 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.712 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.712 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839700+0000", 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.571e-06 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839704+0000", 2024-09-16T03:38:21.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8399999999999997e-06 2024-09-16T03:38:21.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839712+0000", 2024-09-16T03:38:21.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9990000000000008e-06 2024-09-16T03:38:21.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839720+0000", 2024-09-16T03:38:21.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0099999999999995e-06 2024-09-16T03:38:21.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839783+0000", 2024-09-16T03:38:21.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3014000000000003e-05 2024-09-16T03:38:21.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839870+0000", 2024-09-16T03:38:21.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7392999999999995e-05 2024-09-16T03:38:21.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840049+0000", 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017825800000000001 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840874+0000", 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082532099999999995 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.040285+0000", 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19941088400000001 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240748+0000", 2024-09-16T03:38:21.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20046322699999999 2024-09-16T03:38:21.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240786+0000", 2024-09-16T03:38:21.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8158000000000003e-05 2024-09-16T03:38:21.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240834+0000", 2024-09-16T03:38:21.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8128999999999999e-05 2024-09-16T03:38:21.717 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.718 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.718 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.718 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:983 3.1 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head [write 0~2138112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:38:21.718 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:38:21.718 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 51.611527905000003, 2024-09-16T03:38:21.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30819799799999997, 2024-09-16T03:38:21.718 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.718 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.718 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 983 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063610+0000", 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2990000000000002e-06 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065735+0000", 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021257799999999999 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065752+0000", 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6895999999999999e-05 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065762+0000", 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6830000000000003e-06 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065871+0000", 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010904900000000001 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066005+0000", 2024-09-16T03:38:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013369799999999999 2024-09-16T03:38:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066135+0000", 2024-09-16T03:38:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000130481 2024-09-16T03:38:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.069822+0000", 2024-09-16T03:38:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036863759999999999 2024-09-16T03:38:21.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272370+0000", 2024-09-16T03:38:21.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20254802599999999 2024-09-16T03:38:21.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371671+0000", 2024-09-16T03:38:21.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099300893000000001 2024-09-16T03:38:21.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371754+0000", 2024-09-16T03:38:21.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3075e-05 2024-09-16T03:38:21.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371798+0000", 2024-09-16T03:38:21.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4742e-05 2024-09-16T03:38:21.724 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.724 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.724 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.725 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1515 3.1 3:9c9e970c:::7249cb65b52cadf4fe8f4ec923c7e548d1de7d9628823acd2c8de19121f1c1ce:head [write 0~1094656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:38:21.725 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:38:21.725 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 36.725402168000002, 2024-09-16T03:38:21.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.305958065, 2024-09-16T03:38:21.725 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.725 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.725 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.725 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.725 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.725 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.725 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1515 2024-09-16T03:38:21.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.725 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949729+0000", 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.785e-06 2024-09-16T03:38:21.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950933+0000", 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001204341 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950950+0000", 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6838e-05 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950960+0000", 2024-09-16T03:38:21.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0316000000000001e-05 2024-09-16T03:38:21.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951065+0000", 2024-09-16T03:38:21.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104128 2024-09-16T03:38:21.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951202+0000", 2024-09-16T03:38:21.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013706600000000001 2024-09-16T03:38:21.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951350+0000", 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000148197 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.953857+0000", 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0025067150000000001 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.958049+0000", 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0041924650000000003 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255547+0000", 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29749841900000001 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255649+0000", 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101513 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255684+0000", 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5281999999999999e-05 2024-09-16T03:38:21.730 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.731 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.731 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.731 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1704 3.1 3:b6a94245:::f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99:head [write 0~453120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:21.731 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:38:21.731 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.882290813000001, 2024-09-16T03:38:21.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40403313699999999, 2024-09-16T03:38:21.731 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.731 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.731 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.731 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.732 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.732 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.732 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1704 2024-09-16T03:38:21.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.732 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:38:21.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:38:21.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792841+0000", 2024-09-16T03:38:21.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.433e-06 2024-09-16T03:38:21.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793329+0000", 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048839200000000004 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793345+0000", 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6079e-05 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793354+0000", 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.32e-06 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793418+0000", 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3944999999999998e-05 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793549+0000", 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013159900000000001 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793666+0000", 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116532 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.797707+0000", 2024-09-16T03:38:21.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040409629999999998 2024-09-16T03:38:21.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897288+0000", 2024-09-16T03:38:21.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099581449000000002 2024-09-16T03:38:21.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196775+0000", 2024-09-16T03:38:21.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948696800000002 2024-09-16T03:38:21.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196814+0000", 2024-09-16T03:38:21.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8336e-05 2024-09-16T03:38:21.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196871+0000", 2024-09-16T03:38:21.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7120999999999997e-05 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:84 3.1 3:b24f289d:::e84532c28f467d2f2a6e2cf44904fa0d4a555e1304bbe1b25db645af42152bd7:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.873518873999998, 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.395455581, 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.739 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:38:21.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:38:21.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801613+0000", 2024-09-16T03:38:21.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9559999999999999e-06 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802117+0000", 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050401600000000003 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802141+0000", 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3618e-05 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802155+0000", 2024-09-16T03:38:21.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4358e-05 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802229+0000", 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3930999999999994e-05 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802355+0000", 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012527299999999999 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802567+0000", 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021250199999999999 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806218+0000", 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036512189999999998 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897323+0000", 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091104892000000007 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196959+0000", 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299635971 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197021+0000", 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1938000000000002e-05 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197065+0000", 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3906999999999998e-05 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.744 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.745 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.745 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.745 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:85 3.1 3:b380f269:::e65351e8a65a0b1a517e3b1a85e04b84c0ab7cfa6655f11fb0da31f56b4dd9af:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:21.745 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:38:21.745 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.872939977000001, 2024-09-16T03:38:21.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39502727300000001, 2024-09-16T03:38:21.745 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.745 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.745 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.745 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802190+0000", 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2419999999999999e-06 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802464+0000", 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027463699999999999 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802478+0000", 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3375e-05 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802484+0000", 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0990000000000004e-06 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803475+0000", 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00099113099999999991 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803651+0000", 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017654300000000001 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803923+0000", 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000271842 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806228+0000", 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023049590000000001 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897563+0000", 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091334724000000006 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197123+0000", 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299559992 2024-09-16T03:38:21.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197177+0000", 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4447999999999998e-05 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197216+0000", 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8281000000000002e-05 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.750 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:86 3.1 3:b3acb53e:::e951eb69ce5151a3f54107fa22c70f80729eff845da1613447a91536a24046e8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.872632882000001, 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.394851071, 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.751 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.752 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:38:21.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.752 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.752 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802496+0000", 2024-09-16T03:38:21.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:38:21.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.752 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.752 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802782+0000", 2024-09-16T03:38:21.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00028523600000000002 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802796+0000", 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4594e-05 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802804+0000", 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3790000000000006e-06 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804787+0000", 2024-09-16T03:38:21.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019825060000000002 2024-09-16T03:38:21.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804895+0000", 2024-09-16T03:38:21.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010813499999999999 2024-09-16T03:38:21.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.805046+0000", 2024-09-16T03:38:21.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150972 2024-09-16T03:38:21.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806681+0000", 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016353839999999999 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.005712+0000", 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19903093199999999 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197278+0000", 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.191565295 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197315+0000", 2024-09-16T03:38:21.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6946000000000002e-05 2024-09-16T03:38:21.756 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.756 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197347+0000", 2024-09-16T03:38:21.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1863000000000003e-05 2024-09-16T03:38:21.756 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.756 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:99 3.1 3:9f9f4ec5:::smithi15736696-22 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e71)", 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:56.682569+0000", 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.992559084, 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30291003799999999, 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682569+0000", 2024-09-16T03:38:21.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682569+0000", 2024-09-16T03:38:21.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682570+0000", 2024-09-16T03:38:21.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7000000000000004e-07 2024-09-16T03:38:21.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.759 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682574+0000", 2024-09-16T03:38:21.759 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3660000000000002e-06 2024-09-16T03:38:21.759 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.759 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682589+0000", 2024-09-16T03:38:21.759 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.49e-05 2024-09-16T03:38:21.759 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.759 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682596+0000", 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0589999999999997e-06 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.783627+0000", 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101030406 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.783680+0000", 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3562e-05 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.760 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.783780+0000", 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100028 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.884772+0000", 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10099123 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.885096+0000", 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00032439900000000001 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.985281+0000", 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100184936 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.985395+0000", 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114166 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.985479+0000", 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4215999999999995e-05 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.762 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2327 3.1 3:8eb88b02:::fa73f0ab3a7395fcfc19219899ff8c0a4a3c96caf8327f9da9656e2450435b27:head [write 0~838144] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:11.870769+0000", 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.8043594899999995, 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20583104499999999, 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.763 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2327 2024-09-16T03:38:21.764 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.764 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.764 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.764 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.870769+0000", 2024-09-16T03:38:21.764 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.764 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.764 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.764 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.870769+0000", 2024-09-16T03:38:21.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.765 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.765 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.870772+0000", 2024-09-16T03:38:21.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9749999999999999e-06 2024-09-16T03:38:21.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.765 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.765 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.872320+0000", 2024-09-16T03:38:21.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0015477049999999999 2024-09-16T03:38:21.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.766 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.766 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.872336+0000", 2024-09-16T03:38:21.766 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6957e-05 2024-09-16T03:38:21.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.766 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.766 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.872347+0000", 2024-09-16T03:38:21.766 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.064e-05 2024-09-16T03:38:21.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.766 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.972564+0000", 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100216702 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.972778+0000", 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021436 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.972892+0000", 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113863 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.767 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.768 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.768 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.974748+0000", 2024-09-16T03:38:21.768 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001856033 2024-09-16T03:38:21.768 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.768 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.768 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.976939+0000", 2024-09-16T03:38:21.768 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021909640000000001 2024-09-16T03:38:21.768 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.768 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.768 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.076519+0000", 2024-09-16T03:38:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099579728000000006 2024-09-16T03:38:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.076559+0000", 2024-09-16T03:38:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0213999999999998e-05 2024-09-16T03:38:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.076600+0000", 2024-09-16T03:38:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0904e-05 2024-09-16T03:38:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.769 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:130 3.1 3:8eb88b02:::fa73f0ab3a7395fcfc19219899ff8c0a4a3c96caf8327f9da9656e2450435b27:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:12.077208+0000", 2024-09-16T03:38:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.5979198760000006, 2024-09-16T03:38:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.22441346200000001, 2024-09-16T03:38:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.770 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.077208+0000", 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.077208+0000", 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.077211+0000", 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0959999999999999e-06 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.077220+0000", 2024-09-16T03:38:21.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.944e-06 2024-09-16T03:38:21.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.077233+0000", 2024-09-16T03:38:21.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.258e-05 2024-09-16T03:38:21.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.077239+0000", 2024-09-16T03:38:21.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3759999999999996e-06 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.177475+0000", 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100235857 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.177621+0000", 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014580699999999999 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.773 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.177915+0000", 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029362399999999998 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.178665+0000", 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00074989499999999999 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.178998+0000", 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00033294999999999999 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.301502+0000", 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.122504408 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.301573+0000", 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0958999999999996e-05 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:12.301622+0000", 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8965999999999998e-05 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.775 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.776 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.776 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.776 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:140 3.1 3:845e3b2c:::21d7b947791b34f7e3fcd39ad119c9f923ad2f3cc7a703501eac7a68189d5437:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:21.776 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:38:21.776 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.5604536280000003, 2024-09-16T03:38:21.776 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39914108999999998, 2024-09-16T03:38:21.776 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.776 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.776 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.776 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.776 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.776 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.776 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114677+0000", 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1610000000000001e-06 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.777 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.778 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114684+0000", 2024-09-16T03:38:21.778 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3150000000000003e-06 2024-09-16T03:38:21.778 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.778 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.778 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.778 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114699+0000", 2024-09-16T03:38:21.778 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4996e-05 2024-09-16T03:38:21.778 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.778 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.778 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.778 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114705+0000", 2024-09-16T03:38:21.778 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0580000000000003e-06 2024-09-16T03:38:21.778 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.214862+0000", 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015700199999999 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.214922+0000", 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0053999999999998e-05 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215023+0000", 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100432 2024-09-16T03:38:21.779 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.780 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.780 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.216100+0000", 2024-09-16T03:38:21.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010776819999999999 2024-09-16T03:38:21.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.780 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.780 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.499919+0000", 2024-09-16T03:38:21.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.28381845900000002 2024-09-16T03:38:21.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.780 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.780 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513755+0000", 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013836477 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513790+0000", 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4295999999999998e-05 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513816+0000", 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6157999999999999e-05 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.781 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.782 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:141 3.1 3:9387c678:::99a87d3b555b50e905a1ff55d99eb4aa025bd0070f07a4910d03cf1af95bbc44:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:21.782 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:38:21.782 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.5604029310000005, 2024-09-16T03:38:21.782 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50160030300000003, 2024-09-16T03:38:21.782 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.782 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.782 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.782 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.782 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:38:21.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.783 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.783 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.783 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:38:21.784 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.784 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.784 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.784 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.784 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114726+0000", 2024-09-16T03:38:21.784 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4399999999999999e-07 2024-09-16T03:38:21.785 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.785 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.785 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.785 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114730+0000", 2024-09-16T03:38:21.785 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1479999999999999e-06 2024-09-16T03:38:21.785 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.786 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114738+0000", 2024-09-16T03:38:21.786 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3200000000000002e-06 2024-09-16T03:38:21.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.787 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114743+0000", 2024-09-16T03:38:21.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.891e-06 2024-09-16T03:38:21.787 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.787 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.787 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215320+0000", 2024-09-16T03:38:21.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100576463 2024-09-16T03:38:21.787 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215376+0000", 2024-09-16T03:38:21.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6310999999999997e-05 2024-09-16T03:38:21.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.788 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215473+0000", 2024-09-16T03:38:21.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7103999999999997e-05 2024-09-16T03:38:21.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.216108+0000", 2024-09-16T03:38:21.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00063422800000000003 2024-09-16T03:38:21.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513851+0000", 2024-09-16T03:38:21.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29774376400000002 2024-09-16T03:38:21.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616086+0000", 2024-09-16T03:38:21.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10223441 2024-09-16T03:38:21.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616240+0000", 2024-09-16T03:38:21.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015453799999999999 2024-09-16T03:38:21.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.792 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616326+0000", 2024-09-16T03:38:21.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5382000000000003e-05 2024-09-16T03:38:21.792 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.792 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.792 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:142 3.1 3:a93b7b74:::70cd8f47898d416018aded7de9c29e0fe002b2336cb6cfabbf3eb6238766c3b1:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.5603735439999999, 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50166088799999997, 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3600000000000003e-07 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114762+0000", 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6039999999999998e-06 2024-09-16T03:38:21.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114769+0000", 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5939999999999999e-06 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114774+0000", 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2179999999999998e-06 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215687+0000", 2024-09-16T03:38:21.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100912974 2024-09-16T03:38:21.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.796 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215733+0000", 2024-09-16T03:38:21.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6419999999999999e-05 2024-09-16T03:38:21.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.796 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215836+0000", 2024-09-16T03:38:21.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102981 2024-09-16T03:38:21.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.796 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316766+0000", 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100929693 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513877+0000", 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19711067800000001 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616358+0000", 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10248180699999999 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616391+0000", 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2551000000000001e-05 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616416+0000", 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4732e-05 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.798 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:143 3.1 3:ac848b49:::adc4f0c8a4d0bc7e52f9d5c3b4981447737a7dbe58b8313a369b04383e10893f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.5603435379999997, 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60333394699999998, 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:38:21.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:38:21.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7899999999999999e-07 2024-09-16T03:38:21.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114788+0000", 2024-09-16T03:38:21.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4569999999999999e-06 2024-09-16T03:38:21.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114792+0000", 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5730000000000001e-06 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114796+0000", 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5449999999999999e-06 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316237+0000", 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20144099300000001 2024-09-16T03:38:21.801 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316417+0000", 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017981099999999999 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316564+0000", 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014693700000000001 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316993+0000", 2024-09-16T03:38:21.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042985499999999998 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.717965+0000", 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40097123699999998 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718005+0000", 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0083000000000002e-05 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.803 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718064+0000", 2024-09-16T03:38:21.804 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9020000000000001e-05 2024-09-16T03:38:21.804 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.804 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.804 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.804 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718119+0000", 2024-09-16T03:38:21.804 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4957000000000003e-05 2024-09-16T03:38:21.804 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.804 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.804 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.804 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.804 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.804 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:149 3.1 3:8eb88b02:::fa73f0ab3a7395fcfc19219899ff8c0a4a3c96caf8327f9da9656e2450435b27:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:21.804 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.5464348900000005, 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.348655887, 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.805 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128698+0000", 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.527e-06 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128714+0000", 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6019e-05 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.806 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128757+0000", 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3318000000000001e-05 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128764+0000", 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.28e-06 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.228916+0000", 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015252199999999 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.807 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.228976+0000", 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0390999999999998e-05 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.229077+0000", 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010053 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.274759+0000", 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.045682404000000003 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.808 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.809 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.809 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.330314+0000", 2024-09-16T03:38:21.809 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.055554916000000003 2024-09-16T03:38:21.809 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.809 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.809 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:21.809 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477196+0000", 2024-09-16T03:38:21.809 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14688130099999999 2024-09-16T03:38:21.809 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.810 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.810 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:21.810 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477291+0000", 2024-09-16T03:38:21.810 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5144999999999998e-05 2024-09-16T03:38:21.810 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.810 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.810 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:21.810 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477349+0000", 2024-09-16T03:38:21.810 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8533999999999999e-05 2024-09-16T03:38:21.810 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.810 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.811 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.811 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.811 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.811 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:21.812 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:38:21.813 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:21.811+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:21.872 INFO:tasks.rados.rados.0.smithi157.stdout:1239: set_chunk oid 27 offset: 1154560 length: 555520 target oid 95c05e941a661e985e96de0326206dc0f5fa565eed10a59dc70cc3ed43a226b3 offset: 0 snap -1 2024-09-16T03:38:21.873 INFO:tasks.rados.rados.0.smithi157.stdout:1240: tier_promote oid 34 2024-09-16T03:38:21.873 INFO:tasks.rados.rados.0.smithi157.stdout:1231: finishing write tid 1 to smithi15736696-23 2024-09-16T03:38:21.873 INFO:tasks.rados.rados.0.smithi157.stdout:1231: finishing write tid 2 to smithi15736696-23 2024-09-16T03:38:21.873 INFO:tasks.rados.rados.0.smithi157.stdout:1231: finishing write tid 3 to smithi15736696-23 2024-09-16T03:38:21.873 INFO:tasks.rados.rados.0.smithi157.stdout:1232: finishing write tid 1 to smithi15736696-11 2024-09-16T03:38:21.874 INFO:tasks.rados.rados.0.smithi157.stdout:1232: finishing write tid 2 to smithi15736696-11 2024-09-16T03:38:21.874 INFO:tasks.rados.rados.0.smithi157.stdout:1232: finishing write tid 3 to smithi15736696-11 2024-09-16T03:38:21.874 INFO:tasks.rados.rados.0.smithi157.stdout:1241: read oid 8 snap -1 2024-09-16T03:38:21.874 INFO:tasks.rados.rados.0.smithi157.stdout:1241: expect deleted 2024-09-16T03:38:21.874 INFO:tasks.rados.rados.0.smithi157.stdout:1242: read oid 12 snap -1 2024-09-16T03:38:21.874 INFO:tasks.rados.rados.0.smithi157.stdout:1242: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:21.875 INFO:tasks.rados.rados.0.smithi157.stdout:1243: tier_promote oid 29 2024-09-16T03:38:21.875 INFO:tasks.rados.rados.0.smithi157.stdout:1231: finishing write tid 4 to smithi15736696-23 2024-09-16T03:38:21.875 INFO:tasks.rados.rados.0.smithi157.stdout:1231: finishing write tid 5 to smithi15736696-23 2024-09-16T03:38:21.875 INFO:tasks.rados.rados.0.smithi157.stdout:1231: oid 23 updating version 0 to 60 2024-09-16T03:38:21.875 INFO:tasks.rados.rados.0.smithi157.stdout:1231: oid 23 version 60 is already newer than 59 2024-09-16T03:38:21.875 INFO:tasks.rados.rados.0.smithi157.stdout:1231: oid 23 updating version 60 to 61 2024-09-16T03:38:21.876 INFO:tasks.rados.rados.0.smithi157.stdout:1231: oid 23 version 61 is already newer than 58 2024-09-16T03:38:21.876 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 61 (ObjNum 232 snap 0 seq_num 232) dirty exists 2024-09-16T03:38:21.876 INFO:tasks.rados.rados.0.smithi157.stdout:1231: left oid 23 (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:38:21.876 INFO:tasks.rados.rados.0.smithi157.stdout:1235: expect (ObjNum 217 snap 0 seq_num 217) 2024-09-16T03:38:21.876 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:21.913 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:21.912+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:21.918 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:21.918 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:21.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.919 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2688 4.f 4.8e328b0f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:21.919 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.869498+0000", 2024-09-16T03:38:21.919 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.047628126999999999, 2024-09-16T03:38:21.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.047646766, 2024-09-16T03:38:21.920 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.920 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.920 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:21.920 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.920 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.921 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.921 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2688 2024-09-16T03:38:21.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.921 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869498+0000", 2024-09-16T03:38:21.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869498+0000", 2024-09-16T03:38:21.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869499+0000", 2024-09-16T03:38:21.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3939999999999999e-06 2024-09-16T03:38:21.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869505+0000", 2024-09-16T03:38:21.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2570000000000004e-06 2024-09-16T03:38:21.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869514+0000", 2024-09-16T03:38:21.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1470000000000007e-06 2024-09-16T03:38:21.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869522+0000", 2024-09-16T03:38:21.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7379999999999994e-06 2024-09-16T03:38:21.927 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.927 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.927 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.927 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2689 4.f 4.8e328b0f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:21.928 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.869534+0000", 2024-09-16T03:38:21.928 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.047591363999999997, 2024-09-16T03:38:21.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.047639984000000003, 2024-09-16T03:38:21.928 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.928 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.929 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:21.929 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.929 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.929 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.929 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2689 2024-09-16T03:38:21.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.930 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869534+0000", 2024-09-16T03:38:21.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.931 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.931 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869534+0000", 2024-09-16T03:38:21.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869535+0000", 2024-09-16T03:38:21.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3000000000000001e-07 2024-09-16T03:38:21.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869537+0000", 2024-09-16T03:38:21.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4789999999999999e-06 2024-09-16T03:38:21.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869540+0000", 2024-09-16T03:38:21.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.27e-06 2024-09-16T03:38:21.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869543+0000", 2024-09-16T03:38:21.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8959999999999999e-06 2024-09-16T03:38:21.935 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.935 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.935 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.936 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2690 4.f 4.8e328b0f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:21.936 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.869580+0000", 2024-09-16T03:38:21.936 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.047545441000000001, 2024-09-16T03:38:21.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.047617171, 2024-09-16T03:38:21.937 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.937 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.937 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:21.938 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.938 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.938 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.938 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2690 2024-09-16T03:38:21.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.939 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.939 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869580+0000", 2024-09-16T03:38:21.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869580+0000", 2024-09-16T03:38:21.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869581+0000", 2024-09-16T03:38:21.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4300000000000001e-07 2024-09-16T03:38:21.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869586+0000", 2024-09-16T03:38:21.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1680000000000003e-06 2024-09-16T03:38:21.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869595+0000", 2024-09-16T03:38:21.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5520000000000005e-06 2024-09-16T03:38:21.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.944 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869600+0000", 2024-09-16T03:38:21.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2410000000000003e-06 2024-09-16T03:38:21.944 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.944 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.944 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.945 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.8.0:103 4.a e91/32)", 2024-09-16T03:38:21.945 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.875875+0000", 2024-09-16T03:38:21.945 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.041250340000000003, 2024-09-16T03:38:21.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.041344400000000003, 2024-09-16T03:38:21.945 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.946 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.946 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "no flag points reached", 2024-09-16T03:38:21.946 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.875875+0000", 2024-09-16T03:38:21.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.875875+0000", 2024-09-16T03:38:21.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.875876+0000", 2024-09-16T03:38:21.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9299999999999999e-07 2024-09-16T03:38:21.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.879619+0000", 2024-09-16T03:38:21.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037430520000000002 2024-09-16T03:38:21.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.879630+0000", 2024-09-16T03:38:21.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0995000000000001e-05 2024-09-16T03:38:21.950 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.951 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.951 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.951 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2683 4.0 4:06b6d050:::smithi15736696-50:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:21.951 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.451040+0000", 2024-09-16T03:38:21.952 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.46608564000000002, 2024-09-16T03:38:21.952 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.46620160300000002, 2024-09-16T03:38:21.952 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:21.952 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:21.952 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:21.953 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:21.953 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:21.953 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:21.953 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2683 2024-09-16T03:38:21.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.953 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:21.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:21.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.451040+0000", 2024-09-16T03:38:21.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:21.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.451040+0000", 2024-09-16T03:38:21.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:21.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:21.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.451043+0000", 2024-09-16T03:38:21.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6539999999999999e-06 2024-09-16T03:38:21.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:21.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.451049+0000", 2024-09-16T03:38:21.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6510000000000004e-06 2024-09-16T03:38:21.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:21.958 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.451065+0000", 2024-09-16T03:38:21.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5758000000000001e-05 2024-09-16T03:38:21.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.958 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.958 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:21.959 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.451076+0000", 2024-09-16T03:38:21.959 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0971999999999999e-05 2024-09-16T03:38:21.959 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.959 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.959 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.959 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.551212+0000", 2024-09-16T03:38:21.960 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10013617 2024-09-16T03:38:21.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-09-16T03:38:21.960 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.551216+0000", 2024-09-16T03:38:21.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6799999999999999e-06 2024-09-16T03:38:21.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:21.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.551390+0000", 2024-09-16T03:38:21.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017363400000000001 2024-09-16T03:38:21.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:21.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:21.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:21.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.551472+0000", 2024-09-16T03:38:21.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2254000000000002e-05 2024-09-16T03:38:21.962 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.963 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:21.963 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.963 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:21.963 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2024-09-16T03:38:21.963 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 5 2024-09-16T03:38:21.963 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:21.965 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:38:22.014 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:22.012+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:22.069 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:22.069 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:22.070 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:22.070 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:22.070 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:22.078 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:38:22.115 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:22.113+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:22.145 INFO:tasks.rados.rados.0.smithi157.stdout:1236: finishing write tid 2 to smithi15736696-15 2024-09-16T03:38:22.145 INFO:tasks.rados.rados.0.smithi157.stdout:1236: finishing write tid 3 to smithi15736696-15 2024-09-16T03:38:22.145 INFO:tasks.rados.rados.0.smithi157.stdout:1236: finishing write tid 4 to smithi15736696-15 2024-09-16T03:38:22.145 INFO:tasks.rados.rados.0.smithi157.stdout:1236: finishing write tid 5 to smithi15736696-15 2024-09-16T03:38:22.145 INFO:tasks.rados.rados.0.smithi157.stdout:1236: oid 15 updating version 0 to 12 2024-09-16T03:38:22.145 INFO:tasks.rados.rados.0.smithi157.stdout:1236: oid 15 version 12 is already newer than 11 2024-09-16T03:38:22.145 INFO:tasks.rados.rados.0.smithi157.stdout:1236: oid 15 version 12 is already newer than 10 2024-09-16T03:38:22.145 INFO:tasks.rados.rados.0.smithi157.stdout:1236: oid 15 updating version 12 to 13 2024-09-16T03:38:22.145 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 13 (ObjNum 235 snap 0 seq_num 235) dirty exists 2024-09-16T03:38:22.145 INFO:tasks.rados.rados.0.smithi157.stdout:1236: left oid 15 (ObjNum 235 snap 0 seq_num 235) 2024-09-16T03:38:22.145 INFO:tasks.rados.rados.0.smithi157.stdout:1232: finishing write tid 4 to smithi15736696-11 2024-09-16T03:38:22.145 INFO:tasks.rados.rados.0.smithi157.stdout:1231: done (11 left) 2024-09-16T03:38:22.145 INFO:tasks.rados.rados.0.smithi157.stdout:1235: done (10 left) 2024-09-16T03:38:22.146 INFO:tasks.rados.rados.0.smithi157.stdout:1236: done (9 left) 2024-09-16T03:38:22.146 INFO:tasks.rados.rados.0.smithi157.stdout:1244: write (excl) oid 46 current snap is 0 2024-09-16T03:38:22.146 INFO:tasks.rados.rados.0.smithi157.stdout:1233: finishing write tid 1 to smithi15736696-45 2024-09-16T03:38:22.146 INFO:tasks.rados.rados.0.smithi157.stdout:1233: finishing write tid 2 to smithi15736696-45 2024-09-16T03:38:22.146 INFO:tasks.rados.rados.0.smithi157.stdout:1233: finishing write tid 3 to smithi15736696-45 2024-09-16T03:38:22.146 INFO:tasks.rados.rados.0.smithi157.stdout:1244: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2024-09-16T03:38:22.151 INFO:tasks.rados.rados.0.smithi157.stdout:1244: writing smithi15736696-46 from 766452 to 1332159 tid 1 2024-09-16T03:38:22.157 INFO:tasks.rados.rados.0.smithi157.stdout:1244: writing smithi15736696-46 from 1860221 to 2329633 tid 2 2024-09-16T03:38:22.164 INFO:tasks.rados.rados.0.smithi157.stdout:1244: writing smithi15736696-46 from 2981238 to 3575433 tid 3 2024-09-16T03:38:22.164 INFO:tasks.rados.rados.0.smithi157.stdout:1245: delete oid 6 current snap is 0 2024-09-16T03:38:22.164 INFO:tasks.rados.rados.0.smithi157.stdout:1237: expect (ObjNum 219 snap 0 seq_num 219) 2024-09-16T03:38:22.164 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:22.184 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:22.184 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:22.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:22.184 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:22.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.185 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:66 4.0 4:023abada:::smithi15736696-17:head [write 1903006~365906] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:22.185 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.200996+0000", 2024-09-16T03:38:22.185 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.979787071999993, 2024-09-16T03:38:22.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50798839299999998, 2024-09-16T03:38:22.185 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.185 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.185 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.185 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.186 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.186 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.186 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 66 2024-09-16T03:38:22.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.186 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.200996+0000", 2024-09-16T03:38:22.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.186 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.200996+0000", 2024-09-16T03:38:22.187 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.187 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.187 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.187 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.187 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.200998+0000", 2024-09-16T03:38:22.187 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5179999999999999e-06 2024-09-16T03:38:22.187 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.187 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.187 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.187 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.201330+0000", 2024-09-16T03:38:22.187 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000332672 2024-09-16T03:38:22.188 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.188 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.188 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.188 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.201340+0000", 2024-09-16T03:38:22.188 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8099999999999992e-06 2024-09-16T03:38:22.188 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.188 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.188 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.188 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301446+0000", 2024-09-16T03:38:22.188 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100105877 2024-09-16T03:38:22.188 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.188 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.188 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301543+0000", 2024-09-16T03:38:22.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7058000000000005e-05 2024-09-16T03:38:22.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.189 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301628+0000", 2024-09-16T03:38:22.189 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5078000000000001e-05 2024-09-16T03:38:22.189 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.189 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.189 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:22.189 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301770+0000", 2024-09-16T03:38:22.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014212400000000001 2024-09-16T03:38:22.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:22.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.303247+0000", 2024-09-16T03:38:22.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001476595 2024-09-16T03:38:22.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.190 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.190 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.190 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.606126+0000", 2024-09-16T03:38:22.190 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30287937799999998 2024-09-16T03:38:22.190 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.191 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.708902+0000", 2024-09-16T03:38:22.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102775402 2024-09-16T03:38:22.191 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.191 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.191 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.191 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.708938+0000", 2024-09-16T03:38:22.191 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5947e-05 2024-09-16T03:38:22.192 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.192 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.192 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.192 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.708984+0000", 2024-09-16T03:38:22.192 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6934000000000002e-05 2024-09-16T03:38:22.192 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.192 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.192 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.192 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.192 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.192 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:69 4.0 4:0a55bbf2:::smithi15736696-18:head [write 685719~711939,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:22.192 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.301546+0000", 2024-09-16T03:38:22.193 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.879237149000005, 2024-09-16T03:38:22.193 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50794553600000003, 2024-09-16T03:38:22.193 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.193 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.193 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.193 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.193 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.193 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.193 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 69 2024-09-16T03:38:22.193 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.193 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.193 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.193 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301546+0000", 2024-09-16T03:38:22.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301546+0000", 2024-09-16T03:38:22.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.194 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.194 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.194 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.301547+0000", 2024-09-16T03:38:22.194 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4799999999999997e-07 2024-09-16T03:38:22.194 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302372+0000", 2024-09-16T03:38:22.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000824828 2024-09-16T03:38:22.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.195 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302381+0000", 2024-09-16T03:38:22.195 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1759999999999999e-06 2024-09-16T03:38:22.195 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.195 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.195 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.302389+0000", 2024-09-16T03:38:22.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4710000000000007e-06 2024-09-16T03:38:22.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.401577+0000", 2024-09-16T03:38:22.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099188002999999997 2024-09-16T03:38:22.196 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.196 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.196 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.196 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.401681+0000", 2024-09-16T03:38:22.196 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000103618 2024-09-16T03:38:22.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.197 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:22.197 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.401797+0000", 2024-09-16T03:38:22.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116177 2024-09-16T03:38:22.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.197 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:22.197 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.403396+0000", 2024-09-16T03:38:22.197 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001598857 2024-09-16T03:38:22.197 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.197 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.606185+0000", 2024-09-16T03:38:22.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202788996 2024-09-16T03:38:22.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.198 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.198 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.809397+0000", 2024-09-16T03:38:22.198 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20321167700000001 2024-09-16T03:38:22.198 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.198 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.199 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.809449+0000", 2024-09-16T03:38:22.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2389000000000003e-05 2024-09-16T03:38:22.199 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.199 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.199 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.199 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.809492+0000", 2024-09-16T03:38:22.199 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2496000000000001e-05 2024-09-16T03:38:22.199 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.199 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.200 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.200 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.200 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.200 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:416 4.0 4:023abada:::smithi15736696-17:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:22.200 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:38:22.200 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 61.871541424999997, 2024-09-16T03:38:22.200 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60401983699999995, 2024-09-16T03:38:22.200 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.201 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.201 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.201 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.201 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.201 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.201 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 416 2024-09-16T03:38:22.201 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.201 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.201 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.201 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:38:22.201 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.202 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.202 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.202 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:38:22.202 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.202 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.202 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.202 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309243+0000", 2024-09-16T03:38:22.202 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.178e-06 2024-09-16T03:38:22.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.202 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309248+0000", 2024-09-16T03:38:22.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:38:22.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309257+0000", 2024-09-16T03:38:22.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4720000000000001e-06 2024-09-16T03:38:22.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309262+0000", 2024-09-16T03:38:22.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0839999999999999e-06 2024-09-16T03:38:22.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.204 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.204 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309297+0000", 2024-09-16T03:38:22.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4997999999999999e-05 2024-09-16T03:38:22.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.204 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.204 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309367+0000", 2024-09-16T03:38:22.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0124999999999997e-05 2024-09-16T03:38:22.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:22.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812055+0000", 2024-09-16T03:38:22.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50268796900000001 2024-09-16T03:38:22.205 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:22.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812614+0000", 2024-09-16T03:38:22.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000558309 2024-09-16T03:38:22.205 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.813040+0000", 2024-09-16T03:38:22.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042652800000000002 2024-09-16T03:38:22.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913189+0000", 2024-09-16T03:38:22.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10014880399999999 2024-09-16T03:38:22.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913220+0000", 2024-09-16T03:38:22.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1353000000000003e-05 2024-09-16T03:38:22.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.207 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913262+0000", 2024-09-16T03:38:22.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1284e-05 2024-09-16T03:38:22.207 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.207 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.207 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.207 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.207 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:494 4.f 4:f817def9:::smithi15736696-45:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:22.207 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:38:22.207 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 60.458601690000002, 2024-09-16T03:38:22.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60451760399999999, 2024-09-16T03:38:22.208 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.208 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.208 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.208 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.208 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.208 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.208 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 494 2024-09-16T03:38:22.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.208 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.208 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:38:22.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.209 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.209 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:38:22.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.209 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.209 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722183+0000", 2024-09-16T03:38:22.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:38:22.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.210 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722189+0000", 2024-09-16T03:38:22.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9429999999999997e-06 2024-09-16T03:38:22.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.210 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722203+0000", 2024-09-16T03:38:22.211 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4367e-05 2024-09-16T03:38:22.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.211 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722210+0000", 2024-09-16T03:38:22.211 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:38:22.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.211 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822367+0000", 2024-09-16T03:38:22.211 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100156933 2024-09-16T03:38:22.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.212 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.212 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822461+0000", 2024-09-16T03:38:22.212 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3216999999999999e-05 2024-09-16T03:38:22.212 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.212 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.212 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:38:22.212 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125593+0000", 2024-09-16T03:38:22.212 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30313253699999998 2024-09-16T03:38:22.212 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.212 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.212 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:22.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126307+0000", 2024-09-16T03:38:22.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071350000000000005 2024-09-16T03:38:22.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.213 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126825+0000", 2024-09-16T03:38:22.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051846100000000001 2024-09-16T03:38:22.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.213 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326612+0000", 2024-09-16T03:38:22.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19978721999999999 2024-09-16T03:38:22.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.214 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.214 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326663+0000", 2024-09-16T03:38:22.214 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0460000000000001e-05 2024-09-16T03:38:22.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.214 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.214 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326699+0000", 2024-09-16T03:38:22.214 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6340000000000001e-05 2024-09-16T03:38:22.214 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.214 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.215 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.215 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.215 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.215 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:784 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=63b,truncate 2840350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:22.215 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:38:22.215 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 56.374193814000002, 2024-09-16T03:38:22.215 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63358405200000001, 2024-09-16T03:38:22.215 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.215 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.215 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.215 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.215 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.215 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.216 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 784 2024-09-16T03:38:22.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.216 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.216 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.216 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.216 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:38:22.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.216 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.216 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.216 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:38:22.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806590+0000", 2024-09-16T03:38:22.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6000000000000003e-07 2024-09-16T03:38:22.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806593+0000", 2024-09-16T03:38:22.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6649999999999999e-06 2024-09-16T03:38:22.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806597+0000", 2024-09-16T03:38:22.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.245e-06 2024-09-16T03:38:22.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806600+0000", 2024-09-16T03:38:22.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.23e-06 2024-09-16T03:38:22.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099381+0000", 2024-09-16T03:38:22.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29278119400000002 2024-09-16T03:38:22.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.219 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.219 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.219 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099438+0000", 2024-09-16T03:38:22.219 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6198e-05 2024-09-16T03:38:22.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.219 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.219 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:22.219 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099527+0000", 2024-09-16T03:38:22.219 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9505000000000004e-05 2024-09-16T03:38:22.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.220 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.220 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:22.220 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200775+0000", 2024-09-16T03:38:22.220 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101247904 2024-09-16T03:38:22.220 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.220 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.220 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.220 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200889+0000", 2024-09-16T03:38:22.220 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114301 2024-09-16T03:38:22.220 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.220 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.221 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.221 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440066+0000", 2024-09-16T03:38:22.221 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.239176367 2024-09-16T03:38:22.221 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.221 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.221 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.221 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440109+0000", 2024-09-16T03:38:22.221 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2941000000000001e-05 2024-09-16T03:38:22.221 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.221 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.221 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.222 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440173+0000", 2024-09-16T03:38:22.222 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4842e-05 2024-09-16T03:38:22.222 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.222 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.222 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.222 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.223 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.223 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:785 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:22.223 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:38:22.223 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 56.374177889000002, 2024-09-16T03:38:22.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63376373200000002, 2024-09-16T03:38:22.223 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.223 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.223 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:22.223 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.223 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.224 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.224 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 785 2024-09-16T03:38:22.224 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.224 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.224 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.224 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.224 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:38:22.224 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.224 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.224 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.224 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.224 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:38:22.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.225 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.225 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.225 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806606+0000", 2024-09-16T03:38:22.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:38:22.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.225 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.225 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.225 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806607+0000", 2024-09-16T03:38:22.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.424e-06 2024-09-16T03:38:22.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.226 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.226 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.226 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806611+0000", 2024-09-16T03:38:22.226 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1080000000000003e-06 2024-09-16T03:38:22.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.226 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.226 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.226 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806614+0000", 2024-09-16T03:38:22.226 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.067e-06 2024-09-16T03:38:22.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.227 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099813+0000", 2024-09-16T03:38:22.227 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29319853699999998 2024-09-16T03:38:22.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:22.227 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099858+0000", 2024-09-16T03:38:22.227 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5667999999999997e-05 2024-09-16T03:38:22.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.228 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440236+0000", 2024-09-16T03:38:22.228 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34037791499999998 2024-09-16T03:38:22.228 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.228 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.228 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.228 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440297+0000", 2024-09-16T03:38:22.228 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0988e-05 2024-09-16T03:38:22.228 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.228 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.228 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.228 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440369+0000", 2024-09-16T03:38:22.229 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1682999999999998e-05 2024-09-16T03:38:22.229 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.229 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.229 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.229 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.229 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.229 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:801 4.0 4:07ad0560:::smithi15736696-25:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:22.229 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:38:22.229 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 56.174699990999997, 2024-09-16T03:38:22.229 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.534510445, 2024-09-16T03:38:22.229 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.230 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.230 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:22.230 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.230 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.230 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.230 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 801 2024-09-16T03:38:22.230 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.230 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.230 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.230 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.230 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:38:22.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.230 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.231 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:38:22.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.231 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.231 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006084+0000", 2024-09-16T03:38:22.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.322e-06 2024-09-16T03:38:22.231 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006088+0000", 2024-09-16T03:38:22.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000001e-06 2024-09-16T03:38:22.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.232 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006098+0000", 2024-09-16T03:38:22.232 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0186999999999999e-05 2024-09-16T03:38:22.232 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.232 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.232 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006104+0000", 2024-09-16T03:38:22.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.271e-06 2024-09-16T03:38:22.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.233 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.233 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.199952+0000", 2024-09-16T03:38:22.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.193847935 2024-09-16T03:38:22.233 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.233 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.233 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.233 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200027+0000", 2024-09-16T03:38:22.233 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4987000000000006e-05 2024-09-16T03:38:22.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.234 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.234 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.540594+0000", 2024-09-16T03:38:22.234 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34056708299999999 2024-09-16T03:38:22.234 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.234 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.234 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.234 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.234 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.234 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1023 4.0 4:06b6d050:::smithi15736696-50:head [write 2246563~579356] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:22.234 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:38:22.234 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.869550920999998, 2024-09-16T03:38:22.235 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60727534699999997, 2024-09-16T03:38:22.235 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.235 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.235 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.235 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.235 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.235 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.235 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1023 2024-09-16T03:38:22.235 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.235 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.235 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.235 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.235 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:38:22.236 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.236 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.236 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.236 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.236 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:38:22.236 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.236 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.236 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.236 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.236 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311233+0000", 2024-09-16T03:38:22.236 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1039999999999999e-06 2024-09-16T03:38:22.237 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.237 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.237 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.237 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311692+0000", 2024-09-16T03:38:22.237 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045908800000000002 2024-09-16T03:38:22.237 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.237 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.237 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.237 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311704+0000", 2024-09-16T03:38:22.237 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1844e-05 2024-09-16T03:38:22.237 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.237 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.238 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.238 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311713+0000", 2024-09-16T03:38:22.238 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2500000000000006e-06 2024-09-16T03:38:22.238 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.238 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.238 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.238 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359204+0000", 2024-09-16T03:38:22.238 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.047491646999999998 2024-09-16T03:38:22.238 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.238 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.238 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.239 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359258+0000", 2024-09-16T03:38:22.239 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4082000000000002e-05 2024-09-16T03:38:22.239 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.239 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.239 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:22.239 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359354+0000", 2024-09-16T03:38:22.239 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5686999999999994e-05 2024-09-16T03:38:22.239 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.239 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.239 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.239 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512245+0000", 2024-09-16T03:38:22.239 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15289135700000001 2024-09-16T03:38:22.240 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.240 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.240 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:22.240 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512956+0000", 2024-09-16T03:38:22.240 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071110300000000004 2024-09-16T03:38:22.240 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.240 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.240 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.240 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918361+0000", 2024-09-16T03:38:22.240 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40540489400000002 2024-09-16T03:38:22.240 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.241 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.241 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.241 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918455+0000", 2024-09-16T03:38:22.241 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3412999999999995e-05 2024-09-16T03:38:22.241 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.241 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.241 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.241 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918508+0000", 2024-09-16T03:38:22.241 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:38:22.241 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.241 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.242 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.242 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.242 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.242 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1024 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3176490] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:22.242 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:38:22.242 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.869057939999998, 2024-09-16T03:38:22.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60701227000000002, 2024-09-16T03:38:22.242 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.242 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.242 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.242 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.243 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.243 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.243 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1024 2024-09-16T03:38:22.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.243 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:38:22.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:38:22.244 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.244 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311726+0000", 2024-09-16T03:38:22.244 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.02e-07 2024-09-16T03:38:22.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.244 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311730+0000", 2024-09-16T03:38:22.244 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1099999999999996e-06 2024-09-16T03:38:22.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.245 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.245 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311735+0000", 2024-09-16T03:38:22.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4680000000000003e-06 2024-09-16T03:38:22.245 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.245 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.245 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311739+0000", 2024-09-16T03:38:22.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9380000000000002e-06 2024-09-16T03:38:22.248 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.248 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.248 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.248 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361468+0000", 2024-09-16T03:38:22.248 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.049729450000000001 2024-09-16T03:38:22.248 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.248 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.248 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.248 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361539+0000", 2024-09-16T03:38:22.248 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0896999999999996e-05 2024-09-16T03:38:22.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.249 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:22.249 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361661+0000", 2024-09-16T03:38:22.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000122404 2024-09-16T03:38:22.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.249 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.249 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512943+0000", 2024-09-16T03:38:22.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.151281481 2024-09-16T03:38:22.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:22.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512959+0000", 2024-09-16T03:38:22.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:38:22.250 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.250 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918588+0000", 2024-09-16T03:38:22.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40562815499999999 2024-09-16T03:38:22.250 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.250 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918646+0000", 2024-09-16T03:38:22.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8566999999999997e-05 2024-09-16T03:38:22.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.251 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.251 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918737+0000", 2024-09-16T03:38:22.251 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1308000000000001e-05 2024-09-16T03:38:22.251 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.251 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.251 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.251 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1025 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:22.251 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:22.252 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.869039442000002, 2024-09-16T03:38:22.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60723649599999996, 2024-09-16T03:38:22.252 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.252 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.252 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:22.252 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.252 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.252 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.252 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1025 2024-09-16T03:38:22.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.252 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.252 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.253 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:22.253 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.253 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:22.253 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:22.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9400000000000001e-07 2024-09-16T03:38:22.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.254 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.254 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311746+0000", 2024-09-16T03:38:22.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2670000000000001e-06 2024-09-16T03:38:22.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.254 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.254 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311752+0000", 2024-09-16T03:38:22.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6389999999999997e-06 2024-09-16T03:38:22.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.255 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311757+0000", 2024-09-16T03:38:22.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3639999999999998e-06 2024-09-16T03:38:22.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.255 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362018+0000", 2024-09-16T03:38:22.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.050260969000000003 2024-09-16T03:38:22.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.256 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.256 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:22.256 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362089+0000", 2024-09-16T03:38:22.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0735000000000006e-05 2024-09-16T03:38:22.256 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.256 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.256 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.256 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918781+0000", 2024-09-16T03:38:22.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55669215400000005 2024-09-16T03:38:22.256 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.256 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.256 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.256 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918867+0000", 2024-09-16T03:38:22.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6019000000000004e-05 2024-09-16T03:38:22.257 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918980+0000", 2024-09-16T03:38:22.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011295500000000001 2024-09-16T03:38:22.257 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.257 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.257 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.257 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.257 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1054 4.10 4:0a55bbf2:::smithi15736696-18:head [write 2633066~610652] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:22.258 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:38:22.258 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.667546234, 2024-09-16T03:38:22.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51115411600000005, 2024-09-16T03:38:22.258 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.258 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.258 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.258 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.258 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.258 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.258 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1054 2024-09-16T03:38:22.259 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.259 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.259 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.259 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:38:22.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.259 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.259 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.259 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:38:22.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.260 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.260 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.260 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.260 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513238+0000", 2024-09-16T03:38:22.260 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1999999999999998e-07 2024-09-16T03:38:22.260 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.260 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.260 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.260 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513695+0000", 2024-09-16T03:38:22.260 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045695099999999998 2024-09-16T03:38:22.261 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513720+0000", 2024-09-16T03:38:22.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.567e-05 2024-09-16T03:38:22.261 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513725+0000", 2024-09-16T03:38:22.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9899999999999997e-06 2024-09-16T03:38:22.261 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613892+0000", 2024-09-16T03:38:22.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166253 2024-09-16T03:38:22.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:38:22.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613898+0000", 2024-09-16T03:38:22.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8710000000000002e-06 2024-09-16T03:38:22.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916870+0000", 2024-09-16T03:38:22.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30297253800000001 2024-09-16T03:38:22.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.263 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.263 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916934+0000", 2024-09-16T03:38:22.263 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3723e-05 2024-09-16T03:38:22.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.263 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:22.263 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.917033+0000", 2024-09-16T03:38:22.263 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8908999999999998e-05 2024-09-16T03:38:22.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.264 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:22.264 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921278+0000", 2024-09-16T03:38:22.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0042456999999999998 2024-09-16T03:38:22.264 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.264 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.264 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.264 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019617+0000", 2024-09-16T03:38:22.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098338910000000002 2024-09-16T03:38:22.264 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.264 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.264 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.264 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024269+0000", 2024-09-16T03:38:22.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046515519999999998 2024-09-16T03:38:22.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024358+0000", 2024-09-16T03:38:22.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9096000000000002e-05 2024-09-16T03:38:22.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024391+0000", 2024-09-16T03:38:22.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3033e-05 2024-09-16T03:38:22.266 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.266 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.266 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.266 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1055 4.10 4:0a55bbf2:::smithi15736696-18:head [setxattr _header (56) in=7b,truncate 3243718] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:22.266 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:38:22.266 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.667046734000003, 2024-09-16T03:38:22.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61098759599999997, 2024-09-16T03:38:22.266 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.266 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.266 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.267 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.267 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.267 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.267 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1055 2024-09-16T03:38:22.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.267 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:38:22.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:38:22.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513737+0000", 2024-09-16T03:38:22.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9399999999999993e-07 2024-09-16T03:38:22.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.269 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513759+0000", 2024-09-16T03:38:22.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1903999999999999e-05 2024-09-16T03:38:22.269 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.269 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513773+0000", 2024-09-16T03:38:22.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3244000000000001e-05 2024-09-16T03:38:22.269 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.269 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513780+0000", 2024-09-16T03:38:22.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999999e-06 2024-09-16T03:38:22.270 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.270 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.270 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714016+0000", 2024-09-16T03:38:22.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20023635000000001 2024-09-16T03:38:22.270 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.270 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:38:22.270 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714024+0000", 2024-09-16T03:38:22.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8359999999999992e-06 2024-09-16T03:38:22.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.271 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918736+0000", 2024-09-16T03:38:22.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471236700000001 2024-09-16T03:38:22.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.271 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918830+0000", 2024-09-16T03:38:22.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3769000000000003e-05 2024-09-16T03:38:22.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:22.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919003+0000", 2024-09-16T03:38:22.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000173198 2024-09-16T03:38:22.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:22.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921589+0000", 2024-09-16T03:38:22.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002586419 2024-09-16T03:38:22.272 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.273 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019655+0000", 2024-09-16T03:38:22.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098065801999999994 2024-09-16T03:38:22.273 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.273 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.273 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124538+0000", 2024-09-16T03:38:22.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10488272899999999 2024-09-16T03:38:22.273 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.273 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.273 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124635+0000", 2024-09-16T03:38:22.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7125000000000003e-05 2024-09-16T03:38:22.274 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.274 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.274 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.274 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124724+0000", 2024-09-16T03:38:22.274 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8858999999999997e-05 2024-09-16T03:38:22.274 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.274 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.274 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.274 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.274 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.275 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1056 4.10 4:0a55bbf2:::smithi15736696-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:22.275 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:22.275 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.666994162999998, 2024-09-16T03:38:22.275 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61111632800000004, 2024-09-16T03:38:22.275 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.275 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.275 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:22.275 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.275 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.275 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.275 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1056 2024-09-16T03:38:22.275 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.276 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:22.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:22.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.277 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.277 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:22.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2499999999999995e-07 2024-09-16T03:38:22.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.277 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.277 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513792+0000", 2024-09-16T03:38:22.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:38:22.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.277 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513797+0000", 2024-09-16T03:38:22.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4940000000000003e-06 2024-09-16T03:38:22.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.278 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513802+0000", 2024-09-16T03:38:22.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5390000000000001e-06 2024-09-16T03:38:22.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.278 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919221+0000", 2024-09-16T03:38:22.279 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40541913899999998 2024-09-16T03:38:22.279 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.279 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.279 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:22.279 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919268+0000", 2024-09-16T03:38:22.279 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7027000000000002e-05 2024-09-16T03:38:22.279 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.279 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.279 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.279 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124766+0000", 2024-09-16T03:38:22.279 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20549842099999999 2024-09-16T03:38:22.279 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.280 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.280 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.280 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124828+0000", 2024-09-16T03:38:22.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1803000000000004e-05 2024-09-16T03:38:22.280 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.280 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.280 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.280 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124905+0000", 2024-09-16T03:38:22.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7163000000000006e-05 2024-09-16T03:38:22.280 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.280 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.280 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.281 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.281 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.281 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1702 4.10 4:0a55bbf2:::smithi15736696-18:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:22.281 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:38:22.281 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.491639172999999, 2024-09-16T03:38:22.281 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51800317600000001, 2024-09-16T03:38:22.281 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.281 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.281 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:22.281 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.281 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.282 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.282 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1702 2024-09-16T03:38:22.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.282 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:38:22.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:38:22.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.283 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.283 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.283 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.283 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689147+0000", 2024-09-16T03:38:22.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:38:22.283 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.284 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.284 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689158+0000", 2024-09-16T03:38:22.284 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0426e-05 2024-09-16T03:38:22.284 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.284 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.284 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689185+0000", 2024-09-16T03:38:22.284 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7299000000000001e-05 2024-09-16T03:38:22.284 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.284 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789328+0000", 2024-09-16T03:38:22.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100143102 2024-09-16T03:38:22.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789532+0000", 2024-09-16T03:38:22.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000204317 2024-09-16T03:38:22.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789671+0000", 2024-09-16T03:38:22.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013852600000000001 2024-09-16T03:38:22.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.207147+0000", 2024-09-16T03:38:22.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.41747625599999999 2024-09-16T03:38:22.286 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.286 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.286 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.287 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2160 4.17 4:ed874e32:::smithi15736696-14:head [assert-version v36,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-09-16T03:38:22.287 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:38:22.287 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.350637753000001, 2024-09-16T03:38:22.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55377054800000003, 2024-09-16T03:38:22.287 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.287 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.287 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.287 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.287 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.288 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.288 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2160 2024-09-16T03:38:22.288 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.288 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.288 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.288 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.288 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:38:22.288 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.288 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.288 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.288 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:38:22.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830149+0000", 2024-09-16T03:38:22.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.743e-06 2024-09-16T03:38:22.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830162+0000", 2024-09-16T03:38:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3283e-05 2024-09-16T03:38:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830191+0000", 2024-09-16T03:38:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8466000000000002e-05 2024-09-16T03:38:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830207+0000", 2024-09-16T03:38:22.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6453e-05 2024-09-16T03:38:22.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830362+0000", 2024-09-16T03:38:22.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015450699999999999 2024-09-16T03:38:22.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830608+0000", 2024-09-16T03:38:22.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024587399999999998 2024-09-16T03:38:22.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.292 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:38:22.292 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.233387+0000", 2024-09-16T03:38:22.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40277970099999999 2024-09-16T03:38:22.292 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.292 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:22.292 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234208+0000", 2024-09-16T03:38:22.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082053900000000001 2024-09-16T03:38:22.292 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234370+0000", 2024-09-16T03:38:22.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000161645 2024-09-16T03:38:22.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383821+0000", 2024-09-16T03:38:22.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14945145300000001 2024-09-16T03:38:22.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.294 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383867+0000", 2024-09-16T03:38:22.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5498999999999999e-05 2024-09-16T03:38:22.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.294 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383916+0000", 2024-09-16T03:38:22.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9385000000000002e-05 2024-09-16T03:38:22.294 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.294 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.294 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.295 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2532 4.0 4:06b6d050:::smithi15736696-50:head [write 2603977~725887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:22.295 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:38:22.295 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.5864903640000003, 2024-09-16T03:38:22.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53439557900000001, 2024-09-16T03:38:22.295 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.295 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.295 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.295 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.295 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.295 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.296 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2532 2024-09-16T03:38:22.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.296 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:38:22.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:38:22.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594298+0000", 2024-09-16T03:38:22.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6030000000000004e-06 2024-09-16T03:38:22.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594876+0000", 2024-09-16T03:38:22.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00057757800000000003 2024-09-16T03:38:22.298 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.298 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.298 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594885+0000", 2024-09-16T03:38:22.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4150000000000005e-06 2024-09-16T03:38:22.298 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.298 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.298 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594892+0000", 2024-09-16T03:38:22.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7229999999999998e-06 2024-09-16T03:38:22.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628138+0000", 2024-09-16T03:38:22.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033246407999999998 2024-09-16T03:38:22.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628252+0000", 2024-09-16T03:38:22.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113755 2024-09-16T03:38:22.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.300 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:22.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628344+0000", 2024-09-16T03:38:22.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2028999999999996e-05 2024-09-16T03:38:22.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.300 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:22.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730215+0000", 2024-09-16T03:38:22.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101870761 2024-09-16T03:38:22.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.732105+0000", 2024-09-16T03:38:22.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001889951 2024-09-16T03:38:22.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128411+0000", 2024-09-16T03:38:22.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.396305764 2024-09-16T03:38:22.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128611+0000", 2024-09-16T03:38:22.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019990599999999999 2024-09-16T03:38:22.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128688+0000", 2024-09-16T03:38:22.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7686000000000001e-05 2024-09-16T03:38:22.302 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.302 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.302 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.303 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2533 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3663926] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:22.303 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:38:22.303 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.585883011, 2024-09-16T03:38:22.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53394911599999995, 2024-09-16T03:38:22.303 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.303 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.303 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.303 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.303 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.303 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.303 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2533 2024-09-16T03:38:22.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.304 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:38:22.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:38:22.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594901+0000", 2024-09-16T03:38:22.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6700000000000003e-07 2024-09-16T03:38:22.305 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594904+0000", 2024-09-16T03:38:22.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1099999999999999e-06 2024-09-16T03:38:22.305 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.306 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594907+0000", 2024-09-16T03:38:22.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1020000000000001e-06 2024-09-16T03:38:22.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.306 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594910+0000", 2024-09-16T03:38:22.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7889999999999999e-06 2024-09-16T03:38:22.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.306 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729537+0000", 2024-09-16T03:38:22.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.134627725 2024-09-16T03:38:22.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729660+0000", 2024-09-16T03:38:22.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012283800000000001 2024-09-16T03:38:22.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:22.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729786+0000", 2024-09-16T03:38:22.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012572000000000001 2024-09-16T03:38:22.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:22.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730385+0000", 2024-09-16T03:38:22.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059949700000000005 2024-09-16T03:38:22.308 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.028223+0000", 2024-09-16T03:38:22.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29783727999999998 2024-09-16T03:38:22.308 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128774+0000", 2024-09-16T03:38:22.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100551027 2024-09-16T03:38:22.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128816+0000", 2024-09-16T03:38:22.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1946000000000001e-05 2024-09-16T03:38:22.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128849+0000", 2024-09-16T03:38:22.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3515e-05 2024-09-16T03:38:22.310 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.310 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.310 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.310 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2534 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:22.310 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:38:22.310 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.5858685890000004, 2024-09-16T03:38:22.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53406219200000005, 2024-09-16T03:38:22.311 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.311 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.311 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:22.311 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.311 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.311 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.311 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2534 2024-09-16T03:38:22.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.311 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.311 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.311 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:38:22.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:38:22.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:38:22.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8999999999999998e-07 2024-09-16T03:38:22.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594916+0000", 2024-09-16T03:38:22.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.344e-06 2024-09-16T03:38:22.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594919+0000", 2024-09-16T03:38:22.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0580000000000002e-06 2024-09-16T03:38:22.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594922+0000", 2024-09-16T03:38:22.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4389999999999999e-06 2024-09-16T03:38:22.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729999+0000", 2024-09-16T03:38:22.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.13507707399999999 2024-09-16T03:38:22.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:22.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730046+0000", 2024-09-16T03:38:22.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7382999999999997e-05 2024-09-16T03:38:22.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128879+0000", 2024-09-16T03:38:22.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39883298499999997 2024-09-16T03:38:22.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128924+0000", 2024-09-16T03:38:22.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:38:22.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128977+0000", 2024-09-16T03:38:22.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2648999999999997e-05 2024-09-16T03:38:22.316 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.316 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.316 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.316 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2621 4.f 4:fa163101:::smithi15736696-10:head [setxattr _header (56) in=7b,truncate 3815189] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:38:22.316 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:38:22.317 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.9380922109999998, 2024-09-16T03:38:22.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55430782499999998, 2024-09-16T03:38:22.317 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.317 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.317 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.317 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.317 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.317 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.317 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2621 2024-09-16T03:38:22.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.317 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:38:22.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:38:22.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242692+0000", 2024-09-16T03:38:22.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7599999999999996e-07 2024-09-16T03:38:22.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242695+0000", 2024-09-16T03:38:22.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.529e-06 2024-09-16T03:38:22.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242699+0000", 2024-09-16T03:38:22.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1010000000000002e-06 2024-09-16T03:38:22.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.320 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.320 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242703+0000", 2024-09-16T03:38:22.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.518e-06 2024-09-16T03:38:22.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.320 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.320 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.494935+0000", 2024-09-16T03:38:22.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25223231200000001 2024-09-16T03:38:22.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495011+0000", 2024-09-16T03:38:22.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5692999999999995e-05 2024-09-16T03:38:22.321 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:38:22.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495117+0000", 2024-09-16T03:38:22.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000105944 2024-09-16T03:38:22.321 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:22.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.596308+0000", 2024-09-16T03:38:22.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101190825 2024-09-16T03:38:22.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796852+0000", 2024-09-16T03:38:22.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20054443199999999 2024-09-16T03:38:22.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:22.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796893+0000", 2024-09-16T03:38:22.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1221999999999999e-05 2024-09-16T03:38:22.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796946+0000", 2024-09-16T03:38:22.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2435000000000002e-05 2024-09-16T03:38:22.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796999+0000", 2024-09-16T03:38:22.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3038000000000003e-05 2024-09-16T03:38:22.323 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.323 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.323 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.324 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2622 4.f 4:fa163101:::smithi15736696-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:38:22.324 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:38:22.324 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.9380763559999998, 2024-09-16T03:38:22.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55448354399999999, 2024-09-16T03:38:22.324 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.324 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.324 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:22.324 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.324 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.324 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.324 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2622 2024-09-16T03:38:22.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.325 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:38:22.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:38:22.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:38:22.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9999999999999999e-07 2024-09-16T03:38:22.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242709+0000", 2024-09-16T03:38:22.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.429e-06 2024-09-16T03:38:22.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242712+0000", 2024-09-16T03:38:22.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6090000000000001e-06 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242716+0000", 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6480000000000001e-06 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495371+0000", 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25265485700000001 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:22.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495424+0000", 2024-09-16T03:38:22.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2927999999999999e-05 2024-09-16T03:38:22.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797042+0000", 2024-09-16T03:38:22.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30161821900000002 2024-09-16T03:38:22.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797114+0000", 2024-09-16T03:38:22.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2445000000000002e-05 2024-09-16T03:38:22.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.329 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797190+0000", 2024-09-16T03:38:22.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6108999999999999e-05 2024-09-16T03:38:22.329 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.329 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.329 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.329 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.329 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.329 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:22.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:38:22.332 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:38:22.333 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:22.330+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:22.370 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:22.368+0000 7f12e4a1e640 -1 osd.5 91 handle_osd_map injecting map crc failure 2024-09-16T03:38:22.372 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:38:22.372 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.12 to 1 (10000) 2024-09-16T03:38:22.387 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-16T03:38:22.431 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:22.429+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:22.437 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:22.437 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:22.437 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:22.437 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:22.446 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:38:22.462 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:22.463 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:22.481 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:22.481 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:22.485 INFO:tasks.rados.rados.0.smithi157.stdout:1239: finishing set_chunk to oid 27 2024-09-16T03:38:22.485 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 1154560 length: 555520 2024-09-16T03:38:22.485 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 0 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T03:38:22.485 INFO:tasks.rados.rados.0.smithi157.stdout:1233: finishing write tid 4 to smithi15736696-45 2024-09-16T03:38:22.485 INFO:tasks.rados.rados.0.smithi157.stdout:1233: finishing write tid 5 to smithi15736696-45 2024-09-16T03:38:22.485 INFO:tasks.rados.rados.0.smithi157.stdout:1233: oid 45 updating version 0 to 73 2024-09-16T03:38:22.485 INFO:tasks.rados.rados.0.smithi157.stdout:1233: oid 45 version 73 is already newer than 72 2024-09-16T03:38:22.485 INFO:tasks.rados.rados.0.smithi157.stdout:1233: oid 45 version 73 is already newer than 71 2024-09-16T03:38:22.485 INFO:tasks.rados.rados.0.smithi157.stdout:1233: oid 45 updating version 73 to 74 2024-09-16T03:38:22.485 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 74 (ObjNum 234 snap 0 seq_num 234) dirty exists 2024-09-16T03:38:22.485 INFO:tasks.rados.rados.0.smithi157.stdout:1233: left oid 45 (ObjNum 234 snap 0 seq_num 234) 2024-09-16T03:38:22.486 INFO:tasks.rados.rados.0.smithi157.stdout:1243: got (0) Success 2024-09-16T03:38:22.486 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 60 (ObjNum 106 snap 0 seq_num 106) dirty exists 2024-09-16T03:38:22.486 INFO:tasks.rados.rados.0.smithi157.stdout:1232: finishing write tid 5 to smithi15736696-11 2024-09-16T03:38:22.486 INFO:tasks.rados.rados.0.smithi157.stdout:1232: oid 11 updating version 0 to 53 2024-09-16T03:38:22.486 INFO:tasks.rados.rados.0.smithi157.stdout:1232: oid 11 version 53 is already newer than 52 2024-09-16T03:38:22.486 INFO:tasks.rados.rados.0.smithi157.stdout:1232: oid 11 version 53 is already newer than 51 2024-09-16T03:38:22.486 INFO:tasks.rados.rados.0.smithi157.stdout:1232: oid 11 version 53 is already newer than 50 2024-09-16T03:38:22.486 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 53 (ObjNum 233 snap 0 seq_num 233) dirty exists 2024-09-16T03:38:22.486 INFO:tasks.rados.rados.0.smithi157.stdout:1232: left oid 11 (ObjNum 233 snap 0 seq_num 233) 2024-09-16T03:38:22.486 INFO:tasks.rados.rados.0.smithi157.stdout:1242: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:22.486 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:22.498 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:22.498 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:22.515 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:22.515 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:22.515 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1367786 2024-09-16T03:38:22.531 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:22.531 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:22.532 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:22.531+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:22.549 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:22.550 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:22.554 INFO:tasks.rados.rados.0.smithi157.stdout:1238: got (0) Success 2024-09-16T03:38:22.554 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 0 (ObjNum 218 snap 0 seq_num 218) dirty exists 2024-09-16T03:38:22.555 INFO:tasks.rados.rados.0.smithi157.stdout:1232: done (10 left) 2024-09-16T03:38:22.555 INFO:tasks.rados.rados.0.smithi157.stdout:1233: done (9 left) 2024-09-16T03:38:22.555 INFO:tasks.rados.rados.0.smithi157.stdout:1237: done (8 left) 2024-09-16T03:38:22.555 INFO:tasks.rados.rados.0.smithi157.stdout:1238: done (7 left) 2024-09-16T03:38:22.555 INFO:tasks.rados.rados.0.smithi157.stdout:1239: done (6 left) 2024-09-16T03:38:22.555 INFO:tasks.rados.rados.0.smithi157.stdout:1241: done (5 left) 2024-09-16T03:38:22.555 INFO:tasks.rados.rados.0.smithi157.stdout:1242: done (4 left) 2024-09-16T03:38:22.556 INFO:tasks.rados.rados.0.smithi157.stdout:1243: done (3 left) 2024-09-16T03:38:22.556 INFO:tasks.rados.rados.0.smithi157.stdout:1245: done (2 left) 2024-09-16T03:38:22.556 INFO:tasks.rados.rados.0.smithi157.stdout:1246: read oid 7 snap -1 2024-09-16T03:38:22.556 INFO:tasks.rados.rados.0.smithi157.stdout:1246: expect deleted 2024-09-16T03:38:22.556 INFO:tasks.rados.rados.0.smithi157.stdout:1247: set_chunk oid 2 target oid 2024-09-16T03:38:22.556 INFO:tasks.rados.rados.0.smithi157.stdout:1247: set_chunk oid 2 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:22.556 INFO:tasks.rados.rados.0.smithi157.stdout:1248: write oid 24 current snap is 0 2024-09-16T03:38:22.556 INFO:tasks.rados.rados.0.smithi157.stdout:1240: got (0) Success 2024-09-16T03:38:22.556 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 87 (ObjNum 205 snap 0 seq_num 205) dirty exists 2024-09-16T03:38:22.556 INFO:tasks.rados.rados.0.smithi157.stdout:1244: finishing write tid 1 to smithi15736696-46 2024-09-16T03:38:22.557 INFO:tasks.rados.rados.0.smithi157.stdout:1244: finishing write tid 2 to smithi15736696-46 2024-09-16T03:38:22.557 INFO:tasks.rados.rados.0.smithi157.stdout:1244: finishing write tid 3 to smithi15736696-46 2024-09-16T03:38:22.557 INFO:tasks.rados.rados.0.smithi157.stdout:1248: seq_num 238 ranges {400066=716205,1860832=139234} 2024-09-16T03:38:22.559 INFO:tasks.rados.rados.0.smithi157.stdout:1248: writing smithi15736696-24 from 400066 to 1116271 tid 1 2024-09-16T03:38:22.565 INFO:tasks.rados.rados.0.smithi157.stdout:1248: writing smithi15736696-24 from 1860832 to 2000066 tid 2 2024-09-16T03:38:22.565 INFO:tasks.rados.rados.0.smithi157.stdout:1240: done (4 left) 2024-09-16T03:38:22.565 INFO:tasks.rados.rados.0.smithi157.stdout:1249: read oid 9 snap -1 2024-09-16T03:38:22.565 INFO:tasks.rados.rados.0.smithi157.stdout:1249: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-16T03:38:22.565 INFO:tasks.rados.rados.0.smithi157.stdout:1250: write oid 8 current snap is 0 2024-09-16T03:38:22.565 INFO:tasks.rados.rados.0.smithi157.stdout:1244: finishing write tid 4 to smithi15736696-46 2024-09-16T03:38:22.566 INFO:tasks.rados.rados.0.smithi157.stdout:1244: finishing write tid 5 to smithi15736696-46 2024-09-16T03:38:22.566 INFO:tasks.rados.rados.0.smithi157.stdout:1244: oid 46 updating version 0 to 92 2024-09-16T03:38:22.566 INFO:tasks.rados.rados.0.smithi157.stdout:1244: oid 46 version 92 is already newer than 90 2024-09-16T03:38:22.566 INFO:tasks.rados.rados.0.smithi157.stdout:1244: oid 46 updating version 92 to 93 2024-09-16T03:38:22.566 INFO:tasks.rados.rados.0.smithi157.stdout:1244: oid 46 version 93 is already newer than 91 2024-09-16T03:38:22.566 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 93 (ObjNum 236 snap 0 seq_num 236) dirty exists 2024-09-16T03:38:22.566 INFO:tasks.rados.rados.0.smithi157.stdout:1244: left oid 46 (ObjNum 236 snap 0 seq_num 236) 2024-09-16T03:38:22.566 INFO:tasks.rados.rados.0.smithi157.stdout:1250: seq_num 239 ranges {416873=791454,1919314=97559} 2024-09-16T03:38:22.568 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:22.568 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:22.570 INFO:tasks.rados.rados.0.smithi157.stdout:1250: writing smithi15736696-8 from 416873 to 1208327 tid 1 2024-09-16T03:38:22.575 INFO:tasks.rados.rados.0.smithi157.stdout:1250: writing smithi15736696-8 from 1919314 to 2016873 tid 2 2024-09-16T03:38:22.576 INFO:tasks.rados.rados.0.smithi157.stdout:1244: done (5 left) 2024-09-16T03:38:22.576 INFO:tasks.rados.rados.0.smithi157.stdout:1246: done (4 left) 2024-09-16T03:38:22.576 INFO:tasks.rados.rados.0.smithi157.stdout:1251: write oid 45 current snap is 0 2024-09-16T03:38:22.576 INFO:tasks.rados.rados.0.smithi157.stdout:1251: seq_num 240 ranges {433680=466703,1577796=455884} 2024-09-16T03:38:22.580 INFO:tasks.rados.rados.0.smithi157.stdout:1251: writing smithi15736696-45 from 433680 to 900383 tid 1 2024-09-16T03:38:22.587 INFO:tasks.rados.rados.0.smithi157.stdout:1251: writing smithi15736696-45 from 1577796 to 2033680 tid 2 2024-09-16T03:38:22.587 INFO:tasks.rados.rados.0.smithi157.stdout:1252: read oid 42 snap -1 2024-09-16T03:38:22.587 INFO:tasks.rados.rados.0.smithi157.stdout:1252: expect (ObjNum 200 snap 0 seq_num 200) 2024-09-16T03:38:22.587 INFO:tasks.rados.rados.0.smithi157.stdout:1253: tier_promote oid 34 2024-09-16T03:38:22.587 INFO:tasks.rados.rados.0.smithi157.stdout:1254: write (excl) oid 40 current snap is 0 2024-09-16T03:38:22.587 INFO:tasks.rados.rados.0.smithi157.stdout:1254: seq_num 241 ranges {450487=541952,1636278=414209} 2024-09-16T03:38:22.592 INFO:tasks.rados.rados.0.smithi157.stdout:1254: writing smithi15736696-40 from 450487 to 992439 tid 1 2024-09-16T03:38:22.593 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:22.594 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:22.594 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:22.594 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:22.594 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:22.598 INFO:tasks.rados.rados.0.smithi157.stdout:1254: writing smithi15736696-40 from 1636278 to 2050487 tid 2 2024-09-16T03:38:22.598 INFO:tasks.rados.rados.0.smithi157.stdout:1255: write (excl) oid 1 current snap is 0 2024-09-16T03:38:22.598 INFO:tasks.rados.rados.0.smithi157.stdout:1255: seq_num 242 ranges {467294=617201,1578407=488887} 2024-09-16T03:38:22.603 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:38:22.604 INFO:tasks.rados.rados.0.smithi157.stdout:1255: writing smithi15736696-1 from 467294 to 1084495 tid 1 2024-09-16T03:38:22.606 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:22.606 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:22.609 INFO:tasks.rados.rados.0.smithi157.stdout:1255: writing smithi15736696-1 from 1578407 to 2067294 tid 2 2024-09-16T03:38:22.609 INFO:tasks.rados.rados.0.smithi157.stdout:1256: delete oid 15 current snap is 0 2024-09-16T03:38:22.610 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:22.608+0000 7f3d4270c640 -1 osd.12 91 handle_osd_map injecting map crc failure 2024-09-16T03:38:22.612 INFO:tasks.rados.rados.0.smithi157.stdout:1255: finishing write tid 1 to smithi15736696-1 2024-09-16T03:38:22.622 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:22.622 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:22.648 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:22.645+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:22.656 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:22.656 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:22.664 INFO:tasks.rados.rados.0.smithi157.stdout:1248: finishing write tid 1 to smithi15736696-24 2024-09-16T03:38:22.664 INFO:tasks.rados.rados.0.smithi157.stdout:1248: finishing write tid 2 to smithi15736696-24 2024-09-16T03:38:22.665 INFO:tasks.rados.rados.0.smithi157.stdout:1248: finishing write tid 3 to smithi15736696-24 2024-09-16T03:38:22.665 INFO:tasks.rados.rados.0.smithi157.stdout:1248: finishing write tid 4 to smithi15736696-24 2024-09-16T03:38:22.665 INFO:tasks.rados.rados.0.smithi157.stdout:1248: oid 24 updating version 0 to 102 2024-09-16T03:38:22.665 INFO:tasks.rados.rados.0.smithi157.stdout:1248: oid 24 version 102 is already newer than 101 2024-09-16T03:38:22.665 INFO:tasks.rados.rados.0.smithi157.stdout:1248: oid 24 updating version 102 to 103 2024-09-16T03:38:22.665 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 103 (ObjNum 238 snap 0 seq_num 238) dirty exists 2024-09-16T03:38:22.665 INFO:tasks.rados.rados.0.smithi157.stdout:1248: left oid 24 (ObjNum 238 snap 0 seq_num 238) 2024-09-16T03:38:22.672 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:22.673 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:22.682 INFO:tasks.rados.rados.0.smithi157.stdout:1250: finishing write tid 1 to smithi15736696-8 2024-09-16T03:38:22.682 INFO:tasks.rados.rados.0.smithi157.stdout:1250: finishing write tid 2 to smithi15736696-8 2024-09-16T03:38:22.688 INFO:tasks.rados.rados.0.smithi157.stdout:1247: finishing set_chunk to oid 2 2024-09-16T03:38:22.688 INFO:tasks.rados.rados.0.smithi157.stdout:1247: got expected ENOENT (src dne) 2024-09-16T03:38:22.689 INFO:tasks.rados.rados.0.smithi157.stdout:1252: expect (ObjNum 200 snap 0 seq_num 200) 2024-09-16T03:38:22.689 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:22.711 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:22.711 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:22.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:22.712 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:22.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.712 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:22.712 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:22.712 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.565561552999995, 2024-09-16T03:38:22.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:38:22.712 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.712 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.712 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.712 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:38:22.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:38:22.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:38:22.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:38:22.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:38:22.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:38:22.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:38:22.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:38:22.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:38:22.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:38:22.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:38:22.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.716 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.565208193000004, 2024-09-16T03:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:22.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:38:22.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:38:22.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.718 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.718 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:38:22.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:38:22.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:38:22.721 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.721 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.721 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.721 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:38:22.721 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:22.721 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.564080395000005, 2024-09-16T03:38:22.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:38:22.721 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.721 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:38:22.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.726 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:38:22.726 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:22.726 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.563092953999998, 2024-09-16T03:38:22.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:38:22.726 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.726 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.726 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.726 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:22.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:38:22.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:38:22.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:38:22.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:38:22.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:38:22.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:38:22.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.730 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.730 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:38:22.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:38:22.730 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.730 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.730 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.730 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.730 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.730 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:38:22.730 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:22.730 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.562336209999998, 2024-09-16T03:38:22.730 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:38:22.730 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.731 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:38:22.732 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:38:22.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:38:22.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:38:22.733 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:38:22.733 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.733 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.733 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.733 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.734 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:86 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'8, mlcod=39'5)", 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.100823973999994, 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20388281899999999, 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:22.735 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608472+0000", 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.227e-06 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.736 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608616+0000", 2024-09-16T03:38:22.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000144177 2024-09-16T03:38:22.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608638+0000", 2024-09-16T03:38:22.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1543e-05 2024-09-16T03:38:22.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.737 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.737 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608645+0000", 2024-09-16T03:38:22.737 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5839999999999997e-06 2024-09-16T03:38:22.737 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.737 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608740+0000", 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4716999999999995e-05 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608763+0000", 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2671e-05 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812264+0000", 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203501194 2024-09-16T03:38:22.738 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812353+0000", 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8706000000000001e-05 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:87 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'9, mlcod=39'5)", 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.100610599999996, 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20372206400000001, 2024-09-16T03:38:22.739 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.740 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608684+0000", 2024-09-16T03:38:22.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6700000000000003e-07 2024-09-16T03:38:22.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608689+0000", 2024-09-16T03:38:22.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.412e-06 2024-09-16T03:38:22.741 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.741 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.741 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.741 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608704+0000", 2024-09-16T03:38:22.741 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4397e-05 2024-09-16T03:38:22.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608713+0000", 2024-09-16T03:38:22.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:38:22.742 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.742 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.742 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.742 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710025+0000", 2024-09-16T03:38:22.742 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101312032 2024-09-16T03:38:22.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.743 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710042+0000", 2024-09-16T03:38:22.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6889e-05 2024-09-16T03:38:22.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.743 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812361+0000", 2024-09-16T03:38:22.743 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10231889600000001 2024-09-16T03:38:22.743 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.743 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.743 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.744 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812405+0000", 2024-09-16T03:38:22.744 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4101999999999999e-05 2024-09-16T03:38:22.744 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.744 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.744 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.744 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.744 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.744 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:101 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'10, mlcod=39'5)", 2024-09-16T03:38:22.744 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:22.744 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.099597162999999, 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20275586900000001, 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:22.745 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609698+0000", 2024-09-16T03:38:22.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.24e-06 2024-09-16T03:38:22.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.746 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609892+0000", 2024-09-16T03:38:22.746 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019465899999999999 2024-09-16T03:38:22.746 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.746 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.746 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609904+0000", 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2031000000000001e-05 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609912+0000", 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7160000000000003e-06 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710228+0000", 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100315682 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.747 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.748 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710242+0000", 2024-09-16T03:38:22.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4201e-05 2024-09-16T03:38:22.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.748 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812411+0000", 2024-09-16T03:38:22.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10216881799999999 2024-09-16T03:38:22.748 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.748 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.748 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.748 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812452+0000", 2024-09-16T03:38:22.748 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1522e-05 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:102 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'11, mlcod=39'5)", 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.097581137999995, 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20134195399999999, 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.749 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.750 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.750 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:22.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.750 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.750 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.750 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:22.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.750 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.750 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.750 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.750 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611714+0000", 2024-09-16T03:38:22.750 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.269e-06 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611926+0000", 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021252700000000001 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611938+0000", 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1437000000000001e-05 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.751 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611944+0000", 2024-09-16T03:38:22.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.993e-06 2024-09-16T03:38:22.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.752 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.752 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811266+0000", 2024-09-16T03:38:22.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.199322306 2024-09-16T03:38:22.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.752 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.752 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.752 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811291+0000", 2024-09-16T03:38:22.752 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4542999999999999e-05 2024-09-16T03:38:22.752 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.752 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813021+0000", 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001730363 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813054+0000", 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3516000000000002e-05 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.753 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:103 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'12, mlcod=39'5)", 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.094374637000001, 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19838012999999999, 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.754 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.755 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614919+0000", 2024-09-16T03:38:22.755 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.755 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.755 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.755 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.756 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614920+0000", 2024-09-16T03:38:22.756 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.192e-06 2024-09-16T03:38:22.756 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.756 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.756 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614927+0000", 2024-09-16T03:38:22.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4640000000000003e-06 2024-09-16T03:38:22.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614937+0000", 2024-09-16T03:38:22.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9599999999999995e-06 2024-09-16T03:38:22.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.757 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.757 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.757 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.614942+0000", 2024-09-16T03:38:22.757 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.31e-06 2024-09-16T03:38:22.757 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812489+0000", 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.197546629 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812511+0000", 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2385e-05 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813279+0000", 2024-09-16T03:38:22.758 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076818100000000005 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813299+0000", 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0009000000000001e-05 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:375 4.f e45/32 4:fcb75e43:::smithi15736696-26:head v 45'23, mlcod=41'20)", 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.401253410000002, 2024-09-16T03:38:22.759 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200786568, 2024-09-16T03:38:22.760 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.760 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.760 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.760 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.760 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.760 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.760 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:22.760 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.760 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.760 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.760 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.760 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:22.760 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308041+0000", 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8299999999999995e-07 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308046+0000", 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.904e-06 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.761 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308056+0000", 2024-09-16T03:38:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9210000000000004e-06 2024-09-16T03:38:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.408122+0000", 2024-09-16T03:38:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100066302 2024-09-16T03:38:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508289+0000", 2024-09-16T03:38:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166531 2024-09-16T03:38:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.762 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508302+0000", 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2641e-05 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508794+0000", 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00049272400000000005 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508827+0000", 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2561999999999998e-05 2024-09-16T03:38:22.763 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:989 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'31, mlcod=46'28)", 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.321484849999997, 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40332266900000002, 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.764 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387812+0000", 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.765 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388467+0000", 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065539500000000002 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388481+0000", 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3905e-05 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488552+0000", 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10007120899999999 2024-09-16T03:38:22.766 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688818+0000", 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20026627599999999 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688845+0000", 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791091+0000", 2024-09-16T03:38:22.767 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10224588699999999 2024-09-16T03:38:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791131+0000", 2024-09-16T03:38:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0293000000000002e-05 2024-09-16T03:38:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:990 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'32, mlcod=46'28)", 2024-09-16T03:38:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:22.768 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.220701177999999, 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30256419200000001, 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:22.769 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488594+0000", 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488616+0000", 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2419e-05 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.770 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488627+0000", 2024-09-16T03:38:22.771 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1251e-05 2024-09-16T03:38:22.771 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.771 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588753+0000", 2024-09-16T03:38:22.771 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012544399999999 2024-09-16T03:38:22.771 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.771 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.771 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690679+0000", 2024-09-16T03:38:22.771 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101926027 2024-09-16T03:38:22.771 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.771 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690695+0000", 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6528000000000002e-05 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791133+0000", 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100438233 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791157+0000", 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-05 2024-09-16T03:38:22.772 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:991 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'33, mlcod=46'28)", 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.120503681000002, 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202786365, 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.773 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588792+0000", 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0530000000000001e-06 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.774 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588807+0000", 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5257e-05 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588832+0000", 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5123e-05 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588846+0000", 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3312999999999999e-05 2024-09-16T03:38:22.775 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.776 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.776 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.776 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790810+0000", 2024-09-16T03:38:22.776 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20196378100000001 2024-09-16T03:38:22.776 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.776 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.776 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.776 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790866+0000", 2024-09-16T03:38:22.776 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6168999999999997e-05 2024-09-16T03:38:22.776 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.776 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.776 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.776 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791542+0000", 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067583699999999999 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791576+0000", 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4832000000000001e-05 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1072 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'35, mlcod=50'32)", 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:38:22.777 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.401261853999998, 2024-09-16T03:38:22.778 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.103370715, 2024-09-16T03:38:22.778 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.778 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.778 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.778 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.778 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.778 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.778 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:38:22.778 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.779 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.779 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308032+0000", 2024-09-16T03:38:22.779 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.779 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.779 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308034+0000", 2024-09-16T03:38:22.779 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4099999999999998e-06 2024-09-16T03:38:22.779 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.779 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.779 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308377+0000", 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00034276400000000001 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.308415+0000", 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7916000000000001e-05 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408542+0000", 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012673699999999 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.780 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408694+0000", 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015200399999999999 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.408760+0000", 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6549000000000003e-05 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.411366+0000", 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002605324 2024-09-16T03:38:22.781 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.411403+0000", 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7011000000000002e-05 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1580 4.f e61/32 4:fa163101:::smithi15736696-10:head v 61'52, mlcod=58'50)", 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.603672154999998, 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.201787367, 2024-09-16T03:38:22.782 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.783 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.784 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.784 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:22.784 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2600000000000001e-07 2024-09-16T03:38:22.784 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.784 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.784 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.784 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106083+0000", 2024-09-16T03:38:22.784 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00046163900000000002 2024-09-16T03:38:22.784 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.784 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.784 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106093+0000", 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9320000000000008e-06 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206190+0000", 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100096716 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206286+0000", 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5982000000000004e-05 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.785 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206312+0000", 2024-09-16T03:38:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6279999999999999e-05 2024-09-16T03:38:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307345+0000", 2024-09-16T03:38:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10103240199999999 2024-09-16T03:38:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307409+0000", 2024-09-16T03:38:22.786 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4189999999999994e-05 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1760 4.f e71/32 4:f101f67d:::smithi15736696-22:head v 71'55, mlcod=61'53)", 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 26.328468188999999, 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20118894600000001, 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.787 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380826+0000", 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1899999999999998e-07 2024-09-16T03:38:22.788 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380831+0000", 2024-09-16T03:38:22.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6049999999999997e-06 2024-09-16T03:38:22.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380851+0000", 2024-09-16T03:38:22.789 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0021e-05 2024-09-16T03:38:22.789 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.789 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.789 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.789 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380862+0000", 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0458e-05 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581184+0000", 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20032164299999999 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581202+0000", 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8410000000000002e-05 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.790 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581965+0000", 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076291200000000003 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.582014+0000", 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9577999999999999e-05 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2112 4.12 4:4a9d235d:::smithi15736696-1:head [sparse-read 0~1423481,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:38:22.791 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:05.542212+0000", 2024-09-16T03:38:22.792 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 17.167082071999999, 2024-09-16T03:38:22.792 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10240218600000001, 2024-09-16T03:38:22.792 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.792 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.792 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:22.792 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:22.792 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:22.792 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:22.792 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2112 2024-09-16T03:38:22.792 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.792 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.792 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.792 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.542212+0000", 2024-09-16T03:38:22.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.542212+0000", 2024-09-16T03:38:22.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.793 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.793 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.793 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.542215+0000", 2024-09-16T03:38:22.793 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6430000000000001e-06 2024-09-16T03:38:22.793 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.542225+0000", 2024-09-16T03:38:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0010000000000001e-05 2024-09-16T03:38:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.542243+0000", 2024-09-16T03:38:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7861000000000001e-05 2024-09-16T03:38:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.642406+0000", 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100162559 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.642547+0000", 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014096099999999999 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.642638+0000", 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1836999999999995e-05 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.795 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:05.644614+0000", 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019753150000000001 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2664 4.f e91/32 4:f817def9:::smithi15736696-45:head v 91'71, mlcod=89'69)", 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.623583+0000", 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.08571112, 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10213971199999999, 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.796 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.797 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.623583+0000", 2024-09-16T03:38:22.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.797 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.623583+0000", 2024-09-16T03:38:22.797 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.797 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.797 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.797 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.623583+0000", 2024-09-16T03:38:22.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9299999999999998e-07 2024-09-16T03:38:22.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.623982+0000", 2024-09-16T03:38:22.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000399214 2024-09-16T03:38:22.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.798 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.798 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.798 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.623997+0000", 2024-09-16T03:38:22.798 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4738e-05 2024-09-16T03:38:22.798 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.624008+0000", 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1011e-05 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.724257+0000", 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10024862 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.724275+0000", 2024-09-16T03:38:22.799 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8507000000000001e-05 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.725686+0000", 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014112669999999999 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.725722+0000", 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5862000000000003e-05 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.800 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2665 4.f e91/32 4:f817def9:::smithi15736696-45:head v 91'72, mlcod=89'69)", 2024-09-16T03:38:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.624918+0000", 2024-09-16T03:38:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.084375804, 2024-09-16T03:38:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.103272765, 2024-09-16T03:38:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.624918+0000", 2024-09-16T03:38:22.801 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:22.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.624918+0000", 2024-09-16T03:38:22.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:22.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:22.802 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.624919+0000", 2024-09-16T03:38:22.802 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1990000000000001e-06 2024-09-16T03:38:22.802 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.802 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.802 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.625229+0000", 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00030959 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.625242+0000", 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3359e-05 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.625250+0000", 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3080000000000008e-06 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.803 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.804 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:22.804 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.725083+0000", 2024-09-16T03:38:22.804 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099832659000000004 2024-09-16T03:38:22.804 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.804 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.804 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:22.804 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.725099+0000", 2024-09-16T03:38:22.804 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6229999999999999e-05 2024-09-16T03:38:22.804 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.804 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.804 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:22.804 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.728164+0000", 2024-09-16T03:38:22.804 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0030643950000000001 2024-09-16T03:38:22.805 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:22.805 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:22.805 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:22.805 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.728191+0000", 2024-09-16T03:38:22.805 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7025e-05 2024-09-16T03:38:22.805 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.805 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.805 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.805 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:22.805 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:22.805 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:22.806 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:22.807 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:22.807 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:22.808 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:22.808 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:38:22.808 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:22.809 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:22.809 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:22.805+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:22.830 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:22.830 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:22.889 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:22.889 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:22.905 INFO:tasks.rados.rados.0.smithi157.stdout:1254: finishing write tid 1 to smithi15736696-40 2024-09-16T03:38:22.905 INFO:tasks.rados.rados.0.smithi157.stdout:1254: finishing write tid 2 to smithi15736696-40 2024-09-16T03:38:22.905 INFO:tasks.rados.rados.0.smithi157.stdout:1254: finishing write tid 3 to smithi15736696-40 2024-09-16T03:38:22.905 INFO:tasks.rados.rados.0.smithi157.stdout:1254: finishing write tid 4 to smithi15736696-40 2024-09-16T03:38:22.905 INFO:tasks.rados.rados.0.smithi157.stdout:1254: oid 40 updating version 0 to 100 2024-09-16T03:38:22.905 INFO:tasks.rados.rados.0.smithi157.stdout:1254: oid 40 updating version 100 to 101 2024-09-16T03:38:22.906 INFO:tasks.rados.rados.0.smithi157.stdout:1254: oid 40 updating version 101 to 102 2024-09-16T03:38:22.906 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 102 (ObjNum 241 snap 0 seq_num 241) dirty exists 2024-09-16T03:38:22.906 INFO:tasks.rados.rados.0.smithi157.stdout:1254: left oid 40 (ObjNum 241 snap 0 seq_num 241) 2024-09-16T03:38:22.906 INFO:tasks.rados.rados.0.smithi157.stdout:1250: finishing write tid 3 to smithi15736696-8 2024-09-16T03:38:22.906 INFO:tasks.rados.rados.0.smithi157.stdout:1247: done (9 left) 2024-09-16T03:38:22.906 INFO:tasks.rados.rados.0.smithi157.stdout:1248: done (8 left) 2024-09-16T03:38:22.906 INFO:tasks.rados.rados.0.smithi157.stdout:1252: done (7 left) 2024-09-16T03:38:22.906 INFO:tasks.rados.rados.0.smithi157.stdout:1254: done (6 left) 2024-09-16T03:38:22.906 INFO:tasks.rados.rados.0.smithi157.stdout:1256: done (5 left) 2024-09-16T03:38:22.906 INFO:tasks.rados.rados.0.smithi157.stdout:1257: read oid 16 snap -1 2024-09-16T03:38:22.906 INFO:tasks.rados.rados.0.smithi157.stdout:1257: expect (ObjNum 224 snap 0 seq_num 224) 2024-09-16T03:38:22.906 INFO:tasks.rados.rados.0.smithi157.stdout:1258: tier_promote oid 22 2024-09-16T03:38:22.906 INFO:tasks.rados.rados.0.smithi157.stdout:1250: finishing write tid 4 to smithi15736696-8 2024-09-16T03:38:22.907 INFO:tasks.rados.rados.0.smithi157.stdout:1250: oid 8 updating version 0 to 76 2024-09-16T03:38:22.907 INFO:tasks.rados.rados.0.smithi157.stdout:1250: oid 8 updating version 76 to 77 2024-09-16T03:38:22.907 INFO:tasks.rados.rados.0.smithi157.stdout:1250: oid 8 version 77 is already newer than 75 2024-09-16T03:38:22.907 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 77 (ObjNum 239 snap 0 seq_num 239) dirty exists 2024-09-16T03:38:22.907 INFO:tasks.rados.rados.0.smithi157.stdout:1250: left oid 8 (ObjNum 239 snap 0 seq_num 239) 2024-09-16T03:38:22.907 INFO:tasks.rados.rados.0.smithi157.stdout:1250: done (6 left) 2024-09-16T03:38:22.907 INFO:tasks.rados.rados.0.smithi157.stdout:1259: read oid 26 snap -1 2024-09-16T03:38:22.907 INFO:tasks.rados.rados.0.smithi157.stdout:1259: expect deleted 2024-09-16T03:38:22.907 INFO:tasks.rados.rados.0.smithi157.stdout:1260: read oid 46 snap -1 2024-09-16T03:38:22.907 INFO:tasks.rados.rados.0.smithi157.stdout:1260: expect (ObjNum 236 snap 0 seq_num 236) 2024-09-16T03:38:22.907 INFO:tasks.rados.rados.0.smithi157.stdout:1261: set_chunk oid 31 target oid 2024-09-16T03:38:22.920 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:22.920 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:22.929 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:38:22.942 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:22.940+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:23.009 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:23.008+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:23.033 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:23.033 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:23.042 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:38:23.110 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:23.107+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:23.122 INFO:tasks.rados.rados.0.smithi157.stdout:1261: set_chunk oid 31 offset: 1722880 length: 1432576 target oid 949809453df8e7127aa2d84be0664ca726a30776518c717f88e4f79755b3be2a offset: 0 snap -1 2024-09-16T03:38:23.122 INFO:tasks.rados.rados.0.smithi157.stdout:1253: got (0) Success 2024-09-16T03:38:23.122 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 87 (ObjNum 205 snap 0 seq_num 205) dirty exists 2024-09-16T03:38:23.122 INFO:tasks.rados.rados.0.smithi157.stdout:1255: finishing write tid 2 to smithi15736696-1 2024-09-16T03:38:23.122 INFO:tasks.rados.rados.0.smithi157.stdout:1255: finishing write tid 3 to smithi15736696-1 2024-09-16T03:38:23.123 INFO:tasks.rados.rados.0.smithi157.stdout:1262: read oid 13 snap -1 2024-09-16T03:38:23.123 INFO:tasks.rados.rados.0.smithi157.stdout:1262: expect (ObjNum 209 snap 0 seq_num 209) 2024-09-16T03:38:23.123 INFO:tasks.rados.rados.0.smithi157.stdout:1253: done (9 left) 2024-09-16T03:38:23.123 INFO:tasks.rados.rados.0.smithi157.stdout:1263: tier_evict oid 39 2024-09-16T03:38:23.123 INFO:tasks.rados.rados.0.smithi157.stdout:1263: tier_evict oid 39 snap -1 2024-09-16T03:38:23.123 INFO:tasks.rados.rados.0.smithi157.stdout:1255: finishing write tid 4 to smithi15736696-1 2024-09-16T03:38:23.123 INFO:tasks.rados.rados.0.smithi157.stdout:1255: oid 1 updating version 0 to 14 2024-09-16T03:38:23.123 INFO:tasks.rados.rados.0.smithi157.stdout:1255: oid 1 updating version 14 to 16 2024-09-16T03:38:23.123 INFO:tasks.rados.rados.0.smithi157.stdout:1255: oid 1 version 16 is already newer than 15 2024-09-16T03:38:23.123 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 16 (ObjNum 242 snap 0 seq_num 242) dirty exists 2024-09-16T03:38:23.123 INFO:tasks.rados.rados.0.smithi157.stdout:1255: left oid 1 (ObjNum 242 snap 0 seq_num 242) 2024-09-16T03:38:23.123 INFO:tasks.rados.rados.0.smithi157.stdout:1249: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-16T03:38:23.148 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:23.148 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:23.158 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:38:23.184 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:23.210 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:23.209+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:23.246 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:23.264 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:23.264 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:23.273 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:38:23.307 INFO:tasks.rados.rados.0.smithi157.stdout:1251: finishing write tid 1 to smithi15736696-45 2024-09-16T03:38:23.308 INFO:tasks.rados.rados.0.smithi157.stdout:1251: finishing write tid 2 to smithi15736696-45 2024-09-16T03:38:23.308 INFO:tasks.rados.rados.0.smithi157.stdout:1251: finishing write tid 3 to smithi15736696-45 2024-09-16T03:38:23.308 INFO:tasks.rados.rados.0.smithi157.stdout:1251: finishing write tid 4 to smithi15736696-45 2024-09-16T03:38:23.308 INFO:tasks.rados.rados.0.smithi157.stdout:1251: oid 45 updating version 0 to 79 2024-09-16T03:38:23.308 INFO:tasks.rados.rados.0.smithi157.stdout:1251: oid 45 version 79 is already newer than 78 2024-09-16T03:38:23.308 INFO:tasks.rados.rados.0.smithi157.stdout:1251: oid 45 updating version 79 to 80 2024-09-16T03:38:23.308 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 80 (ObjNum 240 snap 0 seq_num 240) dirty exists 2024-09-16T03:38:23.308 INFO:tasks.rados.rados.0.smithi157.stdout:1251: left oid 45 (ObjNum 240 snap 0 seq_num 240) 2024-09-16T03:38:23.308 INFO:tasks.rados.rados.0.smithi157.stdout:1249: done (9 left) 2024-09-16T03:38:23.308 INFO:tasks.rados.rados.0.smithi157.stdout:1251: done (8 left) 2024-09-16T03:38:23.309 INFO:tasks.rados.rados.0.smithi157.stdout:1255: done (7 left) 2024-09-16T03:38:23.309 INFO:tasks.rados.rados.0.smithi157.stdout:1264: write oid 27 current snap is 0 2024-09-16T03:38:23.309 INFO:tasks.rados.rados.0.smithi157.stdout:1264: seq_num 244 ranges {500908=484052,1411724=689184} 2024-09-16T03:38:23.311 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:23.310+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:23.312 INFO:tasks.rados.rados.0.smithi157.stdout:1264: writing smithi15736696-27 from 500908 to 984960 tid 1 2024-09-16T03:38:23.319 INFO:tasks.rados.rados.0.smithi157.stdout:1264: writing smithi15736696-27 from 1411724 to 2100908 tid 2 2024-09-16T03:38:23.319 INFO:tasks.rados.rados.0.smithi157.stdout:1265: read oid 2 snap -1 2024-09-16T03:38:23.319 INFO:tasks.rados.rados.0.smithi157.stdout:1265: expect deleted 2024-09-16T03:38:23.319 INFO:tasks.rados.rados.0.smithi157.stdout:1266: read oid 11 snap -1 2024-09-16T03:38:23.319 INFO:tasks.rados.rados.0.smithi157.stdout:1266: expect (ObjNum 233 snap 0 seq_num 233) 2024-09-16T03:38:23.319 INFO:tasks.rados.rados.0.smithi157.stdout:1267: read oid 48 snap -1 2024-09-16T03:38:23.319 INFO:tasks.rados.rados.0.smithi157.stdout:1267: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-16T03:38:23.319 INFO:tasks.rados.rados.0.smithi157.stdout:1268: read oid 44 snap -1 2024-09-16T03:38:23.320 INFO:tasks.rados.rados.0.smithi157.stdout:1268: expect (ObjNum 219 snap 0 seq_num 219) 2024-09-16T03:38:23.320 INFO:tasks.rados.rados.0.smithi157.stdout:1269: write oid 40 current snap is 0 2024-09-16T03:38:23.320 INFO:tasks.rados.rados.0.smithi157.stdout:1269: seq_num 245 ranges {517715=559301,1870206=247509} 2024-09-16T03:38:23.325 INFO:tasks.rados.rados.0.smithi157.stdout:1269: writing smithi15736696-40 from 517715 to 1077016 tid 1 2024-09-16T03:38:23.332 INFO:tasks.rados.rados.0.smithi157.stdout:1269: writing smithi15736696-40 from 1870206 to 2117715 tid 2 2024-09-16T03:38:23.332 INFO:tasks.rados.rados.0.smithi157.stdout:1270: set_chunk oid 7 target oid 2024-09-16T03:38:23.332 INFO:tasks.rados.rados.0.smithi157.stdout:1270: set_chunk oid 7 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:23.332 INFO:tasks.rados.rados.0.smithi157.stdout:1271: read oid 37 snap -1 2024-09-16T03:38:23.332 INFO:tasks.rados.rados.0.smithi157.stdout:1271: expect (ObjNum 217 snap 0 seq_num 217) 2024-09-16T03:38:23.332 INFO:tasks.rados.rados.0.smithi157.stdout:1257: expect (ObjNum 224 snap 0 seq_num 224) 2024-09-16T03:38:23.332 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 455796 2024-09-16T03:38:23.377 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.14 to 1 (10000) 2024-09-16T03:38:23.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-16T03:38:23.412 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:23.411+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:23.416 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:23.416 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:23.425 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:38:23.442 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:23.502 INFO:tasks.rados.rados.0.smithi157.stdout:1260: expect (ObjNum 236 snap 0 seq_num 236) 2024-09-16T03:38:23.513 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:23.512+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:23.545 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:23.546 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:23.556 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:38:23.615 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:23.613+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:23.660 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:23.660 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:23.668 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:38:23.715 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:23.713+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:23.770 INFO:tasks.rados.rados.0.smithi157.stdout:1263: got (22) Invalid argument 2024-09-16T03:38:23.771 INFO:tasks.rados.rados.0.smithi157.stdout:1258: got (2) No such file or directory 2024-09-16T03:38:23.771 INFO:tasks.rados.rados.0.smithi157.stdout:1258: got expected ENOENT (src dne) 2024-09-16T03:38:23.771 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 80 (ObjNum 129 snap 0 seq_num 3495000344) clean dne 2024-09-16T03:38:23.771 INFO:tasks.rados.rados.0.smithi157.stdout:1262: expect (ObjNum 209 snap 0 seq_num 209) 2024-09-16T03:38:23.812 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:23.812 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:23.816 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:23.814+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:23.819 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:23.817+0000 7f3a1f71e640 -1 osd.6 92 handle_osd_map injecting map crc failure 2024-09-16T03:38:23.821 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:38:23.917 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:23.915+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:23.965 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:23.965 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:23.974 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:38:24.018 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:24.017+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:24.024 INFO:tasks.rados.rados.0.smithi157.stdout:1261: finishing set_chunk to oid 31 2024-09-16T03:38:24.024 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 58 (ObjNum 226 snap 0 seq_num 226) dirty exists 2024-09-16T03:38:24.024 INFO:tasks.rados.rados.0.smithi157.stdout:1266: expect (ObjNum 233 snap 0 seq_num 233) 2024-09-16T03:38:24.028 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1731388 2024-09-16T03:38:24.058 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:24.119 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:24.119 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:24.119 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:24.118+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:24.127 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:38:24.147 INFO:tasks.rados.rados.0.smithi157.stdout:1269: finishing write tid 1 to smithi15736696-40 2024-09-16T03:38:24.147 INFO:tasks.rados.rados.0.smithi157.stdout:1268: expect (ObjNum 219 snap 0 seq_num 219) 2024-09-16T03:38:24.148 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:24.220 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:24.217+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:24.252 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:24.252 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:24.260 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:38:24.321 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:24.319+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:24.359 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:24.364 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:24.364 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:24.373 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:38:24.376 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.15 to 1 (10000) 2024-09-16T03:38:24.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:38:24.395 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T03:38:24.395 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:38:24.421 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:24.420+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:24.466 INFO:tasks.rados.rados.0.smithi157.stdout:1271: expect (ObjNum 217 snap 0 seq_num 217) 2024-09-16T03:38:24.470 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 758864 2024-09-16T03:38:24.495 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:24.495 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:24.505 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:38:24.522 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:24.521+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:24.624 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:24.622+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:24.628 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:24.655 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:24.655 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:24.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:38:24.719 INFO:tasks.rados.rados.0.smithi157.stdout:1267: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-16T03:38:24.725 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:24.723+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:24.746 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:38:24.747 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:38:24.752 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:24.759 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-16T03:38:24.376196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":191270255,"num_objects":108,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":46,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":0,"num_read":2630,"num_read_kb":1682489,"num_write":1649,"num_write_kb":430217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":26334576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":15,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":686744,"kb_used_data":301724,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1131775336,"statfs":{"total":1159641169920,"available":1158937944064,"internally_reserved":0,"allocated":308965376,"data_stored":314150837,"data_compressed":6906899,"data_compressed_allocated":7450624,"data_compressed_original":17088512,"omap_allocated":19135,"internal_metadata":370193729},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1953341,"num_objects":-3,"num_object_clones":0,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":160,"num_read_kb":137171,"num_write":99,"num_write_kb":33395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019452"},"pg_stats":[{"pgid":"4.19","version":"0'0","reported_seq":143,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920381+0000","last_change":"2024-09-16T03:37:52.558405+0000","last_active":"2024-09-16T03:38:23.920381+0000","last_peered":"2024-09-16T03:38:23.920381+0000","last_clean":"2024-09-16T03:38:23.920381+0000","last_became_active":"2024-09-16T03:37:52.558043+0000","last_became_peered":"2024-09-16T03:37:52.558043+0000","last_unstale":"2024-09-16T03:38:23.920381+0000","last_undegraded":"2024-09-16T03:38:23.920381+0000","last_fullsized":"2024-09-16T03:38:23.920381+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:39.955354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.955354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:52.907292+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"80'39","reported_seq":235,"reported_epoch":93,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:23.820070+0000","last_change":"2024-09-16T03:37:54.660680+0000","last_active":"2024-09-16T03:38:23.820070+0000","last_peered":"2024-09-16T03:38:23.820070+0000","last_clean":"2024-09-16T03:37:50.435122+0000","last_became_active":"2024-09-16T03:37:54.659250+0000","last_became_peered":"2024-09-16T03:37:54.659250+0000","last_unstale":"2024-09-16T03:38:23.820070+0000","last_undegraded":"2024-09-16T03:38:23.820070+0000","last_fullsized":"2024-09-16T03:38:23.820070+0000","mapping_epoch":68,"log_start":"79'35","ondisk_log_start":"79'35","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'29","last_scrub_stamp":"2024-09-16T03:37:39.786314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.786314+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":32,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":781,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260558,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20261,"num_write":9,"num_write_kb":2547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"83'41","reported_seq":261,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.375248+0000","last_change":"2024-09-16T03:37:47.616917+0000","last_active":"2024-09-16T03:38:23.375248+0000","last_peered":"2024-09-16T03:38:23.375248+0000","last_clean":"2024-09-16T03:38:23.375248+0000","last_became_active":"2024-09-16T03:37:31.713115+0000","last_became_peered":"2024-09-16T03:37:31.713115+0000","last_unstale":"2024-09-16T03:38:23.375248+0000","last_undegraded":"2024-09-16T03:38:23.375248+0000","last_fullsized":"2024-09-16T03:38:23.375248+0000","mapping_epoch":50,"log_start":"73'38","ondisk_log_start":"73'38","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":37,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:16.011301+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6084101,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":38220,"num_write":6,"num_write_kb":1554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.16","version":"85'61","reported_seq":275,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.420737+0000","last_change":"2024-09-16T03:37:57.546992+0000","last_active":"2024-09-16T03:38:23.420737+0000","last_peered":"2024-09-16T03:38:23.420737+0000","last_clean":"2024-09-16T03:38:23.420737+0000","last_became_active":"2024-09-16T03:37:57.544402+0000","last_became_peered":"2024-09-16T03:37:57.544402+0000","last_unstale":"2024-09-16T03:38:23.420737+0000","last_undegraded":"2024-09-16T03:38:23.420737+0000","last_fullsized":"2024-09-16T03:38:23.420737+0000","mapping_epoch":71,"log_start":"85'58","ondisk_log_start":"85'58","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'32","last_scrub_stamp":"2024-09-16T03:37:35.189364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.189364+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:43.166755+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6857256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":61874,"num_write":25,"num_write_kb":9792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2311899,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.15","version":"91'53","reported_seq":268,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.604989+0000","last_change":"2024-09-16T03:37:47.078728+0000","last_active":"2024-09-16T03:38:21.604989+0000","last_peered":"2024-09-16T03:38:21.604989+0000","last_clean":"2024-09-16T03:38:21.604989+0000","last_became_active":"2024-09-16T03:37:31.349745+0000","last_became_peered":"2024-09-16T03:37:31.349745+0000","last_unstale":"2024-09-16T03:38:21.604989+0000","last_undegraded":"2024-09-16T03:38:21.604989+0000","last_fullsized":"2024-09-16T03:38:21.604989+0000","mapping_epoch":50,"log_start":"88'49","ondisk_log_start":"88'49","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":47,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:01.863665+0000","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7680799,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":70541,"num_write":23,"num_write_kb":6606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.14","version":"79'16","reported_seq":193,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.887426+0000","last_change":"2024-09-16T03:37:39.953719+0000","last_active":"2024-09-16T03:38:20.887426+0000","last_peered":"2024-09-16T03:38:20.887426+0000","last_clean":"2024-09-16T03:38:20.887426+0000","last_became_active":"2024-09-16T03:37:31.149621+0000","last_became_peered":"2024-09-16T03:37:31.149621+0000","last_unstale":"2024-09-16T03:38:20.887426+0000","last_undegraded":"2024-09-16T03:38:20.887426+0000","last_fullsized":"2024-09-16T03:38:20.887426+0000","mapping_epoch":50,"log_start":"57'13","ondisk_log_start":"57'13","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'13","last_scrub_stamp":"2024-09-16T03:37:39.953656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.953656+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":10,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:48.840913+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3159716,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":5791,"num_write":4,"num_write_kb":1127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.13","version":"84'59","reported_seq":278,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.887352+0000","last_change":"2024-09-16T03:37:57.653492+0000","last_active":"2024-09-16T03:38:20.887352+0000","last_peered":"2024-09-16T03:38:20.887352+0000","last_clean":"2024-09-16T03:38:20.887352+0000","last_became_active":"2024-09-16T03:37:57.640119+0000","last_became_peered":"2024-09-16T03:37:57.640119+0000","last_unstale":"2024-09-16T03:38:20.887352+0000","last_undegraded":"2024-09-16T03:38:20.887352+0000","last_fullsized":"2024-09-16T03:38:20.887352+0000","mapping_epoch":71,"log_start":"83'56","ondisk_log_start":"83'56","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'21","last_scrub_stamp":"2024-09-16T03:37:34.916681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.916681+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":54,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:35.558675+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361400,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":56277,"num_write":61,"num_write_kb":14773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4939757,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.12","version":"92'16","reported_seq":215,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920410+0000","last_change":"2024-09-16T03:37:57.652056+0000","last_active":"2024-09-16T03:38:23.920410+0000","last_peered":"2024-09-16T03:38:23.920410+0000","last_clean":"2024-09-16T03:38:23.920410+0000","last_became_active":"2024-09-16T03:37:57.651635+0000","last_became_peered":"2024-09-16T03:37:57.651635+0000","last_unstale":"2024-09-16T03:38:23.920410+0000","last_undegraded":"2024-09-16T03:38:23.920410+0000","last_fullsized":"2024-09-16T03:38:23.920410+0000","mapping_epoch":71,"log_start":"82'13","ondisk_log_start":"82'13","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":13,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:57.128705+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067294,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":31619,"num_write":12,"num_write_kb":3255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1423481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,11],"acting":[14,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.11","version":"72'12","reported_seq":177,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.564663+0000","last_change":"2024-09-16T03:37:50.756196+0000","last_active":"2024-09-16T03:38:20.564663+0000","last_peered":"2024-09-16T03:38:20.564663+0000","last_clean":"2024-09-16T03:38:20.564663+0000","last_became_active":"2024-09-16T03:37:49.624845+0000","last_became_peered":"2024-09-16T03:37:49.624845+0000","last_unstale":"2024-09-16T03:38:20.564663+0000","last_undegraded":"2024-09-16T03:38:20.564663+0000","last_fullsized":"2024-09-16T03:38:20.564663+0000","mapping_epoch":63,"log_start":"62'9","ondisk_log_start":"62'9","created":50,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'9","last_scrub_stamp":"2024-09-16T03:37:50.756120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:50.756120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":9,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:00.529911+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":41724,"num_write":4,"num_write_kb":1177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"85'55","reported_seq":278,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.375401+0000","last_change":"2024-09-16T03:37:58.667607+0000","last_active":"2024-09-16T03:38:23.375401+0000","last_peered":"2024-09-16T03:38:23.375401+0000","last_clean":"2024-09-16T03:38:23.375401+0000","last_became_active":"2024-09-16T03:37:58.567094+0000","last_became_peered":"2024-09-16T03:37:58.567094+0000","last_unstale":"2024-09-16T03:38:23.375401+0000","last_undegraded":"2024-09-16T03:38:23.375401+0000","last_fullsized":"2024-09-16T03:38:23.375401+0000","mapping_epoch":72,"log_start":"73'52","ondisk_log_start":"73'52","created":50,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:46.798884+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":30360,"num_write":9,"num_write_kb":1874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1836181,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":158,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.417482+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:38:23.417482+0000","last_peered":"2024-09-16T03:38:23.417482+0000","last_clean":"2024-09-16T03:38:23.417482+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:38:23.417482+0000","last_undegraded":"2024-09-16T03:38:23.417482+0000","last_fullsized":"2024-09-16T03:38:23.417482+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.232558+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":148,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T03:38:19.355542+0000","last_change":"2024-09-16T03:37:39.701376+0000","last_active":"2024-09-16T03:38:19.355542+0000","last_peered":"2024-09-16T03:38:19.355542+0000","last_clean":"2024-09-16T03:38:19.355542+0000","last_became_active":"2024-09-16T03:37:31.054119+0000","last_became_peered":"2024-09-16T03:37:31.054119+0000","last_unstale":"2024-09-16T03:38:19.355542+0000","last_undegraded":"2024-09-16T03:38:19.355542+0000","last_fullsized":"2024-09-16T03:38:19.355542+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:37:39.701279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.701279+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:03.319077+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":165,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920429+0000","last_change":"2024-09-16T03:38:12.041048+0000","last_active":"2024-09-16T03:38:23.920429+0000","last_peered":"2024-09-16T03:38:23.920429+0000","last_clean":"2024-09-16T03:38:23.920429+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:38:23.920429+0000","last_undegraded":"2024-09-16T03:38:23.920429+0000","last_fullsized":"2024-09-16T03:38:23.920429+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:12.040917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.040917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.877949+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.0","version":"92'105","reported_seq":395,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.534584+0000","last_change":"2024-09-16T03:37:39.774110+0000","last_active":"2024-09-16T03:38:23.534584+0000","last_peered":"2024-09-16T03:38:23.534584+0000","last_clean":"2024-09-16T03:38:23.534584+0000","last_became_active":"2024-09-16T03:37:31.151595+0000","last_became_peered":"2024-09-16T03:37:31.151595+0000","last_unstale":"2024-09-16T03:38:23.534584+0000","last_undegraded":"2024-09-16T03:38:23.534584+0000","last_fullsized":"2024-09-16T03:38:23.534584+0000","mapping_epoch":32,"log_start":"92'102","ondisk_log_start":"92'102","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'52","last_scrub_stamp":"2024-09-16T03:37:39.774014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.774014+0000","objects_scrubbed":5,"log_size":3,"log_dups_size":102,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:05.310361+0000","scrub_duration":647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15361598,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":163,"num_read_kb":161344,"num_write":59,"num_write_kb":17582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.564750+0000","last_change":"2024-09-16T03:37:54.655500+0000","last_active":"2024-09-16T03:38:20.564750+0000","last_peered":"2024-09-16T03:38:20.564750+0000","last_clean":"2024-09-16T03:38:20.564750+0000","last_became_active":"2024-09-16T03:37:54.655185+0000","last_became_peered":"2024-09-16T03:37:54.655185+0000","last_unstale":"2024-09-16T03:38:20.564750+0000","last_undegraded":"2024-09-16T03:38:20.564750+0000","last_fullsized":"2024-09-16T03:38:20.564750+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"86'33","reported_seq":216,"reported_epoch":90,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:20.887371+0000","last_change":"2024-09-16T03:37:55.142030+0000","last_active":"2024-09-16T03:38:20.887371+0000","last_peered":"2024-09-16T03:38:20.887371+0000","last_clean":"2024-09-16T03:37:51.442314+0000","last_became_active":"2024-09-16T03:37:54.760424+0000","last_became_peered":"2024-09-16T03:37:54.760424+0000","last_unstale":"2024-09-16T03:38:20.887371+0000","last_undegraded":"2024-09-16T03:38:20.887371+0000","last_fullsized":"2024-09-16T03:38:20.887371+0000","mapping_epoch":68,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'21","last_scrub_stamp":"2024-09-16T03:37:39.685904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.685904+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":28,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":949,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613505,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":7257,"num_write":14,"num_write_kb":3834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,5],"avail_no_missing":["11","5","8"],"object_location_counts":[{"shards":"5,8,11","objects":1}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.149696+0000","last_change":"2024-09-16T03:37:52.652365+0000","last_active":"2024-09-16T03:38:21.149696+0000","last_peered":"2024-09-16T03:38:21.149696+0000","last_clean":"2024-09-16T03:38:21.149696+0000","last_became_active":"2024-09-16T03:37:52.651907+0000","last_became_peered":"2024-09-16T03:37:52.651907+0000","last_unstale":"2024-09-16T03:38:21.149696+0000","last_undegraded":"2024-09-16T03:38:21.149696+0000","last_fullsized":"2024-09-16T03:38:21.149696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:07.210820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":166,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.397988+0000","last_change":"2024-09-16T03:37:39.853712+0000","last_active":"2024-09-16T03:38:21.397988+0000","last_peered":"2024-09-16T03:38:21.397988+0000","last_clean":"2024-09-16T03:38:21.397988+0000","last_became_active":"2024-09-16T03:37:31.150931+0000","last_became_peered":"2024-09-16T03:37:31.150931+0000","last_unstale":"2024-09-16T03:38:21.397988+0000","last_undegraded":"2024-09-16T03:38:21.397988+0000","last_fullsized":"2024-09-16T03:38:21.397988+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:39.853620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.853620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:57.206393+0000","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"90'174","reported_seq":316,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.398129+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:38:21.398129+0000","last_peered":"2024-09-16T03:38:21.398129+0000","last_clean":"2024-09-16T03:38:21.398129+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:38:21.398129+0000","last_undegraded":"2024-09-16T03:38:21.398129+0000","last_fullsized":"2024-09-16T03:38:21.398129+0000","mapping_epoch":31,"log_start":"87'171","ondisk_log_start":"87'171","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":183,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.679811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18597303,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":153,"num_read_kb":3806,"num_write":208,"num_write_kb":54453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.417413+0000","last_change":"2024-09-16T03:38:12.795978+0000","last_active":"2024-09-16T03:38:23.417413+0000","last_peered":"2024-09-16T03:38:23.417413+0000","last_clean":"2024-09-16T03:38:23.417413+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:38:23.417413+0000","last_undegraded":"2024-09-16T03:38:23.417413+0000","last_fullsized":"2024-09-16T03:38:23.417413+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:38:12.795855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.795855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:13.088740+0000","scrub_duration":501,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.4","version":"87'30","reported_seq":222,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.887534+0000","last_change":"2024-09-16T03:37:46.884372+0000","last_active":"2024-09-16T03:38:20.887534+0000","last_peered":"2024-09-16T03:38:20.887534+0000","last_clean":"2024-09-16T03:38:20.887534+0000","last_became_active":"2024-09-16T03:37:31.249346+0000","last_became_peered":"2024-09-16T03:37:31.249346+0000","last_unstale":"2024-09-16T03:38:20.887534+0000","last_undegraded":"2024-09-16T03:38:20.887534+0000","last_fullsized":"2024-09-16T03:38:20.887534+0000","mapping_epoch":32,"log_start":"82'27","ondisk_log_start":"82'27","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'15","last_scrub_stamp":"2024-09-16T03:37:46.884284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:46.884284+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":27,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:55.612449+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":9455,"num_write":19,"num_write_kb":4676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.0","version":"88'181","reported_seq":332,"reported_epoch":89,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:19.355671+0000","last_change":"2024-09-16T03:37:56.179604+0000","last_active":"2024-09-16T03:38:19.355671+0000","last_peered":"2024-09-16T03:38:19.355671+0000","last_clean":"2024-09-16T03:37:50.666338+0000","last_became_active":"2024-09-16T03:37:55.757919+0000","last_became_peered":"2024-09-16T03:37:55.757919+0000","last_unstale":"2024-09-16T03:38:19.355671+0000","last_undegraded":"2024-09-16T03:38:19.355671+0000","last_fullsized":"2024-09-16T03:38:19.355671+0000","mapping_epoch":69,"log_start":"87'178","ondisk_log_start":"87'178","created":31,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":199,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18575520,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":162,"num_read_kb":6928,"num_write":218,"num_write_kb":63387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,9,1],"avail_no_missing":["4","1","9"],"object_location_counts":[{"shards":"1,4,9","objects":17}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":151,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.397974+0000","last_change":"2024-09-16T03:38:15.949627+0000","last_active":"2024-09-16T03:38:21.397974+0000","last_peered":"2024-09-16T03:38:21.397974+0000","last_clean":"2024-09-16T03:38:21.397974+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:38:21.397974+0000","last_undegraded":"2024-09-16T03:38:21.397974+0000","last_fullsized":"2024-09-16T03:38:21.397974+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:15.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:24.286324+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.7","version":"88'55","reported_seq":262,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.375331+0000","last_change":"2024-09-16T03:37:48.400988+0000","last_active":"2024-09-16T03:38:23.375331+0000","last_peered":"2024-09-16T03:38:23.375331+0000","last_clean":"2024-09-16T03:38:23.375331+0000","last_became_active":"2024-09-16T03:37:31.153814+0000","last_became_peered":"2024-09-16T03:37:31.153814+0000","last_unstale":"2024-09-16T03:38:23.375331+0000","last_undegraded":"2024-09-16T03:38:23.375331+0000","last_fullsized":"2024-09-16T03:38:23.375331+0000","mapping_epoch":32,"log_start":"86'52","ondisk_log_start":"86'52","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'32","last_scrub_stamp":"2024-09-16T03:37:48.400911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:48.400911+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":50,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:10.129191+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6857256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":80936,"num_write":28,"num_write_kb":8438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1","version":"86'130","reported_seq":271,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.249782+0000","last_change":"2024-09-16T03:38:15.890094+0000","last_active":"2024-09-16T03:38:21.249782+0000","last_peered":"2024-09-16T03:38:21.249782+0000","last_clean":"2024-09-16T03:38:21.249782+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:38:21.249782+0000","last_undegraded":"2024-09-16T03:38:21.249782+0000","last_fullsized":"2024-09-16T03:38:21.249782+0000","mapping_epoch":31,"log_start":"85'127","ondisk_log_start":"85'127","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'125","last_scrub_stamp":"2024-09-16T03:38:15.889996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.889996+0000","objects_scrubbed":18,"log_size":3,"log_dups_size":138,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:35.350472+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11382467,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":110,"num_read_kb":743,"num_write":153,"num_write_kb":40662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":158,"reported_epoch":93,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:38:24.257066+0000","last_change":"2024-09-16T03:38:24.155601+0000","last_active":"2024-09-16T03:38:24.257066+0000","last_peered":"2024-09-16T03:38:24.257066+0000","last_clean":"2024-09-16T03:38:24.257066+0000","last_became_active":"2024-09-16T03:37:55.758504+0000","last_became_peered":"2024-09-16T03:37:55.758504+0000","last_unstale":"2024-09-16T03:38:24.257066+0000","last_undegraded":"2024-09-16T03:38:24.257066+0000","last_fullsized":"2024-09-16T03:38:24.257066+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,10],"acting":[8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.6","version":"75'39","reported_seq":268,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.420798+0000","last_change":"2024-09-16T03:37:56.444659+0000","last_active":"2024-09-16T03:38:23.420798+0000","last_peered":"2024-09-16T03:38:23.420798+0000","last_clean":"2024-09-16T03:38:23.420798+0000","last_became_active":"2024-09-16T03:37:56.444134+0000","last_became_peered":"2024-09-16T03:37:56.444134+0000","last_unstale":"2024-09-16T03:38:23.420798+0000","last_undegraded":"2024-09-16T03:38:23.420798+0000","last_fullsized":"2024-09-16T03:38:23.420798+0000","mapping_epoch":70,"log_start":"63'35","ondisk_log_start":"63'35","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'32","last_scrub_stamp":"2024-09-16T03:37:47.613282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.613282+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":34,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:52.751036+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924418,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":24426,"num_write":9,"num_write_kb":2208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1264675,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"89'150","reported_seq":358,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.887390+0000","last_change":"2024-09-16T03:38:01.603352+0000","last_active":"2024-09-16T03:38:20.887390+0000","last_peered":"2024-09-16T03:38:20.887390+0000","last_clean":"2024-09-16T03:38:20.887390+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:38:20.887390+0000","last_undegraded":"2024-09-16T03:38:20.887390+0000","last_fullsized":"2024-09-16T03:38:20.887390+0000","mapping_epoch":74,"log_start":"84'147","ondisk_log_start":"84'147","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":156,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19248469,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":139,"num_read_kb":7648,"num_write":174,"num_write_kb":52654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920256+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:38:23.920256+0000","last_peered":"2024-09-16T03:38:23.920256+0000","last_clean":"2024-09-16T03:38:23.920256+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:38:23.920256+0000","last_undegraded":"2024-09-16T03:38:23.920256+0000","last_fullsized":"2024-09-16T03:38:23.920256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:01.672864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":287,"reported_epoch":90,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:20.887313+0000","last_change":"2024-09-16T03:37:54.760571+0000","last_active":"2024-09-16T03:38:20.887313+0000","last_peered":"2024-09-16T03:38:20.887313+0000","last_clean":"2024-09-16T03:37:51.442238+0000","last_became_active":"2024-09-16T03:37:54.660154+0000","last_became_peered":"2024-09-16T03:37:54.660154+0000","last_unstale":"2024-09-16T03:38:20.887313+0000","last_undegraded":"2024-09-16T03:38:20.887313+0000","last_fullsized":"2024-09-16T03:38:20.887313+0000","mapping_epoch":68,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:36:50.058111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,7],"acting":[11,0,5],"avail_no_missing":["11","0","5"],"object_location_counts":[{"shards":"0,5,11","objects":2}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.5","version":"90'66","reported_seq":278,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.498233+0000","last_change":"2024-09-16T03:37:47.718455+0000","last_active":"2024-09-16T03:38:21.498233+0000","last_peered":"2024-09-16T03:38:21.498233+0000","last_clean":"2024-09-16T03:38:21.498233+0000","last_became_active":"2024-09-16T03:37:31.248101+0000","last_became_peered":"2024-09-16T03:37:31.248101+0000","last_unstale":"2024-09-16T03:38:21.498233+0000","last_undegraded":"2024-09-16T03:38:21.498233+0000","last_fullsized":"2024-09-16T03:38:21.498233+0000","mapping_epoch":32,"log_start":"89'63","ondisk_log_start":"89'63","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'44","last_scrub_stamp":"2024-09-16T03:37:47.718363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.718363+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":62,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:48.218136+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3529470,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":62544,"num_write":24,"num_write_kb":6749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.8","version":"91'62","reported_seq":272,"reported_epoch":93,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:23.820079+0000","last_change":"2024-09-16T03:37:54.958763+0000","last_active":"2024-09-16T03:38:23.820079+0000","last_peered":"2024-09-16T03:38:23.820079+0000","last_clean":"2024-09-16T03:37:41.106904+0000","last_became_active":"2024-09-16T03:37:54.759604+0000","last_became_peered":"2024-09-16T03:37:54.759604+0000","last_unstale":"2024-09-16T03:38:23.820079+0000","last_undegraded":"2024-09-16T03:38:23.820079+0000","last_fullsized":"2024-09-16T03:38:23.820079+0000","mapping_epoch":68,"log_start":"90'59","ondisk_log_start":"90'59","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'38","last_scrub_stamp":"2024-09-16T03:37:40.343801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:40.343801+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3899224,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":58565,"num_write":28,"num_write_kb":8269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":147,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920369+0000","last_change":"2024-09-16T03:37:52.558364+0000","last_active":"2024-09-16T03:38:23.920369+0000","last_peered":"2024-09-16T03:38:23.920369+0000","last_clean":"2024-09-16T03:38:23.920369+0000","last_became_active":"2024-09-16T03:37:52.558058+0000","last_became_peered":"2024-09-16T03:37:52.558058+0000","last_unstale":"2024-09-16T03:38:23.920369+0000","last_undegraded":"2024-09-16T03:38:23.920369+0000","last_fullsized":"2024-09-16T03:38:23.920369+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:46.812106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:46.812106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:50.592391+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.a","version":"92'108","reported_seq":413,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.518020+0000","last_change":"2024-09-16T03:37:06.386519+0000","last_active":"2024-09-16T03:38:23.518020+0000","last_peered":"2024-09-16T03:38:23.518020+0000","last_clean":"2024-09-16T03:38:23.518020+0000","last_became_active":"2024-09-16T03:37:06.286063+0000","last_became_peered":"2024-09-16T03:37:06.286063+0000","last_unstale":"2024-09-16T03:38:23.518020+0000","last_undegraded":"2024-09-16T03:38:23.518020+0000","last_fullsized":"2024-09-16T03:38:23.518020+0000","mapping_epoch":32,"log_start":"92'105","ondisk_log_start":"92'105","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":105,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16437312,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":394,"num_read_kb":298233,"num_write":110,"num_write_kb":33612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.b","version":"86'89","reported_seq":347,"reported_epoch":91,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:21.429878+0000","last_change":"2024-09-16T03:37:55.559928+0000","last_active":"2024-09-16T03:38:21.429878+0000","last_peered":"2024-09-16T03:38:21.429878+0000","last_clean":"2024-09-16T03:37:52.454742+0000","last_became_active":"2024-09-16T03:37:55.559094+0000","last_became_peered":"2024-09-16T03:37:55.559094+0000","last_unstale":"2024-09-16T03:38:21.429878+0000","last_undegraded":"2024-09-16T03:38:21.429878+0000","last_fullsized":"2024-09-16T03:38:21.429878+0000","mapping_epoch":69,"log_start":"85'86","ondisk_log_start":"85'86","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":86,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13143074,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":281,"num_read_kb":257121,"num_write":96,"num_write_kb":30271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,12],"acting":[15,4,9],"avail_no_missing":["15","4","9"],"object_location_counts":[{"shards":"4,9,15","objects":4}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.c","version":"90'55","reported_seq":231,"reported_epoch":92,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:22.711971+0000","last_change":"2024-09-16T03:37:54.763651+0000","last_active":"2024-09-16T03:38:22.711971+0000","last_peered":"2024-09-16T03:38:22.711971+0000","last_clean":"2024-09-16T03:37:50.453398+0000","last_became_active":"2024-09-16T03:37:54.762708+0000","last_became_peered":"2024-09-16T03:37:54.762708+0000","last_unstale":"2024-09-16T03:38:22.711971+0000","last_undegraded":"2024-09-16T03:38:22.711971+0000","last_fullsized":"2024-09-16T03:38:22.711971+0000","mapping_epoch":68,"log_start":"84'51","ondisk_log_start":"84'51","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":51,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3865610,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":29471,"num_write":59,"num_write_kb":16573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,11],"acting":[12,15,1],"avail_no_missing":["12","1","15"],"object_location_counts":[{"shards":"1,12,15","objects":1}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.d","version":"92'14","reported_seq":203,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-16T03:38:22.646482+0000","last_change":"2024-09-16T03:38:17.347261+0000","last_active":"2024-09-16T03:38:22.646482+0000","last_peered":"2024-09-16T03:38:22.646482+0000","last_clean":"2024-09-16T03:38:22.646482+0000","last_became_active":"2024-09-16T03:37:58.450690+0000","last_became_peered":"2024-09-16T03:37:58.450690+0000","last_unstale":"2024-09-16T03:38:22.646482+0000","last_undegraded":"2024-09-16T03:38:22.646482+0000","last_fullsized":"2024-09-16T03:38:22.646482+0000","mapping_epoch":72,"log_start":"91'11","ondisk_log_start":"91'11","created":32,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:46.832564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16968,"num_write":14,"num_write_kb":3863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2160080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.e","version":"87'54","reported_seq":244,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T03:38:19.455824+0000","last_change":"2024-09-16T03:38:11.825753+0000","last_active":"2024-09-16T03:38:19.455824+0000","last_peered":"2024-09-16T03:38:19.455824+0000","last_clean":"2024-09-16T03:38:19.455824+0000","last_became_active":"2024-09-16T03:37:06.037035+0000","last_became_peered":"2024-09-16T03:37:06.037035+0000","last_unstale":"2024-09-16T03:38:19.455824+0000","last_undegraded":"2024-09-16T03:38:19.455824+0000","last_fullsized":"2024-09-16T03:38:19.455824+0000","mapping_epoch":32,"log_start":"86'50","ondisk_log_start":"86'50","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'42","last_scrub_stamp":"2024-09-16T03:38:11.825685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:11.825685+0000","objects_scrubbed":3,"log_size":4,"log_dups_size":50,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:13.309016+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420340,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":92779,"num_write":56,"num_write_kb":16043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"92'81","reported_seq":328,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.375274+0000","last_change":"2024-09-16T03:38:11.380208+0000","last_active":"2024-09-16T03:38:23.375274+0000","last_peered":"2024-09-16T03:38:23.375274+0000","last_clean":"2024-09-16T03:38:23.375274+0000","last_became_active":"2024-09-16T03:37:06.386648+0000","last_became_peered":"2024-09-16T03:37:06.386648+0000","last_unstale":"2024-09-16T03:38:23.375274+0000","last_undegraded":"2024-09-16T03:38:23.375274+0000","last_fullsized":"2024-09-16T03:38:23.375274+0000","mapping_epoch":32,"log_start":"92'78","ondisk_log_start":"92'78","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'64","last_scrub_stamp":"2024-09-16T03:38:11.380094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:11.380094+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":78,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.995078+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7865742,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":207525,"num_write":80,"num_write_kb":21564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":122876109,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":1980,"num_read_kb":1663291,"num_write":749,"num_write_kb":216387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13936073,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":15,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":231383040,"data_stored":230974806,"data_compressed":684382,"data_compressed_allocated":733184,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":78,"acting":78,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":67803759,"num_objects":65,"num_object_clones":0,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":564,"num_read_kb":19125,"num_write":753,"num_write_kb":211156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":124596224,"data_stored":134349436,"data_compressed":8211514,"data_compressed_allocated":8773632,"data_compressed_original":18796544,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":1586784,"data_compressed":93744,"data_compressed_allocated":122880,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247828,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84852,"kb_used_data":50516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94286988,"statfs":{"total":96636764160,"available":96549875712,"internally_reserved":0,"allocated":51728384,"data_stored":52030200,"data_compressed":869744,"data_compressed_allocated":909312,"data_compressed_original":1728512,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247829,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66860,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304980,"statfs":{"total":96636764160,"available":96568299520,"internally_reserved":0,"allocated":34947072,"data_stored":36022264,"data_compressed":747424,"data_compressed_allocated":823296,"data_compressed_original":2371584,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247828,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53784,"kb_used_data":22596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318056,"statfs":{"total":96636764160,"available":96581689344,"internally_reserved":0,"allocated":23138304,"data_stored":23597166,"data_compressed":519317,"data_compressed_allocated":573440,"data_compressed_original":1421312,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280534,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46100,"kb_used_data":14792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325740,"statfs":{"total":96636764160,"available":96589557760,"internally_reserved":0,"allocated":15147008,"data_stored":16233105,"data_compressed":637211,"data_compressed_allocated":704512,"data_compressed_original":2084864,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280534,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280533,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313238,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35928,"kb_used_data":6332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335912,"statfs":{"total":96636764160,"available":96599973888,"internally_reserved":0,"allocated":6483968,"data_stored":6226197,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280534,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55140,"kb_used_data":20524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316700,"statfs":{"total":96636764160,"available":96580300800,"internally_reserved":0,"allocated":21016576,"data_stored":21485657,"data_compressed":457444,"data_compressed_allocated":507904,"data_compressed_original":1302528,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280532,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92132,"kb_used_data":57128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94279708,"statfs":{"total":96636764160,"available":96542420992,"internally_reserved":0,"allocated":58499072,"data_stored":60414403,"data_compressed":1642814,"data_compressed_allocated":1785856,"data_compressed_original":4239360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313238,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280533,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38736,"kb_used_data":7392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333104,"statfs":{"total":96636764160,"available":96597098496,"internally_reserved":0,"allocated":7569408,"data_stored":7319754,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280533,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65712,"kb_used_data":31172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94306128,"statfs":{"total":96636764160,"available":96569475072,"internally_reserved":0,"allocated":31920128,"data_stored":32305674,"data_compressed":934183,"data_compressed_allocated":978944,"data_compressed_original":1724416,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":0,"seq":94489280533,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280534,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66568,"kb_used_data":37764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94305272,"statfs":{"total":96636764160,"available":96568598528,"internally_reserved":0,"allocated":38670336,"data_stored":38542593,"data_compressed":237846,"data_compressed_allocated":258048,"data_compressed_original":544768,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,6,7,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313237,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35092,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336748,"statfs":{"total":96636764160,"available":96600829952,"internally_reserved":0,"allocated":4841472,"data_stored":4579760,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45840,"kb_used_data":14652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326000,"statfs":{"total":96636764160,"available":96589824000,"internally_reserved":0,"allocated":15003648,"data_stored":15394064,"data_compressed":835356,"data_compressed_allocated":872448,"data_compressed_original":1597440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10067968,"data_stored":10938272,"data_compressed":597443,"data_compressed_allocated":651264,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9564160,"data_stored":10197664,"data_compressed":614635,"data_compressed_allocated":651264,"data_compressed_original":1310720,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12275712,"data_stored":12973722,"data_compressed":826836,"data_compressed_allocated":860160,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6619136,"data_stored":7282263,"data_compressed":361389,"data_compressed_allocated":401408,"data_compressed_original":1085440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25677824,"data_stored":28022081,"data_compressed":1682388,"data_compressed_allocated":1818624,"data_compressed_original":4214784,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5505024,"data_stored":6198979,"data_compressed":387998,"data_compressed_allocated":430080,"data_compressed_original":1134592,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10113024,"data_stored":10723430,"data_compressed":838128,"data_compressed_allocated":872448,"data_compressed_original":1507328,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":9564160,"data_stored":10197664,"data_compressed":614635,"data_compressed_allocated":651264,"data_compressed_original":1310720,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":10682368,"data_stored":11705310,"data_compressed":673261,"data_compressed_allocated":733184,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":13021184,"data_stored":13711548,"data_compressed":826836,"data_compressed_allocated":860160,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4280320,"data_stored":4251638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7155712,"data_stored":7111032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4349952,"data_stored":4327588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13676544,"data_stored":13690638,"data_compressed":87535,"data_compressed_allocated":94208,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33468416,"data_stored":33404292,"data_compressed":88404,"data_compressed_allocated":94208,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6242304,"data_stored":6199120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16748544,"data_stored":16807005,"data_compressed":122799,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6864896,"data_stored":6814948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":21086208,"data_stored":21075976,"data_compressed":87535,"data_compressed_allocated":94208,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":9072640,"data_stored":9025673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":37957632,"data_stored":38037194,"data_compressed":229326,"data_compressed_allocated":245760,"data_compressed_original":520192,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":21082112,"data_stored":21005106,"data_compressed":34395,"data_compressed_allocated":36864,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4087808,"data_stored":4063988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5746688,"data_stored":5712182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":38473728,"data_stored":38368386,"data_compressed":34388,"data_compressed_allocated":36864,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:38:24.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:38:24.773 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:24.773 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:24.782 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:38:24.825 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:24.824+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:24.850 INFO:tasks.rados.rados.0.smithi157.stdout:1270: finishing set_chunk to oid 7 2024-09-16T03:38:24.850 INFO:tasks.rados.rados.0.smithi157.stdout:1270: got expected ENOENT (src dne) 2024-09-16T03:38:24.850 INFO:tasks.rados.rados.0.smithi157.stdout:1269: finishing write tid 2 to smithi15736696-40 2024-09-16T03:38:24.850 INFO:tasks.rados.rados.0.smithi157.stdout:1264: finishing write tid 1 to smithi15736696-27 2024-09-16T03:38:24.850 INFO:tasks.rados.rados.0.smithi157.stdout:1264: finishing write tid 2 to smithi15736696-27 2024-09-16T03:38:24.850 INFO:tasks.rados.rados.0.smithi157.stdout:1264: finishing write tid 3 to smithi15736696-27 2024-09-16T03:38:24.850 INFO:tasks.rados.rados.0.smithi157.stdout:1264: finishing write tid 4 to smithi15736696-27 2024-09-16T03:38:24.850 INFO:tasks.rados.rados.0.smithi157.stdout:1264: oid 27 updating version 0 to 107 2024-09-16T03:38:24.850 INFO:tasks.rados.rados.0.smithi157.stdout:1264: oid 27 updating version 107 to 108 2024-09-16T03:38:24.850 INFO:tasks.rados.rados.0.smithi157.stdout:1264: oid 27 version 108 is already newer than 106 2024-09-16T03:38:24.851 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 108 (ObjNum 244 snap 0 seq_num 244) dirty exists 2024-09-16T03:38:24.851 INFO:tasks.rados.rados.0.smithi157.stdout:1264: left oid 27 (ObjNum 244 snap 0 seq_num 244) 2024-09-16T03:38:24.851 INFO:tasks.rados.rados.0.smithi157.stdout:1269: finishing write tid 3 to smithi15736696-40 2024-09-16T03:38:24.851 INFO:tasks.rados.rados.0.smithi157.stdout:1269: finishing write tid 4 to smithi15736696-40 2024-09-16T03:38:24.851 INFO:tasks.rados.rados.0.smithi157.stdout:1269: oid 40 updating version 0 to 105 2024-09-16T03:38:24.851 INFO:tasks.rados.rados.0.smithi157.stdout:1269: oid 40 version 105 is already newer than 103 2024-09-16T03:38:24.851 INFO:tasks.rados.rados.0.smithi157.stdout:1269: oid 40 version 105 is already newer than 104 2024-09-16T03:38:24.851 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 105 (ObjNum 245 snap 0 seq_num 245) dirty exists 2024-09-16T03:38:24.851 INFO:tasks.rados.rados.0.smithi157.stdout:1269: left oid 40 (ObjNum 245 snap 0 seq_num 245) 2024-09-16T03:38:24.851 INFO:tasks.rados.rados.0.smithi157.stdout:1272: read oid 19 snap -1 2024-09-16T03:38:24.851 INFO:tasks.rados.rados.0.smithi157.stdout:1272: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:24.851 INFO:tasks.rados.rados.0.smithi157.stdout:1257: done (15 left) 2024-09-16T03:38:24.852 INFO:tasks.rados.rados.0.smithi157.stdout:1258: done (14 left) 2024-09-16T03:38:24.852 INFO:tasks.rados.rados.0.smithi157.stdout:1259: done (13 left) 2024-09-16T03:38:24.852 INFO:tasks.rados.rados.0.smithi157.stdout:1260: done (12 left) 2024-09-16T03:38:24.853 INFO:tasks.rados.rados.0.smithi157.stdout:1261: done (11 left) 2024-09-16T03:38:24.853 INFO:tasks.rados.rados.0.smithi157.stdout:1262: done (10 left) 2024-09-16T03:38:24.853 INFO:tasks.rados.rados.0.smithi157.stdout:1263: done (9 left) 2024-09-16T03:38:24.853 INFO:tasks.rados.rados.0.smithi157.stdout:1264: done (8 left) 2024-09-16T03:38:24.853 INFO:tasks.rados.rados.0.smithi157.stdout:1265: done (7 left) 2024-09-16T03:38:24.853 INFO:tasks.rados.rados.0.smithi157.stdout:1266: done (6 left) 2024-09-16T03:38:24.853 INFO:tasks.rados.rados.0.smithi157.stdout:1267: done (5 left) 2024-09-16T03:38:24.853 INFO:tasks.rados.rados.0.smithi157.stdout:1268: done (4 left) 2024-09-16T03:38:24.853 INFO:tasks.rados.rados.0.smithi157.stdout:1269: done (3 left) 2024-09-16T03:38:24.854 INFO:tasks.rados.rados.0.smithi157.stdout:1270: done (2 left) 2024-09-16T03:38:24.854 INFO:tasks.rados.rados.0.smithi157.stdout:1271: done (1 left) 2024-09-16T03:38:24.854 INFO:tasks.rados.rados.0.smithi157.stdout:1273: write oid 29 current snap is 0 2024-09-16T03:38:24.854 INFO:tasks.rados.rados.0.smithi157.stdout:1273: seq_num 246 ranges {534522=634550,1812335=322187} 2024-09-16T03:38:24.855 INFO:tasks.rados.rados.0.smithi157.stdout:1273: writing smithi15736696-29 from 534522 to 1169072 tid 1 2024-09-16T03:38:24.861 INFO:tasks.rados.rados.0.smithi157.stdout:1273: writing smithi15736696-29 from 1812335 to 2134522 tid 2 2024-09-16T03:38:24.861 INFO:tasks.rados.rados.0.smithi157.stdout:1274: read oid 46 snap -1 2024-09-16T03:38:24.861 INFO:tasks.rados.rados.0.smithi157.stdout:1274: expect (ObjNum 236 snap 0 seq_num 236) 2024-09-16T03:38:24.861 INFO:tasks.rados.rados.0.smithi157.stdout:1275: tier_flush oid 38 2024-09-16T03:38:24.862 INFO:tasks.rados.rados.0.smithi157.stdout:1275: tier_flush oid 38 snap -1 2024-09-16T03:38:24.862 INFO:tasks.rados.rados.0.smithi157.stdout:1272: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:24.862 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:24.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:38:24.926 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:24.924+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:24.927 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1210722 2024-09-16T03:38:24.936 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:24.936 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:24.946 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:38:25.016 INFO:tasks.rados.rados.0.smithi157.stdout:1273: finishing write tid 1 to smithi15736696-29 2024-09-16T03:38:25.016 INFO:tasks.rados.rados.0.smithi157.stdout:1274: expect (ObjNum 236 snap 0 seq_num 236) 2024-09-16T03:38:25.027 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:25.025+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:25.053 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:25.054 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:25.086 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:38:25.087 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:38:25.088 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:25.088 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:25.098 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-16T03:38:24.376196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":191270255,"num_objects":108,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":46,"num_objects_unfound":0,"num_objects_dirty":107,"num_whiteouts":0,"num_read":2630,"num_read_kb":1682489,"num_write":1649,"num_write_kb":430217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":26334576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":15,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":686744,"kb_used_data":301724,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1131775336,"statfs":{"total":1159641169920,"available":1158937944064,"internally_reserved":0,"allocated":308965376,"data_stored":314150837,"data_compressed":6906899,"data_compressed_allocated":7450624,"data_compressed_original":17088512,"omap_allocated":19135,"internal_metadata":370193729},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1953341,"num_objects":-3,"num_object_clones":0,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":160,"num_read_kb":137171,"num_write":99,"num_write_kb":33395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019452"},"pg_stats":[{"pgid":"4.19","version":"0'0","reported_seq":143,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920381+0000","last_change":"2024-09-16T03:37:52.558405+0000","last_active":"2024-09-16T03:38:23.920381+0000","last_peered":"2024-09-16T03:38:23.920381+0000","last_clean":"2024-09-16T03:38:23.920381+0000","last_became_active":"2024-09-16T03:37:52.558043+0000","last_became_peered":"2024-09-16T03:37:52.558043+0000","last_unstale":"2024-09-16T03:38:23.920381+0000","last_undegraded":"2024-09-16T03:38:23.920381+0000","last_fullsized":"2024-09-16T03:38:23.920381+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:39.955354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.955354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:52.907292+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"80'39","reported_seq":235,"reported_epoch":93,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:23.820070+0000","last_change":"2024-09-16T03:37:54.660680+0000","last_active":"2024-09-16T03:38:23.820070+0000","last_peered":"2024-09-16T03:38:23.820070+0000","last_clean":"2024-09-16T03:37:50.435122+0000","last_became_active":"2024-09-16T03:37:54.659250+0000","last_became_peered":"2024-09-16T03:37:54.659250+0000","last_unstale":"2024-09-16T03:38:23.820070+0000","last_undegraded":"2024-09-16T03:38:23.820070+0000","last_fullsized":"2024-09-16T03:38:23.820070+0000","mapping_epoch":68,"log_start":"79'35","ondisk_log_start":"79'35","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'29","last_scrub_stamp":"2024-09-16T03:37:39.786314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.786314+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":32,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":781,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260558,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20261,"num_write":9,"num_write_kb":2547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"83'41","reported_seq":261,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.375248+0000","last_change":"2024-09-16T03:37:47.616917+0000","last_active":"2024-09-16T03:38:23.375248+0000","last_peered":"2024-09-16T03:38:23.375248+0000","last_clean":"2024-09-16T03:38:23.375248+0000","last_became_active":"2024-09-16T03:37:31.713115+0000","last_became_peered":"2024-09-16T03:37:31.713115+0000","last_unstale":"2024-09-16T03:38:23.375248+0000","last_undegraded":"2024-09-16T03:38:23.375248+0000","last_fullsized":"2024-09-16T03:38:23.375248+0000","mapping_epoch":50,"log_start":"73'38","ondisk_log_start":"73'38","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":37,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:16.011301+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6084101,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":38220,"num_write":6,"num_write_kb":1554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.16","version":"85'61","reported_seq":275,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.420737+0000","last_change":"2024-09-16T03:37:57.546992+0000","last_active":"2024-09-16T03:38:23.420737+0000","last_peered":"2024-09-16T03:38:23.420737+0000","last_clean":"2024-09-16T03:38:23.420737+0000","last_became_active":"2024-09-16T03:37:57.544402+0000","last_became_peered":"2024-09-16T03:37:57.544402+0000","last_unstale":"2024-09-16T03:38:23.420737+0000","last_undegraded":"2024-09-16T03:38:23.420737+0000","last_fullsized":"2024-09-16T03:38:23.420737+0000","mapping_epoch":71,"log_start":"85'58","ondisk_log_start":"85'58","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'32","last_scrub_stamp":"2024-09-16T03:37:35.189364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.189364+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:43.166755+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6857256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":61874,"num_write":25,"num_write_kb":9792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2311899,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.15","version":"91'53","reported_seq":268,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.604989+0000","last_change":"2024-09-16T03:37:47.078728+0000","last_active":"2024-09-16T03:38:21.604989+0000","last_peered":"2024-09-16T03:38:21.604989+0000","last_clean":"2024-09-16T03:38:21.604989+0000","last_became_active":"2024-09-16T03:37:31.349745+0000","last_became_peered":"2024-09-16T03:37:31.349745+0000","last_unstale":"2024-09-16T03:38:21.604989+0000","last_undegraded":"2024-09-16T03:38:21.604989+0000","last_fullsized":"2024-09-16T03:38:21.604989+0000","mapping_epoch":50,"log_start":"88'49","ondisk_log_start":"88'49","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":47,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:01.863665+0000","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7680799,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":70541,"num_write":23,"num_write_kb":6606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.14","version":"79'16","reported_seq":193,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.887426+0000","last_change":"2024-09-16T03:37:39.953719+0000","last_active":"2024-09-16T03:38:20.887426+0000","last_peered":"2024-09-16T03:38:20.887426+0000","last_clean":"2024-09-16T03:38:20.887426+0000","last_became_active":"2024-09-16T03:37:31.149621+0000","last_became_peered":"2024-09-16T03:37:31.149621+0000","last_unstale":"2024-09-16T03:38:20.887426+0000","last_undegraded":"2024-09-16T03:38:20.887426+0000","last_fullsized":"2024-09-16T03:38:20.887426+0000","mapping_epoch":50,"log_start":"57'13","ondisk_log_start":"57'13","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'13","last_scrub_stamp":"2024-09-16T03:37:39.953656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.953656+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":10,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:48.840913+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3159716,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":5791,"num_write":4,"num_write_kb":1127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.13","version":"84'59","reported_seq":278,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.887352+0000","last_change":"2024-09-16T03:37:57.653492+0000","last_active":"2024-09-16T03:38:20.887352+0000","last_peered":"2024-09-16T03:38:20.887352+0000","last_clean":"2024-09-16T03:38:20.887352+0000","last_became_active":"2024-09-16T03:37:57.640119+0000","last_became_peered":"2024-09-16T03:37:57.640119+0000","last_unstale":"2024-09-16T03:38:20.887352+0000","last_undegraded":"2024-09-16T03:38:20.887352+0000","last_fullsized":"2024-09-16T03:38:20.887352+0000","mapping_epoch":71,"log_start":"83'56","ondisk_log_start":"83'56","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'21","last_scrub_stamp":"2024-09-16T03:37:34.916681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.916681+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":54,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:35.558675+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361400,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":56277,"num_write":61,"num_write_kb":14773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4939757,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.12","version":"92'16","reported_seq":215,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920410+0000","last_change":"2024-09-16T03:37:57.652056+0000","last_active":"2024-09-16T03:38:23.920410+0000","last_peered":"2024-09-16T03:38:23.920410+0000","last_clean":"2024-09-16T03:38:23.920410+0000","last_became_active":"2024-09-16T03:37:57.651635+0000","last_became_peered":"2024-09-16T03:37:57.651635+0000","last_unstale":"2024-09-16T03:38:23.920410+0000","last_undegraded":"2024-09-16T03:38:23.920410+0000","last_fullsized":"2024-09-16T03:38:23.920410+0000","mapping_epoch":71,"log_start":"82'13","ondisk_log_start":"82'13","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":13,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:57.128705+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067294,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":31619,"num_write":12,"num_write_kb":3255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1423481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,11],"acting":[14,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.11","version":"72'12","reported_seq":177,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.564663+0000","last_change":"2024-09-16T03:37:50.756196+0000","last_active":"2024-09-16T03:38:20.564663+0000","last_peered":"2024-09-16T03:38:20.564663+0000","last_clean":"2024-09-16T03:38:20.564663+0000","last_became_active":"2024-09-16T03:37:49.624845+0000","last_became_peered":"2024-09-16T03:37:49.624845+0000","last_unstale":"2024-09-16T03:38:20.564663+0000","last_undegraded":"2024-09-16T03:38:20.564663+0000","last_fullsized":"2024-09-16T03:38:20.564663+0000","mapping_epoch":63,"log_start":"62'9","ondisk_log_start":"62'9","created":50,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'9","last_scrub_stamp":"2024-09-16T03:37:50.756120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:50.756120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":9,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:00.529911+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":41724,"num_write":4,"num_write_kb":1177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"85'55","reported_seq":278,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.375401+0000","last_change":"2024-09-16T03:37:58.667607+0000","last_active":"2024-09-16T03:38:23.375401+0000","last_peered":"2024-09-16T03:38:23.375401+0000","last_clean":"2024-09-16T03:38:23.375401+0000","last_became_active":"2024-09-16T03:37:58.567094+0000","last_became_peered":"2024-09-16T03:37:58.567094+0000","last_unstale":"2024-09-16T03:38:23.375401+0000","last_undegraded":"2024-09-16T03:38:23.375401+0000","last_fullsized":"2024-09-16T03:38:23.375401+0000","mapping_epoch":72,"log_start":"73'52","ondisk_log_start":"73'52","created":50,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:46.798884+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":30360,"num_write":9,"num_write_kb":1874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1836181,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":158,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.417482+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:38:23.417482+0000","last_peered":"2024-09-16T03:38:23.417482+0000","last_clean":"2024-09-16T03:38:23.417482+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:38:23.417482+0000","last_undegraded":"2024-09-16T03:38:23.417482+0000","last_fullsized":"2024-09-16T03:38:23.417482+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.232558+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":148,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T03:38:19.355542+0000","last_change":"2024-09-16T03:37:39.701376+0000","last_active":"2024-09-16T03:38:19.355542+0000","last_peered":"2024-09-16T03:38:19.355542+0000","last_clean":"2024-09-16T03:38:19.355542+0000","last_became_active":"2024-09-16T03:37:31.054119+0000","last_became_peered":"2024-09-16T03:37:31.054119+0000","last_unstale":"2024-09-16T03:38:19.355542+0000","last_undegraded":"2024-09-16T03:38:19.355542+0000","last_fullsized":"2024-09-16T03:38:19.355542+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:37:39.701279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.701279+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:03.319077+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":165,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920429+0000","last_change":"2024-09-16T03:38:12.041048+0000","last_active":"2024-09-16T03:38:23.920429+0000","last_peered":"2024-09-16T03:38:23.920429+0000","last_clean":"2024-09-16T03:38:23.920429+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:38:23.920429+0000","last_undegraded":"2024-09-16T03:38:23.920429+0000","last_fullsized":"2024-09-16T03:38:23.920429+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:12.040917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.040917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.877949+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.0","version":"92'105","reported_seq":395,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.534584+0000","last_change":"2024-09-16T03:37:39.774110+0000","last_active":"2024-09-16T03:38:23.534584+0000","last_peered":"2024-09-16T03:38:23.534584+0000","last_clean":"2024-09-16T03:38:23.534584+0000","last_became_active":"2024-09-16T03:37:31.151595+0000","last_became_peered":"2024-09-16T03:37:31.151595+0000","last_unstale":"2024-09-16T03:38:23.534584+0000","last_undegraded":"2024-09-16T03:38:23.534584+0000","last_fullsized":"2024-09-16T03:38:23.534584+0000","mapping_epoch":32,"log_start":"92'102","ondisk_log_start":"92'102","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'52","last_scrub_stamp":"2024-09-16T03:37:39.774014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.774014+0000","objects_scrubbed":5,"log_size":3,"log_dups_size":102,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:05.310361+0000","scrub_duration":647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15361598,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":163,"num_read_kb":161344,"num_write":59,"num_write_kb":17582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.564750+0000","last_change":"2024-09-16T03:37:54.655500+0000","last_active":"2024-09-16T03:38:20.564750+0000","last_peered":"2024-09-16T03:38:20.564750+0000","last_clean":"2024-09-16T03:38:20.564750+0000","last_became_active":"2024-09-16T03:37:54.655185+0000","last_became_peered":"2024-09-16T03:37:54.655185+0000","last_unstale":"2024-09-16T03:38:20.564750+0000","last_undegraded":"2024-09-16T03:38:20.564750+0000","last_fullsized":"2024-09-16T03:38:20.564750+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"86'33","reported_seq":216,"reported_epoch":90,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:20.887371+0000","last_change":"2024-09-16T03:37:55.142030+0000","last_active":"2024-09-16T03:38:20.887371+0000","last_peered":"2024-09-16T03:38:20.887371+0000","last_clean":"2024-09-16T03:37:51.442314+0000","last_became_active":"2024-09-16T03:37:54.760424+0000","last_became_peered":"2024-09-16T03:37:54.760424+0000","last_unstale":"2024-09-16T03:38:20.887371+0000","last_undegraded":"2024-09-16T03:38:20.887371+0000","last_fullsized":"2024-09-16T03:38:20.887371+0000","mapping_epoch":68,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'21","last_scrub_stamp":"2024-09-16T03:37:39.685904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.685904+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":28,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":949,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613505,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":7257,"num_write":14,"num_write_kb":3834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,5],"avail_no_missing":["11","5","8"],"object_location_counts":[{"shards":"5,8,11","objects":1}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.149696+0000","last_change":"2024-09-16T03:37:52.652365+0000","last_active":"2024-09-16T03:38:21.149696+0000","last_peered":"2024-09-16T03:38:21.149696+0000","last_clean":"2024-09-16T03:38:21.149696+0000","last_became_active":"2024-09-16T03:37:52.651907+0000","last_became_peered":"2024-09-16T03:37:52.651907+0000","last_unstale":"2024-09-16T03:38:21.149696+0000","last_undegraded":"2024-09-16T03:38:21.149696+0000","last_fullsized":"2024-09-16T03:38:21.149696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:07.210820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":166,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.397988+0000","last_change":"2024-09-16T03:37:39.853712+0000","last_active":"2024-09-16T03:38:21.397988+0000","last_peered":"2024-09-16T03:38:21.397988+0000","last_clean":"2024-09-16T03:38:21.397988+0000","last_became_active":"2024-09-16T03:37:31.150931+0000","last_became_peered":"2024-09-16T03:37:31.150931+0000","last_unstale":"2024-09-16T03:38:21.397988+0000","last_undegraded":"2024-09-16T03:38:21.397988+0000","last_fullsized":"2024-09-16T03:38:21.397988+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:39.853620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.853620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:57.206393+0000","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"90'174","reported_seq":316,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.398129+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:38:21.398129+0000","last_peered":"2024-09-16T03:38:21.398129+0000","last_clean":"2024-09-16T03:38:21.398129+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:38:21.398129+0000","last_undegraded":"2024-09-16T03:38:21.398129+0000","last_fullsized":"2024-09-16T03:38:21.398129+0000","mapping_epoch":31,"log_start":"87'171","ondisk_log_start":"87'171","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":183,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.679811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18597303,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":153,"num_read_kb":3806,"num_write":208,"num_write_kb":54453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.417413+0000","last_change":"2024-09-16T03:38:12.795978+0000","last_active":"2024-09-16T03:38:23.417413+0000","last_peered":"2024-09-16T03:38:23.417413+0000","last_clean":"2024-09-16T03:38:23.417413+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:38:23.417413+0000","last_undegraded":"2024-09-16T03:38:23.417413+0000","last_fullsized":"2024-09-16T03:38:23.417413+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:38:12.795855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.795855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:13.088740+0000","scrub_duration":501,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.4","version":"87'30","reported_seq":222,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.887534+0000","last_change":"2024-09-16T03:37:46.884372+0000","last_active":"2024-09-16T03:38:20.887534+0000","last_peered":"2024-09-16T03:38:20.887534+0000","last_clean":"2024-09-16T03:38:20.887534+0000","last_became_active":"2024-09-16T03:37:31.249346+0000","last_became_peered":"2024-09-16T03:37:31.249346+0000","last_unstale":"2024-09-16T03:38:20.887534+0000","last_undegraded":"2024-09-16T03:38:20.887534+0000","last_fullsized":"2024-09-16T03:38:20.887534+0000","mapping_epoch":32,"log_start":"82'27","ondisk_log_start":"82'27","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'15","last_scrub_stamp":"2024-09-16T03:37:46.884284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:46.884284+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":27,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:55.612449+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":9455,"num_write":19,"num_write_kb":4676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.0","version":"88'181","reported_seq":332,"reported_epoch":89,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:19.355671+0000","last_change":"2024-09-16T03:37:56.179604+0000","last_active":"2024-09-16T03:38:19.355671+0000","last_peered":"2024-09-16T03:38:19.355671+0000","last_clean":"2024-09-16T03:37:50.666338+0000","last_became_active":"2024-09-16T03:37:55.757919+0000","last_became_peered":"2024-09-16T03:37:55.757919+0000","last_unstale":"2024-09-16T03:38:19.355671+0000","last_undegraded":"2024-09-16T03:38:19.355671+0000","last_fullsized":"2024-09-16T03:38:19.355671+0000","mapping_epoch":69,"log_start":"87'178","ondisk_log_start":"87'178","created":31,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":199,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18575520,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":162,"num_read_kb":6928,"num_write":218,"num_write_kb":63387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,9,1],"avail_no_missing":["4","1","9"],"object_location_counts":[{"shards":"1,4,9","objects":17}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":151,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.397974+0000","last_change":"2024-09-16T03:38:15.949627+0000","last_active":"2024-09-16T03:38:21.397974+0000","last_peered":"2024-09-16T03:38:21.397974+0000","last_clean":"2024-09-16T03:38:21.397974+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:38:21.397974+0000","last_undegraded":"2024-09-16T03:38:21.397974+0000","last_fullsized":"2024-09-16T03:38:21.397974+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:15.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:24.286324+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.7","version":"88'55","reported_seq":262,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.375331+0000","last_change":"2024-09-16T03:37:48.400988+0000","last_active":"2024-09-16T03:38:23.375331+0000","last_peered":"2024-09-16T03:38:23.375331+0000","last_clean":"2024-09-16T03:38:23.375331+0000","last_became_active":"2024-09-16T03:37:31.153814+0000","last_became_peered":"2024-09-16T03:37:31.153814+0000","last_unstale":"2024-09-16T03:38:23.375331+0000","last_undegraded":"2024-09-16T03:38:23.375331+0000","last_fullsized":"2024-09-16T03:38:23.375331+0000","mapping_epoch":32,"log_start":"86'52","ondisk_log_start":"86'52","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'32","last_scrub_stamp":"2024-09-16T03:37:48.400911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:48.400911+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":50,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:10.129191+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6857256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":80936,"num_write":28,"num_write_kb":8438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1","version":"86'130","reported_seq":271,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.249782+0000","last_change":"2024-09-16T03:38:15.890094+0000","last_active":"2024-09-16T03:38:21.249782+0000","last_peered":"2024-09-16T03:38:21.249782+0000","last_clean":"2024-09-16T03:38:21.249782+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:38:21.249782+0000","last_undegraded":"2024-09-16T03:38:21.249782+0000","last_fullsized":"2024-09-16T03:38:21.249782+0000","mapping_epoch":31,"log_start":"85'127","ondisk_log_start":"85'127","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'125","last_scrub_stamp":"2024-09-16T03:38:15.889996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.889996+0000","objects_scrubbed":18,"log_size":3,"log_dups_size":138,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:35.350472+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11382467,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":110,"num_read_kb":743,"num_write":153,"num_write_kb":40662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":158,"reported_epoch":93,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:38:24.257066+0000","last_change":"2024-09-16T03:38:24.155601+0000","last_active":"2024-09-16T03:38:24.257066+0000","last_peered":"2024-09-16T03:38:24.257066+0000","last_clean":"2024-09-16T03:38:24.257066+0000","last_became_active":"2024-09-16T03:37:55.758504+0000","last_became_peered":"2024-09-16T03:37:55.758504+0000","last_unstale":"2024-09-16T03:38:24.257066+0000","last_undegraded":"2024-09-16T03:38:24.257066+0000","last_fullsized":"2024-09-16T03:38:24.257066+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,10],"acting":[8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.6","version":"75'39","reported_seq":268,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.420798+0000","last_change":"2024-09-16T03:37:56.444659+0000","last_active":"2024-09-16T03:38:23.420798+0000","last_peered":"2024-09-16T03:38:23.420798+0000","last_clean":"2024-09-16T03:38:23.420798+0000","last_became_active":"2024-09-16T03:37:56.444134+0000","last_became_peered":"2024-09-16T03:37:56.444134+0000","last_unstale":"2024-09-16T03:38:23.420798+0000","last_undegraded":"2024-09-16T03:38:23.420798+0000","last_fullsized":"2024-09-16T03:38:23.420798+0000","mapping_epoch":70,"log_start":"63'35","ondisk_log_start":"63'35","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'32","last_scrub_stamp":"2024-09-16T03:37:47.613282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.613282+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":34,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:52.751036+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924418,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":24426,"num_write":9,"num_write_kb":2208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1264675,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"89'150","reported_seq":358,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.887390+0000","last_change":"2024-09-16T03:38:01.603352+0000","last_active":"2024-09-16T03:38:20.887390+0000","last_peered":"2024-09-16T03:38:20.887390+0000","last_clean":"2024-09-16T03:38:20.887390+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:38:20.887390+0000","last_undegraded":"2024-09-16T03:38:20.887390+0000","last_fullsized":"2024-09-16T03:38:20.887390+0000","mapping_epoch":74,"log_start":"84'147","ondisk_log_start":"84'147","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":156,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19248469,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":139,"num_read_kb":7648,"num_write":174,"num_write_kb":52654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920256+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:38:23.920256+0000","last_peered":"2024-09-16T03:38:23.920256+0000","last_clean":"2024-09-16T03:38:23.920256+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:38:23.920256+0000","last_undegraded":"2024-09-16T03:38:23.920256+0000","last_fullsized":"2024-09-16T03:38:23.920256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:01.672864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":287,"reported_epoch":90,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:20.887313+0000","last_change":"2024-09-16T03:37:54.760571+0000","last_active":"2024-09-16T03:38:20.887313+0000","last_peered":"2024-09-16T03:38:20.887313+0000","last_clean":"2024-09-16T03:37:51.442238+0000","last_became_active":"2024-09-16T03:37:54.660154+0000","last_became_peered":"2024-09-16T03:37:54.660154+0000","last_unstale":"2024-09-16T03:38:20.887313+0000","last_undegraded":"2024-09-16T03:38:20.887313+0000","last_fullsized":"2024-09-16T03:38:20.887313+0000","mapping_epoch":68,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:36:50.058111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,7],"acting":[11,0,5],"avail_no_missing":["11","0","5"],"object_location_counts":[{"shards":"0,5,11","objects":2}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.5","version":"90'66","reported_seq":278,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.498233+0000","last_change":"2024-09-16T03:37:47.718455+0000","last_active":"2024-09-16T03:38:21.498233+0000","last_peered":"2024-09-16T03:38:21.498233+0000","last_clean":"2024-09-16T03:38:21.498233+0000","last_became_active":"2024-09-16T03:37:31.248101+0000","last_became_peered":"2024-09-16T03:37:31.248101+0000","last_unstale":"2024-09-16T03:38:21.498233+0000","last_undegraded":"2024-09-16T03:38:21.498233+0000","last_fullsized":"2024-09-16T03:38:21.498233+0000","mapping_epoch":32,"log_start":"89'63","ondisk_log_start":"89'63","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'44","last_scrub_stamp":"2024-09-16T03:37:47.718363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.718363+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":62,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:48.218136+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3529470,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":62544,"num_write":24,"num_write_kb":6749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.8","version":"91'62","reported_seq":272,"reported_epoch":93,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:23.820079+0000","last_change":"2024-09-16T03:37:54.958763+0000","last_active":"2024-09-16T03:38:23.820079+0000","last_peered":"2024-09-16T03:38:23.820079+0000","last_clean":"2024-09-16T03:37:41.106904+0000","last_became_active":"2024-09-16T03:37:54.759604+0000","last_became_peered":"2024-09-16T03:37:54.759604+0000","last_unstale":"2024-09-16T03:38:23.820079+0000","last_undegraded":"2024-09-16T03:38:23.820079+0000","last_fullsized":"2024-09-16T03:38:23.820079+0000","mapping_epoch":68,"log_start":"90'59","ondisk_log_start":"90'59","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'38","last_scrub_stamp":"2024-09-16T03:37:40.343801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:40.343801+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3899224,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":58565,"num_write":28,"num_write_kb":8269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":147,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920369+0000","last_change":"2024-09-16T03:37:52.558364+0000","last_active":"2024-09-16T03:38:23.920369+0000","last_peered":"2024-09-16T03:38:23.920369+0000","last_clean":"2024-09-16T03:38:23.920369+0000","last_became_active":"2024-09-16T03:37:52.558058+0000","last_became_peered":"2024-09-16T03:37:52.558058+0000","last_unstale":"2024-09-16T03:38:23.920369+0000","last_undegraded":"2024-09-16T03:38:23.920369+0000","last_fullsized":"2024-09-16T03:38:23.920369+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:46.812106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:46.812106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:50.592391+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.a","version":"92'108","reported_seq":413,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.518020+0000","last_change":"2024-09-16T03:37:06.386519+0000","last_active":"2024-09-16T03:38:23.518020+0000","last_peered":"2024-09-16T03:38:23.518020+0000","last_clean":"2024-09-16T03:38:23.518020+0000","last_became_active":"2024-09-16T03:37:06.286063+0000","last_became_peered":"2024-09-16T03:37:06.286063+0000","last_unstale":"2024-09-16T03:38:23.518020+0000","last_undegraded":"2024-09-16T03:38:23.518020+0000","last_fullsized":"2024-09-16T03:38:23.518020+0000","mapping_epoch":32,"log_start":"92'105","ondisk_log_start":"92'105","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":105,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16437312,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":394,"num_read_kb":298233,"num_write":110,"num_write_kb":33612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.b","version":"86'89","reported_seq":347,"reported_epoch":91,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:21.429878+0000","last_change":"2024-09-16T03:37:55.559928+0000","last_active":"2024-09-16T03:38:21.429878+0000","last_peered":"2024-09-16T03:38:21.429878+0000","last_clean":"2024-09-16T03:37:52.454742+0000","last_became_active":"2024-09-16T03:37:55.559094+0000","last_became_peered":"2024-09-16T03:37:55.559094+0000","last_unstale":"2024-09-16T03:38:21.429878+0000","last_undegraded":"2024-09-16T03:38:21.429878+0000","last_fullsized":"2024-09-16T03:38:21.429878+0000","mapping_epoch":69,"log_start":"85'86","ondisk_log_start":"85'86","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":86,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13143074,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":281,"num_read_kb":257121,"num_write":96,"num_write_kb":30271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,12],"acting":[15,4,9],"avail_no_missing":["15","4","9"],"object_location_counts":[{"shards":"4,9,15","objects":4}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.c","version":"90'55","reported_seq":231,"reported_epoch":92,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:22.711971+0000","last_change":"2024-09-16T03:37:54.763651+0000","last_active":"2024-09-16T03:38:22.711971+0000","last_peered":"2024-09-16T03:38:22.711971+0000","last_clean":"2024-09-16T03:37:50.453398+0000","last_became_active":"2024-09-16T03:37:54.762708+0000","last_became_peered":"2024-09-16T03:37:54.762708+0000","last_unstale":"2024-09-16T03:38:22.711971+0000","last_undegraded":"2024-09-16T03:38:22.711971+0000","last_fullsized":"2024-09-16T03:38:22.711971+0000","mapping_epoch":68,"log_start":"84'51","ondisk_log_start":"84'51","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":51,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3865610,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":29471,"num_write":59,"num_write_kb":16573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,11],"acting":[12,15,1],"avail_no_missing":["12","1","15"],"object_location_counts":[{"shards":"1,12,15","objects":1}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.d","version":"92'14","reported_seq":203,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-16T03:38:22.646482+0000","last_change":"2024-09-16T03:38:17.347261+0000","last_active":"2024-09-16T03:38:22.646482+0000","last_peered":"2024-09-16T03:38:22.646482+0000","last_clean":"2024-09-16T03:38:22.646482+0000","last_became_active":"2024-09-16T03:37:58.450690+0000","last_became_peered":"2024-09-16T03:37:58.450690+0000","last_unstale":"2024-09-16T03:38:22.646482+0000","last_undegraded":"2024-09-16T03:38:22.646482+0000","last_fullsized":"2024-09-16T03:38:22.646482+0000","mapping_epoch":72,"log_start":"91'11","ondisk_log_start":"91'11","created":32,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:46.832564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16968,"num_write":14,"num_write_kb":3863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2160080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.e","version":"87'54","reported_seq":244,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-16T03:38:19.455824+0000","last_change":"2024-09-16T03:38:11.825753+0000","last_active":"2024-09-16T03:38:19.455824+0000","last_peered":"2024-09-16T03:38:19.455824+0000","last_clean":"2024-09-16T03:38:19.455824+0000","last_became_active":"2024-09-16T03:37:06.037035+0000","last_became_peered":"2024-09-16T03:37:06.037035+0000","last_unstale":"2024-09-16T03:38:19.455824+0000","last_undegraded":"2024-09-16T03:38:19.455824+0000","last_fullsized":"2024-09-16T03:38:19.455824+0000","mapping_epoch":32,"log_start":"86'50","ondisk_log_start":"86'50","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'42","last_scrub_stamp":"2024-09-16T03:38:11.825685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:11.825685+0000","objects_scrubbed":3,"log_size":4,"log_dups_size":50,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:13.309016+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420340,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":112,"num_read_kb":92779,"num_write":56,"num_write_kb":16043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"92'81","reported_seq":328,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.375274+0000","last_change":"2024-09-16T03:38:11.380208+0000","last_active":"2024-09-16T03:38:23.375274+0000","last_peered":"2024-09-16T03:38:23.375274+0000","last_clean":"2024-09-16T03:38:23.375274+0000","last_became_active":"2024-09-16T03:37:06.386648+0000","last_became_peered":"2024-09-16T03:37:06.386648+0000","last_unstale":"2024-09-16T03:38:23.375274+0000","last_undegraded":"2024-09-16T03:38:23.375274+0000","last_fullsized":"2024-09-16T03:38:23.375274+0000","mapping_epoch":32,"log_start":"92'78","ondisk_log_start":"92'78","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'64","last_scrub_stamp":"2024-09-16T03:38:11.380094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:11.380094+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":78,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.995078+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7865742,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":207525,"num_write":80,"num_write_kb":21564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":122876109,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":1980,"num_read_kb":1663291,"num_write":749,"num_write_kb":216387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13936073,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":15,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":231383040,"data_stored":230974806,"data_compressed":684382,"data_compressed_allocated":733184,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":78,"acting":78,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":67803759,"num_objects":65,"num_object_clones":0,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":564,"num_read_kb":19125,"num_write":753,"num_write_kb":211156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":124596224,"data_stored":134349436,"data_compressed":8211514,"data_compressed_allocated":8773632,"data_compressed_original":18796544,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":1586784,"data_compressed":93744,"data_compressed_allocated":122880,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247828,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84852,"kb_used_data":50516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94286988,"statfs":{"total":96636764160,"available":96549875712,"internally_reserved":0,"allocated":51728384,"data_stored":52030200,"data_compressed":869744,"data_compressed_allocated":909312,"data_compressed_original":1728512,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247829,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66860,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304980,"statfs":{"total":96636764160,"available":96568299520,"internally_reserved":0,"allocated":34947072,"data_stored":36022264,"data_compressed":747424,"data_compressed_allocated":823296,"data_compressed_original":2371584,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247828,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53784,"kb_used_data":22596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318056,"statfs":{"total":96636764160,"available":96581689344,"internally_reserved":0,"allocated":23138304,"data_stored":23597166,"data_compressed":519317,"data_compressed_allocated":573440,"data_compressed_original":1421312,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280534,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46100,"kb_used_data":14792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325740,"statfs":{"total":96636764160,"available":96589557760,"internally_reserved":0,"allocated":15147008,"data_stored":16233105,"data_compressed":637211,"data_compressed_allocated":704512,"data_compressed_original":2084864,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280534,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280533,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313238,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35928,"kb_used_data":6332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335912,"statfs":{"total":96636764160,"available":96599973888,"internally_reserved":0,"allocated":6483968,"data_stored":6226197,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280534,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55140,"kb_used_data":20524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316700,"statfs":{"total":96636764160,"available":96580300800,"internally_reserved":0,"allocated":21016576,"data_stored":21485657,"data_compressed":457444,"data_compressed_allocated":507904,"data_compressed_original":1302528,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280532,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92132,"kb_used_data":57128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94279708,"statfs":{"total":96636764160,"available":96542420992,"internally_reserved":0,"allocated":58499072,"data_stored":60414403,"data_compressed":1642814,"data_compressed_allocated":1785856,"data_compressed_original":4239360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313238,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280533,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38736,"kb_used_data":7392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333104,"statfs":{"total":96636764160,"available":96597098496,"internally_reserved":0,"allocated":7569408,"data_stored":7319754,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280533,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65712,"kb_used_data":31172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94306128,"statfs":{"total":96636764160,"available":96569475072,"internally_reserved":0,"allocated":31920128,"data_stored":32305674,"data_compressed":934183,"data_compressed_allocated":978944,"data_compressed_original":1724416,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":0,"seq":94489280533,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280534,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66568,"kb_used_data":37764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94305272,"statfs":{"total":96636764160,"available":96568598528,"internally_reserved":0,"allocated":38670336,"data_stored":38542593,"data_compressed":237846,"data_compressed_allocated":258048,"data_compressed_original":544768,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,6,7,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313237,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35092,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336748,"statfs":{"total":96636764160,"available":96600829952,"internally_reserved":0,"allocated":4841472,"data_stored":4579760,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45840,"kb_used_data":14652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326000,"statfs":{"total":96636764160,"available":96589824000,"internally_reserved":0,"allocated":15003648,"data_stored":15394064,"data_compressed":835356,"data_compressed_allocated":872448,"data_compressed_original":1597440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10067968,"data_stored":10938272,"data_compressed":597443,"data_compressed_allocated":651264,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9564160,"data_stored":10197664,"data_compressed":614635,"data_compressed_allocated":651264,"data_compressed_original":1310720,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12275712,"data_stored":12973722,"data_compressed":826836,"data_compressed_allocated":860160,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6619136,"data_stored":7282263,"data_compressed":361389,"data_compressed_allocated":401408,"data_compressed_original":1085440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25677824,"data_stored":28022081,"data_compressed":1682388,"data_compressed_allocated":1818624,"data_compressed_original":4214784,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5505024,"data_stored":6198979,"data_compressed":387998,"data_compressed_allocated":430080,"data_compressed_original":1134592,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10113024,"data_stored":10723430,"data_compressed":838128,"data_compressed_allocated":872448,"data_compressed_original":1507328,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":9564160,"data_stored":10197664,"data_compressed":614635,"data_compressed_allocated":651264,"data_compressed_original":1310720,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":10682368,"data_stored":11705310,"data_compressed":673261,"data_compressed_allocated":733184,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":13021184,"data_stored":13711548,"data_compressed":826836,"data_compressed_allocated":860160,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4280320,"data_stored":4251638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7155712,"data_stored":7111032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4349952,"data_stored":4327588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13676544,"data_stored":13690638,"data_compressed":87535,"data_compressed_allocated":94208,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33468416,"data_stored":33404292,"data_compressed":88404,"data_compressed_allocated":94208,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6242304,"data_stored":6199120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16748544,"data_stored":16807005,"data_compressed":122799,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6864896,"data_stored":6814948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":21086208,"data_stored":21075976,"data_compressed":87535,"data_compressed_allocated":94208,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":9072640,"data_stored":9025673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":37957632,"data_stored":38037194,"data_compressed":229326,"data_compressed_allocated":245760,"data_compressed_original":520192,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":21082112,"data_stored":21005106,"data_compressed":34395,"data_compressed_allocated":36864,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4087808,"data_stored":4063988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5746688,"data_stored":5712182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":38473728,"data_stored":38368386,"data_compressed":34388,"data_compressed_allocated":36864,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:38:25.102 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T03:38:25.102 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:38:25.102 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:38:25.103 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T03:38:25.103 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:38:25.121 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:25.121 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:25.127 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:25.126+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:25.134 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:25.134 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:25.138 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:25.139 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:25.144 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:38:25.156 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:25.157 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:25.167 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3935132 2024-09-16T03:38:25.174 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:25.174 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:25.181 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1185330 2024-09-16T03:38:25.207 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:25.207 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:25.227 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:25.228 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:25.228 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:25.226+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:25.253 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:25.253 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:25.254 INFO:tasks.rados.rados.0.smithi157.stdout:1273: finishing write tid 2 to smithi15736696-29 2024-09-16T03:38:25.254 INFO:tasks.rados.rados.0.smithi157.stdout:1273: finishing write tid 3 to smithi15736696-29 2024-09-16T03:38:25.254 INFO:tasks.rados.rados.0.smithi157.stdout:1273: finishing write tid 4 to smithi15736696-29 2024-09-16T03:38:25.254 INFO:tasks.rados.rados.0.smithi157.stdout:1273: oid 29 updating version 0 to 106 2024-09-16T03:38:25.255 INFO:tasks.rados.rados.0.smithi157.stdout:1273: oid 29 updating version 106 to 107 2024-09-16T03:38:25.255 INFO:tasks.rados.rados.0.smithi157.stdout:1273: oid 29 updating version 107 to 108 2024-09-16T03:38:25.255 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 108 (ObjNum 246 snap 0 seq_num 246) dirty exists 2024-09-16T03:38:25.255 INFO:tasks.rados.rados.0.smithi157.stdout:1273: left oid 29 (ObjNum 246 snap 0 seq_num 246) 2024-09-16T03:38:25.255 INFO:tasks.rados.rados.0.smithi157.stdout:1272: done (3 left) 2024-09-16T03:38:25.255 INFO:tasks.rados.rados.0.smithi157.stdout:1273: done (2 left) 2024-09-16T03:38:25.255 INFO:tasks.rados.rados.0.smithi157.stdout:1274: done (1 left) 2024-09-16T03:38:25.255 INFO:tasks.rados.rados.0.smithi157.stdout:1276: read oid 6 snap -1 2024-09-16T03:38:25.255 INFO:tasks.rados.rados.0.smithi157.stdout:1276: expect deleted 2024-09-16T03:38:25.255 INFO:tasks.rados.rados.0.smithi157.stdout:1277: set_chunk oid 30 target oid 2024-09-16T03:38:25.270 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:25.271 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:38:25.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:38:25.271 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:25.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.271 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:34 4.e 4:773b934b:::smithi15736696-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:25.271 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:38:25.271 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 73.213438988999997, 2024-09-16T03:38:25.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50346527299999999, 2024-09-16T03:38:25.271 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.271 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.271 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-16T03:38:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:38:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054413+0000", 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0669999999999999e-06 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054525+0000", 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111462 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054538+0000", 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3341e-05 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054545+0000", 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.584e-06 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054608+0000", 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2953999999999994e-05 2024-09-16T03:38:25.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:25.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054613+0000", 2024-09-16T03:38:25.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6110000000000001e-06 2024-09-16T03:38:25.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154852+0000", 2024-09-16T03:38:25.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10023838 2024-09-16T03:38:25.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154915+0000", 2024-09-16T03:38:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3138000000000004e-05 2024-09-16T03:38:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155001+0000", 2024-09-16T03:38:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5730999999999996e-05 2024-09-16T03:38:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156166+0000", 2024-09-16T03:38:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001165485 2024-09-16T03:38:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255909+0000", 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099742558999999995 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557800+0000", 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30189130600000003 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557836+0000", 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6566000000000002e-05 2024-09-16T03:38:25.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557877+0000", 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0089000000000002e-05 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:35 4.e 4:773b934b:::smithi15736696-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 73.213290658000005, 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50344320499999995, 2024-09-16T03:38:25.278 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.279 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.279 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.279 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.279 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.279 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.279 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-16T03:38:25.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.279 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:38:25.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054564+0000", 2024-09-16T03:38:25.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5710000000000002e-06 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054569+0000", 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0490000000000003e-06 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054578+0000", 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054639+0000", 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1061000000000004e-05 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054644+0000", 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999996e-06 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155607+0000", 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100963372 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155652+0000", 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155722+0000", 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0130999999999997e-05 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.157921+0000", 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021990159999999998 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255940+0000", 2024-09-16T03:38:25.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098018569999999999 2024-09-16T03:38:25.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557930+0000", 2024-09-16T03:38:25.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.301990333 2024-09-16T03:38:25.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557963+0000", 2024-09-16T03:38:25.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2348000000000003e-05 2024-09-16T03:38:25.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558003+0000", 2024-09-16T03:38:25.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:38:25.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.285 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.285 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:36 4.e 4:773b934b:::smithi15736696-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:25.285 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:38:25.285 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 73.213251968999998, 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50353811800000003, 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:38:25.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:38:25.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054599+0000", 2024-09-16T03:38:25.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:38:25.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054601+0000", 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4360000000000001e-06 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054606+0000", 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1019999999999996e-06 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054610+0000", 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.681e-06 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054662+0000", 2024-09-16T03:38:25.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1545000000000002e-05 2024-09-16T03:38:25.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:25.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054664+0000", 2024-09-16T03:38:25.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1919999999999999e-06 2024-09-16T03:38:25.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155903+0000", 2024-09-16T03:38:25.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101239068 2024-09-16T03:38:25.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:25.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155970+0000", 2024-09-16T03:38:25.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6934999999999995e-05 2024-09-16T03:38:25.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558029+0000", 2024-09-16T03:38:25.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40205888899999997 2024-09-16T03:38:25.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558078+0000", 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9842999999999997e-05 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558136+0000", 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7985000000000001e-05 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.292 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:799 4.e 4:765e9179:::smithi15736696-44:head [setxattr _header (56) in=7b,truncate 2857157] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:25.292 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:38:25.292 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.261399742000002, 2024-09-16T03:38:25.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50104159800000003, 2024-09-16T03:38:25.292 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.292 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.292 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.292 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.292 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.292 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.292 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 799 2024-09-16T03:38:25.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.292 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:38:25.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006454+0000", 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5739999999999998e-06 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006458+0000", 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6179999999999997e-06 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006462+0000", 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4999999999999999e-06 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204504+0000", 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19804184999999999 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204702+0000", 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019868199999999999 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:25.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204888+0000", 2024-09-16T03:38:25.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185701 2024-09-16T03:38:25.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.206549+0000", 2024-09-16T03:38:25.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016612790000000001 2024-09-16T03:38:25.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507371+0000", 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30082157399999998 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507414+0000", 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3053999999999998e-05 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507451+0000", 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6847999999999997e-05 2024-09-16T03:38:25.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507492+0000", 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1267999999999999e-05 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:800 4.e 4:765e9179:::smithi15736696-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.261376407, 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50119258099999997, 2024-09-16T03:38:25.298 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 800 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:38:25.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:38:25.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1899999999999998e-07 2024-09-16T03:38:25.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006478+0000", 2024-09-16T03:38:25.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006484+0000", 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6620000000000002e-06 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006488+0000", 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8700000000000002e-06 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205228+0000", 2024-09-16T03:38:25.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19873985899999999 2024-09-16T03:38:25.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:25.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205283+0000", 2024-09-16T03:38:25.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5878999999999998e-05 2024-09-16T03:38:25.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507527+0000", 2024-09-16T03:38:25.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302243924 2024-09-16T03:38:25.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507593+0000", 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6061000000000003e-05 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507666+0000", 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3059000000000006e-05 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.303 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:90 3.0 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.7112388, 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62288137399999999, 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:38:25.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556614+0000", 2024-09-16T03:38:25.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3439999999999999e-06 2024-09-16T03:38:25.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556621+0000", 2024-09-16T03:38:25.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7440000000000001e-06 2024-09-16T03:38:25.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556632+0000", 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1725e-05 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556640+0000", 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0239999999999997e-06 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556709+0000", 2024-09-16T03:38:25.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8787000000000004e-05 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556718+0000", 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6920000000000008e-06 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758133+0000", 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201414977 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758273+0000", 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140684 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758479+0000", 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020508599999999999 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862766+0000", 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104287952 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.878363+0000", 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015596601999999999 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960000+0000", 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081636630000000002 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960162+0000", 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000162597 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179366+0000", 2024-09-16T03:38:25.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21920357100000001 2024-09-16T03:38:25.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179411+0000", 2024-09-16T03:38:25.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5461999999999999e-05 2024-09-16T03:38:25.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179493+0000", 2024-09-16T03:38:25.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1496999999999996e-05 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:91 3.0 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.711193842, 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70460932600000004, 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556657+0000", 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:38:25.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556663+0000", 2024-09-16T03:38:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6579999999999999e-06 2024-09-16T03:38:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556670+0000", 2024-09-16T03:38:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3189999999999997e-06 2024-09-16T03:38:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556675+0000", 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000002e-06 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656929+0000", 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10025371299999999 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656944+0000", 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5460000000000001e-05 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861436+0000", 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.204491969 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861676+0000", 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024051899999999999 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861828+0000", 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015110199999999999 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862775+0000", 2024-09-16T03:38:25.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094718600000000003 2024-09-16T03:38:25.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079130+0000", 2024-09-16T03:38:25.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21635549200000001 2024-09-16T03:38:25.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179557+0000", 2024-09-16T03:38:25.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10042638399999999 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179972+0000", 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041574 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261034+0000", 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081061604999999995 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261142+0000", 2024-09-16T03:38:25.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107688 2024-09-16T03:38:25.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261266+0000", 2024-09-16T03:38:25.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124082 2024-09-16T03:38:25.320 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.320 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.320 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.320 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:94 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:38:25.320 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:38:25.320 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.711167716999999, 2024-09-16T03:38:25.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70479513699999996, 2024-09-16T03:38:25.320 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.320 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.320 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.320 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:38:25.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556685+0000", 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4279999999999999e-06 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556689+0000", 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.709e-06 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556693+0000", 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2939999999999999e-06 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657036+0000", 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10034209600000001 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657054+0000", 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8162999999999999e-05 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862279+0000", 2024-09-16T03:38:25.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20522560200000001 2024-09-16T03:38:25.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862380+0000", 2024-09-16T03:38:25.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101178 2024-09-16T03:38:25.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:25.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862509+0000", 2024-09-16T03:38:25.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128909 2024-09-16T03:38:25.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.863063+0000", 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055327600000000001 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079172+0000", 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21610910899999999 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179932+0000", 2024-09-16T03:38:25.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100760575 2024-09-16T03:38:25.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179997+0000", 2024-09-16T03:38:25.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5018999999999995e-05 2024-09-16T03:38:25.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261318+0000", 2024-09-16T03:38:25.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081320838000000006 2024-09-16T03:38:25.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261385+0000", 2024-09-16T03:38:25.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7115999999999999e-05 2024-09-16T03:38:25.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261478+0000", 2024-09-16T03:38:25.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2360000000000003e-05 2024-09-16T03:38:25.328 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.328 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.328 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.328 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1742 3.0 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head [write 0~1659392] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T03:38:25.328 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:38:25.328 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.711139317000001, 2024-09-16T03:38:25.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62317617400000003, 2024-09-16T03:38:25.329 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.329 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.329 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.329 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.329 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1742 2024-09-16T03:38:25.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.329 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:38:25.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:38:25.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556713+0000", 2024-09-16T03:38:25.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7209999999999999e-06 2024-09-16T03:38:25.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558531+0000", 2024-09-16T03:38:25.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018184480000000001 2024-09-16T03:38:25.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558539+0000", 2024-09-16T03:38:25.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0830000000000004e-06 2024-09-16T03:38:25.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558546+0000", 2024-09-16T03:38:25.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3369999999999996e-06 2024-09-16T03:38:25.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759148+0000", 2024-09-16T03:38:25.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20060235500000001 2024-09-16T03:38:25.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759230+0000", 2024-09-16T03:38:25.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2391000000000004e-05 2024-09-16T03:38:25.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:25.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759329+0000", 2024-09-16T03:38:25.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9147000000000005e-05 2024-09-16T03:38:25.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862771+0000", 2024-09-16T03:38:25.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103441085 2024-09-16T03:38:25.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960102+0000", 2024-09-16T03:38:25.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097331259000000003 2024-09-16T03:38:25.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960218+0000", 2024-09-16T03:38:25.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011581100000000001 2024-09-16T03:38:25.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079074+0000", 2024-09-16T03:38:25.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118856436 2024-09-16T03:38:25.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179758+0000", 2024-09-16T03:38:25.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100683884 2024-09-16T03:38:25.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179800+0000", 2024-09-16T03:38:25.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1928000000000002e-05 2024-09-16T03:38:25.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179887+0000", 2024-09-16T03:38:25.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7288999999999997e-05 2024-09-16T03:38:25.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.338 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:89 3.0 3:1c07e18b:::d3cef63b99e391058a88d444a56984ba6fc25b22efbc98e24ba274246c542d3c:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:25.338 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:38:25.338 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.931736821000001, 2024-09-16T03:38:25.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50388158999999999, 2024-09-16T03:38:25.338 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.338 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.339 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:38:25.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:38:25.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336115+0000", 2024-09-16T03:38:25.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.866e-06 2024-09-16T03:38:25.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336123+0000", 2024-09-16T03:38:25.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6990000000000003e-06 2024-09-16T03:38:25.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336136+0000", 2024-09-16T03:38:25.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3342e-05 2024-09-16T03:38:25.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336143+0000", 2024-09-16T03:38:25.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2099999999999998e-06 2024-09-16T03:38:25.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336214+0000", 2024-09-16T03:38:25.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1538000000000005e-05 2024-09-16T03:38:25.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336342+0000", 2024-09-16T03:38:25.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127408 2024-09-16T03:38:25.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:25.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336593+0000", 2024-09-16T03:38:25.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025150500000000001 2024-09-16T03:38:25.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337284+0000", 2024-09-16T03:38:25.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069060900000000001 2024-09-16T03:38:25.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338068+0000", 2024-09-16T03:38:25.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078438900000000003 2024-09-16T03:38:25.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338107+0000", 2024-09-16T03:38:25.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8858e-05 2024-09-16T03:38:25.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538164+0000", 2024-09-16T03:38:25.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200056971 2024-09-16T03:38:25.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839843+0000", 2024-09-16T03:38:25.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30167921599999997 2024-09-16T03:38:25.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839905+0000", 2024-09-16T03:38:25.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1542000000000001e-05 2024-09-16T03:38:25.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839995+0000", 2024-09-16T03:38:25.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0437000000000002e-05 2024-09-16T03:38:25.346 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.346 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.346 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.346 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:98 3.0 3:04dfaaf9:::3de53e1d584ced9b0f970855867dbc9a14b3fc90962aa20a81c1ed25ff4b961f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:25.346 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:38:25.346 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.930408920000001, 2024-09-16T03:38:25.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70310345299999999, 2024-09-16T03:38:25.346 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.347 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.347 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.347 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:38:25.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.347 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:38:25.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337442+0000", 2024-09-16T03:38:25.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0729999999999999e-06 2024-09-16T03:38:25.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337447+0000", 2024-09-16T03:38:25.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3259999999999997e-06 2024-09-16T03:38:25.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337457+0000", 2024-09-16T03:38:25.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-09-16T03:38:25.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337462+0000", 2024-09-16T03:38:25.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0780000000000003e-06 2024-09-16T03:38:25.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337511+0000", 2024-09-16T03:38:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:38:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337574+0000", 2024-09-16T03:38:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3114999999999995e-05 2024-09-16T03:38:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:25.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337723+0000", 2024-09-16T03:38:25.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014882299999999999 2024-09-16T03:38:25.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.339039+0000", 2024-09-16T03:38:25.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001315524 2024-09-16T03:38:25.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538225+0000", 2024-09-16T03:38:25.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19918607199999999 2024-09-16T03:38:25.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638838+0000", 2024-09-16T03:38:25.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100613466 2024-09-16T03:38:25.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.739726+0000", 2024-09-16T03:38:25.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100887854 2024-09-16T03:38:25.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040319+0000", 2024-09-16T03:38:25.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30059265699999999 2024-09-16T03:38:25.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040398+0000", 2024-09-16T03:38:25.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8852999999999999e-05 2024-09-16T03:38:25.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040545+0000", 2024-09-16T03:38:25.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147236 2024-09-16T03:38:25.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.354 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.354 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.354 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:101 3.0 3:090a0a71:::0ecec3e4e81d718f460a9826fb4cef2398bf4c951ee653be84827f33946eef5f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:25.354 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:38:25.354 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.831238173999999, 2024-09-16T03:38:25.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60435370600000005, 2024-09-16T03:38:25.355 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.355 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.355 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.355 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:38:25.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:38:25.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436614+0000", 2024-09-16T03:38:25.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.844e-06 2024-09-16T03:38:25.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.356 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436623+0000", 2024-09-16T03:38:25.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0739999999999999e-06 2024-09-16T03:38:25.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436636+0000", 2024-09-16T03:38:25.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2629999999999999e-05 2024-09-16T03:38:25.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436643+0000", 2024-09-16T03:38:25.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7489999999999998e-06 2024-09-16T03:38:25.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536906+0000", 2024-09-16T03:38:25.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10026262599999999 2024-09-16T03:38:25.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536979+0000", 2024-09-16T03:38:25.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3311000000000003e-05 2024-09-16T03:38:25.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537109+0000", 2024-09-16T03:38:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013015800000000001 2024-09-16T03:38:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538241+0000", 2024-09-16T03:38:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001131726 2024-09-16T03:38:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638904+0000", 2024-09-16T03:38:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066240799999999 2024-09-16T03:38:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.940200+0000", 2024-09-16T03:38:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30129597299999999 2024-09-16T03:38:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040604+0000", 2024-09-16T03:38:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040478999999999 2024-09-16T03:38:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040759+0000", 2024-09-16T03:38:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015445500000000001 2024-09-16T03:38:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040824+0000", 2024-09-16T03:38:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5633e-05 2024-09-16T03:38:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040966+0000", 2024-09-16T03:38:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014132899999999999 2024-09-16T03:38:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:102 3.0 3:17742417:::632266e231c4c385ace473a8f4464285cf620fbc8799f68869353e2aa4660567:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:38:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.831191348000001, 2024-09-16T03:38:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.604540823, 2024-09-16T03:38:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:38:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:38:25.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436660+0000", 2024-09-16T03:38:25.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:38:25.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436663+0000", 2024-09-16T03:38:25.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1599999999999998e-06 2024-09-16T03:38:25.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436670+0000", 2024-09-16T03:38:25.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.742e-06 2024-09-16T03:38:25.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436675+0000", 2024-09-16T03:38:25.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3839999999999996e-06 2024-09-16T03:38:25.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537499+0000", 2024-09-16T03:38:25.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100823987 2024-09-16T03:38:25.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537602+0000", 2024-09-16T03:38:25.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102837 2024-09-16T03:38:25.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:25.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537793+0000", 2024-09-16T03:38:25.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019123499999999999 2024-09-16T03:38:25.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638962+0000", 2024-09-16T03:38:25.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101168944 2024-09-16T03:38:25.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.840035+0000", 2024-09-16T03:38:25.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201073481 2024-09-16T03:38:25.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040706+0000", 2024-09-16T03:38:25.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200670606 2024-09-16T03:38:25.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041009+0000", 2024-09-16T03:38:25.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030271600000000002 2024-09-16T03:38:25.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041087+0000", 2024-09-16T03:38:25.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8310000000000001e-05 2024-09-16T03:38:25.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.369 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041127+0000", 2024-09-16T03:38:25.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:38:25.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041200+0000", 2024-09-16T03:38:25.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2381999999999999e-05 2024-09-16T03:38:25.370 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.370 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.370 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.370 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:103 3.0 3:28447854:::c811732c201b76a96e1532c4d9a18d4f4f2a8ddec3bf79945689a1a62c8a5a25:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:25.370 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:38:25.370 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.831146099000001, 2024-09-16T03:38:25.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60467526100000002, 2024-09-16T03:38:25.371 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.371 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.371 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.371 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:38:25.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:38:25.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436705+0000", 2024-09-16T03:38:25.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.23e-07 2024-09-16T03:38:25.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436719+0000", 2024-09-16T03:38:25.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3957999999999999e-05 2024-09-16T03:38:25.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436725+0000", 2024-09-16T03:38:25.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6100000000000002e-06 2024-09-16T03:38:25.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436730+0000", 2024-09-16T03:38:25.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0609999999999997e-06 2024-09-16T03:38:25.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538257+0000", 2024-09-16T03:38:25.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10152739500000001 2024-09-16T03:38:25.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538314+0000", 2024-09-16T03:38:25.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7355000000000002e-05 2024-09-16T03:38:25.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:25.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538435+0000", 2024-09-16T03:38:25.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120967 2024-09-16T03:38:25.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638968+0000", 2024-09-16T03:38:25.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100532994 2024-09-16T03:38:25.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040662+0000", 2024-09-16T03:38:25.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40169386299999998 2024-09-16T03:38:25.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041059+0000", 2024-09-16T03:38:25.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039723900000000002 2024-09-16T03:38:25.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041247+0000", 2024-09-16T03:38:25.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018756900000000001 2024-09-16T03:38:25.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041286+0000", 2024-09-16T03:38:25.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8971999999999999e-05 2024-09-16T03:38:25.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041318+0000", 2024-09-16T03:38:25.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1628999999999998e-05 2024-09-16T03:38:25.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041379+0000", 2024-09-16T03:38:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1926000000000003e-05 2024-09-16T03:38:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(osd.11.23:98 3.0 e74/69 ondisk, result = 0)", 2024-09-16T03:38:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:38:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.729329731, 2024-09-16T03:38:25.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50203460700000002, 2024-09-16T03:38:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:38:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:38:25.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:38:25.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0899999999999997e-07 2024-09-16T03:38:25.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538523+0000", 2024-09-16T03:38:25.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-09-16T03:38:25.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538527+0000", 2024-09-16T03:38:25.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0539999999999996e-06 2024-09-16T03:38:25.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538529+0000", 2024-09-16T03:38:25.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7420000000000002e-06 2024-09-16T03:38:25.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040301+0000", 2024-09-16T03:38:25.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50177146399999994 2024-09-16T03:38:25.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040312+0000", 2024-09-16T03:38:25.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1039999999999999e-05 2024-09-16T03:38:25.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040555+0000", 2024-09-16T03:38:25.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243212 2024-09-16T03:38:25.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.383 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.383 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(osd.11.23:103 3.0 e75/69 ondisk, result = 0)", 2024-09-16T03:38:25.383 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.540430+0000", 2024-09-16T03:38:25.383 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.727420262999999, 2024-09-16T03:38:25.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50024338400000001, 2024-09-16T03:38:25.383 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.384 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.384 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:25.384 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540430+0000", 2024-09-16T03:38:25.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540430+0000", 2024-09-16T03:38:25.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540431+0000", 2024-09-16T03:38:25.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3600000000000002e-07 2024-09-16T03:38:25.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540434+0000", 2024-09-16T03:38:25.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5149999999999998e-06 2024-09-16T03:38:25.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540442+0000", 2024-09-16T03:38:25.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6340000000000001e-06 2024-09-16T03:38:25.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540448+0000", 2024-09-16T03:38:25.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.285e-06 2024-09-16T03:38:25.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040646+0000", 2024-09-16T03:38:25.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50019782800000001 2024-09-16T03:38:25.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040655+0000", 2024-09-16T03:38:25.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3429999999999996e-06 2024-09-16T03:38:25.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040673+0000", 2024-09-16T03:38:25.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8842999999999999e-05 2024-09-16T03:38:25.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.388 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.388 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.388 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2473 4.e 4:765e9179:::smithi15736696-44:head [write 1830877~681726] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:25.388 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:38:25.388 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.2571608740000002, 2024-09-16T03:38:25.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50285070099999996, 2024-09-16T03:38:25.388 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.389 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.389 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.389 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.389 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.389 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.389 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2473 2024-09-16T03:38:25.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.389 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:38:25.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:38:25.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010691+0000", 2024-09-16T03:38:25.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-06 2024-09-16T03:38:25.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011744+0000", 2024-09-16T03:38:25.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001052486 2024-09-16T03:38:25.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011755+0000", 2024-09-16T03:38:25.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1073000000000001e-05 2024-09-16T03:38:25.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011761+0000", 2024-09-16T03:38:25.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1600000000000003e-06 2024-09-16T03:38:25.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011804+0000", 2024-09-16T03:38:25.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2697999999999997e-05 2024-09-16T03:38:25.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011885+0000", 2024-09-16T03:38:25.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.119e-05 2024-09-16T03:38:25.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:25.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.012013+0000", 2024-09-16T03:38:25.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128066 2024-09-16T03:38:25.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.013923+0000", 2024-09-16T03:38:25.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019104739999999999 2024-09-16T03:38:25.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.412920+0000", 2024-09-16T03:38:25.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39899685200000001 2024-09-16T03:38:25.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513347+0000", 2024-09-16T03:38:25.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100427167 2024-09-16T03:38:25.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513490+0000", 2024-09-16T03:38:25.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142163 2024-09-16T03:38:25.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513540+0000", 2024-09-16T03:38:25.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0482000000000002e-05 2024-09-16T03:38:25.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.395 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.395 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.396 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:135 3.0 3:03e4b4cc:::36686daa84b9b41d23533e7d0b5d4ef41f745a0756e58cc66c36c36f24851219:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:25.396 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.114458+0000", 2024-09-16T03:38:25.396 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1533920979999994, 2024-09-16T03:38:25.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.502137163, 2024-09-16T03:38:25.396 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.396 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.396 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.396 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114458+0000", 2024-09-16T03:38:25.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114458+0000", 2024-09-16T03:38:25.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114459+0000", 2024-09-16T03:38:25.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.26e-06 2024-09-16T03:38:25.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114464+0000", 2024-09-16T03:38:25.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8049999999999997e-06 2024-09-16T03:38:25.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114474+0000", 2024-09-16T03:38:25.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4369999999999996e-06 2024-09-16T03:38:25.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114479+0000", 2024-09-16T03:38:25.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3669999999999996e-06 2024-09-16T03:38:25.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114539+0000", 2024-09-16T03:38:25.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:38:25.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114652+0000", 2024-09-16T03:38:25.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112948 2024-09-16T03:38:25.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:25.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114886+0000", 2024-09-16T03:38:25.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000233666 2024-09-16T03:38:25.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116749+0000", 2024-09-16T03:38:25.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001863314 2024-09-16T03:38:25.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116765+0000", 2024-09-16T03:38:25.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5528000000000001e-05 2024-09-16T03:38:25.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.413659+0000", 2024-09-16T03:38:25.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.296894245 2024-09-16T03:38:25.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414169+0000", 2024-09-16T03:38:25.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050972500000000004 2024-09-16T03:38:25.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616394+0000", 2024-09-16T03:38:25.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20222523000000001 2024-09-16T03:38:25.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616467+0000", 2024-09-16T03:38:25.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2929000000000006e-05 2024-09-16T03:38:25.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616595+0000", 2024-09-16T03:38:25.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012867400000000001 2024-09-16T03:38:25.403 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.404 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.404 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:136 3.0 3:1b68d773:::374e1d62491a062bcb5530630b4c28c4696d618a79860b96215d227206cd62aa:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:25.404 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:38:25.404 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1533563349999998, 2024-09-16T03:38:25.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50236015199999995, 2024-09-16T03:38:25.404 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.404 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.404 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.405 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:38:25.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:38:25.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:38:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4499999999999997e-07 2024-09-16T03:38:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114497+0000", 2024-09-16T03:38:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:38:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114506+0000", 2024-09-16T03:38:25.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5759999999999995e-06 2024-09-16T03:38:25.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114511+0000", 2024-09-16T03:38:25.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0649999999999998e-06 2024-09-16T03:38:25.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115277+0000", 2024-09-16T03:38:25.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076592800000000003 2024-09-16T03:38:25.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115368+0000", 2024-09-16T03:38:25.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1037000000000003e-05 2024-09-16T03:38:25.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:25.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115573+0000", 2024-09-16T03:38:25.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000204594 2024-09-16T03:38:25.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116773+0000", 2024-09-16T03:38:25.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001200062 2024-09-16T03:38:25.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.121514+0000", 2024-09-16T03:38:25.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047407270000000001 2024-09-16T03:38:25.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414056+0000", 2024-09-16T03:38:25.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29254249700000001 2024-09-16T03:38:25.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414192+0000", 2024-09-16T03:38:25.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013572000000000001 2024-09-16T03:38:25.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616663+0000", 2024-09-16T03:38:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20247094800000001 2024-09-16T03:38:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616728+0000", 2024-09-16T03:38:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5217999999999998e-05 2024-09-16T03:38:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616854+0000", 2024-09-16T03:38:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126064 2024-09-16T03:38:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.411 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.412 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:137 3.0 3:20b08a9e:::0ba23f501764ab3447cc323c825ee8bff84c3c6e340c68c08117252e1f0b39bb:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:25.412 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:38:25.412 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.15331239, 2024-09-16T03:38:25.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50256677599999999, 2024-09-16T03:38:25.412 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.412 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.412 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:38:25.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:38:25.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:38:25.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1200000000000003e-07 2024-09-16T03:38:25.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114542+0000", 2024-09-16T03:38:25.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7400000000000002e-06 2024-09-16T03:38:25.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114547+0000", 2024-09-16T03:38:25.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.42e-06 2024-09-16T03:38:25.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114550+0000", 2024-09-16T03:38:25.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4470000000000002e-06 2024-09-16T03:38:25.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115951+0000", 2024-09-16T03:38:25.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014005459999999999 2024-09-16T03:38:25.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116047+0000", 2024-09-16T03:38:25.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6899999999999997e-05 2024-09-16T03:38:25.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:25.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116270+0000", 2024-09-16T03:38:25.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022291699999999999 2024-09-16T03:38:25.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.117236+0000", 2024-09-16T03:38:25.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096522299999999997 2024-09-16T03:38:25.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.218932+0000", 2024-09-16T03:38:25.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10169644799999999 2024-09-16T03:38:25.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414271+0000", 2024-09-16T03:38:25.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19533916300000001 2024-09-16T03:38:25.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.514717+0000", 2024-09-16T03:38:25.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100445879 2024-09-16T03:38:25.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616919+0000", 2024-09-16T03:38:25.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10220199100000001 2024-09-16T03:38:25.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616980+0000", 2024-09-16T03:38:25.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0977999999999998e-05 2024-09-16T03:38:25.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.617105+0000", 2024-09-16T03:38:25.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012461199999999999 2024-09-16T03:38:25.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.419 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.420 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.420 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:25.421 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:25.383+0000 7f928a16a640 -1 osd.14 94 handle_osd_map injecting map crc failure 2024-09-16T03:38:25.422 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:25.412+0000 7fa3886e6640 -1 osd.10 94 handle_osd_map injecting map crc failure 2024-09-16T03:38:25.422 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:25.419+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:25.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:38:25.424 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:25.424 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:25.425 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:25.425 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:25.426 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:25.426 INFO:tasks.rados.rados.0.smithi157.stdout:1277: set_chunk oid 30 offset: 213504 length: 1767936 target oid 578b1d87c8afbdf4591f27985998b393e1a072532bab099daa6ebc8f3d0583ba offset: 0 snap -1 2024-09-16T03:38:25.426 INFO:tasks.rados.rados.0.smithi157.stdout:1278: read oid 37 snap -1 2024-09-16T03:38:25.426 INFO:tasks.rados.rados.0.smithi157.stdout:1278: expect (ObjNum 217 snap 0 seq_num 217) 2024-09-16T03:38:25.426 INFO:tasks.rados.rados.0.smithi157.stdout:1279: read oid 35 snap -1 2024-09-16T03:38:25.427 INFO:tasks.rados.rados.0.smithi157.stdout:1279: expect (ObjNum 190 snap 0 seq_num 190) 2024-09-16T03:38:25.427 INFO:tasks.rados.rados.0.smithi157.stdout:1280: write (excl) oid 49 current snap is 0 2024-09-16T03:38:25.427 INFO:tasks.rados.rados.0.smithi157.stdout:1275: got (0) Success 2024-09-16T03:38:25.427 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 0 (ObjNum 215 snap 0 seq_num 215) dirty exists 2024-09-16T03:38:25.427 INFO:tasks.rados.rados.0.smithi157.stdout:1280: seq_num 247 ranges {551329=709799,1870817=280512} 2024-09-16T03:38:25.427 INFO:tasks.rados.rados.0.smithi157.stdout:1280: writing smithi15736696-49 from 551329 to 1261128 tid 1 2024-09-16T03:38:25.427 INFO:tasks.rados.rados.0.smithi157.stdout:1280: writing smithi15736696-49 from 1870817 to 2151329 tid 2 2024-09-16T03:38:25.427 INFO:tasks.rados.rados.0.smithi157.stdout:1275: done (5 left) 2024-09-16T03:38:25.427 INFO:tasks.rados.rados.0.smithi157.stdout:1281: write oid 17 current snap is 0 2024-09-16T03:38:25.427 INFO:tasks.rados.rados.0.smithi157.stdout:1281: seq_num 248 ranges {568136=785048,1929299=238837} 2024-09-16T03:38:25.427 INFO:tasks.rados.rados.0.smithi157.stdout:1281: writing smithi15736696-17 from 568136 to 1353184 tid 1 2024-09-16T03:38:25.428 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:25.428 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:25.428 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:25.428 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:25.429 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:25.429 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:25.429 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:25.431 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:38:25.431 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:25.433 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:25.433 INFO:tasks.rados.rados.0.smithi157.stdout:1281: writing smithi15736696-17 from 1929299 to 2168136 tid 2 2024-09-16T03:38:25.433 INFO:tasks.rados.rados.0.smithi157.stdout:1282: read oid 20 snap -1 2024-09-16T03:38:25.433 INFO:tasks.rados.rados.0.smithi157.stdout:1282: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-16T03:38:25.433 INFO:tasks.rados.rados.0.smithi157.stdout:1283: read oid 45 snap -1 2024-09-16T03:38:25.433 INFO:tasks.rados.rados.0.smithi157.stdout:1283: expect (ObjNum 240 snap 0 seq_num 240) 2024-09-16T03:38:25.433 INFO:tasks.rados.rados.0.smithi157.stdout:1284: read oid 7 snap -1 2024-09-16T03:38:25.433 INFO:tasks.rados.rados.0.smithi157.stdout:1284: expect deleted 2024-09-16T03:38:25.433 INFO:tasks.rados.rados.0.smithi157.stdout:1285: read oid 16 snap -1 2024-09-16T03:38:25.433 INFO:tasks.rados.rados.0.smithi157.stdout:1285: expect (ObjNum 224 snap 0 seq_num 224) 2024-09-16T03:38:25.434 INFO:tasks.rados.rados.0.smithi157.stdout:1286: read oid 24 snap -1 2024-09-16T03:38:25.434 INFO:tasks.rados.rados.0.smithi157.stdout:1286: expect (ObjNum 238 snap 0 seq_num 238) 2024-09-16T03:38:25.434 INFO:tasks.rados.rados.0.smithi157.stdout:1287: delete oid 2 current snap is 0 2024-09-16T03:38:25.434 INFO:tasks.rados.rados.0.smithi157.stdout:1278: expect (ObjNum 217 snap 0 seq_num 217) 2024-09-16T03:38:25.472 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:38:25.472 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:38:25.474 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:25.472+0000 7eff7884b640 -1 osd.3 94 handle_osd_map injecting map crc failure 2024-09-16T03:38:25.483 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-09-16T03:38:25.385673+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189906354,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":2663,"num_read_kb":1727932,"num_write":1657,"num_write_kb":432055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":26334576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":16,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1132462080,"kb_used":687416,"kb_used_data":303668,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1131774664,"statfs":{"total":1159641169920,"available":1158937255936,"internally_reserved":0,"allocated":310956032,"data_stored":316182153,"data_compressed":6953468,"data_compressed_allocated":7495680,"data_compressed_original":17080320,"omap_allocated":19135,"internal_metadata":370193729},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3279794,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":186,"num_read_kb":182608,"num_write":95,"num_write_kb":32094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017239"},"pg_stats":[{"pgid":"4.19","version":"0'0","reported_seq":143,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920381+0000","last_change":"2024-09-16T03:37:52.558405+0000","last_active":"2024-09-16T03:38:23.920381+0000","last_peered":"2024-09-16T03:38:23.920381+0000","last_clean":"2024-09-16T03:38:23.920381+0000","last_became_active":"2024-09-16T03:37:52.558043+0000","last_became_peered":"2024-09-16T03:37:52.558043+0000","last_unstale":"2024-09-16T03:38:23.920381+0000","last_undegraded":"2024-09-16T03:38:23.920381+0000","last_fullsized":"2024-09-16T03:38:23.920381+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:39.955354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.955354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:52.907292+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"80'39","reported_seq":235,"reported_epoch":93,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:23.820070+0000","last_change":"2024-09-16T03:37:54.660680+0000","last_active":"2024-09-16T03:38:23.820070+0000","last_peered":"2024-09-16T03:38:23.820070+0000","last_clean":"2024-09-16T03:37:50.435122+0000","last_became_active":"2024-09-16T03:37:54.659250+0000","last_became_peered":"2024-09-16T03:37:54.659250+0000","last_unstale":"2024-09-16T03:38:23.820070+0000","last_undegraded":"2024-09-16T03:38:23.820070+0000","last_fullsized":"2024-09-16T03:38:23.820070+0000","mapping_epoch":68,"log_start":"79'35","ondisk_log_start":"79'35","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'29","last_scrub_stamp":"2024-09-16T03:37:39.786314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.786314+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":32,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":781,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260558,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20261,"num_write":9,"num_write_kb":2547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"83'41","reported_seq":261,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.375248+0000","last_change":"2024-09-16T03:37:47.616917+0000","last_active":"2024-09-16T03:38:23.375248+0000","last_peered":"2024-09-16T03:38:23.375248+0000","last_clean":"2024-09-16T03:38:23.375248+0000","last_became_active":"2024-09-16T03:37:31.713115+0000","last_became_peered":"2024-09-16T03:37:31.713115+0000","last_unstale":"2024-09-16T03:38:23.375248+0000","last_undegraded":"2024-09-16T03:38:23.375248+0000","last_fullsized":"2024-09-16T03:38:23.375248+0000","mapping_epoch":50,"log_start":"73'38","ondisk_log_start":"73'38","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":37,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:16.011301+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6084101,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":38220,"num_write":6,"num_write_kb":1554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.16","version":"85'61","reported_seq":275,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.420737+0000","last_change":"2024-09-16T03:37:57.546992+0000","last_active":"2024-09-16T03:38:23.420737+0000","last_peered":"2024-09-16T03:38:23.420737+0000","last_clean":"2024-09-16T03:38:23.420737+0000","last_became_active":"2024-09-16T03:37:57.544402+0000","last_became_peered":"2024-09-16T03:37:57.544402+0000","last_unstale":"2024-09-16T03:38:23.420737+0000","last_undegraded":"2024-09-16T03:38:23.420737+0000","last_fullsized":"2024-09-16T03:38:23.420737+0000","mapping_epoch":71,"log_start":"85'58","ondisk_log_start":"85'58","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'32","last_scrub_stamp":"2024-09-16T03:37:35.189364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.189364+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:43.166755+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6857256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":61874,"num_write":25,"num_write_kb":9792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2311899,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.15","version":"91'53","reported_seq":268,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.604989+0000","last_change":"2024-09-16T03:37:47.078728+0000","last_active":"2024-09-16T03:38:21.604989+0000","last_peered":"2024-09-16T03:38:21.604989+0000","last_clean":"2024-09-16T03:38:21.604989+0000","last_became_active":"2024-09-16T03:37:31.349745+0000","last_became_peered":"2024-09-16T03:37:31.349745+0000","last_unstale":"2024-09-16T03:38:21.604989+0000","last_undegraded":"2024-09-16T03:38:21.604989+0000","last_fullsized":"2024-09-16T03:38:21.604989+0000","mapping_epoch":50,"log_start":"88'49","ondisk_log_start":"88'49","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":47,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:01.863665+0000","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7680799,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":70541,"num_write":23,"num_write_kb":6606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.14","version":"79'16","reported_seq":193,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.887426+0000","last_change":"2024-09-16T03:37:39.953719+0000","last_active":"2024-09-16T03:38:20.887426+0000","last_peered":"2024-09-16T03:38:20.887426+0000","last_clean":"2024-09-16T03:38:20.887426+0000","last_became_active":"2024-09-16T03:37:31.149621+0000","last_became_peered":"2024-09-16T03:37:31.149621+0000","last_unstale":"2024-09-16T03:38:20.887426+0000","last_undegraded":"2024-09-16T03:38:20.887426+0000","last_fullsized":"2024-09-16T03:38:20.887426+0000","mapping_epoch":50,"log_start":"57'13","ondisk_log_start":"57'13","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'13","last_scrub_stamp":"2024-09-16T03:37:39.953656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.953656+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":10,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:48.840913+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3159716,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":5791,"num_write":4,"num_write_kb":1127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.13","version":"84'59","reported_seq":278,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.887352+0000","last_change":"2024-09-16T03:37:57.653492+0000","last_active":"2024-09-16T03:38:20.887352+0000","last_peered":"2024-09-16T03:38:20.887352+0000","last_clean":"2024-09-16T03:38:20.887352+0000","last_became_active":"2024-09-16T03:37:57.640119+0000","last_became_peered":"2024-09-16T03:37:57.640119+0000","last_unstale":"2024-09-16T03:38:20.887352+0000","last_undegraded":"2024-09-16T03:38:20.887352+0000","last_fullsized":"2024-09-16T03:38:20.887352+0000","mapping_epoch":71,"log_start":"83'56","ondisk_log_start":"83'56","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'21","last_scrub_stamp":"2024-09-16T03:37:34.916681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.916681+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":54,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:35.558675+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361400,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":56277,"num_write":61,"num_write_kb":14773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4939757,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.12","version":"92'16","reported_seq":215,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920410+0000","last_change":"2024-09-16T03:37:57.652056+0000","last_active":"2024-09-16T03:38:23.920410+0000","last_peered":"2024-09-16T03:38:23.920410+0000","last_clean":"2024-09-16T03:38:23.920410+0000","last_became_active":"2024-09-16T03:37:57.651635+0000","last_became_peered":"2024-09-16T03:37:57.651635+0000","last_unstale":"2024-09-16T03:38:23.920410+0000","last_undegraded":"2024-09-16T03:38:23.920410+0000","last_fullsized":"2024-09-16T03:38:23.920410+0000","mapping_epoch":71,"log_start":"82'13","ondisk_log_start":"82'13","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":13,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:57.128705+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067294,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":31619,"num_write":12,"num_write_kb":3255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1423481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,11],"acting":[14,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.11","version":"72'12","reported_seq":177,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.564663+0000","last_change":"2024-09-16T03:37:50.756196+0000","last_active":"2024-09-16T03:38:20.564663+0000","last_peered":"2024-09-16T03:38:20.564663+0000","last_clean":"2024-09-16T03:38:20.564663+0000","last_became_active":"2024-09-16T03:37:49.624845+0000","last_became_peered":"2024-09-16T03:37:49.624845+0000","last_unstale":"2024-09-16T03:38:20.564663+0000","last_undegraded":"2024-09-16T03:38:20.564663+0000","last_fullsized":"2024-09-16T03:38:20.564663+0000","mapping_epoch":63,"log_start":"62'9","ondisk_log_start":"62'9","created":50,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'9","last_scrub_stamp":"2024-09-16T03:37:50.756120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:50.756120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":9,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:00.529911+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":41724,"num_write":4,"num_write_kb":1177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"85'55","reported_seq":278,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.375401+0000","last_change":"2024-09-16T03:37:58.667607+0000","last_active":"2024-09-16T03:38:23.375401+0000","last_peered":"2024-09-16T03:38:23.375401+0000","last_clean":"2024-09-16T03:38:23.375401+0000","last_became_active":"2024-09-16T03:37:58.567094+0000","last_became_peered":"2024-09-16T03:37:58.567094+0000","last_unstale":"2024-09-16T03:38:23.375401+0000","last_undegraded":"2024-09-16T03:38:23.375401+0000","last_fullsized":"2024-09-16T03:38:23.375401+0000","mapping_epoch":72,"log_start":"73'52","ondisk_log_start":"73'52","created":50,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:46.798884+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":29,"num_read_kb":30360,"num_write":9,"num_write_kb":1874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1836181,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":158,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.417482+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:38:23.417482+0000","last_peered":"2024-09-16T03:38:23.417482+0000","last_clean":"2024-09-16T03:38:23.417482+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:38:23.417482+0000","last_undegraded":"2024-09-16T03:38:23.417482+0000","last_fullsized":"2024-09-16T03:38:23.417482+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.232558+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":158,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T03:38:24.378979+0000","last_change":"2024-09-16T03:37:39.701376+0000","last_active":"2024-09-16T03:38:24.378979+0000","last_peered":"2024-09-16T03:38:24.378979+0000","last_clean":"2024-09-16T03:38:24.378979+0000","last_became_active":"2024-09-16T03:37:31.054119+0000","last_became_peered":"2024-09-16T03:37:31.054119+0000","last_unstale":"2024-09-16T03:38:24.378979+0000","last_undegraded":"2024-09-16T03:38:24.378979+0000","last_fullsized":"2024-09-16T03:38:24.378979+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:37:39.701279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.701279+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:03.319077+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":165,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920429+0000","last_change":"2024-09-16T03:38:12.041048+0000","last_active":"2024-09-16T03:38:23.920429+0000","last_peered":"2024-09-16T03:38:23.920429+0000","last_clean":"2024-09-16T03:38:23.920429+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:38:23.920429+0000","last_undegraded":"2024-09-16T03:38:23.920429+0000","last_fullsized":"2024-09-16T03:38:23.920429+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:12.040917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.040917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.877949+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.0","version":"92'105","reported_seq":395,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.534584+0000","last_change":"2024-09-16T03:37:39.774110+0000","last_active":"2024-09-16T03:38:23.534584+0000","last_peered":"2024-09-16T03:38:23.534584+0000","last_clean":"2024-09-16T03:38:23.534584+0000","last_became_active":"2024-09-16T03:37:31.151595+0000","last_became_peered":"2024-09-16T03:37:31.151595+0000","last_unstale":"2024-09-16T03:38:23.534584+0000","last_undegraded":"2024-09-16T03:38:23.534584+0000","last_fullsized":"2024-09-16T03:38:23.534584+0000","mapping_epoch":32,"log_start":"92'102","ondisk_log_start":"92'102","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'52","last_scrub_stamp":"2024-09-16T03:37:39.774014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.774014+0000","objects_scrubbed":5,"log_size":3,"log_dups_size":102,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:05.310361+0000","scrub_duration":647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15361598,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":163,"num_read_kb":161344,"num_write":59,"num_write_kb":17582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.564750+0000","last_change":"2024-09-16T03:37:54.655500+0000","last_active":"2024-09-16T03:38:20.564750+0000","last_peered":"2024-09-16T03:38:20.564750+0000","last_clean":"2024-09-16T03:38:20.564750+0000","last_became_active":"2024-09-16T03:37:54.655185+0000","last_became_peered":"2024-09-16T03:37:54.655185+0000","last_unstale":"2024-09-16T03:38:20.564750+0000","last_undegraded":"2024-09-16T03:38:20.564750+0000","last_fullsized":"2024-09-16T03:38:20.564750+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"86'33","reported_seq":216,"reported_epoch":90,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:20.887371+0000","last_change":"2024-09-16T03:37:55.142030+0000","last_active":"2024-09-16T03:38:20.887371+0000","last_peered":"2024-09-16T03:38:20.887371+0000","last_clean":"2024-09-16T03:37:51.442314+0000","last_became_active":"2024-09-16T03:37:54.760424+0000","last_became_peered":"2024-09-16T03:37:54.760424+0000","last_unstale":"2024-09-16T03:38:20.887371+0000","last_undegraded":"2024-09-16T03:38:20.887371+0000","last_fullsized":"2024-09-16T03:38:20.887371+0000","mapping_epoch":68,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'21","last_scrub_stamp":"2024-09-16T03:37:39.685904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.685904+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":28,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":949,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613505,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":7257,"num_write":14,"num_write_kb":3834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,5],"avail_no_missing":["11","5","8"],"object_location_counts":[{"shards":"5,8,11","objects":1}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.149696+0000","last_change":"2024-09-16T03:37:52.652365+0000","last_active":"2024-09-16T03:38:21.149696+0000","last_peered":"2024-09-16T03:38:21.149696+0000","last_clean":"2024-09-16T03:38:21.149696+0000","last_became_active":"2024-09-16T03:37:52.651907+0000","last_became_peered":"2024-09-16T03:37:52.651907+0000","last_unstale":"2024-09-16T03:38:21.149696+0000","last_undegraded":"2024-09-16T03:38:21.149696+0000","last_fullsized":"2024-09-16T03:38:21.149696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:07.210820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":166,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.397988+0000","last_change":"2024-09-16T03:37:39.853712+0000","last_active":"2024-09-16T03:38:21.397988+0000","last_peered":"2024-09-16T03:38:21.397988+0000","last_clean":"2024-09-16T03:38:21.397988+0000","last_became_active":"2024-09-16T03:37:31.150931+0000","last_became_peered":"2024-09-16T03:37:31.150931+0000","last_unstale":"2024-09-16T03:38:21.397988+0000","last_undegraded":"2024-09-16T03:38:21.397988+0000","last_fullsized":"2024-09-16T03:38:21.397988+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:39.853620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.853620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:57.206393+0000","scrub_duration":374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"90'174","reported_seq":316,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.398129+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:38:21.398129+0000","last_peered":"2024-09-16T03:38:21.398129+0000","last_clean":"2024-09-16T03:38:21.398129+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:38:21.398129+0000","last_undegraded":"2024-09-16T03:38:21.398129+0000","last_fullsized":"2024-09-16T03:38:21.398129+0000","mapping_epoch":31,"log_start":"87'171","ondisk_log_start":"87'171","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":183,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.679811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18597303,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":153,"num_read_kb":3806,"num_write":208,"num_write_kb":54453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.417413+0000","last_change":"2024-09-16T03:38:12.795978+0000","last_active":"2024-09-16T03:38:23.417413+0000","last_peered":"2024-09-16T03:38:23.417413+0000","last_clean":"2024-09-16T03:38:23.417413+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:38:23.417413+0000","last_undegraded":"2024-09-16T03:38:23.417413+0000","last_fullsized":"2024-09-16T03:38:23.417413+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:38:12.795855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.795855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:13.088740+0000","scrub_duration":501,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.4","version":"87'30","reported_seq":222,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.887534+0000","last_change":"2024-09-16T03:37:46.884372+0000","last_active":"2024-09-16T03:38:20.887534+0000","last_peered":"2024-09-16T03:38:20.887534+0000","last_clean":"2024-09-16T03:38:20.887534+0000","last_became_active":"2024-09-16T03:37:31.249346+0000","last_became_peered":"2024-09-16T03:37:31.249346+0000","last_unstale":"2024-09-16T03:38:20.887534+0000","last_undegraded":"2024-09-16T03:38:20.887534+0000","last_fullsized":"2024-09-16T03:38:20.887534+0000","mapping_epoch":32,"log_start":"82'27","ondisk_log_start":"82'27","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'15","last_scrub_stamp":"2024-09-16T03:37:46.884284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:46.884284+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":27,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:55.612449+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":9455,"num_write":19,"num_write_kb":4676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.0","version":"92'184","reported_seq":347,"reported_epoch":94,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:24.479219+0000","last_change":"2024-09-16T03:37:56.179604+0000","last_active":"2024-09-16T03:38:24.479219+0000","last_peered":"2024-09-16T03:38:24.479219+0000","last_clean":"2024-09-16T03:37:50.666338+0000","last_became_active":"2024-09-16T03:37:55.757919+0000","last_became_peered":"2024-09-16T03:37:55.757919+0000","last_unstale":"2024-09-16T03:38:24.479219+0000","last_undegraded":"2024-09-16T03:38:24.479219+0000","last_fullsized":"2024-09-16T03:38:24.479219+0000","mapping_epoch":69,"log_start":"88'181","ondisk_log_start":"88'181","created":31,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":202,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17127584,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":167,"num_read_kb":11031,"num_write":221,"num_write_kb":63387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,9,1],"avail_no_missing":["4","1","9"],"object_location_counts":[{"shards":"1,4,9","objects":14}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":151,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.397974+0000","last_change":"2024-09-16T03:38:15.949627+0000","last_active":"2024-09-16T03:38:21.397974+0000","last_peered":"2024-09-16T03:38:21.397974+0000","last_clean":"2024-09-16T03:38:21.397974+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:38:21.397974+0000","last_undegraded":"2024-09-16T03:38:21.397974+0000","last_fullsized":"2024-09-16T03:38:21.397974+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:15.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:24.286324+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.7","version":"88'55","reported_seq":262,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.375331+0000","last_change":"2024-09-16T03:37:48.400988+0000","last_active":"2024-09-16T03:38:23.375331+0000","last_peered":"2024-09-16T03:38:23.375331+0000","last_clean":"2024-09-16T03:38:23.375331+0000","last_became_active":"2024-09-16T03:37:31.153814+0000","last_became_peered":"2024-09-16T03:37:31.153814+0000","last_unstale":"2024-09-16T03:38:23.375331+0000","last_undegraded":"2024-09-16T03:38:23.375331+0000","last_fullsized":"2024-09-16T03:38:23.375331+0000","mapping_epoch":32,"log_start":"86'52","ondisk_log_start":"86'52","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'32","last_scrub_stamp":"2024-09-16T03:37:48.400911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:48.400911+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":50,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:10.129191+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6857256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":80936,"num_write":28,"num_write_kb":8438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1","version":"86'130","reported_seq":271,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.249782+0000","last_change":"2024-09-16T03:38:15.890094+0000","last_active":"2024-09-16T03:38:21.249782+0000","last_peered":"2024-09-16T03:38:21.249782+0000","last_clean":"2024-09-16T03:38:21.249782+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:38:21.249782+0000","last_undegraded":"2024-09-16T03:38:21.249782+0000","last_fullsized":"2024-09-16T03:38:21.249782+0000","mapping_epoch":31,"log_start":"85'127","ondisk_log_start":"85'127","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'125","last_scrub_stamp":"2024-09-16T03:38:15.889996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.889996+0000","objects_scrubbed":18,"log_size":3,"log_dups_size":138,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:35.350472+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11382467,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":110,"num_read_kb":743,"num_write":153,"num_write_kb":40662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":158,"reported_epoch":93,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:38:24.257066+0000","last_change":"2024-09-16T03:38:24.155601+0000","last_active":"2024-09-16T03:38:24.257066+0000","last_peered":"2024-09-16T03:38:24.257066+0000","last_clean":"2024-09-16T03:38:24.257066+0000","last_became_active":"2024-09-16T03:37:55.758504+0000","last_became_peered":"2024-09-16T03:37:55.758504+0000","last_unstale":"2024-09-16T03:38:24.257066+0000","last_undegraded":"2024-09-16T03:38:24.257066+0000","last_fullsized":"2024-09-16T03:38:24.257066+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,10],"acting":[8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.6","version":"75'39","reported_seq":268,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.420798+0000","last_change":"2024-09-16T03:37:56.444659+0000","last_active":"2024-09-16T03:38:23.420798+0000","last_peered":"2024-09-16T03:38:23.420798+0000","last_clean":"2024-09-16T03:38:23.420798+0000","last_became_active":"2024-09-16T03:37:56.444134+0000","last_became_peered":"2024-09-16T03:37:56.444134+0000","last_unstale":"2024-09-16T03:38:23.420798+0000","last_undegraded":"2024-09-16T03:38:23.420798+0000","last_fullsized":"2024-09-16T03:38:23.420798+0000","mapping_epoch":70,"log_start":"63'35","ondisk_log_start":"63'35","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'32","last_scrub_stamp":"2024-09-16T03:37:47.613282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.613282+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":34,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:52.751036+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924418,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":24426,"num_write":9,"num_write_kb":2208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1264675,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"89'150","reported_seq":358,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-16T03:38:20.887390+0000","last_change":"2024-09-16T03:38:01.603352+0000","last_active":"2024-09-16T03:38:20.887390+0000","last_peered":"2024-09-16T03:38:20.887390+0000","last_clean":"2024-09-16T03:38:20.887390+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:38:20.887390+0000","last_undegraded":"2024-09-16T03:38:20.887390+0000","last_fullsized":"2024-09-16T03:38:20.887390+0000","mapping_epoch":74,"log_start":"84'147","ondisk_log_start":"84'147","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":156,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19248469,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":139,"num_read_kb":7648,"num_write":174,"num_write_kb":52654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920256+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:38:23.920256+0000","last_peered":"2024-09-16T03:38:23.920256+0000","last_clean":"2024-09-16T03:38:23.920256+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:38:23.920256+0000","last_undegraded":"2024-09-16T03:38:23.920256+0000","last_fullsized":"2024-09-16T03:38:23.920256+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:01.672864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":287,"reported_epoch":90,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:20.887313+0000","last_change":"2024-09-16T03:37:54.760571+0000","last_active":"2024-09-16T03:38:20.887313+0000","last_peered":"2024-09-16T03:38:20.887313+0000","last_clean":"2024-09-16T03:37:51.442238+0000","last_became_active":"2024-09-16T03:37:54.660154+0000","last_became_peered":"2024-09-16T03:37:54.660154+0000","last_unstale":"2024-09-16T03:38:20.887313+0000","last_undegraded":"2024-09-16T03:38:20.887313+0000","last_fullsized":"2024-09-16T03:38:20.887313+0000","mapping_epoch":68,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:36:50.058111+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,7],"acting":[11,0,5],"avail_no_missing":["11","0","5"],"object_location_counts":[{"shards":"0,5,11","objects":2}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.5","version":"90'66","reported_seq":278,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-16T03:38:21.498233+0000","last_change":"2024-09-16T03:37:47.718455+0000","last_active":"2024-09-16T03:38:21.498233+0000","last_peered":"2024-09-16T03:38:21.498233+0000","last_clean":"2024-09-16T03:38:21.498233+0000","last_became_active":"2024-09-16T03:37:31.248101+0000","last_became_peered":"2024-09-16T03:37:31.248101+0000","last_unstale":"2024-09-16T03:38:21.498233+0000","last_undegraded":"2024-09-16T03:38:21.498233+0000","last_fullsized":"2024-09-16T03:38:21.498233+0000","mapping_epoch":32,"log_start":"89'63","ondisk_log_start":"89'63","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'44","last_scrub_stamp":"2024-09-16T03:37:47.718363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.718363+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":62,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:48.218136+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3529470,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":62544,"num_write":24,"num_write_kb":6749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.8","version":"91'62","reported_seq":272,"reported_epoch":93,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:23.820079+0000","last_change":"2024-09-16T03:37:54.958763+0000","last_active":"2024-09-16T03:38:23.820079+0000","last_peered":"2024-09-16T03:38:23.820079+0000","last_clean":"2024-09-16T03:37:41.106904+0000","last_became_active":"2024-09-16T03:37:54.759604+0000","last_became_peered":"2024-09-16T03:37:54.759604+0000","last_unstale":"2024-09-16T03:38:23.820079+0000","last_undegraded":"2024-09-16T03:38:23.820079+0000","last_fullsized":"2024-09-16T03:38:23.820079+0000","mapping_epoch":68,"log_start":"90'59","ondisk_log_start":"90'59","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'38","last_scrub_stamp":"2024-09-16T03:37:40.343801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:40.343801+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3899224,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":58565,"num_write":28,"num_write_kb":8269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":147,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.920369+0000","last_change":"2024-09-16T03:37:52.558364+0000","last_active":"2024-09-16T03:38:23.920369+0000","last_peered":"2024-09-16T03:38:23.920369+0000","last_clean":"2024-09-16T03:38:23.920369+0000","last_became_active":"2024-09-16T03:37:52.558058+0000","last_became_peered":"2024-09-16T03:37:52.558058+0000","last_unstale":"2024-09-16T03:38:23.920369+0000","last_undegraded":"2024-09-16T03:38:23.920369+0000","last_fullsized":"2024-09-16T03:38:23.920369+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:46.812106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:46.812106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:50.592391+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.a","version":"92'108","reported_seq":413,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.518020+0000","last_change":"2024-09-16T03:37:06.386519+0000","last_active":"2024-09-16T03:38:23.518020+0000","last_peered":"2024-09-16T03:38:23.518020+0000","last_clean":"2024-09-16T03:38:23.518020+0000","last_became_active":"2024-09-16T03:37:06.286063+0000","last_became_peered":"2024-09-16T03:37:06.286063+0000","last_unstale":"2024-09-16T03:38:23.518020+0000","last_undegraded":"2024-09-16T03:38:23.518020+0000","last_fullsized":"2024-09-16T03:38:23.518020+0000","mapping_epoch":32,"log_start":"92'105","ondisk_log_start":"92'105","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":105,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16437312,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":394,"num_read_kb":298233,"num_write":110,"num_write_kb":33612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.b","version":"86'89","reported_seq":347,"reported_epoch":91,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:21.429878+0000","last_change":"2024-09-16T03:37:55.559928+0000","last_active":"2024-09-16T03:38:21.429878+0000","last_peered":"2024-09-16T03:38:21.429878+0000","last_clean":"2024-09-16T03:37:52.454742+0000","last_became_active":"2024-09-16T03:37:55.559094+0000","last_became_peered":"2024-09-16T03:37:55.559094+0000","last_unstale":"2024-09-16T03:38:21.429878+0000","last_undegraded":"2024-09-16T03:38:21.429878+0000","last_fullsized":"2024-09-16T03:38:21.429878+0000","mapping_epoch":69,"log_start":"85'86","ondisk_log_start":"85'86","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":86,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13143074,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":281,"num_read_kb":257121,"num_write":96,"num_write_kb":30271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,12],"acting":[15,4,9],"avail_no_missing":["15","4","9"],"object_location_counts":[{"shards":"4,9,15","objects":4}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.c","version":"90'55","reported_seq":231,"reported_epoch":92,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:22.711971+0000","last_change":"2024-09-16T03:37:54.763651+0000","last_active":"2024-09-16T03:38:22.711971+0000","last_peered":"2024-09-16T03:38:22.711971+0000","last_clean":"2024-09-16T03:37:50.453398+0000","last_became_active":"2024-09-16T03:37:54.762708+0000","last_became_peered":"2024-09-16T03:37:54.762708+0000","last_unstale":"2024-09-16T03:38:22.711971+0000","last_undegraded":"2024-09-16T03:38:22.711971+0000","last_fullsized":"2024-09-16T03:38:22.711971+0000","mapping_epoch":68,"log_start":"84'51","ondisk_log_start":"84'51","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":51,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3865610,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":59,"num_read_kb":29471,"num_write":59,"num_write_kb":16573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,11],"acting":[12,15,1],"avail_no_missing":["12","1","15"],"object_location_counts":[{"shards":"1,12,15","objects":1}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.d","version":"92'14","reported_seq":203,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-16T03:38:22.646482+0000","last_change":"2024-09-16T03:38:17.347261+0000","last_active":"2024-09-16T03:38:22.646482+0000","last_peered":"2024-09-16T03:38:22.646482+0000","last_clean":"2024-09-16T03:38:22.646482+0000","last_became_active":"2024-09-16T03:37:58.450690+0000","last_became_peered":"2024-09-16T03:37:58.450690+0000","last_unstale":"2024-09-16T03:38:22.646482+0000","last_undegraded":"2024-09-16T03:38:22.646482+0000","last_fullsized":"2024-09-16T03:38:22.646482+0000","mapping_epoch":72,"log_start":"91'11","ondisk_log_start":"91'11","created":32,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:46.832564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":28,"num_read_kb":16968,"num_write":14,"num_write_kb":3863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2160080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.e","version":"92'59","reported_seq":270,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-16T03:38:24.379194+0000","last_change":"2024-09-16T03:38:11.825753+0000","last_active":"2024-09-16T03:38:24.379194+0000","last_peered":"2024-09-16T03:38:24.379194+0000","last_clean":"2024-09-16T03:38:24.379194+0000","last_became_active":"2024-09-16T03:37:06.037035+0000","last_became_peered":"2024-09-16T03:37:06.037035+0000","last_unstale":"2024-09-16T03:38:24.379194+0000","last_undegraded":"2024-09-16T03:38:24.379194+0000","last_fullsized":"2024-09-16T03:38:24.379194+0000","mapping_epoch":32,"log_start":"89'56","ondisk_log_start":"89'56","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'42","last_scrub_stamp":"2024-09-16T03:38:11.825685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:11.825685+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:13.309016+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10504375,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":140,"num_read_kb":134119,"num_write":61,"num_write_kb":17881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"92'81","reported_seq":328,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-16T03:38:23.375274+0000","last_change":"2024-09-16T03:38:11.380208+0000","last_active":"2024-09-16T03:38:23.375274+0000","last_peered":"2024-09-16T03:38:23.375274+0000","last_clean":"2024-09-16T03:38:23.375274+0000","last_became_active":"2024-09-16T03:37:06.386648+0000","last_became_peered":"2024-09-16T03:37:06.386648+0000","last_unstale":"2024-09-16T03:38:23.375274+0000","last_undegraded":"2024-09-16T03:38:23.375274+0000","last_fullsized":"2024-09-16T03:38:23.375274+0000","mapping_epoch":32,"log_start":"92'78","ondisk_log_start":"92'78","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'64","last_scrub_stamp":"2024-09-16T03:38:11.380094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:11.380094+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":78,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.995078+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7865742,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":240,"num_read_kb":207525,"num_write":80,"num_write_kb":21564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":122960144,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":2008,"num_read_kb":1704631,"num_write":754,"num_write_kb":218225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":13936073,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":16,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":231448576,"data_stored":231092655,"data_compressed":703374,"data_compressed_allocated":753664,"data_compressed_original":1626112,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":78,"acting":78,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":66355823,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":569,"num_read_kb":23228,"num_write":756,"num_write_kb":211156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":125169664,"data_stored":134745554,"data_compressed":8102593,"data_compressed_allocated":8658944,"data_compressed_original":18526208,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":1586784,"data_compressed":93744,"data_compressed_allocated":122880,"data_compressed_original":1536000,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247828,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84852,"kb_used_data":50516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94286988,"statfs":{"total":96636764160,"available":96549875712,"internally_reserved":0,"allocated":51728384,"data_stored":52030200,"data_compressed":869744,"data_compressed_allocated":909312,"data_compressed_original":1728512,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247829,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66860,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94304980,"statfs":{"total":96636764160,"available":96568299520,"internally_reserved":0,"allocated":34947072,"data_stored":36022264,"data_compressed":747424,"data_compressed_allocated":823296,"data_compressed_original":2371584,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,5,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247828,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53784,"kb_used_data":22596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318056,"statfs":{"total":96636764160,"available":96581689344,"internally_reserved":0,"allocated":23138304,"data_stored":23597166,"data_compressed":519317,"data_compressed_allocated":573440,"data_compressed_original":1421312,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280534,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46100,"kb_used_data":14792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325740,"statfs":{"total":96636764160,"available":96589557760,"internally_reserved":0,"allocated":15147008,"data_stored":16233105,"data_compressed":637211,"data_compressed_allocated":704512,"data_compressed_original":2084864,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280534,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280533,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313238,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35928,"kb_used_data":6332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335912,"statfs":{"total":96636764160,"available":96599973888,"internally_reserved":0,"allocated":6483968,"data_stored":6226197,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280534,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55140,"kb_used_data":20524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316700,"statfs":{"total":96636764160,"available":96580300800,"internally_reserved":0,"allocated":21016576,"data_stored":21485657,"data_compressed":457444,"data_compressed_allocated":507904,"data_compressed_original":1302528,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280533,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92804,"kb_used_data":59072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94279036,"statfs":{"total":96636764160,"available":96541732864,"internally_reserved":0,"allocated":60489728,"data_stored":62445719,"data_compressed":1689383,"data_compressed_allocated":1830912,"data_compressed_original":4231168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313238,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280533,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38736,"kb_used_data":7392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333104,"statfs":{"total":96636764160,"available":96597098496,"internally_reserved":0,"allocated":7569408,"data_stored":7319754,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280533,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65712,"kb_used_data":31172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94306128,"statfs":{"total":96636764160,"available":96569475072,"internally_reserved":0,"allocated":31920128,"data_stored":32305674,"data_compressed":934183,"data_compressed_allocated":978944,"data_compressed_original":1724416,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":0,"seq":94489280533,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280534,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66568,"kb_used_data":37764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94305272,"statfs":{"total":96636764160,"available":96568598528,"internally_reserved":0,"allocated":38670336,"data_stored":38542593,"data_compressed":237846,"data_compressed_allocated":258048,"data_compressed_original":544768,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,6,7,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313237,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35092,"kb_used_data":4728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336748,"statfs":{"total":96636764160,"available":96600829952,"internally_reserved":0,"allocated":4841472,"data_stored":4579760,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,3,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45840,"kb_used_data":14652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326000,"statfs":{"total":96636764160,"available":96589824000,"internally_reserved":0,"allocated":15003648,"data_stored":15394064,"data_compressed":835356,"data_compressed_allocated":872448,"data_compressed_original":1597440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10067968,"data_stored":10938272,"data_compressed":597443,"data_compressed_allocated":651264,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9564160,"data_stored":10197664,"data_compressed":614635,"data_compressed_allocated":651264,"data_compressed_original":1310720,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12275712,"data_stored":12973722,"data_compressed":826836,"data_compressed_allocated":860160,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6619136,"data_stored":7282263,"data_compressed":361389,"data_compressed_allocated":401408,"data_compressed_original":1085440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":26251264,"data_stored":28418199,"data_compressed":1573467,"data_compressed_allocated":1703936,"data_compressed_original":3944448,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5505024,"data_stored":6198979,"data_compressed":387998,"data_compressed_allocated":430080,"data_compressed_original":1134592,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10113024,"data_stored":10723430,"data_compressed":838128,"data_compressed_allocated":872448,"data_compressed_original":1507328,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":9564160,"data_stored":10197664,"data_compressed":614635,"data_compressed_allocated":651264,"data_compressed_original":1310720,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":10682368,"data_stored":11705310,"data_compressed":673261,"data_compressed_allocated":733184,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":13021184,"data_stored":13711548,"data_compressed":826836,"data_compressed_allocated":860160,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4280320,"data_stored":4251638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7155712,"data_stored":7111032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4349952,"data_stored":4327588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13676544,"data_stored":13690638,"data_compressed":87535,"data_compressed_allocated":94208,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":33522141,"data_compressed":107396,"data_compressed_allocated":114688,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6242304,"data_stored":6199120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16748544,"data_stored":16807005,"data_compressed":122799,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":6864896,"data_stored":6814948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":21086208,"data_stored":21075976,"data_compressed":87535,"data_compressed_allocated":94208,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":9072640,"data_stored":9025673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":37957632,"data_stored":38037194,"data_compressed":229326,"data_compressed_allocated":245760,"data_compressed_original":520192,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":21082112,"data_stored":21005106,"data_compressed":34395,"data_compressed_allocated":36864,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4087808,"data_stored":4063988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5746688,"data_stored":5712182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":38473728,"data_stored":38368386,"data_compressed":34388,"data_compressed_allocated":36864,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:38:25.486 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-16T03:38:25.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 36 2024-09-16T03:38:25.522 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:25.520+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:25.575 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:25.575 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:25.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.575 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2819 4.a 4:5e3c0760:::smithi15736696-24:head [sparse-read 0~2000066] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-16T03:38:25.575 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:25.428627+0000", 2024-09-16T03:38:25.575 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.14477087699999999, 2024-09-16T03:38:25.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.14480199799999999, 2024-09-16T03:38:25.575 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.575 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.576 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:25.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.576 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2819 2024-09-16T03:38:25.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.576 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.428627+0000", 2024-09-16T03:38:25.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.428627+0000", 2024-09-16T03:38:25.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.428628+0000", 2024-09-16T03:38:25.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:38:25.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.428630+0000", 2024-09-16T03:38:25.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2230000000000001e-06 2024-09-16T03:38:25.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.428634+0000", 2024-09-16T03:38:25.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7309999999999999e-06 2024-09-16T03:38:25.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.528716+0000", 2024-09-16T03:38:25.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10008234100000001 2024-09-16T03:38:25.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.573304+0000", 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044587688 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.573366+0000", 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1921999999999994e-05 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.579 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:38:25.580 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-09-16T03:38:25.580 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:25.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:38:25.628 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:25.626+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:25.628 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:25.626+0000 7f6528e68640 -1 osd.1 94 handle_osd_map injecting map crc failure 2024-09-16T03:38:25.689 INFO:tasks.rados.rados.0.smithi157.stdout:1285: expect (ObjNum 224 snap 0 seq_num 224) 2024-09-16T03:38:25.694 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:38:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:38:25.695 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:25.704 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:38:25.724 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:25.721+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:25.824 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:25.823+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:25.852 INFO:tasks.rados.rados.0.smithi157.stdout:1286: expect (ObjNum 238 snap 0 seq_num 238) 2024-09-16T03:38:25.855 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:25.856 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:38:25.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:38:25.856 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:25.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.856 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4338.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:25.856 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:38:25.856 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 90.755906873000001, 2024-09-16T03:38:25.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30502727499999999, 2024-09-16T03:38:25.856 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.856 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.856 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.856 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4338", 2024-09-16T03:38:25.857 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3073758685", 2024-09-16T03:38:25.857 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-16T03:38:25.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.857 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:38:25.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097101+0000", 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.915e-06 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097111+0000", 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0589e-05 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097130+0000", 2024-09-16T03:38:25.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8958999999999999e-05 2024-09-16T03:38:25.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097141+0000", 2024-09-16T03:38:25.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0616e-05 2024-09-16T03:38:25.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097227+0000", 2024-09-16T03:38:25.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6322000000000005e-05 2024-09-16T03:38:25.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097232+0000", 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2059999999999998e-06 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200235+0000", 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103002945 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200318+0000", 2024-09-16T03:38:25.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2515999999999994e-05 2024-09-16T03:38:25.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,10", 2024-09-16T03:38:25.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200429+0000", 2024-09-16T03:38:25.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011158999999999999 2024-09-16T03:38:25.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201478+0000", 2024-09-16T03:38:25.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00104893 2024-09-16T03:38:25.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201942+0000", 2024-09-16T03:38:25.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046354700000000003 2024-09-16T03:38:25.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402048+0000", 2024-09-16T03:38:25.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20010616000000001 2024-09-16T03:38:25.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402089+0000", 2024-09-16T03:38:25.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0762e-05 2024-09-16T03:38:25.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402125+0000", 2024-09-16T03:38:25.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6217999999999998e-05 2024-09-16T03:38:25.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.863 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.863 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:73 4.a 4:532814f2:::smithi15736696-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:25.863 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:38:25.863 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 73.541744737000002, 2024-09-16T03:38:25.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.362642662, 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311261+0000", 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.562e-06 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311925+0000", 2024-09-16T03:38:25.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066385499999999998 2024-09-16T03:38:25.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311937+0000", 2024-09-16T03:38:25.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1881e-05 2024-09-16T03:38:25.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311943+0000", 2024-09-16T03:38:25.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6999999999999996e-06 2024-09-16T03:38:25.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311988+0000", 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5358000000000002e-05 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312070+0000", 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1735000000000003e-05 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312167+0000", 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6685e-05 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.313661+0000", 2024-09-16T03:38:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001494324 2024-09-16T03:38:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.414759+0000", 2024-09-16T03:38:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10109850300000001 2024-09-16T03:38:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673821+0000", 2024-09-16T03:38:25.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.25906127400000001 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673858+0000", 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7620999999999998e-05 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673902+0000", 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4163999999999999e-05 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:303 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.441285906999994, 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50956986000000004, 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 303 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.870 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:38:25.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:38:25.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411721+0000", 2024-09-16T03:38:25.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0200000000000001e-06 2024-09-16T03:38:25.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411729+0000", 2024-09-16T03:38:25.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1240000000000005e-06 2024-09-16T03:38:25.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411744+0000", 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099000000000001e-05 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411751+0000", 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.029e-06 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511892+0000", 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100140879 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511961+0000", 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9379000000000001e-05 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.613647+0000", 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101685994 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.614294+0000", 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064692799999999996 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.714962+0000", 2024-09-16T03:38:25.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066765 2024-09-16T03:38:25.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921157+0000", 2024-09-16T03:38:25.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206195463 2024-09-16T03:38:25.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921233+0000", 2024-09-16T03:38:25.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5446999999999998e-05 2024-09-16T03:38:25.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921288+0000", 2024-09-16T03:38:25.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5847999999999998e-05 2024-09-16T03:38:25.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.876 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.876 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:503 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:25.876 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:38:25.876 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.031211354999996, 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60521621599999997, 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 503 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:38:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:38:25.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821794+0000", 2024-09-16T03:38:25.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-09-16T03:38:25.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821798+0000", 2024-09-16T03:38:25.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2030000000000002e-06 2024-09-16T03:38:25.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821804+0000", 2024-09-16T03:38:25.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:38:25.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821810+0000", 2024-09-16T03:38:25.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4249999999999997e-06 2024-09-16T03:38:25.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922212+0000", 2024-09-16T03:38:25.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040228800000001 2024-09-16T03:38:25.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922307+0000", 2024-09-16T03:38:25.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5089999999999999e-05 2024-09-16T03:38:25.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:25.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.126007+0000", 2024-09-16T03:38:25.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20369931499999999 2024-09-16T03:38:25.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.127605+0000", 2024-09-16T03:38:25.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001598213 2024-09-16T03:38:25.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.327263+0000", 2024-09-16T03:38:25.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199657905 2024-09-16T03:38:25.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426900+0000", 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099637217 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426963+0000", 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.427009+0000", 2024-09-16T03:38:25.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6533000000000003e-05 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1227 4.a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v18,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.177035091, 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.44160531400000003, 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.883 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.884 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.884 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1227 2024-09-16T03:38:25.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.884 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:38:25.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:38:25.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675972+0000", 2024-09-16T03:38:25.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7269999999999999e-06 2024-09-16T03:38:25.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675981+0000", 2024-09-16T03:38:25.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.166e-06 2024-09-16T03:38:25.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675996+0000", 2024-09-16T03:38:25.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4901e-05 2024-09-16T03:38:25.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676005+0000", 2024-09-16T03:38:25.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4579999999999993e-06 2024-09-16T03:38:25.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676081+0000", 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6157999999999998e-05 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676188+0000", 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106871 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.016594+0000", 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34040622300000001 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017089+0000", 2024-09-16T03:38:25.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049523899999999997 2024-09-16T03:38:25.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017779+0000", 2024-09-16T03:38:25.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069032200000000005 2024-09-16T03:38:25.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117504+0000", 2024-09-16T03:38:25.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099724207999999995 2024-09-16T03:38:25.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117543+0000", 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9208999999999997e-05 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117575+0000", 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1832000000000003e-05 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1523 4.a 4:50107449:::smithi15736696-27:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.489993765000001, 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.410522742, 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1523 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363012+0000", 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.269e-06 2024-09-16T03:38:25.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363016+0000", 2024-09-16T03:38:25.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8920000000000002e-06 2024-09-16T03:38:25.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363027+0000", 2024-09-16T03:38:25.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1079e-05 2024-09-16T03:38:25.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463160+0000", 2024-09-16T03:38:25.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10013267100000001 2024-09-16T03:38:25.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463297+0000", 2024-09-16T03:38:25.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013717699999999999 2024-09-16T03:38:25.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463381+0000", 2024-09-16T03:38:25.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3701000000000004e-05 2024-09-16T03:38:25.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.773533+0000", 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31015295300000001 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1677 4.13 e64/50 4:c8492bfc:::smithi15736696-42:head v 64'37, mlcod=60'35)", 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.136995+0000", 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.716009474000003, 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30289115700000002, 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.894 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.136995+0000", 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.136995+0000", 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.136997+0000", 2024-09-16T03:38:25.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2450000000000001e-06 2024-09-16T03:38:25.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.137887+0000", 2024-09-16T03:38:25.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088958899999999998 2024-09-16T03:38:25.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.137904+0000", 2024-09-16T03:38:25.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6843000000000001e-05 2024-09-16T03:38:25.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.237997+0000", 2024-09-16T03:38:25.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100093631 2024-09-16T03:38:25.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.338117+0000", 2024-09-16T03:38:25.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100120158 2024-09-16T03:38:25.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.338133+0000", 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5624000000000002e-05 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.439797+0000", 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101664168 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.439886+0000", 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8899000000000004e-05 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.898 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.899 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.899 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1690 4.a 4:51338165:::smithi15736696-4:head [assert-version v63,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:25.899 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:38:25.899 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.402721098000001, 2024-09-16T03:38:25.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52002349699999995, 2024-09-16T03:38:25.899 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.899 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.899 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.899 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.899 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.899 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.900 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1690 2024-09-16T03:38:25.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.900 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:38:25.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:38:25.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450286+0000", 2024-09-16T03:38:25.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-16T03:38:25.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450293+0000", 2024-09-16T03:38:25.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5089999999999996e-06 2024-09-16T03:38:25.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450308+0000", 2024-09-16T03:38:25.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4955e-05 2024-09-16T03:38:25.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450316+0000", 2024-09-16T03:38:25.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7079999999999997e-06 2024-09-16T03:38:25.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450386+0000", 2024-09-16T03:38:25.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9715999999999995e-05 2024-09-16T03:38:25.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450478+0000", 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2610999999999998e-05 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.665802+0000", 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21532342700000001 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668724+0000", 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029220600000000002 2024-09-16T03:38:25.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668758+0000", 2024-09-16T03:38:25.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.434e-05 2024-09-16T03:38:25.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970243+0000", 2024-09-16T03:38:25.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30148501 2024-09-16T03:38:25.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970278+0000", 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4965000000000002e-05 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970307+0000", 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8968000000000001e-05 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1654 4.12 4:4a9d235d:::smithi15736696-1:head [write 652138~461367 in=461367b,stat out=16b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:38:25.905 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.310967607000002, 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42790198699999998, 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1654 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:38:25.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542039+0000", 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0250000000000001e-06 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542682+0000", 2024-09-16T03:38:25.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064305099999999995 2024-09-16T03:38:25.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542698+0000", 2024-09-16T03:38:25.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6469000000000001e-05 2024-09-16T03:38:25.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542710+0000", 2024-09-16T03:38:25.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1173e-05 2024-09-16T03:38:25.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542767+0000", 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7584000000000003e-05 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542775+0000", 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6149999999999999e-06 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666597+0000", 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123821815 2024-09-16T03:38:25.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666745+0000", 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148675 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666935+0000", 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018963899999999999 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969207+0000", 2024-09-16T03:38:25.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302272031 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969374+0000", 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016686699999999999 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969533+0000", 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015909299999999999 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969560+0000", 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7148000000000002e-05 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969631+0000", 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1174000000000007e-05 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969685+0000", 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3613000000000003e-05 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969764+0000", 2024-09-16T03:38:25.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9281000000000002e-05 2024-09-16T03:38:25.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969939+0000", 2024-09-16T03:38:25.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473400000000001 2024-09-16T03:38:25.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.913 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1655 4.12 4:4a9d235d:::smithi15736696-1:head [write 1574940~677198 in=677198b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:38:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.310275488999999, 2024-09-16T03:38:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42773147700000003, 2024-09-16T03:38:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1655 2024-09-16T03:38:25.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.915 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:38:25.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:38:25.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542730+0000", 2024-09-16T03:38:25.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.085e-06 2024-09-16T03:38:25.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543338+0000", 2024-09-16T03:38:25.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060826100000000002 2024-09-16T03:38:25.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543351+0000", 2024-09-16T03:38:25.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2789000000000001e-05 2024-09-16T03:38:25.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543360+0000", 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1970000000000002e-06 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543420+0000", 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9562999999999999e-05 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543431+0000", 2024-09-16T03:38:25.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0866999999999999e-05 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.768963+0000", 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.225532487 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769029+0000", 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5881000000000002e-05 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:38:25.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769134+0000", 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104923 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969639+0000", 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20050517900000001 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969978+0000", 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033848599999999998 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970206+0000", 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022867300000000001 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970248+0000", 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1433999999999997e-05 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970299+0000", 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0677999999999998e-05 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970338+0000", 2024-09-16T03:38:25.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9646e-05 2024-09-16T03:38:25.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970377+0000", 2024-09-16T03:38:25.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8439999999999998e-05 2024-09-16T03:38:25.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970460+0000", 2024-09-16T03:38:25.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3887999999999994e-05 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1656 4.12 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2252138] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.309629416999996, 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72904596899999996, 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.922 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1656 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543376+0000", 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4200000000000004e-07 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543383+0000", 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6290000000000004e-06 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543390+0000", 2024-09-16T03:38:25.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:38:25.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543396+0000", 2024-09-16T03:38:25.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.02e-06 2024-09-16T03:38:25.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543461+0000", 2024-09-16T03:38:25.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4876999999999996e-05 2024-09-16T03:38:25.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543465+0000", 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2139999999999998e-06 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970490+0000", 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42702478300000002 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970556+0000", 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6137999999999996e-05 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:38:25.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970670+0000", 2024-09-16T03:38:25.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114416 2024-09-16T03:38:25.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971375+0000", 2024-09-16T03:38:25.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070496000000000003 2024-09-16T03:38:25.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071773+0000", 2024-09-16T03:38:25.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100397945 2024-09-16T03:38:25.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071823+0000", 2024-09-16T03:38:25.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:38:25.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.072067+0000", 2024-09-16T03:38:25.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243856 2024-09-16T03:38:25.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.171991+0000", 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099924052999999999 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272176+0000", 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100184963 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272251+0000", 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5902000000000006e-05 2024-09-16T03:38:25.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272421+0000", 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016962800000000001 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1657 4.12 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.309597431, 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72934181899999995, 2024-09-16T03:38:25.930 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.931 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.931 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:25.931 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.933 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.933 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.933 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1657 2024-09-16T03:38:25.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.933 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:38:25.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:38:25.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543408+0000", 2024-09-16T03:38:25.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.13e-07 2024-09-16T03:38:25.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543410+0000", 2024-09-16T03:38:25.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7030000000000002e-06 2024-09-16T03:38:25.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543417+0000", 2024-09-16T03:38:25.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.567e-06 2024-09-16T03:38:25.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543423+0000", 2024-09-16T03:38:25.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0619999999999997e-06 2024-09-16T03:38:25.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543489+0000", 2024-09-16T03:38:25.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6315999999999993e-05 2024-09-16T03:38:25.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:25.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543498+0000", 2024-09-16T03:38:25.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7239999999999998e-06 2024-09-16T03:38:25.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970952+0000", 2024-09-16T03:38:25.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42745424399999998 2024-09-16T03:38:25.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:25.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971012+0000", 2024-09-16T03:38:25.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9771000000000001e-05 2024-09-16T03:38:25.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272463+0000", 2024-09-16T03:38:25.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30145090800000002 2024-09-16T03:38:25.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272558+0000", 2024-09-16T03:38:25.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4863000000000003e-05 2024-09-16T03:38:25.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272749+0000", 2024-09-16T03:38:25.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191048 2024-09-16T03:38:25.938 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.938 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.938 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2095 4.a 4:503187b3:::smithi15736696-34:head [assert-version v60,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e75)", 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.032567+0000", 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.820437645999998, 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30423177099999998, 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2095 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.939 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032567+0000", 2024-09-16T03:38:25.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032567+0000", 2024-09-16T03:38:25.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032571+0000", 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9609999999999999e-06 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032585+0000", 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3752e-05 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032608+0000", 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3683999999999999e-05 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032621+0000", 2024-09-16T03:38:25.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3087e-05 2024-09-16T03:38:25.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032707+0000", 2024-09-16T03:38:25.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5498e-05 2024-09-16T03:38:25.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.032814+0000", 2024-09-16T03:38:25.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107268 2024-09-16T03:38:25.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:25.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.235147+0000", 2024-09-16T03:38:25.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20233309899999999 2024-09-16T03:38:25.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.236234+0000", 2024-09-16T03:38:25.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010869969999999999 2024-09-16T03:38:25.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.236347+0000", 2024-09-16T03:38:25.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112335 2024-09-16T03:38:25.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.336668+0000", 2024-09-16T03:38:25.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10032134500000001 2024-09-16T03:38:25.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.336740+0000", 2024-09-16T03:38:25.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2047999999999999e-05 2024-09-16T03:38:25.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.336799+0000", 2024-09-16T03:38:25.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8696999999999997e-05 2024-09-16T03:38:25.945 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.945 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.945 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.945 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2105 4.a 4:51338165:::smithi15736696-4:head [sparse-read 0~887916,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:38:25.945 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:38:25.945 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.804133236999999, 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40244934799999998, 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2105 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:38:25.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:38:25.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048881+0000", 2024-09-16T03:38:25.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6080000000000002e-06 2024-09-16T03:38:25.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048889+0000", 2024-09-16T03:38:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7209999999999994e-06 2024-09-16T03:38:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048903+0000", 2024-09-16T03:38:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4538e-05 2024-09-16T03:38:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048912+0000", 2024-09-16T03:38:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2390000000000008e-06 2024-09-16T03:38:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149039+0000", 2024-09-16T03:38:25.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100126563 2024-09-16T03:38:25.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:38:25.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149044+0000", 2024-09-16T03:38:25.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0340000000000004e-06 2024-09-16T03:38:25.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451087+0000", 2024-09-16T03:38:25.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30204318000000002 2024-09-16T03:38:25.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451159+0000", 2024-09-16T03:38:25.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1627999999999999e-05 2024-09-16T03:38:25.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451321+0000", 2024-09-16T03:38:25.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016183699999999999 2024-09-16T03:38:25.950 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2106 4.a 4:51338165:::smithi15736696-4:head [sparse-read 0~887916] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.804072477999998, 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40254453200000001, 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.952 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2106 2024-09-16T03:38:25.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.952 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:38:25.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:38:25.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048933+0000", 2024-09-16T03:38:25.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:38:25.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048936+0000", 2024-09-16T03:38:25.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3890000000000001e-06 2024-09-16T03:38:25.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048943+0000", 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1250000000000004e-06 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048953+0000", 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8190000000000003e-06 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149113+0000", 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100160007 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149118+0000", 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7650000000000001e-06 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451341+0000", 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30222344699999998 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451381+0000", 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9517000000000001e-05 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451477+0000", 2024-09-16T03:38:25.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5656999999999996e-05 2024-09-16T03:38:25.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.956 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.956 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2107 4.a 4:51338165:::smithi15736696-4:head [read 1698310~486600,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:38:25.956 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:38:25.956 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.804042415000001, 2024-09-16T03:38:25.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40288642400000002, 2024-09-16T03:38:25.956 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2107 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:38:25.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048963+0000", 2024-09-16T03:38:25.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:38:25.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048967+0000", 2024-09-16T03:38:25.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9060000000000004e-06 2024-09-16T03:38:25.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048972+0000", 2024-09-16T03:38:25.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5910000000000003e-06 2024-09-16T03:38:25.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048977+0000", 2024-09-16T03:38:25.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9629999999999997e-06 2024-09-16T03:38:25.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149135+0000", 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015828 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149138+0000", 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.779e-06 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451506+0000", 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302367891 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451558+0000", 2024-09-16T03:38:25.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2374999999999999e-05 2024-09-16T03:38:25.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451849+0000", 2024-09-16T03:38:25.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029009899999999999 2024-09-16T03:38:25.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.961 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.961 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.961 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:2473 4.e e85/32 4:765e9179:::smithi15736696-44:head v 85'44, mlcod=78'41)", 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.012085+0000", 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8409199160000007, 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30349153200000001, 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.012085+0000", 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.012085+0000", 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.012086+0000", 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5689999999999999e-06 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.012353+0000", 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026733900000000001 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.012370+0000", 2024-09-16T03:38:25.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6392000000000002e-05 2024-09-16T03:38:25.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.112444+0000", 2024-09-16T03:38:25.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100074303 2024-09-16T03:38:25.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.312737+0000", 2024-09-16T03:38:25.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20029266900000001 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.312759+0000", 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2084999999999999e-05 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.315537+0000", 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027784659999999998 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.315576+0000", 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8708999999999998e-05 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2537 4.a 4:503187b3:::smithi15736696-34:head [assert-version v87,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:17.047079+0000", 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8059255850000007, 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.32976008000000001, 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.966 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2537 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047079+0000", 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047079+0000", 2024-09-16T03:38:25.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047081+0000", 2024-09-16T03:38:25.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6929999999999999e-06 2024-09-16T03:38:25.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047086+0000", 2024-09-16T03:38:25.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8680000000000004e-06 2024-09-16T03:38:25.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047099+0000", 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2309e-05 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047105+0000", 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7050000000000002e-06 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047157+0000", 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1057999999999998e-05 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047219+0000", 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2553999999999998e-05 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.149371+0000", 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102152112 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.149846+0000", 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047467400000000002 2024-09-16T03:38:25.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.183065+0000", 2024-09-16T03:38:25.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033219374000000003 2024-09-16T03:38:25.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.376793+0000", 2024-09-16T03:38:25.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.193728023 2024-09-16T03:38:25.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.376819+0000", 2024-09-16T03:38:25.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6142000000000001e-05 2024-09-16T03:38:25.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.376839+0000", 2024-09-16T03:38:25.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9568e-05 2024-09-16T03:38:25.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.972 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.972 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2543 4.a 4:5e3c0760:::smithi15736696-24:head [write 497540~612664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:25.972 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:17.072565+0000", 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7804395260000003, 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30335864600000001, 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2543 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.072565+0000", 2024-09-16T03:38:25.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.072565+0000", 2024-09-16T03:38:25.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.072567+0000", 2024-09-16T03:38:25.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1019999999999999e-06 2024-09-16T03:38:25.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.073277+0000", 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070954799999999995 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.073289+0000", 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2379e-05 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.073296+0000", 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8510000000000001e-06 2024-09-16T03:38:25.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.073346+0000", 2024-09-16T03:38:25.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0284000000000002e-05 2024-09-16T03:38:25.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.073398+0000", 2024-09-16T03:38:25.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2061999999999997e-05 2024-09-16T03:38:25.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:25.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.073481+0000", 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2756999999999994e-05 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.076141+0000", 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026604350000000001 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.180591+0000", 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10444935599999999 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.375845+0000", 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19525411600000001 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.375890+0000", 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5528000000000002e-05 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.375924+0000", 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3228000000000001e-05 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.978 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2687 4.a 4:503187b3:::smithi15736696-34:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:21.869464+0000", 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9835401419999998, 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31565330200000002, 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2687 2024-09-16T03:38:25.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.980 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:25.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:25.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869464+0000", 2024-09-16T03:38:25.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:25.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869464+0000", 2024-09-16T03:38:25.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:25.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869465+0000", 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.51e-07 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869467+0000", 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869473+0000", 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9560000000000002e-06 2024-09-16T03:38:25.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869488+0000", 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5333e-05 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869667+0000", 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017900699999999999 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869710+0000", 2024-09-16T03:38:25.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2749e-05 2024-09-16T03:38:25.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:25.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:25.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:25.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:22.185118+0000", 2024-09-16T03:38:25.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31540757600000002 2024-09-16T03:38:25.983 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.983 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.983 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.983 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:25.983 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:25.983 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:25.984 INFO:tasks.rados.rados.0.smithi157.stdout:1279: expect (ObjNum 190 snap 0 seq_num 190) 2024-09-16T03:38:25.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:38:25.986 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:25.983+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:26.078 INFO:tasks.rados.rados.0.smithi157.stdout:1280: finishing write tid 1 to smithi15736696-49 2024-09-16T03:38:26.078 INFO:tasks.rados.rados.0.smithi157.stdout:1282: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-16T03:38:26.085 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:26.084+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:26.134 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:26.134 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:26.134 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:38:26.134 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:26.143 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:38:26.186 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:26.185+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:26.207 INFO:tasks.rados.rados.0.smithi157.stdout:1280: finishing write tid 2 to smithi15736696-49 2024-09-16T03:38:26.207 INFO:tasks.rados.rados.0.smithi157.stdout:1280: finishing write tid 3 to smithi15736696-49 2024-09-16T03:38:26.207 INFO:tasks.rados.rados.0.smithi157.stdout:1280: finishing write tid 4 to smithi15736696-49 2024-09-16T03:38:26.207 INFO:tasks.rados.rados.0.smithi157.stdout:1280: oid 49 updating version 0 to 57 2024-09-16T03:38:26.207 INFO:tasks.rados.rados.0.smithi157.stdout:1280: oid 49 version 57 is already newer than 56 2024-09-16T03:38:26.207 INFO:tasks.rados.rados.0.smithi157.stdout:1280: oid 49 updating version 57 to 58 2024-09-16T03:38:26.207 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 58 (ObjNum 247 snap 0 seq_num 247) dirty exists 2024-09-16T03:38:26.207 INFO:tasks.rados.rados.0.smithi157.stdout:1280: left oid 49 (ObjNum 247 snap 0 seq_num 247) 2024-09-16T03:38:26.207 INFO:tasks.rados.rados.0.smithi157.stdout:1276: done (11 left) 2024-09-16T03:38:26.207 INFO:tasks.rados.rados.0.smithi157.stdout:1278: done (10 left) 2024-09-16T03:38:26.207 INFO:tasks.rados.rados.0.smithi157.stdout:1279: done (9 left) 2024-09-16T03:38:26.207 INFO:tasks.rados.rados.0.smithi157.stdout:1280: done (8 left) 2024-09-16T03:38:26.208 INFO:tasks.rados.rados.0.smithi157.stdout:1282: done (7 left) 2024-09-16T03:38:26.208 INFO:tasks.rados.rados.0.smithi157.stdout:1284: done (6 left) 2024-09-16T03:38:26.208 INFO:tasks.rados.rados.0.smithi157.stdout:1285: done (5 left) 2024-09-16T03:38:26.208 INFO:tasks.rados.rados.0.smithi157.stdout:1286: done (4 left) 2024-09-16T03:38:26.208 INFO:tasks.rados.rados.0.smithi157.stdout:1287: done (3 left) 2024-09-16T03:38:26.208 INFO:tasks.rados.rados.0.smithi157.stdout:1288: read oid 42 snap -1 2024-09-16T03:38:26.208 INFO:tasks.rados.rados.0.smithi157.stdout:1288: expect (ObjNum 200 snap 0 seq_num 200) 2024-09-16T03:38:26.208 INFO:tasks.rados.rados.0.smithi157.stdout:1289: read oid 14 snap -1 2024-09-16T03:38:26.208 INFO:tasks.rados.rados.0.smithi157.stdout:1289: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-16T03:38:26.208 INFO:tasks.rados.rados.0.smithi157.stdout:1290: read oid 44 snap -1 2024-09-16T03:38:26.209 INFO:tasks.rados.rados.0.smithi157.stdout:1290: expect (ObjNum 219 snap 0 seq_num 219) 2024-09-16T03:38:26.209 INFO:tasks.rados.rados.0.smithi157.stdout:1283: expect (ObjNum 240 snap 0 seq_num 240) 2024-09-16T03:38:26.250 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:26.250 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:26.250 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:38:26.250 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:38:26.250 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:26.258 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:38:26.269 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1017741 2024-09-16T03:38:26.285 INFO:tasks.rados.rados.0.smithi157.stdout:1281: finishing write tid 1 to smithi15736696-17 2024-09-16T03:38:26.285 INFO:tasks.rados.rados.0.smithi157.stdout:1281: finishing write tid 2 to smithi15736696-17 2024-09-16T03:38:26.285 INFO:tasks.rados.rados.0.smithi157.stdout:1277: finishing set_chunk to oid 30 2024-09-16T03:38:26.285 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 39 (ObjNum 194 snap 0 seq_num 194) dirty exists 2024-09-16T03:38:26.285 INFO:tasks.rados.rados.0.smithi157.stdout:1281: finishing write tid 3 to smithi15736696-17 2024-09-16T03:38:26.285 INFO:tasks.rados.rados.0.smithi157.stdout:1281: finishing write tid 4 to smithi15736696-17 2024-09-16T03:38:26.285 INFO:tasks.rados.rados.0.smithi157.stdout:1281: oid 17 updating version 0 to 109 2024-09-16T03:38:26.285 INFO:tasks.rados.rados.0.smithi157.stdout:1281: oid 17 updating version 109 to 110 2024-09-16T03:38:26.285 INFO:tasks.rados.rados.0.smithi157.stdout:1281: oid 17 updating version 110 to 111 2024-09-16T03:38:26.285 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 111 (ObjNum 248 snap 0 seq_num 248) dirty exists 2024-09-16T03:38:26.285 INFO:tasks.rados.rados.0.smithi157.stdout:1281: left oid 17 (ObjNum 248 snap 0 seq_num 248) 2024-09-16T03:38:26.286 INFO:tasks.rados.rados.0.smithi157.stdout:1291: read oid 26 snap -1 2024-09-16T03:38:26.286 INFO:tasks.rados.rados.0.smithi157.stdout:1291: expect deleted 2024-09-16T03:38:26.286 INFO:tasks.rados.rados.0.smithi157.stdout:1277: done (6 left) 2024-09-16T03:38:26.286 INFO:tasks.rados.rados.0.smithi157.stdout:1281: done (5 left) 2024-09-16T03:38:26.286 INFO:tasks.rados.rados.0.smithi157.stdout:1283: done (4 left) 2024-09-16T03:38:26.286 INFO:tasks.rados.rados.0.smithi157.stdout:1292: delete oid 36 current snap is 0 2024-09-16T03:38:26.287 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:26.285+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:26.310 INFO:tasks.rados.rados.0.smithi157.stdout:1288: expect (ObjNum 200 snap 0 seq_num 200) 2024-09-16T03:38:26.310 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:26.351 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 4 pg_num to 36 2024-09-16T03:38:26.388 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:26.386+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:26.411 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:38:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:38:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:432 4.c 4:34492f38:::smithi15736696-33:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:38:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.494983594999994, 2024-09-16T03:38:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.404460861, 2024-09-16T03:38:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.412 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 432 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:38:26.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913833+0000", 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0610000000000001e-06 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913839+0000", 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9479999999999996e-06 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913848+0000", 2024-09-16T03:38:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5629999999999992e-06 2024-09-16T03:38:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913857+0000", 2024-09-16T03:38:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0399999999999998e-06 2024-09-16T03:38:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914022+0000", 2024-09-16T03:38:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000165422 2024-09-16T03:38:26.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914101+0000", 2024-09-16T03:38:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8462999999999997e-05 2024-09-16T03:38:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217171+0000", 2024-09-16T03:38:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30307040000000002 2024-09-16T03:38:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217854+0000", 2024-09-16T03:38:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068278700000000002 2024-09-16T03:38:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.218098+0000", 2024-09-16T03:38:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243825 2024-09-16T03:38:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318164+0000", 2024-09-16T03:38:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100066451 2024-09-16T03:38:26.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318232+0000", 2024-09-16T03:38:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8113000000000003e-05 2024-09-16T03:38:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318293+0000", 2024-09-16T03:38:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0788000000000001e-05 2024-09-16T03:38:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:790 4.d 4:b3fe05b2:::smithi15736696-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:38:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.520843521000003, 2024-09-16T03:38:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41456747799999999, 2024-09-16T03:38:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.419 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 790 2024-09-16T03:38:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:38:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:38:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887975+0000", 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0369999999999999e-06 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887987+0000", 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1606e-05 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888016+0000", 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.917e-05 2024-09-16T03:38:26.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888032+0000", 2024-09-16T03:38:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5616000000000001e-05 2024-09-16T03:38:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888178+0000", 2024-09-16T03:38:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612100000000001 2024-09-16T03:38:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888307+0000", 2024-09-16T03:38:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012953899999999999 2024-09-16T03:38:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.302540+0000", 2024-09-16T03:38:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41423238899999998 2024-09-16T03:38:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.423 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1418 4.c 4:3313e97c:::smithi15736696-21:head [write 615156~555834] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.182732348999998, 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40891108799999998, 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1418 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226087+0000", 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8890000000000004e-06 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226827+0000", 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000739298 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226848+0000", 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1418000000000001e-05 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226858+0000", 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0434e-05 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327036+0000", 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10017769 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327098+0000", 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1760999999999994e-05 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327230+0000", 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131839 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434369+0000", 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107139788 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434444+0000", 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4171999999999994e-05 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634830+0000", 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20038660599999999 2024-09-16T03:38:26.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634937+0000", 2024-09-16T03:38:26.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010716499999999999 2024-09-16T03:38:26.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634994+0000", 2024-09-16T03:38:26.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7027999999999997e-05 2024-09-16T03:38:26.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.429 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.430 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1419 4.c 4:3313e97c:::smithi15736696-21:head [write 1603467~416361] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:26.430 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:38:26.430 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.178066960000002, 2024-09-16T03:38:26.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40430432100000002, 2024-09-16T03:38:26.430 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.430 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.430 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1419 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230752+0000", 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.354e-06 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231164+0000", 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041208599999999998 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231179+0000", 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4885999999999999e-05 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231186+0000", 2024-09-16T03:38:26.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8759999999999999e-06 2024-09-16T03:38:26.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328726+0000", 2024-09-16T03:38:26.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097539855999999994 2024-09-16T03:38:26.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328786+0000", 2024-09-16T03:38:26.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0059000000000002e-05 2024-09-16T03:38:26.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:26.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328890+0000", 2024-09-16T03:38:26.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104338 2024-09-16T03:38:26.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434475+0000", 2024-09-16T03:38:26.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.105584709 2024-09-16T03:38:26.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.534594+0000", 2024-09-16T03:38:26.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10011914099999999 2024-09-16T03:38:26.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635003+0000", 2024-09-16T03:38:26.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100408834 2024-09-16T03:38:26.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635029+0000", 2024-09-16T03:38:26.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6551999999999999e-05 2024-09-16T03:38:26.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635053+0000", 2024-09-16T03:38:26.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3629999999999999e-05 2024-09-16T03:38:26.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.436 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.437 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.437 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1420 4.c 4:3313e97c:::smithi15736696-21:head [write 2816389~693495] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:26.437 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:38:26.437 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.170152649000002, 2024-09-16T03:38:26.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39653502600000001, 2024-09-16T03:38:26.437 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.437 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1420 2024-09-16T03:38:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:38:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:38:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238666+0000", 2024-09-16T03:38:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.163e-06 2024-09-16T03:38:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239395+0000", 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072868599999999998 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239410+0000", 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5330999999999999e-05 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239419+0000", 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7360000000000007e-06 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332338+0000", 2024-09-16T03:38:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092919165999999997 2024-09-16T03:38:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332435+0000", 2024-09-16T03:38:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6972999999999995e-05 2024-09-16T03:38:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:26.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332544+0000", 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010898700000000001 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634808+0000", 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30226407700000002 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635058+0000", 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025009300000000001 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635103+0000", 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4539000000000002e-05 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635145+0000", 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1938999999999999e-05 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635198+0000", 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3335999999999999e-05 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.443 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1421 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.169384200000003, 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.497492088, 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.444 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1421 2024-09-16T03:38:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:38:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:38:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239432+0000", 2024-09-16T03:38:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5999999999999991e-07 2024-09-16T03:38:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239436+0000", 2024-09-16T03:38:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:38:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239443+0000", 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.968e-06 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239447+0000", 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6969999999999999e-06 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635225+0000", 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39577807399999998 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635290+0000", 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4583000000000002e-05 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635410+0000", 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120684 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636180+0000", 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076997899999999995 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636799+0000", 2024-09-16T03:38:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061906600000000004 2024-09-16T03:38:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736742+0000", 2024-09-16T03:38:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099943041999999996 2024-09-16T03:38:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736843+0000", 2024-09-16T03:38:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010076799999999999 2024-09-16T03:38:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736924+0000", 2024-09-16T03:38:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0419000000000004e-05 2024-09-16T03:38:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1422 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.169362919999998, 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49764818700000002, 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1422 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239455+0000", 2024-09-16T03:38:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9860000000000001e-06 2024-09-16T03:38:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239461+0000", 2024-09-16T03:38:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:38:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239464+0000", 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2619999999999999e-06 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635736+0000", 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.396272287 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635790+0000", 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4252000000000002e-05 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736964+0000", 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101173319 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737024+0000", 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9975000000000001e-05 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737101+0000", 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7303000000000002e-05 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1684 4.d 4:b3fe05b2:::smithi15736696-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.254467804000001, 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39825815199999998, 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1684 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.456 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:26.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9999999999999998e-07 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154351+0000", 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.249e-06 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154356+0000", 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7690000000000002e-06 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154361+0000", 2024-09-16T03:38:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4209999999999997e-06 2024-09-16T03:38:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.253488+0000", 2024-09-16T03:38:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099126895000000007 2024-09-16T03:38:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.253585+0000", 2024-09-16T03:38:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7204e-05 2024-09-16T03:38:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552451+0000", 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.298866358 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552521+0000", 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9873e-05 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552606+0000", 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4882999999999994e-05 2024-09-16T03:38:26.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1749 4.c 4:34492f38:::smithi15736696-33:head [assert-version v40,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:56.368656+0000", 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.040159524, 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30391657799999999, 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1749 2024-09-16T03:38:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368656+0000", 2024-09-16T03:38:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368656+0000", 2024-09-16T03:38:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368660+0000", 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.693e-06 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368673+0000", 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2944e-05 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368702+0000", 2024-09-16T03:38:26.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9365000000000001e-05 2024-09-16T03:38:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368719+0000", 2024-09-16T03:38:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6756999999999999e-05 2024-09-16T03:38:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368857+0000", 2024-09-16T03:38:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000138166 2024-09-16T03:38:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368950+0000", 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2843000000000005e-05 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.471081+0000", 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102130647 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.471807+0000", 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072618800000000001 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.547167+0000", 2024-09-16T03:38:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075359801000000004 2024-09-16T03:38:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.572606+0000", 2024-09-16T03:38:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025439215000000001 2024-09-16T03:38:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.672443+0000", 2024-09-16T03:38:26.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099837710999999996 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.672493+0000", 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9645999999999999e-05 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.672573+0000", 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9602000000000001e-05 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2334 4.c 4:34492f38:::smithi15736696-33:head [write 3060800~250179] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.094561+0000", 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.314254779000001, 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30670181299999999, 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2334 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.094561+0000", 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.094561+0000", 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.094562+0000", 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.026e-06 2024-09-16T03:38:26.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095003+0000", 2024-09-16T03:38:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000441366 2024-09-16T03:38:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095011+0000", 2024-09-16T03:38:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1380000000000007e-06 2024-09-16T03:38:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095016+0000", 2024-09-16T03:38:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2459999999999997e-06 2024-09-16T03:38:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.188911+0000", 2024-09-16T03:38:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093895270000000003 2024-09-16T03:38:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.188978+0000", 2024-09-16T03:38:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7021999999999996e-05 2024-09-16T03:38:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.189075+0000", 2024-09-16T03:38:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7535999999999996e-05 2024-09-16T03:38:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.190081+0000", 2024-09-16T03:38:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001005875 2024-09-16T03:38:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.190314+0000", 2024-09-16T03:38:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000233131 2024-09-16T03:38:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.202489+0000", 2024-09-16T03:38:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012174574000000001 2024-09-16T03:38:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.401033+0000", 2024-09-16T03:38:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19854360099999999 2024-09-16T03:38:26.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.401107+0000", 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3924000000000006e-05 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.401263+0000", 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015610400000000001 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2335 4.c 4:34492f38:::smithi15736696-33:head [setxattr _header (56) in=7b,truncate 3310979] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.095024+0000", 2024-09-16T03:38:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.313791601, 2024-09-16T03:38:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.306438074, 2024-09-16T03:38:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2335 2024-09-16T03:38:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.475 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095024+0000", 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095024+0000", 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095025+0000", 2024-09-16T03:38:26.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8100000000000003e-07 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095027+0000", 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5579999999999999e-06 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095031+0000", 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.585e-06 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095034+0000", 2024-09-16T03:38:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8710000000000001e-06 2024-09-16T03:38:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.189600+0000", 2024-09-16T03:38:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094566501999999997 2024-09-16T03:38:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.189658+0000", 2024-09-16T03:38:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7969e-05 2024-09-16T03:38:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.189782+0000", 2024-09-16T03:38:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012398800000000001 2024-09-16T03:38:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.190356+0000", 2024-09-16T03:38:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057428700000000004 2024-09-16T03:38:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.202888+0000", 2024-09-16T03:38:26.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012531588 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.390802+0000", 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18791423199999999 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.401345+0000", 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010542924 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.401383+0000", 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7781000000000003e-05 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.401462+0000", 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9208000000000005e-05 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2336 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.313775802, 2024-09-16T03:38:26.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.407180869, 2024-09-16T03:38:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2336 2024-09-16T03:38:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:38:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:38:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:38:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2399999999999999e-07 2024-09-16T03:38:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095042+0000", 2024-09-16T03:38:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.567e-06 2024-09-16T03:38:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095046+0000", 2024-09-16T03:38:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.639e-06 2024-09-16T03:38:26.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195114+0000", 2024-09-16T03:38:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10006748 2024-09-16T03:38:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195173+0000", 2024-09-16T03:38:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8727000000000002e-05 2024-09-16T03:38:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195278+0000", 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010522900000000001 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.501699+0000", 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30642159299999999 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.501995+0000", 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000295822 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.502221+0000", 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022548800000000001 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2450 4.c 4:34492f38:::smithi15736696-33:head [write 2113302~674388] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:15.459245+0000", 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.949570691, 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30558659100000002, 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.487 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2450 2024-09-16T03:38:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.459245+0000", 2024-09-16T03:38:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.459245+0000", 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.459247+0000", 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9250000000000002e-06 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.460572+0000", 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013251649999999999 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.460581+0000", 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3060000000000003e-06 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.460586+0000", 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9350000000000002e-06 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463062+0000", 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024754730000000002 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463119+0000", 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7635e-05 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463219+0000", 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9729999999999996e-05 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.560780+0000", 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097560981000000005 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.566417+0000", 2024-09-16T03:38:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056371629999999997 2024-09-16T03:38:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.566537+0000", 2024-09-16T03:38:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011941499999999999 2024-09-16T03:38:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.764722+0000", 2024-09-16T03:38:26.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19818529800000001 2024-09-16T03:38:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.764761+0000", 2024-09-16T03:38:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8831000000000001e-05 2024-09-16T03:38:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.764832+0000", 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0734000000000004e-05 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2452 4.c 4:34492f38:::smithi15736696-33:head [setxattr _header (56) in=7b,truncate 3479049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:15.463009+0000", 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.945806489000001, 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40224235800000002, 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.494 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2452 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463009+0000", 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463009+0000", 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463010+0000", 2024-09-16T03:38:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6000000000000004e-07 2024-09-16T03:38:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463012+0000", 2024-09-16T03:38:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5100000000000001e-06 2024-09-16T03:38:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463017+0000", 2024-09-16T03:38:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5959999999999997e-06 2024-09-16T03:38:26.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463020+0000", 2024-09-16T03:38:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2339999999999999e-06 2024-09-16T03:38:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.465322+0000", 2024-09-16T03:38:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002301464 2024-09-16T03:38:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.465374+0000", 2024-09-16T03:38:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2945999999999998e-05 2024-09-16T03:38:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.465470+0000", 2024-09-16T03:38:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5445000000000005e-05 2024-09-16T03:38:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.560795+0000", 2024-09-16T03:38:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095324675999999997 2024-09-16T03:38:26.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.566503+0000", 2024-09-16T03:38:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057087659999999997 2024-09-16T03:38:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.566596+0000", 2024-09-16T03:38:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3107000000000002e-05 2024-09-16T03:38:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865113+0000", 2024-09-16T03:38:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29851622100000003 2024-09-16T03:38:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865160+0000", 2024-09-16T03:38:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6863000000000002e-05 2024-09-16T03:38:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865252+0000", 2024-09-16T03:38:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1970000000000002e-05 2024-09-16T03:38:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.500 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2453 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:38:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:15.463024+0000", 2024-09-16T03:38:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.945791505000001, 2024-09-16T03:38:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40260837500000002, 2024-09-16T03:38:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.501 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2453 2024-09-16T03:38:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463024+0000", 2024-09-16T03:38:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463024+0000", 2024-09-16T03:38:26.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463025+0000", 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4499999999999998e-07 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463026+0000", 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4500000000000001e-06 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463030+0000", 2024-09-16T03:38:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5449999999999997e-06 2024-09-16T03:38:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463034+0000", 2024-09-16T03:38:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0309999999999999e-06 2024-09-16T03:38:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.560713+0000", 2024-09-16T03:38:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097679777999999995 2024-09-16T03:38:26.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.560765+0000", 2024-09-16T03:38:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2021000000000002e-05 2024-09-16T03:38:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865314+0000", 2024-09-16T03:38:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30454909000000002 2024-09-16T03:38:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865423+0000", 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108342 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865633+0000", 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020977300000000001 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2642 4.c 4:34492f38:::smithi15736696-33:head [write 665610~797860,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:38:26.506 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.5326290570000003, 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40488026700000002, 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2642 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876189+0000", 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0549999999999998e-06 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877003+0000", 2024-09-16T03:38:26.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081467900000000001 2024-09-16T03:38:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877018+0000", 2024-09-16T03:38:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.471e-05 2024-09-16T03:38:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877025+0000", 2024-09-16T03:38:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.379e-06 2024-09-16T03:38:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977237+0000", 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100211914 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977354+0000", 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116416 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977480+0000", 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012629899999999999 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.984474+0000", 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069943230000000002 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084635+0000", 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100161018 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185809+0000", 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10117406700000001 2024-09-16T03:38:26.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.280666+0000", 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094856491000000001 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.280857+0000", 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019133999999999999 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.281067+0000", 2024-09-16T03:38:26.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000209576 2024-09-16T03:38:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2643 4.c 4:34492f38:::smithi15736696-33:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:38:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.5279621710000004, 2024-09-16T03:38:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50004631700000002, 2024-09-16T03:38:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.513 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2643 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880854+0000", 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6599999999999994e-07 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881441+0000", 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058613000000000003 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881450+0000", 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6260000000000007e-06 2024-09-16T03:38:26.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881455+0000", 2024-09-16T03:38:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.515e-06 2024-09-16T03:38:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979409+0000", 2024-09-16T03:38:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097954645000000007 2024-09-16T03:38:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979512+0000", 2024-09-16T03:38:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102701 2024-09-16T03:38:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979684+0000", 2024-09-16T03:38:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017176299999999999 2024-09-16T03:38:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084699+0000", 2024-09-16T03:38:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10501545599999999 2024-09-16T03:38:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084714+0000", 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4738e-05 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185858+0000", 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101144074 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380720+0000", 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.194862072 2024-09-16T03:38:26.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380783+0000", 2024-09-16T03:38:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3262000000000005e-05 2024-09-16T03:38:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380900+0000", 2024-09-16T03:38:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116369 2024-09-16T03:38:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2644 4.c 4:34492f38:::smithi15736696-33:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.5232856760000004, 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.495569859, 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2644 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.520 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885531+0000", 2024-09-16T03:38:26.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885813+0000", 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028188399999999999 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885824+0000", 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885830+0000", 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6289999999999998e-06 2024-09-16T03:38:26.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983163+0000", 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097333127000000005 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983273+0000", 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109868 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983458+0000", 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018554899999999999 2024-09-16T03:38:26.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084722+0000", 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101263244 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185717+0000", 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100995575 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185904+0000", 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186682 2024-09-16T03:38:26.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380965+0000", 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.195060921 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381030+0000", 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5174999999999999e-05 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381100+0000", 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9877999999999997e-05 2024-09-16T03:38:26.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2645 4.c 4:34492f38:::smithi15736696-33:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:26.526 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:38:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.5229761750000002, 2024-09-16T03:38:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59563304800000005, 2024-09-16T03:38:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.528 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2645 2024-09-16T03:38:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:38:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:38:26.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885840+0000", 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885843+0000", 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.785e-06 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885848+0000", 2024-09-16T03:38:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3979999999999998e-06 2024-09-16T03:38:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885852+0000", 2024-09-16T03:38:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6509999999999999e-06 2024-09-16T03:38:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084768+0000", 2024-09-16T03:38:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19891652000000001 2024-09-16T03:38:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084831+0000", 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2717000000000003e-05 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084964+0000", 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132731 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185772+0000", 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10080797399999999 2024-09-16T03:38:26.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185790+0000", 2024-09-16T03:38:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7966999999999999e-05 2024-09-16T03:38:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185959+0000", 2024-09-16T03:38:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168777 2024-09-16T03:38:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:26.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481235+0000", 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29527644800000002 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481307+0000", 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1539999999999996e-05 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481473+0000", 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016595600000000001 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2646 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.5229594859999995, 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59596245699999995, 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2646 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.535 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885857+0000", 2024-09-16T03:38:26.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:38:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885860+0000", 2024-09-16T03:38:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8940000000000003e-06 2024-09-16T03:38:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885864+0000", 2024-09-16T03:38:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.602e-06 2024-09-16T03:38:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885867+0000", 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0089999999999999e-06 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085196+0000", 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19932923399999999 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085287+0000", 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0480000000000001e-05 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481521+0000", 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39623439799999999 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481613+0000", 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2276999999999998e-05 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481819+0000", 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020521499999999999 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.539 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.540 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.540 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:26.540 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:26.540 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:26.541 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:26.541 INFO:tasks.rados.rados.0.smithi157.stdout:1289: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-16T03:38:26.541 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:26.542 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:38:26.543 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:26.540+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:26.543 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1739977 2024-09-16T03:38:26.582 INFO:tasks.rados.rados.0.smithi157.stdout:1290: expect (ObjNum 219 snap 0 seq_num 219) 2024-09-16T03:38:26.642 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:26.640+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:26.658 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:26.658 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:26.658 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:26.658 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:26.667 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:38:26.743 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:26.742+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:26.770 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:26.770 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:26.779 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:38:26.787 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 828260 2024-09-16T03:38:26.844 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:26.842+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:26.883 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:21 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'4, mlcod=0'0)", 2024-09-16T03:38:26.883 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:38:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 74.693519142, 2024-09-16T03:38:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10231100799999999, 2024-09-16T03:38:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:38:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:38:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187300+0000", 2024-09-16T03:38:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0360000000000001e-06 2024-09-16T03:38:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187905+0000", 2024-09-16T03:38:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060508100000000004 2024-09-16T03:38:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187920+0000", 2024-09-16T03:38:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4314000000000001e-05 2024-09-16T03:38:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187928+0000", 2024-09-16T03:38:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7150000000000004e-06 2024-09-16T03:38:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187984+0000", 2024-09-16T03:38:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5872999999999999e-05 2024-09-16T03:38:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188016+0000", 2024-09-16T03:38:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2045000000000002e-05 2024-09-16T03:38:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289567+0000", 2024-09-16T03:38:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101550875 2024-09-16T03:38:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289609+0000", 2024-09-16T03:38:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2069e-05 2024-09-16T03:38:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:81 4.c e39/32 4:3313e97c:::smithi15736696-21:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.398719+0000", 2024-09-16T03:38:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 74.482098121000007, 2024-09-16T03:38:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102114725, 2024-09-16T03:38:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.889 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.398719+0000", 2024-09-16T03:38:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.398719+0000", 2024-09-16T03:38:26.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.398722+0000", 2024-09-16T03:38:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7199999999999998e-06 2024-09-16T03:38:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.399189+0000", 2024-09-16T03:38:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046724000000000001 2024-09-16T03:38:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.399200+0000", 2024-09-16T03:38:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1068e-05 2024-09-16T03:38:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499313+0000", 2024-09-16T03:38:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100112249 2024-09-16T03:38:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499411+0000", 2024-09-16T03:38:26.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8930000000000003e-05 2024-09-16T03:38:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.499444+0000", 2024-09-16T03:38:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2966000000000003e-05 2024-09-16T03:38:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.500783+0000", 2024-09-16T03:38:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013380880000000001 2024-09-16T03:38:26.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.500834+0000", 2024-09-16T03:38:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1464000000000001e-05 2024-09-16T03:38:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.21:5 3.0 e45/44 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'39, mlcod=45'37)", 2024-09-16T03:38:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:38:26.894 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 65.966222107999997, 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20119541799999999, 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:38:26.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914597+0000", 2024-09-16T03:38:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.494e-06 2024-09-16T03:38:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914603+0000", 2024-09-16T03:38:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7349999999999999e-06 2024-09-16T03:38:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914614+0000", 2024-09-16T03:38:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0995999999999999e-05 2024-09-16T03:38:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914623+0000", 2024-09-16T03:38:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3899999999999993e-06 2024-09-16T03:38:26.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.114941+0000", 2024-09-16T03:38:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20031812399999999 2024-09-16T03:38:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115001+0000", 2024-09-16T03:38:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0161999999999998e-05 2024-09-16T03:38:26.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115750+0000", 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074851699999999998 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115791+0000", 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1e-05 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.5.21:36 3.0 e52/44 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 52'62, mlcod=51'60)", 2024-09-16T03:38:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:38:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 54.996191449000001, 2024-09-16T03:38:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20104840199999999, 2024-09-16T03:38:26.900 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:38:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:38:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884627+0000", 2024-09-16T03:38:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.322e-06 2024-09-16T03:38:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884634+0000", 2024-09-16T03:38:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:38:26.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884643+0000", 2024-09-16T03:38:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7680000000000003e-06 2024-09-16T03:38:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984740+0000", 2024-09-16T03:38:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100096066 2024-09-16T03:38:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084931+0000", 2024-09-16T03:38:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100191408 2024-09-16T03:38:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084952+0000", 2024-09-16T03:38:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.128e-05 2024-09-16T03:38:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085645+0000", 2024-09-16T03:38:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00069309500000000002 2024-09-16T03:38:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085674+0000", 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9025000000000001e-05 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1537 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'35, mlcod=58'33)", 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 41.160438155000001, 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102969106, 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.905 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:38:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:38:26.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720381+0000", 2024-09-16T03:38:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.767e-06 2024-09-16T03:38:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721134+0000", 2024-09-16T03:38:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00075281999999999996 2024-09-16T03:38:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721148+0000", 2024-09-16T03:38:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4534e-05 2024-09-16T03:38:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721160+0000", 2024-09-16T03:38:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1595e-05 2024-09-16T03:38:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721256+0000", 2024-09-16T03:38:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6136000000000002e-05 2024-09-16T03:38:26.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721274+0000", 2024-09-16T03:38:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8006e-05 2024-09-16T03:38:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823312+0000", 2024-09-16T03:38:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102038064 2024-09-16T03:38:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823348+0000", 2024-09-16T03:38:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6183999999999998e-05 2024-09-16T03:38:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1538 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'36, mlcod=58'33)", 2024-09-16T03:38:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 41.159283817999999, 2024-09-16T03:38:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102397932, 2024-09-16T03:38:26.910 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5199999999999996e-07 2024-09-16T03:38:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722162+0000", 2024-09-16T03:38:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062758599999999996 2024-09-16T03:38:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722171+0000", 2024-09-16T03:38:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7139999999999999e-06 2024-09-16T03:38:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722177+0000", 2024-09-16T03:38:26.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.866e-06 2024-09-16T03:38:26.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822283+0000", 2024-09-16T03:38:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10010508899999999 2024-09-16T03:38:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822298+0000", 2024-09-16T03:38:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5497000000000001e-05 2024-09-16T03:38:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823907+0000", 2024-09-16T03:38:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001609166 2024-09-16T03:38:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823931+0000", 2024-09-16T03:38:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4261999999999999e-05 2024-09-16T03:38:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1539 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'37, mlcod=58'33)", 2024-09-16T03:38:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:38:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 41.158083757, 2024-09-16T03:38:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103545683, 2024-09-16T03:38:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:38:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722735+0000", 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8400000000000002e-07 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722740+0000", 2024-09-16T03:38:26.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3129999999999998e-06 2024-09-16T03:38:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722747+0000", 2024-09-16T03:38:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0600000000000002e-06 2024-09-16T03:38:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822841+0000", 2024-09-16T03:38:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100094477 2024-09-16T03:38:26.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823376+0000", 2024-09-16T03:38:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00053497199999999999 2024-09-16T03:38:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823399+0000", 2024-09-16T03:38:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2680999999999998e-05 2024-09-16T03:38:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.826259+0000", 2024-09-16T03:38:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028598500000000002 2024-09-16T03:38:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.826279+0000", 2024-09-16T03:38:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0346000000000001e-05 2024-09-16T03:38:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.922 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:38:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:38:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 36.730892851999997, 2024-09-16T03:38:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20169899299999999, 2024-09-16T03:38:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:38:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:38:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149927+0000", 2024-09-16T03:38:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9299999999999999e-06 2024-09-16T03:38:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151090+0000", 2024-09-16T03:38:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011624459999999999 2024-09-16T03:38:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151105+0000", 2024-09-16T03:38:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5238999999999999e-05 2024-09-16T03:38:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251193+0000", 2024-09-16T03:38:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100088025 2024-09-16T03:38:26.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251556+0000", 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036245599999999999 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251578+0000", 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2651e-05 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351582+0000", 2024-09-16T03:38:26.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100003811 2024-09-16T03:38:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351624+0000", 2024-09-16T03:38:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1434999999999999e-05 2024-09-16T03:38:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:90 3.0 e70/69 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head v 70'103, mlcod=0'0)", 2024-09-16T03:38:26.927 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.105054569, 2024-09-16T03:38:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102185333, 2024-09-16T03:38:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:26.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-16T03:38:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775769+0000", 2024-09-16T03:38:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7379999999999999e-06 2024-09-16T03:38:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775782+0000", 2024-09-16T03:38:26.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2989e-05 2024-09-16T03:38:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775791+0000", 2024-09-16T03:38:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5369999999999997e-06 2024-09-16T03:38:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876295+0000", 2024-09-16T03:38:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100504817 2024-09-16T03:38:26.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876316+0000", 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0409999999999999e-05 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877916+0000", 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016003790000000001 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877948+0000", 2024-09-16T03:38:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1949999999999997e-05 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1742 3.0 e70/69 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head v 70'104, mlcod=0'0)", 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.105017873000001, 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20271830699999999, 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:26.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:38:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777455+0000", 2024-09-16T03:38:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001655524 2024-09-16T03:38:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777468+0000", 2024-09-16T03:38:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2486999999999999e-05 2024-09-16T03:38:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777474+0000", 2024-09-16T03:38:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:38:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876577+0000", 2024-09-16T03:38:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099103360000000001 2024-09-16T03:38:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876594+0000", 2024-09-16T03:38:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6449000000000001e-05 2024-09-16T03:38:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978469+0000", 2024-09-16T03:38:26.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101875521 2024-09-16T03:38:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978518+0000", 2024-09-16T03:38:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8488000000000001e-05 2024-09-16T03:38:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:91 3.0 e70/69 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head v 70'105, mlcod=0'0)", 2024-09-16T03:38:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:38:26.938 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.018806004000002, 2024-09-16T03:38:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.116532025, 2024-09-16T03:38:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:38:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:38:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862015+0000", 2024-09-16T03:38:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3569999999999998e-06 2024-09-16T03:38:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862029+0000", 2024-09-16T03:38:26.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4082e-05 2024-09-16T03:38:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862055+0000", 2024-09-16T03:38:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6570000000000001e-05 2024-09-16T03:38:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862070+0000", 2024-09-16T03:38:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4473e-05 2024-09-16T03:38:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877631+0000", 2024-09-16T03:38:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.015560744 2024-09-16T03:38:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877674+0000", 2024-09-16T03:38:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3615999999999997e-05 2024-09-16T03:38:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978525+0000", 2024-09-16T03:38:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100850696 2024-09-16T03:38:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978543+0000", 2024-09-16T03:38:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8487000000000001e-05 2024-09-16T03:38:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:94 3.0 e70/69 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head v 70'106, mlcod=0'0)", 2024-09-16T03:38:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:38:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 31.018269785000001, 2024-09-16T03:38:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11627603, 2024-09-16T03:38:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:38:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:38:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862549+0000", 2024-09-16T03:38:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1650000000000001e-06 2024-09-16T03:38:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862556+0000", 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8709999999999999e-06 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862566+0000", 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0271e-05 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862573+0000", 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.861e-06 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978123+0000", 2024-09-16T03:38:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11554993199999999 2024-09-16T03:38:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978163+0000", 2024-09-16T03:38:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0201999999999999e-05 2024-09-16T03:38:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978796+0000", 2024-09-16T03:38:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063316700000000002 2024-09-16T03:38:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978824+0000", 2024-09-16T03:38:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7560999999999999e-05 2024-09-16T03:38:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1947 4.8 e74/68 4:146f84f0:::smithi15736696-6:head v 74'47, mlcod=73'45)", 2024-09-16T03:38:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:38:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 25.755653646999999, 2024-09-16T03:38:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103866244, 2024-09-16T03:38:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:38:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:38:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125165+0000", 2024-09-16T03:38:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.513e-06 2024-09-16T03:38:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126855+0000", 2024-09-16T03:38:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016901010000000001 2024-09-16T03:38:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126870+0000", 2024-09-16T03:38:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4487999999999999e-05 2024-09-16T03:38:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227009+0000", 2024-09-16T03:38:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100138888 2024-09-16T03:38:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227117+0000", 2024-09-16T03:38:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107984 2024-09-16T03:38:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227136+0000", 2024-09-16T03:38:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9247999999999999e-05 2024-09-16T03:38:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229002+0000", 2024-09-16T03:38:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001866143 2024-09-16T03:38:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229030+0000", 2024-09-16T03:38:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7878999999999999e-05 2024-09-16T03:38:26.953 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1979 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'32, mlcod=0'0)", 2024-09-16T03:38:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:38:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 25.0430791, 2024-09-16T03:38:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20429768500000001, 2024-09-16T03:38:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.954 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:38:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:38:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837740+0000", 2024-09-16T03:38:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.779e-06 2024-09-16T03:38:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838356+0000", 2024-09-16T03:38:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061634500000000002 2024-09-16T03:38:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838371+0000", 2024-09-16T03:38:26.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5007e-05 2024-09-16T03:38:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838380+0000", 2024-09-16T03:38:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1009999999999992e-06 2024-09-16T03:38:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038674+0000", 2024-09-16T03:38:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20029450500000001 2024-09-16T03:38:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038710+0000", 2024-09-16T03:38:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6100999999999999e-05 2024-09-16T03:38:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042001+0000", 2024-09-16T03:38:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032909430000000002 2024-09-16T03:38:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042036+0000", 2024-09-16T03:38:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4904000000000003e-05 2024-09-16T03:38:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1980 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'33, mlcod=0'0)", 2024-09-16T03:38:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:38:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 25.041202504000001, 2024-09-16T03:38:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20296795100000001, 2024-09-16T03:38:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.959 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:38:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:38:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839616+0000", 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3099999999999996e-07 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840706+0000", 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010900980000000001 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840720+0000", 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.378e-05 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840727+0000", 2024-09-16T03:38:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.525e-06 2024-09-16T03:38:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039940+0000", 2024-09-16T03:38:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19921313199999999 2024-09-16T03:38:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039967+0000", 2024-09-16T03:38:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6393999999999998e-05 2024-09-16T03:38:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042552+0000", 2024-09-16T03:38:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025850999999999999 2024-09-16T03:38:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042583+0000", 2024-09-16T03:38:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0991000000000002e-05 2024-09-16T03:38:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1981 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'34, mlcod=0'0)", 2024-09-16T03:38:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:38:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 25.039180149, 2024-09-16T03:38:26.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201270175, 2024-09-16T03:38:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:38:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841638+0000", 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841644+0000", 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8050000000000003e-06 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841653+0000", 2024-09-16T03:38:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6580000000000008e-06 2024-09-16T03:38:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841660+0000", 2024-09-16T03:38:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9290000000000001e-06 2024-09-16T03:38:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041673+0000", 2024-09-16T03:38:26.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20001311499999999 2024-09-16T03:38:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041696+0000", 2024-09-16T03:38:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3365000000000001e-05 2024-09-16T03:38:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042878+0000", 2024-09-16T03:38:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011814499999999999 2024-09-16T03:38:26.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042907+0000", 2024-09-16T03:38:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9655999999999999e-05 2024-09-16T03:38:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2117 3.0 e76/69 3:3c8eb54b:::593314a3a723c40dec2b25351adb282089b02c0264cb0891ee3f17edde05dfe9:head v 76'138, mlcod=75'136)", 2024-09-16T03:38:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:38:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 21.228772460999998, 2024-09-16T03:38:26.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10374235, 2024-09-16T03:38:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:38:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:38:26.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8099999999999999e-07 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654214+0000", 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002169191 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654230+0000", 2024-09-16T03:38:26.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6144e-05 2024-09-16T03:38:26.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654242+0000", 2024-09-16T03:38:26.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1748e-05 2024-09-16T03:38:26.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.754387+0000", 2024-09-16T03:38:26.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10014505899999999 2024-09-16T03:38:26.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.754409+0000", 2024-09-16T03:38:26.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2181999999999999e-05 2024-09-16T03:38:26.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.755736+0000", 2024-09-16T03:38:26.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013261690000000001 2024-09-16T03:38:26.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.755787+0000", 2024-09-16T03:38:26.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1675999999999998e-05 2024-09-16T03:38:26.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.974 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.974 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2278 4.18 e80/68 4:1f63731d:::smithi15736696-30:head v 80'37, mlcod=75'34)", 2024-09-16T03:38:26.974 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:38:26.975 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.355362262, 2024-09-16T03:38:26.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10326089300000001, 2024-09-16T03:38:26.975 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.975 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.975 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.975 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:38:26.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:38:26.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525456+0000", 2024-09-16T03:38:26.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.248e-06 2024-09-16T03:38:26.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525936+0000", 2024-09-16T03:38:26.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047982899999999998 2024-09-16T03:38:26.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525945+0000", 2024-09-16T03:38:26.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8489999999999995e-06 2024-09-16T03:38:26.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525955+0000", 2024-09-16T03:38:26.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.043e-05 2024-09-16T03:38:26.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.526339+0000", 2024-09-16T03:38:26.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038385200000000001 2024-09-16T03:38:26.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.526358+0000", 2024-09-16T03:38:26.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8802000000000001e-05 2024-09-16T03:38:26.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.628679+0000", 2024-09-16T03:38:26.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102321304 2024-09-16T03:38:26.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.628716+0000", 2024-09-16T03:38:26.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6579000000000003e-05 2024-09-16T03:38:26.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.979 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.980 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2536 3.0 e86/69 3:10a17b1b:::13dc2fab1f287170fabf96291b912bf80e0b081f291f2c6d41d494566f2d1f8c:head v 86'176, mlcod=86'174)", 2024-09-16T03:38:26.980 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:38:26.980 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.041305770999999, 2024-09-16T03:38:26.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20664157499999999, 2024-09-16T03:38:26.980 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.980 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.980 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.980 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839515+0000", 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3330000000000001e-06 2024-09-16T03:38:26.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843392+0000", 2024-09-16T03:38:26.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038769329999999999 2024-09-16T03:38:26.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843412+0000", 2024-09-16T03:38:26.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0608e-05 2024-09-16T03:38:26.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.943546+0000", 2024-09-16T03:38:26.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100133176 2024-09-16T03:38:26.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043824+0000", 2024-09-16T03:38:26.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100278785 2024-09-16T03:38:26.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043869+0000", 2024-09-16T03:38:26.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4845999999999997e-05 2024-09-16T03:38:26.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046096+0000", 2024-09-16T03:38:26.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022262559999999998 2024-09-16T03:38:26.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046153+0000", 2024-09-16T03:38:26.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7637999999999999e-05 2024-09-16T03:38:26.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.985 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.985 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.985 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2654 4.8 e90/68 4:148e627d:::smithi15736696-23:head v 90'58, mlcod=80'56)", 2024-09-16T03:38:26.985 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:38:26.985 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.5059184129999998, 2024-09-16T03:38:26.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.106604212, 2024-09-16T03:38:26.985 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:26.985 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:26.985 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:26.985 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:26.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374901+0000", 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1749999999999999e-06 2024-09-16T03:38:26.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:26.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375810+0000", 2024-09-16T03:38:26.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00090911400000000004 2024-09-16T03:38:26.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:26.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375827+0000", 2024-09-16T03:38:26.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7010000000000001e-05 2024-09-16T03:38:26.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:26.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476010+0000", 2024-09-16T03:38:26.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100182879 2024-09-16T03:38:26.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:26.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476184+0000", 2024-09-16T03:38:26.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017408899999999999 2024-09-16T03:38:26.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:26.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476206+0000", 2024-09-16T03:38:26.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2013000000000001e-05 2024-09-16T03:38:26.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:26.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481458+0000", 2024-09-16T03:38:26.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052519120000000001 2024-09-16T03:38:26.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:26.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:26.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:26.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481503+0000", 2024-09-16T03:38:26.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5019999999999999e-05 2024-09-16T03:38:26.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.989 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.990 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:26.990 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:26.990 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:26.992 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:38:26.993 INFO:tasks.rados.rados.0.smithi157.stdout:1288: done (4 left) 2024-09-16T03:38:26.994 INFO:tasks.rados.rados.0.smithi157.stdout:1289: done (3 left) 2024-09-16T03:38:26.994 INFO:tasks.rados.rados.0.smithi157.stdout:1290: done (2 left) 2024-09-16T03:38:26.994 INFO:tasks.rados.rados.0.smithi157.stdout:1291: done (1 left) 2024-09-16T03:38:26.994 INFO:tasks.rados.rados.0.smithi157.stdout:1292: done (0 left) 2024-09-16T03:38:26.994 INFO:tasks.rados.rados.0.smithi157.stdout:1293: read oid 7 snap -1 2024-09-16T03:38:26.994 INFO:tasks.rados.rados.0.smithi157.stdout:1293: expect deleted 2024-09-16T03:38:26.994 INFO:tasks.rados.rados.0.smithi157.stdout:1294: write (excl) oid 39 current snap is 0 2024-09-16T03:38:26.995 INFO:tasks.rados.rados.0.smithi157.stdout:1294: seq_num 251 ranges {618557=727148,2104745=113812} 2024-09-16T03:38:26.995 INFO:tasks.rados.rados.0.smithi157.stdout:1294: writing smithi15736696-39 from 618557 to 1345705 tid 1 2024-09-16T03:38:26.995 INFO:tasks.rados.rados.0.smithi157.stdout:1294: writing smithi15736696-39 from 2104745 to 2218557 tid 2 2024-09-16T03:38:26.995 INFO:tasks.rados.rados.0.smithi157.stdout:1295: read oid 42 snap -1 2024-09-16T03:38:26.995 INFO:tasks.rados.rados.0.smithi157.stdout:1295: expect (ObjNum 200 snap 0 seq_num 200) 2024-09-16T03:38:26.995 INFO:tasks.rados.rados.0.smithi157.stdout:1296: write oid 28 current snap is 0 2024-09-16T03:38:26.995 INFO:tasks.rados.rados.0.smithi157.stdout:1296: seq_num 252 ranges {635364=402397,1763227=472137} 2024-09-16T03:38:26.995 INFO:tasks.rados.rados.0.smithi157.stdout:1296: writing smithi15736696-28 from 635364 to 1037761 tid 1 2024-09-16T03:38:26.995 INFO:tasks.rados.rados.0.smithi157.stdout:1296: writing smithi15736696-28 from 1763227 to 2235364 tid 2 2024-09-16T03:38:26.995 INFO:tasks.rados.rados.0.smithi157.stdout:1297: write (excl) oid 30 current snap is 0 2024-09-16T03:38:26.995 INFO:tasks.rados.rados.0.smithi157.stdout:1297: seq_num 253 ranges {652171=477646,1821709=430462} 2024-09-16T03:38:26.995 INFO:tasks.rados.rados.0.smithi157.stdout:1297: writing smithi15736696-30 from 652171 to 1129817 tid 1 2024-09-16T03:38:26.995 INFO:tasks.rados.rados.0.smithi157.stdout:1297: writing smithi15736696-30 from 1821709 to 2252171 tid 2 2024-09-16T03:38:26.996 INFO:tasks.rados.rados.0.smithi157.stdout:1295: expect (ObjNum 200 snap 0 seq_num 200) 2024-09-16T03:38:26.996 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:26.990+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:27.025 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2337802 2024-09-16T03:38:27.048 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 110026 2024-09-16T03:38:27.091 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:27.090+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:27.097 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:27.095+0000 7fedb5739640 -1 osd.9 95 handle_osd_map injecting map crc failure 2024-09-16T03:38:27.101 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:27.101 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:27.101 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:27.101 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:27.110 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:38:27.117 INFO:tasks.rados.rados.0.smithi157.stdout:1294: finishing write tid 1 to smithi15736696-39 2024-09-16T03:38:27.117 INFO:tasks.rados.rados.0.smithi157.stdout:1294: finishing write tid 2 to smithi15736696-39 2024-09-16T03:38:27.117 INFO:tasks.rados.rados.0.smithi157.stdout:1297: finishing write tid 1 to smithi15736696-30 2024-09-16T03:38:27.117 INFO:tasks.rados.rados.0.smithi157.stdout:1294: finishing write tid 3 to smithi15736696-39 2024-09-16T03:38:27.117 INFO:tasks.rados.rados.0.smithi157.stdout:1298: read oid 18 snap -1 2024-09-16T03:38:27.117 INFO:tasks.rados.rados.0.smithi157.stdout:1298: expect deleted 2024-09-16T03:38:27.117 INFO:tasks.rados.rados.0.smithi157.stdout:1293: done (5 left) 2024-09-16T03:38:27.118 INFO:tasks.rados.rados.0.smithi157.stdout:1295: done (4 left) 2024-09-16T03:38:27.118 INFO:tasks.rados.rados.0.smithi157.stdout:1299: tier_flush oid 21 2024-09-16T03:38:27.118 INFO:tasks.rados.rados.0.smithi157.stdout:1299: tier_flush oid 21 snap -1 2024-09-16T03:38:27.118 INFO:tasks.rados.rados.0.smithi157.stdout:1300: read oid 37 snap -1 2024-09-16T03:38:27.118 INFO:tasks.rados.rados.0.smithi157.stdout:1300: expect (ObjNum 217 snap 0 seq_num 217) 2024-09-16T03:38:27.118 INFO:tasks.rados.rados.0.smithi157.stdout:1301: tier_evict oid 31 2024-09-16T03:38:27.118 INFO:tasks.rados.rados.0.smithi157.stdout:1301: tier_evict oid 31 snap -1 2024-09-16T03:38:27.118 INFO:tasks.rados.rados.0.smithi157.stdout:1302: write (excl) oid 49 current snap is 0 2024-09-16T03:38:27.118 INFO:tasks.rados.rados.0.smithi157.stdout:1302: seq_num 254 ranges {668978=552895,1763838=505140} 2024-09-16T03:38:27.125 INFO:tasks.rados.rados.0.smithi157.stdout:1302: writing smithi15736696-49 from 668978 to 1221873 tid 1 2024-09-16T03:38:27.131 INFO:tasks.rados.rados.0.smithi157.stdout:1302: writing smithi15736696-49 from 1763838 to 2268978 tid 2 2024-09-16T03:38:27.131 INFO:tasks.rados.rados.0.smithi157.stdout:1303: read oid 13 snap -1 2024-09-16T03:38:27.131 INFO:tasks.rados.rados.0.smithi157.stdout:1303: expect (ObjNum 209 snap 0 seq_num 209) 2024-09-16T03:38:27.131 INFO:tasks.rados.rados.0.smithi157.stdout:1304: read oid 43 snap -1 2024-09-16T03:38:27.131 INFO:tasks.rados.rados.0.smithi157.stdout:1304: expect (ObjNum 210 snap 0 seq_num 210) 2024-09-16T03:38:27.131 INFO:tasks.rados.rados.0.smithi157.stdout:1305: read oid 47 snap -1 2024-09-16T03:38:27.131 INFO:tasks.rados.rados.0.smithi157.stdout:1305: expect (ObjNum 214 snap 0 seq_num 214) 2024-09-16T03:38:27.131 INFO:tasks.rados.rados.0.smithi157.stdout:1301: got (22) Invalid argument 2024-09-16T03:38:27.131 INFO:tasks.rados.rados.0.smithi157.stdout:1306: read oid 41 snap -1 2024-09-16T03:38:27.131 INFO:tasks.rados.rados.0.smithi157.stdout:1306: expect (ObjNum 112 snap 0 seq_num 112) 2024-09-16T03:38:27.132 INFO:tasks.rados.rados.0.smithi157.stdout:1301: done (11 left) 2024-09-16T03:38:27.132 INFO:tasks.rados.rados.0.smithi157.stdout:1307: read oid 42 snap -1 2024-09-16T03:38:27.132 INFO:tasks.rados.rados.0.smithi157.stdout:1307: expect (ObjNum 200 snap 0 seq_num 200) 2024-09-16T03:38:27.132 INFO:tasks.rados.rados.0.smithi157.stdout:1308: read oid 46 snap -1 2024-09-16T03:38:27.132 INFO:tasks.rados.rados.0.smithi157.stdout:1308: expect (ObjNum 236 snap 0 seq_num 236) 2024-09-16T03:38:27.132 INFO:tasks.rados.rados.0.smithi157.stdout:1309: tier_promote oid 20 2024-09-16T03:38:27.132 INFO:tasks.rados.rados.0.smithi157.stdout:1299: got (2) No such file or directory 2024-09-16T03:38:27.132 INFO:tasks.rados.rados.0.smithi157.stdout:1299: got expected ENOENT (src dne) 2024-09-16T03:38:27.132 INFO:tasks.rados.rados.0.smithi157.stdout:1297: finishing write tid 2 to smithi15736696-30 2024-09-16T03:38:27.132 INFO:tasks.rados.rados.0.smithi157.stdout:1299: done (13 left) 2024-09-16T03:38:27.132 INFO:tasks.rados.rados.0.smithi157.stdout:1310: read oid 24 snap -1 2024-09-16T03:38:27.132 INFO:tasks.rados.rados.0.smithi157.stdout:1310: expect (ObjNum 238 snap 0 seq_num 238) 2024-09-16T03:38:27.132 INFO:tasks.rados.rados.0.smithi157.stdout:1311: read oid 5 snap -1 2024-09-16T03:38:27.133 INFO:tasks.rados.rados.0.smithi157.stdout:1311: expect deleted 2024-09-16T03:38:27.133 INFO:tasks.rados.rados.0.smithi157.stdout:1312: read oid 33 snap -1 2024-09-16T03:38:27.133 INFO:tasks.rados.rados.0.smithi157.stdout:1312: expect (ObjNum 230 snap 0 seq_num 230) 2024-09-16T03:38:27.133 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:27.192 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:27.190+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:27.200 INFO:tasks.rados.rados.0.smithi157.stdout:1294: finishing write tid 4 to smithi15736696-39 2024-09-16T03:38:27.200 INFO:tasks.rados.rados.0.smithi157.stdout:1294: oid 39 updating version 0 to 69 2024-09-16T03:38:27.200 INFO:tasks.rados.rados.0.smithi157.stdout:1294: oid 39 version 69 is already newer than 68 2024-09-16T03:38:27.200 INFO:tasks.rados.rados.0.smithi157.stdout:1294: oid 39 version 69 is already newer than 67 2024-09-16T03:38:27.200 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 69 (ObjNum 251 snap 0 seq_num 251) dirty exists 2024-09-16T03:38:27.200 INFO:tasks.rados.rados.0.smithi157.stdout:1294: left oid 39 (ObjNum 251 snap 0 seq_num 251) 2024-09-16T03:38:27.201 INFO:tasks.rados.rados.0.smithi157.stdout:1294: done (15 left) 2024-09-16T03:38:27.201 INFO:tasks.rados.rados.0.smithi157.stdout:1313: read oid 19 snap -1 2024-09-16T03:38:27.201 INFO:tasks.rados.rados.0.smithi157.stdout:1313: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:27.201 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:27.201 INFO:tasks.rados.rados.0.smithi157.stdout:1310: expect (ObjNum 238 snap 0 seq_num 238) 2024-09-16T03:38:27.213 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:27.213 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:27.213 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:27.214 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:27.214 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:27.223 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:38:27.224 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:27.270 INFO:tasks.rados.rados.0.smithi157.stdout:1297: finishing write tid 3 to smithi15736696-30 2024-09-16T03:38:27.270 INFO:tasks.rados.rados.0.smithi157.stdout:1297: finishing write tid 4 to smithi15736696-30 2024-09-16T03:38:27.270 INFO:tasks.rados.rados.0.smithi157.stdout:1297: oid 30 updating version 0 to 41 2024-09-16T03:38:27.270 INFO:tasks.rados.rados.0.smithi157.stdout:1297: oid 30 updating version 41 to 42 2024-09-16T03:38:27.270 INFO:tasks.rados.rados.0.smithi157.stdout:1297: oid 30 updating version 42 to 43 2024-09-16T03:38:27.271 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 43 (ObjNum 253 snap 0 seq_num 253) dirty exists 2024-09-16T03:38:27.271 INFO:tasks.rados.rados.0.smithi157.stdout:1297: left oid 30 (ObjNum 253 snap 0 seq_num 253) 2024-09-16T03:38:27.271 INFO:tasks.rados.rados.0.smithi157.stdout:1300: expect (ObjNum 217 snap 0 seq_num 217) 2024-09-16T03:38:27.271 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:27.294 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:27.291+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:27.327 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:27.327 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:27.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:27.327 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:27.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.327 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:25 4.6 4:6ece7830:::smithi15736696-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:27.328 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:38:27.328 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 75.204512172999998, 2024-09-16T03:38:27.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40559452699999998, 2024-09-16T03:38:27.328 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.328 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.328 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.328 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.328 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 25 2024-09-16T03:38:27.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.328 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:38:27.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:38:27.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119830+0000", 2024-09-16T03:38:27.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9200000000000001e-07 2024-09-16T03:38:27.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120013+0000", 2024-09-16T03:38:27.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018367200000000001 2024-09-16T03:38:27.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120021+0000", 2024-09-16T03:38:27.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9179999999999994e-06 2024-09-16T03:38:27.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120027+0000", 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6710000000000004e-06 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220194+0000", 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10016689500000001 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220399+0000", 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020502899999999999 2024-09-16T03:38:27.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220613+0000", 2024-09-16T03:38:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021383900000000001 2024-09-16T03:38:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224640+0000", 2024-09-16T03:38:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040268099999999996 2024-09-16T03:38:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423348+0000", 2024-09-16T03:38:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19870853399999999 2024-09-16T03:38:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525343+0000", 2024-09-16T03:38:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101995081 2024-09-16T03:38:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525381+0000", 2024-09-16T03:38:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7842000000000001e-05 2024-09-16T03:38:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525424+0000", 2024-09-16T03:38:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2644e-05 2024-09-16T03:38:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:26 4.6 4:6ece7830:::smithi15736696-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:38:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 75.204299946000006, 2024-09-16T03:38:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550111300000002, 2024-09-16T03:38:27.334 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 26 2024-09-16T03:38:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:38:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:38:27.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120042+0000", 2024-09-16T03:38:27.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7999999999999995e-07 2024-09-16T03:38:27.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120210+0000", 2024-09-16T03:38:27.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016779499999999999 2024-09-16T03:38:27.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120219+0000", 2024-09-16T03:38:27.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6859999999999995e-06 2024-09-16T03:38:27.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120223+0000", 2024-09-16T03:38:27.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0330000000000002e-06 2024-09-16T03:38:27.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221494+0000", 2024-09-16T03:38:27.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101271812 2024-09-16T03:38:27.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221572+0000", 2024-09-16T03:38:27.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7733999999999998e-05 2024-09-16T03:38:27.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:27.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221697+0000", 2024-09-16T03:38:27.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012474900000000001 2024-09-16T03:38:27.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224670+0000", 2024-09-16T03:38:27.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029729589999999998 2024-09-16T03:38:27.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423398+0000", 2024-09-16T03:38:27.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19872784900000001 2024-09-16T03:38:27.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525465+0000", 2024-09-16T03:38:27.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102067557 2024-09-16T03:38:27.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525494+0000", 2024-09-16T03:38:27.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9113999999999999e-05 2024-09-16T03:38:27.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525542+0000", 2024-09-16T03:38:27.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8145000000000001e-05 2024-09-16T03:38:27.340 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.341 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.341 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:27 4.6 4:6ece7830:::smithi15736696-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:27.341 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:38:27.341 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 75.204112361, 2024-09-16T03:38:27.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40541507399999999, 2024-09-16T03:38:27.342 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.342 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.342 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.342 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.342 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.342 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.342 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27 2024-09-16T03:38:27.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.342 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:38:27.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:38:27.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120230+0000", 2024-09-16T03:38:27.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5799999999999999e-07 2024-09-16T03:38:27.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120233+0000", 2024-09-16T03:38:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.055e-06 2024-09-16T03:38:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120237+0000", 2024-09-16T03:38:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7099999999999998e-06 2024-09-16T03:38:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120244+0000", 2024-09-16T03:38:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4810000000000003e-06 2024-09-16T03:38:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222660+0000", 2024-09-16T03:38:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102416089 2024-09-16T03:38:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222705+0000", 2024-09-16T03:38:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5368999999999999e-05 2024-09-16T03:38:27.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222782+0000", 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6623000000000001e-05 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224696+0000", 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019143929999999999 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423402+0000", 2024-09-16T03:38:27.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198705305 2024-09-16T03:38:27.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525582+0000", 2024-09-16T03:38:27.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102180178 2024-09-16T03:38:27.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525615+0000", 2024-09-16T03:38:27.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3303000000000003e-05 2024-09-16T03:38:27.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525644+0000", 2024-09-16T03:38:27.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8909999999999999e-05 2024-09-16T03:38:27.348 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.348 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.348 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.348 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:28 4.6 4:6ece7830:::smithi15736696-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:27.348 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:27.348 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 75.204091613000003, 2024-09-16T03:38:27.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550787799999999, 2024-09-16T03:38:27.349 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.349 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.349 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:27.349 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.349 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.349 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.349 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 28 2024-09-16T03:38:27.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.349 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:27.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:27.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:27.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7999999999999996e-07 2024-09-16T03:38:27.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120252+0000", 2024-09-16T03:38:27.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.249e-06 2024-09-16T03:38:27.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120257+0000", 2024-09-16T03:38:27.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1969999999999998e-06 2024-09-16T03:38:27.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120261+0000", 2024-09-16T03:38:27.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1130000000000003e-06 2024-09-16T03:38:27.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222976+0000", 2024-09-16T03:38:27.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102715397 2024-09-16T03:38:27.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:27.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223049+0000", 2024-09-16T03:38:27.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2846e-05 2024-09-16T03:38:27.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525666+0000", 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30261668800000002 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525704+0000", 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8025999999999998e-05 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525758+0000", 2024-09-16T03:38:27.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3881999999999997e-05 2024-09-16T03:38:27.354 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.354 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.354 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.354 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:49 4.6 4:6df3bbbe:::smithi15736696-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:27.354 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:27.354 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 75.204076549000007, 2024-09-16T03:38:27.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40556222800000002, 2024-09-16T03:38:27.354 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.354 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.354 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.355 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.355 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.355 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.355 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49 2024-09-16T03:38:27.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.355 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:27.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:27.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:27.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8700000000000002e-07 2024-09-16T03:38:27.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120454+0000", 2024-09-16T03:38:27.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018909299999999999 2024-09-16T03:38:27.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120465+0000", 2024-09-16T03:38:27.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.083e-05 2024-09-16T03:38:27.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120468+0000", 2024-09-16T03:38:27.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3869999999999999e-06 2024-09-16T03:38:27.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223082+0000", 2024-09-16T03:38:27.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102613379 2024-09-16T03:38:27.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223189+0000", 2024-09-16T03:38:27.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107728 2024-09-16T03:38:27.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:27.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223368+0000", 2024-09-16T03:38:27.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179011 2024-09-16T03:38:27.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.226087+0000", 2024-09-16T03:38:27.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027183390000000002 2024-09-16T03:38:27.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423408+0000", 2024-09-16T03:38:27.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19732106499999999 2024-09-16T03:38:27.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525779+0000", 2024-09-16T03:38:27.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10237133 2024-09-16T03:38:27.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525805+0000", 2024-09-16T03:38:27.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5967000000000001e-05 2024-09-16T03:38:27.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525827+0000", 2024-09-16T03:38:27.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1812000000000001e-05 2024-09-16T03:38:27.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.361 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.361 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:50 4.6 4:6df3bbbe:::smithi15736696-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:27.362 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:38:27.362 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 75.203868240000006, 2024-09-16T03:38:27.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50526462999999999, 2024-09-16T03:38:27.362 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.362 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.362 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.362 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.362 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.362 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.362 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 50 2024-09-16T03:38:27.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.362 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:38:27.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:38:27.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120474+0000", 2024-09-16T03:38:27.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4600000000000005e-07 2024-09-16T03:38:27.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120901+0000", 2024-09-16T03:38:27.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00042690000000000002 2024-09-16T03:38:27.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120907+0000", 2024-09-16T03:38:27.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7710000000000001e-06 2024-09-16T03:38:27.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120911+0000", 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0500000000000002e-06 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224717+0000", 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103806175 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224763+0000", 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5629e-05 2024-09-16T03:38:27.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:27.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224838+0000", 2024-09-16T03:38:27.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4548999999999994e-05 2024-09-16T03:38:27.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423390+0000", 2024-09-16T03:38:27.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19855242400000001 2024-09-16T03:38:27.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423411+0000", 2024-09-16T03:38:27.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0888000000000001e-05 2024-09-16T03:38:27.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625680+0000", 2024-09-16T03:38:27.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20226923099999999 2024-09-16T03:38:27.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625709+0000", 2024-09-16T03:38:27.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9232000000000001e-05 2024-09-16T03:38:27.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625738+0000", 2024-09-16T03:38:27.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8235e-05 2024-09-16T03:38:27.368 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.368 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.368 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.369 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:51 4.6 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:27.369 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:27.369 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 75.203423658000006, 2024-09-16T03:38:27.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50490003699999997, 2024-09-16T03:38:27.369 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.369 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.369 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.369 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.369 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.369 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.369 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 51 2024-09-16T03:38:27.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:27.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8699999999999995e-07 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120920+0000", 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0820000000000001e-06 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120924+0000", 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120927+0000", 2024-09-16T03:38:27.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5809999999999999e-06 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225783+0000", 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10485588699999999 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225832+0000", 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9750000000000003e-05 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:27.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225910+0000", 2024-09-16T03:38:27.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7084999999999998e-05 2024-09-16T03:38:27.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423414+0000", 2024-09-16T03:38:27.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19750438200000001 2024-09-16T03:38:27.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423456+0000", 2024-09-16T03:38:27.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2386999999999999e-05 2024-09-16T03:38:27.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625766+0000", 2024-09-16T03:38:27.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20231011300000001 2024-09-16T03:38:27.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625790+0000", 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3672999999999998e-05 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625818+0000", 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7576000000000002e-05 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:52 4.6 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 75.203411462000005, 2024-09-16T03:38:27.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50502012900000004, 2024-09-16T03:38:27.376 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.376 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.376 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:27.376 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.376 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.376 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.376 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 52 2024-09-16T03:38:27.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.376 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:27.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:27.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:27.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:38:27.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120932+0000", 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.181e-06 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120935+0000", 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7690000000000001e-06 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120938+0000", 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6639999999999998e-06 2024-09-16T03:38:27.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423188+0000", 2024-09-16T03:38:27.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30224989200000002 2024-09-16T03:38:27.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:27.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423318+0000", 2024-09-16T03:38:27.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013045999999999999 2024-09-16T03:38:27.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625844+0000", 2024-09-16T03:38:27.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20252547600000001 2024-09-16T03:38:27.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625895+0000", 2024-09-16T03:38:27.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1242000000000002e-05 2024-09-16T03:38:27.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625950+0000", 2024-09-16T03:38:27.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5164999999999998e-05 2024-09-16T03:38:27.381 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.381 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.381 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.381 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:279 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:27.381 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:38:27.381 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 72.127459674999997, 2024-09-16T03:38:27.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50446570800000001, 2024-09-16T03:38:27.381 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.381 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.382 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.382 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.382 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.382 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.382 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 279 2024-09-16T03:38:27.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.382 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:38:27.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:38:27.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196883+0000", 2024-09-16T03:38:27.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.48e-06 2024-09-16T03:38:27.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196889+0000", 2024-09-16T03:38:27.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6239999999999999e-06 2024-09-16T03:38:27.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196902+0000", 2024-09-16T03:38:27.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2804e-05 2024-09-16T03:38:27.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196909+0000", 2024-09-16T03:38:27.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9379999999999992e-06 2024-09-16T03:38:27.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196963+0000", 2024-09-16T03:38:27.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3437999999999999e-05 2024-09-16T03:38:27.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op must wait for map", 2024-09-16T03:38:27.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196969+0000", 2024-09-16T03:38:27.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1709999999999999e-06 2024-09-16T03:38:27.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302538+0000", 2024-09-16T03:38:27.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10556889899999999 2024-09-16T03:38:27.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302659+0000", 2024-09-16T03:38:27.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121441 2024-09-16T03:38:27.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:38:27.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500040+0000", 2024-09-16T03:38:27.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19738082900000001 2024-09-16T03:38:27.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500589+0000", 2024-09-16T03:38:27.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00054827000000000001 2024-09-16T03:38:27.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701118+0000", 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20052917100000001 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701205+0000", 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7120999999999995e-05 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701289+0000", 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4457999999999997e-05 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701347+0000", 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8063999999999998e-05 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:306 4.6 4:6ece7830:::smithi15736696-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 70.811457360000006, 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40927214699999998, 2024-09-16T03:38:27.390 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.391 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.391 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.391 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.391 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.391 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.391 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 306 2024-09-16T03:38:27.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.391 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:38:27.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:38:27.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512885+0000", 2024-09-16T03:38:27.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.277e-06 2024-09-16T03:38:27.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512890+0000", 2024-09-16T03:38:27.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6689999999999997e-06 2024-09-16T03:38:27.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512900+0000", 2024-09-16T03:38:27.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0393e-05 2024-09-16T03:38:27.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512906+0000", 2024-09-16T03:38:27.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4569999999999996e-06 2024-09-16T03:38:27.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512951+0000", 2024-09-16T03:38:27.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4891000000000001e-05 2024-09-16T03:38:27.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.513000+0000", 2024-09-16T03:38:27.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9187999999999997e-05 2024-09-16T03:38:27.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:27.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.720758+0000", 2024-09-16T03:38:27.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20775775699999999 2024-09-16T03:38:27.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.721960+0000", 2024-09-16T03:38:27.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001202107 2024-09-16T03:38:27.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922010+0000", 2024-09-16T03:38:27.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20004993900000001 2024-09-16T03:38:27.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922055+0000", 2024-09-16T03:38:27.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4981000000000002e-05 2024-09-16T03:38:27.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922103+0000", 2024-09-16T03:38:27.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8097999999999999e-05 2024-09-16T03:38:27.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922156+0000", 2024-09-16T03:38:27.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3390000000000002e-05 2024-09-16T03:38:27.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.397 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.397 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:419 4.6 4:64ce87bc:::smithi15736696-20:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:27.397 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:38:27.397 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 66.410657745999998, 2024-09-16T03:38:27.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40437787200000003, 2024-09-16T03:38:27.397 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.397 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.397 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.398 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.398 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.398 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.398 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 419 2024-09-16T03:38:27.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.398 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:38:27.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913687+0000", 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3299999999999999e-06 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913700+0000", 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2830000000000001e-05 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913733+0000", 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2815999999999999e-05 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913750+0000", 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7073000000000001e-05 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913920+0000", 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000170015 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913981+0000", 2024-09-16T03:38:27.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1631999999999995e-05 2024-09-16T03:38:27.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:27.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216580+0000", 2024-09-16T03:38:27.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30259888200000001 2024-09-16T03:38:27.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217205+0000", 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062461099999999996 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317915+0000", 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10071053000000001 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317996+0000", 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0440999999999998e-05 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318032+0000", 2024-09-16T03:38:27.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5944e-05 2024-09-16T03:38:27.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318061+0000", 2024-09-16T03:38:27.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9768e-05 2024-09-16T03:38:27.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.403 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.403 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:479 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:27.403 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 65.602801600000006, 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.504011706, 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 479 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721542+0000", 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3199999999999998e-06 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721553+0000", 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1413000000000001e-05 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721570+0000", 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.685e-05 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721596+0000", 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5647999999999999e-05 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.921950+0000", 2024-09-16T03:38:27.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20035402899999999 2024-09-16T03:38:27.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.922009+0000", 2024-09-16T03:38:27.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9046999999999999e-05 2024-09-16T03:38:27.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:38:27.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.023850+0000", 2024-09-16T03:38:27.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10184130199999999 2024-09-16T03:38:27.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.024503+0000", 2024-09-16T03:38:27.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065257399999999995 2024-09-16T03:38:27.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.125105+0000", 2024-09-16T03:38:27.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100602046 2024-09-16T03:38:27.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225461+0000", 2024-09-16T03:38:27.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100355835 2024-09-16T03:38:27.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225512+0000", 2024-09-16T03:38:27.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0862999999999997e-05 2024-09-16T03:38:27.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225551+0000", 2024-09-16T03:38:27.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9779e-05 2024-09-16T03:38:27.410 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.410 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.410 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.410 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:996 4.8 4:148e627d:::smithi15736696-23:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:27.410 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:38:27.410 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 56.276998693000003, 2024-09-16T03:38:27.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.73588397999999999, 2024-09-16T03:38:27.410 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.410 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.410 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:27.410 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.411 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.411 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.411 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 996 2024-09-16T03:38:27.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.411 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:38:27.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:38:27.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047344+0000", 2024-09-16T03:38:27.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.846e-06 2024-09-16T03:38:27.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047351+0000", 2024-09-16T03:38:27.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.973e-06 2024-09-16T03:38:27.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047365+0000", 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3556e-05 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047382+0000", 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6994e-05 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047453+0000", 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1458000000000006e-05 2024-09-16T03:38:27.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047465+0000", 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1119e-05 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449411+0000", 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40194680399999999 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449535+0000", 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124043 2024-09-16T03:38:27.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783227+0000", 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.333691187 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1048 4.16 4:6df3bbbe:::smithi15736696-13:head [write 1622009~726683] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.748554575999997, 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40682847599999999, 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.415 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.416 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.416 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.416 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.416 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.416 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1048 2024-09-16T03:38:27.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.416 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:27.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576941+0000", 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011532230000000001 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576949+0000", 2024-09-16T03:38:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1089999999999998e-06 2024-09-16T03:38:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576956+0000", 2024-09-16T03:38:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000001e-06 2024-09-16T03:38:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878710+0000", 2024-09-16T03:38:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30175434299999998 2024-09-16T03:38:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878765+0000", 2024-09-16T03:38:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4988000000000003e-05 2024-09-16T03:38:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878852+0000", 2024-09-16T03:38:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7201999999999996e-05 2024-09-16T03:38:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881535+0000", 2024-09-16T03:38:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026828310000000001 2024-09-16T03:38:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883968+0000", 2024-09-16T03:38:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024330659999999998 2024-09-16T03:38:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982541+0000", 2024-09-16T03:38:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098572876000000004 2024-09-16T03:38:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982580+0000", 2024-09-16T03:38:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8714000000000002e-05 2024-09-16T03:38:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982615+0000", 2024-09-16T03:38:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5438000000000002e-05 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1049 4.16 4:6df3bbbe:::smithi15736696-13:head [write 3121996~104915] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.747376426999999, 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40579890600000001, 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1049 2024-09-16T03:38:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:38:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:38:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576966+0000", 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8199999999999999e-07 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577069+0000", 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103583 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577076+0000", 2024-09-16T03:38:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8190000000000003e-06 2024-09-16T03:38:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577082+0000", 2024-09-16T03:38:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5130000000000004e-06 2024-09-16T03:38:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880130+0000", 2024-09-16T03:38:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.303048127 2024-09-16T03:38:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880183+0000", 2024-09-16T03:38:27.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3127999999999997e-05 2024-09-16T03:38:27.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:27.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880282+0000", 2024-09-16T03:38:27.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8881000000000004e-05 2024-09-16T03:38:27.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883101+0000", 2024-09-16T03:38:27.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028194180000000002 2024-09-16T03:38:27.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.979716+0000", 2024-09-16T03:38:27.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.096614899000000004 2024-09-16T03:38:27.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982707+0000", 2024-09-16T03:38:27.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029914249999999998 2024-09-16T03:38:27.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982739+0000", 2024-09-16T03:38:27.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1250000000000001e-05 2024-09-16T03:38:27.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982764+0000", 2024-09-16T03:38:27.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5080999999999999e-05 2024-09-16T03:38:27.428 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.428 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.428 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1050 4.16 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 3226911] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.747251832000003, 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40784258099999998, 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1050 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577090+0000", 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000003e-07 2024-09-16T03:38:27.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577094+0000", 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4340000000000001e-06 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577099+0000", 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8599999999999998e-06 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577104+0000", 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0010000000000001e-06 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881242+0000", 2024-09-16T03:38:27.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30413779000000002 2024-09-16T03:38:27.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881290+0000", 2024-09-16T03:38:27.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7298e-05 2024-09-16T03:38:27.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:27.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881367+0000", 2024-09-16T03:38:27.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7750000000000006e-05 2024-09-16T03:38:27.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883117+0000", 2024-09-16T03:38:27.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001750096 2024-09-16T03:38:27.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.980218+0000", 2024-09-16T03:38:27.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097100724999999999 2024-09-16T03:38:27.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984859+0000", 2024-09-16T03:38:27.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046412950000000001 2024-09-16T03:38:27.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984900+0000", 2024-09-16T03:38:27.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0300999999999999e-05 2024-09-16T03:38:27.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984932+0000", 2024-09-16T03:38:27.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2332999999999997e-05 2024-09-16T03:38:27.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.435 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.435 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1051 4.16 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:27.435 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:27.435 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.747229552, 2024-09-16T03:38:27.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50807376999999998, 2024-09-16T03:38:27.435 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.435 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.435 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:27.436 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.436 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.436 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.436 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1051 2024-09-16T03:38:27.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.436 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:27.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:27.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:27.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:38:27.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577115+0000", 2024-09-16T03:38:27.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3980000000000002e-06 2024-09-16T03:38:27.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577120+0000", 2024-09-16T03:38:27.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5589999999999996e-06 2024-09-16T03:38:27.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577125+0000", 2024-09-16T03:38:27.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9910000000000002e-06 2024-09-16T03:38:27.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882780+0000", 2024-09-16T03:38:27.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30565438499999997 2024-09-16T03:38:27.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:27.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882825+0000", 2024-09-16T03:38:27.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5739999999999999e-05 2024-09-16T03:38:27.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085039+0000", 2024-09-16T03:38:27.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20221376499999999 2024-09-16T03:38:27.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085096+0000", 2024-09-16T03:38:27.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6666000000000003e-05 2024-09-16T03:38:27.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085186+0000", 2024-09-16T03:38:27.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9716000000000006e-05 2024-09-16T03:38:27.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.440 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1359 4.6 4:64ce87bc:::smithi15736696-20:head [write 1768928~405354] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 47.158206413999999, 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40239613499999999, 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.441 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1359 2024-09-16T03:38:27.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.442 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:38:27.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:38:27.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166136+0000", 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166923+0000", 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00078683500000000005 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166933+0000", 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0445000000000001e-05 2024-09-16T03:38:27.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166938+0000", 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0119999999999996e-06 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264901+0000", 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097962831 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264963+0000", 2024-09-16T03:38:27.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1812999999999999e-05 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.265085+0000", 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121914 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.269430+0000", 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0043455000000000004 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.372259+0000", 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102828645 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568289+0000", 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19603034999999999 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568475+0000", 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018613599999999999 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568531+0000", 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5729000000000001e-05 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1676 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v29,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 37.194377551000002, 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50424139300000004, 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.447 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.448 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:27.448 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:27.448 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:27.448 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1676 2024-09-16T03:38:27.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.448 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:38:27.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:38:27.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129968+0000", 2024-09-16T03:38:27.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8430000000000003e-06 2024-09-16T03:38:27.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129982+0000", 2024-09-16T03:38:27.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4514e-05 2024-09-16T03:38:27.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130013+0000", 2024-09-16T03:38:27.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1081999999999998e-05 2024-09-16T03:38:27.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130029+0000", 2024-09-16T03:38:27.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5849999999999999e-05 2024-09-16T03:38:27.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130169+0000", 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000139973 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130240+0000", 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0638999999999999e-05 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.532892+0000", 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402652654 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533546+0000", 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065390700000000001 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533943+0000", 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039703999999999998 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634089+0000", 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100146098 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634153+0000", 2024-09-16T03:38:27.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3581e-05 2024-09-16T03:38:27.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634205+0000", 2024-09-16T03:38:27.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2212000000000001e-05 2024-09-16T03:38:27.453 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.453 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.453 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.453 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:77 3.2 e73/68 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head v 73'117, mlcod=66'115)", 2024-09-16T03:38:27.454 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:38:27.454 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.145339273000001, 2024-09-16T03:38:27.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.401393467, 2024-09-16T03:38:27.454 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:27.454 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:27.454 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:27.454 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:27.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:27.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:38:27.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:38:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179005+0000", 2024-09-16T03:38:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.553e-06 2024-09-16T03:38:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179029+0000", 2024-09-16T03:38:27.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4039000000000001e-05 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179046+0000", 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7139999999999999e-05 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.279144+0000", 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100098245 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:27.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379323+0000", 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100178621 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379348+0000", 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5315000000000001e-05 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580315+0000", 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20096734599999999 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:27.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580395+0000", 2024-09-16T03:38:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0208000000000002e-05 2024-09-16T03:38:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:27.458 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:27.458 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:27.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:38:27.459 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:38:27.460 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2231602 2024-09-16T03:38:27.461 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:27.458+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:27.462 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:38:27.531 INFO:tasks.rados.rados.0.smithi157.stdout:1313: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:27.560 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:27.558+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:27.565 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:27.565 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:27.565 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:27.574 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:38:27.609 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:27.609 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:27.638 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:27.638 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:27.655 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:27.655 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:27.661 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:27.659+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:27.673 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:27.673 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:27.677 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:27.677 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:27.677 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:27.677 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:27.677 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:27.678 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2525647 2024-09-16T03:38:27.681 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:27.680+0000 7f8be56cd640 -1 osd.7 96 handle_osd_map injecting map crc failure 2024-09-16T03:38:27.686 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:38:27.687 INFO:tasks.rados.rados.0.smithi157.stdout:1307: expect (ObjNum 200 snap 0 seq_num 200) 2024-09-16T03:38:27.689 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:27.689 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:27.706 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:27.706 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:27.723 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:27.723 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:27.739 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:27.739 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:27.755 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:27.755 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:27.769 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:27.768+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:27.771 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:27.772 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:27.788 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:27.789 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:27.804 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:27.805 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:27.820 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:27.821 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:27.830 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:27.830 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:27.837 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:27.837 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:27.840 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:38:27.854 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:27.854 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:27.856 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3148868 2024-09-16T03:38:27.862 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:27.861+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:27.863 INFO:tasks.rados.rados.0.smithi157.stdout:1312: expect (ObjNum 230 snap 0 seq_num 230) 2024-09-16T03:38:27.871 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:27.871 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:27.960 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:27.963 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:27.961+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:27.983 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:27.983 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:27.992 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:38:28.063 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 799606 2024-09-16T03:38:28.064 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:28.062+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:28.094 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:28.095 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:28.103 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:38:28.148 INFO:tasks.rados.rados.0.smithi157.stdout:1296: finishing write tid 1 to smithi15736696-28 2024-09-16T03:38:28.148 INFO:tasks.rados.rados.0.smithi157.stdout:1296: finishing write tid 2 to smithi15736696-28 2024-09-16T03:38:28.148 INFO:tasks.rados.rados.0.smithi157.stdout:1296: finishing write tid 3 to smithi15736696-28 2024-09-16T03:38:28.148 INFO:tasks.rados.rados.0.smithi157.stdout:1296: finishing write tid 4 to smithi15736696-28 2024-09-16T03:38:28.148 INFO:tasks.rados.rados.0.smithi157.stdout:1296: oid 28 updating version 0 to 96 2024-09-16T03:38:28.149 INFO:tasks.rados.rados.0.smithi157.stdout:1296: oid 28 version 96 is already newer than 95 2024-09-16T03:38:28.149 INFO:tasks.rados.rados.0.smithi157.stdout:1296: oid 28 version 96 is already newer than 94 2024-09-16T03:38:28.149 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 96 (ObjNum 252 snap 0 seq_num 252) dirty exists 2024-09-16T03:38:28.149 INFO:tasks.rados.rados.0.smithi157.stdout:1296: left oid 28 (ObjNum 252 snap 0 seq_num 252) 2024-09-16T03:38:28.149 INFO:tasks.rados.rados.0.smithi157.stdout:1304: expect (ObjNum 210 snap 0 seq_num 210) 2024-09-16T03:38:28.190 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:28.188+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:28.231 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 544895 2024-09-16T03:38:28.247 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:28.247 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:28.257 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:38:28.264 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:28.263+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:28.355 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:28.353+0000 7f3d40708640 -1 osd.12 97 handle_osd_map injecting map crc failure 2024-09-16T03:38:28.362 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:38:28.365 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:28.364+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:28.374 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:28.374 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:28.383 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:38:28.417 INFO:tasks.rados.rados.0.smithi157.stdout:1305: expect (ObjNum 214 snap 0 seq_num 214) 2024-09-16T03:38:28.466 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:28.464+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:28.487 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:28.487 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:28.497 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:38:28.567 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:28.565+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:28.600 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:28.600 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:28.607 INFO:tasks.rados.rados.0.smithi157.stdout:1303: expect (ObjNum 209 snap 0 seq_num 209) 2024-09-16T03:38:28.609 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:38:28.663 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3171674 2024-09-16T03:38:28.668 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:28.666+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:28.712 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:28.712 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:28.721 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:38:28.750 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1340630 2024-09-16T03:38:28.768 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:28.766+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:28.810 INFO:tasks.rados.rados.0.smithi157.stdout:1308: expect (ObjNum 236 snap 0 seq_num 236) 2024-09-16T03:38:28.825 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:28.825 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:28.834 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:38:28.869 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:28.868+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:28.890 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:28.970 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:28.969+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:28.976 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:28.976 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:28.984 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:38:28.995 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1815375 2024-09-16T03:38:29.053 INFO:tasks.rados.rados.0.smithi157.stdout:1302: finishing write tid 1 to smithi15736696-49 2024-09-16T03:38:29.053 INFO:tasks.rados.rados.0.smithi157.stdout:1309: got (0) Success 2024-09-16T03:38:29.054 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 39 (ObjNum 174 snap 0 seq_num 174) dirty exists 2024-09-16T03:38:29.054 INFO:tasks.rados.rados.0.smithi157.stdout:1302: finishing write tid 2 to smithi15736696-49 2024-09-16T03:38:29.054 INFO:tasks.rados.rados.0.smithi157.stdout:1296: done (15 left) 2024-09-16T03:38:29.054 INFO:tasks.rados.rados.0.smithi157.stdout:1297: done (14 left) 2024-09-16T03:38:29.054 INFO:tasks.rados.rados.0.smithi157.stdout:1298: done (13 left) 2024-09-16T03:38:29.054 INFO:tasks.rados.rados.0.smithi157.stdout:1300: done (12 left) 2024-09-16T03:38:29.054 INFO:tasks.rados.rados.0.smithi157.stdout:1303: done (11 left) 2024-09-16T03:38:29.054 INFO:tasks.rados.rados.0.smithi157.stdout:1304: done (10 left) 2024-09-16T03:38:29.054 INFO:tasks.rados.rados.0.smithi157.stdout:1305: done (9 left) 2024-09-16T03:38:29.054 INFO:tasks.rados.rados.0.smithi157.stdout:1307: done (8 left) 2024-09-16T03:38:29.054 INFO:tasks.rados.rados.0.smithi157.stdout:1308: done (7 left) 2024-09-16T03:38:29.054 INFO:tasks.rados.rados.0.smithi157.stdout:1309: done (6 left) 2024-09-16T03:38:29.054 INFO:tasks.rados.rados.0.smithi157.stdout:1310: done (5 left) 2024-09-16T03:38:29.055 INFO:tasks.rados.rados.0.smithi157.stdout:1311: done (4 left) 2024-09-16T03:38:29.055 INFO:tasks.rados.rados.0.smithi157.stdout:1312: done (3 left) 2024-09-16T03:38:29.055 INFO:tasks.rados.rados.0.smithi157.stdout:1313: done (2 left) 2024-09-16T03:38:29.055 INFO:tasks.rados.rados.0.smithi157.stdout:1314: set_chunk oid 5 target oid 2024-09-16T03:38:29.055 INFO:tasks.rados.rados.0.smithi157.stdout:1314: set_chunk oid 5 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:29.055 INFO:tasks.rados.rados.0.smithi157.stdout:1315: tier_flush oid 10 2024-09-16T03:38:29.055 INFO:tasks.rados.rados.0.smithi157.stdout:1315: tier_flush oid 10 snap -1 2024-09-16T03:38:29.055 INFO:tasks.rados.rados.0.smithi157.stdout:1306: expect (ObjNum 112 snap 0 seq_num 112) 2024-09-16T03:38:29.055 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:29.062 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:29.061+0000 7f4212d00640 -1 osd.8 97 handle_osd_map injecting map crc failure 2024-09-16T03:38:29.062 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:29.061+0000 7f8be56cd640 -1 osd.7 97 handle_osd_map injecting map crc failure 2024-09-16T03:38:29.064 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:29.063+0000 7f12e2a1a640 -1 osd.5 97 handle_osd_map injecting map crc failure 2024-09-16T03:38:29.071 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:29.069+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:29.092 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:29.092 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:29.101 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:38:29.172 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:29.169+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:29.206 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:29.206 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:29.215 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:38:29.220 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:29.219+0000 7efeef6d5640 -1 osd.13 97 handle_osd_map injecting map crc failure 2024-09-16T03:38:29.253 INFO:tasks.rados.rados.0.smithi157.stdout:1302: finishing write tid 3 to smithi15736696-49 2024-09-16T03:38:29.253 INFO:tasks.rados.rados.0.smithi157.stdout:1302: finishing write tid 4 to smithi15736696-49 2024-09-16T03:38:29.253 INFO:tasks.rados.rados.0.smithi157.stdout:1302: oid 49 updating version 0 to 61 2024-09-16T03:38:29.253 INFO:tasks.rados.rados.0.smithi157.stdout:1302: oid 49 version 61 is already newer than 59 2024-09-16T03:38:29.253 INFO:tasks.rados.rados.0.smithi157.stdout:1302: oid 49 version 61 is already newer than 60 2024-09-16T03:38:29.253 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 61 (ObjNum 254 snap 0 seq_num 254) dirty exists 2024-09-16T03:38:29.254 INFO:tasks.rados.rados.0.smithi157.stdout:1302: left oid 49 (ObjNum 254 snap 0 seq_num 254) 2024-09-16T03:38:29.254 INFO:tasks.rados.rados.0.smithi157.stdout:1314: finishing set_chunk to oid 5 2024-09-16T03:38:29.254 INFO:tasks.rados.rados.0.smithi157.stdout:1314: got expected ENOENT (src dne) 2024-09-16T03:38:29.254 INFO:tasks.rados.rados.0.smithi157.stdout:1302: done (3 left) 2024-09-16T03:38:29.254 INFO:tasks.rados.rados.0.smithi157.stdout:1306: done (2 left) 2024-09-16T03:38:29.254 INFO:tasks.rados.rados.0.smithi157.stdout:1314: done (1 left) 2024-09-16T03:38:29.254 INFO:tasks.rados.rados.0.smithi157.stdout:1316: delete oid 32 current snap is 0 2024-09-16T03:38:29.299 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:29.297+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:29.361 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:29.361 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:29.369 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:38:29.410 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:29.408+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:29.475 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:29.472+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:29.513 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:29.514 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:29.522 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:38:29.556 INFO:tasks.rados.rados.0.smithi157.stdout:1316: done (1 left) 2024-09-16T03:38:29.556 INFO:tasks.rados.rados.0.smithi157.stdout:1317: delete oid 30 current snap is 0 2024-09-16T03:38:29.570 INFO:tasks.rados.rados.0.smithi157.stdout:1315: got (0) Success 2024-09-16T03:38:29.570 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 0 (ObjNum 227 snap 0 seq_num 227) dirty exists 2024-09-16T03:38:29.576 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:29.575+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:29.665 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:29.666 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:29.674 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:38:29.677 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:29.675+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:29.759 INFO:tasks.rados.rados.0.smithi157.stdout:1315: done (1 left) 2024-09-16T03:38:29.759 INFO:tasks.rados.rados.0.smithi157.stdout:1317: done (0 left) 2024-09-16T03:38:29.759 INFO:tasks.rados.rados.0.smithi157.stdout:1318: read oid 34 snap -1 2024-09-16T03:38:29.759 INFO:tasks.rados.rados.0.smithi157.stdout:1318: expect (ObjNum 205 snap 0 seq_num 205) 2024-09-16T03:38:29.759 INFO:tasks.rados.rados.0.smithi157.stdout:1319: write oid 7 current snap is 0 2024-09-16T03:38:29.759 INFO:tasks.rados.rados.0.smithi157.stdout:1319: seq_num 257 ranges {719399=778642,1939284=380115} 2024-09-16T03:38:29.766 INFO:tasks.rados.rados.0.smithi157.stdout:1319: writing smithi15736696-7 from 719399 to 1498041 tid 1 2024-09-16T03:38:29.771 INFO:tasks.rados.rados.0.smithi157.stdout:1319: writing smithi15736696-7 from 1939284 to 2319399 tid 2 2024-09-16T03:38:29.771 INFO:tasks.rados.rados.0.smithi157.stdout:1320: set_chunk oid 50 target oid 2024-09-16T03:38:29.778 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:29.776+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:29.778 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:29.778 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:29.787 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:38:29.878 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:29.876+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:29.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:38:29.893 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:29.893 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:29.901 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:38:29.979 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:29.977+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:30.014 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:30.014 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:30.032 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:30.033 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:30.043 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:30.043 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:30.049 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:30.049 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:30.052 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:38:30.065 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:30.066 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:30.080 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:30.078+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:30.081 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:30.082 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:30.100 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:30.100 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:30.117 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:30.117 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:30.134 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:30.134 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:30.150 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:30.151 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:30.156 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:983 3.1 e50/31 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head v 50'54, mlcod=49'52)", 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.088217557, 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.105768445, 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.066318+0000", 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7700000000000001e-07 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.070158+0000", 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0038403170000000002 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.070168+0000", 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9019999999999994e-06 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.070178+0000", 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6860000000000001e-06 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.170328+0000", 2024-09-16T03:38:30.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150196 2024-09-16T03:38:30.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.170363+0000", 2024-09-16T03:38:30.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4588999999999997e-05 2024-09-16T03:38:30.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.172053+0000", 2024-09-16T03:38:30.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016902900000000001 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.172086+0000", 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3287999999999997e-05 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:988 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'30, mlcod=46'28)", 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.381297+0000", 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.773238470000003, 2024-09-16T03:38:30.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10422211300000001, 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.381297+0000", 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.381297+0000", 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.381298+0000", 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5409999999999999e-06 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.382086+0000", 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078808600000000002 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.382097+0000", 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0557000000000001e-05 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.482188+0000", 2024-09-16T03:38:30.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100091478 2024-09-16T03:38:30.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.482261+0000", 2024-09-16T03:38:30.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2992999999999997e-05 2024-09-16T03:38:30.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.482287+0000", 2024-09-16T03:38:30.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5211999999999998e-05 2024-09-16T03:38:30.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.485483+0000", 2024-09-16T03:38:30.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031964939999999998 2024-09-16T03:38:30.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.485519+0000", 2024-09-16T03:38:30.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5751999999999999e-05 2024-09-16T03:38:30.165 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.165 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.166 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.166 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1071 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'34, mlcod=50'32)", 2024-09-16T03:38:30.166 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:32.306208+0000", 2024-09-16T03:38:30.166 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 57.848327341999997, 2024-09-16T03:38:30.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103114758, 2024-09-16T03:38:30.166 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.166 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.166 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.166 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.306208+0000", 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.306208+0000", 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.306210+0000", 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6619999999999999e-06 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307136+0000", 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092664400000000001 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307149+0000", 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2615e-05 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307159+0000", 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0341e-05 2024-09-16T03:38:30.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307221+0000", 2024-09-16T03:38:30.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1871000000000004e-05 2024-09-16T03:38:30.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307239+0000", 2024-09-16T03:38:30.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7876999999999998e-05 2024-09-16T03:38:30.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409282+0000", 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102042801 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409323+0000", 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0946999999999999e-05 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1072 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'35, mlcod=50'32)", 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 57.846591678999999, 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10192738899999999, 2024-09-16T03:38:30.170 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.307944+0000", 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9999999999999996e-07 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.308383+0000", 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000438794 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.308391+0000", 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6149999999999999e-06 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.308396+0000", 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1719999999999998e-06 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.408530+0000", 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013438700000001 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.408547+0000", 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6554000000000001e-05 2024-09-16T03:38:30.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409844+0000", 2024-09-16T03:38:30.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001297166 2024-09-16T03:38:30.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.409871+0000", 2024-09-16T03:38:30.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6900999999999999e-05 2024-09-16T03:38:30.174 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.174 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.174 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.175 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1073 4.f e52/32 4:f0d14c71:::smithi15736696-8:head v 52'36, mlcod=50'32)", 2024-09-16T03:38:30.175 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:32.311182+0000", 2024-09-16T03:38:30.175 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 57.843353321999999, 2024-09-16T03:38:30.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101901307, 2024-09-16T03:38:30.175 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.175 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.175 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.175 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311182+0000", 2024-09-16T03:38:30.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311182+0000", 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311183+0000", 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311720+0000", 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00053692400000000004 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.311728+0000", 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7849999999999992e-06 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.411763+0000", 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100035393 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.411838+0000", 2024-09-16T03:38:30.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4658000000000002e-05 2024-09-16T03:38:30.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.411853+0000", 2024-09-16T03:38:30.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5268e-05 2024-09-16T03:38:30.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.178 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.178 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.413058+0000", 2024-09-16T03:38:30.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001204751 2024-09-16T03:38:30.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:32.413083+0000", 2024-09-16T03:38:30.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5609000000000002e-05 2024-09-16T03:38:30.179 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.179 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.179 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.179 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1193 4.a e54/32 4:503187b3:::smithi15736696-34:head v 54'43, mlcod=53'41)", 2024-09-16T03:38:30.179 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:36.160789+0000", 2024-09-16T03:38:30.179 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 53.993746354000002, 2024-09-16T03:38:30.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10275106000000001, 2024-09-16T03:38:30.180 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.180 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.180 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.180 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.160789+0000", 2024-09-16T03:38:30.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.160789+0000", 2024-09-16T03:38:30.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.160791+0000", 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.638e-06 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.161609+0000", 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00081820899999999997 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.161635+0000", 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5939999999999999e-05 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.261770+0000", 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100135416 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.261913+0000", 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000142695 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.261950+0000", 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7324999999999999e-05 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.263493+0000", 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015430800000000001 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:36.263540+0000", 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6757e-05 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.183 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1515 3.1 e59/31 3:9c9e970c:::7249cb65b52cadf4fe8f4ec923c7e548d1de7d9628823acd2c8de19121f1c1ce:head v 59'69, mlcod=56'67)", 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 45.202884849, 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.203376156, 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:38:30.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:38:30.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951651+0000", 2024-09-16T03:38:30.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8299999999999996e-07 2024-09-16T03:38:30.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953610+0000", 2024-09-16T03:38:30.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019588769999999999 2024-09-16T03:38:30.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953622+0000", 2024-09-16T03:38:30.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1846e-05 2024-09-16T03:38:30.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953632+0000", 2024-09-16T03:38:30.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0202e-05 2024-09-16T03:38:30.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.053808+0000", 2024-09-16T03:38:30.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100175615 2024-09-16T03:38:30.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.053846+0000", 2024-09-16T03:38:30.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8383000000000002e-05 2024-09-16T03:38:30.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.155006+0000", 2024-09-16T03:38:30.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101160131 2024-09-16T03:38:30.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.155027+0000", 2024-09-16T03:38:30.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0319e-05 2024-09-16T03:38:30.188 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.188 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.188 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.188 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1704 3.1 e65/31 3:b6a94245:::f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99:head v 65'77, mlcod=64'75)", 2024-09-16T03:38:30.189 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:38:30.189 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 39.360749575, 2024-09-16T03:38:30.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30251757499999998, 2024-09-16T03:38:30.189 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.189 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.189 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.189 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:38:30.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:38:30.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793788+0000", 2024-09-16T03:38:30.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:38:30.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794391+0000", 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060354199999999999 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794406+0000", 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5045000000000001e-05 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794432+0000", 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5996999999999999e-05 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.191 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.994704+0000", 2024-09-16T03:38:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20027177700000001 2024-09-16T03:38:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.994719+0000", 2024-09-16T03:38:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5021000000000001e-05 2024-09-16T03:38:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096277+0000", 2024-09-16T03:38:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101557437 2024-09-16T03:38:30.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096303+0000", 2024-09-16T03:38:30.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6732000000000001e-05 2024-09-16T03:38:30.193 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.193 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.193 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.193 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:84 3.1 e65/31 3:b24f289d:::e84532c28f467d2f2a6e2cf44904fa0d4a555e1304bbe1b25db645af42152bd7:head v 65'78, mlcod=64'75)", 2024-09-16T03:38:30.193 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:38:30.193 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 39.351734411999999, 2024-09-16T03:38:30.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29353782099999998, 2024-09-16T03:38:30.194 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.194 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.194 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.194 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:38:30.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:38:30.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802805+0000", 2024-09-16T03:38:30.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1200000000000004e-06 2024-09-16T03:38:30.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803910+0000", 2024-09-16T03:38:30.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011051660000000001 2024-09-16T03:38:30.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803928+0000", 2024-09-16T03:38:30.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7880000000000002e-05 2024-09-16T03:38:30.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803938+0000", 2024-09-16T03:38:30.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6690000000000001e-06 2024-09-16T03:38:30.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095473+0000", 2024-09-16T03:38:30.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29153552399999999 2024-09-16T03:38:30.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.197 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095486+0000", 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2455e-05 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096306+0000", 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00082031900000000002 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096339+0000", 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2688000000000003e-05 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.198 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.199 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:85 3.1 e65/31 3:b380f269:::e65351e8a65a0b1a517e3b1a85e04b84c0ab7cfa6655f11fb0da31f56b4dd9af:head v 65'79, mlcod=64'75)", 2024-09-16T03:38:30.199 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:38:30.199 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 39.350453834, 2024-09-16T03:38:30.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29260815400000001, 2024-09-16T03:38:30.199 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.199 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.199 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.199 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804083+0000", 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804266+0000", 2024-09-16T03:38:30.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018268099999999999 2024-09-16T03:38:30.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804275+0000", 2024-09-16T03:38:30.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4439999999999997e-06 2024-09-16T03:38:30.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904371+0000", 2024-09-16T03:38:30.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100095276 2024-09-16T03:38:30.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095883+0000", 2024-09-16T03:38:30.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.191512444 2024-09-16T03:38:30.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095901+0000", 2024-09-16T03:38:30.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8462999999999999e-05 2024-09-16T03:38:30.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096658+0000", 2024-09-16T03:38:30.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00075622000000000005 2024-09-16T03:38:30.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096690+0000", 2024-09-16T03:38:30.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1919999999999999e-05 2024-09-16T03:38:30.203 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.203 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.203 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.204 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:86 3.1 e65/31 3:b3acb53e:::e951eb69ce5151a3f54107fa22c70f80729eff845da1613447a91536a24046e8:head v 65'80, mlcod=64'75)", 2024-09-16T03:38:30.204 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:38:30.204 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 39.250150417999997, 2024-09-16T03:38:30.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29244589199999999, 2024-09-16T03:38:30.204 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.204 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.204 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.204 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:38:30.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:38:30.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904386+0000", 2024-09-16T03:38:30.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3e-07 2024-09-16T03:38:30.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904537+0000", 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015122700000000001 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904544+0000", 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.024e-06 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904547+0000", 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6540000000000001e-06 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196016+0000", 2024-09-16T03:38:30.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29146812599999999 2024-09-16T03:38:30.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196027+0000", 2024-09-16T03:38:30.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1172000000000001e-05 2024-09-16T03:38:30.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196791+0000", 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00076418700000000005 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196831+0000", 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9771999999999998e-05 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:91 3.0 e70/69 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head v 70'105, mlcod=0'0)", 2024-09-16T03:38:30.208 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:55.960027+0000", 2024-09-16T03:38:30.209 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.194507909999999, 2024-09-16T03:38:30.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200596892, 2024-09-16T03:38:30.209 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.209 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.209 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.209 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960027+0000", 2024-09-16T03:38:30.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960027+0000", 2024-09-16T03:38:30.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960030+0000", 2024-09-16T03:38:30.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9909999999999998e-06 2024-09-16T03:38:30.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.210 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.210 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960043+0000", 2024-09-16T03:38:30.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2428e-05 2024-09-16T03:38:30.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960076+0000", 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3494999999999997e-05 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960102+0000", 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5692000000000001e-05 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960181+0000", 2024-09-16T03:38:30.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8827999999999998e-05 2024-09-16T03:38:30.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960200+0000", 2024-09-16T03:38:30.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9463e-05 2024-09-16T03:38:30.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.160597+0000", 2024-09-16T03:38:30.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200396611 2024-09-16T03:38:30.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.160624+0000", 2024-09-16T03:38:30.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7384000000000001e-05 2024-09-16T03:38:30.213 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.213 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.213 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.213 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:94 3.0 e70/69 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head v 70'106, mlcod=0'0)", 2024-09-16T03:38:30.213 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:55.960145+0000", 2024-09-16T03:38:30.213 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.194390724000002, 2024-09-16T03:38:30.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20090359099999999, 2024-09-16T03:38:30.213 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.213 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960145+0000", 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960145+0000", 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960145+0000", 2024-09-16T03:38:30.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7300000000000002e-07 2024-09-16T03:38:30.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960147+0000", 2024-09-16T03:38:30.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.509e-06 2024-09-16T03:38:30.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960151+0000", 2024-09-16T03:38:30.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:38:30.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.060262+0000", 2024-09-16T03:38:30.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100110847 2024-09-16T03:38:30.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.160465+0000", 2024-09-16T03:38:30.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100203264 2024-09-16T03:38:30.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.160483+0000", 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.753e-05 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.161026+0000", 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00054326199999999998 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.161048+0000", 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1951999999999999e-05 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.217 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.11.23:98 3.0 e74/69 3:04dfaaf9:::3de53e1d584ced9b0f970855867dbc9a14b3fc90962aa20a81c1ed25ff4b961f:head v 74'124, mlcod=73'121)", 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:01.337812+0000", 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 28.816722956, 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20064802300000001, 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.337812+0000", 2024-09-16T03:38:30.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.337812+0000", 2024-09-16T03:38:30.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.337813+0000", 2024-09-16T03:38:30.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2699999999999998e-07 2024-09-16T03:38:30.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.337818+0000", 2024-09-16T03:38:30.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4440000000000001e-06 2024-09-16T03:38:30.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.337825+0000", 2024-09-16T03:38:30.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0269999999999998e-06 2024-09-16T03:38:30.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.437892+0000", 2024-09-16T03:38:30.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100067131 2024-09-16T03:38:30.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.538006+0000", 2024-09-16T03:38:30.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100114093 2024-09-16T03:38:30.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.538030+0000", 2024-09-16T03:38:30.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3944e-05 2024-09-16T03:38:30.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.538424+0000", 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00039373000000000001 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:01.538460+0000", 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6727000000000003e-05 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:141 3.1 e85/31 3:9387c678:::99a87d3b555b50e905a1ff55d99eb4aa025bd0070f07a4910d03cf1af95bbc44:head v 85'127, mlcod=82'124)", 2024-09-16T03:38:30.222 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.399323+0000", 2024-09-16T03:38:30.223 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.755211863, 2024-09-16T03:38:30.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.116304349, 2024-09-16T03:38:30.223 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.223 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.223 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.223 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.399323+0000", 2024-09-16T03:38:30.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.399323+0000", 2024-09-16T03:38:30.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.399325+0000", 2024-09-16T03:38:30.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.579e-06 2024-09-16T03:38:30.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.399348+0000", 2024-09-16T03:38:30.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2969e-05 2024-09-16T03:38:30.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.399359+0000", 2024-09-16T03:38:30.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0920999999999999e-05 2024-09-16T03:38:30.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.499435+0000", 2024-09-16T03:38:30.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100076279 2024-09-16T03:38:30.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.514890+0000", 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.015454927 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.514909+0000", 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8658e-05 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.515587+0000", 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00067845100000000005 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.515628+0000", 2024-09-16T03:38:30.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0565000000000002e-05 2024-09-16T03:38:30.227 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.227 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.227 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.227 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2543 4.a e86/32 4:5e3c0760:::smithi15736696-24:head v 86'88, mlcod=84'86)", 2024-09-16T03:38:30.227 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:17.073635+0000", 2024-09-16T03:38:30.227 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.080899915, 2024-09-16T03:38:30.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20396895300000001, 2024-09-16T03:38:30.227 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.227 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.073635+0000", 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.073635+0000", 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.073636+0000", 2024-09-16T03:38:30.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7099999999999997e-07 2024-09-16T03:38:30.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.074518+0000", 2024-09-16T03:38:30.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000882532 2024-09-16T03:38:30.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.074530+0000", 2024-09-16T03:38:30.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1799e-05 2024-09-16T03:38:30.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.174624+0000", 2024-09-16T03:38:30.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100094445 2024-09-16T03:38:30.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.274761+0000", 2024-09-16T03:38:30.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013633199999999 2024-09-16T03:38:30.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.274785+0000", 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3731e-05 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.277559+0000", 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027741179999999999 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.277604+0000", 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5624999999999998e-05 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.234 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.235 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.235 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2544 4.a e86/32 4:5e3c0760:::smithi15736696-24:head v 86'89, mlcod=84'86)", 2024-09-16T03:38:30.235 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:17.174656+0000", 2024-09-16T03:38:30.235 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 12.979879287999999, 2024-09-16T03:38:30.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102972786, 2024-09-16T03:38:30.235 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.235 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.235 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.235 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.174656+0000", 2024-09-16T03:38:30.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.174656+0000", 2024-09-16T03:38:30.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.174657+0000", 2024-09-16T03:38:30.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.438e-06 2024-09-16T03:38:30.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.174962+0000", 2024-09-16T03:38:30.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030437500000000001 2024-09-16T03:38:30.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.174970+0000", 2024-09-16T03:38:30.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6249999999999996e-06 2024-09-16T03:38:30.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.275042+0000", 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10007197600000001 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.275956+0000", 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00091321299999999996 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.275977+0000", 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1676e-05 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.277606+0000", 2024-09-16T03:38:30.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0016289640000000001 2024-09-16T03:38:30.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.277629+0000", 2024-09-16T03:38:30.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2518999999999999e-05 2024-09-16T03:38:30.239 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.239 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.239 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.240 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2618 4.f e89/32 4:fa163101:::smithi15736696-10:head v 89'67, mlcod=87'65)", 2024-09-16T03:38:30.240 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.330289+0000", 2024-09-16T03:38:30.240 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.8242463410000003, 2024-09-16T03:38:30.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.13640091100000001, 2024-09-16T03:38:30.240 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.240 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.240 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.240 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.330289+0000", 2024-09-16T03:38:30.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.330289+0000", 2024-09-16T03:38:30.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.330289+0000", 2024-09-16T03:38:30.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4999999999999999e-07 2024-09-16T03:38:30.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.330797+0000", 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00050832399999999999 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.330806+0000", 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3650000000000004e-06 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.430876+0000", 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100069751 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.464201+0000", 2024-09-16T03:38:30.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.033325191999999997 2024-09-16T03:38:30.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.464215+0000", 2024-09-16T03:38:30.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4335000000000001e-05 2024-09-16T03:38:30.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.466650+0000", 2024-09-16T03:38:30.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024354099999999998 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.466690+0000", 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9384000000000001e-05 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2664 4.f e91/32 4:f817def9:::smithi15736696-45:head v 91'71, mlcod=89'69)", 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.623500+0000", 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.5310348699999992, 2024-09-16T03:38:30.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20350020799999999, 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.623500+0000", 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.623500+0000", 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.623501+0000", 2024-09-16T03:38:30.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.99e-07 2024-09-16T03:38:30.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624286+0000", 2024-09-16T03:38:30.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078502800000000001 2024-09-16T03:38:30.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624295+0000", 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8630000000000003e-06 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624302+0000", 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0589999999999999e-06 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624347+0000", 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5531000000000002e-05 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624362+0000", 2024-09-16T03:38:30.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4571e-05 2024-09-16T03:38:30.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.826968+0000", 2024-09-16T03:38:30.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20260592199999999 2024-09-16T03:38:30.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.827001+0000", 2024-09-16T03:38:30.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3034999999999998e-05 2024-09-16T03:38:30.248 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.249 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.249 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.249 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2665 4.f e91/32 4:f817def9:::smithi15736696-45:head v 91'72, mlcod=89'69)", 2024-09-16T03:38:30.249 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.624911+0000", 2024-09-16T03:38:30.249 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.5296239450000009, 2024-09-16T03:38:30.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20261111400000001, 2024-09-16T03:38:30.249 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.249 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.249 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.249 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624911+0000", 2024-09-16T03:38:30.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624911+0000", 2024-09-16T03:38:30.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624912+0000", 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7000000000000004e-07 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.625313+0000", 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040085000000000001 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.625321+0000", 2024-09-16T03:38:30.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6210000000000004e-06 2024-09-16T03:38:30.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.625325+0000", 2024-09-16T03:38:30.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3800000000000004e-06 2024-09-16T03:38:30.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.825578+0000", 2024-09-16T03:38:30.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20025301500000001 2024-09-16T03:38:30.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.825591+0000", 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3326000000000001e-05 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.827497+0000", 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001905728 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.827522+0000", 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5423999999999999e-05 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.253 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.254 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.254 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.254 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.254 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:30.255 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:30.255 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:30.254+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:30.256 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:30.256 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:30.257 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:30.257 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:38:30.257 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:30.258 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:30.258 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:30.258 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:30.259 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:30.259 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:30.259 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:30.259 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:30.262 INFO:tasks.rados.rados.0.smithi157.stdout:1320: set_chunk oid 50 offset: 335872 length: 2716672 target oid 0cfdb6f916d94e75d5157501428af2b49bc98f21f1235407f8b06e7f5271f6e8 offset: 0 snap -1 2024-09-16T03:38:30.262 INFO:tasks.rados.rados.0.smithi157.stdout:1321: read oid 48 snap -1 2024-09-16T03:38:30.262 INFO:tasks.rados.rados.0.smithi157.stdout:1321: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-16T03:38:30.262 INFO:tasks.rados.rados.0.smithi157.stdout:1322: write (excl) oid 39 current snap is 0 2024-09-16T03:38:30.262 INFO:tasks.rados.rados.0.smithi157.stdout:1322: seq_num 258 ranges {736206=453891,1881413=454793} 2024-09-16T03:38:30.268 INFO:tasks.rados.rados.0.smithi157.stdout:1322: writing smithi15736696-39 from 736206 to 1190097 tid 1 2024-09-16T03:38:30.271 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:30.272 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:30.275 INFO:tasks.rados.rados.0.smithi157.stdout:1322: writing smithi15736696-39 from 1881413 to 2336206 tid 2 2024-09-16T03:38:30.275 INFO:tasks.rados.rados.0.smithi157.stdout:1323: read oid 33 snap -1 2024-09-16T03:38:30.275 INFO:tasks.rados.rados.0.smithi157.stdout:1323: expect (ObjNum 230 snap 0 seq_num 230) 2024-09-16T03:38:30.275 INFO:tasks.rados.rados.0.smithi157.stdout:1324: write oid 8 current snap is 0 2024-09-16T03:38:30.275 INFO:tasks.rados.rados.0.smithi157.stdout:1324: seq_num 259 ranges {753013=645493,2056248=296765} 2024-09-16T03:38:30.283 INFO:tasks.rados.rados.0.smithi157.stdout:1324: writing smithi15736696-8 from 753013 to 1398506 tid 1 2024-09-16T03:38:30.289 INFO:tasks.rados.rados.0.smithi157.stdout:1324: writing smithi15736696-8 from 2056248 to 2353013 tid 2 2024-09-16T03:38:30.289 INFO:tasks.rados.rados.0.smithi157.stdout:1325: write oid 45 current snap is 0 2024-09-16T03:38:30.289 INFO:tasks.rados.rados.0.smithi157.stdout:1319: finishing write tid 1 to smithi15736696-7 2024-09-16T03:38:30.289 INFO:tasks.rados.rados.0.smithi157.stdout:1325: seq_num 260 ranges {769820=720742,2114730=255090} 2024-09-16T03:38:30.297 INFO:tasks.rados.rados.0.smithi157.stdout:1325: writing smithi15736696-45 from 769820 to 1490562 tid 1 2024-09-16T03:38:30.302 INFO:tasks.rados.rados.0.smithi157.stdout:1325: writing smithi15736696-45 from 2114730 to 2369820 tid 2 2024-09-16T03:38:30.302 INFO:tasks.rados.rados.0.smithi157.stdout:1326: write oid 11 current snap is 0 2024-09-16T03:38:30.302 INFO:tasks.rados.rados.0.smithi157.stdout:1319: finishing write tid 2 to smithi15736696-7 2024-09-16T03:38:30.302 INFO:tasks.rados.rados.0.smithi157.stdout:1318: expect (ObjNum 205 snap 0 seq_num 205) 2024-09-16T03:38:30.356 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:30.354+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:30.369 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:30.369 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:30.369 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:38:30.369 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:30.376 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:38:30.379 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:38:30.382 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2603224 2024-09-16T03:38:30.432 INFO:tasks.rados.rados.0.smithi157.stdout:1319: finishing write tid 3 to smithi15736696-7 2024-09-16T03:38:30.432 INFO:tasks.rados.rados.0.smithi157.stdout:1319: finishing write tid 4 to smithi15736696-7 2024-09-16T03:38:30.432 INFO:tasks.rados.rados.0.smithi157.stdout:1319: oid 7 updating version 0 to 63 2024-09-16T03:38:30.432 INFO:tasks.rados.rados.0.smithi157.stdout:1319: oid 7 updating version 63 to 64 2024-09-16T03:38:30.432 INFO:tasks.rados.rados.0.smithi157.stdout:1319: oid 7 version 64 is already newer than 62 2024-09-16T03:38:30.432 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 64 (ObjNum 257 snap 0 seq_num 257) dirty exists 2024-09-16T03:38:30.432 INFO:tasks.rados.rados.0.smithi157.stdout:1319: left oid 7 (ObjNum 257 snap 0 seq_num 257) 2024-09-16T03:38:30.432 INFO:tasks.rados.rados.0.smithi157.stdout:1326: seq_num 261 ranges {786627=795991,2173212=213415} 2024-09-16T03:38:30.441 INFO:tasks.rados.rados.0.smithi157.stdout:1326: writing smithi15736696-11 from 786627 to 1582618 tid 1 2024-09-16T03:38:30.446 INFO:tasks.rados.rados.0.smithi157.stdout:1326: writing smithi15736696-11 from 2173212 to 2386627 tid 2 2024-09-16T03:38:30.446 INFO:tasks.rados.rados.0.smithi157.stdout:1318: done (8 left) 2024-09-16T03:38:30.446 INFO:tasks.rados.rados.0.smithi157.stdout:1319: done (7 left) 2024-09-16T03:38:30.447 INFO:tasks.rados.rados.0.smithi157.stdout:1327: tier_promote oid 28 2024-09-16T03:38:30.447 INFO:tasks.rados.rados.0.smithi157.stdout:1320: finishing set_chunk to oid 50 2024-09-16T03:38:30.447 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 50 set_chunk returned error code -95 offset: 335872 length: 2716672 2024-09-16T03:38:30.447 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 0 (ObjNum 218 snap 0 seq_num 218) dirty exists 2024-09-16T03:38:30.447 INFO:tasks.rados.rados.0.smithi157.stdout:1320: done (7 left) 2024-09-16T03:38:30.447 INFO:tasks.rados.rados.0.smithi157.stdout:1328: write (excl) oid 47 current snap is 0 2024-09-16T03:38:30.447 INFO:tasks.rados.rados.0.smithi157.stdout:1323: expect (ObjNum 230 snap 0 seq_num 230) 2024-09-16T03:38:30.456 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:30.454+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:30.492 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:30.492 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:30.492 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:38:30.492 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:38:30.492 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:30.501 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:38:30.558 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:30.556+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:30.609 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:30.609 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:38:30.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:38:30.610 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:30.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.610 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:30.610 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:38:30.610 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.420710779000004, 2024-09-16T03:38:30.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102630163, 2024-09-16T03:38:30.610 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.610 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.610 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.610 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:38:30.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:38:30.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186081+0000", 2024-09-16T03:38:30.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5600000000000005e-07 2024-09-16T03:38:30.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186970+0000", 2024-09-16T03:38:30.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00088889000000000004 2024-09-16T03:38:30.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186979+0000", 2024-09-16T03:38:30.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0620000000000007e-06 2024-09-16T03:38:30.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186986+0000", 2024-09-16T03:38:30.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:38:30.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287184+0000", 2024-09-16T03:38:30.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100198036 2024-09-16T03:38:30.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287199+0000", 2024-09-16T03:38:30.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5458999999999999e-05 2024-09-16T03:38:30.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288659+0000", 2024-09-16T03:38:30.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014601900000000001 2024-09-16T03:38:30.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288711+0000", 2024-09-16T03:38:30.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1088000000000003e-05 2024-09-16T03:38:30.615 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.615 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.615 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.615 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:30.615 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:38:30.616 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.419674345999994, 2024-09-16T03:38:30.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10205370900000001, 2024-09-16T03:38:30.616 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.616 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.616 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.616 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:38:30.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:38:30.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187118+0000", 2024-09-16T03:38:30.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8599999999999998e-07 2024-09-16T03:38:30.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188062+0000", 2024-09-16T03:38:30.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094447300000000004 2024-09-16T03:38:30.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188070+0000", 2024-09-16T03:38:30.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.695e-06 2024-09-16T03:38:30.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188074+0000", 2024-09-16T03:38:30.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0709999999999996e-06 2024-09-16T03:38:30.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287842+0000", 2024-09-16T03:38:30.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099768478999999993 2024-09-16T03:38:30.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287864+0000", 2024-09-16T03:38:30.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1679999999999999e-05 2024-09-16T03:38:30.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289137+0000", 2024-09-16T03:38:30.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012732430000000001 2024-09-16T03:38:30.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289171+0000", 2024-09-16T03:38:30.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3382e-05 2024-09-16T03:38:30.620 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.620 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.621 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.621 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:42 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:30.621 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:38:30.621 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.349592438000002, 2024-09-16T03:38:30.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10172858, 2024-09-16T03:38:30.621 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.621 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.621 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.621 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:38:30.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:38:30.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257201+0000", 2024-09-16T03:38:30.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:38:30.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257536+0000", 2024-09-16T03:38:30.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033513899999999998 2024-09-16T03:38:30.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257555+0000", 2024-09-16T03:38:30.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8547999999999999e-05 2024-09-16T03:38:30.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257566+0000", 2024-09-16T03:38:30.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1379999999999999e-05 2024-09-16T03:38:30.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357720+0000", 2024-09-16T03:38:30.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100153532 2024-09-16T03:38:30.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.625 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.625 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357734+0000", 2024-09-16T03:38:30.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4491000000000001e-05 2024-09-16T03:38:30.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.625 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.625 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358896+0000", 2024-09-16T03:38:30.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011620599999999999 2024-09-16T03:38:30.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.625 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.626 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358927+0000", 2024-09-16T03:38:30.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1158999999999998e-05 2024-09-16T03:38:30.626 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.626 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.626 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.626 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:43 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'3, mlcod=0'0)", 2024-09-16T03:38:30.626 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:38:30.626 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.347808525999994, 2024-09-16T03:38:30.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101764453, 2024-09-16T03:38:30.626 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.627 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.627 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.627 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:38:30.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:38:30.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258984+0000", 2024-09-16T03:38:30.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5689999999999999e-06 2024-09-16T03:38:30.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258994+0000", 2024-09-16T03:38:30.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-16T03:38:30.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259008+0000", 2024-09-16T03:38:30.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4151999999999999e-05 2024-09-16T03:38:30.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359073+0000", 2024-09-16T03:38:30.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100065628 2024-09-16T03:38:30.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359110+0000", 2024-09-16T03:38:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7045999999999997e-05 2024-09-16T03:38:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359124+0000", 2024-09-16T03:38:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.382e-05 2024-09-16T03:38:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.630 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.631 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.631 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360716+0000", 2024-09-16T03:38:30.631 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015919829999999999 2024-09-16T03:38:30.631 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.631 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.631 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.631 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360747+0000", 2024-09-16T03:38:30.631 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0874999999999998e-05 2024-09-16T03:38:30.631 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.631 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.631 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.632 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.632 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.632 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:492 4.5 e45/32 4:a11e1d7c:::smithi15736696-43:head v 45'22, mlcod=45'20)", 2024-09-16T03:38:30.632 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:38:30.632 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 68.783440239000001, 2024-09-16T03:38:30.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100797151, 2024-09-16T03:38:30.632 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.632 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.632 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.632 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.632 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.633 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.633 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:38:30.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.633 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.633 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:38:30.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823353+0000", 2024-09-16T03:38:30.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2450000000000001e-06 2024-09-16T03:38:30.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823363+0000", 2024-09-16T03:38:30.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0074999999999999e-05 2024-09-16T03:38:30.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823378+0000", 2024-09-16T03:38:30.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4534e-05 2024-09-16T03:38:30.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923461+0000", 2024-09-16T03:38:30.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100082859 2024-09-16T03:38:30.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923525+0000", 2024-09-16T03:38:30.636 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4616999999999995e-05 2024-09-16T03:38:30.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923549+0000", 2024-09-16T03:38:30.636 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.357e-05 2024-09-16T03:38:30.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.637 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.637 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924106+0000", 2024-09-16T03:38:30.637 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000556753 2024-09-16T03:38:30.637 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.637 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.637 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.637 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924148+0000", 2024-09-16T03:38:30.637 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2497999999999999e-05 2024-09-16T03:38:30.637 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.637 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.637 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.638 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.638 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.638 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:806 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'25, mlcod=45'22)", 2024-09-16T03:38:30.638 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.019370+0000", 2024-09-16T03:38:30.638 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.587420911999999, 2024-09-16T03:38:30.638 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0048461379999999998, 2024-09-16T03:38:30.638 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.638 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.638 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.638 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.638 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019370+0000", 2024-09-16T03:38:30.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019370+0000", 2024-09-16T03:38:30.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.640 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019372+0000", 2024-09-16T03:38:30.640 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.424e-06 2024-09-16T03:38:30.640 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.640 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.640 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.640 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019541+0000", 2024-09-16T03:38:30.640 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016961999999999999 2024-09-16T03:38:30.640 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.640 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.640 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.640 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019553+0000", 2024-09-16T03:38:30.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1678000000000001e-05 2024-09-16T03:38:30.641 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.641 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.641 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.641 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.019561+0000", 2024-09-16T03:38:30.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7579999999999992e-06 2024-09-16T03:38:30.641 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.641 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.641 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.641 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022299+0000", 2024-09-16T03:38:30.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002738061 2024-09-16T03:38:30.641 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.642 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.642 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.642 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.022326+0000", 2024-09-16T03:38:30.642 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7515999999999999e-05 2024-09-16T03:38:30.642 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.642 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.642 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.642 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024179+0000", 2024-09-16T03:38:30.642 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001853005 2024-09-16T03:38:30.642 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.642 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.643 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.643 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.024216+0000", 2024-09-16T03:38:30.643 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7076000000000002e-05 2024-09-16T03:38:30.643 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.643 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.643 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.643 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.643 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.643 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:808 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'27, mlcod=45'22)", 2024-09-16T03:38:30.643 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:38:30.643 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.583746963999999, 2024-09-16T03:38:30.644 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.097081386000000006, 2024-09-16T03:38:30.644 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.644 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.644 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.644 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.644 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.644 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.644 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:38:30.644 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.645 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.645 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.645 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:38:30.645 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.645 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.645 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.645 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023046+0000", 2024-09-16T03:38:30.645 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1059999999999998e-06 2024-09-16T03:38:30.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.646 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.646 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023056+0000", 2024-09-16T03:38:30.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3719999999999994e-06 2024-09-16T03:38:30.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.646 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.646 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023067+0000", 2024-09-16T03:38:30.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1246999999999999e-05 2024-09-16T03:38:30.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.647 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.647 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023075+0000", 2024-09-16T03:38:30.647 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4710000000000007e-06 2024-09-16T03:38:30.647 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.647 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.647 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.119708+0000", 2024-09-16T03:38:30.647 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096632179999999998 2024-09-16T03:38:30.647 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.648 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.648 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.119725+0000", 2024-09-16T03:38:30.648 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7754999999999999e-05 2024-09-16T03:38:30.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.648 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.648 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.120092+0000", 2024-09-16T03:38:30.648 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00036668099999999999 2024-09-16T03:38:30.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.648 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.649 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.120126+0000", 2024-09-16T03:38:30.649 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3574000000000001e-05 2024-09-16T03:38:30.649 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.649 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.649 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.649 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.649 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.649 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:53'30,epoch:57/50,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:38:30.649 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:38:30.649 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.208088623000002, 2024-09-16T03:38:30.649 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100230893, 2024-09-16T03:38:30.650 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.650 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.650 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:30.650 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.650 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.650 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.650 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:38:30.650 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.650 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.650 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.650 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:38:30.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.651 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.651 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398705+0000", 2024-09-16T03:38:30.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5689999999999999e-06 2024-09-16T03:38:30.651 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.651 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398722+0000", 2024-09-16T03:38:30.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6558e-05 2024-09-16T03:38:30.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398735+0000", 2024-09-16T03:38:30.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3499999999999999e-05 2024-09-16T03:38:30.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398746+0000", 2024-09-16T03:38:30.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1182999999999999e-05 2024-09-16T03:38:30.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498898+0000", 2024-09-16T03:38:30.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015112599999999 2024-09-16T03:38:30.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498906+0000", 2024-09-16T03:38:30.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3189999999999995e-06 2024-09-16T03:38:30.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.654 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.654 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498933+0000", 2024-09-16T03:38:30.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7637999999999999e-05 2024-09-16T03:38:30.654 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.654 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.654 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.654 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1570 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'31, mlcod=48'29)", 2024-09-16T03:38:30.654 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:38:30.654 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.432481627999998, 2024-09-16T03:38:30.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10253683500000001, 2024-09-16T03:38:30.655 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.655 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.655 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.655 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:38:30.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.655 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:38:30.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:38:30.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8500000000000002e-07 2024-09-16T03:38:30.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174689+0000", 2024-09-16T03:38:30.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037958600000000001 2024-09-16T03:38:30.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174699+0000", 2024-09-16T03:38:30.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0015e-05 2024-09-16T03:38:30.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174706+0000", 2024-09-16T03:38:30.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6250000000000001e-06 2024-09-16T03:38:30.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274865+0000", 2024-09-16T03:38:30.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100159157 2024-09-16T03:38:30.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274879+0000", 2024-09-16T03:38:30.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3983e-05 2024-09-16T03:38:30.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.659 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276820+0000", 2024-09-16T03:38:30.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001941248 2024-09-16T03:38:30.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.659 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.659 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276846+0000", 2024-09-16T03:38:30.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5936e-05 2024-09-16T03:38:30.659 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.659 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.659 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.660 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1571 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'32, mlcod=48'29)", 2024-09-16T03:38:30.660 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:38:30.660 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.432030873999999, 2024-09-16T03:38:30.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10249543, 2024-09-16T03:38:30.660 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.660 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.660 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.660 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.660 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:38:30.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:38:30.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174761+0000", 2024-09-16T03:38:30.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0800000000000004e-07 2024-09-16T03:38:30.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175053+0000", 2024-09-16T03:38:30.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029151600000000002 2024-09-16T03:38:30.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175063+0000", 2024-09-16T03:38:30.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0538e-05 2024-09-16T03:38:30.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.663 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175070+0000", 2024-09-16T03:38:30.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7819999999999996e-06 2024-09-16T03:38:30.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.663 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275895+0000", 2024-09-16T03:38:30.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10082516 2024-09-16T03:38:30.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.664 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275909+0000", 2024-09-16T03:38:30.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4197e-05 2024-09-16T03:38:30.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.664 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277232+0000", 2024-09-16T03:38:30.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013231009999999999 2024-09-16T03:38:30.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.664 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277256+0000", 2024-09-16T03:38:30.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3428e-05 2024-09-16T03:38:30.665 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.665 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.665 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.665 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1572 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'33, mlcod=48'29)", 2024-09-16T03:38:30.665 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:38:30.665 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.431689574000004, 2024-09-16T03:38:30.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10216886, 2024-09-16T03:38:30.665 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.666 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.666 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.666 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.666 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:38:30.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.666 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:38:30.667 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.667 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.667 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.667 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.667 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:38:30.667 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8400000000000004e-07 2024-09-16T03:38:30.667 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.667 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.667 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.667 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175107+0000", 2024-09-16T03:38:30.668 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.578e-06 2024-09-16T03:38:30.668 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.668 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.668 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.668 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175114+0000", 2024-09-16T03:38:30.668 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.888e-06 2024-09-16T03:38:30.668 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.668 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.668 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.668 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175119+0000", 2024-09-16T03:38:30.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.074e-06 2024-09-16T03:38:30.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276662+0000", 2024-09-16T03:38:30.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101542753 2024-09-16T03:38:30.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276674+0000", 2024-09-16T03:38:30.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1960000000000001e-05 2024-09-16T03:38:30.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277258+0000", 2024-09-16T03:38:30.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058431900000000003 2024-09-16T03:38:30.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277270+0000", 2024-09-16T03:38:30.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2604e-05 2024-09-16T03:38:30.670 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.670 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.671 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.671 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:60'33,epoch:61/50,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:38:30.671 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:38:30.671 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.929768320000001, 2024-09-16T03:38:30.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10023048699999999, 2024-09-16T03:38:30.671 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.671 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.671 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:30.671 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.672 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:38:30.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.672 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.672 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:38:30.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.672 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.673 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:38:30.673 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8599999999999999e-07 2024-09-16T03:38:30.673 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.673 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.673 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.673 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677027+0000", 2024-09-16T03:38:30.673 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7359999999999999e-06 2024-09-16T03:38:30.673 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.673 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.673 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.673 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677042+0000", 2024-09-16T03:38:30.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4888e-05 2024-09-16T03:38:30.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.674 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.674 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677052+0000", 2024-09-16T03:38:30.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7750000000000004e-06 2024-09-16T03:38:30.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.674 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.674 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777182+0000", 2024-09-16T03:38:30.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100130394 2024-09-16T03:38:30.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777195+0000", 2024-09-16T03:38:30.675 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3373999999999999e-05 2024-09-16T03:38:30.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777253+0000", 2024-09-16T03:38:30.675 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7933999999999998e-05 2024-09-16T03:38:30.675 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.675 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.676 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.676 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.676 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.676 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2454 4.15 e84/50 4:aff4cc41:::smithi15736696-11:head v 84'41, mlcod=79'39)", 2024-09-16T03:38:30.676 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:38:30.676 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.143552211999999, 2024-09-16T03:38:30.676 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101244889, 2024-09-16T03:38:30.676 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.676 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.676 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.676 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.676 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.677 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.677 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:38:30.677 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.677 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.677 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.677 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:38:30.677 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.677 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.677 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.678 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:38:30.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2899999999999999e-07 2024-09-16T03:38:30.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.678 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.678 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463249+0000", 2024-09-16T03:38:30.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8039999999999996e-06 2024-09-16T03:38:30.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.678 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.678 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463267+0000", 2024-09-16T03:38:30.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7955e-05 2024-09-16T03:38:30.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563340+0000", 2024-09-16T03:38:30.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100072999 2024-09-16T03:38:30.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563396+0000", 2024-09-16T03:38:30.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6141999999999999e-05 2024-09-16T03:38:30.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.680 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.680 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563414+0000", 2024-09-16T03:38:30.680 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8088999999999999e-05 2024-09-16T03:38:30.680 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.680 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.680 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.680 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.564427+0000", 2024-09-16T03:38:30.680 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010127230000000001 2024-09-16T03:38:30.680 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.680 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.680 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.681 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.564484+0000", 2024-09-16T03:38:30.681 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6747999999999999e-05 2024-09-16T03:38:30.681 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.681 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.681 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.681 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.681 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.681 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2467 4.5 e85/50)", 2024-09-16T03:38:30.681 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:38:30.681 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.455660086, 2024-09-16T03:38:30.681 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102439455, 2024-09-16T03:38:30.682 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.682 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.682 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.682 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.682 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.682 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.682 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:38:30.682 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.682 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.682 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.683 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.683 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:38:30.683 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.683 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.683 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.683 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.683 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151134+0000", 2024-09-16T03:38:30.683 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2030000000000001e-06 2024-09-16T03:38:30.683 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.683 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.684 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151480+0000", 2024-09-16T03:38:30.684 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034613999999999999 2024-09-16T03:38:30.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.684 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151500+0000", 2024-09-16T03:38:30.684 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9582999999999999e-05 2024-09-16T03:38:30.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.684 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151513+0000", 2024-09-16T03:38:30.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:38:30.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.685 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251653+0000", 2024-09-16T03:38:30.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10014007900000001 2024-09-16T03:38:30.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.685 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251679+0000", 2024-09-16T03:38:30.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5721e-05 2024-09-16T03:38:30.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.686 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.686 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253532+0000", 2024-09-16T03:38:30.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001852701 2024-09-16T03:38:30.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.686 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.686 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253571+0000", 2024-09-16T03:38:30.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8677999999999998e-05 2024-09-16T03:38:30.686 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.687 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.687 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.687 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.687 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2468 4.5 e85/50)", 2024-09-16T03:38:30.687 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:38:30.687 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.454191557, 2024-09-16T03:38:30.687 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101505239, 2024-09-16T03:38:30.687 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.687 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.687 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.688 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.688 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.688 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.688 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:38:30.688 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.688 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.688 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.688 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.688 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:38:30.688 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.688 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.689 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.689 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.689 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152601+0000", 2024-09-16T03:38:30.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6869999999999999e-06 2024-09-16T03:38:30.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.689 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.689 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.689 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152719+0000", 2024-09-16T03:38:30.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117508 2024-09-16T03:38:30.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.689 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152732+0000", 2024-09-16T03:38:30.690 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:38:30.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152740+0000", 2024-09-16T03:38:30.690 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7090000000000002e-06 2024-09-16T03:38:30.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.691 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.691 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252704+0000", 2024-09-16T03:38:30.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099964259999999999 2024-09-16T03:38:30.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.691 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.691 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.691 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252727+0000", 2024-09-16T03:38:30.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3601e-05 2024-09-16T03:38:30.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.691 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.691 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.692 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254075+0000", 2024-09-16T03:38:30.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00134772 2024-09-16T03:38:30.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.692 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.692 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254105+0000", 2024-09-16T03:38:30.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9802e-05 2024-09-16T03:38:30.692 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.692 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.692 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.693 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2623 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'62, mlcod=85'60)", 2024-09-16T03:38:30.693 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:38:30.693 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.257553927, 2024-09-16T03:38:30.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.104590794, 2024-09-16T03:38:30.693 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.693 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.693 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.693 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:38:30.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:38:30.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:38:30.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1899999999999999e-07 2024-09-16T03:38:30.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349496+0000", 2024-09-16T03:38:30.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000258934 2024-09-16T03:38:30.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349506+0000", 2024-09-16T03:38:30.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9389999999999992e-06 2024-09-16T03:38:30.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349513+0000", 2024-09-16T03:38:30.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6209999999999998e-06 2024-09-16T03:38:30.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449668+0000", 2024-09-16T03:38:30.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015550099999999 2024-09-16T03:38:30.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449695+0000", 2024-09-16T03:38:30.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6253000000000001e-05 2024-09-16T03:38:30.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453784+0000", 2024-09-16T03:38:30.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040896609999999996 2024-09-16T03:38:30.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453828+0000", 2024-09-16T03:38:30.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3665999999999998e-05 2024-09-16T03:38:30.698 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.698 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.698 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.698 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2624 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'63, mlcod=85'60)", 2024-09-16T03:38:30.698 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:38:30.698 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.256206689000001, 2024-09-16T03:38:30.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103286796, 2024-09-16T03:38:30.698 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.699 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.699 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.699 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:38:30.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:38:30.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350586+0000", 2024-09-16T03:38:30.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0020000000000001e-06 2024-09-16T03:38:30.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350794+0000", 2024-09-16T03:38:30.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020824099999999999 2024-09-16T03:38:30.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.701 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.701 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350805+0000", 2024-09-16T03:38:30.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1443999999999999e-05 2024-09-16T03:38:30.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.701 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.701 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450876+0000", 2024-09-16T03:38:30.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100070612 2024-09-16T03:38:30.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.702 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450960+0000", 2024-09-16T03:38:30.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4444999999999995e-05 2024-09-16T03:38:30.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.702 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450980+0000", 2024-09-16T03:38:30.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9728000000000002e-05 2024-09-16T03:38:30.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453831+0000", 2024-09-16T03:38:30.703 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002850923 2024-09-16T03:38:30.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453871+0000", 2024-09-16T03:38:30.703 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0401000000000002e-05 2024-09-16T03:38:30.703 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.703 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.703 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.704 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2626 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'65, mlcod=85'60)", 2024-09-16T03:38:30.704 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:38:30.704 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.155502712000001, 2024-09-16T03:38:30.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100625902, 2024-09-16T03:38:30.704 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.704 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.704 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.704 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:38:30.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:38:30.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451289+0000", 2024-09-16T03:38:30.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7100000000000002e-07 2024-09-16T03:38:30.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451295+0000", 2024-09-16T03:38:30.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2890000000000003e-06 2024-09-16T03:38:30.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451300+0000", 2024-09-16T03:38:30.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0710000000000002e-06 2024-09-16T03:38:30.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.707 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551399+0000", 2024-09-16T03:38:30.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10009881 2024-09-16T03:38:30.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.707 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551482+0000", 2024-09-16T03:38:30.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2738000000000006e-05 2024-09-16T03:38:30.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.708 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551496+0000", 2024-09-16T03:38:30.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4394e-05 2024-09-16T03:38:30.708 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.708 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.708 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551881+0000", 2024-09-16T03:38:30.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038522300000000002 2024-09-16T03:38:30.708 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.709 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551914+0000", 2024-09-16T03:38:30.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2906e-05 2024-09-16T03:38:30.709 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.709 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.709 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.709 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.709 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.709 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2661 4.15 e91/50 4:aff4cc41:::smithi15736696-11:head v 91'52, mlcod=88'48)", 2024-09-16T03:38:30.709 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:38:30.709 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.1976064589999993, 2024-09-16T03:38:30.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10164756699999999, 2024-09-16T03:38:30.710 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.710 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.710 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.710 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:38:30.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:38:30.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409186+0000", 2024-09-16T03:38:30.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-06 2024-09-16T03:38:30.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409568+0000", 2024-09-16T03:38:30.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038196600000000002 2024-09-16T03:38:30.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.712 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409578+0000", 2024-09-16T03:38:30.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.781e-06 2024-09-16T03:38:30.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.712 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409583+0000", 2024-09-16T03:38:30.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3070000000000002e-06 2024-09-16T03:38:30.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.713 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.713 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409633+0000", 2024-09-16T03:38:30.713 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0358000000000001e-05 2024-09-16T03:38:30.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.713 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.713 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409649+0000", 2024-09-16T03:38:30.713 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5875999999999998e-05 2024-09-16T03:38:30.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.713 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510800+0000", 2024-09-16T03:38:30.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101150728 2024-09-16T03:38:30.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510832+0000", 2024-09-16T03:38:30.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2500999999999999e-05 2024-09-16T03:38:30.714 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.714 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.714 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.715 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2662 4.15 e91/50 4:aff4cc41:::smithi15736696-11:head v 91'53, mlcod=88'48)", 2024-09-16T03:38:30.715 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:38:30.715 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.1963985089999998, 2024-09-16T03:38:30.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100650287, 2024-09-16T03:38:30.715 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.715 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.715 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:30.715 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:38:30.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:38:30.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410394+0000", 2024-09-16T03:38:30.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6299999999999993e-07 2024-09-16T03:38:30.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410399+0000", 2024-09-16T03:38:30.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9400000000000001e-06 2024-09-16T03:38:30.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410407+0000", 2024-09-16T03:38:30.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8859999999999993e-06 2024-09-16T03:38:30.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410412+0000", 2024-09-16T03:38:30.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9339999999999997e-06 2024-09-16T03:38:30.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:30.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510516+0000", 2024-09-16T03:38:30.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100103159 2024-09-16T03:38:30.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:30.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510536+0000", 2024-09-16T03:38:30.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0072999999999999e-05 2024-09-16T03:38:30.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:30.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.511018+0000", 2024-09-16T03:38:30.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00048191899999999998 2024-09-16T03:38:30.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:30.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.511043+0000", 2024-09-16T03:38:30.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5412999999999999e-05 2024-09-16T03:38:30.720 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.720 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.720 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.720 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.720 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.720 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:30.722 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:38:30.723 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:30.720+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:30.742 INFO:tasks.rados.rados.0.smithi157.stdout:1324: finishing write tid 1 to smithi15736696-8 2024-09-16T03:38:30.742 INFO:tasks.rados.rados.0.smithi157.stdout:1328: seq_num 262 ranges {403434=471240,1315341=400756} 2024-09-16T03:38:30.747 INFO:tasks.rados.rados.0.smithi157.stdout:1328: writing smithi15736696-47 from 403434 to 874674 tid 1 2024-09-16T03:38:30.752 INFO:tasks.rados.rados.0.smithi157.stdout:1328: writing smithi15736696-47 from 1315341 to 1716097 tid 2 2024-09-16T03:38:30.752 INFO:tasks.rados.rados.0.smithi157.stdout:1323: done (7 left) 2024-09-16T03:38:30.752 INFO:tasks.rados.rados.0.smithi157.stdout:1329: read oid 33 snap -1 2024-09-16T03:38:30.752 INFO:tasks.rados.rados.0.smithi157.stdout:1329: expect (ObjNum 230 snap 0 seq_num 230) 2024-09-16T03:38:30.752 INFO:tasks.rados.rados.0.smithi157.stdout:1330: write (excl) oid 41 current snap is 0 2024-09-16T03:38:30.752 INFO:tasks.rados.rados.0.smithi157.stdout:1330: seq_num 263 ranges {420241=546489,1373823=544414} 2024-09-16T03:38:30.757 INFO:tasks.rados.rados.0.smithi157.stdout:1330: writing smithi15736696-41 from 420241 to 966730 tid 1 2024-09-16T03:38:30.763 INFO:tasks.rados.rados.0.smithi157.stdout:1330: writing smithi15736696-41 from 1373823 to 1918237 tid 2 2024-09-16T03:38:30.763 INFO:tasks.rados.rados.0.smithi157.stdout:1331: read oid 31 snap -1 2024-09-16T03:38:30.763 INFO:tasks.rados.rados.0.smithi157.stdout:1331: expect (ObjNum 226 snap 0 seq_num 226) 2024-09-16T03:38:30.763 INFO:tasks.rados.rados.0.smithi157.stdout:1332: write (excl) oid 43 current snap is 0 2024-09-16T03:38:30.763 INFO:tasks.rados.rados.0.smithi157.stdout:1332: seq_num 264 ranges {437048=621738,1832305=404425} 2024-09-16T03:38:30.768 INFO:tasks.rados.rados.0.smithi157.stdout:1332: writing smithi15736696-43 from 437048 to 1058786 tid 1 2024-09-16T03:38:30.775 INFO:tasks.rados.rados.0.smithi157.stdout:1332: writing smithi15736696-43 from 1832305 to 2236730 tid 2 2024-09-16T03:38:30.775 INFO:tasks.rados.rados.0.smithi157.stdout:1333: read oid 5 snap -1 2024-09-16T03:38:30.775 INFO:tasks.rados.rados.0.smithi157.stdout:1333: expect deleted 2024-09-16T03:38:30.775 INFO:tasks.rados.rados.0.smithi157.stdout:1334: write (excl) oid 35 current snap is 0 2024-09-16T03:38:30.775 INFO:tasks.rados.rados.0.smithi157.stdout:1321: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-16T03:38:30.823 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:30.822+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:30.832 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2950 4.13 4.fa0244f3 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:38:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:30.774588+0000", 2024-09-16T03:38:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.056457350000000003, 2024-09-16T03:38:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.056475740000000003, 2024-09-16T03:38:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.832 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "no flag points reached", 2024-09-16T03:38:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2950 2024-09-16T03:38:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.774588+0000", 2024-09-16T03:38:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.774588+0000", 2024-09-16T03:38:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.774591+0000", 2024-09-16T03:38:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8789999999999999e-06 2024-09-16T03:38:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.774603+0000", 2024-09-16T03:38:30.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1319e-05 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.774618+0000", 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5301000000000001e-05 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2947 4.5 e98/50)", 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:30.775202+0000", 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.055844183999999998, 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055891047999999999, 2024-09-16T03:38:30.835 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.775202+0000", 2024-09-16T03:38:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.775202+0000", 2024-09-16T03:38:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:30.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.775203+0000", 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.72e-06 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.775431+0000", 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00022778699999999999 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.775445+0000", 2024-09-16T03:38:30.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4171999999999999e-05 2024-09-16T03:38:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.775454+0000", 2024-09-16T03:38:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7609999999999996e-06 2024-09-16T03:38:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-16T03:38:30.838 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 2 2024-09-16T03:38:30.838 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:30.841 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:38:30.889 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2169102 2024-09-16T03:38:30.915 INFO:tasks.rados.rados.0.smithi157.stdout:1322: finishing write tid 1 to smithi15736696-39 2024-09-16T03:38:30.915 INFO:tasks.rados.rados.0.smithi157.stdout:1334: seq_num 265 ranges {453855=696987,1890787=548083} 2024-09-16T03:38:30.923 INFO:tasks.rados.rados.0.smithi157.stdout:1334: writing smithi15736696-35 from 453855 to 1150842 tid 1 2024-09-16T03:38:30.923 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:30.922+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:30.933 INFO:tasks.rados.rados.0.smithi157.stdout:1334: writing smithi15736696-35 from 1890787 to 2438870 tid 2 2024-09-16T03:38:30.933 INFO:tasks.rados.rados.0.smithi157.stdout:1321: done (12 left) 2024-09-16T03:38:30.933 INFO:tasks.rados.rados.0.smithi157.stdout:1335: read oid 42 snap -1 2024-09-16T03:38:30.933 INFO:tasks.rados.rados.0.smithi157.stdout:1335: expect (ObjNum 200 snap 0 seq_num 200) 2024-09-16T03:38:30.933 INFO:tasks.rados.rados.0.smithi157.stdout:1336: read oid 7 snap -1 2024-09-16T03:38:30.933 INFO:tasks.rados.rados.0.smithi157.stdout:1336: expect (ObjNum 257 snap 0 seq_num 257) 2024-09-16T03:38:30.933 INFO:tasks.rados.rados.0.smithi157.stdout:1337: read oid 40 snap -1 2024-09-16T03:38:30.933 INFO:tasks.rados.rados.0.smithi157.stdout:1337: expect (ObjNum 245 snap 0 seq_num 245) 2024-09-16T03:38:30.933 INFO:tasks.rados.rados.0.smithi157.stdout:1338: read oid 30 snap -1 2024-09-16T03:38:30.933 INFO:tasks.rados.rados.0.smithi157.stdout:1338: expect deleted 2024-09-16T03:38:30.933 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout:1324: finishing write tid 2 to smithi15736696-8 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout:1324: finishing write tid 3 to smithi15736696-8 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout:1324: finishing write tid 4 to smithi15736696-8 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout:1324: oid 8 updating version 0 to 85 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout:1324: oid 8 version 85 is already newer than 84 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout:1324: oid 8 version 85 is already newer than 83 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 85 (ObjNum 259 snap 0 seq_num 259) dirty exists 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout:1324: left oid 8 (ObjNum 259 snap 0 seq_num 259) 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout:1325: finishing write tid 1 to smithi15736696-45 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout:1322: finishing write tid 2 to smithi15736696-39 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout:1324: done (15 left) 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout:1339: read oid 17 snap -1 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout:1339: expect (ObjNum 248 snap 0 seq_num 248) 2024-09-16T03:38:30.934 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:30.935 INFO:tasks.rados.rados.0.smithi157.stdout:1322: finishing write tid 3 to smithi15736696-39 2024-09-16T03:38:30.935 INFO:tasks.rados.rados.0.smithi157.stdout:1322: finishing write tid 4 to smithi15736696-39 2024-09-16T03:38:30.935 INFO:tasks.rados.rados.0.smithi157.stdout:1322: oid 39 updating version 0 to 71 2024-09-16T03:38:30.935 INFO:tasks.rados.rados.0.smithi157.stdout:1322: oid 39 version 71 is already newer than 70 2024-09-16T03:38:30.935 INFO:tasks.rados.rados.0.smithi157.stdout:1322: oid 39 updating version 71 to 72 2024-09-16T03:38:30.935 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 72 (ObjNum 258 snap 0 seq_num 258) dirty exists 2024-09-16T03:38:30.935 INFO:tasks.rados.rados.0.smithi157.stdout:1322: left oid 39 (ObjNum 258 snap 0 seq_num 258) 2024-09-16T03:38:30.935 INFO:tasks.rados.rados.0.smithi157.stdout:1325: finishing write tid 2 to smithi15736696-45 2024-09-16T03:38:30.935 INFO:tasks.rados.rados.0.smithi157.stdout:1325: finishing write tid 3 to smithi15736696-45 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:1325: finishing write tid 4 to smithi15736696-45 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:1325: oid 45 updating version 0 to 86 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:1325: oid 45 updating version 86 to 87 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:1325: oid 45 updating version 87 to 88 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 88 (ObjNum 260 snap 0 seq_num 260) dirty exists 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:1325: left oid 45 (ObjNum 260 snap 0 seq_num 260) 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:1326: finishing write tid 1 to smithi15736696-11 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:1326: finishing write tid 2 to smithi15736696-11 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:1326: finishing write tid 3 to smithi15736696-11 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:1326: finishing write tid 4 to smithi15736696-11 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:1326: oid 11 updating version 0 to 54 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:1326: oid 11 updating version 54 to 56 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:1326: oid 11 version 56 is already newer than 55 2024-09-16T03:38:30.936 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 56 (ObjNum 261 snap 0 seq_num 261) dirty exists 2024-09-16T03:38:30.937 INFO:tasks.rados.rados.0.smithi157.stdout:1326: left oid 11 (ObjNum 261 snap 0 seq_num 261) 2024-09-16T03:38:30.937 INFO:tasks.rados.rados.0.smithi157.stdout:1327: got (0) Success 2024-09-16T03:38:30.937 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 96 (ObjNum 252 snap 0 seq_num 252) dirty exists 2024-09-16T03:38:30.937 INFO:tasks.rados.rados.0.smithi157.stdout:1329: expect (ObjNum 230 snap 0 seq_num 230) 2024-09-16T03:38:30.942 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1426732 2024-09-16T03:38:30.993 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:30.993 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:30.993 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:38:30.993 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:38:30.993 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:31.002 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:38:31.043 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:31.041+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:31.125 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:31.124+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:31.152 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:31.152 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:38:31.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:38:31.152 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:31.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.152 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T03:38:31.152 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:38:31.152 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 100.071350729, 2024-09-16T03:38:31.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.88203620400000005, 2024-09-16T03:38:31.152 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.153 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.153 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.153 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.153 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.153 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.153 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:38:31.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.153 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:38:31.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076767+0000", 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9300000000000002e-06 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076774+0000", 2024-09-16T03:38:31.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6630000000000004e-06 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076785+0000", 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1221e-05 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076796+0000", 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0428000000000001e-05 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076856+0000", 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076863+0000", 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9840000000000004e-06 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855770+0000", 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.77890736999999999 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855902+0000", 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013207 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.856054+0000", 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015220899999999999 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.858407+0000", 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023530069999999998 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958598+0000", 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100190194 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958716+0000", 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118589 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958761+0000", 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5120000000000002e-05 2024-09-16T03:38:31.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958802+0000", 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0383999999999998e-05 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 96.552934551999996, 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70508991300000001, 2024-09-16T03:38:31.159 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 52 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6300000000000005e-07 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595187+0000", 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9790000000000001e-06 2024-09-16T03:38:31.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595194+0000", 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6490000000000002e-06 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595208+0000", 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4139e-05 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801931+0000", 2024-09-16T03:38:31.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.206723083 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801990+0000", 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8625999999999997e-05 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802066+0000", 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6553999999999999e-05 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802504+0000", 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043775200000000002 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.199998+0000", 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39749365599999997 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300151+0000", 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015310099999999 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300210+0000", 2024-09-16T03:38:31.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9376999999999997e-05 2024-09-16T03:38:31.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.165 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.165 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300271+0000", 2024-09-16T03:38:31.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1334000000000006e-05 2024-09-16T03:38:31.165 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.165 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.165 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.165 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.165 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:70 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 94.778064779999994, 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60616296300000005, 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 70 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370051+0000", 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370052+0000", 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8899999999999999e-07 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370055+0000", 2024-09-16T03:38:31.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7709999999999998e-06 2024-09-16T03:38:31.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370060+0000", 2024-09-16T03:38:31.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6910000000000002e-06 2024-09-16T03:38:31.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370064+0000", 2024-09-16T03:38:31.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.619e-06 2024-09-16T03:38:31.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.572988+0000", 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20292439300000001 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.573049+0000", 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0418999999999999e-05 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.573133+0000", 2024-09-16T03:38:31.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4324000000000001e-05 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674789+0000", 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101655658 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.875723+0000", 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20093406699999999 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976140+0000", 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10041718199999999 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976172+0000", 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2169000000000003e-05 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976214+0000", 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2080999999999999e-05 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.171 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.172 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.172 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.172 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.172 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:38:31.172 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 94.778043569000005, 2024-09-16T03:38:31.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60623559900000001, 2024-09-16T03:38:31.172 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.172 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.172 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.172 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.172 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.172 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.172 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 71 2024-09-16T03:38:31.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.173 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:38:31.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.173 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.173 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:38:31.173 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.173 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.173 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370077+0000", 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9940000000000002e-06 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370081+0000", 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5850000000000001e-06 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.174 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370087+0000", 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673522+0000", 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.303434969 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673567+0000", 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5080000000000002e-05 2024-09-16T03:38:31.175 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673651+0000", 2024-09-16T03:38:31.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3652999999999993e-05 2024-09-16T03:38:31.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775115+0000", 2024-09-16T03:38:31.176 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101464178 2024-09-16T03:38:31.176 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.176 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.176 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.176 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975918+0000", 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20080318699999999 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976254+0000", 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033619200000000002 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976285+0000", 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0679000000000002e-05 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.177 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976308+0000", 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3051e-05 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 94.778020209000005, 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60632925100000001, 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.178 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.179 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.179 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.179 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 72 2024-09-16T03:38:31.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.179 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:38:31.179 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.179 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.179 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.179 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.179 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:38:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:38:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:38:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370150+0000", 2024-09-16T03:38:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3906999999999997e-05 2024-09-16T03:38:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.180 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370157+0000", 2024-09-16T03:38:31.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:38:31.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370162+0000", 2024-09-16T03:38:31.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7249999999999997e-06 2024-09-16T03:38:31.181 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.181 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.181 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.181 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673969+0000", 2024-09-16T03:38:31.181 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30380713199999998 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674030+0000", 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1141000000000003e-05 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674124+0000", 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3859999999999999e-05 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775687+0000", 2024-09-16T03:38:31.182 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10156293299999999 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976103+0000", 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20041631400000001 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976355+0000", 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000251611 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.183 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976385+0000", 2024-09-16T03:38:31.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0883999999999998e-05 2024-09-16T03:38:31.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.184 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.184 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976425+0000", 2024-09-16T03:38:31.184 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9666000000000003e-05 2024-09-16T03:38:31.184 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.184 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.184 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.184 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.184 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.184 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.184 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 94.777947725000004, 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60763126999999995, 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 73 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.185 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370169+0000", 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6199999999999998e-07 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.186 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370171+0000", 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370175+0000", 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.596e-06 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470295+0000", 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100119967 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.187 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775148+0000", 2024-09-16T03:38:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30485311799999998 2024-09-16T03:38:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775203+0000", 2024-09-16T03:38:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4592999999999997e-05 2024-09-16T03:38:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775293+0000", 2024-09-16T03:38:31.188 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0859999999999994e-05 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976223+0000", 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20092996599999999 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976463+0000", 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000239695 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.189 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977730+0000", 2024-09-16T03:38:31.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012667690000000001 2024-09-16T03:38:31.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977773+0000", 2024-09-16T03:38:31.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2604000000000001e-05 2024-09-16T03:38:31.190 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.190 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.190 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.190 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977800+0000", 2024-09-16T03:38:31.190 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7137000000000001e-05 2024-09-16T03:38:31.190 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.190 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 94.677808870000007, 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60633810099999996, 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 74 2024-09-16T03:38:31.191 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.192 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:38:31.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.192 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:38:31.192 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.192 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.192 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.192 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470308+0000", 2024-09-16T03:38:31.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0950000000000001e-06 2024-09-16T03:38:31.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470313+0000", 2024-09-16T03:38:31.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:38:31.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.193 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.193 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.193 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470320+0000", 2024-09-16T03:38:31.193 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5089999999999999e-06 2024-09-16T03:38:31.193 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470327+0000", 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976483+0000", 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50615579499999996 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976533+0000", 2024-09-16T03:38:31.194 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0096000000000003e-05 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976643+0000", 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109481 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977808+0000", 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011649869999999999 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.195 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.981699+0000", 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003891579 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076496+0000", 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.094797333999999997 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076583+0000", 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6571999999999997e-05 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076645+0000", 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2331999999999999e-05 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 94.677778735000004, 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71251765099999997, 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.197 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 75 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.198 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470338+0000", 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2500000000000005e-07 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470389+0000", 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.134e-05 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.199 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470397+0000", 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8140000000000001e-06 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570537+0000", 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100139425 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977018+0000", 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40648102899999999 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.200 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977063+0000", 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5247000000000002e-05 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977143+0000", 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0054000000000003e-05 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.978132+0000", 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00098896399999999995 2024-09-16T03:38:31.201 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182725+0000", 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.204592626 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182783+0000", 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8285000000000002e-05 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182825+0000", 2024-09-16T03:38:31.202 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2398999999999998e-05 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182855+0000", 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9842999999999999e-05 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 94.577552167999997, 2024-09-16T03:38:31.203 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71263271800000005, 2024-09-16T03:38:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 76 2024-09-16T03:38:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:38:31.204 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570566+0000", 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570577+0000", 2024-09-16T03:38:31.205 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0672999999999999e-05 2024-09-16T03:38:31.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570593+0000", 2024-09-16T03:38:31.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5979999999999999e-05 2024-09-16T03:38:31.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.206 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570604+0000", 2024-09-16T03:38:31.206 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.077e-05 2024-09-16T03:38:31.206 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.206 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.206 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979890+0000", 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40928624200000002 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979945+0000", 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5651e-05 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980038+0000", 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2603999999999996e-05 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.207 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980669+0000", 2024-09-16T03:38:31.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063121900000000003 2024-09-16T03:38:31.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283050+0000", 2024-09-16T03:38:31.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30238120800000001 2024-09-16T03:38:31.208 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.208 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.208 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.208 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283132+0000", 2024-09-16T03:38:31.208 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1841999999999993e-05 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283168+0000", 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5438999999999997e-05 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283197+0000", 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8960000000000001e-05 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.209 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.210 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.210 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.210 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:38:31.210 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.763017069, 2024-09-16T03:38:31.210 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70799990400000001, 2024-09-16T03:38:31.210 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.210 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.210 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.210 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.210 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.210 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.210 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 94 2024-09-16T03:38:31.210 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.211 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:38:31.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.211 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:38:31.211 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.211 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.211 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.211 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385100+0000", 2024-09-16T03:38:31.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9599999999999998e-07 2024-09-16T03:38:31.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385104+0000", 2024-09-16T03:38:31.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7579999999999998e-06 2024-09-16T03:38:31.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385128+0000", 2024-09-16T03:38:31.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4700999999999999e-05 2024-09-16T03:38:31.212 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385139+0000", 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0587e-05 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787499+0000", 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40235976099999998 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787545+0000", 2024-09-16T03:38:31.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6418999999999997e-05 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787623+0000", 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7897000000000003e-05 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888605+0000", 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100982321 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990721+0000", 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102115922 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093019+0000", 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10229788500000001 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093051+0000", 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2255999999999997e-05 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093099+0000", 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7500999999999998e-05 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.762965746000006, 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80404594699999998, 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.216 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 95 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385151+0000", 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385165+0000", 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4175999999999999e-05 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385175+0000", 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385183+0000", 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888643+0000", 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50345988200000003 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888695+0000", 2024-09-16T03:38:31.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2834999999999998e-05 2024-09-16T03:38:31.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888802+0000", 2024-09-16T03:38:31.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010697899999999999 2024-09-16T03:38:31.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990658+0000", 2024-09-16T03:38:31.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185585699999999 2024-09-16T03:38:31.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990917+0000", 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025853600000000002 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189054+0000", 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198136812 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189135+0000", 2024-09-16T03:38:31.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1482000000000004e-05 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189196+0000", 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1175999999999998e-05 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.762925119000002, 2024-09-16T03:38:31.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80412488400000004, 2024-09-16T03:38:31.223 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.223 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.223 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.223 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.223 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 96 2024-09-16T03:38:31.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.223 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:38:31.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385192+0000", 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385267+0000", 2024-09-16T03:38:31.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5535000000000001e-05 2024-09-16T03:38:31.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385277+0000", 2024-09-16T03:38:31.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6239999999999997e-06 2024-09-16T03:38:31.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385284+0000", 2024-09-16T03:38:31.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4800000000000004e-06 2024-09-16T03:38:31.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989283+0000", 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60399849500000002 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989410+0000", 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127298 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989615+0000", 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000205045 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991036+0000", 2024-09-16T03:38:31.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001421042 2024-09-16T03:38:31.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189228+0000", 2024-09-16T03:38:31.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19819188500000001 2024-09-16T03:38:31.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189249+0000", 2024-09-16T03:38:31.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0996999999999999e-05 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189293+0000", 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3912000000000002e-05 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189316+0000", 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2739e-05 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.229 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.229 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:38:31.229 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.762821388000006, 2024-09-16T03:38:31.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80642264100000005, 2024-09-16T03:38:31.229 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.229 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.229 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.229 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.229 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.229 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.229 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 97 2024-09-16T03:38:31.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385296+0000", 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.006e-06 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385301+0000", 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2630000000000003e-06 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385308+0000", 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6030000000000001e-06 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485459+0000", 2024-09-16T03:38:31.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150929 2024-09-16T03:38:31.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990238+0000", 2024-09-16T03:38:31.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50477992000000005 2024-09-16T03:38:31.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990287+0000", 2024-09-16T03:38:31.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8863999999999998e-05 2024-09-16T03:38:31.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990377+0000", 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9900000000000003e-05 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093109+0000", 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102732055 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189351+0000", 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096241766000000006 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191570+0000", 2024-09-16T03:38:31.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00221915 2024-09-16T03:38:31.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191653+0000", 2024-09-16T03:38:31.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3046000000000004e-05 2024-09-16T03:38:31.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191717+0000", 2024-09-16T03:38:31.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4139000000000004e-05 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.662630930999995, 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90688294999999997, 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.235 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 98 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485487+0000", 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1799999999999999e-06 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485498+0000", 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0648999999999999e-05 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485514+0000", 2024-09-16T03:38:31.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5943999999999998e-05 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485525+0000", 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.065e-05 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991102+0000", 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50557748400000002 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991161+0000", 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8801999999999997e-05 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991255+0000", 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3936000000000003e-05 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093112+0000", 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185709499999999 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291911+0000", 2024-09-16T03:38:31.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19879897699999999 2024-09-16T03:38:31.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392167+0000", 2024-09-16T03:38:31.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10025634999999999 2024-09-16T03:38:31.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392249+0000", 2024-09-16T03:38:31.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2175000000000005e-05 2024-09-16T03:38:31.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392368+0000", 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118708 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.662577307999996, 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90707203199999997, 2024-09-16T03:38:31.241 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 99 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485540+0000", 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-06 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485661+0000", 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120835 2024-09-16T03:38:31.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485680+0000", 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8885999999999998e-05 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485693+0000", 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3297e-05 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991546+0000", 2024-09-16T03:38:31.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.505853098 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991598+0000", 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1921e-05 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991681+0000", 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2502000000000004e-05 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093115+0000", 2024-09-16T03:38:31.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10143424299999999 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291977+0000", 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198862172 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392541+0000", 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100563758 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392579+0000", 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7796999999999997e-05 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392611+0000", 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2073999999999998e-05 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:38:31.247 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.662398414999998, 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0073149939999999, 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 100 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:38:31.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7400000000000003e-07 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485721+0000", 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4530000000000001e-06 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485726+0000", 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0590000000000002e-06 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485729+0000", 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092342+0000", 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60661343700000003 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092469+0000", 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012658799999999999 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092671+0000", 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020206 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392452+0000", 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29978163699999999 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392486+0000", 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3736999999999999e-05 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492812+0000", 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100325621 2024-09-16T03:38:31.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492901+0000", 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8702000000000006e-05 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.493033+0000", 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000132185 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 90.487247894000006, 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60874581500000002, 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 123 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660869+0000", 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9400000000000005e-07 2024-09-16T03:38:31.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660921+0000", 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2312000000000003e-05 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660927+0000", 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9329999999999998e-06 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660931+0000", 2024-09-16T03:38:31.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.967991+0000", 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30706032700000002 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968039+0000", 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7477e-05 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968117+0000", 2024-09-16T03:38:31.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8589000000000003e-05 2024-09-16T03:38:31.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067678+0000", 2024-09-16T03:38:31.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099560406000000004 2024-09-16T03:38:31.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170175+0000", 2024-09-16T03:38:31.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102496927 2024-09-16T03:38:31.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269491+0000", 2024-09-16T03:38:31.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099316205000000005 2024-09-16T03:38:31.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269574+0000", 2024-09-16T03:38:31.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2903000000000002e-05 2024-09-16T03:38:31.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269614+0000", 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0188000000000002e-05 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 90.487179196, 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60879245500000001, 2024-09-16T03:38:31.262 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 124 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:38:31.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660938+0000", 2024-09-16T03:38:31.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:38:31.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660940+0000", 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2529999999999999e-06 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660943+0000", 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.01e-06 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660946+0000", 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8159999999999998e-06 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012417+0000", 2024-09-16T03:38:31.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.35147120599999998 2024-09-16T03:38:31.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012497+0000", 2024-09-16T03:38:31.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9963000000000007e-05 2024-09-16T03:38:31.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:31.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012585+0000", 2024-09-16T03:38:31.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067680+0000", 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055095203000000002 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170243+0000", 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10256290799999999 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269662+0000", 2024-09-16T03:38:31.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099418896000000007 2024-09-16T03:38:31.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269701+0000", 2024-09-16T03:38:31.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8829999999999999e-05 2024-09-16T03:38:31.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.268 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.268 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269729+0000", 2024-09-16T03:38:31.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8569e-05 2024-09-16T03:38:31.268 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.268 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.269 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.269 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1550 4.13 4:c8492bfc:::smithi15736696-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:38:31.269 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:38:31.269 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 45.481451055999997, 2024-09-16T03:38:31.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62706974999999998, 2024-09-16T03:38:31.269 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.269 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.269 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:31.269 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.269 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.269 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.270 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1550 2024-09-16T03:38:31.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.270 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:38:31.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:38:31.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0699999999999998e-07 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666669+0000", 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.393e-06 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666673+0000", 2024-09-16T03:38:31.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:38:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666682+0000", 2024-09-16T03:38:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6510000000000007e-06 2024-09-16T03:38:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.913940+0000", 2024-09-16T03:38:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.24725763000000001 2024-09-16T03:38:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.272 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:31.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.914034+0000", 2024-09-16T03:38:31.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3923000000000002e-05 2024-09-16T03:38:31.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293650+0000", 2024-09-16T03:38:31.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.37961662200000001 2024-09-16T03:38:31.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293688+0000", 2024-09-16T03:38:31.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8096000000000003e-05 2024-09-16T03:38:31.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293735+0000", 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6451e-05 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:77 3.2 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 31.069574435, 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.61452436499999996, 2024-09-16T03:38:31.274 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.275 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.275 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:38:31.275 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:38:31.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:38:31.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.276 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.276 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078543+0000", 2024-09-16T03:38:31.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.553e-06 2024-09-16T03:38:31.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.276 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.276 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078551+0000", 2024-09-16T03:38:31.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0790000000000001e-06 2024-09-16T03:38:31.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.276 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.276 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078564+0000", 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2943e-05 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178635+0000", 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10007044599999999 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178684+0000", 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9592999999999998e-05 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178747+0000", 2024-09-16T03:38:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2286000000000006e-05 2024-09-16T03:38:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4,5", 2024-09-16T03:38:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178857+0000", 2024-09-16T03:38:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110807 2024-09-16T03:38:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280364+0000", 2024-09-16T03:38:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101506518 2024-09-16T03:38:31.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280373+0000", 2024-09-16T03:38:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6899999999999998e-06 2024-09-16T03:38:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.369249+0000", 2024-09-16T03:38:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.088876172000000003 2024-09-16T03:38:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693044+0000", 2024-09-16T03:38:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.32379486499999999 2024-09-16T03:38:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693066+0000", 2024-09-16T03:38:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2413e-05 2024-09-16T03:38:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.280 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.280 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:31.282 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:31.280+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:31.282 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:38:31.283 INFO:tasks.rados.rados.0.smithi157.stdout:1330: finishing write tid 1 to smithi15736696-41 2024-09-16T03:38:31.283 INFO:tasks.rados.rados.0.smithi157.stdout:1332: finishing write tid 1 to smithi15736696-43 2024-09-16T03:38:31.283 INFO:tasks.rados.rados.0.smithi157.stdout:1332: finishing write tid 2 to smithi15736696-43 2024-09-16T03:38:31.283 INFO:tasks.rados.rados.0.smithi157.stdout:1335: expect (ObjNum 200 snap 0 seq_num 200) 2024-09-16T03:38:31.283 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1334411 2024-09-16T03:38:31.329 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3158474 2024-09-16T03:38:31.337 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:31.352 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:38:31.367 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15] out_osds: [1, 5, 9, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:38:31.367 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:38:31.367 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:38:31.367 INFO:tasks.thrashosds.thrasher:Adding osd 9 2024-09-16T03:38:31.367 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 9 2024-09-16T03:38:31.383 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:31.382+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:31.409 INFO:tasks.rados.rados.0.smithi157.stdout:1337: expect (ObjNum 245 snap 0 seq_num 245) 2024-09-16T03:38:31.409 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:31.428 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:31.429 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:31.429 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:38:31.429 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:31.437 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:38:31.455 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 550723 2024-09-16T03:38:31.484 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:31.482+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:31.498 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1144062 2024-09-16T03:38:31.522 INFO:tasks.rados.rados.0.smithi157.stdout:1328: finishing write tid 1 to smithi15736696-47 2024-09-16T03:38:31.522 INFO:tasks.rados.rados.0.smithi157.stdout:1328: finishing write tid 2 to smithi15736696-47 2024-09-16T03:38:31.522 INFO:tasks.rados.rados.0.smithi157.stdout:1328: finishing write tid 3 to smithi15736696-47 2024-09-16T03:38:31.522 INFO:tasks.rados.rados.0.smithi157.stdout:1328: finishing write tid 4 to smithi15736696-47 2024-09-16T03:38:31.522 INFO:tasks.rados.rados.0.smithi157.stdout:1328: oid 47 updating version 0 to 99 2024-09-16T03:38:31.522 INFO:tasks.rados.rados.0.smithi157.stdout:1328: oid 47 updating version 99 to 100 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1328: oid 47 updating version 100 to 101 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 101 (ObjNum 262 snap 0 seq_num 262) dirty exists 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1328: left oid 47 (ObjNum 262 snap 0 seq_num 262) 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1322: done (15 left) 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1325: done (14 left) 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1326: done (13 left) 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1327: done (12 left) 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1328: done (11 left) 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1329: done (10 left) 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1333: done (9 left) 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1335: done (8 left) 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1337: done (7 left) 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1338: done (6 left) 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1340: read oid 48 snap -1 2024-09-16T03:38:31.523 INFO:tasks.rados.rados.0.smithi157.stdout:1340: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-16T03:38:31.524 INFO:tasks.rados.rados.0.smithi157.stdout:1341: set_chunk oid 2 target oid 2024-09-16T03:38:31.524 INFO:tasks.rados.rados.0.smithi157.stdout:1341: set_chunk oid 2 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:31.524 INFO:tasks.rados.rados.0.smithi157.stdout:1342: write (excl) oid 50 current snap is 0 2024-09-16T03:38:31.524 INFO:tasks.rados.rados.0.smithi157.stdout:1331: expect (ObjNum 226 snap 0 seq_num 226) 2024-09-16T03:38:31.560 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:31.561 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:31.561 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:38:31.561 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:38:31.561 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:31.569 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:38:31.583 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:31.585 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:31.584+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:31.677 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:31.677 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:64 4.5 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 79.417281527, 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50484290399999998, 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.678 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 64 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.680 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257341+0000", 2024-09-16T03:38:31.680 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:38:31.680 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.680 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.680 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.680 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257343+0000", 2024-09-16T03:38:31.680 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3889999999999999e-06 2024-09-16T03:38:31.680 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.680 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.680 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.680 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257346+0000", 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257349+0000", 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8559999999999998e-06 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357057+0000", 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099708121999999996 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.681 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:31.682 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357156+0000", 2024-09-16T03:38:31.682 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9162999999999999e-05 2024-09-16T03:38:31.682 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.682 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.682 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.682 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762060+0000", 2024-09-16T03:38:31.682 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40490364400000001 2024-09-16T03:38:31.682 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.682 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.682 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.682 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762112+0000", 2024-09-16T03:38:31.682 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1882000000000002e-05 2024-09-16T03:38:31.682 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.683 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.683 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.683 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762183+0000", 2024-09-16T03:38:31.683 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1297000000000005e-05 2024-09-16T03:38:31.683 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.683 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.683 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.683 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.683 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.683 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.10.22:22 3.3 3:f84e13a5:::smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:31.683 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:38:31.683 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 71.365134291999993, 2024-09-16T03:38:31.684 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50224271099999995, 2024-09-16T03:38:31.684 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.684 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.684 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.684 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.684 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:38:31.684 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.684 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:38:31.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.685 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309489+0000", 2024-09-16T03:38:31.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4309999999999999e-06 2024-09-16T03:38:31.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.685 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309496+0000", 2024-09-16T03:38:31.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5810000000000002e-06 2024-09-16T03:38:31.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309505+0000", 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6889999999999999e-06 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309513+0000", 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3749999999999997e-06 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.406932+0000", 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.097419631000000007 2024-09-16T03:38:31.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407009+0000", 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7220999999999998e-05 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407143+0000", 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013323799999999999 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407732+0000", 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058938399999999998 2024-09-16T03:38:31.687 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.688 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.688 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.689 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.408133+0000", 2024-09-16T03:38:31.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040090500000000001 2024-09-16T03:38:31.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.689 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.689 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.689 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811599+0000", 2024-09-16T03:38:31.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40346633399999998 2024-09-16T03:38:31.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.689 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.689 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.689 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811682+0000", 2024-09-16T03:38:31.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2463e-05 2024-09-16T03:38:31.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811730+0000", 2024-09-16T03:38:31.690 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8458999999999998e-05 2024-09-16T03:38:31.690 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.690 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.690 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.690 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:465 4.5 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:31.690 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:38:31.690 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.658916754000003, 2024-09-16T03:38:31.690 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60529649299999999, 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 465 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.692 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.692 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:38:31.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.692 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.692 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015706+0000", 2024-09-16T03:38:31.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.091e-06 2024-09-16T03:38:31.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015712+0000", 2024-09-16T03:38:31.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-16T03:38:31.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015724+0000", 2024-09-16T03:38:31.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1408e-05 2024-09-16T03:38:31.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015731+0000", 2024-09-16T03:38:31.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:38:31.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216049+0000", 2024-09-16T03:38:31.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200317895 2024-09-16T03:38:31.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216178+0000", 2024-09-16T03:38:31.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012848800000000001 2024-09-16T03:38:31.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:31.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620010+0000", 2024-09-16T03:38:31.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40383207199999999 2024-09-16T03:38:31.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620710+0000", 2024-09-16T03:38:31.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069988400000000001 2024-09-16T03:38:31.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620902+0000", 2024-09-16T03:38:31.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019197800000000001 2024-09-16T03:38:31.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620933+0000", 2024-09-16T03:38:31.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1513999999999997e-05 2024-09-16T03:38:31.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.696 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.696 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620968+0000", 2024-09-16T03:38:31.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4524999999999999e-05 2024-09-16T03:38:31.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.621002+0000", 2024-09-16T03:38:31.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3955000000000003e-05 2024-09-16T03:38:31.697 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.697 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.697 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.697 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:10 3.3 3:deac805c:::smithi15736696-44 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:31.697 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:38:31.697 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 65.267421777999999, 2024-09-16T03:38:31.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53191748100000003, 2024-09-16T03:38:31.697 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407202+0000", 2024-09-16T03:38:31.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.855e-06 2024-09-16T03:38:31.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407208+0000", 2024-09-16T03:38:31.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9819999999999996e-06 2024-09-16T03:38:31.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407220+0000", 2024-09-16T03:38:31.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2191999999999999e-05 2024-09-16T03:38:31.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407236+0000", 2024-09-16T03:38:31.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5698000000000001e-05 2024-09-16T03:38:31.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538167+0000", 2024-09-16T03:38:31.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.13093111499999999 2024-09-16T03:38:31.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.701 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538229+0000", 2024-09-16T03:38:31.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2441999999999996e-05 2024-09-16T03:38:31.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.701 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:38:31.701 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538346+0000", 2024-09-16T03:38:31.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011716400000000001 2024-09-16T03:38:31.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.701 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.701 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540757+0000", 2024-09-16T03:38:31.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024105089999999999 2024-09-16T03:38:31.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.639501+0000", 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098743545000000002 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939038+0000", 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29953765700000001 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939075+0000", 2024-09-16T03:38:31.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6600999999999997e-05 2024-09-16T03:38:31.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939118+0000", 2024-09-16T03:38:31.703 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2721e-05 2024-09-16T03:38:31.703 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.703 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.703 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.703 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:984 4.15 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:31.703 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:38:31.703 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 61.302501849999999, 2024-09-16T03:38:31.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.97836159599999994, 2024-09-16T03:38:31.704 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.704 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.704 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:31.704 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.704 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.704 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.704 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 984 2024-09-16T03:38:31.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.704 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372122+0000", 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.934e-06 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372129+0000", 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8029999999999999e-06 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372141+0000", 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2006e-05 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472234+0000", 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100092985 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472318+0000", 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4257000000000003e-05 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472326+0000", 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8960000000000003e-06 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350293+0000", 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.87796699899999997 2024-09-16T03:38:31.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.708 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.708 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350433+0000", 2024-09-16T03:38:31.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013999600000000001 2024-09-16T03:38:31.708 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.708 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.708 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350482+0000", 2024-09-16T03:38:31.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8720000000000001e-05 2024-09-16T03:38:31.708 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.708 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.708 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.708 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1000 4.15 4:aff4cc41:::smithi15736696-11:head [read 0~2873964,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 61.084062764999999, 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.761707841, 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1000 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.709 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590560+0000", 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:38:31.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590562+0000", 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.17e-06 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590568+0000", 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7939999999999997e-06 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590572+0000", 2024-09-16T03:38:31.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1300000000000001e-06 2024-09-16T03:38:31.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.712 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590619+0000", 2024-09-16T03:38:31.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7148999999999999e-05 2024-09-16T03:38:31.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:31.712 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590625+0000", 2024-09-16T03:38:31.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5679999999999996e-06 2024-09-16T03:38:31.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.713 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350858+0000", 2024-09-16T03:38:31.713 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76023295000000002 2024-09-16T03:38:31.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.713 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.713 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350973+0000", 2024-09-16T03:38:31.713 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115023 2024-09-16T03:38:31.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.713 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352267+0000", 2024-09-16T03:38:31.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012935970000000001 2024-09-16T03:38:31.714 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.714 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.714 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.714 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1001 4.15 4:aff4cc41:::smithi15736696-11:head [read 1254674~1619290,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:31.714 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:38:31.714 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 61.084044179999999, 2024-09-16T03:38:31.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76275827799999996, 2024-09-16T03:38:31.715 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.715 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.715 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:31.715 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.715 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.715 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.715 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1001 2024-09-16T03:38:31.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.715 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590580+0000", 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590584+0000", 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8800000000000001e-06 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590587+0000", 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6630000000000002e-06 2024-09-16T03:38:31.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590650+0000", 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3950999999999998e-05 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590654+0000", 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.89e-06 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352309+0000", 2024-09-16T03:38:31.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76165495699999997 2024-09-16T03:38:31.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352407+0000", 2024-09-16T03:38:31.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7188999999999994e-05 2024-09-16T03:38:31.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.353336+0000", 2024-09-16T03:38:31.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092939400000000005 2024-09-16T03:38:31.719 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.719 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.719 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.720 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1002 4.15 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~1527025] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:31.720 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:38:31.720 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 61.084014926000002, 2024-09-16T03:38:31.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85963523600000002, 2024-09-16T03:38:31.720 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.720 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.720 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:31.720 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.720 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.720 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.721 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1002 2024-09-16T03:38:31.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.721 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:38:31.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:38:31.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.15e-07 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590609+0000", 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.606e-06 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590615+0000", 2024-09-16T03:38:31.722 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3040000000000001e-06 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590618+0000", 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.309e-06 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590669+0000", 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1048000000000003e-05 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:31.723 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590673+0000", 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.061e-06 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.449932+0000", 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85925979699999999 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450032+0000", 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9858000000000005e-05 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.724 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.725 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450242+0000", 2024-09-16T03:38:31.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020993800000000001 2024-09-16T03:38:31.725 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.725 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.725 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.725 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.725 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.725 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2467 4.5 4:a11e1d7c:::smithi15736696-43:head [write 1772395~538068] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:31.725 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:38:31.725 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.683338555000001, 2024-09-16T03:38:31.725 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.56288137400000005, 2024-09-16T03:38:31.725 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.725 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2467 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.726 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991285+0000", 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0990000000000001e-06 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991447+0000", 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016170099999999999 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.727 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991458+0000", 2024-09-16T03:38:31.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.128e-05 2024-09-16T03:38:31.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991464+0000", 2024-09-16T03:38:31.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3800000000000002e-06 2024-09-16T03:38:31.728 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.728 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.728 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.728 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150823+0000", 2024-09-16T03:38:31.728 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15935943799999999 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150899+0000", 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5359999999999997e-05 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151007+0000", 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108317 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.729 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253992+0000", 2024-09-16T03:38:31.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102985249 2024-09-16T03:38:31.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.730 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.730 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.730 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.353797+0000", 2024-09-16T03:38:31.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099805039999999998 2024-09-16T03:38:31.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.730 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.730 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.730 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554039+0000", 2024-09-16T03:38:31.730 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20024175899999999 2024-09-16T03:38:31.730 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.730 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554129+0000", 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0067000000000004e-05 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554165+0000", 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5683999999999999e-05 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.731 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2468 4.5 4:a11e1d7c:::smithi15736696-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:31.732 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:38:31.732 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.677917775999999, 2024-09-16T03:38:31.732 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.55765803899999999, 2024-09-16T03:38:31.732 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.732 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.732 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.732 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.732 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.732 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.732 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2468 2024-09-16T03:38:31.732 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.732 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.732 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996708+0000", 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5149999999999998e-06 2024-09-16T03:38:31.733 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996921+0000", 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000213692 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996937+0000", 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5699999999999999e-05 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996946+0000", 2024-09-16T03:38:31.734 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7159999999999992e-06 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152362+0000", 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15541682800000001 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152421+0000", 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:31.735 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152515+0000", 2024-09-16T03:38:31.736 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3659999999999994e-05 2024-09-16T03:38:31.736 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.736 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.736 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.736 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253995+0000", 2024-09-16T03:38:31.736 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10148022499999999 2024-09-16T03:38:31.736 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.736 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.736 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.736 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.453908+0000", 2024-09-16T03:38:31.736 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19991299700000001 2024-09-16T03:38:31.736 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.736 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554321+0000", 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100412977 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554344+0000", 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.262e-05 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554362+0000", 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8539999999999999e-05 2024-09-16T03:38:31.737 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.738 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.738 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.738 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.738 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.738 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2469 4.5 4:a11e1d7c:::smithi15736696-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:31.738 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:38:31.738 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.677662413, 2024-09-16T03:38:31.738 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.75785074799999996, 2024-09-16T03:38:31.738 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.738 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.738 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.738 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.738 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2469 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.739 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.740 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.740 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.740 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996960+0000", 2024-09-16T03:38:31.740 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3000000000000001e-07 2024-09-16T03:38:31.740 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.740 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.740 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.740 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996963+0000", 2024-09-16T03:38:31.740 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8679999999999999e-06 2024-09-16T03:38:31.740 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.740 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.740 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.740 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996967+0000", 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.622e-06 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996971+0000", 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1429999999999998e-06 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253536+0000", 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25656582900000002 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.741 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253632+0000", 2024-09-16T03:38:31.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6067e-05 2024-09-16T03:38:31.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:31.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253745+0000", 2024-09-16T03:38:31.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011256699999999999 2024-09-16T03:38:31.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.742 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.742 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.742 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254218+0000", 2024-09-16T03:38:31.742 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00047321400000000001 2024-09-16T03:38:31.742 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.743 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.743 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.743 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554397+0000", 2024-09-16T03:38:31.743 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30017854399999999 2024-09-16T03:38:31.743 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.743 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.743 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.743 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754557+0000", 2024-09-16T03:38:31.743 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200159848 2024-09-16T03:38:31.743 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.743 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.743 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.743 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754646+0000", 2024-09-16T03:38:31.744 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9370999999999995e-05 2024-09-16T03:38:31.744 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.744 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.744 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.744 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754810+0000", 2024-09-16T03:38:31.744 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016414500000000001 2024-09-16T03:38:31.744 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.744 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.744 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.744 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.744 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.744 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2470 4.5 4:a11e1d7c:::smithi15736696-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.677646083999999, 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76022711799999998, 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2470 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.745 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.746 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:38:31.746 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.746 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.746 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.746 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.746 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:38:31.746 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.746 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.746 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.746 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.746 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:38:31.746 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.65e-07 2024-09-16T03:38:31.746 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996977+0000", 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4079999999999999e-06 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996981+0000", 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996988+0000", 2024-09-16T03:38:31.747 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0890000000000002e-06 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253947+0000", 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25695931100000002 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253985+0000", 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7939999999999999e-05 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.748 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757090+0000", 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50310477499999995 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757140+0000", 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9567999999999997e-05 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757203+0000", 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3215999999999999e-05 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.749 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.750 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.750 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.750 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2487 4.15 4:ad0b9852:::smithi15736696-16:head [write 1947841~685395] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:31.750 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:38:31.750 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.258601494000001, 2024-09-16T03:38:31.750 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.74140762199999999, 2024-09-16T03:38:31.750 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.750 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.750 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.750 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.750 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.750 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.750 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2487 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.751 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.752 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416023+0000", 2024-09-16T03:38:31.752 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:38:31.752 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.752 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.752 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.752 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416341+0000", 2024-09-16T03:38:31.752 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031881699999999999 2024-09-16T03:38:31.752 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.752 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.752 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.752 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416353+0000", 2024-09-16T03:38:31.752 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.189e-05 2024-09-16T03:38:31.752 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.753 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.753 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.753 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416360+0000", 2024-09-16T03:38:31.753 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5719999999999999e-06 2024-09-16T03:38:31.753 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.753 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.753 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.753 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755578+0000", 2024-09-16T03:38:31.753 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33921859199999999 2024-09-16T03:38:31.753 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.753 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.753 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.753 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755639+0000", 2024-09-16T03:38:31.754 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0615000000000002e-05 2024-09-16T03:38:31.754 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.754 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.754 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:31.754 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755735+0000", 2024-09-16T03:38:31.754 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5798999999999995e-05 2024-09-16T03:38:31.754 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.754 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.754 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.754 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858633+0000", 2024-09-16T03:38:31.754 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10289859599999999 2024-09-16T03:38:31.754 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.755 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858695+0000", 2024-09-16T03:38:31.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2000000000000003e-05 2024-09-16T03:38:31.755 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.755 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157315+0000", 2024-09-16T03:38:31.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29861931600000002 2024-09-16T03:38:31.755 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157375+0000", 2024-09-16T03:38:31.756 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0268e-05 2024-09-16T03:38:31.756 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157428+0000", 2024-09-16T03:38:31.756 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2939999999999998e-05 2024-09-16T03:38:31.756 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.756 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.756 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.756 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.757 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.757 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2488 4.15 4:ad0b9852:::smithi15736696-16:head [write 3383215~196676] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:31.757 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:38:31.757 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.252928843999999, 2024-09-16T03:38:31.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73589615900000005, 2024-09-16T03:38:31.757 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.757 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.757 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.757 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.757 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.757 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.757 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2488 2024-09-16T03:38:31.757 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.758 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.758 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.758 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.758 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:38:31.758 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.758 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.758 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.758 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.758 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:38:31.758 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.758 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.758 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421695+0000", 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1469999999999999e-06 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421975+0000", 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028008299999999998 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421987+0000", 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1294000000000001e-05 2024-09-16T03:38:31.759 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.760 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.760 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421992+0000", 2024-09-16T03:38:31.760 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9349999999999999e-06 2024-09-16T03:38:31.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.760 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.760 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757205+0000", 2024-09-16T03:38:31.760 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33521242800000001 2024-09-16T03:38:31.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.760 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.760 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757269+0000", 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3761000000000002e-05 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757376+0000", 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010706599999999999 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858684+0000", 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101308255 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.761 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858700+0000", 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6138999999999999e-05 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157491+0000", 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29879056999999998 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157546+0000", 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5668999999999998e-05 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.763 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157589+0000", 2024-09-16T03:38:31.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2812000000000003e-05 2024-09-16T03:38:31.763 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.763 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.763 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.763 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2489 4.15 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=7b,truncate 3579891] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:31.763 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:38:31.763 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.252620022, 2024-09-16T03:38:31.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73575981700000004, 2024-09-16T03:38:31.763 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.764 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.764 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.764 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.764 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.764 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.764 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2489 2024-09-16T03:38:31.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.764 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.764 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.764 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:38:31.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:38:31.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:38:31.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0200000000000002e-07 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422005+0000", 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7379999999999999e-06 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422009+0000", 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6219999999999998e-06 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422016+0000", 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5789999999999998e-06 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858071+0000", 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.436054302 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858130+0000", 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9493000000000001e-05 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.767 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:31.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858226+0000", 2024-09-16T03:38:31.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6174000000000004e-05 2024-09-16T03:38:31.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:38:31.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10106483199999999 2024-09-16T03:38:31.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959639+0000", 2024-09-16T03:38:31.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034802999999999999 2024-09-16T03:38:31.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157640+0000", 2024-09-16T03:38:31.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19800045299999999 2024-09-16T03:38:31.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157702+0000", 2024-09-16T03:38:31.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2218000000000006e-05 2024-09-16T03:38:31.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157762+0000", 2024-09-16T03:38:31.770 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9774000000000001e-05 2024-09-16T03:38:31.770 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.770 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.770 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.770 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.770 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.770 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2490 4.15 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:31.770 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:38:31.770 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.25260143, 2024-09-16T03:38:31.770 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73597824999999994, 2024-09-16T03:38:31.770 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.770 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.770 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:31.771 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.771 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.771 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.771 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2490 2024-09-16T03:38:31.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.771 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.771 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.771 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:38:31.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.771 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.772 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:38:31.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.772 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.772 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422021+0000", 2024-09-16T03:38:31.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8099999999999999e-07 2024-09-16T03:38:31.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.772 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.772 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422022+0000", 2024-09-16T03:38:31.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.573e-06 2024-09-16T03:38:31.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422027+0000", 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3250000000000001e-06 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422030+0000", 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.948e-06 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858510+0000", 2024-09-16T03:38:31.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.43648032599999997 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858588+0000", 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7888999999999999e-05 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157803+0000", 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.299215642 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.774 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157893+0000", 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9565000000000001e-05 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157999+0000", 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105701 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2492 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:38:31.775 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.252588920999999, 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53721099400000005, 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2492 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:38:31.776 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:38:31.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:38:31.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:38:31.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422036+0000", 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5900000000000002e-06 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422043+0000", 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3069999999999997e-06 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422048+0000", 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6619999999999996e-06 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958774+0000", 2024-09-16T03:38:31.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53672593499999999 2024-09-16T03:38:31.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958868+0000", 2024-09-16T03:38:31.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4408999999999997e-05 2024-09-16T03:38:31.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959244+0000", 2024-09-16T03:38:31.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037581099999999998 2024-09-16T03:38:31.780 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.780 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.780 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.780 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.780 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.780 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2493 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:31.780 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:38:31.780 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.252569142, 2024-09-16T03:38:31.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53757571299999996, 2024-09-16T03:38:31.780 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.780 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.780 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:31.780 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.781 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2493 2024-09-16T03:38:31.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.781 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.781 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.781 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.781 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:38:31.781 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.781 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.781 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.781 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3600000000000004e-07 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422055+0000", 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7460000000000001e-06 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422058+0000", 2024-09-16T03:38:31.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:38:31.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422061+0000", 2024-09-16T03:38:31.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6759999999999999e-06 2024-09-16T03:38:31.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:38:31.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53723057900000004 2024-09-16T03:38:31.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959367+0000", 2024-09-16T03:38:31.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5158000000000001e-05 2024-09-16T03:38:31.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959628+0000", 2024-09-16T03:38:31.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026175800000000001 2024-09-16T03:38:31.784 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.784 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.784 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.785 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2494 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 1738052~782998] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:31.785 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:38:31.785 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.252558111999999, 2024-09-16T03:38:31.785 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53787640199999998, 2024-09-16T03:38:31.785 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.785 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.785 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:31.785 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.785 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2494 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:38:31.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6100000000000002e-07 2024-09-16T03:38:31.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422065+0000", 2024-09-16T03:38:31.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2839999999999999e-06 2024-09-16T03:38:31.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422069+0000", 2024-09-16T03:38:31.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7239999999999998e-06 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422072+0000", 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6560000000000001e-06 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959661+0000", 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53758982600000005 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959739+0000", 2024-09-16T03:38:31.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7581000000000001e-05 2024-09-16T03:38:31.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959940+0000", 2024-09-16T03:38:31.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020107 2024-09-16T03:38:31.789 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.789 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.789 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.789 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2623 4.5 4:a4fbca73:::smithi15736696-39:head [write 631996~531009] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:38:31.789 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.248471+0000", 2024-09-16T03:38:31.790 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 11.426150867, 2024-09-16T03:38:31.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50345695400000001, 2024-09-16T03:38:31.790 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:31.790 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:31.790 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:31.790 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:31.790 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:31.790 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:31.790 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2623 2024-09-16T03:38:31.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.790 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:31.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:31.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.248471+0000", 2024-09-16T03:38:31.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:31.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.248471+0000", 2024-09-16T03:38:31.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:31.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:31.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.248473+0000", 2024-09-16T03:38:31.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.609e-06 2024-09-16T03:38:31.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:31.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.248587+0000", 2024-09-16T03:38:31.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114089 2024-09-16T03:38:31.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:31.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.248600+0000", 2024-09-16T03:38:31.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3061000000000001e-05 2024-09-16T03:38:31.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:31.793 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.348733+0000", 2024-09-16T03:38:31.793 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100133028 2024-09-16T03:38:31.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.793 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.793 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:31.793 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.348844+0000", 2024-09-16T03:38:31.793 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011157 2024-09-16T03:38:31.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.793 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.793 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:31.793 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.348913+0000", 2024-09-16T03:38:31.793 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8554999999999997e-05 2024-09-16T03:38:31.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.794 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:31.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349017+0000", 2024-09-16T03:38:31.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010454300000000001 2024-09-16T03:38:31.794 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.794 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:31.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.352857+0000", 2024-09-16T03:38:31.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0038398460000000001 2024-09-16T03:38:31.794 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.794 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.454106+0000", 2024-09-16T03:38:31.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10124849399999999 2024-09-16T03:38:31.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:31.795 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.751825+0000", 2024-09-16T03:38:31.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29771889000000001 2024-09-16T03:38:31.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:31.795 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.751882+0000", 2024-09-16T03:38:31.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7524e-05 2024-09-16T03:38:31.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:31.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:31.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:31.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.751928+0000", 2024-09-16T03:38:31.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5745000000000003e-05 2024-09-16T03:38:31.796 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.796 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.796 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.796 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:31.796 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:31.796 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:31.797 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-09-16T03:38:31.717+0000 7fdb04aef640 -1 mon.c@2(leader).osd e99 definitely_dead 0 2024-09-16T03:38:31.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:38:31.799 INFO:tasks.rados.rados.0.smithi157.stdout:1336: expect (ObjNum 257 snap 0 seq_num 257) 2024-09-16T03:38:31.799 INFO:tasks.rados.rados.0.smithi157.stdout:1332: finishing write tid 3 to smithi15736696-43 2024-09-16T03:38:31.799 INFO:tasks.rados.rados.0.smithi157.stdout:1332: finishing write tid 4 to smithi15736696-43 2024-09-16T03:38:31.799 INFO:tasks.rados.rados.0.smithi157.stdout:1332: oid 43 updating version 0 to 73 2024-09-16T03:38:31.799 INFO:tasks.rados.rados.0.smithi157.stdout:1332: oid 43 updating version 73 to 74 2024-09-16T03:38:31.799 INFO:tasks.rados.rados.0.smithi157.stdout:1332: oid 43 updating version 74 to 75 2024-09-16T03:38:31.800 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 75 (ObjNum 264 snap 0 seq_num 264) dirty exists 2024-09-16T03:38:31.800 INFO:tasks.rados.rados.0.smithi157.stdout:1332: left oid 43 (ObjNum 264 snap 0 seq_num 264) 2024-09-16T03:38:31.800 INFO:tasks.rados.rados.0.smithi157.stdout:1330: finishing write tid 2 to smithi15736696-41 2024-09-16T03:38:31.800 INFO:tasks.rados.rados.0.smithi157.stdout:1342: seq_num 266 ranges {470662=772236,1832916=408094} 2024-09-16T03:38:31.800 INFO:tasks.rados.rados.0.smithi157.stdout:1342: writing smithi15736696-50 from 470662 to 1242898 tid 1 2024-09-16T03:38:31.800 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:31.796+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:31.801 INFO:tasks.rados.rados.0.smithi157.stdout:1342: writing smithi15736696-50 from 1832916 to 2241010 tid 2 2024-09-16T03:38:31.801 INFO:tasks.rados.rados.0.smithi157.stdout:1331: done (8 left) 2024-09-16T03:38:31.801 INFO:tasks.rados.rados.0.smithi157.stdout:1332: done (7 left) 2024-09-16T03:38:31.801 INFO:tasks.rados.rados.0.smithi157.stdout:1336: done (6 left) 2024-09-16T03:38:31.802 INFO:tasks.rados.rados.0.smithi157.stdout:1343: read oid 49 snap -1 2024-09-16T03:38:31.802 INFO:tasks.rados.rados.0.smithi157.stdout:1343: expect (ObjNum 254 snap 0 seq_num 254) 2024-09-16T03:38:31.802 INFO:tasks.rados.rados.0.smithi157.stdout:1344: read oid 25 snap -1 2024-09-16T03:38:31.802 INFO:tasks.rados.rados.0.smithi157.stdout:1344: expect (ObjNum 216 snap 0 seq_num 216) 2024-09-16T03:38:31.802 INFO:tasks.rados.rados.0.smithi157.stdout:1345: write oid 36 current snap is 0 2024-09-16T03:38:31.802 INFO:tasks.rados.rados.0.smithi157.stdout:1339: expect (ObjNum 248 snap 0 seq_num 248) 2024-09-16T03:38:31.837 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:31.899 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:31.897+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:31.907 INFO:tasks.rados.rados.0.smithi157.stdout:1334: finishing write tid 1 to smithi15736696-35 2024-09-16T03:38:31.908 INFO:tasks.rados.rados.0.smithi157.stdout:1334: finishing write tid 2 to smithi15736696-35 2024-09-16T03:38:31.908 INFO:tasks.rados.rados.0.smithi157.stdout:1345: seq_num 267 ranges {487469=563838,1607751=551752} 2024-09-16T03:38:31.908 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:31.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.10.22:196 3.2 e99/74)", 2024-09-16T03:38:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.903927+0000", 2024-09-16T03:38:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0029288529999999999, 2024-09-16T03:38:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029425269999999999, 2024-09-16T03:38:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "no flag points reached", 2024-09-16T03:38:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.903927+0000", 2024-09-16T03:38:31.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.903927+0000", 2024-09-16T03:38:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.903929+0000", 2024-09-16T03:38:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-16T03:38:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:31.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.903937+0000", 2024-09-16T03:38:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0590000000000003e-06 2024-09-16T03:38:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.903951+0000", 2024-09-16T03:38:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3298e-05 2024-09-16T03:38:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:38:31.911 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-09-16T03:38:31.912 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:31.913 INFO:tasks.rados.rados.0.smithi157.stdout:1345: writing smithi15736696-36 from 487469 to 1051307 tid 1 2024-09-16T03:38:31.917 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:38:31.921 INFO:tasks.rados.rados.0.smithi157.stdout:1345: writing smithi15736696-36 from 1607751 to 2159503 tid 2 2024-09-16T03:38:31.921 INFO:tasks.rados.rados.0.smithi157.stdout:1339: done (8 left) 2024-09-16T03:38:31.921 INFO:tasks.rados.rados.0.smithi157.stdout:1346: tier_evict oid 40 2024-09-16T03:38:31.921 INFO:tasks.rados.rados.0.smithi157.stdout:1346: tier_evict oid 40 snap -1 2024-09-16T03:38:31.921 INFO:tasks.rados.rados.0.smithi157.stdout:1334: finishing write tid 3 to smithi15736696-35 2024-09-16T03:38:31.921 INFO:tasks.rados.rados.0.smithi157.stdout:1347: read oid 23 snap -1 2024-09-16T03:38:31.921 INFO:tasks.rados.rados.0.smithi157.stdout:1347: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:38:31.921 INFO:tasks.rados.rados.0.smithi157.stdout:1348: read oid 24 snap -1 2024-09-16T03:38:31.921 INFO:tasks.rados.rados.0.smithi157.stdout:1348: expect (ObjNum 238 snap 0 seq_num 238) 2024-09-16T03:38:31.921 INFO:tasks.rados.rados.0.smithi157.stdout:1349: read oid 16 snap -1 2024-09-16T03:38:31.921 INFO:tasks.rados.rados.0.smithi157.stdout:1349: expect (ObjNum 224 snap 0 seq_num 224) 2024-09-16T03:38:31.921 INFO:tasks.rados.rados.0.smithi157.stdout:1350: tier_promote oid 33 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:1334: finishing write tid 4 to smithi15736696-35 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:1334: oid 35 updating version 0 to 102 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:1334: oid 35 updating version 102 to 104 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:1334: oid 35 version 104 is already newer than 103 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 104 (ObjNum 265 snap 0 seq_num 265) dirty exists 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:1334: left oid 35 (ObjNum 265 snap 0 seq_num 265) 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:1330: finishing write tid 3 to smithi15736696-41 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:1330: finishing write tid 4 to smithi15736696-41 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:1330: oid 41 updating version 0 to 42 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:1330: oid 41 updating version 42 to 44 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:1330: oid 41 version 44 is already newer than 43 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 44 (ObjNum 263 snap 0 seq_num 263) dirty exists 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:1330: left oid 41 (ObjNum 263 snap 0 seq_num 263) 2024-09-16T03:38:31.922 INFO:tasks.rados.rados.0.smithi157.stdout:1341: finishing set_chunk to oid 2 2024-09-16T03:38:31.923 INFO:tasks.rados.rados.0.smithi157.stdout:1341: got expected ENOENT (src dne) 2024-09-16T03:38:31.923 INFO:tasks.rados.rados.0.smithi157.stdout:1330: done (12 left) 2024-09-16T03:38:31.923 INFO:tasks.rados.rados.0.smithi157.stdout:1334: done (11 left) 2024-09-16T03:38:31.923 INFO:tasks.rados.rados.0.smithi157.stdout:1341: done (10 left) 2024-09-16T03:38:31.923 INFO:tasks.rados.rados.0.smithi157.stdout:1351: read oid 29 snap -1 2024-09-16T03:38:31.923 INFO:tasks.rados.rados.0.smithi157.stdout:1351: expect (ObjNum 246 snap 0 seq_num 246) 2024-09-16T03:38:31.923 INFO:tasks.rados.rados.0.smithi157.stdout:1352: delete oid 45 current snap is 0 2024-09-16T03:38:31.923 INFO:tasks.rados.rados.0.smithi157.stdout:1340: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-16T03:38:31.983 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2816890 2024-09-16T03:38:32.000 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:31.998+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:32.005 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 293053 2024-09-16T03:38:32.026 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:32.026 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:32.026 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:38:32.026 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:38:32.026 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:32.035 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:38:32.055 INFO:tasks.rados.rados.0.smithi157.stdout:1343: expect (ObjNum 254 snap 0 seq_num 254) 2024-09-16T03:38:32.101 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:32.098+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:32.134 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:32.147 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:32.147 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:38:32.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:38:32.147 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:32.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.147 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:39 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'21, mlcod=26'19)", 2024-09-16T03:38:32.147 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:38:32.147 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 97.548416621000001, 2024-09-16T03:38:32.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30197182500000003, 2024-09-16T03:38:32.147 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.148 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:38:32.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:38:32.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595320+0000", 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.962e-06 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595336+0000", 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5568000000000001e-05 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595349+0000", 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3203e-05 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695424+0000", 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007542899999999 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695495+0000", 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0450999999999993e-05 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695538+0000", 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3433000000000002e-05 2024-09-16T03:38:32.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897256+0000", 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20171840699999999 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897290+0000", 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3371999999999998e-05 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.152 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1226 3.2 e54/31 3:6f0527c9:::7f01c73e0f253c35d7776d11c46f7e5d9edb0e9d6a760d7305f052eba477b6c2:head v 54'82, mlcod=54'80)", 2024-09-16T03:38:32.152 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.472805+0000", 2024-09-16T03:38:32.152 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.670929573999999, 2024-09-16T03:38:32.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20253328100000001, 2024-09-16T03:38:32.152 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.152 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.152 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.472805+0000", 2024-09-16T03:38:32.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.472805+0000", 2024-09-16T03:38:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.472807+0000", 2024-09-16T03:38:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9489999999999999e-06 2024-09-16T03:38:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.473596+0000", 2024-09-16T03:38:32.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078904500000000005 2024-09-16T03:38:32.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.473614+0000", 2024-09-16T03:38:32.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7832000000000001e-05 2024-09-16T03:38:32.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.573713+0000", 2024-09-16T03:38:32.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100098627 2024-09-16T03:38:32.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.673879+0000", 2024-09-16T03:38:32.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100166088 2024-09-16T03:38:32.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.673918+0000", 2024-09-16T03:38:32.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9035999999999997e-05 2024-09-16T03:38:32.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675283+0000", 2024-09-16T03:38:32.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001365668 2024-09-16T03:38:32.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675338+0000", 2024-09-16T03:38:32.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5036e-05 2024-09-16T03:38:32.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.156 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.156 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1588 3.2 e61/31 3:57b21e08:::c9c74db4cc3b05b40ab52a9ed9c665f10e27723063858665cdba49de2698a01f:head v 61'105, mlcod=59'103)", 2024-09-16T03:38:32.157 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:46.917498+0000", 2024-09-16T03:38:32.157 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.226236241999999, 2024-09-16T03:38:32.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206346578, 2024-09-16T03:38:32.157 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.157 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.157 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.157 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.917498+0000", 2024-09-16T03:38:32.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.917498+0000", 2024-09-16T03:38:32.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.917502+0000", 2024-09-16T03:38:32.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0230000000000003e-06 2024-09-16T03:38:32.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.921109+0000", 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036070149999999999 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.921129+0000", 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9741999999999998e-05 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.921139+0000", 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5079999999999994e-06 2024-09-16T03:38:32.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.121366+0000", 2024-09-16T03:38:32.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20022730599999999 2024-09-16T03:38:32.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.121386+0000", 2024-09-16T03:38:32.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9856000000000001e-05 2024-09-16T03:38:32.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.123808+0000", 2024-09-16T03:38:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024219979999999999 2024-09-16T03:38:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.123845+0000", 2024-09-16T03:38:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7129999999999999e-05 2024-09-16T03:38:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1947 4.8 4:146f84f0:::smithi15736696-6:head [write 2063988~686387] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:32.161 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124065+0000", 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.019669262000001, 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203590252, 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1947 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124065+0000", 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124065+0000", 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124067+0000", 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.66e-06 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124770+0000", 2024-09-16T03:38:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070269099999999997 2024-09-16T03:38:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124783+0000", 2024-09-16T03:38:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2989999999999999e-05 2024-09-16T03:38:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124791+0000", 2024-09-16T03:38:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5159999999999994e-06 2024-09-16T03:38:32.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124848+0000", 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6787999999999999e-05 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124933+0000", 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4753999999999995e-05 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.125040+0000", 2024-09-16T03:38:32.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106827 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127507+0000", 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024672119999999999 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127880+0000", 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000373237 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.131974+0000", 2024-09-16T03:38:32.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040936230000000002 2024-09-16T03:38:32.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.226110+0000", 2024-09-16T03:38:32.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094136475999999997 2024-09-16T03:38:32.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.327424+0000", 2024-09-16T03:38:32.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101313954 2024-09-16T03:38:32.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.327503+0000", 2024-09-16T03:38:32.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8850000000000006e-05 2024-09-16T03:38:32.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.327656+0000", 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152674 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1948 4.8 4:146f84f0:::smithi15736696-6:head [setxattr _header (55) in=7b,truncate 2756348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.018927815000001, 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40326984799999999, 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.170 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.171 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.171 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.171 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1948 2024-09-16T03:38:32.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.171 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:38:32.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:38:32.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124808+0000", 2024-09-16T03:38:32.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9099999999999991e-07 2024-09-16T03:38:32.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124813+0000", 2024-09-16T03:38:32.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9599999999999999e-06 2024-09-16T03:38:32.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124821+0000", 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0490000000000004e-06 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124827+0000", 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3339999999999998e-06 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126739+0000", 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00191147 2024-09-16T03:38:32.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126830+0000", 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1751999999999998e-05 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127005+0000", 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174227 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127981+0000", 2024-09-16T03:38:32.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097622799999999999 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.128193+0000", 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021176999999999999 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.135281+0000", 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070882150000000001 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.427785+0000", 2024-09-16T03:38:32.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29250390599999998 2024-09-16T03:38:32.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527932+0000", 2024-09-16T03:38:32.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10014759500000001 2024-09-16T03:38:32.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527980+0000", 2024-09-16T03:38:32.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7725000000000001e-05 2024-09-16T03:38:32.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528077+0000", 2024-09-16T03:38:32.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6626000000000006e-05 2024-09-16T03:38:32.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.177 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.177 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1949 4.8 4:146f84f0:::smithi15736696-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:32.177 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:38:32.177 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.018897071000001, 2024-09-16T03:38:32.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40346921899999999, 2024-09-16T03:38:32.177 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1949 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:38:32.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:38:32.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:38:32.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124841+0000", 2024-09-16T03:38:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5890000000000001e-06 2024-09-16T03:38:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124847+0000", 2024-09-16T03:38:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0820000000000004e-06 2024-09-16T03:38:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124852+0000", 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1070000000000004e-06 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127536+0000", 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026844619999999999 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127585+0000", 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8915999999999997e-05 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528126+0000", 2024-09-16T03:38:32.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40054061899999999 2024-09-16T03:38:32.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528193+0000", 2024-09-16T03:38:32.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6717999999999994e-05 2024-09-16T03:38:32.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528307+0000", 2024-09-16T03:38:32.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011413600000000001 2024-09-16T03:38:32.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.183 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.183 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1979 4.18 4:1f63731d:::smithi15736696-30:head [write 789962~534747] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:32.183 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:38:32.183 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.507889759000001, 2024-09-16T03:38:32.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.506754182, 2024-09-16T03:38:32.183 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1979 2024-09-16T03:38:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:38:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:38:32.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635850+0000", 2024-09-16T03:38:32.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6670000000000004e-06 2024-09-16T03:38:32.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636429+0000", 2024-09-16T03:38:32.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000579745 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636441+0000", 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1965e-05 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736594+0000", 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100152291 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837326+0000", 2024-09-16T03:38:32.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10073201399999999 2024-09-16T03:38:32.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837437+0000", 2024-09-16T03:38:32.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111914 2024-09-16T03:38:32.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:32.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837625+0000", 2024-09-16T03:38:32.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000187391 2024-09-16T03:38:32.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939449+0000", 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10182371599999999 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939476+0000", 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7117000000000001e-05 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142297+0000", 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20282091499999999 2024-09-16T03:38:32.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142347+0000", 2024-09-16T03:38:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0800000000000002e-05 2024-09-16T03:38:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142485+0000", 2024-09-16T03:38:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013799999999999999 2024-09-16T03:38:32.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142526+0000", 2024-09-16T03:38:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0269999999999999e-05 2024-09-16T03:38:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142599+0000", 2024-09-16T03:38:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3376999999999999e-05 2024-09-16T03:38:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1980 4.18 4:1f63731d:::smithi15736696-30:head [write 1780952~690056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.407088598000001, 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40615011400000001, 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1980 2024-09-16T03:38:32.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.192 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:38:32.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:38:32.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736656+0000", 2024-09-16T03:38:32.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0339999999999999e-05 2024-09-16T03:38:32.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737092+0000", 2024-09-16T03:38:32.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435957 2024-09-16T03:38:32.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737102+0000", 2024-09-16T03:38:32.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7860000000000008e-06 2024-09-16T03:38:32.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737107+0000", 2024-09-16T03:38:32.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:38:32.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839282+0000", 2024-09-16T03:38:32.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10217523100000001 2024-09-16T03:38:32.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839389+0000", 2024-09-16T03:38:32.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010619399999999999 2024-09-16T03:38:32.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:32.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839567+0000", 2024-09-16T03:38:32.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017886200000000001 2024-09-16T03:38:32.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:32.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939485+0000", 2024-09-16T03:38:32.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099917168000000001 2024-09-16T03:38:32.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943655+0000", 2024-09-16T03:38:32.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004170287 2024-09-16T03:38:32.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142373+0000", 2024-09-16T03:38:32.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19871839599999999 2024-09-16T03:38:32.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142431+0000", 2024-09-16T03:38:32.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7880999999999997e-05 2024-09-16T03:38:32.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142627+0000", 2024-09-16T03:38:32.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019580500000000001 2024-09-16T03:38:32.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142723+0000", 2024-09-16T03:38:32.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6209e-05 2024-09-16T03:38:32.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142796+0000", 2024-09-16T03:38:32.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2978000000000005e-05 2024-09-16T03:38:32.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2277 4.18 4:1f63731d:::smithi15736696-30:head [write 460558~707131] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.720952898, 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39421539100000003, 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.199 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.200 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2277 2024-09-16T03:38:32.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.200 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:38:32.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422785+0000", 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9809999999999999e-06 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423586+0000", 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080133600000000004 2024-09-16T03:38:32.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423607+0000", 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1039e-05 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423616+0000", 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0189999999999995e-06 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523776+0000", 2024-09-16T03:38:32.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015945399999999 2024-09-16T03:38:32.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523851+0000", 2024-09-16T03:38:32.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5549999999999993e-05 2024-09-16T03:38:32.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:32.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523972+0000", 2024-09-16T03:38:32.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120683 2024-09-16T03:38:32.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:32.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527436+0000", 2024-09-16T03:38:32.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034638870000000001 2024-09-16T03:38:32.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636588+0000", 2024-09-16T03:38:32.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109152601 2024-09-16T03:38:32.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636665+0000", 2024-09-16T03:38:32.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6661000000000003e-05 2024-09-16T03:38:32.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637128+0000", 2024-09-16T03:38:32.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046316199999999998 2024-09-16T03:38:32.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816886+0000", 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17975745200000001 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816927+0000", 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1529000000000002e-05 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816997+0000", 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0036999999999994e-05 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2278 4.18 4:1f63731d:::smithi15736696-30:head [write 1869389~508716] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.714383897000001, 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48788153400000001, 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2278 2024-09-16T03:38:32.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.209 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:38:32.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:38:32.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429352+0000", 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.604e-06 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429895+0000", 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054308899999999996 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429909+0000", 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3518e-05 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429917+0000", 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525208+0000", 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095291736000000002 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525303+0000", 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4304999999999999e-05 2024-09-16T03:38:32.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:32.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525410+0000", 2024-09-16T03:38:32.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010755399999999999 2024-09-16T03:38:32.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636785+0000", 2024-09-16T03:38:32.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111375037 2024-09-16T03:38:32.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637092+0000", 2024-09-16T03:38:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030685599999999999 2024-09-16T03:38:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637157+0000", 2024-09-16T03:38:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4593999999999999e-05 2024-09-16T03:38:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637236+0000", 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9563999999999999e-05 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917119+0000", 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.279882931 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917158+0000", 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9096e-05 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917232+0000", 2024-09-16T03:38:32.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4091000000000006e-05 2024-09-16T03:38:32.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.215 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.215 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2279 4.18 4:1f63731d:::smithi15736696-30:head [write 3094884~165674] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:32.215 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:38:32.215 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.709695102000001, 2024-09-16T03:38:32.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483372209, 2024-09-16T03:38:32.215 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.215 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.216 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2279 2024-09-16T03:38:32.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.216 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:38:32.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:38:32.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434041+0000", 2024-09-16T03:38:32.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.195e-06 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434191+0000", 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000150488 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434202+0000", 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0978000000000001e-05 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434208+0000", 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526820+0000", 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092612229000000004 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526894+0000", 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3435000000000004e-05 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:32.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526990+0000", 2024-09-16T03:38:32.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6112000000000004e-05 2024-09-16T03:38:32.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637050+0000", 2024-09-16T03:38:32.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110059859 2024-09-16T03:38:32.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:32.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637097+0000", 2024-09-16T03:38:32.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6848000000000003e-05 2024-09-16T03:38:32.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637183+0000", 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6798000000000005e-05 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637262+0000", 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9049999999999997e-05 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917287+0000", 2024-09-16T03:38:32.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280025036 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917326+0000", 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.871e-05 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917412+0000", 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5638999999999998e-05 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.223 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2280 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 3260558] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:32.223 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:38:32.223 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.709516821000001, 2024-09-16T03:38:32.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48353995599999999, 2024-09-16T03:38:32.223 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.223 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.223 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2280 2024-09-16T03:38:32.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:38:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:38:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434221+0000", 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8329999999999998e-06 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434225+0000", 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7589999999999999e-06 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434228+0000", 2024-09-16T03:38:32.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.971e-06 2024-09-16T03:38:32.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527459+0000", 2024-09-16T03:38:32.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093231249000000002 2024-09-16T03:38:32.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527519+0000", 2024-09-16T03:38:32.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9471e-05 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527606+0000", 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6776999999999999e-05 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637084+0000", 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109478228 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:32.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637100+0000", 2024-09-16T03:38:32.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6711999999999999e-05 2024-09-16T03:38:32.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637210+0000", 2024-09-16T03:38:32.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010909200000000001 2024-09-16T03:38:32.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637296+0000", 2024-09-16T03:38:32.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6718000000000006e-05 2024-09-16T03:38:32.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917637+0000", 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280340537 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917671+0000", 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3983999999999999e-05 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917758+0000", 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7017999999999999e-05 2024-09-16T03:38:32.229 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.230 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.230 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.230 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2281 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:32.230 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:38:32.230 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.709502526000001, 2024-09-16T03:38:32.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483761683, 2024-09-16T03:38:32.230 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.230 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.230 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:32.230 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.230 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2281 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.96e-07 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434234+0000", 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4050000000000001e-06 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434237+0000", 2024-09-16T03:38:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1939999999999998e-06 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434240+0000", 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.74e-06 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530037+0000", 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095796830999999999 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:32.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530089+0000", 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917823+0000", 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38773380000000002 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917891+0000", 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7676999999999996e-05 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917994+0000", 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010286200000000001 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2654 4.8 4:148e627d:::smithi15736696-23:head [write 699224~548358] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:21.373900+0000", 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.769834521, 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20820516, 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.235 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2654 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.373900+0000", 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.373900+0000", 2024-09-16T03:38:32.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.373904+0000", 2024-09-16T03:38:32.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3670000000000001e-06 2024-09-16T03:38:32.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.374531+0000", 2024-09-16T03:38:32.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062763499999999998 2024-09-16T03:38:32.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.374550+0000", 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8612000000000001e-05 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.374559+0000", 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8729999999999996e-06 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.374626+0000", 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7638999999999994e-05 2024-09-16T03:38:32.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.374707+0000", 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0620999999999999e-05 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.374824+0000", 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011693600000000001 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.377249+0000", 2024-09-16T03:38:32.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024252140000000002 2024-09-16T03:38:32.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381000+0000", 2024-09-16T03:38:32.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003751037 2024-09-16T03:38:32.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.391544+0000", 2024-09-16T03:38:32.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010544174999999999 2024-09-16T03:38:32.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.576324+0000", 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18477964199999999 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.581868+0000", 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055443480000000002 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.581949+0000", 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0403999999999998e-05 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.582105+0000", 2024-09-16T03:38:32.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015665699999999999 2024-09-16T03:38:32.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.242 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.242 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2657 4.8 4:148e627d:::smithi15736696-23:head [setxattr _header (56) in=7b,truncate 3899224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:32.242 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:21.388211+0000", 2024-09-16T03:38:32.242 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.755524080000001, 2024-09-16T03:38:32.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.293904257, 2024-09-16T03:38:32.242 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2657 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388211+0000", 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388211+0000", 2024-09-16T03:38:32.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388211+0000", 2024-09-16T03:38:32.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.02e-07 2024-09-16T03:38:32.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388217+0000", 2024-09-16T03:38:32.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6960000000000002e-06 2024-09-16T03:38:32.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388223+0000", 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3349999999999997e-06 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388226+0000", 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2629999999999999e-06 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.391612+0000", 2024-09-16T03:38:32.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033864239999999999 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.391673+0000", 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0418999999999999e-05 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.391771+0000", 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8638e-05 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:32.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.392525+0000", 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075383199999999996 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.551525+0000", 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15900014600000001 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.577048+0000", 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025522566 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.582681+0000", 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056329830000000003 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.681849+0000", 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099168439999999997 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.681936+0000", 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6720999999999999e-05 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.682115+0000", 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017889199999999999 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2658 4.8 4:148e627d:::smithi15736696-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:21.388241+0000", 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.755493669, 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29431808300000001, 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2658 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388241+0000", 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388241+0000", 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388242+0000", 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2699999999999999e-07 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388244+0000", 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2089999999999999e-06 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388250+0000", 2024-09-16T03:38:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.409e-06 2024-09-16T03:38:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388254+0000", 2024-09-16T03:38:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4489999999999999e-06 2024-09-16T03:38:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.392379+0000", 2024-09-16T03:38:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004125325 2024-09-16T03:38:32.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.392451+0000", 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2152999999999998e-05 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.682233+0000", 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.28978181400000003 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.682337+0000", 2024-09-16T03:38:32.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104258 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.682559+0000", 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022193899999999999 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2790 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v39,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.7396550709999996, 2024-09-16T03:38:32.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42581759000000002, 2024-09-16T03:38:32.255 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.255 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.255 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.255 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2790 2024-09-16T03:38:32.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.255 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:38:32.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:38:32.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404083+0000", 2024-09-16T03:38:32.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2770000000000001e-06 2024-09-16T03:38:32.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404095+0000", 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1780000000000001e-05 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404116+0000", 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0882000000000001e-05 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404127+0000", 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1078e-05 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489837+0000", 2024-09-16T03:38:32.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085709977000000007 2024-09-16T03:38:32.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489956+0000", 2024-09-16T03:38:32.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119856 2024-09-16T03:38:32.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.592758+0000", 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10280165400000001 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.593693+0000", 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000935285 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.594164+0000", 2024-09-16T03:38:32.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047097000000000003 2024-09-16T03:38:32.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.628426+0000", 2024-09-16T03:38:32.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034262070999999998 2024-09-16T03:38:32.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.732623+0000", 2024-09-16T03:38:32.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10419628 2024-09-16T03:38:32.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829726+0000", 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097103098999999998 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829790+0000", 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4677000000000004e-05 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829897+0000", 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106704 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2849 4.18 4:1f63731d:::smithi15736696-30:head [write 1821709~430462] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920227+0000", 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2235072550000003, 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203899736, 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.262 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2849 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920227+0000", 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920227+0000", 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920229+0000", 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5820000000000001e-06 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920587+0000", 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035828299999999998 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920601+0000", 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3633e-05 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920607+0000", 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7139999999999998e-06 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920651+0000", 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4409999999999997e-05 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920720+0000", 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9109999999999994e-05 2024-09-16T03:38:32.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:32.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920815+0000", 2024-09-16T03:38:32.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4932000000000005e-05 2024-09-16T03:38:32.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:32.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.023363+0000", 2024-09-16T03:38:32.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102547948 2024-09-16T03:38:32.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123125+0000", 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099761876999999999 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123179+0000", 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4072e-05 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123780+0000", 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060145499999999998 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123961+0000", 2024-09-16T03:38:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000180922 2024-09-16T03:38:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.124016+0000", 2024-09-16T03:38:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4826e-05 2024-09-16T03:38:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.124127+0000", 2024-09-16T03:38:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011097199999999999 2024-09-16T03:38:32.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.269 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.269 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2850 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 2252171] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:38:32.269 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:38:32.269 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2231147890000003, 2024-09-16T03:38:32.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30323033799999999, 2024-09-16T03:38:32.269 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:32.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.269 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2850 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920621+0000", 2024-09-16T03:38:32.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.47e-07 2024-09-16T03:38:32.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920625+0000", 2024-09-16T03:38:32.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1620000000000001e-06 2024-09-16T03:38:32.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920632+0000", 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0180000000000004e-06 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920638+0000", 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0950000000000001e-06 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922161+0000", 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015229880000000001 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922222+0000", 2024-09-16T03:38:32.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0903999999999998e-05 2024-09-16T03:38:32.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:32.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922323+0000", 2024-09-16T03:38:32.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101425 2024-09-16T03:38:32.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:32.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.023371+0000", 2024-09-16T03:38:32.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101047493 2024-09-16T03:38:32.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123227+0000", 2024-09-16T03:38:32.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099856057999999998 2024-09-16T03:38:32.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123840+0000", 2024-09-16T03:38:32.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061324999999999997 2024-09-16T03:38:32.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123902+0000", 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1447999999999998e-05 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223497+0000", 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099595499000000004 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223665+0000", 2024-09-16T03:38:32.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016807399999999999 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223850+0000", 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018507699999999999 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2851 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2230786929999997, 2024-09-16T03:38:32.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30354833399999998, 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2851 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920657+0000", 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0100000000000004e-07 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920660+0000", 2024-09-16T03:38:32.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6519999999999999e-06 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920668+0000", 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1919999999999999e-06 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920674+0000", 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126299+0000", 2024-09-16T03:38:32.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20562534700000001 2024-09-16T03:38:32.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:32.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126358+0000", 2024-09-16T03:38:32.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8459999999999999e-05 2024-09-16T03:38:32.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:32.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223913+0000", 2024-09-16T03:38:32.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097555263000000003 2024-09-16T03:38:32.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:32.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224009+0000", 2024-09-16T03:38:32.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5723999999999994e-05 2024-09-16T03:38:32.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:32.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:32.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:32.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224204+0000", 2024-09-16T03:38:32.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195498 2024-09-16T03:38:32.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.281 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:32.281 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:32.282 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:32.282 INFO:tasks.rados.rados.0.smithi157.stdout:1342: finishing write tid 1 to smithi15736696-50 2024-09-16T03:38:32.283 INFO:tasks.rados.rados.0.smithi157.stdout:1349: expect (ObjNum 224 snap 0 seq_num 224) 2024-09-16T03:38:32.283 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:38:32.284 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:38:32.284 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:32.281+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:32.288 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:32.342 INFO:tasks.rados.rados.0.smithi157.stdout:1348: expect (ObjNum 238 snap 0 seq_num 238) 2024-09-16T03:38:32.362 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:32.361+0000 7f8be76d1640 -1 osd.7 99 handle_osd_map injecting map crc failure 2024-09-16T03:38:32.365 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:32.367 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:32.365+0000 7fedb5739640 -1 osd.9 99 handle_osd_map injecting map crc failure 2024-09-16T03:38:32.368 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:32.365+0000 7f4212d00640 -1 osd.8 99 handle_osd_map injecting map crc failure 2024-09-16T03:38:32.368 INFO:teuthology.orchestra.run.smithi064.stderr:marked in osd.9. 2024-09-16T03:38:32.369 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:32.367+0000 7f4212d00640 -1 osd.8 99 handle_osd_map injecting map crc failure 2024-09-16T03:38:32.383 INFO:tasks.thrashosds.thrasher:Added osd 9 2024-09-16T03:38:32.384 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:32.382+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:32.400 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:32.400 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:32.401 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:38:32.401 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:32.410 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:38:32.412 INFO:tasks.rados.rados.0.smithi157.stdout:1350: got (0) Success 2024-09-16T03:38:32.412 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 55 (ObjNum 230 snap 0 seq_num 230) dirty exists 2024-09-16T03:38:32.412 INFO:tasks.rados.rados.0.smithi157.stdout:1347: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:38:32.416 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:32.417 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:32.434 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:32.434 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:32.452 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:32.453 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:32.469 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:32.469 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:32.484 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:32.483+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:32.488 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:32.488 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:32.506 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:32.506 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:32.522 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:32.523 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:32.527 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:32.527 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:32.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:38:32.539 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:32.539 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:32.556 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:32.557 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:32.574 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:32.574 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:32.580 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:32.585 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:32.583+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:32.591 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:32.591 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:32.608 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:32.609 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:32.626 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:32.626 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:32.643 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:32.643 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:32.648 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:32.649 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:32.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:38:32.660 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:32.660 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:32.672 INFO:tasks.rados.rados.0.smithi157.stdout:1344: expect (ObjNum 216 snap 0 seq_num 216) 2024-09-16T03:38:32.676 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:32.677 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:32.686 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:32.683+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:32.730 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:32.768 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:32.768 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:32.777 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:38:32.787 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:32.785+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:32.787 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:32.844 INFO:tasks.rados.rados.0.smithi157.stdout:1342: finishing write tid 2 to smithi15736696-50 2024-09-16T03:38:32.845 INFO:tasks.rados.rados.0.smithi157.stdout:1342: finishing write tid 3 to smithi15736696-50 2024-09-16T03:38:32.845 INFO:tasks.rados.rados.0.smithi157.stdout:1342: finishing write tid 4 to smithi15736696-50 2024-09-16T03:38:32.845 INFO:tasks.rados.rados.0.smithi157.stdout:1342: oid 50 updating version 0 to 114 2024-09-16T03:38:32.845 INFO:tasks.rados.rados.0.smithi157.stdout:1342: oid 50 version 114 is already newer than 112 2024-09-16T03:38:32.845 INFO:tasks.rados.rados.0.smithi157.stdout:1342: oid 50 version 114 is already newer than 113 2024-09-16T03:38:32.845 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 114 (ObjNum 266 snap 0 seq_num 266) dirty exists 2024-09-16T03:38:32.845 INFO:tasks.rados.rados.0.smithi157.stdout:1342: left oid 50 (ObjNum 266 snap 0 seq_num 266) 2024-09-16T03:38:32.845 INFO:tasks.rados.rados.0.smithi157.stdout:1346: got (22) Invalid argument 2024-09-16T03:38:32.845 INFO:tasks.rados.rados.0.smithi157.stdout:1345: finishing write tid 1 to smithi15736696-36 2024-09-16T03:38:32.845 INFO:tasks.rados.rados.0.smithi157.stdout:1351: expect (ObjNum 246 snap 0 seq_num 246) 2024-09-16T03:38:32.877 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1880936 2024-09-16T03:38:32.884 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:32.884 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:32.887 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:32.886+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:32.893 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:38:32.961 INFO:tasks.rados.rados.0.smithi157.stdout:1345: finishing write tid 2 to smithi15736696-36 2024-09-16T03:38:32.961 INFO:tasks.rados.rados.0.smithi157.stdout:1345: finishing write tid 3 to smithi15736696-36 2024-09-16T03:38:32.961 INFO:tasks.rados.rados.0.smithi157.stdout:1345: finishing write tid 4 to smithi15736696-36 2024-09-16T03:38:32.961 INFO:tasks.rados.rados.0.smithi157.stdout:1345: oid 36 updating version 0 to 111 2024-09-16T03:38:32.961 INFO:tasks.rados.rados.0.smithi157.stdout:1345: oid 36 updating version 111 to 112 2024-09-16T03:38:32.961 INFO:tasks.rados.rados.0.smithi157.stdout:1345: oid 36 version 112 is already newer than 110 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 112 (ObjNum 267 snap 0 seq_num 267) dirty exists 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1345: left oid 36 (ObjNum 267 snap 0 seq_num 267) 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1340: done (11 left) 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1342: done (10 left) 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1343: done (9 left) 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1344: done (8 left) 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1345: done (7 left) 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1346: done (6 left) 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1347: done (5 left) 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1348: done (4 left) 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1349: done (3 left) 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1350: done (2 left) 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1351: done (1 left) 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1352: done (0 left) 2024-09-16T03:38:32.962 INFO:tasks.rados.rados.0.smithi157.stdout:1353: read oid 32 snap -1 2024-09-16T03:38:32.963 INFO:tasks.rados.rados.0.smithi157.stdout:1353: expect deleted 2024-09-16T03:38:32.963 INFO:tasks.rados.rados.0.smithi157.stdout:1354: write oid 18 current snap is 0 2024-09-16T03:38:32.963 INFO:tasks.rados.rados.0.smithi157.stdout:1354: seq_num 269 ranges {521083=714336,1724715=555421} 2024-09-16T03:38:32.966 INFO:tasks.rados.rados.0.smithi157.stdout:1354: writing smithi15736696-18 from 521083 to 1235419 tid 1 2024-09-16T03:38:32.973 INFO:tasks.rados.rados.0.smithi157.stdout:1354: writing smithi15736696-18 from 1724715 to 2280136 tid 2 2024-09-16T03:38:32.973 INFO:tasks.rados.rados.0.smithi157.stdout:1355: read oid 34 snap -1 2024-09-16T03:38:32.973 INFO:tasks.rados.rados.0.smithi157.stdout:1355: expect (ObjNum 205 snap 0 seq_num 205) 2024-09-16T03:38:32.973 INFO:tasks.rados.rados.0.smithi157.stdout:1356: write (excl) oid 16 current snap is 0 2024-09-16T03:38:32.973 INFO:tasks.rados.rados.0.smithi157.stdout:1356: seq_num 270 ranges {537890=789585,1783197=699079} 2024-09-16T03:38:32.980 INFO:tasks.rados.rados.0.smithi157.stdout:1356: writing smithi15736696-16 from 537890 to 1327475 tid 1 2024-09-16T03:38:32.986 INFO:tasks.rados.rados.0.smithi157.stdout:1356: writing smithi15736696-16 from 1783197 to 2482276 tid 2 2024-09-16T03:38:32.986 INFO:tasks.rados.rados.0.smithi157.stdout:1357: write oid 24 current snap is 0 2024-09-16T03:38:32.986 INFO:tasks.rados.rados.0.smithi157.stdout:1357: seq_num 271 ranges {554697=464834,1725326=559090} 2024-09-16T03:38:32.989 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:32.987+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:32.992 INFO:tasks.rados.rados.0.smithi157.stdout:1357: writing smithi15736696-24 from 554697 to 1019531 tid 1 2024-09-16T03:38:33.000 INFO:tasks.rados.rados.0.smithi157.stdout:1357: writing smithi15736696-24 from 1725326 to 2284416 tid 2 2024-09-16T03:38:33.000 INFO:tasks.rados.rados.0.smithi157.stdout:1358: read oid 36 snap -1 2024-09-16T03:38:33.000 INFO:tasks.rados.rados.0.smithi157.stdout:1358: expect (ObjNum 267 snap 0 seq_num 267) 2024-09-16T03:38:33.000 INFO:tasks.rados.rados.0.smithi157.stdout:1359: read oid 15 snap -1 2024-09-16T03:38:33.000 INFO:tasks.rados.rados.0.smithi157.stdout:1359: expect deleted 2024-09-16T03:38:33.000 INFO:tasks.rados.rados.0.smithi157.stdout:1360: tier_evict oid 7 2024-09-16T03:38:33.000 INFO:tasks.rados.rados.0.smithi157.stdout:1360: tier_evict oid 7 snap -1 2024-09-16T03:38:33.000 INFO:tasks.rados.rados.0.smithi157.stdout:1361: read oid 44 snap -1 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1361: expect (ObjNum 219 snap 0 seq_num 219) 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1362: read oid 9 snap -1 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1362: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1363: read oid 41 snap -1 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1363: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1364: read oid 33 snap -1 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1364: expect (ObjNum 230 snap 0 seq_num 230) 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1365: read oid 12 snap -1 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1365: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1366: read oid 10 snap -1 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1366: expect (ObjNum 227 snap 0 seq_num 227) 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1367: read oid 19 snap -1 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1367: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:33.001 INFO:tasks.rados.rados.0.smithi157.stdout:1368: read oid 20 snap -1 2024-09-16T03:38:33.002 INFO:tasks.rados.rados.0.smithi157.stdout:1368: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-16T03:38:33.002 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:33.002 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:33.002 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:33.010 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:38:33.060 INFO:tasks.rados.rados.0.smithi157.stdout:1353: done (15 left) 2024-09-16T03:38:33.061 INFO:tasks.rados.rados.0.smithi157.stdout:1369: delete oid 21 current snap is 0 2024-09-16T03:38:33.090 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:33.088+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:33.104 INFO:tasks.rados.rados.0.smithi157.stdout:1365: expect (ObjNum 148 snap 0 seq_num 148) 2024-09-16T03:38:33.157 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:33.157 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:33.158 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 301227 2024-09-16T03:38:33.166 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:38:33.185 INFO:tasks.rados.rados.0.smithi157.stdout:1356: finishing write tid 1 to smithi15736696-16 2024-09-16T03:38:33.185 INFO:tasks.rados.rados.0.smithi157.stdout:1356: finishing write tid 2 to smithi15736696-16 2024-09-16T03:38:33.185 INFO:tasks.rados.rados.0.smithi157.stdout:1356: finishing write tid 3 to smithi15736696-16 2024-09-16T03:38:33.190 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:33.189+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:33.273 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:33.273 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:33.274 INFO:tasks.rados.rados.0.smithi157.stdout:1354: finishing write tid 1 to smithi15736696-18 2024-09-16T03:38:33.274 INFO:tasks.rados.rados.0.smithi157.stdout:1360: got (22) Invalid argument 2024-09-16T03:38:33.276 INFO:tasks.rados.rados.0.smithi157.stdout:1354: finishing write tid 2 to smithi15736696-18 2024-09-16T03:38:33.277 INFO:tasks.rados.rados.0.smithi157.stdout:1354: finishing write tid 3 to smithi15736696-18 2024-09-16T03:38:33.278 INFO:tasks.rados.rados.0.smithi157.stdout:1368: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-16T03:38:33.282 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:38:33.291 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:33.290+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:33.342 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1738320 2024-09-16T03:38:33.367 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:38:33.379 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:33.376+0000 7f12e2a1a640 -1 osd.5 100 handle_osd_map injecting map crc failure 2024-09-16T03:38:33.392 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:33.390+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:33.393 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:33.393 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:33.403 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:38:33.408 INFO:tasks.rados.rados.0.smithi157.stdout:1354: finishing write tid 4 to smithi15736696-18 2024-09-16T03:38:33.408 INFO:tasks.rados.rados.0.smithi157.stdout:1354: oid 18 updating version 0 to 58 2024-09-16T03:38:33.409 INFO:tasks.rados.rados.0.smithi157.stdout:1354: oid 18 version 58 is already newer than 57 2024-09-16T03:38:33.409 INFO:tasks.rados.rados.0.smithi157.stdout:1354: oid 18 version 58 is already newer than 56 2024-09-16T03:38:33.409 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 58 (ObjNum 269 snap 0 seq_num 269) dirty exists 2024-09-16T03:38:33.409 INFO:tasks.rados.rados.0.smithi157.stdout:1354: left oid 18 (ObjNum 269 snap 0 seq_num 269) 2024-09-16T03:38:33.409 INFO:tasks.rados.rados.0.smithi157.stdout:1363: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-16T03:38:33.409 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:33.493 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:33.491+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:33.517 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:33.518 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:33.527 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:38:33.540 INFO:tasks.rados.rados.0.smithi157.stdout:1356: finishing write tid 4 to smithi15736696-16 2024-09-16T03:38:33.540 INFO:tasks.rados.rados.0.smithi157.stdout:1356: oid 16 updating version 0 to 59 2024-09-16T03:38:33.540 INFO:tasks.rados.rados.0.smithi157.stdout:1356: oid 16 version 59 is already newer than 58 2024-09-16T03:38:33.540 INFO:tasks.rados.rados.0.smithi157.stdout:1356: oid 16 version 59 is already newer than 57 2024-09-16T03:38:33.541 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 59 (ObjNum 270 snap 0 seq_num 270) dirty exists 2024-09-16T03:38:33.541 INFO:tasks.rados.rados.0.smithi157.stdout:1356: left oid 16 (ObjNum 270 snap 0 seq_num 270) 2024-09-16T03:38:33.592 INFO:tasks.rados.rados.0.smithi157.stdout:1361: expect (ObjNum 219 snap 0 seq_num 219) 2024-09-16T03:38:33.594 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:33.591+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:33.631 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:33.631 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:33.639 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:38:33.695 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:33.694+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:33.784 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:33.784 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:33.793 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:38:33.796 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:33.793+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:33.897 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:33.894+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:33.897 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:33.897 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:33.904 INFO:tasks.rados.rados.0.smithi157.stdout:1362: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-16T03:38:33.906 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:38:33.998 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:33.995+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:34.009 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:34.010 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:34.018 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:38:34.067 INFO:tasks.rados.rados.0.smithi157.stdout:1355: expect (ObjNum 205 snap 0 seq_num 205) 2024-09-16T03:38:34.099 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:34.096+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:34.121 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:34.122 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:34.130 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:38:34.142 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2250823 2024-09-16T03:38:34.161 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:34.199 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:34.198+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:34.212 INFO:tasks.rados.rados.0.smithi157.stdout:1358: expect (ObjNum 267 snap 0 seq_num 267) 2024-09-16T03:38:34.270 INFO:tasks.rados.rados.0.smithi157.stdout:1367: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:34.270 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:34.274 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:34.274 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:34.283 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:38:34.326 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:34.324+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:34.381 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:34.383 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:34.380+0000 7f49419d0640 -1 osd.4 101 handle_osd_map injecting map crc failure 2024-09-16T03:38:34.388 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:38:34.391 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:34.391 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:34.400 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:38:34.401 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:34.400+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:34.438 INFO:tasks.rados.rados.0.smithi157.stdout:1364: expect (ObjNum 230 snap 0 seq_num 230) 2024-09-16T03:38:34.438 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:34.502 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:34.499+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:34.545 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:34.545 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:34.554 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:38:34.603 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:34.600+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:34.659 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:34.660 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:34.668 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:38:34.688 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:38:34.710 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:34.708+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:34.711 INFO:tasks.rados.rados.0.smithi157.stdout:1357: finishing write tid 1 to smithi15736696-24 2024-09-16T03:38:34.712 INFO:tasks.rados.rados.0.smithi157.stdout:1357: finishing write tid 2 to smithi15736696-24 2024-09-16T03:38:34.712 INFO:tasks.rados.rados.0.smithi157.stdout:1357: finishing write tid 3 to smithi15736696-24 2024-09-16T03:38:34.712 INFO:tasks.rados.rados.0.smithi157.stdout:1366: expect (ObjNum 227 snap 0 seq_num 227) 2024-09-16T03:38:34.788 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:34.805 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:34.803+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:34.814 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:34.814 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:34.816 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:34.816 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:34.823 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:38:34.834 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:34.834 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:34.852 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:34.853 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:34.869 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:34.869 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:34.885 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:34.885 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:34.902 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:34.903 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:34.906 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:34.904+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:34.920 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:34.920 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:34.928 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:34.928 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:34.937 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:34.937 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:34.938 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:38:34.943 INFO:tasks.rados.rados.0.smithi157.stdout:1357: finishing write tid 4 to smithi15736696-24 2024-09-16T03:38:34.944 INFO:tasks.rados.rados.0.smithi157.stdout:1357: oid 24 updating version 0 to 113 2024-09-16T03:38:34.944 INFO:tasks.rados.rados.0.smithi157.stdout:1357: oid 24 updating version 113 to 114 2024-09-16T03:38:34.944 INFO:tasks.rados.rados.0.smithi157.stdout:1357: oid 24 updating version 114 to 115 2024-09-16T03:38:34.944 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 115 (ObjNum 271 snap 0 seq_num 271) dirty exists 2024-09-16T03:38:34.944 INFO:tasks.rados.rados.0.smithi157.stdout:1357: left oid 24 (ObjNum 271 snap 0 seq_num 271) 2024-09-16T03:38:34.944 INFO:tasks.rados.rados.0.smithi157.stdout:1354: done (15 left) 2024-09-16T03:38:34.944 INFO:tasks.rados.rados.0.smithi157.stdout:1355: done (14 left) 2024-09-16T03:38:34.944 INFO:tasks.rados.rados.0.smithi157.stdout:1356: done (13 left) 2024-09-16T03:38:34.944 INFO:tasks.rados.rados.0.smithi157.stdout:1357: done (12 left) 2024-09-16T03:38:34.944 INFO:tasks.rados.rados.0.smithi157.stdout:1358: done (11 left) 2024-09-16T03:38:34.944 INFO:tasks.rados.rados.0.smithi157.stdout:1359: done (10 left) 2024-09-16T03:38:34.944 INFO:tasks.rados.rados.0.smithi157.stdout:1360: done (9 left) 2024-09-16T03:38:34.944 INFO:tasks.rados.rados.0.smithi157.stdout:1361: done (8 left) 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1362: done (7 left) 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1363: done (6 left) 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1364: done (5 left) 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1365: done (4 left) 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1366: done (3 left) 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1367: done (2 left) 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1368: done (1 left) 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1369: done (0 left) 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1370: read oid 7 snap -1 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1370: expect (ObjNum 257 snap 0 seq_num 257) 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1371: read oid 16 snap -1 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1371: expect (ObjNum 270 snap 0 seq_num 270) 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1372: read oid 3 snap -1 2024-09-16T03:38:34.945 INFO:tasks.rados.rados.0.smithi157.stdout:1372: expect deleted 2024-09-16T03:38:34.946 INFO:tasks.rados.rados.0.smithi157.stdout:1373: tier_promote oid 13 2024-09-16T03:38:34.946 INFO:tasks.rados.rados.0.smithi157.stdout:1374: write (excl) oid 45 current snap is 0 2024-09-16T03:38:34.946 INFO:tasks.rados.rados.0.smithi157.stdout:1374: seq_num 273 ranges {588311=615332,1842290=562759} 2024-09-16T03:38:34.949 INFO:tasks.rados.rados.0.smithi157.stdout:1374: writing smithi15736696-45 from 588311 to 1203643 tid 1 2024-09-16T03:38:34.953 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:34.953 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:34.955 INFO:tasks.rados.rados.0.smithi157.stdout:1374: writing smithi15736696-45 from 1842290 to 2405049 tid 2 2024-09-16T03:38:34.955 INFO:tasks.rados.rados.0.smithi157.stdout:1375: tier_promote oid 5 2024-09-16T03:38:34.955 INFO:tasks.rados.rados.0.smithi157.stdout:1370: expect (ObjNum 257 snap 0 seq_num 257) 2024-09-16T03:38:34.959 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1551796 2024-09-16T03:38:34.977 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 960028 2024-09-16T03:38:35.007 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:35.004+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:35.017 INFO:tasks.rados.rados.0.smithi157.stdout:1375: got (2) No such file or directory 2024-09-16T03:38:35.017 INFO:tasks.rados.rados.0.smithi157.stdout:1375: got expected ENOENT (src dne) 2024-09-16T03:38:35.017 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 59 (ObjNum 97 snap 0 seq_num 2524317228) clean dne 2024-09-16T03:38:35.017 INFO:tasks.rados.rados.0.smithi157.stdout:1370: done (5 left) 2024-09-16T03:38:35.017 INFO:tasks.rados.rados.0.smithi157.stdout:1372: done (4 left) 2024-09-16T03:38:35.017 INFO:tasks.rados.rados.0.smithi157.stdout:1375: done (3 left) 2024-09-16T03:38:35.017 INFO:tasks.rados.rados.0.smithi157.stdout:1376: delete oid 25 current snap is 0 2024-09-16T03:38:35.018 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:35.018 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:35.035 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:35.035 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:35.052 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:35.052 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:35.058 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:35.058 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:35.058 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:35.058 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:35.068 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:38:35.068 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:35.068 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:35.084 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:35.085 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:35.095 INFO:tasks.rados.rados.0.smithi157.stdout:1373: got (0) Success 2024-09-16T03:38:35.095 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 58 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-09-16T03:38:35.101 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:35.101 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:35.112 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:35.110+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:35.145 INFO:tasks.rados.rados.0.smithi157.stdout:1371: expect (ObjNum 270 snap 0 seq_num 270) 2024-09-16T03:38:35.145 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:35.161 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:35.161 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:35.172 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:35.172 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:35.172 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:35.172 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:35.172 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:35.181 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:38:35.208 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:35.207+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:35.232 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:35.281 INFO:tasks.rados.rados.0.smithi157.stdout:1371: done (3 left) 2024-09-16T03:38:35.281 INFO:tasks.rados.rados.0.smithi157.stdout:1373: done (2 left) 2024-09-16T03:38:35.281 INFO:tasks.rados.rados.0.smithi157.stdout:1376: done (1 left) 2024-09-16T03:38:35.282 INFO:tasks.rados.rados.0.smithi157.stdout:1377: read oid 17 snap -1 2024-09-16T03:38:35.282 INFO:tasks.rados.rados.0.smithi157.stdout:1377: expect (ObjNum 248 snap 0 seq_num 248) 2024-09-16T03:38:35.282 INFO:tasks.rados.rados.0.smithi157.stdout:1378: write oid 42 current snap is 0 2024-09-16T03:38:35.282 INFO:tasks.rados.rados.0.smithi157.stdout:1374: finishing write tid 1 to smithi15736696-45 2024-09-16T03:38:35.282 INFO:tasks.rados.rados.0.smithi157.stdout:1374: finishing write tid 2 to smithi15736696-45 2024-09-16T03:38:35.282 INFO:tasks.rados.rados.0.smithi157.stdout:1378: seq_num 275 ranges {621925=482183,1559254=566428} 2024-09-16T03:38:35.287 INFO:tasks.rados.rados.0.smithi157.stdout:1378: writing smithi15736696-42 from 621925 to 1104108 tid 1 2024-09-16T03:38:35.293 INFO:tasks.rados.rados.0.smithi157.stdout:1378: writing smithi15736696-42 from 1559254 to 2125682 tid 2 2024-09-16T03:38:35.293 INFO:tasks.rados.rados.0.smithi157.stdout:1379: read oid 43 snap -1 2024-09-16T03:38:35.293 INFO:tasks.rados.rados.0.smithi157.stdout:1379: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-16T03:38:35.293 INFO:tasks.rados.rados.0.smithi157.stdout:1380: write (excl) oid 10 current snap is 0 2024-09-16T03:38:35.293 INFO:tasks.rados.rados.0.smithi157.stdout:1380: seq_num 276 ranges {638732=557432,1617736=710086} 2024-09-16T03:38:35.299 INFO:tasks.rados.rados.0.smithi157.stdout:1380: writing smithi15736696-10 from 638732 to 1196164 tid 1 2024-09-16T03:38:35.306 INFO:tasks.rados.rados.0.smithi157.stdout:1380: writing smithi15736696-10 from 1617736 to 2327822 tid 2 2024-09-16T03:38:35.306 INFO:tasks.rados.rados.0.smithi157.stdout:1381: set_chunk oid 34 target oid 2024-09-16T03:38:35.309 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:35.307+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:35.329 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:35.329 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:35.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:35.329 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:35.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.330 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:5 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'1, mlcod=0'0)", 2024-09-16T03:38:35.330 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:38:35.330 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 83.358777118999996, 2024-09-16T03:38:35.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.105742802, 2024-09-16T03:38:35.330 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.330 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.330 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.331 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.331 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.331 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:38:35.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.332 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.332 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:38:35.332 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.332 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.332 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968147+0000", 2024-09-16T03:38:35.332 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4e-07 2024-09-16T03:38:35.333 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.333 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.333 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968752+0000", 2024-09-16T03:38:35.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060437900000000005 2024-09-16T03:38:35.333 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.333 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.333 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968762+0000", 2024-09-16T03:38:35.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0284e-05 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968778+0000", 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6124e-05 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069797+0000", 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10101895700000001 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069833+0000", 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6013000000000002e-05 2024-09-16T03:38:35.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073838+0000", 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040049049999999996 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073889+0000", 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.13e-05 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.335 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:7 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'3, mlcod=0'0)", 2024-09-16T03:38:35.336 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:38:35.336 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 83.351884620000007, 2024-09-16T03:38:35.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20061241399999999, 2024-09-16T03:38:35.336 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.336 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.336 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.336 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:38:35.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:38:35.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.337 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975042+0000", 2024-09-16T03:38:35.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5289999999999998e-06 2024-09-16T03:38:35.337 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975079+0000", 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7626999999999997e-05 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975104+0000", 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4958999999999999e-05 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.075211+0000", 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100106814 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080573+0000", 2024-09-16T03:38:35.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0053618650000000004 2024-09-16T03:38:35.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080592+0000", 2024-09-16T03:38:35.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9079000000000001e-05 2024-09-16T03:38:35.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175497+0000", 2024-09-16T03:38:35.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.094904832999999994 2024-09-16T03:38:35.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175651+0000", 2024-09-16T03:38:35.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015470800000000001 2024-09-16T03:38:35.340 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.341 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.341 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.341 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:35.341 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:35.341 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 83.140931448000003, 2024-09-16T03:38:35.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10311670100000001, 2024-09-16T03:38:35.341 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.341 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.341 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.341 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:38:35.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186761+0000", 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076880999999999998 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186772+0000", 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0986e-05 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186799+0000", 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6254e-05 2024-09-16T03:38:35.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186876+0000", 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7523999999999998e-05 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186896+0000", 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0302999999999999e-05 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289077+0000", 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10218086899999999 2024-09-16T03:38:35.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289109+0000", 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1625000000000003e-05 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36)", 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 83.139725659000007, 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102408122, 2024-09-16T03:38:35.345 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187198+0000", 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.187199+0000", 2024-09-16T03:38:35.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6199999999999996e-07 2024-09-16T03:38:35.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188184+0000", 2024-09-16T03:38:35.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000985442 2024-09-16T03:38:35.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188194+0000", 2024-09-16T03:38:35.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5659999999999996e-06 2024-09-16T03:38:35.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.188210+0000", 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5858999999999998e-05 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288363+0000", 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100152872 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.288390+0000", 2024-09-16T03:38:35.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7081999999999999e-05 2024-09-16T03:38:35.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289577+0000", 2024-09-16T03:38:35.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011875740000000001 2024-09-16T03:38:35.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289606+0000", 2024-09-16T03:38:35.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8864999999999999e-05 2024-09-16T03:38:35.349 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.349 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.349 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.5.21:19 3.3 e47/31 3:fb369346:::smithi15736696-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 47'44, mlcod=47'41)", 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.339975+0000", 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 68.986948943000002, 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10201615, 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339975+0000", 2024-09-16T03:38:35.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339975+0000", 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339976+0000", 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.204e-06 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339982+0000", 2024-09-16T03:38:35.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339992+0000", 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0196e-05 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.339998+0000", 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.463e-06 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.438719+0000", 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098721712000000003 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.438747+0000", 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7475e-05 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.441935+0000", 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0031878480000000001 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.441991+0000", 2024-09-16T03:38:35.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6100999999999997e-05 2024-09-16T03:38:35.354 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.354 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.354 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.354 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:44 3.3 e49/31 3:dc284e8d:::smithi15736696-28 low_tier:head v 49'55, mlcod=49'53)", 2024-09-16T03:38:35.354 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:38:35.354 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 65.203960223999999, 2024-09-16T03:38:35.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.149341897, 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122964+0000", 2024-09-16T03:38:35.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.91e-07 2024-09-16T03:38:35.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122968+0000", 2024-09-16T03:38:35.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1470000000000003e-06 2024-09-16T03:38:35.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122979+0000", 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1090000000000001e-05 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.223110+0000", 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100131357 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271688+0000", 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048578192999999999 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271712+0000", 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4066e-05 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272260+0000", 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054737000000000004 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272305+0000", 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5482999999999998e-05 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.359 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1007 3.3 e50/31 3:fda66a0d:::920f29fe85d0745d4cd24a887714ac53f1b9a978ec2208ff473c99287fdf8feb:head v 50'57, mlcod=49'55)", 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 64.732333670000003, 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20236485200000001, 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:38:35.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594592+0000", 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0700000000000001e-06 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595508+0000", 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00091599499999999996 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595521+0000", 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595531+0000", 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2499999999999995e-06 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795762+0000", 2024-09-16T03:38:35.363 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200230977 2024-09-16T03:38:35.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.363 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.363 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795778+0000", 2024-09-16T03:38:35.363 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:38:35.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.363 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.363 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796912+0000", 2024-09-16T03:38:35.363 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011338520000000001 2024-09-16T03:38:35.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.364 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.364 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796955+0000", 2024-09-16T03:38:35.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2768000000000001e-05 2024-09-16T03:38:35.364 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.364 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.364 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.364 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.364 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.364 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1247 4.11 4:89839b7e:::smithi15736696-12:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-16T03:38:35.364 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:38:35.364 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 58.371346488999997, 2024-09-16T03:38:35.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202294525, 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1247 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955579+0000", 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3240000000000001e-06 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955587+0000", 2024-09-16T03:38:35.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:38:35.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955600+0000", 2024-09-16T03:38:35.367 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2723e-05 2024-09-16T03:38:35.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955612+0000", 2024-09-16T03:38:35.367 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2422999999999999e-05 2024-09-16T03:38:35.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055837+0000", 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10022539699999999 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055974+0000", 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000136995 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,7,8,11,12", 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.056092+0000", 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011738 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057209+0000", 2024-09-16T03:38:35.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011170329999999999 2024-09-16T03:38:35.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057262+0000", 2024-09-16T03:38:35.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2859000000000003e-05 2024-09-16T03:38:35.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157594+0000", 2024-09-16T03:38:35.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10033241699999999 2024-09-16T03:38:35.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157648+0000", 2024-09-16T03:38:35.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3863999999999998e-05 2024-09-16T03:38:35.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157692+0000", 2024-09-16T03:38:35.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4406999999999997e-05 2024-09-16T03:38:35.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157732+0000", 2024-09-16T03:38:35.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9156000000000003e-05 2024-09-16T03:38:35.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157780+0000", 2024-09-16T03:38:35.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8801999999999998e-05 2024-09-16T03:38:35.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157872+0000", 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1311999999999996e-05 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1518 3.3 e59/31 3:df1be05b:::e06092e68680905b5745d5057e93f6c7581518aed07ca5db2802546e9c29aeef:head v 59'95, mlcod=58'93)", 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.068902815999998, 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104523454, 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.372 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258022+0000", 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.372e-06 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258462+0000", 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00043970100000000002 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258476+0000", 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.376e-05 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258483+0000", 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.294e-06 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358679+0000", 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100196177 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358718+0000", 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9295999999999998e-05 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362496+0000", 2024-09-16T03:38:35.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037772069999999999 2024-09-16T03:38:35.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362544+0000", 2024-09-16T03:38:35.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8646999999999997e-05 2024-09-16T03:38:35.376 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.376 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.376 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.376 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1923 4.4 e73/50 4:239dbd85:::smithi15736696-2:head v 73'19, mlcod=62'17)", 2024-09-16T03:38:35.377 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:38:35.377 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.363041994, 2024-09-16T03:38:35.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104144848, 2024-09-16T03:38:35.377 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.377 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.377 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.377 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.377 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:38:35.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.377 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:38:35.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963884+0000", 2024-09-16T03:38:35.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:38:35.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964597+0000", 2024-09-16T03:38:35.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071314700000000002 2024-09-16T03:38:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964615+0000", 2024-09-16T03:38:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8209000000000001e-05 2024-09-16T03:38:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964626+0000", 2024-09-16T03:38:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0511e-05 2024-09-16T03:38:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064779+0000", 2024-09-16T03:38:35.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100153495 2024-09-16T03:38:35.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064794+0000", 2024-09-16T03:38:35.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4946000000000001e-05 2024-09-16T03:38:35.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.067999+0000", 2024-09-16T03:38:35.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032050709999999999 2024-09-16T03:38:35.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068026+0000", 2024-09-16T03:38:35.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7066000000000001e-05 2024-09-16T03:38:35.381 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.381 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.381 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.381 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1924 4.4 e73/50 4:239dbd85:::smithi15736696-2:head v 73'20, mlcod=62'17)", 2024-09-16T03:38:35.381 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:59.965625+0000", 2024-09-16T03:38:35.381 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.361298574999999, 2024-09-16T03:38:35.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102804671, 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.965625+0000", 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.965625+0000", 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.965627+0000", 2024-09-16T03:38:35.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5179999999999999e-06 2024-09-16T03:38:35.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.966256+0000", 2024-09-16T03:38:35.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00062937300000000002 2024-09-16T03:38:35.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.966274+0000", 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8257000000000002e-05 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.966284+0000", 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4299999999999995e-06 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.066012+0000", 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099728368999999997 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.066028+0000", 2024-09-16T03:38:35.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6512e-05 2024-09-16T03:38:35.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068405+0000", 2024-09-16T03:38:35.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023766740000000001 2024-09-16T03:38:35.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068430+0000", 2024-09-16T03:38:35.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4538000000000001e-05 2024-09-16T03:38:35.386 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.386 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.386 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.386 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:106 3.3 e73/31 3:dbb0d6f3:::b417679517d63919e63cf420e20217e40570d74b9322ef8b960ef314107040f0:head v 73'120, mlcod=73'116)", 2024-09-16T03:38:35.386 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:35.386 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 35.048821068000002, 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.1035437, 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:35.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:35.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4900000000000004e-07 2024-09-16T03:38:35.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278120+0000", 2024-09-16T03:38:35.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6214000000000001e-05 2024-09-16T03:38:35.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278129+0000", 2024-09-16T03:38:35.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.115e-06 2024-09-16T03:38:35.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278138+0000", 2024-09-16T03:38:35.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.329e-06 2024-09-16T03:38:35.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.378298+0000", 2024-09-16T03:38:35.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100159945 2024-09-16T03:38:35.390 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.378320+0000", 2024-09-16T03:38:35.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1810999999999998e-05 2024-09-16T03:38:35.390 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.381607+0000", 2024-09-16T03:38:35.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032868210000000001 2024-09-16T03:38:35.390 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.381646+0000", 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9515999999999999e-05 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2145 4.4 e77/50 4:239dbd85:::smithi15736696-2:head v 77'22, mlcod=73'20)", 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.410542+0000", 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.916381270999999, 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10336585099999999, 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.391 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.392 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.410542+0000", 2024-09-16T03:38:35.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.410542+0000", 2024-09-16T03:38:35.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.393 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.410543+0000", 2024-09-16T03:38:35.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0599999999999999e-07 2024-09-16T03:38:35.393 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.393 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.393 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.393 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.411160+0000", 2024-09-16T03:38:35.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061733000000000003 2024-09-16T03:38:35.393 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.393 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.393 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.393 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.411172+0000", 2024-09-16T03:38:35.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2153e-05 2024-09-16T03:38:35.393 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.394 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.394 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.394 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511308+0000", 2024-09-16T03:38:35.394 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10013572900000001 2024-09-16T03:38:35.394 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.394 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.394 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.394 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511607+0000", 2024-09-16T03:38:35.394 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029879100000000002 2024-09-16T03:38:35.394 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.394 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.394 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.395 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511647+0000", 2024-09-16T03:38:35.395 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0241999999999999e-05 2024-09-16T03:38:35.395 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.395 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.395 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.395 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.513870+0000", 2024-09-16T03:38:35.395 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022230499999999999 2024-09-16T03:38:35.395 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.395 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.395 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.395 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.513908+0000", 2024-09-16T03:38:35.395 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8349999999999997e-05 2024-09-16T03:38:35.395 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.396 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.396 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.396 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.396 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.396 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2152 4.11 4:89839b7e:::smithi15736696-12:head [read 0~2487436,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:38:35.396 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:38:35.396 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.815349588, 2024-09-16T03:38:35.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10362692900000001, 2024-09-16T03:38:35.396 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.396 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.396 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:35.396 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2152 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:38:35.397 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511576+0000", 2024-09-16T03:38:35.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:38:35.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511584+0000", 2024-09-16T03:38:35.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5939999999999994e-06 2024-09-16T03:38:35.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511612+0000", 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7855e-05 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.611754+0000", 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100141641 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613255+0000", 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001500744 2024-09-16T03:38:35.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.400 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.400 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613366+0000", 2024-09-16T03:38:35.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000111088 2024-09-16T03:38:35.400 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.400 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.400 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.615201+0000", 2024-09-16T03:38:35.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0018353009999999999 2024-09-16T03:38:35.400 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.400 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.400 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.400 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.401 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2316 4.11 4:89839b7e:::smithi15736696-12:head [read 0~2487436,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:35.401 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:11.770262+0000", 2024-09-16T03:38:35.401 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 23.556661284, 2024-09-16T03:38:35.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10250772399999999, 2024-09-16T03:38:35.401 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.401 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.401 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:35.401 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:35.401 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:35.401 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:35.401 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2316 2024-09-16T03:38:35.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.402 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770262+0000", 2024-09-16T03:38:35.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770262+0000", 2024-09-16T03:38:35.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770265+0000", 2024-09-16T03:38:35.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8930000000000001e-06 2024-09-16T03:38:35.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770275+0000", 2024-09-16T03:38:35.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3810000000000005e-06 2024-09-16T03:38:35.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770287+0000", 2024-09-16T03:38:35.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2889e-05 2024-09-16T03:38:35.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.870382+0000", 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100094409 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.870584+0000", 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020166199999999999 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.870806+0000", 2024-09-16T03:38:35.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022220200000000001 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.872770+0000", 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019642879999999998 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2445 4.11 4:89839b7e:::smithi15736696-12:head [sparse-read 1818507~599510,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.882760166000001, 2024-09-16T03:38:35.405 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200613392, 2024-09-16T03:38:35.406 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.406 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.406 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:35.406 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:35.406 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:35.406 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:35.406 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2445 2024-09-16T03:38:35.406 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.406 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.406 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.406 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.406 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:38:35.406 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.407 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:38:35.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.407 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444167+0000", 2024-09-16T03:38:35.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6650000000000001e-06 2024-09-16T03:38:35.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444182+0000", 2024-09-16T03:38:35.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4678000000000001e-05 2024-09-16T03:38:35.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.408 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444207+0000", 2024-09-16T03:38:35.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4768000000000001e-05 2024-09-16T03:38:35.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.408 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.544359+0000", 2024-09-16T03:38:35.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100152804 2024-09-16T03:38:35.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.409 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.409 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.409 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644561+0000", 2024-09-16T03:38:35.409 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10020214500000001 2024-09-16T03:38:35.409 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.409 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.409 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.409 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644624+0000", 2024-09-16T03:38:35.409 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2444e-05 2024-09-16T03:38:35.409 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.409 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.409 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.409 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644777+0000", 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000152888 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2629 3.3 e89/31 3:e23364e2:::e447860b11e9f9fde68b75cb1d2f204a74952889ec51a2417ef18c1427ff9a70:head v 89'173, mlcod=87'171)", 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.875017473, 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10848095100000001, 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.410 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451907+0000", 2024-09-16T03:38:35.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6599999999999996e-07 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456443+0000", 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0045365509999999998 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456470+0000", 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6897e-05 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456496+0000", 2024-09-16T03:38:35.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.618e-05 2024-09-16T03:38:35.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.413 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556666+0000", 2024-09-16T03:38:35.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100169678 2024-09-16T03:38:35.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.413 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556686+0000", 2024-09-16T03:38:35.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9740000000000001e-05 2024-09-16T03:38:35.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560345+0000", 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003658821 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560387+0000", 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2518000000000002e-05 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.415 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2692 4.11 4:89839b7e:::smithi15736696-12:head [read 1367786~1119650,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:35.415 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.869603+0000", 2024-09-16T03:38:35.415 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.457320168000001, 2024-09-16T03:38:35.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10206304200000001, 2024-09-16T03:38:35.415 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.415 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.415 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:35.415 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:35.415 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:35.415 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:35.415 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2692 2024-09-16T03:38:35.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.415 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869603+0000", 2024-09-16T03:38:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869603+0000", 2024-09-16T03:38:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869604+0000", 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7100000000000001e-07 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869608+0000", 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3789999999999999e-06 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869616+0000", 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0639999999999996e-06 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869622+0000", 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1630000000000001e-06 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.971042+0000", 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10142073 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.971125+0000", 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2360000000000004e-05 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.419 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.419 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.971666+0000", 2024-09-16T03:38:35.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054167500000000003 2024-09-16T03:38:35.419 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.419 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.419 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.419 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.419 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.419 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2693 4.11 4:89839b7e:::smithi15736696-12:head [sparse-read 990224~701806] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:35.419 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.869631+0000", 2024-09-16T03:38:35.419 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.457292278000001, 2024-09-16T03:38:35.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102196038, 2024-09-16T03:38:35.419 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.420 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.420 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:35.420 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:35.420 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:35.420 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:35.420 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2693 2024-09-16T03:38:35.420 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.420 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.420 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.420 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.420 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869631+0000", 2024-09-16T03:38:35.420 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.420 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869631+0000", 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869632+0000", 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0200000000000002e-07 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869635+0000", 2024-09-16T03:38:35.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.518e-06 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869644+0000", 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6710000000000005e-06 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.869650+0000", 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3759999999999996e-06 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.422 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.971685+0000", 2024-09-16T03:38:35.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10203509199999999 2024-09-16T03:38:35.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.971724+0000", 2024-09-16T03:38:35.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8795999999999999e-05 2024-09-16T03:38:35.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.971827+0000", 2024-09-16T03:38:35.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000103083 2024-09-16T03:38:35.423 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.423 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.424 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.424 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.424 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:209 3.3 e99/31 3:f2e0c7d0:::6bf49184c86dbc717b1f919afcf6cb40a4ca9c2465eb100f7ba59687540c40ed:head v 99'201, mlcod=98'194)", 2024-09-16T03:38:35.424 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:38:35.424 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.1208872759999999, 2024-09-16T03:38:35.424 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200559603, 2024-09-16T03:38:35.424 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.424 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206037+0000", 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9999999999999999e-07 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206039+0000", 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2579999999999998e-06 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206041+0000", 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.542e-06 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306162+0000", 2024-09-16T03:38:35.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100120348 2024-09-16T03:38:35.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306250+0000", 2024-09-16T03:38:35.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:38:35.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.428 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.428 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.428 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306268+0000", 2024-09-16T03:38:35.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8127000000000001e-05 2024-09-16T03:38:35.428 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.428 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.428 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.428 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.406549+0000", 2024-09-16T03:38:35.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10028121199999999 2024-09-16T03:38:35.428 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.429 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.406596+0000", 2024-09-16T03:38:35.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6700999999999999e-05 2024-09-16T03:38:35.429 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.429 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.429 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.429 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.429 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.429 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:35.430 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:35.379+0000 7f3d40708640 -1 osd.12 102 handle_osd_map injecting map crc failure 2024-09-16T03:38:35.431 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:35.429+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:35.431 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:38:35.531 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:35.529+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:35.576 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:35.576 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:35.576 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:35.576 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:35.585 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:38:35.638 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:35.637+0000 7eff7884b640 -1 osd.3 102 handle_osd_map injecting map crc failure 2024-09-16T03:38:35.669 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:35.667+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:35.729 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:35.729 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:35.729 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:35.729 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:35.729 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:35.733 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:35.730+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:35.737 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:38:35.833 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:35.830+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:35.883 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:35.884 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:35.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:35.884 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:35.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.884 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:19 3.1 3:9aba8983:::smithi15736696-5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:35.884 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:38:35.884 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 79.462434599000005, 2024-09-16T03:38:35.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30153978799999998, 2024-09-16T03:38:35.884 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.884 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.884 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.885 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:38:35.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418432+0000", 2024-09-16T03:38:35.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418452+0000", 2024-09-16T03:38:35.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0363000000000001e-05 2024-09-16T03:38:35.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418470+0000", 2024-09-16T03:38:35.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7506000000000001e-05 2024-09-16T03:38:35.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418504+0000", 2024-09-16T03:38:35.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4542999999999998e-05 2024-09-16T03:38:35.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418522+0000", 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8070000000000001e-05 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418662+0000", 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013946299999999999 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418826+0000", 2024-09-16T03:38:35.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016469299999999999 2024-09-16T03:38:35.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:35.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.418989+0000", 2024-09-16T03:38:35.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000162759 2024-09-16T03:38:35.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.419750+0000", 2024-09-16T03:38:35.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076104299999999998 2024-09-16T03:38:35.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.419865+0000", 2024-09-16T03:38:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114338 2024-09-16T03:38:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719876+0000", 2024-09-16T03:38:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30001127300000002 2024-09-16T03:38:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719920+0000", 2024-09-16T03:38:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4549999999999999e-05 2024-09-16T03:38:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:16.719972+0000", 2024-09-16T03:38:35.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1187000000000003e-05 2024-09-16T03:38:35.890 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.890 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.890 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.890 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:779 3.1 3:abdaa5c9:::2782d12a88f9842f7f754fca32c4222cf10fb780cd136830d32faaa700e687ba:head [write 0~45056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:35.890 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:38:35.890 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.394286283, 2024-09-16T03:38:35.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30397390600000002, 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 779 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:38:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486583+0000", 2024-09-16T03:38:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.762e-06 2024-09-16T03:38:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486649+0000", 2024-09-16T03:38:35.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6143999999999996e-05 2024-09-16T03:38:35.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486669+0000", 2024-09-16T03:38:35.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9837e-05 2024-09-16T03:38:35.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486681+0000", 2024-09-16T03:38:35.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2194999999999999e-05 2024-09-16T03:38:35.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486746+0000", 2024-09-16T03:38:35.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5480000000000003e-05 2024-09-16T03:38:35.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-09-16T03:38:35.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486752+0000", 2024-09-16T03:38:35.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4659999999999998e-06 2024-09-16T03:38:35.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588316+0000", 2024-09-16T03:38:35.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10156393399999999 2024-09-16T03:38:35.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588539+0000", 2024-09-16T03:38:35.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022308300000000001 2024-09-16T03:38:35.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:35.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588647+0000", 2024-09-16T03:38:35.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000108047 2024-09-16T03:38:35.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.589516+0000", 2024-09-16T03:38:35.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086926899999999999 2024-09-16T03:38:35.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.789989+0000", 2024-09-16T03:38:35.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20047296100000001 2024-09-16T03:38:35.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790478+0000", 2024-09-16T03:38:35.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048882099999999996 2024-09-16T03:38:35.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790518+0000", 2024-09-16T03:38:35.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0343000000000003e-05 2024-09-16T03:38:35.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790554+0000", 2024-09-16T03:38:35.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5564e-05 2024-09-16T03:38:35.899 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.899 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.899 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.899 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:38 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:35.899 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:38:35.899 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.042459093000005, 2024-09-16T03:38:35.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30216061300000002, 2024-09-16T03:38:35.899 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.899 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.900 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.900 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:38:35.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838407+0000", 2024-09-16T03:38:35.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.901 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838411+0000", 2024-09-16T03:38:35.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3780000000000001e-06 2024-09-16T03:38:35.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.901 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838422+0000", 2024-09-16T03:38:35.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1223000000000001e-05 2024-09-16T03:38:35.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838450+0000", 2024-09-16T03:38:35.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7878999999999999e-05 2024-09-16T03:38:35.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838464+0000", 2024-09-16T03:38:35.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3811e-05 2024-09-16T03:38:35.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838539+0000", 2024-09-16T03:38:35.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5685000000000005e-05 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838604+0000", 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4980999999999993e-05 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.838732+0000", 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012821100000000001 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839340+0000", 2024-09-16T03:38:35.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060762399999999997 2024-09-16T03:38:35.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840716+0000", 2024-09-16T03:38:35.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001375692 2024-09-16T03:38:35.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140491+0000", 2024-09-16T03:38:35.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29977497600000003 2024-09-16T03:38:35.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140534+0000", 2024-09-16T03:38:35.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3411999999999997e-05 2024-09-16T03:38:35.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.140568+0000", 2024-09-16T03:38:35.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3741000000000001e-05 2024-09-16T03:38:35.905 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.905 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.905 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:39 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:38:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.041168874999997, 2024-09-16T03:38:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40113680400000001, 2024-09-16T03:38:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:38:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:38:35.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839700+0000", 2024-09-16T03:38:35.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.571e-06 2024-09-16T03:38:35.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839704+0000", 2024-09-16T03:38:35.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8399999999999997e-06 2024-09-16T03:38:35.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839712+0000", 2024-09-16T03:38:35.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9990000000000008e-06 2024-09-16T03:38:35.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839720+0000", 2024-09-16T03:38:35.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0099999999999995e-06 2024-09-16T03:38:35.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839783+0000", 2024-09-16T03:38:35.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3014000000000003e-05 2024-09-16T03:38:35.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839870+0000", 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7392999999999995e-05 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840049+0000", 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017825800000000001 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840874+0000", 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082532099999999995 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.040285+0000", 2024-09-16T03:38:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19941088400000001 2024-09-16T03:38:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240748+0000", 2024-09-16T03:38:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20046322699999999 2024-09-16T03:38:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240786+0000", 2024-09-16T03:38:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8158000000000003e-05 2024-09-16T03:38:35.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240834+0000", 2024-09-16T03:38:35.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8128999999999999e-05 2024-09-16T03:38:35.912 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.912 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.912 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.912 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:983 3.1 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head [write 0~2138112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:38:35.912 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:38:35.912 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 65.817265934999995, 2024-09-16T03:38:35.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30819799799999997, 2024-09-16T03:38:35.913 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.913 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.913 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.913 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:35.913 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:35.913 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:35.913 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 983 2024-09-16T03:38:35.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.913 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.913 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:38:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:38:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063610+0000", 2024-09-16T03:38:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2990000000000002e-06 2024-09-16T03:38:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065735+0000", 2024-09-16T03:38:35.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021257799999999999 2024-09-16T03:38:35.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065752+0000", 2024-09-16T03:38:35.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6895999999999999e-05 2024-09-16T03:38:35.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065762+0000", 2024-09-16T03:38:35.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6830000000000003e-06 2024-09-16T03:38:35.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065871+0000", 2024-09-16T03:38:35.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010904900000000001 2024-09-16T03:38:35.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066005+0000", 2024-09-16T03:38:35.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013369799999999999 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066135+0000", 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000130481 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.069822+0000", 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036863759999999999 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272370+0000", 2024-09-16T03:38:35.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20254802599999999 2024-09-16T03:38:35.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371671+0000", 2024-09-16T03:38:35.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099300893000000001 2024-09-16T03:38:35.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371754+0000", 2024-09-16T03:38:35.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3075e-05 2024-09-16T03:38:35.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371798+0000", 2024-09-16T03:38:35.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4742e-05 2024-09-16T03:38:35.920 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.920 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.920 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.920 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1515 3.1 3:9c9e970c:::7249cb65b52cadf4fe8f4ec923c7e548d1de7d9628823acd2c8de19121f1c1ce:head [write 0~1094656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:38:35.920 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:38:35.920 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.931140198000001, 2024-09-16T03:38:35.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.305958065, 2024-09-16T03:38:35.920 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.920 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.920 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.921 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:35.921 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:35.921 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:35.921 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1515 2024-09-16T03:38:35.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.921 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:38:35.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949729+0000", 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.785e-06 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950933+0000", 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001204341 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950950+0000", 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6838e-05 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950960+0000", 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0316000000000001e-05 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951065+0000", 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104128 2024-09-16T03:38:35.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951202+0000", 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013706600000000001 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951350+0000", 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000148197 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.953857+0000", 2024-09-16T03:38:35.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0025067150000000001 2024-09-16T03:38:35.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.958049+0000", 2024-09-16T03:38:35.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0041924650000000003 2024-09-16T03:38:35.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255547+0000", 2024-09-16T03:38:35.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29749841900000001 2024-09-16T03:38:35.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255649+0000", 2024-09-16T03:38:35.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101513 2024-09-16T03:38:35.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255684+0000", 2024-09-16T03:38:35.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5281999999999999e-05 2024-09-16T03:38:35.927 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.927 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.927 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.927 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1704 3.1 3:b6a94245:::f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99:head [write 0~453120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:35.927 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:38:35.927 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 45.088028842999996, 2024-09-16T03:38:35.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40403313699999999, 2024-09-16T03:38:35.928 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.928 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.928 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.928 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:35.928 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:35.928 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:35.928 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1704 2024-09-16T03:38:35.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.928 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:38:35.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:38:35.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792841+0000", 2024-09-16T03:38:35.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.433e-06 2024-09-16T03:38:35.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.931 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793329+0000", 2024-09-16T03:38:35.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048839200000000004 2024-09-16T03:38:35.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.931 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.931 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793345+0000", 2024-09-16T03:38:35.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6079e-05 2024-09-16T03:38:35.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.931 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.931 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793354+0000", 2024-09-16T03:38:35.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.32e-06 2024-09-16T03:38:35.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793418+0000", 2024-09-16T03:38:35.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3944999999999998e-05 2024-09-16T03:38:35.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793549+0000", 2024-09-16T03:38:35.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013159900000000001 2024-09-16T03:38:35.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:35.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793666+0000", 2024-09-16T03:38:35.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116532 2024-09-16T03:38:35.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.797707+0000", 2024-09-16T03:38:35.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040409629999999998 2024-09-16T03:38:35.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897288+0000", 2024-09-16T03:38:35.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099581449000000002 2024-09-16T03:38:35.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196775+0000", 2024-09-16T03:38:35.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948696800000002 2024-09-16T03:38:35.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196814+0000", 2024-09-16T03:38:35.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8336e-05 2024-09-16T03:38:35.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196871+0000", 2024-09-16T03:38:35.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7120999999999997e-05 2024-09-16T03:38:35.936 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.936 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.936 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.936 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:84 3.1 3:b24f289d:::e84532c28f467d2f2a6e2cf44904fa0d4a555e1304bbe1b25db645af42152bd7:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:35.936 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:38:35.936 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 45.079256903999998, 2024-09-16T03:38:35.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.395455581, 2024-09-16T03:38:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:38:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:38:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801613+0000", 2024-09-16T03:38:35.938 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9559999999999999e-06 2024-09-16T03:38:35.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.939 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802117+0000", 2024-09-16T03:38:35.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050401600000000003 2024-09-16T03:38:35.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802141+0000", 2024-09-16T03:38:35.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3618e-05 2024-09-16T03:38:35.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802155+0000", 2024-09-16T03:38:35.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4358e-05 2024-09-16T03:38:35.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802229+0000", 2024-09-16T03:38:35.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3930999999999994e-05 2024-09-16T03:38:35.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802355+0000", 2024-09-16T03:38:35.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012527299999999999 2024-09-16T03:38:35.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:35.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802567+0000", 2024-09-16T03:38:35.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021250199999999999 2024-09-16T03:38:35.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.944 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.944 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.944 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806218+0000", 2024-09-16T03:38:35.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036512189999999998 2024-09-16T03:38:35.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.945 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.945 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897323+0000", 2024-09-16T03:38:35.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091104892000000007 2024-09-16T03:38:35.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196959+0000", 2024-09-16T03:38:35.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299635971 2024-09-16T03:38:35.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197021+0000", 2024-09-16T03:38:35.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1938000000000002e-05 2024-09-16T03:38:35.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197065+0000", 2024-09-16T03:38:35.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3906999999999998e-05 2024-09-16T03:38:35.948 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.948 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.948 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.949 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:85 3.1 3:b380f269:::e65351e8a65a0b1a517e3b1a85e04b84c0ab7cfa6655f11fb0da31f56b4dd9af:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:35.949 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:38:35.949 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 45.078678007000001, 2024-09-16T03:38:35.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39502727300000001, 2024-09-16T03:38:35.949 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.950 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.950 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.950 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:38:35.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.951 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.951 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:38:35.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.952 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.952 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.952 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802190+0000", 2024-09-16T03:38:35.952 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2419999999999999e-06 2024-09-16T03:38:35.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.953 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.953 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.953 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802464+0000", 2024-09-16T03:38:35.953 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027463699999999999 2024-09-16T03:38:35.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.953 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802478+0000", 2024-09-16T03:38:35.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3375e-05 2024-09-16T03:38:35.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802484+0000", 2024-09-16T03:38:35.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0990000000000004e-06 2024-09-16T03:38:35.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803475+0000", 2024-09-16T03:38:35.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00099113099999999991 2024-09-16T03:38:35.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803651+0000", 2024-09-16T03:38:35.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017654300000000001 2024-09-16T03:38:35.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:35.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803923+0000", 2024-09-16T03:38:35.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000271842 2024-09-16T03:38:35.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.958 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.958 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.958 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806228+0000", 2024-09-16T03:38:35.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023049590000000001 2024-09-16T03:38:35.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.959 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.959 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.959 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897563+0000", 2024-09-16T03:38:35.959 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091334724000000006 2024-09-16T03:38:35.959 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.959 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.960 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197123+0000", 2024-09-16T03:38:35.960 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299559992 2024-09-16T03:38:35.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197177+0000", 2024-09-16T03:38:35.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4447999999999998e-05 2024-09-16T03:38:35.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197216+0000", 2024-09-16T03:38:35.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8281000000000002e-05 2024-09-16T03:38:35.962 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.962 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.962 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.963 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:86 3.1 3:b3acb53e:::e951eb69ce5151a3f54107fa22c70f80729eff845da1613447a91536a24046e8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:35.963 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:38:35.963 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 45.078370911999997, 2024-09-16T03:38:35.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.394851071, 2024-09-16T03:38:35.963 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.963 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.964 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.964 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:38:35.965 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.965 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.965 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.965 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:38:35.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.969 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802496+0000", 2024-09-16T03:38:35.969 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:38:35.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.969 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.969 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802782+0000", 2024-09-16T03:38:35.970 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00028523600000000002 2024-09-16T03:38:35.970 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.970 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.970 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.970 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802796+0000", 2024-09-16T03:38:35.970 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4594e-05 2024-09-16T03:38:35.971 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.971 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.971 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.971 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802804+0000", 2024-09-16T03:38:35.971 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3790000000000006e-06 2024-09-16T03:38:35.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804787+0000", 2024-09-16T03:38:35.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019825060000000002 2024-09-16T03:38:35.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.973 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804895+0000", 2024-09-16T03:38:35.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010813499999999999 2024-09-16T03:38:35.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:35.974 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.805046+0000", 2024-09-16T03:38:35.974 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150972 2024-09-16T03:38:35.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.975 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806681+0000", 2024-09-16T03:38:35.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016353839999999999 2024-09-16T03:38:35.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.975 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.975 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.005712+0000", 2024-09-16T03:38:35.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19903093199999999 2024-09-16T03:38:35.976 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.976 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197278+0000", 2024-09-16T03:38:35.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.191565295 2024-09-16T03:38:35.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.977 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.977 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197315+0000", 2024-09-16T03:38:35.977 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6946000000000002e-05 2024-09-16T03:38:35.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197347+0000", 2024-09-16T03:38:35.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1863000000000003e-05 2024-09-16T03:38:35.978 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.979 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.979 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.979 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:99 3.1 3:9f9f4ec5:::smithi15736696-22 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e71)", 2024-09-16T03:38:35.979 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:56.682569+0000", 2024-09-16T03:38:35.980 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 39.198297113999999, 2024-09-16T03:38:35.980 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30291003799999999, 2024-09-16T03:38:35.980 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.980 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.980 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.980 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.980 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.980 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682569+0000", 2024-09-16T03:38:35.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.981 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.981 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682569+0000", 2024-09-16T03:38:35.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682570+0000", 2024-09-16T03:38:35.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7000000000000004e-07 2024-09-16T03:38:35.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682574+0000", 2024-09-16T03:38:35.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3660000000000002e-06 2024-09-16T03:38:35.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682589+0000", 2024-09-16T03:38:35.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.49e-05 2024-09-16T03:38:35.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.682596+0000", 2024-09-16T03:38:35.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0589999999999997e-06 2024-09-16T03:38:35.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.783627+0000", 2024-09-16T03:38:35.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101030406 2024-09-16T03:38:35.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.783680+0000", 2024-09-16T03:38:35.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3562e-05 2024-09-16T03:38:35.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:35.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.783780+0000", 2024-09-16T03:38:35.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100028 2024-09-16T03:38:35.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.884772+0000", 2024-09-16T03:38:35.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10099123 2024-09-16T03:38:35.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.885096+0000", 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00032439900000000001 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.985281+0000", 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100184936 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.985395+0000", 2024-09-16T03:38:35.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114166 2024-09-16T03:38:35.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.987 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.987 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.987 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.985479+0000", 2024-09-16T03:38:35.987 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4215999999999995e-05 2024-09-16T03:38:35.987 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.987 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.987 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.987 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.987 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:140 3.1 3:845e3b2c:::21d7b947791b34f7e3fcd39ad119c9f923ad2f3cc7a703501eac7a68189d5437:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:35.987 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:38:35.987 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.766191658, 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39914108999999998, 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:38:35.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.989 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114677+0000", 2024-09-16T03:38:35.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1610000000000001e-06 2024-09-16T03:38:35.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.989 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114684+0000", 2024-09-16T03:38:35.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3150000000000003e-06 2024-09-16T03:38:35.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114699+0000", 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4996e-05 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114705+0000", 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0580000000000003e-06 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.214862+0000", 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015700199999999 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.214922+0000", 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0053999999999998e-05 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215023+0000", 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100432 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.216100+0000", 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010776819999999999 2024-09-16T03:38:35.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.499919+0000", 2024-09-16T03:38:35.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.28381845900000002 2024-09-16T03:38:35.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513755+0000", 2024-09-16T03:38:35.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013836477 2024-09-16T03:38:35.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:35.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513790+0000", 2024-09-16T03:38:35.993 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4295999999999998e-05 2024-09-16T03:38:35.993 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.993 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.993 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:35.993 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513816+0000", 2024-09-16T03:38:35.993 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6157999999999999e-05 2024-09-16T03:38:35.993 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.993 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:35.993 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:35.993 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:141 3.1 3:9387c678:::99a87d3b555b50e905a1ff55d99eb4aa025bd0070f07a4910d03cf1af95bbc44:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.766140961000001, 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50160030300000003, 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:35.995 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:38:35.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:35.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:35.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114726+0000", 2024-09-16T03:38:35.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4399999999999999e-07 2024-09-16T03:38:35.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:35.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114730+0000", 2024-09-16T03:38:35.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1479999999999999e-06 2024-09-16T03:38:35.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:35.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114738+0000", 2024-09-16T03:38:35.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3200000000000002e-06 2024-09-16T03:38:35.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.997 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:35.997 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114743+0000", 2024-09-16T03:38:35.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.891e-06 2024-09-16T03:38:35.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.997 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:35.997 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215320+0000", 2024-09-16T03:38:35.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100576463 2024-09-16T03:38:35.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.998 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:35.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215376+0000", 2024-09-16T03:38:35.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6310999999999997e-05 2024-09-16T03:38:35.998 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.998 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:35.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215473+0000", 2024-09-16T03:38:35.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7103999999999997e-05 2024-09-16T03:38:35.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:35.999 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.216108+0000", 2024-09-16T03:38:35.999 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00063422800000000003 2024-09-16T03:38:35.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:35.999 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513851+0000", 2024-09-16T03:38:35.999 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29774376400000002 2024-09-16T03:38:35.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:35.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:35.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616086+0000", 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10223441 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616240+0000", 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015453799999999999 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616326+0000", 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5382000000000003e-05 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.000 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.001 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.001 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:142 3.1 3:a93b7b74:::70cd8f47898d416018aded7de9c29e0fe002b2336cb6cfabbf3eb6238766c3b1:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:36.001 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:38:36.001 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.766111574, 2024-09-16T03:38:36.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50166088799999997, 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.002 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3600000000000003e-07 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114762+0000", 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6039999999999998e-06 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114769+0000", 2024-09-16T03:38:36.003 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5939999999999999e-06 2024-09-16T03:38:36.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114774+0000", 2024-09-16T03:38:36.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2179999999999998e-06 2024-09-16T03:38:36.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215687+0000", 2024-09-16T03:38:36.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100912974 2024-09-16T03:38:36.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.005 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.005 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.005 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215733+0000", 2024-09-16T03:38:36.005 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6419999999999999e-05 2024-09-16T03:38:36.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.005 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.005 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:36.005 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215836+0000", 2024-09-16T03:38:36.005 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102981 2024-09-16T03:38:36.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.005 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316766+0000", 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100929693 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513877+0000", 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19711067800000001 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616358+0000", 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10248180699999999 2024-09-16T03:38:36.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616391+0000", 2024-09-16T03:38:36.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2551000000000001e-05 2024-09-16T03:38:36.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616416+0000", 2024-09-16T03:38:36.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4732e-05 2024-09-16T03:38:36.007 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.007 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.007 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.008 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:143 3.1 3:ac848b49:::adc4f0c8a4d0bc7e52f9d5c3b4981447737a7dbe58b8313a369b04383e10893f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:36.008 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:38:36.008 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.766081568000001, 2024-09-16T03:38:36.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60333394699999998, 2024-09-16T03:38:36.008 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.008 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.008 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.008 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:38:36.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7899999999999999e-07 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114788+0000", 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4569999999999999e-06 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114792+0000", 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5730000000000001e-06 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114796+0000", 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5449999999999999e-06 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316237+0000", 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20144099300000001 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316417+0000", 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017981099999999999 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316564+0000", 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014693700000000001 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316993+0000", 2024-09-16T03:38:36.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042985499999999998 2024-09-16T03:38:36.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.717965+0000", 2024-09-16T03:38:36.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40097123699999998 2024-09-16T03:38:36.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718005+0000", 2024-09-16T03:38:36.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0083000000000002e-05 2024-09-16T03:38:36.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718064+0000", 2024-09-16T03:38:36.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9020000000000001e-05 2024-09-16T03:38:36.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718119+0000", 2024-09-16T03:38:36.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4957000000000003e-05 2024-09-16T03:38:36.013 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.013 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.013 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:149 3.1 3:8eb88b02:::fa73f0ab3a7395fcfc19219899ff8c0a4a3c96caf8327f9da9656e2450435b27:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 18.75217292, 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.348655887, 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:38:36.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128698+0000", 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.527e-06 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128714+0000", 2024-09-16T03:38:36.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6019e-05 2024-09-16T03:38:36.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128757+0000", 2024-09-16T03:38:36.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3318000000000001e-05 2024-09-16T03:38:36.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128764+0000", 2024-09-16T03:38:36.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.28e-06 2024-09-16T03:38:36.017 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.017 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.017 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.228916+0000", 2024-09-16T03:38:36.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015252199999999 2024-09-16T03:38:36.017 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.017 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.017 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.228976+0000", 2024-09-16T03:38:36.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0390999999999998e-05 2024-09-16T03:38:36.017 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.229077+0000", 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010053 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.274759+0000", 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.045682404000000003 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.330314+0000", 2024-09-16T03:38:36.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.055554916000000003 2024-09-16T03:38:36.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477196+0000", 2024-09-16T03:38:36.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14688130099999999 2024-09-16T03:38:36.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477291+0000", 2024-09-16T03:38:36.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5144999999999998e-05 2024-09-16T03:38:36.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477349+0000", 2024-09-16T03:38:36.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8533999999999999e-05 2024-09-16T03:38:36.020 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.020 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.020 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.020 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:168 3.1 3:a655680f:::65b183ceded8ac2b61dbe9ba8fb4fd4f9f3a4854c219da4322bb1e81ebb7e2e5:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:36.020 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.648756+0000", 2024-09-16T03:38:36.020 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.232109897999999, 2024-09-16T03:38:36.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30591469500000001, 2024-09-16T03:38:36.021 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.021 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.021 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.021 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648756+0000", 2024-09-16T03:38:36.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648756+0000", 2024-09-16T03:38:36.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648757+0000", 2024-09-16T03:38:36.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7299999999999995e-07 2024-09-16T03:38:36.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648799+0000", 2024-09-16T03:38:36.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.227e-05 2024-09-16T03:38:36.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648805+0000", 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4790000000000004e-06 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648809+0000", 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7890000000000001e-06 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.650787+0000", 2024-09-16T03:38:36.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019778700000000001 2024-09-16T03:38:36.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.650914+0000", 2024-09-16T03:38:36.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012707899999999999 2024-09-16T03:38:36.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:36.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651123+0000", 2024-09-16T03:38:36.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020929100000000001 2024-09-16T03:38:36.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.652287+0000", 2024-09-16T03:38:36.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011644859999999999 2024-09-16T03:38:36.025 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.653716+0000", 2024-09-16T03:38:36.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014290889999999999 2024-09-16T03:38:36.025 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954577+0000", 2024-09-16T03:38:36.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30086063800000001 2024-09-16T03:38:36.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.026 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.026 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954621+0000", 2024-09-16T03:38:36.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3820999999999999e-05 2024-09-16T03:38:36.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.026 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.026 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954671+0000", 2024-09-16T03:38:36.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0210000000000002e-05 2024-09-16T03:38:36.026 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.026 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.027 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.027 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:169 3.1 3:8ca52973:::737e1241b430663b0c460fc4c0e6bf0a2c97ca3f584ee4e3541536e0014b01da:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:36.027 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.648814+0000", 2024-09-16T03:38:36.027 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.232052073, 2024-09-16T03:38:36.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30599283399999999, 2024-09-16T03:38:36.027 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.027 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.027 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.028 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648814+0000", 2024-09-16T03:38:36.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648814+0000", 2024-09-16T03:38:36.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648815+0000", 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9900000000000001e-07 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648861+0000", 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6786999999999998e-05 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648867+0000", 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1830000000000002e-06 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648872+0000", 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9169999999999997e-06 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651624+0000", 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027522150000000001 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651689+0000", 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5065000000000001e-05 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651814+0000", 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012493199999999999 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.653881+0000", 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020671830000000002 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.752238+0000", 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098357336000000004 2024-09-16T03:38:36.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954726+0000", 2024-09-16T03:38:36.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20248775199999999 2024-09-16T03:38:36.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954780+0000", 2024-09-16T03:38:36.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3859000000000001e-05 2024-09-16T03:38:36.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954807+0000", 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7206e-05 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:171 3.1 3:93178f8f:::777eb2bf7c36938cba5735a17f90bb453157d7a9a42a0a6f52227fe7530ec353:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.648876+0000", 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.231990028, 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30603207999999998, 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.033 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648876+0000", 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648876+0000", 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648877+0000", 2024-09-16T03:38:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8799999999999998e-07 2024-09-16T03:38:36.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648918+0000", 2024-09-16T03:38:36.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1488e-05 2024-09-16T03:38:36.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648923+0000", 2024-09-16T03:38:36.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0320000000000003e-06 2024-09-16T03:38:36.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648926+0000", 2024-09-16T03:38:36.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1690000000000001e-06 2024-09-16T03:38:36.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.652727+0000", 2024-09-16T03:38:36.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038005909999999999 2024-09-16T03:38:36.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.652854+0000", 2024-09-16T03:38:36.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012661700000000001 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.653069+0000", 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021573799999999999 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.654043+0000", 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00097351999999999996 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.854187+0000", 2024-09-16T03:38:36.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20014437400000001 2024-09-16T03:38:36.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954834+0000", 2024-09-16T03:38:36.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100646293 2024-09-16T03:38:36.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954875+0000", 2024-09-16T03:38:36.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1637999999999997e-05 2024-09-16T03:38:36.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.040 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.040 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954908+0000", 2024-09-16T03:38:36.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3232000000000003e-05 2024-09-16T03:38:36.040 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.040 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.040 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.040 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:205 3.1 3:a655680f:::65b183ceded8ac2b61dbe9ba8fb4fd4f9f3a4854c219da4322bb1e81ebb7e2e5:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:38:36.040 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:32.005649+0000", 2024-09-16T03:38:36.040 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 3.8752177100000003, 2024-09-16T03:38:36.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30163399899999999, 2024-09-16T03:38:36.041 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.041 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.041 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.041 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.005649+0000", 2024-09-16T03:38:36.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.005649+0000", 2024-09-16T03:38:36.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.005649+0000", 2024-09-16T03:38:36.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0900000000000002e-07 2024-09-16T03:38:36.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.005652+0000", 2024-09-16T03:38:36.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9440000000000001e-06 2024-09-16T03:38:36.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.005657+0000", 2024-09-16T03:38:36.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.561e-06 2024-09-16T03:38:36.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.005663+0000", 2024-09-16T03:38:36.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9179999999999999e-06 2024-09-16T03:38:36.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.106426+0000", 2024-09-16T03:38:36.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100763407 2024-09-16T03:38:36.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.106498+0000", 2024-09-16T03:38:36.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2026999999999993e-05 2024-09-16T03:38:36.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:36.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.106634+0000", 2024-09-16T03:38:36.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013559800000000001 2024-09-16T03:38:36.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.207334+0000", 2024-09-16T03:38:36.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100700002 2024-09-16T03:38:36.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.209664+0000", 2024-09-16T03:38:36.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023305259999999999 2024-09-16T03:38:36.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.307220+0000", 2024-09-16T03:38:36.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.097555904999999998 2024-09-16T03:38:36.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.307253+0000", 2024-09-16T03:38:36.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2830000000000002e-05 2024-09-16T03:38:36.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.307283+0000", 2024-09-16T03:38:36.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9771999999999999e-05 2024-09-16T03:38:36.050 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.051 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.051 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.051 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.051 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.051 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:36.053 INFO:tasks.rados.rados.0.smithi157.stdout:1381: set_chunk oid 34 offset: 432640 length: 1679360 target oid 6a9007270aa32b63b7f7e5355768bf791fa2d1832cb8adf59666d35d1b037cad offset: 0 snap -1 2024-09-16T03:38:36.053 INFO:tasks.rados.rados.0.smithi157.stdout:1382: write (excl) oid 50 current snap is 0 2024-09-16T03:38:36.053 INFO:tasks.rados.rados.0.smithi157.stdout:1377: expect (ObjNum 248 snap 0 seq_num 248) 2024-09-16T03:38:36.054 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:38:36.054 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:36.050+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:36.151 INFO:tasks.rados.rados.0.smithi157.stdout:1374: finishing write tid 3 to smithi15736696-45 2024-09-16T03:38:36.151 INFO:tasks.rados.rados.0.smithi157.stdout:1374: finishing write tid 4 to smithi15736696-45 2024-09-16T03:38:36.151 INFO:tasks.rados.rados.0.smithi157.stdout:1374: oid 45 updating version 0 to 91 2024-09-16T03:38:36.151 INFO:tasks.rados.rados.0.smithi157.stdout:1374: oid 45 updating version 91 to 92 2024-09-16T03:38:36.151 INFO:tasks.rados.rados.0.smithi157.stdout:1374: oid 45 version 92 is already newer than 90 2024-09-16T03:38:36.151 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 92 (ObjNum 273 snap 0 seq_num 273) dirty exists 2024-09-16T03:38:36.151 INFO:tasks.rados.rados.0.smithi157.stdout:1374: left oid 45 (ObjNum 273 snap 0 seq_num 273) 2024-09-16T03:38:36.151 INFO:tasks.rados.rados.0.smithi157.stdout:1378: finishing write tid 1 to smithi15736696-42 2024-09-16T03:38:36.151 INFO:tasks.rados.rados.0.smithi157.stdout:1382: seq_num 277 ranges {655539=632681,2076218=570097} 2024-09-16T03:38:36.154 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:36.152+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:36.157 INFO:tasks.rados.rados.0.smithi157.stdout:1382: writing smithi15736696-50 from 655539 to 1288220 tid 1 2024-09-16T03:38:36.165 INFO:tasks.rados.rados.0.smithi157.stdout:1382: writing smithi15736696-50 from 2076218 to 2646315 tid 2 2024-09-16T03:38:36.165 INFO:tasks.rados.rados.0.smithi157.stdout:1374: done (6 left) 2024-09-16T03:38:36.166 INFO:tasks.rados.rados.0.smithi157.stdout:1377: done (5 left) 2024-09-16T03:38:36.166 INFO:tasks.rados.rados.0.smithi157.stdout:1383: set_chunk oid 25 target oid 2024-09-16T03:38:36.166 INFO:tasks.rados.rados.0.smithi157.stdout:1383: set_chunk oid 25 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:36.166 INFO:tasks.rados.rados.0.smithi157.stdout:1384: read oid 11 snap -1 2024-09-16T03:38:36.166 INFO:tasks.rados.rados.0.smithi157.stdout:1384: expect (ObjNum 261 snap 0 seq_num 261) 2024-09-16T03:38:36.166 INFO:tasks.rados.rados.0.smithi157.stdout:1385: read oid 17 snap -1 2024-09-16T03:38:36.166 INFO:tasks.rados.rados.0.smithi157.stdout:1385: expect (ObjNum 248 snap 0 seq_num 248) 2024-09-16T03:38:36.166 INFO:tasks.rados.rados.0.smithi157.stdout:1386: set_chunk oid 41 target oid 2024-09-16T03:38:36.201 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:36.201 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:36.201 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.201 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3092 4.20 4.6a0b5e0 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:36.201 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:36.165184+0000", 2024-09-16T03:38:36.201 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.034197558000000003, 2024-09-16T03:38:36.201 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034213003999999998, 2024-09-16T03:38:36.201 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.201 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.201 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:36.201 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.202 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.202 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.202 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3092 2024-09-16T03:38:36.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.202 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.202 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.202 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.202 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165184+0000", 2024-09-16T03:38:36.202 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.202 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.202 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.202 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.202 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165184+0000", 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165184+0000", 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9299999999999999e-07 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165186+0000", 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9819999999999998e-06 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.203 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165190+0000", 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165194+0000", 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3129999999999999e-06 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3096 4.0 4.5b5d5c40 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:36.204 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:36.165197+0000", 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.034184738999999999, 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034229796999999999, 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3096 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.205 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165197+0000", 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165197+0000", 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165197+0000", 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9700000000000002e-07 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.206 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165203+0000", 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3279999999999996e-06 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165206+0000", 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9890000000000001e-06 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165208+0000", 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5459999999999998e-06 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.207 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3097 4.0 4.5b5d5c40 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:36.165211+0000", 2024-09-16T03:38:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.034170565999999999, 2024-09-16T03:38:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034263170000000003, 2024-09-16T03:38:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "no flag points reached", 2024-09-16T03:38:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.209 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3097 2024-09-16T03:38:36.209 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.209 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.209 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165211+0000", 2024-09-16T03:38:36.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.209 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.209 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.209 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.209 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165211+0000", 2024-09-16T03:38:36.209 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.210 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165211+0000", 2024-09-16T03:38:36.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.79e-07 2024-09-16T03:38:36.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.210 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.210 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.210 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165213+0000", 2024-09-16T03:38:36.210 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7379999999999999e-06 2024-09-16T03:38:36.210 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165217+0000", 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.053e-06 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "rep_scrubmap(4.20 e100 from shard 4 PREEMPTED)", 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:36.166325+0000", 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.033056976000000002, 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033170872999999997, 2024-09-16T03:38:36.211 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.166325+0000", 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.166325+0000", 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.212 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.166326+0000", 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.099e-06 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.166330+0000", 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6210000000000003e-06 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.166342+0000", 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1541e-05 2024-09-16T03:38:36.213 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.166348+0000", 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1500000000000004e-06 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3088 4.20 4:06b6d050:::smithi15736696-50:head [write 655539~632681 in=632681b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:36.157083+0000", 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.042298898000000001, 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.042439531000000003, 2024-09-16T03:38:36.214 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3088 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.157083+0000", 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.215 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.157083+0000", 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.157085+0000", 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.26e-06 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.157738+0000", 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065297100000000002 2024-09-16T03:38:36.216 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.157751+0000", 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2683e-05 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.157775+0000", 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4397e-05 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.157818+0000", 2024-09-16T03:38:36.217 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2451999999999999e-05 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.157924+0000", 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000106743 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.158040+0000", 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000115293 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.218 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.160084+0000", 2024-09-16T03:38:36.219 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002044082 2024-09-16T03:38:36.219 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.219 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.219 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.219 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.219 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.219 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3089 4.20 4:06b6d050:::smithi15736696-50:head [write 2076218~570097 in=570097b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:36.219 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:36.164597+0000", 2024-09-16T03:38:36.219 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.034785101999999998, 2024-09-16T03:38:36.219 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034962211999999999, 2024-09-16T03:38:36.219 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.219 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.219 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3089 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.164597+0000", 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.220 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.164597+0000", 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.164599+0000", 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3099999999999999e-06 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165075+0000", 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000475613 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.221 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165089+0000", 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4146e-05 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165113+0000", 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4199e-05 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165179+0000", 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6576999999999996e-05 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.222 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.223 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.223 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165293+0000", 2024-09-16T03:38:36.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011315999999999999 2024-09-16T03:38:36.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.223 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.223 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:36.223 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165483+0000", 2024-09-16T03:38:36.223 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019021899999999999 2024-09-16T03:38:36.223 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.223 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.223 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.223 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.223 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3090 4.20 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=63b,truncate 2655539] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:36.165127+0000", 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.034255061000000003, 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034465161000000001, 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3090 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.224 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165127+0000", 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165127+0000", 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165127+0000", 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6400000000000002e-07 2024-09-16T03:38:36.225 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165130+0000", 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.774e-06 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165133+0000", 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3969999999999998e-06 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165136+0000", 2024-09-16T03:38:36.226 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.004e-06 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.167104+0000", 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001967711 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.167216+0000", 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000111821 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.167382+0000", 2024-09-16T03:38:36.227 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000166231 2024-09-16T03:38:36.228 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.228 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.228 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.228 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.228 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.228 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3091 4.20 4.a0b6d60 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:36.228 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:36.165164+0000", 2024-09-16T03:38:36.228 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.034217495000000001, 2024-09-16T03:38:36.228 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.034457844000000001, 2024-09-16T03:38:36.228 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.228 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.228 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3091 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165164+0000", 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.229 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165164+0000", 2024-09-16T03:38:36.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.230 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.230 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.230 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.230 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165165+0000", 2024-09-16T03:38:36.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8700000000000001e-07 2024-09-16T03:38:36.230 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.230 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.230 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.230 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165167+0000", 2024-09-16T03:38:36.230 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0040000000000002e-06 2024-09-16T03:38:36.230 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.230 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165175+0000", 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2009999999999999e-06 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:36.165180+0000", 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0579999999999997e-06 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 8 2024-09-16T03:38:36.231 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:36.233 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:38:36.276 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:36.274+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:36.338 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:36.338 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:36.338 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:36.338 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:36.338 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:36.346 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:38:36.350 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:36.348+0000 7ff26836a640 -1 osd.11 103 handle_osd_map injecting map crc failure 2024-09-16T03:38:36.355 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:36.352+0000 7f6528e68640 -1 osd.1 103 handle_osd_map injecting map crc failure 2024-09-16T03:38:36.355 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:36.352+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:36.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:38:36.455 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:36.454+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:36.492 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:36.492 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:36.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:416 4.0 4:023abada:::smithi15736696-17:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 76.180368625, 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60401983699999995, 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 416 2024-09-16T03:38:36.493 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.494 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.494 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:38:36.494 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.494 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.494 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:38:36.494 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.494 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.495 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309243+0000", 2024-09-16T03:38:36.495 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.178e-06 2024-09-16T03:38:36.495 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.495 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.496 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309248+0000", 2024-09-16T03:38:36.496 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:38:36.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.496 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.496 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309257+0000", 2024-09-16T03:38:36.496 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4720000000000001e-06 2024-09-16T03:38:36.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.496 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.496 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309262+0000", 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0839999999999999e-06 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309297+0000", 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4997999999999999e-05 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309367+0000", 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0124999999999997e-05 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:36.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812055+0000", 2024-09-16T03:38:36.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50268796900000001 2024-09-16T03:38:36.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.498 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812614+0000", 2024-09-16T03:38:36.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000558309 2024-09-16T03:38:36.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.498 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.813040+0000", 2024-09-16T03:38:36.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042652800000000002 2024-09-16T03:38:36.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.499 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.499 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.499 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913189+0000", 2024-09-16T03:38:36.499 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10014880399999999 2024-09-16T03:38:36.499 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.499 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.499 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.499 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913220+0000", 2024-09-16T03:38:36.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1353000000000003e-05 2024-09-16T03:38:36.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.500 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913262+0000", 2024-09-16T03:38:36.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1284e-05 2024-09-16T03:38:36.500 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.500 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.500 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.500 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:494 4.f 4:f817def9:::smithi15736696-45:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:36.501 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:38:36.501 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 74.767428890000005, 2024-09-16T03:38:36.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60451760399999999, 2024-09-16T03:38:36.501 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.501 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.501 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.501 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.501 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.502 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.502 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 494 2024-09-16T03:38:36.502 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.502 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.502 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.502 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.502 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:38:36.502 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.502 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.502 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.502 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.502 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:38:36.502 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.503 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.503 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.503 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722183+0000", 2024-09-16T03:38:36.503 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:38:36.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.503 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.503 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.503 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722189+0000", 2024-09-16T03:38:36.503 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9429999999999997e-06 2024-09-16T03:38:36.504 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.504 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.504 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.504 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722203+0000", 2024-09-16T03:38:36.504 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4367e-05 2024-09-16T03:38:36.504 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.504 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.504 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.504 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722210+0000", 2024-09-16T03:38:36.505 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:38:36.505 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.505 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.505 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.505 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822367+0000", 2024-09-16T03:38:36.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100156933 2024-09-16T03:38:36.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.506 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.506 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822461+0000", 2024-09-16T03:38:36.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3216999999999999e-05 2024-09-16T03:38:36.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.506 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.506 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:38:36.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125593+0000", 2024-09-16T03:38:36.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30313253699999998 2024-09-16T03:38:36.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.507 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.507 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.507 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126307+0000", 2024-09-16T03:38:36.507 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071350000000000005 2024-09-16T03:38:36.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.507 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.507 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.507 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126825+0000", 2024-09-16T03:38:36.507 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051846100000000001 2024-09-16T03:38:36.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.507 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.507 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.508 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326612+0000", 2024-09-16T03:38:36.508 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19978721999999999 2024-09-16T03:38:36.508 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.508 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.508 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.508 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326663+0000", 2024-09-16T03:38:36.508 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0460000000000001e-05 2024-09-16T03:38:36.508 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.508 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.508 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.508 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326699+0000", 2024-09-16T03:38:36.508 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6340000000000001e-05 2024-09-16T03:38:36.508 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.509 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.509 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.509 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.509 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.509 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:784 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=63b,truncate 2840350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:36.509 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:38:36.509 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.683021014000005, 2024-09-16T03:38:36.509 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63358405200000001, 2024-09-16T03:38:36.510 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.510 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.510 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.510 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.510 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.510 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.510 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 784 2024-09-16T03:38:36.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.510 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806590+0000", 2024-09-16T03:38:36.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6000000000000003e-07 2024-09-16T03:38:36.512 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.512 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.512 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.512 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806593+0000", 2024-09-16T03:38:36.512 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6649999999999999e-06 2024-09-16T03:38:36.512 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.512 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.512 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.512 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806597+0000", 2024-09-16T03:38:36.512 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.245e-06 2024-09-16T03:38:36.512 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.513 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.513 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.513 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806600+0000", 2024-09-16T03:38:36.513 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.23e-06 2024-09-16T03:38:36.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.513 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.513 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.513 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099381+0000", 2024-09-16T03:38:36.513 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29278119400000002 2024-09-16T03:38:36.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.513 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.514 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.514 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099438+0000", 2024-09-16T03:38:36.514 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6198e-05 2024-09-16T03:38:36.514 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.514 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.514 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:36.514 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099527+0000", 2024-09-16T03:38:36.514 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9505000000000004e-05 2024-09-16T03:38:36.514 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.514 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.514 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.514 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200775+0000", 2024-09-16T03:38:36.514 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101247904 2024-09-16T03:38:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200889+0000", 2024-09-16T03:38:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114301 2024-09-16T03:38:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440066+0000", 2024-09-16T03:38:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.239176367 2024-09-16T03:38:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.515 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.516 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.516 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440109+0000", 2024-09-16T03:38:36.516 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2941000000000001e-05 2024-09-16T03:38:36.516 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.516 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.516 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.516 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440173+0000", 2024-09-16T03:38:36.516 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4842e-05 2024-09-16T03:38:36.516 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.516 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.517 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.517 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.517 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.517 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:785 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:36.517 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:38:36.517 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.683005089000005, 2024-09-16T03:38:36.517 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63376373200000002, 2024-09-16T03:38:36.517 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.517 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.517 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 785 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.518 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:38:36.519 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.519 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.519 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.519 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.519 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806606+0000", 2024-09-16T03:38:36.519 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:38:36.519 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.519 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.519 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.520 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806607+0000", 2024-09-16T03:38:36.520 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.424e-06 2024-09-16T03:38:36.520 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.520 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.520 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.520 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806611+0000", 2024-09-16T03:38:36.520 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1080000000000003e-06 2024-09-16T03:38:36.520 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.520 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.520 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.520 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806614+0000", 2024-09-16T03:38:36.521 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.067e-06 2024-09-16T03:38:36.521 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.521 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.521 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.521 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099813+0000", 2024-09-16T03:38:36.521 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29319853699999998 2024-09-16T03:38:36.521 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.521 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.521 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:36.521 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099858+0000", 2024-09-16T03:38:36.521 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5667999999999997e-05 2024-09-16T03:38:36.522 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.522 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.522 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.522 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440236+0000", 2024-09-16T03:38:36.522 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34037791499999998 2024-09-16T03:38:36.522 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.522 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.522 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.522 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440297+0000", 2024-09-16T03:38:36.522 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0988e-05 2024-09-16T03:38:36.522 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.522 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.523 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.523 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440369+0000", 2024-09-16T03:38:36.523 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1682999999999998e-05 2024-09-16T03:38:36.523 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.523 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.523 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.523 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.523 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.523 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:801 4.0 4:07ad0560:::smithi15736696-25:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:36.523 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:38:36.523 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.483527190999993, 2024-09-16T03:38:36.523 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.534510445, 2024-09-16T03:38:36.523 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.524 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.524 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:36.524 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.524 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.524 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.524 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 801 2024-09-16T03:38:36.524 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.524 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.524 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.524 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.524 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:38:36.525 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.525 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.525 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.525 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.525 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:38:36.525 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.525 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.525 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.525 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.525 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006084+0000", 2024-09-16T03:38:36.525 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.322e-06 2024-09-16T03:38:36.526 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.526 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.526 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.526 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006088+0000", 2024-09-16T03:38:36.526 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000001e-06 2024-09-16T03:38:36.526 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.526 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.526 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.526 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006098+0000", 2024-09-16T03:38:36.526 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0186999999999999e-05 2024-09-16T03:38:36.527 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.527 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.527 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.527 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006104+0000", 2024-09-16T03:38:36.527 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.271e-06 2024-09-16T03:38:36.527 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.527 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.527 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.527 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.199952+0000", 2024-09-16T03:38:36.527 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.193847935 2024-09-16T03:38:36.527 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.527 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.527 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.528 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200027+0000", 2024-09-16T03:38:36.528 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4987000000000006e-05 2024-09-16T03:38:36.528 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.528 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.528 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.528 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.540594+0000", 2024-09-16T03:38:36.528 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34056708299999999 2024-09-16T03:38:36.528 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.528 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.528 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.528 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.528 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.529 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1023 4.0 4:06b6d050:::smithi15736696-50:head [write 2246563~579356] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:36.529 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:38:36.529 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 65.178378120999994, 2024-09-16T03:38:36.529 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60727534699999997, 2024-09-16T03:38:36.529 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.529 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.529 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.529 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.529 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.530 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.530 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1023 2024-09-16T03:38:36.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.530 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.530 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.530 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.530 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:38:36.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.530 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.530 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311233+0000", 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1039999999999999e-06 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311692+0000", 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045908800000000002 2024-09-16T03:38:36.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.532 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.532 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.532 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311704+0000", 2024-09-16T03:38:36.532 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1844e-05 2024-09-16T03:38:36.532 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.532 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.532 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.532 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311713+0000", 2024-09-16T03:38:36.532 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2500000000000006e-06 2024-09-16T03:38:36.532 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.532 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.532 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.532 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359204+0000", 2024-09-16T03:38:36.533 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.047491646999999998 2024-09-16T03:38:36.533 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.533 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.533 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.533 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359258+0000", 2024-09-16T03:38:36.533 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4082000000000002e-05 2024-09-16T03:38:36.533 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.533 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.533 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:36.533 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359354+0000", 2024-09-16T03:38:36.533 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5686999999999994e-05 2024-09-16T03:38:36.533 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.534 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.534 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.534 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512245+0000", 2024-09-16T03:38:36.534 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15289135700000001 2024-09-16T03:38:36.534 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.534 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.534 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.534 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512956+0000", 2024-09-16T03:38:36.534 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071110300000000004 2024-09-16T03:38:36.534 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.534 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.534 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.534 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918361+0000", 2024-09-16T03:38:36.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40540489400000002 2024-09-16T03:38:36.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.535 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918455+0000", 2024-09-16T03:38:36.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3412999999999995e-05 2024-09-16T03:38:36.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.535 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918508+0000", 2024-09-16T03:38:36.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:38:36.535 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.535 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.536 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.536 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.536 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.536 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1024 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3176490] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:36.536 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:38:36.536 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 65.177885140000001, 2024-09-16T03:38:36.536 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60701227000000002, 2024-09-16T03:38:36.536 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1024 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.537 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.538 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.538 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:38:36.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.538 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.538 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.538 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.538 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311726+0000", 2024-09-16T03:38:36.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.02e-07 2024-09-16T03:38:36.538 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.538 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.538 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.538 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311730+0000", 2024-09-16T03:38:36.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1099999999999996e-06 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311735+0000", 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4680000000000003e-06 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311739+0000", 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9380000000000002e-06 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.539 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361468+0000", 2024-09-16T03:38:36.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.049729450000000001 2024-09-16T03:38:36.540 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.540 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.540 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.540 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361539+0000", 2024-09-16T03:38:36.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0896999999999996e-05 2024-09-16T03:38:36.540 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.540 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.540 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:36.540 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361661+0000", 2024-09-16T03:38:36.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000122404 2024-09-16T03:38:36.540 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.540 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512943+0000", 2024-09-16T03:38:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.151281481 2024-09-16T03:38:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512959+0000", 2024-09-16T03:38:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:38:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918588+0000", 2024-09-16T03:38:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40562815499999999 2024-09-16T03:38:36.542 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.542 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.542 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918646+0000", 2024-09-16T03:38:36.542 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8566999999999997e-05 2024-09-16T03:38:36.542 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.542 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.542 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918737+0000", 2024-09-16T03:38:36.542 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1308000000000001e-05 2024-09-16T03:38:36.542 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.542 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.542 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1025 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 65.177866641999998, 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60723649599999996, 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1025 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.543 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:36.544 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9400000000000001e-07 2024-09-16T03:38:36.545 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.545 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311746+0000", 2024-09-16T03:38:36.545 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2670000000000001e-06 2024-09-16T03:38:36.545 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.545 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311752+0000", 2024-09-16T03:38:36.545 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6389999999999997e-06 2024-09-16T03:38:36.545 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.545 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.546 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311757+0000", 2024-09-16T03:38:36.546 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3639999999999998e-06 2024-09-16T03:38:36.546 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.546 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.546 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.546 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362018+0000", 2024-09-16T03:38:36.546 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.050260969000000003 2024-09-16T03:38:36.546 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.546 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.546 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:36.546 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362089+0000", 2024-09-16T03:38:36.546 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0735000000000006e-05 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918781+0000", 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55669215400000005 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918867+0000", 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6019000000000004e-05 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.547 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918980+0000", 2024-09-16T03:38:36.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011295500000000001 2024-09-16T03:38:36.548 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.548 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.548 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.548 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1054 4.10 4:0a55bbf2:::smithi15736696-18:head [write 2633066~610652] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:36.548 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:38:36.548 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 64.976373433999996, 2024-09-16T03:38:36.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51115411600000005, 2024-09-16T03:38:36.548 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.548 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.548 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1054 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.549 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513238+0000", 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1999999999999998e-07 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513695+0000", 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045695099999999998 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.550 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513720+0000", 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.567e-05 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513725+0000", 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9899999999999997e-06 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613892+0000", 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166253 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.551 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613898+0000", 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8710000000000002e-06 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916870+0000", 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30297253800000001 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916934+0000", 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3723e-05 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.552 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.553 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:36.553 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.917033+0000", 2024-09-16T03:38:36.553 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8908999999999998e-05 2024-09-16T03:38:36.553 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.553 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.553 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.553 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921278+0000", 2024-09-16T03:38:36.553 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0042456999999999998 2024-09-16T03:38:36.553 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.553 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.553 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.553 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019617+0000", 2024-09-16T03:38:36.553 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098338910000000002 2024-09-16T03:38:36.554 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.554 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.554 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.554 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024269+0000", 2024-09-16T03:38:36.554 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046515519999999998 2024-09-16T03:38:36.554 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.554 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.554 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.554 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024358+0000", 2024-09-16T03:38:36.554 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9096000000000002e-05 2024-09-16T03:38:36.554 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.554 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024391+0000", 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3033e-05 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1055 4.10 4:0a55bbf2:::smithi15736696-18:head [setxattr _header (56) in=7b,truncate 3243718] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 64.975873934000006, 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61098759599999997, 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.555 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.556 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.556 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.556 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.556 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.556 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1055 2024-09-16T03:38:36.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.556 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.556 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:38:36.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:38:36.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513737+0000", 2024-09-16T03:38:36.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9399999999999993e-07 2024-09-16T03:38:36.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513759+0000", 2024-09-16T03:38:36.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1903999999999999e-05 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513773+0000", 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3244000000000001e-05 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513780+0000", 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999999e-06 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.558 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714016+0000", 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20023635000000001 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714024+0000", 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8359999999999992e-06 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918736+0000", 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471236700000001 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.559 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918830+0000", 2024-09-16T03:38:36.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3769000000000003e-05 2024-09-16T03:38:36.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.560 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.560 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:36.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919003+0000", 2024-09-16T03:38:36.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000173198 2024-09-16T03:38:36.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.560 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.560 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921589+0000", 2024-09-16T03:38:36.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002586419 2024-09-16T03:38:36.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019655+0000", 2024-09-16T03:38:36.561 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098065801999999994 2024-09-16T03:38:36.561 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124538+0000", 2024-09-16T03:38:36.561 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10488272899999999 2024-09-16T03:38:36.561 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124635+0000", 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7125000000000003e-05 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124724+0000", 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8858999999999997e-05 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1056 4.10 4:0a55bbf2:::smithi15736696-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:36.562 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 64.975821362999994, 2024-09-16T03:38:36.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61111632800000004, 2024-09-16T03:38:36.563 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.563 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.563 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:36.563 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.563 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.563 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.563 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1056 2024-09-16T03:38:36.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.563 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.563 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:36.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.564 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.564 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:36.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.564 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.564 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:36.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2499999999999995e-07 2024-09-16T03:38:36.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513792+0000", 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513797+0000", 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4940000000000003e-06 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513802+0000", 2024-09-16T03:38:36.565 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5390000000000001e-06 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919221+0000", 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40541913899999998 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919268+0000", 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7027000000000002e-05 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124766+0000", 2024-09-16T03:38:36.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20549842099999999 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124828+0000", 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1803000000000004e-05 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124905+0000", 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7163000000000006e-05 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1702 4.10 4:0a55bbf2:::smithi15736696-18:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 45.800466372999999, 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51800317600000001, 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1702 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.568 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689147+0000", 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:38:36.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689158+0000", 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0426e-05 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689185+0000", 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7299000000000001e-05 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789328+0000", 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100143102 2024-09-16T03:38:36.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.571 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789532+0000", 2024-09-16T03:38:36.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000204317 2024-09-16T03:38:36.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.571 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789671+0000", 2024-09-16T03:38:36.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013852600000000001 2024-09-16T03:38:36.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.207147+0000", 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.41747625599999999 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2160 4.17 4:ed874e32:::smithi15736696-14:head [assert-version v36,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 29.659464953000001, 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55377054800000003, 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.572 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.573 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.573 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.573 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.573 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2160 2024-09-16T03:38:36.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.573 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:38:36.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:38:36.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830149+0000", 2024-09-16T03:38:36.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.743e-06 2024-09-16T03:38:36.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830162+0000", 2024-09-16T03:38:36.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3283e-05 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830191+0000", 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8466000000000002e-05 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830207+0000", 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6453e-05 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830362+0000", 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015450699999999999 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830608+0000", 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024587399999999998 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.233387+0000", 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40277970099999999 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.576 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234208+0000", 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082053900000000001 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234370+0000", 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000161645 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383821+0000", 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14945145300000001 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.578 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383867+0000", 2024-09-16T03:38:36.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5498999999999999e-05 2024-09-16T03:38:36.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.578 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383916+0000", 2024-09-16T03:38:36.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9385000000000002e-05 2024-09-16T03:38:36.581 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.581 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.581 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.581 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2532 4.0 4:06b6d050:::smithi15736696-50:head [write 2603977~725887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:36.581 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:38:36.581 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.895317563999999, 2024-09-16T03:38:36.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53439557900000001, 2024-09-16T03:38:36.581 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.582 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.582 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.582 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.582 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.582 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.582 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2532 2024-09-16T03:38:36.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.582 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:38:36.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:38:36.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594298+0000", 2024-09-16T03:38:36.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6030000000000004e-06 2024-09-16T03:38:36.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594876+0000", 2024-09-16T03:38:36.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00057757800000000003 2024-09-16T03:38:36.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594885+0000", 2024-09-16T03:38:36.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4150000000000005e-06 2024-09-16T03:38:36.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594892+0000", 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7229999999999998e-06 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628138+0000", 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033246407999999998 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628252+0000", 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113755 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628344+0000", 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2028999999999996e-05 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730215+0000", 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101870761 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.732105+0000", 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001889951 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128411+0000", 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.396305764 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128611+0000", 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019990599999999999 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128688+0000", 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7686000000000001e-05 2024-09-16T03:38:36.587 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.588 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.588 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.588 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2533 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3663926] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:36.588 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:38:36.588 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.894710211, 2024-09-16T03:38:36.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53394911599999995, 2024-09-16T03:38:36.588 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.588 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.588 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.588 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.589 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.589 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.589 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2533 2024-09-16T03:38:36.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.589 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:38:36.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:38:36.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594901+0000", 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6700000000000003e-07 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594904+0000", 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1099999999999999e-06 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594907+0000", 2024-09-16T03:38:36.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1020000000000001e-06 2024-09-16T03:38:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594910+0000", 2024-09-16T03:38:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7889999999999999e-06 2024-09-16T03:38:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729537+0000", 2024-09-16T03:38:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.134627725 2024-09-16T03:38:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729660+0000", 2024-09-16T03:38:36.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012283800000000001 2024-09-16T03:38:36.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:36.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729786+0000", 2024-09-16T03:38:36.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012572000000000001 2024-09-16T03:38:36.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730385+0000", 2024-09-16T03:38:36.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059949700000000005 2024-09-16T03:38:36.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.028223+0000", 2024-09-16T03:38:36.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29783727999999998 2024-09-16T03:38:36.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128774+0000", 2024-09-16T03:38:36.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100551027 2024-09-16T03:38:36.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128816+0000", 2024-09-16T03:38:36.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1946000000000001e-05 2024-09-16T03:38:36.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128849+0000", 2024-09-16T03:38:36.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3515e-05 2024-09-16T03:38:36.595 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.595 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.595 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.595 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2534 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:36.596 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:38:36.596 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.894695789, 2024-09-16T03:38:36.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53406219200000005, 2024-09-16T03:38:36.596 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.596 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.596 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:36.596 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.596 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.597 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.597 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2534 2024-09-16T03:38:36.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.597 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:38:36.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:38:36.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:38:36.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8999999999999998e-07 2024-09-16T03:38:36.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594916+0000", 2024-09-16T03:38:36.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.344e-06 2024-09-16T03:38:36.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594919+0000", 2024-09-16T03:38:36.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0580000000000002e-06 2024-09-16T03:38:36.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594922+0000", 2024-09-16T03:38:36.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4389999999999999e-06 2024-09-16T03:38:36.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729999+0000", 2024-09-16T03:38:36.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.13507707399999999 2024-09-16T03:38:36.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:36.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730046+0000", 2024-09-16T03:38:36.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7382999999999997e-05 2024-09-16T03:38:36.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128879+0000", 2024-09-16T03:38:36.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39883298499999997 2024-09-16T03:38:36.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128924+0000", 2024-09-16T03:38:36.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:38:36.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128977+0000", 2024-09-16T03:38:36.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2648999999999997e-05 2024-09-16T03:38:36.601 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.601 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.601 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.602 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2621 4.f 4:fa163101:::smithi15736696-10:head [setxattr _header (56) in=7b,truncate 3815189] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:38:36.602 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:38:36.602 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.246919411, 2024-09-16T03:38:36.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55430782499999998, 2024-09-16T03:38:36.602 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.602 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.602 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.602 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.602 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.603 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.603 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2621 2024-09-16T03:38:36.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.603 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:38:36.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:38:36.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242692+0000", 2024-09-16T03:38:36.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7599999999999996e-07 2024-09-16T03:38:36.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242695+0000", 2024-09-16T03:38:36.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.529e-06 2024-09-16T03:38:36.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242699+0000", 2024-09-16T03:38:36.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1010000000000002e-06 2024-09-16T03:38:36.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242703+0000", 2024-09-16T03:38:36.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.518e-06 2024-09-16T03:38:36.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.494935+0000", 2024-09-16T03:38:36.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25223231200000001 2024-09-16T03:38:36.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495011+0000", 2024-09-16T03:38:36.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5692999999999995e-05 2024-09-16T03:38:36.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:38:36.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495117+0000", 2024-09-16T03:38:36.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000105944 2024-09-16T03:38:36.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.596308+0000", 2024-09-16T03:38:36.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101190825 2024-09-16T03:38:36.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796852+0000", 2024-09-16T03:38:36.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20054443199999999 2024-09-16T03:38:36.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796893+0000", 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1221999999999999e-05 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796946+0000", 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2435000000000002e-05 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796999+0000", 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3038000000000003e-05 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.608 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2622 4.f 4:fa163101:::smithi15736696-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 16.246903555999999, 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55448354399999999, 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2622 2024-09-16T03:38:36.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.610 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:38:36.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:38:36.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:38:36.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9999999999999999e-07 2024-09-16T03:38:36.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242709+0000", 2024-09-16T03:38:36.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.429e-06 2024-09-16T03:38:36.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242712+0000", 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6090000000000001e-06 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242716+0000", 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6480000000000001e-06 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495371+0000", 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25265485700000001 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:36.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495424+0000", 2024-09-16T03:38:36.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2927999999999999e-05 2024-09-16T03:38:36.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797042+0000", 2024-09-16T03:38:36.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30161821900000002 2024-09-16T03:38:36.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797114+0000", 2024-09-16T03:38:36.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2445000000000002e-05 2024-09-16T03:38:36.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797190+0000", 2024-09-16T03:38:36.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6108999999999999e-05 2024-09-16T03:38:36.614 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.614 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.614 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.615 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2941 4.17 4:eb60a28d:::smithi15736696-41:head [setxattr _header (56) in=7b,truncate 2420241] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:38:36.615 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:30.762823+0000", 2024-09-16T03:38:36.615 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.7267872510000002, 2024-09-16T03:38:36.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51643088500000001, 2024-09-16T03:38:36.615 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.615 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.615 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:36.615 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.615 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.615 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.615 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2941 2024-09-16T03:38:36.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.616 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762823+0000", 2024-09-16T03:38:36.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762823+0000", 2024-09-16T03:38:36.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762824+0000", 2024-09-16T03:38:36.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3200000000000005e-07 2024-09-16T03:38:36.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762826+0000", 2024-09-16T03:38:36.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7590000000000002e-06 2024-09-16T03:38:36.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762830+0000", 2024-09-16T03:38:36.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4879999999999999e-06 2024-09-16T03:38:36.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762834+0000", 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7249999999999999e-06 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.763900+0000", 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001066851 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.763982+0000", 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1225999999999998e-05 2024-09-16T03:38:36.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.764130+0000", 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014830999999999999 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.767207+0000", 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0030773189999999998 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.062538+0000", 2024-09-16T03:38:36.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29533109499999999 2024-09-16T03:38:36.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:36.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279028+0000", 2024-09-16T03:38:36.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.21648964100000001 2024-09-16T03:38:36.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:36.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279214+0000", 2024-09-16T03:38:36.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000185606 2024-09-16T03:38:36.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279254+0000", 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0333000000000002e-05 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2942 4.17 4:eb60a28d:::smithi15736696-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:30.762852+0000", 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.7267580589999998, 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51654379900000003, 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:36.621 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:36.622 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:36.622 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:36.622 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:36.622 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2942 2024-09-16T03:38:36.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.622 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:36.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:36.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762852+0000", 2024-09-16T03:38:36.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:36.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762852+0000", 2024-09-16T03:38:36.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:36.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:36.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762853+0000", 2024-09-16T03:38:36.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:38:36.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:36.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762855+0000", 2024-09-16T03:38:36.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0810000000000001e-06 2024-09-16T03:38:36.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:36.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762860+0000", 2024-09-16T03:38:36.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8720000000000001e-06 2024-09-16T03:38:36.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:36.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762863+0000", 2024-09-16T03:38:36.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2930000000000001e-06 2024-09-16T03:38:36.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.764919+0000", 2024-09-16T03:38:36.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020556979999999999 2024-09-16T03:38:36.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:36.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.764964+0000", 2024-09-16T03:38:36.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5788999999999998e-05 2024-09-16T03:38:36.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:36.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279291+0000", 2024-09-16T03:38:36.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51432677999999998 2024-09-16T03:38:36.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:36.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279337+0000", 2024-09-16T03:38:36.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5892e-05 2024-09-16T03:38:36.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:36.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:36.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:36.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279396+0000", 2024-09-16T03:38:36.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9018000000000003e-05 2024-09-16T03:38:36.627 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.627 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.627 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.627 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:36.627 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:36.628 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:36.629 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:38:36.630 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:36.627+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:36.657 INFO:tasks.rados.rados.0.smithi157.stdout:1386: set_chunk oid 41 offset: 177152 length: 1584128 target oid a63ac782fa4bce1e96397c8301f2734ce3d5850a2dc454a54c42fc5451a8d72d offset: 0 snap -1 2024-09-16T03:38:36.657 INFO:tasks.rados.rados.0.smithi157.stdout:1387: read oid 40 snap -1 2024-09-16T03:38:36.658 INFO:tasks.rados.rados.0.smithi157.stdout:1387: expect (ObjNum 245 snap 0 seq_num 245) 2024-09-16T03:38:36.658 INFO:tasks.rados.rados.0.smithi157.stdout:1388: delete oid 48 current snap is 0 2024-09-16T03:38:36.658 INFO:tasks.rados.rados.0.smithi157.stdout:1378: finishing write tid 2 to smithi15736696-42 2024-09-16T03:38:36.658 INFO:tasks.rados.rados.0.smithi157.stdout:1379: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-16T03:38:36.681 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1637023 2024-09-16T03:38:36.730 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:36.728+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:36.775 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:36.775 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:36.775 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:36.775 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:36.785 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:38:36.831 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:36.829+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:36.888 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:36.889 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:36.889 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:36.898 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:38:36.918 INFO:tasks.rados.rados.0.smithi157.stdout:1378: finishing write tid 3 to smithi15736696-42 2024-09-16T03:38:36.918 INFO:tasks.rados.rados.0.smithi157.stdout:1378: finishing write tid 4 to smithi15736696-42 2024-09-16T03:38:36.918 INFO:tasks.rados.rados.0.smithi157.stdout:1378: oid 42 updating version 0 to 63 2024-09-16T03:38:36.918 INFO:tasks.rados.rados.0.smithi157.stdout:1378: oid 42 version 63 is already newer than 62 2024-09-16T03:38:36.918 INFO:tasks.rados.rados.0.smithi157.stdout:1378: oid 42 version 63 is already newer than 61 2024-09-16T03:38:36.918 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 63 (ObjNum 275 snap 0 seq_num 275) dirty exists 2024-09-16T03:38:36.919 INFO:tasks.rados.rados.0.smithi157.stdout:1378: left oid 42 (ObjNum 275 snap 0 seq_num 275) 2024-09-16T03:38:36.919 INFO:tasks.rados.rados.0.smithi157.stdout:1380: finishing write tid 1 to smithi15736696-10 2024-09-16T03:38:36.919 INFO:tasks.rados.rados.0.smithi157.stdout:1378: done (10 left) 2024-09-16T03:38:36.919 INFO:tasks.rados.rados.0.smithi157.stdout:1379: done (9 left) 2024-09-16T03:38:36.919 INFO:tasks.rados.rados.0.smithi157.stdout:1388: done (8 left) 2024-09-16T03:38:36.919 INFO:tasks.rados.rados.0.smithi157.stdout:1389: write oid 28 current snap is 0 2024-09-16T03:38:36.919 INFO:tasks.rados.rados.0.smithi157.stdout:1380: finishing write tid 2 to smithi15736696-10 2024-09-16T03:38:36.919 INFO:tasks.rados.rados.0.smithi157.stdout:1389: seq_num 279 ranges {689153=783179,2076829=457413} 2024-09-16T03:38:36.925 INFO:tasks.rados.rados.0.smithi157.stdout:1389: writing smithi15736696-28 from 689153 to 1472332 tid 1 2024-09-16T03:38:36.932 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:36.929+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:36.933 INFO:tasks.rados.rados.0.smithi157.stdout:1389: writing smithi15736696-28 from 2076829 to 2534242 tid 2 2024-09-16T03:38:36.933 INFO:tasks.rados.rados.0.smithi157.stdout:1390: read oid 47 snap -1 2024-09-16T03:38:36.933 INFO:tasks.rados.rados.0.smithi157.stdout:1390: expect (ObjNum 262 snap 0 seq_num 262) 2024-09-16T03:38:36.933 INFO:tasks.rados.rados.0.smithi157.stdout:1391: read oid 36 snap -1 2024-09-16T03:38:36.933 INFO:tasks.rados.rados.0.smithi157.stdout:1391: expect (ObjNum 267 snap 0 seq_num 267) 2024-09-16T03:38:36.933 INFO:tasks.rados.rados.0.smithi157.stdout:1392: read oid 39 snap -1 2024-09-16T03:38:36.933 INFO:tasks.rados.rados.0.smithi157.stdout:1392: expect (ObjNum 258 snap 0 seq_num 258) 2024-09-16T03:38:36.933 INFO:tasks.rados.rados.0.smithi157.stdout:1393: tier_flush oid 42 2024-09-16T03:38:36.933 INFO:tasks.rados.rados.0.smithi157.stdout:1393: tier_flush oid 42 snap -1 2024-09-16T03:38:36.934 INFO:tasks.rados.rados.0.smithi157.stdout:1380: finishing write tid 3 to smithi15736696-10 2024-09-16T03:38:36.934 INFO:tasks.rados.rados.0.smithi157.stdout:1394: read oid 1 snap -1 2024-09-16T03:38:36.934 INFO:tasks.rados.rados.0.smithi157.stdout:1394: expect (ObjNum 242 snap 0 seq_num 242) 2024-09-16T03:38:36.934 INFO:tasks.rados.rados.0.smithi157.stdout:1395: read oid 2 snap -1 2024-09-16T03:38:36.934 INFO:tasks.rados.rados.0.smithi157.stdout:1395: expect deleted 2024-09-16T03:38:36.934 INFO:tasks.rados.rados.0.smithi157.stdout:1396: write oid 32 current snap is 0 2024-09-16T03:38:36.934 INFO:tasks.rados.rados.0.smithi157.stdout:1396: seq_num 280 ranges {705960=458428,1735311=717424} 2024-09-16T03:38:36.938 INFO:tasks.rados.rados.0.smithi157.stdout:1396: writing smithi15736696-32 from 705960 to 1164388 tid 1 2024-09-16T03:38:36.946 INFO:tasks.rados.rados.0.smithi157.stdout:1396: writing smithi15736696-32 from 1735311 to 2452735 tid 2 2024-09-16T03:38:36.946 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:36.946 INFO:tasks.rados.rados.0.smithi157.stdout:1380: finishing write tid 4 to smithi15736696-10 2024-09-16T03:38:36.946 INFO:tasks.rados.rados.0.smithi157.stdout:1380: oid 10 updating version 0 to 94 2024-09-16T03:38:36.946 INFO:tasks.rados.rados.0.smithi157.stdout:1380: oid 10 updating version 94 to 95 2024-09-16T03:38:36.947 INFO:tasks.rados.rados.0.smithi157.stdout:1380: oid 10 version 95 is already newer than 93 2024-09-16T03:38:36.947 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 95 (ObjNum 276 snap 0 seq_num 276) dirty exists 2024-09-16T03:38:36.947 INFO:tasks.rados.rados.0.smithi157.stdout:1380: left oid 10 (ObjNum 276 snap 0 seq_num 276) 2024-09-16T03:38:36.947 INFO:tasks.rados.rados.0.smithi157.stdout:1381: finishing set_chunk to oid 34 2024-09-16T03:38:36.947 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 34 set_chunk returned error code -95 offset: 432640 length: 1679360 2024-09-16T03:38:36.947 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 0 (ObjNum 205 snap 0 seq_num 205) dirty exists 2024-09-16T03:38:36.947 INFO:tasks.rados.rados.0.smithi157.stdout:1380: done (15 left) 2024-09-16T03:38:36.947 INFO:tasks.rados.rados.0.smithi157.stdout:1381: done (14 left) 2024-09-16T03:38:36.947 INFO:tasks.rados.rados.0.smithi157.stdout:1397: set_chunk oid 3 target oid 2024-09-16T03:38:36.947 INFO:tasks.rados.rados.0.smithi157.stdout:1397: set_chunk oid 3 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:36.948 INFO:tasks.rados.rados.0.smithi157.stdout:1398: write oid 5 current snap is 0 2024-09-16T03:38:36.948 INFO:tasks.rados.rados.0.smithi157.stdout:1398: seq_num 281 ranges {722767=533677,1793793=461082} 2024-09-16T03:38:36.952 INFO:tasks.rados.rados.0.smithi157.stdout:1398: writing smithi15736696-5 from 722767 to 1256444 tid 1 2024-09-16T03:38:36.958 INFO:tasks.rados.rados.0.smithi157.stdout:1398: writing smithi15736696-5 from 1793793 to 2254875 tid 2 2024-09-16T03:38:36.958 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:36.958 INFO:tasks.rados.rados.0.smithi157.stdout:1382: finishing write tid 1 to smithi15736696-50 2024-09-16T03:38:36.958 INFO:tasks.rados.rados.0.smithi157.stdout:1384: expect (ObjNum 261 snap 0 seq_num 261) 2024-09-16T03:38:37.005 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:37.005 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:37.005 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.859370638000001, 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:37.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:38:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:38:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:38:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:38:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:38:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:38:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:38:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:38:37.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:38:37.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.859017277999996, 2024-09-16T03:38:37.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:38:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:38:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:38:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:38:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:38:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:38:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:38:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:38:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:38:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:38:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:38:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.857889479999997, 2024-09-16T03:38:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:38:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.015 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:38:37.017 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:38:37.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.856902039000005, 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:37.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:38:37.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:38:37.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:38:37.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:38:37.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:38:37.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:38:37.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:38:37.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:38:37.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:38:37.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:38:37.024 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.024 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.024 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.024 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:38:37.025 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:37.025 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.856145295000005, 2024-09-16T03:38:37.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:38:37.025 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.025 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.025 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.025 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:37.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:37.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:38:37.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:38:37.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:38:37.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:38:37.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:38:37.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:86 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'8, mlcod=39'5)", 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.394633059, 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20388281899999999, 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:37.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608472+0000", 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.227e-06 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608616+0000", 2024-09-16T03:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000144177 2024-09-16T03:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608638+0000", 2024-09-16T03:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1543e-05 2024-09-16T03:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608645+0000", 2024-09-16T03:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5839999999999997e-06 2024-09-16T03:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608740+0000", 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4716999999999995e-05 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608763+0000", 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2671e-05 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812264+0000", 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203501194 2024-09-16T03:38:37.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812353+0000", 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8706000000000001e-05 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:87 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'9, mlcod=39'5)", 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.394419685000003, 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20372206400000001, 2024-09-16T03:38:37.035 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.036 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.036 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.036 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:37.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:37.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608684+0000", 2024-09-16T03:38:37.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6700000000000003e-07 2024-09-16T03:38:37.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608689+0000", 2024-09-16T03:38:37.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.412e-06 2024-09-16T03:38:37.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608704+0000", 2024-09-16T03:38:37.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4397e-05 2024-09-16T03:38:37.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608713+0000", 2024-09-16T03:38:37.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:38:37.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710025+0000", 2024-09-16T03:38:37.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101312032 2024-09-16T03:38:37.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710042+0000", 2024-09-16T03:38:37.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6889e-05 2024-09-16T03:38:37.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812361+0000", 2024-09-16T03:38:37.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10231889600000001 2024-09-16T03:38:37.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.040 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.040 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812405+0000", 2024-09-16T03:38:37.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4101999999999999e-05 2024-09-16T03:38:37.040 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.040 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.040 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.040 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:101 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'10, mlcod=39'5)", 2024-09-16T03:38:37.040 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:37.041 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.393406248000005, 2024-09-16T03:38:37.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20275586900000001, 2024-09-16T03:38:37.041 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.041 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.041 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.041 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:37.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609698+0000", 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.24e-06 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609892+0000", 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019465899999999999 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609904+0000", 2024-09-16T03:38:37.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2031000000000001e-05 2024-09-16T03:38:37.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609912+0000", 2024-09-16T03:38:37.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7160000000000003e-06 2024-09-16T03:38:37.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710228+0000", 2024-09-16T03:38:37.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100315682 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710242+0000", 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4201e-05 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812411+0000", 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10216881799999999 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812452+0000", 2024-09-16T03:38:37.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1522e-05 2024-09-16T03:38:37.045 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.045 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.045 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.045 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:102 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'11, mlcod=39'5)", 2024-09-16T03:38:37.045 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:37.045 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.391390223000002, 2024-09-16T03:38:37.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20134195399999999, 2024-09-16T03:38:37.045 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.045 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.045 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611714+0000", 2024-09-16T03:38:37.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.269e-06 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611926+0000", 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021252700000000001 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611938+0000", 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1437000000000001e-05 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611944+0000", 2024-09-16T03:38:37.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.993e-06 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811266+0000", 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.199322306 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811291+0000", 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4542999999999999e-05 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813021+0000", 2024-09-16T03:38:37.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001730363 2024-09-16T03:38:37.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813054+0000", 2024-09-16T03:38:37.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3516000000000002e-05 2024-09-16T03:38:37.049 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.049 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.049 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.049 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:375 4.f e45/32 4:fcb75e43:::smithi15736696-26:head v 45'23, mlcod=41'20)", 2024-09-16T03:38:37.049 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 76.695062495000002, 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200786568, 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:37.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308041+0000", 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8299999999999995e-07 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308046+0000", 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.904e-06 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308056+0000", 2024-09-16T03:38:37.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9210000000000004e-06 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.408122+0000", 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100066302 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508289+0000", 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166531 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508302+0000", 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2641e-05 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508794+0000", 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00049272400000000005 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508827+0000", 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2561999999999998e-05 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.054 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:989 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'31, mlcod=46'28)", 2024-09-16T03:38:37.054 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:37.054 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 66.615293934999997, 2024-09-16T03:38:37.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40332266900000002, 2024-09-16T03:38:37.054 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.054 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.054 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.054 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:37.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387812+0000", 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388467+0000", 2024-09-16T03:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065539500000000002 2024-09-16T03:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388481+0000", 2024-09-16T03:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3905e-05 2024-09-16T03:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488552+0000", 2024-09-16T03:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10007120899999999 2024-09-16T03:38:37.056 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688818+0000", 2024-09-16T03:38:37.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20026627599999999 2024-09-16T03:38:37.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688845+0000", 2024-09-16T03:38:37.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:38:37.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791091+0000", 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10224588699999999 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791131+0000", 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0293000000000002e-05 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:990 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'32, mlcod=46'28)", 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:37.058 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 66.514510263000005, 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30256419200000001, 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:37.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488594+0000", 2024-09-16T03:38:37.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:38:37.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488616+0000", 2024-09-16T03:38:37.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2419e-05 2024-09-16T03:38:37.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488627+0000", 2024-09-16T03:38:37.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1251e-05 2024-09-16T03:38:37.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.061 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588753+0000", 2024-09-16T03:38:37.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012544399999999 2024-09-16T03:38:37.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.061 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690679+0000", 2024-09-16T03:38:37.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101926027 2024-09-16T03:38:37.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690695+0000", 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6528000000000002e-05 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791133+0000", 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100438233 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791157+0000", 2024-09-16T03:38:37.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-05 2024-09-16T03:38:37.063 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.063 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.063 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.063 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:991 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'33, mlcod=46'28)", 2024-09-16T03:38:37.063 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:37.063 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 66.414312765999995, 2024-09-16T03:38:37.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202786365, 2024-09-16T03:38:37.063 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.063 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.063 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.063 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588792+0000", 2024-09-16T03:38:37.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0530000000000001e-06 2024-09-16T03:38:37.065 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.065 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.065 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588807+0000", 2024-09-16T03:38:37.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5257e-05 2024-09-16T03:38:37.065 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.065 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.065 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588832+0000", 2024-09-16T03:38:37.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5123e-05 2024-09-16T03:38:37.065 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.065 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588846+0000", 2024-09-16T03:38:37.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3312999999999999e-05 2024-09-16T03:38:37.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790810+0000", 2024-09-16T03:38:37.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20196378100000001 2024-09-16T03:38:37.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790866+0000", 2024-09-16T03:38:37.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6168999999999997e-05 2024-09-16T03:38:37.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791542+0000", 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067583699999999999 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791576+0000", 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4832000000000001e-05 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.067 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1580 4.f e61/32 4:fa163101:::smithi15736696-10:head v 61'52, mlcod=58'50)", 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.897481239999998, 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.201787367, 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2600000000000001e-07 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106083+0000", 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00046163900000000002 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106093+0000", 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9320000000000008e-06 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206190+0000", 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100096716 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206286+0000", 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5982000000000004e-05 2024-09-16T03:38:37.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206312+0000", 2024-09-16T03:38:37.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6279999999999999e-05 2024-09-16T03:38:37.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307345+0000", 2024-09-16T03:38:37.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10103240199999999 2024-09-16T03:38:37.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307409+0000", 2024-09-16T03:38:37.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4189999999999994e-05 2024-09-16T03:38:37.072 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.072 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.072 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.072 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1760 4.f e71/32 4:f101f67d:::smithi15736696-22:head v 71'55, mlcod=61'53)", 2024-09-16T03:38:37.072 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:37.072 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 40.622277273999998, 2024-09-16T03:38:37.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20118894600000001, 2024-09-16T03:38:37.073 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.073 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.073 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.073 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:37.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:37.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380826+0000", 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1899999999999998e-07 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380831+0000", 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6049999999999997e-06 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380851+0000", 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0021e-05 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380862+0000", 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0458e-05 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581184+0000", 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20032164299999999 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581202+0000", 2024-09-16T03:38:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8410000000000002e-05 2024-09-16T03:38:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581965+0000", 2024-09-16T03:38:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076291200000000003 2024-09-16T03:38:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.582014+0000", 2024-09-16T03:38:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9577999999999999e-05 2024-09-16T03:38:37.076 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.077 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.077 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.077 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2729 4.12 4:4a9d235d:::smithi15736696-1:head [write 1578407~488887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:38:37.077 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:22.608814+0000", 2024-09-16T03:38:37.077 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.394288535999999, 2024-09-16T03:38:37.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203332443, 2024-09-16T03:38:37.077 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.077 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.077 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.077 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:37.077 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2729 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.608814+0000", 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.608814+0000", 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.608816+0000", 2024-09-16T03:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0480000000000001e-06 2024-09-16T03:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609287+0000", 2024-09-16T03:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000470524 2024-09-16T03:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609298+0000", 2024-09-16T03:38:37.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1629e-05 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609306+0000", 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5480000000000004e-06 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609354+0000", 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7688999999999997e-05 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609448+0000", 2024-09-16T03:38:37.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4060000000000004e-05 2024-09-16T03:38:37.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 2,11", 2024-09-16T03:38:37.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609571+0000", 2024-09-16T03:38:37.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012361699999999999 2024-09-16T03:38:37.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:37.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.611263+0000", 2024-09-16T03:38:37.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016921099999999999 2024-09-16T03:38:37.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.613981+0000", 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027178340000000001 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812075+0000", 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19809358399999999 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812114+0000", 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9082000000000003e-05 2024-09-16T03:38:37.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812147+0000", 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2718000000000001e-05 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2730 4.12 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2067294] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:22.609319+0000", 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.393783875, 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202937274, 2024-09-16T03:38:37.083 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2730 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609319+0000", 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609319+0000", 2024-09-16T03:38:37.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609320+0000", 2024-09-16T03:38:37.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3099999999999997e-07 2024-09-16T03:38:37.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609324+0000", 2024-09-16T03:38:37.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.98e-06 2024-09-16T03:38:37.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609330+0000", 2024-09-16T03:38:37.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4640000000000003e-06 2024-09-16T03:38:37.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609337+0000", 2024-09-16T03:38:37.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.374e-06 2024-09-16T03:38:37.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.610575+0000", 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001237414 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.610663+0000", 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8268000000000003e-05 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 2,11", 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.610822+0000", 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015895199999999999 2024-09-16T03:38:37.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.613696+0000", 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0028742239999999999 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.614015+0000", 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00031823499999999999 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812188+0000", 2024-09-16T03:38:37.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.198173138 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812222+0000", 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4724999999999997e-05 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812256+0000", 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3769000000000001e-05 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2731 4.12 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:22.609352+0000", 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.393750426, 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203034889, 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2731 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.090 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609352+0000", 2024-09-16T03:38:37.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609352+0000", 2024-09-16T03:38:37.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609353+0000", 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4099999999999999e-07 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609355+0000", 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4439999999999998e-06 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609360+0000", 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609363+0000", 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3780000000000001e-06 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.611279+0000", 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019158140000000001 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.611318+0000", 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8312999999999997e-05 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812286+0000", 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20096847600000001 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812332+0000", 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6313000000000002e-05 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812387+0000", 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4857e-05 2024-09-16T03:38:37.094 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.095 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.095 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.095 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3081 4.1f e102/100 4:fa163101:::smithi15736696-10:head v 102'93, mlcod=102'91)", 2024-09-16T03:38:37.095 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:38:37.095 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.669387795, 2024-09-16T03:38:37.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20437049600000001, 2024-09-16T03:38:37.095 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.095 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.095 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.095 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:38:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:38:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333718+0000", 2024-09-16T03:38:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3720000000000001e-06 2024-09-16T03:38:37.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334338+0000", 2024-09-16T03:38:37.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061959099999999996 2024-09-16T03:38:37.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334355+0000", 2024-09-16T03:38:37.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.677e-05 2024-09-16T03:38:37.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334364+0000", 2024-09-16T03:38:37.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3780000000000007e-06 2024-09-16T03:38:37.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334424+0000", 2024-09-16T03:38:37.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0084000000000003e-05 2024-09-16T03:38:37.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334447+0000", 2024-09-16T03:38:37.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3113e-05 2024-09-16T03:38:37.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538030+0000", 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20358272399999999 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538085+0000", 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5464000000000003e-05 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3082 4.1f e102/100 4:fa163101:::smithi15736696-10:head v 102'94, mlcod=102'91)", 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:35.335460+0000", 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 1.667642837, 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264421699999999, 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335460+0000", 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:37.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335460+0000", 2024-09-16T03:38:37.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:37.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:37.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335462+0000", 2024-09-16T03:38:37.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9929999999999998e-06 2024-09-16T03:38:37.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:37.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335889+0000", 2024-09-16T03:38:37.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042691199999999999 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335898+0000", 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6660000000000003e-06 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335906+0000", 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0720000000000002e-06 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:37.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335960+0000", 2024-09-16T03:38:37.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4456000000000002e-05 2024-09-16T03:38:37.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:37.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335978+0000", 2024-09-16T03:38:37.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8082000000000001e-05 2024-09-16T03:38:37.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:37.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538089+0000", 2024-09-16T03:38:37.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202111078 2024-09-16T03:38:37.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:37.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:37.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:37.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538104+0000", 2024-09-16T03:38:37.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4958e-05 2024-09-16T03:38:37.104 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.104 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.104 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.104 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:37.104 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:37.104 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:37.106 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:38:37.107 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:37.107 INFO:tasks.rados.rados.0.smithi157.stdout:1382: finishing write tid 2 to smithi15736696-50 2024-09-16T03:38:37.107 INFO:tasks.rados.rados.0.smithi157.stdout:1384: done (15 left) 2024-09-16T03:38:37.107 INFO:tasks.rados.rados.0.smithi157.stdout:1399: write oid 13 current snap is 0 2024-09-16T03:38:37.108 INFO:tasks.rados.rados.0.smithi157.stdout:1399: seq_num 282 ranges {739574=725279,1968628=721093} 2024-09-16T03:38:37.108 INFO:tasks.rados.rados.0.smithi157.stdout:1399: writing smithi15736696-13 from 739574 to 1464853 tid 1 2024-09-16T03:38:37.108 INFO:tasks.rados.rados.0.smithi157.stdout:1399: writing smithi15736696-13 from 1968628 to 2689721 tid 2 2024-09-16T03:38:37.108 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:37.108 INFO:tasks.rados.rados.0.smithi157.stdout:1383: finishing set_chunk to oid 25 2024-09-16T03:38:37.108 INFO:tasks.rados.rados.0.smithi157.stdout:1383: got expected ENOENT (src dne) 2024-09-16T03:38:37.108 INFO:tasks.rados.rados.0.smithi157.stdout:1385: expect (ObjNum 248 snap 0 seq_num 248) 2024-09-16T03:38:37.108 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 84107 2024-09-16T03:38:37.108 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:37.103+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:37.163 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2058280 2024-09-16T03:38:37.164 INFO:tasks.rados.rados.0.smithi157.stdout:1382: finishing write tid 3 to smithi15736696-50 2024-09-16T03:38:37.164 INFO:tasks.rados.rados.0.smithi157.stdout:1382: finishing write tid 4 to smithi15736696-50 2024-09-16T03:38:37.164 INFO:tasks.rados.rados.0.smithi157.stdout:1382: oid 50 updating version 0 to 116 2024-09-16T03:38:37.164 INFO:tasks.rados.rados.0.smithi157.stdout:1382: oid 50 updating version 116 to 117 2024-09-16T03:38:37.164 INFO:tasks.rados.rados.0.smithi157.stdout:1382: oid 50 updating version 117 to 118 2024-09-16T03:38:37.164 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 118 (ObjNum 277 snap 0 seq_num 277) dirty exists 2024-09-16T03:38:37.164 INFO:tasks.rados.rados.0.smithi157.stdout:1382: left oid 50 (ObjNum 277 snap 0 seq_num 277) 2024-09-16T03:38:37.164 INFO:tasks.rados.rados.0.smithi157.stdout:1387: expect (ObjNum 245 snap 0 seq_num 245) 2024-09-16T03:38:37.173 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:38:37.204 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 173085 2024-09-16T03:38:37.213 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:37.213 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:37.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.213 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.11.23:139 3.1 e104/31)", 2024-09-16T03:38:37.214 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:37.142810+0000", 2024-09-16T03:38:37.214 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.069692555000000003, 2024-09-16T03:38:37.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.069708539, 2024-09-16T03:38:37.214 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:37.214 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:37.214 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:37.214 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:37.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.142810+0000", 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.142810+0000", 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.142811+0000", 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8200000000000001e-07 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:37.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.142865+0000", 2024-09-16T03:38:37.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4914000000000003e-05 2024-09-16T03:38:37.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:37.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.142871+0000", 2024-09-16T03:38:37.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1250000000000001e-06 2024-09-16T03:38:37.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.142874+0000", 2024-09-16T03:38:37.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.304e-06 2024-09-16T03:38:37.217 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:37.217 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:37.217 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:37.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.217 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.11.23:144 3.1 e104/31)", 2024-09-16T03:38:37.217 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:37.172033+0000", 2024-09-16T03:38:37.217 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.040469406999999999, 2024-09-16T03:38:37.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.040508980999999999, 2024-09-16T03:38:37.217 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:37.217 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:37.217 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:37.218 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:37.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:37.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.172033+0000", 2024-09-16T03:38:37.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:37.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:37.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.172033+0000", 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.172035+0000", 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.88e-06 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.172283+0000", 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00024775600000000002 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:37.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.172308+0000", 2024-09-16T03:38:37.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4892000000000002e-05 2024-09-16T03:38:37.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.172318+0000", 2024-09-16T03:38:37.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0112000000000001e-05 2024-09-16T03:38:37.221 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:37.221 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:37.221 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:37.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.221 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.11.23:146 3.1 e104/31)", 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:37.172360+0000", 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.040142918, 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.040202266, 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.172360+0000", 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.172360+0000", 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.172360+0000", 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2399999999999998e-07 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.172569+0000", 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020891700000000001 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.172576+0000", 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0249999999999997e-06 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.172581+0000", 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2270000000000001e-06 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-16T03:38:37.224 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 3 2024-09-16T03:38:37.225 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:37.225 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:37.214+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:37.226 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:38:37.284 INFO:tasks.rados.rados.0.smithi157.stdout:1382: done (15 left) 2024-09-16T03:38:37.284 INFO:tasks.rados.rados.0.smithi157.stdout:1383: done (14 left) 2024-09-16T03:38:37.284 INFO:tasks.rados.rados.0.smithi157.stdout:1385: done (13 left) 2024-09-16T03:38:37.284 INFO:tasks.rados.rados.0.smithi157.stdout:1387: done (12 left) 2024-09-16T03:38:37.285 INFO:tasks.rados.rados.0.smithi157.stdout:1395: done (11 left) 2024-09-16T03:38:37.285 INFO:tasks.rados.rados.0.smithi157.stdout:1400: read oid 46 snap -1 2024-09-16T03:38:37.285 INFO:tasks.rados.rados.0.smithi157.stdout:1400: expect (ObjNum 236 snap 0 seq_num 236) 2024-09-16T03:38:37.285 INFO:tasks.rados.rados.0.smithi157.stdout:1401: write (excl) oid 37 current snap is 0 2024-09-16T03:38:37.285 INFO:tasks.rados.rados.0.smithi157.stdout:1392: expect (ObjNum 258 snap 0 seq_num 258) 2024-09-16T03:38:37.285 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:37.298 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:37.299 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:37.307 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:37.305+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:37.317 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:37.317 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:37.332 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:38:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:38:37.333 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:37.334 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:37.335 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:37.338 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:37.341 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:38:37.353 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:37.354 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:37.354 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:38:37.372 INFO:tasks.rados.rados.0.smithi157.stdout:1394: expect (ObjNum 242 snap 0 seq_num 242) 2024-09-16T03:38:37.373 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:37.384 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 9] out_osds: [1, 5, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:38:37.384 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:38:37.384 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:38:37.384 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T03:38:37.385 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:38:37.408 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:37.406+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:37.416 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:37.448 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:37.449 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:37.456 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:37.456 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:37.458 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:38:37.471 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1155020 2024-09-16T03:38:37.474 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:37.474 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:37.492 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:37.492 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:37.494 INFO:tasks.rados.rados.0.smithi157.stdout:1397: finishing set_chunk to oid 3 2024-09-16T03:38:37.494 INFO:tasks.rados.rados.0.smithi157.stdout:1397: got expected ENOENT (src dne) 2024-09-16T03:38:37.494 INFO:tasks.rados.rados.0.smithi157.stdout:1391: expect (ObjNum 267 snap 0 seq_num 267) 2024-09-16T03:38:37.503 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1691814 2024-09-16T03:38:37.509 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:37.507+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:37.509 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:37.509 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:37.512 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:37.526 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:37.526 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:37.544 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:37.544 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:37.561 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:37.561 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:37.564 INFO:tasks.rados.rados.0.smithi157.stdout:1389: finishing write tid 1 to smithi15736696-28 2024-09-16T03:38:37.564 INFO:tasks.rados.rados.0.smithi157.stdout:1389: finishing write tid 2 to smithi15736696-28 2024-09-16T03:38:37.564 INFO:tasks.rados.rados.0.smithi157.stdout:1389: finishing write tid 3 to smithi15736696-28 2024-09-16T03:38:37.564 INFO:tasks.rados.rados.0.smithi157.stdout:1389: finishing write tid 4 to smithi15736696-28 2024-09-16T03:38:37.564 INFO:tasks.rados.rados.0.smithi157.stdout:1389: oid 28 updating version 0 to 105 2024-09-16T03:38:37.564 INFO:tasks.rados.rados.0.smithi157.stdout:1389: oid 28 updating version 105 to 106 2024-09-16T03:38:37.564 INFO:tasks.rados.rados.0.smithi157.stdout:1389: oid 28 updating version 106 to 107 2024-09-16T03:38:37.564 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 107 (ObjNum 279 snap 0 seq_num 279) dirty exists 2024-09-16T03:38:37.564 INFO:tasks.rados.rados.0.smithi157.stdout:1389: left oid 28 (ObjNum 279 snap 0 seq_num 279) 2024-09-16T03:38:37.564 INFO:tasks.rados.rados.0.smithi157.stdout:1396: finishing write tid 1 to smithi15736696-32 2024-09-16T03:38:37.565 INFO:tasks.rados.rados.0.smithi157.stdout:1401: seq_num 283 ranges {756381=400528,1910757=464751} 2024-09-16T03:38:37.570 INFO:tasks.rados.rados.0.smithi157.stdout:1401: writing smithi15736696-37 from 756381 to 1156909 tid 1 2024-09-16T03:38:37.578 INFO:tasks.rados.rados.0.smithi157.stdout:1401: writing smithi15736696-37 from 1910757 to 2375508 tid 2 2024-09-16T03:38:37.578 INFO:tasks.rados.rados.0.smithi157.stdout:1389: done (12 left) 2024-09-16T03:38:37.578 INFO:tasks.rados.rados.0.smithi157.stdout:1391: done (11 left) 2024-09-16T03:38:37.579 INFO:tasks.rados.rados.0.smithi157.stdout:1392: done (10 left) 2024-09-16T03:38:37.579 INFO:tasks.rados.rados.0.smithi157.stdout:1394: done (9 left) 2024-09-16T03:38:37.579 INFO:tasks.rados.rados.0.smithi157.stdout:1397: done (8 left) 2024-09-16T03:38:37.579 INFO:tasks.rados.rados.0.smithi157.stdout:1402: read oid 6 snap -1 2024-09-16T03:38:37.579 INFO:tasks.rados.rados.0.smithi157.stdout:1402: expect deleted 2024-09-16T03:38:37.580 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:37.580 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:37.582 INFO:tasks.rados.rados.0.smithi157.stdout:1403: set_chunk oid 30 target oid 2024-09-16T03:38:37.583 INFO:tasks.rados.rados.0.smithi157.stdout:1403: set_chunk oid 30 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:37.583 INFO:tasks.rados.rados.0.smithi157.stdout:1404: tier_flush oid 10 2024-09-16T03:38:37.583 INFO:tasks.rados.rados.0.smithi157.stdout:1404: tier_flush oid 10 snap -1 2024-09-16T03:38:37.583 INFO:tasks.rados.rados.0.smithi157.stdout:1386: finishing set_chunk to oid 41 2024-09-16T03:38:37.583 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 44 (ObjNum 263 snap 0 seq_num 263) dirty exists 2024-09-16T03:38:37.583 INFO:tasks.rados.rados.0.smithi157.stdout:1396: finishing write tid 2 to smithi15736696-32 2024-09-16T03:38:37.584 INFO:tasks.rados.rados.0.smithi157.stdout:1386: done (10 left) 2024-09-16T03:38:37.584 INFO:tasks.rados.rados.0.smithi157.stdout:1405: read oid 9 snap -1 2024-09-16T03:38:37.584 INFO:tasks.rados.rados.0.smithi157.stdout:1405: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-16T03:38:37.584 INFO:tasks.rados.rados.0.smithi157.stdout:1406: read oid 18 snap -1 2024-09-16T03:38:37.584 INFO:tasks.rados.rados.0.smithi157.stdout:1406: expect (ObjNum 269 snap 0 seq_num 269) 2024-09-16T03:38:37.584 INFO:tasks.rados.rados.0.smithi157.stdout:1407: tier_evict oid 8 2024-09-16T03:38:37.585 INFO:tasks.rados.rados.0.smithi157.stdout:1407: tier_evict oid 8 snap -1 2024-09-16T03:38:37.585 INFO:tasks.rados.rados.0.smithi157.stdout:1396: finishing write tid 3 to smithi15736696-32 2024-09-16T03:38:37.585 INFO:tasks.rados.rados.0.smithi157.stdout:1396: finishing write tid 4 to smithi15736696-32 2024-09-16T03:38:37.585 INFO:tasks.rados.rados.0.smithi157.stdout:1396: oid 32 updating version 0 to 18 2024-09-16T03:38:37.585 INFO:tasks.rados.rados.0.smithi157.stdout:1396: oid 32 updating version 18 to 19 2024-09-16T03:38:37.585 INFO:tasks.rados.rados.0.smithi157.stdout:1396: oid 32 updating version 19 to 20 2024-09-16T03:38:37.585 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 20 (ObjNum 280 snap 0 seq_num 280) dirty exists 2024-09-16T03:38:37.585 INFO:tasks.rados.rados.0.smithi157.stdout:1396: left oid 32 (ObjNum 280 snap 0 seq_num 280) 2024-09-16T03:38:37.585 INFO:tasks.rados.rados.0.smithi157.stdout:1398: finishing write tid 1 to smithi15736696-5 2024-09-16T03:38:37.585 INFO:tasks.rados.rados.0.smithi157.stdout:1398: finishing write tid 2 to smithi15736696-5 2024-09-16T03:38:37.585 INFO:tasks.rados.rados.0.smithi157.stdout:1398: finishing write tid 3 to smithi15736696-5 2024-09-16T03:38:37.585 INFO:tasks.rados.rados.0.smithi157.stdout:1399: finishing write tid 1 to smithi15736696-13 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:1399: finishing write tid 2 to smithi15736696-13 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:1398: finishing write tid 4 to smithi15736696-5 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:1398: oid 5 updating version 0 to 65 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:1398: oid 5 version 65 is already newer than 64 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:1398: oid 5 updating version 65 to 66 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 66 (ObjNum 281 snap 0 seq_num 281) dirty exists 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:1398: left oid 5 (ObjNum 281 snap 0 seq_num 281) 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:1399: finishing write tid 3 to smithi15736696-13 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:1399: finishing write tid 4 to smithi15736696-13 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:1399: oid 13 updating version 0 to 68 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:1399: oid 13 version 68 is already newer than 67 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:1399: oid 13 updating version 68 to 69 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 69 (ObjNum 282 snap 0 seq_num 282) dirty exists 2024-09-16T03:38:37.586 INFO:tasks.rados.rados.0.smithi157.stdout:1399: left oid 13 (ObjNum 282 snap 0 seq_num 282) 2024-09-16T03:38:37.587 INFO:tasks.rados.rados.0.smithi157.stdout:1396: done (12 left) 2024-09-16T03:38:37.587 INFO:tasks.rados.rados.0.smithi157.stdout:1398: done (11 left) 2024-09-16T03:38:37.587 INFO:tasks.rados.rados.0.smithi157.stdout:1399: done (10 left) 2024-09-16T03:38:37.587 INFO:tasks.rados.rados.0.smithi157.stdout:1408: set_chunk oid 27 target oid 2024-09-16T03:38:37.587 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:37.588 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:37.589 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:38:37.595 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:37.596 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:37.609 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:37.607+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:37.613 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:37.613 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:37.703 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:37.703 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:37.710 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:37.708+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:37.716 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:38:37.717 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:38:37.719 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:37.719 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:37.728 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-16T03:38:37.698911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":172459964,"num_objects":122,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":0,"num_read":3199,"num_read_kb":2131144,"num_write":1902,"num_write_kb":472136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":47356892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":14,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":147,"acting":147,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":13,"num_per_pool_osds":13,"num_per_pool_omap_osds":13,"kb":1226833920,"kb_used":703932,"kb_used_data":300084,"kb_used_omap":20,"kb_used_meta":396843,"kb_avail":1226129988,"statfs":{"total":1256277934080,"available":1255557107712,"internally_reserved":0,"allocated":307286016,"data_stored":312637415,"data_compressed":7571540,"data_compressed_allocated":8126464,"data_compressed_original":18063360,"omap_allocated":20735,"internal_metadata":406368001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2931004,"num_objects":-24,"num_object_clones":0,"num_object_copies":-72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-41,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":227,"num_read_kb":183433,"num_write":65,"num_write_kb":12378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7530695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000967"},"pg_stats":[{"pgid":"4.23","version":"94'34","reported_seq":277,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T03:38:36.452649+0000","last_change":"2024-09-16T03:38:34.893610+0000","last_active":"2024-09-16T03:38:36.452649+0000","last_peered":"2024-09-16T03:38:36.452649+0000","last_clean":"2024-09-16T03:38:36.452649+0000","last_became_active":"2024-09-16T03:38:33.881123+0000","last_became_peered":"2024-09-16T03:38:33.881123+0000","last_unstale":"2024-09-16T03:38:36.452649+0000","last_undegraded":"2024-09-16T03:38:36.452649+0000","last_fullsized":"2024-09-16T03:38:36.452649+0000","mapping_epoch":100,"log_start":"73'29","ondisk_log_start":"73'29","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:34.893524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.893524+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:59.573531+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613505,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.22","version":"45'6","reported_seq":208,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T03:38:36.352498+0000","last_change":"2024-09-16T03:38:34.836413+0000","last_active":"2024-09-16T03:38:36.352498+0000","last_peered":"2024-09-16T03:38:36.352498+0000","last_clean":"2024-09-16T03:38:36.352498+0000","last_became_active":"2024-09-16T03:38:33.689595+0000","last_became_peered":"2024-09-16T03:38:33.689595+0000","last_unstale":"2024-09-16T03:38:36.352498+0000","last_undegraded":"2024-09-16T03:38:36.352498+0000","last_fullsized":"2024-09-16T03:38:36.352498+0000","mapping_epoch":100,"log_start":"39'3","ondisk_log_start":"39'3","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:34.836343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.836343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:37.424576+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.21","version":"48'7","reported_seq":55,"reported_epoch":103,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:35.726337+0000","last_change":"2024-09-16T03:38:35.726255+0000","last_active":"2024-09-16T03:38:35.726337+0000","last_peered":"2024-09-16T03:38:35.726337+0000","last_clean":"2024-09-16T03:37:28.467886+0000","last_became_active":"2024-09-16T03:38:35.624748+0000","last_became_peered":"2024-09-16T03:38:35.624748+0000","last_unstale":"2024-09-16T03:38:35.726337+0000","last_undegraded":"2024-09-16T03:38:35.726337+0000","last_fullsized":"2024-09-16T03:38:35.726337+0000","mapping_epoch":102,"log_start":"40'4","ondisk_log_start":"40'4","created":100,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":6,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:37:39.701279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.701279+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":760509,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,12],"acting":[2,4,7],"avail_no_missing":["2","4","7","8","11","12"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"99'114","reported_seq":445,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.782846+0000","last_change":"2024-09-16T03:38:33.782236+0000","last_active":"2024-09-16T03:38:33.782846+0000","last_peered":"2024-09-16T03:38:33.782846+0000","last_clean":"2024-09-16T03:38:33.782846+0000","last_became_active":"2024-09-16T03:38:33.781861+0000","last_became_peered":"2024-09-16T03:38:33.781861+0000","last_unstale":"2024-09-16T03:38:33.782846+0000","last_undegraded":"2024-09-16T03:38:33.782846+0000","last_fullsized":"2024-09-16T03:38:33.782846+0000","mapping_epoch":100,"log_start":"95'111","ondisk_log_start":"95'111","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"54'52","last_scrub_stamp":"2024-09-16T03:37:39.774014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.774014+0000","objects_scrubbed":5,"log_size":3,"log_dups_size":111,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6260674,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":93,"num_read_kb":89849,"num_write":36,"num_write_kb":10336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.1f","version":"99'89","reported_seq":369,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:34.083810+0000","last_change":"2024-09-16T03:38:33.882581+0000","last_active":"2024-09-16T03:38:34.083810+0000","last_peered":"2024-09-16T03:38:34.083810+0000","last_clean":"2024-09-16T03:38:34.083810+0000","last_became_active":"2024-09-16T03:38:33.782199+0000","last_became_peered":"2024-09-16T03:38:33.782199+0000","last_unstale":"2024-09-16T03:38:34.083810+0000","last_undegraded":"2024-09-16T03:38:34.083810+0000","last_fullsized":"2024-09-16T03:38:34.083810+0000","mapping_epoch":100,"log_start":"98'86","ondisk_log_start":"98'86","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"75'64","last_scrub_stamp":"2024-09-16T03:38:11.380094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:11.380094+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":86,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3084101,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":133,"num_read_kb":117797,"num_write":45,"num_write_kb":11720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.1e","version":"92'59","reported_seq":300,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-16T03:38:34.385285+0000","last_change":"2024-09-16T03:38:33.680537+0000","last_active":"2024-09-16T03:38:34.385285+0000","last_peered":"2024-09-16T03:38:34.385285+0000","last_clean":"2024-09-16T03:38:34.385285+0000","last_became_active":"2024-09-16T03:38:33.680215+0000","last_became_peered":"2024-09-16T03:38:33.680215+0000","last_unstale":"2024-09-16T03:38:34.385285+0000","last_undegraded":"2024-09-16T03:38:34.385285+0000","last_fullsized":"2024-09-16T03:38:34.385285+0000","mapping_epoch":100,"log_start":"89'56","ondisk_log_start":"89'56","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"78'42","last_scrub_stamp":"2024-09-16T03:38:11.825685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:11.825685+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5252188,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":76183,"num_write":31,"num_write_kb":8941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"92'14","reported_seq":237,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.979596+0000","last_change":"2024-09-16T03:38:33.979036+0000","last_active":"2024-09-16T03:38:33.979596+0000","last_peered":"2024-09-16T03:38:33.979596+0000","last_clean":"2024-09-16T03:38:33.979596+0000","last_became_active":"2024-09-16T03:38:33.878609+0000","last_became_peered":"2024-09-16T03:38:33.878609+0000","last_unstale":"2024-09-16T03:38:33.979596+0000","last_undegraded":"2024-09-16T03:38:33.979596+0000","last_fullsized":"2024-09-16T03:38:33.979596+0000","mapping_epoch":100,"log_start":"91'11","ondisk_log_start":"91'11","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:27.634300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:27.634300+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":8484,"num_write":7,"num_write_kb":1932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1080040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.1c","version":"99'56","reported_seq":261,"reported_epoch":101,"state":"unknown","last_fresh":"2024-09-16T03:38:33.375140+0000","last_change":"2024-09-16T03:38:34.079842+0000","last_active":"2024-09-16T03:38:33.375140+0000","last_peered":"2024-09-16T03:38:32.121236+0000","last_clean":"2024-09-16T03:37:50.453398+0000","last_became_active":"2024-09-16T03:37:54.762708+0000","last_became_peered":"2024-09-16T03:37:54.762708+0000","last_unstale":"2024-09-16T03:38:33.375140+0000","last_undegraded":"2024-09-16T03:38:33.375140+0000","last_fullsized":"2024-09-16T03:38:33.375140+0000","mapping_epoch":100,"log_start":"90'53","ondisk_log_start":"90'53","created":100,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932805,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":42,"num_read_kb":32274,"num_write":30,"num_write_kb":8287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.1b","version":"98'104","reported_seq":442,"reported_epoch":104,"state":"active+clean+remapped","last_fresh":"2024-09-16T03:38:36.597610+0000","last_change":"2024-09-16T03:38:36.597050+0000","last_active":"2024-09-16T03:38:36.597610+0000","last_peered":"2024-09-16T03:38:36.597610+0000","last_clean":"2024-09-16T03:38:36.597610+0000","last_became_active":"2024-09-16T03:38:35.585330+0000","last_became_peered":"2024-09-16T03:38:35.585330+0000","last_unstale":"2024-09-16T03:38:36.597610+0000","last_undegraded":"2024-09-16T03:38:36.597610+0000","last_fullsized":"2024-09-16T03:38:36.597610+0000","mapping_epoch":102,"log_start":"98'98","ondisk_log_start":"98'98","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":98,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5529553,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":138299,"num_write":57,"num_write_kb":17461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8906666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,12],"avail_no_missing":["15","4","9","12"],"object_location_counts":[{"shards":"4,12,15","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.1a","version":"101'115","reported_seq":476,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:34.098622+0000","last_change":"2024-09-16T03:38:33.889878+0000","last_active":"2024-09-16T03:38:34.098622+0000","last_peered":"2024-09-16T03:38:34.098622+0000","last_clean":"2024-09-16T03:38:34.098622+0000","last_became_active":"2024-09-16T03:38:33.888393+0000","last_became_peered":"2024-09-16T03:38:33.888393+0000","last_unstale":"2024-09-16T03:38:34.098622+0000","last_undegraded":"2024-09-16T03:38:34.098622+0000","last_fullsized":"2024-09-16T03:38:34.098622+0000","mapping_epoch":100,"log_start":"99'112","ondisk_log_start":"99'112","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'108","last_scrub_stamp":"2024-09-16T03:38:25.831424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:25.831424+0000","objects_scrubbed":6,"log_size":3,"log_dups_size":112,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8151445,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":225,"num_read_kb":163564,"num_write":62,"num_write_kb":18351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":161,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.475905+0000","last_change":"2024-09-16T03:37:52.558405+0000","last_active":"2024-09-16T03:38:33.475905+0000","last_peered":"2024-09-16T03:38:33.475905+0000","last_clean":"2024-09-16T03:38:33.475905+0000","last_became_active":"2024-09-16T03:37:52.558043+0000","last_became_peered":"2024-09-16T03:37:52.558043+0000","last_unstale":"2024-09-16T03:38:33.475905+0000","last_undegraded":"2024-09-16T03:38:33.475905+0000","last_fullsized":"2024-09-16T03:38:33.475905+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:39.955354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.955354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:01.568207+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"98'44","reported_seq":271,"reported_epoch":101,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:33.880442+0000","last_change":"2024-09-16T03:38:33.880442+0000","last_active":"2024-09-16T03:38:33.880442+0000","last_peered":"2024-09-16T03:38:33.880442+0000","last_clean":"2024-09-16T03:37:50.435122+0000","last_became_active":"2024-09-16T03:38:33.779761+0000","last_became_peered":"2024-09-16T03:38:33.779761+0000","last_unstale":"2024-09-16T03:38:33.880442+0000","last_undegraded":"2024-09-16T03:38:33.880442+0000","last_fullsized":"2024-09-16T03:38:33.880442+0000","mapping_epoch":100,"log_start":"96'41","ondisk_log_start":"96'41","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'29","last_scrub_stamp":"2024-09-16T03:37:39.786314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.786314+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":38,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":781,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":21989,"num_write":14,"num_write_kb":3435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,5,1],"avail_no_missing":["0","1","4","5","9","10"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"98'44","reported_seq":292,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.575925+0000","last_change":"2024-09-16T03:37:47.616917+0000","last_active":"2024-09-16T03:38:33.575925+0000","last_peered":"2024-09-16T03:38:33.575925+0000","last_clean":"2024-09-16T03:38:33.575925+0000","last_became_active":"2024-09-16T03:37:31.713115+0000","last_became_peered":"2024-09-16T03:37:31.713115+0000","last_unstale":"2024-09-16T03:38:33.575925+0000","last_undegraded":"2024-09-16T03:38:33.575925+0000","last_fullsized":"2024-09-16T03:38:33.575925+0000","mapping_epoch":50,"log_start":"83'41","ondisk_log_start":"83'41","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:50.714861+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4621958,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":62985,"num_write":10,"num_write_kb":2620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.16","version":"98'64","reported_seq":309,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.474705+0000","last_change":"2024-09-16T03:37:57.546992+0000","last_active":"2024-09-16T03:38:33.474705+0000","last_peered":"2024-09-16T03:38:33.474705+0000","last_clean":"2024-09-16T03:38:33.474705+0000","last_became_active":"2024-09-16T03:37:57.544402+0000","last_became_peered":"2024-09-16T03:37:57.544402+0000","last_unstale":"2024-09-16T03:38:33.474705+0000","last_undegraded":"2024-09-16T03:38:33.474705+0000","last_fullsized":"2024-09-16T03:38:33.474705+0000","mapping_epoch":71,"log_start":"85'61","ondisk_log_start":"85'61","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'32","last_scrub_stamp":"2024-09-16T03:37:35.189364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.189364+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:51.659275+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9176655,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":85795,"num_write":29,"num_write_kb":10925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2311899,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.15","version":"100'59","reported_seq":324,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T03:38:36.352571+0000","last_change":"2024-09-16T03:37:47.078728+0000","last_active":"2024-09-16T03:38:36.352571+0000","last_peered":"2024-09-16T03:38:36.352571+0000","last_clean":"2024-09-16T03:38:36.352571+0000","last_became_active":"2024-09-16T03:37:31.349745+0000","last_became_peered":"2024-09-16T03:37:31.349745+0000","last_unstale":"2024-09-16T03:38:36.352571+0000","last_undegraded":"2024-09-16T03:38:36.352571+0000","last_fullsized":"2024-09-16T03:38:36.352571+0000","mapping_epoch":50,"log_start":"98'56","ondisk_log_start":"98'56","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":54,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:12.367451+0000","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4924517,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":138,"num_read_kb":135684,"num_write":31,"num_write_kb":9048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.14","version":"98'17","reported_seq":224,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T03:38:36.452653+0000","last_change":"2024-09-16T03:37:39.953719+0000","last_active":"2024-09-16T03:38:36.452653+0000","last_peered":"2024-09-16T03:38:36.452653+0000","last_clean":"2024-09-16T03:38:36.452653+0000","last_became_active":"2024-09-16T03:37:31.149621+0000","last_became_peered":"2024-09-16T03:37:31.149621+0000","last_unstale":"2024-09-16T03:38:36.452653+0000","last_undegraded":"2024-09-16T03:38:36.452653+0000","last_fullsized":"2024-09-16T03:38:36.452653+0000","mapping_epoch":50,"log_start":"79'14","ondisk_log_start":"79'14","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'13","last_scrub_stamp":"2024-09-16T03:37:39.953656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.953656+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:46.353232+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":5791,"num_write":5,"num_write_kb":1127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.13","version":"102'63","reported_seq":328,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T03:38:36.552709+0000","last_change":"2024-09-16T03:37:57.653492+0000","last_active":"2024-09-16T03:38:36.552709+0000","last_peered":"2024-09-16T03:38:36.552709+0000","last_clean":"2024-09-16T03:38:36.552709+0000","last_became_active":"2024-09-16T03:37:57.640119+0000","last_became_peered":"2024-09-16T03:37:57.640119+0000","last_unstale":"2024-09-16T03:38:36.552709+0000","last_undegraded":"2024-09-16T03:38:36.552709+0000","last_fullsized":"2024-09-16T03:38:36.552709+0000","mapping_epoch":71,"log_start":"102'60","ondisk_log_start":"102'60","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'21","last_scrub_stamp":"2024-09-16T03:37:34.916681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.916681+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:38.040111+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621925,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":123,"num_read_kb":105662,"num_write":65,"num_write_kb":15798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4939757,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.12","version":"92'16","reported_seq":230,"reported_epoch":100,"state":"remapped+peering","last_fresh":"2024-09-16T03:38:32.563222+0000","last_change":"2024-09-16T03:38:33.377523+0000","last_active":"2024-09-16T03:38:32.563222+0000","last_peered":"2024-09-16T03:38:32.309081+0000","last_clean":"2024-09-16T03:38:32.309081+0000","last_became_active":"2024-09-16T03:37:57.651635+0000","last_became_peered":"2024-09-16T03:37:57.651635+0000","last_unstale":"2024-09-16T03:38:32.563222+0000","last_undegraded":"2024-09-16T03:38:32.563222+0000","last_fullsized":"2024-09-16T03:38:32.563222+0000","mapping_epoch":101,"log_start":"82'13","ondisk_log_start":"82'13","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":13,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067294,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":31619,"num_write":12,"num_write_kb":3255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1423481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,9,11],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.11","version":"72'12","reported_seq":211,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T03:38:35.383116+0000","last_change":"2024-09-16T03:37:50.756196+0000","last_active":"2024-09-16T03:38:35.383116+0000","last_peered":"2024-09-16T03:38:35.383116+0000","last_clean":"2024-09-16T03:38:35.383116+0000","last_became_active":"2024-09-16T03:37:49.624845+0000","last_became_peered":"2024-09-16T03:37:49.624845+0000","last_unstale":"2024-09-16T03:38:35.383116+0000","last_undegraded":"2024-09-16T03:38:35.383116+0000","last_fullsized":"2024-09-16T03:38:35.383116+0000","mapping_epoch":63,"log_start":"62'9","ondisk_log_start":"62'9","created":50,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'9","last_scrub_stamp":"2024-09-16T03:37:50.756120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:50.756120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":9,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:55.866226+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":62,"num_read_kb":56090,"num_write":4,"num_write_kb":1177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"100'58","reported_seq":300,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.475360+0000","last_change":"2024-09-16T03:37:58.667607+0000","last_active":"2024-09-16T03:38:33.475360+0000","last_peered":"2024-09-16T03:38:33.475360+0000","last_clean":"2024-09-16T03:38:33.475360+0000","last_became_active":"2024-09-16T03:37:58.567094+0000","last_became_peered":"2024-09-16T03:37:58.567094+0000","last_unstale":"2024-09-16T03:38:33.475360+0000","last_undegraded":"2024-09-16T03:38:33.475360+0000","last_fullsized":"2024-09-16T03:38:33.475360+0000","mapping_epoch":72,"log_start":"85'55","ondisk_log_start":"85'55","created":50,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":52,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:55.378004+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521083,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30361,"num_write":13,"num_write_kb":3115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1836181,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":174,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.479241+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:38:33.479241+0000","last_peered":"2024-09-16T03:38:33.479241+0000","last_clean":"2024-09-16T03:38:33.479241+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:38:33.479241+0000","last_undegraded":"2024-09-16T03:38:33.479241+0000","last_fullsized":"2024-09-16T03:38:33.479241+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.232558+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":186,"reported_epoch":102,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:38:34.727260+0000","last_change":"2024-09-16T03:38:34.726993+0000","last_active":"2024-09-16T03:38:34.727260+0000","last_peered":"2024-09-16T03:38:34.727260+0000","last_clean":"2024-09-16T03:38:34.727260+0000","last_became_active":"2024-09-16T03:38:33.484091+0000","last_became_peered":"2024-09-16T03:38:33.484091+0000","last_unstale":"2024-09-16T03:38:34.727260+0000","last_undegraded":"2024-09-16T03:38:34.727260+0000","last_fullsized":"2024-09-16T03:38:34.727260+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:37:39.701279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.701279+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":181,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.475853+0000","last_change":"2024-09-16T03:38:12.041048+0000","last_active":"2024-09-16T03:38:33.475853+0000","last_peered":"2024-09-16T03:38:33.475853+0000","last_clean":"2024-09-16T03:38:33.475853+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:38:33.475853+0000","last_undegraded":"2024-09-16T03:38:33.475853+0000","last_fullsized":"2024-09-16T03:38:33.475853+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:12.040917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.040917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.877949+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.0","version":"99'114","reported_seq":447,"reported_epoch":101,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:38:34.079055+0000","last_change":"2024-09-16T03:38:34.078879+0000","last_active":"2024-09-16T03:38:34.079055+0000","last_peered":"2024-09-16T03:38:34.079055+0000","last_clean":"2024-09-16T03:38:34.079055+0000","last_became_active":"2024-09-16T03:38:33.581141+0000","last_became_peered":"2024-09-16T03:38:33.581141+0000","last_unstale":"2024-09-16T03:38:34.079055+0000","last_undegraded":"2024-09-16T03:38:34.079055+0000","last_fullsized":"2024-09-16T03:38:34.079055+0000","mapping_epoch":32,"log_start":"95'111","ondisk_log_start":"95'111","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'52","last_scrub_stamp":"2024-09-16T03:37:39.774014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.774014+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":111,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6260673,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":89848,"num_write":35,"num_write_kb":10336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.481251+0000","last_change":"2024-09-16T03:38:33.480743+0000","last_active":"2024-09-16T03:38:33.481251+0000","last_peered":"2024-09-16T03:38:33.481251+0000","last_clean":"2024-09-16T03:38:33.481251+0000","last_became_active":"2024-09-16T03:38:33.480354+0000","last_became_peered":"2024-09-16T03:38:33.480354+0000","last_unstale":"2024-09-16T03:38:33.481251+0000","last_undegraded":"2024-09-16T03:38:33.481251+0000","last_fullsized":"2024-09-16T03:38:33.481251+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.291947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.3","version":"94'34","reported_seq":271,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T03:38:36.452704+0000","last_change":"2024-09-16T03:38:34.382583+0000","last_active":"2024-09-16T03:38:36.452704+0000","last_peered":"2024-09-16T03:38:36.452704+0000","last_clean":"2024-09-16T03:38:36.452704+0000","last_became_active":"2024-09-16T03:38:33.580022+0000","last_became_peered":"2024-09-16T03:38:33.580022+0000","last_unstale":"2024-09-16T03:38:36.452704+0000","last_undegraded":"2024-09-16T03:38:36.452704+0000","last_fullsized":"2024-09-16T03:38:36.452704+0000","mapping_epoch":96,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'21","last_scrub_stamp":"2024-09-16T03:37:39.685904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.685904+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:03.695550+0000","scrub_duration":949,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1806752,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3628,"num_write":7,"num_write_kb":1917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":973741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":178,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T03:38:35.393781+0000","last_change":"2024-09-16T03:37:52.652365+0000","last_active":"2024-09-16T03:38:35.393781+0000","last_peered":"2024-09-16T03:38:35.393781+0000","last_clean":"2024-09-16T03:38:35.393781+0000","last_became_active":"2024-09-16T03:37:52.651907+0000","last_became_peered":"2024-09-16T03:37:52.651907+0000","last_unstale":"2024-09-16T03:38:35.393781+0000","last_undegraded":"2024-09-16T03:38:35.393781+0000","last_fullsized":"2024-09-16T03:38:35.393781+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:07.210820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":208,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T03:38:36.452527+0000","last_change":"2024-09-16T03:38:35.828277+0000","last_active":"2024-09-16T03:38:36.452527+0000","last_peered":"2024-09-16T03:38:36.452527+0000","last_clean":"2024-09-16T03:38:36.452527+0000","last_became_active":"2024-09-16T03:38:33.778361+0000","last_became_peered":"2024-09-16T03:38:33.778361+0000","last_unstale":"2024-09-16T03:38:36.452527+0000","last_undegraded":"2024-09-16T03:38:36.452527+0000","last_fullsized":"2024-09-16T03:38:36.452527+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:35.828203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.828203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:43.253669+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"103'205","reported_seq":374,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T03:38:36.352760+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:38:36.352760+0000","last_peered":"2024-09-16T03:38:36.352760+0000","last_clean":"2024-09-16T03:38:36.352760+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:38:36.352760+0000","last_undegraded":"2024-09-16T03:38:36.352760+0000","last_fullsized":"2024-09-16T03:38:36.352760+0000","mapping_epoch":31,"log_start":"99'202","ondisk_log_start":"99'202","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":214,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.679811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17107639,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":184,"num_read_kb":3952,"num_write":256,"num_write_kb":58603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":175,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.477383+0000","last_change":"2024-09-16T03:38:12.795978+0000","last_active":"2024-09-16T03:38:33.477383+0000","last_peered":"2024-09-16T03:38:33.477383+0000","last_clean":"2024-09-16T03:38:33.477383+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:38:33.477383+0000","last_undegraded":"2024-09-16T03:38:33.477383+0000","last_fullsized":"2024-09-16T03:38:33.477383+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:38:12.795855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.795855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:13.088740+0000","scrub_duration":501,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.4","version":"95'31","reported_seq":253,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T03:38:36.452690+0000","last_change":"2024-09-16T03:37:46.884372+0000","last_active":"2024-09-16T03:38:36.452690+0000","last_peered":"2024-09-16T03:38:36.452690+0000","last_clean":"2024-09-16T03:38:36.452690+0000","last_became_active":"2024-09-16T03:37:31.249346+0000","last_became_peered":"2024-09-16T03:37:31.249346+0000","last_unstale":"2024-09-16T03:38:36.452690+0000","last_undegraded":"2024-09-16T03:38:36.452690+0000","last_fullsized":"2024-09-16T03:38:36.452690+0000","mapping_epoch":32,"log_start":"83'28","ondisk_log_start":"83'28","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'15","last_scrub_stamp":"2024-09-16T03:37:46.884284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:46.884284+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":28,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:50.726238+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":9455,"num_write":19,"num_write_kb":4676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.0","version":"98'194","reported_seq":367,"reported_epoch":98,"state":"remapped+peering","last_fresh":"2024-09-16T03:38:29.568717+0000","last_change":"2024-09-16T03:38:33.377357+0000","last_active":"2024-09-16T03:38:33.377357+0000","last_peered":"2024-09-16T03:38:29.568717+0000","last_clean":"2024-09-16T03:37:50.666338+0000","last_became_active":"2024-09-16T03:37:55.757919+0000","last_became_peered":"2024-09-16T03:37:55.757919+0000","last_unstale":"2024-09-16T03:38:29.568717+0000","last_undegraded":"2024-09-16T03:38:29.568717+0000","last_fullsized":"2024-09-16T03:38:29.568717+0000","mapping_epoch":101,"log_start":"98'191","ondisk_log_start":"98'191","created":31,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":212,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16965709,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":176,"num_read_kb":11034,"num_write":237,"num_write_kb":67475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,4],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":177,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T03:38:36.452466+0000","last_change":"2024-09-16T03:38:15.949627+0000","last_active":"2024-09-16T03:38:36.452466+0000","last_peered":"2024-09-16T03:38:36.452466+0000","last_clean":"2024-09-16T03:38:36.452466+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:38:36.452466+0000","last_undegraded":"2024-09-16T03:38:36.452466+0000","last_fullsized":"2024-09-16T03:38:36.452466+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:15.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:24.286324+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.7","version":"96'61","reported_seq":297,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.479696+0000","last_change":"2024-09-16T03:37:48.400988+0000","last_active":"2024-09-16T03:38:33.479696+0000","last_peered":"2024-09-16T03:38:33.479696+0000","last_clean":"2024-09-16T03:38:33.479696+0000","last_became_active":"2024-09-16T03:37:31.153814+0000","last_became_peered":"2024-09-16T03:37:31.153814+0000","last_unstale":"2024-09-16T03:38:33.479696+0000","last_undegraded":"2024-09-16T03:38:33.479696+0000","last_fullsized":"2024-09-16T03:38:33.479696+0000","mapping_epoch":32,"log_start":"95'58","ondisk_log_start":"95'58","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'32","last_scrub_stamp":"2024-09-16T03:37:48.400911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:48.400911+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:17.094567+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916097,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":106188,"num_write":36,"num_write_kb":10440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1","version":"103'158","reported_seq":325,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-16T03:38:35.740896+0000","last_change":"2024-09-16T03:38:15.890094+0000","last_active":"2024-09-16T03:38:35.740896+0000","last_peered":"2024-09-16T03:38:35.740896+0000","last_clean":"2024-09-16T03:38:35.740896+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:38:35.740896+0000","last_undegraded":"2024-09-16T03:38:35.740896+0000","last_fullsized":"2024-09-16T03:38:35.740896+0000","mapping_epoch":31,"log_start":"99'155","ondisk_log_start":"99'155","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'125","last_scrub_stamp":"2024-09-16T03:38:15.889996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.889996+0000","objects_scrubbed":18,"log_size":3,"log_dups_size":166,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:35.350472+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11141037,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":136,"num_read_kb":754,"num_write":194,"num_write_kb":46212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":186,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.580920+0000","last_change":"2024-09-16T03:38:33.580846+0000","last_active":"2024-09-16T03:38:33.580920+0000","last_peered":"2024-09-16T03:38:33.580920+0000","last_clean":"2024-09-16T03:38:33.580920+0000","last_became_active":"2024-09-16T03:38:33.580519+0000","last_became_peered":"2024-09-16T03:38:33.580519+0000","last_unstale":"2024-09-16T03:38:33.580920+0000","last_undegraded":"2024-09-16T03:38:33.580920+0000","last_fullsized":"2024-09-16T03:38:33.580920+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:24.358339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:24.358339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:53.576553+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.6","version":"96'40","reported_seq":293,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.574860+0000","last_change":"2024-09-16T03:37:56.444659+0000","last_active":"2024-09-16T03:38:33.574860+0000","last_peered":"2024-09-16T03:38:33.574860+0000","last_clean":"2024-09-16T03:38:33.574860+0000","last_became_active":"2024-09-16T03:37:56.444134+0000","last_became_peered":"2024-09-16T03:37:56.444134+0000","last_unstale":"2024-09-16T03:38:33.574860+0000","last_undegraded":"2024-09-16T03:38:33.574860+0000","last_fullsized":"2024-09-16T03:38:33.574860+0000","mapping_epoch":70,"log_start":"75'37","ondisk_log_start":"75'37","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'32","last_scrub_stamp":"2024-09-16T03:37:47.613282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.613282+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":36,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:16.654617+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924418,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":41,"num_read_kb":33828,"num_write":9,"num_write_kb":2208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1264675,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"103'182","reported_seq":418,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T03:38:36.452904+0000","last_change":"2024-09-16T03:38:01.603352+0000","last_active":"2024-09-16T03:38:36.452904+0000","last_peered":"2024-09-16T03:38:36.452904+0000","last_clean":"2024-09-16T03:38:36.452904+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:38:36.452904+0000","last_undegraded":"2024-09-16T03:38:36.452904+0000","last_fullsized":"2024-09-16T03:38:36.452904+0000","mapping_epoch":74,"log_start":"99'178","ondisk_log_start":"99'178","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":187,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.902901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14673668,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":171,"num_read_kb":7665,"num_write":221,"num_write_kb":56043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":60,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.476000+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:38:33.476000+0000","last_peered":"2024-09-16T03:38:33.476000+0000","last_clean":"2024-09-16T03:38:33.476000+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:38:33.476000+0000","last_undegraded":"2024-09-16T03:38:33.476000+0000","last_fullsized":"2024-09-16T03:38:33.476000+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:01.672864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":354,"reported_epoch":104,"state":"activating","last_fresh":"2024-09-16T03:38:36.453074+0000","last_change":"2024-09-16T03:38:36.453161+0000","last_active":"2024-09-16T03:38:35.383186+0000","last_peered":"2024-09-16T03:38:34.792650+0000","last_clean":"2024-09-16T03:38:34.792650+0000","last_became_active":"2024-09-16T03:38:34.584475+0000","last_became_peered":"2024-09-16T03:38:34.584475+0000","last_unstale":"2024-09-16T03:38:36.453074+0000","last_undegraded":"2024-09-16T03:38:36.453074+0000","last_fullsized":"2024-09-16T03:38:36.453074+0000","mapping_epoch":103,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:38:28.063690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:38:28.063690+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":["11","0","7","9"],"object_location_counts":[{"shards":"0,9,11","objects":2}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.5","version":"98'75","reported_seq":324,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-16T03:38:36.352604+0000","last_change":"2024-09-16T03:37:47.718455+0000","last_active":"2024-09-16T03:38:36.352604+0000","last_peered":"2024-09-16T03:38:36.352604+0000","last_clean":"2024-09-16T03:38:36.352604+0000","last_became_active":"2024-09-16T03:37:31.248101+0000","last_became_peered":"2024-09-16T03:37:31.248101+0000","last_unstale":"2024-09-16T03:38:36.352604+0000","last_undegraded":"2024-09-16T03:38:36.352604+0000","last_fullsized":"2024-09-16T03:38:36.352604+0000","mapping_epoch":32,"log_start":"98'72","ondisk_log_start":"98'72","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'44","last_scrub_stamp":"2024-09-16T03:37:47.718363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.718363+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":71,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:04.533062+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4773254,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":75847,"num_write":36,"num_write_kb":9464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.8","version":"91'62","reported_seq":306,"reported_epoch":100,"state":"remapped+peering","last_fresh":"2024-09-16T03:38:32.472013+0000","last_change":"2024-09-16T03:38:33.478175+0000","last_active":"2024-09-16T03:38:32.472013+0000","last_peered":"2024-09-16T03:38:32.124703+0000","last_clean":"2024-09-16T03:38:32.124703+0000","last_became_active":"2024-09-16T03:38:27.551282+0000","last_became_peered":"2024-09-16T03:38:27.551282+0000","last_unstale":"2024-09-16T03:38:32.472013+0000","last_undegraded":"2024-09-16T03:38:32.472013+0000","last_fullsized":"2024-09-16T03:38:32.472013+0000","mapping_epoch":101,"log_start":"90'59","ondisk_log_start":"90'59","created":32,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'38","last_scrub_stamp":"2024-09-16T03:37:40.343801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:40.343801+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3899224,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":58565,"num_write":28,"num_write_kb":8269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1989538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,9,10],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":165,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.475781+0000","last_change":"2024-09-16T03:37:52.558364+0000","last_active":"2024-09-16T03:38:33.475781+0000","last_peered":"2024-09-16T03:38:33.475781+0000","last_clean":"2024-09-16T03:38:33.475781+0000","last_became_active":"2024-09-16T03:37:52.558058+0000","last_became_peered":"2024-09-16T03:37:52.558058+0000","last_unstale":"2024-09-16T03:38:33.475781+0000","last_undegraded":"2024-09-16T03:38:33.475781+0000","last_fullsized":"2024-09-16T03:38:33.475781+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:46.812106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:46.812106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:56.090023+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.a","version":"99'112","reported_seq":477,"reported_epoch":101,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:38:34.188096+0000","last_change":"2024-09-16T03:38:34.187993+0000","last_active":"2024-09-16T03:38:34.188096+0000","last_peered":"2024-09-16T03:38:34.188096+0000","last_clean":"2024-09-16T03:38:34.188096+0000","last_became_active":"2024-09-16T03:38:33.582063+0000","last_became_peered":"2024-09-16T03:38:33.582063+0000","last_unstale":"2024-09-16T03:38:34.188096+0000","last_undegraded":"2024-09-16T03:38:34.188096+0000","last_fullsized":"2024-09-16T03:38:34.188096+0000","mapping_epoch":32,"log_start":"95'109","ondisk_log_start":"95'109","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'108","last_scrub_stamp":"2024-09-16T03:38:25.831424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:25.831424+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7596813,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":235,"num_read_kb":186465,"num_write":57,"num_write_kb":17351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.b","version":"98'104","reported_seq":443,"reported_epoch":103,"state":"peering","last_fresh":"2024-09-16T03:38:35.838715+0000","last_change":"2024-09-16T03:38:36.352883+0000","last_active":"2024-09-16T03:38:36.352883+0000","last_peered":"2024-09-16T03:38:35.838715+0000","last_clean":"2024-09-16T03:38:35.838715+0000","last_became_active":"2024-09-16T03:38:34.583215+0000","last_became_peered":"2024-09-16T03:38:34.583215+0000","last_unstale":"2024-09-16T03:38:35.838715+0000","last_undegraded":"2024-09-16T03:38:35.838715+0000","last_fullsized":"2024-09-16T03:38:35.838715+0000","mapping_epoch":104,"log_start":"98'98","ondisk_log_start":"98'98","created":32,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":98,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5529552,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":138298,"num_write":57,"num_write_kb":17461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8094515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,9,12],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.c","version":"99'56","reported_seq":273,"reported_epoch":101,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-16T03:38:33.980644+0000","last_change":"2024-09-16T03:38:33.880038+0000","last_active":"2024-09-16T03:38:33.980644+0000","last_peered":"2024-09-16T03:38:33.980644+0000","last_clean":"2024-09-16T03:37:50.453398+0000","last_became_active":"2024-09-16T03:38:33.778095+0000","last_became_peered":"2024-09-16T03:38:33.778095+0000","last_unstale":"2024-09-16T03:38:33.980644+0000","last_undegraded":"2024-09-16T03:38:33.980644+0000","last_fullsized":"2024-09-16T03:38:33.980644+0000","mapping_epoch":100,"log_start":"90'53","ondisk_log_start":"90'53","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:04.919920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932805,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":42622,"num_write":29,"num_write_kb":8286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,1],"avail_no_missing":["12","1","9","11","15"],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.d","version":"92'14","reported_seq":241,"reported_epoch":101,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:38:34.046645+0000","last_change":"2024-09-16T03:38:34.046371+0000","last_active":"2024-09-16T03:38:34.046645+0000","last_peered":"2024-09-16T03:38:34.046645+0000","last_clean":"2024-09-16T03:38:34.046645+0000","last_became_active":"2024-09-16T03:38:33.677674+0000","last_became_peered":"2024-09-16T03:38:33.677674+0000","last_unstale":"2024-09-16T03:38:34.046645+0000","last_undegraded":"2024-09-16T03:38:34.046645+0000","last_fullsized":"2024-09-16T03:38:34.046645+0000","mapping_epoch":72,"log_start":"91'11","ondisk_log_start":"91'11","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:27.634300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:27.634300+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":8484,"num_write":7,"num_write_kb":1931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1080040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.e","version":"92'59","reported_seq":310,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-16T03:38:34.385421+0000","last_change":"2024-09-16T03:38:34.385421+0000","last_active":"2024-09-16T03:38:34.385421+0000","last_peered":"2024-09-16T03:38:34.385421+0000","last_clean":"2024-09-16T03:38:34.385421+0000","last_became_active":"2024-09-16T03:38:33.581495+0000","last_became_peered":"2024-09-16T03:38:33.581495+0000","last_unstale":"2024-09-16T03:38:34.385421+0000","last_undegraded":"2024-09-16T03:38:34.385421+0000","last_fullsized":"2024-09-16T03:38:34.385421+0000","mapping_epoch":32,"log_start":"89'56","ondisk_log_start":"89'56","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'42","last_scrub_stamp":"2024-09-16T03:38:11.825685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:11.825685+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:03.876925+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5252187,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":92,"num_read_kb":85834,"num_write":30,"num_write_kb":8940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"99'89","reported_seq":365,"reported_epoch":101,"state":"active+clean","last_fresh":"2024-09-16T03:38:33.581791+0000","last_change":"2024-09-16T03:38:33.581662+0000","last_active":"2024-09-16T03:38:33.581791+0000","last_peered":"2024-09-16T03:38:33.581791+0000","last_clean":"2024-09-16T03:38:33.581791+0000","last_became_active":"2024-09-16T03:38:33.581172+0000","last_became_peered":"2024-09-16T03:38:33.581172+0000","last_unstale":"2024-09-16T03:38:33.581791+0000","last_undegraded":"2024-09-16T03:38:33.581791+0000","last_fullsized":"2024-09-16T03:38:33.581791+0000","mapping_epoch":32,"log_start":"98'86","ondisk_log_start":"98'86","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'64","last_scrub_stamp":"2024-09-16T03:38:11.380094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:11.380094+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":86,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3084101,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":106178,"num_write":44,"num_write_kb":11719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":36,"stat_sum":{"num_bytes":111981524,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":2446,"num_read_kb":2107666,"num_write":847,"num_write_kb":241129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":33900533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":14,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205115392,"data_stored":204920686,"data_compressed":687978,"data_compressed_allocated":737280,"data_compressed_original":1560576,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":108,"acting":108,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":59888053,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":667,"num_read_kb":23405,"num_write":908,"num_write_kb":228333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":117174272,"data_stored":125850501,"data_compressed":7999494,"data_compressed_allocated":8515584,"data_compressed_original":17510400,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":2115712,"data_compressed":126405,"data_compressed_allocated":163840,"data_compressed_original":2060288,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71940,"kb_used_data":39748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94299900,"statfs":{"total":96636764160,"available":96563097600,"internally_reserved":0,"allocated":40701952,"data_stored":41034547,"data_compressed":904304,"data_compressed_allocated":946176,"data_compressed_original":1728512,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247832,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59484,"kb_used_data":26828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94312356,"statfs":{"total":96636764160,"available":96575852544,"internally_reserved":0,"allocated":27471872,"data_stored":28325461,"data_compressed":597893,"data_compressed_allocated":655360,"data_compressed_original":1888256,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247831,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57632,"kb_used_data":24976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94314208,"statfs":{"total":96636764160,"available":96577748992,"internally_reserved":0,"allocated":25575424,"data_stored":25987333,"data_compressed":560136,"data_compressed_allocated":610304,"data_compressed_original":1372160,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280536,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42524,"kb_used_data":11332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329316,"statfs":{"total":96636764160,"available":96593219584,"internally_reserved":0,"allocated":11603968,"data_stored":12611997,"data_compressed":589603,"data_compressed_allocated":651264,"data_compressed_original":1953792,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280536,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280535,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313240,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":5460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":5591040,"data_stored":5318429,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280536,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51180,"kb_used_data":19984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320660,"statfs":{"total":96636764160,"available":96584355840,"internally_reserved":0,"allocated":20463616,"data_stored":20885082,"data_compressed":559697,"data_compressed_allocated":610304,"data_compressed_original":1368064,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[2,4,5,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280535,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91744,"kb_used_data":60548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94280096,"statfs":{"total":96636764160,"available":96542818304,"internally_reserved":0,"allocated":62001152,"data_stored":63679600,"data_compressed":1663277,"data_compressed_allocated":1794048,"data_compressed_original":3960832,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313240,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280535,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39224,"kb_used_data":6460,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332616,"statfs":{"total":96636764160,"available":96596598784,"internally_reserved":0,"allocated":6615040,"data_stored":6835105,"data_compressed":41181,"data_compressed_allocated":53248,"data_compressed_original":548864,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280535,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62140,"kb_used_data":30368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94309700,"statfs":{"total":96636764160,"available":96573132800,"internally_reserved":0,"allocated":31096832,"data_stored":31556355,"data_compressed":972847,"data_compressed_allocated":1019904,"data_compressed_original":1855488,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280535,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38080,"kb_used_data":11048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333760,"statfs":{"total":96636764160,"available":96597770240,"internally_reserved":0,"allocated":11313152,"data_stored":11584257,"data_compressed":542404,"data_compressed_allocated":577536,"data_compressed_original":1130496,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280536,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69620,"kb_used_data":37840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302220,"statfs":{"total":96636764160,"available":96565473280,"internally_reserved":0,"allocated":38748160,"data_stored":38604457,"data_compressed":237846,"data_compressed_allocated":258048,"data_compressed_original":544768,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,3,4,5,6,7,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313239,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43820,"kb_used_data":12632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328020,"statfs":{"total":96636764160,"available":96591892480,"internally_reserved":0,"allocated":12935168,"data_stored":12614727,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44048,"kb_used_data":12860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327792,"statfs":{"total":96636764160,"available":96591659008,"internally_reserved":0,"allocated":13168640,"data_stored":13600065,"data_compressed":885312,"data_compressed_allocated":925696,"data_compressed_original":1662976,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8728576,"data_stored":9528367,"data_compressed":549835,"data_compressed_allocated":598016,"data_compressed_original":1417216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10506240,"data_stored":11012984,"data_compressed":533884,"data_compressed_allocated":565248,"data_compressed_original":1105920,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9400320,"data_stored":10104536,"data_compressed":876792,"data_compressed_allocated":913408,"data_compressed_original":1638400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6774784,"data_stored":7399467,"data_compressed":463642,"data_compressed_allocated":503808,"data_compressed_original":1150976,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":26009600,"data_stored":27940818,"data_compressed":1547361,"data_compressed_allocated":1667072,"data_compressed_original":3674112,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6402048,"data_stored":7006125,"data_compressed":428817,"data_compressed_allocated":466944,"data_compressed_original":1085440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":9871360,"data_stored":10567777,"data_compressed":876792,"data_compressed_allocated":913408,"data_compressed_original":1638400,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":10506240,"data_stored":11012984,"data_compressed":533884,"data_compressed_allocated":565248,"data_compressed_original":1105920,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8069120,"data_stored":8774404,"data_compressed":523730,"data_compressed_allocated":565248,"data_compressed_original":1286144,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":9400320,"data_stored":10104536,"data_compressed":876792,"data_compressed_allocated":913408,"data_compressed_original":1638400,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2002944,"data_stored":1989538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2875392,"data_stored":2859394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2457600,"data_stored":2443231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13303808,"data_stored":13333529,"data_compressed":87535,"data_compressed_allocated":94208,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":35180544,"data_stored":35157770,"data_compressed":107396,"data_compressed_allocated":114688,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":18337792,"data_stored":18391539,"data_compressed":122799,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5775360,"data_stored":5735414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20414464,"data_stored":20416064,"data_compressed":87535,"data_compressed_allocated":94208,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":37941248,"data_stored":38031996,"data_compressed":229326,"data_compressed_allocated":245760,"data_compressed_original":520192,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":18423808,"data_stored":18365911,"data_compressed":34395,"data_compressed_allocated":36864,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":12095488,"data_stored":12039496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4771840,"data_stored":4745228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":30445568,"data_stored":30331536,"data_compressed":18992,"data_compressed_allocated":20480,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:38:37.732 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-16T03:38:37.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 46 2024-09-16T03:38:37.737 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:37.738 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:37.746 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:38:37.801 INFO:tasks.rados.rados.0.smithi157.stdout:1408: set_chunk oid 27 offset: 5632 length: 2053632 target oid f0da7483c2b661d0844ade71413e5e48f85d4939c1834eb6480cde7256cbb935 offset: 0 snap -1 2024-09-16T03:38:37.801 INFO:tasks.rados.rados.0.smithi157.stdout:1409: delete oid 3 current snap is 0 2024-09-16T03:38:37.801 INFO:tasks.rados.rados.0.smithi157.stdout:1393: got (0) Success 2024-09-16T03:38:37.801 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 0 (ObjNum 275 snap 0 seq_num 275) dirty exists 2024-09-16T03:38:37.802 INFO:tasks.rados.rados.0.smithi157.stdout:1403: finishing set_chunk to oid 30 2024-09-16T03:38:37.802 INFO:tasks.rados.rados.0.smithi157.stdout:1403: got expected ENOENT (src dne) 2024-09-16T03:38:37.802 INFO:tasks.rados.rados.0.smithi157.stdout:1405: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-16T03:38:37.806 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1003719 2024-09-16T03:38:37.810 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:37.809+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:37.853 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:37.886 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:37.884+0000 7f6528e68640 -1 osd.1 104 handle_osd_map injecting map crc failure 2024-09-16T03:38:37.893 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:37.894 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:37.903 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:38:37.911 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:37.909+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:37.966 INFO:tasks.rados.rados.0.smithi157.stdout:1406: expect (ObjNum 269 snap 0 seq_num 269) 2024-09-16T03:38:38.012 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:38.010+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:38.046 INFO:tasks.rados.rados.0.smithi157.stdout:1390: expect (ObjNum 262 snap 0 seq_num 262) 2024-09-16T03:38:38.048 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:38.048 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:38.053 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 966252 2024-09-16T03:38:38.057 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:38:38.141 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:38.139+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:38.166 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2233012 2024-09-16T03:38:38.175 INFO:tasks.rados.rados.0.smithi157.stdout:1407: got (22) Invalid argument 2024-09-16T03:38:38.175 INFO:tasks.rados.rados.0.smithi157.stdout:1408: finishing set_chunk to oid 27 2024-09-16T03:38:38.175 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 27 set_chunk returned error code -95 offset: 5632 length: 2053632 2024-09-16T03:38:38.175 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 0 (ObjNum 244 snap 0 seq_num 244) dirty exists 2024-09-16T03:38:38.175 INFO:tasks.rados.rados.0.smithi157.stdout:1401: finishing write tid 1 to smithi15736696-37 2024-09-16T03:38:38.175 INFO:tasks.rados.rados.0.smithi157.stdout:1401: finishing write tid 2 to smithi15736696-37 2024-09-16T03:38:38.175 INFO:tasks.rados.rados.0.smithi157.stdout:1390: done (11 left) 2024-09-16T03:38:38.175 INFO:tasks.rados.rados.0.smithi157.stdout:1393: done (10 left) 2024-09-16T03:38:38.175 INFO:tasks.rados.rados.0.smithi157.stdout:1402: done (9 left) 2024-09-16T03:38:38.175 INFO:tasks.rados.rados.0.smithi157.stdout:1403: done (8 left) 2024-09-16T03:38:38.176 INFO:tasks.rados.rados.0.smithi157.stdout:1405: done (7 left) 2024-09-16T03:38:38.176 INFO:tasks.rados.rados.0.smithi157.stdout:1406: done (6 left) 2024-09-16T03:38:38.176 INFO:tasks.rados.rados.0.smithi157.stdout:1407: done (5 left) 2024-09-16T03:38:38.176 INFO:tasks.rados.rados.0.smithi157.stdout:1408: done (4 left) 2024-09-16T03:38:38.176 INFO:tasks.rados.rados.0.smithi157.stdout:1409: done (3 left) 2024-09-16T03:38:38.176 INFO:tasks.rados.rados.0.smithi157.stdout:1410: tier_evict oid 34 2024-09-16T03:38:38.176 INFO:tasks.rados.rados.0.smithi157.stdout:1410: tier_evict oid 34 snap -1 2024-09-16T03:38:38.176 INFO:tasks.rados.rados.0.smithi157.stdout:1401: finishing write tid 3 to smithi15736696-37 2024-09-16T03:38:38.176 INFO:tasks.rados.rados.0.smithi157.stdout:1401: finishing write tid 4 to smithi15736696-37 2024-09-16T03:38:38.176 INFO:tasks.rados.rados.0.smithi157.stdout:1401: oid 37 updating version 0 to 64 2024-09-16T03:38:38.176 INFO:tasks.rados.rados.0.smithi157.stdout:1401: oid 37 version 64 is already newer than 62 2024-09-16T03:38:38.176 INFO:tasks.rados.rados.0.smithi157.stdout:1401: oid 37 version 64 is already newer than 63 2024-09-16T03:38:38.177 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 64 (ObjNum 283 snap 0 seq_num 283) dirty exists 2024-09-16T03:38:38.177 INFO:tasks.rados.rados.0.smithi157.stdout:1401: left oid 37 (ObjNum 283 snap 0 seq_num 283) 2024-09-16T03:38:38.177 INFO:tasks.rados.rados.0.smithi157.stdout:1401: done (3 left) 2024-09-16T03:38:38.177 INFO:tasks.rados.rados.0.smithi157.stdout:1411: read oid 15 snap -1 2024-09-16T03:38:38.177 INFO:tasks.rados.rados.0.smithi157.stdout:1411: expect deleted 2024-09-16T03:38:38.177 INFO:tasks.rados.rados.0.smithi157.stdout:1412: tier_promote oid 2 2024-09-16T03:38:38.177 INFO:tasks.rados.rados.0.smithi157.stdout:1413: read oid 4 snap -1 2024-09-16T03:38:38.177 INFO:tasks.rados.rados.0.smithi157.stdout:1413: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:38.177 INFO:tasks.rados.rados.0.smithi157.stdout:1414: write oid 38 current snap is 0 2024-09-16T03:38:38.177 INFO:tasks.rados.rados.0.smithi157.stdout:1414: seq_num 285 ranges {789995=551026,2027721=468420} 2024-09-16T03:38:38.181 INFO:tasks.rados.rados.0.smithi157.stdout:1414: writing smithi15736696-38 from 789995 to 1341021 tid 1 2024-09-16T03:38:38.188 INFO:tasks.rados.rados.0.smithi157.stdout:1414: writing smithi15736696-38 from 2027721 to 2496141 tid 2 2024-09-16T03:38:38.189 INFO:tasks.rados.rados.0.smithi157.stdout:1415: set_chunk oid 21 target oid 2024-09-16T03:38:38.189 INFO:tasks.rados.rados.0.smithi157.stdout:1415: set_chunk oid 21 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:38.189 INFO:tasks.rados.rados.0.smithi157.stdout:1416: read oid 31 snap -1 2024-09-16T03:38:38.189 INFO:tasks.rados.rados.0.smithi157.stdout:1416: expect (ObjNum 226 snap 0 seq_num 226) 2024-09-16T03:38:38.189 INFO:tasks.rados.rados.0.smithi157.stdout:1417: delete oid 45 current snap is 0 2024-09-16T03:38:38.189 INFO:tasks.rados.rados.0.smithi157.stdout:1412: got (2) No such file or directory 2024-09-16T03:38:38.189 INFO:tasks.rados.rados.0.smithi157.stdout:1412: got expected ENOENT (src dne) 2024-09-16T03:38:38.189 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 2 v 30 (ObjNum 81 snap 0 seq_num 1) clean dne 2024-09-16T03:38:38.190 INFO:tasks.rados.rados.0.smithi157.stdout:1415: finishing set_chunk to oid 21 2024-09-16T03:38:38.190 INFO:tasks.rados.rados.0.smithi157.stdout:1415: got expected ENOENT (src dne) 2024-09-16T03:38:38.191 INFO:tasks.rados.rados.0.smithi157.stdout:1411: done (9 left) 2024-09-16T03:38:38.191 INFO:tasks.rados.rados.0.smithi157.stdout:1412: done (8 left) 2024-09-16T03:38:38.191 INFO:tasks.rados.rados.0.smithi157.stdout:1415: done (7 left) 2024-09-16T03:38:38.191 INFO:tasks.rados.rados.0.smithi157.stdout:1417: done (6 left) 2024-09-16T03:38:38.191 INFO:tasks.rados.rados.0.smithi157.stdout:1418: read oid 18 snap -1 2024-09-16T03:38:38.191 INFO:tasks.rados.rados.0.smithi157.stdout:1418: expect (ObjNum 269 snap 0 seq_num 269) 2024-09-16T03:38:38.191 INFO:tasks.rados.rados.0.smithi157.stdout:1419: read oid 41 snap -1 2024-09-16T03:38:38.191 INFO:tasks.rados.rados.0.smithi157.stdout:1419: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-16T03:38:38.191 INFO:tasks.rados.rados.0.smithi157.stdout:1420: read oid 32 snap -1 2024-09-16T03:38:38.191 INFO:tasks.rados.rados.0.smithi157.stdout:1420: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-16T03:38:38.192 INFO:tasks.rados.rados.0.smithi157.stdout:1421: read oid 19 snap -1 2024-09-16T03:38:38.192 INFO:tasks.rados.rados.0.smithi157.stdout:1421: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:38.192 INFO:tasks.rados.rados.0.smithi157.stdout:1422: tier_flush oid 1 2024-09-16T03:38:38.192 INFO:tasks.rados.rados.0.smithi157.stdout:1422: tier_flush oid 1 snap -1 2024-09-16T03:38:38.192 INFO:tasks.rados.rados.0.smithi157.stdout:1423: set_chunk oid 26 target oid 2024-09-16T03:38:38.192 INFO:tasks.rados.rados.0.smithi157.stdout:1423: set_chunk oid 26 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:38.192 INFO:tasks.rados.rados.0.smithi157.stdout:1424: write (excl) oid 44 current snap is 0 2024-09-16T03:38:38.192 INFO:tasks.rados.rados.0.smithi157.stdout:1424: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-09-16T03:38:38.196 INFO:tasks.rados.rados.0.smithi157.stdout:1424: writing smithi15736696-44 from 423609 to 1125133 tid 1 2024-09-16T03:38:38.202 INFO:tasks.rados.rados.0.smithi157.stdout:1424: writing smithi15736696-44 from 1628332 to 2100421 tid 2 2024-09-16T03:38:38.205 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:38.205 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:38.206 INFO:tasks.rados.rados.0.smithi157.stdout:1424: writing smithi15736696-44 from 2648987 to 2823609 tid 3 2024-09-16T03:38:38.206 INFO:tasks.rados.rados.0.smithi157.stdout:1425: read oid 36 snap -1 2024-09-16T03:38:38.206 INFO:tasks.rados.rados.0.smithi157.stdout:1425: expect (ObjNum 267 snap 0 seq_num 267) 2024-09-16T03:38:38.206 INFO:tasks.rados.rados.0.smithi157.stdout:1426: write (excl) oid 20 current snap is 0 2024-09-16T03:38:38.206 INFO:tasks.rados.rados.0.smithi157.stdout:1426: seq_num 288 ranges {440416=776773,1686814=732100} 2024-09-16T03:38:38.213 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:38.211+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:38.213 INFO:tasks.rados.rados.0.smithi157.stdout:1426: writing smithi15736696-20 from 440416 to 1217189 tid 1 2024-09-16T03:38:38.214 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:38:38.220 INFO:tasks.rados.rados.0.smithi157.stdout:1426: writing smithi15736696-20 from 1686814 to 2418914 tid 2 2024-09-16T03:38:38.220 INFO:tasks.rados.rados.0.smithi157.stdout:1427: read oid 14 snap -1 2024-09-16T03:38:38.220 INFO:tasks.rados.rados.0.smithi157.stdout:1427: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-16T03:38:38.220 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:38.293 INFO:tasks.rados.rados.0.smithi157.stdout:1420: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-16T03:38:38.313 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:38.311+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:38.343 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:38.343 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:38.351 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2413897 2024-09-16T03:38:38.352 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:38:38.357 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:38.354+0000 7f49419d0640 -1 osd.4 105 handle_osd_map injecting map crc failure 2024-09-16T03:38:38.358 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 4 pg_num to 46 2024-09-16T03:38:38.359 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1368901 2024-09-16T03:38:38.377 INFO:tasks.rados.rados.0.smithi157.stdout:1416: expect (ObjNum 226 snap 0 seq_num 226) 2024-09-16T03:38:38.414 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:38.412+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:38.458 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:38.459 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:38.467 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:38:38.472 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2442485 2024-09-16T03:38:38.513 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:38.515 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:38.513+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:38.577 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:38.577 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:38.585 INFO:tasks.rados.rados.0.smithi157.stdout:1410: got (22) Invalid argument 2024-09-16T03:38:38.585 INFO:tasks.rados.rados.0.smithi157.stdout:1413: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-16T03:38:38.586 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:38:38.616 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:38.614+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:38.662 INFO:tasks.rados.rados.0.smithi157.stdout:1404: got (0) Success 2024-09-16T03:38:38.662 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 0 (ObjNum 276 snap 0 seq_num 276) dirty exists 2024-09-16T03:38:38.662 INFO:tasks.rados.rados.0.smithi157.stdout:1418: expect (ObjNum 269 snap 0 seq_num 269) 2024-09-16T03:38:38.691 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:38.718 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:38.716+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:38.733 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:38.733 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:38.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:38:38.748 INFO:tasks.rados.rados.0.smithi157.stdout:1423: finishing set_chunk to oid 26 2024-09-16T03:38:38.748 INFO:tasks.rados.rados.0.smithi157.stdout:1423: got expected ENOENT (src dne) 2024-09-16T03:38:38.748 INFO:tasks.rados.rados.0.smithi157.stdout:1419: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-16T03:38:38.753 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 910713 2024-09-16T03:38:38.792 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:38.818 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:38.816+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:38.839 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:38.851 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:38.851 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:38.859 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:38:38.886 INFO:tasks.rados.rados.0.smithi157.stdout:1426: finishing write tid 1 to smithi15736696-20 2024-09-16T03:38:38.886 INFO:tasks.rados.rados.0.smithi157.stdout:1404: done (15 left) 2024-09-16T03:38:38.886 INFO:tasks.rados.rados.0.smithi157.stdout:1410: done (14 left) 2024-09-16T03:38:38.887 INFO:tasks.rados.rados.0.smithi157.stdout:1413: done (13 left) 2024-09-16T03:38:38.887 INFO:tasks.rados.rados.0.smithi157.stdout:1416: done (12 left) 2024-09-16T03:38:38.887 INFO:tasks.rados.rados.0.smithi157.stdout:1418: done (11 left) 2024-09-16T03:38:38.887 INFO:tasks.rados.rados.0.smithi157.stdout:1419: done (10 left) 2024-09-16T03:38:38.887 INFO:tasks.rados.rados.0.smithi157.stdout:1420: done (9 left) 2024-09-16T03:38:38.887 INFO:tasks.rados.rados.0.smithi157.stdout:1423: done (8 left) 2024-09-16T03:38:38.887 INFO:tasks.rados.rados.0.smithi157.stdout:1428: write (excl) oid 17 current snap is 0 2024-09-16T03:38:38.887 INFO:tasks.rados.rados.0.smithi157.stdout:1428: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-09-16T03:38:38.891 INFO:tasks.rados.rados.0.smithi157.stdout:1428: writing smithi15736696-17 from 457223 to 1025598 tid 1 2024-09-16T03:38:38.896 INFO:tasks.rados.rados.0.smithi157.stdout:1428: writing smithi15736696-17 from 1461649 to 1937407 tid 2 2024-09-16T03:38:38.903 INFO:tasks.rados.rados.0.smithi157.stdout:1428: writing smithi15736696-17 from 2420186 to 2857223 tid 3 2024-09-16T03:38:38.903 INFO:tasks.rados.rados.0.smithi157.stdout:1429: read oid 29 snap -1 2024-09-16T03:38:38.903 INFO:tasks.rados.rados.0.smithi157.stdout:1429: expect (ObjNum 246 snap 0 seq_num 246) 2024-09-16T03:38:38.903 INFO:tasks.rados.rados.0.smithi157.stdout:1430: read oid 42 snap -1 2024-09-16T03:38:38.903 INFO:tasks.rados.rados.0.smithi157.stdout:1430: expect (ObjNum 275 snap 0 seq_num 275) 2024-09-16T03:38:38.903 INFO:tasks.rados.rados.0.smithi157.stdout:1431: delete oid 18 current snap is 0 2024-09-16T03:38:38.903 INFO:tasks.rados.rados.0.smithi157.stdout:1427: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-16T03:38:38.920 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:38.918+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:38.927 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:38.988 INFO:tasks.rados.rados.0.smithi157.stdout:1400: expect (ObjNum 236 snap 0 seq_num 236) 2024-09-16T03:38:39.007 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:39.007 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:39.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:38:39.061 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:39.058+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:39.122 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:39.121+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:39.124 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:39.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:38:39.166 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 723888 2024-09-16T03:38:39.223 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:39.221+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:39.246 INFO:tasks.rados.rados.0.smithi157.stdout:1414: finishing write tid 1 to smithi15736696-38 2024-09-16T03:38:39.246 INFO:tasks.rados.rados.0.smithi157.stdout:1414: finishing write tid 2 to smithi15736696-38 2024-09-16T03:38:39.246 INFO:tasks.rados.rados.0.smithi157.stdout:1400: done (11 left) 2024-09-16T03:38:39.246 INFO:tasks.rados.rados.0.smithi157.stdout:1427: done (10 left) 2024-09-16T03:38:39.246 INFO:tasks.rados.rados.0.smithi157.stdout:1431: done (9 left) 2024-09-16T03:38:39.246 INFO:tasks.rados.rados.0.smithi157.stdout:1432: write oid 31 current snap is 0 2024-09-16T03:38:39.247 INFO:tasks.rados.rados.0.smithi157.stdout:1421: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-16T03:38:39.282 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:39.282 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:39.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:38:39.324 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:39.322+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:39.350 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:39.367 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:38:39.405 INFO:tasks.rados.rados.0.smithi157.stdout:1425: expect (ObjNum 267 snap 0 seq_num 267) 2024-09-16T03:38:39.425 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:39.424+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:39.439 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:39.440 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:39.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:38:39.455 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 421596 2024-09-16T03:38:39.480 INFO:tasks.rados.rados.0.smithi157.stdout:1426: finishing write tid 2 to smithi15736696-20 2024-09-16T03:38:39.480 INFO:tasks.rados.rados.0.smithi157.stdout:1426: finishing write tid 3 to smithi15736696-20 2024-09-16T03:38:39.480 INFO:tasks.rados.rados.0.smithi157.stdout:1432: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-09-16T03:38:39.486 INFO:tasks.rados.rados.0.smithi157.stdout:1432: writing smithi15736696-31 from 490837 to 1209710 tid 1 2024-09-16T03:38:39.492 INFO:tasks.rados.rados.0.smithi157.stdout:1432: writing smithi15736696-31 from 1862260 to 2341687 tid 2 2024-09-16T03:38:39.496 INFO:tasks.rados.rados.0.smithi157.stdout:1432: writing smithi15736696-31 from 2875032 to 2890837 tid 3 2024-09-16T03:38:39.496 INFO:tasks.rados.rados.0.smithi157.stdout:1421: done (9 left) 2024-09-16T03:38:39.496 INFO:tasks.rados.rados.0.smithi157.stdout:1425: done (8 left) 2024-09-16T03:38:39.496 INFO:tasks.rados.rados.0.smithi157.stdout:1433: read oid 48 snap -1 2024-09-16T03:38:39.496 INFO:tasks.rados.rados.0.smithi157.stdout:1433: expect deleted 2024-09-16T03:38:39.496 INFO:tasks.rados.rados.0.smithi157.stdout:1434: set_chunk oid 10 target oid 2024-09-16T03:38:39.527 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:39.524+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:39.560 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:39.561 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:39.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:38:39.654 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:39.651+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:39.718 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:39.718 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:39.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:38:39.728 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:39.727+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:39.735 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:38:39.809 INFO:tasks.rados.rados.0.smithi157.stdout:1434: set_chunk oid 10 offset: 22528 length: 437760 target oid 6b46815293b63c523a0d78f7f1696adb187f23d99ae7e88cdcfc300779153c86 offset: 0 snap -1 2024-09-16T03:38:39.810 INFO:tasks.rados.rados.0.smithi157.stdout:1435: set_chunk oid 28 target oid 2024-09-16T03:38:39.829 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:39.828+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:39.877 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:39.877 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:39.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:38:39.893 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:39.894 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:39.912 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:39.912 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:39.930 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:39.928+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:39.930 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:39.930 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:39.948 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:39.948 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:39.966 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:39.966 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:39.985 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:39.985 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:39.996 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:39.996 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:40.000 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:40.001 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:40.006 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:38:40.019 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:40.019 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:40.030 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:40.028+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:40.035 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:40.035 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:40.052 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:40.053 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:40.086 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:40.086 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:40.103 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:40.103 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:40.117 INFO:tasks.rados.rados.0.smithi157.stdout:1435: set_chunk oid 28 offset: 666624 length: 1219072 target oid a66ed998f6a4574e959d95e9d2d1c90dc34e0e29de8459152c5d370a337bd09a offset: 0 snap -1 2024-09-16T03:38:40.117 INFO:tasks.rados.rados.0.smithi157.stdout:1436: write (excl) oid 47 current snap is 0 2024-09-16T03:38:40.118 INFO:tasks.rados.rados.0.smithi157.stdout:1426: finishing write tid 4 to smithi15736696-20 2024-09-16T03:38:40.118 INFO:tasks.rados.rados.0.smithi157.stdout:1426: oid 20 updating version 0 to 41 2024-09-16T03:38:40.118 INFO:tasks.rados.rados.0.smithi157.stdout:1426: oid 20 updating version 41 to 43 2024-09-16T03:38:40.118 INFO:tasks.rados.rados.0.smithi157.stdout:1426: oid 20 version 43 is already newer than 42 2024-09-16T03:38:40.118 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 43 (ObjNum 288 snap 0 seq_num 288) dirty exists 2024-09-16T03:38:40.118 INFO:tasks.rados.rados.0.smithi157.stdout:1426: left oid 20 (ObjNum 288 snap 0 seq_num 288) 2024-09-16T03:38:40.118 INFO:tasks.rados.rados.0.smithi157.stdout:1414: finishing write tid 3 to smithi15736696-38 2024-09-16T03:38:40.118 INFO:tasks.rados.rados.0.smithi157.stdout:1424: finishing write tid 1 to smithi15736696-44 2024-09-16T03:38:40.118 INFO:tasks.rados.rados.0.smithi157.stdout:1424: finishing write tid 2 to smithi15736696-44 2024-09-16T03:38:40.118 INFO:tasks.rados.rados.0.smithi157.stdout:1424: finishing write tid 3 to smithi15736696-44 2024-09-16T03:38:40.118 INFO:tasks.rados.rados.0.smithi157.stdout:1424: finishing write tid 4 to smithi15736696-44 2024-09-16T03:38:40.118 INFO:tasks.rados.rados.0.smithi157.stdout:1436: seq_num 292 ranges {507644=794122,1920742=623085} 2024-09-16T03:38:40.120 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:40.120 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:40.124 INFO:tasks.rados.rados.0.smithi157.stdout:1436: writing smithi15736696-47 from 507644 to 1301766 tid 1 2024-09-16T03:38:40.132 INFO:tasks.rados.rados.0.smithi157.stdout:1436: writing smithi15736696-47 from 1920742 to 2543827 tid 2 2024-09-16T03:38:40.132 INFO:tasks.rados.rados.0.smithi157.stdout:1426: done (11 left) 2024-09-16T03:38:40.132 INFO:tasks.rados.rados.0.smithi157.stdout:1437: read oid 40 snap -1 2024-09-16T03:38:40.132 INFO:tasks.rados.rados.0.smithi157.stdout:1437: expect (ObjNum 245 snap 0 seq_num 245) 2024-09-16T03:38:40.132 INFO:tasks.rados.rados.0.smithi157.stdout:1438: read oid 35 snap -1 2024-09-16T03:38:40.132 INFO:tasks.rados.rados.0.smithi157.stdout:1438: expect (ObjNum 265 snap 0 seq_num 265) 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:1439: tier_flush oid 18 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:1439: tier_flush oid 18 snap -1 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:1424: finishing write tid 5 to smithi15736696-44 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:1424: oid 44 updating version 0 to 62 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:1424: oid 44 version 62 is already newer than 61 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:1424: oid 44 version 62 is already newer than 60 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:1424: oid 44 updating version 62 to 63 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 63 (ObjNum 287 snap 0 seq_num 287) dirty exists 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:1424: left oid 44 (ObjNum 287 snap 0 seq_num 287) 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:1422: got (0) Success 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 0 (ObjNum 242 snap 0 seq_num 242) dirty exists 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:1422: done (13 left) 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:1424: done (12 left) 2024-09-16T03:38:40.133 INFO:tasks.rados.rados.0.smithi157.stdout:1440: read oid 25 snap -1 2024-09-16T03:38:40.134 INFO:tasks.rados.rados.0.smithi157.stdout:1440: expect deleted 2024-09-16T03:38:40.134 INFO:tasks.rados.rados.0.smithi157.stdout:1441: delete oid 39 current snap is 0 2024-09-16T03:38:40.134 INFO:tasks.rados.rados.0.smithi157.stdout:1414: finishing write tid 4 to smithi15736696-38 2024-09-16T03:38:40.134 INFO:tasks.rados.rados.0.smithi157.stdout:1414: oid 38 updating version 0 to 35 2024-09-16T03:38:40.134 INFO:tasks.rados.rados.0.smithi157.stdout:1414: oid 38 updating version 35 to 36 2024-09-16T03:38:40.134 INFO:tasks.rados.rados.0.smithi157.stdout:1414: oid 38 updating version 36 to 37 2024-09-16T03:38:40.134 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 37 (ObjNum 285 snap 0 seq_num 285) dirty exists 2024-09-16T03:38:40.134 INFO:tasks.rados.rados.0.smithi157.stdout:1414: left oid 38 (ObjNum 285 snap 0 seq_num 285) 2024-09-16T03:38:40.134 INFO:tasks.rados.rados.0.smithi157.stdout:1428: finishing write tid 1 to smithi15736696-17 2024-09-16T03:38:40.134 INFO:tasks.rados.rados.0.smithi157.stdout:1428: finishing write tid 2 to smithi15736696-17 2024-09-16T03:38:40.134 INFO:tasks.rados.rados.0.smithi157.stdout:1429: expect (ObjNum 246 snap 0 seq_num 246) 2024-09-16T03:38:40.135 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:40.129+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:40.138 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:40.139 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:40.139 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:40.139 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:40.139 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:38:40.140 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:40.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:38:40.155 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:40.156 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:40.173 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:40.174 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:40.180 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1362 2024-09-16T03:38:40.227 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:40.231 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:40.230+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:40.264 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:40.264 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:40.264 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:38:40.265 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:38:40.265 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:40.272 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:38:40.273 INFO:tasks.rados.rados.0.smithi157.stdout:1430: expect (ObjNum 275 snap 0 seq_num 275) 2024-09-16T03:38:40.333 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:40.330+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:40.382 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:40.382 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:38:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:38:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:432 4.c 4:34492f38:::smithi15736696-33:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:38:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.465914574999999, 2024-09-16T03:38:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.404460861, 2024-09-16T03:38:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.383 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 432 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913833+0000", 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0610000000000001e-06 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913839+0000", 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9479999999999996e-06 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913848+0000", 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5629999999999992e-06 2024-09-16T03:38:40.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913857+0000", 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0399999999999998e-06 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914022+0000", 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000165422 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914101+0000", 2024-09-16T03:38:40.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8462999999999997e-05 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217171+0000", 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30307040000000002 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217854+0000", 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068278700000000002 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.218098+0000", 2024-09-16T03:38:40.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243825 2024-09-16T03:38:40.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318164+0000", 2024-09-16T03:38:40.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100066451 2024-09-16T03:38:40.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318232+0000", 2024-09-16T03:38:40.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8113000000000003e-05 2024-09-16T03:38:40.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318293+0000", 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0788000000000001e-05 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:790 4.d 4:b3fe05b2:::smithi15736696-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.491774500999995, 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41456747799999999, 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.389 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 790 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887975+0000", 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0369999999999999e-06 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887987+0000", 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1606e-05 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888016+0000", 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.917e-05 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888032+0000", 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5616000000000001e-05 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888178+0000", 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612100000000001 2024-09-16T03:38:40.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888307+0000", 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012953899999999999 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.302540+0000", 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41423238899999998 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1418 4.c 4:3313e97c:::smithi15736696-21:head [write 615156~555834] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.153663328999997, 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40891108799999998, 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1418 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226087+0000", 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8890000000000004e-06 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226827+0000", 2024-09-16T03:38:40.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000739298 2024-09-16T03:38:40.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226848+0000", 2024-09-16T03:38:40.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1418000000000001e-05 2024-09-16T03:38:40.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226858+0000", 2024-09-16T03:38:40.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0434e-05 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327036+0000", 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10017769 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327098+0000", 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1760999999999994e-05 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:40.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327230+0000", 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131839 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434369+0000", 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107139788 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434444+0000", 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4171999999999994e-05 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634830+0000", 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20038660599999999 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634937+0000", 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010716499999999999 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634994+0000", 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7027999999999997e-05 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.399 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.400 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.400 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1419 4.c 4:3313e97c:::smithi15736696-21:head [write 1603467~416361] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:40.400 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:38:40.400 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.148997940000001, 2024-09-16T03:38:40.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40430432100000002, 2024-09-16T03:38:40.400 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.400 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.400 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.400 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1419 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230752+0000", 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.354e-06 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231164+0000", 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041208599999999998 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231179+0000", 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4885999999999999e-05 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231186+0000", 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8759999999999999e-06 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328726+0000", 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097539855999999994 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328786+0000", 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0059000000000002e-05 2024-09-16T03:38:40.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:40.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328890+0000", 2024-09-16T03:38:40.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104338 2024-09-16T03:38:40.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434475+0000", 2024-09-16T03:38:40.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.105584709 2024-09-16T03:38:40.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.534594+0000", 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10011914099999999 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635003+0000", 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100408834 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635029+0000", 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6551999999999999e-05 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635053+0000", 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3629999999999999e-05 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1420 4.c 4:3313e97c:::smithi15736696-21:head [write 2816389~693495] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.141083629000001, 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39653502600000001, 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.406 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1420 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238663+0000", 2024-09-16T03:38:40.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.238666+0000", 2024-09-16T03:38:40.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.163e-06 2024-09-16T03:38:40.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239395+0000", 2024-09-16T03:38:40.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072868599999999998 2024-09-16T03:38:40.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239410+0000", 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5330999999999999e-05 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239419+0000", 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7360000000000007e-06 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332338+0000", 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092919165999999997 2024-09-16T03:38:40.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332435+0000", 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6972999999999995e-05 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.332544+0000", 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010898700000000001 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634808+0000", 2024-09-16T03:38:40.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30226407700000002 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635058+0000", 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025009300000000001 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635103+0000", 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4539000000000002e-05 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635145+0000", 2024-09-16T03:38:40.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1938999999999999e-05 2024-09-16T03:38:40.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635198+0000", 2024-09-16T03:38:40.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3335999999999999e-05 2024-09-16T03:38:40.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.412 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.412 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1421 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:40.412 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.140315180000002, 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.497492088, 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1421 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239432+0000", 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5999999999999991e-07 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239436+0000", 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239443+0000", 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.968e-06 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239447+0000", 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6969999999999999e-06 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635225+0000", 2024-09-16T03:38:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39577807399999998 2024-09-16T03:38:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635290+0000", 2024-09-16T03:38:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4583000000000002e-05 2024-09-16T03:38:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:38:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635410+0000", 2024-09-16T03:38:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120684 2024-09-16T03:38:40.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636180+0000", 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076997899999999995 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636799+0000", 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061906600000000004 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736742+0000", 2024-09-16T03:38:40.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099943041999999996 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736843+0000", 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010076799999999999 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736924+0000", 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0419000000000004e-05 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.419 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1422 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:38:40.419 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:40.419 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.140293900000003, 2024-09-16T03:38:40.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49764818700000002, 2024-09-16T03:38:40.419 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.419 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.419 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:40.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.419 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.419 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1422 2024-09-16T03:38:40.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.420 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:40.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:40.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:38:40.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:38:40.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239455+0000", 2024-09-16T03:38:40.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9860000000000001e-06 2024-09-16T03:38:40.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239461+0000", 2024-09-16T03:38:40.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:38:40.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239464+0000", 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2619999999999999e-06 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635736+0000", 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.396272287 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635790+0000", 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4252000000000002e-05 2024-09-16T03:38:40.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736964+0000", 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101173319 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737024+0000", 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9975000000000001e-05 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737101+0000", 2024-09-16T03:38:40.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7303000000000002e-05 2024-09-16T03:38:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1684 4.d 4:b3fe05b2:::smithi15736696-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.225398783999999, 2024-09-16T03:38:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39825815199999998, 2024-09-16T03:38:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:40.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1684 2024-09-16T03:38:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154348+0000", 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9999999999999998e-07 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154351+0000", 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.249e-06 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154356+0000", 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7690000000000002e-06 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.154361+0000", 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4209999999999997e-06 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.253488+0000", 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099126895000000007 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.253585+0000", 2024-09-16T03:38:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7204e-05 2024-09-16T03:38:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552451+0000", 2024-09-16T03:38:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.298866358 2024-09-16T03:38:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552521+0000", 2024-09-16T03:38:40.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9873e-05 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.552606+0000", 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4882999999999994e-05 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1749 4.c 4:34492f38:::smithi15736696-33:head [assert-version v40,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:56.368656+0000", 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.011090504000002, 2024-09-16T03:38:40.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30391657799999999, 2024-09-16T03:38:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1749 2024-09-16T03:38:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368656+0000", 2024-09-16T03:38:40.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368656+0000", 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368660+0000", 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.693e-06 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368673+0000", 2024-09-16T03:38:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2944e-05 2024-09-16T03:38:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368702+0000", 2024-09-16T03:38:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9365000000000001e-05 2024-09-16T03:38:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368719+0000", 2024-09-16T03:38:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6756999999999999e-05 2024-09-16T03:38:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368857+0000", 2024-09-16T03:38:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000138166 2024-09-16T03:38:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.368950+0000", 2024-09-16T03:38:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2843000000000005e-05 2024-09-16T03:38:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.471081+0000", 2024-09-16T03:38:40.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102130647 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.471807+0000", 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072618800000000001 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.547167+0000", 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075359801000000004 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.572606+0000", 2024-09-16T03:38:40.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025439215000000001 2024-09-16T03:38:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.672443+0000", 2024-09-16T03:38:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099837710999999996 2024-09-16T03:38:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.672493+0000", 2024-09-16T03:38:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9645999999999999e-05 2024-09-16T03:38:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.672573+0000", 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9602000000000001e-05 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2334 4.c 4:34492f38:::smithi15736696-33:head [write 3060800~250179] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.094561+0000", 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.285185759000001, 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30670181299999999, 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.436 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2334 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.094561+0000", 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.094561+0000", 2024-09-16T03:38:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.094562+0000", 2024-09-16T03:38:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.026e-06 2024-09-16T03:38:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095003+0000", 2024-09-16T03:38:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000441366 2024-09-16T03:38:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095011+0000", 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1380000000000007e-06 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095016+0000", 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2459999999999997e-06 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.188911+0000", 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093895270000000003 2024-09-16T03:38:40.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.188978+0000", 2024-09-16T03:38:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7021999999999996e-05 2024-09-16T03:38:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.189075+0000", 2024-09-16T03:38:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7535999999999996e-05 2024-09-16T03:38:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.190081+0000", 2024-09-16T03:38:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001005875 2024-09-16T03:38:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.190314+0000", 2024-09-16T03:38:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000233131 2024-09-16T03:38:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.202489+0000", 2024-09-16T03:38:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012174574000000001 2024-09-16T03:38:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.401033+0000", 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19854360099999999 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.401107+0000", 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3924000000000006e-05 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.401263+0000", 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015610400000000001 2024-09-16T03:38:40.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2335 4.c 4:34492f38:::smithi15736696-33:head [setxattr _header (56) in=7b,truncate 3310979] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.095024+0000", 2024-09-16T03:38:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.284722581, 2024-09-16T03:38:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.306438074, 2024-09-16T03:38:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2335 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095024+0000", 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095024+0000", 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095025+0000", 2024-09-16T03:38:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8100000000000003e-07 2024-09-16T03:38:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095027+0000", 2024-09-16T03:38:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5579999999999999e-06 2024-09-16T03:38:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095031+0000", 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.585e-06 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095034+0000", 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8710000000000001e-06 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.189600+0000", 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094566501999999997 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.189658+0000", 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7969e-05 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.189782+0000", 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012398800000000001 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.190356+0000", 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057428700000000004 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.202888+0000", 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012531588 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.390802+0000", 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18791423199999999 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.401345+0000", 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010542924 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.401383+0000", 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7781000000000003e-05 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.401462+0000", 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9208000000000005e-05 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.449 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2336 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.284706782000001, 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.407180869, 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2336 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:38:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:38:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:38:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2399999999999999e-07 2024-09-16T03:38:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095042+0000", 2024-09-16T03:38:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.567e-06 2024-09-16T03:38:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095046+0000", 2024-09-16T03:38:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.639e-06 2024-09-16T03:38:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195114+0000", 2024-09-16T03:38:40.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10006748 2024-09-16T03:38:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195173+0000", 2024-09-16T03:38:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8727000000000002e-05 2024-09-16T03:38:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195278+0000", 2024-09-16T03:38:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010522900000000001 2024-09-16T03:38:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.501699+0000", 2024-09-16T03:38:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30642159299999999 2024-09-16T03:38:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.501995+0000", 2024-09-16T03:38:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000295822 2024-09-16T03:38:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.502221+0000", 2024-09-16T03:38:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022548800000000001 2024-09-16T03:38:40.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2450 4.c 4:34492f38:::smithi15736696-33:head [write 2113302~674388] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:38:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:15.459245+0000", 2024-09-16T03:38:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.920501671, 2024-09-16T03:38:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30558659100000002, 2024-09-16T03:38:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2450 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.459245+0000", 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.459245+0000", 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.459247+0000", 2024-09-16T03:38:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9250000000000002e-06 2024-09-16T03:38:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.460572+0000", 2024-09-16T03:38:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013251649999999999 2024-09-16T03:38:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.460581+0000", 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3060000000000003e-06 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.460586+0000", 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9350000000000002e-06 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463062+0000", 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024754730000000002 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463119+0000", 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7635e-05 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463219+0000", 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9729999999999996e-05 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.560780+0000", 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097560981000000005 2024-09-16T03:38:40.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.566417+0000", 2024-09-16T03:38:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056371629999999997 2024-09-16T03:38:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.566537+0000", 2024-09-16T03:38:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011941499999999999 2024-09-16T03:38:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.764722+0000", 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19818529800000001 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.764761+0000", 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8831000000000001e-05 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.764832+0000", 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0734000000000004e-05 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2452 4.c 4:34492f38:::smithi15736696-33:head [setxattr _header (56) in=7b,truncate 3479049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:15.463009+0000", 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.916737469000001, 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40224235800000002, 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2452 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463009+0000", 2024-09-16T03:38:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463009+0000", 2024-09-16T03:38:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463010+0000", 2024-09-16T03:38:40.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6000000000000004e-07 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463012+0000", 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5100000000000001e-06 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463017+0000", 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5959999999999997e-06 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463020+0000", 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2339999999999999e-06 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.465322+0000", 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002301464 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.465374+0000", 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2945999999999998e-05 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.465470+0000", 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5445000000000005e-05 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.560795+0000", 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095324675999999997 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.566503+0000", 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057087659999999997 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.566596+0000", 2024-09-16T03:38:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3107000000000002e-05 2024-09-16T03:38:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865113+0000", 2024-09-16T03:38:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29851622100000003 2024-09-16T03:38:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865160+0000", 2024-09-16T03:38:40.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6863000000000002e-05 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865252+0000", 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1970000000000002e-05 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2453 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:15.463024+0000", 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.916722485000001, 2024-09-16T03:38:40.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40260837500000002, 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2453 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463024+0000", 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463024+0000", 2024-09-16T03:38:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463025+0000", 2024-09-16T03:38:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4499999999999998e-07 2024-09-16T03:38:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463026+0000", 2024-09-16T03:38:40.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4500000000000001e-06 2024-09-16T03:38:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463030+0000", 2024-09-16T03:38:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5449999999999997e-06 2024-09-16T03:38:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463034+0000", 2024-09-16T03:38:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0309999999999999e-06 2024-09-16T03:38:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.560713+0000", 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097679777999999995 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.560765+0000", 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2021000000000002e-05 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865314+0000", 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30454909000000002 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865423+0000", 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108342 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865633+0000", 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020977300000000001 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2642 4.c 4:34492f38:::smithi15736696-33:head [write 665610~797860,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:40.475 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.503560037, 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40488026700000002, 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2642 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:38:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:38:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876189+0000", 2024-09-16T03:38:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0549999999999998e-06 2024-09-16T03:38:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877003+0000", 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081467900000000001 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877018+0000", 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.471e-05 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877025+0000", 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.379e-06 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977237+0000", 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100211914 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977354+0000", 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116416 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977480+0000", 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012629899999999999 2024-09-16T03:38:40.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.984474+0000", 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069943230000000002 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084635+0000", 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100161018 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185809+0000", 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10117406700000001 2024-09-16T03:38:40.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.280666+0000", 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094856491000000001 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.280857+0000", 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019133999999999999 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.281067+0000", 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000209576 2024-09-16T03:38:40.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2643 4.c 4:34492f38:::smithi15736696-33:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:38:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.498893151000001, 2024-09-16T03:38:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50004631700000002, 2024-09-16T03:38:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2643 2024-09-16T03:38:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:38:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:38:40.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880854+0000", 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6599999999999994e-07 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881441+0000", 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058613000000000003 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881450+0000", 2024-09-16T03:38:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6260000000000007e-06 2024-09-16T03:38:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881455+0000", 2024-09-16T03:38:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.515e-06 2024-09-16T03:38:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979409+0000", 2024-09-16T03:38:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097954645000000007 2024-09-16T03:38:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979512+0000", 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102701 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979684+0000", 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017176299999999999 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084699+0000", 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10501545599999999 2024-09-16T03:38:40.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084714+0000", 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4738e-05 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185858+0000", 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101144074 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380720+0000", 2024-09-16T03:38:40.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.194862072 2024-09-16T03:38:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380783+0000", 2024-09-16T03:38:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3262000000000005e-05 2024-09-16T03:38:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380900+0000", 2024-09-16T03:38:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116369 2024-09-16T03:38:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.488 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2644 4.c 4:34492f38:::smithi15736696-33:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.494216655999999, 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.495569859, 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2644 2024-09-16T03:38:40.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:38:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:38:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885531+0000", 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885813+0000", 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028188399999999999 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885824+0000", 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885830+0000", 2024-09-16T03:38:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6289999999999998e-06 2024-09-16T03:38:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983163+0000", 2024-09-16T03:38:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097333127000000005 2024-09-16T03:38:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983273+0000", 2024-09-16T03:38:40.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109868 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983458+0000", 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018554899999999999 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084722+0000", 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101263244 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185717+0000", 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100995575 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185904+0000", 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186682 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380965+0000", 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.195060921 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381030+0000", 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5174999999999999e-05 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381100+0000", 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9877999999999997e-05 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2645 4.c 4:34492f38:::smithi15736696-33:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:40.495 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.493907154999999, 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59563304800000005, 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2645 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:38:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:38:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885840+0000", 2024-09-16T03:38:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-09-16T03:38:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885843+0000", 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.785e-06 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885848+0000", 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3979999999999998e-06 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885852+0000", 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6509999999999999e-06 2024-09-16T03:38:40.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084768+0000", 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19891652000000001 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084831+0000", 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2717000000000003e-05 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084964+0000", 2024-09-16T03:38:40.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132731 2024-09-16T03:38:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185772+0000", 2024-09-16T03:38:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10080797399999999 2024-09-16T03:38:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185790+0000", 2024-09-16T03:38:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7966999999999999e-05 2024-09-16T03:38:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185959+0000", 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168777 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481235+0000", 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29527644800000002 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481307+0000", 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1539999999999996e-05 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481473+0000", 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016595600000000001 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2646 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.493890466, 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59596245699999995, 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:40.502 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2646 2024-09-16T03:38:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:38:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885857+0000", 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885860+0000", 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8940000000000003e-06 2024-09-16T03:38:40.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885864+0000", 2024-09-16T03:38:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.602e-06 2024-09-16T03:38:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885867+0000", 2024-09-16T03:38:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0089999999999999e-06 2024-09-16T03:38:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085196+0000", 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19932923399999999 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085287+0000", 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0480000000000001e-05 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481521+0000", 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39623439799999999 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481613+0000", 2024-09-16T03:38:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2276999999999998e-05 2024-09-16T03:38:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481819+0000", 2024-09-16T03:38:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020521499999999999 2024-09-16T03:38:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:40.508 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:40.508 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:40.509 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:40.400+0000 7f66f5209640 -1 osd.15 106 handle_osd_map injecting map crc failure 2024-09-16T03:38:40.510 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:38:40.512 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:38:40.512 INFO:tasks.rados.rados.0.smithi157.stdout:1428: finishing write tid 3 to smithi15736696-17 2024-09-16T03:38:40.512 INFO:tasks.rados.rados.0.smithi157.stdout:1428: finishing write tid 4 to smithi15736696-17 2024-09-16T03:38:40.512 INFO:tasks.rados.rados.0.smithi157.stdout:1428: finishing write tid 5 to smithi15736696-17 2024-09-16T03:38:40.512 INFO:tasks.rados.rados.0.smithi157.stdout:1428: oid 17 updating version 0 to 117 2024-09-16T03:38:40.512 INFO:tasks.rados.rados.0.smithi157.stdout:1428: oid 17 version 117 is already newer than 115 2024-09-16T03:38:40.513 INFO:tasks.rados.rados.0.smithi157.stdout:1428: oid 17 version 117 is already newer than 116 2024-09-16T03:38:40.513 INFO:tasks.rados.rados.0.smithi157.stdout:1428: oid 17 updating version 117 to 118 2024-09-16T03:38:40.513 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 118 (ObjNum 289 snap 0 seq_num 289) dirty exists 2024-09-16T03:38:40.513 INFO:tasks.rados.rados.0.smithi157.stdout:1428: left oid 17 (ObjNum 289 snap 0 seq_num 289) 2024-09-16T03:38:40.513 INFO:tasks.rados.rados.0.smithi157.stdout:1432: finishing write tid 1 to smithi15736696-31 2024-09-16T03:38:40.513 INFO:tasks.rados.rados.0.smithi157.stdout:1414: done (13 left) 2024-09-16T03:38:40.513 INFO:tasks.rados.rados.0.smithi157.stdout:1428: done (12 left) 2024-09-16T03:38:40.513 INFO:tasks.rados.rados.0.smithi157.stdout:1429: done (11 left) 2024-09-16T03:38:40.513 INFO:tasks.rados.rados.0.smithi157.stdout:1430: done (10 left) 2024-09-16T03:38:40.513 INFO:tasks.rados.rados.0.smithi157.stdout:1441: done (9 left) 2024-09-16T03:38:40.513 INFO:tasks.rados.rados.0.smithi157.stdout:1442: read oid 11 snap -1 2024-09-16T03:38:40.513 INFO:tasks.rados.rados.0.smithi157.stdout:1442: expect (ObjNum 261 snap 0 seq_num 261) 2024-09-16T03:38:40.513 INFO:tasks.rados.rados.0.smithi157.stdout:1443: read oid 34 snap -1 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:1443: expect (ObjNum 205 snap 0 seq_num 205) 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:1444: read oid 43 snap -1 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:1444: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:1445: tier_flush oid 24 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:1445: tier_flush oid 24 snap -1 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:1432: finishing write tid 2 to smithi15736696-31 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:1432: finishing write tid 3 to smithi15736696-31 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:1432: finishing write tid 4 to smithi15736696-31 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:1432: finishing write tid 5 to smithi15736696-31 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:1432: oid 31 updating version 0 to 67 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:1432: oid 31 version 67 is already newer than 66 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:1432: oid 31 version 67 is already newer than 64 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:1432: oid 31 version 67 is already newer than 65 2024-09-16T03:38:40.514 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 67 (ObjNum 291 snap 0 seq_num 291) dirty exists 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:1432: left oid 31 (ObjNum 291 snap 0 seq_num 291) 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:1434: finishing set_chunk to oid 10 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 10 set_chunk returned error code -95 offset: 22528 length: 437760 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 0 (ObjNum 276 snap 0 seq_num 276) dirty exists 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:1436: finishing write tid 1 to smithi15736696-47 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:1432: done (12 left) 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:1433: done (11 left) 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:1434: done (10 left) 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:1446: write (excl) oid 8 current snap is 0 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:1439: got (2) No such file or directory 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:1439: got expected ENOENT (src dne) 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:1437: expect (ObjNum 245 snap 0 seq_num 245) 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:40.515 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1028794 2024-09-16T03:38:40.516 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:40.507+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:40.535 INFO:tasks.rados.rados.0.smithi157.stdout:1435: finishing set_chunk to oid 28 2024-09-16T03:38:40.535 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 107 (ObjNum 279 snap 0 seq_num 279) dirty exists 2024-09-16T03:38:40.535 INFO:tasks.rados.rados.0.smithi157.stdout:1436: finishing write tid 2 to smithi15736696-47 2024-09-16T03:38:40.536 INFO:tasks.rados.rados.0.smithi157.stdout:1436: finishing write tid 3 to smithi15736696-47 2024-09-16T03:38:40.536 INFO:tasks.rados.rados.0.smithi157.stdout:1436: finishing write tid 4 to smithi15736696-47 2024-09-16T03:38:40.536 INFO:tasks.rados.rados.0.smithi157.stdout:1436: oid 47 updating version 0 to 107 2024-09-16T03:38:40.536 INFO:tasks.rados.rados.0.smithi157.stdout:1436: oid 47 version 107 is already newer than 106 2024-09-16T03:38:40.536 INFO:tasks.rados.rados.0.smithi157.stdout:1436: oid 47 version 107 is already newer than 105 2024-09-16T03:38:40.536 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 107 (ObjNum 292 snap 0 seq_num 292) dirty exists 2024-09-16T03:38:40.536 INFO:tasks.rados.rados.0.smithi157.stdout:1436: left oid 47 (ObjNum 292 snap 0 seq_num 292) 2024-09-16T03:38:40.536 INFO:tasks.rados.rados.0.smithi157.stdout:1446: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-09-16T03:38:40.541 INFO:tasks.rados.rados.0.smithi157.stdout:1446: writing smithi15736696-8 from 541258 to 1085878 tid 1 2024-09-16T03:38:40.548 INFO:tasks.rados.rados.0.smithi157.stdout:1446: writing smithi15736696-8 from 1637706 to 2264460 tid 2 2024-09-16T03:38:40.552 INFO:tasks.rados.rados.0.smithi157.stdout:1446: writing smithi15736696-8 from 2840948 to 2941258 tid 3 2024-09-16T03:38:40.552 INFO:tasks.rados.rados.0.smithi157.stdout:1435: done (10 left) 2024-09-16T03:38:40.552 INFO:tasks.rados.rados.0.smithi157.stdout:1436: done (9 left) 2024-09-16T03:38:40.552 INFO:tasks.rados.rados.0.smithi157.stdout:1437: done (8 left) 2024-09-16T03:38:40.552 INFO:tasks.rados.rados.0.smithi157.stdout:1439: done (7 left) 2024-09-16T03:38:40.552 INFO:tasks.rados.rados.0.smithi157.stdout:1440: done (6 left) 2024-09-16T03:38:40.552 INFO:tasks.rados.rados.0.smithi157.stdout:1447: write (excl) oid 25 current snap is 0 2024-09-16T03:38:40.552 INFO:tasks.rados.rados.0.smithi157.stdout:1438: expect (ObjNum 265 snap 0 seq_num 265) 2024-09-16T03:38:40.552 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:40.605 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1095490 2024-09-16T03:38:40.610 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:40.608+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:40.630 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:40.630 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:40.630 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:40.630 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:40.639 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:38:40.641 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:40.689 INFO:tasks.rados.rados.0.smithi157.stdout:1442: expect (ObjNum 261 snap 0 seq_num 261) 2024-09-16T03:38:40.711 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:40.709+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:40.742 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:40.743 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:40.743 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:40.743 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:40.743 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:40.751 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:38:40.765 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 796275 2024-09-16T03:38:40.789 INFO:tasks.rados.rados.0.smithi157.stdout:1444: expect (ObjNum 264 snap 0 seq_num 264) 2024-09-16T03:38:40.812 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:40.810+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:40.856 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:40.857 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:40.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:40.857 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:40.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.857 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:21 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'4, mlcod=0'0)", 2024-09-16T03:38:40.857 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:38:40.857 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.666964903999997, 2024-09-16T03:38:40.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10231100799999999, 2024-09-16T03:38:40.857 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.858 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.858 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.858 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:38:40.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:38:40.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187300+0000", 2024-09-16T03:38:40.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0360000000000001e-06 2024-09-16T03:38:40.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187905+0000", 2024-09-16T03:38:40.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060508100000000004 2024-09-16T03:38:40.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187920+0000", 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4314000000000001e-05 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187928+0000", 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7150000000000004e-06 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187984+0000", 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5872999999999999e-05 2024-09-16T03:38:40.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188016+0000", 2024-09-16T03:38:40.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2045000000000002e-05 2024-09-16T03:38:40.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289567+0000", 2024-09-16T03:38:40.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101550875 2024-09-16T03:38:40.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289609+0000", 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2069e-05 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.21:5 3.0 e45/44 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'39, mlcod=45'37)", 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 79.939667869999994, 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20119541799999999, 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:38:40.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:38:40.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914597+0000", 2024-09-16T03:38:40.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.494e-06 2024-09-16T03:38:40.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914603+0000", 2024-09-16T03:38:40.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7349999999999999e-06 2024-09-16T03:38:40.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914614+0000", 2024-09-16T03:38:40.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0995999999999999e-05 2024-09-16T03:38:40.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914623+0000", 2024-09-16T03:38:40.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3899999999999993e-06 2024-09-16T03:38:40.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.114941+0000", 2024-09-16T03:38:40.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20031812399999999 2024-09-16T03:38:40.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115001+0000", 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0161999999999998e-05 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115750+0000", 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074851699999999998 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115791+0000", 2024-09-16T03:38:40.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1e-05 2024-09-16T03:38:40.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.867 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.867 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.5.21:36 3.0 e52/44 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 52'62, mlcod=51'60)", 2024-09-16T03:38:40.867 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:38:40.867 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 68.969637211000006, 2024-09-16T03:38:40.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20104840199999999, 2024-09-16T03:38:40.867 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.867 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.867 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.867 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:38:40.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:38:40.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884627+0000", 2024-09-16T03:38:40.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.322e-06 2024-09-16T03:38:40.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884634+0000", 2024-09-16T03:38:40.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:38:40.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884643+0000", 2024-09-16T03:38:40.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7680000000000003e-06 2024-09-16T03:38:40.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984740+0000", 2024-09-16T03:38:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100096066 2024-09-16T03:38:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084931+0000", 2024-09-16T03:38:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100191408 2024-09-16T03:38:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084952+0000", 2024-09-16T03:38:40.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.128e-05 2024-09-16T03:38:40.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085645+0000", 2024-09-16T03:38:40.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00069309500000000002 2024-09-16T03:38:40.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085674+0000", 2024-09-16T03:38:40.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9025000000000001e-05 2024-09-16T03:38:40.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.871 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.872 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1537 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'35, mlcod=58'33)", 2024-09-16T03:38:40.872 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:38:40.872 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.133883916999999, 2024-09-16T03:38:40.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102969106, 2024-09-16T03:38:40.872 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.872 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.872 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.872 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:38:40.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:38:40.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720381+0000", 2024-09-16T03:38:40.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.767e-06 2024-09-16T03:38:40.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721134+0000", 2024-09-16T03:38:40.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00075281999999999996 2024-09-16T03:38:40.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721148+0000", 2024-09-16T03:38:40.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4534e-05 2024-09-16T03:38:40.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721160+0000", 2024-09-16T03:38:40.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1595e-05 2024-09-16T03:38:40.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721256+0000", 2024-09-16T03:38:40.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6136000000000002e-05 2024-09-16T03:38:40.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721274+0000", 2024-09-16T03:38:40.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8006e-05 2024-09-16T03:38:40.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823312+0000", 2024-09-16T03:38:40.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102038064 2024-09-16T03:38:40.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823348+0000", 2024-09-16T03:38:40.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6183999999999998e-05 2024-09-16T03:38:40.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.877 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.877 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1538 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'36, mlcod=58'33)", 2024-09-16T03:38:40.878 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:40.878 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.132729580000003, 2024-09-16T03:38:40.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102397932, 2024-09-16T03:38:40.878 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.878 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.878 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.878 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:40.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5199999999999996e-07 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722162+0000", 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062758599999999996 2024-09-16T03:38:40.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722171+0000", 2024-09-16T03:38:40.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7139999999999999e-06 2024-09-16T03:38:40.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722177+0000", 2024-09-16T03:38:40.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.866e-06 2024-09-16T03:38:40.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822283+0000", 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10010508899999999 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822298+0000", 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5497000000000001e-05 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823907+0000", 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001609166 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823931+0000", 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4261999999999999e-05 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1539 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'37, mlcod=58'33)", 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.131529518999997, 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103545683, 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.882 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722735+0000", 2024-09-16T03:38:40.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8400000000000002e-07 2024-09-16T03:38:40.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722740+0000", 2024-09-16T03:38:40.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3129999999999998e-06 2024-09-16T03:38:40.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722747+0000", 2024-09-16T03:38:40.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0600000000000002e-06 2024-09-16T03:38:40.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822841+0000", 2024-09-16T03:38:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100094477 2024-09-16T03:38:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823376+0000", 2024-09-16T03:38:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00053497199999999999 2024-09-16T03:38:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823399+0000", 2024-09-16T03:38:40.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2680999999999998e-05 2024-09-16T03:38:40.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.826259+0000", 2024-09-16T03:38:40.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028598500000000002 2024-09-16T03:38:40.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.826279+0000", 2024-09-16T03:38:40.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0346000000000001e-05 2024-09-16T03:38:40.886 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.886 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.886 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.887 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:38:40.887 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:38:40.887 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.704338614000001, 2024-09-16T03:38:40.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20169899299999999, 2024-09-16T03:38:40.887 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.887 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.887 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.887 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:38:40.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:38:40.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149927+0000", 2024-09-16T03:38:40.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9299999999999999e-06 2024-09-16T03:38:40.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151090+0000", 2024-09-16T03:38:40.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011624459999999999 2024-09-16T03:38:40.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151105+0000", 2024-09-16T03:38:40.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5238999999999999e-05 2024-09-16T03:38:40.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251193+0000", 2024-09-16T03:38:40.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100088025 2024-09-16T03:38:40.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251556+0000", 2024-09-16T03:38:40.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036245599999999999 2024-09-16T03:38:40.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251578+0000", 2024-09-16T03:38:40.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2651e-05 2024-09-16T03:38:40.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351582+0000", 2024-09-16T03:38:40.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100003811 2024-09-16T03:38:40.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351624+0000", 2024-09-16T03:38:40.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1434999999999999e-05 2024-09-16T03:38:40.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.891 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.892 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:90 3.0 e70/69 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head v 70'103, mlcod=0'0)", 2024-09-16T03:38:40.892 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:40.892 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 45.078500331000001, 2024-09-16T03:38:40.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102185333, 2024-09-16T03:38:40.892 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.892 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.892 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.892 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:40.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775769+0000", 2024-09-16T03:38:40.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7379999999999999e-06 2024-09-16T03:38:40.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775782+0000", 2024-09-16T03:38:40.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2989e-05 2024-09-16T03:38:40.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775791+0000", 2024-09-16T03:38:40.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5369999999999997e-06 2024-09-16T03:38:40.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876295+0000", 2024-09-16T03:38:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100504817 2024-09-16T03:38:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876316+0000", 2024-09-16T03:38:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0409999999999999e-05 2024-09-16T03:38:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877916+0000", 2024-09-16T03:38:40.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016003790000000001 2024-09-16T03:38:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877948+0000", 2024-09-16T03:38:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1949999999999997e-05 2024-09-16T03:38:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1742 3.0 e70/69 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head v 70'104, mlcod=0'0)", 2024-09-16T03:38:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:40.896 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 45.078463634999999, 2024-09-16T03:38:40.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20271830699999999, 2024-09-16T03:38:40.897 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.897 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.897 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.897 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:40.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:38:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777455+0000", 2024-09-16T03:38:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001655524 2024-09-16T03:38:40.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777468+0000", 2024-09-16T03:38:40.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2486999999999999e-05 2024-09-16T03:38:40.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777474+0000", 2024-09-16T03:38:40.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:38:40.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876577+0000", 2024-09-16T03:38:40.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099103360000000001 2024-09-16T03:38:40.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876594+0000", 2024-09-16T03:38:40.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6449000000000001e-05 2024-09-16T03:38:40.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978469+0000", 2024-09-16T03:38:40.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101875521 2024-09-16T03:38:40.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978518+0000", 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8488000000000001e-05 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:91 3.0 e70/69 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head v 70'105, mlcod=0'0)", 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 44.992251766000003, 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.116532025, 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.901 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.902 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.902 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:38:40.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:38:40.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862015+0000", 2024-09-16T03:38:40.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3569999999999998e-06 2024-09-16T03:38:40.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862029+0000", 2024-09-16T03:38:40.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4082e-05 2024-09-16T03:38:40.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862055+0000", 2024-09-16T03:38:40.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6570000000000001e-05 2024-09-16T03:38:40.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862070+0000", 2024-09-16T03:38:40.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4473e-05 2024-09-16T03:38:40.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877631+0000", 2024-09-16T03:38:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.015560744 2024-09-16T03:38:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877674+0000", 2024-09-16T03:38:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3615999999999997e-05 2024-09-16T03:38:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978525+0000", 2024-09-16T03:38:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100850696 2024-09-16T03:38:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978543+0000", 2024-09-16T03:38:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8487000000000001e-05 2024-09-16T03:38:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:94 3.0 e70/69 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head v 70'106, mlcod=0'0)", 2024-09-16T03:38:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:38:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 44.991715546999998, 2024-09-16T03:38:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11627603, 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862549+0000", 2024-09-16T03:38:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1650000000000001e-06 2024-09-16T03:38:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862556+0000", 2024-09-16T03:38:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8709999999999999e-06 2024-09-16T03:38:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862566+0000", 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0271e-05 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862573+0000", 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.861e-06 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978123+0000", 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11554993199999999 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978163+0000", 2024-09-16T03:38:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0201999999999999e-05 2024-09-16T03:38:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978796+0000", 2024-09-16T03:38:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063316700000000002 2024-09-16T03:38:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978824+0000", 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7560999999999999e-05 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1947 4.8 e74/68 4:146f84f0:::smithi15736696-6:head v 74'47, mlcod=73'45)", 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 39.729099409, 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103866244, 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:38:40.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:38:40.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125165+0000", 2024-09-16T03:38:40.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.513e-06 2024-09-16T03:38:40.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126855+0000", 2024-09-16T03:38:40.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016901010000000001 2024-09-16T03:38:40.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126870+0000", 2024-09-16T03:38:40.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4487999999999999e-05 2024-09-16T03:38:40.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227009+0000", 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100138888 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227117+0000", 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107984 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227136+0000", 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9247999999999999e-05 2024-09-16T03:38:40.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229002+0000", 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001866143 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229030+0000", 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7878999999999999e-05 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.916 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1979 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'32, mlcod=0'0)", 2024-09-16T03:38:40.916 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:38:40.916 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 39.016524861999997, 2024-09-16T03:38:40.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20429768500000001, 2024-09-16T03:38:40.916 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.916 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.916 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.916 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:38:40.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:38:40.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837740+0000", 2024-09-16T03:38:40.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.779e-06 2024-09-16T03:38:40.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838356+0000", 2024-09-16T03:38:40.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061634500000000002 2024-09-16T03:38:40.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838371+0000", 2024-09-16T03:38:40.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5007e-05 2024-09-16T03:38:40.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838380+0000", 2024-09-16T03:38:40.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1009999999999992e-06 2024-09-16T03:38:40.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038674+0000", 2024-09-16T03:38:40.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20029450500000001 2024-09-16T03:38:40.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038710+0000", 2024-09-16T03:38:40.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6100999999999999e-05 2024-09-16T03:38:40.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042001+0000", 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032909430000000002 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042036+0000", 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4904000000000003e-05 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.920 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1980 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'33, mlcod=0'0)", 2024-09-16T03:38:40.921 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:38:40.921 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 39.014648266000002, 2024-09-16T03:38:40.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20296795100000001, 2024-09-16T03:38:40.921 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.921 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.921 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.921 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:38:40.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:38:40.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839616+0000", 2024-09-16T03:38:40.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3099999999999996e-07 2024-09-16T03:38:40.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840706+0000", 2024-09-16T03:38:40.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010900980000000001 2024-09-16T03:38:40.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840720+0000", 2024-09-16T03:38:40.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.378e-05 2024-09-16T03:38:40.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840727+0000", 2024-09-16T03:38:40.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.525e-06 2024-09-16T03:38:40.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039940+0000", 2024-09-16T03:38:40.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19921313199999999 2024-09-16T03:38:40.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039967+0000", 2024-09-16T03:38:40.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6393999999999998e-05 2024-09-16T03:38:40.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042552+0000", 2024-09-16T03:38:40.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025850999999999999 2024-09-16T03:38:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042583+0000", 2024-09-16T03:38:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0991000000000002e-05 2024-09-16T03:38:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1981 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'34, mlcod=0'0)", 2024-09-16T03:38:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:38:40.925 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 39.012625911000001, 2024-09-16T03:38:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201270175, 2024-09-16T03:38:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:38:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:38:40.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841638+0000", 2024-09-16T03:38:40.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-16T03:38:40.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841644+0000", 2024-09-16T03:38:40.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8050000000000003e-06 2024-09-16T03:38:40.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841653+0000", 2024-09-16T03:38:40.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6580000000000008e-06 2024-09-16T03:38:40.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841660+0000", 2024-09-16T03:38:40.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9290000000000001e-06 2024-09-16T03:38:40.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041673+0000", 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20001311499999999 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041696+0000", 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3365000000000001e-05 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042878+0000", 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011814499999999999 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042907+0000", 2024-09-16T03:38:40.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9655999999999999e-05 2024-09-16T03:38:40.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.930 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.930 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2117 3.0 e76/69 3:3c8eb54b:::593314a3a723c40dec2b25351adb282089b02c0264cb0891ee3f17edde05dfe9:head v 76'138, mlcod=75'136)", 2024-09-16T03:38:40.930 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:38:40.930 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 35.202218223000003, 2024-09-16T03:38:40.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10374235, 2024-09-16T03:38:40.930 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.930 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.931 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.931 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:38:40.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:38:40.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:38:40.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8099999999999999e-07 2024-09-16T03:38:40.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654214+0000", 2024-09-16T03:38:40.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002169191 2024-09-16T03:38:40.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654230+0000", 2024-09-16T03:38:40.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6144e-05 2024-09-16T03:38:40.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654242+0000", 2024-09-16T03:38:40.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1748e-05 2024-09-16T03:38:40.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.754387+0000", 2024-09-16T03:38:40.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10014505899999999 2024-09-16T03:38:40.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.754409+0000", 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2181999999999999e-05 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.755736+0000", 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013261690000000001 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.755787+0000", 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1675999999999998e-05 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.934 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2278 4.18 e80/68 4:1f63731d:::smithi15736696-30:head v 80'37, mlcod=75'34)", 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.328808024000001, 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10326089300000001, 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:38:40.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:38:40.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525456+0000", 2024-09-16T03:38:40.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.248e-06 2024-09-16T03:38:40.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525936+0000", 2024-09-16T03:38:40.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047982899999999998 2024-09-16T03:38:40.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525945+0000", 2024-09-16T03:38:40.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8489999999999995e-06 2024-09-16T03:38:40.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525955+0000", 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.043e-05 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.526339+0000", 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038385200000000001 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.526358+0000", 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8802000000000001e-05 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.628679+0000", 2024-09-16T03:38:40.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102321304 2024-09-16T03:38:40.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.628716+0000", 2024-09-16T03:38:40.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6579000000000003e-05 2024-09-16T03:38:40.939 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.939 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.939 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.939 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2536 3.0 e86/69 3:10a17b1b:::13dc2fab1f287170fabf96291b912bf80e0b081f291f2c6d41d494566f2d1f8c:head v 86'176, mlcod=86'174)", 2024-09-16T03:38:40.940 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:38:40.940 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 24.014751532999998, 2024-09-16T03:38:40.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20664157499999999, 2024-09-16T03:38:40.940 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.940 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.940 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.940 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:38:40.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:38:40.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839515+0000", 2024-09-16T03:38:40.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3330000000000001e-06 2024-09-16T03:38:40.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843392+0000", 2024-09-16T03:38:40.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038769329999999999 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843412+0000", 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0608e-05 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.943546+0000", 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100133176 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043824+0000", 2024-09-16T03:38:40.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100278785 2024-09-16T03:38:40.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043869+0000", 2024-09-16T03:38:40.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4845999999999997e-05 2024-09-16T03:38:40.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046096+0000", 2024-09-16T03:38:40.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022262559999999998 2024-09-16T03:38:40.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046153+0000", 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7637999999999999e-05 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2654 4.8 e90/68 4:148e627d:::smithi15736696-23:head v 90'58, mlcod=80'56)", 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.479364175000001, 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.106604212, 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.944 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.945 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:40.945 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:38:40.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:38:40.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374901+0000", 2024-09-16T03:38:40.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1749999999999999e-06 2024-09-16T03:38:40.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375810+0000", 2024-09-16T03:38:40.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00090911400000000004 2024-09-16T03:38:40.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375827+0000", 2024-09-16T03:38:40.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7010000000000001e-05 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476010+0000", 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100182879 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476184+0000", 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017408899999999999 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:40.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476206+0000", 2024-09-16T03:38:40.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2013000000000001e-05 2024-09-16T03:38:40.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:40.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481458+0000", 2024-09-16T03:38:40.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052519120000000001 2024-09-16T03:38:40.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481503+0000", 2024-09-16T03:38:40.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5019999999999999e-05 2024-09-16T03:38:40.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.948 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "pg_update_log_missing(4.c epoch 99/68 rep_tid 69 entries 99'56 (0'0) error 4:34492f38:::smithi15736696-33:head by client.4628.0:3000 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 90'53 roll_forward_to 90'54)", 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 8.9344060239999994, 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201155684, 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:40.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919858+0000", 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919871+0000", 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3433e-05 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919929+0000", 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7543000000000001e-05 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919942+0000", 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3122000000000001e-05 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:40.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.120302+0000", 2024-09-16T03:38:40.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036052800000001 2024-09-16T03:38:40.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:40.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:40.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:40.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.121013+0000", 2024-09-16T03:38:40.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071046800000000002 2024-09-16T03:38:40.952 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.952 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.952 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.952 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:40.952 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:40.953 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:40.954 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:40.954 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:38:40.955 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:40.952+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:41.040 INFO:tasks.rados.rados.0.smithi157.stdout:1443: expect (ObjNum 205 snap 0 seq_num 205) 2024-09-16T03:38:41.054 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:41.053+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:41.056 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:41.056 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:41.057 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:41.064 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2792488 2024-09-16T03:38:41.065 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:38:41.095 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1975014 2024-09-16T03:38:41.135 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2144289 2024-09-16T03:38:41.154 INFO:tasks.rados.rados.0.smithi157.stdout:1446: finishing write tid 1 to smithi15736696-8 2024-09-16T03:38:41.154 INFO:tasks.rados.rados.0.smithi157.stdout:1446: finishing write tid 2 to smithi15736696-8 2024-09-16T03:38:41.155 INFO:tasks.rados.rados.0.smithi157.stdout:1446: finishing write tid 3 to smithi15736696-8 2024-09-16T03:38:41.155 INFO:tasks.rados.rados.0.smithi157.stdout:1446: finishing write tid 4 to smithi15736696-8 2024-09-16T03:38:41.155 INFO:tasks.rados.rados.0.smithi157.stdout:1446: finishing write tid 5 to smithi15736696-8 2024-09-16T03:38:41.155 INFO:tasks.rados.rados.0.smithi157.stdout:1446: oid 8 updating version 0 to 92 2024-09-16T03:38:41.155 INFO:tasks.rados.rados.0.smithi157.stdout:1446: oid 8 updating version 92 to 93 2024-09-16T03:38:41.155 INFO:tasks.rados.rados.0.smithi157.stdout:1446: oid 8 version 93 is already newer than 91 2024-09-16T03:38:41.155 INFO:tasks.rados.rados.0.smithi157.stdout:1446: oid 8 version 93 is already newer than 90 2024-09-16T03:38:41.155 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 93 (ObjNum 294 snap 0 seq_num 294) dirty exists 2024-09-16T03:38:41.155 INFO:tasks.rados.rados.0.smithi157.stdout:1446: left oid 8 (ObjNum 294 snap 0 seq_num 294) 2024-09-16T03:38:41.155 INFO:tasks.rados.rados.0.smithi157.stdout:1447: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-09-16T03:38:41.155 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:41.153+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:41.160 INFO:tasks.rados.rados.0.smithi157.stdout:1447: writing smithi15736696-25 from 558065 to 1177934 tid 1 2024-09-16T03:38:41.165 INFO:tasks.rados.rados.0.smithi157.stdout:1447: writing smithi15736696-25 from 1579835 to 2066600 tid 2 2024-09-16T03:38:41.167 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:41.168 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:41.168 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:41.168 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:41.168 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:41.171 INFO:tasks.rados.rados.0.smithi157.stdout:1447: writing smithi15736696-25 from 2468371 to 2958065 tid 3 2024-09-16T03:38:41.171 INFO:tasks.rados.rados.0.smithi157.stdout:1438: done (6 left) 2024-09-16T03:38:41.171 INFO:tasks.rados.rados.0.smithi157.stdout:1442: done (5 left) 2024-09-16T03:38:41.171 INFO:tasks.rados.rados.0.smithi157.stdout:1443: done (4 left) 2024-09-16T03:38:41.172 INFO:tasks.rados.rados.0.smithi157.stdout:1444: done (3 left) 2024-09-16T03:38:41.172 INFO:tasks.rados.rados.0.smithi157.stdout:1446: done (2 left) 2024-09-16T03:38:41.172 INFO:tasks.rados.rados.0.smithi157.stdout:1448: write oid 4 current snap is 0 2024-09-16T03:38:41.172 INFO:tasks.rados.rados.0.smithi157.stdout:1448: seq_num 296 ranges {574872=695118,2038317=630423} 2024-09-16T03:38:41.176 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:38:41.178 INFO:tasks.rados.rados.0.smithi157.stdout:1448: writing smithi15736696-4 from 574872 to 1269990 tid 1 2024-09-16T03:38:41.186 INFO:tasks.rados.rados.0.smithi157.stdout:1448: writing smithi15736696-4 from 2038317 to 2668740 tid 2 2024-09-16T03:38:41.186 INFO:tasks.rados.rados.0.smithi157.stdout:1449: write oid 12 current snap is 0 2024-09-16T03:38:41.186 INFO:tasks.rados.rados.0.smithi157.stdout:1449: seq_num 297 ranges {591679=486720,1813152=490434} 2024-09-16T03:38:41.191 INFO:tasks.rados.rados.0.smithi157.stdout:1449: writing smithi15736696-12 from 591679 to 1078399 tid 1 2024-09-16T03:38:41.199 INFO:tasks.rados.rados.0.smithi157.stdout:1449: writing smithi15736696-12 from 1813152 to 2303586 tid 2 2024-09-16T03:38:41.199 INFO:tasks.rados.rados.0.smithi157.stdout:1450: set_chunk oid 22 target oid 2024-09-16T03:38:41.199 INFO:tasks.rados.rados.0.smithi157.stdout:1450: set_chunk oid 22 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:41.199 INFO:tasks.rados.rados.0.smithi157.stdout:1451: read oid 28 snap -1 2024-09-16T03:38:41.199 INFO:tasks.rados.rados.0.smithi157.stdout:1451: expect (ObjNum 279 snap 0 seq_num 279) 2024-09-16T03:38:41.199 INFO:tasks.rados.rados.0.smithi157.stdout:1449: finishing write tid 1 to smithi15736696-12 2024-09-16T03:38:41.199 INFO:tasks.rados.rados.0.smithi157.stdout:1452: read oid 27 snap -1 2024-09-16T03:38:41.199 INFO:tasks.rados.rados.0.smithi157.stdout:1452: expect (ObjNum 244 snap 0 seq_num 244) 2024-09-16T03:38:41.199 INFO:tasks.rados.rados.0.smithi157.stdout:1453: tier_promote oid 36 2024-09-16T03:38:41.199 INFO:tasks.rados.rados.0.smithi157.stdout:1454: write (excl) oid 35 current snap is 0 2024-09-16T03:38:41.199 INFO:tasks.rados.rados.0.smithi157.stdout:1454: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-09-16T03:38:41.205 INFO:tasks.rados.rados.0.smithi157.stdout:1454: writing smithi15736696-35 from 608486 to 1170455 tid 1 2024-09-16T03:38:41.212 INFO:tasks.rados.rados.0.smithi157.stdout:1454: writing smithi15736696-35 from 1871634 to 2505726 tid 2 2024-09-16T03:38:41.215 INFO:tasks.rados.rados.0.smithi157.stdout:1454: writing smithi15736696-35 from 2950640 to 3008486 tid 3 2024-09-16T03:38:41.215 INFO:tasks.rados.rados.0.smithi157.stdout:1455: set_chunk oid 5 target oid 2024-09-16T03:38:41.220 INFO:tasks.rados.rados.0.smithi157.stdout:1455: set_chunk oid 5 offset: 1915392 length: 580608 target oid 33b1bbaf413b7db734c5944096710db971df0affeadd1a0dea75c013cf26ec22 offset: 0 snap -1 2024-09-16T03:38:41.220 INFO:tasks.rados.rados.0.smithi157.stdout:1456: read oid 29 snap -1 2024-09-16T03:38:41.220 INFO:tasks.rados.rados.0.smithi157.stdout:1456: expect (ObjNum 246 snap 0 seq_num 246) 2024-09-16T03:38:41.220 INFO:tasks.rados.rados.0.smithi157.stdout:1457: set_chunk oid 39 target oid 2024-09-16T03:38:41.220 INFO:tasks.rados.rados.0.smithi157.stdout:1457: set_chunk oid 39 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:41.220 INFO:tasks.rados.rados.0.smithi157.stdout:1458: set_chunk oid 20 target oid 2024-09-16T03:38:41.256 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:41.254+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:41.281 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:41.281 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:41.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:41.281 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:41.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.281 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:25 4.6 4:6ece7830:::smithi15736696-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:41.281 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:38:41.281 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 89.158230802000006, 2024-09-16T03:38:41.281 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40559452699999998, 2024-09-16T03:38:41.281 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.282 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.282 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.282 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.284 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.284 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.284 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 25 2024-09-16T03:38:41.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.284 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:38:41.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:38:41.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.285 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119830+0000", 2024-09-16T03:38:41.285 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9200000000000001e-07 2024-09-16T03:38:41.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120013+0000", 2024-09-16T03:38:41.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018367200000000001 2024-09-16T03:38:41.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120021+0000", 2024-09-16T03:38:41.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9179999999999994e-06 2024-09-16T03:38:41.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120027+0000", 2024-09-16T03:38:41.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6710000000000004e-06 2024-09-16T03:38:41.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220194+0000", 2024-09-16T03:38:41.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10016689500000001 2024-09-16T03:38:41.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220399+0000", 2024-09-16T03:38:41.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020502899999999999 2024-09-16T03:38:41.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:41.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220613+0000", 2024-09-16T03:38:41.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021383900000000001 2024-09-16T03:38:41.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224640+0000", 2024-09-16T03:38:41.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040268099999999996 2024-09-16T03:38:41.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423348+0000", 2024-09-16T03:38:41.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19870853399999999 2024-09-16T03:38:41.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525343+0000", 2024-09-16T03:38:41.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101995081 2024-09-16T03:38:41.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525381+0000", 2024-09-16T03:38:41.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7842000000000001e-05 2024-09-16T03:38:41.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525424+0000", 2024-09-16T03:38:41.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2644e-05 2024-09-16T03:38:41.291 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.291 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.291 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.291 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:26 4.6 4:6ece7830:::smithi15736696-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:41.291 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:38:41.291 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 89.158018575, 2024-09-16T03:38:41.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550111300000002, 2024-09-16T03:38:41.291 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.291 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.291 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.292 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.292 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.292 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.292 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 26 2024-09-16T03:38:41.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.292 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:38:41.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:38:41.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.293 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.293 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120042+0000", 2024-09-16T03:38:41.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7999999999999995e-07 2024-09-16T03:38:41.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120210+0000", 2024-09-16T03:38:41.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016779499999999999 2024-09-16T03:38:41.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120219+0000", 2024-09-16T03:38:41.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6859999999999995e-06 2024-09-16T03:38:41.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120223+0000", 2024-09-16T03:38:41.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0330000000000002e-06 2024-09-16T03:38:41.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221494+0000", 2024-09-16T03:38:41.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101271812 2024-09-16T03:38:41.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221572+0000", 2024-09-16T03:38:41.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7733999999999998e-05 2024-09-16T03:38:41.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:41.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221697+0000", 2024-09-16T03:38:41.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012474900000000001 2024-09-16T03:38:41.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224670+0000", 2024-09-16T03:38:41.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029729589999999998 2024-09-16T03:38:41.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423398+0000", 2024-09-16T03:38:41.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19872784900000001 2024-09-16T03:38:41.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525465+0000", 2024-09-16T03:38:41.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102067557 2024-09-16T03:38:41.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525494+0000", 2024-09-16T03:38:41.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9113999999999999e-05 2024-09-16T03:38:41.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525542+0000", 2024-09-16T03:38:41.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8145000000000001e-05 2024-09-16T03:38:41.299 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.299 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.299 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.300 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:27 4.6 4:6ece7830:::smithi15736696-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:41.300 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:38:41.300 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 89.157830989999994, 2024-09-16T03:38:41.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40541507399999999, 2024-09-16T03:38:41.300 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.300 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.300 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.300 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.300 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.301 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.301 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27 2024-09-16T03:38:41.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.301 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:38:41.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.302 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:38:41.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.302 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120230+0000", 2024-09-16T03:38:41.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5799999999999999e-07 2024-09-16T03:38:41.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120233+0000", 2024-09-16T03:38:41.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.055e-06 2024-09-16T03:38:41.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120237+0000", 2024-09-16T03:38:41.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7099999999999998e-06 2024-09-16T03:38:41.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120244+0000", 2024-09-16T03:38:41.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4810000000000003e-06 2024-09-16T03:38:41.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222660+0000", 2024-09-16T03:38:41.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102416089 2024-09-16T03:38:41.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222705+0000", 2024-09-16T03:38:41.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5368999999999999e-05 2024-09-16T03:38:41.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:41.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222782+0000", 2024-09-16T03:38:41.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6623000000000001e-05 2024-09-16T03:38:41.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224696+0000", 2024-09-16T03:38:41.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019143929999999999 2024-09-16T03:38:41.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423402+0000", 2024-09-16T03:38:41.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198705305 2024-09-16T03:38:41.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525582+0000", 2024-09-16T03:38:41.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102180178 2024-09-16T03:38:41.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525615+0000", 2024-09-16T03:38:41.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3303000000000003e-05 2024-09-16T03:38:41.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525644+0000", 2024-09-16T03:38:41.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8909999999999999e-05 2024-09-16T03:38:41.307 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.307 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.308 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.308 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:28 4.6 4:6ece7830:::smithi15736696-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:41.308 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:41.308 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 89.157810241999996, 2024-09-16T03:38:41.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550787799999999, 2024-09-16T03:38:41.308 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.308 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.308 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:41.308 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.308 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.309 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.309 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 28 2024-09-16T03:38:41.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.309 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:41.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:41.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:41.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7999999999999996e-07 2024-09-16T03:38:41.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120252+0000", 2024-09-16T03:38:41.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.249e-06 2024-09-16T03:38:41.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120257+0000", 2024-09-16T03:38:41.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1969999999999998e-06 2024-09-16T03:38:41.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120261+0000", 2024-09-16T03:38:41.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1130000000000003e-06 2024-09-16T03:38:41.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222976+0000", 2024-09-16T03:38:41.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102715397 2024-09-16T03:38:41.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:41.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223049+0000", 2024-09-16T03:38:41.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2846e-05 2024-09-16T03:38:41.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525666+0000", 2024-09-16T03:38:41.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30261668800000002 2024-09-16T03:38:41.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525704+0000", 2024-09-16T03:38:41.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8025999999999998e-05 2024-09-16T03:38:41.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525758+0000", 2024-09-16T03:38:41.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3881999999999997e-05 2024-09-16T03:38:41.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.314 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.314 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:49 4.6 4:6df3bbbe:::smithi15736696-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:41.314 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:41.314 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 89.157795178000001, 2024-09-16T03:38:41.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40556222800000002, 2024-09-16T03:38:41.314 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.314 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.315 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.315 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.315 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.315 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.315 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49 2024-09-16T03:38:41.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.315 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:41.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:41.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:41.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8700000000000002e-07 2024-09-16T03:38:41.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120454+0000", 2024-09-16T03:38:41.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018909299999999999 2024-09-16T03:38:41.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120465+0000", 2024-09-16T03:38:41.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.083e-05 2024-09-16T03:38:41.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120468+0000", 2024-09-16T03:38:41.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3869999999999999e-06 2024-09-16T03:38:41.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223082+0000", 2024-09-16T03:38:41.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102613379 2024-09-16T03:38:41.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223189+0000", 2024-09-16T03:38:41.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107728 2024-09-16T03:38:41.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:41.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223368+0000", 2024-09-16T03:38:41.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179011 2024-09-16T03:38:41.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.226087+0000", 2024-09-16T03:38:41.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027183390000000002 2024-09-16T03:38:41.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423408+0000", 2024-09-16T03:38:41.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19732106499999999 2024-09-16T03:38:41.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525779+0000", 2024-09-16T03:38:41.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10237133 2024-09-16T03:38:41.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525805+0000", 2024-09-16T03:38:41.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5967000000000001e-05 2024-09-16T03:38:41.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525827+0000", 2024-09-16T03:38:41.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1812000000000001e-05 2024-09-16T03:38:41.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.322 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.322 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:50 4.6 4:6df3bbbe:::smithi15736696-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:41.322 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:38:41.322 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 89.157586868999999, 2024-09-16T03:38:41.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50526462999999999, 2024-09-16T03:38:41.322 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.322 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.323 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.323 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.323 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 50 2024-09-16T03:38:41.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.323 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:38:41.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:38:41.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120474+0000", 2024-09-16T03:38:41.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4600000000000005e-07 2024-09-16T03:38:41.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120901+0000", 2024-09-16T03:38:41.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00042690000000000002 2024-09-16T03:38:41.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120907+0000", 2024-09-16T03:38:41.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7710000000000001e-06 2024-09-16T03:38:41.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120911+0000", 2024-09-16T03:38:41.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0500000000000002e-06 2024-09-16T03:38:41.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224717+0000", 2024-09-16T03:38:41.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103806175 2024-09-16T03:38:41.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224763+0000", 2024-09-16T03:38:41.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5629e-05 2024-09-16T03:38:41.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:41.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224838+0000", 2024-09-16T03:38:41.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4548999999999994e-05 2024-09-16T03:38:41.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423390+0000", 2024-09-16T03:38:41.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19855242400000001 2024-09-16T03:38:41.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423411+0000", 2024-09-16T03:38:41.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0888000000000001e-05 2024-09-16T03:38:41.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625680+0000", 2024-09-16T03:38:41.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20226923099999999 2024-09-16T03:38:41.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625709+0000", 2024-09-16T03:38:41.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9232000000000001e-05 2024-09-16T03:38:41.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625738+0000", 2024-09-16T03:38:41.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8235e-05 2024-09-16T03:38:41.330 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.330 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.330 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.330 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:51 4.6 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:41.330 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:41.330 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 89.157142286999999, 2024-09-16T03:38:41.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50490003699999997, 2024-09-16T03:38:41.330 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.330 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.331 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.331 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.331 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.331 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.331 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 51 2024-09-16T03:38:41.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.331 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:41.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:41.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:41.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8699999999999995e-07 2024-09-16T03:38:41.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120920+0000", 2024-09-16T03:38:41.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0820000000000001e-06 2024-09-16T03:38:41.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120924+0000", 2024-09-16T03:38:41.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:38:41.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120927+0000", 2024-09-16T03:38:41.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5809999999999999e-06 2024-09-16T03:38:41.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225783+0000", 2024-09-16T03:38:41.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10485588699999999 2024-09-16T03:38:41.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225832+0000", 2024-09-16T03:38:41.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9750000000000003e-05 2024-09-16T03:38:41.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:41.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225910+0000", 2024-09-16T03:38:41.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7084999999999998e-05 2024-09-16T03:38:41.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423414+0000", 2024-09-16T03:38:41.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19750438200000001 2024-09-16T03:38:41.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423456+0000", 2024-09-16T03:38:41.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2386999999999999e-05 2024-09-16T03:38:41.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625766+0000", 2024-09-16T03:38:41.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20231011300000001 2024-09-16T03:38:41.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625790+0000", 2024-09-16T03:38:41.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3672999999999998e-05 2024-09-16T03:38:41.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625818+0000", 2024-09-16T03:38:41.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7576000000000002e-05 2024-09-16T03:38:41.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.338 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.338 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:52 4.6 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:41.338 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:41.338 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 89.157130090999999, 2024-09-16T03:38:41.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50502012900000004, 2024-09-16T03:38:41.338 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.338 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.338 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:41.339 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.339 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.339 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.339 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 52 2024-09-16T03:38:41.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.339 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:41.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:41.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:41.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:38:41.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120932+0000", 2024-09-16T03:38:41.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.181e-06 2024-09-16T03:38:41.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120935+0000", 2024-09-16T03:38:41.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7690000000000001e-06 2024-09-16T03:38:41.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120938+0000", 2024-09-16T03:38:41.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6639999999999998e-06 2024-09-16T03:38:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423188+0000", 2024-09-16T03:38:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30224989200000002 2024-09-16T03:38:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423318+0000", 2024-09-16T03:38:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013045999999999999 2024-09-16T03:38:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625844+0000", 2024-09-16T03:38:41.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20252547600000001 2024-09-16T03:38:41.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625895+0000", 2024-09-16T03:38:41.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1242000000000002e-05 2024-09-16T03:38:41.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625950+0000", 2024-09-16T03:38:41.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5164999999999998e-05 2024-09-16T03:38:41.344 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.344 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.344 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.344 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:279 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:41.344 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:38:41.344 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 86.081178304000005, 2024-09-16T03:38:41.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50446570800000001, 2024-09-16T03:38:41.345 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.345 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.345 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.345 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.345 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.345 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.345 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 279 2024-09-16T03:38:41.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.345 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:38:41.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:38:41.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196883+0000", 2024-09-16T03:38:41.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.48e-06 2024-09-16T03:38:41.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196889+0000", 2024-09-16T03:38:41.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6239999999999999e-06 2024-09-16T03:38:41.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196902+0000", 2024-09-16T03:38:41.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2804e-05 2024-09-16T03:38:41.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196909+0000", 2024-09-16T03:38:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9379999999999992e-06 2024-09-16T03:38:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196963+0000", 2024-09-16T03:38:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3437999999999999e-05 2024-09-16T03:38:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op must wait for map", 2024-09-16T03:38:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196969+0000", 2024-09-16T03:38:41.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1709999999999999e-06 2024-09-16T03:38:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302538+0000", 2024-09-16T03:38:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10556889899999999 2024-09-16T03:38:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302659+0000", 2024-09-16T03:38:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121441 2024-09-16T03:38:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:38:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500040+0000", 2024-09-16T03:38:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19738082900000001 2024-09-16T03:38:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500589+0000", 2024-09-16T03:38:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00054827000000000001 2024-09-16T03:38:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701118+0000", 2024-09-16T03:38:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20052917100000001 2024-09-16T03:38:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701205+0000", 2024-09-16T03:38:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7120999999999995e-05 2024-09-16T03:38:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701289+0000", 2024-09-16T03:38:41.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4457999999999997e-05 2024-09-16T03:38:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701347+0000", 2024-09-16T03:38:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8063999999999998e-05 2024-09-16T03:38:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:306 4.6 4:6ece7830:::smithi15736696-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:41.353 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:38:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.765175988999999, 2024-09-16T03:38:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40927214699999998, 2024-09-16T03:38:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 306 2024-09-16T03:38:41.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:38:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:38:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512885+0000", 2024-09-16T03:38:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.277e-06 2024-09-16T03:38:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512890+0000", 2024-09-16T03:38:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6689999999999997e-06 2024-09-16T03:38:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512900+0000", 2024-09-16T03:38:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0393e-05 2024-09-16T03:38:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512906+0000", 2024-09-16T03:38:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4569999999999996e-06 2024-09-16T03:38:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512951+0000", 2024-09-16T03:38:41.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4891000000000001e-05 2024-09-16T03:38:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.513000+0000", 2024-09-16T03:38:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9187999999999997e-05 2024-09-16T03:38:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.720758+0000", 2024-09-16T03:38:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20775775699999999 2024-09-16T03:38:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.721960+0000", 2024-09-16T03:38:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001202107 2024-09-16T03:38:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922010+0000", 2024-09-16T03:38:41.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20004993900000001 2024-09-16T03:38:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922055+0000", 2024-09-16T03:38:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4981000000000002e-05 2024-09-16T03:38:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922103+0000", 2024-09-16T03:38:41.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8097999999999999e-05 2024-09-16T03:38:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922156+0000", 2024-09-16T03:38:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3390000000000002e-05 2024-09-16T03:38:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:419 4.6 4:64ce87bc:::smithi15736696-20:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:38:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.364376375000006, 2024-09-16T03:38:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40437787200000003, 2024-09-16T03:38:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 419 2024-09-16T03:38:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:38:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:38:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913687+0000", 2024-09-16T03:38:41.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3299999999999999e-06 2024-09-16T03:38:41.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913700+0000", 2024-09-16T03:38:41.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2830000000000001e-05 2024-09-16T03:38:41.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913733+0000", 2024-09-16T03:38:41.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2815999999999999e-05 2024-09-16T03:38:41.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913750+0000", 2024-09-16T03:38:41.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7073000000000001e-05 2024-09-16T03:38:41.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913920+0000", 2024-09-16T03:38:41.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000170015 2024-09-16T03:38:41.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913981+0000", 2024-09-16T03:38:41.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1631999999999995e-05 2024-09-16T03:38:41.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:41.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216580+0000", 2024-09-16T03:38:41.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30259888200000001 2024-09-16T03:38:41.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217205+0000", 2024-09-16T03:38:41.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062461099999999996 2024-09-16T03:38:41.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317915+0000", 2024-09-16T03:38:41.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10071053000000001 2024-09-16T03:38:41.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317996+0000", 2024-09-16T03:38:41.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0440999999999998e-05 2024-09-16T03:38:41.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318032+0000", 2024-09-16T03:38:41.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5944e-05 2024-09-16T03:38:41.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318061+0000", 2024-09-16T03:38:41.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9768e-05 2024-09-16T03:38:41.369 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.370 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.370 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:479 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:41.370 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:38:41.370 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 79.556520229, 2024-09-16T03:38:41.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.504011706, 2024-09-16T03:38:41.370 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.370 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.370 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.370 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.371 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.371 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.371 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 479 2024-09-16T03:38:41.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.371 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:38:41.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:38:41.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721542+0000", 2024-09-16T03:38:41.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3199999999999998e-06 2024-09-16T03:38:41.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721553+0000", 2024-09-16T03:38:41.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1413000000000001e-05 2024-09-16T03:38:41.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721570+0000", 2024-09-16T03:38:41.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.685e-05 2024-09-16T03:38:41.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721596+0000", 2024-09-16T03:38:41.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5647999999999999e-05 2024-09-16T03:38:41.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.921950+0000", 2024-09-16T03:38:41.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20035402899999999 2024-09-16T03:38:41.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.922009+0000", 2024-09-16T03:38:41.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9046999999999999e-05 2024-09-16T03:38:41.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:38:41.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.023850+0000", 2024-09-16T03:38:41.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10184130199999999 2024-09-16T03:38:41.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.024503+0000", 2024-09-16T03:38:41.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065257399999999995 2024-09-16T03:38:41.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.125105+0000", 2024-09-16T03:38:41.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100602046 2024-09-16T03:38:41.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225461+0000", 2024-09-16T03:38:41.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100355835 2024-09-16T03:38:41.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225512+0000", 2024-09-16T03:38:41.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0862999999999997e-05 2024-09-16T03:38:41.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225551+0000", 2024-09-16T03:38:41.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9779e-05 2024-09-16T03:38:41.377 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.378 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.378 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.378 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:996 4.8 4:148e627d:::smithi15736696-23:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:41.378 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:38:41.378 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 70.230717322000004, 2024-09-16T03:38:41.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.73588397999999999, 2024-09-16T03:38:41.378 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.378 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.378 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:41.378 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.379 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.379 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.379 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 996 2024-09-16T03:38:41.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.379 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:38:41.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:38:41.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047344+0000", 2024-09-16T03:38:41.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.846e-06 2024-09-16T03:38:41.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047351+0000", 2024-09-16T03:38:41.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.973e-06 2024-09-16T03:38:41.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047365+0000", 2024-09-16T03:38:41.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3556e-05 2024-09-16T03:38:41.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047382+0000", 2024-09-16T03:38:41.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6994e-05 2024-09-16T03:38:41.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047453+0000", 2024-09-16T03:38:41.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1458000000000006e-05 2024-09-16T03:38:41.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047465+0000", 2024-09-16T03:38:41.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1119e-05 2024-09-16T03:38:41.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449411+0000", 2024-09-16T03:38:41.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40194680399999999 2024-09-16T03:38:41.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449535+0000", 2024-09-16T03:38:41.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124043 2024-09-16T03:38:41.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783227+0000", 2024-09-16T03:38:41.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.333691187 2024-09-16T03:38:41.384 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.384 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.384 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.384 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1048 4.16 4:6df3bbbe:::smithi15736696-13:head [write 1622009~726683] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:41.384 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:41.384 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.702273204999997, 2024-09-16T03:38:41.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40682847599999999, 2024-09-16T03:38:41.384 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.384 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.385 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.385 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.385 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.385 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.385 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1048 2024-09-16T03:38:41.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.385 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:41.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:41.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:41.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-16T03:38:41.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576941+0000", 2024-09-16T03:38:41.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011532230000000001 2024-09-16T03:38:41.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576949+0000", 2024-09-16T03:38:41.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1089999999999998e-06 2024-09-16T03:38:41.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576956+0000", 2024-09-16T03:38:41.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000001e-06 2024-09-16T03:38:41.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878710+0000", 2024-09-16T03:38:41.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30175434299999998 2024-09-16T03:38:41.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878765+0000", 2024-09-16T03:38:41.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4988000000000003e-05 2024-09-16T03:38:41.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:41.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878852+0000", 2024-09-16T03:38:41.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7201999999999996e-05 2024-09-16T03:38:41.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881535+0000", 2024-09-16T03:38:41.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026828310000000001 2024-09-16T03:38:41.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883968+0000", 2024-09-16T03:38:41.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024330659999999998 2024-09-16T03:38:41.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982541+0000", 2024-09-16T03:38:41.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098572876000000004 2024-09-16T03:38:41.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982580+0000", 2024-09-16T03:38:41.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8714000000000002e-05 2024-09-16T03:38:41.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982615+0000", 2024-09-16T03:38:41.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5438000000000002e-05 2024-09-16T03:38:41.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.391 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.392 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1049 4.16 4:6df3bbbe:::smithi15736696-13:head [write 3121996~104915] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:41.392 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:38:41.392 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.701095056, 2024-09-16T03:38:41.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40579890600000001, 2024-09-16T03:38:41.392 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.392 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.392 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.392 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.392 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.393 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.393 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1049 2024-09-16T03:38:41.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.393 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:38:41.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:38:41.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576966+0000", 2024-09-16T03:38:41.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8199999999999999e-07 2024-09-16T03:38:41.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577069+0000", 2024-09-16T03:38:41.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103583 2024-09-16T03:38:41.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577076+0000", 2024-09-16T03:38:41.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8190000000000003e-06 2024-09-16T03:38:41.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577082+0000", 2024-09-16T03:38:41.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5130000000000004e-06 2024-09-16T03:38:41.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880130+0000", 2024-09-16T03:38:41.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.303048127 2024-09-16T03:38:41.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880183+0000", 2024-09-16T03:38:41.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3127999999999997e-05 2024-09-16T03:38:41.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:41.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880282+0000", 2024-09-16T03:38:41.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8881000000000004e-05 2024-09-16T03:38:41.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883101+0000", 2024-09-16T03:38:41.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028194180000000002 2024-09-16T03:38:41.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.979716+0000", 2024-09-16T03:38:41.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.096614899000000004 2024-09-16T03:38:41.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982707+0000", 2024-09-16T03:38:41.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029914249999999998 2024-09-16T03:38:41.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982739+0000", 2024-09-16T03:38:41.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1250000000000001e-05 2024-09-16T03:38:41.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982764+0000", 2024-09-16T03:38:41.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5080999999999999e-05 2024-09-16T03:38:41.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.400 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.400 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1050 4.16 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 3226911] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:41.400 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:38:41.400 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.700970460999997, 2024-09-16T03:38:41.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40784258099999998, 2024-09-16T03:38:41.401 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.401 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.401 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.401 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.401 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.401 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.401 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1050 2024-09-16T03:38:41.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.401 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:38:41.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:38:41.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577090+0000", 2024-09-16T03:38:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000003e-07 2024-09-16T03:38:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577094+0000", 2024-09-16T03:38:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4340000000000001e-06 2024-09-16T03:38:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577099+0000", 2024-09-16T03:38:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8599999999999998e-06 2024-09-16T03:38:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577104+0000", 2024-09-16T03:38:41.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0010000000000001e-06 2024-09-16T03:38:41.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881242+0000", 2024-09-16T03:38:41.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30413779000000002 2024-09-16T03:38:41.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881290+0000", 2024-09-16T03:38:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7298e-05 2024-09-16T03:38:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881367+0000", 2024-09-16T03:38:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7750000000000006e-05 2024-09-16T03:38:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883117+0000", 2024-09-16T03:38:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001750096 2024-09-16T03:38:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.980218+0000", 2024-09-16T03:38:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097100724999999999 2024-09-16T03:38:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984859+0000", 2024-09-16T03:38:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046412950000000001 2024-09-16T03:38:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984900+0000", 2024-09-16T03:38:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0300999999999999e-05 2024-09-16T03:38:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984932+0000", 2024-09-16T03:38:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2332999999999997e-05 2024-09-16T03:38:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1051 4.16 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.700948181000001, 2024-09-16T03:38:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50807376999999998, 2024-09-16T03:38:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1051 2024-09-16T03:38:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:38:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577115+0000", 2024-09-16T03:38:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3980000000000002e-06 2024-09-16T03:38:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577120+0000", 2024-09-16T03:38:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5589999999999996e-06 2024-09-16T03:38:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577125+0000", 2024-09-16T03:38:41.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9910000000000002e-06 2024-09-16T03:38:41.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882780+0000", 2024-09-16T03:38:41.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30565438499999997 2024-09-16T03:38:41.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:41.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882825+0000", 2024-09-16T03:38:41.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5739999999999999e-05 2024-09-16T03:38:41.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085039+0000", 2024-09-16T03:38:41.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20221376499999999 2024-09-16T03:38:41.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085096+0000", 2024-09-16T03:38:41.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6666000000000003e-05 2024-09-16T03:38:41.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085186+0000", 2024-09-16T03:38:41.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9716000000000006e-05 2024-09-16T03:38:41.414 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.414 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.414 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.414 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1359 4.6 4:64ce87bc:::smithi15736696-20:head [write 1768928~405354] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T03:38:41.414 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:38:41.415 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 61.111925042999999, 2024-09-16T03:38:41.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40239613499999999, 2024-09-16T03:38:41.415 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.415 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.415 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.415 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.415 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.415 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.415 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1359 2024-09-16T03:38:41.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.415 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:38:41.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:38:41.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166136+0000", 2024-09-16T03:38:41.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:38:41.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166923+0000", 2024-09-16T03:38:41.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00078683500000000005 2024-09-16T03:38:41.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166933+0000", 2024-09-16T03:38:41.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0445000000000001e-05 2024-09-16T03:38:41.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166938+0000", 2024-09-16T03:38:41.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0119999999999996e-06 2024-09-16T03:38:41.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264901+0000", 2024-09-16T03:38:41.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097962831 2024-09-16T03:38:41.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264963+0000", 2024-09-16T03:38:41.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1812999999999999e-05 2024-09-16T03:38:41.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:41.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.265085+0000", 2024-09-16T03:38:41.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121914 2024-09-16T03:38:41.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.269430+0000", 2024-09-16T03:38:41.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0043455000000000004 2024-09-16T03:38:41.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.372259+0000", 2024-09-16T03:38:41.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102828645 2024-09-16T03:38:41.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568289+0000", 2024-09-16T03:38:41.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19603034999999999 2024-09-16T03:38:41.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568475+0000", 2024-09-16T03:38:41.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018613599999999999 2024-09-16T03:38:41.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568531+0000", 2024-09-16T03:38:41.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5729000000000001e-05 2024-09-16T03:38:41.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.422 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.422 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1676 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v29,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:41.422 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:38:41.422 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.148096180000003, 2024-09-16T03:38:41.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50424139300000004, 2024-09-16T03:38:41.423 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.423 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.423 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.423 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.423 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.423 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.423 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1676 2024-09-16T03:38:41.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.423 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:38:41.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:38:41.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129968+0000", 2024-09-16T03:38:41.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8430000000000003e-06 2024-09-16T03:38:41.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129982+0000", 2024-09-16T03:38:41.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4514e-05 2024-09-16T03:38:41.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130013+0000", 2024-09-16T03:38:41.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1081999999999998e-05 2024-09-16T03:38:41.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130029+0000", 2024-09-16T03:38:41.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5849999999999999e-05 2024-09-16T03:38:41.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130169+0000", 2024-09-16T03:38:41.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000139973 2024-09-16T03:38:41.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130240+0000", 2024-09-16T03:38:41.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0638999999999999e-05 2024-09-16T03:38:41.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:38:41.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.532892+0000", 2024-09-16T03:38:41.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402652654 2024-09-16T03:38:41.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533546+0000", 2024-09-16T03:38:41.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065390700000000001 2024-09-16T03:38:41.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533943+0000", 2024-09-16T03:38:41.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039703999999999998 2024-09-16T03:38:41.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634089+0000", 2024-09-16T03:38:41.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100146098 2024-09-16T03:38:41.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634153+0000", 2024-09-16T03:38:41.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3581e-05 2024-09-16T03:38:41.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634205+0000", 2024-09-16T03:38:41.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2212000000000001e-05 2024-09-16T03:38:41.429 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.430 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.430 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.430 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:77 3.2 e73/68 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head v 73'117, mlcod=66'115)", 2024-09-16T03:38:41.430 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:38:41.430 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 41.099057901999998, 2024-09-16T03:38:41.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.401393467, 2024-09-16T03:38:41.430 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.430 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.430 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.431 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:38:41.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:38:41.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179005+0000", 2024-09-16T03:38:41.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.553e-06 2024-09-16T03:38:41.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179029+0000", 2024-09-16T03:38:41.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4039000000000001e-05 2024-09-16T03:38:41.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179046+0000", 2024-09-16T03:38:41.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7139999999999999e-05 2024-09-16T03:38:41.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.279144+0000", 2024-09-16T03:38:41.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100098245 2024-09-16T03:38:41.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379323+0000", 2024-09-16T03:38:41.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100178621 2024-09-16T03:38:41.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379348+0000", 2024-09-16T03:38:41.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5315000000000001e-05 2024-09-16T03:38:41.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580315+0000", 2024-09-16T03:38:41.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20096734599999999 2024-09-16T03:38:41.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580395+0000", 2024-09-16T03:38:41.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0208000000000002e-05 2024-09-16T03:38:41.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.435 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.435 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.435 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:41.437 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:41.350+0000 7f4214d04640 -1 osd.8 107 handle_osd_map injecting map crc failure 2024-09-16T03:38:41.437 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:41.435+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:41.438 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:38:41.538 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:41.536+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:41.542 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:41.542 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:41.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.543 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:54 3.0 3:04086e73:::a30f14ebc2b326cba32a5aae93ab4cf09bd2a972b24cfcba49e65d7da3ac52de:head [call cas.chunk_create_or_get_ref in=131071b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:38:41.543 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:38:41.543 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.18626189900000001, 2024-09-16T03:38:41.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.18628636900000001, 2024-09-16T03:38:41.543 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.543 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.543 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:41.543 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.543 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.543 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:38:41.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:38:41.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355311+0000", 2024-09-16T03:38:41.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3250000000000002e-06 2024-09-16T03:38:41.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355502+0000", 2024-09-16T03:38:41.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019106 2024-09-16T03:38:41.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355515+0000", 2024-09-16T03:38:41.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3730000000000001e-05 2024-09-16T03:38:41.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355526+0000", 2024-09-16T03:38:41.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0473e-05 2024-09-16T03:38:41.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355567+0000", 2024-09-16T03:38:41.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1455000000000002e-05 2024-09-16T03:38:41.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355576+0000", 2024-09-16T03:38:41.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2840000000000006e-06 2024-09-16T03:38:41.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.457986+0000", 2024-09-16T03:38:41.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10241070200000001 2024-09-16T03:38:41.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.458087+0000", 2024-09-16T03:38:41.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000100715 2024-09-16T03:38:41.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:41.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.458252+0000", 2024-09-16T03:38:41.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016499 2024-09-16T03:38:41.547 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.547 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.547 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.548 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:58 3.0 3:07e63342:::0385a732ea8ffd270f01137d752687ebb6ebb56bc894ea96f00cc7a27fcd2563:head [call cas.chunk_create_or_get_ref in=284238b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:38:41.548 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:38:41.548 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.18602906499999999, 2024-09-16T03:38:41.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.186093814, 2024-09-16T03:38:41.548 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.548 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.548 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:41.548 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355542+0000", 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1699999999999997e-07 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355761+0000", 2024-09-16T03:38:41.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021889699999999999 2024-09-16T03:38:41.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355769+0000", 2024-09-16T03:38:41.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.695e-06 2024-09-16T03:38:41.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355775+0000", 2024-09-16T03:38:41.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5239999999999997e-06 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355851+0000", 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6103e-05 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355860+0000", 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0710000000000001e-06 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459143+0000", 2024-09-16T03:38:41.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10328306399999999 2024-09-16T03:38:41.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459292+0000", 2024-09-16T03:38:41.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000148221 2024-09-16T03:38:41.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:41.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459540+0000", 2024-09-16T03:38:41.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024846100000000001 2024-09-16T03:38:41.552 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.552 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.552 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:60 3.0 3.12430b68 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.18578399400000001, 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.18588184899999999, 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "waiting for peered", 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:38:41.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:38:41.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355787+0000", 2024-09-16T03:38:41.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-16T03:38:41.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355791+0000", 2024-09-16T03:38:41.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.631e-06 2024-09-16T03:38:41.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355797+0000", 2024-09-16T03:38:41.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7379999999999999e-06 2024-09-16T03:38:41.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355802+0000", 2024-09-16T03:38:41.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.322e-06 2024-09-16T03:38:41.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.356898+0000", 2024-09-16T03:38:41.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010963710000000001 2024-09-16T03:38:41.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.557 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.557 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.356902+0000", 2024-09-16T03:38:41.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0110000000000002e-06 2024-09-16T03:38:41.557 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.557 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.557 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.557 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.557 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2024-09-16T03:38:41.557 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 3 2024-09-16T03:38:41.557 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:41.558 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:38:41.639 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:41.638+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:41.661 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:41.661 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:41.662 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:41.662 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:41.662 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:41.670 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:38:41.716 INFO:tasks.rados.rados.0.smithi157.stdout:1458: set_chunk oid 20 offset: 1744896 length: 133632 target oid 0011d3c40d9c06ad735a2dc95936761b4600a08ac9cd7ee354ba7a837b3ddaa5 offset: 0 snap -1 2024-09-16T03:38:41.716 INFO:tasks.rados.rados.0.smithi157.stdout:1459: read oid 23 snap -1 2024-09-16T03:38:41.716 INFO:tasks.rados.rados.0.smithi157.stdout:1459: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:38:41.716 INFO:tasks.rados.rados.0.smithi157.stdout:1460: write oid 42 current snap is 0 2024-09-16T03:38:41.716 INFO:tasks.rados.rados.0.smithi157.stdout:1460: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-09-16T03:38:41.722 INFO:tasks.rados.rados.0.smithi157.stdout:1460: writing smithi15736696-42 from 625293 to 1262511 tid 1 2024-09-16T03:38:41.729 INFO:tasks.rados.rados.0.smithi157.stdout:1460: writing smithi15736696-42 from 1813763 to 2307866 tid 2 2024-09-16T03:38:41.733 INFO:tasks.rados.rados.0.smithi157.stdout:1460: writing smithi15736696-42 from 2978063 to 3025293 tid 3 2024-09-16T03:38:41.733 INFO:tasks.rados.rados.0.smithi157.stdout:1461: read oid 46 snap -1 2024-09-16T03:38:41.733 INFO:tasks.rados.rados.0.smithi157.stdout:1461: expect (ObjNum 236 snap 0 seq_num 236) 2024-09-16T03:38:41.733 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:41.733 INFO:tasks.rados.rados.0.smithi157.stdout:1451: expect (ObjNum 279 snap 0 seq_num 279) 2024-09-16T03:38:41.733 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:41.740 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:41.738+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:41.775 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:41.775 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:41.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:41.775 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:41.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:3 3.0 3.f934d18 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:38:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 86.373030666999995, 2024-09-16T03:38:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78381636700000001, 2024-09-16T03:38:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:38:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:38:41.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399429+0000", 2024-09-16T03:38:41.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:38:41.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399434+0000", 2024-09-16T03:38:41.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1050000000000003e-06 2024-09-16T03:38:41.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399442+0000", 2024-09-16T03:38:41.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0160000000000008e-06 2024-09-16T03:38:41.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399448+0000", 2024-09-16T03:38:41.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.879e-06 2024-09-16T03:38:41.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399489+0000", 2024-09-16T03:38:41.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0815999999999997e-05 2024-09-16T03:38:41.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399495+0000", 2024-09-16T03:38:41.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3439999999999997e-06 2024-09-16T03:38:41.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.779 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.779 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183226+0000", 2024-09-16T03:38:41.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78373077400000002 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183244+0000", 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8128999999999999e-05 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:5 3.0 3.aefbbec0 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 86.273263205000006, 2024-09-16T03:38:41.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68447923499999996, 2024-09-16T03:38:41.781 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.781 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.781 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:41.781 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:38:41.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:38:41.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499196+0000", 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3480000000000001e-06 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499202+0000", 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4219999999999999e-06 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499212+0000", 2024-09-16T03:38:41.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:38:41.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499218+0000", 2024-09-16T03:38:41.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4920000000000001e-06 2024-09-16T03:38:41.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499283+0000", 2024-09-16T03:38:41.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4906000000000005e-05 2024-09-16T03:38:41.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499293+0000", 2024-09-16T03:38:41.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0513000000000001e-05 2024-09-16T03:38:41.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183665+0000", 2024-09-16T03:38:41.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68437196600000005 2024-09-16T03:38:41.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183674+0000", 2024-09-16T03:38:41.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9700000000000005e-06 2024-09-16T03:38:41.785 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.785 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.786 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.786 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:6 3.0 3.3bcda120 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:41.786 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:41.786 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 86.273202596000004, 2024-09-16T03:38:41.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78474037699999999, 2024-09-16T03:38:41.786 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.786 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.786 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:41.786 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:41.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:41.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:41.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0500000000000002e-07 2024-09-16T03:38:41.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499260+0000", 2024-09-16T03:38:41.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6669999999999998e-06 2024-09-16T03:38:41.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499266+0000", 2024-09-16T03:38:41.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:38:41.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499274+0000", 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2250000000000003e-06 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499316+0000", 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2670000000000003e-05 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499323+0000", 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.37e-06 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.789 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283987+0000", 2024-09-16T03:38:41.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78466433700000005 2024-09-16T03:38:41.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.790 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.790 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283996+0000", 2024-09-16T03:38:41.790 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8650000000000007e-06 2024-09-16T03:38:41.790 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.790 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.790 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.790 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:15 3.0 3.7ec93288 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 86.164010586000003, 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57518172599999995, 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.791 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:38:41.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608449+0000", 2024-09-16T03:38:41.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.184e-06 2024-09-16T03:38:41.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.792 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.792 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608459+0000", 2024-09-16T03:38:41.792 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0553999999999999e-05 2024-09-16T03:38:41.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608470+0000", 2024-09-16T03:38:41.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0524e-05 2024-09-16T03:38:41.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.793 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.793 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608493+0000", 2024-09-16T03:38:41.793 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2980999999999999e-05 2024-09-16T03:38:41.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608537+0000", 2024-09-16T03:38:41.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3850000000000002e-05 2024-09-16T03:38:41.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.794 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.794 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608542+0000", 2024-09-16T03:38:41.794 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:38:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183623+0000", 2024-09-16T03:38:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.575080649 2024-09-16T03:38:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183629+0000", 2024-09-16T03:38:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6950000000000003e-06 2024-09-16T03:38:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.795 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:11 3.0 3.77005160 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:38:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 86.070510275999993, 2024-09-16T03:38:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.48153733399999998, 2024-09-16T03:38:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.796 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:38:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:38:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701949+0000", 2024-09-16T03:38:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.314e-06 2024-09-16T03:38:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.797 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701954+0000", 2024-09-16T03:38:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:38:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701964+0000", 2024-09-16T03:38:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9550000000000004e-06 2024-09-16T03:38:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701986+0000", 2024-09-16T03:38:41.798 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1931999999999999e-05 2024-09-16T03:38:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802129+0000", 2024-09-16T03:38:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100142986 2024-09-16T03:38:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802135+0000", 2024-09-16T03:38:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6350000000000003e-06 2024-09-16T03:38:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183475+0000", 2024-09-16T03:38:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38133958099999998 2024-09-16T03:38:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183485+0000", 2024-09-16T03:38:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0642000000000001e-05 2024-09-16T03:38:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:6 3.0 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:38:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.789606363000004, 2024-09-16T03:38:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60469085300000003, 2024-09-16T03:38:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:38:41.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:38:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982853+0000", 2024-09-16T03:38:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.393e-06 2024-09-16T03:38:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982858+0000", 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.062e-06 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982867+0000", 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982879+0000", 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2241e-05 2024-09-16T03:38:41.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982996+0000", 2024-09-16T03:38:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116996 2024-09-16T03:38:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983007+0000", 2024-09-16T03:38:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0694e-05 2024-09-16T03:38:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284499+0000", 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30149171400000002 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284690+0000", 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000191032 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284908+0000", 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021827399999999999 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487092+0000", 2024-09-16T03:38:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.202184156 2024-09-16T03:38:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487110+0000", 2024-09-16T03:38:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7856999999999999e-05 2024-09-16T03:38:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587335+0000", 2024-09-16T03:38:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100224802 2024-09-16T03:38:41.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587410+0000", 2024-09-16T03:38:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5343e-05 2024-09-16T03:38:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587543+0000", 2024-09-16T03:38:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000132492 2024-09-16T03:38:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:2 3.0 3:16550cfd:::smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.789534171, 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40289696000000003, 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982925+0000", 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982928+0000", 2024-09-16T03:38:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4130000000000002e-06 2024-09-16T03:38:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982936+0000", 2024-09-16T03:38:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.875e-06 2024-09-16T03:38:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982942+0000", 2024-09-16T03:38:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:38:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983093+0000", 2024-09-16T03:38:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015144400000000001 2024-09-16T03:38:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983098+0000", 2024-09-16T03:38:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4909999999999999e-06 2024-09-16T03:38:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385695+0000", 2024-09-16T03:38:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402597539 2024-09-16T03:38:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385821+0000", 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125418 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:9 3.0 3:2c9b9e7c:::smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.789517982000007, 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60615655800000001, 2024-09-16T03:38:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:38:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:38:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982941+0000", 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0860000000000001e-06 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982946+0000", 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982954+0000", 2024-09-16T03:38:41.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8520000000000004e-06 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982958+0000", 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4870000000000004e-06 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983116+0000", 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015774699999999999 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983120+0000", 2024-09-16T03:38:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:38:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486023+0000", 2024-09-16T03:38:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50290281400000003 2024-09-16T03:38:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486423+0000", 2024-09-16T03:38:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040079299999999998 2024-09-16T03:38:41.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486707+0000", 2024-09-16T03:38:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028336600000000001 2024-09-16T03:38:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588861+0000", 2024-09-16T03:38:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102154756 2024-09-16T03:38:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588952+0000", 2024-09-16T03:38:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0630000000000005e-05 2024-09-16T03:38:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588984+0000", 2024-09-16T03:38:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2116999999999998e-05 2024-09-16T03:38:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589022+0000", 2024-09-16T03:38:41.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7910000000000001e-05 2024-09-16T03:38:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589097+0000", 2024-09-16T03:38:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4536999999999995e-05 2024-09-16T03:38:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:12 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:38:41.819 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.789442633999997, 2024-09-16T03:38:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60616102900000002, 2024-09-16T03:38:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:38:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.820 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:38:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983017+0000", 2024-09-16T03:38:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.483e-06 2024-09-16T03:38:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983022+0000", 2024-09-16T03:38:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:38:41.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983032+0000", 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0036e-05 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983038+0000", 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6800000000000004e-06 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983155+0000", 2024-09-16T03:38:41.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116401 2024-09-16T03:38:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983163+0000", 2024-09-16T03:38:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6219999999999998e-06 2024-09-16T03:38:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589120+0000", 2024-09-16T03:38:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60595653199999999 2024-09-16T03:38:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589176+0000", 2024-09-16T03:38:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6542000000000002e-05 2024-09-16T03:38:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:30 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:38:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.556124127999993, 2024-09-16T03:38:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40312788999999999, 2024-09-16T03:38:41.824 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:38:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:38:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216338+0000", 2024-09-16T03:38:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0799999999999999e-06 2024-09-16T03:38:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216354+0000", 2024-09-16T03:38:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6439e-05 2024-09-16T03:38:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216392+0000", 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7162000000000001e-05 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216444+0000", 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2082e-05 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417447+0000", 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20100358400000001 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417583+0000", 2024-09-16T03:38:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135537 2024-09-16T03:38:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417777+0000", 2024-09-16T03:38:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019431200000000001 2024-09-16T03:38:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418479+0000", 2024-09-16T03:38:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00070148499999999996 2024-09-16T03:38:41.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418755+0000", 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027627100000000003 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619285+0000", 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200530138 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619370+0000", 2024-09-16T03:38:41.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4567999999999994e-05 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619462+0000", 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2231999999999994e-05 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:18 3.0 3:037ddf75:::smithi15736696-34 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 74.836529857000002, 2024-09-16T03:38:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30645336299999998, 2024-09-16T03:38:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:38:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935928+0000", 2024-09-16T03:38:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935929+0000", 2024-09-16T03:38:41.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4600000000000003e-07 2024-09-16T03:38:41.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935936+0000", 2024-09-16T03:38:41.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3199999999999996e-06 2024-09-16T03:38:41.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935943+0000", 2024-09-16T03:38:41.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8900000000000007e-06 2024-09-16T03:38:41.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.935954+0000", 2024-09-16T03:38:41.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0246e-05 2024-09-16T03:38:41.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140278+0000", 2024-09-16T03:38:41.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20432396999999999 2024-09-16T03:38:41.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140342+0000", 2024-09-16T03:38:41.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4442000000000004e-05 2024-09-16T03:38:41.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:41.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.140468+0000", 2024-09-16T03:38:41.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125718 2024-09-16T03:38:41.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.141460+0000", 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00099208400000000006 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242240+0000", 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100779733 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242316+0000", 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6038999999999995e-05 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242351+0000", 2024-09-16T03:38:41.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4928999999999997e-05 2024-09-16T03:38:41.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:27.242382+0000", 2024-09-16T03:38:41.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1046000000000001e-05 2024-09-16T03:38:41.836 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.836 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.836 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.836 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:34 3.0 3:08307d25:::2b3cb797ae2d3abc24b2cae4a012f26325debe2fc38ccb08bd4503e949d305ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 70.297668071000004, 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30798127600000003, 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474791+0000", 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6000000000000003e-07 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474865+0000", 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4504000000000004e-05 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474882+0000", 2024-09-16T03:38:41.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6765999999999998e-05 2024-09-16T03:38:41.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474890+0000", 2024-09-16T03:38:41.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.14e-06 2024-09-16T03:38:41.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681242+0000", 2024-09-16T03:38:41.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20635221200000001 2024-09-16T03:38:41.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681355+0000", 2024-09-16T03:38:41.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000112211 2024-09-16T03:38:41.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:41.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681505+0000", 2024-09-16T03:38:41.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015040100000000001 2024-09-16T03:38:41.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682352+0000", 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084686400000000002 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682989+0000", 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063749800000000001 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782695+0000", 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099706063999999997 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782735+0000", 2024-09-16T03:38:41.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9304000000000002e-05 2024-09-16T03:38:41.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782771+0000", 2024-09-16T03:38:41.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6551999999999998e-05 2024-09-16T03:38:41.842 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.842 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.842 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.843 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:87 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:41.843 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:38:41.843 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.968473347, 2024-09-16T03:38:41.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402368742, 2024-09-16T03:38:41.843 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.843 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.843 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.843 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:38:41.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803986+0000", 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9400000000000004e-07 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804061+0000", 2024-09-16T03:38:41.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5230999999999999e-05 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804069+0000", 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7530000000000001e-06 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804073+0000", 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9660000000000004e-06 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907236+0000", 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103162566 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907324+0000", 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7894999999999998e-05 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907474+0000", 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014983999999999999 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.908330+0000", 2024-09-16T03:38:41.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085600100000000001 2024-09-16T03:38:41.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107220+0000", 2024-09-16T03:38:41.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198889812 2024-09-16T03:38:41.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206269+0000", 2024-09-16T03:38:41.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099049611999999995 2024-09-16T03:38:41.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206305+0000", 2024-09-16T03:38:41.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5679000000000002e-05 2024-09-16T03:38:41.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206353+0000", 2024-09-16T03:38:41.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8593e-05 2024-09-16T03:38:41.848 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.849 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.849 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.849 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:3086 3.0 3:1e42685c:::6a9007270aa32b63b7f7e5355768bf791fa2d1832cb8adf59666d35d1b037cad:head [write 0~1679360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:41.849 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:38:41.849 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.3606358199999997, 2024-09-16T03:38:41.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57631983099999995, 2024-09-16T03:38:41.849 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.849 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.849 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.849 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.849 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.850 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.850 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 3086 2024-09-16T03:38:41.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.850 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:38:41.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:38:41.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411824+0000", 2024-09-16T03:38:41.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.849e-06 2024-09-16T03:38:41.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414291+0000", 2024-09-16T03:38:41.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024666900000000001 2024-09-16T03:38:41.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414304+0000", 2024-09-16T03:38:41.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2886e-05 2024-09-16T03:38:41.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414310+0000", 2024-09-16T03:38:41.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4219999999999997e-06 2024-09-16T03:38:41.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783484+0000", 2024-09-16T03:38:41.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.36917337700000002 2024-09-16T03:38:41.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783601+0000", 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011796899999999999 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783711+0000", 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110012 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887652+0000", 2024-09-16T03:38:41.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103940373 2024-09-16T03:38:41.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887669+0000", 2024-09-16T03:38:41.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6864e-05 2024-09-16T03:38:41.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942330+0000", 2024-09-16T03:38:41.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.054661333999999999 2024-09-16T03:38:41.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987927+0000", 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045596722999999999 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987993+0000", 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6141000000000003e-05 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.988142+0000", 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014919099999999999 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.855 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.856 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.856 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:211 3.0 3:09d857e3:::85925bf7ff4b61513b387a03c5bd27d81ce6bd960cfaf104686b99338ce39220:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:41.856 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:38:41.856 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.1344567110000003, 2024-09-16T03:38:41.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.55082720799999996, 2024-09-16T03:38:41.856 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.856 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.856 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.856 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638003+0000", 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8250000000000001e-06 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638010+0000", 2024-09-16T03:38:41.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4130000000000003e-06 2024-09-16T03:38:41.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638022+0000", 2024-09-16T03:38:41.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2016e-05 2024-09-16T03:38:41.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638028+0000", 2024-09-16T03:38:41.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0390000000000001e-06 2024-09-16T03:38:41.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886321+0000", 2024-09-16T03:38:41.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.248293081 2024-09-16T03:38:41.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886428+0000", 2024-09-16T03:38:41.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000106791 2024-09-16T03:38:41.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:38:41.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886630+0000", 2024-09-16T03:38:41.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020280499999999999 2024-09-16T03:38:41.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889701+0000", 2024-09-16T03:38:41.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030706650000000002 2024-09-16T03:38:41.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942392+0000", 2024-09-16T03:38:41.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.052691156000000003 2024-09-16T03:38:41.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987675+0000", 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045283229000000001 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188710+0000", 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20103479899999999 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188753+0000", 2024-09-16T03:38:41.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2762000000000002e-05 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188829+0000", 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5626999999999999e-05 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:214 3.0 3:2fb3d035:::5dba7feb0009e61b5e6a50f88cc7ba99127287f7eeb210342a4b40e996304a6b:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.1343426369999996, 2024-09-16T03:38:41.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.55085424699999996, 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638116+0000", 2024-09-16T03:38:41.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1200000000000002e-07 2024-09-16T03:38:41.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638120+0000", 2024-09-16T03:38:41.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3400000000000002e-06 2024-09-16T03:38:41.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638126+0000", 2024-09-16T03:38:41.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2339999999999999e-06 2024-09-16T03:38:41.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638130+0000", 2024-09-16T03:38:41.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6349999999999997e-06 2024-09-16T03:38:41.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887101+0000", 2024-09-16T03:38:41.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.248970462 2024-09-16T03:38:41.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887191+0000", 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0087000000000007e-05 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887375+0000", 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018401600000000001 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889706+0000", 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023305869999999998 2024-09-16T03:38:41.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942851+0000", 2024-09-16T03:38:41.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.053145489999999997 2024-09-16T03:38:41.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.088369+0000", 2024-09-16T03:38:41.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.14551832200000001 2024-09-16T03:38:41.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188856+0000", 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100486244 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188902+0000", 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6134999999999998e-05 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188970+0000", 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7983000000000003e-05 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.868 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:219 3.0 3:21e0c4e8:::f1ac9f6b1092b2dbfccdb2f32bdf689fb55e405da82aefeec91d0d73cdce51e1:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 6.0345363550000002, 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.71756203500000004, 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737924+0000", 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8500000000000001e-06 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737931+0000", 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9879999999999998e-06 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737942+0000", 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1837e-05 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.838013+0000", 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100070438 2024-09-16T03:38:41.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887695+0000", 2024-09-16T03:38:41.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.049682624000000002 2024-09-16T03:38:41.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887831+0000", 2024-09-16T03:38:41.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013565499999999999 2024-09-16T03:38:41.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:38:41.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887948+0000", 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000117353 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889709+0000", 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017605629999999999 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.043145+0000", 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.153436136 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188664+0000", 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.14551857000000001 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455373+0000", 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.26670951199999998 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455413+0000", 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9517999999999997e-05 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455484+0000", 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0990999999999998e-05 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:3100 3.0 3:2db88641:::a63ac782fa4bce1e96397c8301f2734ce3d5850a2dc454a54c42fc5451a8d72d:head [write 0~1584128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.5009560979999996, 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38491842399999998, 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.875 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.876 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.876 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:41.876 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:41.876 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:41.876 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 3100 2024-09-16T03:38:41.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.876 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:38:41.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271523+0000", 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0649000000000002e-05 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273460+0000", 2024-09-16T03:38:41.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019373820000000001 2024-09-16T03:38:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273488+0000", 2024-09-16T03:38:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7537000000000001e-05 2024-09-16T03:38:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273503+0000", 2024-09-16T03:38:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5438999999999999e-05 2024-09-16T03:38:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273645+0000", 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000142013 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273820+0000", 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017546200000000001 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273943+0000", 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012269099999999999 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.276532+0000", 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025884010000000002 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455527+0000", 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.17899536199999999 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.556259+0000", 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100731821 2024-09-16T03:38:41.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656291+0000", 2024-09-16T03:38:41.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100032144 2024-09-16T03:38:41.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656330+0000", 2024-09-16T03:38:41.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8600000000000003e-05 2024-09-16T03:38:41.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656420+0000", 2024-09-16T03:38:41.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0922999999999997e-05 2024-09-16T03:38:41.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.882 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.882 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:54 3.0 3:04086e73:::a30f14ebc2b326cba32a5aae93ab4cf09bd2a972b24cfcba49e65d7da3ac52de:head [call cas.chunk_create_or_get_ref in=131071b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:38:41.882 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:38:41.882 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.41714972299999997, 2024-09-16T03:38:41.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30746864499999998, 2024-09-16T03:38:41.882 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.882 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.882 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355311+0000", 2024-09-16T03:38:41.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3250000000000002e-06 2024-09-16T03:38:41.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355502+0000", 2024-09-16T03:38:41.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019106 2024-09-16T03:38:41.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355515+0000", 2024-09-16T03:38:41.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3730000000000001e-05 2024-09-16T03:38:41.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355526+0000", 2024-09-16T03:38:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0473e-05 2024-09-16T03:38:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355567+0000", 2024-09-16T03:38:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1455000000000002e-05 2024-09-16T03:38:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355576+0000", 2024-09-16T03:38:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2840000000000006e-06 2024-09-16T03:38:41.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.457986+0000", 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10241070200000001 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.458087+0000", 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000100715 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.458252+0000", 2024-09-16T03:38:41.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016499 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560797+0000", 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10254485300000001 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662591+0000", 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10179377100000001 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662693+0000", 2024-09-16T03:38:41.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000102174 2024-09-16T03:38:41.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662745+0000", 2024-09-16T03:38:41.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1887e-05 2024-09-16T03:38:41.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662777+0000", 2024-09-16T03:38:41.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2225999999999999e-05 2024-09-16T03:38:41.888 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.888 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.889 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:58 3.0 3:07e63342:::0385a732ea8ffd270f01137d752687ebb6ebb56bc894ea96f00cc7a27fcd2563:head [call cas.chunk_create_or_get_ref in=284238b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:38:41.889 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:38:41.889 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 0.41691688900000001, 2024-09-16T03:38:41.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30738563099999999, 2024-09-16T03:38:41.889 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:41.889 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:41.889 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:41.889 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:41.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:41.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:38:41.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:41.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:38:41.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:41.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:41.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355542+0000", 2024-09-16T03:38:41.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1699999999999997e-07 2024-09-16T03:38:41.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355761+0000", 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021889699999999999 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355769+0000", 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.695e-06 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355775+0000", 2024-09-16T03:38:41.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5239999999999997e-06 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355851+0000", 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6103e-05 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355860+0000", 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0710000000000001e-06 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:41.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459143+0000", 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10328306399999999 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459292+0000", 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000148221 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459540+0000", 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024846100000000001 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662661+0000", 2024-09-16T03:38:41.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.203120936 2024-09-16T03:38:41.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:41.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662669+0000", 2024-09-16T03:38:41.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6520000000000006e-06 2024-09-16T03:38:41.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662824+0000", 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015512699999999999 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662885+0000", 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0789999999999999e-05 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662927+0000", 2024-09-16T03:38:41.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2172999999999998e-05 2024-09-16T03:38:41.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.898 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:41.898 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:41.898 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:41.900 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:41.900 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:41.900 INFO:tasks.rados.rados.0.smithi157.stdout:1450: finishing set_chunk to oid 22 2024-09-16T03:38:41.900 INFO:tasks.rados.rados.0.smithi157.stdout:1450: got expected ENOENT (src dne) 2024-09-16T03:38:41.900 INFO:tasks.rados.rados.0.smithi157.stdout:1447: finishing write tid 1 to smithi15736696-25 2024-09-16T03:38:41.900 INFO:tasks.rados.rados.0.smithi157.stdout:1447: finishing write tid 2 to smithi15736696-25 2024-09-16T03:38:41.900 INFO:tasks.rados.rados.0.smithi157.stdout:1448: finishing write tid 1 to smithi15736696-4 2024-09-16T03:38:41.900 INFO:tasks.rados.rados.0.smithi157.stdout:1448: finishing write tid 2 to smithi15736696-4 2024-09-16T03:38:41.900 INFO:tasks.rados.rados.0.smithi157.stdout:1448: finishing write tid 3 to smithi15736696-4 2024-09-16T03:38:41.900 INFO:tasks.rados.rados.0.smithi157.stdout:1448: finishing write tid 4 to smithi15736696-4 2024-09-16T03:38:41.901 INFO:tasks.rados.rados.0.smithi157.stdout:1448: oid 4 updating version 0 to 114 2024-09-16T03:38:41.901 INFO:tasks.rados.rados.0.smithi157.stdout:1448: oid 4 version 114 is already newer than 113 2024-09-16T03:38:41.901 INFO:tasks.rados.rados.0.smithi157.stdout:1448: oid 4 updating version 114 to 115 2024-09-16T03:38:41.901 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 115 (ObjNum 296 snap 0 seq_num 296) dirty exists 2024-09-16T03:38:41.901 INFO:tasks.rados.rados.0.smithi157.stdout:1448: left oid 4 (ObjNum 296 snap 0 seq_num 296) 2024-09-16T03:38:41.901 INFO:tasks.rados.rados.0.smithi157.stdout:1453: got (0) Success 2024-09-16T03:38:41.901 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 112 (ObjNum 267 snap 0 seq_num 267) dirty exists 2024-09-16T03:38:41.901 INFO:tasks.rados.rados.0.smithi157.stdout:1447: finishing write tid 3 to smithi15736696-25 2024-09-16T03:38:41.901 INFO:tasks.rados.rados.0.smithi157.stdout:1452: expect (ObjNum 244 snap 0 seq_num 244) 2024-09-16T03:38:41.902 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:38:41.902 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:41.897+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:41.926 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:41.981 INFO:tasks.rados.rados.0.smithi157.stdout:1454: finishing write tid 1 to smithi15736696-35 2024-09-16T03:38:41.981 INFO:tasks.rados.rados.0.smithi157.stdout:1457: finishing set_chunk to oid 39 2024-09-16T03:38:41.981 INFO:tasks.rados.rados.0.smithi157.stdout:1457: got expected ENOENT (src dne) 2024-09-16T03:38:41.981 INFO:tasks.rados.rados.0.smithi157.stdout:1447: finishing write tid 4 to smithi15736696-25 2024-09-16T03:38:41.981 INFO:tasks.rados.rados.0.smithi157.stdout:1448: done (15 left) 2024-09-16T03:38:41.982 INFO:tasks.rados.rados.0.smithi157.stdout:1450: done (14 left) 2024-09-16T03:38:41.982 INFO:tasks.rados.rados.0.smithi157.stdout:1451: done (13 left) 2024-09-16T03:38:41.982 INFO:tasks.rados.rados.0.smithi157.stdout:1452: done (12 left) 2024-09-16T03:38:41.982 INFO:tasks.rados.rados.0.smithi157.stdout:1453: done (11 left) 2024-09-16T03:38:41.982 INFO:tasks.rados.rados.0.smithi157.stdout:1457: done (10 left) 2024-09-16T03:38:41.982 INFO:tasks.rados.rados.0.smithi157.stdout:1462: set_chunk oid 37 target oid 2024-09-16T03:38:41.989 INFO:tasks.rados.rados.0.smithi157.stdout:1462: set_chunk oid 37 offset: 126464 length: 664064 target oid 9ba17af5240d2c83c8b18fa9cd08567cb20b93e30b107e56dd30df31b8b06bf1 offset: 0 snap -1 2024-09-16T03:38:41.990 INFO:tasks.rados.rados.0.smithi157.stdout:1463: delete oid 38 current snap is 0 2024-09-16T03:38:41.990 INFO:tasks.rados.rados.0.smithi157.stdout:1447: finishing write tid 5 to smithi15736696-25 2024-09-16T03:38:41.990 INFO:tasks.rados.rados.0.smithi157.stdout:1447: oid 25 updating version 0 to 122 2024-09-16T03:38:41.990 INFO:tasks.rados.rados.0.smithi157.stdout:1447: oid 25 version 122 is already newer than 119 2024-09-16T03:38:41.990 INFO:tasks.rados.rados.0.smithi157.stdout:1447: oid 25 version 122 is already newer than 120 2024-09-16T03:38:41.990 INFO:tasks.rados.rados.0.smithi157.stdout:1447: oid 25 version 122 is already newer than 121 2024-09-16T03:38:41.990 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 122 (ObjNum 295 snap 0 seq_num 295) dirty exists 2024-09-16T03:38:41.990 INFO:tasks.rados.rados.0.smithi157.stdout:1447: left oid 25 (ObjNum 295 snap 0 seq_num 295) 2024-09-16T03:38:41.990 INFO:tasks.rados.rados.0.smithi157.stdout:1456: expect (ObjNum 246 snap 0 seq_num 246) 2024-09-16T03:38:42.000 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:41.999+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:42.001 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1548990 2024-09-16T03:38:42.043 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:42.043 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:42.043 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:42.043 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:42.052 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:38:42.062 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:42.101 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:42.099+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:42.109 INFO:tasks.rados.rados.0.smithi157.stdout:1449: finishing write tid 2 to smithi15736696-12 2024-09-16T03:38:42.109 INFO:tasks.rados.rados.0.smithi157.stdout:1454: finishing write tid 2 to smithi15736696-35 2024-09-16T03:38:42.109 INFO:tasks.rados.rados.0.smithi157.stdout:1454: finishing write tid 3 to smithi15736696-35 2024-09-16T03:38:42.109 INFO:tasks.rados.rados.0.smithi157.stdout:1454: finishing write tid 4 to smithi15736696-35 2024-09-16T03:38:42.109 INFO:tasks.rados.rados.0.smithi157.stdout:1454: finishing write tid 5 to smithi15736696-35 2024-09-16T03:38:42.110 INFO:tasks.rados.rados.0.smithi157.stdout:1454: oid 35 updating version 0 to 110 2024-09-16T03:38:42.110 INFO:tasks.rados.rados.0.smithi157.stdout:1454: oid 35 updating version 110 to 111 2024-09-16T03:38:42.110 INFO:tasks.rados.rados.0.smithi157.stdout:1454: oid 35 version 111 is already newer than 109 2024-09-16T03:38:42.110 INFO:tasks.rados.rados.0.smithi157.stdout:1454: oid 35 version 111 is already newer than 108 2024-09-16T03:38:42.110 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 111 (ObjNum 298 snap 0 seq_num 298) dirty exists 2024-09-16T03:38:42.110 INFO:tasks.rados.rados.0.smithi157.stdout:1454: left oid 35 (ObjNum 298 snap 0 seq_num 298) 2024-09-16T03:38:42.110 INFO:tasks.rados.rados.0.smithi157.stdout:1447: done (11 left) 2024-09-16T03:38:42.110 INFO:tasks.rados.rados.0.smithi157.stdout:1454: done (10 left) 2024-09-16T03:38:42.110 INFO:tasks.rados.rados.0.smithi157.stdout:1456: done (9 left) 2024-09-16T03:38:42.110 INFO:tasks.rados.rados.0.smithi157.stdout:1463: done (8 left) 2024-09-16T03:38:42.110 INFO:tasks.rados.rados.0.smithi157.stdout:1464: read oid 31 snap -1 2024-09-16T03:38:42.110 INFO:tasks.rados.rados.0.smithi157.stdout:1464: expect (ObjNum 291 snap 0 seq_num 291) 2024-09-16T03:38:42.111 INFO:tasks.rados.rados.0.smithi157.stdout:1465: write oid 47 current snap is 0 2024-09-16T03:38:42.111 INFO:tasks.rados.rados.0.smithi157.stdout:1455: finishing set_chunk to oid 5 2024-09-16T03:38:42.111 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 66 (ObjNum 281 snap 0 seq_num 281) dirty exists 2024-09-16T03:38:42.111 INFO:tasks.rados.rados.0.smithi157.stdout:1449: finishing write tid 3 to smithi15736696-12 2024-09-16T03:38:42.111 INFO:tasks.rados.rados.0.smithi157.stdout:1449: finishing write tid 4 to smithi15736696-12 2024-09-16T03:38:42.111 INFO:tasks.rados.rados.0.smithi157.stdout:1449: oid 12 updating version 0 to 14 2024-09-16T03:38:42.111 INFO:tasks.rados.rados.0.smithi157.stdout:1449: oid 12 version 14 is already newer than 13 2024-09-16T03:38:42.111 INFO:tasks.rados.rados.0.smithi157.stdout:1449: oid 12 updating version 14 to 15 2024-09-16T03:38:42.111 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 15 (ObjNum 297 snap 0 seq_num 297) dirty exists 2024-09-16T03:38:42.111 INFO:tasks.rados.rados.0.smithi157.stdout:1449: left oid 12 (ObjNum 297 snap 0 seq_num 297) 2024-09-16T03:38:42.111 INFO:tasks.rados.rados.0.smithi157.stdout:1459: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:38:42.155 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:42.155 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:42.155 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:42.155 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:42.155 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:42.164 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:38:42.186 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:38:42.188 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:42.205 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:42.203+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:42.305 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:42.303+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:42.307 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:42.307 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:42.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:42.307 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:42.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.307 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:57 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:42.307 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:42.307 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 90.180141966999997, 2024-09-16T03:38:42.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20372090900000001, 2024-09-16T03:38:42.307 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:42.308 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:42.308 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:42.308 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:42.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:42.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:42.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:42.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:42.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:42.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125313+0000", 2024-09-16T03:38:42.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7180000000000001e-06 2024-09-16T03:38:42.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:42.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126730+0000", 2024-09-16T03:38:42.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00141731 2024-09-16T03:38:42.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:42.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126746+0000", 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6455000000000001e-05 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126756+0000", 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227068+0000", 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100312308 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:42.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227085+0000", 2024-09-16T03:38:42.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6898e-05 2024-09-16T03:38:42.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:42.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328995+0000", 2024-09-16T03:38:42.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101909903 2024-09-16T03:38:42.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:42.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329031+0000", 2024-09-16T03:38:42.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5717999999999999e-05 2024-09-16T03:38:42.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.312 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:42.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.312 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:58 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:42.312 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:42.312 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 90.176808836999996, 2024-09-16T03:38:42.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084524500000001, 2024-09-16T03:38:42.312 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:42.312 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:42.312 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:42.312 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:42.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128644+0000", 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.251e-06 2024-09-16T03:38:42.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:42.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128913+0000", 2024-09-16T03:38:42.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026856200000000002 2024-09-16T03:38:42.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.314 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:42.314 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128925+0000", 2024-09-16T03:38:42.314 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2109999999999999e-05 2024-09-16T03:38:42.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:42.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128933+0000", 2024-09-16T03:38:42.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6260000000000003e-06 2024-09-16T03:38:42.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:42.315 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328196+0000", 2024-09-16T03:38:42.315 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19926303200000001 2024-09-16T03:38:42.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.315 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:42.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328253+0000", 2024-09-16T03:38:42.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7262999999999997e-05 2024-09-16T03:38:42.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:42.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329458+0000", 2024-09-16T03:38:42.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012051939999999999 2024-09-16T03:38:42.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.316 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:42.316 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329488+0000", 2024-09-16T03:38:42.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0207000000000001e-05 2024-09-16T03:38:42.316 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:59 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'3, mlcod=0'0)", 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 90.176313403999998, 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036954900000001, 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:42.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:42.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:42.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:42.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:42.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129139+0000", 2024-09-16T03:38:42.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6099999999999999e-07 2024-09-16T03:38:42.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:42.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129158+0000", 2024-09-16T03:38:42.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088e-05 2024-09-16T03:38:42.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:42.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129168+0000", 2024-09-16T03:38:42.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.048e-05 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.229233+0000", 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100065168 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328815+0000", 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099581694999999998 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328845+0000", 2024-09-16T03:38:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0193e-05 2024-09-16T03:38:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329492+0000", 2024-09-16T03:38:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064671800000000001 2024-09-16T03:38:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329508+0000", 2024-09-16T03:38:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6246e-05 2024-09-16T03:38:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.0:15 4.d e47/32 4:b3fe05b2:::smithi15736696-15:head v 47'5, mlcod=39'3)", 2024-09-16T03:38:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.003018678000004, 2024-09-16T03:38:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100920751, 2024-09-16T03:38:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302435+0000", 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2340000000000001e-06 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302446+0000", 2024-09-16T03:38:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0596e-05 2024-09-16T03:38:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302461+0000", 2024-09-16T03:38:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5082000000000001e-05 2024-09-16T03:38:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402592+0000", 2024-09-16T03:38:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100130488 2024-09-16T03:38:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402712+0000", 2024-09-16T03:38:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012013100000000001 2024-09-16T03:38:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402741+0000", 2024-09-16T03:38:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9717999999999999e-05 2024-09-16T03:38:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403309+0000", 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056725700000000005 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403354+0000", 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5244999999999998e-05 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:38:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 52.155519968999997, 2024-09-16T03:38:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032593800000000001, 2024-09-16T03:38:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149943+0000", 2024-09-16T03:38:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0817e-05 2024-09-16T03:38:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151026+0000", 2024-09-16T03:38:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010835350000000001 2024-09-16T03:38:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151040+0000", 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3884e-05 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151049+0000", 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151107+0000", 2024-09-16T03:38:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8199000000000003e-05 2024-09-16T03:38:42.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:42.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151124+0000", 2024-09-16T03:38:42.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7756000000000001e-05 2024-09-16T03:38:42.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:42.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153142+0000", 2024-09-16T03:38:42.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020177569999999998 2024-09-16T03:38:42.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:42.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153191+0000", 2024-09-16T03:38:42.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9051999999999997e-05 2024-09-16T03:38:42.331 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.331 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:42.331 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.332 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1682 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'7, mlcod=40'4)", 2024-09-16T03:38:42.332 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:42.332 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 52.051922779000002, 2024-09-16T03:38:42.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10169502499999999, 2024-09-16T03:38:42.332 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:42.332 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:42.332 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:42.332 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:42.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:42.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:42.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253540+0000", 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0888e-05 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:42.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254015+0000", 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047485499999999999 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254030+0000", 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4899000000000001e-05 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254039+0000", 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5000000000000005e-06 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:42.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254107+0000", 2024-09-16T03:38:42.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7298000000000005e-05 2024-09-16T03:38:42.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:42.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254122+0000", 2024-09-16T03:38:42.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5928999999999999e-05 2024-09-16T03:38:42.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355184+0000", 2024-09-16T03:38:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101061703 2024-09-16T03:38:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355224+0000", 2024-09-16T03:38:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9953000000000002e-05 2024-09-16T03:38:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1683 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'8, mlcod=40'4)", 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 52.051385230999998, 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101630885, 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:42.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:42.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:42.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254068+0000", 2024-09-16T03:38:42.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5700000000000001e-07 2024-09-16T03:38:42.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:42.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254075+0000", 2024-09-16T03:38:42.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2710000000000003e-06 2024-09-16T03:38:42.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:42.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254083+0000", 2024-09-16T03:38:42.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0220000000000004e-06 2024-09-16T03:38:42.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:42.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254089+0000", 2024-09-16T03:38:42.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8699999999999997e-06 2024-09-16T03:38:42.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:42.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354968+0000", 2024-09-16T03:38:42.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100879625 2024-09-16T03:38:42.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:42.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354992+0000", 2024-09-16T03:38:42.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-16T03:38:42.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:42.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355665+0000", 2024-09-16T03:38:42.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00067371100000000001 2024-09-16T03:38:42.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:42.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355698+0000", 2024-09-16T03:38:42.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2291e-05 2024-09-16T03:38:42.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.341 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:42.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.341 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.0:177 4.10 e65/57 4:0a55bbf2:::smithi15736696-18:head v 65'47, mlcod=0'0)", 2024-09-16T03:38:42.341 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:42.341 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 51.098438643000001, 2024-09-16T03:38:42.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074461099999999995, 2024-09-16T03:38:42.342 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:42.342 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:42.342 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:42.342 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:42.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:42.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:42.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:42.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5300000000000001e-07 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207018+0000", 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7210000000000002e-06 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.343 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:42.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207031+0000", 2024-09-16T03:38:42.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2767e-05 2024-09-16T03:38:42.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:42.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207040+0000", 2024-09-16T03:38:42.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5400000000000001e-06 2024-09-16T03:38:42.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:42.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207107+0000", 2024-09-16T03:38:42.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6625999999999995e-05 2024-09-16T03:38:42.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:42.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207131+0000", 2024-09-16T03:38:42.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4128e-05 2024-09-16T03:38:42.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:42.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207712+0000", 2024-09-16T03:38:42.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00058124700000000001 2024-09-16T03:38:42.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:42.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207758+0000", 2024-09-16T03:38:42.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5429000000000002e-05 2024-09-16T03:38:42.346 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.346 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:42.346 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.346 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1723 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'48, mlcod=53'46)", 2024-09-16T03:38:42.346 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:42.346 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.656371462000003, 2024-09-16T03:38:42.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00088284599999999996, 2024-09-16T03:38:42.346 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:42.346 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:42.346 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:42.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649081+0000", 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8299999999999995e-07 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649101+0000", 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0126e-05 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649141+0000", 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9609e-05 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:42.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649161+0000", 2024-09-16T03:38:42.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9599999999999999e-05 2024-09-16T03:38:42.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:42.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649299+0000", 2024-09-16T03:38:42.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013822199999999999 2024-09-16T03:38:42.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:42.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649318+0000", 2024-09-16T03:38:42.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9273999999999999e-05 2024-09-16T03:38:42.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649919+0000", 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060074100000000001 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649963+0000", 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4391000000000002e-05 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1724 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'49, mlcod=53'46)", 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.652706496999997, 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064536300000000003, 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:42.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:42.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:42.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652747+0000", 2024-09-16T03:38:42.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.291e-06 2024-09-16T03:38:42.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:42.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652753+0000", 2024-09-16T03:38:42.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4450000000000001e-06 2024-09-16T03:38:42.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:42.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652782+0000", 2024-09-16T03:38:42.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9094999999999998e-05 2024-09-16T03:38:42.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:42.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652797+0000", 2024-09-16T03:38:42.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5021999999999999e-05 2024-09-16T03:38:42.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652854+0000", 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6586999999999999e-05 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652885+0000", 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.146e-05 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653350+0000", 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046443599999999998 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653391+0000", 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1026999999999999e-05 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1725 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'50, mlcod=53'46)", 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 50.651913673999999, 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056324100000000002, 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:42.355 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653539+0000", 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9699999999999994e-07 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653545+0000", 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9309999999999996e-06 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653557+0000", 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2340999999999999e-05 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653564+0000", 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2779999999999999e-06 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653605+0000", 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1301000000000003e-05 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653621+0000", 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5951e-05 2024-09-16T03:38:42.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654064+0000", 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000442843 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654101+0000", 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7598999999999997e-05 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:42.359 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:42.360 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:42.360 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:42.361 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:42.361 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:42.362 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:42.362 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:38:42.362 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:42.363 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:42.370 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:42.370 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:42.371 INFO:tasks.rados.rados.0.smithi157.stdout:1460: finishing write tid 1 to smithi15736696-42 2024-09-16T03:38:42.371 INFO:tasks.rados.rados.0.smithi157.stdout:1460: finishing write tid 2 to smithi15736696-42 2024-09-16T03:38:42.371 INFO:tasks.rados.rados.0.smithi157.stdout:1460: finishing write tid 3 to smithi15736696-42 2024-09-16T03:38:42.372 INFO:tasks.rados.rados.0.smithi157.stdout:1445: got (0) Success 2024-09-16T03:38:42.372 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 0 (ObjNum 271 snap 0 seq_num 271) dirty exists 2024-09-16T03:38:42.372 INFO:tasks.rados.rados.0.smithi157.stdout:1460: finishing write tid 4 to smithi15736696-42 2024-09-16T03:38:42.372 INFO:tasks.rados.rados.0.smithi157.stdout:1460: finishing write tid 5 to smithi15736696-42 2024-09-16T03:38:42.372 INFO:tasks.rados.rados.0.smithi157.stdout:1460: oid 42 updating version 0 to 72 2024-09-16T03:38:42.372 INFO:tasks.rados.rados.0.smithi157.stdout:1460: oid 42 version 72 is already newer than 70 2024-09-16T03:38:42.372 INFO:tasks.rados.rados.0.smithi157.stdout:1460: oid 42 version 72 is already newer than 69 2024-09-16T03:38:42.372 INFO:tasks.rados.rados.0.smithi157.stdout:1460: oid 42 version 72 is already newer than 71 2024-09-16T03:38:42.372 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 72 (ObjNum 299 snap 0 seq_num 299) dirty exists 2024-09-16T03:38:42.372 INFO:tasks.rados.rados.0.smithi157.stdout:1460: left oid 42 (ObjNum 299 snap 0 seq_num 299) 2024-09-16T03:38:42.372 INFO:tasks.rados.rados.0.smithi157.stdout:1461: expect (ObjNum 236 snap 0 seq_num 236) 2024-09-16T03:38:42.386 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:42.386 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:42.403 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:42.403 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:42.406 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:42.404+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:42.421 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:42.421 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:42.437 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:42.437 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:42.443 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:42.453 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:42.453 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:42.470 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:42.470 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:42.479 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:42.480 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:42.487 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:42.487 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:42.489 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:38:42.503 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:42.504 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:42.509 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:42.507+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:42.513 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:38:42.520 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:42.520 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:42.529 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:42.536 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:42.537 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:42.554 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:42.554 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:42.570 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:42.571 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:42.610 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:42.608+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:42.616 INFO:tasks.rados.rados.0.smithi157.stdout:1458: finishing set_chunk to oid 20 2024-09-16T03:38:42.616 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 43 (ObjNum 288 snap 0 seq_num 288) dirty exists 2024-09-16T03:38:42.616 INFO:tasks.rados.rados.0.smithi157.stdout:1462: finishing set_chunk to oid 37 2024-09-16T03:38:42.616 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 64 (ObjNum 283 snap 0 seq_num 283) dirty exists 2024-09-16T03:38:42.616 INFO:tasks.rados.rados.0.smithi157.stdout:1464: expect (ObjNum 291 snap 0 seq_num 291) 2024-09-16T03:38:42.634 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:42.634 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:42.643 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:38:42.711 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:42.709+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:42.746 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:42.746 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:42.755 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:38:42.755 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:42.812 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:42.810+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:42.826 INFO:tasks.rados.rados.0.smithi157.stdout:1465: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-09-16T03:38:42.833 INFO:tasks.rados.rados.0.smithi157.stdout:1465: writing smithi15736696-47 from 658907 to 1446623 tid 1 2024-09-16T03:38:42.839 INFO:tasks.rados.rados.0.smithi157.stdout:1465: writing smithi15736696-47 from 1930727 to 2428499 tid 2 2024-09-16T03:38:42.843 INFO:tasks.rados.rados.0.smithi157.stdout:1465: writing smithi15736696-47 from 3032909 to 3058907 tid 3 2024-09-16T03:38:42.843 INFO:tasks.rados.rados.0.smithi157.stdout:1445: done (9 left) 2024-09-16T03:38:42.844 INFO:tasks.rados.rados.0.smithi157.stdout:1449: done (8 left) 2024-09-16T03:38:42.844 INFO:tasks.rados.rados.0.smithi157.stdout:1455: done (7 left) 2024-09-16T03:38:42.844 INFO:tasks.rados.rados.0.smithi157.stdout:1458: done (6 left) 2024-09-16T03:38:42.844 INFO:tasks.rados.rados.0.smithi157.stdout:1459: done (5 left) 2024-09-16T03:38:42.844 INFO:tasks.rados.rados.0.smithi157.stdout:1460: done (4 left) 2024-09-16T03:38:42.844 INFO:tasks.rados.rados.0.smithi157.stdout:1461: done (3 left) 2024-09-16T03:38:42.844 INFO:tasks.rados.rados.0.smithi157.stdout:1462: done (2 left) 2024-09-16T03:38:42.844 INFO:tasks.rados.rados.0.smithi157.stdout:1464: done (1 left) 2024-09-16T03:38:42.844 INFO:tasks.rados.rados.0.smithi157.stdout:1466: read oid 7 snap -1 2024-09-16T03:38:42.844 INFO:tasks.rados.rados.0.smithi157.stdout:1466: expect (ObjNum 257 snap 0 seq_num 257) 2024-09-16T03:38:42.844 INFO:tasks.rados.rados.0.smithi157.stdout:1467: set_chunk oid 37 target oid 2024-09-16T03:38:42.861 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:42.861 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:42.870 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:38:42.913 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:42.911+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:42.957 INFO:tasks.rados.rados.0.smithi157.stdout:1467: set_chunk oid 37 offset: 725504 length: 1651712 target oid ff80442105a743685d3f05bcb10f5081ef2dceeec3d6d3ad2c6d4a775cd96dd1 offset: 0 snap -1 2024-09-16T03:38:42.957 INFO:tasks.rados.rados.0.smithi157.stdout:1468: read oid 16 snap -1 2024-09-16T03:38:42.957 INFO:tasks.rados.rados.0.smithi157.stdout:1468: expect (ObjNum 270 snap 0 seq_num 270) 2024-09-16T03:38:42.957 INFO:tasks.rados.rados.0.smithi157.stdout:1469: write (excl) oid 40 current snap is 0 2024-09-16T03:38:42.957 INFO:tasks.rados.rados.0.smithi157.stdout:1469: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-09-16T03:38:42.962 INFO:tasks.rados.rados.0.smithi157.stdout:1469: writing smithi15736696-40 from 675714 to 1138679 tid 1 2024-09-16T03:38:42.968 INFO:tasks.rados.rados.0.smithi157.stdout:1469: writing smithi15736696-40 from 1589209 to 2230639 tid 2 2024-09-16T03:38:42.973 INFO:tasks.rados.rados.0.smithi157.stdout:1469: writing smithi15736696-40 from 2943979 to 3075714 tid 3 2024-09-16T03:38:42.973 INFO:tasks.rados.rados.0.smithi157.stdout:1470: write (excl) oid 34 current snap is 0 2024-09-16T03:38:42.974 INFO:tasks.rados.rados.0.smithi157.stdout:1470: seq_num 303 ranges {692521=538214,1931338=785088} 2024-09-16T03:38:42.975 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:42.975 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:42.979 INFO:tasks.rados.rados.0.smithi157.stdout:1470: writing smithi15736696-34 from 692521 to 1230735 tid 1 2024-09-16T03:38:42.983 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:38:42.988 INFO:tasks.rados.rados.0.smithi157.stdout:1470: writing smithi15736696-34 from 1931338 to 2716426 tid 2 2024-09-16T03:38:42.988 INFO:tasks.rados.rados.0.smithi157.stdout:1471: set_chunk oid 1 target oid 2024-09-16T03:38:43.014 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:43.013+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:43.088 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:43.089 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:43.092 INFO:tasks.rados.rados.0.smithi157.stdout:1471: set_chunk oid 1 offset: 567808 length: 287232 target oid a7062e3e3f2b5371e9b37a03f0f785bed2bb5dcbdd278653d8e89acdb02d2e74 offset: 0 snap -1 2024-09-16T03:38:43.092 INFO:tasks.rados.rados.0.smithi157.stdout:1472: write oid 13 current snap is 0 2024-09-16T03:38:43.092 INFO:tasks.rados.rados.0.smithi157.stdout:1472: seq_num 304 ranges {709328=613463,1989820=645099} 2024-09-16T03:38:43.097 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:38:43.099 INFO:tasks.rados.rados.0.smithi157.stdout:1472: writing smithi15736696-13 from 709328 to 1322791 tid 1 2024-09-16T03:38:43.106 INFO:tasks.rados.rados.0.smithi157.stdout:1472: writing smithi15736696-13 from 1989820 to 2634919 tid 2 2024-09-16T03:38:43.106 INFO:tasks.rados.rados.0.smithi157.stdout:1473: write (excl) oid 19 current snap is 0 2024-09-16T03:38:43.106 INFO:tasks.rados.rados.0.smithi157.stdout:1473: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-09-16T03:38:43.112 INFO:tasks.rados.rados.0.smithi157.stdout:1473: writing smithi15736696-19 from 726135 to 1131200 tid 1 2024-09-16T03:38:43.120 INFO:tasks.rados.rados.0.smithi157.stdout:1473: writing smithi15736696-19 from 1764655 to 2553412 tid 2 2024-09-16T03:38:43.123 INFO:tasks.rados.rados.0.smithi157.stdout:1473: writing smithi15736696-19 from 3026248 to 3126135 tid 3 2024-09-16T03:38:43.123 INFO:tasks.rados.rados.0.smithi157.stdout:1474: write (excl) oid 36 current snap is 0 2024-09-16T03:38:43.123 INFO:tasks.rados.rados.0.smithi157.stdout:1474: seq_num 306 ranges {742942=480314,1823137=648768} 2024-09-16T03:38:43.129 INFO:tasks.rados.rados.0.smithi157.stdout:1474: writing smithi15736696-36 from 742942 to 1223256 tid 1 2024-09-16T03:38:43.136 INFO:tasks.rados.rados.0.smithi157.stdout:1474: writing smithi15736696-36 from 1823137 to 2471905 tid 2 2024-09-16T03:38:43.136 INFO:tasks.rados.rados.0.smithi157.stdout:1475: set_chunk oid 49 target oid 2024-09-16T03:38:43.141 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:43.139+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:43.203 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:43.203 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:43.212 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:38:43.216 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:43.215+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:43.317 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:43.316+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:43.345 INFO:tasks.rados.rados.0.smithi157.stdout:1475: set_chunk oid 49 offset: 772608 length: 852992 target oid 38e7227acd42582cc5e84cf66aa38bc8c8ea690c8b1e6a4e4a6847c2defec17d offset: 0 snap -1 2024-09-16T03:38:43.345 INFO:tasks.rados.rados.0.smithi157.stdout:1476: read oid 27 snap -1 2024-09-16T03:38:43.345 INFO:tasks.rados.rados.0.smithi157.stdout:1476: expect (ObjNum 244 snap 0 seq_num 244) 2024-09-16T03:38:43.345 INFO:tasks.rados.rados.0.smithi157.stdout:1477: write (excl) oid 35 current snap is 0 2024-09-16T03:38:43.345 INFO:tasks.rados.rados.0.smithi157.stdout:1465: finishing write tid 1 to smithi15736696-47 2024-09-16T03:38:43.346 INFO:tasks.rados.rados.0.smithi157.stdout:1466: expect (ObjNum 257 snap 0 seq_num 257) 2024-09-16T03:38:43.356 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:43.356 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:43.365 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:38:43.370 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:43.372 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 9] out_osds: [1, 5, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:38:43.373 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:38:43.373 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:38:43.373 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T03:38:43.373 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:38:43.396 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:43.412 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:38:43.418 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:43.416+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:43.421 INFO:tasks.rados.rados.0.smithi157.stdout:1467: finishing set_chunk to oid 37 2024-09-16T03:38:43.421 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 37 set_chunk returned error code -95 offset: 725504 length: 1651712 2024-09-16T03:38:43.421 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 0 (ObjNum 283 snap 0 seq_num 283) dirty exists 2024-09-16T03:38:43.421 INFO:tasks.rados.rados.0.smithi157.stdout:1465: finishing write tid 2 to smithi15736696-47 2024-09-16T03:38:43.421 INFO:tasks.rados.rados.0.smithi157.stdout:1465: finishing write tid 3 to smithi15736696-47 2024-09-16T03:38:43.422 INFO:tasks.rados.rados.0.smithi157.stdout:1469: finishing write tid 1 to smithi15736696-40 2024-09-16T03:38:43.422 INFO:tasks.rados.rados.0.smithi157.stdout:1477: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2024-09-16T03:38:43.428 INFO:tasks.rados.rados.0.smithi157.stdout:1477: writing smithi15736696-35 from 759749 to 1315312 tid 1 2024-09-16T03:38:43.435 INFO:tasks.rados.rados.0.smithi157.stdout:1477: writing smithi15736696-35 from 1765266 to 2557692 tid 2 2024-09-16T03:38:43.439 INFO:tasks.rados.rados.0.smithi157.stdout:1477: writing smithi15736696-35 from 2964741 to 3159749 tid 3 2024-09-16T03:38:43.439 INFO:tasks.rados.rados.0.smithi157.stdout:1466: done (12 left) 2024-09-16T03:38:43.439 INFO:tasks.rados.rados.0.smithi157.stdout:1467: done (11 left) 2024-09-16T03:38:43.439 INFO:tasks.rados.rados.0.smithi157.stdout:1478: tier_evict oid 38 2024-09-16T03:38:43.439 INFO:tasks.rados.rados.0.smithi157.stdout:1478: tier_evict oid 38 snap -1 2024-09-16T03:38:43.439 INFO:tasks.rados.rados.0.smithi157.stdout:1471: finishing set_chunk to oid 1 2024-09-16T03:38:43.439 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 1 set_chunk returned error code -95 offset: 567808 length: 287232 2024-09-16T03:38:43.440 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 0 (ObjNum 242 snap 0 seq_num 242) dirty exists 2024-09-16T03:38:43.440 INFO:tasks.rados.rados.0.smithi157.stdout:1465: finishing write tid 4 to smithi15736696-47 2024-09-16T03:38:43.440 INFO:tasks.rados.rados.0.smithi157.stdout:1465: finishing write tid 5 to smithi15736696-47 2024-09-16T03:38:43.440 INFO:tasks.rados.rados.0.smithi157.stdout:1465: oid 47 updating version 0 to 113 2024-09-16T03:38:43.440 INFO:tasks.rados.rados.0.smithi157.stdout:1465: oid 47 version 113 is already newer than 112 2024-09-16T03:38:43.440 INFO:tasks.rados.rados.0.smithi157.stdout:1465: oid 47 updating version 113 to 115 2024-09-16T03:38:43.440 INFO:tasks.rados.rados.0.smithi157.stdout:1465: oid 47 version 115 is already newer than 114 2024-09-16T03:38:43.440 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 115 (ObjNum 301 snap 0 seq_num 301) dirty exists 2024-09-16T03:38:43.440 INFO:tasks.rados.rados.0.smithi157.stdout:1465: left oid 47 (ObjNum 301 snap 0 seq_num 301) 2024-09-16T03:38:43.440 INFO:tasks.rados.rados.0.smithi157.stdout:1468: expect (ObjNum 270 snap 0 seq_num 270) 2024-09-16T03:38:43.440 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:43.469 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:43.469 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:43.477 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:38:43.487 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 163242 2024-09-16T03:38:43.519 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:43.517+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:43.549 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:43.548+0000 7f12e2a1a640 -1 osd.5 108 handle_osd_map injecting map crc failure 2024-09-16T03:38:43.556 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2205438 2024-09-16T03:38:43.573 INFO:tasks.rados.rados.0.smithi157.stdout:1472: finishing write tid 1 to smithi15736696-13 2024-09-16T03:38:43.573 INFO:tasks.rados.rados.0.smithi157.stdout:1469: finishing write tid 2 to smithi15736696-40 2024-09-16T03:38:43.573 INFO:tasks.rados.rados.0.smithi157.stdout:1469: finishing write tid 3 to smithi15736696-40 2024-09-16T03:38:43.573 INFO:tasks.rados.rados.0.smithi157.stdout:1472: finishing write tid 2 to smithi15736696-13 2024-09-16T03:38:43.573 INFO:tasks.rados.rados.0.smithi157.stdout:1472: finishing write tid 3 to smithi15736696-13 2024-09-16T03:38:43.573 INFO:tasks.rados.rados.0.smithi157.stdout:1472: finishing write tid 4 to smithi15736696-13 2024-09-16T03:38:43.573 INFO:tasks.rados.rados.0.smithi157.stdout:1472: oid 13 updating version 0 to 70 2024-09-16T03:38:43.573 INFO:tasks.rados.rados.0.smithi157.stdout:1472: oid 13 updating version 70 to 72 2024-09-16T03:38:43.573 INFO:tasks.rados.rados.0.smithi157.stdout:1472: oid 13 version 72 is already newer than 71 2024-09-16T03:38:43.574 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 72 (ObjNum 304 snap 0 seq_num 304) dirty exists 2024-09-16T03:38:43.574 INFO:tasks.rados.rados.0.smithi157.stdout:1472: left oid 13 (ObjNum 304 snap 0 seq_num 304) 2024-09-16T03:38:43.574 INFO:tasks.rados.rados.0.smithi157.stdout:1469: finishing write tid 4 to smithi15736696-40 2024-09-16T03:38:43.574 INFO:tasks.rados.rados.0.smithi157.stdout:1465: done (11 left) 2024-09-16T03:38:43.574 INFO:tasks.rados.rados.0.smithi157.stdout:1468: done (10 left) 2024-09-16T03:38:43.574 INFO:tasks.rados.rados.0.smithi157.stdout:1471: done (9 left) 2024-09-16T03:38:43.574 INFO:tasks.rados.rados.0.smithi157.stdout:1472: done (8 left) 2024-09-16T03:38:43.574 INFO:tasks.rados.rados.0.smithi157.stdout:1479: set_chunk oid 25 target oid 2024-09-16T03:38:43.620 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:43.618+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:43.623 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:43.623 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:43.632 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:38:43.679 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:38:43.680 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:38:43.690 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-16T03:38:43.409930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":173371459,"num_objects":144,"num_object_clones":0,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":2026,"num_read_kb":1084393,"num_write":1551,"num_write_kb":355439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":44608194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":14,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":147,"acting":147,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":13,"num_per_pool_osds":13,"num_per_pool_omap_osds":13,"kb":1226833920,"kb_used":740444,"kb_used_data":318256,"kb_used_omap":20,"kb_used_meta":401003,"kb_avail":1226093476,"statfs":{"total":1256277934080,"available":1255519719424,"internally_reserved":0,"allocated":325894144,"data_stored":331601796,"data_compressed":8229725,"data_compressed_allocated":8847360,"data_compressed_original":19521536,"omap_allocated":20735,"internal_metadata":410627841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,0,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":911495,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":-1173,"num_read_kb":-1046751,"num_write":-351,"num_write_kb":-116697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":-2748698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.711018"},"pg_stats":[{"pgid":"4.23","version":"105'37","reported_seq":290,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.397469+0000","last_change":"2024-09-16T03:38:34.893610+0000","last_active":"2024-09-16T03:38:41.397469+0000","last_peered":"2024-09-16T03:38:41.397469+0000","last_clean":"2024-09-16T03:38:41.397469+0000","last_became_active":"2024-09-16T03:38:33.881123+0000","last_became_peered":"2024-09-16T03:38:33.881123+0000","last_unstale":"2024-09-16T03:38:41.397469+0000","last_undegraded":"2024-09-16T03:38:41.397469+0000","last_fullsized":"2024-09-16T03:38:41.397469+0000","mapping_epoch":100,"log_start":"94'34","ondisk_log_start":"94'34","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:34.893524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.893524+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":33,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.098251+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2789995,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.22","version":"45'6","reported_seq":217,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.458548+0000","last_change":"2024-09-16T03:38:34.836413+0000","last_active":"2024-09-16T03:38:41.458548+0000","last_peered":"2024-09-16T03:38:41.458548+0000","last_clean":"2024-09-16T03:38:41.458548+0000","last_became_active":"2024-09-16T03:38:33.689595+0000","last_became_peered":"2024-09-16T03:38:33.689595+0000","last_unstale":"2024-09-16T03:38:41.458548+0000","last_undegraded":"2024-09-16T03:38:41.458548+0000","last_fullsized":"2024-09-16T03:38:41.458548+0000","mapping_epoch":100,"log_start":"39'3","ondisk_log_start":"39'3","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:34.836343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.836343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:01.708631+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.21","version":"48'7","reported_seq":63,"reported_epoch":107,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:40.565286+0000","last_change":"2024-09-16T03:38:35.726255+0000","last_active":"2024-09-16T03:38:40.565286+0000","last_peered":"2024-09-16T03:38:40.565286+0000","last_clean":"2024-09-16T03:37:28.467886+0000","last_became_active":"2024-09-16T03:38:35.624748+0000","last_became_peered":"2024-09-16T03:38:35.624748+0000","last_unstale":"2024-09-16T03:38:40.565286+0000","last_undegraded":"2024-09-16T03:38:40.565286+0000","last_fullsized":"2024-09-16T03:38:40.565286+0000","mapping_epoch":102,"log_start":"40'4","ondisk_log_start":"40'4","created":100,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":6,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:37:39.701279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.701279+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":760509,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,12],"acting":[2,4,7],"avail_no_missing":["2","4","7","8","11","12"],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.20","version":"103'118","reported_seq":472,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.393270+0000","last_change":"2024-09-16T03:38:36.698284+0000","last_active":"2024-09-16T03:38:38.393270+0000","last_peered":"2024-09-16T03:38:38.393270+0000","last_clean":"2024-09-16T03:38:38.393270+0000","last_became_active":"2024-09-16T03:38:33.781861+0000","last_became_peered":"2024-09-16T03:38:33.781861+0000","last_unstale":"2024-09-16T03:38:38.393270+0000","last_undegraded":"2024-09-16T03:38:38.393270+0000","last_fullsized":"2024-09-16T03:38:38.393270+0000","mapping_epoch":100,"log_start":"102'115","ondisk_log_start":"102'115","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'118","last_scrub_stamp":"2024-09-16T03:38:36.698150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:36.698150+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":115,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:40.200896+0000","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4773254,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6706,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.1f","version":"105'97","reported_seq":398,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.394104+0000","last_change":"2024-09-16T03:38:37.202045+0000","last_active":"2024-09-16T03:38:38.394104+0000","last_peered":"2024-09-16T03:38:38.394104+0000","last_clean":"2024-09-16T03:38:38.394104+0000","last_became_active":"2024-09-16T03:38:33.782199+0000","last_became_peered":"2024-09-16T03:38:33.782199+0000","last_unstale":"2024-09-16T03:38:38.394104+0000","last_undegraded":"2024-09-16T03:38:38.394104+0000","last_fullsized":"2024-09-16T03:38:38.394104+0000","mapping_epoch":100,"log_start":"102'93","ondisk_log_start":"102'93","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'95","last_scrub_stamp":"2024-09-16T03:38:37.201959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:37.201959+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":93,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.013775+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2638732,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.1e","version":"92'59","reported_seq":317,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.358978+0000","last_change":"2024-09-16T03:38:35.977708+0000","last_active":"2024-09-16T03:38:38.358978+0000","last_peered":"2024-09-16T03:38:38.358978+0000","last_clean":"2024-09-16T03:38:38.358978+0000","last_became_active":"2024-09-16T03:38:33.680215+0000","last_became_peered":"2024-09-16T03:38:33.680215+0000","last_unstale":"2024-09-16T03:38:38.358978+0000","last_undegraded":"2024-09-16T03:38:38.358978+0000","last_fullsized":"2024-09-16T03:38:38.358978+0000","mapping_epoch":100,"log_start":"89'56","ondisk_log_start":"89'56","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'59","last_scrub_stamp":"2024-09-16T03:38:35.977646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.977646+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:38.000887+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"92'14","reported_seq":256,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.456327+0000","last_change":"2024-09-16T03:38:35.267486+0000","last_active":"2024-09-16T03:38:38.456327+0000","last_peered":"2024-09-16T03:38:38.456327+0000","last_clean":"2024-09-16T03:38:38.456327+0000","last_became_active":"2024-09-16T03:38:33.878609+0000","last_became_peered":"2024-09-16T03:38:33.878609+0000","last_unstale":"2024-09-16T03:38:38.456327+0000","last_undegraded":"2024-09-16T03:38:38.456327+0000","last_fullsized":"2024-09-16T03:38:38.456327+0000","mapping_epoch":100,"log_start":"91'11","ondisk_log_start":"91'11","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:35.267359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.267359+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:58.109061+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.1c","version":"99'56","reported_seq":298,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.456306+0000","last_change":"2024-09-16T03:38:37.557541+0000","last_active":"2024-09-16T03:38:38.456306+0000","last_peered":"2024-09-16T03:38:38.456306+0000","last_clean":"2024-09-16T03:38:38.456306+0000","last_became_active":"2024-09-16T03:38:37.557219+0000","last_became_peered":"2024-09-16T03:38:37.557219+0000","last_unstale":"2024-09-16T03:38:38.456306+0000","last_undegraded":"2024-09-16T03:38:38.456306+0000","last_fullsized":"2024-09-16T03:38:38.456306+0000","mapping_epoch":104,"log_start":"90'53","ondisk_log_start":"90'53","created":100,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":5,"last_scrub":"99'56","last_scrub_stamp":"2024-09-16T03:38:36.187596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:36.187596+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.499035+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.1b","version":"106'108","reported_seq":477,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.412106+0000","last_change":"2024-09-16T03:38:39.905868+0000","last_active":"2024-09-16T03:38:41.412106+0000","last_peered":"2024-09-16T03:38:41.412106+0000","last_clean":"2024-09-16T03:38:41.412106+0000","last_became_active":"2024-09-16T03:38:38.557289+0000","last_became_peered":"2024-09-16T03:38:38.557289+0000","last_unstale":"2024-09-16T03:38:41.412106+0000","last_undegraded":"2024-09-16T03:38:41.412106+0000","last_fullsized":"2024-09-16T03:38:41.412106+0000","mapping_epoch":105,"log_start":"104'105","ondisk_log_start":"104'105","created":100,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'107","last_scrub_stamp":"2024-09-16T03:38:39.905697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.905697+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":99,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:46.763459+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6655605,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":23291,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.1a","version":"101'115","reported_seq":501,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.676391+0000","last_change":"2024-09-16T03:38:35.254581+0000","last_active":"2024-09-16T03:38:38.676391+0000","last_peered":"2024-09-16T03:38:38.676391+0000","last_clean":"2024-09-16T03:38:38.676391+0000","last_became_active":"2024-09-16T03:38:33.888393+0000","last_became_peered":"2024-09-16T03:38:33.888393+0000","last_unstale":"2024-09-16T03:38:38.676391+0000","last_undegraded":"2024-09-16T03:38:38.676391+0000","last_fullsized":"2024-09-16T03:38:38.676391+0000","mapping_epoch":100,"log_start":"99'112","ondisk_log_start":"99'112","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'115","last_scrub_stamp":"2024-09-16T03:38:35.254488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.254488+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":112,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:39.749615+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5042166,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":12852,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":172,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.364545+0000","last_change":"2024-09-16T03:37:52.558405+0000","last_active":"2024-09-16T03:38:38.364545+0000","last_peered":"2024-09-16T03:38:38.364545+0000","last_clean":"2024-09-16T03:38:38.364545+0000","last_became_active":"2024-09-16T03:37:52.558043+0000","last_became_peered":"2024-09-16T03:37:52.558043+0000","last_unstale":"2024-09-16T03:38:38.364545+0000","last_undegraded":"2024-09-16T03:38:38.364545+0000","last_fullsized":"2024-09-16T03:38:38.364545+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:39.955354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.955354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:51.622468+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"98'44","reported_seq":281,"reported_epoch":106,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:38:38.396334+0000","last_change":"2024-09-16T03:38:33.880442+0000","last_active":"2024-09-16T03:38:38.396334+0000","last_peered":"2024-09-16T03:38:38.396334+0000","last_clean":"2024-09-16T03:37:50.435122+0000","last_became_active":"2024-09-16T03:38:33.779761+0000","last_became_peered":"2024-09-16T03:38:33.779761+0000","last_unstale":"2024-09-16T03:38:38.396334+0000","last_undegraded":"2024-09-16T03:38:38.396334+0000","last_fullsized":"2024-09-16T03:38:38.396334+0000","mapping_epoch":100,"log_start":"96'41","ondisk_log_start":"96'41","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'29","last_scrub_stamp":"2024-09-16T03:37:39.786314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.786314+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":38,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":781,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":24,"num_read_kb":21989,"num_write":14,"num_write_kb":3435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,5,1],"avail_no_missing":["0","1","4","5","9","10"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"104'45","reported_seq":311,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.594513+0000","last_change":"2024-09-16T03:37:47.616917+0000","last_active":"2024-09-16T03:38:38.594513+0000","last_peered":"2024-09-16T03:38:38.594513+0000","last_clean":"2024-09-16T03:38:38.594513+0000","last_became_active":"2024-09-16T03:37:31.713115+0000","last_became_peered":"2024-09-16T03:37:31.713115+0000","last_unstale":"2024-09-16T03:38:38.594513+0000","last_undegraded":"2024-09-16T03:38:38.594513+0000","last_fullsized":"2024-09-16T03:38:38.594513+0000","mapping_epoch":50,"log_start":"98'42","ondisk_log_start":"98'42","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":41,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:17.517572+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4621958,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":83172,"num_write":10,"num_write_kb":2620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.16","version":"104'69","reported_seq":329,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.395637+0000","last_change":"2024-09-16T03:37:57.546992+0000","last_active":"2024-09-16T03:38:38.395637+0000","last_peered":"2024-09-16T03:38:38.395637+0000","last_clean":"2024-09-16T03:38:38.395637+0000","last_became_active":"2024-09-16T03:37:57.544402+0000","last_became_peered":"2024-09-16T03:37:57.544402+0000","last_unstale":"2024-09-16T03:38:38.395637+0000","last_undegraded":"2024-09-16T03:38:38.395637+0000","last_fullsized":"2024-09-16T03:38:38.395637+0000","mapping_epoch":71,"log_start":"104'66","ondisk_log_start":"104'66","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'32","last_scrub_stamp":"2024-09-16T03:37:35.189364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.189364+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":64,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:01.317395+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5058973,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":91088,"num_write":34,"num_write_kb":12339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2311899,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.15","version":"100'59","reported_seq":336,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.458644+0000","last_change":"2024-09-16T03:37:47.078728+0000","last_active":"2024-09-16T03:38:41.458644+0000","last_peered":"2024-09-16T03:38:41.458644+0000","last_clean":"2024-09-16T03:38:41.458644+0000","last_became_active":"2024-09-16T03:37:31.349745+0000","last_became_peered":"2024-09-16T03:37:31.349745+0000","last_unstale":"2024-09-16T03:38:41.458644+0000","last_undegraded":"2024-09-16T03:38:41.458644+0000","last_fullsized":"2024-09-16T03:38:41.458644+0000","mapping_epoch":50,"log_start":"98'56","ondisk_log_start":"98'56","created":50,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":54,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:15.293798+0000","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4924517,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":146,"num_read_kb":141080,"num_write":31,"num_write_kb":9048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.14","version":"104'20","reported_seq":240,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.397509+0000","last_change":"2024-09-16T03:37:39.953719+0000","last_active":"2024-09-16T03:38:41.397509+0000","last_peered":"2024-09-16T03:38:41.397509+0000","last_clean":"2024-09-16T03:38:41.397509+0000","last_became_active":"2024-09-16T03:37:31.149621+0000","last_became_peered":"2024-09-16T03:37:31.149621+0000","last_unstale":"2024-09-16T03:38:41.397509+0000","last_undegraded":"2024-09-16T03:38:41.397509+0000","last_fullsized":"2024-09-16T03:38:41.397509+0000","mapping_epoch":50,"log_start":"98'17","ondisk_log_start":"98'17","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":5,"last_scrub":"57'13","last_scrub_stamp":"2024-09-16T03:37:39.953656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.953656+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:59.243798+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2705960,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":10129,"num_write":9,"num_write_kb":2276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.13","version":"108'68","reported_seq":347,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.397555+0000","last_change":"2024-09-16T03:37:57.653492+0000","last_active":"2024-09-16T03:38:41.397555+0000","last_peered":"2024-09-16T03:38:41.397555+0000","last_clean":"2024-09-16T03:38:41.397555+0000","last_became_active":"2024-09-16T03:37:57.640119+0000","last_became_peered":"2024-09-16T03:37:57.640119+0000","last_unstale":"2024-09-16T03:38:41.397555+0000","last_undegraded":"2024-09-16T03:38:41.397555+0000","last_fullsized":"2024-09-16T03:38:41.397555+0000","mapping_epoch":71,"log_start":"104'65","ondisk_log_start":"104'65","created":50,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'21","last_scrub_stamp":"2024-09-16T03:37:34.916681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.916681+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":63,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:54.228271+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5344692,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":132,"num_read_kb":111918,"num_write":69,"num_write_kb":16771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4939757,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.12","version":"106'17","reported_seq":266,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.364619+0000","last_change":"2024-09-16T03:38:36.456574+0000","last_active":"2024-09-16T03:38:38.364619+0000","last_peered":"2024-09-16T03:38:38.364619+0000","last_clean":"2024-09-16T03:38:38.364619+0000","last_became_active":"2024-09-16T03:38:36.456181+0000","last_became_peered":"2024-09-16T03:38:36.456181+0000","last_unstale":"2024-09-16T03:38:38.364619+0000","last_undegraded":"2024-09-16T03:38:38.364619+0000","last_fullsized":"2024-09-16T03:38:38.364619+0000","mapping_epoch":103,"log_start":"82'13","ondisk_log_start":"82'13","created":50,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:03.304067+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067294,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":50,"num_read_kb":41481,"num_write":12,"num_write_kb":3255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2662118,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.11","version":"72'12","reported_seq":220,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T03:38:40.565287+0000","last_change":"2024-09-16T03:37:50.756196+0000","last_active":"2024-09-16T03:38:40.565287+0000","last_peered":"2024-09-16T03:38:40.565287+0000","last_clean":"2024-09-16T03:38:40.565287+0000","last_became_active":"2024-09-16T03:37:49.624845+0000","last_became_peered":"2024-09-16T03:37:49.624845+0000","last_unstale":"2024-09-16T03:38:40.565287+0000","last_undegraded":"2024-09-16T03:38:40.565287+0000","last_fullsized":"2024-09-16T03:38:40.565287+0000","mapping_epoch":63,"log_start":"62'9","ondisk_log_start":"62'9","created":50,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'9","last_scrub_stamp":"2024-09-16T03:37:50.756120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:50.756120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":9,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:11.554148+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":62,"num_read_kb":56090,"num_write":4,"num_write_kb":1177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.10","version":"106'59","reported_seq":318,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.394039+0000","last_change":"2024-09-16T03:37:58.667607+0000","last_active":"2024-09-16T03:38:38.394039+0000","last_peered":"2024-09-16T03:38:38.394039+0000","last_clean":"2024-09-16T03:38:38.394039+0000","last_became_active":"2024-09-16T03:37:58.567094+0000","last_became_peered":"2024-09-16T03:37:58.567094+0000","last_unstale":"2024-09-16T03:38:38.394039+0000","last_undegraded":"2024-09-16T03:38:38.394039+0000","last_fullsized":"2024-09-16T03:38:38.394039+0000","mapping_epoch":72,"log_start":"100'56","ondisk_log_start":"100'56","created":50,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:46.505713+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":41471,"num_write":14,"num_write_kb":3115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1836181,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.690697+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:38:38.690697+0000","last_peered":"2024-09-16T03:38:38.690697+0000","last_clean":"2024-09-16T03:38:38.690697+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:38:38.690697+0000","last_undegraded":"2024-09-16T03:38:38.690697+0000","last_fullsized":"2024-09-16T03:38:38.690697+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.232558+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":200,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.358996+0000","last_change":"2024-09-16T03:38:34.928371+0000","last_active":"2024-09-16T03:38:38.358996+0000","last_peered":"2024-09-16T03:38:38.358996+0000","last_clean":"2024-09-16T03:38:38.358996+0000","last_became_active":"2024-09-16T03:38:33.484091+0000","last_became_peered":"2024-09-16T03:38:33.484091+0000","last_unstale":"2024-09-16T03:38:38.358996+0000","last_undegraded":"2024-09-16T03:38:38.358996+0000","last_fullsized":"2024-09-16T03:38:38.358996+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:34.928298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.928298+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:59.189853+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":191,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.364488+0000","last_change":"2024-09-16T03:38:12.041048+0000","last_active":"2024-09-16T03:38:38.364488+0000","last_peered":"2024-09-16T03:38:38.364488+0000","last_clean":"2024-09-16T03:38:38.364488+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:38:38.364488+0000","last_undegraded":"2024-09-16T03:38:38.364488+0000","last_fullsized":"2024-09-16T03:38:38.364488+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:12.040917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.040917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.877949+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.0","version":"106'118","reported_seq":469,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:39.002145+0000","last_change":"2024-09-16T03:38:34.422457+0000","last_active":"2024-09-16T03:38:39.002145+0000","last_peered":"2024-09-16T03:38:39.002145+0000","last_clean":"2024-09-16T03:38:39.002145+0000","last_became_active":"2024-09-16T03:38:33.581141+0000","last_became_peered":"2024-09-16T03:38:33.581141+0000","last_unstale":"2024-09-16T03:38:39.002145+0000","last_undegraded":"2024-09-16T03:38:39.002145+0000","last_fullsized":"2024-09-16T03:38:39.002145+0000","mapping_epoch":32,"log_start":"106'115","ondisk_log_start":"106'115","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'52","last_scrub_stamp":"2024-09-16T03:37:39.774014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:39.774014+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":112,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":647,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6949760,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":107554,"num_write":40,"num_write_kb":11784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.357943+0000","last_change":"2024-09-16T03:38:33.480743+0000","last_active":"2024-09-16T03:38:38.357943+0000","last_peered":"2024-09-16T03:38:38.357943+0000","last_clean":"2024-09-16T03:38:38.357943+0000","last_became_active":"2024-09-16T03:38:33.480354+0000","last_became_peered":"2024-09-16T03:38:33.480354+0000","last_unstale":"2024-09-16T03:38:38.357943+0000","last_undegraded":"2024-09-16T03:38:38.357943+0000","last_fullsized":"2024-09-16T03:38:38.357943+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.291947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.3","version":"94'34","reported_seq":288,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.397620+0000","last_change":"2024-09-16T03:38:39.081942+0000","last_active":"2024-09-16T03:38:41.397620+0000","last_peered":"2024-09-16T03:38:41.397620+0000","last_clean":"2024-09-16T03:38:41.397620+0000","last_became_active":"2024-09-16T03:38:33.580022+0000","last_became_peered":"2024-09-16T03:38:33.580022+0000","last_unstale":"2024-09-16T03:38:41.397620+0000","last_undegraded":"2024-09-16T03:38:41.397620+0000","last_fullsized":"2024-09-16T03:38:41.397620+0000","mapping_epoch":96,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:39.081810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.081810+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:43.224921+0000","scrub_duration":473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":186,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T03:38:40.565429+0000","last_change":"2024-09-16T03:37:52.652365+0000","last_active":"2024-09-16T03:38:40.565429+0000","last_peered":"2024-09-16T03:38:40.565429+0000","last_clean":"2024-09-16T03:38:40.565429+0000","last_became_active":"2024-09-16T03:37:52.651907+0000","last_became_peered":"2024-09-16T03:37:52.651907+0000","last_unstale":"2024-09-16T03:38:40.565429+0000","last_undegraded":"2024-09-16T03:38:40.565429+0000","last_fullsized":"2024-09-16T03:38:40.565429+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:07.210820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":217,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.358498+0000","last_change":"2024-09-16T03:38:35.828277+0000","last_active":"2024-09-16T03:38:41.358498+0000","last_peered":"2024-09-16T03:38:41.358498+0000","last_clean":"2024-09-16T03:38:41.358498+0000","last_became_active":"2024-09-16T03:38:33.778361+0000","last_became_peered":"2024-09-16T03:38:33.778361+0000","last_unstale":"2024-09-16T03:38:41.358498+0000","last_undegraded":"2024-09-16T03:38:41.358498+0000","last_fullsized":"2024-09-16T03:38:41.358498+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:35.828203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.828203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:56.485832+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.3","version":"108'222","reported_seq":399,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.512415+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:38:41.512415+0000","last_peered":"2024-09-16T03:38:41.512415+0000","last_clean":"2024-09-16T03:38:41.512415+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:38:41.512415+0000","last_undegraded":"2024-09-16T03:38:41.512415+0000","last_fullsized":"2024-09-16T03:38:41.512415+0000","mapping_epoch":31,"log_start":"106'219","ondisk_log_start":"106'219","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":231,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.679811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20403386,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":199,"num_read_kb":3959,"num_write":281,"num_write_kb":63286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":185,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.676665+0000","last_change":"2024-09-16T03:38:12.795978+0000","last_active":"2024-09-16T03:38:38.676665+0000","last_peered":"2024-09-16T03:38:38.676665+0000","last_clean":"2024-09-16T03:38:38.676665+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:38:38.676665+0000","last_undegraded":"2024-09-16T03:38:38.676665+0000","last_fullsized":"2024-09-16T03:38:38.676665+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:38:12.795855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.795855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:13.088740+0000","scrub_duration":501,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.4","version":"105'32","reported_seq":263,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.397601+0000","last_change":"2024-09-16T03:37:46.884372+0000","last_active":"2024-09-16T03:38:41.397601+0000","last_peered":"2024-09-16T03:38:41.397601+0000","last_clean":"2024-09-16T03:38:41.397601+0000","last_became_active":"2024-09-16T03:37:31.249346+0000","last_became_peered":"2024-09-16T03:37:31.249346+0000","last_unstale":"2024-09-16T03:38:41.397601+0000","last_undegraded":"2024-09-16T03:38:41.397601+0000","last_fullsized":"2024-09-16T03:38:41.397601+0000","mapping_epoch":32,"log_start":"83'29","ondisk_log_start":"83'29","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'15","last_scrub_stamp":"2024-09-16T03:37:46.884284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:46.884284+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:53.163594+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":9455,"num_write":19,"num_write_kb":4676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.0","version":"106'208","reported_seq":443,"reported_epoch":106,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:38:39.403242+0000","last_change":"2024-09-16T03:38:34.787655+0000","last_active":"2024-09-16T03:38:39.403242+0000","last_peered":"2024-09-16T03:38:39.403242+0000","last_clean":"2024-09-16T03:37:50.666338+0000","last_became_active":"2024-09-16T03:38:34.586943+0000","last_became_peered":"2024-09-16T03:38:34.586943+0000","last_unstale":"2024-09-16T03:38:39.403242+0000","last_undegraded":"2024-09-16T03:38:39.403242+0000","last_fullsized":"2024-09-16T03:38:39.403242+0000","mapping_epoch":101,"log_start":"105'205","ondisk_log_start":"105'205","created":31,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":226,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20585078,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":188,"num_read_kb":11039,"num_write":257,"num_write_kb":72281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":14158129,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,1],"avail_no_missing":["9","1","4"],"object_location_counts":[{"shards":"1,4,9","objects":22}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":185,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.458530+0000","last_change":"2024-09-16T03:38:15.949627+0000","last_active":"2024-09-16T03:38:41.458530+0000","last_peered":"2024-09-16T03:38:41.458530+0000","last_clean":"2024-09-16T03:38:41.458530+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:38:41.458530+0000","last_undegraded":"2024-09-16T03:38:41.458530+0000","last_fullsized":"2024-09-16T03:38:41.458530+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:15.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:24.286324+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.7","version":"105'64","reported_seq":312,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.394411+0000","last_change":"2024-09-16T03:37:48.400988+0000","last_active":"2024-09-16T03:38:38.394411+0000","last_peered":"2024-09-16T03:38:38.394411+0000","last_clean":"2024-09-16T03:38:38.394411+0000","last_became_active":"2024-09-16T03:37:31.153814+0000","last_became_peered":"2024-09-16T03:37:31.153814+0000","last_unstale":"2024-09-16T03:38:38.394411+0000","last_undegraded":"2024-09-16T03:38:38.394411+0000","last_fullsized":"2024-09-16T03:38:38.394411+0000","mapping_epoch":32,"log_start":"96'61","ondisk_log_start":"96'61","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'32","last_scrub_stamp":"2024-09-16T03:37:48.400911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:48.400911+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:51.787641+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5025359,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":106189,"num_write":40,"num_write_kb":11286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1","version":"107'174","reported_seq":349,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-16T03:38:40.745955+0000","last_change":"2024-09-16T03:38:15.890094+0000","last_active":"2024-09-16T03:38:40.745955+0000","last_peered":"2024-09-16T03:38:40.745955+0000","last_clean":"2024-09-16T03:38:40.745955+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:38:40.745955+0000","last_undegraded":"2024-09-16T03:38:40.745955+0000","last_fullsized":"2024-09-16T03:38:40.745955+0000","mapping_epoch":31,"log_start":"106'171","ondisk_log_start":"106'171","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'125","last_scrub_stamp":"2024-09-16T03:38:15.889996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.889996+0000","objects_scrubbed":18,"log_size":3,"log_dups_size":182,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:35.350472+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10993346,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":152,"num_read_kb":759,"num_write":221,"num_write_kb":48685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":196,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.690601+0000","last_change":"2024-09-16T03:38:33.580846+0000","last_active":"2024-09-16T03:38:38.690601+0000","last_peered":"2024-09-16T03:38:38.690601+0000","last_clean":"2024-09-16T03:38:38.690601+0000","last_became_active":"2024-09-16T03:38:33.580519+0000","last_became_peered":"2024-09-16T03:38:33.580519+0000","last_unstale":"2024-09-16T03:38:38.690601+0000","last_undegraded":"2024-09-16T03:38:38.690601+0000","last_fullsized":"2024-09-16T03:38:38.690601+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:24.358339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:24.358339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:53.576553+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.6","version":"105'43","reported_seq":308,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.696524+0000","last_change":"2024-09-16T03:37:56.444659+0000","last_active":"2024-09-16T03:38:38.696524+0000","last_peered":"2024-09-16T03:38:38.696524+0000","last_clean":"2024-09-16T03:38:38.696524+0000","last_became_active":"2024-09-16T03:37:56.444134+0000","last_became_peered":"2024-09-16T03:37:56.444134+0000","last_unstale":"2024-09-16T03:38:38.696524+0000","last_undegraded":"2024-09-16T03:38:38.696524+0000","last_fullsized":"2024-09-16T03:38:38.696524+0000","mapping_epoch":70,"log_start":"96'40","ondisk_log_start":"96'40","created":32,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'32","last_scrub_stamp":"2024-09-16T03:37:47.613282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.613282+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":39,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:05.636213+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2840416,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":43,"num_read_kb":33829,"num_write":13,"num_write_kb":3682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1264675,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.2","version":"107'200","reported_seq":444,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.397601+0000","last_change":"2024-09-16T03:38:01.603352+0000","last_active":"2024-09-16T03:38:41.397601+0000","last_peered":"2024-09-16T03:38:41.397601+0000","last_clean":"2024-09-16T03:38:41.397601+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:38:41.397601+0000","last_undegraded":"2024-09-16T03:38:41.397601+0000","last_fullsized":"2024-09-16T03:38:41.397601+0000","mapping_epoch":74,"log_start":"107'197","ondisk_log_start":"107'197","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":206,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.902901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18195813,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":187,"num_read_kb":7668,"num_write":250,"num_write_kb":59917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":70,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.496475+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:38:38.496475+0000","last_peered":"2024-09-16T03:38:38.496475+0000","last_clean":"2024-09-16T03:38:38.496475+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:38:38.496475+0000","last_undegraded":"2024-09-16T03:38:38.496475+0000","last_fullsized":"2024-09-16T03:38:38.496475+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:01.672864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":370,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.397441+0000","last_change":"2024-09-16T03:38:36.757156+0000","last_active":"2024-09-16T03:38:41.397441+0000","last_peered":"2024-09-16T03:38:41.397441+0000","last_clean":"2024-09-16T03:38:41.397441+0000","last_became_active":"2024-09-16T03:38:36.756855+0000","last_became_peered":"2024-09-16T03:38:36.756855+0000","last_unstale":"2024-09-16T03:38:41.397441+0000","last_undegraded":"2024-09-16T03:38:41.397441+0000","last_fullsized":"2024-09-16T03:38:41.397441+0000","mapping_epoch":103,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:38:28.063690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:38:28.063690+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:28.682469+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.5","version":"106'76","reported_seq":340,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.358593+0000","last_change":"2024-09-16T03:37:47.718455+0000","last_active":"2024-09-16T03:38:41.358593+0000","last_peered":"2024-09-16T03:38:41.358593+0000","last_clean":"2024-09-16T03:38:41.358593+0000","last_became_active":"2024-09-16T03:37:31.248101+0000","last_became_peered":"2024-09-16T03:37:31.248101+0000","last_unstale":"2024-09-16T03:38:41.358593+0000","last_undegraded":"2024-09-16T03:38:41.358593+0000","last_fullsized":"2024-09-16T03:38:41.358593+0000","mapping_epoch":32,"log_start":"98'73","ondisk_log_start":"98'73","created":32,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'44","last_scrub_stamp":"2024-09-16T03:37:47.718363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.718363+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":72,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:08.856188+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2437048,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":110,"num_read_kb":91786,"num_write":37,"num_write_kb":9464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.8","version":"105'63","reported_seq":339,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.396645+0000","last_change":"2024-09-16T03:38:36.653385+0000","last_active":"2024-09-16T03:38:38.396645+0000","last_peered":"2024-09-16T03:38:38.396645+0000","last_clean":"2024-09-16T03:38:38.396645+0000","last_became_active":"2024-09-16T03:38:36.652849+0000","last_became_peered":"2024-09-16T03:38:36.652849+0000","last_unstale":"2024-09-16T03:38:38.396645+0000","last_undegraded":"2024-09-16T03:38:38.396645+0000","last_fullsized":"2024-09-16T03:38:38.396645+0000","mapping_epoch":103,"log_start":"90'60","ondisk_log_start":"90'60","created":32,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"56'38","last_scrub_stamp":"2024-09-16T03:37:40.343801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:40.343801+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:09.940930+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3899224,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":58565,"num_write":28,"num_write_kb":8269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3979076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":176,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.464602+0000","last_change":"2024-09-16T03:37:52.558364+0000","last_active":"2024-09-16T03:38:38.464602+0000","last_peered":"2024-09-16T03:38:38.464602+0000","last_clean":"2024-09-16T03:38:38.464602+0000","last_became_active":"2024-09-16T03:37:52.558058+0000","last_became_peered":"2024-09-16T03:37:52.558058+0000","last_unstale":"2024-09-16T03:38:38.464602+0000","last_undegraded":"2024-09-16T03:38:38.464602+0000","last_fullsized":"2024-09-16T03:38:38.464602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:46.812106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:46.812106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:14.026640+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.a","version":"99'112","reported_seq":504,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:39.259985+0000","last_change":"2024-09-16T03:38:39.259985+0000","last_active":"2024-09-16T03:38:39.259985+0000","last_peered":"2024-09-16T03:38:39.259985+0000","last_clean":"2024-09-16T03:38:39.259985+0000","last_became_active":"2024-09-16T03:38:33.582063+0000","last_became_peered":"2024-09-16T03:38:33.582063+0000","last_unstale":"2024-09-16T03:38:39.259985+0000","last_undegraded":"2024-09-16T03:38:39.259985+0000","last_fullsized":"2024-09-16T03:38:39.259985+0000","mapping_epoch":32,"log_start":"95'109","ondisk_log_start":"95'109","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'112","last_scrub_stamp":"2024-09-16T03:38:39.259882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.259882+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.072309+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10706092,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.b","version":"107'111","reported_seq":483,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-16T03:38:41.512864+0000","last_change":"2024-09-16T03:38:38.759024+0000","last_active":"2024-09-16T03:38:41.512864+0000","last_peered":"2024-09-16T03:38:41.512864+0000","last_clean":"2024-09-16T03:38:41.512864+0000","last_became_active":"2024-09-16T03:38:37.588212+0000","last_became_peered":"2024-09-16T03:38:37.588212+0000","last_unstale":"2024-09-16T03:38:41.512864+0000","last_undegraded":"2024-09-16T03:38:41.512864+0000","last_fullsized":"2024-09-16T03:38:41.512864+0000","mapping_epoch":104,"log_start":"106'107","ondisk_log_start":"106'107","created":32,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'104","last_scrub_stamp":"2024-09-16T03:38:38.758962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:38.758962+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":107,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:58.549474+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5916130,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":12248,"num_write":9,"num_write_kb":2611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.c","version":"101'57","reported_seq":307,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.456340+0000","last_change":"2024-09-16T03:38:37.251799+0000","last_active":"2024-09-16T03:38:38.456340+0000","last_peered":"2024-09-16T03:38:38.456340+0000","last_clean":"2024-09-16T03:38:38.456340+0000","last_became_active":"2024-09-16T03:38:36.654714+0000","last_became_peered":"2024-09-16T03:38:36.654714+0000","last_unstale":"2024-09-16T03:38:38.456340+0000","last_undegraded":"2024-09-16T03:38:38.456340+0000","last_fullsized":"2024-09-16T03:38:38.456340+0000","mapping_epoch":103,"log_start":"90'54","ondisk_log_start":"90'54","created":32,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'57","last_scrub_stamp":"2024-09-16T03:38:37.251676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:37.251676+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":54,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:41.675429+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3865610,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.d","version":"92'14","reported_seq":260,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:39.133471+0000","last_change":"2024-09-16T03:38:39.133471+0000","last_active":"2024-09-16T03:38:39.133471+0000","last_peered":"2024-09-16T03:38:39.133471+0000","last_clean":"2024-09-16T03:38:39.133471+0000","last_became_active":"2024-09-16T03:38:33.677674+0000","last_became_peered":"2024-09-16T03:38:33.677674+0000","last_unstale":"2024-09-16T03:38:39.133471+0000","last_undegraded":"2024-09-16T03:38:39.133471+0000","last_fullsized":"2024-09-16T03:38:39.133471+0000","mapping_epoch":72,"log_start":"91'11","ondisk_log_start":"91'11","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:39.133386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.133386+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:56.858692+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.e","version":"106'67","reported_seq":341,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:39.696922+0000","last_change":"2024-09-16T03:38:39.194330+0000","last_active":"2024-09-16T03:38:39.696922+0000","last_peered":"2024-09-16T03:38:39.696922+0000","last_clean":"2024-09-16T03:38:39.696922+0000","last_became_active":"2024-09-16T03:38:33.581495+0000","last_became_peered":"2024-09-16T03:38:33.581495+0000","last_unstale":"2024-09-16T03:38:39.696922+0000","last_undegraded":"2024-09-16T03:38:39.696922+0000","last_fullsized":"2024-09-16T03:38:39.696922+0000","mapping_epoch":32,"log_start":"106'64","ondisk_log_start":"106'64","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:39.194237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.194237+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":64,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:06.017886+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8739706,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"99'89","reported_seq":384,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T03:38:38.394226+0000","last_change":"2024-09-16T03:38:35.350916+0000","last_active":"2024-09-16T03:38:38.394226+0000","last_peered":"2024-09-16T03:38:38.394226+0000","last_clean":"2024-09-16T03:38:38.394226+0000","last_became_active":"2024-09-16T03:38:33.581172+0000","last_became_peered":"2024-09-16T03:38:33.581172+0000","last_unstale":"2024-09-16T03:38:38.394226+0000","last_undegraded":"2024-09-16T03:38:38.394226+0000","last_fullsized":"2024-09-16T03:38:38.394226+0000","mapping_epoch":32,"log_start":"98'86","ondisk_log_start":"98'86","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'89","last_scrub_stamp":"2024-09-16T03:38:35.350835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.350835+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":86,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:51.480338+0000","scrub_duration":319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353013,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":36,"stat_sum":{"num_bytes":102603449,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":1214,"num_read_kb":1060895,"num_write":395,"num_write_kb":108596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":16993706,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":14,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":193859584,"data_stored":193435412,"data_compressed":537471,"data_compressed_allocated":577536,"data_compressed_original":1167360,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":108,"acting":108,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":70177623,"num_objects":103,"num_object_clones":0,"num_object_copies":309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":726,"num_read_kb":23425,"num_write":1009,"num_write_kb":244169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":26556632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":146399232,"data_stored":156197044,"data_compressed":8935803,"data_compressed_allocated":9531392,"data_compressed_original":19808256,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":2644640,"data_compressed":159066,"data_compressed_allocated":204800,"data_compressed_original":2584576,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75036,"kb_used_data":42732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94296804,"statfs":{"total":96636764160,"available":96559927296,"internally_reserved":0,"allocated":43757568,"data_stored":44135995,"data_compressed":1044582,"data_compressed_allocated":1105920,"data_compressed_original":1978368,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59592,"kb_used_data":28400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94312248,"statfs":{"total":96636764160,"available":96575741952,"internally_reserved":0,"allocated":29081600,"data_stored":29989031,"data_compressed":632450,"data_compressed_allocated":692224,"data_compressed_original":2019328,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247832,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54312,"kb_used_data":23124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94317528,"statfs":{"total":96636764160,"available":96581148672,"internally_reserved":0,"allocated":23678976,"data_stored":23928035,"data_compressed":443378,"data_compressed_allocated":483328,"data_compressed_original":1105920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280537,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42872,"kb_used_data":11684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328968,"statfs":{"total":96636764160,"available":96592863232,"internally_reserved":0,"allocated":11964416,"data_stored":12842036,"data_compressed":563498,"data_compressed_allocated":618496,"data_compressed_original":1822720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280537,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280536,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313241,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":5104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":5226496,"data_stored":4934759,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280537,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66420,"kb_used_data":31068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94305420,"statfs":{"total":96636764160,"available":96568750080,"internally_reserved":0,"allocated":31813632,"data_stored":32698568,"data_compressed":1054435,"data_compressed_allocated":1134592,"data_compressed_original":2416640,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,5,7,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280536,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95960,"kb_used_data":59056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94275880,"statfs":{"total":96636764160,"available":96538501120,"internally_reserved":0,"allocated":60473344,"data_stored":62099589,"data_compressed":1569071,"data_compressed_allocated":1691648,"data_compressed_original":3837952,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313241,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280536,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37664,"kb_used_data":6472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334176,"statfs":{"total":96636764160,"available":96598196224,"internally_reserved":0,"allocated":6627328,"data_stored":6348320,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,5,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280536,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65024,"kb_used_data":30904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94306816,"statfs":{"total":96636764160,"available":96570179584,"internally_reserved":0,"allocated":31645696,"data_stored":32174307,"data_compressed":1074576,"data_compressed_allocated":1138688,"data_compressed_original":2060288,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280536,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55124,"kb_used_data":28096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94316716,"statfs":{"total":96636764160,"available":96580317184,"internally_reserved":0,"allocated":28770304,"data_stored":29618209,"data_compressed":647457,"data_compressed_allocated":696320,"data_compressed_original":1925120,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280537,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68916,"kb_used_data":32124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302924,"statfs":{"total":96636764160,"available":96566194176,"internally_reserved":0,"allocated":32894976,"data_stored":32683052,"data_compressed":187677,"data_compressed_allocated":204800,"data_compressed_original":413696,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,3,4,5,6,7,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313240,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41580,"kb_used_data":4872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330260,"statfs":{"total":96636764160,"available":96594186240,"internally_reserved":0,"allocated":4988928,"data_stored":4681362,"data_compressed":8520,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45808,"kb_used_data":14620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326032,"statfs":{"total":96636764160,"available":96589856768,"internally_reserved":0,"allocated":14970880,"data_stored":15468533,"data_compressed":987041,"data_compressed_allocated":1044480,"data_compressed_original":1867776,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8880128,"data_stored":9567942,"data_compressed":523730,"data_compressed_allocated":565248,"data_compressed_original":1286144,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13422592,"data_stored":14113910,"data_compressed":606276,"data_compressed_allocated":643072,"data_compressed_original":1376256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10682368,"data_stored":11471446,"data_compressed":1017070,"data_compressed_allocated":1073152,"data_compressed_original":1888256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20615168,"data_stored":21886587,"data_compressed":979594,"data_compressed_allocated":1052672,"data_compressed_original":2396160,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":29761536,"data_stored":31718102,"data_compressed":1503324,"data_compressed_allocated":1617920,"data_compressed_original":3682304,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6209536,"data_stored":6733948,"data_compressed":362228,"data_compressed_allocated":393216,"data_compressed_original":950272,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10440704,"data_stored":11227056,"data_compressed":978521,"data_compressed_allocated":1032192,"data_compressed_original":1843200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":13422592,"data_stored":14113910,"data_compressed":606276,"data_compressed_allocated":643072,"data_compressed_original":1376256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":10641408,"data_stored":11360562,"data_compressed":553749,"data_compressed_allocated":593920,"data_compressed_original":1351680,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":10817536,"data_stored":11605078,"data_compressed":1017070,"data_compressed_allocated":1073152,"data_compressed_original":1888256,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2002944,"data_stored":1989538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3641344,"data_stored":3619083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11603968,"data_stored":11637959,"data_compressed":87535,"data_compressed_allocated":94208,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":29855744,"data_stored":29767564,"data_compressed":57227,"data_compressed_allocated":61440,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16584704,"data_stored":16571081,"data_compressed":72630,"data_compressed_allocated":77824,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5775360,"data_stored":5735414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20332544,"data_stored":20332594,"data_compressed":87535,"data_compressed_allocated":94208,"data_compressed_original":192512,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14430208,"data_stored":14361955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":30556160,"data_stored":30598664,"data_compressed":179157,"data_compressed_allocated":192512,"data_compressed_original":389120,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17379328,"data_stored":17324967,"data_compressed":34395,"data_compressed_allocated":36864,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4087808,"data_stored":4063988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4345856,"data_stored":4319415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":32174080,"data_stored":32033150,"data_compressed":18992,"data_compressed_allocated":20480,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:38:43.693 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-16T03:38:43.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 56 2024-09-16T03:38:43.721 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:43.719+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:43.737 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:43.737 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:43.746 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:38:43.821 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:43.820+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:43.830 INFO:tasks.rados.rados.0.smithi157.stdout:1479: set_chunk oid 25 offset: 2092032 length: 355328 target oid 8fe19cde73751101a7d6e4029e7ad044607da7a914f3bfaee0751a9eadfcb17f offset: 0 snap -1 2024-09-16T03:38:43.830 INFO:tasks.rados.rados.0.smithi157.stdout:1480: read oid 47 snap -1 2024-09-16T03:38:43.831 INFO:tasks.rados.rados.0.smithi157.stdout:1480: expect (ObjNum 301 snap 0 seq_num 301) 2024-09-16T03:38:43.831 INFO:tasks.rados.rados.0.smithi157.stdout:1481: tier_evict oid 30 2024-09-16T03:38:43.831 INFO:tasks.rados.rados.0.smithi157.stdout:1481: tier_evict oid 30 snap -1 2024-09-16T03:38:43.831 INFO:tasks.rados.rados.0.smithi157.stdout:1469: finishing write tid 5 to smithi15736696-40 2024-09-16T03:38:43.831 INFO:tasks.rados.rados.0.smithi157.stdout:1469: oid 40 updating version 0 to 123 2024-09-16T03:38:43.831 INFO:tasks.rados.rados.0.smithi157.stdout:1469: oid 40 updating version 123 to 125 2024-09-16T03:38:43.831 INFO:tasks.rados.rados.0.smithi157.stdout:1469: oid 40 version 125 is already newer than 124 2024-09-16T03:38:43.831 INFO:tasks.rados.rados.0.smithi157.stdout:1469: oid 40 updating version 125 to 126 2024-09-16T03:38:43.831 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 126 (ObjNum 302 snap 0 seq_num 302) dirty exists 2024-09-16T03:38:43.831 INFO:tasks.rados.rados.0.smithi157.stdout:1469: left oid 40 (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:38:43.831 INFO:tasks.rados.rados.0.smithi157.stdout:1470: finishing write tid 1 to smithi15736696-34 2024-09-16T03:38:43.831 INFO:tasks.rados.rados.0.smithi157.stdout:1470: finishing write tid 2 to smithi15736696-34 2024-09-16T03:38:43.831 INFO:tasks.rados.rados.0.smithi157.stdout:1470: finishing write tid 3 to smithi15736696-34 2024-09-16T03:38:43.832 INFO:tasks.rados.rados.0.smithi157.stdout:1470: finishing write tid 4 to smithi15736696-34 2024-09-16T03:38:43.832 INFO:tasks.rados.rados.0.smithi157.stdout:1470: oid 34 updating version 0 to 116 2024-09-16T03:38:43.832 INFO:tasks.rados.rados.0.smithi157.stdout:1470: oid 34 updating version 116 to 117 2024-09-16T03:38:43.832 INFO:tasks.rados.rados.0.smithi157.stdout:1470: oid 34 updating version 117 to 118 2024-09-16T03:38:43.832 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 118 (ObjNum 303 snap 0 seq_num 303) dirty exists 2024-09-16T03:38:43.832 INFO:tasks.rados.rados.0.smithi157.stdout:1470: left oid 34 (ObjNum 303 snap 0 seq_num 303) 2024-09-16T03:38:43.832 INFO:tasks.rados.rados.0.smithi157.stdout:1473: finishing write tid 1 to smithi15736696-19 2024-09-16T03:38:43.832 INFO:tasks.rados.rados.0.smithi157.stdout:1473: finishing write tid 2 to smithi15736696-19 2024-09-16T03:38:43.832 INFO:tasks.rados.rados.0.smithi157.stdout:1469: done (10 left) 2024-09-16T03:38:43.832 INFO:tasks.rados.rados.0.smithi157.stdout:1470: done (9 left) 2024-09-16T03:38:43.832 INFO:tasks.rados.rados.0.smithi157.stdout:1482: read oid 2 snap -1 2024-09-16T03:38:43.832 INFO:tasks.rados.rados.0.smithi157.stdout:1482: expect deleted 2024-09-16T03:38:43.832 INFO:tasks.rados.rados.0.smithi157.stdout:1483: read oid 34 snap -1 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:1483: expect (ObjNum 303 snap 0 seq_num 303) 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:1484: delete oid 11 current snap is 0 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:1478: got (22) Invalid argument 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:1473: finishing write tid 3 to smithi15736696-19 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:1473: finishing write tid 4 to smithi15736696-19 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:1473: finishing write tid 5 to smithi15736696-19 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:1473: oid 19 updating version 0 to 121 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:1473: oid 19 version 121 is already newer than 120 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:1473: oid 19 version 121 is already newer than 119 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:1473: oid 19 updating version 121 to 122 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 122 (ObjNum 305 snap 0 seq_num 305) dirty exists 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:1473: left oid 19 (ObjNum 305 snap 0 seq_num 305) 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:1477: finishing write tid 1 to smithi15736696-35 2024-09-16T03:38:43.833 INFO:tasks.rados.rados.0.smithi157.stdout:1477: finishing write tid 2 to smithi15736696-35 2024-09-16T03:38:43.834 INFO:tasks.rados.rados.0.smithi157.stdout:1474: finishing write tid 1 to smithi15736696-36 2024-09-16T03:38:43.834 INFO:tasks.rados.rados.0.smithi157.stdout:1476: expect (ObjNum 244 snap 0 seq_num 244) 2024-09-16T03:38:43.850 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:43.851 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:43.859 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:38:43.887 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:43.916 INFO:tasks.rados.rados.0.smithi157.stdout:1474: finishing write tid 2 to smithi15736696-36 2024-09-16T03:38:43.916 INFO:tasks.rados.rados.0.smithi157.stdout:1475: finishing set_chunk to oid 49 2024-09-16T03:38:43.916 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 61 (ObjNum 254 snap 0 seq_num 254) dirty exists 2024-09-16T03:38:43.916 INFO:tasks.rados.rados.0.smithi157.stdout:1477: finishing write tid 3 to smithi15736696-35 2024-09-16T03:38:43.916 INFO:tasks.rados.rados.0.smithi157.stdout:1474: finishing write tid 3 to smithi15736696-36 2024-09-16T03:38:43.916 INFO:tasks.rados.rados.0.smithi157.stdout:1474: finishing write tid 4 to smithi15736696-36 2024-09-16T03:38:43.916 INFO:tasks.rados.rados.0.smithi157.stdout:1474: oid 36 updating version 0 to 120 2024-09-16T03:38:43.916 INFO:tasks.rados.rados.0.smithi157.stdout:1474: oid 36 version 120 is already newer than 118 2024-09-16T03:38:43.916 INFO:tasks.rados.rados.0.smithi157.stdout:1474: oid 36 version 120 is already newer than 119 2024-09-16T03:38:43.916 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 120 (ObjNum 306 snap 0 seq_num 306) dirty exists 2024-09-16T03:38:43.916 INFO:tasks.rados.rados.0.smithi157.stdout:1474: left oid 36 (ObjNum 306 snap 0 seq_num 306) 2024-09-16T03:38:43.916 INFO:tasks.rados.rados.0.smithi157.stdout:1477: finishing write tid 4 to smithi15736696-35 2024-09-16T03:38:43.917 INFO:tasks.rados.rados.0.smithi157.stdout:1477: finishing write tid 5 to smithi15736696-35 2024-09-16T03:38:43.917 INFO:tasks.rados.rados.0.smithi157.stdout:1477: oid 35 updating version 0 to 116 2024-09-16T03:38:43.917 INFO:tasks.rados.rados.0.smithi157.stdout:1477: oid 35 updating version 116 to 119 2024-09-16T03:38:43.917 INFO:tasks.rados.rados.0.smithi157.stdout:1477: oid 35 version 119 is already newer than 117 2024-09-16T03:38:43.917 INFO:tasks.rados.rados.0.smithi157.stdout:1477: oid 35 version 119 is already newer than 118 2024-09-16T03:38:43.917 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 119 (ObjNum 307 snap 0 seq_num 307) dirty exists 2024-09-16T03:38:43.917 INFO:tasks.rados.rados.0.smithi157.stdout:1477: left oid 35 (ObjNum 307 snap 0 seq_num 307) 2024-09-16T03:38:43.922 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:43.921+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:43.930 INFO:tasks.rados.rados.0.smithi157.stdout:1481: got (2) No such file or directory 2024-09-16T03:38:43.930 INFO:tasks.rados.rados.0.smithi157.stdout:1481: got expected ENOENT (src dne) 2024-09-16T03:38:43.932 INFO:tasks.rados.rados.0.smithi157.stdout:1473: done (11 left) 2024-09-16T03:38:43.932 INFO:tasks.rados.rados.0.smithi157.stdout:1474: done (10 left) 2024-09-16T03:38:43.932 INFO:tasks.rados.rados.0.smithi157.stdout:1475: done (9 left) 2024-09-16T03:38:43.932 INFO:tasks.rados.rados.0.smithi157.stdout:1476: done (8 left) 2024-09-16T03:38:43.932 INFO:tasks.rados.rados.0.smithi157.stdout:1477: done (7 left) 2024-09-16T03:38:43.932 INFO:tasks.rados.rados.0.smithi157.stdout:1478: done (6 left) 2024-09-16T03:38:43.932 INFO:tasks.rados.rados.0.smithi157.stdout:1481: done (5 left) 2024-09-16T03:38:43.932 INFO:tasks.rados.rados.0.smithi157.stdout:1482: done (4 left) 2024-09-16T03:38:43.933 INFO:tasks.rados.rados.0.smithi157.stdout:1484: done (3 left) 2024-09-16T03:38:43.933 INFO:tasks.rados.rados.0.smithi157.stdout:1485: read oid 46 snap -1 2024-09-16T03:38:43.933 INFO:tasks.rados.rados.0.smithi157.stdout:1485: expect (ObjNum 236 snap 0 seq_num 236) 2024-09-16T03:38:43.933 INFO:tasks.rados.rados.0.smithi157.stdout:1486: tier_promote oid 12 2024-09-16T03:38:43.933 INFO:tasks.rados.rados.0.smithi157.stdout:1487: read oid 41 snap -1 2024-09-16T03:38:43.933 INFO:tasks.rados.rados.0.smithi157.stdout:1487: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-16T03:38:43.933 INFO:tasks.rados.rados.0.smithi157.stdout:1488: tier_promote oid 43 2024-09-16T03:38:43.933 INFO:tasks.rados.rados.0.smithi157.stdout:1489: read oid 27 snap -1 2024-09-16T03:38:43.936 INFO:tasks.rados.rados.0.smithi157.stdout:1489: expect (ObjNum 244 snap 0 seq_num 244) 2024-09-16T03:38:43.936 INFO:tasks.rados.rados.0.smithi157.stdout:1490: read oid 50 snap -1 2024-09-16T03:38:43.936 INFO:tasks.rados.rados.0.smithi157.stdout:1490: expect (ObjNum 277 snap 0 seq_num 277) 2024-09-16T03:38:43.936 INFO:tasks.rados.rados.0.smithi157.stdout:1491: write oid 28 current snap is 0 2024-09-16T03:38:43.937 INFO:tasks.rados.rados.0.smithi157.stdout:1491: seq_num 309 ranges {793363=706061,2282230=796095} 2024-09-16T03:38:43.941 INFO:tasks.rados.rados.0.smithi157.stdout:1491: writing smithi15736696-28 from 793363 to 1499424 tid 1 2024-09-16T03:38:43.951 INFO:tasks.rados.rados.0.smithi157.stdout:1491: writing smithi15736696-28 from 2282230 to 3078325 tid 2 2024-09-16T03:38:43.951 INFO:tasks.rados.rados.0.smithi157.stdout:1492: tier_evict oid 21 2024-09-16T03:38:43.951 INFO:tasks.rados.rados.0.smithi157.stdout:1492: tier_evict oid 21 snap -1 2024-09-16T03:38:43.951 INFO:tasks.rados.rados.0.smithi157.stdout:1486: got (0) Success 2024-09-16T03:38:43.951 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 15 (ObjNum 297 snap 0 seq_num 297) dirty exists 2024-09-16T03:38:43.951 INFO:tasks.rados.rados.0.smithi157.stdout:1480: expect (ObjNum 301 snap 0 seq_num 301) 2024-09-16T03:38:43.961 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 918010 2024-09-16T03:38:43.971 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:43.972 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:43.981 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:38:44.018 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:44.024 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:44.022+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:44.125 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:44.123+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:44.126 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:44.126 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:44.135 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:38:44.165 INFO:tasks.rados.rados.0.smithi157.stdout:1483: expect (ObjNum 303 snap 0 seq_num 303) 2024-09-16T03:38:44.225 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:44.225 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:44.223+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:44.238 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:44.238 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:44.247 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:38:44.326 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:44.324+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:44.344 INFO:tasks.rados.rados.0.smithi157.stdout:1487: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-16T03:38:44.344 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:44.391 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:44.392 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:44.400 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:38:44.413 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1850956 2024-09-16T03:38:44.425 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 4 pg_num to 56 2024-09-16T03:38:44.427 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:44.425+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:44.456 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1863662 2024-09-16T03:38:44.474 INFO:tasks.rados.rados.0.smithi157.stdout:1485: expect (ObjNum 236 snap 0 seq_num 236) 2024-09-16T03:38:44.509 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:44.509 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:44.517 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:38:44.528 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:44.526+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:44.538 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3646560 2024-09-16T03:38:44.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:38:44.609 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3311094 2024-09-16T03:38:44.621 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:44.629 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:44.627+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:44.645 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:44.645 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:44.653 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:38:44.712 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:44.794 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:44.795 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:44.795 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:44.795 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:44.796 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:44.796 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:44.796 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:44.796 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:44.797 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:44.797 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:44.797 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:44.798 INFO:tasks.rados.rados.0.smithi157.stdout:1488: got (0) Success 2024-09-16T03:38:44.798 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 75 (ObjNum 264 snap 0 seq_num 264) dirty exists 2024-09-16T03:38:44.798 INFO:tasks.rados.rados.0.smithi157.stdout:1490: expect (ObjNum 277 snap 0 seq_num 277) 2024-09-16T03:38:44.799 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:44.799 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:44.799 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:38:44.800 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:44.801 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:44.797+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:44.812 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:44.812 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:44.828 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:44.829 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:44.844 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:44.844 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:44.861 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:44.861 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:44.878 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:44.878 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:44.894 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:44.894 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:44.899 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:44.898+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:44.905 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:38:44.905 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:38:44.905 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:44.908 INFO:tasks.rados.rados.0.smithi157.stdout:1491: finishing write tid 1 to smithi15736696-28 2024-09-16T03:38:44.908 INFO:tasks.rados.rados.0.smithi157.stdout:1492: got (2) No such file or directory 2024-09-16T03:38:44.908 INFO:tasks.rados.rados.0.smithi157.stdout:1492: got expected ENOENT (src dne) 2024-09-16T03:38:44.908 INFO:tasks.rados.rados.0.smithi157.stdout:1489: expect (ObjNum 244 snap 0 seq_num 244) 2024-09-16T03:38:44.910 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:44.910 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:44.914 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:38:44.927 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:44.927 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:44.945 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:44.945 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:44.955 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:44.961 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:44.961 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:44.984 INFO:tasks.rados.rados.0.smithi157.stdout:1491: finishing write tid 2 to smithi15736696-28 2024-09-16T03:38:44.984 INFO:tasks.rados.rados.0.smithi157.stdout:1491: finishing write tid 3 to smithi15736696-28 2024-09-16T03:38:44.984 INFO:tasks.rados.rados.0.smithi157.stdout:1491: finishing write tid 4 to smithi15736696-28 2024-09-16T03:38:44.984 INFO:tasks.rados.rados.0.smithi157.stdout:1491: oid 28 updating version 0 to 109 2024-09-16T03:38:44.984 INFO:tasks.rados.rados.0.smithi157.stdout:1491: oid 28 updating version 109 to 110 2024-09-16T03:38:44.984 INFO:tasks.rados.rados.0.smithi157.stdout:1491: oid 28 updating version 110 to 111 2024-09-16T03:38:44.984 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 111 (ObjNum 309 snap 0 seq_num 309) dirty exists 2024-09-16T03:38:44.984 INFO:tasks.rados.rados.0.smithi157.stdout:1491: left oid 28 (ObjNum 309 snap 0 seq_num 309) 2024-09-16T03:38:44.984 INFO:tasks.rados.rados.0.smithi157.stdout:1479: finishing set_chunk to oid 25 2024-09-16T03:38:44.985 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 122 (ObjNum 295 snap 0 seq_num 295) dirty exists 2024-09-16T03:38:44.985 INFO:tasks.rados.rados.0.smithi157.stdout:1479: done (10 left) 2024-09-16T03:38:44.985 INFO:tasks.rados.rados.0.smithi157.stdout:1480: done (9 left) 2024-09-16T03:38:44.985 INFO:tasks.rados.rados.0.smithi157.stdout:1483: done (8 left) 2024-09-16T03:38:44.985 INFO:tasks.rados.rados.0.smithi157.stdout:1485: done (7 left) 2024-09-16T03:38:44.985 INFO:tasks.rados.rados.0.smithi157.stdout:1486: done (6 left) 2024-09-16T03:38:44.985 INFO:tasks.rados.rados.0.smithi157.stdout:1487: done (5 left) 2024-09-16T03:38:44.985 INFO:tasks.rados.rados.0.smithi157.stdout:1488: done (4 left) 2024-09-16T03:38:44.985 INFO:tasks.rados.rados.0.smithi157.stdout:1489: done (3 left) 2024-09-16T03:38:44.985 INFO:tasks.rados.rados.0.smithi157.stdout:1490: done (2 left) 2024-09-16T03:38:44.985 INFO:tasks.rados.rados.0.smithi157.stdout:1491: done (1 left) 2024-09-16T03:38:44.985 INFO:tasks.rados.rados.0.smithi157.stdout:1492: done (0 left) 2024-09-16T03:38:44.985 INFO:tasks.rados.rados.0.smithi157.stdout:1493: delete oid 46 current snap is 0 2024-09-16T03:38:45.001 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:44.999+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:45.021 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:45.021 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:38:45.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:38:45.021 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:45.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.021 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T03:38:45.021 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:38:45.021 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.941569988, 2024-09-16T03:38:45.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.88203620400000005, 2024-09-16T03:38:45.022 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.022 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.022 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.022 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.022 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.022 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.022 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:38:45.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.022 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:38:45.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:38:45.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076767+0000", 2024-09-16T03:38:45.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9300000000000002e-06 2024-09-16T03:38:45.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076774+0000", 2024-09-16T03:38:45.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6630000000000004e-06 2024-09-16T03:38:45.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076785+0000", 2024-09-16T03:38:45.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1221e-05 2024-09-16T03:38:45.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076796+0000", 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0428000000000001e-05 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076856+0000", 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076863+0000", 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9840000000000004e-06 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855770+0000", 2024-09-16T03:38:45.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.77890736999999999 2024-09-16T03:38:45.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855902+0000", 2024-09-16T03:38:45.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013207 2024-09-16T03:38:45.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.856054+0000", 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015220899999999999 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.858407+0000", 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023530069999999998 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958598+0000", 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100190194 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958716+0000", 2024-09-16T03:38:45.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118589 2024-09-16T03:38:45.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958761+0000", 2024-09-16T03:38:45.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5120000000000002e-05 2024-09-16T03:38:45.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958802+0000", 2024-09-16T03:38:45.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0383999999999998e-05 2024-09-16T03:38:45.029 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.029 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.029 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.029 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:38:45.029 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:38:45.029 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 110.42315381100001, 2024-09-16T03:38:45.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70508991300000001, 2024-09-16T03:38:45.030 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.030 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.030 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.030 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.030 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.031 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.031 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 52 2024-09-16T03:38:45.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.032 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:38:45.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:38:45.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:38:45.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6300000000000005e-07 2024-09-16T03:38:45.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595187+0000", 2024-09-16T03:38:45.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9790000000000001e-06 2024-09-16T03:38:45.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595194+0000", 2024-09-16T03:38:45.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6490000000000002e-06 2024-09-16T03:38:45.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595208+0000", 2024-09-16T03:38:45.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4139e-05 2024-09-16T03:38:45.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801931+0000", 2024-09-16T03:38:45.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.206723083 2024-09-16T03:38:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801990+0000", 2024-09-16T03:38:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8625999999999997e-05 2024-09-16T03:38:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802066+0000", 2024-09-16T03:38:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6553999999999999e-05 2024-09-16T03:38:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802504+0000", 2024-09-16T03:38:45.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043775200000000002 2024-09-16T03:38:45.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.199998+0000", 2024-09-16T03:38:45.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39749365599999997 2024-09-16T03:38:45.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300151+0000", 2024-09-16T03:38:45.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015310099999999 2024-09-16T03:38:45.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300210+0000", 2024-09-16T03:38:45.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9376999999999997e-05 2024-09-16T03:38:45.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300271+0000", 2024-09-16T03:38:45.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1334000000000006e-05 2024-09-16T03:38:45.037 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.037 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.037 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.038 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.038 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:38:45.038 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 108.648262828, 2024-09-16T03:38:45.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60623559900000001, 2024-09-16T03:38:45.038 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.038 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.038 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.038 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.038 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.038 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.038 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 71 2024-09-16T03:38:45.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.039 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:38:45.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:38:45.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:38:45.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:38:45.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370077+0000", 2024-09-16T03:38:45.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9940000000000002e-06 2024-09-16T03:38:45.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370081+0000", 2024-09-16T03:38:45.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5850000000000001e-06 2024-09-16T03:38:45.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370087+0000", 2024-09-16T03:38:45.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:38:45.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673522+0000", 2024-09-16T03:38:45.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.303434969 2024-09-16T03:38:45.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673567+0000", 2024-09-16T03:38:45.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5080000000000002e-05 2024-09-16T03:38:45.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673651+0000", 2024-09-16T03:38:45.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3652999999999993e-05 2024-09-16T03:38:45.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775115+0000", 2024-09-16T03:38:45.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101464178 2024-09-16T03:38:45.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975918+0000", 2024-09-16T03:38:45.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20080318699999999 2024-09-16T03:38:45.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976254+0000", 2024-09-16T03:38:45.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033619200000000002 2024-09-16T03:38:45.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976285+0000", 2024-09-16T03:38:45.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0679000000000002e-05 2024-09-16T03:38:45.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976308+0000", 2024-09-16T03:38:45.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3051e-05 2024-09-16T03:38:45.044 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.044 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.045 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.045 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.045 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:38:45.045 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 108.648239468, 2024-09-16T03:38:45.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60632925100000001, 2024-09-16T03:38:45.045 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.045 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.045 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.045 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 72 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:38:45.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370150+0000", 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3906999999999997e-05 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370157+0000", 2024-09-16T03:38:45.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:38:45.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370162+0000", 2024-09-16T03:38:45.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7249999999999997e-06 2024-09-16T03:38:45.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673969+0000", 2024-09-16T03:38:45.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30380713199999998 2024-09-16T03:38:45.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674030+0000", 2024-09-16T03:38:45.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1141000000000003e-05 2024-09-16T03:38:45.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674124+0000", 2024-09-16T03:38:45.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3859999999999999e-05 2024-09-16T03:38:45.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775687+0000", 2024-09-16T03:38:45.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10156293299999999 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976103+0000", 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20041631400000001 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976355+0000", 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000251611 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976385+0000", 2024-09-16T03:38:45.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0883999999999998e-05 2024-09-16T03:38:45.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976425+0000", 2024-09-16T03:38:45.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9666000000000003e-05 2024-09-16T03:38:45.051 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.051 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.051 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.051 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.051 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:38:45.052 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 108.648166984, 2024-09-16T03:38:45.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60763126999999995, 2024-09-16T03:38:45.052 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.052 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.052 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.052 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.052 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.052 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.052 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 73 2024-09-16T03:38:45.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370169+0000", 2024-09-16T03:38:45.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6199999999999998e-07 2024-09-16T03:38:45.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370171+0000", 2024-09-16T03:38:45.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:38:45.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370175+0000", 2024-09-16T03:38:45.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.596e-06 2024-09-16T03:38:45.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470295+0000", 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100119967 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775148+0000", 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30485311799999998 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775203+0000", 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4592999999999997e-05 2024-09-16T03:38:45.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775293+0000", 2024-09-16T03:38:45.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0859999999999994e-05 2024-09-16T03:38:45.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976223+0000", 2024-09-16T03:38:45.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20092996599999999 2024-09-16T03:38:45.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976463+0000", 2024-09-16T03:38:45.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000239695 2024-09-16T03:38:45.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977730+0000", 2024-09-16T03:38:45.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012667690000000001 2024-09-16T03:38:45.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977773+0000", 2024-09-16T03:38:45.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2604000000000001e-05 2024-09-16T03:38:45.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977800+0000", 2024-09-16T03:38:45.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7137000000000001e-05 2024-09-16T03:38:45.058 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.058 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.058 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.058 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.058 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:38:45.058 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 108.548028129, 2024-09-16T03:38:45.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60633810099999996, 2024-09-16T03:38:45.058 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 74 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:38:45.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470308+0000", 2024-09-16T03:38:45.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0950000000000001e-06 2024-09-16T03:38:45.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470313+0000", 2024-09-16T03:38:45.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:38:45.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470320+0000", 2024-09-16T03:38:45.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5089999999999999e-06 2024-09-16T03:38:45.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470327+0000", 2024-09-16T03:38:45.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:38:45.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976483+0000", 2024-09-16T03:38:45.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50615579499999996 2024-09-16T03:38:45.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976533+0000", 2024-09-16T03:38:45.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0096000000000003e-05 2024-09-16T03:38:45.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976643+0000", 2024-09-16T03:38:45.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109481 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977808+0000", 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011649869999999999 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.981699+0000", 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003891579 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076496+0000", 2024-09-16T03:38:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.094797333999999997 2024-09-16T03:38:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076583+0000", 2024-09-16T03:38:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6571999999999997e-05 2024-09-16T03:38:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076645+0000", 2024-09-16T03:38:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2331999999999999e-05 2024-09-16T03:38:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.064 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 108.547997994, 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71251765099999997, 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.065 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 75 2024-09-16T03:38:45.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.066 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:38:45.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:38:45.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470338+0000", 2024-09-16T03:38:45.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2500000000000005e-07 2024-09-16T03:38:45.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470389+0000", 2024-09-16T03:38:45.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.134e-05 2024-09-16T03:38:45.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470397+0000", 2024-09-16T03:38:45.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8140000000000001e-06 2024-09-16T03:38:45.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570537+0000", 2024-09-16T03:38:45.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100139425 2024-09-16T03:38:45.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977018+0000", 2024-09-16T03:38:45.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40648102899999999 2024-09-16T03:38:45.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977063+0000", 2024-09-16T03:38:45.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5247000000000002e-05 2024-09-16T03:38:45.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977143+0000", 2024-09-16T03:38:45.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0054000000000003e-05 2024-09-16T03:38:45.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.978132+0000", 2024-09-16T03:38:45.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00098896399999999995 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182725+0000", 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.204592626 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182783+0000", 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8285000000000002e-05 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182825+0000", 2024-09-16T03:38:45.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2398999999999998e-05 2024-09-16T03:38:45.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182855+0000", 2024-09-16T03:38:45.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9842999999999999e-05 2024-09-16T03:38:45.071 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.071 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.071 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.071 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.071 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:38:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 108.44777142700001, 2024-09-16T03:38:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71263271800000005, 2024-09-16T03:38:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 76 2024-09-16T03:38:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:38:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:38:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570566+0000", 2024-09-16T03:38:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:38:45.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570577+0000", 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0672999999999999e-05 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570593+0000", 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5979999999999999e-05 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570604+0000", 2024-09-16T03:38:45.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.077e-05 2024-09-16T03:38:45.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979890+0000", 2024-09-16T03:38:45.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40928624200000002 2024-09-16T03:38:45.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979945+0000", 2024-09-16T03:38:45.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5651e-05 2024-09-16T03:38:45.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980038+0000", 2024-09-16T03:38:45.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2603999999999996e-05 2024-09-16T03:38:45.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980669+0000", 2024-09-16T03:38:45.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063121900000000003 2024-09-16T03:38:45.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283050+0000", 2024-09-16T03:38:45.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30238120800000001 2024-09-16T03:38:45.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283132+0000", 2024-09-16T03:38:45.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1841999999999993e-05 2024-09-16T03:38:45.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283168+0000", 2024-09-16T03:38:45.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5438999999999997e-05 2024-09-16T03:38:45.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283197+0000", 2024-09-16T03:38:45.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8960000000000001e-05 2024-09-16T03:38:45.078 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.078 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.078 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.078 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.078 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:38:45.078 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 106.633236328, 2024-09-16T03:38:45.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70799990400000001, 2024-09-16T03:38:45.078 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.079 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.079 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.079 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.079 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.079 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.079 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 94 2024-09-16T03:38:45.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.079 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:38:45.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:38:45.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385100+0000", 2024-09-16T03:38:45.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9599999999999998e-07 2024-09-16T03:38:45.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385104+0000", 2024-09-16T03:38:45.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7579999999999998e-06 2024-09-16T03:38:45.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385128+0000", 2024-09-16T03:38:45.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4700999999999999e-05 2024-09-16T03:38:45.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385139+0000", 2024-09-16T03:38:45.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0587e-05 2024-09-16T03:38:45.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787499+0000", 2024-09-16T03:38:45.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40235976099999998 2024-09-16T03:38:45.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787545+0000", 2024-09-16T03:38:45.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6418999999999997e-05 2024-09-16T03:38:45.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787623+0000", 2024-09-16T03:38:45.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7897000000000003e-05 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888605+0000", 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100982321 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990721+0000", 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102115922 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093019+0000", 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10229788500000001 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093051+0000", 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2255999999999997e-05 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093099+0000", 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7500999999999998e-05 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.084 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.085 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.085 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:38:45.085 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 106.633185005, 2024-09-16T03:38:45.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80404594699999998, 2024-09-16T03:38:45.085 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.085 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.085 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.085 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.085 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.085 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.085 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 95 2024-09-16T03:38:45.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.086 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:38:45.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:38:45.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385151+0000", 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385165+0000", 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4175999999999999e-05 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385175+0000", 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:38:45.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385183+0000", 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888643+0000", 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50345988200000003 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888695+0000", 2024-09-16T03:38:45.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2834999999999998e-05 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888802+0000", 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010697899999999999 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990658+0000", 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185585699999999 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990917+0000", 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025853600000000002 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189054+0000", 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198136812 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189135+0000", 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1482000000000004e-05 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189196+0000", 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1175999999999998e-05 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 106.633144378, 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80412488400000004, 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.091 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 96 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:38:45.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.093 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.093 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385192+0000", 2024-09-16T03:38:45.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:38:45.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.093 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.093 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385267+0000", 2024-09-16T03:38:45.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5535000000000001e-05 2024-09-16T03:38:45.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385277+0000", 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6239999999999997e-06 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385284+0000", 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4800000000000004e-06 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989283+0000", 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60399849500000002 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989410+0000", 2024-09-16T03:38:45.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127298 2024-09-16T03:38:45.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989615+0000", 2024-09-16T03:38:45.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000205045 2024-09-16T03:38:45.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991036+0000", 2024-09-16T03:38:45.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001421042 2024-09-16T03:38:45.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189228+0000", 2024-09-16T03:38:45.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19819188500000001 2024-09-16T03:38:45.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189249+0000", 2024-09-16T03:38:45.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0996999999999999e-05 2024-09-16T03:38:45.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189293+0000", 2024-09-16T03:38:45.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3912000000000002e-05 2024-09-16T03:38:45.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189316+0000", 2024-09-16T03:38:45.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2739e-05 2024-09-16T03:38:45.097 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.097 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.097 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.098 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.098 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:38:45.098 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 106.633040647, 2024-09-16T03:38:45.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80642264100000005, 2024-09-16T03:38:45.098 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.098 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.098 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.098 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.098 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.098 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.098 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 97 2024-09-16T03:38:45.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.099 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:38:45.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:38:45.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385296+0000", 2024-09-16T03:38:45.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.006e-06 2024-09-16T03:38:45.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385301+0000", 2024-09-16T03:38:45.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2630000000000003e-06 2024-09-16T03:38:45.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385308+0000", 2024-09-16T03:38:45.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6030000000000001e-06 2024-09-16T03:38:45.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485459+0000", 2024-09-16T03:38:45.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150929 2024-09-16T03:38:45.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990238+0000", 2024-09-16T03:38:45.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50477992000000005 2024-09-16T03:38:45.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990287+0000", 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8863999999999998e-05 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990377+0000", 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9900000000000003e-05 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093109+0000", 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102732055 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189351+0000", 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096241766000000006 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191570+0000", 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00221915 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191653+0000", 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3046000000000004e-05 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191717+0000", 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4139000000000004e-05 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 106.53285019, 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90688294999999997, 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.104 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.105 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.105 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.105 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.105 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.105 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 98 2024-09-16T03:38:45.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.105 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:38:45.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485487+0000", 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1799999999999999e-06 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485498+0000", 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0648999999999999e-05 2024-09-16T03:38:45.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485514+0000", 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5943999999999998e-05 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485525+0000", 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.065e-05 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991102+0000", 2024-09-16T03:38:45.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50557748400000002 2024-09-16T03:38:45.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991161+0000", 2024-09-16T03:38:45.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8801999999999997e-05 2024-09-16T03:38:45.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991255+0000", 2024-09-16T03:38:45.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3936000000000003e-05 2024-09-16T03:38:45.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093112+0000", 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185709499999999 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291911+0000", 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19879897699999999 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392167+0000", 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10025634999999999 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392249+0000", 2024-09-16T03:38:45.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2175000000000005e-05 2024-09-16T03:38:45.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392368+0000", 2024-09-16T03:38:45.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118708 2024-09-16T03:38:45.110 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.110 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.110 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.111 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.111 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:38:45.111 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 106.53279656700001, 2024-09-16T03:38:45.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90707203199999997, 2024-09-16T03:38:45.111 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.111 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.111 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.111 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.111 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.112 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.112 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 99 2024-09-16T03:38:45.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.112 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.112 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:38:45.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.112 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:38:45.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485540+0000", 2024-09-16T03:38:45.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-06 2024-09-16T03:38:45.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485661+0000", 2024-09-16T03:38:45.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120835 2024-09-16T03:38:45.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485680+0000", 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8885999999999998e-05 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485693+0000", 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3297e-05 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991546+0000", 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.505853098 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991598+0000", 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1921e-05 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991681+0000", 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2502000000000004e-05 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093115+0000", 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10143424299999999 2024-09-16T03:38:45.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291977+0000", 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198862172 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392541+0000", 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100563758 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392579+0000", 2024-09-16T03:38:45.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7796999999999997e-05 2024-09-16T03:38:45.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392611+0000", 2024-09-16T03:38:45.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2073999999999998e-05 2024-09-16T03:38:45.117 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.117 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.117 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.117 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.117 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:38:45.117 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 106.53261767399999, 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0073149939999999, 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 100 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:38:45.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7400000000000003e-07 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485721+0000", 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4530000000000001e-06 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485726+0000", 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0590000000000002e-06 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485729+0000", 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092342+0000", 2024-09-16T03:38:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60661343700000003 2024-09-16T03:38:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092469+0000", 2024-09-16T03:38:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012658799999999999 2024-09-16T03:38:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092671+0000", 2024-09-16T03:38:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020206 2024-09-16T03:38:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392452+0000", 2024-09-16T03:38:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29978163699999999 2024-09-16T03:38:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392486+0000", 2024-09-16T03:38:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3736999999999999e-05 2024-09-16T03:38:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492812+0000", 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100325621 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492901+0000", 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8702000000000006e-05 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.493033+0000", 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000132185 2024-09-16T03:38:45.123 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:38:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 104.357467153, 2024-09-16T03:38:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60874581500000002, 2024-09-16T03:38:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.124 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 123 2024-09-16T03:38:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:38:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:38:45.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660869+0000", 2024-09-16T03:38:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9400000000000005e-07 2024-09-16T03:38:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660921+0000", 2024-09-16T03:38:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2312000000000003e-05 2024-09-16T03:38:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660927+0000", 2024-09-16T03:38:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9329999999999998e-06 2024-09-16T03:38:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660931+0000", 2024-09-16T03:38:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:38:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.967991+0000", 2024-09-16T03:38:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30706032700000002 2024-09-16T03:38:45.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968039+0000", 2024-09-16T03:38:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7477e-05 2024-09-16T03:38:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968117+0000", 2024-09-16T03:38:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8589000000000003e-05 2024-09-16T03:38:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067678+0000", 2024-09-16T03:38:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099560406000000004 2024-09-16T03:38:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170175+0000", 2024-09-16T03:38:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102496927 2024-09-16T03:38:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269491+0000", 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099316205000000005 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269574+0000", 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2903000000000002e-05 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269614+0000", 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0188000000000002e-05 2024-09-16T03:38:45.130 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:38:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 104.35739845499999, 2024-09-16T03:38:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60879245500000001, 2024-09-16T03:38:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.131 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 124 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660938+0000", 2024-09-16T03:38:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:38:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660940+0000", 2024-09-16T03:38:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2529999999999999e-06 2024-09-16T03:38:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660943+0000", 2024-09-16T03:38:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.01e-06 2024-09-16T03:38:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660946+0000", 2024-09-16T03:38:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8159999999999998e-06 2024-09-16T03:38:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012417+0000", 2024-09-16T03:38:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.35147120599999998 2024-09-16T03:38:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012497+0000", 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9963000000000007e-05 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012585+0000", 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067680+0000", 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055095203000000002 2024-09-16T03:38:45.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170243+0000", 2024-09-16T03:38:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10256290799999999 2024-09-16T03:38:45.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269662+0000", 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099418896000000007 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269701+0000", 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8829999999999999e-05 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269729+0000", 2024-09-16T03:38:45.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8569e-05 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1550 4.13 4:c8492bfc:::smithi15736696-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.351670315, 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62706974999999998, 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.139 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1550 2024-09-16T03:38:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:38:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:38:45.140 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0699999999999998e-07 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666669+0000", 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.393e-06 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.141 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666673+0000", 2024-09-16T03:38:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:38:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666682+0000", 2024-09-16T03:38:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6510000000000007e-06 2024-09-16T03:38:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.913940+0000", 2024-09-16T03:38:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.24725763000000001 2024-09-16T03:38:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.914034+0000", 2024-09-16T03:38:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3923000000000002e-05 2024-09-16T03:38:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293650+0000", 2024-09-16T03:38:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.37961662200000001 2024-09-16T03:38:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293688+0000", 2024-09-16T03:38:45.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8096000000000003e-05 2024-09-16T03:38:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293735+0000", 2024-09-16T03:38:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6451e-05 2024-09-16T03:38:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:77 3.2 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:38:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:38:45.144 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.939793694000002, 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.61452436499999996, 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:38:45.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078543+0000", 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.553e-06 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078551+0000", 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0790000000000001e-06 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.146 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078564+0000", 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2943e-05 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178635+0000", 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10007044599999999 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178684+0000", 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9592999999999998e-05 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178747+0000", 2024-09-16T03:38:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2286000000000006e-05 2024-09-16T03:38:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4,5", 2024-09-16T03:38:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178857+0000", 2024-09-16T03:38:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110807 2024-09-16T03:38:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280364+0000", 2024-09-16T03:38:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101506518 2024-09-16T03:38:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280373+0000", 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6899999999999998e-06 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.369249+0000", 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.088876172000000003 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693044+0000", 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.32379486499999999 2024-09-16T03:38:45.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693066+0000", 2024-09-16T03:38:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2413e-05 2024-09-16T03:38:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3119 4.13 4:c8492bfc:::smithi15736696-42:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:38:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:38:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 8.0868966380000007, 2024-09-16T03:38:45.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.64286237700000004, 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3119 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:38:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931441+0000", 2024-09-16T03:38:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3599999999999999e-06 2024-09-16T03:38:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931449+0000", 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4080000000000001e-06 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931463+0000", 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.488e-05 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931473+0000", 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8109999999999997e-06 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.031604+0000", 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013079499999999 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.031668+0000", 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4226999999999993e-05 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.574301+0000", 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54263289599999998 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.154 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.155 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.155 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:45.156 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:38:45.157 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:45.155+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:45.257 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:45.255+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:45.300 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:45.300 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:45.300 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:38:45.300 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:45.309 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:38:45.357 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:45.356+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:45.411 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:45.412 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:45.412 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:38:45.412 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:38:45.412 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:45.420 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:38:45.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:38:45.458 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:45.456+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:45.533 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:45.533 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:38:45.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:38:45.533 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:45.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.534 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:64 4.5 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:45.534 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:38:45.534 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 93.271058221000004, 2024-09-16T03:38:45.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50484290399999998, 2024-09-16T03:38:45.535 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.535 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.535 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:45.535 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.535 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.535 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.535 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 64 2024-09-16T03:38:45.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.535 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:38:45.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:38:45.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257341+0000", 2024-09-16T03:38:45.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:38:45.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257343+0000", 2024-09-16T03:38:45.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3889999999999999e-06 2024-09-16T03:38:45.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257346+0000", 2024-09-16T03:38:45.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:38:45.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257349+0000", 2024-09-16T03:38:45.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8559999999999998e-06 2024-09-16T03:38:45.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357057+0000", 2024-09-16T03:38:45.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099708121999999996 2024-09-16T03:38:45.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:45.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357156+0000", 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9162999999999999e-05 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762060+0000", 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40490364400000001 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762112+0000", 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1882000000000002e-05 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762183+0000", 2024-09-16T03:38:45.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1297000000000005e-05 2024-09-16T03:38:45.543 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.543 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.543 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.543 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.10.22:22 3.3 3:f84e13a5:::smithi15736696-17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:45.543 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:38:45.543 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 85.218910985999997, 2024-09-16T03:38:45.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50224271099999995, 2024-09-16T03:38:45.543 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.543 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309488+0000", 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309489+0000", 2024-09-16T03:38:45.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4309999999999999e-06 2024-09-16T03:38:45.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309496+0000", 2024-09-16T03:38:45.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5810000000000002e-06 2024-09-16T03:38:45.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309505+0000", 2024-09-16T03:38:45.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6889999999999999e-06 2024-09-16T03:38:45.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.309513+0000", 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3749999999999997e-06 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.406932+0000", 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.097419631000000007 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407009+0000", 2024-09-16T03:38:45.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7220999999999998e-05 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407143+0000", 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013323799999999999 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.407732+0000", 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058938399999999998 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.408133+0000", 2024-09-16T03:38:45.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040090500000000001 2024-09-16T03:38:45.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811599+0000", 2024-09-16T03:38:45.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40346633399999998 2024-09-16T03:38:45.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811682+0000", 2024-09-16T03:38:45.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2463e-05 2024-09-16T03:38:45.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:20.811730+0000", 2024-09-16T03:38:45.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8458999999999998e-05 2024-09-16T03:38:45.549 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.549 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.549 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.549 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:465 4.5 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:45.549 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:38:45.550 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 84.512693447999993, 2024-09-16T03:38:45.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60529649299999999, 2024-09-16T03:38:45.550 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.550 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.550 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.550 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.550 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.550 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.551 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 465 2024-09-16T03:38:45.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.551 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:38:45.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:38:45.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015706+0000", 2024-09-16T03:38:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.091e-06 2024-09-16T03:38:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015712+0000", 2024-09-16T03:38:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-16T03:38:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015724+0000", 2024-09-16T03:38:45.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1408e-05 2024-09-16T03:38:45.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015731+0000", 2024-09-16T03:38:45.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:38:45.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216049+0000", 2024-09-16T03:38:45.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200317895 2024-09-16T03:38:45.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216178+0000", 2024-09-16T03:38:45.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012848800000000001 2024-09-16T03:38:45.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:45.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620010+0000", 2024-09-16T03:38:45.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40383207199999999 2024-09-16T03:38:45.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620710+0000", 2024-09-16T03:38:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069988400000000001 2024-09-16T03:38:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620902+0000", 2024-09-16T03:38:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019197800000000001 2024-09-16T03:38:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620933+0000", 2024-09-16T03:38:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1513999999999997e-05 2024-09-16T03:38:45.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620968+0000", 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4524999999999999e-05 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.621002+0000", 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3955000000000003e-05 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.558 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:10 3.3 3:deac805c:::smithi15736696-44 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:45.558 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:38:45.558 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 79.121198472000003, 2024-09-16T03:38:45.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53191748100000003, 2024-09-16T03:38:45.558 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.558 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.558 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.558 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407202+0000", 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.855e-06 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407208+0000", 2024-09-16T03:38:45.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9819999999999996e-06 2024-09-16T03:38:45.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407220+0000", 2024-09-16T03:38:45.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2191999999999999e-05 2024-09-16T03:38:45.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407236+0000", 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5698000000000001e-05 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538167+0000", 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.13093111499999999 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538229+0000", 2024-09-16T03:38:45.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2441999999999996e-05 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538346+0000", 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011716400000000001 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540757+0000", 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024105089999999999 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.639501+0000", 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098743545000000002 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939038+0000", 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29953765700000001 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939075+0000", 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6600999999999997e-05 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939118+0000", 2024-09-16T03:38:45.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2721e-05 2024-09-16T03:38:45.564 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.564 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.564 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.564 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:984 4.15 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:45.564 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:38:45.564 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 75.156278544000003, 2024-09-16T03:38:45.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.97836159599999994, 2024-09-16T03:38:45.564 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.564 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 984 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372122+0000", 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.934e-06 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372129+0000", 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8029999999999999e-06 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372141+0000", 2024-09-16T03:38:45.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2006e-05 2024-09-16T03:38:45.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472234+0000", 2024-09-16T03:38:45.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100092985 2024-09-16T03:38:45.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472318+0000", 2024-09-16T03:38:45.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4257000000000003e-05 2024-09-16T03:38:45.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:45.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472326+0000", 2024-09-16T03:38:45.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8960000000000003e-06 2024-09-16T03:38:45.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350293+0000", 2024-09-16T03:38:45.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.87796699899999997 2024-09-16T03:38:45.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350433+0000", 2024-09-16T03:38:45.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013999600000000001 2024-09-16T03:38:45.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350482+0000", 2024-09-16T03:38:45.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8720000000000001e-05 2024-09-16T03:38:45.569 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.570 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.570 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.570 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1000 4.15 4:aff4cc41:::smithi15736696-11:head [read 0~2873964,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:45.570 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:38:45.570 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.937839459000003, 2024-09-16T03:38:45.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.761707841, 2024-09-16T03:38:45.570 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.572 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.572 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:45.572 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.572 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.572 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1000 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590560+0000", 2024-09-16T03:38:45.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:38:45.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590562+0000", 2024-09-16T03:38:45.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.17e-06 2024-09-16T03:38:45.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590568+0000", 2024-09-16T03:38:45.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7939999999999997e-06 2024-09-16T03:38:45.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590572+0000", 2024-09-16T03:38:45.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1300000000000001e-06 2024-09-16T03:38:45.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590619+0000", 2024-09-16T03:38:45.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7148999999999999e-05 2024-09-16T03:38:45.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:45.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590625+0000", 2024-09-16T03:38:45.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5679999999999996e-06 2024-09-16T03:38:45.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350858+0000", 2024-09-16T03:38:45.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76023295000000002 2024-09-16T03:38:45.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350973+0000", 2024-09-16T03:38:45.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115023 2024-09-16T03:38:45.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352267+0000", 2024-09-16T03:38:45.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012935970000000001 2024-09-16T03:38:45.579 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.579 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.579 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.580 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1001 4.15 4:aff4cc41:::smithi15736696-11:head [read 1254674~1619290,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:45.580 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:38:45.580 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.937820873999996, 2024-09-16T03:38:45.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76275827799999996, 2024-09-16T03:38:45.580 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.580 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.580 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:45.580 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.581 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.581 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.581 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1001 2024-09-16T03:38:45.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.581 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:38:45.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:38:45.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:38:45.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:38:45.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590580+0000", 2024-09-16T03:38:45.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:38:45.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590584+0000", 2024-09-16T03:38:45.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8800000000000001e-06 2024-09-16T03:38:45.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590587+0000", 2024-09-16T03:38:45.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6630000000000002e-06 2024-09-16T03:38:45.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590650+0000", 2024-09-16T03:38:45.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3950999999999998e-05 2024-09-16T03:38:45.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:45.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590654+0000", 2024-09-16T03:38:45.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.89e-06 2024-09-16T03:38:45.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352309+0000", 2024-09-16T03:38:45.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76165495699999997 2024-09-16T03:38:45.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352407+0000", 2024-09-16T03:38:45.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7188999999999994e-05 2024-09-16T03:38:45.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.353336+0000", 2024-09-16T03:38:45.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092939400000000005 2024-09-16T03:38:45.588 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.588 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.588 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.588 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1002 4.15 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~1527025] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:45.589 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:38:45.589 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.937791619999999, 2024-09-16T03:38:45.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85963523600000002, 2024-09-16T03:38:45.589 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.589 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.589 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:45.589 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.589 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.589 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.590 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1002 2024-09-16T03:38:45.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.590 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:38:45.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:38:45.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:38:45.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.15e-07 2024-09-16T03:38:45.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590609+0000", 2024-09-16T03:38:45.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.606e-06 2024-09-16T03:38:45.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590615+0000", 2024-09-16T03:38:45.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3040000000000001e-06 2024-09-16T03:38:45.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590618+0000", 2024-09-16T03:38:45.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.309e-06 2024-09-16T03:38:45.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590669+0000", 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1048000000000003e-05 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590673+0000", 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.061e-06 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.449932+0000", 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85925979699999999 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450032+0000", 2024-09-16T03:38:45.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9858000000000005e-05 2024-09-16T03:38:45.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450242+0000", 2024-09-16T03:38:45.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020993800000000001 2024-09-16T03:38:45.596 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.596 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.596 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.596 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2467 4.5 4:a11e1d7c:::smithi15736696-43:head [write 1772395~538068] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:45.597 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:38:45.597 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.537115248999999, 2024-09-16T03:38:45.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.56288137400000005, 2024-09-16T03:38:45.597 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.597 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.597 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.597 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.597 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.597 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.597 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2467 2024-09-16T03:38:45.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.597 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991285+0000", 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0990000000000001e-06 2024-09-16T03:38:45.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991447+0000", 2024-09-16T03:38:45.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016170099999999999 2024-09-16T03:38:45.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991458+0000", 2024-09-16T03:38:45.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.128e-05 2024-09-16T03:38:45.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991464+0000", 2024-09-16T03:38:45.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3800000000000002e-06 2024-09-16T03:38:45.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150823+0000", 2024-09-16T03:38:45.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15935943799999999 2024-09-16T03:38:45.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150899+0000", 2024-09-16T03:38:45.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5359999999999997e-05 2024-09-16T03:38:45.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:45.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151007+0000", 2024-09-16T03:38:45.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108317 2024-09-16T03:38:45.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253992+0000", 2024-09-16T03:38:45.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102985249 2024-09-16T03:38:45.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.353797+0000", 2024-09-16T03:38:45.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099805039999999998 2024-09-16T03:38:45.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554039+0000", 2024-09-16T03:38:45.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20024175899999999 2024-09-16T03:38:45.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554129+0000", 2024-09-16T03:38:45.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0067000000000004e-05 2024-09-16T03:38:45.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554165+0000", 2024-09-16T03:38:45.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5683999999999999e-05 2024-09-16T03:38:45.604 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.604 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.605 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.605 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2468 4.5 4:a11e1d7c:::smithi15736696-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:45.605 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:38:45.605 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.531694470000001, 2024-09-16T03:38:45.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.55765803899999999, 2024-09-16T03:38:45.605 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.605 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.605 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.606 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.606 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.606 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.606 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2468 2024-09-16T03:38:45.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.606 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:38:45.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:38:45.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996708+0000", 2024-09-16T03:38:45.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5149999999999998e-06 2024-09-16T03:38:45.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996921+0000", 2024-09-16T03:38:45.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000213692 2024-09-16T03:38:45.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.608 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996937+0000", 2024-09-16T03:38:45.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5699999999999999e-05 2024-09-16T03:38:45.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.608 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996946+0000", 2024-09-16T03:38:45.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7159999999999992e-06 2024-09-16T03:38:45.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152362+0000", 2024-09-16T03:38:45.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15541682800000001 2024-09-16T03:38:45.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152421+0000", 2024-09-16T03:38:45.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-16T03:38:45.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.610 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:45.610 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152515+0000", 2024-09-16T03:38:45.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3659999999999994e-05 2024-09-16T03:38:45.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253995+0000", 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10148022499999999 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.453908+0000", 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19991299700000001 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554321+0000", 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100412977 2024-09-16T03:38:45.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554344+0000", 2024-09-16T03:38:45.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.262e-05 2024-09-16T03:38:45.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554362+0000", 2024-09-16T03:38:45.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8539999999999999e-05 2024-09-16T03:38:45.612 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.612 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.612 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.613 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2469 4.5 4:a11e1d7c:::smithi15736696-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:45.613 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:38:45.613 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.531439107000001, 2024-09-16T03:38:45.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.75785074799999996, 2024-09-16T03:38:45.613 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.613 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.613 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.613 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.613 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.613 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.614 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2469 2024-09-16T03:38:45.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.614 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:38:45.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:38:45.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996960+0000", 2024-09-16T03:38:45.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3000000000000001e-07 2024-09-16T03:38:45.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996963+0000", 2024-09-16T03:38:45.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8679999999999999e-06 2024-09-16T03:38:45.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996967+0000", 2024-09-16T03:38:45.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.622e-06 2024-09-16T03:38:45.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996971+0000", 2024-09-16T03:38:45.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1429999999999998e-06 2024-09-16T03:38:45.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253536+0000", 2024-09-16T03:38:45.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25656582900000002 2024-09-16T03:38:45.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253632+0000", 2024-09-16T03:38:45.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6067e-05 2024-09-16T03:38:45.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:45.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253745+0000", 2024-09-16T03:38:45.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011256699999999999 2024-09-16T03:38:45.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.617 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.617 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254218+0000", 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00047321400000000001 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554397+0000", 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30017854399999999 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754557+0000", 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200159848 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754646+0000", 2024-09-16T03:38:45.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9370999999999995e-05 2024-09-16T03:38:45.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754810+0000", 2024-09-16T03:38:45.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016414500000000001 2024-09-16T03:38:45.619 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.619 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.619 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.619 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2470 4.5 4:a11e1d7c:::smithi15736696-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:45.620 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:38:45.620 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.531422778, 2024-09-16T03:38:45.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76022711799999998, 2024-09-16T03:38:45.620 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.620 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.620 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:45.620 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.620 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.620 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.620 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2470 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:38:45.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.65e-07 2024-09-16T03:38:45.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996977+0000", 2024-09-16T03:38:45.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4079999999999999e-06 2024-09-16T03:38:45.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996981+0000", 2024-09-16T03:38:45.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-16T03:38:45.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996988+0000", 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0890000000000002e-06 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253947+0000", 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25695931100000002 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253985+0000", 2024-09-16T03:38:45.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7939999999999999e-05 2024-09-16T03:38:45.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757090+0000", 2024-09-16T03:38:45.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50310477499999995 2024-09-16T03:38:45.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.624 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757140+0000", 2024-09-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9567999999999997e-05 2024-09-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757203+0000", 2024-09-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3215999999999999e-05 2024-09-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.625 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2487 4.15 4:ad0b9852:::smithi15736696-16:head [write 1947841~685395] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:45.626 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:38:45.626 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.112378188000001, 2024-09-16T03:38:45.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.74140762199999999, 2024-09-16T03:38:45.626 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.626 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.626 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.626 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.626 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.626 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.627 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2487 2024-09-16T03:38:45.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.627 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:38:45.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:38:45.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416023+0000", 2024-09-16T03:38:45.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:38:45.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416341+0000", 2024-09-16T03:38:45.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031881699999999999 2024-09-16T03:38:45.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416353+0000", 2024-09-16T03:38:45.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.189e-05 2024-09-16T03:38:45.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416360+0000", 2024-09-16T03:38:45.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5719999999999999e-06 2024-09-16T03:38:45.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.630 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755578+0000", 2024-09-16T03:38:45.630 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33921859199999999 2024-09-16T03:38:45.630 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.630 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.630 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.630 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755639+0000", 2024-09-16T03:38:45.630 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0615000000000002e-05 2024-09-16T03:38:45.630 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.630 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.630 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:45.631 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755735+0000", 2024-09-16T03:38:45.631 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5798999999999995e-05 2024-09-16T03:38:45.631 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.631 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.631 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.631 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858633+0000", 2024-09-16T03:38:45.631 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10289859599999999 2024-09-16T03:38:45.631 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.631 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.631 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.631 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858695+0000", 2024-09-16T03:38:45.631 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2000000000000003e-05 2024-09-16T03:38:45.631 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157315+0000", 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29861931600000002 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157375+0000", 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0268e-05 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157428+0000", 2024-09-16T03:38:45.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2939999999999998e-05 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2488 4.15 4:ad0b9852:::smithi15736696-16:head [write 3383215~196676] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.106705538, 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73589615900000005, 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.633 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.634 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.634 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2488 2024-09-16T03:38:45.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.634 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:38:45.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:38:45.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421695+0000", 2024-09-16T03:38:45.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1469999999999999e-06 2024-09-16T03:38:45.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421975+0000", 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028008299999999998 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421987+0000", 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1294000000000001e-05 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421992+0000", 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9349999999999999e-06 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757205+0000", 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33521242800000001 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757269+0000", 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3761000000000002e-05 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757376+0000", 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010706599999999999 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.637 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.638 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.638 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858684+0000", 2024-09-16T03:38:45.638 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101308255 2024-09-16T03:38:45.638 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.638 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.638 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.638 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858700+0000", 2024-09-16T03:38:45.638 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6138999999999999e-05 2024-09-16T03:38:45.638 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.638 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.638 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.638 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157491+0000", 2024-09-16T03:38:45.638 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29879056999999998 2024-09-16T03:38:45.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157546+0000", 2024-09-16T03:38:45.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5668999999999998e-05 2024-09-16T03:38:45.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.640 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157589+0000", 2024-09-16T03:38:45.640 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2812000000000003e-05 2024-09-16T03:38:45.640 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.640 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.640 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.640 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.640 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.640 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2489 4.15 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=7b,truncate 3579891] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:45.640 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:38:45.640 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.106396715999999, 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73575981700000004, 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2489 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:38:45.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.642 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.642 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.642 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.642 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:38:45.642 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.642 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.642 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.642 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.642 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:38:45.642 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0200000000000002e-07 2024-09-16T03:38:45.642 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.642 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.642 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.643 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422005+0000", 2024-09-16T03:38:45.643 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7379999999999999e-06 2024-09-16T03:38:45.643 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.643 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.643 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.643 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422009+0000", 2024-09-16T03:38:45.643 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6219999999999998e-06 2024-09-16T03:38:45.643 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.643 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.643 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.643 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422016+0000", 2024-09-16T03:38:45.643 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5789999999999998e-06 2024-09-16T03:38:45.643 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858071+0000", 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.436054302 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858130+0000", 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9493000000000001e-05 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858226+0000", 2024-09-16T03:38:45.644 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6174000000000004e-05 2024-09-16T03:38:45.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.645 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.645 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.645 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:38:45.645 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10106483199999999 2024-09-16T03:38:45.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.645 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.645 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.645 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959639+0000", 2024-09-16T03:38:45.645 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034802999999999999 2024-09-16T03:38:45.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.645 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.645 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157640+0000", 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19800045299999999 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157702+0000", 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2218000000000006e-05 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157762+0000", 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9774000000000001e-05 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.646 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.647 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.647 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.647 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2490 4.15 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:45.647 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:38:45.647 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.106378123999999, 2024-09-16T03:38:45.647 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73597824999999994, 2024-09-16T03:38:45.647 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.647 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.647 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:45.647 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.647 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.647 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.648 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2490 2024-09-16T03:38:45.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.648 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.648 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.648 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:38:45.648 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.648 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.648 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:38:45.648 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.649 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.650 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.650 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422021+0000", 2024-09-16T03:38:45.650 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8099999999999999e-07 2024-09-16T03:38:45.650 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.650 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.650 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.650 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422022+0000", 2024-09-16T03:38:45.650 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.573e-06 2024-09-16T03:38:45.650 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.650 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.650 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422027+0000", 2024-09-16T03:38:45.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3250000000000001e-06 2024-09-16T03:38:45.651 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.651 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422030+0000", 2024-09-16T03:38:45.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.948e-06 2024-09-16T03:38:45.651 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.651 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858510+0000", 2024-09-16T03:38:45.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.43648032599999997 2024-09-16T03:38:45.651 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858588+0000", 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7888999999999999e-05 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157803+0000", 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.299215642 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157893+0000", 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9565000000000001e-05 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157999+0000", 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105701 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2492 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:38:45.653 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.106365615000001, 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53721099400000005, 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2492 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:38:45.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:38:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:38:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:38:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422036+0000", 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5900000000000002e-06 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422043+0000", 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3069999999999997e-06 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422048+0000", 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6619999999999996e-06 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958774+0000", 2024-09-16T03:38:45.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53672593499999999 2024-09-16T03:38:45.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958868+0000", 2024-09-16T03:38:45.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4408999999999997e-05 2024-09-16T03:38:45.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959244+0000", 2024-09-16T03:38:45.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037581099999999998 2024-09-16T03:38:45.658 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.658 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.658 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.658 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2493 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:45.658 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:38:45.658 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.106345835999999, 2024-09-16T03:38:45.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53757571299999996, 2024-09-16T03:38:45.658 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.658 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.658 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:45.659 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.659 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.659 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.659 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2493 2024-09-16T03:38:45.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.659 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.659 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.659 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:38:45.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.659 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3600000000000004e-07 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422055+0000", 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7460000000000001e-06 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422058+0000", 2024-09-16T03:38:45.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:38:45.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422061+0000", 2024-09-16T03:38:45.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6759999999999999e-06 2024-09-16T03:38:45.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:38:45.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53723057900000004 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959367+0000", 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5158000000000001e-05 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959628+0000", 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026175800000000001 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2494 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 1738052~782998] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.106334806, 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53787640199999998, 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2494 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.663 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.664 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:38:45.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.664 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:38:45.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.664 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:38:45.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6100000000000002e-07 2024-09-16T03:38:45.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.665 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422065+0000", 2024-09-16T03:38:45.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2839999999999999e-06 2024-09-16T03:38:45.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.665 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.665 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422069+0000", 2024-09-16T03:38:45.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7239999999999998e-06 2024-09-16T03:38:45.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.665 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422072+0000", 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6560000000000001e-06 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959661+0000", 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53758982600000005 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959739+0000", 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7581000000000001e-05 2024-09-16T03:38:45.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959940+0000", 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020107 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2623 4.5 4:a4fbca73:::smithi15736696-39:head [write 631996~531009] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.248471+0000", 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.279927561000001, 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50345695400000001, 2024-09-16T03:38:45.667 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:45.668 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:45.668 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:45.668 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:45.668 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:45.668 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:45.668 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2623 2024-09-16T03:38:45.668 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.668 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:45.668 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.668 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:45.668 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.248471+0000", 2024-09-16T03:38:45.668 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.668 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.248471+0000", 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.248473+0000", 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.609e-06 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.248587+0000", 2024-09-16T03:38:45.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000114089 2024-09-16T03:38:45.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:45.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.248600+0000", 2024-09-16T03:38:45.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3061000000000001e-05 2024-09-16T03:38:45.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:45.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.348733+0000", 2024-09-16T03:38:45.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100133028 2024-09-16T03:38:45.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.348844+0000", 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011157 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.348913+0000", 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8554999999999997e-05 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349017+0000", 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010454300000000001 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:45.672 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.352857+0000", 2024-09-16T03:38:45.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0038398460000000001 2024-09-16T03:38:45.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.672 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.672 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.454106+0000", 2024-09-16T03:38:45.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10124849399999999 2024-09-16T03:38:45.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.672 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.751825+0000", 2024-09-16T03:38:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29771889000000001 2024-09-16T03:38:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.751882+0000", 2024-09-16T03:38:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7524e-05 2024-09-16T03:38:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.751928+0000", 2024-09-16T03:38:45.673 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5745000000000003e-05 2024-09-16T03:38:45.674 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.674 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.674 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.674 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:45.674 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:45.674 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:45.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:38:45.676 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:45.673+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:45.776 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:45.775+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:45.785 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:45.786 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:38:45.786 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:45.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:38:45.877 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:45.875+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:45.903 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:38:45.903 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:38:45.903 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:45.912 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:38:45.982 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:45.979+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:46.022 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:46.022 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:38:46.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:38:46.022 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:46.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.022 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:39 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'21, mlcod=26'19)", 2024-09-16T03:38:46.022 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:38:46.023 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 111.423797201, 2024-09-16T03:38:46.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30197182500000003, 2024-09-16T03:38:46.023 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.023 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.023 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.023 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:38:46.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:38:46.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595320+0000", 2024-09-16T03:38:46.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.962e-06 2024-09-16T03:38:46.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595336+0000", 2024-09-16T03:38:46.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5568000000000001e-05 2024-09-16T03:38:46.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595349+0000", 2024-09-16T03:38:46.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3203e-05 2024-09-16T03:38:46.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695424+0000", 2024-09-16T03:38:46.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007542899999999 2024-09-16T03:38:46.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695495+0000", 2024-09-16T03:38:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0450999999999993e-05 2024-09-16T03:38:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695538+0000", 2024-09-16T03:38:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3433000000000002e-05 2024-09-16T03:38:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897256+0000", 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20171840699999999 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897290+0000", 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3371999999999998e-05 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:1588 3.2 e61/31 3:57b21e08:::c9c74db4cc3b05b40ab52a9ed9c665f10e27723063858665cdba49de2698a01f:head v 61'105, mlcod=59'103)", 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:46.917498+0000", 2024-09-16T03:38:46.027 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.101616821999997, 2024-09-16T03:38:46.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206346578, 2024-09-16T03:38:46.028 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.028 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.917498+0000", 2024-09-16T03:38:46.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.917498+0000", 2024-09-16T03:38:46.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.917502+0000", 2024-09-16T03:38:46.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0230000000000003e-06 2024-09-16T03:38:46.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.921109+0000", 2024-09-16T03:38:46.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0036070149999999999 2024-09-16T03:38:46.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.921129+0000", 2024-09-16T03:38:46.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9741999999999998e-05 2024-09-16T03:38:46.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:46.921139+0000", 2024-09-16T03:38:46.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5079999999999994e-06 2024-09-16T03:38:46.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.121366+0000", 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20022730599999999 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.121386+0000", 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9856000000000001e-05 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.123808+0000", 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024219979999999999 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:47.123845+0000", 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7129999999999999e-05 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1947 4.8 4:146f84f0:::smithi15736696-6:head [write 2063988~686387] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124065+0000", 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.895049841999999, 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203590252, 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1947 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124065+0000", 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124065+0000", 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124067+0000", 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.66e-06 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124770+0000", 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070269099999999997 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124783+0000", 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2989999999999999e-05 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124791+0000", 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5159999999999994e-06 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124848+0000", 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6787999999999999e-05 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124933+0000", 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4753999999999995e-05 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.125040+0000", 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106827 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127507+0000", 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024672119999999999 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127880+0000", 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000373237 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.131974+0000", 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0040936230000000002 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.226110+0000", 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094136475999999997 2024-09-16T03:38:46.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.327424+0000", 2024-09-16T03:38:46.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101313954 2024-09-16T03:38:46.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.327503+0000", 2024-09-16T03:38:46.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8850000000000006e-05 2024-09-16T03:38:46.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.327656+0000", 2024-09-16T03:38:46.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000152674 2024-09-16T03:38:46.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.040 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.040 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1948 4.8 4:146f84f0:::smithi15736696-6:head [setxattr _header (55) in=7b,truncate 2756348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:46.040 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:38:46.040 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.894308395000003, 2024-09-16T03:38:46.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40326984799999999, 2024-09-16T03:38:46.040 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1948 2024-09-16T03:38:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:38:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:38:46.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124808+0000", 2024-09-16T03:38:46.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9099999999999991e-07 2024-09-16T03:38:46.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124813+0000", 2024-09-16T03:38:46.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9599999999999999e-06 2024-09-16T03:38:46.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124821+0000", 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0490000000000004e-06 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124827+0000", 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3339999999999998e-06 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126739+0000", 2024-09-16T03:38:46.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00191147 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126830+0000", 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1751999999999998e-05 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127005+0000", 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174227 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127981+0000", 2024-09-16T03:38:46.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097622799999999999 2024-09-16T03:38:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.128193+0000", 2024-09-16T03:38:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021176999999999999 2024-09-16T03:38:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.135281+0000", 2024-09-16T03:38:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070882150000000001 2024-09-16T03:38:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.427785+0000", 2024-09-16T03:38:46.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29250390599999998 2024-09-16T03:38:46.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527932+0000", 2024-09-16T03:38:46.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10014759500000001 2024-09-16T03:38:46.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527980+0000", 2024-09-16T03:38:46.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7725000000000001e-05 2024-09-16T03:38:46.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528077+0000", 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6626000000000006e-05 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1949 4.8 4:146f84f0:::smithi15736696-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.894277651000003, 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40346921899999999, 2024-09-16T03:38:46.047 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.048 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.048 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:46.048 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.048 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.048 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.048 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1949 2024-09-16T03:38:46.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.049 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:38:46.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:38:46.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:38:46.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:38:46.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124841+0000", 2024-09-16T03:38:46.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5890000000000001e-06 2024-09-16T03:38:46.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124847+0000", 2024-09-16T03:38:46.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0820000000000004e-06 2024-09-16T03:38:46.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124852+0000", 2024-09-16T03:38:46.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1070000000000004e-06 2024-09-16T03:38:46.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127536+0000", 2024-09-16T03:38:46.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026844619999999999 2024-09-16T03:38:46.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:46.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127585+0000", 2024-09-16T03:38:46.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8915999999999997e-05 2024-09-16T03:38:46.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528126+0000", 2024-09-16T03:38:46.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40054061899999999 2024-09-16T03:38:46.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528193+0000", 2024-09-16T03:38:46.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6717999999999994e-05 2024-09-16T03:38:46.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528307+0000", 2024-09-16T03:38:46.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011413600000000001 2024-09-16T03:38:46.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.059 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.059 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1979 4.18 4:1f63731d:::smithi15736696-30:head [write 789962~534747] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:46.060 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:38:46.060 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.383270338999999, 2024-09-16T03:38:46.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.506754182, 2024-09-16T03:38:46.060 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.060 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.061 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.061 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.061 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.061 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1979 2024-09-16T03:38:46.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.061 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:38:46.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:38:46.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635850+0000", 2024-09-16T03:38:46.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6670000000000004e-06 2024-09-16T03:38:46.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636429+0000", 2024-09-16T03:38:46.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000579745 2024-09-16T03:38:46.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636441+0000", 2024-09-16T03:38:46.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1965e-05 2024-09-16T03:38:46.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736594+0000", 2024-09-16T03:38:46.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100152291 2024-09-16T03:38:46.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837326+0000", 2024-09-16T03:38:46.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10073201399999999 2024-09-16T03:38:46.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837437+0000", 2024-09-16T03:38:46.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111914 2024-09-16T03:38:46.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:46.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837625+0000", 2024-09-16T03:38:46.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000187391 2024-09-16T03:38:46.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939449+0000", 2024-09-16T03:38:46.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10182371599999999 2024-09-16T03:38:46.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939476+0000", 2024-09-16T03:38:46.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7117000000000001e-05 2024-09-16T03:38:46.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142297+0000", 2024-09-16T03:38:46.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20282091499999999 2024-09-16T03:38:46.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142347+0000", 2024-09-16T03:38:46.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0800000000000002e-05 2024-09-16T03:38:46.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142485+0000", 2024-09-16T03:38:46.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013799999999999999 2024-09-16T03:38:46.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142526+0000", 2024-09-16T03:38:46.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0269999999999999e-05 2024-09-16T03:38:46.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142599+0000", 2024-09-16T03:38:46.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3376999999999999e-05 2024-09-16T03:38:46.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.076 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.077 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1980 4.18 4:1f63731d:::smithi15736696-30:head [write 1780952~690056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:46.077 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:38:46.077 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.282469177999999, 2024-09-16T03:38:46.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40615011400000001, 2024-09-16T03:38:46.077 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.077 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.078 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.078 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1980 2024-09-16T03:38:46.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.078 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:38:46.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:38:46.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736656+0000", 2024-09-16T03:38:46.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0339999999999999e-05 2024-09-16T03:38:46.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737092+0000", 2024-09-16T03:38:46.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435957 2024-09-16T03:38:46.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737102+0000", 2024-09-16T03:38:46.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7860000000000008e-06 2024-09-16T03:38:46.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737107+0000", 2024-09-16T03:38:46.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:38:46.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839282+0000", 2024-09-16T03:38:46.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10217523100000001 2024-09-16T03:38:46.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839389+0000", 2024-09-16T03:38:46.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010619399999999999 2024-09-16T03:38:46.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:46.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839567+0000", 2024-09-16T03:38:46.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017886200000000001 2024-09-16T03:38:46.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939485+0000", 2024-09-16T03:38:46.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099917168000000001 2024-09-16T03:38:46.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943655+0000", 2024-09-16T03:38:46.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004170287 2024-09-16T03:38:46.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142373+0000", 2024-09-16T03:38:46.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19871839599999999 2024-09-16T03:38:46.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142431+0000", 2024-09-16T03:38:46.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7880999999999997e-05 2024-09-16T03:38:46.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142627+0000", 2024-09-16T03:38:46.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019580500000000001 2024-09-16T03:38:46.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142723+0000", 2024-09-16T03:38:46.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6209e-05 2024-09-16T03:38:46.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142796+0000", 2024-09-16T03:38:46.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2978000000000005e-05 2024-09-16T03:38:46.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.089 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.089 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.090 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2277 4.18 4:1f63731d:::smithi15736696-30:head [write 460558~707131] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:46.090 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:38:46.090 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.596333477999998, 2024-09-16T03:38:46.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39421539100000003, 2024-09-16T03:38:46.090 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.090 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.090 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.090 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.090 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.090 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.091 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2277 2024-09-16T03:38:46.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.091 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:38:46.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:38:46.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422785+0000", 2024-09-16T03:38:46.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9809999999999999e-06 2024-09-16T03:38:46.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423586+0000", 2024-09-16T03:38:46.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080133600000000004 2024-09-16T03:38:46.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423607+0000", 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1039e-05 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423616+0000", 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0189999999999995e-06 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523776+0000", 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015945399999999 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523851+0000", 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5549999999999993e-05 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523972+0000", 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120683 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527436+0000", 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034638870000000001 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636588+0000", 2024-09-16T03:38:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109152601 2024-09-16T03:38:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636665+0000", 2024-09-16T03:38:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6661000000000003e-05 2024-09-16T03:38:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637128+0000", 2024-09-16T03:38:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046316199999999998 2024-09-16T03:38:46.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816886+0000", 2024-09-16T03:38:46.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17975745200000001 2024-09-16T03:38:46.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816927+0000", 2024-09-16T03:38:46.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1529000000000002e-05 2024-09-16T03:38:46.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816997+0000", 2024-09-16T03:38:46.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0036999999999994e-05 2024-09-16T03:38:46.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.097 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.097 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.097 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2278 4.18 4:1f63731d:::smithi15736696-30:head [write 1869389~508716] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:46.097 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:38:46.097 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.589764477000003, 2024-09-16T03:38:46.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48788153400000001, 2024-09-16T03:38:46.097 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2278 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:38:46.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429352+0000", 2024-09-16T03:38:46.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.604e-06 2024-09-16T03:38:46.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429895+0000", 2024-09-16T03:38:46.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054308899999999996 2024-09-16T03:38:46.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429909+0000", 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3518e-05 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429917+0000", 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525208+0000", 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095291736000000002 2024-09-16T03:38:46.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525303+0000", 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4304999999999999e-05 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525410+0000", 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010755399999999999 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636785+0000", 2024-09-16T03:38:46.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111375037 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637092+0000", 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030685599999999999 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637157+0000", 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4593999999999999e-05 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637236+0000", 2024-09-16T03:38:46.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9563999999999999e-05 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917119+0000", 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.279882931 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917158+0000", 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9096e-05 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917232+0000", 2024-09-16T03:38:46.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4091000000000006e-05 2024-09-16T03:38:46.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.104 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.104 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2279 4.18 4:1f63731d:::smithi15736696-30:head [write 3094884~165674] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:46.104 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:38:46.104 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.585075682000003, 2024-09-16T03:38:46.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483372209, 2024-09-16T03:38:46.104 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.104 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.104 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2279 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434041+0000", 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.195e-06 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434191+0000", 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000150488 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434202+0000", 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0978000000000001e-05 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434208+0000", 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526820+0000", 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092612229000000004 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526894+0000", 2024-09-16T03:38:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3435000000000004e-05 2024-09-16T03:38:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526990+0000", 2024-09-16T03:38:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6112000000000004e-05 2024-09-16T03:38:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637050+0000", 2024-09-16T03:38:46.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110059859 2024-09-16T03:38:46.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637097+0000", 2024-09-16T03:38:46.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6848000000000003e-05 2024-09-16T03:38:46.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637183+0000", 2024-09-16T03:38:46.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6798000000000005e-05 2024-09-16T03:38:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637262+0000", 2024-09-16T03:38:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9049999999999997e-05 2024-09-16T03:38:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917287+0000", 2024-09-16T03:38:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280025036 2024-09-16T03:38:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917326+0000", 2024-09-16T03:38:46.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.871e-05 2024-09-16T03:38:46.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917412+0000", 2024-09-16T03:38:46.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5638999999999998e-05 2024-09-16T03:38:46.111 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.112 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.112 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.112 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2280 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 3260558] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:46.112 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:38:46.112 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.584897400999999, 2024-09-16T03:38:46.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48353995599999999, 2024-09-16T03:38:46.112 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.112 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.112 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.112 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.112 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2280 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:38:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-09-16T03:38:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434221+0000", 2024-09-16T03:38:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8329999999999998e-06 2024-09-16T03:38:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434225+0000", 2024-09-16T03:38:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7589999999999999e-06 2024-09-16T03:38:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434228+0000", 2024-09-16T03:38:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.971e-06 2024-09-16T03:38:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527459+0000", 2024-09-16T03:38:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093231249000000002 2024-09-16T03:38:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527519+0000", 2024-09-16T03:38:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9471e-05 2024-09-16T03:38:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527606+0000", 2024-09-16T03:38:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6776999999999999e-05 2024-09-16T03:38:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637084+0000", 2024-09-16T03:38:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109478228 2024-09-16T03:38:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637100+0000", 2024-09-16T03:38:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6711999999999999e-05 2024-09-16T03:38:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637210+0000", 2024-09-16T03:38:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010909200000000001 2024-09-16T03:38:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637296+0000", 2024-09-16T03:38:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6718000000000006e-05 2024-09-16T03:38:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917637+0000", 2024-09-16T03:38:46.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280340537 2024-09-16T03:38:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917671+0000", 2024-09-16T03:38:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3983999999999999e-05 2024-09-16T03:38:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917758+0000", 2024-09-16T03:38:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7017999999999999e-05 2024-09-16T03:38:46.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2281 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:38:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:38:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.584883105999999, 2024-09-16T03:38:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483761683, 2024-09-16T03:38:46.121 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2281 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:38:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:38:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.96e-07 2024-09-16T03:38:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434234+0000", 2024-09-16T03:38:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4050000000000001e-06 2024-09-16T03:38:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434237+0000", 2024-09-16T03:38:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1939999999999998e-06 2024-09-16T03:38:46.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434240+0000", 2024-09-16T03:38:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.74e-06 2024-09-16T03:38:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530037+0000", 2024-09-16T03:38:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095796830999999999 2024-09-16T03:38:46.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:46.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530089+0000", 2024-09-16T03:38:46.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:38:46.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917823+0000", 2024-09-16T03:38:46.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38773380000000002 2024-09-16T03:38:46.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917891+0000", 2024-09-16T03:38:46.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7676999999999996e-05 2024-09-16T03:38:46.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917994+0000", 2024-09-16T03:38:46.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010286200000000001 2024-09-16T03:38:46.127 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.127 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.127 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.128 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2654 4.8 4:148e627d:::smithi15736696-23:head [write 699224~548358] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:46.128 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:21.373900+0000", 2024-09-16T03:38:46.128 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.645215101000002, 2024-09-16T03:38:46.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20820516, 2024-09-16T03:38:46.128 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.128 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.128 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.128 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.129 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.129 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.129 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2654 2024-09-16T03:38:46.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.129 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.373900+0000", 2024-09-16T03:38:46.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.373900+0000", 2024-09-16T03:38:46.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.373904+0000", 2024-09-16T03:38:46.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3670000000000001e-06 2024-09-16T03:38:46.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.374531+0000", 2024-09-16T03:38:46.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062763499999999998 2024-09-16T03:38:46.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.374550+0000", 2024-09-16T03:38:46.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8612000000000001e-05 2024-09-16T03:38:46.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.374559+0000", 2024-09-16T03:38:46.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8729999999999996e-06 2024-09-16T03:38:46.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.374626+0000", 2024-09-16T03:38:46.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7638999999999994e-05 2024-09-16T03:38:46.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.374707+0000", 2024-09-16T03:38:46.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0620999999999999e-05 2024-09-16T03:38:46.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:46.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.374824+0000", 2024-09-16T03:38:46.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011693600000000001 2024-09-16T03:38:46.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.377249+0000", 2024-09-16T03:38:46.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024252140000000002 2024-09-16T03:38:46.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381000+0000", 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003751037 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.391544+0000", 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010544174999999999 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.576324+0000", 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18477964199999999 2024-09-16T03:38:46.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.581868+0000", 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055443480000000002 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.581949+0000", 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0403999999999998e-05 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.582105+0000", 2024-09-16T03:38:46.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015665699999999999 2024-09-16T03:38:46.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.137 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.137 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2657 4.8 4:148e627d:::smithi15736696-23:head [setxattr _header (56) in=7b,truncate 3899224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:46.137 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:21.388211+0000", 2024-09-16T03:38:46.137 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.630904659999999, 2024-09-16T03:38:46.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.293904257, 2024-09-16T03:38:46.138 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.138 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.138 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.138 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.138 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.138 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.138 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2657 2024-09-16T03:38:46.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.138 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388211+0000", 2024-09-16T03:38:46.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388211+0000", 2024-09-16T03:38:46.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388211+0000", 2024-09-16T03:38:46.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.02e-07 2024-09-16T03:38:46.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388217+0000", 2024-09-16T03:38:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6960000000000002e-06 2024-09-16T03:38:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388223+0000", 2024-09-16T03:38:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3349999999999997e-06 2024-09-16T03:38:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388226+0000", 2024-09-16T03:38:46.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2629999999999999e-06 2024-09-16T03:38:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.391612+0000", 2024-09-16T03:38:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033864239999999999 2024-09-16T03:38:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.391673+0000", 2024-09-16T03:38:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0418999999999999e-05 2024-09-16T03:38:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:46.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.391771+0000", 2024-09-16T03:38:46.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8638e-05 2024-09-16T03:38:46.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.392525+0000", 2024-09-16T03:38:46.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075383199999999996 2024-09-16T03:38:46.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.551525+0000", 2024-09-16T03:38:46.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15900014600000001 2024-09-16T03:38:46.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.577048+0000", 2024-09-16T03:38:46.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025522566 2024-09-16T03:38:46.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.582681+0000", 2024-09-16T03:38:46.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056329830000000003 2024-09-16T03:38:46.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.681849+0000", 2024-09-16T03:38:46.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099168439999999997 2024-09-16T03:38:46.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.681936+0000", 2024-09-16T03:38:46.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6720999999999999e-05 2024-09-16T03:38:46.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.682115+0000", 2024-09-16T03:38:46.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017889199999999999 2024-09-16T03:38:46.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.145 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.145 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2658 4.8 4:148e627d:::smithi15736696-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:38:46.145 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:21.388241+0000", 2024-09-16T03:38:46.145 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.630874249000001, 2024-09-16T03:38:46.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29431808300000001, 2024-09-16T03:38:46.146 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.146 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.146 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:46.146 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.146 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.146 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.146 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2658 2024-09-16T03:38:46.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.146 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388241+0000", 2024-09-16T03:38:46.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388241+0000", 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388242+0000", 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2699999999999999e-07 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388244+0000", 2024-09-16T03:38:46.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2089999999999999e-06 2024-09-16T03:38:46.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388250+0000", 2024-09-16T03:38:46.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.409e-06 2024-09-16T03:38:46.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388254+0000", 2024-09-16T03:38:46.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4489999999999999e-06 2024-09-16T03:38:46.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.392379+0000", 2024-09-16T03:38:46.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004125325 2024-09-16T03:38:46.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:46.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.392451+0000", 2024-09-16T03:38:46.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2152999999999998e-05 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.682233+0000", 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.28978181400000003 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.682337+0000", 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104258 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.682559+0000", 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022193899999999999 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2790 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v39,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.615035650999999, 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42581759000000002, 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.152 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.152 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.152 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2790 2024-09-16T03:38:46.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.152 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:38:46.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:38:46.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404083+0000", 2024-09-16T03:38:46.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2770000000000001e-06 2024-09-16T03:38:46.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404095+0000", 2024-09-16T03:38:46.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1780000000000001e-05 2024-09-16T03:38:46.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404116+0000", 2024-09-16T03:38:46.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0882000000000001e-05 2024-09-16T03:38:46.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404127+0000", 2024-09-16T03:38:46.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1078e-05 2024-09-16T03:38:46.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489837+0000", 2024-09-16T03:38:46.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085709977000000007 2024-09-16T03:38:46.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489956+0000", 2024-09-16T03:38:46.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119856 2024-09-16T03:38:46.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:46.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.592758+0000", 2024-09-16T03:38:46.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10280165400000001 2024-09-16T03:38:46.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.593693+0000", 2024-09-16T03:38:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000935285 2024-09-16T03:38:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.594164+0000", 2024-09-16T03:38:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047097000000000003 2024-09-16T03:38:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.628426+0000", 2024-09-16T03:38:46.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034262070999999998 2024-09-16T03:38:46.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.732623+0000", 2024-09-16T03:38:46.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10419628 2024-09-16T03:38:46.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829726+0000", 2024-09-16T03:38:46.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097103098999999998 2024-09-16T03:38:46.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829790+0000", 2024-09-16T03:38:46.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4677000000000004e-05 2024-09-16T03:38:46.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829897+0000", 2024-09-16T03:38:46.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106704 2024-09-16T03:38:46.159 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.159 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2849 4.18 4:1f63731d:::smithi15736696-30:head [write 1821709~430462] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920227+0000", 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.098887834999999, 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.203899736, 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.160 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2849 2024-09-16T03:38:46.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.161 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920227+0000", 2024-09-16T03:38:46.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920227+0000", 2024-09-16T03:38:46.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920229+0000", 2024-09-16T03:38:46.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5820000000000001e-06 2024-09-16T03:38:46.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920587+0000", 2024-09-16T03:38:46.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00035828299999999998 2024-09-16T03:38:46.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920601+0000", 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3633e-05 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920607+0000", 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7139999999999998e-06 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920651+0000", 2024-09-16T03:38:46.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4409999999999997e-05 2024-09-16T03:38:46.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920720+0000", 2024-09-16T03:38:46.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9109999999999994e-05 2024-09-16T03:38:46.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:46.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920815+0000", 2024-09-16T03:38:46.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4932000000000005e-05 2024-09-16T03:38:46.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.023363+0000", 2024-09-16T03:38:46.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102547948 2024-09-16T03:38:46.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123125+0000", 2024-09-16T03:38:46.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099761876999999999 2024-09-16T03:38:46.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123179+0000", 2024-09-16T03:38:46.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4072e-05 2024-09-16T03:38:46.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123780+0000", 2024-09-16T03:38:46.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060145499999999998 2024-09-16T03:38:46.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123961+0000", 2024-09-16T03:38:46.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000180922 2024-09-16T03:38:46.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.124016+0000", 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4826e-05 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.124127+0000", 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011097199999999999 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2850 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 2252171] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.098495368999998, 2024-09-16T03:38:46.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30323033799999999, 2024-09-16T03:38:46.168 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.168 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.168 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2850 2024-09-16T03:38:46.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.169 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:38:46.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:38:46.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920621+0000", 2024-09-16T03:38:46.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.47e-07 2024-09-16T03:38:46.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920625+0000", 2024-09-16T03:38:46.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1620000000000001e-06 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920632+0000", 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0180000000000004e-06 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920638+0000", 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0950000000000001e-06 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922161+0000", 2024-09-16T03:38:46.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015229880000000001 2024-09-16T03:38:46.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922222+0000", 2024-09-16T03:38:46.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0903999999999998e-05 2024-09-16T03:38:46.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:38:46.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922323+0000", 2024-09-16T03:38:46.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101425 2024-09-16T03:38:46.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.023371+0000", 2024-09-16T03:38:46.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101047493 2024-09-16T03:38:46.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123227+0000", 2024-09-16T03:38:46.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099856057999999998 2024-09-16T03:38:46.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123840+0000", 2024-09-16T03:38:46.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061324999999999997 2024-09-16T03:38:46.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123902+0000", 2024-09-16T03:38:46.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1447999999999998e-05 2024-09-16T03:38:46.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223497+0000", 2024-09-16T03:38:46.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099595499000000004 2024-09-16T03:38:46.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223665+0000", 2024-09-16T03:38:46.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016807399999999999 2024-09-16T03:38:46.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223850+0000", 2024-09-16T03:38:46.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018507699999999999 2024-09-16T03:38:46.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.177 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.177 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.177 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2851 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:38:46.177 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:38:46.178 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.098459273, 2024-09-16T03:38:46.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30354833399999998, 2024-09-16T03:38:46.178 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.178 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:46.179 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.179 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.179 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.179 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2851 2024-09-16T03:38:46.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.179 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:38:46.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:38:46.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920657+0000", 2024-09-16T03:38:46.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0100000000000004e-07 2024-09-16T03:38:46.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920660+0000", 2024-09-16T03:38:46.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6519999999999999e-06 2024-09-16T03:38:46.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920668+0000", 2024-09-16T03:38:46.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1919999999999999e-06 2024-09-16T03:38:46.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920674+0000", 2024-09-16T03:38:46.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-16T03:38:46.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126299+0000", 2024-09-16T03:38:46.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20562534700000001 2024-09-16T03:38:46.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:46.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126358+0000", 2024-09-16T03:38:46.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8459999999999999e-05 2024-09-16T03:38:46.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223913+0000", 2024-09-16T03:38:46.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097555263000000003 2024-09-16T03:38:46.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224009+0000", 2024-09-16T03:38:46.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5723999999999994e-05 2024-09-16T03:38:46.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224204+0000", 2024-09-16T03:38:46.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195498 2024-09-16T03:38:46.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.190 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.191 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2993 4.8 4:148e627d:::smithi15736696-23:head [read 0~3899224,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:38:46.191 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.919321+0000", 2024-09-16T03:38:46.191 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.099794256999999, 2024-09-16T03:38:46.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.205436542, 2024-09-16T03:38:46.191 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.192 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.192 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:46.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.193 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2993 2024-09-16T03:38:46.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.193 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919321+0000", 2024-09-16T03:38:46.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919321+0000", 2024-09-16T03:38:46.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919322+0000", 2024-09-16T03:38:46.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.61e-07 2024-09-16T03:38:46.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919326+0000", 2024-09-16T03:38:46.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.861e-06 2024-09-16T03:38:46.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919333+0000", 2024-09-16T03:38:46.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5549999999999997e-06 2024-09-16T03:38:46.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919339+0000", 2024-09-16T03:38:46.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6869999999999999e-06 2024-09-16T03:38:46.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.121754+0000", 2024-09-16T03:38:46.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20241543000000001 2024-09-16T03:38:46.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.121869+0000", 2024-09-16T03:38:46.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011435800000000001 2024-09-16T03:38:46.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.124758+0000", 2024-09-16T03:38:46.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028889900000000001 2024-09-16T03:38:46.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.201 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.201 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.201 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:46.204 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:38:46.205 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:46.201+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:46.304 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:46.303+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:46.314 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:46.314 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:46.314 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:38:46.314 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:46.322 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:38:46.356 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:38:46.405 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:46.403+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:46.435 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:46.435 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:46.435 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:38:46.435 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:38:46.435 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:46.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:38:46.506 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:46.505+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:46.554 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:46.555 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:38:46.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:38:46.555 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:46.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.555 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:34 4.e 4:773b934b:::smithi15736696-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:46.555 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:38:46.555 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.497308777000001, 2024-09-16T03:38:46.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50346527299999999, 2024-09-16T03:38:46.555 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.555 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.555 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.555 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.555 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.556 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.556 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-16T03:38:46.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.556 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:38:46.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:38:46.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054413+0000", 2024-09-16T03:38:46.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0669999999999999e-06 2024-09-16T03:38:46.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054525+0000", 2024-09-16T03:38:46.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111462 2024-09-16T03:38:46.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054538+0000", 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3341e-05 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054545+0000", 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.584e-06 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054608+0000", 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2953999999999994e-05 2024-09-16T03:38:46.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:46.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054613+0000", 2024-09-16T03:38:46.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6110000000000001e-06 2024-09-16T03:38:46.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154852+0000", 2024-09-16T03:38:46.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10023838 2024-09-16T03:38:46.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154915+0000", 2024-09-16T03:38:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3138000000000004e-05 2024-09-16T03:38:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155001+0000", 2024-09-16T03:38:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5730999999999996e-05 2024-09-16T03:38:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156166+0000", 2024-09-16T03:38:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001165485 2024-09-16T03:38:46.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255909+0000", 2024-09-16T03:38:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099742558999999995 2024-09-16T03:38:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557800+0000", 2024-09-16T03:38:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30189130600000003 2024-09-16T03:38:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557836+0000", 2024-09-16T03:38:46.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6566000000000002e-05 2024-09-16T03:38:46.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557877+0000", 2024-09-16T03:38:46.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0089000000000002e-05 2024-09-16T03:38:46.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.562 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:35 4.e 4:773b934b:::smithi15736696-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:46.562 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:38:46.563 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.497160445999995, 2024-09-16T03:38:46.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50344320499999995, 2024-09-16T03:38:46.563 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.563 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.563 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-16T03:38:46.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.563 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:38:46.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:38:46.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:38:46.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:38:46.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054564+0000", 2024-09-16T03:38:46.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5710000000000002e-06 2024-09-16T03:38:46.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054569+0000", 2024-09-16T03:38:46.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0490000000000003e-06 2024-09-16T03:38:46.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054578+0000", 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054639+0000", 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1061000000000004e-05 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054644+0000", 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999996e-06 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155607+0000", 2024-09-16T03:38:46.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100963372 2024-09-16T03:38:46.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155652+0000", 2024-09-16T03:38:46.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:38:46.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:46.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155722+0000", 2024-09-16T03:38:46.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0130999999999997e-05 2024-09-16T03:38:46.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.157921+0000", 2024-09-16T03:38:46.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021990159999999998 2024-09-16T03:38:46.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255940+0000", 2024-09-16T03:38:46.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098018569999999999 2024-09-16T03:38:46.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557930+0000", 2024-09-16T03:38:46.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.301990333 2024-09-16T03:38:46.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557963+0000", 2024-09-16T03:38:46.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2348000000000003e-05 2024-09-16T03:38:46.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558003+0000", 2024-09-16T03:38:46.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:38:46.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.570 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.570 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.571 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:36 4.e 4:773b934b:::smithi15736696-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:46.571 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:38:46.571 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.497121757000002, 2024-09-16T03:38:46.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50353811800000003, 2024-09-16T03:38:46.571 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.571 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:46.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.571 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.572 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.572 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-16T03:38:46.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.572 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:38:46.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:38:46.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054599+0000", 2024-09-16T03:38:46.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:38:46.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054601+0000", 2024-09-16T03:38:46.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4360000000000001e-06 2024-09-16T03:38:46.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054606+0000", 2024-09-16T03:38:46.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1019999999999996e-06 2024-09-16T03:38:46.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054610+0000", 2024-09-16T03:38:46.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.681e-06 2024-09-16T03:38:46.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054662+0000", 2024-09-16T03:38:46.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1545000000000002e-05 2024-09-16T03:38:46.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:46.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054664+0000", 2024-09-16T03:38:46.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1919999999999999e-06 2024-09-16T03:38:46.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155903+0000", 2024-09-16T03:38:46.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101239068 2024-09-16T03:38:46.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:46.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155970+0000", 2024-09-16T03:38:46.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6934999999999995e-05 2024-09-16T03:38:46.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558029+0000", 2024-09-16T03:38:46.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40205888899999997 2024-09-16T03:38:46.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558078+0000", 2024-09-16T03:38:46.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9842999999999997e-05 2024-09-16T03:38:46.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558136+0000", 2024-09-16T03:38:46.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7985000000000001e-05 2024-09-16T03:38:46.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.578 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:799 4.e 4:765e9179:::smithi15736696-44:head [setxattr _header (56) in=7b,truncate 2857157] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:46.578 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:38:46.578 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.545269529999999, 2024-09-16T03:38:46.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50104159800000003, 2024-09-16T03:38:46.578 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.578 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.578 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.578 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.578 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.579 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.579 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 799 2024-09-16T03:38:46.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.579 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:38:46.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:38:46.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:38:46.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:38:46.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006454+0000", 2024-09-16T03:38:46.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5739999999999998e-06 2024-09-16T03:38:46.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006458+0000", 2024-09-16T03:38:46.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6179999999999997e-06 2024-09-16T03:38:46.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006462+0000", 2024-09-16T03:38:46.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4999999999999999e-06 2024-09-16T03:38:46.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204504+0000", 2024-09-16T03:38:46.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19804184999999999 2024-09-16T03:38:46.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204702+0000", 2024-09-16T03:38:46.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019868199999999999 2024-09-16T03:38:46.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:46.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204888+0000", 2024-09-16T03:38:46.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185701 2024-09-16T03:38:46.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.206549+0000", 2024-09-16T03:38:46.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016612790000000001 2024-09-16T03:38:46.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507371+0000", 2024-09-16T03:38:46.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30082157399999998 2024-09-16T03:38:46.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507414+0000", 2024-09-16T03:38:46.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3053999999999998e-05 2024-09-16T03:38:46.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507451+0000", 2024-09-16T03:38:46.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6847999999999997e-05 2024-09-16T03:38:46.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507492+0000", 2024-09-16T03:38:46.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1267999999999999e-05 2024-09-16T03:38:46.584 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:800 4.e 4:765e9179:::smithi15736696-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.545246195000004, 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50119258099999997, 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.585 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.586 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 800 2024-09-16T03:38:46.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.586 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:38:46.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:38:46.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1899999999999998e-07 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006478+0000", 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006484+0000", 2024-09-16T03:38:46.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6620000000000002e-06 2024-09-16T03:38:46.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006488+0000", 2024-09-16T03:38:46.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8700000000000002e-06 2024-09-16T03:38:46.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205228+0000", 2024-09-16T03:38:46.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19873985899999999 2024-09-16T03:38:46.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:46.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205283+0000", 2024-09-16T03:38:46.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5878999999999998e-05 2024-09-16T03:38:46.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507527+0000", 2024-09-16T03:38:46.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302243924 2024-09-16T03:38:46.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507593+0000", 2024-09-16T03:38:46.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6061000000000003e-05 2024-09-16T03:38:46.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507666+0000", 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3059000000000006e-05 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:90 3.0 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.995108588000001, 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62288137399999999, 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.594 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.594 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:38:46.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:38:46.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556614+0000", 2024-09-16T03:38:46.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3439999999999999e-06 2024-09-16T03:38:46.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556621+0000", 2024-09-16T03:38:46.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7440000000000001e-06 2024-09-16T03:38:46.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556632+0000", 2024-09-16T03:38:46.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1725e-05 2024-09-16T03:38:46.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556640+0000", 2024-09-16T03:38:46.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0239999999999997e-06 2024-09-16T03:38:46.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556709+0000", 2024-09-16T03:38:46.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8787000000000004e-05 2024-09-16T03:38:46.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:46.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556718+0000", 2024-09-16T03:38:46.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6920000000000008e-06 2024-09-16T03:38:46.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758133+0000", 2024-09-16T03:38:46.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201414977 2024-09-16T03:38:46.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758273+0000", 2024-09-16T03:38:46.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140684 2024-09-16T03:38:46.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:46.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758479+0000", 2024-09-16T03:38:46.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020508599999999999 2024-09-16T03:38:46.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862766+0000", 2024-09-16T03:38:46.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104287952 2024-09-16T03:38:46.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.878363+0000", 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015596601999999999 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960000+0000", 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081636630000000002 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960162+0000", 2024-09-16T03:38:46.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000162597 2024-09-16T03:38:46.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179366+0000", 2024-09-16T03:38:46.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21920357100000001 2024-09-16T03:38:46.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179411+0000", 2024-09-16T03:38:46.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5461999999999999e-05 2024-09-16T03:38:46.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179493+0000", 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1496999999999996e-05 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:91 3.0 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.995063629999997, 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70460932600000004, 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.601 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.602 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.602 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:38:46.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:38:46.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556657+0000", 2024-09-16T03:38:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:38:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556663+0000", 2024-09-16T03:38:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6579999999999999e-06 2024-09-16T03:38:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556670+0000", 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3189999999999997e-06 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556675+0000", 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000002e-06 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656929+0000", 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10025371299999999 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:46.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656944+0000", 2024-09-16T03:38:46.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5460000000000001e-05 2024-09-16T03:38:46.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861436+0000", 2024-09-16T03:38:46.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.204491969 2024-09-16T03:38:46.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861676+0000", 2024-09-16T03:38:46.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024051899999999999 2024-09-16T03:38:46.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:46.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861828+0000", 2024-09-16T03:38:46.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015110199999999999 2024-09-16T03:38:46.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862775+0000", 2024-09-16T03:38:46.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094718600000000003 2024-09-16T03:38:46.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079130+0000", 2024-09-16T03:38:46.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21635549200000001 2024-09-16T03:38:46.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179557+0000", 2024-09-16T03:38:46.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10042638399999999 2024-09-16T03:38:46.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179972+0000", 2024-09-16T03:38:46.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041574 2024-09-16T03:38:46.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261034+0000", 2024-09-16T03:38:46.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081061604999999995 2024-09-16T03:38:46.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261142+0000", 2024-09-16T03:38:46.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107688 2024-09-16T03:38:46.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261266+0000", 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124082 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:94 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.995037504999999, 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70479513699999996, 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.609 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.610 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.610 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:38:46.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:38:46.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556685+0000", 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4279999999999999e-06 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556689+0000", 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.709e-06 2024-09-16T03:38:46.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556693+0000", 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2939999999999999e-06 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657036+0000", 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10034209600000001 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657054+0000", 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8162999999999999e-05 2024-09-16T03:38:46.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862279+0000", 2024-09-16T03:38:46.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20522560200000001 2024-09-16T03:38:46.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862380+0000", 2024-09-16T03:38:46.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101178 2024-09-16T03:38:46.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862509+0000", 2024-09-16T03:38:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128909 2024-09-16T03:38:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.863063+0000", 2024-09-16T03:38:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055327600000000001 2024-09-16T03:38:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079172+0000", 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21610910899999999 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179932+0000", 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100760575 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179997+0000", 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5018999999999995e-05 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261318+0000", 2024-09-16T03:38:46.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081320838000000006 2024-09-16T03:38:46.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261385+0000", 2024-09-16T03:38:46.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7115999999999999e-05 2024-09-16T03:38:46.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261478+0000", 2024-09-16T03:38:46.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2360000000000003e-05 2024-09-16T03:38:46.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.617 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.617 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.617 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1742 3.0 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head [write 0~1659392] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T03:38:46.617 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:38:46.617 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.995009105000001, 2024-09-16T03:38:46.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62317617400000003, 2024-09-16T03:38:46.617 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.617 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.617 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.618 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.618 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.618 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1742 2024-09-16T03:38:46.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.618 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:38:46.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:38:46.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556713+0000", 2024-09-16T03:38:46.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7209999999999999e-06 2024-09-16T03:38:46.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558531+0000", 2024-09-16T03:38:46.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018184480000000001 2024-09-16T03:38:46.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558539+0000", 2024-09-16T03:38:46.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0830000000000004e-06 2024-09-16T03:38:46.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558546+0000", 2024-09-16T03:38:46.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3369999999999996e-06 2024-09-16T03:38:46.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759148+0000", 2024-09-16T03:38:46.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20060235500000001 2024-09-16T03:38:46.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759230+0000", 2024-09-16T03:38:46.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2391000000000004e-05 2024-09-16T03:38:46.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:46.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759329+0000", 2024-09-16T03:38:46.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9147000000000005e-05 2024-09-16T03:38:46.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862771+0000", 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103441085 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960102+0000", 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097331259000000003 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960218+0000", 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011581100000000001 2024-09-16T03:38:46.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079074+0000", 2024-09-16T03:38:46.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118856436 2024-09-16T03:38:46.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179758+0000", 2024-09-16T03:38:46.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100683884 2024-09-16T03:38:46.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179800+0000", 2024-09-16T03:38:46.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1928000000000002e-05 2024-09-16T03:38:46.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179887+0000", 2024-09-16T03:38:46.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7288999999999997e-05 2024-09-16T03:38:46.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.624 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.624 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:89 3.0 3:1c07e18b:::d3cef63b99e391058a88d444a56984ba6fc25b22efbc98e24ba274246c542d3c:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:46.625 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:38:46.625 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.215606608999998, 2024-09-16T03:38:46.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50388158999999999, 2024-09-16T03:38:46.625 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.625 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.625 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.625 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:38:46.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336115+0000", 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.866e-06 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336123+0000", 2024-09-16T03:38:46.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6990000000000003e-06 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336136+0000", 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3342e-05 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336143+0000", 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2099999999999998e-06 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336214+0000", 2024-09-16T03:38:46.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1538000000000005e-05 2024-09-16T03:38:46.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336342+0000", 2024-09-16T03:38:46.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127408 2024-09-16T03:38:46.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:46.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336593+0000", 2024-09-16T03:38:46.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025150500000000001 2024-09-16T03:38:46.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337284+0000", 2024-09-16T03:38:46.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069060900000000001 2024-09-16T03:38:46.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338068+0000", 2024-09-16T03:38:46.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078438900000000003 2024-09-16T03:38:46.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338107+0000", 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8858e-05 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538164+0000", 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200056971 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839843+0000", 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30167921599999997 2024-09-16T03:38:46.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839905+0000", 2024-09-16T03:38:46.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1542000000000001e-05 2024-09-16T03:38:46.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839995+0000", 2024-09-16T03:38:46.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0437000000000002e-05 2024-09-16T03:38:46.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.631 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.632 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:98 3.0 3:04dfaaf9:::3de53e1d584ced9b0f970855867dbc9a14b3fc90962aa20a81c1ed25ff4b961f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:46.632 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:38:46.632 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.214278708000002, 2024-09-16T03:38:46.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70310345299999999, 2024-09-16T03:38:46.632 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.632 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.632 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.632 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:38:46.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:38:46.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337442+0000", 2024-09-16T03:38:46.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0729999999999999e-06 2024-09-16T03:38:46.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337447+0000", 2024-09-16T03:38:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3259999999999997e-06 2024-09-16T03:38:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337457+0000", 2024-09-16T03:38:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-09-16T03:38:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337462+0000", 2024-09-16T03:38:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0780000000000003e-06 2024-09-16T03:38:46.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337511+0000", 2024-09-16T03:38:46.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:38:46.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337574+0000", 2024-09-16T03:38:46.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3114999999999995e-05 2024-09-16T03:38:46.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:46.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337723+0000", 2024-09-16T03:38:46.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014882299999999999 2024-09-16T03:38:46.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.339039+0000", 2024-09-16T03:38:46.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001315524 2024-09-16T03:38:46.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538225+0000", 2024-09-16T03:38:46.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19918607199999999 2024-09-16T03:38:46.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638838+0000", 2024-09-16T03:38:46.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100613466 2024-09-16T03:38:46.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.739726+0000", 2024-09-16T03:38:46.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100887854 2024-09-16T03:38:46.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040319+0000", 2024-09-16T03:38:46.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30059265699999999 2024-09-16T03:38:46.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040398+0000", 2024-09-16T03:38:46.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8852999999999999e-05 2024-09-16T03:38:46.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040545+0000", 2024-09-16T03:38:46.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147236 2024-09-16T03:38:46.638 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.638 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.638 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.639 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:101 3.0 3:090a0a71:::0ecec3e4e81d718f460a9826fb4cef2398bf4c951ee653be84827f33946eef5f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:46.639 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:38:46.639 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.115107962000003, 2024-09-16T03:38:46.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60435370600000005, 2024-09-16T03:38:46.639 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.639 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.639 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.639 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436614+0000", 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.844e-06 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436623+0000", 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0739999999999999e-06 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436636+0000", 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2629999999999999e-05 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436643+0000", 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7489999999999998e-06 2024-09-16T03:38:46.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536906+0000", 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10026262599999999 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536979+0000", 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3311000000000003e-05 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537109+0000", 2024-09-16T03:38:46.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013015800000000001 2024-09-16T03:38:46.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538241+0000", 2024-09-16T03:38:46.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001131726 2024-09-16T03:38:46.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638904+0000", 2024-09-16T03:38:46.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066240799999999 2024-09-16T03:38:46.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.940200+0000", 2024-09-16T03:38:46.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30129597299999999 2024-09-16T03:38:46.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040604+0000", 2024-09-16T03:38:46.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040478999999999 2024-09-16T03:38:46.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040759+0000", 2024-09-16T03:38:46.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015445500000000001 2024-09-16T03:38:46.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040824+0000", 2024-09-16T03:38:46.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5633e-05 2024-09-16T03:38:46.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040966+0000", 2024-09-16T03:38:46.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014132899999999999 2024-09-16T03:38:46.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.646 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.646 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:102 3.0 3:17742417:::632266e231c4c385ace473a8f4464285cf620fbc8799f68869353e2aa4660567:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:46.646 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:38:46.646 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.115061136000001, 2024-09-16T03:38:46.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.604540823, 2024-09-16T03:38:46.646 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.646 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.646 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436660+0000", 2024-09-16T03:38:46.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:38:46.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436663+0000", 2024-09-16T03:38:46.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1599999999999998e-06 2024-09-16T03:38:46.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436670+0000", 2024-09-16T03:38:46.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.742e-06 2024-09-16T03:38:46.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436675+0000", 2024-09-16T03:38:46.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3839999999999996e-06 2024-09-16T03:38:46.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537499+0000", 2024-09-16T03:38:46.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100823987 2024-09-16T03:38:46.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537602+0000", 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102837 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537793+0000", 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019123499999999999 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638962+0000", 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101168944 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.840035+0000", 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201073481 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040706+0000", 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200670606 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041009+0000", 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030271600000000002 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041087+0000", 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8310000000000001e-05 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041127+0000", 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041200+0000", 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2381999999999999e-05 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.652 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:103 3.0 3:28447854:::c811732c201b76a96e1532c4d9a18d4f4f2a8ddec3bf79945689a1a62c8a5a25:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.115015886999998, 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60467526100000002, 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436705+0000", 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.23e-07 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436719+0000", 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3957999999999999e-05 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436725+0000", 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6100000000000002e-06 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436730+0000", 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0609999999999997e-06 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538257+0000", 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10152739500000001 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538314+0000", 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7355000000000002e-05 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538435+0000", 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120967 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638968+0000", 2024-09-16T03:38:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100532994 2024-09-16T03:38:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040662+0000", 2024-09-16T03:38:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40169386299999998 2024-09-16T03:38:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041059+0000", 2024-09-16T03:38:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039723900000000002 2024-09-16T03:38:46.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041247+0000", 2024-09-16T03:38:46.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018756900000000001 2024-09-16T03:38:46.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041286+0000", 2024-09-16T03:38:46.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8971999999999999e-05 2024-09-16T03:38:46.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041318+0000", 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1628999999999998e-05 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041379+0000", 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1926000000000003e-05 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(osd.11.23:98 3.0 e74/69 ondisk, result = 0)", 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:38:46.659 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.013199518999997, 2024-09-16T03:38:46.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50203460700000002, 2024-09-16T03:38:46.660 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.660 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.660 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:46.660 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:38:46.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:38:46.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:38:46.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0899999999999997e-07 2024-09-16T03:38:46.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538523+0000", 2024-09-16T03:38:46.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-09-16T03:38:46.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538527+0000", 2024-09-16T03:38:46.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0539999999999996e-06 2024-09-16T03:38:46.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538529+0000", 2024-09-16T03:38:46.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7420000000000002e-06 2024-09-16T03:38:46.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040301+0000", 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50177146399999994 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040312+0000", 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1039999999999999e-05 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040555+0000", 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243212 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.663 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.664 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.664 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(osd.11.23:103 3.0 e75/69 ondisk, result = 0)", 2024-09-16T03:38:46.664 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.540430+0000", 2024-09-16T03:38:46.664 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.011290051000003, 2024-09-16T03:38:46.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50024338400000001, 2024-09-16T03:38:46.664 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.664 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.664 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:46.664 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540430+0000", 2024-09-16T03:38:46.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540430+0000", 2024-09-16T03:38:46.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540431+0000", 2024-09-16T03:38:46.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3600000000000002e-07 2024-09-16T03:38:46.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540434+0000", 2024-09-16T03:38:46.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5149999999999998e-06 2024-09-16T03:38:46.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540442+0000", 2024-09-16T03:38:46.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6340000000000001e-06 2024-09-16T03:38:46.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.540448+0000", 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.285e-06 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040646+0000", 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50019782800000001 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040655+0000", 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3429999999999996e-06 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040673+0000", 2024-09-16T03:38:46.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8842999999999999e-05 2024-09-16T03:38:46.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.668 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.668 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2473 4.e 4:765e9179:::smithi15736696-44:head [write 1830877~681726] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:46.668 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:38:46.668 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.541030662000001, 2024-09-16T03:38:46.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50285070099999996, 2024-09-16T03:38:46.669 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.669 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.669 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.669 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:46.669 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:46.669 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:46.669 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2473 2024-09-16T03:38:46.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.669 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:38:46.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:38:46.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010691+0000", 2024-09-16T03:38:46.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-06 2024-09-16T03:38:46.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011744+0000", 2024-09-16T03:38:46.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001052486 2024-09-16T03:38:46.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011755+0000", 2024-09-16T03:38:46.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1073000000000001e-05 2024-09-16T03:38:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011761+0000", 2024-09-16T03:38:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1600000000000003e-06 2024-09-16T03:38:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011804+0000", 2024-09-16T03:38:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2697999999999997e-05 2024-09-16T03:38:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011885+0000", 2024-09-16T03:38:46.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.119e-05 2024-09-16T03:38:46.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:38:46.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.012013+0000", 2024-09-16T03:38:46.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128066 2024-09-16T03:38:46.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.013923+0000", 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019104739999999999 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.412920+0000", 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39899685200000001 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513347+0000", 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100427167 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513490+0000", 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142163 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513540+0000", 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0482000000000002e-05 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:135 3.0 3:03e4b4cc:::36686daa84b9b41d23533e7d0b5d4ef41f745a0756e58cc66c36c36f24851219:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:46.675 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.114458+0000", 2024-09-16T03:38:46.676 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.437261886000002, 2024-09-16T03:38:46.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.502137163, 2024-09-16T03:38:46.676 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.676 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.676 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.676 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114458+0000", 2024-09-16T03:38:46.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114458+0000", 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114459+0000", 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.26e-06 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114464+0000", 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8049999999999997e-06 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114474+0000", 2024-09-16T03:38:46.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4369999999999996e-06 2024-09-16T03:38:46.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114479+0000", 2024-09-16T03:38:46.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3669999999999996e-06 2024-09-16T03:38:46.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114539+0000", 2024-09-16T03:38:46.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114652+0000", 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112948 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114886+0000", 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000233666 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116749+0000", 2024-09-16T03:38:46.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001863314 2024-09-16T03:38:46.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116765+0000", 2024-09-16T03:38:46.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5528000000000001e-05 2024-09-16T03:38:46.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.413659+0000", 2024-09-16T03:38:46.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.296894245 2024-09-16T03:38:46.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414169+0000", 2024-09-16T03:38:46.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050972500000000004 2024-09-16T03:38:46.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616394+0000", 2024-09-16T03:38:46.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20222523000000001 2024-09-16T03:38:46.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616467+0000", 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2929000000000006e-05 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616595+0000", 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012867400000000001 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:136 3.0 3:1b68d773:::374e1d62491a062bcb5530630b4c28c4696d618a79860b96215d227206cd62aa:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:46.682 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.437226122999999, 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50236015199999995, 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:38:46.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:38:46.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4499999999999997e-07 2024-09-16T03:38:46.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114497+0000", 2024-09-16T03:38:46.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:38:46.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114506+0000", 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5759999999999995e-06 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114511+0000", 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0649999999999998e-06 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115277+0000", 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076592800000000003 2024-09-16T03:38:46.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115368+0000", 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1037000000000003e-05 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115573+0000", 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000204594 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116773+0000", 2024-09-16T03:38:46.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001200062 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.121514+0000", 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047407270000000001 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414056+0000", 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29254249700000001 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414192+0000", 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013572000000000001 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616663+0000", 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20247094800000001 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616728+0000", 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5217999999999998e-05 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616854+0000", 2024-09-16T03:38:46.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126064 2024-09-16T03:38:46.689 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.689 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.689 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.689 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:137 3.0 3:20b08a9e:::0ba23f501764ab3447cc323c825ee8bff84c3c6e340c68c08117252e1f0b39bb:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:46.689 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:38:46.689 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.437182178, 2024-09-16T03:38:46.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50256677599999999, 2024-09-16T03:38:46.689 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:46.690 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:46.690 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:46.690 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:46.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:46.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:38:46.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:46.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:38:46.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:46.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1200000000000003e-07 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114542+0000", 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7400000000000002e-06 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114547+0000", 2024-09-16T03:38:46.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.42e-06 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114550+0000", 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4470000000000002e-06 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115951+0000", 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014005459999999999 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116047+0000", 2024-09-16T03:38:46.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6899999999999997e-05 2024-09-16T03:38:46.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:38:46.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116270+0000", 2024-09-16T03:38:46.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022291699999999999 2024-09-16T03:38:46.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:46.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.117236+0000", 2024-09-16T03:38:46.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096522299999999997 2024-09-16T03:38:46.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.218932+0000", 2024-09-16T03:38:46.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10169644799999999 2024-09-16T03:38:46.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414271+0000", 2024-09-16T03:38:46.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19533916300000001 2024-09-16T03:38:46.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.514717+0000", 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100445879 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616919+0000", 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10220199100000001 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616980+0000", 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0977999999999998e-05 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:46.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:46.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.617105+0000", 2024-09-16T03:38:46.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012461199999999999 2024-09-16T03:38:46.696 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.696 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.696 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.696 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:46.696 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:46.696 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:46.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:38:46.698 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:46.695+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:46.798 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:46.796+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:46.807 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:46.807 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:46.807 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:38:46.807 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:46.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:38:46.899 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:46.897+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:46.925 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:46.925 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:46.925 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:38:46.926 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:38:46.926 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:46.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:38:46.973 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:38:47.001 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:47.000+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:47.046 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:47.046 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:38:47.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:38:47.046 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:47.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.046 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4338.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:38:47.046 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:38:47.046 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 111.94583462999999, 2024-09-16T03:38:47.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30502727499999999, 2024-09-16T03:38:47.047 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.047 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:47.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4338", 2024-09-16T03:38:47.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.64:0/3073758685", 2024-09-16T03:38:47.047 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-16T03:38:47.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.047 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:38:47.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097098+0000", 2024-09-16T03:38:47.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097101+0000", 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.915e-06 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097111+0000", 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0589e-05 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097130+0000", 2024-09-16T03:38:47.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8958999999999999e-05 2024-09-16T03:38:47.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097141+0000", 2024-09-16T03:38:47.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0616e-05 2024-09-16T03:38:47.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097227+0000", 2024-09-16T03:38:47.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6322000000000005e-05 2024-09-16T03:38:47.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.097232+0000", 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2059999999999998e-06 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200235+0000", 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103002945 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200318+0000", 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2515999999999994e-05 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,10", 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.200429+0000", 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011158999999999999 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201478+0000", 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00104893 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.201942+0000", 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046354700000000003 2024-09-16T03:38:47.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402048+0000", 2024-09-16T03:38:47.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20010616000000001 2024-09-16T03:38:47.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:47.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402089+0000", 2024-09-16T03:38:47.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0762e-05 2024-09-16T03:38:47.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:55.402125+0000", 2024-09-16T03:38:47.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6217999999999998e-05 2024-09-16T03:38:47.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.054 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.054 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.054 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:73 4.a 4:532814f2:::smithi15736696-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:47.054 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:38:47.054 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.731672493999994, 2024-09-16T03:38:47.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.362642662, 2024-09-16T03:38:47.054 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.054 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311261+0000", 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.562e-06 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311925+0000", 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066385499999999998 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311937+0000", 2024-09-16T03:38:47.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1881e-05 2024-09-16T03:38:47.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311943+0000", 2024-09-16T03:38:47.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6999999999999996e-06 2024-09-16T03:38:47.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311988+0000", 2024-09-16T03:38:47.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5358000000000002e-05 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312070+0000", 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1735000000000003e-05 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312167+0000", 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6685e-05 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:47.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.313661+0000", 2024-09-16T03:38:47.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001494324 2024-09-16T03:38:47.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.414759+0000", 2024-09-16T03:38:47.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10109850300000001 2024-09-16T03:38:47.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673821+0000", 2024-09-16T03:38:47.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.25906127400000001 2024-09-16T03:38:47.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673858+0000", 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7620999999999998e-05 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673902+0000", 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4163999999999999e-05 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.061 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:303 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 90.631213664000001, 2024-09-16T03:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50956986000000004, 2024-09-16T03:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 303 2024-09-16T03:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411721+0000", 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0200000000000001e-06 2024-09-16T03:38:47.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411729+0000", 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1240000000000005e-06 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411744+0000", 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099000000000001e-05 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411751+0000", 2024-09-16T03:38:47.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.029e-06 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511892+0000", 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100140879 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511961+0000", 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9379000000000001e-05 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:47.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.613647+0000", 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101685994 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.614294+0000", 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064692799999999996 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.714962+0000", 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066765 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921157+0000", 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206195463 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921233+0000", 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5446999999999998e-05 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921288+0000", 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5847999999999998e-05 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:503 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 85.221139112000003, 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60521621599999997, 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.068 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 503 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821794+0000", 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821798+0000", 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2030000000000002e-06 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821804+0000", 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821810+0000", 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4249999999999997e-06 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922212+0000", 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040228800000001 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922307+0000", 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5089999999999999e-05 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.126007+0000", 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20369931499999999 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.127605+0000", 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001598213 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.327263+0000", 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199657905 2024-09-16T03:38:47.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426900+0000", 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099637217 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426963+0000", 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.427009+0000", 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6533000000000003e-05 2024-09-16T03:38:47.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1227 4.a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v18,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.366962848, 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.44160531400000003, 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1227 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675972+0000", 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7269999999999999e-06 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675981+0000", 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.166e-06 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675996+0000", 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4901e-05 2024-09-16T03:38:47.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676005+0000", 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4579999999999993e-06 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676081+0000", 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6157999999999998e-05 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676188+0000", 2024-09-16T03:38:47.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106871 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.016594+0000", 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34040622300000001 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017089+0000", 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049523899999999997 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017779+0000", 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069032200000000005 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117504+0000", 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099724207999999995 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117543+0000", 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9208999999999997e-05 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117575+0000", 2024-09-16T03:38:47.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1832000000000003e-05 2024-09-16T03:38:47.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.080 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.080 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1523 4.a 4:50107449:::smithi15736696-27:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:38:47.080 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:38:47.080 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.679921522000001, 2024-09-16T03:38:47.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.410522742, 2024-09-16T03:38:47.080 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.080 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1523 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363012+0000", 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.269e-06 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363016+0000", 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8920000000000002e-06 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363027+0000", 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1079e-05 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463160+0000", 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10013267100000001 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463297+0000", 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013717699999999999 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463381+0000", 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3701000000000004e-05 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.773533+0000", 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31015295300000001 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.084 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1690 4.a 4:51338165:::smithi15736696-4:head [assert-version v63,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.592648855, 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52002349699999995, 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1690 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450286+0000", 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450293+0000", 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5089999999999996e-06 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450308+0000", 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4955e-05 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450316+0000", 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7079999999999997e-06 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450386+0000", 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9715999999999995e-05 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450478+0000", 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2610999999999998e-05 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.665802+0000", 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21532342700000001 2024-09-16T03:38:47.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668724+0000", 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029220600000000002 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668758+0000", 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.434e-05 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970243+0000", 2024-09-16T03:38:47.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30148501 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970278+0000", 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4965000000000002e-05 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970307+0000", 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8968000000000001e-05 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1654 4.12 4:4a9d235d:::smithi15736696-1:head [write 652138~461367 in=461367b,stat out=16b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.500895364000002, 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42790198699999998, 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1654 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542039+0000", 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0250000000000001e-06 2024-09-16T03:38:47.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542682+0000", 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064305099999999995 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542698+0000", 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6469000000000001e-05 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542710+0000", 2024-09-16T03:38:47.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1173e-05 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542767+0000", 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7584000000000003e-05 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542775+0000", 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6149999999999999e-06 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666597+0000", 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123821815 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666745+0000", 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148675 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666935+0000", 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018963899999999999 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969207+0000", 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302272031 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969374+0000", 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016686699999999999 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969533+0000", 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015909299999999999 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969560+0000", 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7148000000000002e-05 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969631+0000", 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1174000000000007e-05 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969685+0000", 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3613000000000003e-05 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:47.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969764+0000", 2024-09-16T03:38:47.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9281000000000002e-05 2024-09-16T03:38:47.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969939+0000", 2024-09-16T03:38:47.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473400000000001 2024-09-16T03:38:47.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.098 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1655 4.12 4:4a9d235d:::smithi15736696-1:head [write 1574940~677198 in=677198b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.500203245999998, 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42773147700000003, 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1655 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542730+0000", 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.085e-06 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543338+0000", 2024-09-16T03:38:47.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060826100000000002 2024-09-16T03:38:47.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543351+0000", 2024-09-16T03:38:47.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2789000000000001e-05 2024-09-16T03:38:47.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543360+0000", 2024-09-16T03:38:47.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1970000000000002e-06 2024-09-16T03:38:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543420+0000", 2024-09-16T03:38:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9562999999999999e-05 2024-09-16T03:38:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543431+0000", 2024-09-16T03:38:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0866999999999999e-05 2024-09-16T03:38:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.768963+0000", 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.225532487 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769029+0000", 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5881000000000002e-05 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769134+0000", 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104923 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:47.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969639+0000", 2024-09-16T03:38:47.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20050517900000001 2024-09-16T03:38:47.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969978+0000", 2024-09-16T03:38:47.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033848599999999998 2024-09-16T03:38:47.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970206+0000", 2024-09-16T03:38:47.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022867300000000001 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970248+0000", 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1433999999999997e-05 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970299+0000", 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0677999999999998e-05 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970338+0000", 2024-09-16T03:38:47.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9646e-05 2024-09-16T03:38:47.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:47.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970377+0000", 2024-09-16T03:38:47.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8439999999999998e-05 2024-09-16T03:38:47.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970460+0000", 2024-09-16T03:38:47.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3887999999999994e-05 2024-09-16T03:38:47.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.106 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1656 4.12 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2252138] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.499557174000003, 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72904596899999996, 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.107 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1656 2024-09-16T03:38:47.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.108 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:38:47.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:38:47.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543376+0000", 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4200000000000004e-07 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543383+0000", 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6290000000000004e-06 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543390+0000", 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:38:47.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543396+0000", 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.02e-06 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543461+0000", 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4876999999999996e-05 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543465+0000", 2024-09-16T03:38:47.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2139999999999998e-06 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970490+0000", 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42702478300000002 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970556+0000", 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6137999999999996e-05 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:38:47.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970670+0000", 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114416 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971375+0000", 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070496000000000003 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071773+0000", 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100397945 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071823+0000", 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.072067+0000", 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243856 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.171991+0000", 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099924052999999999 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272176+0000", 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100184963 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272251+0000", 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5902000000000006e-05 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272421+0000", 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016962800000000001 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.114 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1657 4.12 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:38:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.499525188, 2024-09-16T03:38:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72934181899999995, 2024-09-16T03:38:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1657 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543408+0000", 2024-09-16T03:38:47.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.13e-07 2024-09-16T03:38:47.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543410+0000", 2024-09-16T03:38:47.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7030000000000002e-06 2024-09-16T03:38:47.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543417+0000", 2024-09-16T03:38:47.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.567e-06 2024-09-16T03:38:47.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543423+0000", 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0619999999999997e-06 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543489+0000", 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6315999999999993e-05 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543498+0000", 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7239999999999998e-06 2024-09-16T03:38:47.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970952+0000", 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42745424399999998 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971012+0000", 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9771000000000001e-05 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272463+0000", 2024-09-16T03:38:47.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30145090800000002 2024-09-16T03:38:47.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272558+0000", 2024-09-16T03:38:47.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4863000000000003e-05 2024-09-16T03:38:47.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272749+0000", 2024-09-16T03:38:47.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191048 2024-09-16T03:38:47.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.120 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2105 4.a 4:51338165:::smithi15736696-4:head [sparse-read 0~887916,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.994060994000002, 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40244934799999998, 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2105 2024-09-16T03:38:47.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:38:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:38:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048881+0000", 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6080000000000002e-06 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048889+0000", 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7209999999999994e-06 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048903+0000", 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4538e-05 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048912+0000", 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2390000000000008e-06 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149039+0000", 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100126563 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149044+0000", 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0340000000000004e-06 2024-09-16T03:38:47.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451087+0000", 2024-09-16T03:38:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30204318000000002 2024-09-16T03:38:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451159+0000", 2024-09-16T03:38:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1627999999999999e-05 2024-09-16T03:38:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451321+0000", 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016183699999999999 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2106 4.a 4:51338165:::smithi15736696-4:head [sparse-read 0~887916] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.994000235000001, 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40254453200000001, 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2106 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:38:47.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048933+0000", 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048936+0000", 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3890000000000001e-06 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048943+0000", 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1250000000000004e-06 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048953+0000", 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8190000000000003e-06 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149113+0000", 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100160007 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149118+0000", 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7650000000000001e-06 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451341+0000", 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30222344699999998 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451381+0000", 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9517000000000001e-05 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451477+0000", 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5656999999999996e-05 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2107 4.a 4:51338165:::smithi15736696-4:head [read 1698310~486600,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.993970171999997, 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40288642400000002, 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.131 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2107 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048963+0000", 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048967+0000", 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9060000000000004e-06 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048972+0000", 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5910000000000003e-06 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048977+0000", 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9629999999999997e-06 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149135+0000", 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015828 2024-09-16T03:38:47.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149138+0000", 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.779e-06 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451506+0000", 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302367891 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451558+0000", 2024-09-16T03:38:47.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2374999999999999e-05 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451849+0000", 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029009899999999999 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2537 4.a 4:503187b3:::smithi15736696-34:head [assert-version v87,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:17.047079+0000", 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.995853342, 2024-09-16T03:38:47.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.32976008000000001, 2024-09-16T03:38:47.137 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.137 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:47.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.137 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2537 2024-09-16T03:38:47.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.137 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047079+0000", 2024-09-16T03:38:47.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047079+0000", 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047081+0000", 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6929999999999999e-06 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047086+0000", 2024-09-16T03:38:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8680000000000004e-06 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047099+0000", 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2309e-05 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047105+0000", 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7050000000000002e-06 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047157+0000", 2024-09-16T03:38:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1057999999999998e-05 2024-09-16T03:38:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047219+0000", 2024-09-16T03:38:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2553999999999998e-05 2024-09-16T03:38:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.149371+0000", 2024-09-16T03:38:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102152112 2024-09-16T03:38:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.149846+0000", 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047467400000000002 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.183065+0000", 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033219374000000003 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.376793+0000", 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.193728023 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.376819+0000", 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6142000000000001e-05 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.376839+0000", 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9568e-05 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2687 4.a 4:503187b3:::smithi15736696-34:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:21.869464+0000", 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.173467898999998, 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31565330200000002, 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2687 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869464+0000", 2024-09-16T03:38:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869464+0000", 2024-09-16T03:38:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869465+0000", 2024-09-16T03:38:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.51e-07 2024-09-16T03:38:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869467+0000", 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869473+0000", 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9560000000000002e-06 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869488+0000", 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5333e-05 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869667+0000", 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017900699999999999 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869710+0000", 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2749e-05 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:22.185118+0000", 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31540757600000002 2024-09-16T03:38:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2986 4.a 4:5a5e9bc2:::smithi15736696-36:head [write 487469~563838] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.912930+0000", 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.130002723, 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39506163799999999, 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2986 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.912930+0000", 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.912930+0000", 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.912932+0000", 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8169999999999999e-06 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913522+0000", 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058972099999999997 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913539+0000", 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7306999999999999e-05 2024-09-16T03:38:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913550+0000", 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0356e-05 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913637+0000", 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7552999999999993e-05 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913699+0000", 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1991999999999999e-05 2024-09-16T03:38:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913797+0000", 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7738000000000005e-05 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.915370+0000", 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015725719999999999 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.916260+0000", 2024-09-16T03:38:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089050100000000003 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.307864+0000", 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39160404900000001 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.307936+0000", 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2025000000000002e-05 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.307991+0000", 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5006999999999997e-05 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2987 4.a 4:5a5e9bc2:::smithi15736696-36:head [write 1607751~551752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.123984683, 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49279943700000001, 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2987 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:38:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918949+0000", 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.86e-06 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919579+0000", 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062909000000000001 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919591+0000", 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2078e-05 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919596+0000", 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019705+0000", 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100108405 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019754+0000", 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9258999999999997e-05 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019847+0000", 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2954999999999993e-05 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.023003+0000", 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031561110000000001 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.122544+0000", 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099540553000000004 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411658+0000", 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.28911426699999998 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411696+0000", 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8430000000000003e-05 2024-09-16T03:38:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411747+0000", 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0745999999999998e-05 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2988 4.a 4:5a5e9bc2:::smithi15736696-36:head [setxattr _header (56) in=7b,truncate 2487469] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.123322944, 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59236646500000001, 2024-09-16T03:38:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2988 2024-09-16T03:38:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:38:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919610+0000", 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.048e-06 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919615+0000", 2024-09-16T03:38:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8239999999999999e-06 2024-09-16T03:38:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919621+0000", 2024-09-16T03:38:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4380000000000003e-06 2024-09-16T03:38:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919626+0000", 2024-09-16T03:38:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2000000000000002e-06 2024-09-16T03:38:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.020922+0000", 2024-09-16T03:38:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101295782 2024-09-16T03:38:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.020995+0000", 2024-09-16T03:38:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3256999999999993e-05 2024-09-16T03:38:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:38:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.021088+0000", 2024-09-16T03:38:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3419999999999997e-05 2024-09-16T03:38:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:38:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.023007+0000", 2024-09-16T03:38:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019186380000000001 2024-09-16T03:38:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.222530+0000", 2024-09-16T03:38:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19952276899999999 2024-09-16T03:38:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511887+0000", 2024-09-16T03:38:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.289357378 2024-09-16T03:38:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:38:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511927+0000", 2024-09-16T03:38:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0357e-05 2024-09-16T03:38:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511976+0000", 2024-09-16T03:38:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8353999999999998e-05 2024-09-16T03:38:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3258 4.1a 4:5e3c0760:::smithi15736696-24:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:38:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.6128720689999998, 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3302292470000001, 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3258 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:38:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:38:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430061+0000", 2024-09-16T03:38:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.27e-07 2024-09-16T03:38:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430062+0000", 2024-09-16T03:38:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6780000000000001e-06 2024-09-16T03:38:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430066+0000", 2024-09-16T03:38:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5269999999999999e-06 2024-09-16T03:38:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430069+0000", 2024-09-16T03:38:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:38:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430119+0000", 2024-09-16T03:38:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0212e-05 2024-09-16T03:38:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:38:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430122+0000", 2024-09-16T03:38:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7319999999999998e-06 2024-09-16T03:38:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:38:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.631493+0000", 2024-09-16T03:38:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20137142499999999 2024-09-16T03:38:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:38:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.631579+0000", 2024-09-16T03:38:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5418000000000001e-05 2024-09-16T03:38:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:38:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:41.760289+0000", 2024-09-16T03:38:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1287106680000001 2024-09-16T03:38:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.174 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:47.176 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:47.174+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:47.177 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:38:47.178 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:47.178 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:47.178 INFO:tasks.rados.rados.0.smithi157.stdout:1493: done (0 left) 2024-09-16T03:38:47.178 INFO:tasks.rados.rados.0.smithi157.stdout:1494: tier_flush oid 24 2024-09-16T03:38:47.179 INFO:tasks.rados.rados.0.smithi157.stdout:1494: tier_flush oid 24 snap -1 2024-09-16T03:38:47.179 INFO:tasks.rados.rados.0.smithi157.stdout:1495: read oid 25 snap -1 2024-09-16T03:38:47.179 INFO:tasks.rados.rados.0.smithi157.stdout:1495: expect (ObjNum 295 snap 0 seq_num 295) 2024-09-16T03:38:47.179 INFO:tasks.rados.rados.0.smithi157.stdout:1496: read oid 40 snap -1 2024-09-16T03:38:47.179 INFO:tasks.rados.rados.0.smithi157.stdout:1496: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:38:47.179 INFO:tasks.rados.rados.0.smithi157.stdout:1497: read oid 2 snap -1 2024-09-16T03:38:47.179 INFO:tasks.rados.rados.0.smithi157.stdout:1497: expect deleted 2024-09-16T03:38:47.179 INFO:tasks.rados.rados.0.smithi157.stdout:1498: read oid 28 snap -1 2024-09-16T03:38:47.179 INFO:tasks.rados.rados.0.smithi157.stdout:1498: expect (ObjNum 309 snap 0 seq_num 309) 2024-09-16T03:38:47.179 INFO:tasks.rados.rados.0.smithi157.stdout:1499: write oid 7 current snap is 0 2024-09-16T03:38:47.179 INFO:tasks.rados.rados.0.smithi157.stdout:1499: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-09-16T03:38:47.179 INFO:tasks.rados.rados.0.smithi157.stdout:1499: writing smithi15736696-7 from 426977 to 883536 tid 1 2024-09-16T03:38:47.179 INFO:tasks.rados.rados.0.smithi157.stdout:1499: writing smithi15736696-7 from 1482841 to 2282605 tid 2 2024-09-16T03:38:47.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:47.180 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:47.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:47.181 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:47.181 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:47.182 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:47.182 INFO:tasks.rados.rados.0.smithi157.stdout:1499: writing smithi15736696-7 from 2958080 to 3226977 tid 3 2024-09-16T03:38:47.182 INFO:tasks.rados.rados.0.smithi157.stdout:1500: tier_evict oid 26 2024-09-16T03:38:47.182 INFO:tasks.rados.rados.0.smithi157.stdout:1500: tier_evict oid 26 snap -1 2024-09-16T03:38:47.182 INFO:tasks.rados.rados.0.smithi157.stdout:1501: delete oid 33 current snap is 0 2024-09-16T03:38:47.182 INFO:tasks.rados.rados.0.smithi157.stdout:1494: got (0) Success 2024-09-16T03:38:47.182 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 115 (ObjNum 271 snap 0 seq_num 271) dirty exists 2024-09-16T03:38:47.183 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:47.183 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:47.189 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:47.189 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:47.208 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:47.209 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:47.225 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:47.225 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:47.277 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:47.275+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:47.280 INFO:tasks.rados.rados.0.smithi157.stdout:1500: got (2) No such file or directory 2024-09-16T03:38:47.280 INFO:tasks.rados.rados.0.smithi157.stdout:1500: got expected ENOENT (src dne) 2024-09-16T03:38:47.292 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:47.292 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3422 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:47.178845+0000", 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.11236916199999999, 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.11238788, 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3422 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:38:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:38:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.178845+0000", 2024-09-16T03:38:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:38:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.178845+0000", 2024-09-16T03:38:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:38:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:38:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.178847+0000", 2024-09-16T03:38:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1069999999999999e-06 2024-09-16T03:38:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:38:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.178854+0000", 2024-09-16T03:38:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6340000000000003e-06 2024-09-16T03:38:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:38:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.178880+0000", 2024-09-16T03:38:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5989000000000001e-05 2024-09-16T03:38:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:38:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:38:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:38:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.178890+0000", 2024-09-16T03:38:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0491e-05 2024-09-16T03:38:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:38:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:38:47.297 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:38:47.297 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-09-16T03:38:47.297 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:47.297 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:47.297 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:47.301 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:38:47.312 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:47.313 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:47.330 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:47.330 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:47.347 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:47.348 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:47.355 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:47.353+0000 7f3d4270c640 -1 osd.12 112 handle_osd_map injecting map crc failure 2024-09-16T03:38:47.357 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:47.355+0000 7f4212d00640 -1 osd.8 112 handle_osd_map injecting map crc failure 2024-09-16T03:38:47.357 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:47.355+0000 7fd103e73640 -1 osd.2 112 handle_osd_map injecting map crc failure 2024-09-16T03:38:47.367 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:47.367 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:47.379 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:47.376+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:47.382 INFO:tasks.rados.rados.0.smithi157.stdout:1494: done (7 left) 2024-09-16T03:38:47.382 INFO:tasks.rados.rados.0.smithi157.stdout:1497: done (6 left) 2024-09-16T03:38:47.382 INFO:tasks.rados.rados.0.smithi157.stdout:1500: done (5 left) 2024-09-16T03:38:47.382 INFO:tasks.rados.rados.0.smithi157.stdout:1501: done (4 left) 2024-09-16T03:38:47.382 INFO:tasks.rados.rados.0.smithi157.stdout:1502: tier_promote oid 37 2024-09-16T03:38:47.382 INFO:tasks.rados.rados.0.smithi157.stdout:1503: write (excl) oid 50 current snap is 0 2024-09-16T03:38:47.382 INFO:tasks.rados.rados.0.smithi157.stdout:1503: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-09-16T03:38:47.384 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:47.385 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:47.388 INFO:tasks.rados.rados.0.smithi157.stdout:1503: writing smithi15736696-50 from 460591 to 1184001 tid 1 2024-09-16T03:38:47.393 INFO:tasks.rados.rados.0.smithi157.stdout:1503: writing smithi15736696-50 from 1716158 to 2119591 tid 2 2024-09-16T03:38:47.399 INFO:tasks.rados.rados.0.smithi157.stdout:1503: writing smithi15736696-50 from 2729279 to 3169419 tid 3 2024-09-16T03:38:47.399 INFO:tasks.rados.rados.0.smithi157.stdout:1504: read oid 4 snap -1 2024-09-16T03:38:47.399 INFO:tasks.rados.rados.0.smithi157.stdout:1504: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-16T03:38:47.400 INFO:tasks.rados.rados.0.smithi157.stdout:1505: read oid 49 snap -1 2024-09-16T03:38:47.400 INFO:tasks.rados.rados.0.smithi157.stdout:1505: expect (ObjNum 254 snap 0 seq_num 254) 2024-09-16T03:38:47.400 INFO:tasks.rados.rados.0.smithi157.stdout:1506: read oid 9 snap -1 2024-09-16T03:38:47.400 INFO:tasks.rados.rados.0.smithi157.stdout:1506: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-16T03:38:47.400 INFO:tasks.rados.rados.0.smithi157.stdout:1507: write (excl) oid 31 current snap is 0 2024-09-16T03:38:47.400 INFO:tasks.rados.rados.0.smithi157.stdout:1507: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-09-16T03:38:47.408 INFO:tasks.rados.rados.0.smithi157.stdout:1507: writing smithi15736696-31 from 477398 to 1276057 tid 1 2024-09-16T03:38:47.412 INFO:tasks.rados.rados.0.smithi157.stdout:1507: writing smithi15736696-31 from 1774640 to 2438084 tid 2 2024-09-16T03:38:47.417 INFO:tasks.rados.rados.0.smithi157.stdout:1507: writing smithi15736696-31 from 2873055 to 3277398 tid 3 2024-09-16T03:38:47.417 INFO:tasks.rados.rados.0.smithi157.stdout:1508: read oid 18 snap -1 2024-09-16T03:38:47.417 INFO:tasks.rados.rados.0.smithi157.stdout:1508: expect deleted 2024-09-16T03:38:47.417 INFO:tasks.rados.rados.0.smithi157.stdout:1509: tier_flush oid 8 2024-09-16T03:38:47.417 INFO:tasks.rados.rados.0.smithi157.stdout:1509: tier_flush oid 8 snap -1 2024-09-16T03:38:47.418 INFO:tasks.rados.rados.0.smithi157.stdout:1505: expect (ObjNum 254 snap 0 seq_num 254) 2024-09-16T03:38:47.423 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:47.423 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:47.431 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1408415 2024-09-16T03:38:47.432 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:38:47.449 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:47.478 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:47.478 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:47.479 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:47.477+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:47.492 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 303890 2024-09-16T03:38:47.495 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:47.495 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:47.530 INFO:tasks.rados.rados.0.smithi157.stdout:1505: done (11 left) 2024-09-16T03:38:47.531 INFO:tasks.rados.rados.0.smithi157.stdout:1510: write oid 26 current snap is 0 2024-09-16T03:38:47.531 INFO:tasks.rados.rados.0.smithi157.stdout:1510: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2024-09-16T03:38:47.535 INFO:tasks.rados.rados.0.smithi157.stdout:1510: writing smithi15736696-26 from 494205 to 968113 tid 1 2024-09-16T03:38:47.540 INFO:tasks.rados.rados.0.smithi157.stdout:1510: writing smithi15736696-26 from 1433122 to 1840224 tid 2 2024-09-16T03:38:47.546 INFO:tasks.rados.rados.0.smithi157.stdout:1510: writing smithi15736696-26 from 2384125 to 2846809 tid 3 2024-09-16T03:38:47.546 INFO:tasks.rados.rados.0.smithi157.stdout:1511: tier_flush oid 6 2024-09-16T03:38:47.546 INFO:tasks.rados.rados.0.smithi157.stdout:1511: tier_flush oid 6 snap -1 2024-09-16T03:38:47.546 INFO:tasks.rados.rados.0.smithi157.stdout:1512: tier_evict oid 44 2024-09-16T03:38:47.546 INFO:tasks.rados.rados.0.smithi157.stdout:1512: tier_evict oid 44 snap -1 2024-09-16T03:38:47.546 INFO:tasks.rados.rados.0.smithi157.stdout:1513: write (excl) oid 49 current snap is 0 2024-09-16T03:38:47.546 INFO:tasks.rados.rados.0.smithi157.stdout:1513: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2024-09-16T03:38:47.551 INFO:tasks.rados.rados.0.smithi157.stdout:1513: writing smithi15736696-49 from 511012 to 1060169 tid 1 2024-09-16T03:38:47.557 INFO:tasks.rados.rados.0.smithi157.stdout:1513: writing smithi15736696-49 from 1775251 to 2326011 tid 2 2024-09-16T03:38:47.563 INFO:tasks.rados.rados.0.smithi157.stdout:1513: writing smithi15736696-49 from 3095195 to 3311012 tid 3 2024-09-16T03:38:47.563 INFO:tasks.rados.rados.0.smithi157.stdout:1514: write (excl) oid 43 current snap is 0 2024-09-16T03:38:47.564 INFO:tasks.rados.rados.0.smithi157.stdout:1514: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2024-09-16T03:38:47.569 INFO:tasks.rados.rados.0.smithi157.stdout:1514: writing smithi15736696-43 from 527819 to 1152225 tid 1 2024-09-16T03:38:47.575 INFO:tasks.rados.rados.0.smithi157.stdout:1514: writing smithi15736696-43 from 1833733 to 2244504 tid 2 2024-09-16T03:38:47.580 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:47.578+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:47.582 INFO:tasks.rados.rados.0.smithi157.stdout:1514: writing smithi15736696-43 from 2722618 to 3324199 tid 3 2024-09-16T03:38:47.583 INFO:tasks.rados.rados.0.smithi157.stdout:1508: done (15 left) 2024-09-16T03:38:47.583 INFO:tasks.rados.rados.0.smithi157.stdout:1515: write oid 47 current snap is 0 2024-09-16T03:38:47.583 INFO:tasks.rados.rados.0.smithi157.stdout:1504: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-16T03:38:47.584 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:47.584 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:47.593 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:38:47.654 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2800368 2024-09-16T03:38:47.681 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:47.679+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:47.700 INFO:tasks.rados.rados.0.smithi157.stdout:1515: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-09-16T03:38:47.704 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:47.705 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:47.706 INFO:tasks.rados.rados.0.smithi157.stdout:1515: writing smithi15736696-47 from 544626 to 1244281 tid 1 2024-09-16T03:38:47.713 INFO:tasks.rados.rados.0.smithi157.stdout:1515: writing smithi15736696-47 from 1892215 to 2446644 tid 2 2024-09-16T03:38:47.714 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:38:47.718 INFO:tasks.rados.rados.0.smithi157.stdout:1515: writing smithi15736696-47 from 3150041 to 3344626 tid 3 2024-09-16T03:38:47.718 INFO:tasks.rados.rados.0.smithi157.stdout:1504: done (15 left) 2024-09-16T03:38:47.718 INFO:tasks.rados.rados.0.smithi157.stdout:1516: tier_flush oid 11 2024-09-16T03:38:47.718 INFO:tasks.rados.rados.0.smithi157.stdout:1516: tier_flush oid 11 snap -1 2024-09-16T03:38:47.718 INFO:tasks.rados.rados.0.smithi157.stdout:1511: got (2) No such file or directory 2024-09-16T03:38:47.718 INFO:tasks.rados.rados.0.smithi157.stdout:1511: got expected ENOENT (src dne) 2024-09-16T03:38:47.718 INFO:tasks.rados.rados.0.smithi157.stdout:1495: expect (ObjNum 295 snap 0 seq_num 295) 2024-09-16T03:38:47.726 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1662156 2024-09-16T03:38:47.740 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:47.773 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:47.782 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:47.781+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:47.807 INFO:tasks.rados.rados.0.smithi157.stdout:1496: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:38:47.857 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:47.858 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:47.858 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:47.866 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:38:47.883 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:47.882+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:47.961 INFO:tasks.rados.rados.0.smithi157.stdout:1498: expect (ObjNum 309 snap 0 seq_num 309) 2024-09-16T03:38:47.961 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:47.969 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:47.970 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:47.978 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:38:48.025 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:48.020+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:48.065 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1850668 2024-09-16T03:38:48.080 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:48.081 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:48.085 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:48.083+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:48.089 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:38:48.164 INFO:tasks.rados.rados.0.smithi157.stdout:1506: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-16T03:38:48.164 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 252284 2024-09-16T03:38:48.185 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:48.184+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:48.224 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:48.231 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:48.232 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:48.240 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:38:48.286 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:48.285+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:48.306 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2133042 2024-09-16T03:38:48.328 INFO:tasks.rados.rados.0.smithi157.stdout:1512: got (22) Invalid argument 2024-09-16T03:38:48.328 INFO:tasks.rados.rados.0.smithi157.stdout:1514: finishing write tid 1 to smithi15736696-43 2024-09-16T03:38:48.328 INFO:tasks.rados.rados.0.smithi157.stdout:1514: finishing write tid 2 to smithi15736696-43 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:1514: finishing write tid 3 to smithi15736696-43 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:1502: got (0) Success 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 64 (ObjNum 283 snap 0 seq_num 283) dirty exists 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:1513: finishing write tid 1 to smithi15736696-49 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:1513: finishing write tid 2 to smithi15736696-49 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:1513: finishing write tid 3 to smithi15736696-49 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:1495: done (15 left) 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:1496: done (14 left) 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:1498: done (13 left) 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:1502: done (12 left) 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:1506: done (11 left) 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:1511: done (10 left) 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:1512: done (9 left) 2024-09-16T03:38:48.329 INFO:tasks.rados.rados.0.smithi157.stdout:1517: read oid 27 snap -1 2024-09-16T03:38:48.330 INFO:tasks.rados.rados.0.smithi157.stdout:1517: expect (ObjNum 244 snap 0 seq_num 244) 2024-09-16T03:38:48.330 INFO:tasks.rados.rados.0.smithi157.stdout:1518: set_chunk oid 30 target oid 2024-09-16T03:38:48.330 INFO:tasks.rados.rados.0.smithi157.stdout:1518: set_chunk oid 30 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:48.330 INFO:tasks.rados.rados.0.smithi157.stdout:1519: read oid 1 snap -1 2024-09-16T03:38:48.330 INFO:tasks.rados.rados.0.smithi157.stdout:1519: expect (ObjNum 242 snap 0 seq_num 242) 2024-09-16T03:38:48.330 INFO:tasks.rados.rados.0.smithi157.stdout:1520: write (excl) oid 18 current snap is 0 2024-09-16T03:38:48.330 INFO:tasks.rados.rados.0.smithi157.stdout:1520: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2024-09-16T03:38:48.334 INFO:tasks.rados.rados.0.smithi157.stdout:1520: writing smithi15736696-18 from 561433 to 1336337 tid 1 2024-09-16T03:38:48.340 INFO:tasks.rados.rados.0.smithi157.stdout:1520: writing smithi15736696-18 from 1950697 to 2365137 tid 2 2024-09-16T03:38:48.342 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:48.343 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:48.347 INFO:tasks.rados.rados.0.smithi157.stdout:1520: writing smithi15736696-18 from 2777464 to 3361433 tid 3 2024-09-16T03:38:48.347 INFO:tasks.rados.rados.0.smithi157.stdout:1521: delete oid 19 current snap is 0 2024-09-16T03:38:48.347 INFO:tasks.rados.rados.0.smithi157.stdout:1513: finishing write tid 4 to smithi15736696-49 2024-09-16T03:38:48.347 INFO:tasks.rados.rados.0.smithi157.stdout:1513: finishing write tid 5 to smithi15736696-49 2024-09-16T03:38:48.347 INFO:tasks.rados.rados.0.smithi157.stdout:1513: oid 49 updating version 0 to 69 2024-09-16T03:38:48.347 INFO:tasks.rados.rados.0.smithi157.stdout:1513: oid 49 updating version 69 to 71 2024-09-16T03:38:48.347 INFO:tasks.rados.rados.0.smithi157.stdout:1513: oid 49 updating version 71 to 72 2024-09-16T03:38:48.347 INFO:tasks.rados.rados.0.smithi157.stdout:1513: oid 49 version 72 is already newer than 70 2024-09-16T03:38:48.347 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 72 (ObjNum 316 snap 0 seq_num 316) dirty exists 2024-09-16T03:38:48.347 INFO:tasks.rados.rados.0.smithi157.stdout:1513: left oid 49 (ObjNum 316 snap 0 seq_num 316) 2024-09-16T03:38:48.348 INFO:tasks.rados.rados.0.smithi157.stdout:1516: got (2) No such file or directory 2024-09-16T03:38:48.348 INFO:tasks.rados.rados.0.smithi157.stdout:1516: got expected ENOENT (src dne) 2024-09-16T03:38:48.348 INFO:tasks.rados.rados.0.smithi157.stdout:1509: got (0) Success 2024-09-16T03:38:48.348 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 294 snap 0 seq_num 294) dirty exists 2024-09-16T03:38:48.348 INFO:tasks.rados.rados.0.smithi157.stdout:1514: finishing write tid 4 to smithi15736696-43 2024-09-16T03:38:48.348 INFO:tasks.rados.rados.0.smithi157.stdout:1514: finishing write tid 5 to smithi15736696-43 2024-09-16T03:38:48.348 INFO:tasks.rados.rados.0.smithi157.stdout:1514: oid 43 updating version 0 to 80 2024-09-16T03:38:48.348 INFO:tasks.rados.rados.0.smithi157.stdout:1514: oid 43 updating version 80 to 81 2024-09-16T03:38:48.348 INFO:tasks.rados.rados.0.smithi157.stdout:1514: oid 43 version 81 is already newer than 79 2024-09-16T03:38:48.348 INFO:tasks.rados.rados.0.smithi157.stdout:1514: oid 43 version 81 is already newer than 78 2024-09-16T03:38:48.348 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 81 (ObjNum 317 snap 0 seq_num 317) dirty exists 2024-09-16T03:38:48.348 INFO:tasks.rados.rados.0.smithi157.stdout:1514: left oid 43 (ObjNum 317 snap 0 seq_num 317) 2024-09-16T03:38:48.348 INFO:tasks.rados.rados.0.smithi157.stdout:1503: finishing write tid 1 to smithi15736696-50 2024-09-16T03:38:48.349 INFO:tasks.rados.rados.0.smithi157.stdout:1507: finishing write tid 1 to smithi15736696-31 2024-09-16T03:38:48.349 INFO:tasks.rados.rados.0.smithi157.stdout:1503: finishing write tid 2 to smithi15736696-50 2024-09-16T03:38:48.349 INFO:tasks.rados.rados.0.smithi157.stdout:1507: finishing write tid 2 to smithi15736696-31 2024-09-16T03:38:48.349 INFO:tasks.rados.rados.0.smithi157.stdout:1507: finishing write tid 3 to smithi15736696-31 2024-09-16T03:38:48.349 INFO:tasks.rados.rados.0.smithi157.stdout:1503: finishing write tid 3 to smithi15736696-50 2024-09-16T03:38:48.349 INFO:tasks.rados.rados.0.smithi157.stdout:1499: finishing write tid 1 to smithi15736696-7 2024-09-16T03:38:48.349 INFO:tasks.rados.rados.0.smithi157.stdout:1499: finishing write tid 2 to smithi15736696-7 2024-09-16T03:38:48.349 INFO:tasks.rados.rados.0.smithi157.stdout:1499: finishing write tid 3 to smithi15736696-7 2024-09-16T03:38:48.349 INFO:tasks.rados.rados.0.smithi157.stdout:1503: finishing write tid 4 to smithi15736696-50 2024-09-16T03:38:48.349 INFO:tasks.rados.rados.0.smithi157.stdout:1503: finishing write tid 5 to smithi15736696-50 2024-09-16T03:38:48.349 INFO:tasks.rados.rados.0.smithi157.stdout:1503: oid 50 updating version 0 to 128 2024-09-16T03:38:48.349 INFO:tasks.rados.rados.0.smithi157.stdout:1503: oid 50 updating version 128 to 130 2024-09-16T03:38:48.349 INFO:tasks.rados.rados.0.smithi157.stdout:1503: oid 50 updating version 130 to 131 2024-09-16T03:38:48.350 INFO:tasks.rados.rados.0.smithi157.stdout:1503: oid 50 version 131 is already newer than 129 2024-09-16T03:38:48.350 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 131 (ObjNum 313 snap 0 seq_num 313) dirty exists 2024-09-16T03:38:48.350 INFO:tasks.rados.rados.0.smithi157.stdout:1503: left oid 50 (ObjNum 313 snap 0 seq_num 313) 2024-09-16T03:38:48.350 INFO:tasks.rados.rados.0.smithi157.stdout:1507: finishing write tid 4 to smithi15736696-31 2024-09-16T03:38:48.350 INFO:tasks.rados.rados.0.smithi157.stdout:1507: finishing write tid 5 to smithi15736696-31 2024-09-16T03:38:48.350 INFO:tasks.rados.rados.0.smithi157.stdout:1507: oid 31 updating version 0 to 71 2024-09-16T03:38:48.350 INFO:tasks.rados.rados.0.smithi157.stdout:1507: oid 31 version 71 is already newer than 68 2024-09-16T03:38:48.350 INFO:tasks.rados.rados.0.smithi157.stdout:1507: oid 31 version 71 is already newer than 70 2024-09-16T03:38:48.350 INFO:tasks.rados.rados.0.smithi157.stdout:1507: oid 31 version 71 is already newer than 69 2024-09-16T03:38:48.350 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 71 (ObjNum 314 snap 0 seq_num 314) dirty exists 2024-09-16T03:38:48.350 INFO:tasks.rados.rados.0.smithi157.stdout:1507: left oid 31 (ObjNum 314 snap 0 seq_num 314) 2024-09-16T03:38:48.350 INFO:tasks.rados.rados.0.smithi157.stdout:1499: finishing write tid 4 to smithi15736696-7 2024-09-16T03:38:48.350 INFO:tasks.rados.rados.0.smithi157.stdout:1499: finishing write tid 5 to smithi15736696-7 2024-09-16T03:38:48.351 INFO:tasks.rados.rados.0.smithi157.stdout:1499: oid 7 updating version 0 to 75 2024-09-16T03:38:48.351 INFO:tasks.rados.rados.0.smithi157.stdout:1499: oid 7 updating version 75 to 76 2024-09-16T03:38:48.351 INFO:tasks.rados.rados.0.smithi157.stdout:1499: oid 7 version 76 is already newer than 74 2024-09-16T03:38:48.351 INFO:tasks.rados.rados.0.smithi157.stdout:1499: oid 7 version 76 is already newer than 73 2024-09-16T03:38:48.351 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 76 (ObjNum 311 snap 0 seq_num 311) dirty exists 2024-09-16T03:38:48.351 INFO:tasks.rados.rados.0.smithi157.stdout:1499: left oid 7 (ObjNum 311 snap 0 seq_num 311) 2024-09-16T03:38:48.351 INFO:tasks.rados.rados.0.smithi157.stdout:1515: finishing write tid 1 to smithi15736696-47 2024-09-16T03:38:48.351 INFO:tasks.rados.rados.0.smithi157.stdout:1515: finishing write tid 2 to smithi15736696-47 2024-09-16T03:38:48.351 INFO:tasks.rados.rados.0.smithi157.stdout:1515: finishing write tid 3 to smithi15736696-47 2024-09-16T03:38:48.351 INFO:tasks.rados.rados.0.smithi157.stdout:1515: finishing write tid 4 to smithi15736696-47 2024-09-16T03:38:48.351 INFO:tasks.rados.rados.0.smithi157.stdout:1515: finishing write tid 5 to smithi15736696-47 2024-09-16T03:38:48.352 INFO:tasks.rados.rados.0.smithi157.stdout:1515: oid 47 updating version 0 to 123 2024-09-16T03:38:48.352 INFO:tasks.rados.rados.0.smithi157.stdout:1515: oid 47 version 123 is already newer than 121 2024-09-16T03:38:48.352 INFO:tasks.rados.rados.0.smithi157.stdout:1515: oid 47 version 123 is already newer than 122 2024-09-16T03:38:48.352 INFO:tasks.rados.rados.0.smithi157.stdout:1515: oid 47 version 123 is already newer than 120 2024-09-16T03:38:48.352 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 123 (ObjNum 318 snap 0 seq_num 318) dirty exists 2024-09-16T03:38:48.352 INFO:tasks.rados.rados.0.smithi157.stdout:1515: left oid 47 (ObjNum 318 snap 0 seq_num 318) 2024-09-16T03:38:48.352 INFO:tasks.rados.rados.0.smithi157.stdout:1518: finishing set_chunk to oid 30 2024-09-16T03:38:48.352 INFO:tasks.rados.rados.0.smithi157.stdout:1518: got expected ENOENT (src dne) 2024-09-16T03:38:48.352 INFO:tasks.rados.rados.0.smithi157.stdout:1517: expect (ObjNum 244 snap 0 seq_num 244) 2024-09-16T03:38:48.353 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:38:48.374 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:38:48.418 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:48.417+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:48.452 INFO:tasks.rados.rados.0.smithi157.stdout:1520: finishing write tid 1 to smithi15736696-18 2024-09-16T03:38:48.452 INFO:tasks.rados.rados.0.smithi157.stdout:1520: finishing write tid 2 to smithi15736696-18 2024-09-16T03:38:48.487 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:48.486+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:48.502 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:48.502 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:48.511 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:38:48.531 INFO:tasks.rados.rados.0.smithi157.stdout:1519: expect (ObjNum 242 snap 0 seq_num 242) 2024-09-16T03:38:48.562 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 218839 2024-09-16T03:38:48.594 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:48.593+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:48.642 INFO:tasks.rados.rados.0.smithi157.stdout:1499: done (13 left) 2024-09-16T03:38:48.642 INFO:tasks.rados.rados.0.smithi157.stdout:1503: done (12 left) 2024-09-16T03:38:48.642 INFO:tasks.rados.rados.0.smithi157.stdout:1507: done (11 left) 2024-09-16T03:38:48.642 INFO:tasks.rados.rados.0.smithi157.stdout:1509: done (10 left) 2024-09-16T03:38:48.643 INFO:tasks.rados.rados.0.smithi157.stdout:1513: done (9 left) 2024-09-16T03:38:48.643 INFO:tasks.rados.rados.0.smithi157.stdout:1514: done (8 left) 2024-09-16T03:38:48.643 INFO:tasks.rados.rados.0.smithi157.stdout:1515: done (7 left) 2024-09-16T03:38:48.643 INFO:tasks.rados.rados.0.smithi157.stdout:1516: done (6 left) 2024-09-16T03:38:48.643 INFO:tasks.rados.rados.0.smithi157.stdout:1517: done (5 left) 2024-09-16T03:38:48.643 INFO:tasks.rados.rados.0.smithi157.stdout:1518: done (4 left) 2024-09-16T03:38:48.643 INFO:tasks.rados.rados.0.smithi157.stdout:1519: done (3 left) 2024-09-16T03:38:48.643 INFO:tasks.rados.rados.0.smithi157.stdout:1521: done (2 left) 2024-09-16T03:38:48.643 INFO:tasks.rados.rados.0.smithi157.stdout:1522: set_chunk oid 12 target oid 2024-09-16T03:38:48.645 INFO:tasks.rados.rados.0.smithi157.stdout:1522: set_chunk oid 12 offset: 2344448 length: 165376 target oid 31da1b6fa9cebad627ea35c5604ae731063d104535c738dbe0eb206eb7a05dce offset: 0 snap -1 2024-09-16T03:38:48.645 INFO:tasks.rados.rados.0.smithi157.stdout:1523: read oid 3 snap -1 2024-09-16T03:38:48.645 INFO:tasks.rados.rados.0.smithi157.stdout:1523: expect deleted 2024-09-16T03:38:48.646 INFO:tasks.rados.rados.0.smithi157.stdout:1524: read oid 38 snap -1 2024-09-16T03:38:48.646 INFO:tasks.rados.rados.0.smithi157.stdout:1524: expect deleted 2024-09-16T03:38:48.646 INFO:tasks.rados.rados.0.smithi157.stdout:1525: write (excl) oid 17 current snap is 0 2024-09-16T03:38:48.646 INFO:tasks.rados.rados.0.smithi157.stdout:1525: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-09-16T03:38:48.651 INFO:tasks.rados.rados.0.smithi157.stdout:1525: writing smithi15736696-17 from 595047 to 1236802 tid 1 2024-09-16T03:38:48.653 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:48.654 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:48.656 INFO:tasks.rados.rados.0.smithi157.stdout:1525: writing smithi15736696-17 from 1667661 to 2085770 tid 2 2024-09-16T03:38:48.662 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:38:48.663 INFO:tasks.rados.rados.0.smithi157.stdout:1525: writing smithi15736696-17 from 2548663 to 3311685 tid 3 2024-09-16T03:38:48.663 INFO:tasks.rados.rados.0.smithi157.stdout:1526: set_chunk oid 25 target oid 2024-09-16T03:38:48.690 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:48.688+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:48.768 INFO:tasks.rados.rados.0.smithi157.stdout:1526: set_chunk oid 25 offset: 1969664 length: 250880 target oid 7d32ec853a23b571de16fb0e2109ba1e64f8e2118f8fdf8866dcc5d80f2f1d8f offset: 0 snap -1 2024-09-16T03:38:48.768 INFO:tasks.rados.rados.0.smithi157.stdout:1527: set_chunk oid 31 target oid 2024-09-16T03:38:48.791 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:48.790+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:48.807 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:48.807 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:48.815 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:38:48.892 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:48.891+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:48.919 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:48.919 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:48.928 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:38:49.012 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:49.010+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:49.071 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:49.071 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:49.073 INFO:tasks.rados.rados.0.smithi157.stdout:1527: set_chunk oid 31 offset: 871424 length: 423424 target oid 22b7bfe24ac52e6ea78a288ad4512e77b9dfbdafc16883b3d764a683e6cab0b4 offset: 0 snap -1 2024-09-16T03:38:49.073 INFO:tasks.rados.rados.0.smithi157.stdout:1528: set_chunk oid 44 target oid 2024-09-16T03:38:49.079 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:38:49.094 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:49.092+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:49.194 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:49.193+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:49.223 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:49.223 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:49.231 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:38:49.278 INFO:tasks.rados.rados.0.smithi157.stdout:1528: set_chunk oid 44 offset: 2283008 length: 94208 target oid dbf2fb242ade52f89f984f60d94b0a752f94820fc720f6f2b4534714e8e1c20d offset: 0 snap -1 2024-09-16T03:38:49.278 INFO:tasks.rados.rados.0.smithi157.stdout:1529: read oid 34 snap -1 2024-09-16T03:38:49.278 INFO:tasks.rados.rados.0.smithi157.stdout:1529: expect (ObjNum 303 snap 0 seq_num 303) 2024-09-16T03:38:49.278 INFO:tasks.rados.rados.0.smithi157.stdout:1530: read oid 37 snap -1 2024-09-16T03:38:49.278 INFO:tasks.rados.rados.0.smithi157.stdout:1530: expect (ObjNum 283 snap 0 seq_num 283) 2024-09-16T03:38:49.278 INFO:tasks.rados.rados.0.smithi157.stdout:1531: write (excl) oid 39 current snap is 0 2024-09-16T03:38:49.279 INFO:tasks.rados.rados.0.smithi157.stdout:1525: finishing write tid 1 to smithi15736696-17 2024-09-16T03:38:49.279 INFO:tasks.rados.rados.0.smithi157.stdout:1531: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-09-16T03:38:49.284 INFO:tasks.rados.rados.0.smithi157.stdout:1531: writing smithi15736696-39 from 611854 to 1328858 tid 1 2024-09-16T03:38:49.291 INFO:tasks.rados.rados.0.smithi157.stdout:1531: writing smithi15736696-39 from 2126143 to 2687910 tid 2 2024-09-16T03:38:49.295 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:49.293+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:49.296 INFO:tasks.rados.rados.0.smithi157.stdout:1531: writing smithi15736696-39 from 3259733 to 3411854 tid 3 2024-09-16T03:38:49.296 INFO:tasks.rados.rados.0.smithi157.stdout:1532: write (excl) oid 27 current snap is 0 2024-09-16T03:38:49.296 INFO:tasks.rados.rados.0.smithi157.stdout:1532: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-09-16T03:38:49.303 INFO:tasks.rados.rados.0.smithi157.stdout:1532: writing smithi15736696-27 from 628661 to 1420914 tid 1 2024-09-16T03:38:49.309 INFO:tasks.rados.rados.0.smithi157.stdout:1532: writing smithi15736696-27 from 2184625 to 2606403 tid 2 2024-09-16T03:38:49.314 INFO:tasks.rados.rados.0.smithi157.stdout:1532: writing smithi15736696-27 from 3403509 to 3428661 tid 3 2024-09-16T03:38:49.314 INFO:tasks.rados.rados.0.smithi157.stdout:1533: read oid 13 snap -1 2024-09-16T03:38:49.314 INFO:tasks.rados.rados.0.smithi157.stdout:1533: expect (ObjNum 304 snap 0 seq_num 304) 2024-09-16T03:38:49.314 INFO:tasks.rados.rados.0.smithi157.stdout:1534: read oid 40 snap -1 2024-09-16T03:38:49.314 INFO:tasks.rados.rados.0.smithi157.stdout:1534: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:38:49.314 INFO:tasks.rados.rados.0.smithi157.stdout:1535: write (excl) oid 19 current snap is 0 2024-09-16T03:38:49.314 INFO:tasks.rados.rados.0.smithi157.stdout:1535: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-09-16T03:38:49.320 INFO:tasks.rados.rados.0.smithi157.stdout:1535: writing smithi15736696-19 from 645468 to 1112970 tid 1 2024-09-16T03:38:49.326 INFO:tasks.rados.rados.0.smithi157.stdout:1535: writing smithi15736696-19 from 1726754 to 2292190 tid 2 2024-09-16T03:38:49.334 INFO:tasks.rados.rados.0.smithi157.stdout:1535: writing smithi15736696-19 from 2798226 to 3395064 tid 3 2024-09-16T03:38:49.334 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:49.334 INFO:tasks.rados.rados.0.smithi157.stdout:1526: finishing set_chunk to oid 25 2024-09-16T03:38:49.334 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk returned error code -95 offset: 1969664 length: 250880 2024-09-16T03:38:49.334 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 295 snap 0 seq_num 295) dirty exists 2024-09-16T03:38:49.334 INFO:tasks.rados.rados.0.smithi157.stdout:1520: finishing write tid 3 to smithi15736696-18 2024-09-16T03:38:49.334 INFO:tasks.rados.rados.0.smithi157.stdout:1520: finishing write tid 4 to smithi15736696-18 2024-09-16T03:38:49.334 INFO:tasks.rados.rados.0.smithi157.stdout:1523: done (15 left) 2024-09-16T03:38:49.334 INFO:tasks.rados.rados.0.smithi157.stdout:1526: done (14 left) 2024-09-16T03:38:49.334 INFO:tasks.rados.rados.0.smithi157.stdout:1536: read oid 25 snap -1 2024-09-16T03:38:49.334 INFO:tasks.rados.rados.0.smithi157.stdout:1536: expect (ObjNum 295 snap 0 seq_num 295) 2024-09-16T03:38:49.335 INFO:tasks.rados.rados.0.smithi157.stdout:1537: read oid 22 snap -1 2024-09-16T03:38:49.335 INFO:tasks.rados.rados.0.smithi157.stdout:1537: expect deleted 2024-09-16T03:38:49.335 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:49.335 INFO:tasks.rados.rados.0.smithi157.stdout:1520: finishing write tid 5 to smithi15736696-18 2024-09-16T03:38:49.335 INFO:tasks.rados.rados.0.smithi157.stdout:1520: oid 18 updating version 0 to 61 2024-09-16T03:38:49.335 INFO:tasks.rados.rados.0.smithi157.stdout:1520: oid 18 updating version 61 to 63 2024-09-16T03:38:49.335 INFO:tasks.rados.rados.0.smithi157.stdout:1520: oid 18 version 63 is already newer than 62 2024-09-16T03:38:49.335 INFO:tasks.rados.rados.0.smithi157.stdout:1520: oid 18 version 63 is already newer than 60 2024-09-16T03:38:49.335 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 63 (ObjNum 319 snap 0 seq_num 319) dirty exists 2024-09-16T03:38:49.335 INFO:tasks.rados.rados.0.smithi157.stdout:1520: left oid 18 (ObjNum 319 snap 0 seq_num 319) 2024-09-16T03:38:49.336 INFO:tasks.rados.rados.0.smithi157.stdout:1525: finishing write tid 2 to smithi15736696-17 2024-09-16T03:38:49.336 INFO:tasks.rados.rados.0.smithi157.stdout:1525: finishing write tid 3 to smithi15736696-17 2024-09-16T03:38:49.336 INFO:tasks.rados.rados.0.smithi157.stdout:1525: finishing write tid 4 to smithi15736696-17 2024-09-16T03:38:49.336 INFO:tasks.rados.rados.0.smithi157.stdout:1520: done (15 left) 2024-09-16T03:38:49.336 INFO:tasks.rados.rados.0.smithi157.stdout:1538: tier_promote oid 10 2024-09-16T03:38:49.336 INFO:tasks.rados.rados.0.smithi157.stdout:1522: finishing set_chunk to oid 12 2024-09-16T03:38:49.336 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 15 (ObjNum 297 snap 0 seq_num 297) dirty exists 2024-09-16T03:38:49.336 INFO:tasks.rados.rados.0.smithi157.stdout:1525: finishing write tid 5 to smithi15736696-17 2024-09-16T03:38:49.336 INFO:tasks.rados.rados.0.smithi157.stdout:1525: oid 17 updating version 0 to 121 2024-09-16T03:38:49.336 INFO:tasks.rados.rados.0.smithi157.stdout:1525: oid 17 version 121 is already newer than 120 2024-09-16T03:38:49.336 INFO:tasks.rados.rados.0.smithi157.stdout:1525: oid 17 version 121 is already newer than 119 2024-09-16T03:38:49.336 INFO:tasks.rados.rados.0.smithi157.stdout:1525: oid 17 updating version 121 to 122 2024-09-16T03:38:49.337 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 122 (ObjNum 321 snap 0 seq_num 321) dirty exists 2024-09-16T03:38:49.337 INFO:tasks.rados.rados.0.smithi157.stdout:1525: left oid 17 (ObjNum 321 snap 0 seq_num 321) 2024-09-16T03:38:49.337 INFO:tasks.rados.rados.0.smithi157.stdout:1510: finishing write tid 1 to smithi15736696-26 2024-09-16T03:38:49.337 INFO:tasks.rados.rados.0.smithi157.stdout:1522: done (15 left) 2024-09-16T03:38:49.337 INFO:tasks.rados.rados.0.smithi157.stdout:1525: done (14 left) 2024-09-16T03:38:49.337 INFO:tasks.rados.rados.0.smithi157.stdout:1539: read oid 5 snap -1 2024-09-16T03:38:49.337 INFO:tasks.rados.rados.0.smithi157.stdout:1539: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-16T03:38:49.338 INFO:tasks.rados.rados.0.smithi157.stdout:1540: write (excl) oid 45 current snap is 0 2024-09-16T03:38:49.338 INFO:tasks.rados.rados.0.smithi157.stdout:1510: finishing write tid 2 to smithi15736696-26 2024-09-16T03:38:49.338 INFO:tasks.rados.rados.0.smithi157.stdout:1510: finishing write tid 3 to smithi15736696-26 2024-09-16T03:38:49.338 INFO:tasks.rados.rados.0.smithi157.stdout:1540: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2024-09-16T03:38:49.339 INFO:tasks.rados.rados.0.smithi157.stdout:1540: writing smithi15736696-45 from 662275 to 1205026 tid 1 2024-09-16T03:38:49.344 INFO:tasks.rados.rados.0.smithi157.stdout:1540: writing smithi15736696-45 from 1785236 to 2210683 tid 2 2024-09-16T03:38:49.345 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:49.345 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:49.351 INFO:tasks.rados.rados.0.smithi157.stdout:1540: writing smithi15736696-45 from 2942002 to 3462275 tid 3 2024-09-16T03:38:49.351 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:49.351 INFO:tasks.rados.rados.0.smithi157.stdout:1527: finishing set_chunk to oid 31 2024-09-16T03:38:49.351 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 71 (ObjNum 314 snap 0 seq_num 314) dirty exists 2024-09-16T03:38:49.351 INFO:tasks.rados.rados.0.smithi157.stdout:1510: finishing write tid 4 to smithi15736696-26 2024-09-16T03:38:49.352 INFO:tasks.rados.rados.0.smithi157.stdout:1527: done (15 left) 2024-09-16T03:38:49.352 INFO:tasks.rados.rados.0.smithi157.stdout:1541: set_chunk oid 24 target oid 2024-09-16T03:38:49.354 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:38:49.362 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:38:49.395 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:49.394+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:49.433 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:49.432+0000 7f12e2a1a640 -1 osd.5 114 handle_osd_map injecting map crc failure 2024-09-16T03:38:49.442 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 9] out_osds: [1, 5, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:38:49.442 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:38:49.442 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:38:49.442 INFO:tasks.thrashosds.thrasher:Reweighting osd 9 to 0.1266657275128433 2024-09-16T03:38:49.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 0.1266657275128433 2024-09-16T03:38:49.462 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:49.462 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:49.472 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:38:49.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:38:49.516 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:49.515+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:49.578 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:49.578 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:49.587 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:38:49.597 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:49.596+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:49.638 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:49.638 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:49.655 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:49.655 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:49.672 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:49.672 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:49.689 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:49.689 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:49.698 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:49.697+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:49.704 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:49.705 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:49.721 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:49.721 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:49.732 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:49.732 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:49.737 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:49.737 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:49.741 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:38:49.754 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:49.754 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:49.770 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:49.770 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:49.787 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:49.787 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:49.799 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:49.798+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:49.803 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:49.804 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:49.821 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:49.821 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:49.837 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:49.837 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:49.845 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:49.845 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:49.845 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:49.846 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:49.846 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:49.848 INFO:tasks.rados.rados.0.smithi157.stdout:1541: set_chunk oid 24 offset: 217088 length: 21504 target oid 747327a0bf031a7444e0a2fafa775284daf571e025125260806741cf3752fd11 offset: 0 snap -1 2024-09-16T03:38:49.848 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:49.848 INFO:tasks.rados.rados.0.smithi157.stdout:1510: finishing write tid 5 to smithi15736696-26 2024-09-16T03:38:49.848 INFO:tasks.rados.rados.0.smithi157.stdout:1510: oid 26 updating version 0 to 101 2024-09-16T03:38:49.848 INFO:tasks.rados.rados.0.smithi157.stdout:1510: oid 26 version 101 is already newer than 99 2024-09-16T03:38:49.848 INFO:tasks.rados.rados.0.smithi157.stdout:1510: oid 26 version 101 is already newer than 100 2024-09-16T03:38:49.848 INFO:tasks.rados.rados.0.smithi157.stdout:1510: oid 26 version 101 is already newer than 98 2024-09-16T03:38:49.848 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 101 (ObjNum 315 snap 0 seq_num 315) dirty exists 2024-09-16T03:38:49.849 INFO:tasks.rados.rados.0.smithi157.stdout:1510: left oid 26 (ObjNum 315 snap 0 seq_num 315) 2024-09-16T03:38:49.849 INFO:tasks.rados.rados.0.smithi157.stdout:1510: done (15 left) 2024-09-16T03:38:49.849 INFO:tasks.rados.rados.0.smithi157.stdout:1542: read oid 14 snap -1 2024-09-16T03:38:49.849 INFO:tasks.rados.rados.0.smithi157.stdout:1542: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-16T03:38:49.849 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:49.849 INFO:tasks.rados.rados.0.smithi157.stdout:1529: expect (ObjNum 303 snap 0 seq_num 303) 2024-09-16T03:38:49.853 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:49.853 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:49.855 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:38:49.870 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:49.870 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:49.886 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:49.887 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:49.901 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:49.900+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:49.960 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:49.961 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:49.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:49.961 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:49.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.961 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:779 3.1 3:abdaa5c9:::2782d12a88f9842f7f754fca32c4222cf10fb780cd136830d32faaa700e687ba:head [write 0~45056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:49.961 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:38:49.961 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.471224012999997, 2024-09-16T03:38:49.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30397390600000002, 2024-09-16T03:38:49.961 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:49.961 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:49.961 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:49.961 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 779 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486580+0000", 2024-09-16T03:38:49.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486583+0000", 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.762e-06 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486649+0000", 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6143999999999996e-05 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:49.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486669+0000", 2024-09-16T03:38:49.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9837e-05 2024-09-16T03:38:49.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:49.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486681+0000", 2024-09-16T03:38:49.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2194999999999999e-05 2024-09-16T03:38:49.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:49.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486746+0000", 2024-09-16T03:38:49.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5480000000000003e-05 2024-09-16T03:38:49.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.965 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op must wait for map", 2024-09-16T03:38:49.965 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.486752+0000", 2024-09-16T03:38:49.965 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4659999999999998e-06 2024-09-16T03:38:49.965 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.965 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:49.965 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588316+0000", 2024-09-16T03:38:49.965 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10156393399999999 2024-09-16T03:38:49.965 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.965 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:49.965 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588539+0000", 2024-09-16T03:38:49.965 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022308300000000001 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.588647+0000", 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000108047 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.589516+0000", 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086926899999999999 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:49.966 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.789989+0000", 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20047296100000001 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790478+0000", 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048882099999999996 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790518+0000", 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0343000000000003e-05 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:49.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.790554+0000", 2024-09-16T03:38:49.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5564e-05 2024-09-16T03:38:49.968 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:49.968 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:49.968 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:49.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.968 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:39 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:49.968 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:38:49.968 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 83.118106604999994, 2024-09-16T03:38:49.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40113680400000001, 2024-09-16T03:38:49.968 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:49.968 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:49.969 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:49.969 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:49.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.969 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:49.969 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:38:49.969 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:49.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.969 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:49.969 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:38:49.969 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:49.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.970 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:49.970 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839700+0000", 2024-09-16T03:38:49.970 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.571e-06 2024-09-16T03:38:49.970 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.970 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.970 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:49.970 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839704+0000", 2024-09-16T03:38:49.970 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8399999999999997e-06 2024-09-16T03:38:49.970 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.970 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.970 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:49.970 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839712+0000", 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9990000000000008e-06 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839720+0000", 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0099999999999995e-06 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839783+0000", 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3014000000000003e-05 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.971 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839870+0000", 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7392999999999995e-05 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840049+0000", 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017825800000000001 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840874+0000", 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082532099999999995 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.040285+0000", 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19941088400000001 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240748+0000", 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20046322699999999 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240786+0000", 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8158000000000003e-05 2024-09-16T03:38:49.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240834+0000", 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8128999999999999e-05 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:983 3.1 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head [write 0~2138112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 79.894203665000006, 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30819799799999997, 2024-09-16T03:38:49.974 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:49.975 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:49.975 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:49.975 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:49.975 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:49.975 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:49.975 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 983 2024-09-16T03:38:49.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.975 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:49.975 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.975 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:49.975 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:38:49.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:49.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063610+0000", 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2990000000000002e-06 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065735+0000", 2024-09-16T03:38:49.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021257799999999999 2024-09-16T03:38:49.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.977 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:49.977 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065752+0000", 2024-09-16T03:38:49.977 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6895999999999999e-05 2024-09-16T03:38:49.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.977 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:49.977 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065762+0000", 2024-09-16T03:38:49.977 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6830000000000003e-06 2024-09-16T03:38:49.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.977 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065871+0000", 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010904900000000001 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066005+0000", 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013369799999999999 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066135+0000", 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000130481 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.069822+0000", 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036863759999999999 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272370+0000", 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20254802599999999 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371671+0000", 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099300893000000001 2024-09-16T03:38:49.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371754+0000", 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3075e-05 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371798+0000", 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4742e-05 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.980 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1515 3.1 3:9c9e970c:::7249cb65b52cadf4fe8f4ec923c7e548d1de7d9628823acd2c8de19121f1c1ce:head [write 0~1094656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 65.008077928000006, 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.305958065, 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1515 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:49.981 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949729+0000", 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.785e-06 2024-09-16T03:38:49.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950933+0000", 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001204341 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950950+0000", 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6838e-05 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950960+0000", 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0316000000000001e-05 2024-09-16T03:38:49.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951065+0000", 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104128 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951202+0000", 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013706600000000001 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951350+0000", 2024-09-16T03:38:49.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000148197 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.953857+0000", 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0025067150000000001 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.958049+0000", 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0041924650000000003 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:49.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255547+0000", 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29749841900000001 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255649+0000", 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101513 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255684+0000", 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5281999999999999e-05 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:49.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1704 3.1 3:b6a94245:::f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99:head [write 0~453120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 59.164966573000001, 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40403313699999999, 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1704 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.987 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:49.988 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:49.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:38:49.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:49.988 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.988 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:49.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:38:49.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:49.988 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.988 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:49.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792841+0000", 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.433e-06 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793329+0000", 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048839200000000004 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793345+0000", 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6079e-05 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793354+0000", 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.32e-06 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793418+0000", 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3944999999999998e-05 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793549+0000", 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013159900000000001 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793666+0000", 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116532 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.797707+0000", 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040409629999999998 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897288+0000", 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099581449000000002 2024-09-16T03:38:49.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196775+0000", 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948696800000002 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196814+0000", 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8336e-05 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196871+0000", 2024-09-16T03:38:49.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7120999999999997e-05 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:84 3.1 3:b24f289d:::e84532c28f467d2f2a6e2cf44904fa0d4a555e1304bbe1b25db645af42152bd7:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 59.156194634000002, 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.395455581, 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:49.993 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801613+0000", 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9559999999999999e-06 2024-09-16T03:38:49.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802117+0000", 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050401600000000003 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802141+0000", 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3618e-05 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802155+0000", 2024-09-16T03:38:49.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4358e-05 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802229+0000", 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3930999999999994e-05 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802355+0000", 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012527299999999999 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:49.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802567+0000", 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021250199999999999 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806218+0000", 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036512189999999998 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897323+0000", 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091104892000000007 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196959+0000", 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299635971 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197021+0000", 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1938000000000002e-05 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197065+0000", 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3906999999999998e-05 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:49.998 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:85 3.1 3:b380f269:::e65351e8a65a0b1a517e3b1a85e04b84c0ab7cfa6655f11fb0da31f56b4dd9af:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 59.155615736999998, 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39502727300000001, 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:38:49.999 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802190+0000", 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2419999999999999e-06 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802464+0000", 2024-09-16T03:38:50.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027463699999999999 2024-09-16T03:38:50.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.001 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.001 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802478+0000", 2024-09-16T03:38:50.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3375e-05 2024-09-16T03:38:50.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.001 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.001 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802484+0000", 2024-09-16T03:38:50.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0990000000000004e-06 2024-09-16T03:38:50.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803475+0000", 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00099113099999999991 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803651+0000", 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017654300000000001 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803923+0000", 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000271842 2024-09-16T03:38:50.002 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806228+0000", 2024-09-16T03:38:50.003 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023049590000000001 2024-09-16T03:38:50.003 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897563+0000", 2024-09-16T03:38:50.003 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091334724000000006 2024-09-16T03:38:50.003 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197123+0000", 2024-09-16T03:38:50.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299559992 2024-09-16T03:38:50.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197177+0000", 2024-09-16T03:38:50.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4447999999999998e-05 2024-09-16T03:38:50.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197216+0000", 2024-09-16T03:38:50.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8281000000000002e-05 2024-09-16T03:38:50.004 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.005 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.005 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.005 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.005 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:86 3.1 3:b3acb53e:::e951eb69ce5151a3f54107fa22c70f80729eff845da1613447a91536a24046e8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:50.005 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:38:50.005 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 59.155308642000001, 2024-09-16T03:38:50.005 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.394851071, 2024-09-16T03:38:50.005 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.005 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.005 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.005 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.005 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802496+0000", 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:38:50.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802782+0000", 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00028523600000000002 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802796+0000", 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4594e-05 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802804+0000", 2024-09-16T03:38:50.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3790000000000006e-06 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804787+0000", 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019825060000000002 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804895+0000", 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010813499999999999 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.805046+0000", 2024-09-16T03:38:50.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150972 2024-09-16T03:38:50.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806681+0000", 2024-09-16T03:38:50.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016353839999999999 2024-09-16T03:38:50.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.005712+0000", 2024-09-16T03:38:50.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19903093199999999 2024-09-16T03:38:50.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197278+0000", 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.191565295 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197315+0000", 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6946000000000002e-05 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197347+0000", 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1863000000000003e-05 2024-09-16T03:38:50.010 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:140 3.1 3:845e3b2c:::21d7b947791b34f7e3fcd39ad119c9f923ad2f3cc7a703501eac7a68189d5437:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 33.843129388000001, 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39914108999999998, 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:38:50.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:38:50.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114677+0000", 2024-09-16T03:38:50.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1610000000000001e-06 2024-09-16T03:38:50.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114684+0000", 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3150000000000003e-06 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114699+0000", 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4996e-05 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114705+0000", 2024-09-16T03:38:50.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0580000000000003e-06 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.214862+0000", 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015700199999999 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.214922+0000", 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0053999999999998e-05 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215023+0000", 2024-09-16T03:38:50.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100432 2024-09-16T03:38:50.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.216100+0000", 2024-09-16T03:38:50.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010776819999999999 2024-09-16T03:38:50.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.499919+0000", 2024-09-16T03:38:50.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.28381845900000002 2024-09-16T03:38:50.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513755+0000", 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013836477 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513790+0000", 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4295999999999998e-05 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513816+0000", 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6157999999999999e-05 2024-09-16T03:38:50.016 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:141 3.1 3:9387c678:::99a87d3b555b50e905a1ff55d99eb4aa025bd0070f07a4910d03cf1af95bbc44:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 33.843078691000002, 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50160030300000003, 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.017 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114726+0000", 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4399999999999999e-07 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114730+0000", 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1479999999999999e-06 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114738+0000", 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3200000000000002e-06 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114743+0000", 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.891e-06 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215320+0000", 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100576463 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215376+0000", 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6310999999999997e-05 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215473+0000", 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7103999999999997e-05 2024-09-16T03:38:50.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.216108+0000", 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00063422800000000003 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513851+0000", 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29774376400000002 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616086+0000", 2024-09-16T03:38:50.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10223441 2024-09-16T03:38:50.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616240+0000", 2024-09-16T03:38:50.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015453799999999999 2024-09-16T03:38:50.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616326+0000", 2024-09-16T03:38:50.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5382000000000003e-05 2024-09-16T03:38:50.022 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.022 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.022 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:142 3.1 3:a93b7b74:::70cd8f47898d416018aded7de9c29e0fe002b2336cb6cfabbf3eb6238766c3b1:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 33.843049303999997, 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50166088799999997, 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3600000000000003e-07 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114762+0000", 2024-09-16T03:38:50.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6039999999999998e-06 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114769+0000", 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5939999999999999e-06 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114774+0000", 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2179999999999998e-06 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215687+0000", 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100912974 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215733+0000", 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6419999999999999e-05 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215836+0000", 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102981 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.026 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316766+0000", 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100929693 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513877+0000", 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19711067800000001 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616358+0000", 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10248180699999999 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616391+0000", 2024-09-16T03:38:50.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2551000000000001e-05 2024-09-16T03:38:50.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616416+0000", 2024-09-16T03:38:50.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4732e-05 2024-09-16T03:38:50.028 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.028 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.028 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:143 3.1 3:ac848b49:::adc4f0c8a4d0bc7e52f9d5c3b4981447737a7dbe58b8313a369b04383e10893f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 33.843019298000002, 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60333394699999998, 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7899999999999999e-07 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114788+0000", 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4569999999999999e-06 2024-09-16T03:38:50.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114792+0000", 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5730000000000001e-06 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114796+0000", 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5449999999999999e-06 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316237+0000", 2024-09-16T03:38:50.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20144099300000001 2024-09-16T03:38:50.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316417+0000", 2024-09-16T03:38:50.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017981099999999999 2024-09-16T03:38:50.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316564+0000", 2024-09-16T03:38:50.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014693700000000001 2024-09-16T03:38:50.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316993+0000", 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042985499999999998 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.717965+0000", 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40097123699999998 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718005+0000", 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0083000000000002e-05 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718064+0000", 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9020000000000001e-05 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718119+0000", 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4957000000000003e-05 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.034 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:149 3.1 3:8eb88b02:::fa73f0ab3a7395fcfc19219899ff8c0a4a3c96caf8327f9da9656e2450435b27:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 32.829110649999997, 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.348655887, 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128698+0000", 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.527e-06 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128714+0000", 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6019e-05 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128757+0000", 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3318000000000001e-05 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128764+0000", 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.28e-06 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.228916+0000", 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015252199999999 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.228976+0000", 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0390999999999998e-05 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.229077+0000", 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010053 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.274759+0000", 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.045682404000000003 2024-09-16T03:38:50.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.330314+0000", 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.055554916000000003 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477196+0000", 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14688130099999999 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477291+0000", 2024-09-16T03:38:50.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5144999999999998e-05 2024-09-16T03:38:50.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.040 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.040 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477349+0000", 2024-09-16T03:38:50.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8533999999999999e-05 2024-09-16T03:38:50.040 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.040 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.040 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.040 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:168 3.1 3:a655680f:::65b183ceded8ac2b61dbe9ba8fb4fd4f9f3a4854c219da4322bb1e81ebb7e2e5:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:50.040 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.648756+0000", 2024-09-16T03:38:50.040 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.309047627999998, 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30591469500000001, 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648756+0000", 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648756+0000", 2024-09-16T03:38:50.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648757+0000", 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7299999999999995e-07 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648799+0000", 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.227e-05 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648805+0000", 2024-09-16T03:38:50.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4790000000000004e-06 2024-09-16T03:38:50.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648809+0000", 2024-09-16T03:38:50.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7890000000000001e-06 2024-09-16T03:38:50.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.650787+0000", 2024-09-16T03:38:50.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019778700000000001 2024-09-16T03:38:50.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.650914+0000", 2024-09-16T03:38:50.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012707899999999999 2024-09-16T03:38:50.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651123+0000", 2024-09-16T03:38:50.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020929100000000001 2024-09-16T03:38:50.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.652287+0000", 2024-09-16T03:38:50.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011644859999999999 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.653716+0000", 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014290889999999999 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954577+0000", 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30086063800000001 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954621+0000", 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3820999999999999e-05 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954671+0000", 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0210000000000002e-05 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:169 3.1 3:8ca52973:::737e1241b430663b0c460fc4c0e6bf0a2c97ca3f584ee4e3541536e0014b01da:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.648814+0000", 2024-09-16T03:38:50.046 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.308989802999999, 2024-09-16T03:38:50.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30599283399999999, 2024-09-16T03:38:50.047 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.047 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.047 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.047 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648814+0000", 2024-09-16T03:38:50.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648814+0000", 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648815+0000", 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9900000000000001e-07 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648861+0000", 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6786999999999998e-05 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648867+0000", 2024-09-16T03:38:50.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1830000000000002e-06 2024-09-16T03:38:50.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648872+0000", 2024-09-16T03:38:50.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9169999999999997e-06 2024-09-16T03:38:50.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651624+0000", 2024-09-16T03:38:50.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027522150000000001 2024-09-16T03:38:50.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651689+0000", 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5065000000000001e-05 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651814+0000", 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012493199999999999 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.653881+0000", 2024-09-16T03:38:50.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020671830000000002 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.752238+0000", 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098357336000000004 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954726+0000", 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20248775199999999 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954780+0000", 2024-09-16T03:38:50.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3859000000000001e-05 2024-09-16T03:38:50.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954807+0000", 2024-09-16T03:38:50.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7206e-05 2024-09-16T03:38:50.052 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.052 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.052 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.052 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:171 3.1 3:93178f8f:::777eb2bf7c36938cba5735a17f90bb453157d7a9a42a0a6f52227fe7530ec353:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:38:50.052 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.648876+0000", 2024-09-16T03:38:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.308927757999999, 2024-09-16T03:38:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30603207999999998, 2024-09-16T03:38:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648876+0000", 2024-09-16T03:38:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648876+0000", 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648877+0000", 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8799999999999998e-07 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648918+0000", 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1488e-05 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648923+0000", 2024-09-16T03:38:50.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0320000000000003e-06 2024-09-16T03:38:50.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648926+0000", 2024-09-16T03:38:50.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1690000000000001e-06 2024-09-16T03:38:50.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.652727+0000", 2024-09-16T03:38:50.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038005909999999999 2024-09-16T03:38:50.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.652854+0000", 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012661700000000001 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.653069+0000", 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021573799999999999 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.654043+0000", 2024-09-16T03:38:50.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00097351999999999996 2024-09-16T03:38:50.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.854187+0000", 2024-09-16T03:38:50.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20014437400000001 2024-09-16T03:38:50.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954834+0000", 2024-09-16T03:38:50.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100646293 2024-09-16T03:38:50.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954875+0000", 2024-09-16T03:38:50.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1637999999999997e-05 2024-09-16T03:38:50.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954908+0000", 2024-09-16T03:38:50.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3232000000000003e-05 2024-09-16T03:38:50.058 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.058 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.058 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.058 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:139 3.1 3:bd600e93:::b81929c6ed6df2fe181809e10b331bcd3c2df1a1077e06024548babcdcd8f550:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 12.916230537000001, 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.43180016300000001, 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.059 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2300000000000001e-07 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041700+0000", 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000126367 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041707+0000", 2024-09-16T03:38:50.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1519999999999997e-06 2024-09-16T03:38:50.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041710+0000", 2024-09-16T03:38:50.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7869999999999999e-06 2024-09-16T03:38:50.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043365+0000", 2024-09-16T03:38:50.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001654914 2024-09-16T03:38:50.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.065 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.065 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043453+0000", 2024-09-16T03:38:50.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7954000000000005e-05 2024-09-16T03:38:50.065 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.065 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.065 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043596+0000", 2024-09-16T03:38:50.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014304500000000001 2024-09-16T03:38:50.065 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.065 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.145131+0000", 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10153440599999999 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.344273+0000", 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19914232200000001 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473290+0000", 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12901701900000001 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473325+0000", 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4656000000000002e-05 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473374+0000", 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9017999999999997e-05 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:144 3.1 3:91c40fc0:::f2d8823355177e50718038dafd7a8313c1a25b0231461a8672e684a7d4e587b4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:38:50.067 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 12.91608628, 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.431801096, 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:38:50.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:38:50.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.75e-07 2024-09-16T03:38:50.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042077+0000", 2024-09-16T03:38:50.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035865099999999998 2024-09-16T03:38:50.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042084+0000", 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2400000000000001e-06 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042088+0000", 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1570000000000002e-06 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144179+0000", 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10209033300000001 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144427+0000", 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000247879 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144656+0000", 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022919800000000001 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.145866+0000", 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001210046 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.344293+0000", 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.198427353 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473432+0000", 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12913908800000001 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473478+0000", 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6165000000000003e-05 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473519+0000", 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0510999999999999e-05 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:165 3.1 3:8166b007:::3555453b0ab6ec5b2f29a1e76cc26fdaf7f55d077a227627cdf2200eb9ca342d:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 8.130569478, 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40346431100000002, 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.073 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5999999999999999e-07 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827239+0000", 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8920000000000002e-06 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827243+0000", 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2760000000000002e-06 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827246+0000", 2024-09-16T03:38:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1990000000000002e-06 2024-09-16T03:38:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.029284+0000", 2024-09-16T03:38:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20203769499999999 2024-09-16T03:38:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.029375+0000", 2024-09-16T03:38:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1419e-05 2024-09-16T03:38:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.029542+0000", 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016613199999999999 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.130201+0000", 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10065989 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230566+0000", 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100364563 2024-09-16T03:38:50.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230623+0000", 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6848000000000002e-05 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230670+0000", 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7117999999999998e-05 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230699+0000", 2024-09-16T03:38:50.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8918999999999999e-05 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:166 3.1 3:91c40fc0:::f2d8823355177e50718038dafd7a8313c1a25b0231461a8672e684a7d4e587b4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.827250+0000", 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 8.1305537670000003, 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40362257099999999, 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827250+0000", 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827250+0000", 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827251+0000", 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827253+0000", 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.928e-06 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827257+0000", 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0249999999999996e-06 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827260+0000", 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3359999999999999e-06 2024-09-16T03:38:50.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.090467+0000", 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.26320751100000001 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.090543+0000", 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5740000000000003e-05 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.090651+0000", 2024-09-16T03:38:50.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000107415 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.130204+0000", 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.039553455000000001 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230753+0000", 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100549096 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230809+0000", 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6243999999999999e-05 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230848+0000", 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9045999999999999e-05 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230873+0000", 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4433000000000001e-05 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.084 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.085 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.085 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.085 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:50.086 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2488428 2024-09-16T03:38:50.086 INFO:tasks.rados.rados.0.smithi157.stdout:1533: expect (ObjNum 304 snap 0 seq_num 304) 2024-09-16T03:38:50.086 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 647658 2024-09-16T03:38:50.087 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:50.084+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:50.087 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:38:50.187 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:50.185+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:50.214 INFO:tasks.rados.rados.0.smithi157.stdout:1530: expect (ObjNum 283 snap 0 seq_num 283) 2024-09-16T03:38:50.214 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 331648 2024-09-16T03:38:50.231 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:50.232 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:50.232 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:50.232 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:50.240 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:38:50.288 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:50.286+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:50.294 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 520588 2024-09-16T03:38:50.343 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:50.343 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:50.343 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:50.343 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:50.343 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:50.351 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:38:50.361 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.9 to 0.126666 (206d) 2024-09-16T03:38:50.362 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:50.361+0000 7f3a1f71e640 -1 osd.6 115 handle_osd_map injecting map crc failure 2024-09-16T03:38:50.363 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:50.362+0000 7fa3886e6640 -1 osd.10 115 handle_osd_map injecting map crc failure 2024-09-16T03:38:50.364 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:50.363+0000 7f3a1f71e640 -1 osd.6 115 handle_osd_map injecting map crc failure 2024-09-16T03:38:50.389 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:50.387+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:50.451 INFO:tasks.rados.rados.0.smithi157.stdout:1531: finishing write tid 1 to smithi15736696-39 2024-09-16T03:38:50.451 INFO:tasks.rados.rados.0.smithi157.stdout:1532: finishing write tid 1 to smithi15736696-27 2024-09-16T03:38:50.451 INFO:tasks.rados.rados.0.smithi157.stdout:1529: done (15 left) 2024-09-16T03:38:50.451 INFO:tasks.rados.rados.0.smithi157.stdout:1530: done (14 left) 2024-09-16T03:38:50.451 INFO:tasks.rados.rados.0.smithi157.stdout:1533: done (13 left) 2024-09-16T03:38:50.451 INFO:tasks.rados.rados.0.smithi157.stdout:1543: tier_promote oid 21 2024-09-16T03:38:50.451 INFO:tasks.rados.rados.0.smithi157.stdout:1534: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:38:50.451 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:50.463 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:50.463 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:50.463 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:50.463 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:50.463 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.463 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:416 4.0 4:023abada:::smithi15736696-17:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:50.463 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:38:50.463 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 90.151173919000001, 2024-09-16T03:38:50.464 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60401983699999995, 2024-09-16T03:38:50.464 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.464 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.464 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.464 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.464 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.464 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.464 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 416 2024-09-16T03:38:50.464 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.464 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.464 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.464 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309243+0000", 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.178e-06 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.465 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309248+0000", 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309257+0000", 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4720000000000001e-06 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309262+0000", 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0839999999999999e-06 2024-09-16T03:38:50.466 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309297+0000", 2024-09-16T03:38:50.467 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4997999999999999e-05 2024-09-16T03:38:50.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309367+0000", 2024-09-16T03:38:50.467 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0124999999999997e-05 2024-09-16T03:38:50.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:50.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812055+0000", 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50268796900000001 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812614+0000", 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000558309 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.813040+0000", 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042652800000000002 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.468 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.469 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913189+0000", 2024-09-16T03:38:50.469 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10014880399999999 2024-09-16T03:38:50.469 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.469 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.469 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.469 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913220+0000", 2024-09-16T03:38:50.469 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1353000000000003e-05 2024-09-16T03:38:50.469 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.469 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.469 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.469 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913262+0000", 2024-09-16T03:38:50.469 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1284e-05 2024-09-16T03:38:50.469 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:494 4.f 4:f817def9:::smithi15736696-45:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 88.738234184000007, 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60451760399999999, 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.470 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.471 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 494 2024-09-16T03:38:50.471 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.471 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.471 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.471 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.471 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:38:50.472 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.472 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.472 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.472 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.472 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:38:50.472 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.472 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.473 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.473 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.473 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722183+0000", 2024-09-16T03:38:50.473 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:38:50.473 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.473 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.473 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.473 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722189+0000", 2024-09-16T03:38:50.473 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9429999999999997e-06 2024-09-16T03:38:50.473 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.473 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.473 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.474 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722203+0000", 2024-09-16T03:38:50.474 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4367e-05 2024-09-16T03:38:50.474 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.474 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.474 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.474 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722210+0000", 2024-09-16T03:38:50.474 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:38:50.474 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.474 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.474 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.474 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822367+0000", 2024-09-16T03:38:50.474 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100156933 2024-09-16T03:38:50.475 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.475 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822461+0000", 2024-09-16T03:38:50.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3216999999999999e-05 2024-09-16T03:38:50.475 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.475 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:38:50.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125593+0000", 2024-09-16T03:38:50.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30313253699999998 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126307+0000", 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071350000000000005 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126825+0000", 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051846100000000001 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.476 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326612+0000", 2024-09-16T03:38:50.477 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19978721999999999 2024-09-16T03:38:50.477 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.477 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.477 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.477 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326663+0000", 2024-09-16T03:38:50.477 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0460000000000001e-05 2024-09-16T03:38:50.477 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.477 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.477 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.477 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326699+0000", 2024-09-16T03:38:50.477 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6340000000000001e-05 2024-09-16T03:38:50.477 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.478 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.478 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.478 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.478 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:784 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=63b,truncate 2840350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:50.478 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:38:50.478 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.653826308000006, 2024-09-16T03:38:50.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63358405200000001, 2024-09-16T03:38:50.478 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.478 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.478 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.478 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.478 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.479 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.479 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 784 2024-09-16T03:38:50.479 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.479 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.479 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:38:50.479 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.479 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.479 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:38:50.479 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806590+0000", 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6000000000000003e-07 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806593+0000", 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6649999999999999e-06 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.480 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806597+0000", 2024-09-16T03:38:50.481 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.245e-06 2024-09-16T03:38:50.481 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.481 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.481 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806600+0000", 2024-09-16T03:38:50.481 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.23e-06 2024-09-16T03:38:50.481 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.481 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.481 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099381+0000", 2024-09-16T03:38:50.481 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29278119400000002 2024-09-16T03:38:50.481 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.481 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.482 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.482 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099438+0000", 2024-09-16T03:38:50.482 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6198e-05 2024-09-16T03:38:50.482 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.482 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.482 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:50.482 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099527+0000", 2024-09-16T03:38:50.482 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9505000000000004e-05 2024-09-16T03:38:50.482 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.482 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.482 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.482 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200775+0000", 2024-09-16T03:38:50.482 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101247904 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200889+0000", 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114301 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440066+0000", 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.239176367 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.483 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440109+0000", 2024-09-16T03:38:50.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2941000000000001e-05 2024-09-16T03:38:50.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.484 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.484 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.484 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440173+0000", 2024-09-16T03:38:50.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4842e-05 2024-09-16T03:38:50.484 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.484 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.484 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.484 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.484 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:785 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:50.485 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:38:50.485 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.653810383000007, 2024-09-16T03:38:50.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63376373200000002, 2024-09-16T03:38:50.485 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.485 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.485 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:50.485 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.485 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.485 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.485 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 785 2024-09-16T03:38:50.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.485 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:38:50.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:38:50.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806606+0000", 2024-09-16T03:38:50.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:38:50.487 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.487 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.487 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.487 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806607+0000", 2024-09-16T03:38:50.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.424e-06 2024-09-16T03:38:50.487 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.487 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.487 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.487 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806611+0000", 2024-09-16T03:38:50.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1080000000000003e-06 2024-09-16T03:38:50.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.488 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806614+0000", 2024-09-16T03:38:50.488 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.067e-06 2024-09-16T03:38:50.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.488 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099813+0000", 2024-09-16T03:38:50.488 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29319853699999998 2024-09-16T03:38:50.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099858+0000", 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5667999999999997e-05 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440236+0000", 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34037791499999998 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440297+0000", 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0988e-05 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.489 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440369+0000", 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1682999999999998e-05 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:801 4.0 4:07ad0560:::smithi15736696-25:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.454332484999995, 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.534510445, 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.490 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.491 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:50.491 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.491 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.491 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.491 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 801 2024-09-16T03:38:50.491 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.491 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.491 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.491 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.492 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:38:50.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.492 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.492 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.492 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:38:50.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.492 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.492 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.492 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006084+0000", 2024-09-16T03:38:50.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.322e-06 2024-09-16T03:38:50.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006088+0000", 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000001e-06 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006098+0000", 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0186999999999999e-05 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006104+0000", 2024-09-16T03:38:50.493 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.271e-06 2024-09-16T03:38:50.494 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.494 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.199952+0000", 2024-09-16T03:38:50.494 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.193847935 2024-09-16T03:38:50.494 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.494 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200027+0000", 2024-09-16T03:38:50.494 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4987000000000006e-05 2024-09-16T03:38:50.494 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.495 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.540594+0000", 2024-09-16T03:38:50.495 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34056708299999999 2024-09-16T03:38:50.495 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.495 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.495 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.495 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.495 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.495 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1023 4.0 4:06b6d050:::smithi15736696-50:head [write 2246563~579356] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:50.495 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:38:50.495 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 79.149183414999996, 2024-09-16T03:38:50.495 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60727534699999997, 2024-09-16T03:38:50.495 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.496 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.496 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.496 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.496 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.496 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.496 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1023 2024-09-16T03:38:50.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.496 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.496 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.496 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:38:50.496 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.497 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.497 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:38:50.497 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.497 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.497 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.497 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311233+0000", 2024-09-16T03:38:50.497 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1039999999999999e-06 2024-09-16T03:38:50.497 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.497 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.497 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311692+0000", 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045908800000000002 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311704+0000", 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1844e-05 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311713+0000", 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2500000000000006e-06 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359204+0000", 2024-09-16T03:38:50.499 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.047491646999999998 2024-09-16T03:38:50.499 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.499 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.499 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.499 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359258+0000", 2024-09-16T03:38:50.499 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4082000000000002e-05 2024-09-16T03:38:50.499 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.499 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.499 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:50.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359354+0000", 2024-09-16T03:38:50.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5686999999999994e-05 2024-09-16T03:38:50.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.500 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512245+0000", 2024-09-16T03:38:50.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15289135700000001 2024-09-16T03:38:50.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.500 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512956+0000", 2024-09-16T03:38:50.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071110300000000004 2024-09-16T03:38:50.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918361+0000", 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40540489400000002 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918455+0000", 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3412999999999995e-05 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918508+0000", 2024-09-16T03:38:50.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:38:50.502 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.502 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.502 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.502 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.502 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.502 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1024 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3176490] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:50.502 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:38:50.502 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 79.148690434000002, 2024-09-16T03:38:50.502 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60701227000000002, 2024-09-16T03:38:50.502 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.502 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.502 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.502 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.503 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.503 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.503 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1024 2024-09-16T03:38:50.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.503 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.503 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.503 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.503 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:38:50.503 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.503 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.503 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.503 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311726+0000", 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.02e-07 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311730+0000", 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1099999999999996e-06 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.504 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.505 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311735+0000", 2024-09-16T03:38:50.505 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4680000000000003e-06 2024-09-16T03:38:50.505 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.505 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.505 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.505 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311739+0000", 2024-09-16T03:38:50.505 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9380000000000002e-06 2024-09-16T03:38:50.505 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.505 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.505 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.505 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361468+0000", 2024-09-16T03:38:50.505 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.049729450000000001 2024-09-16T03:38:50.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.506 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.506 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361539+0000", 2024-09-16T03:38:50.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0896999999999996e-05 2024-09-16T03:38:50.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.506 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.506 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:50.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361661+0000", 2024-09-16T03:38:50.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000122404 2024-09-16T03:38:50.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.506 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.506 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.507 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512943+0000", 2024-09-16T03:38:50.507 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.151281481 2024-09-16T03:38:50.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.507 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.507 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.507 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512959+0000", 2024-09-16T03:38:50.507 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:38:50.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.507 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.507 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.507 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918588+0000", 2024-09-16T03:38:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40562815499999999 2024-09-16T03:38:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918646+0000", 2024-09-16T03:38:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8566999999999997e-05 2024-09-16T03:38:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918737+0000", 2024-09-16T03:38:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1308000000000001e-05 2024-09-16T03:38:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.509 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.509 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.509 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.509 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.509 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1025 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:50.509 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:50.509 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 79.148671936, 2024-09-16T03:38:50.509 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60723649599999996, 2024-09-16T03:38:50.509 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.509 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.509 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:50.509 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.509 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1025 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9400000000000001e-07 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311746+0000", 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2670000000000001e-06 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311752+0000", 2024-09-16T03:38:50.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6389999999999997e-06 2024-09-16T03:38:50.512 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.512 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.512 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.512 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311757+0000", 2024-09-16T03:38:50.512 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3639999999999998e-06 2024-09-16T03:38:50.512 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.512 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.512 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.512 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362018+0000", 2024-09-16T03:38:50.512 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.050260969000000003 2024-09-16T03:38:50.512 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.512 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.512 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:50.513 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362089+0000", 2024-09-16T03:38:50.513 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0735000000000006e-05 2024-09-16T03:38:50.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.513 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.513 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.513 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918781+0000", 2024-09-16T03:38:50.513 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55669215400000005 2024-09-16T03:38:50.513 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.513 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.513 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.513 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918867+0000", 2024-09-16T03:38:50.514 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6019000000000004e-05 2024-09-16T03:38:50.514 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.514 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.514 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.514 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918980+0000", 2024-09-16T03:38:50.514 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011295500000000001 2024-09-16T03:38:50.514 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.514 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.514 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.514 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.514 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.515 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1054 4.10 4:0a55bbf2:::smithi15736696-18:head [write 2633066~610652] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:50.515 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:38:50.515 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 78.947178727999997, 2024-09-16T03:38:50.515 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51115411600000005, 2024-09-16T03:38:50.515 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.515 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.515 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.515 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.515 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.515 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.515 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1054 2024-09-16T03:38:50.515 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.516 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.516 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.516 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.516 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:38:50.516 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.516 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.516 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.516 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.516 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:38:50.517 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.517 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.517 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.517 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.517 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513238+0000", 2024-09-16T03:38:50.517 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1999999999999998e-07 2024-09-16T03:38:50.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.518 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.518 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.518 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513695+0000", 2024-09-16T03:38:50.518 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045695099999999998 2024-09-16T03:38:50.518 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.519 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.519 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.519 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513720+0000", 2024-09-16T03:38:50.519 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.567e-05 2024-09-16T03:38:50.519 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.519 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.520 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.520 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513725+0000", 2024-09-16T03:38:50.520 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9899999999999997e-06 2024-09-16T03:38:50.520 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.520 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.520 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.521 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613892+0000", 2024-09-16T03:38:50.521 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166253 2024-09-16T03:38:50.521 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.521 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.521 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:38:50.521 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613898+0000", 2024-09-16T03:38:50.521 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8710000000000002e-06 2024-09-16T03:38:50.522 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.522 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.522 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.522 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916870+0000", 2024-09-16T03:38:50.522 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30297253800000001 2024-09-16T03:38:50.522 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.522 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.523 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.523 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916934+0000", 2024-09-16T03:38:50.523 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3723e-05 2024-09-16T03:38:50.523 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.523 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.523 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:50.524 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.917033+0000", 2024-09-16T03:38:50.524 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8908999999999998e-05 2024-09-16T03:38:50.524 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.524 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.524 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.525 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921278+0000", 2024-09-16T03:38:50.525 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0042456999999999998 2024-09-16T03:38:50.525 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.525 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.525 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.525 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019617+0000", 2024-09-16T03:38:50.526 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098338910000000002 2024-09-16T03:38:50.526 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.526 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.526 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.526 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024269+0000", 2024-09-16T03:38:50.527 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046515519999999998 2024-09-16T03:38:50.527 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.527 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.527 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.527 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024358+0000", 2024-09-16T03:38:50.527 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9096000000000002e-05 2024-09-16T03:38:50.528 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.528 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.528 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.528 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024391+0000", 2024-09-16T03:38:50.528 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3033e-05 2024-09-16T03:38:50.529 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.529 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.529 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.529 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.529 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.529 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1055 4.10 4:0a55bbf2:::smithi15736696-18:head [setxattr _header (56) in=7b,truncate 3243718] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:50.529 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:38:50.530 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 78.946679227999994, 2024-09-16T03:38:50.530 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61098759599999997, 2024-09-16T03:38:50.530 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.530 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.530 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.530 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.531 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.531 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.531 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1055 2024-09-16T03:38:50.531 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.531 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.531 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.531 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.532 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:38:50.532 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.532 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.532 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.532 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.532 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:38:50.533 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.533 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.533 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.533 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.533 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513737+0000", 2024-09-16T03:38:50.534 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9399999999999993e-07 2024-09-16T03:38:50.534 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.534 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.534 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.534 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513759+0000", 2024-09-16T03:38:50.534 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1903999999999999e-05 2024-09-16T03:38:50.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.535 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.535 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.535 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513773+0000", 2024-09-16T03:38:50.535 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3244000000000001e-05 2024-09-16T03:38:50.535 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.536 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.536 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.536 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513780+0000", 2024-09-16T03:38:50.536 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999999e-06 2024-09-16T03:38:50.536 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.536 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.537 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.537 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714016+0000", 2024-09-16T03:38:50.537 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20023635000000001 2024-09-16T03:38:50.537 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.537 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.538 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:38:50.538 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714024+0000", 2024-09-16T03:38:50.538 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8359999999999992e-06 2024-09-16T03:38:50.538 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.538 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.539 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918736+0000", 2024-09-16T03:38:50.539 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471236700000001 2024-09-16T03:38:50.539 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.539 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.539 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.540 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918830+0000", 2024-09-16T03:38:50.540 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3769000000000003e-05 2024-09-16T03:38:50.540 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.540 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.540 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:50.541 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919003+0000", 2024-09-16T03:38:50.541 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000173198 2024-09-16T03:38:50.541 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.542 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.542 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.542 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921589+0000", 2024-09-16T03:38:50.542 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002586419 2024-09-16T03:38:50.542 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.542 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.543 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.543 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019655+0000", 2024-09-16T03:38:50.543 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098065801999999994 2024-09-16T03:38:50.543 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.543 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.544 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.544 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124538+0000", 2024-09-16T03:38:50.544 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10488272899999999 2024-09-16T03:38:50.544 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.544 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.544 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124635+0000", 2024-09-16T03:38:50.545 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7125000000000003e-05 2024-09-16T03:38:50.545 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.545 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.545 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.545 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124724+0000", 2024-09-16T03:38:50.546 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8858999999999997e-05 2024-09-16T03:38:50.546 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.546 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.546 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.546 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.546 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.547 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1056 4.10 4:0a55bbf2:::smithi15736696-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:50.547 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:50.547 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 78.946626656999996, 2024-09-16T03:38:50.547 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61111632800000004, 2024-09-16T03:38:50.547 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.547 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.547 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:50.547 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.547 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.547 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.548 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1056 2024-09-16T03:38:50.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.548 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.548 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.548 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:50.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.548 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.548 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.548 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.548 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:50.548 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.549 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.549 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.549 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:38:50.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2499999999999995e-07 2024-09-16T03:38:50.549 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.549 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.549 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.549 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513792+0000", 2024-09-16T03:38:50.549 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:38:50.549 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.550 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.550 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.550 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513797+0000", 2024-09-16T03:38:50.550 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4940000000000003e-06 2024-09-16T03:38:50.550 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.550 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.550 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.550 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513802+0000", 2024-09-16T03:38:50.550 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5390000000000001e-06 2024-09-16T03:38:50.551 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.551 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.551 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.551 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919221+0000", 2024-09-16T03:38:50.551 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40541913899999998 2024-09-16T03:38:50.551 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.551 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.552 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:50.552 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919268+0000", 2024-09-16T03:38:50.552 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7027000000000002e-05 2024-09-16T03:38:50.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.552 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.552 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.552 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124766+0000", 2024-09-16T03:38:50.552 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20549842099999999 2024-09-16T03:38:50.552 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.552 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.552 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.552 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124828+0000", 2024-09-16T03:38:50.553 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1803000000000004e-05 2024-09-16T03:38:50.553 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.553 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.553 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.553 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124905+0000", 2024-09-16T03:38:50.553 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7163000000000006e-05 2024-09-16T03:38:50.553 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.553 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.553 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.553 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.553 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.553 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1702 4.10 4:0a55bbf2:::smithi15736696-18:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 59.771271667000001, 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51800317600000001, 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1702 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.554 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:38:50.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.555 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.555 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:38:50.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.555 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.555 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.555 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.555 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689147+0000", 2024-09-16T03:38:50.555 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:38:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689158+0000", 2024-09-16T03:38:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0426e-05 2024-09-16T03:38:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689185+0000", 2024-09-16T03:38:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7299000000000001e-05 2024-09-16T03:38:50.556 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789328+0000", 2024-09-16T03:38:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100143102 2024-09-16T03:38:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789532+0000", 2024-09-16T03:38:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000204317 2024-09-16T03:38:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.557 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789671+0000", 2024-09-16T03:38:50.558 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013852600000000001 2024-09-16T03:38:50.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.558 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.558 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.207147+0000", 2024-09-16T03:38:50.558 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.41747625599999999 2024-09-16T03:38:50.558 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.558 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.558 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.558 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.558 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.558 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2160 4.17 4:ed874e32:::smithi15736696-14:head [assert-version v36,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-09-16T03:38:50.558 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 43.630270246999999, 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55377054800000003, 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2160 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.559 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:38:50.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.560 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.560 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:38:50.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.560 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.560 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.560 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830149+0000", 2024-09-16T03:38:50.560 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.743e-06 2024-09-16T03:38:50.560 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830162+0000", 2024-09-16T03:38:50.561 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3283e-05 2024-09-16T03:38:50.561 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830191+0000", 2024-09-16T03:38:50.561 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8466000000000002e-05 2024-09-16T03:38:50.561 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.561 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.561 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.561 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830207+0000", 2024-09-16T03:38:50.562 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6453e-05 2024-09-16T03:38:50.562 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.562 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.562 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.562 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830362+0000", 2024-09-16T03:38:50.562 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015450699999999999 2024-09-16T03:38:50.562 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.563 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.563 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830608+0000", 2024-09-16T03:38:50.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024587399999999998 2024-09-16T03:38:50.563 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.563 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:38:50.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.233387+0000", 2024-09-16T03:38:50.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40277970099999999 2024-09-16T03:38:50.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.564 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.564 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234208+0000", 2024-09-16T03:38:50.565 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082053900000000001 2024-09-16T03:38:50.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.565 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.565 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234370+0000", 2024-09-16T03:38:50.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000161645 2024-09-16T03:38:50.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383821+0000", 2024-09-16T03:38:50.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14945145300000001 2024-09-16T03:38:50.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383867+0000", 2024-09-16T03:38:50.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5498999999999999e-05 2024-09-16T03:38:50.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383916+0000", 2024-09-16T03:38:50.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9385000000000002e-05 2024-09-16T03:38:50.568 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.568 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.568 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.568 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.568 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2532 4.0 4:06b6d050:::smithi15736696-50:head [write 2603977~725887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:50.568 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:38:50.568 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 33.866122857999997, 2024-09-16T03:38:50.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53439557900000001, 2024-09-16T03:38:50.569 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.569 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.569 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.569 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.569 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.569 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.569 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2532 2024-09-16T03:38:50.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.570 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.570 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.570 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:38:50.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.570 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.570 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:38:50.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.571 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594298+0000", 2024-09-16T03:38:50.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6030000000000004e-06 2024-09-16T03:38:50.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.571 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594876+0000", 2024-09-16T03:38:50.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00057757800000000003 2024-09-16T03:38:50.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.572 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594885+0000", 2024-09-16T03:38:50.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4150000000000005e-06 2024-09-16T03:38:50.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.572 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594892+0000", 2024-09-16T03:38:50.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7229999999999998e-06 2024-09-16T03:38:50.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628138+0000", 2024-09-16T03:38:50.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033246407999999998 2024-09-16T03:38:50.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628252+0000", 2024-09-16T03:38:50.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113755 2024-09-16T03:38:50.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:50.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628344+0000", 2024-09-16T03:38:50.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2028999999999996e-05 2024-09-16T03:38:50.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730215+0000", 2024-09-16T03:38:50.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101870761 2024-09-16T03:38:50.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.732105+0000", 2024-09-16T03:38:50.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001889951 2024-09-16T03:38:50.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.576 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128411+0000", 2024-09-16T03:38:50.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.396305764 2024-09-16T03:38:50.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128611+0000", 2024-09-16T03:38:50.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019990599999999999 2024-09-16T03:38:50.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.578 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128688+0000", 2024-09-16T03:38:50.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7686000000000001e-05 2024-09-16T03:38:50.579 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.579 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.579 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.579 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.579 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.579 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2533 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3663926] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:50.579 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:38:50.579 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 33.865515504999998, 2024-09-16T03:38:50.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53394911599999995, 2024-09-16T03:38:50.579 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.580 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.580 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.580 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.580 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.580 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.580 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2533 2024-09-16T03:38:50.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.581 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.581 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.581 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:38:50.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:38:50.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594901+0000", 2024-09-16T03:38:50.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6700000000000003e-07 2024-09-16T03:38:50.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594904+0000", 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1099999999999999e-06 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594907+0000", 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1020000000000001e-06 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594910+0000", 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7889999999999999e-06 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729537+0000", 2024-09-16T03:38:50.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.134627725 2024-09-16T03:38:50.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729660+0000", 2024-09-16T03:38:50.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012283800000000001 2024-09-16T03:38:50.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:38:50.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729786+0000", 2024-09-16T03:38:50.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012572000000000001 2024-09-16T03:38:50.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730385+0000", 2024-09-16T03:38:50.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059949700000000005 2024-09-16T03:38:50.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.028223+0000", 2024-09-16T03:38:50.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29783727999999998 2024-09-16T03:38:50.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128774+0000", 2024-09-16T03:38:50.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100551027 2024-09-16T03:38:50.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.588 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.588 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128816+0000", 2024-09-16T03:38:50.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1946000000000001e-05 2024-09-16T03:38:50.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.588 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.588 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128849+0000", 2024-09-16T03:38:50.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3515e-05 2024-09-16T03:38:50.588 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.588 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.588 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.589 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2534 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:38:50.589 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:38:50.589 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 33.865501082999998, 2024-09-16T03:38:50.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53406219200000005, 2024-09-16T03:38:50.589 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.589 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.589 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:50.589 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.589 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.589 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.589 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2534 2024-09-16T03:38:50.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8999999999999998e-07 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594916+0000", 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.344e-06 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594919+0000", 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0580000000000002e-06 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594922+0000", 2024-09-16T03:38:50.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4389999999999999e-06 2024-09-16T03:38:50.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729999+0000", 2024-09-16T03:38:50.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.13507707399999999 2024-09-16T03:38:50.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:50.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730046+0000", 2024-09-16T03:38:50.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7382999999999997e-05 2024-09-16T03:38:50.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128879+0000", 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39883298499999997 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128924+0000", 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128977+0000", 2024-09-16T03:38:50.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2648999999999997e-05 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2621 4.f 4:fa163101:::smithi15736696-10:head [setxattr _header (56) in=7b,truncate 3815189] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.217724704999998, 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55430782499999998, 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.594 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2621 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242692+0000", 2024-09-16T03:38:50.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7599999999999996e-07 2024-09-16T03:38:50.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242695+0000", 2024-09-16T03:38:50.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.529e-06 2024-09-16T03:38:50.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242699+0000", 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1010000000000002e-06 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242703+0000", 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.518e-06 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.494935+0000", 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25223231200000001 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495011+0000", 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5692999999999995e-05 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495117+0000", 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000105944 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.596308+0000", 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101190825 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796852+0000", 2024-09-16T03:38:50.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20054443199999999 2024-09-16T03:38:50.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796893+0000", 2024-09-16T03:38:50.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1221999999999999e-05 2024-09-16T03:38:50.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796946+0000", 2024-09-16T03:38:50.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2435000000000002e-05 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796999+0000", 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3038000000000003e-05 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2622 4.f 4:fa163101:::smithi15736696-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.217708850000001, 2024-09-16T03:38:50.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55448354399999999, 2024-09-16T03:38:50.601 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.601 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.601 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:50.601 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.601 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.601 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.601 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2622 2024-09-16T03:38:50.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.601 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:38:50.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9999999999999999e-07 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242709+0000", 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.429e-06 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242712+0000", 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6090000000000001e-06 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242716+0000", 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6480000000000001e-06 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495371+0000", 2024-09-16T03:38:50.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25265485700000001 2024-09-16T03:38:50.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:50.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495424+0000", 2024-09-16T03:38:50.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2927999999999999e-05 2024-09-16T03:38:50.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797042+0000", 2024-09-16T03:38:50.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30161821900000002 2024-09-16T03:38:50.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797114+0000", 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2445000000000002e-05 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797190+0000", 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6108999999999999e-05 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2941 4.17 4:eb60a28d:::smithi15736696-41:head [setxattr _header (56) in=7b,truncate 2420241] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:30.762823+0000", 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.697592544999999, 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51643088500000001, 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2941 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762823+0000", 2024-09-16T03:38:50.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762823+0000", 2024-09-16T03:38:50.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762824+0000", 2024-09-16T03:38:50.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3200000000000005e-07 2024-09-16T03:38:50.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762826+0000", 2024-09-16T03:38:50.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7590000000000002e-06 2024-09-16T03:38:50.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762830+0000", 2024-09-16T03:38:50.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4879999999999999e-06 2024-09-16T03:38:50.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762834+0000", 2024-09-16T03:38:50.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7249999999999999e-06 2024-09-16T03:38:50.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.763900+0000", 2024-09-16T03:38:50.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001066851 2024-09-16T03:38:50.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.763982+0000", 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1225999999999998e-05 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.764130+0000", 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014830999999999999 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.767207+0000", 2024-09-16T03:38:50.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0030773189999999998 2024-09-16T03:38:50.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.062538+0000", 2024-09-16T03:38:50.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29533109499999999 2024-09-16T03:38:50.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:50.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279028+0000", 2024-09-16T03:38:50.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.21648964100000001 2024-09-16T03:38:50.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279214+0000", 2024-09-16T03:38:50.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000185606 2024-09-16T03:38:50.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279254+0000", 2024-09-16T03:38:50.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0333000000000002e-05 2024-09-16T03:38:50.614 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.614 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.614 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.615 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2942 4.17 4:eb60a28d:::smithi15736696-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:38:50.615 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:30.762852+0000", 2024-09-16T03:38:50.615 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.697563353, 2024-09-16T03:38:50.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51654379900000003, 2024-09-16T03:38:50.615 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.615 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.615 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:50.615 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:50.615 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:50.616 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:50.616 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2942 2024-09-16T03:38:50.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.616 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762852+0000", 2024-09-16T03:38:50.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762852+0000", 2024-09-16T03:38:50.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762853+0000", 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762855+0000", 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0810000000000001e-06 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762860+0000", 2024-09-16T03:38:50.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8720000000000001e-06 2024-09-16T03:38:50.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762863+0000", 2024-09-16T03:38:50.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2930000000000001e-06 2024-09-16T03:38:50.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.764919+0000", 2024-09-16T03:38:50.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020556979999999999 2024-09-16T03:38:50.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.764964+0000", 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5788999999999998e-05 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279291+0000", 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51432677999999998 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279337+0000", 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5892e-05 2024-09-16T03:38:50.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:50.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279396+0000", 2024-09-16T03:38:50.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9018000000000003e-05 2024-09-16T03:38:50.620 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.620 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.620 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.620 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:50.620 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:50.620 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:50.621 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2285749 2024-09-16T03:38:50.622 INFO:tasks.rados.rados.0.smithi157.stdout:1536: expect (ObjNum 295 snap 0 seq_num 295) 2024-09-16T03:38:50.622 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:38:50.623 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:50.620+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:50.672 INFO:tasks.rados.rados.0.smithi157.stdout:1540: finishing write tid 1 to smithi15736696-45 2024-09-16T03:38:50.672 INFO:tasks.rados.rados.0.smithi157.stdout:1528: finishing set_chunk to oid 44 2024-09-16T03:38:50.672 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 63 (ObjNum 287 snap 0 seq_num 287) dirty exists 2024-09-16T03:38:50.672 INFO:tasks.rados.rados.0.smithi157.stdout:1524: done (13 left) 2024-09-16T03:38:50.673 INFO:tasks.rados.rados.0.smithi157.stdout:1528: done (12 left) 2024-09-16T03:38:50.673 INFO:tasks.rados.rados.0.smithi157.stdout:1534: done (11 left) 2024-09-16T03:38:50.673 INFO:tasks.rados.rados.0.smithi157.stdout:1536: done (10 left) 2024-09-16T03:38:50.673 INFO:tasks.rados.rados.0.smithi157.stdout:1537: done (9 left) 2024-09-16T03:38:50.673 INFO:tasks.rados.rados.0.smithi157.stdout:1544: read oid 49 snap -1 2024-09-16T03:38:50.673 INFO:tasks.rados.rados.0.smithi157.stdout:1544: expect (ObjNum 316 snap 0 seq_num 316) 2024-09-16T03:38:50.673 INFO:tasks.rados.rados.0.smithi157.stdout:1545: read oid 41 snap -1 2024-09-16T03:38:50.673 INFO:tasks.rados.rados.0.smithi157.stdout:1545: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-16T03:38:50.673 INFO:tasks.rados.rados.0.smithi157.stdout:1546: set_chunk oid 43 target oid 2024-09-16T03:38:50.722 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:50.721+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:50.762 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:50.760+0000 7f8be56cd640 -1 osd.7 115 handle_osd_map injecting map crc failure 2024-09-16T03:38:50.762 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:50.760+0000 7efeef6d5640 -1 osd.13 115 handle_osd_map injecting map crc failure 2024-09-16T03:38:50.768 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:50.768 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:50.768 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:50.768 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:50.777 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:38:50.824 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:50.822+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:50.881 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:50.881 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:50.882 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:50.882 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:50.882 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:50.890 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:38:50.925 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:50.924+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:50.982 INFO:tasks.rados.rados.0.smithi157.stdout:1546: set_chunk oid 43 offset: 2595328 length: 616448 target oid 54ba4afd36df654e33b8e658b959f8126d986d4c35bf73d0d0c23f1193e54795 offset: 0 snap -1 2024-09-16T03:38:50.982 INFO:tasks.rados.rados.0.smithi157.stdout:1547: write oid 46 current snap is 0 2024-09-16T03:38:50.982 INFO:tasks.rados.rados.0.smithi157.stdout:1531: finishing write tid 2 to smithi15736696-39 2024-09-16T03:38:50.982 INFO:tasks.rados.rados.0.smithi157.stdout:1531: finishing write tid 3 to smithi15736696-39 2024-09-16T03:38:50.982 INFO:tasks.rados.rados.0.smithi157.stdout:1531: finishing write tid 4 to smithi15736696-39 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:1538: got (0) Success 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 95 (ObjNum 276 snap 0 seq_num 276) dirty exists 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:1540: finishing write tid 2 to smithi15736696-45 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:1540: finishing write tid 3 to smithi15736696-45 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:1531: finishing write tid 5 to smithi15736696-39 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:1531: oid 39 updating version 0 to 78 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:1531: oid 39 updating version 78 to 80 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:1531: oid 39 updating version 80 to 81 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:1531: oid 39 version 81 is already newer than 79 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 81 (ObjNum 322 snap 0 seq_num 322) dirty exists 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:1531: left oid 39 (ObjNum 322 snap 0 seq_num 322) 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:1532: finishing write tid 2 to smithi15736696-27 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:1532: finishing write tid 3 to smithi15736696-27 2024-09-16T03:38:50.983 INFO:tasks.rados.rados.0.smithi157.stdout:1532: finishing write tid 4 to smithi15736696-27 2024-09-16T03:38:50.984 INFO:tasks.rados.rados.0.smithi157.stdout:1547: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2024-09-16T03:38:50.988 INFO:tasks.rados.rados.0.smithi157.stdout:1547: writing smithi15736696-46 from 679082 to 1297082 tid 1 2024-09-16T03:38:50.994 INFO:tasks.rados.rados.0.smithi157.stdout:1547: writing smithi15736696-46 from 1843718 to 2412823 tid 2 2024-09-16T03:38:50.995 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:50.995 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:50.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:50.995 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:50.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.995 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:50.995 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:50.995 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 98.848887019000003, 2024-09-16T03:38:50.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:38:50.996 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:50.996 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:50.996 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:50.996 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:50.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:50.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:50.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:50.997 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:38:50.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:50.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.997 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:50.997 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:38:50.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:38:50.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.997 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:50.997 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:38:50.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:38:50.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:38:50.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:38:50.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:50.999 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:38:50.999 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:38:50.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:50.999 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:38:50.999 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:38:50.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:50.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:50.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:38:51.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:38:51.000 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.000 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.000 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.000 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.000 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:38:51.000 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:51.000 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 98.848533658999997, 2024-09-16T03:38:51.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:38:51.000 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.002 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:38:51.003 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:38:51.003 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:38:51.003 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:38:51.003 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:38:51.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:38:51.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:38:51.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:38:51.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:38:51.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:38:51.005 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.005 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.005 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.005 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.005 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:38:51.005 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:51.005 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 98.847405860999999, 2024-09-16T03:38:51.005 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:38:51.005 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.005 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.005 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.006 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:51.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:38:51.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:38:51.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:38:51.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:38:51.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:38:51.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:38:51.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:38:51.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:38:51.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:38:51.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:38:51.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:38:51.009 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.009 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 98.846418420000006, 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:51.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:38:51.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:38:51.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:38:51.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:38:51.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:38:51.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:38:51.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:38:51.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:38:51.014 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:51.015 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 98.845661676000006, 2024-09-16T03:38:51.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:38:51.015 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.015 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.015 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.015 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:51.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:38:51.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:38:51.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:38:51.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:38:51.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:38:51.017 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.017 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.017 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:38:51.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:38:51.017 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.017 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.017 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:38:51.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:38:51.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:38:51.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:38:51.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:38:51.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:38:51.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:38:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:38:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:38:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:38:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.019 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:86 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'8, mlcod=39'5)", 2024-09-16T03:38:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 98.384149440000002, 2024-09-16T03:38:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20388281899999999, 2024-09-16T03:38:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:38:51.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608472+0000", 2024-09-16T03:38:51.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.227e-06 2024-09-16T03:38:51.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608616+0000", 2024-09-16T03:38:51.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000144177 2024-09-16T03:38:51.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608638+0000", 2024-09-16T03:38:51.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1543e-05 2024-09-16T03:38:51.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608645+0000", 2024-09-16T03:38:51.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5839999999999997e-06 2024-09-16T03:38:51.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608740+0000", 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4716999999999995e-05 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608763+0000", 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2671e-05 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812264+0000", 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203501194 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812353+0000", 2024-09-16T03:38:51.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8706000000000001e-05 2024-09-16T03:38:51.024 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.024 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.024 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.024 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:87 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'9, mlcod=39'5)", 2024-09-16T03:38:51.024 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:51.024 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 98.383936066000004, 2024-09-16T03:38:51.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20372206400000001, 2024-09-16T03:38:51.024 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.025 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.025 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.025 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:51.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.025 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:38:51.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.026 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.026 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608684+0000", 2024-09-16T03:38:51.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6700000000000003e-07 2024-09-16T03:38:51.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.026 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.026 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608689+0000", 2024-09-16T03:38:51.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.412e-06 2024-09-16T03:38:51.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608704+0000", 2024-09-16T03:38:51.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4397e-05 2024-09-16T03:38:51.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608713+0000", 2024-09-16T03:38:51.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:38:51.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710025+0000", 2024-09-16T03:38:51.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101312032 2024-09-16T03:38:51.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710042+0000", 2024-09-16T03:38:51.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6889e-05 2024-09-16T03:38:51.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812361+0000", 2024-09-16T03:38:51.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10231889600000001 2024-09-16T03:38:51.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812405+0000", 2024-09-16T03:38:51.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4101999999999999e-05 2024-09-16T03:38:51.029 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.029 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.029 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.029 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:101 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'10, mlcod=39'5)", 2024-09-16T03:38:51.029 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:51.029 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 98.382922629000007, 2024-09-16T03:38:51.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20275586900000001, 2024-09-16T03:38:51.030 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.030 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.030 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.030 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:51.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:38:51.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609698+0000", 2024-09-16T03:38:51.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.24e-06 2024-09-16T03:38:51.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609892+0000", 2024-09-16T03:38:51.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019465899999999999 2024-09-16T03:38:51.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609904+0000", 2024-09-16T03:38:51.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2031000000000001e-05 2024-09-16T03:38:51.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609912+0000", 2024-09-16T03:38:51.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7160000000000003e-06 2024-09-16T03:38:51.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710228+0000", 2024-09-16T03:38:51.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100315682 2024-09-16T03:38:51.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.033 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.033 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710242+0000", 2024-09-16T03:38:51.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4201e-05 2024-09-16T03:38:51.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.033 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.033 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812411+0000", 2024-09-16T03:38:51.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10216881799999999 2024-09-16T03:38:51.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812452+0000", 2024-09-16T03:38:51.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1522e-05 2024-09-16T03:38:51.034 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.034 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.034 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.034 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:102 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'11, mlcod=39'5)", 2024-09-16T03:38:51.034 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:51.034 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 98.380906604000003, 2024-09-16T03:38:51.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20134195399999999, 2024-09-16T03:38:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:38:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611714+0000", 2024-09-16T03:38:51.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.269e-06 2024-09-16T03:38:51.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611926+0000", 2024-09-16T03:38:51.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021252700000000001 2024-09-16T03:38:51.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611938+0000", 2024-09-16T03:38:51.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1437000000000001e-05 2024-09-16T03:38:51.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611944+0000", 2024-09-16T03:38:51.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.993e-06 2024-09-16T03:38:51.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811266+0000", 2024-09-16T03:38:51.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.199322306 2024-09-16T03:38:51.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811291+0000", 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4542999999999999e-05 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813021+0000", 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001730363 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813054+0000", 2024-09-16T03:38:51.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3516000000000002e-05 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:375 4.f e45/32 4:fcb75e43:::smithi15736696-26:head v 45'23, mlcod=41'20)", 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 90.684578876000003, 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200786568, 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.040 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.040 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:51.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.040 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.040 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:38:51.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.040 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.040 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308041+0000", 2024-09-16T03:38:51.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8299999999999995e-07 2024-09-16T03:38:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308046+0000", 2024-09-16T03:38:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.904e-06 2024-09-16T03:38:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308056+0000", 2024-09-16T03:38:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9210000000000004e-06 2024-09-16T03:38:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.408122+0000", 2024-09-16T03:38:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100066302 2024-09-16T03:38:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508289+0000", 2024-09-16T03:38:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166531 2024-09-16T03:38:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508302+0000", 2024-09-16T03:38:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2641e-05 2024-09-16T03:38:51.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508794+0000", 2024-09-16T03:38:51.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00049272400000000005 2024-09-16T03:38:51.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508827+0000", 2024-09-16T03:38:51.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2561999999999998e-05 2024-09-16T03:38:51.043 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.043 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.043 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.044 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:989 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'31, mlcod=46'28)", 2024-09-16T03:38:51.044 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:51.044 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 80.604810315999998, 2024-09-16T03:38:51.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40332266900000002, 2024-09-16T03:38:51.044 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.044 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.044 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.044 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:51.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:38:51.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387812+0000", 2024-09-16T03:38:51.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:38:51.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388467+0000", 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065539500000000002 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388481+0000", 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3905e-05 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488552+0000", 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10007120899999999 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688818+0000", 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20026627599999999 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688845+0000", 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791091+0000", 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10224588699999999 2024-09-16T03:38:51.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791131+0000", 2024-09-16T03:38:51.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0293000000000002e-05 2024-09-16T03:38:51.048 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.048 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.048 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.048 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:990 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'32, mlcod=46'28)", 2024-09-16T03:38:51.048 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:51.048 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 80.504026644000007, 2024-09-16T03:38:51.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30256419200000001, 2024-09-16T03:38:51.049 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.049 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.049 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.049 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:51.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488594+0000", 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488616+0000", 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2419e-05 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488627+0000", 2024-09-16T03:38:51.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1251e-05 2024-09-16T03:38:51.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588753+0000", 2024-09-16T03:38:51.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012544399999999 2024-09-16T03:38:51.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690679+0000", 2024-09-16T03:38:51.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101926027 2024-09-16T03:38:51.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690695+0000", 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6528000000000002e-05 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791133+0000", 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100438233 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791157+0000", 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-05 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.052 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.053 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.053 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:991 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'33, mlcod=46'28)", 2024-09-16T03:38:51.053 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:51.053 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 80.403829146999996, 2024-09-16T03:38:51.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202786365, 2024-09-16T03:38:51.053 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.053 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.053 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.053 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588792+0000", 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0530000000000001e-06 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588807+0000", 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5257e-05 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588832+0000", 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5123e-05 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588846+0000", 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3312999999999999e-05 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790810+0000", 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20196378100000001 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790866+0000", 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6168999999999997e-05 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791542+0000", 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067583699999999999 2024-09-16T03:38:51.056 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791576+0000", 2024-09-16T03:38:51.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4832000000000001e-05 2024-09-16T03:38:51.057 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.057 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.057 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.057 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1580 4.f e61/32 4:fa163101:::smithi15736696-10:head v 61'52, mlcod=58'50)", 2024-09-16T03:38:51.057 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:51.057 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 63.886997620999999, 2024-09-16T03:38:51.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.201787367, 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2600000000000001e-07 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106083+0000", 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00046163900000000002 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106093+0000", 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9320000000000008e-06 2024-09-16T03:38:51.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206190+0000", 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100096716 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206286+0000", 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5982000000000004e-05 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206312+0000", 2024-09-16T03:38:51.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6279999999999999e-05 2024-09-16T03:38:51.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.061 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307345+0000", 2024-09-16T03:38:51.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10103240199999999 2024-09-16T03:38:51.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.061 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307409+0000", 2024-09-16T03:38:51.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4189999999999994e-05 2024-09-16T03:38:51.061 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.061 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1760 4.f e71/32 4:f101f67d:::smithi15736696-22:head v 71'55, mlcod=61'53)", 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.611793655, 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20118894600000001, 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380826+0000", 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1899999999999998e-07 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380831+0000", 2024-09-16T03:38:51.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6049999999999997e-06 2024-09-16T03:38:51.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380851+0000", 2024-09-16T03:38:51.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0021e-05 2024-09-16T03:38:51.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380862+0000", 2024-09-16T03:38:51.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0458e-05 2024-09-16T03:38:51.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.065 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581184+0000", 2024-09-16T03:38:51.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20032164299999999 2024-09-16T03:38:51.065 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.065 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.065 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581202+0000", 2024-09-16T03:38:51.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8410000000000002e-05 2024-09-16T03:38:51.065 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.065 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.065 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581965+0000", 2024-09-16T03:38:51.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076291200000000003 2024-09-16T03:38:51.065 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.582014+0000", 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9577999999999999e-05 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2729 4.12 4:4a9d235d:::smithi15736696-1:head [write 1578407~488887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:22.608814+0000", 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.383804916999999, 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203332443, 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.066 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.067 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.067 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:51.067 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:51.067 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.067 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2729 2024-09-16T03:38:51.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.067 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.608814+0000", 2024-09-16T03:38:51.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.608814+0000", 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.608816+0000", 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0480000000000001e-06 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609287+0000", 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000470524 2024-09-16T03:38:51.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609298+0000", 2024-09-16T03:38:51.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1629e-05 2024-09-16T03:38:51.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609306+0000", 2024-09-16T03:38:51.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5480000000000004e-06 2024-09-16T03:38:51.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609354+0000", 2024-09-16T03:38:51.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7688999999999997e-05 2024-09-16T03:38:51.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609448+0000", 2024-09-16T03:38:51.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4060000000000004e-05 2024-09-16T03:38:51.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 2,11", 2024-09-16T03:38:51.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609571+0000", 2024-09-16T03:38:51.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012361699999999999 2024-09-16T03:38:51.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.611263+0000", 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016921099999999999 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.613981+0000", 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027178340000000001 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812075+0000", 2024-09-16T03:38:51.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19809358399999999 2024-09-16T03:38:51.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812114+0000", 2024-09-16T03:38:51.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9082000000000003e-05 2024-09-16T03:38:51.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812147+0000", 2024-09-16T03:38:51.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2718000000000001e-05 2024-09-16T03:38:51.072 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.072 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.073 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.073 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2730 4.12 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2067294] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:38:51.073 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:22.609319+0000", 2024-09-16T03:38:51.073 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.383300255999998, 2024-09-16T03:38:51.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202937274, 2024-09-16T03:38:51.073 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.073 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.073 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.073 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:51.073 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:51.074 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.074 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2730 2024-09-16T03:38:51.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.074 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609319+0000", 2024-09-16T03:38:51.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609319+0000", 2024-09-16T03:38:51.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609320+0000", 2024-09-16T03:38:51.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3099999999999997e-07 2024-09-16T03:38:51.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609324+0000", 2024-09-16T03:38:51.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.98e-06 2024-09-16T03:38:51.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609330+0000", 2024-09-16T03:38:51.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4640000000000003e-06 2024-09-16T03:38:51.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609337+0000", 2024-09-16T03:38:51.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.374e-06 2024-09-16T03:38:51.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.610575+0000", 2024-09-16T03:38:51.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001237414 2024-09-16T03:38:51.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.610663+0000", 2024-09-16T03:38:51.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8268000000000003e-05 2024-09-16T03:38:51.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 2,11", 2024-09-16T03:38:51.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.610822+0000", 2024-09-16T03:38:51.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015895199999999999 2024-09-16T03:38:51.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.613696+0000", 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0028742239999999999 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.614015+0000", 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00031823499999999999 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812188+0000", 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.198173138 2024-09-16T03:38:51.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812222+0000", 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4724999999999997e-05 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812256+0000", 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3769000000000001e-05 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.079 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2731 4.12 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:38:51.080 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:22.609352+0000", 2024-09-16T03:38:51.080 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.383266806999998, 2024-09-16T03:38:51.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203034889, 2024-09-16T03:38:51.080 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.080 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.080 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:51.080 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:51.080 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:51.080 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.080 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2731 2024-09-16T03:38:51.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.080 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609352+0000", 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609352+0000", 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609353+0000", 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4099999999999999e-07 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609355+0000", 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4439999999999998e-06 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609360+0000", 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609363+0000", 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3780000000000001e-06 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.611279+0000", 2024-09-16T03:38:51.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019158140000000001 2024-09-16T03:38:51.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:51.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.611318+0000", 2024-09-16T03:38:51.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8312999999999997e-05 2024-09-16T03:38:51.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812286+0000", 2024-09-16T03:38:51.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20096847600000001 2024-09-16T03:38:51.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812332+0000", 2024-09-16T03:38:51.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6313000000000002e-05 2024-09-16T03:38:51.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812387+0000", 2024-09-16T03:38:51.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4857e-05 2024-09-16T03:38:51.084 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.084 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.085 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.085 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3081 4.1f e102/100 4:fa163101:::smithi15736696-10:head v 102'93, mlcod=102'91)", 2024-09-16T03:38:51.085 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:38:51.085 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.658904176, 2024-09-16T03:38:51.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20437049600000001, 2024-09-16T03:38:51.085 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.085 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.085 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.085 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:38:51.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:38:51.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333718+0000", 2024-09-16T03:38:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3720000000000001e-06 2024-09-16T03:38:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334338+0000", 2024-09-16T03:38:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061959099999999996 2024-09-16T03:38:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334355+0000", 2024-09-16T03:38:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.677e-05 2024-09-16T03:38:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334364+0000", 2024-09-16T03:38:51.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3780000000000007e-06 2024-09-16T03:38:51.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334424+0000", 2024-09-16T03:38:51.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0084000000000003e-05 2024-09-16T03:38:51.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334447+0000", 2024-09-16T03:38:51.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3113e-05 2024-09-16T03:38:51.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538030+0000", 2024-09-16T03:38:51.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20358272399999999 2024-09-16T03:38:51.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538085+0000", 2024-09-16T03:38:51.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5464000000000003e-05 2024-09-16T03:38:51.090 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.090 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.090 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.090 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3082 4.1f e102/100 4:fa163101:::smithi15736696-10:head v 102'94, mlcod=102'91)", 2024-09-16T03:38:51.090 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:35.335460+0000", 2024-09-16T03:38:51.090 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.657159218, 2024-09-16T03:38:51.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264421699999999, 2024-09-16T03:38:51.090 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:51.090 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:51.090 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.091 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:51.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:51.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335460+0000", 2024-09-16T03:38:51.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:51.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335460+0000", 2024-09-16T03:38:51.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:51.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335462+0000", 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9929999999999998e-06 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335889+0000", 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042691199999999999 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335898+0000", 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6660000000000003e-06 2024-09-16T03:38:51.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335906+0000", 2024-09-16T03:38:51.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0720000000000002e-06 2024-09-16T03:38:51.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:51.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335960+0000", 2024-09-16T03:38:51.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4456000000000002e-05 2024-09-16T03:38:51.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:51.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335978+0000", 2024-09-16T03:38:51.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8082000000000001e-05 2024-09-16T03:38:51.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:51.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538089+0000", 2024-09-16T03:38:51.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202111078 2024-09-16T03:38:51.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:51.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:51.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:51.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538104+0000", 2024-09-16T03:38:51.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4958e-05 2024-09-16T03:38:51.095 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.095 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.095 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.095 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:51.095 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:51.095 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:51.097 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:38:51.100 INFO:tasks.rados.rados.0.smithi157.stdout:1547: writing smithi15736696-46 from 2853072 to 3479082 tid 3 2024-09-16T03:38:51.100 INFO:tasks.rados.rados.0.smithi157.stdout:1531: done (12 left) 2024-09-16T03:38:51.100 INFO:tasks.rados.rados.0.smithi157.stdout:1538: done (11 left) 2024-09-16T03:38:51.100 INFO:tasks.rados.rados.0.smithi157.stdout:1548: set_chunk oid 38 target oid 2024-09-16T03:38:51.100 INFO:tasks.rados.rados.0.smithi157.stdout:1548: set_chunk oid 38 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:51.100 INFO:tasks.rados.rados.0.smithi157.stdout:1549: write (excl) oid 6 current snap is 0 2024-09-16T03:38:51.100 INFO:tasks.rados.rados.0.smithi157.stdout:1532: finishing write tid 5 to smithi15736696-27 2024-09-16T03:38:51.100 INFO:tasks.rados.rados.0.smithi157.stdout:1532: oid 27 updating version 0 to 124 2024-09-16T03:38:51.100 INFO:tasks.rados.rados.0.smithi157.stdout:1532: oid 27 updating version 124 to 126 2024-09-16T03:38:51.100 INFO:tasks.rados.rados.0.smithi157.stdout:1532: oid 27 updating version 126 to 127 2024-09-16T03:38:51.100 INFO:tasks.rados.rados.0.smithi157.stdout:1532: oid 27 version 127 is already newer than 125 2024-09-16T03:38:51.100 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 127 (ObjNum 323 snap 0 seq_num 323) dirty exists 2024-09-16T03:38:51.100 INFO:tasks.rados.rados.0.smithi157.stdout:1532: left oid 27 (ObjNum 323 snap 0 seq_num 323) 2024-09-16T03:38:51.101 INFO:tasks.rados.rados.0.smithi157.stdout:1535: finishing write tid 1 to smithi15736696-19 2024-09-16T03:38:51.101 INFO:tasks.rados.rados.0.smithi157.stdout:1549: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-09-16T03:38:51.101 INFO:tasks.rados.rados.0.smithi157.stdout:1549: writing smithi15736696-6 from 695889 to 1105491 tid 1 2024-09-16T03:38:51.101 INFO:tasks.rados.rados.0.smithi157.stdout:1549: writing smithi15736696-6 from 1618553 to 2331316 tid 2 2024-09-16T03:38:51.101 INFO:tasks.rados.rados.0.smithi157.stdout:1549: writing smithi15736696-6 from 2996848 to 3495889 tid 3 2024-09-16T03:38:51.101 INFO:tasks.rados.rados.0.smithi157.stdout:1532: done (12 left) 2024-09-16T03:38:51.101 INFO:tasks.rados.rados.0.smithi157.stdout:1550: read oid 34 snap -1 2024-09-16T03:38:51.101 INFO:tasks.rados.rados.0.smithi157.stdout:1550: expect (ObjNum 303 snap 0 seq_num 303) 2024-09-16T03:38:51.101 INFO:tasks.rados.rados.0.smithi157.stdout:1551: read oid 50 snap -1 2024-09-16T03:38:51.101 INFO:tasks.rados.rados.0.smithi157.stdout:1551: expect (ObjNum 313 snap 0 seq_num 313) 2024-09-16T03:38:51.101 INFO:tasks.rados.rados.0.smithi157.stdout:1552: write oid 33 current snap is 0 2024-09-16T03:38:51.101 INFO:tasks.rados.rados.0.smithi157.stdout:1541: finishing set_chunk to oid 24 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 217088 length: 21504 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 0 (ObjNum 271 snap 0 seq_num 271) dirty exists 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:1540: finishing write tid 4 to smithi15736696-45 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:1535: finishing write tid 2 to smithi15736696-19 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:1552: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:1552: writing smithi15736696-33 from 712696 to 1197547 tid 1 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:1552: writing smithi15736696-33 from 1960682 to 2533456 tid 2 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:1552: writing smithi15736696-33 from 3307918 to 3512696 tid 3 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:1541: done (14 left) 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:1553: read oid 27 snap -1 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:1553: expect (ObjNum 323 snap 0 seq_num 323) 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:1554: write oid 28 current snap is 0 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:1554: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:1554: writing smithi15736696-28 from 729503 to 1289603 tid 1 2024-09-16T03:38:51.102 INFO:tasks.rados.rados.0.smithi157.stdout:1554: writing smithi15736696-28 from 2019164 to 2735596 tid 2 2024-09-16T03:38:51.103 INFO:tasks.rados.rados.0.smithi157.stdout:1554: writing smithi15736696-28 from 3335341 to 3529503 tid 3 2024-09-16T03:38:51.103 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:51.103 INFO:tasks.rados.rados.0.smithi157.stdout:1535: finishing write tid 3 to smithi15736696-19 2024-09-16T03:38:51.103 INFO:tasks.rados.rados.0.smithi157.stdout:1535: finishing write tid 4 to smithi15736696-19 2024-09-16T03:38:51.103 INFO:tasks.rados.rados.0.smithi157.stdout:1542: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-16T03:38:51.103 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2085476 2024-09-16T03:38:51.103 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:51.094+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:51.126 INFO:tasks.rados.rados.0.smithi157.stdout:1540: finishing write tid 5 to smithi15736696-45 2024-09-16T03:38:51.126 INFO:tasks.rados.rados.0.smithi157.stdout:1540: oid 45 updating version 0 to 106 2024-09-16T03:38:51.126 INFO:tasks.rados.rados.0.smithi157.stdout:1540: oid 45 updating version 106 to 107 2024-09-16T03:38:51.126 INFO:tasks.rados.rados.0.smithi157.stdout:1540: oid 45 version 107 is already newer than 102 2024-09-16T03:38:51.126 INFO:tasks.rados.rados.0.smithi157.stdout:1540: oid 45 version 107 is already newer than 105 2024-09-16T03:38:51.126 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 107 (ObjNum 325 snap 0 seq_num 325) dirty exists 2024-09-16T03:38:51.126 INFO:tasks.rados.rados.0.smithi157.stdout:1540: left oid 45 (ObjNum 325 snap 0 seq_num 325) 2024-09-16T03:38:51.126 INFO:tasks.rados.rados.0.smithi157.stdout:1540: done (15 left) 2024-09-16T03:38:51.126 INFO:tasks.rados.rados.0.smithi157.stdout:1542: done (14 left) 2024-09-16T03:38:51.127 INFO:tasks.rados.rados.0.smithi157.stdout:1555: delete oid 39 current snap is 0 2024-09-16T03:38:51.127 INFO:tasks.rados.rados.0.smithi157.stdout:1535: finishing write tid 5 to smithi15736696-19 2024-09-16T03:38:51.127 INFO:tasks.rados.rados.0.smithi157.stdout:1535: oid 19 updating version 0 to 128 2024-09-16T03:38:51.127 INFO:tasks.rados.rados.0.smithi157.stdout:1535: oid 19 updating version 128 to 131 2024-09-16T03:38:51.127 INFO:tasks.rados.rados.0.smithi157.stdout:1535: oid 19 version 131 is already newer than 129 2024-09-16T03:38:51.127 INFO:tasks.rados.rados.0.smithi157.stdout:1535: oid 19 version 131 is already newer than 130 2024-09-16T03:38:51.127 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 131 (ObjNum 324 snap 0 seq_num 324) dirty exists 2024-09-16T03:38:51.127 INFO:tasks.rados.rados.0.smithi157.stdout:1535: left oid 19 (ObjNum 324 snap 0 seq_num 324) 2024-09-16T03:38:51.127 INFO:tasks.rados.rados.0.smithi157.stdout:1545: expect (ObjNum 263 snap 0 seq_num 263) 2024-09-16T03:38:51.175 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:51.197 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:51.195+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:51.211 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop_reply(client.4628.0:3567 4.1b e116/115)", 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.156595+0000", 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.053858766000000002, 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.053873706, 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "no flag points reached", 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156595+0000", 2024-09-16T03:38:51.212 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156595+0000", 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156595+0000", 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6899999999999998e-07 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.213 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156598+0000", 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1550000000000001e-06 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156603+0000", 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2179999999999998e-06 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3592 4.1b 4.7a696d3b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:38:51.214 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.053840322000000003, 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.053878967999999999, 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3592 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:38:51.215 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156615+0000", 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.714e-06 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.216 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156623+0000", 2024-09-16T03:38:51.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5939999999999996e-06 2024-09-16T03:38:51.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156634+0000", 2024-09-16T03:38:51.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.093e-05 2024-09-16T03:38:51.217 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.217 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.217 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156640+0000", 2024-09-16T03:38:51.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1890000000000004e-06 2024-09-16T03:38:51.217 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.218 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.218 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.218 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3593 4.1b 4.7a696d3b (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:38:51.218 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:38:51.218 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.053805588000000001, 2024-09-16T03:38:51.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.053870121999999999, 2024-09-16T03:38:51.218 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.218 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.218 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:51.218 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3593 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:38:51.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156649+0000", 2024-09-16T03:38:51.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6400000000000002e-07 2024-09-16T03:38:51.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156652+0000", 2024-09-16T03:38:51.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9579999999999999e-06 2024-09-16T03:38:51.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156658+0000", 2024-09-16T03:38:51.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4729999999999997e-06 2024-09-16T03:38:51.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156667+0000", 2024-09-16T03:38:51.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1940000000000004e-06 2024-09-16T03:38:51.221 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.221 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.221 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.221 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3567 4.1b 4:dbb349e8:::smithi15736696-46:head [setxattr _header (56) in=63b,truncate 3479082] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:38:51.222 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:50.999803+0000", 2024-09-16T03:38:51.222 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.210651227, 2024-09-16T03:38:51.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.21074689999999999, 2024-09-16T03:38:51.222 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.222 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.222 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:51.222 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:51.222 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:51.222 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.222 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3567 2024-09-16T03:38:51.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.222 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999803+0000", 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999803+0000", 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999803+0000", 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7199999999999998e-07 2024-09-16T03:38:51.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999807+0000", 2024-09-16T03:38:51.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.782e-06 2024-09-16T03:38:51.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999813+0000", 2024-09-16T03:38:51.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8300000000000001e-06 2024-09-16T03:38:51.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999816+0000", 2024-09-16T03:38:51.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3050000000000001e-06 2024-09-16T03:38:51.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001310+0000", 2024-09-16T03:38:51.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001494057 2024-09-16T03:38:51.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001397+0000", 2024-09-16T03:38:51.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6458000000000005e-05 2024-09-16T03:38:51.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001551+0000", 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015404000000000001 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001975+0000", 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042458099999999999 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044734+0000", 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.042758530000000003 2024-09-16T03:38:51.226 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.227 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.227 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.227 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3568 4.1b 4:dbb349e8:::smithi15736696-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:38:51.227 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:50.999832+0000", 2024-09-16T03:38:51.227 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.21062180699999999, 2024-09-16T03:38:51.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.210753998, 2024-09-16T03:38:51.227 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.227 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.227 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for rw locks", 2024-09-16T03:38:51.227 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:51.227 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:51.228 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.228 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3568 2024-09-16T03:38:51.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.228 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999832+0000", 2024-09-16T03:38:51.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999832+0000", 2024-09-16T03:38:51.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999833+0000", 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.51e-07 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999835+0000", 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5330000000000001e-06 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999841+0000", 2024-09-16T03:38:51.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9939999999999997e-06 2024-09-16T03:38:51.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999846+0000", 2024-09-16T03:38:51.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.31e-06 2024-09-16T03:38:51.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001833+0000", 2024-09-16T03:38:51.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019864890000000001 2024-09-16T03:38:51.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001875+0000", 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2113999999999997e-05 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3589 4.1b 4:dcb6965e:::smithi15736696-28:head [write 729503~560100 in=560100b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.044505+0000", 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.165949019, 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.166110643, 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.231 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.232 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:51.232 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:51.232 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:51.232 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.232 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3589 2024-09-16T03:38:51.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.232 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044505+0000", 2024-09-16T03:38:51.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044505+0000", 2024-09-16T03:38:51.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044508+0000", 2024-09-16T03:38:51.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5190000000000001e-06 2024-09-16T03:38:51.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044633+0000", 2024-09-16T03:38:51.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000124447 2024-09-16T03:38:51.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044653+0000", 2024-09-16T03:38:51.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0032999999999999e-05 2024-09-16T03:38:51.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044663+0000", 2024-09-16T03:38:51.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0487e-05 2024-09-16T03:38:51.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052467+0000", 2024-09-16T03:38:51.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0078037280000000002 2024-09-16T03:38:51.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052561+0000", 2024-09-16T03:38:51.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3879e-05 2024-09-16T03:38:51.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:38:51.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052665+0000", 2024-09-16T03:38:51.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000104613 2024-09-16T03:38:51.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:51.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.054365+0000", 2024-09-16T03:38:51.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001699983 2024-09-16T03:38:51.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:51.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156645+0000", 2024-09-16T03:38:51.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10227989699999999 2024-09-16T03:38:51.236 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.236 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.236 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.237 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3590 4.1b 4:dcb6965e:::smithi15736696-28:head [write 2019164~716432 in=716432b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:38:51.237 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.052101+0000", 2024-09-16T03:38:51.237 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.15835262, 2024-09-16T03:38:51.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.158548996, 2024-09-16T03:38:51.237 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.237 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.237 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:51.237 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:51.237 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:51.237 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3590 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052101+0000", 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052101+0000", 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052103+0000", 2024-09-16T03:38:51.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.44e-06 2024-09-16T03:38:51.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052404+0000", 2024-09-16T03:38:51.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030129000000000001 2024-09-16T03:38:51.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052416+0000", 2024-09-16T03:38:51.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1868999999999999e-05 2024-09-16T03:38:51.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052422+0000", 2024-09-16T03:38:51.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:38:51.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.103424+0000", 2024-09-16T03:38:51.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051001643999999999 2024-09-16T03:38:51.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.103503+0000", 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9871999999999997e-05 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.103590+0000", 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6268999999999997e-05 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.105959+0000", 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023694179999999999 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.241 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3591 4.1b 4.7a696d3b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.15404847399999999, 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15427664399999999, 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3591 2024-09-16T03:38:51.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056407+0000", 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8920000000000001e-06 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056502+0000", 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4838000000000002e-05 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056511+0000", 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3579999999999992e-06 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156581+0000", 2024-09-16T03:38:51.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100069254 2024-09-16T03:38:51.245 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.245 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.245 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.245 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.245 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-16T03:38:51.245 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 8 2024-09-16T03:38:51.245 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:51.246 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:38:51.273 INFO:tasks.rados.rados.0.smithi157.stdout:1544: expect (ObjNum 316 snap 0 seq_num 316) 2024-09-16T03:38:51.297 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:51.296+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:51.356 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:51.354+0000 7f66f5209640 -1 osd.15 116 handle_osd_map injecting map crc failure 2024-09-16T03:38:51.358 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:51.356+0000 7efeef6d5640 -1 osd.13 116 handle_osd_map injecting map crc failure 2024-09-16T03:38:51.360 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:51.359+0000 7f3a1f71e640 -1 osd.6 116 handle_osd_map injecting map crc failure 2024-09-16T03:38:51.365 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1526324 2024-09-16T03:38:51.378 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:38:51.379 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:51.379 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:51.379 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:38:51.379 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:38:51.379 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:51.387 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:38:51.430 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:51.429+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:51.467 INFO:tasks.rados.rados.0.smithi157.stdout:1547: finishing write tid 1 to smithi15736696-46 2024-09-16T03:38:51.467 INFO:tasks.rados.rados.0.smithi157.stdout:1547: finishing write tid 2 to smithi15736696-46 2024-09-16T03:38:51.467 INFO:tasks.rados.rados.0.smithi157.stdout:1548: finishing set_chunk to oid 38 2024-09-16T03:38:51.467 INFO:tasks.rados.rados.0.smithi157.stdout:1548: got expected ENOENT (src dne) 2024-09-16T03:38:51.467 INFO:tasks.rados.rados.0.smithi157.stdout:1551: expect (ObjNum 313 snap 0 seq_num 313) 2024-09-16T03:38:51.497 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:51.497 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:38:51.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:38:51.497 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:51.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.497 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1515 3.1 e59/31 3:9c9e970c:::7249cb65b52cadf4fe8f4ec923c7e548d1de7d9628823acd2c8de19121f1c1ce:head v 59'69, mlcod=56'67)", 2024-09-16T03:38:51.497 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:38:51.497 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.540330006999994, 2024-09-16T03:38:51.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.203376156, 2024-09-16T03:38:51.498 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.498 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.498 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.498 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:38:51.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:38:51.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951651+0000", 2024-09-16T03:38:51.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8299999999999996e-07 2024-09-16T03:38:51.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953610+0000", 2024-09-16T03:38:51.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019588769999999999 2024-09-16T03:38:51.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953622+0000", 2024-09-16T03:38:51.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1846e-05 2024-09-16T03:38:51.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953632+0000", 2024-09-16T03:38:51.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0202e-05 2024-09-16T03:38:51.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.053808+0000", 2024-09-16T03:38:51.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100175615 2024-09-16T03:38:51.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.053846+0000", 2024-09-16T03:38:51.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8383000000000002e-05 2024-09-16T03:38:51.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.155006+0000", 2024-09-16T03:38:51.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101160131 2024-09-16T03:38:51.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.155027+0000", 2024-09-16T03:38:51.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0319e-05 2024-09-16T03:38:51.502 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.502 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.502 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.502 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1704 3.1 e65/31 3:b6a94245:::f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99:head v 65'77, mlcod=64'75)", 2024-09-16T03:38:51.502 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.698194733000001, 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30251757499999998, 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:38:51.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793788+0000", 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794391+0000", 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060354199999999999 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794406+0000", 2024-09-16T03:38:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5045000000000001e-05 2024-09-16T03:38:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794432+0000", 2024-09-16T03:38:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5996999999999999e-05 2024-09-16T03:38:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.994704+0000", 2024-09-16T03:38:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20027177700000001 2024-09-16T03:38:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.994719+0000", 2024-09-16T03:38:51.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5021000000000001e-05 2024-09-16T03:38:51.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096277+0000", 2024-09-16T03:38:51.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101557437 2024-09-16T03:38:51.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096303+0000", 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6732000000000001e-05 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:84 3.1 e65/31 3:b24f289d:::e84532c28f467d2f2a6e2cf44904fa0d4a555e1304bbe1b25db645af42152bd7:head v 65'78, mlcod=64'75)", 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.68917957, 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29353782099999998, 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.507 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802805+0000", 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1200000000000004e-06 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803910+0000", 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011051660000000001 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803928+0000", 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7880000000000002e-05 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803938+0000", 2024-09-16T03:38:51.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6690000000000001e-06 2024-09-16T03:38:51.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095473+0000", 2024-09-16T03:38:51.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29153552399999999 2024-09-16T03:38:51.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095486+0000", 2024-09-16T03:38:51.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2455e-05 2024-09-16T03:38:51.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096306+0000", 2024-09-16T03:38:51.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00082031900000000002 2024-09-16T03:38:51.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096339+0000", 2024-09-16T03:38:51.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2688000000000003e-05 2024-09-16T03:38:51.511 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.511 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.511 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:85 3.1 e65/31 3:b380f269:::e65351e8a65a0b1a517e3b1a85e04b84c0ab7cfa6655f11fb0da31f56b4dd9af:head v 65'79, mlcod=64'75)", 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.687898992000001, 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29260815400000001, 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:38:51.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804083+0000", 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804266+0000", 2024-09-16T03:38:51.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018268099999999999 2024-09-16T03:38:51.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804275+0000", 2024-09-16T03:38:51.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4439999999999997e-06 2024-09-16T03:38:51.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904371+0000", 2024-09-16T03:38:51.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100095276 2024-09-16T03:38:51.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095883+0000", 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.191512444 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095901+0000", 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8462999999999999e-05 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096658+0000", 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00075622000000000005 2024-09-16T03:38:51.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096690+0000", 2024-09-16T03:38:51.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1919999999999999e-05 2024-09-16T03:38:51.516 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.516 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.516 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.516 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:86 3.1 e65/31 3:b3acb53e:::e951eb69ce5151a3f54107fa22c70f80729eff845da1613447a91536a24046e8:head v 65'80, mlcod=64'75)", 2024-09-16T03:38:51.516 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:38:51.516 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 60.587595575999998, 2024-09-16T03:38:51.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29244589199999999, 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904386+0000", 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3e-07 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904537+0000", 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015122700000000001 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904544+0000", 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.024e-06 2024-09-16T03:38:51.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904547+0000", 2024-09-16T03:38:51.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6540000000000001e-06 2024-09-16T03:38:51.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196016+0000", 2024-09-16T03:38:51.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29146812599999999 2024-09-16T03:38:51.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196027+0000", 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1172000000000001e-05 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196791+0000", 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00076418700000000005 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196831+0000", 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9771999999999998e-05 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.520 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.521 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:94 3.0 e70/69 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head v 70'106, mlcod=0'0)", 2024-09-16T03:38:51.521 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:55.960145+0000", 2024-09-16T03:38:51.521 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.531835882000003, 2024-09-16T03:38:51.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20090359099999999, 2024-09-16T03:38:51.521 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.521 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.521 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.521 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960145+0000", 2024-09-16T03:38:51.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960145+0000", 2024-09-16T03:38:51.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960145+0000", 2024-09-16T03:38:51.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7300000000000002e-07 2024-09-16T03:38:51.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960147+0000", 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.509e-06 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:55.960151+0000", 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.060262+0000", 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100110847 2024-09-16T03:38:51.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.160465+0000", 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100203264 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.160483+0000", 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.753e-05 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.161026+0000", 2024-09-16T03:38:51.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00054326199999999998 2024-09-16T03:38:51.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:56.161048+0000", 2024-09-16T03:38:51.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1951999999999999e-05 2024-09-16T03:38:51.525 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.525 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.525 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.525 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2543 4.a e86/32 4:5e3c0760:::smithi15736696-24:head v 86'88, mlcod=84'86)", 2024-09-16T03:38:51.525 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:17.073635+0000", 2024-09-16T03:38:51.525 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 34.418345072999998, 2024-09-16T03:38:51.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20396895300000001, 2024-09-16T03:38:51.526 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.526 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.526 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.526 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.073635+0000", 2024-09-16T03:38:51.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.073635+0000", 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.073636+0000", 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7099999999999997e-07 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.074518+0000", 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000882532 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.074530+0000", 2024-09-16T03:38:51.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1799e-05 2024-09-16T03:38:51.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.174624+0000", 2024-09-16T03:38:51.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100094445 2024-09-16T03:38:51.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.274761+0000", 2024-09-16T03:38:51.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013633199999999 2024-09-16T03:38:51.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.274785+0000", 2024-09-16T03:38:51.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3731e-05 2024-09-16T03:38:51.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.277559+0000", 2024-09-16T03:38:51.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027741179999999999 2024-09-16T03:38:51.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.277604+0000", 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5624999999999998e-05 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2664 4.f e91/32 4:f817def9:::smithi15736696-45:head v 91'71, mlcod=89'69)", 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.623500+0000", 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.868480028, 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20350020799999999, 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.530 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.623500+0000", 2024-09-16T03:38:51.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.623500+0000", 2024-09-16T03:38:51.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.623501+0000", 2024-09-16T03:38:51.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.99e-07 2024-09-16T03:38:51.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624286+0000", 2024-09-16T03:38:51.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078502800000000001 2024-09-16T03:38:51.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624295+0000", 2024-09-16T03:38:51.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8630000000000003e-06 2024-09-16T03:38:51.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624302+0000", 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0589999999999999e-06 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624347+0000", 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5531000000000002e-05 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624362+0000", 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4571e-05 2024-09-16T03:38:51.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.826968+0000", 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20260592199999999 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.827001+0000", 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3034999999999998e-05 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2665 4.f e91/32 4:f817def9:::smithi15736696-45:head v 91'72, mlcod=89'69)", 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.624911+0000", 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.867069102999999, 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20261111400000001, 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624911+0000", 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624911+0000", 2024-09-16T03:38:51.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624912+0000", 2024-09-16T03:38:51.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7000000000000004e-07 2024-09-16T03:38:51.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.625313+0000", 2024-09-16T03:38:51.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040085000000000001 2024-09-16T03:38:51.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.625321+0000", 2024-09-16T03:38:51.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6210000000000004e-06 2024-09-16T03:38:51.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.625325+0000", 2024-09-16T03:38:51.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3800000000000004e-06 2024-09-16T03:38:51.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.825578+0000", 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20025301500000001 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.825591+0000", 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3326000000000001e-05 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.827497+0000", 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001905728 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.827522+0000", 2024-09-16T03:38:51.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5423999999999999e-05 2024-09-16T03:38:51.539 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.539 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.539 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.539 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:3081 4.1f e102/100 4:fa163101:::smithi15736696-10:head v 102'93, mlcod=102'91)", 2024-09-16T03:38:51.539 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:35.333773+0000", 2024-09-16T03:38:51.539 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.158207106999999, 2024-09-16T03:38:51.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20354451600000001, 2024-09-16T03:38:51.539 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.539 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.333773+0000", 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.333773+0000", 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.333774+0000", 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1570000000000001e-06 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.334419+0000", 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00064475100000000005 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.334444+0000", 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5123e-05 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.434544+0000", 2024-09-16T03:38:51.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10009976499999999 2024-09-16T03:38:51.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.434625+0000", 2024-09-16T03:38:51.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0538999999999996e-05 2024-09-16T03:38:51.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.434652+0000", 2024-09-16T03:38:51.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.779e-05 2024-09-16T03:38:51.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.537288+0000", 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102635448 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.537318+0000", 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9943000000000001e-05 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:219 3.0 e103/101 3:21e0c4e8:::f1ac9f6b1092b2dbfccdb2f32bdf689fb55e405da82aefeec91d0d73cdce51e1:head v 103'198, mlcod=0'0)", 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:35.987822+0000", 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.504158215, 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30141685800000001, 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.987822+0000", 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.987822+0000", 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.987823+0000", 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0900000000000002e-07 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.987826+0000", 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8199999999999998e-06 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.987831+0000", 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0990000000000001e-06 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.087895+0000", 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100064234 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.087948+0000", 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2781999999999997e-05 2024-09-16T03:38:51.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.087971+0000", 2024-09-16T03:38:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3108e-05 2024-09-16T03:38:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.289201+0000", 2024-09-16T03:38:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.201230722 2024-09-16T03:38:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.289239+0000", 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7684e-05 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "MOSDPGPush(3.0 103/101 [PushOp(3:10a17b1b:::13dc2fab1f287170fabf96291b912bf80e0b081f291f2c6d41d494566f2d1f8c:head, version: 86'177, data_included: [327680~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:10a17b1b:::13dc2fab1f287170fabf96291b912bf80e0b081f291f2c6d41d494566f2d1f8c:head@86'177, size: 2098688, copy_subset: [(0, 2098688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1376256, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:36.087954+0000", 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.404026824000001, 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20123582400000001, 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:51.548 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.087954+0000", 2024-09-16T03:38:51.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.087954+0000", 2024-09-16T03:38:51.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.087954+0000", 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.22e-07 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.088619+0000", 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00066490699999999995 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.088636+0000", 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6779e-05 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.088653+0000", 2024-09-16T03:38:51.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7200000000000001e-05 2024-09-16T03:38:51.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.288227+0000", 2024-09-16T03:38:51.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19957375199999999 2024-09-16T03:38:51.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.288238+0000", 2024-09-16T03:38:51.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1268e-05 2024-09-16T03:38:51.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.289189+0000", 2024-09-16T03:38:51.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00095159599999999997 2024-09-16T03:38:51.552 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.552 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.552 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.552 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:3100 3.0 e103/101 3:2db88641:::a63ac782fa4bce1e96397c8301f2734ce3d5850a2dc454a54c42fc5451a8d72d:head v 103'199, mlcod=103'196)", 2024-09-16T03:38:51.552 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:36.295147+0000", 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.196833452, 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20098897099999999, 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.295147+0000", 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.295147+0000", 2024-09-16T03:38:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.295149+0000", 2024-09-16T03:38:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8729999999999999e-06 2024-09-16T03:38:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.295161+0000", 2024-09-16T03:38:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2459e-05 2024-09-16T03:38:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.295172+0000", 2024-09-16T03:38:51.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0509000000000001e-05 2024-09-16T03:38:51.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.295179+0000", 2024-09-16T03:38:51.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7220000000000002e-06 2024-09-16T03:38:51.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.495408+0000", 2024-09-16T03:38:51.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200229243 2024-09-16T03:38:51.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.495447+0000", 2024-09-16T03:38:51.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8955999999999998e-05 2024-09-16T03:38:51.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.496051+0000", 2024-09-16T03:38:51.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060449099999999997 2024-09-16T03:38:51.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.496136+0000", 2024-09-16T03:38:51.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4717999999999998e-05 2024-09-16T03:38:51.557 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.557 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.557 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.557 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:224 3.0 e104/101 3:2db88641:::a63ac782fa4bce1e96397c8301f2734ce3d5850a2dc454a54c42fc5451a8d72d:head v 104'200, mlcod=103'198)", 2024-09-16T03:38:51.557 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:36.758244+0000", 2024-09-16T03:38:51.557 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.733736432000001, 2024-09-16T03:38:51.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20094310700000001, 2024-09-16T03:38:51.557 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.558 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.558 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.558 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.758244+0000", 2024-09-16T03:38:51.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.758244+0000", 2024-09-16T03:38:51.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.758245+0000", 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4300000000000002e-07 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.758250+0000", 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9559999999999996e-06 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.758256+0000", 2024-09-16T03:38:51.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.674e-06 2024-09-16T03:38:51.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.758260+0000", 2024-09-16T03:38:51.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0940000000000001e-06 2024-09-16T03:38:51.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.958724+0000", 2024-09-16T03:38:51.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20046372700000001 2024-09-16T03:38:51.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.958742+0000", 2024-09-16T03:38:51.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7869000000000001e-05 2024-09-16T03:38:51.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.959156+0000", 2024-09-16T03:38:51.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00041438199999999998 2024-09-16T03:38:51.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.959187+0000", 2024-09-16T03:38:51.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0661999999999999e-05 2024-09-16T03:38:51.562 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.562 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.562 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.562 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:3260 4.f e107/100 4:f0d14c71:::smithi15736696-8:head v 107'91, mlcod=0'0)", 2024-09-16T03:38:51.562 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:40.646237+0000", 2024-09-16T03:38:51.562 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.845743206, 2024-09-16T03:38:51.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20284116699999999, 2024-09-16T03:38:51.562 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.562 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.562 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.562 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.646237+0000", 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.646237+0000", 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.646238+0000", 2024-09-16T03:38:51.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0300000000000001e-06 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.647035+0000", 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00079685300000000004 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.647045+0000", 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7140000000000005e-06 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.647050+0000", 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0030000000000002e-06 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.747166+0000", 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100116127 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.747190+0000", 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4474e-05 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.849053+0000", 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101862482 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.849078+0000", 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5483999999999999e-05 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:3261 4.f e107/100 4:f0d14c71:::smithi15736696-8:head v 107'92, mlcod=0'0)", 2024-09-16T03:38:51.566 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:40.648159+0000", 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.843821760999999, 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20121599200000001, 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.648159+0000", 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.648159+0000", 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.648160+0000", 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9000000000000005e-07 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.648250+0000", 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0435999999999999e-05 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.648259+0000", 2024-09-16T03:38:51.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5720000000000003e-06 2024-09-16T03:38:51.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.648267+0000", 2024-09-16T03:38:51.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5369999999999997e-06 2024-09-16T03:38:51.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.848733+0000", 2024-09-16T03:38:51.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200465325 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.848747+0000", 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4948e-05 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.849341+0000", 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00059359600000000005 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.849375+0000", 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3587999999999998e-05 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3402 4.1b 4.1792cddb (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:44.983846+0000", 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 6.5081348649999997, 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53764449299999995, 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:38:51.571 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:51.572 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:51.572 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.572 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3402 2024-09-16T03:38:51.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.572 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983846+0000", 2024-09-16T03:38:51.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983846+0000", 2024-09-16T03:38:51.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983855+0000", 2024-09-16T03:38:51.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2059999999999996e-06 2024-09-16T03:38:51.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983860+0000", 2024-09-16T03:38:51.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3990000000000003e-06 2024-09-16T03:38:51.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983869+0000", 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5609999999999999e-06 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983875+0000", 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9900000000000002e-06 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983933+0000", 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8519e-05 2024-09-16T03:38:51.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983940+0000", 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1579999999999999e-06 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:45.521480+0000", 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53753933300000001 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:45.521490+0000", 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0327e-05 2024-09-16T03:38:51.575 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.576 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.576 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.576 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3567 4.1b 4:dbb349e8:::smithi15736696-46:head [setxattr _header (56) in=63b,truncate 3479082] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:38:51.576 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:50.999803+0000", 2024-09-16T03:38:51.576 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.49217786099999999, 2024-09-16T03:38:51.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.257763882, 2024-09-16T03:38:51.576 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.576 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.576 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.576 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:51.576 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3567 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999803+0000", 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999803+0000", 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999803+0000", 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7199999999999998e-07 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999807+0000", 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.782e-06 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999813+0000", 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8300000000000001e-06 2024-09-16T03:38:51.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999816+0000", 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3050000000000001e-06 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001310+0000", 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001494057 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001397+0000", 2024-09-16T03:38:51.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6458000000000005e-05 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001551+0000", 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015404000000000001 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001975+0000", 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042458099999999999 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:51.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044734+0000", 2024-09-16T03:38:51.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.042758530000000003 2024-09-16T03:38:51.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:51.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257487+0000", 2024-09-16T03:38:51.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.212752841 2024-09-16T03:38:51.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257519+0000", 2024-09-16T03:38:51.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2113000000000002e-05 2024-09-16T03:38:51.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257566+0000", 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7573e-05 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3568 4.1b 4:dbb349e8:::smithi15736696-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:50.999832+0000", 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.49214844099999999, 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25787906900000002, 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.582 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.583 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:51.583 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:51.583 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:51.583 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.583 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3568 2024-09-16T03:38:51.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.583 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999832+0000", 2024-09-16T03:38:51.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999832+0000", 2024-09-16T03:38:51.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999833+0000", 2024-09-16T03:38:51.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.51e-07 2024-09-16T03:38:51.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999835+0000", 2024-09-16T03:38:51.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5330000000000001e-06 2024-09-16T03:38:51.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999841+0000", 2024-09-16T03:38:51.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9939999999999997e-06 2024-09-16T03:38:51.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999846+0000", 2024-09-16T03:38:51.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.31e-06 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001833+0000", 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019864890000000001 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001875+0000", 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2113999999999997e-05 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257607+0000", 2024-09-16T03:38:51.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25573209000000002 2024-09-16T03:38:51.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257651+0000", 2024-09-16T03:38:51.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4233000000000002e-05 2024-09-16T03:38:51.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.587 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257711+0000", 2024-09-16T03:38:51.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9755e-05 2024-09-16T03:38:51.587 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.587 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.588 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.588 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3589 4.1b 4:dcb6965e:::smithi15736696-28:head [write 729503~560100 in=560100b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:38:51.588 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.044505+0000", 2024-09-16T03:38:51.588 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.44747565299999997, 2024-09-16T03:38:51.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.21331082500000001, 2024-09-16T03:38:51.588 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.588 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.588 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.588 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:38:51.588 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:38:51.588 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3589 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044505+0000", 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044505+0000", 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044508+0000", 2024-09-16T03:38:51.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5190000000000001e-06 2024-09-16T03:38:51.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044633+0000", 2024-09-16T03:38:51.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000124447 2024-09-16T03:38:51.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044653+0000", 2024-09-16T03:38:51.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0032999999999999e-05 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044663+0000", 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0487e-05 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052467+0000", 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0078037280000000002 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052561+0000", 2024-09-16T03:38:51.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3879e-05 2024-09-16T03:38:51.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:38:51.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052665+0000", 2024-09-16T03:38:51.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000104613 2024-09-16T03:38:51.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:51.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.054365+0000", 2024-09-16T03:38:51.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001699983 2024-09-16T03:38:51.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156645+0000", 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10227989699999999 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257746+0000", 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10110100800000001 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257780+0000", 2024-09-16T03:38:51.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4001999999999997e-05 2024-09-16T03:38:51.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257816+0000", 2024-09-16T03:38:51.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5228999999999998e-05 2024-09-16T03:38:51.594 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.594 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.594 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.594 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.594 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.594 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:51.596 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:38:51.596 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 179482 2024-09-16T03:38:51.597 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:51.594+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:51.685 INFO:tasks.rados.rados.0.smithi157.stdout:1550: expect (ObjNum 303 snap 0 seq_num 303) 2024-09-16T03:38:51.685 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:51.697 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:51.695+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:51.740 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:51.740 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:51.740 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:38:51.740 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:51.749 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:38:51.797 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:51.796+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:51.809 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 942512 2024-09-16T03:38:51.851 INFO:tasks.rados.rados.0.smithi157.stdout:1553: expect (ObjNum 323 snap 0 seq_num 323) 2024-09-16T03:38:51.854 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:51.854 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:38:51.854 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:38:51.854 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:38:51.854 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:51.863 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:38:51.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:38:51.899 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:51.897+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:51.905 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:51.969 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:51.970 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:38:51.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:38:51.970 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:51.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.970 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:51.970 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:38:51.970 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 99.780723257999995, 2024-09-16T03:38:51.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102630163, 2024-09-16T03:38:51.970 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.970 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.970 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.970 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:38:51.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:38:51.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186081+0000", 2024-09-16T03:38:51.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5600000000000005e-07 2024-09-16T03:38:51.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186970+0000", 2024-09-16T03:38:51.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00088889000000000004 2024-09-16T03:38:51.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186979+0000", 2024-09-16T03:38:51.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0620000000000007e-06 2024-09-16T03:38:51.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186986+0000", 2024-09-16T03:38:51.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:38:51.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287184+0000", 2024-09-16T03:38:51.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100198036 2024-09-16T03:38:51.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287199+0000", 2024-09-16T03:38:51.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5458999999999999e-05 2024-09-16T03:38:51.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288659+0000", 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014601900000000001 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288711+0000", 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1088000000000003e-05 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.975 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:51.975 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:38:51.975 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 99.779686824999999, 2024-09-16T03:38:51.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10205370900000001, 2024-09-16T03:38:51.975 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.975 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.975 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.975 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:38:51.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:38:51.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187118+0000", 2024-09-16T03:38:51.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8599999999999998e-07 2024-09-16T03:38:51.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188062+0000", 2024-09-16T03:38:51.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094447300000000004 2024-09-16T03:38:51.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188070+0000", 2024-09-16T03:38:51.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.695e-06 2024-09-16T03:38:51.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188074+0000", 2024-09-16T03:38:51.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0709999999999996e-06 2024-09-16T03:38:51.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287842+0000", 2024-09-16T03:38:51.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099768478999999993 2024-09-16T03:38:51.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287864+0000", 2024-09-16T03:38:51.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1679999999999999e-05 2024-09-16T03:38:51.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289137+0000", 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012732430000000001 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289171+0000", 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3382e-05 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:42 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 99.709604916999993, 2024-09-16T03:38:51.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10172858, 2024-09-16T03:38:51.980 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.980 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.980 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.980 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:38:51.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:38:51.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257201+0000", 2024-09-16T03:38:51.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:38:51.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257536+0000", 2024-09-16T03:38:51.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033513899999999998 2024-09-16T03:38:51.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257555+0000", 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8547999999999999e-05 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257566+0000", 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1379999999999999e-05 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357720+0000", 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100153532 2024-09-16T03:38:51.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357734+0000", 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4491000000000001e-05 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358896+0000", 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011620599999999999 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358927+0000", 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1158999999999998e-05 2024-09-16T03:38:51.983 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.984 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.984 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:43 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'3, mlcod=0'0)", 2024-09-16T03:38:51.984 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:38:51.984 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 99.707821005, 2024-09-16T03:38:51.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101764453, 2024-09-16T03:38:51.984 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.984 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.984 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.984 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:38:51.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:38:51.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258984+0000", 2024-09-16T03:38:51.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5689999999999999e-06 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258994+0000", 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259008+0000", 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4151999999999999e-05 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359073+0000", 2024-09-16T03:38:51.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100065628 2024-09-16T03:38:51.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359110+0000", 2024-09-16T03:38:51.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7045999999999997e-05 2024-09-16T03:38:51.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359124+0000", 2024-09-16T03:38:51.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.382e-05 2024-09-16T03:38:51.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360716+0000", 2024-09-16T03:38:51.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015919829999999999 2024-09-16T03:38:51.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360747+0000", 2024-09-16T03:38:51.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0874999999999998e-05 2024-09-16T03:38:51.988 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.988 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.988 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.989 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:492 4.5 e45/32 4:a11e1d7c:::smithi15736696-43:head v 45'22, mlcod=45'20)", 2024-09-16T03:38:51.989 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:38:51.989 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 90.143452718000006, 2024-09-16T03:38:51.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100797151, 2024-09-16T03:38:51.989 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.989 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.989 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.989 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:38:51.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:38:51.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823353+0000", 2024-09-16T03:38:51.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2450000000000001e-06 2024-09-16T03:38:51.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823363+0000", 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0074999999999999e-05 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823378+0000", 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4534e-05 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923461+0000", 2024-09-16T03:38:51.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100082859 2024-09-16T03:38:51.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923525+0000", 2024-09-16T03:38:51.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4616999999999995e-05 2024-09-16T03:38:51.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923549+0000", 2024-09-16T03:38:51.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.357e-05 2024-09-16T03:38:51.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924106+0000", 2024-09-16T03:38:51.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000556753 2024-09-16T03:38:51.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924148+0000", 2024-09-16T03:38:51.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2497999999999999e-05 2024-09-16T03:38:51.993 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.993 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.993 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.993 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:808 4.5 e47/32 4:aff4cc41:::smithi15736696-11:head v 47'27, mlcod=45'22)", 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 85.943759443000005, 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.097081386000000006, 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023044+0000", 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023046+0000", 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1059999999999998e-06 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023056+0000", 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3719999999999994e-06 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:51.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023067+0000", 2024-09-16T03:38:51.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1246999999999999e-05 2024-09-16T03:38:51.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:51.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.023075+0000", 2024-09-16T03:38:51.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4710000000000007e-06 2024-09-16T03:38:51.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:51.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.119708+0000", 2024-09-16T03:38:51.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096632179999999998 2024-09-16T03:38:51.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:51.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.119725+0000", 2024-09-16T03:38:51.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7754999999999999e-05 2024-09-16T03:38:51.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:51.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.120092+0000", 2024-09-16T03:38:51.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00036668099999999999 2024-09-16T03:38:51.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:51.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.120126+0000", 2024-09-16T03:38:51.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3574000000000001e-05 2024-09-16T03:38:51.998 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.998 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:51.998 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:51.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.998 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:53'30,epoch:57/50,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:38:51.998 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:38:51.998 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 71.568101102, 2024-09-16T03:38:51.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100230893, 2024-09-16T03:38:51.998 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:51.998 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:51.998 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:51.999 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:51.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:51.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:38:51.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:51.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:51.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:38:51.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:51.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:51.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398705+0000", 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5689999999999999e-06 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398722+0000", 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6558e-05 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398735+0000", 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3499999999999999e-05 2024-09-16T03:38:52.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398746+0000", 2024-09-16T03:38:52.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1182999999999999e-05 2024-09-16T03:38:52.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498898+0000", 2024-09-16T03:38:52.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015112599999999 2024-09-16T03:38:52.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498906+0000", 2024-09-16T03:38:52.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3189999999999995e-06 2024-09-16T03:38:52.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498933+0000", 2024-09-16T03:38:52.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7637999999999999e-05 2024-09-16T03:38:52.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.002 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.002 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1570 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'31, mlcod=48'29)", 2024-09-16T03:38:52.002 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:38:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 65.792494106999996, 2024-09-16T03:38:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10253683500000001, 2024-09-16T03:38:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:38:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8500000000000002e-07 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174689+0000", 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037958600000000001 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174699+0000", 2024-09-16T03:38:52.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0015e-05 2024-09-16T03:38:52.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174706+0000", 2024-09-16T03:38:52.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6250000000000001e-06 2024-09-16T03:38:52.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274865+0000", 2024-09-16T03:38:52.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100159157 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274879+0000", 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3983e-05 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276820+0000", 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001941248 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276846+0000", 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5936e-05 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1571 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'32, mlcod=48'29)", 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 65.792043352999997, 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10249543, 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:52.007 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:38:52.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:38:52.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174761+0000", 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0800000000000004e-07 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175053+0000", 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029151600000000002 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175063+0000", 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0538e-05 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175070+0000", 2024-09-16T03:38:52.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7819999999999996e-06 2024-09-16T03:38:52.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275895+0000", 2024-09-16T03:38:52.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10082516 2024-09-16T03:38:52.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275909+0000", 2024-09-16T03:38:52.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4197e-05 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277232+0000", 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013231009999999999 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277256+0000", 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3428e-05 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.012 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1572 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'33, mlcod=48'29)", 2024-09-16T03:38:52.012 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:38:52.012 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 65.791702052999995, 2024-09-16T03:38:52.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10216886, 2024-09-16T03:38:52.012 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.012 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.012 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:52.012 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:38:52.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8400000000000004e-07 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175107+0000", 2024-09-16T03:38:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.578e-06 2024-09-16T03:38:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175114+0000", 2024-09-16T03:38:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.888e-06 2024-09-16T03:38:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175119+0000", 2024-09-16T03:38:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.074e-06 2024-09-16T03:38:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276662+0000", 2024-09-16T03:38:52.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101542753 2024-09-16T03:38:52.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276674+0000", 2024-09-16T03:38:52.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1960000000000001e-05 2024-09-16T03:38:52.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:52.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277258+0000", 2024-09-16T03:38:52.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058431900000000003 2024-09-16T03:38:52.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277270+0000", 2024-09-16T03:38:52.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2604e-05 2024-09-16T03:38:52.016 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.016 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.016 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.016 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:60'33,epoch:61/50,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:38:52.016 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:38:52.016 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 65.289780798999999, 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10023048699999999, 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:38:52.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:38:52.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8599999999999999e-07 2024-09-16T03:38:52.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677027+0000", 2024-09-16T03:38:52.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7359999999999999e-06 2024-09-16T03:38:52.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677042+0000", 2024-09-16T03:38:52.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4888e-05 2024-09-16T03:38:52.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677052+0000", 2024-09-16T03:38:52.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7750000000000004e-06 2024-09-16T03:38:52.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777182+0000", 2024-09-16T03:38:52.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100130394 2024-09-16T03:38:52.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777195+0000", 2024-09-16T03:38:52.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3373999999999999e-05 2024-09-16T03:38:52.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777253+0000", 2024-09-16T03:38:52.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7933999999999998e-05 2024-09-16T03:38:52.020 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.020 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.020 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.021 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2454 4.15 e84/50 4:aff4cc41:::smithi15736696-11:head v 84'41, mlcod=79'39)", 2024-09-16T03:38:52.021 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:38:52.021 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.503564691000001, 2024-09-16T03:38:52.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101244889, 2024-09-16T03:38:52.021 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.021 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.021 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:52.021 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:38:52.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:38:52.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:38:52.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2899999999999999e-07 2024-09-16T03:38:52.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463249+0000", 2024-09-16T03:38:52.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8039999999999996e-06 2024-09-16T03:38:52.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463267+0000", 2024-09-16T03:38:52.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7955e-05 2024-09-16T03:38:52.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563340+0000", 2024-09-16T03:38:52.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100072999 2024-09-16T03:38:52.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563396+0000", 2024-09-16T03:38:52.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6141999999999999e-05 2024-09-16T03:38:52.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563414+0000", 2024-09-16T03:38:52.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8088999999999999e-05 2024-09-16T03:38:52.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:52.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.564427+0000", 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010127230000000001 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.564484+0000", 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6747999999999999e-05 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2467 4.5 e85/50)", 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:38:52.025 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.815672565, 2024-09-16T03:38:52.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102439455, 2024-09-16T03:38:52.026 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.026 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.026 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:52.026 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:38:52.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151134+0000", 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2030000000000001e-06 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151480+0000", 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034613999999999999 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151500+0000", 2024-09-16T03:38:52.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9582999999999999e-05 2024-09-16T03:38:52.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151513+0000", 2024-09-16T03:38:52.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:38:52.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251653+0000", 2024-09-16T03:38:52.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10014007900000001 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251679+0000", 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5721e-05 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253532+0000", 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001852701 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253571+0000", 2024-09-16T03:38:52.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8677999999999998e-05 2024-09-16T03:38:52.030 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.030 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.030 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.030 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2468 4.5 e85/50)", 2024-09-16T03:38:52.030 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:38:52.030 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.814204036, 2024-09-16T03:38:52.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101505239, 2024-09-16T03:38:52.030 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.030 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.030 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152601+0000", 2024-09-16T03:38:52.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6869999999999999e-06 2024-09-16T03:38:52.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152719+0000", 2024-09-16T03:38:52.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117508 2024-09-16T03:38:52.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152732+0000", 2024-09-16T03:38:52.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:38:52.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152740+0000", 2024-09-16T03:38:52.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7090000000000002e-06 2024-09-16T03:38:52.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252704+0000", 2024-09-16T03:38:52.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099964259999999999 2024-09-16T03:38:52.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252727+0000", 2024-09-16T03:38:52.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3601e-05 2024-09-16T03:38:52.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:52.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254075+0000", 2024-09-16T03:38:52.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00134772 2024-09-16T03:38:52.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254105+0000", 2024-09-16T03:38:52.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9802e-05 2024-09-16T03:38:52.034 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.034 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.034 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2623 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'62, mlcod=85'60)", 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 31.617566406000002, 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.104590794, 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:38:52.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:38:52.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:38:52.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1899999999999999e-07 2024-09-16T03:38:52.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349496+0000", 2024-09-16T03:38:52.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000258934 2024-09-16T03:38:52.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349506+0000", 2024-09-16T03:38:52.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9389999999999992e-06 2024-09-16T03:38:52.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349513+0000", 2024-09-16T03:38:52.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6209999999999998e-06 2024-09-16T03:38:52.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449668+0000", 2024-09-16T03:38:52.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015550099999999 2024-09-16T03:38:52.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449695+0000", 2024-09-16T03:38:52.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6253000000000001e-05 2024-09-16T03:38:52.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:52.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453784+0000", 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040896609999999996 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453828+0000", 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3665999999999998e-05 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2624 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'63, mlcod=85'60)", 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:38:52.039 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 31.616219168000001, 2024-09-16T03:38:52.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103286796, 2024-09-16T03:38:52.040 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.040 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.040 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:52.040 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:38:52.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:38:52.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350586+0000", 2024-09-16T03:38:52.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0020000000000001e-06 2024-09-16T03:38:52.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350794+0000", 2024-09-16T03:38:52.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020824099999999999 2024-09-16T03:38:52.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350805+0000", 2024-09-16T03:38:52.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1443999999999999e-05 2024-09-16T03:38:52.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450876+0000", 2024-09-16T03:38:52.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100070612 2024-09-16T03:38:52.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450960+0000", 2024-09-16T03:38:52.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4444999999999995e-05 2024-09-16T03:38:52.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450980+0000", 2024-09-16T03:38:52.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9728000000000002e-05 2024-09-16T03:38:52.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:52.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453831+0000", 2024-09-16T03:38:52.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002850923 2024-09-16T03:38:52.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453871+0000", 2024-09-16T03:38:52.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0401000000000002e-05 2024-09-16T03:38:52.044 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.044 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.044 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.044 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2626 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'65, mlcod=85'60)", 2024-09-16T03:38:52.044 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:38:52.044 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 31.515515190999999, 2024-09-16T03:38:52.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100625902, 2024-09-16T03:38:52.045 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.045 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.045 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:52.045 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:38:52.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451289+0000", 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7100000000000002e-07 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451295+0000", 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2890000000000003e-06 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451300+0000", 2024-09-16T03:38:52.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0710000000000002e-06 2024-09-16T03:38:52.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551399+0000", 2024-09-16T03:38:52.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10009881 2024-09-16T03:38:52.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551482+0000", 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2738000000000006e-05 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551496+0000", 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4394e-05 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551881+0000", 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038522300000000002 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551914+0000", 2024-09-16T03:38:52.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2906e-05 2024-09-16T03:38:52.049 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.049 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.049 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.049 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2661 4.15 e91/50 4:aff4cc41:::smithi15736696-11:head v 91'52, mlcod=88'48)", 2024-09-16T03:38:52.049 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:38:52.049 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.557618938000001, 2024-09-16T03:38:52.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10164756699999999, 2024-09-16T03:38:52.049 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409186+0000", 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-06 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409568+0000", 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038196600000000002 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409578+0000", 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.781e-06 2024-09-16T03:38:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409583+0000", 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3070000000000002e-06 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409633+0000", 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0358000000000001e-05 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409649+0000", 2024-09-16T03:38:52.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5875999999999998e-05 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510800+0000", 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101150728 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510832+0000", 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2500999999999999e-05 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.054 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2662 4.15 e91/50 4:aff4cc41:::smithi15736696-11:head v 91'53, mlcod=88'48)", 2024-09-16T03:38:52.054 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:38:52.054 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.556410988, 2024-09-16T03:38:52.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100650287, 2024-09-16T03:38:52.054 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.054 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.054 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:52.054 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410394+0000", 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6299999999999993e-07 2024-09-16T03:38:52.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410399+0000", 2024-09-16T03:38:52.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9400000000000001e-06 2024-09-16T03:38:52.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410407+0000", 2024-09-16T03:38:52.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8859999999999993e-06 2024-09-16T03:38:52.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410412+0000", 2024-09-16T03:38:52.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9339999999999997e-06 2024-09-16T03:38:52.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510516+0000", 2024-09-16T03:38:52.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100103159 2024-09-16T03:38:52.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510536+0000", 2024-09-16T03:38:52.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0072999999999999e-05 2024-09-16T03:38:52.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:52.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.511018+0000", 2024-09-16T03:38:52.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00048191899999999998 2024-09-16T03:38:52.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.511043+0000", 2024-09-16T03:38:52.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5412999999999999e-05 2024-09-16T03:38:52.058 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.058 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.058 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2914 4.5 e98/50 4:a4fbca73:::smithi15736696-39:head v 98'72, mlcod=96'69)", 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:30.471084+0000", 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 21.495719416, 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10075181900000001, 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471084+0000", 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471084+0000", 2024-09-16T03:38:52.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471087+0000", 2024-09-16T03:38:52.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3580000000000001e-06 2024-09-16T03:38:52.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471099+0000", 2024-09-16T03:38:52.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.255e-05 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471116+0000", 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6867999999999998e-05 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471128+0000", 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2313999999999999e-05 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571266+0000", 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013743999999999 2024-09-16T03:38:52.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571286+0000", 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0313999999999999e-05 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571787+0000", 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00050111600000000002 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571836+0000", 2024-09-16T03:38:52.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8859000000000001e-05 2024-09-16T03:38:52.063 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.063 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.063 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.063 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.063 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.063 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:52.064 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:52.064 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1859412 2024-09-16T03:38:52.064 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:52.065 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:52.065 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:52.066 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:38:52.067 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:52.067 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:52.068 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:52.062+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:52.070 INFO:tasks.rados.rados.0.smithi157.stdout:1547: finishing write tid 3 to smithi15736696-46 2024-09-16T03:38:52.070 INFO:tasks.rados.rados.0.smithi157.stdout:1547: finishing write tid 4 to smithi15736696-46 2024-09-16T03:38:52.070 INFO:tasks.rados.rados.0.smithi157.stdout:1547: finishing write tid 5 to smithi15736696-46 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:1547: oid 46 updating version 0 to 113 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:1547: oid 46 updating version 113 to 115 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:1547: oid 46 updating version 115 to 116 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:1547: oid 46 version 116 is already newer than 114 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 116 (ObjNum 326 snap 0 seq_num 326) dirty exists 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:1547: left oid 46 (ObjNum 326 snap 0 seq_num 326) 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:1554: finishing write tid 1 to smithi15736696-28 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:1535: done (14 left) 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:1544: done (13 left) 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:1545: done (12 left) 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:1547: done (11 left) 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:1548: done (10 left) 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:1550: done (9 left) 2024-09-16T03:38:52.071 INFO:tasks.rados.rados.0.smithi157.stdout:1551: done (8 left) 2024-09-16T03:38:52.072 INFO:tasks.rados.rados.0.smithi157.stdout:1553: done (7 left) 2024-09-16T03:38:52.072 INFO:tasks.rados.rados.0.smithi157.stdout:1555: done (6 left) 2024-09-16T03:38:52.072 INFO:tasks.rados.rados.0.smithi157.stdout:1556: write (excl) oid 19 current snap is 0 2024-09-16T03:38:52.072 INFO:tasks.rados.rados.0.smithi157.stdout:1546: finishing set_chunk to oid 43 2024-09-16T03:38:52.072 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 81 (ObjNum 317 snap 0 seq_num 317) dirty exists 2024-09-16T03:38:52.072 INFO:tasks.rados.rados.0.smithi157.stdout:1556: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2024-09-16T03:38:52.072 INFO:tasks.rados.rados.0.smithi157.stdout:1556: writing smithi15736696-19 from 763117 to 1473715 tid 1 2024-09-16T03:38:52.072 INFO:tasks.rados.rados.0.smithi157.stdout:1556: writing smithi15736696-19 from 2136128 to 2856229 tid 2 2024-09-16T03:38:52.072 INFO:tasks.rados.rados.0.smithi157.stdout:1556: writing smithi15736696-19 from 3390187 to 3563117 tid 3 2024-09-16T03:38:52.072 INFO:tasks.rados.rados.0.smithi157.stdout:1546: done (6 left) 2024-09-16T03:38:52.072 INFO:tasks.rados.rados.0.smithi157.stdout:1557: tier_evict oid 35 2024-09-16T03:38:52.072 INFO:tasks.rados.rados.0.smithi157.stdout:1557: tier_evict oid 35 snap -1 2024-09-16T03:38:52.072 INFO:tasks.rados.rados.0.smithi157.stdout:1554: finishing write tid 2 to smithi15736696-28 2024-09-16T03:38:52.073 INFO:tasks.rados.rados.0.smithi157.stdout:1554: finishing write tid 3 to smithi15736696-28 2024-09-16T03:38:52.073 INFO:tasks.rados.rados.0.smithi157.stdout:1554: finishing write tid 4 to smithi15736696-28 2024-09-16T03:38:52.073 INFO:tasks.rados.rados.0.smithi157.stdout:1554: finishing write tid 5 to smithi15736696-28 2024-09-16T03:38:52.073 INFO:tasks.rados.rados.0.smithi157.stdout:1554: oid 28 updating version 0 to 119 2024-09-16T03:38:52.073 INFO:tasks.rados.rados.0.smithi157.stdout:1554: oid 28 version 119 is already newer than 117 2024-09-16T03:38:52.073 INFO:tasks.rados.rados.0.smithi157.stdout:1554: oid 28 version 119 is already newer than 118 2024-09-16T03:38:52.073 INFO:tasks.rados.rados.0.smithi157.stdout:1554: oid 28 updating version 119 to 120 2024-09-16T03:38:52.073 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 120 (ObjNum 329 snap 0 seq_num 329) dirty exists 2024-09-16T03:38:52.073 INFO:tasks.rados.rados.0.smithi157.stdout:1554: left oid 28 (ObjNum 329 snap 0 seq_num 329) 2024-09-16T03:38:52.073 INFO:tasks.rados.rados.0.smithi157.stdout:1554: done (6 left) 2024-09-16T03:38:52.073 INFO:tasks.rados.rados.0.smithi157.stdout:1558: read oid 37 snap -1 2024-09-16T03:38:52.073 INFO:tasks.rados.rados.0.smithi157.stdout:1558: expect (ObjNum 283 snap 0 seq_num 283) 2024-09-16T03:38:52.073 INFO:tasks.rados.rados.0.smithi157.stdout:1559: read oid 30 snap -1 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1559: expect deleted 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1560: read oid 16 snap -1 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1560: expect (ObjNum 270 snap 0 seq_num 270) 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1561: tier_flush oid 9 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1561: tier_flush oid 9 snap -1 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1562: write (excl) oid 41 current snap is 0 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1562: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1562: writing smithi15736696-41 from 779924 to 1565771 tid 1 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1562: writing smithi15736696-41 from 2078257 to 2658369 tid 2 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1562: writing smithi15736696-41 from 3301257 to 3579924 tid 3 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1563: write oid 24 current snap is 0 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1563: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1563: writing smithi15736696-24 from 796731 to 1257827 tid 1 2024-09-16T03:38:52.074 INFO:tasks.rados.rados.0.smithi157.stdout:1563: writing smithi15736696-24 from 1736739 to 2460509 tid 2 2024-09-16T03:38:52.075 INFO:tasks.rados.rados.0.smithi157.stdout:1563: writing smithi15736696-24 from 2928680 to 3596731 tid 3 2024-09-16T03:38:52.075 INFO:tasks.rados.rados.0.smithi157.stdout:1564: set_chunk oid 18 target oid 2024-09-16T03:38:52.077 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:52.077 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:52.093 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:52.093 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:52.111 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:52.111 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:52.128 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:52.128 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:52.146 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:52.146 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:52.162 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:52.162 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:52.165 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:52.163+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:52.217 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:38:52.217 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:38:52.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.217 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:78 3.2 3:6a11a5ad:::9ab3848d08148830c844f2e4402374f58aa4b3e9e8e9c3823807d389d026e16e:head [call cas.chunk_create_or_get_ref in=263496b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e117)", 2024-09-16T03:38:52.217 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:52.133856+0000", 2024-09-16T03:38:52.217 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.081603287999999996, 2024-09-16T03:38:52.217 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.081628999999999993, 2024-09-16T03:38:52.217 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.217 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.217 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:52.217 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.217 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.133856+0000", 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.133856+0000", 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.133857+0000", 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-06 2024-09-16T03:38:52.218 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134060+0000", 2024-09-16T03:38:52.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000203117 2024-09-16T03:38:52.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134068+0000", 2024-09-16T03:38:52.219 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0849999999999997e-06 2024-09-16T03:38:52.219 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.219 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.219 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.219 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134075+0000", 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2589999999999997e-06 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134874+0000", 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00079926499999999996 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.135006+0000", 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013237899999999999 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.220 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4", 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.135329+0000", 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00032249200000000002 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.138913+0000", 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0035846010000000002 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:80 3.2 3:76ddb3eb:::f9b88e4cb70678bbb52ec514510758ed89a245d67de945acb3373553defc4a2f:head [call cas.chunk_create_or_get_ref in=117607b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e117)", 2024-09-16T03:38:52.221 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:52.134087+0000", 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.081371794999999997, 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.081446107000000004, 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134087+0000", 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.222 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134087+0000", 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134088+0000", 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7000000000000004e-07 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134177+0000", 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8648999999999998e-05 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.223 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134186+0000", 2024-09-16T03:38:52.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5040000000000008e-06 2024-09-16T03:38:52.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134194+0000", 2024-09-16T03:38:52.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8669999999999994e-06 2024-09-16T03:38:52.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.224 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.224 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.136307+0000", 2024-09-16T03:38:52.224 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021133150000000002 2024-09-16T03:38:52.224 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.224 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.136423+0000", 2024-09-16T03:38:52.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011567500000000001 2024-09-16T03:38:52.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4", 2024-09-16T03:38:52.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.136576+0000", 2024-09-16T03:38:52.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015260999999999999 2024-09-16T03:38:52.225 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.225 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.225 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:52.225 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.138959+0000", 2024-09-16T03:38:52.225 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023834210000000002 2024-09-16T03:38:52.226 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.226 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.226 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.226 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.226 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.226 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:82 3.2 3:59711a04:::571f1a13ea01dc6ad2cd516e413c2c2201e75dca9b15ed482f116983a8f36525:head [call cas.chunk_create_or_get_ref in=479810b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e117)", 2024-09-16T03:38:52.226 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:52.134203+0000", 2024-09-16T03:38:52.226 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.081256604999999996, 2024-09-16T03:38:52.226 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.081364755999999996, 2024-09-16T03:38:52.226 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.226 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.226 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:52.226 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134203+0000", 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134203+0000", 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134203+0000", 2024-09-16T03:38:52.227 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8000000000000005e-07 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134663+0000", 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00045926199999999999 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134671+0000", 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8170000000000005e-06 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134679+0000", 2024-09-16T03:38:52.228 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3989999999999998e-06 2024-09-16T03:38:52.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.137039+0000", 2024-09-16T03:38:52.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023600219999999998 2024-09-16T03:38:52.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.229 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.137124+0000", 2024-09-16T03:38:52.229 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5329999999999998e-05 2024-09-16T03:38:52.229 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.229 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.229 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4", 2024-09-16T03:38:52.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.137302+0000", 2024-09-16T03:38:52.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017766 2024-09-16T03:38:52.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.230 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:52.230 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.139031+0000", 2024-09-16T03:38:52.230 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001728764 2024-09-16T03:38:52.230 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.230 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.230 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.230 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.230 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:86 3.2 3:45787732:::181469fdfd885e9f7eabb30f6c72c38096bbc58d6098a2242d5937f8341074ea:head [call cas.chunk_create_or_get_ref in=143859b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e117)", 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:52.134689+0000", 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.080770550999999996, 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.080911548, 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134689+0000", 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.231 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134689+0000", 2024-09-16T03:38:52.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134689+0000", 2024-09-16T03:38:52.232 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3699999999999999e-07 2024-09-16T03:38:52.232 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.232 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.232 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.232 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134802+0000", 2024-09-16T03:38:52.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112537 2024-09-16T03:38:52.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134810+0000", 2024-09-16T03:38:52.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.593e-06 2024-09-16T03:38:52.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.233 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.233 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134819+0000", 2024-09-16T03:38:52.233 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2809999999999997e-06 2024-09-16T03:38:52.233 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.233 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.137921+0000", 2024-09-16T03:38:52.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003102058 2024-09-16T03:38:52.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.137991+0000", 2024-09-16T03:38:52.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9664000000000003e-05 2024-09-16T03:38:52.234 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.234 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.234 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4", 2024-09-16T03:38:52.234 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.138146+0000", 2024-09-16T03:38:52.234 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000155322 2024-09-16T03:38:52.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.235 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:52.235 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.139060+0000", 2024-09-16T03:38:52.235 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00091396000000000001 2024-09-16T03:38:52.235 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.235 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.235 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.235 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.235 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.235 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:89 3.2 3.80b63106 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e117)", 2024-09-16T03:38:52.235 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:52.134830+0000", 2024-09-16T03:38:52.235 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.080628798000000002, 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.080799994, 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134830+0000", 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134830+0000", 2024-09-16T03:38:52.236 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134831+0000", 2024-09-16T03:38:52.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0999999999999997e-07 2024-09-16T03:38:52.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.237 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134901+0000", 2024-09-16T03:38:52.237 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9760999999999998e-05 2024-09-16T03:38:52.237 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.237 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.237 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134908+0000", 2024-09-16T03:38:52.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2869999999999999e-06 2024-09-16T03:38:52.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.238 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.238 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134917+0000", 2024-09-16T03:38:52.238 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4449999999999996e-06 2024-09-16T03:38:52.238 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.238 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.238 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.238 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.238 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.238 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:91 3.2 3.f566efbe (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e117)", 2024-09-16T03:38:52.239 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:52.134925+0000", 2024-09-16T03:38:52.239 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.080533837999999996, 2024-09-16T03:38:52.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.080725194, 2024-09-16T03:38:52.239 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.239 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.239 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "no flag points reached", 2024-09-16T03:38:52.239 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.239 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.239 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134925+0000", 2024-09-16T03:38:52.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.239 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134925+0000", 2024-09-16T03:38:52.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134926+0000", 2024-09-16T03:38:52.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5300000000000003e-07 2024-09-16T03:38:52.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.135877+0000", 2024-09-16T03:38:52.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00095047899999999995 2024-09-16T03:38:52.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.135891+0000", 2024-09-16T03:38:52.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4212e-05 2024-09-16T03:38:52.241 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.241 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.241 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.241 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:3617 4.1a e117/111)", 2024-09-16T03:38:52.241 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:52.156419+0000", 2024-09-16T03:38:52.241 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.059040079000000002, 2024-09-16T03:38:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.059248787999999997, 2024-09-16T03:38:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "no flag points reached", 2024-09-16T03:38:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.156419+0000", 2024-09-16T03:38:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.156419+0000", 2024-09-16T03:38:52.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.156422+0000", 2024-09-16T03:38:52.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.399e-06 2024-09-16T03:38:52.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.156450+0000", 2024-09-16T03:38:52.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7817000000000002e-05 2024-09-16T03:38:52.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.156470+0000", 2024-09-16T03:38:52.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9573999999999999e-05 2024-09-16T03:38:52.244 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.244 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.244 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.244 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:77 3.2 3:5595a68d:::07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541:head [call cas.chunk_create_or_get_ref in=524387b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e117)", 2024-09-16T03:38:52.244 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:52.133154+0000", 2024-09-16T03:38:52.244 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.082305301999999997, 2024-09-16T03:38:52.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.082535563000000006, 2024-09-16T03:38:52.245 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:38:52.245 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:38:52.245 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:38:52.245 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:38:52.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:38:52.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.133154+0000", 2024-09-16T03:38:52.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:38:52.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.133154+0000", 2024-09-16T03:38:52.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:38:52.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:38:52.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.133157+0000", 2024-09-16T03:38:52.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7319999999999998e-06 2024-09-16T03:38:52.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:38:52.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.133773+0000", 2024-09-16T03:38:52.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00061617099999999999 2024-09-16T03:38:52.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.133806+0000", 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3602000000000001e-05 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.133822+0000", 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5688e-05 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134135+0000", 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031339600000000002 2024-09-16T03:38:52.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:38:52.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134260+0000", 2024-09-16T03:38:52.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012485999999999999 2024-09-16T03:38:52.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4", 2024-09-16T03:38:52.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.134484+0000", 2024-09-16T03:38:52.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000224145 2024-09-16T03:38:52.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:38:52.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.136985+0000", 2024-09-16T03:38:52.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0025009089999999999 2024-09-16T03:38:52.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:38:52.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:38:52.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:52.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:52.137019+0000", 2024-09-16T03:38:52.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4066000000000002e-05 2024-09-16T03:38:52.249 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.249 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:38:52.249 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.249 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:38:52.249 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-16T03:38:52.249 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 8 2024-09-16T03:38:52.249 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:38:52.250 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:38:52.253 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:52.254 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:52.270 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:52.271 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:52.276 INFO:tasks.rados.rados.0.smithi157.stdout:1564: set_chunk oid 18 offset: 564736 length: 2429440 target oid b93497f70debd179a82519568f31393a4bc15f22e64a03250eb2dd8580fbc91a offset: 0 snap -1 2024-09-16T03:38:52.276 INFO:tasks.rados.rados.0.smithi157.stdout:1565: read oid 40 snap -1 2024-09-16T03:38:52.276 INFO:tasks.rados.rados.0.smithi157.stdout:1565: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:38:52.276 INFO:tasks.rados.rados.0.smithi157.stdout:1566: read oid 43 snap -1 2024-09-16T03:38:52.276 INFO:tasks.rados.rados.0.smithi157.stdout:1566: expect (ObjNum 317 snap 0 seq_num 317) 2024-09-16T03:38:52.276 INFO:tasks.rados.rados.0.smithi157.stdout:1567: set_chunk oid 34 target oid 2024-09-16T03:38:52.277 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:52.273+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:52.289 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:52.289 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:52.305 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:52.305 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:52.322 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:52.322 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:52.339 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:52.339 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:52.356 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:52.354+0000 7f12e4a1e640 -1 osd.5 117 handle_osd_map injecting map crc failure 2024-09-16T03:38:52.356 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:52.356 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:52.359 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:38:52.360 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:52.358+0000 7f49419d0640 -1 osd.4 117 handle_osd_map injecting map crc failure 2024-09-16T03:38:52.366 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:38:52.367 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:52.365+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:52.376 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:52.373+0000 7f66f5209640 -1 osd.15 117 handle_osd_map injecting map crc failure 2024-09-16T03:38:52.467 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:52.464+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:52.483 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:52.483 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:52.492 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:38:52.495 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:52.495 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:52.565 INFO:tasks.rados.rados.0.smithi157.stdout:1567: set_chunk oid 34 offset: 2139648 length: 95744 target oid bd01b52c60cccb31921d95bc79115f245264f797f0abcd2d9da69820e0a6e18d offset: 0 snap -1 2024-09-16T03:38:52.565 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:52.565 INFO:tasks.rados.rados.0.smithi157.stdout:1558: expect (ObjNum 283 snap 0 seq_num 283) 2024-09-16T03:38:52.566 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:52.568 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:52.566+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:52.639 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:52.639 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:52.648 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:38:52.689 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:52.687+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:52.728 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:52.769 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:52.768+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:52.794 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:52.794 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:52.803 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:38:52.812 INFO:tasks.rados.rados.0.smithi157.stdout:1556: finishing write tid 1 to smithi15736696-19 2024-09-16T03:38:52.812 INFO:tasks.rados.rados.0.smithi157.stdout:1558: done (15 left) 2024-09-16T03:38:52.813 INFO:tasks.rados.rados.0.smithi157.stdout:1559: done (14 left) 2024-09-16T03:38:52.813 INFO:tasks.rados.rados.0.smithi157.stdout:1568: write oid 15 current snap is 0 2024-09-16T03:38:52.813 INFO:tasks.rados.rados.0.smithi157.stdout:1556: finishing write tid 2 to smithi15736696-19 2024-09-16T03:38:52.813 INFO:tasks.rados.rados.0.smithi157.stdout:1563: finishing write tid 1 to smithi15736696-24 2024-09-16T03:38:52.813 INFO:tasks.rados.rados.0.smithi157.stdout:1556: finishing write tid 3 to smithi15736696-19 2024-09-16T03:38:52.813 INFO:tasks.rados.rados.0.smithi157.stdout:1565: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:38:52.813 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:52.870 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:52.869+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:52.918 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:52.918 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:52.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:38:52.960 INFO:tasks.rados.rados.0.smithi157.stdout:1566: expect (ObjNum 317 snap 0 seq_num 317) 2024-09-16T03:38:52.971 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:52.970+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:53.036 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:53.036 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:53.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:38:53.060 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:53.072 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:53.069+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:53.154 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:53.154 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:53.163 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:38:53.206 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:53.205+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:53.264 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2670009 2024-09-16T03:38:53.273 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:53.271+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:53.274 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:53.274 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:53.278 INFO:tasks.rados.rados.0.smithi157.stdout:1556: finishing write tid 4 to smithi15736696-19 2024-09-16T03:38:53.278 INFO:tasks.rados.rados.0.smithi157.stdout:1556: finishing write tid 5 to smithi15736696-19 2024-09-16T03:38:53.278 INFO:tasks.rados.rados.0.smithi157.stdout:1556: oid 19 updating version 0 to 133 2024-09-16T03:38:53.278 INFO:tasks.rados.rados.0.smithi157.stdout:1556: oid 19 version 133 is already newer than 132 2024-09-16T03:38:53.278 INFO:tasks.rados.rados.0.smithi157.stdout:1556: oid 19 updating version 133 to 135 2024-09-16T03:38:53.278 INFO:tasks.rados.rados.0.smithi157.stdout:1556: oid 19 version 135 is already newer than 134 2024-09-16T03:38:53.278 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 135 (ObjNum 331 snap 0 seq_num 331) dirty exists 2024-09-16T03:38:53.279 INFO:tasks.rados.rados.0.smithi157.stdout:1556: left oid 19 (ObjNum 331 snap 0 seq_num 331) 2024-09-16T03:38:53.279 INFO:tasks.rados.rados.0.smithi157.stdout:1562: finishing write tid 1 to smithi15736696-41 2024-09-16T03:38:53.279 INFO:tasks.rados.rados.0.smithi157.stdout:1568: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2024-09-16T03:38:53.283 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:38:53.284 INFO:tasks.rados.rados.0.smithi157.stdout:1568: writing smithi15736696-15 from 413538 to 949883 tid 1 2024-09-16T03:38:53.288 INFO:tasks.rados.rados.0.smithi157.stdout:1568: writing smithi15736696-15 from 1395221 to 1979002 tid 2 2024-09-16T03:38:53.295 INFO:tasks.rados.rados.0.smithi157.stdout:1568: writing smithi15736696-15 from 2556103 to 3214720 tid 3 2024-09-16T03:38:53.295 INFO:tasks.rados.rados.0.smithi157.stdout:1556: done (14 left) 2024-09-16T03:38:53.295 INFO:tasks.rados.rados.0.smithi157.stdout:1565: done (13 left) 2024-09-16T03:38:53.296 INFO:tasks.rados.rados.0.smithi157.stdout:1566: done (12 left) 2024-09-16T03:38:53.296 INFO:tasks.rados.rados.0.smithi157.stdout:1569: write (excl) oid 50 current snap is 0 2024-09-16T03:38:53.296 INFO:tasks.rados.rados.0.smithi157.stdout:1562: finishing write tid 2 to smithi15736696-41 2024-09-16T03:38:53.296 INFO:tasks.rados.rados.0.smithi157.stdout:1562: finishing write tid 3 to smithi15736696-41 2024-09-16T03:38:53.296 INFO:tasks.rados.rados.0.smithi157.stdout:1569: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-09-16T03:38:53.301 INFO:tasks.rados.rados.0.smithi157.stdout:1569: writing smithi15736696-50 from 430345 to 1158292 tid 1 2024-09-16T03:38:53.307 INFO:tasks.rados.rados.0.smithi157.stdout:1569: writing smithi15736696-50 from 1570056 to 2297495 tid 2 2024-09-16T03:38:53.312 INFO:tasks.rados.rados.0.smithi157.stdout:1569: writing smithi15736696-50 from 2699879 to 3169768 tid 3 2024-09-16T03:38:53.312 INFO:tasks.rados.rados.0.smithi157.stdout:1570: write oid 26 current snap is 0 2024-09-16T03:38:53.312 INFO:tasks.rados.rados.0.smithi157.stdout:1570: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-09-16T03:38:53.316 INFO:tasks.rados.rados.0.smithi157.stdout:1570: writing smithi15736696-26 from 447152 to 850348 tid 1 2024-09-16T03:38:53.323 INFO:tasks.rados.rados.0.smithi157.stdout:1570: writing smithi15736696-26 from 1512185 to 2099635 tid 2 2024-09-16T03:38:53.332 INFO:tasks.rados.rados.0.smithi157.stdout:1570: writing smithi15736696-26 from 2727302 to 3408463 tid 3 2024-09-16T03:38:53.333 INFO:tasks.rados.rados.0.smithi157.stdout:1571: read oid 43 snap -1 2024-09-16T03:38:53.333 INFO:tasks.rados.rados.0.smithi157.stdout:1571: expect (ObjNum 317 snap 0 seq_num 317) 2024-09-16T03:38:53.333 INFO:tasks.rados.rados.0.smithi157.stdout:1572: write oid 27 current snap is 0 2024-09-16T03:38:53.333 INFO:tasks.rados.rados.0.smithi157.stdout:1572: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2024-09-16T03:38:53.335 INFO:tasks.rados.rados.0.smithi157.stdout:1572: writing smithi15736696-27 from 463959 to 942404 tid 1 2024-09-16T03:38:53.342 INFO:tasks.rados.rados.0.smithi157.stdout:1572: writing smithi15736696-27 from 1570667 to 2301775 tid 2 2024-09-16T03:38:53.349 INFO:tasks.rados.rados.0.smithi157.stdout:1572: writing smithi15736696-27 from 3038372 to 3530805 tid 3 2024-09-16T03:38:53.349 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:53.349 INFO:tasks.rados.rados.0.smithi157.stdout:1562: finishing write tid 4 to smithi15736696-41 2024-09-16T03:38:53.349 INFO:tasks.rados.rados.0.smithi157.stdout:1562: finishing write tid 5 to smithi15736696-41 2024-09-16T03:38:53.349 INFO:tasks.rados.rados.0.smithi157.stdout:1562: oid 41 updating version 0 to 48 2024-09-16T03:38:53.349 INFO:tasks.rados.rados.0.smithi157.stdout:1562: oid 41 updating version 48 to 49 2024-09-16T03:38:53.349 INFO:tasks.rados.rados.0.smithi157.stdout:1562: oid 41 version 49 is already newer than 46 2024-09-16T03:38:53.349 INFO:tasks.rados.rados.0.smithi157.stdout:1562: oid 41 version 49 is already newer than 47 2024-09-16T03:38:53.349 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 49 (ObjNum 332 snap 0 seq_num 332) dirty exists 2024-09-16T03:38:53.349 INFO:tasks.rados.rados.0.smithi157.stdout:1562: left oid 41 (ObjNum 332 snap 0 seq_num 332) 2024-09-16T03:38:53.349 INFO:tasks.rados.rados.0.smithi157.stdout:1539: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-16T03:38:53.360 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:53.359+0000 7fd101e6f640 -1 osd.2 118 handle_osd_map injecting map crc failure 2024-09-16T03:38:53.362 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:53.359+0000 7ff266366640 -1 osd.11 118 handle_osd_map injecting map crc failure 2024-09-16T03:38:53.368 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:53.366+0000 7f12e2a1a640 -1 osd.5 118 handle_osd_map injecting map crc failure 2024-09-16T03:38:53.369 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:53.366+0000 7fedb5739640 -1 osd.9 118 handle_osd_map injecting map crc failure 2024-09-16T03:38:53.374 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:53.371+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:53.385 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:53.425 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:53.423+0000 7efeef6d5640 -1 osd.13 118 handle_osd_map injecting map crc failure 2024-09-16T03:38:53.436 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:53.436 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:53.440 INFO:tasks.rados.rados.0.smithi157.stdout:1560: expect (ObjNum 270 snap 0 seq_num 270) 2024-09-16T03:38:53.441 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:53.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:38:53.474 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:53.472+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:53.490 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:53.555 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:53.555 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:53.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:38:53.572 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2499755 2024-09-16T03:38:53.576 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:53.574+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:53.577 INFO:tasks.rados.rados.0.smithi157.stdout:1557: got (22) Invalid argument 2024-09-16T03:38:53.577 INFO:tasks.rados.rados.0.smithi157.stdout:1567: finishing set_chunk to oid 34 2024-09-16T03:38:53.577 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 118 (ObjNum 303 snap 0 seq_num 303) dirty exists 2024-09-16T03:38:53.577 INFO:tasks.rados.rados.0.smithi157.stdout:1564: finishing set_chunk to oid 18 2024-09-16T03:38:53.577 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 63 (ObjNum 319 snap 0 seq_num 319) dirty exists 2024-09-16T03:38:53.577 INFO:tasks.rados.rados.0.smithi157.stdout:1563: finishing write tid 2 to smithi15736696-24 2024-09-16T03:38:53.577 INFO:tasks.rados.rados.0.smithi157.stdout:1539: done (15 left) 2024-09-16T03:38:53.577 INFO:tasks.rados.rados.0.smithi157.stdout:1557: done (14 left) 2024-09-16T03:38:53.577 INFO:tasks.rados.rados.0.smithi157.stdout:1560: done (13 left) 2024-09-16T03:38:53.577 INFO:tasks.rados.rados.0.smithi157.stdout:1562: done (12 left) 2024-09-16T03:38:53.577 INFO:tasks.rados.rados.0.smithi157.stdout:1564: done (11 left) 2024-09-16T03:38:53.578 INFO:tasks.rados.rados.0.smithi157.stdout:1567: done (10 left) 2024-09-16T03:38:53.578 INFO:tasks.rados.rados.0.smithi157.stdout:1573: set_chunk oid 34 target oid 2024-09-16T03:38:53.671 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:53.671 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:53.676 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:53.675+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:53.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:38:53.778 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:53.777+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:53.828 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:53.828 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:53.837 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:38:53.879 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:53.877+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:53.930 INFO:tasks.rados.rados.0.smithi157.stdout:1573: set_chunk oid 34 offset: 1839616 length: 1052672 target oid 9f02d4ebd8c8fcce54315cb319da77895061fd98a5845e00d58bc4abe5ce267d offset: 0 snap -1 2024-09-16T03:38:53.931 INFO:tasks.rados.rados.0.smithi157.stdout:1574: write oid 28 current snap is 0 2024-09-16T03:38:53.931 INFO:tasks.rados.rados.0.smithi157.stdout:1543: got (2) No such file or directory 2024-09-16T03:38:53.931 INFO:tasks.rados.rados.0.smithi157.stdout:1543: got expected ENOENT (src dne) 2024-09-16T03:38:53.931 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 55 (ObjNum 129 snap 0 seq_num 3489785888) clean dne 2024-09-16T03:38:53.931 INFO:tasks.rados.rados.0.smithi157.stdout:1563: finishing write tid 3 to smithi15736696-24 2024-09-16T03:38:53.931 INFO:tasks.rados.rados.0.smithi157.stdout:1563: finishing write tid 4 to smithi15736696-24 2024-09-16T03:38:53.931 INFO:tasks.rados.rados.0.smithi157.stdout:1563: finishing write tid 5 to smithi15736696-24 2024-09-16T03:38:53.931 INFO:tasks.rados.rados.0.smithi157.stdout:1563: oid 24 updating version 0 to 121 2024-09-16T03:38:53.931 INFO:tasks.rados.rados.0.smithi157.stdout:1563: oid 24 updating version 121 to 124 2024-09-16T03:38:53.931 INFO:tasks.rados.rados.0.smithi157.stdout:1563: oid 24 version 124 is already newer than 123 2024-09-16T03:38:53.931 INFO:tasks.rados.rados.0.smithi157.stdout:1563: oid 24 version 124 is already newer than 122 2024-09-16T03:38:53.931 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 124 (ObjNum 333 snap 0 seq_num 333) dirty exists 2024-09-16T03:38:53.931 INFO:tasks.rados.rados.0.smithi157.stdout:1563: left oid 24 (ObjNum 333 snap 0 seq_num 333) 2024-09-16T03:38:53.932 INFO:tasks.rados.rados.0.smithi157.stdout:1552: finishing write tid 1 to smithi15736696-33 2024-09-16T03:38:53.932 INFO:tasks.rados.rados.0.smithi157.stdout:1552: finishing write tid 2 to smithi15736696-33 2024-09-16T03:38:53.932 INFO:tasks.rados.rados.0.smithi157.stdout:1552: finishing write tid 3 to smithi15736696-33 2024-09-16T03:38:53.932 INFO:tasks.rados.rados.0.smithi157.stdout:1552: finishing write tid 4 to smithi15736696-33 2024-09-16T03:38:53.932 INFO:tasks.rados.rados.0.smithi157.stdout:1574: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-09-16T03:38:53.935 INFO:tasks.rados.rados.0.smithi157.stdout:1574: writing smithi15736696-28 from 480766 to 1034460 tid 1 2024-09-16T03:38:53.945 INFO:tasks.rados.rados.0.smithi157.stdout:1574: writing smithi15736696-28 from 1629149 to 2220268 tid 2 2024-09-16T03:38:53.945 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:53.946 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:53.946 INFO:tasks.rados.rados.0.smithi157.stdout:1574: writing smithi15736696-28 from 2782148 to 3202206 tid 3 2024-09-16T03:38:53.947 INFO:tasks.rados.rados.0.smithi157.stdout:1543: done (11 left) 2024-09-16T03:38:53.947 INFO:tasks.rados.rados.0.smithi157.stdout:1563: done (10 left) 2024-09-16T03:38:53.947 INFO:tasks.rados.rados.0.smithi157.stdout:1575: set_chunk oid 18 target oid 2024-09-16T03:38:53.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:38:53.999 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:53.997+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:54.064 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:54.065 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:54.073 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:38:54.080 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:54.079+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:54.182 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:54.181+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:54.223 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:54.224 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:54.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:38:54.283 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:54.281+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:54.288 INFO:tasks.rados.rados.0.smithi157.stdout:1575: set_chunk oid 18 offset: 216064 length: 2754048 target oid d9015cee2a6f10198bb99733ba558627ba3a39d37f6c963e9a92808417696f28 offset: 0 snap -1 2024-09-16T03:38:54.289 INFO:tasks.rados.rados.0.smithi157.stdout:1576: read oid 23 snap -1 2024-09-16T03:38:54.289 INFO:tasks.rados.rados.0.smithi157.stdout:1576: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:38:54.289 INFO:tasks.rados.rados.0.smithi157.stdout:1577: tier_flush oid 47 2024-09-16T03:38:54.289 INFO:tasks.rados.rados.0.smithi157.stdout:1577: tier_flush oid 47 snap -1 2024-09-16T03:38:54.289 INFO:tasks.rados.rados.0.smithi157.stdout:1552: finishing write tid 5 to smithi15736696-33 2024-09-16T03:38:54.289 INFO:tasks.rados.rados.0.smithi157.stdout:1552: oid 33 updating version 0 to 61 2024-09-16T03:38:54.289 INFO:tasks.rados.rados.0.smithi157.stdout:1552: oid 33 version 61 is already newer than 60 2024-09-16T03:38:54.289 INFO:tasks.rados.rados.0.smithi157.stdout:1552: oid 33 version 61 is already newer than 59 2024-09-16T03:38:54.289 INFO:tasks.rados.rados.0.smithi157.stdout:1552: oid 33 updating version 61 to 62 2024-09-16T03:38:54.289 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 62 (ObjNum 328 snap 0 seq_num 328) dirty exists 2024-09-16T03:38:54.289 INFO:tasks.rados.rados.0.smithi157.stdout:1552: left oid 33 (ObjNum 328 snap 0 seq_num 328) 2024-09-16T03:38:54.289 INFO:tasks.rados.rados.0.smithi157.stdout:1552: done (12 left) 2024-09-16T03:38:54.290 INFO:tasks.rados.rados.0.smithi157.stdout:1578: write oid 35 current snap is 0 2024-09-16T03:38:54.290 INFO:tasks.rados.rados.0.smithi157.stdout:1561: got (0) Success 2024-09-16T03:38:54.290 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 0 (ObjNum 180 snap 0 seq_num 180) dirty exists 2024-09-16T03:38:54.290 INFO:tasks.rados.rados.0.smithi157.stdout:1578: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2024-09-16T03:38:54.294 INFO:tasks.rados.rados.0.smithi157.stdout:1578: writing smithi15736696-35 from 497573 to 1126516 tid 1 2024-09-16T03:38:54.301 INFO:tasks.rados.rados.0.smithi157.stdout:1578: writing smithi15736696-35 from 1687631 to 2422408 tid 2 2024-09-16T03:38:54.309 INFO:tasks.rados.rados.0.smithi157.stdout:1578: writing smithi15736696-35 from 3093218 to 3697573 tid 3 2024-09-16T03:38:54.309 INFO:tasks.rados.rados.0.smithi157.stdout:1561: done (12 left) 2024-09-16T03:38:54.309 INFO:tasks.rados.rados.0.smithi157.stdout:1579: set_chunk oid 11 target oid 2024-09-16T03:38:54.309 INFO:tasks.rados.rados.0.smithi157.stdout:1579: set_chunk oid 11 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:54.309 INFO:tasks.rados.rados.0.smithi157.stdout:1580: read oid 41 snap -1 2024-09-16T03:38:54.309 INFO:tasks.rados.rados.0.smithi157.stdout:1580: expect (ObjNum 332 snap 0 seq_num 332) 2024-09-16T03:38:54.309 INFO:tasks.rados.rados.0.smithi157.stdout:1581: set_chunk oid 3 target oid 2024-09-16T03:38:54.310 INFO:tasks.rados.rados.0.smithi157.stdout:1581: set_chunk oid 3 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:38:54.310 INFO:tasks.rados.rados.0.smithi157.stdout:1582: set_chunk oid 29 target oid 2024-09-16T03:38:54.342 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:54.342 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:54.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:38:54.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:38:54.398 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:54.396+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:38:54.412 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:54.411+0000 7f9288166640 -1 osd.14 119 handle_osd_map injecting map crc failure 2024-09-16T03:38:54.466 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:54.466 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:54.475 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:38:54.485 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:54.483+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:54.507 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:38:54.585 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:38:54.583+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:38:54.592 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:54.592 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:54.601 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:38:54.637 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:54.637 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:54.655 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:54.655 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:54.673 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:54.673 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:54.686 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:54.685+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:54.691 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:54.691 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:54.707 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:54.707 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:54.725 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:54.725 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:54.742 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:54.743 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:54.746 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:54.746 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:54.746 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:54.746 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:54.746 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:54.755 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:38:54.760 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:54.760 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:54.776 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:54.777 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:54.787 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:54.784+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:54.793 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:54.794 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:54.809 INFO:tasks.rados.rados.0.smithi157.stdout:1582: set_chunk oid 29 offset: 674304 length: 973824 target oid 5468a7f28c81b54feb043029f03600247cb0fa6c0edaf011502c30f57166db38 offset: 0 snap -1 2024-09-16T03:38:54.809 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:54.809 INFO:tasks.rados.rados.0.smithi157.stdout:1568: finishing write tid 1 to smithi15736696-15 2024-09-16T03:38:54.809 INFO:tasks.rados.rados.0.smithi157.stdout:1571: expect (ObjNum 317 snap 0 seq_num 317) 2024-09-16T03:38:54.809 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:54.810 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:54.810 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:54.827 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:54.827 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:54.843 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:54.843 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:54.859 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:54.860 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:54.876 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:54.877 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:54.888 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:54.887+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:54.899 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:54.899 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:54.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:54.899 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:54.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.900 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:21 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'4, mlcod=0'0)", 2024-09-16T03:38:54.900 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:38:54.900 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 102.70981532, 2024-09-16T03:38:54.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10231100799999999, 2024-09-16T03:38:54.900 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.900 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.900 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.900 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:38:54.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:38:54.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187300+0000", 2024-09-16T03:38:54.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0360000000000001e-06 2024-09-16T03:38:54.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187905+0000", 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060508100000000004 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187920+0000", 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4314000000000001e-05 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187928+0000", 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7150000000000004e-06 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187984+0000", 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5872999999999999e-05 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188016+0000", 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2045000000000002e-05 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289567+0000", 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101550875 2024-09-16T03:38:54.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289609+0000", 2024-09-16T03:38:54.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2069e-05 2024-09-16T03:38:54.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.904 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.904 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.21:5 3.0 e45/44 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'39, mlcod=45'37)", 2024-09-16T03:38:54.904 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:38:54.904 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 93.982518286000001, 2024-09-16T03:38:54.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20119541799999999, 2024-09-16T03:38:54.905 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.905 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.905 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.905 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:38:54.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:38:54.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914597+0000", 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.494e-06 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914603+0000", 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7349999999999999e-06 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914614+0000", 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0995999999999999e-05 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914623+0000", 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3899999999999993e-06 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.114941+0000", 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20031812399999999 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115001+0000", 2024-09-16T03:38:54.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0161999999999998e-05 2024-09-16T03:38:54.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115750+0000", 2024-09-16T03:38:54.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074851699999999998 2024-09-16T03:38:54.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115791+0000", 2024-09-16T03:38:54.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1e-05 2024-09-16T03:38:54.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.909 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.909 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.909 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.5.21:36 3.0 e52/44 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 52'62, mlcod=51'60)", 2024-09-16T03:38:54.909 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:38:54.909 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.012487626999999, 2024-09-16T03:38:54.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20104840199999999, 2024-09-16T03:38:54.909 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.909 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.909 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.909 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:38:54.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:38:54.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884627+0000", 2024-09-16T03:38:54.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.322e-06 2024-09-16T03:38:54.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884634+0000", 2024-09-16T03:38:54.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:38:54.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884643+0000", 2024-09-16T03:38:54.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7680000000000003e-06 2024-09-16T03:38:54.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984740+0000", 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100096066 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084931+0000", 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100191408 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084952+0000", 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.128e-05 2024-09-16T03:38:54.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085645+0000", 2024-09-16T03:38:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00069309500000000002 2024-09-16T03:38:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085674+0000", 2024-09-16T03:38:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9025000000000001e-05 2024-09-16T03:38:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.914 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1537 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'35, mlcod=58'33)", 2024-09-16T03:38:54.914 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:38:54.914 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.176734332999999, 2024-09-16T03:38:54.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102969106, 2024-09-16T03:38:54.914 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.914 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.914 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.914 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:38:54.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720381+0000", 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.767e-06 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721134+0000", 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00075281999999999996 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721148+0000", 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4534e-05 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721160+0000", 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1595e-05 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721256+0000", 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6136000000000002e-05 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721274+0000", 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8006e-05 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823312+0000", 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102038064 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823348+0000", 2024-09-16T03:38:54.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6183999999999998e-05 2024-09-16T03:38:54.918 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.918 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.918 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.918 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1538 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'36, mlcod=58'33)", 2024-09-16T03:38:54.918 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:54.918 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.175579995999996, 2024-09-16T03:38:54.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102397932, 2024-09-16T03:38:54.918 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5199999999999996e-07 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722162+0000", 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062758599999999996 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722171+0000", 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7139999999999999e-06 2024-09-16T03:38:54.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722177+0000", 2024-09-16T03:38:54.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.866e-06 2024-09-16T03:38:54.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822283+0000", 2024-09-16T03:38:54.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10010508899999999 2024-09-16T03:38:54.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822298+0000", 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5497000000000001e-05 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823907+0000", 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001609166 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823931+0000", 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4261999999999999e-05 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.922 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1539 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'37, mlcod=58'33)", 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.174379935000005, 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103545683, 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:38:54.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:38:54.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722735+0000", 2024-09-16T03:38:54.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8400000000000002e-07 2024-09-16T03:38:54.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722740+0000", 2024-09-16T03:38:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3129999999999998e-06 2024-09-16T03:38:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722747+0000", 2024-09-16T03:38:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0600000000000002e-06 2024-09-16T03:38:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822841+0000", 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100094477 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823376+0000", 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00053497199999999999 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823399+0000", 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2680999999999998e-05 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.826259+0000", 2024-09-16T03:38:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028598500000000002 2024-09-16T03:38:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.826279+0000", 2024-09-16T03:38:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0346000000000001e-05 2024-09-16T03:38:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.927 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:38:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:38:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.747189030000001, 2024-09-16T03:38:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20169899299999999, 2024-09-16T03:38:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:38:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:38:54.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149927+0000", 2024-09-16T03:38:54.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9299999999999999e-06 2024-09-16T03:38:54.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151090+0000", 2024-09-16T03:38:54.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011624459999999999 2024-09-16T03:38:54.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151105+0000", 2024-09-16T03:38:54.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5238999999999999e-05 2024-09-16T03:38:54.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251193+0000", 2024-09-16T03:38:54.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100088025 2024-09-16T03:38:54.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251556+0000", 2024-09-16T03:38:54.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036245599999999999 2024-09-16T03:38:54.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251578+0000", 2024-09-16T03:38:54.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2651e-05 2024-09-16T03:38:54.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351582+0000", 2024-09-16T03:38:54.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100003811 2024-09-16T03:38:54.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351624+0000", 2024-09-16T03:38:54.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1434999999999999e-05 2024-09-16T03:38:54.932 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.932 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.932 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.932 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:90 3.0 e70/69 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head v 70'103, mlcod=0'0)", 2024-09-16T03:38:54.932 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:54.932 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 59.121350747000001, 2024-09-16T03:38:54.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102185333, 2024-09-16T03:38:54.933 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.933 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.933 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.933 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:54.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:54.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:38:54.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-16T03:38:54.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775769+0000", 2024-09-16T03:38:54.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7379999999999999e-06 2024-09-16T03:38:54.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775782+0000", 2024-09-16T03:38:54.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2989e-05 2024-09-16T03:38:54.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775791+0000", 2024-09-16T03:38:54.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5369999999999997e-06 2024-09-16T03:38:54.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876295+0000", 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100504817 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876316+0000", 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0409999999999999e-05 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877916+0000", 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016003790000000001 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877948+0000", 2024-09-16T03:38:54.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1949999999999997e-05 2024-09-16T03:38:54.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.937 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.937 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1742 3.0 e70/69 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head v 70'104, mlcod=0'0)", 2024-09-16T03:38:54.937 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:54.937 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 59.121314050999999, 2024-09-16T03:38:54.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20271830699999999, 2024-09-16T03:38:54.937 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.938 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.938 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.938 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:54.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:54.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:38:54.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:38:54.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777455+0000", 2024-09-16T03:38:54.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001655524 2024-09-16T03:38:54.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777468+0000", 2024-09-16T03:38:54.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2486999999999999e-05 2024-09-16T03:38:54.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777474+0000", 2024-09-16T03:38:54.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:38:54.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876577+0000", 2024-09-16T03:38:54.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099103360000000001 2024-09-16T03:38:54.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876594+0000", 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6449000000000001e-05 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978469+0000", 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101875521 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978518+0000", 2024-09-16T03:38:54.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8488000000000001e-05 2024-09-16T03:38:54.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.942 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.942 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:91 3.0 e70/69 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head v 70'105, mlcod=0'0)", 2024-09-16T03:38:54.942 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:38:54.942 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 59.035102182000003, 2024-09-16T03:38:54.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.116532025, 2024-09-16T03:38:54.942 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.942 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.942 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.942 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:38:54.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:38:54.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862015+0000", 2024-09-16T03:38:54.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3569999999999998e-06 2024-09-16T03:38:54.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862029+0000", 2024-09-16T03:38:54.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4082e-05 2024-09-16T03:38:54.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862055+0000", 2024-09-16T03:38:54.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6570000000000001e-05 2024-09-16T03:38:54.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862070+0000", 2024-09-16T03:38:54.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4473e-05 2024-09-16T03:38:54.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877631+0000", 2024-09-16T03:38:54.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.015560744 2024-09-16T03:38:54.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877674+0000", 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3615999999999997e-05 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978525+0000", 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100850696 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978543+0000", 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8487000000000001e-05 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.946 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.947 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:94 3.0 e70/69 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head v 70'106, mlcod=0'0)", 2024-09-16T03:38:54.947 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:38:54.947 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 59.034565962999999, 2024-09-16T03:38:54.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11627603, 2024-09-16T03:38:54.947 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.947 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.947 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.947 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:38:54.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:38:54.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862549+0000", 2024-09-16T03:38:54.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1650000000000001e-06 2024-09-16T03:38:54.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862556+0000", 2024-09-16T03:38:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8709999999999999e-06 2024-09-16T03:38:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862566+0000", 2024-09-16T03:38:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0271e-05 2024-09-16T03:38:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862573+0000", 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.861e-06 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978123+0000", 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11554993199999999 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978163+0000", 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0201999999999999e-05 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978796+0000", 2024-09-16T03:38:54.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063316700000000002 2024-09-16T03:38:54.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978824+0000", 2024-09-16T03:38:54.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7560999999999999e-05 2024-09-16T03:38:54.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.951 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.951 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1947 4.8 e74/68 4:146f84f0:::smithi15736696-6:head v 74'47, mlcod=73'45)", 2024-09-16T03:38:54.952 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:38:54.952 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.771949825, 2024-09-16T03:38:54.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103866244, 2024-09-16T03:38:54.952 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.952 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.952 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.952 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:38:54.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:38:54.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125165+0000", 2024-09-16T03:38:54.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.513e-06 2024-09-16T03:38:54.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126855+0000", 2024-09-16T03:38:54.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016901010000000001 2024-09-16T03:38:54.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126870+0000", 2024-09-16T03:38:54.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4487999999999999e-05 2024-09-16T03:38:54.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227009+0000", 2024-09-16T03:38:54.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100138888 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227117+0000", 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107984 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227136+0000", 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9247999999999999e-05 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229002+0000", 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001866143 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229030+0000", 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7878999999999999e-05 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1979 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'32, mlcod=0'0)", 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:38:54.956 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.059375277999997, 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20429768500000001, 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837740+0000", 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.779e-06 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838356+0000", 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061634500000000002 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838371+0000", 2024-09-16T03:38:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5007e-05 2024-09-16T03:38:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838380+0000", 2024-09-16T03:38:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1009999999999992e-06 2024-09-16T03:38:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038674+0000", 2024-09-16T03:38:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20029450500000001 2024-09-16T03:38:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038710+0000", 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6100999999999999e-05 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042001+0000", 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032909430000000002 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042036+0000", 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4904000000000003e-05 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1980 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'33, mlcod=0'0)", 2024-09-16T03:38:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:38:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.057498682000002, 2024-09-16T03:38:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20296795100000001, 2024-09-16T03:38:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:38:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:38:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839616+0000", 2024-09-16T03:38:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3099999999999996e-07 2024-09-16T03:38:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840706+0000", 2024-09-16T03:38:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010900980000000001 2024-09-16T03:38:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840720+0000", 2024-09-16T03:38:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.378e-05 2024-09-16T03:38:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840727+0000", 2024-09-16T03:38:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.525e-06 2024-09-16T03:38:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039940+0000", 2024-09-16T03:38:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19921313199999999 2024-09-16T03:38:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039967+0000", 2024-09-16T03:38:54.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6393999999999998e-05 2024-09-16T03:38:54.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042552+0000", 2024-09-16T03:38:54.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025850999999999999 2024-09-16T03:38:54.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042583+0000", 2024-09-16T03:38:54.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0991000000000002e-05 2024-09-16T03:38:54.968 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.968 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.968 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1981 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'34, mlcod=0'0)", 2024-09-16T03:38:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:38:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 53.055476327000001, 2024-09-16T03:38:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201270175, 2024-09-16T03:38:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:38:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:38:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841638+0000", 2024-09-16T03:38:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-16T03:38:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841644+0000", 2024-09-16T03:38:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8050000000000003e-06 2024-09-16T03:38:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841653+0000", 2024-09-16T03:38:54.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6580000000000008e-06 2024-09-16T03:38:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841660+0000", 2024-09-16T03:38:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9290000000000001e-06 2024-09-16T03:38:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041673+0000", 2024-09-16T03:38:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20001311499999999 2024-09-16T03:38:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041696+0000", 2024-09-16T03:38:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3365000000000001e-05 2024-09-16T03:38:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042878+0000", 2024-09-16T03:38:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011814499999999999 2024-09-16T03:38:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042907+0000", 2024-09-16T03:38:54.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9655999999999999e-05 2024-09-16T03:38:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2117 3.0 e76/69 3:3c8eb54b:::593314a3a723c40dec2b25351adb282089b02c0264cb0891ee3f17edde05dfe9:head v 76'138, mlcod=75'136)", 2024-09-16T03:38:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:38:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 49.245068639000003, 2024-09-16T03:38:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10374235, 2024-09-16T03:38:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.974 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:38:54.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:38:54.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:38:54.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8099999999999999e-07 2024-09-16T03:38:54.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654214+0000", 2024-09-16T03:38:54.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002169191 2024-09-16T03:38:54.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654230+0000", 2024-09-16T03:38:54.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6144e-05 2024-09-16T03:38:54.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654242+0000", 2024-09-16T03:38:54.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1748e-05 2024-09-16T03:38:54.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.754387+0000", 2024-09-16T03:38:54.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10014505899999999 2024-09-16T03:38:54.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.754409+0000", 2024-09-16T03:38:54.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2181999999999999e-05 2024-09-16T03:38:54.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.755736+0000", 2024-09-16T03:38:54.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013261690000000001 2024-09-16T03:38:54.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.755787+0000", 2024-09-16T03:38:54.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1675999999999998e-05 2024-09-16T03:38:54.978 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.978 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.979 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2278 4.18 e80/68 4:1f63731d:::smithi15736696-30:head v 80'37, mlcod=75'34)", 2024-09-16T03:38:54.979 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:38:54.979 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 44.371658439999997, 2024-09-16T03:38:54.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10326089300000001, 2024-09-16T03:38:54.979 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.979 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.979 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.979 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:38:54.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:38:54.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525456+0000", 2024-09-16T03:38:54.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.248e-06 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525936+0000", 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047982899999999998 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525945+0000", 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8489999999999995e-06 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525955+0000", 2024-09-16T03:38:54.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.043e-05 2024-09-16T03:38:54.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.526339+0000", 2024-09-16T03:38:54.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038385200000000001 2024-09-16T03:38:54.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.526358+0000", 2024-09-16T03:38:54.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8802000000000001e-05 2024-09-16T03:38:54.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.628679+0000", 2024-09-16T03:38:54.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102321304 2024-09-16T03:38:54.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.628716+0000", 2024-09-16T03:38:54.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6579000000000003e-05 2024-09-16T03:38:54.983 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.983 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.983 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.984 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2536 3.0 e86/69 3:10a17b1b:::13dc2fab1f287170fabf96291b912bf80e0b081f291f2c6d41d494566f2d1f8c:head v 86'176, mlcod=86'174)", 2024-09-16T03:38:54.984 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:38:54.984 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 38.057601949000002, 2024-09-16T03:38:54.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20664157499999999, 2024-09-16T03:38:54.984 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.984 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.984 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.984 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:38:54.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839515+0000", 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3330000000000001e-06 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843392+0000", 2024-09-16T03:38:54.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038769329999999999 2024-09-16T03:38:54.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843412+0000", 2024-09-16T03:38:54.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0608e-05 2024-09-16T03:38:54.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.943546+0000", 2024-09-16T03:38:54.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100133176 2024-09-16T03:38:54.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043824+0000", 2024-09-16T03:38:54.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100278785 2024-09-16T03:38:54.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043869+0000", 2024-09-16T03:38:54.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4845999999999997e-05 2024-09-16T03:38:54.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046096+0000", 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022262559999999998 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046153+0000", 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7637999999999999e-05 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2654 4.8 e90/68 4:148e627d:::smithi15736696-23:head v 90'58, mlcod=80'56)", 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:38:54.988 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 33.522214591000001, 2024-09-16T03:38:54.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.106604212, 2024-09-16T03:38:54.989 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.989 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.989 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:54.989 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:38:54.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:38:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374901+0000", 2024-09-16T03:38:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1749999999999999e-06 2024-09-16T03:38:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375810+0000", 2024-09-16T03:38:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00090911400000000004 2024-09-16T03:38:54.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375827+0000", 2024-09-16T03:38:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7010000000000001e-05 2024-09-16T03:38:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476010+0000", 2024-09-16T03:38:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100182879 2024-09-16T03:38:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476184+0000", 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017408899999999999 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476206+0000", 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2013000000000001e-05 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481458+0000", 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052519120000000001 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481503+0000", 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5019999999999999e-05 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "pg_update_log_missing(4.c epoch 99/68 rep_tid 69 entries 99'56 (0'0) error 4:34492f38:::smithi15736696-33:head by client.4628.0:3000 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 90'53 roll_forward_to 90'54)", 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 22.977256440000001, 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201155684, 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:54.993 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:54.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919858+0000", 2024-09-16T03:38:54.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:38:54.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:54.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919871+0000", 2024-09-16T03:38:54.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3433e-05 2024-09-16T03:38:54.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:54.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919929+0000", 2024-09-16T03:38:54.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7543000000000001e-05 2024-09-16T03:38:54.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:54.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919942+0000", 2024-09-16T03:38:54.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3122000000000001e-05 2024-09-16T03:38:54.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:54.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.120302+0000", 2024-09-16T03:38:54.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036052800000001 2024-09-16T03:38:54.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:54.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:54.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:54.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.121013+0000", 2024-09-16T03:38:54.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071046800000000002 2024-09-16T03:38:54.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.997 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:54.997 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:54.997 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:54.998 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:54.999 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:38:55.000 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:55.001 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:54.997+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:55.040 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:55.099 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:55.097+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:55.104 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:55.104 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:55.104 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:55.104 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:55.113 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:38:55.156 INFO:tasks.rados.rados.0.smithi157.stdout:1570: finishing write tid 1 to smithi15736696-26 2024-09-16T03:38:55.156 INFO:tasks.rados.rados.0.smithi157.stdout:1570: finishing write tid 2 to smithi15736696-26 2024-09-16T03:38:55.156 INFO:tasks.rados.rados.0.smithi157.stdout:1549: finishing write tid 1 to smithi15736696-6 2024-09-16T03:38:55.157 INFO:tasks.rados.rados.0.smithi157.stdout:1549: finishing write tid 2 to smithi15736696-6 2024-09-16T03:38:55.157 INFO:tasks.rados.rados.0.smithi157.stdout:1568: finishing write tid 2 to smithi15736696-15 2024-09-16T03:38:55.157 INFO:tasks.rados.rados.0.smithi157.stdout:1572: finishing write tid 1 to smithi15736696-27 2024-09-16T03:38:55.157 INFO:tasks.rados.rados.0.smithi157.stdout:1570: finishing write tid 3 to smithi15736696-26 2024-09-16T03:38:55.157 INFO:tasks.rados.rados.0.smithi157.stdout:1549: finishing write tid 3 to smithi15736696-6 2024-09-16T03:38:55.157 INFO:tasks.rados.rados.0.smithi157.stdout:1549: finishing write tid 4 to smithi15736696-6 2024-09-16T03:38:55.157 INFO:tasks.rados.rados.0.smithi157.stdout:1568: finishing write tid 3 to smithi15736696-15 2024-09-16T03:38:55.157 INFO:tasks.rados.rados.0.smithi157.stdout:1568: finishing write tid 4 to smithi15736696-15 2024-09-16T03:38:55.157 INFO:tasks.rados.rados.0.smithi157.stdout:1568: finishing write tid 5 to smithi15736696-15 2024-09-16T03:38:55.157 INFO:tasks.rados.rados.0.smithi157.stdout:1568: oid 15 updating version 0 to 18 2024-09-16T03:38:55.157 INFO:tasks.rados.rados.0.smithi157.stdout:1568: oid 15 version 18 is already newer than 15 2024-09-16T03:38:55.157 INFO:tasks.rados.rados.0.smithi157.stdout:1568: oid 15 version 18 is already newer than 16 2024-09-16T03:38:55.157 INFO:tasks.rados.rados.0.smithi157.stdout:1568: oid 15 version 18 is already newer than 17 2024-09-16T03:38:55.158 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 18 (ObjNum 334 snap 0 seq_num 334) dirty exists 2024-09-16T03:38:55.158 INFO:tasks.rados.rados.0.smithi157.stdout:1568: left oid 15 (ObjNum 334 snap 0 seq_num 334) 2024-09-16T03:38:55.158 INFO:tasks.rados.rados.0.smithi157.stdout:1572: finishing write tid 2 to smithi15736696-27 2024-09-16T03:38:55.158 INFO:tasks.rados.rados.0.smithi157.stdout:1568: done (15 left) 2024-09-16T03:38:55.158 INFO:tasks.rados.rados.0.smithi157.stdout:1571: done (14 left) 2024-09-16T03:38:55.158 INFO:tasks.rados.rados.0.smithi157.stdout:1583: read oid 15 snap -1 2024-09-16T03:38:55.158 INFO:tasks.rados.rados.0.smithi157.stdout:1583: expect (ObjNum 334 snap 0 seq_num 334) 2024-09-16T03:38:55.158 INFO:tasks.rados.rados.0.smithi157.stdout:1584: set_chunk oid 13 target oid 2024-09-16T03:38:55.200 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:55.199+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:55.215 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:55.215 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:55.216 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:55.224 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:38:55.308 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:55.306+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:55.360 INFO:tasks.rados.rados.0.smithi157.stdout:1584: set_chunk oid 13 offset: 1225216 length: 128000 target oid b4ee10e4ed588720ea4c4929d33b0c13809105cad76800eeca69352e39b5743a offset: 0 snap -1 2024-09-16T03:38:55.360 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:1572: finishing write tid 3 to smithi15736696-27 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:1570: finishing write tid 4 to smithi15736696-26 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:1570: finishing write tid 5 to smithi15736696-26 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:1570: oid 26 updating version 0 to 110 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:1570: oid 26 updating version 110 to 111 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:1570: oid 26 version 111 is already newer than 108 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:1570: oid 26 version 111 is already newer than 109 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 111 (ObjNum 336 snap 0 seq_num 336) dirty exists 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:1570: left oid 26 (ObjNum 336 snap 0 seq_num 336) 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:1572: finishing write tid 4 to smithi15736696-27 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:1572: finishing write tid 5 to smithi15736696-27 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:1572: oid 27 updating version 0 to 137 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:1572: oid 27 updating version 137 to 138 2024-09-16T03:38:55.361 INFO:tasks.rados.rados.0.smithi157.stdout:1572: oid 27 updating version 138 to 139 2024-09-16T03:38:55.362 INFO:tasks.rados.rados.0.smithi157.stdout:1572: oid 27 updating version 139 to 140 2024-09-16T03:38:55.362 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 140 (ObjNum 337 snap 0 seq_num 337) dirty exists 2024-09-16T03:38:55.362 INFO:tasks.rados.rados.0.smithi157.stdout:1572: left oid 27 (ObjNum 337 snap 0 seq_num 337) 2024-09-16T03:38:55.362 INFO:tasks.rados.rados.0.smithi157.stdout:1569: finishing write tid 1 to smithi15736696-50 2024-09-16T03:38:55.362 INFO:tasks.rados.rados.0.smithi157.stdout:1570: done (15 left) 2024-09-16T03:38:55.362 INFO:tasks.rados.rados.0.smithi157.stdout:1572: done (14 left) 2024-09-16T03:38:55.362 INFO:tasks.rados.rados.0.smithi157.stdout:1585: read oid 2 snap -1 2024-09-16T03:38:55.362 INFO:tasks.rados.rados.0.smithi157.stdout:1585: expect deleted 2024-09-16T03:38:55.362 INFO:tasks.rados.rados.0.smithi157.stdout:1586: read oid 17 snap -1 2024-09-16T03:38:55.362 INFO:tasks.rados.rados.0.smithi157.stdout:1586: expect (ObjNum 321 snap 0 seq_num 321) 2024-09-16T03:38:55.362 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:55.362 INFO:tasks.rados.rados.0.smithi157.stdout:1569: finishing write tid 2 to smithi15736696-50 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:1549: finishing write tid 5 to smithi15736696-6 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:1549: oid 6 updating version 0 to 65 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:1549: oid 6 version 65 is already newer than 64 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:1549: oid 6 updating version 65 to 67 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:1549: oid 6 version 67 is already newer than 66 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 67 (ObjNum 327 snap 0 seq_num 327) dirty exists 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:1549: left oid 6 (ObjNum 327 snap 0 seq_num 327) 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:1569: finishing write tid 3 to smithi15736696-50 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:1573: finishing set_chunk to oid 34 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 34 set_chunk returned error code -95 offset: 1839616 length: 1052672 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 0 (ObjNum 303 snap 0 seq_num 303) dirty exists 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:1569: finishing write tid 4 to smithi15736696-50 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:1569: finishing write tid 5 to smithi15736696-50 2024-09-16T03:38:55.363 INFO:tasks.rados.rados.0.smithi157.stdout:1569: oid 50 updating version 0 to 133 2024-09-16T03:38:55.364 INFO:tasks.rados.rados.0.smithi157.stdout:1569: oid 50 version 133 is already newer than 132 2024-09-16T03:38:55.364 INFO:tasks.rados.rados.0.smithi157.stdout:1569: oid 50 updating version 133 to 134 2024-09-16T03:38:55.364 INFO:tasks.rados.rados.0.smithi157.stdout:1569: oid 50 updating version 134 to 135 2024-09-16T03:38:55.364 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 135 (ObjNum 335 snap 0 seq_num 335) dirty exists 2024-09-16T03:38:55.364 INFO:tasks.rados.rados.0.smithi157.stdout:1569: left oid 50 (ObjNum 335 snap 0 seq_num 335) 2024-09-16T03:38:55.364 INFO:tasks.rados.rados.0.smithi157.stdout:1574: finishing write tid 1 to smithi15736696-28 2024-09-16T03:38:55.364 INFO:tasks.rados.rados.0.smithi157.stdout:1574: finishing write tid 2 to smithi15736696-28 2024-09-16T03:38:55.364 INFO:tasks.rados.rados.0.smithi157.stdout:1574: finishing write tid 3 to smithi15736696-28 2024-09-16T03:38:55.364 INFO:tasks.rados.rados.0.smithi157.stdout:1574: finishing write tid 4 to smithi15736696-28 2024-09-16T03:38:55.364 INFO:tasks.rados.rados.0.smithi157.stdout:1574: finishing write tid 5 to smithi15736696-28 2024-09-16T03:38:55.364 INFO:tasks.rados.rados.0.smithi157.stdout:1574: oid 28 updating version 0 to 122 2024-09-16T03:38:55.364 INFO:tasks.rados.rados.0.smithi157.stdout:1574: oid 28 updating version 122 to 124 2024-09-16T03:38:55.364 INFO:tasks.rados.rados.0.smithi157.stdout:1574: oid 28 version 124 is already newer than 121 2024-09-16T03:38:55.365 INFO:tasks.rados.rados.0.smithi157.stdout:1574: oid 28 version 124 is already newer than 123 2024-09-16T03:38:55.365 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 124 (ObjNum 338 snap 0 seq_num 338) dirty exists 2024-09-16T03:38:55.365 INFO:tasks.rados.rados.0.smithi157.stdout:1574: left oid 28 (ObjNum 338 snap 0 seq_num 338) 2024-09-16T03:38:55.365 INFO:tasks.rados.rados.0.smithi157.stdout:1575: finishing set_chunk to oid 18 2024-09-16T03:38:55.365 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 18 set_chunk returned error code -95 offset: 216064 length: 2754048 2024-09-16T03:38:55.365 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 0 (ObjNum 319 snap 0 seq_num 319) dirty exists 2024-09-16T03:38:55.365 INFO:tasks.rados.rados.0.smithi157.stdout:1579: finishing set_chunk to oid 11 2024-09-16T03:38:55.365 INFO:tasks.rados.rados.0.smithi157.stdout:1579: got expected ENOENT (src dne) 2024-09-16T03:38:55.365 INFO:tasks.rados.rados.0.smithi157.stdout:1576: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:38:55.365 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:55.366 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:38:55.368 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:55.368 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:55.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:25 4.6 4:6ece7830:::smithi15736696-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:38:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 103.24597511499999, 2024-09-16T03:38:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40559452699999998, 2024-09-16T03:38:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.369 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 25 2024-09-16T03:38:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:38:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:38:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119830+0000", 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9200000000000001e-07 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120013+0000", 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018367200000000001 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120021+0000", 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9179999999999994e-06 2024-09-16T03:38:55.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120027+0000", 2024-09-16T03:38:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6710000000000004e-06 2024-09-16T03:38:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220194+0000", 2024-09-16T03:38:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10016689500000001 2024-09-16T03:38:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220399+0000", 2024-09-16T03:38:55.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020502899999999999 2024-09-16T03:38:55.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:55.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220613+0000", 2024-09-16T03:38:55.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021383900000000001 2024-09-16T03:38:55.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224640+0000", 2024-09-16T03:38:55.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040268099999999996 2024-09-16T03:38:55.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423348+0000", 2024-09-16T03:38:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19870853399999999 2024-09-16T03:38:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525343+0000", 2024-09-16T03:38:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101995081 2024-09-16T03:38:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525381+0000", 2024-09-16T03:38:55.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7842000000000001e-05 2024-09-16T03:38:55.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525424+0000", 2024-09-16T03:38:55.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2644e-05 2024-09-16T03:38:55.375 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.375 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.375 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.375 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:26 4.6 4:6ece7830:::smithi15736696-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:55.375 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:38:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 103.245762888, 2024-09-16T03:38:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550111300000002, 2024-09-16T03:38:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 26 2024-09-16T03:38:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.376 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:38:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:38:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120042+0000", 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7999999999999995e-07 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120210+0000", 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016779499999999999 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120219+0000", 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6859999999999995e-06 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120223+0000", 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0330000000000002e-06 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221494+0000", 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101271812 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221572+0000", 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7733999999999998e-05 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221697+0000", 2024-09-16T03:38:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012474900000000001 2024-09-16T03:38:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224670+0000", 2024-09-16T03:38:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029729589999999998 2024-09-16T03:38:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423398+0000", 2024-09-16T03:38:55.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19872784900000001 2024-09-16T03:38:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525465+0000", 2024-09-16T03:38:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102067557 2024-09-16T03:38:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525494+0000", 2024-09-16T03:38:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9113999999999999e-05 2024-09-16T03:38:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525542+0000", 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8145000000000001e-05 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:27 4.6 4:6ece7830:::smithi15736696-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 103.245575303, 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40541507399999999, 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.382 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27 2024-09-16T03:38:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:38:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:38:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120230+0000", 2024-09-16T03:38:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5799999999999999e-07 2024-09-16T03:38:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120233+0000", 2024-09-16T03:38:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.055e-06 2024-09-16T03:38:55.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120237+0000", 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7099999999999998e-06 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120244+0000", 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4810000000000003e-06 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222660+0000", 2024-09-16T03:38:55.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102416089 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222705+0000", 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5368999999999999e-05 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222782+0000", 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6623000000000001e-05 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224696+0000", 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019143929999999999 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423402+0000", 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198705305 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525582+0000", 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102180178 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525615+0000", 2024-09-16T03:38:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3303000000000003e-05 2024-09-16T03:38:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525644+0000", 2024-09-16T03:38:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8909999999999999e-05 2024-09-16T03:38:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.388 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:28 4.6 4:6ece7830:::smithi15736696-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 103.245554555, 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550787799999999, 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 28 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:38:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7999999999999996e-07 2024-09-16T03:38:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120252+0000", 2024-09-16T03:38:55.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.249e-06 2024-09-16T03:38:55.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120257+0000", 2024-09-16T03:38:55.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1969999999999998e-06 2024-09-16T03:38:55.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120261+0000", 2024-09-16T03:38:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1130000000000003e-06 2024-09-16T03:38:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222976+0000", 2024-09-16T03:38:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102715397 2024-09-16T03:38:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223049+0000", 2024-09-16T03:38:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2846e-05 2024-09-16T03:38:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525666+0000", 2024-09-16T03:38:55.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30261668800000002 2024-09-16T03:38:55.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525704+0000", 2024-09-16T03:38:55.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8025999999999998e-05 2024-09-16T03:38:55.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525758+0000", 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3881999999999997e-05 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:49 4.6 4:6df3bbbe:::smithi15736696-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 103.245539491, 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40556222800000002, 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.394 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:55.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8700000000000002e-07 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120454+0000", 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018909299999999999 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120465+0000", 2024-09-16T03:38:55.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.083e-05 2024-09-16T03:38:55.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120468+0000", 2024-09-16T03:38:55.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3869999999999999e-06 2024-09-16T03:38:55.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223082+0000", 2024-09-16T03:38:55.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102613379 2024-09-16T03:38:55.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223189+0000", 2024-09-16T03:38:55.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107728 2024-09-16T03:38:55.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:55.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223368+0000", 2024-09-16T03:38:55.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179011 2024-09-16T03:38:55.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.226087+0000", 2024-09-16T03:38:55.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027183390000000002 2024-09-16T03:38:55.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423408+0000", 2024-09-16T03:38:55.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19732106499999999 2024-09-16T03:38:55.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525779+0000", 2024-09-16T03:38:55.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10237133 2024-09-16T03:38:55.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525805+0000", 2024-09-16T03:38:55.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5967000000000001e-05 2024-09-16T03:38:55.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525827+0000", 2024-09-16T03:38:55.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1812000000000001e-05 2024-09-16T03:38:55.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.400 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.400 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:50 4.6 4:6df3bbbe:::smithi15736696-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 103.245331182, 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50526462999999999, 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 50 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120474+0000", 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4600000000000005e-07 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120901+0000", 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00042690000000000002 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120907+0000", 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7710000000000001e-06 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120911+0000", 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0500000000000002e-06 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224717+0000", 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103806175 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224763+0000", 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5629e-05 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224838+0000", 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4548999999999994e-05 2024-09-16T03:38:55.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423390+0000", 2024-09-16T03:38:55.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19855242400000001 2024-09-16T03:38:55.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423411+0000", 2024-09-16T03:38:55.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0888000000000001e-05 2024-09-16T03:38:55.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625680+0000", 2024-09-16T03:38:55.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20226923099999999 2024-09-16T03:38:55.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625709+0000", 2024-09-16T03:38:55.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9232000000000001e-05 2024-09-16T03:38:55.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625738+0000", 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8235e-05 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:51 4.6 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 103.2448866, 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50490003699999997, 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.407 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.408 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.408 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.408 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.408 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.408 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 51 2024-09-16T03:38:55.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.408 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:55.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8699999999999995e-07 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120920+0000", 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0820000000000001e-06 2024-09-16T03:38:55.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120924+0000", 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120927+0000", 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5809999999999999e-06 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225783+0000", 2024-09-16T03:38:55.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10485588699999999 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225832+0000", 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9750000000000003e-05 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225910+0000", 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7084999999999998e-05 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423414+0000", 2024-09-16T03:38:55.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19750438200000001 2024-09-16T03:38:55.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423456+0000", 2024-09-16T03:38:55.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2386999999999999e-05 2024-09-16T03:38:55.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625766+0000", 2024-09-16T03:38:55.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20231011300000001 2024-09-16T03:38:55.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625790+0000", 2024-09-16T03:38:55.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3672999999999998e-05 2024-09-16T03:38:55.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625818+0000", 2024-09-16T03:38:55.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7576000000000002e-05 2024-09-16T03:38:55.413 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.413 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.413 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:52 4.6 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 103.244874404, 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50502012900000004, 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 52 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.414 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:55.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:55.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:38:55.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:38:55.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120932+0000", 2024-09-16T03:38:55.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.181e-06 2024-09-16T03:38:55.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120935+0000", 2024-09-16T03:38:55.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7690000000000001e-06 2024-09-16T03:38:55.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120938+0000", 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6639999999999998e-06 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423188+0000", 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30224989200000002 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423318+0000", 2024-09-16T03:38:55.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013045999999999999 2024-09-16T03:38:55.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625844+0000", 2024-09-16T03:38:55.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20252547600000001 2024-09-16T03:38:55.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625895+0000", 2024-09-16T03:38:55.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1242000000000002e-05 2024-09-16T03:38:55.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625950+0000", 2024-09-16T03:38:55.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5164999999999998e-05 2024-09-16T03:38:55.419 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.419 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.419 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.419 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:279 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:55.419 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:38:55.419 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 100.16892261700001, 2024-09-16T03:38:55.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50446570800000001, 2024-09-16T03:38:55.419 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.420 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.420 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.420 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.420 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.420 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.420 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 279 2024-09-16T03:38:55.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.420 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:38:55.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196883+0000", 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.48e-06 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196889+0000", 2024-09-16T03:38:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6239999999999999e-06 2024-09-16T03:38:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196902+0000", 2024-09-16T03:38:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2804e-05 2024-09-16T03:38:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196909+0000", 2024-09-16T03:38:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9379999999999992e-06 2024-09-16T03:38:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196963+0000", 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3437999999999999e-05 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op must wait for map", 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196969+0000", 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1709999999999999e-06 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302538+0000", 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10556889899999999 2024-09-16T03:38:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302659+0000", 2024-09-16T03:38:55.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121441 2024-09-16T03:38:55.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:38:55.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500040+0000", 2024-09-16T03:38:55.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19738082900000001 2024-09-16T03:38:55.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500589+0000", 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00054827000000000001 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701118+0000", 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20052917100000001 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701205+0000", 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7120999999999995e-05 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701289+0000", 2024-09-16T03:38:55.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4457999999999997e-05 2024-09-16T03:38:55.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701347+0000", 2024-09-16T03:38:55.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8063999999999998e-05 2024-09-16T03:38:55.426 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.426 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.426 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.427 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:306 4.6 4:6ece7830:::smithi15736696-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:55.427 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:38:55.427 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 98.852920302000001, 2024-09-16T03:38:55.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40927214699999998, 2024-09-16T03:38:55.427 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.427 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.427 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.427 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.427 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.427 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.427 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 306 2024-09-16T03:38:55.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.427 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:38:55.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:38:55.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512885+0000", 2024-09-16T03:38:55.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.277e-06 2024-09-16T03:38:55.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512890+0000", 2024-09-16T03:38:55.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6689999999999997e-06 2024-09-16T03:38:55.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512900+0000", 2024-09-16T03:38:55.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0393e-05 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512906+0000", 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4569999999999996e-06 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512951+0000", 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4891000000000001e-05 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.513000+0000", 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9187999999999997e-05 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.720758+0000", 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20775775699999999 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.721960+0000", 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001202107 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922010+0000", 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20004993900000001 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922055+0000", 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4981000000000002e-05 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922103+0000", 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8097999999999999e-05 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922156+0000", 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3390000000000002e-05 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:419 4.6 4:64ce87bc:::smithi15736696-20:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 94.452120687999994, 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40437787200000003, 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 419 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:38:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913687+0000", 2024-09-16T03:38:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3299999999999999e-06 2024-09-16T03:38:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913700+0000", 2024-09-16T03:38:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2830000000000001e-05 2024-09-16T03:38:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913733+0000", 2024-09-16T03:38:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2815999999999999e-05 2024-09-16T03:38:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913750+0000", 2024-09-16T03:38:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7073000000000001e-05 2024-09-16T03:38:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913920+0000", 2024-09-16T03:38:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000170015 2024-09-16T03:38:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913981+0000", 2024-09-16T03:38:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1631999999999995e-05 2024-09-16T03:38:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216580+0000", 2024-09-16T03:38:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30259888200000001 2024-09-16T03:38:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217205+0000", 2024-09-16T03:38:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062461099999999996 2024-09-16T03:38:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317915+0000", 2024-09-16T03:38:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10071053000000001 2024-09-16T03:38:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317996+0000", 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0440999999999998e-05 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318032+0000", 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5944e-05 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318061+0000", 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9768e-05 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:479 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 93.644264542000002, 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.504011706, 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.440 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 479 2024-09-16T03:38:55.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.441 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:38:55.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:38:55.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721542+0000", 2024-09-16T03:38:55.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3199999999999998e-06 2024-09-16T03:38:55.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721553+0000", 2024-09-16T03:38:55.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1413000000000001e-05 2024-09-16T03:38:55.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721570+0000", 2024-09-16T03:38:55.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.685e-05 2024-09-16T03:38:55.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721596+0000", 2024-09-16T03:38:55.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5647999999999999e-05 2024-09-16T03:38:55.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.921950+0000", 2024-09-16T03:38:55.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20035402899999999 2024-09-16T03:38:55.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.922009+0000", 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9046999999999999e-05 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.023850+0000", 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10184130199999999 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.024503+0000", 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065257399999999995 2024-09-16T03:38:55.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.125105+0000", 2024-09-16T03:38:55.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100602046 2024-09-16T03:38:55.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225461+0000", 2024-09-16T03:38:55.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100355835 2024-09-16T03:38:55.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225512+0000", 2024-09-16T03:38:55.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0862999999999997e-05 2024-09-16T03:38:55.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225551+0000", 2024-09-16T03:38:55.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9779e-05 2024-09-16T03:38:55.446 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.446 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.446 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.446 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:996 4.8 4:148e627d:::smithi15736696-23:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:38:55.446 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:38:55.447 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.318461635000006, 2024-09-16T03:38:55.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.73588397999999999, 2024-09-16T03:38:55.447 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.447 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.447 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:55.447 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.447 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.447 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.447 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 996 2024-09-16T03:38:55.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.447 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:38:55.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:38:55.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047344+0000", 2024-09-16T03:38:55.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.846e-06 2024-09-16T03:38:55.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047351+0000", 2024-09-16T03:38:55.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.973e-06 2024-09-16T03:38:55.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047365+0000", 2024-09-16T03:38:55.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3556e-05 2024-09-16T03:38:55.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047382+0000", 2024-09-16T03:38:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6994e-05 2024-09-16T03:38:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047453+0000", 2024-09-16T03:38:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1458000000000006e-05 2024-09-16T03:38:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047465+0000", 2024-09-16T03:38:55.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1119e-05 2024-09-16T03:38:55.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449411+0000", 2024-09-16T03:38:55.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40194680399999999 2024-09-16T03:38:55.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449535+0000", 2024-09-16T03:38:55.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124043 2024-09-16T03:38:55.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783227+0000", 2024-09-16T03:38:55.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.333691187 2024-09-16T03:38:55.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.452 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.452 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1048 4.16 4:6df3bbbe:::smithi15736696-13:head [write 1622009~726683] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:55.452 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:55.452 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.790017517999999, 2024-09-16T03:38:55.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40682847599999999, 2024-09-16T03:38:55.452 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.452 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.453 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.453 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.453 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.453 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.453 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1048 2024-09-16T03:38:55.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.453 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:55.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:55.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:38:55.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-16T03:38:55.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576941+0000", 2024-09-16T03:38:55.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011532230000000001 2024-09-16T03:38:55.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576949+0000", 2024-09-16T03:38:55.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1089999999999998e-06 2024-09-16T03:38:55.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576956+0000", 2024-09-16T03:38:55.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000001e-06 2024-09-16T03:38:55.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878710+0000", 2024-09-16T03:38:55.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30175434299999998 2024-09-16T03:38:55.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878765+0000", 2024-09-16T03:38:55.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4988000000000003e-05 2024-09-16T03:38:55.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:55.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878852+0000", 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7201999999999996e-05 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881535+0000", 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026828310000000001 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883968+0000", 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024330659999999998 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982541+0000", 2024-09-16T03:38:55.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098572876000000004 2024-09-16T03:38:55.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982580+0000", 2024-09-16T03:38:55.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8714000000000002e-05 2024-09-16T03:38:55.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982615+0000", 2024-09-16T03:38:55.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5438000000000002e-05 2024-09-16T03:38:55.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.459 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.459 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1049 4.16 4:6df3bbbe:::smithi15736696-13:head [write 3121996~104915] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:55.459 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:38:55.459 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.788839369000002, 2024-09-16T03:38:55.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40579890600000001, 2024-09-16T03:38:55.459 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.459 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.460 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.460 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.460 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.460 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.460 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1049 2024-09-16T03:38:55.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.460 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:38:55.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:38:55.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576966+0000", 2024-09-16T03:38:55.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8199999999999999e-07 2024-09-16T03:38:55.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577069+0000", 2024-09-16T03:38:55.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103583 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577076+0000", 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8190000000000003e-06 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577082+0000", 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5130000000000004e-06 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880130+0000", 2024-09-16T03:38:55.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.303048127 2024-09-16T03:38:55.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880183+0000", 2024-09-16T03:38:55.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3127999999999997e-05 2024-09-16T03:38:55.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:55.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880282+0000", 2024-09-16T03:38:55.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8881000000000004e-05 2024-09-16T03:38:55.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883101+0000", 2024-09-16T03:38:55.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028194180000000002 2024-09-16T03:38:55.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.979716+0000", 2024-09-16T03:38:55.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.096614899000000004 2024-09-16T03:38:55.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982707+0000", 2024-09-16T03:38:55.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029914249999999998 2024-09-16T03:38:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982739+0000", 2024-09-16T03:38:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1250000000000001e-05 2024-09-16T03:38:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982764+0000", 2024-09-16T03:38:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5080999999999999e-05 2024-09-16T03:38:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.465 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1050 4.16 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 3226911] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:38:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.788714773999999, 2024-09-16T03:38:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40784258099999998, 2024-09-16T03:38:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1050 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577090+0000", 2024-09-16T03:38:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000003e-07 2024-09-16T03:38:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577094+0000", 2024-09-16T03:38:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4340000000000001e-06 2024-09-16T03:38:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577099+0000", 2024-09-16T03:38:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8599999999999998e-06 2024-09-16T03:38:55.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577104+0000", 2024-09-16T03:38:55.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0010000000000001e-06 2024-09-16T03:38:55.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881242+0000", 2024-09-16T03:38:55.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30413779000000002 2024-09-16T03:38:55.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881290+0000", 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7298e-05 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881367+0000", 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7750000000000006e-05 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883117+0000", 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001750096 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.980218+0000", 2024-09-16T03:38:55.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097100724999999999 2024-09-16T03:38:55.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984859+0000", 2024-09-16T03:38:55.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046412950000000001 2024-09-16T03:38:55.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984900+0000", 2024-09-16T03:38:55.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0300999999999999e-05 2024-09-16T03:38:55.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984932+0000", 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2332999999999997e-05 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1051 4.16 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.788692494000003, 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50807376999999998, 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.472 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:38:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1051 2024-09-16T03:38:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577115+0000", 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3980000000000002e-06 2024-09-16T03:38:55.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577120+0000", 2024-09-16T03:38:55.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5589999999999996e-06 2024-09-16T03:38:55.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577125+0000", 2024-09-16T03:38:55.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9910000000000002e-06 2024-09-16T03:38:55.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882780+0000", 2024-09-16T03:38:55.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30565438499999997 2024-09-16T03:38:55.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:55.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882825+0000", 2024-09-16T03:38:55.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5739999999999999e-05 2024-09-16T03:38:55.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085039+0000", 2024-09-16T03:38:55.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20221376499999999 2024-09-16T03:38:55.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085096+0000", 2024-09-16T03:38:55.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6666000000000003e-05 2024-09-16T03:38:55.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085186+0000", 2024-09-16T03:38:55.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9716000000000006e-05 2024-09-16T03:38:55.477 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.477 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.477 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.478 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1359 4.6 4:64ce87bc:::smithi15736696-20:head [write 1768928~405354] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T03:38:55.478 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:38:55.478 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 75.199669356000001, 2024-09-16T03:38:55.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40239613499999999, 2024-09-16T03:38:55.478 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.478 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.478 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.478 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.478 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.478 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.478 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1359 2024-09-16T03:38:55.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:38:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:38:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166136+0000", 2024-09-16T03:38:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:38:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166923+0000", 2024-09-16T03:38:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00078683500000000005 2024-09-16T03:38:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166933+0000", 2024-09-16T03:38:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0445000000000001e-05 2024-09-16T03:38:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166938+0000", 2024-09-16T03:38:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0119999999999996e-06 2024-09-16T03:38:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264901+0000", 2024-09-16T03:38:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097962831 2024-09-16T03:38:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264963+0000", 2024-09-16T03:38:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1812999999999999e-05 2024-09-16T03:38:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:38:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.265085+0000", 2024-09-16T03:38:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121914 2024-09-16T03:38:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.269430+0000", 2024-09-16T03:38:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0043455000000000004 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.372259+0000", 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102828645 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568289+0000", 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19603034999999999 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568475+0000", 2024-09-16T03:38:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018613599999999999 2024-09-16T03:38:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568531+0000", 2024-09-16T03:38:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5729000000000001e-05 2024-09-16T03:38:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1676 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v29,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:38:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:38:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 65.235840492999998, 2024-09-16T03:38:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50424139300000004, 2024-09-16T03:38:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1676 2024-09-16T03:38:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:38:55.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:38:55.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129968+0000", 2024-09-16T03:38:55.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8430000000000003e-06 2024-09-16T03:38:55.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129982+0000", 2024-09-16T03:38:55.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4514e-05 2024-09-16T03:38:55.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130013+0000", 2024-09-16T03:38:55.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1081999999999998e-05 2024-09-16T03:38:55.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130029+0000", 2024-09-16T03:38:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5849999999999999e-05 2024-09-16T03:38:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130169+0000", 2024-09-16T03:38:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000139973 2024-09-16T03:38:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130240+0000", 2024-09-16T03:38:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0638999999999999e-05 2024-09-16T03:38:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.532892+0000", 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402652654 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533546+0000", 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065390700000000001 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533943+0000", 2024-09-16T03:38:55.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039703999999999998 2024-09-16T03:38:55.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634089+0000", 2024-09-16T03:38:55.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100146098 2024-09-16T03:38:55.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634153+0000", 2024-09-16T03:38:55.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3581e-05 2024-09-16T03:38:55.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634205+0000", 2024-09-16T03:38:55.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2212000000000001e-05 2024-09-16T03:38:55.491 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.491 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.491 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.491 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:77 3.2 e73/68 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head v 73'117, mlcod=66'115)", 2024-09-16T03:38:55.491 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:38:55.491 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.186802215, 2024-09-16T03:38:55.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.401393467, 2024-09-16T03:38:55.491 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.491 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.492 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.492 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:38:55.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:38:55.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179005+0000", 2024-09-16T03:38:55.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.553e-06 2024-09-16T03:38:55.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179029+0000", 2024-09-16T03:38:55.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4039000000000001e-05 2024-09-16T03:38:55.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179046+0000", 2024-09-16T03:38:55.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7139999999999999e-05 2024-09-16T03:38:55.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.279144+0000", 2024-09-16T03:38:55.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100098245 2024-09-16T03:38:55.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379323+0000", 2024-09-16T03:38:55.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100178621 2024-09-16T03:38:55.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379348+0000", 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5315000000000001e-05 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580315+0000", 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20096734599999999 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580395+0000", 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0208000000000002e-05 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.495 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.496 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.496 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.496 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.496 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:55.496 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 9] out_osds: [1, 5, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:38:55.496 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T03:38:55.498 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:55.498 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:55.495+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:55.499 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:38:55.598 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:55.596+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:55.624 INFO:tasks.rados.rados.0.smithi157.stdout:1578: finishing write tid 1 to smithi15736696-35 2024-09-16T03:38:55.624 INFO:tasks.rados.rados.0.smithi157.stdout:1581: finishing set_chunk to oid 3 2024-09-16T03:38:55.624 INFO:tasks.rados.rados.0.smithi157.stdout:1581: got expected ENOENT (src dne) 2024-09-16T03:38:55.624 INFO:tasks.rados.rados.0.smithi157.stdout:1580: expect (ObjNum 332 snap 0 seq_num 332) 2024-09-16T03:38:55.624 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:55.643 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:55.643 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:55.643 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:55.643 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:55.654 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:38:55.699 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:55.697+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:55.721 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2974658 2024-09-16T03:38:55.735 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 464680 2024-09-16T03:38:55.758 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:55.758 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:55.758 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:55.767 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:38:55.790 INFO:tasks.rados.rados.0.smithi157.stdout:1578: finishing write tid 2 to smithi15736696-35 2024-09-16T03:38:55.791 INFO:tasks.rados.rados.0.smithi157.stdout:1578: finishing write tid 3 to smithi15736696-35 2024-09-16T03:38:55.791 INFO:tasks.rados.rados.0.smithi157.stdout:1578: finishing write tid 4 to smithi15736696-35 2024-09-16T03:38:55.791 INFO:tasks.rados.rados.0.smithi157.stdout:1578: finishing write tid 5 to smithi15736696-35 2024-09-16T03:38:55.791 INFO:tasks.rados.rados.0.smithi157.stdout:1578: oid 35 updating version 0 to 122 2024-09-16T03:38:55.791 INFO:tasks.rados.rados.0.smithi157.stdout:1578: oid 35 version 122 is already newer than 120 2024-09-16T03:38:55.791 INFO:tasks.rados.rados.0.smithi157.stdout:1578: oid 35 version 122 is already newer than 121 2024-09-16T03:38:55.791 INFO:tasks.rados.rados.0.smithi157.stdout:1578: oid 35 updating version 122 to 123 2024-09-16T03:38:55.791 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 123 (ObjNum 339 snap 0 seq_num 339) dirty exists 2024-09-16T03:38:55.791 INFO:tasks.rados.rados.0.smithi157.stdout:1578: left oid 35 (ObjNum 339 snap 0 seq_num 339) 2024-09-16T03:38:55.791 INFO:tasks.rados.rados.0.smithi157.stdout:1577: got (0) Success 2024-09-16T03:38:55.791 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 0 (ObjNum 318 snap 0 seq_num 318) dirty exists 2024-09-16T03:38:55.792 INFO:tasks.rados.rados.0.smithi157.stdout:1583: expect (ObjNum 334 snap 0 seq_num 334) 2024-09-16T03:38:55.800 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:55.797+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:55.801 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:55.799+0000 7efeef6d5640 -1 osd.13 120 handle_osd_map injecting map crc failure 2024-09-16T03:38:55.804 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2596587 2024-09-16T03:38:55.829 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2658927 2024-09-16T03:38:55.869 INFO:tasks.rados.rados.0.smithi157.stdout:1582: finishing set_chunk to oid 29 2024-09-16T03:38:55.869 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 108 (ObjNum 246 snap 0 seq_num 246) dirty exists 2024-09-16T03:38:55.869 INFO:tasks.rados.rados.0.smithi157.stdout:1584: finishing set_chunk to oid 13 2024-09-16T03:38:55.869 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 72 (ObjNum 304 snap 0 seq_num 304) dirty exists 2024-09-16T03:38:55.869 INFO:tasks.rados.rados.0.smithi157.stdout:1586: expect (ObjNum 321 snap 0 seq_num 321) 2024-09-16T03:38:55.869 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:55.872 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:55.872 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:55.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:55.872 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:55.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.872 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:3 3.0 3.f934d18 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:55.872 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:38:55.872 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 100.469633694, 2024-09-16T03:38:55.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78381636700000001, 2024-09-16T03:38:55.873 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.873 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.873 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:55.873 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:38:55.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399429+0000", 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399434+0000", 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1050000000000003e-06 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399442+0000", 2024-09-16T03:38:55.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0160000000000008e-06 2024-09-16T03:38:55.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399448+0000", 2024-09-16T03:38:55.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.879e-06 2024-09-16T03:38:55.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399489+0000", 2024-09-16T03:38:55.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0815999999999997e-05 2024-09-16T03:38:55.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:55.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399495+0000", 2024-09-16T03:38:55.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3439999999999997e-06 2024-09-16T03:38:55.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183226+0000", 2024-09-16T03:38:55.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78373077400000002 2024-09-16T03:38:55.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183244+0000", 2024-09-16T03:38:55.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8128999999999999e-05 2024-09-16T03:38:55.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.877 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.877 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:5 3.0 3.aefbbec0 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:55.877 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:38:55.877 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 100.36986623200001, 2024-09-16T03:38:55.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68447923499999996, 2024-09-16T03:38:55.877 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.877 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.878 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:55.878 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:38:55.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:38:55.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499196+0000", 2024-09-16T03:38:55.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3480000000000001e-06 2024-09-16T03:38:55.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499202+0000", 2024-09-16T03:38:55.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4219999999999999e-06 2024-09-16T03:38:55.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499212+0000", 2024-09-16T03:38:55.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:38:55.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499218+0000", 2024-09-16T03:38:55.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4920000000000001e-06 2024-09-16T03:38:55.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499283+0000", 2024-09-16T03:38:55.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4906000000000005e-05 2024-09-16T03:38:55.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:55.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499293+0000", 2024-09-16T03:38:55.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0513000000000001e-05 2024-09-16T03:38:55.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183665+0000", 2024-09-16T03:38:55.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68437196600000005 2024-09-16T03:38:55.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183674+0000", 2024-09-16T03:38:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9700000000000005e-06 2024-09-16T03:38:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:6 3.0 3.3bcda120 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:55.882 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 100.369805623, 2024-09-16T03:38:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78474037699999999, 2024-09-16T03:38:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:55.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:38:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0500000000000002e-07 2024-09-16T03:38:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499260+0000", 2024-09-16T03:38:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6669999999999998e-06 2024-09-16T03:38:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499266+0000", 2024-09-16T03:38:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:38:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499274+0000", 2024-09-16T03:38:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2250000000000003e-06 2024-09-16T03:38:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499316+0000", 2024-09-16T03:38:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2670000000000003e-05 2024-09-16T03:38:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499323+0000", 2024-09-16T03:38:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.37e-06 2024-09-16T03:38:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283987+0000", 2024-09-16T03:38:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78466433700000005 2024-09-16T03:38:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283996+0000", 2024-09-16T03:38:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8650000000000007e-06 2024-09-16T03:38:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:15 3.0 3.7ec93288 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:38:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 100.260613613, 2024-09-16T03:38:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57518172599999995, 2024-09-16T03:38:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:38:55.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:38:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608449+0000", 2024-09-16T03:38:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.184e-06 2024-09-16T03:38:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608459+0000", 2024-09-16T03:38:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0553999999999999e-05 2024-09-16T03:38:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608470+0000", 2024-09-16T03:38:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0524e-05 2024-09-16T03:38:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608493+0000", 2024-09-16T03:38:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2980999999999999e-05 2024-09-16T03:38:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608537+0000", 2024-09-16T03:38:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3850000000000002e-05 2024-09-16T03:38:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608542+0000", 2024-09-16T03:38:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:38:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183623+0000", 2024-09-16T03:38:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.575080649 2024-09-16T03:38:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183629+0000", 2024-09-16T03:38:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6950000000000003e-06 2024-09-16T03:38:55.892 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:11 3.0 3.77005160 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:38:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:38:55.893 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 100.16711330299999, 2024-09-16T03:38:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.48153733399999998, 2024-09-16T03:38:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:38:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:38:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701949+0000", 2024-09-16T03:38:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.314e-06 2024-09-16T03:38:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701954+0000", 2024-09-16T03:38:55.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:38:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701964+0000", 2024-09-16T03:38:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9550000000000004e-06 2024-09-16T03:38:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701986+0000", 2024-09-16T03:38:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1931999999999999e-05 2024-09-16T03:38:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802129+0000", 2024-09-16T03:38:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100142986 2024-09-16T03:38:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802135+0000", 2024-09-16T03:38:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6350000000000003e-06 2024-09-16T03:38:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183475+0000", 2024-09-16T03:38:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38133958099999998 2024-09-16T03:38:55.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183485+0000", 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0642000000000001e-05 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:6 3.0 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 98.886209390000005, 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60469085300000003, 2024-09-16T03:38:55.898 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:38:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:38:55.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982853+0000", 2024-09-16T03:38:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.393e-06 2024-09-16T03:38:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982858+0000", 2024-09-16T03:38:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.062e-06 2024-09-16T03:38:55.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982867+0000", 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982879+0000", 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2241e-05 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982996+0000", 2024-09-16T03:38:55.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116996 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983007+0000", 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0694e-05 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284499+0000", 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30149171400000002 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284690+0000", 2024-09-16T03:38:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000191032 2024-09-16T03:38:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284908+0000", 2024-09-16T03:38:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021827399999999999 2024-09-16T03:38:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487092+0000", 2024-09-16T03:38:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.202184156 2024-09-16T03:38:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487110+0000", 2024-09-16T03:38:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7856999999999999e-05 2024-09-16T03:38:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587335+0000", 2024-09-16T03:38:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100224802 2024-09-16T03:38:55.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587410+0000", 2024-09-16T03:38:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5343e-05 2024-09-16T03:38:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587543+0000", 2024-09-16T03:38:55.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000132492 2024-09-16T03:38:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:2 3.0 3:16550cfd:::smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:38:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 98.886137198, 2024-09-16T03:38:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40289696000000003, 2024-09-16T03:38:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.906 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:38:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:38:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982925+0000", 2024-09-16T03:38:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:38:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982928+0000", 2024-09-16T03:38:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4130000000000002e-06 2024-09-16T03:38:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982936+0000", 2024-09-16T03:38:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.875e-06 2024-09-16T03:38:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982942+0000", 2024-09-16T03:38:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:38:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983093+0000", 2024-09-16T03:38:55.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015144400000000001 2024-09-16T03:38:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983098+0000", 2024-09-16T03:38:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4909999999999999e-06 2024-09-16T03:38:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385695+0000", 2024-09-16T03:38:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402597539 2024-09-16T03:38:55.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385821+0000", 2024-09-16T03:38:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125418 2024-09-16T03:38:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.911 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:9 3.0 3:2c9b9e7c:::smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:38:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:38:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 98.886121008999993, 2024-09-16T03:38:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60615655800000001, 2024-09-16T03:38:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:38:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:38:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982941+0000", 2024-09-16T03:38:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0860000000000001e-06 2024-09-16T03:38:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982946+0000", 2024-09-16T03:38:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:38:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982954+0000", 2024-09-16T03:38:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8520000000000004e-06 2024-09-16T03:38:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982958+0000", 2024-09-16T03:38:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4870000000000004e-06 2024-09-16T03:38:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983116+0000", 2024-09-16T03:38:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015774699999999999 2024-09-16T03:38:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983120+0000", 2024-09-16T03:38:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:38:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486023+0000", 2024-09-16T03:38:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50290281400000003 2024-09-16T03:38:55.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486423+0000", 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040079299999999998 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486707+0000", 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028336600000000001 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588861+0000", 2024-09-16T03:38:55.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102154756 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588952+0000", 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0630000000000005e-05 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588984+0000", 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2116999999999998e-05 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589022+0000", 2024-09-16T03:38:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7910000000000001e-05 2024-09-16T03:38:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589097+0000", 2024-09-16T03:38:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4536999999999995e-05 2024-09-16T03:38:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:12 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:38:55.919 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:38:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 98.886045660999997, 2024-09-16T03:38:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60616102900000002, 2024-09-16T03:38:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:38:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:38:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:38:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983017+0000", 2024-09-16T03:38:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.483e-06 2024-09-16T03:38:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983022+0000", 2024-09-16T03:38:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:38:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983032+0000", 2024-09-16T03:38:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0036e-05 2024-09-16T03:38:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983038+0000", 2024-09-16T03:38:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6800000000000004e-06 2024-09-16T03:38:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983155+0000", 2024-09-16T03:38:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116401 2024-09-16T03:38:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983163+0000", 2024-09-16T03:38:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6219999999999998e-06 2024-09-16T03:38:55.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589120+0000", 2024-09-16T03:38:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60595653199999999 2024-09-16T03:38:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589176+0000", 2024-09-16T03:38:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6542000000000002e-05 2024-09-16T03:38:55.924 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:30 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:38:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:38:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 94.652727154999994, 2024-09-16T03:38:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40312788999999999, 2024-09-16T03:38:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.925 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216338+0000", 2024-09-16T03:38:55.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0799999999999999e-06 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216354+0000", 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6439e-05 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216392+0000", 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7162000000000001e-05 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216444+0000", 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2082e-05 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417447+0000", 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20100358400000001 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417583+0000", 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135537 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417777+0000", 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019431200000000001 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418479+0000", 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00070148499999999996 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418755+0000", 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027627100000000003 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619285+0000", 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200530138 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619370+0000", 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4567999999999994e-05 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619462+0000", 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2231999999999994e-05 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.930 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:34 3.0 3:08307d25:::2b3cb797ae2d3abc24b2cae4a012f26325debe2fc38ccb08bd4503e949d305ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.394271098000004, 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30798127600000003, 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:38:55.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474791+0000", 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6000000000000003e-07 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474865+0000", 2024-09-16T03:38:55.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4504000000000004e-05 2024-09-16T03:38:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474882+0000", 2024-09-16T03:38:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6765999999999998e-05 2024-09-16T03:38:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474890+0000", 2024-09-16T03:38:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.14e-06 2024-09-16T03:38:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681242+0000", 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20635221200000001 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681355+0000", 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000112211 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681505+0000", 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015040100000000001 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682352+0000", 2024-09-16T03:38:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084686400000000002 2024-09-16T03:38:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682989+0000", 2024-09-16T03:38:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063749800000000001 2024-09-16T03:38:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782695+0000", 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099706063999999997 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782735+0000", 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9304000000000002e-05 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782771+0000", 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6551999999999998e-05 2024-09-16T03:38:55.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:87 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:38:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:38:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 65.065076374, 2024-09-16T03:38:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402368742, 2024-09-16T03:38:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.937 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:38:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:38:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803986+0000", 2024-09-16T03:38:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9400000000000004e-07 2024-09-16T03:38:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804061+0000", 2024-09-16T03:38:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5230999999999999e-05 2024-09-16T03:38:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804069+0000", 2024-09-16T03:38:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7530000000000001e-06 2024-09-16T03:38:55.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804073+0000", 2024-09-16T03:38:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9660000000000004e-06 2024-09-16T03:38:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907236+0000", 2024-09-16T03:38:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103162566 2024-09-16T03:38:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907324+0000", 2024-09-16T03:38:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7894999999999998e-05 2024-09-16T03:38:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:38:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907474+0000", 2024-09-16T03:38:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014983999999999999 2024-09-16T03:38:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.908330+0000", 2024-09-16T03:38:55.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085600100000000001 2024-09-16T03:38:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107220+0000", 2024-09-16T03:38:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198889812 2024-09-16T03:38:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206269+0000", 2024-09-16T03:38:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099049611999999995 2024-09-16T03:38:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206305+0000", 2024-09-16T03:38:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5679000000000002e-05 2024-09-16T03:38:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206353+0000", 2024-09-16T03:38:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8593e-05 2024-09-16T03:38:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:3086 3.0 3:1e42685c:::6a9007270aa32b63b7f7e5355768bf791fa2d1832cb8adf59666d35d1b037cad:head [write 0~1679360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.457238846999999, 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57631983099999995, 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 3086 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:38:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:38:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411824+0000", 2024-09-16T03:38:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.849e-06 2024-09-16T03:38:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414291+0000", 2024-09-16T03:38:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024666900000000001 2024-09-16T03:38:55.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414304+0000", 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2886e-05 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414310+0000", 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4219999999999997e-06 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783484+0000", 2024-09-16T03:38:55.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.36917337700000002 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783601+0000", 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011796899999999999 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783711+0000", 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110012 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887652+0000", 2024-09-16T03:38:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103940373 2024-09-16T03:38:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887669+0000", 2024-09-16T03:38:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6864e-05 2024-09-16T03:38:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942330+0000", 2024-09-16T03:38:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.054661333999999999 2024-09-16T03:38:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987927+0000", 2024-09-16T03:38:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045596722999999999 2024-09-16T03:38:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987993+0000", 2024-09-16T03:38:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6141000000000003e-05 2024-09-16T03:38:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.988142+0000", 2024-09-16T03:38:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014919099999999999 2024-09-16T03:38:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:211 3.0 3:09d857e3:::85925bf7ff4b61513b387a03c5bd27d81ce6bd960cfaf104686b99338ce39220:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:38:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.231059737999999, 2024-09-16T03:38:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.55082720799999996, 2024-09-16T03:38:55.951 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:38:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:38:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638003+0000", 2024-09-16T03:38:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8250000000000001e-06 2024-09-16T03:38:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638010+0000", 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4130000000000003e-06 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638022+0000", 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2016e-05 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638028+0000", 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0390000000000001e-06 2024-09-16T03:38:55.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886321+0000", 2024-09-16T03:38:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.248293081 2024-09-16T03:38:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886428+0000", 2024-09-16T03:38:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000106791 2024-09-16T03:38:55.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:38:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886630+0000", 2024-09-16T03:38:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020280499999999999 2024-09-16T03:38:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889701+0000", 2024-09-16T03:38:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030706650000000002 2024-09-16T03:38:55.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942392+0000", 2024-09-16T03:38:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.052691156000000003 2024-09-16T03:38:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987675+0000", 2024-09-16T03:38:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045283229000000001 2024-09-16T03:38:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188710+0000", 2024-09-16T03:38:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20103479899999999 2024-09-16T03:38:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188753+0000", 2024-09-16T03:38:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2762000000000002e-05 2024-09-16T03:38:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188829+0000", 2024-09-16T03:38:55.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5626999999999999e-05 2024-09-16T03:38:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:214 3.0 3:2fb3d035:::5dba7feb0009e61b5e6a50f88cc7ba99127287f7eeb210342a4b40e996304a6b:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:38:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.230945664, 2024-09-16T03:38:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.55085424699999996, 2024-09-16T03:38:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.959 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:38:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:38:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638116+0000", 2024-09-16T03:38:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1200000000000002e-07 2024-09-16T03:38:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638120+0000", 2024-09-16T03:38:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3400000000000002e-06 2024-09-16T03:38:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638126+0000", 2024-09-16T03:38:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2339999999999999e-06 2024-09-16T03:38:55.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638130+0000", 2024-09-16T03:38:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6349999999999997e-06 2024-09-16T03:38:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887101+0000", 2024-09-16T03:38:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.248970462 2024-09-16T03:38:55.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887191+0000", 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0087000000000007e-05 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887375+0000", 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018401600000000001 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889706+0000", 2024-09-16T03:38:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023305869999999998 2024-09-16T03:38:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942851+0000", 2024-09-16T03:38:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.053145489999999997 2024-09-16T03:38:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.088369+0000", 2024-09-16T03:38:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.14551832200000001 2024-09-16T03:38:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188856+0000", 2024-09-16T03:38:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100486244 2024-09-16T03:38:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188902+0000", 2024-09-16T03:38:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6134999999999998e-05 2024-09-16T03:38:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188970+0000", 2024-09-16T03:38:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7983000000000003e-05 2024-09-16T03:38:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:219 3.0 3:21e0c4e8:::f1ac9f6b1092b2dbfccdb2f32bdf689fb55e405da82aefeec91d0d73cdce51e1:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:38:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 20.131139382000001, 2024-09-16T03:38:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.71756203500000004, 2024-09-16T03:38:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:38:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:38:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737924+0000", 2024-09-16T03:38:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8500000000000001e-06 2024-09-16T03:38:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737931+0000", 2024-09-16T03:38:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9879999999999998e-06 2024-09-16T03:38:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737942+0000", 2024-09-16T03:38:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1837e-05 2024-09-16T03:38:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.838013+0000", 2024-09-16T03:38:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100070438 2024-09-16T03:38:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887695+0000", 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.049682624000000002 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887831+0000", 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013565499999999999 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887948+0000", 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000117353 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889709+0000", 2024-09-16T03:38:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017605629999999999 2024-09-16T03:38:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.043145+0000", 2024-09-16T03:38:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.153436136 2024-09-16T03:38:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188664+0000", 2024-09-16T03:38:55.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.14551857000000001 2024-09-16T03:38:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455373+0000", 2024-09-16T03:38:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.26670951199999998 2024-09-16T03:38:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455413+0000", 2024-09-16T03:38:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9517999999999997e-05 2024-09-16T03:38:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455484+0000", 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0990999999999998e-05 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:3100 3.0 3:2db88641:::a63ac782fa4bce1e96397c8301f2734ce3d5850a2dc454a54c42fc5451a8d72d:head [write 0~1584128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 19.597559125, 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38491842399999998, 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.973 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 3100 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271523+0000", 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0649000000000002e-05 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273460+0000", 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019373820000000001 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273488+0000", 2024-09-16T03:38:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7537000000000001e-05 2024-09-16T03:38:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273503+0000", 2024-09-16T03:38:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5438999999999999e-05 2024-09-16T03:38:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273645+0000", 2024-09-16T03:38:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000142013 2024-09-16T03:38:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273820+0000", 2024-09-16T03:38:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017546200000000001 2024-09-16T03:38:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:38:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273943+0000", 2024-09-16T03:38:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012269099999999999 2024-09-16T03:38:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.276532+0000", 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025884010000000002 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455527+0000", 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.17899536199999999 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.556259+0000", 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100731821 2024-09-16T03:38:55.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656291+0000", 2024-09-16T03:38:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100032144 2024-09-16T03:38:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656330+0000", 2024-09-16T03:38:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8600000000000003e-05 2024-09-16T03:38:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656420+0000", 2024-09-16T03:38:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0922999999999997e-05 2024-09-16T03:38:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:54 3.0 3:04086e73:::a30f14ebc2b326cba32a5aae93ab4cf09bd2a972b24cfcba49e65d7da3ac52de:head [call cas.chunk_create_or_get_ref in=131071b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:38:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:38:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.51375275, 2024-09-16T03:38:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30746864499999998, 2024-09-16T03:38:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.980 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355311+0000", 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3250000000000002e-06 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355502+0000", 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019106 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355515+0000", 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3730000000000001e-05 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355526+0000", 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0473e-05 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355567+0000", 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1455000000000002e-05 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355576+0000", 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2840000000000006e-06 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.457986+0000", 2024-09-16T03:38:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10241070200000001 2024-09-16T03:38:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.458087+0000", 2024-09-16T03:38:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000100715 2024-09-16T03:38:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:55.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.458252+0000", 2024-09-16T03:38:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016499 2024-09-16T03:38:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560797+0000", 2024-09-16T03:38:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10254485300000001 2024-09-16T03:38:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662591+0000", 2024-09-16T03:38:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10179377100000001 2024-09-16T03:38:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662693+0000", 2024-09-16T03:38:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000102174 2024-09-16T03:38:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662745+0000", 2024-09-16T03:38:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1887e-05 2024-09-16T03:38:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662777+0000", 2024-09-16T03:38:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2225999999999999e-05 2024-09-16T03:38:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.987 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:58 3.0 3:07e63342:::0385a732ea8ffd270f01137d752687ebb6ebb56bc894ea96f00cc7a27fcd2563:head [call cas.chunk_create_or_get_ref in=284238b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:38:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:38:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.513519916, 2024-09-16T03:38:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30738563099999999, 2024-09-16T03:38:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:38:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:38:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355542+0000", 2024-09-16T03:38:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1699999999999997e-07 2024-09-16T03:38:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355761+0000", 2024-09-16T03:38:55.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021889699999999999 2024-09-16T03:38:55.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355769+0000", 2024-09-16T03:38:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.695e-06 2024-09-16T03:38:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355775+0000", 2024-09-16T03:38:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5239999999999997e-06 2024-09-16T03:38:55.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355851+0000", 2024-09-16T03:38:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6103e-05 2024-09-16T03:38:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355860+0000", 2024-09-16T03:38:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0710000000000001e-06 2024-09-16T03:38:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459143+0000", 2024-09-16T03:38:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10328306399999999 2024-09-16T03:38:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459292+0000", 2024-09-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000148221 2024-09-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459540+0000", 2024-09-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024846100000000001 2024-09-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662661+0000", 2024-09-16T03:38:55.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.203120936 2024-09-16T03:38:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662669+0000", 2024-09-16T03:38:55.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6520000000000006e-06 2024-09-16T03:38:55.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:55.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662824+0000", 2024-09-16T03:38:55.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015512699999999999 2024-09-16T03:38:55.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662885+0000", 2024-09-16T03:38:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0789999999999999e-05 2024-09-16T03:38:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662927+0000", 2024-09-16T03:38:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2172999999999998e-05 2024-09-16T03:38:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:55.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.000 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:60 3.0 3:16d0c248:::a2b7d44d389d3ac8752762fa1d216869a8025d7e5ae15e8811d7e7f2543095f6:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:38:56.000 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:38:56.000 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.513274845, 2024-09-16T03:38:56.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50513552500000003, 2024-09-16T03:38:56.000 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:56.000 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:56.000 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.000 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:56.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:38:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:38:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:56.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355787+0000", 2024-09-16T03:38:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-16T03:38:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355791+0000", 2024-09-16T03:38:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.631e-06 2024-09-16T03:38:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355797+0000", 2024-09-16T03:38:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7379999999999999e-06 2024-09-16T03:38:56.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355802+0000", 2024-09-16T03:38:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.322e-06 2024-09-16T03:38:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.356898+0000", 2024-09-16T03:38:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010963710000000001 2024-09-16T03:38:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:38:56.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.356902+0000", 2024-09-16T03:38:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0110000000000002e-06 2024-09-16T03:38:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560272+0000", 2024-09-16T03:38:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20336954199999999 2024-09-16T03:38:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560376+0000", 2024-09-16T03:38:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103885 2024-09-16T03:38:56.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:38:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560508+0000", 2024-09-16T03:38:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013254599999999999 2024-09-16T03:38:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:38:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662786+0000", 2024-09-16T03:38:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10227757699999999 2024-09-16T03:38:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:56.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860745+0000", 2024-09-16T03:38:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19795950700000001 2024-09-16T03:38:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860843+0000", 2024-09-16T03:38:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7829000000000001e-05 2024-09-16T03:38:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860890+0000", 2024-09-16T03:38:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7005999999999997e-05 2024-09-16T03:38:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:56.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860922+0000", 2024-09-16T03:38:56.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1680000000000002e-05 2024-09-16T03:38:56.007 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.007 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.007 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.007 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.007 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.007 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:56.008 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:56.009 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:38:56.010 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:56.007+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:56.052 INFO:tasks.rados.rados.0.smithi157.stdout:1549: done (15 left) 2024-09-16T03:38:56.052 INFO:tasks.rados.rados.0.smithi157.stdout:1569: done (14 left) 2024-09-16T03:38:56.053 INFO:tasks.rados.rados.0.smithi157.stdout:1573: done (13 left) 2024-09-16T03:38:56.053 INFO:tasks.rados.rados.0.smithi157.stdout:1574: done (12 left) 2024-09-16T03:38:56.053 INFO:tasks.rados.rados.0.smithi157.stdout:1575: done (11 left) 2024-09-16T03:38:56.053 INFO:tasks.rados.rados.0.smithi157.stdout:1576: done (10 left) 2024-09-16T03:38:56.053 INFO:tasks.rados.rados.0.smithi157.stdout:1577: done (9 left) 2024-09-16T03:38:56.053 INFO:tasks.rados.rados.0.smithi157.stdout:1578: done (8 left) 2024-09-16T03:38:56.053 INFO:tasks.rados.rados.0.smithi157.stdout:1579: done (7 left) 2024-09-16T03:38:56.054 INFO:tasks.rados.rados.0.smithi157.stdout:1580: done (6 left) 2024-09-16T03:38:56.054 INFO:tasks.rados.rados.0.smithi157.stdout:1581: done (5 left) 2024-09-16T03:38:56.054 INFO:tasks.rados.rados.0.smithi157.stdout:1582: done (4 left) 2024-09-16T03:38:56.054 INFO:tasks.rados.rados.0.smithi157.stdout:1583: done (3 left) 2024-09-16T03:38:56.054 INFO:tasks.rados.rados.0.smithi157.stdout:1584: done (2 left) 2024-09-16T03:38:56.054 INFO:tasks.rados.rados.0.smithi157.stdout:1585: done (1 left) 2024-09-16T03:38:56.054 INFO:tasks.rados.rados.0.smithi157.stdout:1586: done (0 left) 2024-09-16T03:38:56.054 INFO:tasks.rados.rados.0.smithi157.stdout:1587: read oid 19 snap -1 2024-09-16T03:38:56.054 INFO:tasks.rados.rados.0.smithi157.stdout:1587: expect (ObjNum 331 snap 0 seq_num 331) 2024-09-16T03:38:56.054 INFO:tasks.rados.rados.0.smithi157.stdout:1588: set_chunk oid 4 target oid 2024-09-16T03:38:56.109 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:56.107+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:56.153 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:56.153 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:56.153 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:38:56.153 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:56.162 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:38:56.210 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:56.209+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:56.264 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:56.264 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:38:56.264 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:38:56.264 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:38:56.265 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:56.273 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:38:56.311 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:56.309+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:56.358 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T03:38:56.372 INFO:tasks.rados.rados.0.smithi157.stdout:1588: set_chunk oid 4 offset: 1543168 length: 1411584 target oid 46b61e4f4444b02d24bec340347a553c3871945dc5b13fa530142a9c4eda3480 offset: 0 snap -1 2024-09-16T03:38:56.373 INFO:tasks.rados.rados.0.smithi157.stdout:1589: tier_evict oid 29 2024-09-16T03:38:56.373 INFO:tasks.rados.rados.0.smithi157.stdout:1589: tier_evict oid 29 snap -1 2024-09-16T03:38:56.373 INFO:tasks.rados.rados.0.smithi157.stdout:1587: expect (ObjNum 331 snap 0 seq_num 331) 2024-09-16T03:38:56.374 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T03:38:56.400 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3368534 2024-09-16T03:38:56.412 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:56.410+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:56.417 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:38:56.417 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:38:56.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:38:56.418 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:38:56.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.418 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:57 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:56.418 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:56.418 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.290820728, 2024-09-16T03:38:56.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20372090900000001, 2024-09-16T03:38:56.418 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:56.418 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:56.418 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.419 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:56.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:56.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:56.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:56.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:38:56.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:56.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125313+0000", 2024-09-16T03:38:56.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7180000000000001e-06 2024-09-16T03:38:56.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:56.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126730+0000", 2024-09-16T03:38:56.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00141731 2024-09-16T03:38:56.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:56.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126746+0000", 2024-09-16T03:38:56.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6455000000000001e-05 2024-09-16T03:38:56.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126756+0000", 2024-09-16T03:38:56.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:38:56.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:56.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227068+0000", 2024-09-16T03:38:56.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100312308 2024-09-16T03:38:56.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:56.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227085+0000", 2024-09-16T03:38:56.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6898e-05 2024-09-16T03:38:56.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:56.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328995+0000", 2024-09-16T03:38:56.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101909903 2024-09-16T03:38:56.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:56.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329031+0000", 2024-09-16T03:38:56.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5717999999999999e-05 2024-09-16T03:38:56.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.423 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.424 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:58 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'2, mlcod=0'0)", 2024-09-16T03:38:56.424 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:56.424 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.287487598, 2024-09-16T03:38:56.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084524500000001, 2024-09-16T03:38:56.425 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:56.425 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:56.425 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.425 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:56.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:56.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:56.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:56.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:38:56.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:56.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128644+0000", 2024-09-16T03:38:56.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.251e-06 2024-09-16T03:38:56.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:56.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128913+0000", 2024-09-16T03:38:56.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026856200000000002 2024-09-16T03:38:56.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:56.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128925+0000", 2024-09-16T03:38:56.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2109999999999999e-05 2024-09-16T03:38:56.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128933+0000", 2024-09-16T03:38:56.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6260000000000003e-06 2024-09-16T03:38:56.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:56.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328196+0000", 2024-09-16T03:38:56.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19926303200000001 2024-09-16T03:38:56.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:56.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328253+0000", 2024-09-16T03:38:56.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7262999999999997e-05 2024-09-16T03:38:56.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:56.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329458+0000", 2024-09-16T03:38:56.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012051939999999999 2024-09-16T03:38:56.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:56.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329488+0000", 2024-09-16T03:38:56.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0207000000000001e-05 2024-09-16T03:38:56.430 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.430 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.430 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.430 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:59 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'3, mlcod=0'0)", 2024-09-16T03:38:56.430 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:56.430 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.286992165, 2024-09-16T03:38:56.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036954900000001, 2024-09-16T03:38:56.431 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:56.431 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:56.431 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.431 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:56.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:56.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:56.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:56.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:38:56.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129139+0000", 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6099999999999999e-07 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129158+0000", 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088e-05 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:56.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129168+0000", 2024-09-16T03:38:56.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.048e-05 2024-09-16T03:38:56.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.229233+0000", 2024-09-16T03:38:56.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100065168 2024-09-16T03:38:56.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:56.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328815+0000", 2024-09-16T03:38:56.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099581694999999998 2024-09-16T03:38:56.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:56.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328845+0000", 2024-09-16T03:38:56.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0193e-05 2024-09-16T03:38:56.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:56.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329492+0000", 2024-09-16T03:38:56.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064671800000000001 2024-09-16T03:38:56.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:56.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329508+0000", 2024-09-16T03:38:56.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6246e-05 2024-09-16T03:38:56.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.435 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.435 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.0:15 4.d e47/32 4:b3fe05b2:::smithi15736696-15:head v 47'5, mlcod=39'3)", 2024-09-16T03:38:56.435 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:56.435 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 90.113697439000006, 2024-09-16T03:38:56.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100920751, 2024-09-16T03:38:56.435 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:56.435 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:56.436 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.436 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:56.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:56.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:56.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:56.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:38:56.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:56.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302435+0000", 2024-09-16T03:38:56.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2340000000000001e-06 2024-09-16T03:38:56.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:56.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302446+0000", 2024-09-16T03:38:56.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0596e-05 2024-09-16T03:38:56.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:56.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302461+0000", 2024-09-16T03:38:56.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5082000000000001e-05 2024-09-16T03:38:56.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402592+0000", 2024-09-16T03:38:56.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100130488 2024-09-16T03:38:56.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:56.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402712+0000", 2024-09-16T03:38:56.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012013100000000001 2024-09-16T03:38:56.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:56.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402741+0000", 2024-09-16T03:38:56.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9717999999999999e-05 2024-09-16T03:38:56.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:56.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403309+0000", 2024-09-16T03:38:56.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056725700000000005 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403354+0000", 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5244999999999998e-05 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 66.266198729999999, 2024-09-16T03:38:56.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032593800000000001, 2024-09-16T03:38:56.441 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:56.441 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:56.441 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.441 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:56.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:56.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:56.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:56.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:38:56.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:56.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149943+0000", 2024-09-16T03:38:56.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0817e-05 2024-09-16T03:38:56.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:56.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151026+0000", 2024-09-16T03:38:56.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010835350000000001 2024-09-16T03:38:56.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:56.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151040+0000", 2024-09-16T03:38:56.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3884e-05 2024-09-16T03:38:56.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151049+0000", 2024-09-16T03:38:56.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:38:56.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:56.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151107+0000", 2024-09-16T03:38:56.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8199000000000003e-05 2024-09-16T03:38:56.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:56.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151124+0000", 2024-09-16T03:38:56.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7756000000000001e-05 2024-09-16T03:38:56.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:56.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153142+0000", 2024-09-16T03:38:56.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020177569999999998 2024-09-16T03:38:56.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:56.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153191+0000", 2024-09-16T03:38:56.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9051999999999997e-05 2024-09-16T03:38:56.446 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.446 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.446 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.446 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1682 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'7, mlcod=40'4)", 2024-09-16T03:38:56.447 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:56.447 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 66.162601539999997, 2024-09-16T03:38:56.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10169502499999999, 2024-09-16T03:38:56.447 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:56.447 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:56.447 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.447 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:56.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:56.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:56.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:56.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:38:56.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:56.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253540+0000", 2024-09-16T03:38:56.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0888e-05 2024-09-16T03:38:56.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:56.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254015+0000", 2024-09-16T03:38:56.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047485499999999999 2024-09-16T03:38:56.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:56.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254030+0000", 2024-09-16T03:38:56.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4899000000000001e-05 2024-09-16T03:38:56.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254039+0000", 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5000000000000005e-06 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254107+0000", 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7298000000000005e-05 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254122+0000", 2024-09-16T03:38:56.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5928999999999999e-05 2024-09-16T03:38:56.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:56.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355184+0000", 2024-09-16T03:38:56.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101061703 2024-09-16T03:38:56.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:56.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355224+0000", 2024-09-16T03:38:56.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9953000000000002e-05 2024-09-16T03:38:56.451 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.452 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.452 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1683 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'8, mlcod=40'4)", 2024-09-16T03:38:56.452 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:56.452 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 66.162063992, 2024-09-16T03:38:56.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101630885, 2024-09-16T03:38:56.452 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:56.452 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:56.452 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.452 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:56.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:56.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:56.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:56.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:38:56.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:56.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254068+0000", 2024-09-16T03:38:56.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5700000000000001e-07 2024-09-16T03:38:56.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:56.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254075+0000", 2024-09-16T03:38:56.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2710000000000003e-06 2024-09-16T03:38:56.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:56.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254083+0000", 2024-09-16T03:38:56.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0220000000000004e-06 2024-09-16T03:38:56.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254089+0000", 2024-09-16T03:38:56.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8699999999999997e-06 2024-09-16T03:38:56.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:56.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354968+0000", 2024-09-16T03:38:56.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100879625 2024-09-16T03:38:56.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:56.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354992+0000", 2024-09-16T03:38:56.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-16T03:38:56.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:56.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355665+0000", 2024-09-16T03:38:56.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00067371100000000001 2024-09-16T03:38:56.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:56.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355698+0000", 2024-09-16T03:38:56.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2291e-05 2024-09-16T03:38:56.457 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.457 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.457 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.457 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.0:177 4.10 e65/57 4:0a55bbf2:::smithi15736696-18:head v 65'47, mlcod=0'0)", 2024-09-16T03:38:56.457 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:56.457 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 65.209117403999997, 2024-09-16T03:38:56.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074461099999999995, 2024-09-16T03:38:56.458 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:56.458 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:56.458 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.458 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:56.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:56.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:56.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:56.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:56.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:56.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:38:56.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5300000000000001e-07 2024-09-16T03:38:56.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:56.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207018+0000", 2024-09-16T03:38:56.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7210000000000002e-06 2024-09-16T03:38:56.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:56.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207031+0000", 2024-09-16T03:38:56.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2767e-05 2024-09-16T03:38:56.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207040+0000", 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5400000000000001e-06 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207107+0000", 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6625999999999995e-05 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207131+0000", 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4128e-05 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207712+0000", 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00058124700000000001 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207758+0000", 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5429000000000002e-05 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1723 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'48, mlcod=53'46)", 2024-09-16T03:38:56.462 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.767050222999998, 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00088284599999999996, 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:56.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:38:56.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:56.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649081+0000", 2024-09-16T03:38:56.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8299999999999995e-07 2024-09-16T03:38:56.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:56.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649101+0000", 2024-09-16T03:38:56.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0126e-05 2024-09-16T03:38:56.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:56.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649141+0000", 2024-09-16T03:38:56.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9609e-05 2024-09-16T03:38:56.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649161+0000", 2024-09-16T03:38:56.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9599999999999999e-05 2024-09-16T03:38:56.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:56.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649299+0000", 2024-09-16T03:38:56.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013822199999999999 2024-09-16T03:38:56.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:56.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649318+0000", 2024-09-16T03:38:56.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9273999999999999e-05 2024-09-16T03:38:56.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:56.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649919+0000", 2024-09-16T03:38:56.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060074100000000001 2024-09-16T03:38:56.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:56.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649963+0000", 2024-09-16T03:38:56.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4391000000000002e-05 2024-09-16T03:38:56.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.468 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.468 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.468 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1724 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'49, mlcod=53'46)", 2024-09-16T03:38:56.468 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:56.468 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.763385258, 2024-09-16T03:38:56.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064536300000000003, 2024-09-16T03:38:56.468 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:56.468 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:56.469 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.469 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:56.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:56.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:56.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:56.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:38:56.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:56.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652747+0000", 2024-09-16T03:38:56.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.291e-06 2024-09-16T03:38:56.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:56.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652753+0000", 2024-09-16T03:38:56.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4450000000000001e-06 2024-09-16T03:38:56.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:56.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652782+0000", 2024-09-16T03:38:56.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9094999999999998e-05 2024-09-16T03:38:56.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652797+0000", 2024-09-16T03:38:56.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5021999999999999e-05 2024-09-16T03:38:56.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:56.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652854+0000", 2024-09-16T03:38:56.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6586999999999999e-05 2024-09-16T03:38:56.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:56.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652885+0000", 2024-09-16T03:38:56.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.146e-05 2024-09-16T03:38:56.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:56.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653350+0000", 2024-09-16T03:38:56.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046443599999999998 2024-09-16T03:38:56.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:56.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653391+0000", 2024-09-16T03:38:56.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1026999999999999e-05 2024-09-16T03:38:56.473 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.473 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.473 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.473 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1725 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'50, mlcod=53'46)", 2024-09-16T03:38:56.474 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:56.474 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 64.762592435000002, 2024-09-16T03:38:56.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056324100000000002, 2024-09-16T03:38:56.474 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:38:56.474 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:38:56.474 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.474 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:38:56.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:38:56.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:56.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:38:56.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:38:56.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:38:56.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:38:56.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653539+0000", 2024-09-16T03:38:56.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9699999999999994e-07 2024-09-16T03:38:56.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:38:56.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653545+0000", 2024-09-16T03:38:56.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9309999999999996e-06 2024-09-16T03:38:56.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:38:56.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653557+0000", 2024-09-16T03:38:56.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2340999999999999e-05 2024-09-16T03:38:56.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653564+0000", 2024-09-16T03:38:56.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2779999999999999e-06 2024-09-16T03:38:56.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:38:56.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653605+0000", 2024-09-16T03:38:56.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1301000000000003e-05 2024-09-16T03:38:56.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:38:56.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653621+0000", 2024-09-16T03:38:56.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5951e-05 2024-09-16T03:38:56.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:38:56.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654064+0000", 2024-09-16T03:38:56.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000442843 2024-09-16T03:38:56.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:38:56.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:38:56.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:38:56.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654101+0000", 2024-09-16T03:38:56.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7598999999999997e-05 2024-09-16T03:38:56.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.479 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.479 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.479 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:38:56.479 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:38:56.479 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:38:56.480 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3327799 2024-09-16T03:38:56.480 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 32750 2024-09-16T03:38:56.481 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:38:56.500 INFO:tasks.rados.rados.0.smithi157.stdout:1589: got (22) Invalid argument 2024-09-16T03:38:56.500 INFO:tasks.rados.rados.0.smithi157.stdout:1587: done (2 left) 2024-09-16T03:38:56.500 INFO:tasks.rados.rados.0.smithi157.stdout:1589: done (1 left) 2024-09-16T03:38:56.500 INFO:tasks.rados.rados.0.smithi157.stdout:1590: read oid 9 snap -1 2024-09-16T03:38:56.501 INFO:tasks.rados.rados.0.smithi157.stdout:1590: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-16T03:38:56.501 INFO:tasks.rados.rados.0.smithi157.stdout:1591: read oid 16 snap -1 2024-09-16T03:38:56.501 INFO:tasks.rados.rados.0.smithi157.stdout:1591: expect (ObjNum 270 snap 0 seq_num 270) 2024-09-16T03:38:56.501 INFO:tasks.rados.rados.0.smithi157.stdout:1592: tier_promote oid 33 2024-09-16T03:38:56.501 INFO:tasks.rados.rados.0.smithi157.stdout:1593: set_chunk oid 34 target oid 2024-09-16T03:38:56.513 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:56.510+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:56.578 INFO:tasks.rados.rados.0.smithi157.stdout:1593: set_chunk oid 34 offset: 1747456 length: 1022976 target oid 1c7a79a7d459478c159643b05834d0cf9e24d0d40af8a204880a457aea5279f2 offset: 0 snap -1 2024-09-16T03:38:56.578 INFO:tasks.rados.rados.0.smithi157.stdout:1594: set_chunk oid 7 target oid 2024-09-16T03:38:56.596 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:56.596 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:56.596 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:56.596 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:56.600 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:56.598+0000 7f3d40708640 -1 osd.12 121 handle_osd_map injecting map crc failure 2024-09-16T03:38:56.600 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:56.599+0000 7f3a1f71e640 -1 osd.6 121 handle_osd_map injecting map crc failure 2024-09-16T03:38:56.606 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:38:56.649 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:56.648+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:56.711 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:56.711 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:56.712 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:56.712 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:56.712 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:56.714 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:56.713+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:56.720 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:38:56.804 INFO:tasks.rados.rados.0.smithi157.stdout:1594: set_chunk oid 7 offset: 1425408 length: 453632 target oid 4d740933c1b660144e02093e4df223011750ab86ed622982aaaf7f62f7981ed2 offset: 0 snap -1 2024-09-16T03:38:56.805 INFO:tasks.rados.rados.0.smithi157.stdout:1595: read oid 26 snap -1 2024-09-16T03:38:56.805 INFO:tasks.rados.rados.0.smithi157.stdout:1595: expect (ObjNum 336 snap 0 seq_num 336) 2024-09-16T03:38:56.805 INFO:tasks.rados.rados.0.smithi157.stdout:1596: write oid 20 current snap is 0 2024-09-16T03:38:56.805 INFO:tasks.rados.rados.0.smithi157.stdout:1596: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-09-16T03:38:56.810 INFO:tasks.rados.rados.0.smithi157.stdout:1596: writing smithi15736696-20 from 514380 to 1218572 tid 1 2024-09-16T03:38:56.815 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:56.813+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:56.815 INFO:tasks.rados.rados.0.smithi157.stdout:1596: writing smithi15736696-20 from 1629760 to 2108195 tid 2 2024-09-16T03:38:56.821 INFO:tasks.rados.rados.0.smithi157.stdout:1596: writing smithi15736696-20 from 2604288 to 3046890 tid 3 2024-09-16T03:38:56.821 INFO:tasks.rados.rados.0.smithi157.stdout:1597: read oid 2 snap -1 2024-09-16T03:38:56.822 INFO:tasks.rados.rados.0.smithi157.stdout:1597: expect deleted 2024-09-16T03:38:56.822 INFO:tasks.rados.rados.0.smithi157.stdout:1598: tier_promote oid 47 2024-09-16T03:38:56.822 INFO:tasks.rados.rados.0.smithi157.stdout:1590: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-16T03:38:56.822 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:56.869 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:56.869 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:38:56.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:38:56.869 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:38:56.869 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.869 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:5 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'1, mlcod=0'0)", 2024-09-16T03:38:56.869 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 104.898498459, 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.105742802, 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968147+0000", 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4e-07 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968752+0000", 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060437900000000005 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.871 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968762+0000", 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0284e-05 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968778+0000", 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6124e-05 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069797+0000", 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10101895700000001 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.872 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069833+0000", 2024-09-16T03:38:56.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6013000000000002e-05 2024-09-16T03:38:56.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073838+0000", 2024-09-16T03:38:56.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040049049999999996 2024-09-16T03:38:56.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073889+0000", 2024-09-16T03:38:56.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.13e-05 2024-09-16T03:38:56.873 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.873 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:7 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'3, mlcod=0'0)", 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 104.89160596000001, 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20061241399999999, 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:38:56.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:38:56.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975042+0000", 2024-09-16T03:38:56.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5289999999999998e-06 2024-09-16T03:38:56.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975079+0000", 2024-09-16T03:38:56.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7626999999999997e-05 2024-09-16T03:38:56.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975104+0000", 2024-09-16T03:38:56.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4958999999999999e-05 2024-09-16T03:38:56.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.075211+0000", 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100106814 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080573+0000", 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0053618650000000004 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080592+0000", 2024-09-16T03:38:56.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9079000000000001e-05 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175497+0000", 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.094904832999999994 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175651+0000", 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015470800000000001 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 104.680652788, 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10311670100000001, 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:56.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.185992+0000", 2024-09-16T03:38:56.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:38:56.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186761+0000", 2024-09-16T03:38:56.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076880999999999998 2024-09-16T03:38:56.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186772+0000", 2024-09-16T03:38:56.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0986e-05 2024-09-16T03:38:56.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186799+0000", 2024-09-16T03:38:56.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6254e-05 2024-09-16T03:38:56.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186876+0000", 2024-09-16T03:38:56.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7523999999999998e-05 2024-09-16T03:38:56.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.186896+0000", 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0302999999999999e-05 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289077+0000", 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10218086899999999 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.289109+0000", 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1625000000000003e-05 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:44 3.3 e49/31 3:dc284e8d:::smithi15736696-28 low_tier:head v 49'55, mlcod=49'53)", 2024-09-16T03:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 86.743681563999999, 2024-09-16T03:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.149341897, 2024-09-16T03:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:38:56.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122964+0000", 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.91e-07 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122968+0000", 2024-09-16T03:38:56.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1470000000000003e-06 2024-09-16T03:38:56.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122979+0000", 2024-09-16T03:38:56.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1090000000000001e-05 2024-09-16T03:38:56.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.223110+0000", 2024-09-16T03:38:56.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100131357 2024-09-16T03:38:56.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271688+0000", 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048578192999999999 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271712+0000", 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4066e-05 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272260+0000", 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054737000000000004 2024-09-16T03:38:56.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272305+0000", 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5482999999999998e-05 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1007 3.3 e50/31 3:fda66a0d:::920f29fe85d0745d4cd24a887714ac53f1b9a978ec2208ff473c99287fdf8feb:head v 50'57, mlcod=49'55)", 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 86.272055010000003, 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20236485200000001, 2024-09-16T03:38:56.888 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.889 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.889 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.889 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:38:56.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:38:56.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594592+0000", 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0700000000000001e-06 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595508+0000", 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00091599499999999996 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595521+0000", 2024-09-16T03:38:56.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:38:56.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595531+0000", 2024-09-16T03:38:56.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2499999999999995e-06 2024-09-16T03:38:56.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795762+0000", 2024-09-16T03:38:56.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200230977 2024-09-16T03:38:56.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795778+0000", 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796912+0000", 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011338520000000001 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796955+0000", 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2768000000000001e-05 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.892 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1247 4.11 4:89839b7e:::smithi15736696-12:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 79.911067829000004, 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202294525, 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:56.893 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1247 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955579+0000", 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3240000000000001e-06 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955587+0000", 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955600+0000", 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2723e-05 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955612+0000", 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2422999999999999e-05 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055837+0000", 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10022539699999999 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055974+0000", 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000136995 2024-09-16T03:38:56.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,7,8,11,12", 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.056092+0000", 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011738 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057209+0000", 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011170329999999999 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057262+0000", 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2859000000000003e-05 2024-09-16T03:38:56.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:56.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157594+0000", 2024-09-16T03:38:56.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10033241699999999 2024-09-16T03:38:56.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:56.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157648+0000", 2024-09-16T03:38:56.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3863999999999998e-05 2024-09-16T03:38:56.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:56.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157692+0000", 2024-09-16T03:38:56.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4406999999999997e-05 2024-09-16T03:38:56.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:56.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157732+0000", 2024-09-16T03:38:56.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9156000000000003e-05 2024-09-16T03:38:56.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157780+0000", 2024-09-16T03:38:56.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8801999999999998e-05 2024-09-16T03:38:56.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157872+0000", 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1311999999999996e-05 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1518 3.3 e59/31 3:df1be05b:::e06092e68680905b5745d5057e93f6c7581518aed07ca5db2802546e9c29aeef:head v 59'95, mlcod=58'93)", 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 71.608624156000005, 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104523454, 2024-09-16T03:38:56.900 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258022+0000", 2024-09-16T03:38:56.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.372e-06 2024-09-16T03:38:56.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258462+0000", 2024-09-16T03:38:56.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00043970100000000002 2024-09-16T03:38:56.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258476+0000", 2024-09-16T03:38:56.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.376e-05 2024-09-16T03:38:56.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258483+0000", 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.294e-06 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358679+0000", 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100196177 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358718+0000", 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9295999999999998e-05 2024-09-16T03:38:56.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362496+0000", 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037772069999999999 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362544+0000", 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8646999999999997e-05 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1923 4.4 e73/50 4:239dbd85:::smithi15736696-2:head v 73'19, mlcod=62'17)", 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 56.902763333999999, 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104144848, 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.906 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.906 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:38:56.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.906 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.906 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963884+0000", 2024-09-16T03:38:56.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:38:56.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964597+0000", 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071314700000000002 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964615+0000", 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8209000000000001e-05 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964626+0000", 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0511e-05 2024-09-16T03:38:56.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064779+0000", 2024-09-16T03:38:56.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100153495 2024-09-16T03:38:56.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064794+0000", 2024-09-16T03:38:56.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4946000000000001e-05 2024-09-16T03:38:56.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.067999+0000", 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032050709999999999 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068026+0000", 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7066000000000001e-05 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1924 4.4 e73/50 4:239dbd85:::smithi15736696-2:head v 73'20, mlcod=62'17)", 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:59.965625+0000", 2024-09-16T03:38:56.909 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 56.901019914999999, 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102804671, 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.965625+0000", 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.965625+0000", 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.965627+0000", 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5179999999999999e-06 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.966256+0000", 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00062937300000000002 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.966274+0000", 2024-09-16T03:38:56.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8257000000000002e-05 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.966284+0000", 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4299999999999995e-06 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.066012+0000", 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099728368999999997 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.066028+0000", 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6512e-05 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068405+0000", 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023766740000000001 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068430+0000", 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4538000000000001e-05 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.913 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:106 3.3 e73/31 3:dbb0d6f3:::b417679517d63919e63cf420e20217e40570d74b9322ef8b960ef314107040f0:head v 73'120, mlcod=73'116)", 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 56.588542408000002, 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.1035437, 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4900000000000004e-07 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278120+0000", 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6214000000000001e-05 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278129+0000", 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.115e-06 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278138+0000", 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.329e-06 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.378298+0000", 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100159945 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.378320+0000", 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1810999999999998e-05 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.381607+0000", 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032868210000000001 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.381646+0000", 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9515999999999999e-05 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2145 4.4 e77/50 4:239dbd85:::smithi15736696-2:head v 77'22, mlcod=73'20)", 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.410542+0000", 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.456102610999999, 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10336585099999999, 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.918 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.410542+0000", 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.410542+0000", 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.410543+0000", 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0599999999999999e-07 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.411160+0000", 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061733000000000003 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.411172+0000", 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2153e-05 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511308+0000", 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10013572900000001 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511607+0000", 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029879100000000002 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511647+0000", 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0241999999999999e-05 2024-09-16T03:38:56.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.513870+0000", 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022230499999999999 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.513908+0000", 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8349999999999997e-05 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2152 4.11 4:89839b7e:::smithi15736696-12:head [read 0~2487436,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.355070928000004, 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10362692900000001, 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2152 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511576+0000", 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:38:56.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511584+0000", 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5939999999999994e-06 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511612+0000", 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7855e-05 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.611754+0000", 2024-09-16T03:38:56.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100141641 2024-09-16T03:38:56.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613255+0000", 2024-09-16T03:38:56.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001500744 2024-09-16T03:38:56.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613366+0000", 2024-09-16T03:38:56.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000111088 2024-09-16T03:38:56.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.615201+0000", 2024-09-16T03:38:56.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0018353009999999999 2024-09-16T03:38:56.927 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.927 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.927 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.927 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2316 4.11 4:89839b7e:::smithi15736696-12:head [read 0~2487436,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:38:56.927 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:11.770262+0000", 2024-09-16T03:38:56.927 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 45.096382624, 2024-09-16T03:38:56.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10250772399999999, 2024-09-16T03:38:56.927 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.927 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.928 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:56.928 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:56.928 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:56.928 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:56.928 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2316 2024-09-16T03:38:56.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.928 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770262+0000", 2024-09-16T03:38:56.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770262+0000", 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770265+0000", 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8930000000000001e-06 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770275+0000", 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3810000000000005e-06 2024-09-16T03:38:56.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.770287+0000", 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2889e-05 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.870382+0000", 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100094409 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.870584+0000", 2024-09-16T03:38:56.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020166199999999999 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.870806+0000", 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022220200000000001 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:11.872770+0000", 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019642879999999998 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2445 4.11 4:89839b7e:::smithi15736696-12:head [sparse-read 1818507~599510,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 41.422481505999997, 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200613392, 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2445 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444167+0000", 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6650000000000001e-06 2024-09-16T03:38:56.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444182+0000", 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4678000000000001e-05 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444207+0000", 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4768000000000001e-05 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.544359+0000", 2024-09-16T03:38:56.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100152804 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644561+0000", 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10020214500000001 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644624+0000", 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2444e-05 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644777+0000", 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000152888 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2629 3.3 e89/31 3:e23364e2:::e447860b11e9f9fde68b75cb1d2f204a74952889ec51a2417ef18c1427ff9a70:head v 89'173, mlcod=87'171)", 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 36.414738813, 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10848095100000001, 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.936 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451907+0000", 2024-09-16T03:38:56.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6599999999999996e-07 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456443+0000", 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0045365509999999998 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456470+0000", 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6897e-05 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456496+0000", 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.618e-05 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556666+0000", 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100169678 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556686+0000", 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9740000000000001e-05 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.939 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560345+0000", 2024-09-16T03:38:56.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003658821 2024-09-16T03:38:56.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560387+0000", 2024-09-16T03:38:56.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2518000000000002e-05 2024-09-16T03:38:56.940 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.940 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.940 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.940 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:209 3.3 e99/31 3:f2e0c7d0:::6bf49184c86dbc717b1f919afcf6cb40a4ca9c2465eb100f7ba59687540c40ed:head v 99'201, mlcod=98'194)", 2024-09-16T03:38:56.941 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:38:56.941 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 24.660608616000001, 2024-09-16T03:38:56.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200559603, 2024-09-16T03:38:56.941 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.941 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.941 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.941 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:38:56.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206037+0000", 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9999999999999999e-07 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206039+0000", 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2579999999999998e-06 2024-09-16T03:38:56.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206041+0000", 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.542e-06 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306162+0000", 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100120348 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306250+0000", 2024-09-16T03:38:56.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306268+0000", 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8127000000000001e-05 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.406549+0000", 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10028121199999999 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.944 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.406596+0000", 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6700999999999999e-05 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3274 4.11 4:89839b7e:::smithi15736696-12:head [write 1813152~490434] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.668845446000001, 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.28154463400000002, 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.945 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3274 2024-09-16T03:38:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:38:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197801+0000", 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9539999999999998e-06 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.198205+0000", 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00040328199999999998 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.198218+0000", 2024-09-16T03:38:56.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3176000000000001e-05 2024-09-16T03:38:56.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298318+0000", 2024-09-16T03:38:56.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100100562 2024-09-16T03:38:56.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298419+0000", 2024-09-16T03:38:56.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100667 2024-09-16T03:38:56.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298489+0000", 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0023000000000004e-05 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,12", 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298583+0000", 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3634000000000005e-05 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300283+0000", 2024-09-16T03:38:56.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00170049 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.307072+0000", 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0067889209999999998 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479243+0000", 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.17217031799999999 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479300+0000", 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7837999999999997e-05 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479344+0000", 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3769000000000001e-05 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3275 4.11 4:89839b7e:::smithi15736696-12:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:38:56.951 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.568300033, 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.276974636, 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3275 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:38:56.952 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298364+0000", 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9273e-05 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298373+0000", 2024-09-16T03:38:56.953 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5809999999999997e-06 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298382+0000", 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6269999999999995e-06 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298390+0000", 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2259999999999999e-06 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299770+0000", 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013803159999999999 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299852+0000", 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.174e-05 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,12", 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299948+0000", 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6669000000000006e-05 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:38:56.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300496+0000", 2024-09-16T03:38:56.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054762799999999998 2024-09-16T03:38:56.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:56.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.401930+0000", 2024-09-16T03:38:56.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101434495 2024-09-16T03:38:56.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:38:56.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575189+0000", 2024-09-16T03:38:56.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.173258996 2024-09-16T03:38:56.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575247+0000", 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7651000000000001e-05 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575320+0000", 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2434000000000005e-05 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3276 4.11 4:89839b7e:::smithi15736696-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 15.568231221, 2024-09-16T03:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.37724456499999998, 2024-09-16T03:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3276 2024-09-16T03:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.958 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298415+0000", 2024-09-16T03:38:56.959 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:38:56.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.960 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298418+0000", 2024-09-16T03:38:56.960 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.766e-06 2024-09-16T03:38:56.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.960 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298426+0000", 2024-09-16T03:38:56.960 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1359999999999999e-06 2024-09-16T03:38:56.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298433+0000", 2024-09-16T03:38:56.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0899999999999999e-06 2024-09-16T03:38:56.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300168+0000", 2024-09-16T03:38:56.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017353869999999999 2024-09-16T03:38:56.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:38:56.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300215+0000", 2024-09-16T03:38:56.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6702000000000001e-05 2024-09-16T03:38:56.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675524+0000", 2024-09-16T03:38:56.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.375309585 2024-09-16T03:38:56.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675596+0000", 2024-09-16T03:38:56.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1921000000000005e-05 2024-09-16T03:38:56.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675658+0000", 2024-09-16T03:38:56.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2052999999999997e-05 2024-09-16T03:38:56.963 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.963 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.963 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.963 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDPGPush(4.16 112/111 [PushOp(4:6df3bbbe:::smithi15736696-13:head, version: 108'72, data_included: [1603978~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:6df3bbbe:::smithi15736696-13:head@108'72, size: 3109328, copy_subset: [(0, 3109328)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2652554, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1603978, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-16T03:38:56.963 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:46.973785+0000", 2024-09-16T03:38:56.963 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.892859477, 2024-09-16T03:38:56.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102422003, 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:46.973785+0000", 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:46.973785+0000", 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:38:56.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.965 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:38:56.965 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:46.973787+0000", 2024-09-16T03:38:56.965 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.906e-06 2024-09-16T03:38:56.965 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.965 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:38:56.965 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:46.974787+0000", 2024-09-16T03:38:56.965 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00099961499999999992 2024-09-16T03:38:56.965 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.965 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:38:56.965 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:46.974807+0000", 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0369000000000001e-05 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:46.974819+0000", 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1507e-05 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:47.074960+0000", 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100141015 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:38:56.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:47.074970+0000", 2024-09-16T03:38:56.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0037000000000001e-05 2024-09-16T03:38:56.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:38:56.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:38:56.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:38:56.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:47.076207+0000", 2024-09-16T03:38:56.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0012375540000000001 2024-09-16T03:38:56.967 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.967 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.967 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.967 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:38:56.967 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:38:56.967 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:56.968 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:56.969 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:38:56.970 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:38:56.968+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:38:56.997 INFO:tasks.rados.rados.0.smithi157.stdout:1591: expect (ObjNum 270 snap 0 seq_num 270) 2024-09-16T03:38:56.998 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:57.001 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:38:57.069 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:57.068+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:57.112 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:57.112 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:57.113 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:38:57.113 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:57.121 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:38:57.126 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:57.127 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:57.128 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2502060 2024-09-16T03:38:57.132 INFO:tasks.rados.rados.0.smithi157.stdout:1592: got (0) Success 2024-09-16T03:38:57.132 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 62 (ObjNum 328 snap 0 seq_num 328) dirty exists 2024-09-16T03:38:57.133 INFO:tasks.rados.rados.0.smithi157.stdout:1593: finishing set_chunk to oid 34 2024-09-16T03:38:57.133 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 34 set_chunk returned error code -95 offset: 1747456 length: 1022976 2024-09-16T03:38:57.133 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 0 (ObjNum 303 snap 0 seq_num 303) dirty exists 2024-09-16T03:38:57.133 INFO:tasks.rados.rados.0.smithi157.stdout:1590: done (9 left) 2024-09-16T03:38:57.133 INFO:tasks.rados.rados.0.smithi157.stdout:1591: done (8 left) 2024-09-16T03:38:57.133 INFO:tasks.rados.rados.0.smithi157.stdout:1592: done (7 left) 2024-09-16T03:38:57.133 INFO:tasks.rados.rados.0.smithi157.stdout:1593: done (6 left) 2024-09-16T03:38:57.133 INFO:tasks.rados.rados.0.smithi157.stdout:1599: set_chunk oid 13 target oid 2024-09-16T03:38:57.144 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:57.145 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:57.162 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:57.162 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:57.170 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:57.169+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:57.178 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:57.178 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:57.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:57.194 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:57.211 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:57.211 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:57.226 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:38:57.227 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:38:57.227 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:38:57.227 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:38:57.227 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:38:57.229 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:57.230 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:57.236 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:38:57.246 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:57.246 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:57.262 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:57.262 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:57.271 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:57.270+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:57.279 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:57.279 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:57.296 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:57.296 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:57.312 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:57.313 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:57.329 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:57.329 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:57.336 INFO:tasks.rados.rados.0.smithi157.stdout:1599: set_chunk oid 13 offset: 907264 length: 693248 target oid fa4d51829b626c1d967af933073bb2ae733ab2be6a82b2fb4183d28c22e9c453 offset: 0 snap -1 2024-09-16T03:38:57.336 INFO:tasks.rados.rados.0.smithi157.stdout:1600: write (excl) oid 21 current snap is 0 2024-09-16T03:38:57.337 INFO:tasks.rados.rados.0.smithi157.stdout:1596: finishing write tid 1 to smithi15736696-20 2024-09-16T03:38:57.337 INFO:tasks.rados.rados.0.smithi157.stdout:1588: finishing set_chunk to oid 4 2024-09-16T03:38:57.337 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 115 (ObjNum 296 snap 0 seq_num 296) dirty exists 2024-09-16T03:38:57.337 INFO:tasks.rados.rados.0.smithi157.stdout:1595: expect (ObjNum 336 snap 0 seq_num 336) 2024-09-16T03:38:57.346 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:57.346 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:57.363 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:57.363 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:57.372 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:38:57.369+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:38:57.380 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:57.380 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:57.381 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:57.381 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:57.389 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:38:57.394 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:57.471 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T03:38:57.473 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:57.472+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:57.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T03:38:57.496 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:57.496 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:57.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:38:57.505 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:38:57.506 INFO:tasks.rados.rados.0.smithi157.stdout:1596: finishing write tid 2 to smithi15736696-20 2024-09-16T03:38:57.506 INFO:tasks.rados.rados.0.smithi157.stdout:1596: finishing write tid 3 to smithi15736696-20 2024-09-16T03:38:57.506 INFO:tasks.rados.rados.0.smithi157.stdout:1596: finishing write tid 4 to smithi15736696-20 2024-09-16T03:38:57.506 INFO:tasks.rados.rados.0.smithi157.stdout:1596: finishing write tid 5 to smithi15736696-20 2024-09-16T03:38:57.506 INFO:tasks.rados.rados.0.smithi157.stdout:1596: oid 20 updating version 0 to 45 2024-09-16T03:38:57.506 INFO:tasks.rados.rados.0.smithi157.stdout:1596: oid 20 updating version 45 to 46 2024-09-16T03:38:57.506 INFO:tasks.rados.rados.0.smithi157.stdout:1596: oid 20 updating version 46 to 47 2024-09-16T03:38:57.506 INFO:tasks.rados.rados.0.smithi157.stdout:1596: oid 20 updating version 47 to 48 2024-09-16T03:38:57.506 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 48 (ObjNum 340 snap 0 seq_num 340) dirty exists 2024-09-16T03:38:57.506 INFO:tasks.rados.rados.0.smithi157.stdout:1596: left oid 20 (ObjNum 340 snap 0 seq_num 340) 2024-09-16T03:38:57.506 INFO:tasks.rados.rados.0.smithi157.stdout:1600: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-09-16T03:38:57.511 INFO:tasks.rados.rados.0.smithi157.stdout:1600: writing smithi15736696-21 from 531187 to 1310628 tid 1 2024-09-16T03:38:57.520 INFO:tasks.rados.rados.0.smithi157.stdout:1600: writing smithi15736696-21 from 2088242 to 2826688 tid 2 2024-09-16T03:38:57.525 INFO:tasks.rados.rados.0.smithi157.stdout:1600: writing smithi15736696-21 from 3431711 to 3731187 tid 3 2024-09-16T03:38:57.525 INFO:tasks.rados.rados.0.smithi157.stdout:1588: done (7 left) 2024-09-16T03:38:57.525 INFO:tasks.rados.rados.0.smithi157.stdout:1595: done (6 left) 2024-09-16T03:38:57.525 INFO:tasks.rados.rados.0.smithi157.stdout:1596: done (5 left) 2024-09-16T03:38:57.526 INFO:tasks.rados.rados.0.smithi157.stdout:1597: done (4 left) 2024-09-16T03:38:57.526 INFO:tasks.rados.rados.0.smithi157.stdout:1601: write oid 34 current snap is 0 2024-09-16T03:38:57.526 INFO:tasks.rados.rados.0.smithi157.stdout:1594: finishing set_chunk to oid 7 2024-09-16T03:38:57.526 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 76 (ObjNum 311 snap 0 seq_num 311) dirty exists 2024-09-16T03:38:57.526 INFO:tasks.rados.rados.0.smithi157.stdout:1598: got (0) Success 2024-09-16T03:38:57.526 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 123 (ObjNum 318 snap 0 seq_num 318) dirty exists 2024-09-16T03:38:57.526 INFO:tasks.rados.rados.0.smithi157.stdout:1599: finishing set_chunk to oid 13 2024-09-16T03:38:57.526 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 13 set_chunk returned error code -95 offset: 907264 length: 693248 2024-09-16T03:38:57.526 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 0 (ObjNum 304 snap 0 seq_num 304) dirty exists 2024-09-16T03:38:57.526 INFO:tasks.rados.rados.0.smithi157.stdout:1601: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-09-16T03:38:57.530 INFO:tasks.rados.rados.0.smithi157.stdout:1601: writing smithi15736696-34 from 547994 to 1002684 tid 1 2024-09-16T03:38:57.536 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:57.535+0000 7f3a1f71e640 -1 osd.6 122 handle_osd_map injecting map crc failure 2024-09-16T03:38:57.537 INFO:tasks.rados.rados.0.smithi157.stdout:1601: writing smithi15736696-34 from 1746724 to 2228828 tid 2 2024-09-16T03:38:57.541 INFO:tasks.rados.rados.0.smithi157.stdout:1601: writing smithi15736696-34 from 2659134 to 3124280 tid 3 2024-09-16T03:38:57.545 INFO:tasks.rados.rados.0.smithi157.stdout:1601: writing smithi15736696-34 from 3605976 to 3747994 tid 4 2024-09-16T03:38:57.545 INFO:tasks.rados.rados.0.smithi157.stdout:1594: done (4 left) 2024-09-16T03:38:57.545 INFO:tasks.rados.rados.0.smithi157.stdout:1598: done (3 left) 2024-09-16T03:38:57.545 INFO:tasks.rados.rados.0.smithi157.stdout:1599: done (2 left) 2024-09-16T03:38:57.545 INFO:tasks.rados.rados.0.smithi157.stdout:1602: read oid 47 snap -1 2024-09-16T03:38:57.545 INFO:tasks.rados.rados.0.smithi157.stdout:1602: expect (ObjNum 318 snap 0 seq_num 318) 2024-09-16T03:38:57.545 INFO:tasks.rados.rados.0.smithi157.stdout:1603: delete oid 9 current snap is 0 2024-09-16T03:38:57.549 INFO:tasks.rados.rados.0.smithi157.stdout:1602: expect (ObjNum 318 snap 0 seq_num 318) 2024-09-16T03:38:57.575 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:57.571+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:57.611 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:57.612 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:57.620 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:38:57.637 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:57.673 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:57.671+0000 7f3d40708640 -1 osd.12 122 handle_osd_map injecting map crc failure 2024-09-16T03:38:57.675 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:57.673+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:57.724 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:57.765 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:57.765 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:57.774 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:38:57.776 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:57.774+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:57.813 INFO:tasks.rados.rados.0.smithi157.stdout:1600: finishing write tid 1 to smithi15736696-21 2024-09-16T03:38:57.813 INFO:tasks.rados.rados.0.smithi157.stdout:1600: finishing write tid 2 to smithi15736696-21 2024-09-16T03:38:57.813 INFO:tasks.rados.rados.0.smithi157.stdout:1600: finishing write tid 3 to smithi15736696-21 2024-09-16T03:38:57.813 INFO:tasks.rados.rados.0.smithi157.stdout:1601: finishing write tid 1 to smithi15736696-34 2024-09-16T03:38:57.813 INFO:tasks.rados.rados.0.smithi157.stdout:1602: done (3 left) 2024-09-16T03:38:57.813 INFO:tasks.rados.rados.0.smithi157.stdout:1603: done (2 left) 2024-09-16T03:38:57.814 INFO:tasks.rados.rados.0.smithi157.stdout:1604: write oid 29 current snap is 0 2024-09-16T03:38:57.814 INFO:tasks.rados.rados.0.smithi157.stdout:1604: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-09-16T03:38:57.819 INFO:tasks.rados.rados.0.smithi157.stdout:1604: writing smithi15736696-29 from 581608 to 1303149 tid 1 2024-09-16T03:38:57.825 INFO:tasks.rados.rados.0.smithi157.stdout:1604: writing smithi15736696-29 from 1863688 to 2349461 tid 2 2024-09-16T03:38:57.833 INFO:tasks.rados.rados.0.smithi157.stdout:1604: writing smithi15736696-29 from 3113980 to 3718023 tid 3 2024-09-16T03:38:57.833 INFO:tasks.rados.rados.0.smithi157.stdout:1605: read oid 7 snap -1 2024-09-16T03:38:57.833 INFO:tasks.rados.rados.0.smithi157.stdout:1605: expect (ObjNum 311 snap 0 seq_num 311) 2024-09-16T03:38:57.833 INFO:tasks.rados.rados.0.smithi157.stdout:1606: read oid 48 snap -1 2024-09-16T03:38:57.833 INFO:tasks.rados.rados.0.smithi157.stdout:1606: expect deleted 2024-09-16T03:38:57.834 INFO:tasks.rados.rados.0.smithi157.stdout:1607: read oid 31 snap -1 2024-09-16T03:38:57.834 INFO:tasks.rados.rados.0.smithi157.stdout:1607: expect (ObjNum 314 snap 0 seq_num 314) 2024-09-16T03:38:57.834 INFO:tasks.rados.rados.0.smithi157.stdout:1608: write (excl) oid 3 current snap is 0 2024-09-16T03:38:57.834 INFO:tasks.rados.rados.0.smithi157.stdout:1608: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-09-16T03:38:57.840 INFO:tasks.rados.rados.0.smithi157.stdout:1608: writing smithi15736696-3 from 598415 to 1395205 tid 1 2024-09-16T03:38:57.846 INFO:tasks.rados.rados.0.smithi157.stdout:1608: writing smithi15736696-3 from 1922170 to 2667954 tid 2 2024-09-16T03:38:57.851 INFO:tasks.rados.rados.0.smithi157.stdout:1608: writing smithi15736696-3 from 3141403 to 3556718 tid 3 2024-09-16T03:38:57.852 INFO:tasks.rados.rados.0.smithi157.stdout:1609: write (excl) oid 20 current snap is 0 2024-09-16T03:38:57.852 INFO:tasks.rados.rados.0.smithi157.stdout:1609: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-09-16T03:38:57.857 INFO:tasks.rados.rados.0.smithi157.stdout:1609: writing smithi15736696-20 from 615222 to 1087261 tid 1 2024-09-16T03:38:57.862 INFO:tasks.rados.rados.0.smithi157.stdout:1609: writing smithi15736696-20 from 1580652 to 2070094 tid 2 2024-09-16T03:38:57.869 INFO:tasks.rados.rados.0.smithi157.stdout:1609: writing smithi15736696-20 from 2768826 to 3395413 tid 3 2024-09-16T03:38:57.870 INFO:tasks.rados.rados.0.smithi157.stdout:1610: tier_evict oid 25 2024-09-16T03:38:57.870 INFO:tasks.rados.rados.0.smithi157.stdout:1610: tier_evict oid 25 snap -1 2024-09-16T03:38:57.870 INFO:tasks.rados.rados.0.smithi157.stdout:1611: set_chunk oid 47 target oid 2024-09-16T03:38:57.876 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:57.875+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:57.878 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:57.878 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:57.887 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:38:57.978 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:57.975+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:57.994 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:57.994 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:58.003 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:38:58.079 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:58.078+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:58.106 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:58.106 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:58.114 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:38:58.180 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:58.179+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:58.186 INFO:tasks.rados.rados.0.smithi157.stdout:1611: set_chunk oid 47 offset: 436224 length: 1513984 target oid 28a8e721c5022e2be3053b23881b028be8e46daa6b34efd87743dc4ed4a463ab offset: 0 snap -1 2024-09-16T03:38:58.186 INFO:tasks.rados.rados.0.smithi157.stdout:1612: write oid 16 current snap is 0 2024-09-16T03:38:58.186 INFO:tasks.rados.rados.0.smithi157.stdout:1601: finishing write tid 2 to smithi15736696-34 2024-09-16T03:38:58.186 INFO:tasks.rados.rados.0.smithi157.stdout:1612: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-09-16T03:38:58.193 INFO:tasks.rados.rados.0.smithi157.stdout:1612: writing smithi15736696-16 from 632029 to 1179317 tid 1 2024-09-16T03:38:58.201 INFO:tasks.rados.rados.0.smithi157.stdout:1612: writing smithi15736696-16 from 1639134 to 2388587 tid 2 2024-09-16T03:38:58.206 INFO:tasks.rados.rados.0.smithi157.stdout:1612: writing smithi15736696-16 from 2796249 to 3234108 tid 3 2024-09-16T03:38:58.211 INFO:tasks.rados.rados.0.smithi157.stdout:1612: writing smithi15736696-16 from 3653524 to 3832029 tid 4 2024-09-16T03:38:58.211 INFO:tasks.rados.rados.0.smithi157.stdout:1613: tier_evict oid 17 2024-09-16T03:38:58.211 INFO:tasks.rados.rados.0.smithi157.stdout:1613: tier_evict oid 17 snap -1 2024-09-16T03:38:58.211 INFO:tasks.rados.rados.0.smithi157.stdout:1610: got (22) Invalid argument 2024-09-16T03:38:58.211 INFO:tasks.rados.rados.0.smithi157.stdout:1600: finishing write tid 4 to smithi15736696-21 2024-09-16T03:38:58.211 INFO:tasks.rados.rados.0.smithi157.stdout:1610: done (11 left) 2024-09-16T03:38:58.211 INFO:tasks.rados.rados.0.smithi157.stdout:1614: tier_evict oid 27 2024-09-16T03:38:58.211 INFO:tasks.rados.rados.0.smithi157.stdout:1614: tier_evict oid 27 snap -1 2024-09-16T03:38:58.212 INFO:tasks.rados.rados.0.smithi157.stdout:1607: expect (ObjNum 314 snap 0 seq_num 314) 2024-09-16T03:38:58.212 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:58.229 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:58.229 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:58.239 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:38:58.281 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:58.279+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:58.344 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:58.344 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:58.353 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:38:58.381 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:38:58.380+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:38:58.442 INFO:tasks.rados.rados.0.smithi157.stdout:1600: finishing write tid 5 to smithi15736696-21 2024-09-16T03:38:58.442 INFO:tasks.rados.rados.0.smithi157.stdout:1600: oid 21 updating version 0 to 62 2024-09-16T03:38:58.442 INFO:tasks.rados.rados.0.smithi157.stdout:1600: oid 21 version 62 is already newer than 60 2024-09-16T03:38:58.442 INFO:tasks.rados.rados.0.smithi157.stdout:1600: oid 21 version 62 is already newer than 59 2024-09-16T03:38:58.442 INFO:tasks.rados.rados.0.smithi157.stdout:1600: oid 21 version 62 is already newer than 61 2024-09-16T03:38:58.442 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 62 (ObjNum 341 snap 0 seq_num 341) dirty exists 2024-09-16T03:38:58.443 INFO:tasks.rados.rados.0.smithi157.stdout:1600: left oid 21 (ObjNum 341 snap 0 seq_num 341) 2024-09-16T03:38:58.443 INFO:tasks.rados.rados.0.smithi157.stdout:1605: expect (ObjNum 311 snap 0 seq_num 311) 2024-09-16T03:38:58.460 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:58.461 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:58.467 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:38:58.481 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:38:58.482 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T03:38:58.482 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:58.479+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:58.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T03:38:58.573 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:58.573 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:58.575 INFO:tasks.rados.rados.0.smithi157.stdout:1609: finishing write tid 1 to smithi15736696-20 2024-09-16T03:38:58.575 INFO:tasks.rados.rados.0.smithi157.stdout:1601: finishing write tid 3 to smithi15736696-34 2024-09-16T03:38:58.576 INFO:tasks.rados.rados.0.smithi157.stdout:1601: finishing write tid 4 to smithi15736696-34 2024-09-16T03:38:58.576 INFO:tasks.rados.rados.0.smithi157.stdout:1601: finishing write tid 5 to smithi15736696-34 2024-09-16T03:38:58.576 INFO:tasks.rados.rados.0.smithi157.stdout:1601: finishing write tid 6 to smithi15736696-34 2024-09-16T03:38:58.576 INFO:tasks.rados.rados.0.smithi157.stdout:1601: oid 34 updating version 0 to 143 2024-09-16T03:38:58.576 INFO:tasks.rados.rados.0.smithi157.stdout:1601: oid 34 updating version 143 to 145 2024-09-16T03:38:58.576 INFO:tasks.rados.rados.0.smithi157.stdout:1601: oid 34 version 145 is already newer than 142 2024-09-16T03:38:58.576 INFO:tasks.rados.rados.0.smithi157.stdout:1601: oid 34 version 145 is already newer than 144 2024-09-16T03:38:58.576 INFO:tasks.rados.rados.0.smithi157.stdout:1601: oid 34 updating version 145 to 146 2024-09-16T03:38:58.576 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 146 (ObjNum 342 snap 0 seq_num 342) dirty exists 2024-09-16T03:38:58.576 INFO:tasks.rados.rados.0.smithi157.stdout:1601: left oid 34 (ObjNum 342 snap 0 seq_num 342) 2024-09-16T03:38:58.576 INFO:tasks.rados.rados.0.smithi157.stdout:1608: finishing write tid 1 to smithi15736696-3 2024-09-16T03:38:58.576 INFO:tasks.rados.rados.0.smithi157.stdout:1608: finishing write tid 2 to smithi15736696-3 2024-09-16T03:38:58.576 INFO:tasks.rados.rados.0.smithi157.stdout:1608: finishing write tid 3 to smithi15736696-3 2024-09-16T03:38:58.577 INFO:tasks.rados.rados.0.smithi157.stdout:1608: finishing write tid 4 to smithi15736696-3 2024-09-16T03:38:58.577 INFO:tasks.rados.rados.0.smithi157.stdout:1608: finishing write tid 5 to smithi15736696-3 2024-09-16T03:38:58.577 INFO:tasks.rados.rados.0.smithi157.stdout:1608: oid 3 updating version 0 to 67 2024-09-16T03:38:58.577 INFO:tasks.rados.rados.0.smithi157.stdout:1608: oid 3 version 67 is already newer than 65 2024-09-16T03:38:58.577 INFO:tasks.rados.rados.0.smithi157.stdout:1608: oid 3 version 67 is already newer than 64 2024-09-16T03:38:58.577 INFO:tasks.rados.rados.0.smithi157.stdout:1608: oid 3 version 67 is already newer than 66 2024-09-16T03:38:58.577 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 67 (ObjNum 345 snap 0 seq_num 345) dirty exists 2024-09-16T03:38:58.577 INFO:tasks.rados.rados.0.smithi157.stdout:1608: left oid 3 (ObjNum 345 snap 0 seq_num 345) 2024-09-16T03:38:58.577 INFO:tasks.rados.rados.0.smithi157.stdout:1600: done (11 left) 2024-09-16T03:38:58.577 INFO:tasks.rados.rados.0.smithi157.stdout:1601: done (10 left) 2024-09-16T03:38:58.577 INFO:tasks.rados.rados.0.smithi157.stdout:1605: done (9 left) 2024-09-16T03:38:58.577 INFO:tasks.rados.rados.0.smithi157.stdout:1606: done (8 left) 2024-09-16T03:38:58.578 INFO:tasks.rados.rados.0.smithi157.stdout:1607: done (7 left) 2024-09-16T03:38:58.578 INFO:tasks.rados.rados.0.smithi157.stdout:1608: done (6 left) 2024-09-16T03:38:58.578 INFO:tasks.rados.rados.0.smithi157.stdout:1615: write (excl) oid 13 current snap is 0 2024-09-16T03:38:58.578 INFO:tasks.rados.rados.0.smithi157.stdout:1615: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-09-16T03:38:58.582 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:38:58.583 INFO:tasks.rados.rados.0.smithi157.stdout:1615: writing smithi15736696-13 from 648836 to 1271373 tid 1 2024-09-16T03:38:58.583 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:38:58.581+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:38:58.589 INFO:tasks.rados.rados.0.smithi157.stdout:1615: writing smithi15736696-13 from 1981263 to 2474374 tid 2 2024-09-16T03:38:58.597 INFO:tasks.rados.rados.0.smithi157.stdout:1615: writing smithi15736696-13 from 3107319 to 3848836 tid 3 2024-09-16T03:38:58.597 INFO:tasks.rados.rados.0.smithi157.stdout:1616: read oid 6 snap -1 2024-09-16T03:38:58.597 INFO:tasks.rados.rados.0.smithi157.stdout:1616: expect (ObjNum 327 snap 0 seq_num 327) 2024-09-16T03:38:58.597 INFO:tasks.rados.rados.0.smithi157.stdout:1617: read oid 2 snap -1 2024-09-16T03:38:58.597 INFO:tasks.rados.rados.0.smithi157.stdout:1617: expect deleted 2024-09-16T03:38:58.597 INFO:tasks.rados.rados.0.smithi157.stdout:1618: set_chunk oid 15 target oid 2024-09-16T03:38:58.684 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:58.682+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:58.687 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:58.687 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:58.697 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:38:58.786 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:38:58.783+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:38:58.843 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:58.844 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:58.852 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:38:58.887 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:58.885+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:58.987 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:38:58.986+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:38:58.991 INFO:tasks.rados.rados.0.smithi157.stdout:1618: set_chunk oid 15 offset: 745984 length: 1016832 target oid e25bb306aca5cb932e3a257d165374c9c66a0b070e8b18ed29e68e73d7687080 offset: 0 snap -1 2024-09-16T03:38:58.991 INFO:tasks.rados.rados.0.smithi157.stdout:1619: write (excl) oid 48 current snap is 0 2024-09-16T03:38:58.991 INFO:tasks.rados.rados.0.smithi157.stdout:1612: finishing write tid 1 to smithi15736696-16 2024-09-16T03:38:58.991 INFO:tasks.rados.rados.0.smithi157.stdout:1614: got (22) Invalid argument 2024-09-16T03:38:58.991 INFO:tasks.rados.rados.0.smithi157.stdout:1609: finishing write tid 2 to smithi15736696-20 2024-09-16T03:38:58.991 INFO:tasks.rados.rados.0.smithi157.stdout:1619: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2024-09-16T03:38:58.995 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:58.995 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:58.998 INFO:tasks.rados.rados.0.smithi157.stdout:1619: writing smithi15736696-48 from 665643 to 1363429 tid 1 2024-09-16T03:38:59.005 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:38:59.007 INFO:tasks.rados.rados.0.smithi157.stdout:1619: writing smithi15736696-48 from 2039745 to 2792867 tid 2 2024-09-16T03:38:59.013 INFO:tasks.rados.rados.0.smithi157.stdout:1619: writing smithi15736696-48 from 3534742 to 3865643 tid 3 2024-09-16T03:38:59.013 INFO:tasks.rados.rados.0.smithi157.stdout:1614: done (10 left) 2024-09-16T03:38:59.013 INFO:tasks.rados.rados.0.smithi157.stdout:1620: tier_evict oid 43 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:1620: tier_evict oid 43 snap -1 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:1609: finishing write tid 3 to smithi15736696-20 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:1604: finishing write tid 1 to smithi15736696-29 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:1604: finishing write tid 2 to smithi15736696-29 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:1604: finishing write tid 3 to smithi15736696-29 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:1609: finishing write tid 4 to smithi15736696-20 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:1621: write oid 50 current snap is 0 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:1609: finishing write tid 5 to smithi15736696-20 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:1609: oid 20 updating version 0 to 51 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:1609: oid 20 version 51 is already newer than 49 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:1609: oid 20 version 51 is already newer than 50 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:1609: oid 20 updating version 51 to 52 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 52 (ObjNum 346 snap 0 seq_num 346) dirty exists 2024-09-16T03:38:59.014 INFO:tasks.rados.rados.0.smithi157.stdout:1609: left oid 20 (ObjNum 346 snap 0 seq_num 346) 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:1611: finishing set_chunk to oid 47 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 47 set_chunk returned error code -95 offset: 436224 length: 1513984 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 0 (ObjNum 318 snap 0 seq_num 318) dirty exists 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:1604: finishing write tid 4 to smithi15736696-29 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:1604: finishing write tid 5 to smithi15736696-29 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:1604: oid 29 updating version 0 to 127 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:1604: oid 29 version 127 is already newer than 126 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:1604: oid 29 version 127 is already newer than 124 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:1604: oid 29 version 127 is already newer than 125 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 127 (ObjNum 344 snap 0 seq_num 344) dirty exists 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:1604: left oid 29 (ObjNum 344 snap 0 seq_num 344) 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:1612: finishing write tid 2 to smithi15736696-16 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:1612: finishing write tid 3 to smithi15736696-16 2024-09-16T03:38:59.015 INFO:tasks.rados.rados.0.smithi157.stdout:1621: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2024-09-16T03:38:59.019 INFO:tasks.rados.rados.0.smithi157.stdout:1621: writing smithi15736696-50 from 682450 to 1171838 tid 1 2024-09-16T03:38:59.025 INFO:tasks.rados.rados.0.smithi157.stdout:1621: writing smithi15736696-50 from 1814580 to 2311360 tid 2 2024-09-16T03:38:59.034 INFO:tasks.rados.rados.0.smithi157.stdout:1621: writing smithi15736696-50 from 2878518 to 3666546 tid 3 2024-09-16T03:38:59.034 INFO:tasks.rados.rados.0.smithi157.stdout:1604: done (11 left) 2024-09-16T03:38:59.034 INFO:tasks.rados.rados.0.smithi157.stdout:1609: done (10 left) 2024-09-16T03:38:59.034 INFO:tasks.rados.rados.0.smithi157.stdout:1611: done (9 left) 2024-09-16T03:38:59.034 INFO:tasks.rados.rados.0.smithi157.stdout:1622: tier_evict oid 36 2024-09-16T03:38:59.034 INFO:tasks.rados.rados.0.smithi157.stdout:1622: tier_evict oid 36 snap -1 2024-09-16T03:38:59.034 INFO:tasks.rados.rados.0.smithi157.stdout:1612: finishing write tid 4 to smithi15736696-16 2024-09-16T03:38:59.034 INFO:tasks.rados.rados.0.smithi157.stdout:1613: got (22) Invalid argument 2024-09-16T03:38:59.034 INFO:tasks.rados.rados.0.smithi157.stdout:1612: finishing write tid 5 to smithi15736696-16 2024-09-16T03:38:59.035 INFO:tasks.rados.rados.0.smithi157.stdout:1612: finishing write tid 6 to smithi15736696-16 2024-09-16T03:38:59.035 INFO:tasks.rados.rados.0.smithi157.stdout:1612: oid 16 updating version 0 to 61 2024-09-16T03:38:59.035 INFO:tasks.rados.rados.0.smithi157.stdout:1612: oid 16 updating version 61 to 65 2024-09-16T03:38:59.035 INFO:tasks.rados.rados.0.smithi157.stdout:1612: oid 16 version 65 is already newer than 63 2024-09-16T03:38:59.035 INFO:tasks.rados.rados.0.smithi157.stdout:1612: oid 16 version 65 is already newer than 64 2024-09-16T03:38:59.035 INFO:tasks.rados.rados.0.smithi157.stdout:1612: oid 16 version 65 is already newer than 62 2024-09-16T03:38:59.035 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 65 (ObjNum 347 snap 0 seq_num 347) dirty exists 2024-09-16T03:38:59.035 INFO:tasks.rados.rados.0.smithi157.stdout:1612: left oid 16 (ObjNum 347 snap 0 seq_num 347) 2024-09-16T03:38:59.035 INFO:tasks.rados.rados.0.smithi157.stdout:1612: done (9 left) 2024-09-16T03:38:59.035 INFO:tasks.rados.rados.0.smithi157.stdout:1613: done (8 left) 2024-09-16T03:38:59.035 INFO:tasks.rados.rados.0.smithi157.stdout:1623: write oid 30 current snap is 0 2024-09-16T03:38:59.035 INFO:tasks.rados.rados.0.smithi157.stdout:1623: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-09-16T03:38:59.040 INFO:tasks.rados.rados.0.smithi157.stdout:1623: writing smithi15736696-30 from 699257 to 1263894 tid 1 2024-09-16T03:38:59.047 INFO:tasks.rados.rados.0.smithi157.stdout:1623: writing smithi15736696-30 from 1873062 to 2513500 tid 2 2024-09-16T03:38:59.055 INFO:tasks.rados.rados.0.smithi157.stdout:1623: writing smithi15736696-30 from 3305941 to 3899257 tid 3 2024-09-16T03:38:59.055 INFO:tasks.rados.rados.0.smithi157.stdout:1624: set_chunk oid 34 target oid 2024-09-16T03:38:59.089 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:38:59.088+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:38:59.111 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:59.112 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:59.120 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:38:59.205 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:59.202+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:38:59.266 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:59.266 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:59.275 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:38:59.291 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:38:59.290+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:38:59.334 INFO:tasks.rados.rados.0.smithi157.stdout:1624: set_chunk oid 34 offset: 1507840 length: 1096704 target oid 0c28b64cb98c4375b8285829cb22a51685910510d1ba74394a66883f781db201 offset: 0 snap -1 2024-09-16T03:38:59.334 INFO:tasks.rados.rados.0.smithi157.stdout:1625: read oid 19 snap -1 2024-09-16T03:38:59.334 INFO:tasks.rados.rados.0.smithi157.stdout:1625: expect (ObjNum 331 snap 0 seq_num 331) 2024-09-16T03:38:59.334 INFO:tasks.rados.rados.0.smithi157.stdout:1626: read oid 40 snap -1 2024-09-16T03:38:59.334 INFO:tasks.rados.rados.0.smithi157.stdout:1626: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:38:59.334 INFO:tasks.rados.rados.0.smithi157.stdout:1627: read oid 39 snap -1 2024-09-16T03:38:59.334 INFO:tasks.rados.rados.0.smithi157.stdout:1627: expect deleted 2024-09-16T03:38:59.334 INFO:tasks.rados.rados.0.smithi157.stdout:1628: read oid 31 snap -1 2024-09-16T03:38:59.334 INFO:tasks.rados.rados.0.smithi157.stdout:1628: expect (ObjNum 314 snap 0 seq_num 314) 2024-09-16T03:38:59.335 INFO:tasks.rados.rados.0.smithi157.stdout:1629: read oid 22 snap -1 2024-09-16T03:38:59.335 INFO:tasks.rados.rados.0.smithi157.stdout:1629: expect deleted 2024-09-16T03:38:59.335 INFO:tasks.rados.rados.0.smithi157.stdout:1630: write oid 3 current snap is 0 2024-09-16T03:38:59.335 INFO:tasks.rados.rados.0.smithi157.stdout:1630: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-09-16T03:38:59.340 INFO:tasks.rados.rados.0.smithi157.stdout:1630: writing smithi15736696-3 from 716064 to 1355950 tid 1 2024-09-16T03:38:59.345 INFO:tasks.rados.rados.0.smithi157.stdout:1630: writing smithi15736696-3 from 1815191 to 2315640 tid 2 2024-09-16T03:38:59.351 INFO:tasks.rados.rados.0.smithi157.stdout:1630: writing smithi15736696-3 from 2817011 to 3343936 tid 3 2024-09-16T03:38:59.351 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:59.351 INFO:tasks.rados.rados.0.smithi157.stdout:1616: expect (ObjNum 327 snap 0 seq_num 327) 2024-09-16T03:38:59.351 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:59.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:38:59.393 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:59.390+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:59.422 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:59.422 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:59.431 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:38:59.454 INFO:tasks.rados.rados.0.smithi157.stdout:1615: finishing write tid 1 to smithi15736696-13 2024-09-16T03:38:59.454 INFO:tasks.rados.rados.0.smithi157.stdout:1615: finishing write tid 2 to smithi15736696-13 2024-09-16T03:38:59.454 INFO:tasks.rados.rados.0.smithi157.stdout:1615: finishing write tid 3 to smithi15736696-13 2024-09-16T03:38:59.454 INFO:tasks.rados.rados.0.smithi157.stdout:1615: finishing write tid 4 to smithi15736696-13 2024-09-16T03:38:59.454 INFO:tasks.rados.rados.0.smithi157.stdout:1615: finishing write tid 5 to smithi15736696-13 2024-09-16T03:38:59.454 INFO:tasks.rados.rados.0.smithi157.stdout:1615: oid 13 updating version 0 to 81 2024-09-16T03:38:59.454 INFO:tasks.rados.rados.0.smithi157.stdout:1615: oid 13 version 81 is already newer than 79 2024-09-16T03:38:59.454 INFO:tasks.rados.rados.0.smithi157.stdout:1615: oid 13 updating version 81 to 82 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:1615: oid 13 version 82 is already newer than 80 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 82 (ObjNum 348 snap 0 seq_num 348) dirty exists 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:1615: left oid 13 (ObjNum 348 snap 0 seq_num 348) 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:1620: got (22) Invalid argument 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:1615: done (15 left) 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:1616: done (14 left) 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:1617: done (13 left) 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:1620: done (12 left) 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:1631: read oid 14 snap -1 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:1631: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:1632: read oid 32 snap -1 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:1632: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:1619: finishing write tid 1 to smithi15736696-48 2024-09-16T03:38:59.455 INFO:tasks.rados.rados.0.smithi157.stdout:1619: finishing write tid 2 to smithi15736696-48 2024-09-16T03:38:59.456 INFO:tasks.rados.rados.0.smithi157.stdout:1621: finishing write tid 1 to smithi15736696-50 2024-09-16T03:38:59.456 INFO:tasks.rados.rados.0.smithi157.stdout:1622: got (22) Invalid argument 2024-09-16T03:38:59.456 INFO:tasks.rados.rados.0.smithi157.stdout:1623: finishing write tid 1 to smithi15736696-30 2024-09-16T03:38:59.456 INFO:tasks.rados.rados.0.smithi157.stdout:1633: read oid 1 snap -1 2024-09-16T03:38:59.456 INFO:tasks.rados.rados.0.smithi157.stdout:1633: expect (ObjNum 242 snap 0 seq_num 242) 2024-09-16T03:38:59.456 INFO:tasks.rados.rados.0.smithi157.stdout:1622: done (14 left) 2024-09-16T03:38:59.456 INFO:tasks.rados.rados.0.smithi157.stdout:1634: read oid 21 snap -1 2024-09-16T03:38:59.456 INFO:tasks.rados.rados.0.smithi157.stdout:1634: expect (ObjNum 341 snap 0 seq_num 341) 2024-09-16T03:38:59.456 INFO:tasks.rados.rados.0.smithi157.stdout:1635: set_chunk oid 12 target oid 2024-09-16T03:38:59.490 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T03:38:59.491 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:38:59.488+0000 7f49439d4640 -1 osd.4 124 handle_osd_map injecting map crc failure 2024-09-16T03:38:59.492 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:38:59.490+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:38:59.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T03:38:59.522 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:38:59.522 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:38:59.541 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:38:59.541 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:38:59.558 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:38:59.558 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:38:59.558 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:59.559 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:59.567 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:38:59.575 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:38:59.575 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:38:59.591 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:38:59.591 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:38:59.593 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:38:59.592+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:38:59.608 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:38:59.608 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:38:59.625 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:38:59.625 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:38:59.649 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:38:59.649 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:38:59.665 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:38:59.665 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:38:59.671 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:38:59.671 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:38:59.680 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:38:59.682 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:38:59.683 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:38:59.694 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:38:59.691+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:38:59.698 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:38:59.699 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:38:59.717 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:38:59.718 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:38:59.751 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:38:59.751 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:38:59.783 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:38:59.783 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:38:59.794 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:38:59.793+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:38:59.809 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:38:59.810 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:38:59.819 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:59.819 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:59.819 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:38:59.819 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:59.826 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:38:59.827 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:38:59.828 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:38:59.860 INFO:tasks.rados.rados.0.smithi157.stdout:1635: set_chunk oid 12 offset: 1724416 length: 742912 target oid 67f135cb5be89ce872b446934c8173bbba66056d48624f7fe26ad5697061c863 offset: 0 snap -1 2024-09-16T03:38:59.860 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:59.860 INFO:tasks.rados.rados.0.smithi157.stdout:1623: finishing write tid 2 to smithi15736696-30 2024-09-16T03:38:59.860 INFO:tasks.rados.rados.0.smithi157.stdout:1619: finishing write tid 3 to smithi15736696-48 2024-09-16T03:38:59.860 INFO:tasks.rados.rados.0.smithi157.stdout:1619: finishing write tid 4 to smithi15736696-48 2024-09-16T03:38:59.860 INFO:tasks.rados.rados.0.smithi157.stdout:1619: finishing write tid 5 to smithi15736696-48 2024-09-16T03:38:59.860 INFO:tasks.rados.rados.0.smithi157.stdout:1619: oid 48 updating version 0 to 84 2024-09-16T03:38:59.861 INFO:tasks.rados.rados.0.smithi157.stdout:1619: oid 48 version 84 is already newer than 83 2024-09-16T03:38:59.861 INFO:tasks.rados.rados.0.smithi157.stdout:1619: oid 48 updating version 84 to 85 2024-09-16T03:38:59.861 INFO:tasks.rados.rados.0.smithi157.stdout:1619: oid 48 updating version 85 to 86 2024-09-16T03:38:59.861 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 86 (ObjNum 349 snap 0 seq_num 349) dirty exists 2024-09-16T03:38:59.861 INFO:tasks.rados.rados.0.smithi157.stdout:1619: left oid 48 (ObjNum 349 snap 0 seq_num 349) 2024-09-16T03:38:59.861 INFO:tasks.rados.rados.0.smithi157.stdout:1621: finishing write tid 2 to smithi15736696-50 2024-09-16T03:38:59.861 INFO:tasks.rados.rados.0.smithi157.stdout:1619: done (15 left) 2024-09-16T03:38:59.861 INFO:tasks.rados.rados.0.smithi157.stdout:1636: read oid 28 snap -1 2024-09-16T03:38:59.861 INFO:tasks.rados.rados.0.smithi157.stdout:1636: expect (ObjNum 338 snap 0 seq_num 338) 2024-09-16T03:38:59.861 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:38:59.861 INFO:tasks.rados.rados.0.smithi157.stdout:1627: done (15 left) 2024-09-16T03:38:59.861 INFO:tasks.rados.rados.0.smithi157.stdout:1629: done (14 left) 2024-09-16T03:38:59.861 INFO:tasks.rados.rados.0.smithi157.stdout:1637: tier_evict oid 43 2024-09-16T03:38:59.862 INFO:tasks.rados.rados.0.smithi157.stdout:1637: tier_evict oid 43 snap -1 2024-09-16T03:38:59.862 INFO:tasks.rados.rados.0.smithi157.stdout:1623: finishing write tid 3 to smithi15736696-30 2024-09-16T03:38:59.862 INFO:tasks.rados.rados.0.smithi157.stdout:1623: finishing write tid 4 to smithi15736696-30 2024-09-16T03:38:59.862 INFO:tasks.rados.rados.0.smithi157.stdout:1623: finishing write tid 5 to smithi15736696-30 2024-09-16T03:38:59.862 INFO:tasks.rados.rados.0.smithi157.stdout:1623: oid 30 updating version 0 to 46 2024-09-16T03:38:59.862 INFO:tasks.rados.rados.0.smithi157.stdout:1623: oid 30 version 46 is already newer than 45 2024-09-16T03:38:59.862 INFO:tasks.rados.rados.0.smithi157.stdout:1623: oid 30 updating version 46 to 47 2024-09-16T03:38:59.862 INFO:tasks.rados.rados.0.smithi157.stdout:1623: oid 30 updating version 47 to 48 2024-09-16T03:38:59.862 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 48 (ObjNum 351 snap 0 seq_num 351) dirty exists 2024-09-16T03:38:59.862 INFO:tasks.rados.rados.0.smithi157.stdout:1623: left oid 30 (ObjNum 351 snap 0 seq_num 351) 2024-09-16T03:38:59.862 INFO:tasks.rados.rados.0.smithi157.stdout:1621: finishing write tid 3 to smithi15736696-50 2024-09-16T03:38:59.863 INFO:tasks.rados.rados.0.smithi157.stdout:1621: finishing write tid 4 to smithi15736696-50 2024-09-16T03:38:59.863 INFO:tasks.rados.rados.0.smithi157.stdout:1625: expect (ObjNum 331 snap 0 seq_num 331) 2024-09-16T03:38:59.895 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:38:59.893+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:38:59.934 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2389400 2024-09-16T03:38:59.953 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:38:59.987 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:38:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:38:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:38:59.987 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:38:59.987 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:38:59.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:38:59.997 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:38:59.993+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:00.007 INFO:tasks.rados.rados.0.smithi157.stdout:1634: expect (ObjNum 341 snap 0 seq_num 341) 2024-09-16T03:39:00.017 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3361409 2024-09-16T03:39:00.090 INFO:tasks.rados.rados.0.smithi157.stdout:1633: expect (ObjNum 242 snap 0 seq_num 242) 2024-09-16T03:39:00.090 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:00.097 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:00.095+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:00.105 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:00.105 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:00.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:00.106 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:00.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.106 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:39 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'21, mlcod=26'19)", 2024-09-16T03:39:00.106 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:39:00.106 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.507232543, 2024-09-16T03:39:00.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30197182500000003, 2024-09-16T03:39:00.106 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.106 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.106 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.106 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:39:00.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:39:00.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595320+0000", 2024-09-16T03:39:00.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.962e-06 2024-09-16T03:39:00.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595336+0000", 2024-09-16T03:39:00.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5568000000000001e-05 2024-09-16T03:39:00.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595349+0000", 2024-09-16T03:39:00.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3203e-05 2024-09-16T03:39:00.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695424+0000", 2024-09-16T03:39:00.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007542899999999 2024-09-16T03:39:00.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695495+0000", 2024-09-16T03:39:00.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0450999999999993e-05 2024-09-16T03:39:00.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695538+0000", 2024-09-16T03:39:00.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3433000000000002e-05 2024-09-16T03:39:00.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897256+0000", 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20171840699999999 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897290+0000", 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3371999999999998e-05 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.111 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1948 4.8 4:146f84f0:::smithi15736696-6:head [setxattr _header (55) in=7b,truncate 2756348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:00.111 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:39:00.111 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.977743736999997, 2024-09-16T03:39:00.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40326984799999999, 2024-09-16T03:39:00.111 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.111 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.111 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.111 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.111 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.111 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.111 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1948 2024-09-16T03:39:00.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.111 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:39:00.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:39:00.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124808+0000", 2024-09-16T03:39:00.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9099999999999991e-07 2024-09-16T03:39:00.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124813+0000", 2024-09-16T03:39:00.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9599999999999999e-06 2024-09-16T03:39:00.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124821+0000", 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0490000000000004e-06 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124827+0000", 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3339999999999998e-06 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126739+0000", 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00191147 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126830+0000", 2024-09-16T03:39:00.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1751999999999998e-05 2024-09-16T03:39:00.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:00.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127005+0000", 2024-09-16T03:39:00.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174227 2024-09-16T03:39:00.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127981+0000", 2024-09-16T03:39:00.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097622799999999999 2024-09-16T03:39:00.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.128193+0000", 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021176999999999999 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.135281+0000", 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070882150000000001 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.427785+0000", 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29250390599999998 2024-09-16T03:39:00.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527932+0000", 2024-09-16T03:39:00.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10014759500000001 2024-09-16T03:39:00.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527980+0000", 2024-09-16T03:39:00.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7725000000000001e-05 2024-09-16T03:39:00.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528077+0000", 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6626000000000006e-05 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1949 4.8 4:146f84f0:::smithi15736696-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.977712992999997, 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40346921899999999, 2024-09-16T03:39:00.119 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.120 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.120 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:00.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.120 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.120 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1949 2024-09-16T03:39:00.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.120 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:00.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:00.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:00.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:39:00.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124841+0000", 2024-09-16T03:39:00.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5890000000000001e-06 2024-09-16T03:39:00.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124847+0000", 2024-09-16T03:39:00.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0820000000000004e-06 2024-09-16T03:39:00.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124852+0000", 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1070000000000004e-06 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127536+0000", 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026844619999999999 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127585+0000", 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8915999999999997e-05 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528126+0000", 2024-09-16T03:39:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40054061899999999 2024-09-16T03:39:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528193+0000", 2024-09-16T03:39:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6717999999999994e-05 2024-09-16T03:39:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528307+0000", 2024-09-16T03:39:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011413600000000001 2024-09-16T03:39:00.124 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.125 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.125 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1979 4.18 4:1f63731d:::smithi15736696-30:head [write 789962~534747] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:00.125 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:39:00.125 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.466705681000001, 2024-09-16T03:39:00.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.506754182, 2024-09-16T03:39:00.125 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.125 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.125 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.125 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.125 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.126 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.126 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1979 2024-09-16T03:39:00.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.126 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:39:00.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:39:00.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635850+0000", 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6670000000000004e-06 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636429+0000", 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000579745 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636441+0000", 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1965e-05 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736594+0000", 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100152291 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837326+0000", 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10073201399999999 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837437+0000", 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111914 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837625+0000", 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000187391 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939449+0000", 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10182371599999999 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939476+0000", 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7117000000000001e-05 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142297+0000", 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20282091499999999 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142347+0000", 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0800000000000002e-05 2024-09-16T03:39:00.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142485+0000", 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013799999999999999 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142526+0000", 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0269999999999999e-05 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142599+0000", 2024-09-16T03:39:00.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3376999999999999e-05 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1980 4.18 4:1f63731d:::smithi15736696-30:head [write 1780952~690056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.365904520000001, 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40615011400000001, 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.132 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1980 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736656+0000", 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0339999999999999e-05 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737092+0000", 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435957 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737102+0000", 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7860000000000008e-06 2024-09-16T03:39:00.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737107+0000", 2024-09-16T03:39:00.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:39:00.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839282+0000", 2024-09-16T03:39:00.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10217523100000001 2024-09-16T03:39:00.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839389+0000", 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010619399999999999 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839567+0000", 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017886200000000001 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939485+0000", 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099917168000000001 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943655+0000", 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004170287 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142373+0000", 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19871839599999999 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142431+0000", 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7880999999999997e-05 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142627+0000", 2024-09-16T03:39:00.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019580500000000001 2024-09-16T03:39:00.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142723+0000", 2024-09-16T03:39:00.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6209e-05 2024-09-16T03:39:00.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142796+0000", 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2978000000000005e-05 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2277 4.18 4:1f63731d:::smithi15736696-30:head [write 460558~707131] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.67976882, 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39421539100000003, 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.139 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.140 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.140 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.140 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2277 2024-09-16T03:39:00.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.140 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:39:00.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:39:00.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422785+0000", 2024-09-16T03:39:00.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9809999999999999e-06 2024-09-16T03:39:00.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423586+0000", 2024-09-16T03:39:00.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080133600000000004 2024-09-16T03:39:00.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423607+0000", 2024-09-16T03:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1039e-05 2024-09-16T03:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423616+0000", 2024-09-16T03:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0189999999999995e-06 2024-09-16T03:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523776+0000", 2024-09-16T03:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015945399999999 2024-09-16T03:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523851+0000", 2024-09-16T03:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5549999999999993e-05 2024-09-16T03:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523972+0000", 2024-09-16T03:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120683 2024-09-16T03:39:00.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527436+0000", 2024-09-16T03:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034638870000000001 2024-09-16T03:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636588+0000", 2024-09-16T03:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109152601 2024-09-16T03:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636665+0000", 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6661000000000003e-05 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637128+0000", 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046316199999999998 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816886+0000", 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17975745200000001 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816927+0000", 2024-09-16T03:39:00.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1529000000000002e-05 2024-09-16T03:39:00.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816997+0000", 2024-09-16T03:39:00.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0036999999999994e-05 2024-09-16T03:39:00.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.146 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.146 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2278 4.18 4:1f63731d:::smithi15736696-30:head [write 1869389~508716] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.673199818999997, 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48788153400000001, 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2278 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429352+0000", 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.604e-06 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429895+0000", 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054308899999999996 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429909+0000", 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3518e-05 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429917+0000", 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-09-16T03:39:00.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525208+0000", 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095291736000000002 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525303+0000", 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4304999999999999e-05 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525410+0000", 2024-09-16T03:39:00.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010755399999999999 2024-09-16T03:39:00.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636785+0000", 2024-09-16T03:39:00.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111375037 2024-09-16T03:39:00.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637092+0000", 2024-09-16T03:39:00.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030685599999999999 2024-09-16T03:39:00.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637157+0000", 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4593999999999999e-05 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637236+0000", 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9563999999999999e-05 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917119+0000", 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.279882931 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917158+0000", 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9096e-05 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917232+0000", 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4091000000000006e-05 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.153 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2279 4.18 4:1f63731d:::smithi15736696-30:head [write 3094884~165674] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.668511023999997, 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483372209, 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2279 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434041+0000", 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.195e-06 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434191+0000", 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000150488 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434202+0000", 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0978000000000001e-05 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434208+0000", 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-09-16T03:39:00.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526820+0000", 2024-09-16T03:39:00.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092612229000000004 2024-09-16T03:39:00.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526894+0000", 2024-09-16T03:39:00.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3435000000000004e-05 2024-09-16T03:39:00.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:00.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526990+0000", 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6112000000000004e-05 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637050+0000", 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110059859 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637097+0000", 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6848000000000003e-05 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637183+0000", 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6798000000000005e-05 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637262+0000", 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9049999999999997e-05 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917287+0000", 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280025036 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917326+0000", 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.871e-05 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917412+0000", 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5638999999999998e-05 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.160 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2280 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 3260558] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.668332743000001, 2024-09-16T03:39:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48353995599999999, 2024-09-16T03:39:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2280 2024-09-16T03:39:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434221+0000", 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8329999999999998e-06 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434225+0000", 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7589999999999999e-06 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434228+0000", 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.971e-06 2024-09-16T03:39:00.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527459+0000", 2024-09-16T03:39:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093231249000000002 2024-09-16T03:39:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527519+0000", 2024-09-16T03:39:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9471e-05 2024-09-16T03:39:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:00.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527606+0000", 2024-09-16T03:39:00.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6776999999999999e-05 2024-09-16T03:39:00.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637084+0000", 2024-09-16T03:39:00.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109478228 2024-09-16T03:39:00.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637100+0000", 2024-09-16T03:39:00.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6711999999999999e-05 2024-09-16T03:39:00.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637210+0000", 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010909200000000001 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637296+0000", 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6718000000000006e-05 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917637+0000", 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280340537 2024-09-16T03:39:00.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917671+0000", 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3983999999999999e-05 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917758+0000", 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7017999999999999e-05 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.168 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2281 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:00.168 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:00.168 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.668318448000001, 2024-09-16T03:39:00.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483761683, 2024-09-16T03:39:00.168 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.168 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:00.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.168 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2281 2024-09-16T03:39:00.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.168 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:00.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.96e-07 2024-09-16T03:39:00.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434234+0000", 2024-09-16T03:39:00.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4050000000000001e-06 2024-09-16T03:39:00.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434237+0000", 2024-09-16T03:39:00.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1939999999999998e-06 2024-09-16T03:39:00.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434240+0000", 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.74e-06 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530037+0000", 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095796830999999999 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530089+0000", 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917823+0000", 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38773380000000002 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917891+0000", 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7676999999999996e-05 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917994+0000", 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010286200000000001 2024-09-16T03:39:00.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2657 4.8 4:148e627d:::smithi15736696-23:head [setxattr _header (56) in=7b,truncate 3899224] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:21.388211+0000", 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.714340002, 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.293904257, 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.173 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2657 2024-09-16T03:39:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388211+0000", 2024-09-16T03:39:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388211+0000", 2024-09-16T03:39:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388211+0000", 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.02e-07 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388217+0000", 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6960000000000002e-06 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388223+0000", 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3349999999999997e-06 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388226+0000", 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2629999999999999e-06 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.391612+0000", 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0033864239999999999 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.391673+0000", 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0418999999999999e-05 2024-09-16T03:39:00.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.391771+0000", 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8638e-05 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.392525+0000", 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075383199999999996 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.551525+0000", 2024-09-16T03:39:00.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.15900014600000001 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.577048+0000", 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025522566 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.582681+0000", 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056329830000000003 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.681849+0000", 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099168439999999997 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.681936+0000", 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6720999999999999e-05 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.682115+0000", 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017889199999999999 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2658 4.8 4:148e627d:::smithi15736696-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:21.388241+0000", 2024-09-16T03:39:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.714309591000003, 2024-09-16T03:39:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29431808300000001, 2024-09-16T03:39:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2658 2024-09-16T03:39:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388241+0000", 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388241+0000", 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388242+0000", 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2699999999999999e-07 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388244+0000", 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2089999999999999e-06 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388250+0000", 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.409e-06 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.388254+0000", 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4489999999999999e-06 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.392379+0000", 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004125325 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.392451+0000", 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2152999999999998e-05 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.682233+0000", 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.28978181400000003 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.682337+0000", 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104258 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.682559+0000", 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022193899999999999 2024-09-16T03:39:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2790 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v39,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-09-16T03:39:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:39:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.698470993000001, 2024-09-16T03:39:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42581759000000002, 2024-09-16T03:39:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2790 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404083+0000", 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2770000000000001e-06 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404095+0000", 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1780000000000001e-05 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404116+0000", 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0882000000000001e-05 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404127+0000", 2024-09-16T03:39:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1078e-05 2024-09-16T03:39:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489837+0000", 2024-09-16T03:39:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085709977000000007 2024-09-16T03:39:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489956+0000", 2024-09-16T03:39:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119856 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.592758+0000", 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10280165400000001 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.593693+0000", 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000935285 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.594164+0000", 2024-09-16T03:39:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047097000000000003 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.628426+0000", 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034262070999999998 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.732623+0000", 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10419628 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829726+0000", 2024-09-16T03:39:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097103098999999998 2024-09-16T03:39:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829790+0000", 2024-09-16T03:39:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4677000000000004e-05 2024-09-16T03:39:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829897+0000", 2024-09-16T03:39:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106704 2024-09-16T03:39:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2850 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 2252171] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:39:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:39:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.181930711, 2024-09-16T03:39:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30323033799999999, 2024-09-16T03:39:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2850 2024-09-16T03:39:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:39:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:39:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920621+0000", 2024-09-16T03:39:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.47e-07 2024-09-16T03:39:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920625+0000", 2024-09-16T03:39:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1620000000000001e-06 2024-09-16T03:39:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920632+0000", 2024-09-16T03:39:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0180000000000004e-06 2024-09-16T03:39:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920638+0000", 2024-09-16T03:39:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0950000000000001e-06 2024-09-16T03:39:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922161+0000", 2024-09-16T03:39:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015229880000000001 2024-09-16T03:39:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922222+0000", 2024-09-16T03:39:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0903999999999998e-05 2024-09-16T03:39:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922323+0000", 2024-09-16T03:39:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101425 2024-09-16T03:39:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.023371+0000", 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101047493 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123227+0000", 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099856057999999998 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123840+0000", 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061324999999999997 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123902+0000", 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1447999999999998e-05 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223497+0000", 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099595499000000004 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223665+0000", 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016807399999999999 2024-09-16T03:39:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223850+0000", 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018507699999999999 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2851 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.181894614999997, 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30354833399999998, 2024-09-16T03:39:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2851 2024-09-16T03:39:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:39:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920657+0000", 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0100000000000004e-07 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920660+0000", 2024-09-16T03:39:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6519999999999999e-06 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920668+0000", 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1919999999999999e-06 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920674+0000", 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126299+0000", 2024-09-16T03:39:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20562534700000001 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126358+0000", 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8459999999999999e-05 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223913+0000", 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097555263000000003 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224009+0000", 2024-09-16T03:39:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5723999999999994e-05 2024-09-16T03:39:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224204+0000", 2024-09-16T03:39:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195498 2024-09-16T03:39:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3570 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:39:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.096761291, 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.353157053, 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3570 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:39:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005796+0000", 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9229999999999996e-06 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006222+0000", 2024-09-16T03:39:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000426282 2024-09-16T03:39:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006240+0000", 2024-09-16T03:39:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7858000000000001e-05 2024-09-16T03:39:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006249+0000", 2024-09-16T03:39:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7120000000000006e-06 2024-09-16T03:39:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006323+0000", 2024-09-16T03:39:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3808999999999998e-05 2024-09-16T03:39:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006333+0000", 2024-09-16T03:39:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0488e-05 2024-09-16T03:39:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358910+0000", 2024-09-16T03:39:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.35257616400000003 2024-09-16T03:39:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358946+0000", 2024-09-16T03:39:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6816999999999997e-05 2024-09-16T03:39:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3571 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:39:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0902561350000006, 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34670431699999998, 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3571 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:39:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012298+0000", 2024-09-16T03:39:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.18e-06 2024-09-16T03:39:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012947+0000", 2024-09-16T03:39:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064929600000000005 2024-09-16T03:39:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012961+0000", 2024-09-16T03:39:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4341e-05 2024-09-16T03:39:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012970+0000", 2024-09-16T03:39:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2740000000000007e-06 2024-09-16T03:39:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.013025+0000", 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5353999999999999e-05 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.013033+0000", 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4619999999999996e-06 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358989+0000", 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34595562699999999 2024-09-16T03:39:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358999+0000", 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7829999999999993e-06 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3572 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0842353790000008, 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34073019300000001, 2024-09-16T03:39:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3572 2024-09-16T03:39:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:39:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018317+0000", 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4610000000000001e-06 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018779+0000", 2024-09-16T03:39:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046202500000000002 2024-09-16T03:39:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018793+0000", 2024-09-16T03:39:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4629e-05 2024-09-16T03:39:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018799+0000", 2024-09-16T03:39:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1589999999999998e-06 2024-09-16T03:39:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018867+0000", 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7879000000000005e-05 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018888+0000", 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0747999999999999e-05 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359036+0000", 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34014758 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359045+0000", 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7119999999999995e-06 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3573 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0837356850000006, 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34027060599999998, 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3573 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018816+0000", 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1400000000000001e-06 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018821+0000", 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4179999999999996e-06 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018829+0000", 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6320000000000008e-06 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018835+0000", 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0399999999999998e-06 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018953+0000", 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117713 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018967+0000", 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4334e-05 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359078+0000", 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34011044499999998 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359086+0000", 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8839999999999994e-06 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3574 4.28 4.f21f628 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0837025300000001, 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.340272882, 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3574 2024-09-16T03:39:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:39:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:39:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018849+0000", 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9599999999999998e-07 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018852+0000", 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9749999999999999e-06 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018867+0000", 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5102e-05 2024-09-16T03:39:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018882+0000", 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5319e-05 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018993+0000", 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110457 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018998+0000", 2024-09-16T03:39:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.028e-06 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359114+0000", 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34011593099999998 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359121+0000", 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2740000000000002e-06 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.229 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:00.230 INFO:tasks.rados.rados.0.smithi157.stdout:1632: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-16T03:39:00.231 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:39:00.231 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:00.228+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:00.251 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:00.280 INFO:tasks.rados.rados.0.smithi157.stdout:1628: expect (ObjNum 314 snap 0 seq_num 314) 2024-09-16T03:39:00.281 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:00.331 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:00.329+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:00.379 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:00.379 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:39:00.379 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:00.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:39:00.432 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:00.430+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:00.485 INFO:tasks.rados.rados.0.smithi157.stdout:1618: finishing set_chunk to oid 15 2024-09-16T03:39:00.485 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 18 (ObjNum 334 snap 0 seq_num 334) dirty exists 2024-09-16T03:39:00.485 INFO:tasks.rados.rados.0.smithi157.stdout:1621: finishing write tid 5 to smithi15736696-50 2024-09-16T03:39:00.485 INFO:tasks.rados.rados.0.smithi157.stdout:1621: oid 50 updating version 0 to 137 2024-09-16T03:39:00.486 INFO:tasks.rados.rados.0.smithi157.stdout:1621: oid 50 updating version 137 to 138 2024-09-16T03:39:00.486 INFO:tasks.rados.rados.0.smithi157.stdout:1621: oid 50 updating version 138 to 139 2024-09-16T03:39:00.486 INFO:tasks.rados.rados.0.smithi157.stdout:1621: oid 50 version 139 is already newer than 136 2024-09-16T03:39:00.486 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 139 (ObjNum 350 snap 0 seq_num 350) dirty exists 2024-09-16T03:39:00.486 INFO:tasks.rados.rados.0.smithi157.stdout:1621: left oid 50 (ObjNum 350 snap 0 seq_num 350) 2024-09-16T03:39:00.486 INFO:tasks.rados.rados.0.smithi157.stdout:1624: finishing set_chunk to oid 34 2024-09-16T03:39:00.486 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 146 (ObjNum 342 snap 0 seq_num 342) dirty exists 2024-09-16T03:39:00.486 INFO:tasks.rados.rados.0.smithi157.stdout:1631: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-16T03:39:00.492 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T03:39:00.498 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:00.498 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:00.498 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:00.498 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:00.498 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:00.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:39:00.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:39:00.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T03:39:00.533 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:00.530+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:00.539 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:00.569 INFO:tasks.rados.rados.0.smithi157.stdout:1630: finishing write tid 1 to smithi15736696-3 2024-09-16T03:39:00.569 INFO:tasks.rados.rados.0.smithi157.stdout:1626: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:39:00.569 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:00.634 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:00.633+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:00.665 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:00.665 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:00.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:00.665 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:00.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.665 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:34 4.e 4:773b934b:::smithi15736696-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:39:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.60754643200001, 2024-09-16T03:39:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50346527299999999, 2024-09-16T03:39:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-16T03:39:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054413+0000", 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0669999999999999e-06 2024-09-16T03:39:00.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054525+0000", 2024-09-16T03:39:00.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111462 2024-09-16T03:39:00.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054538+0000", 2024-09-16T03:39:00.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3341e-05 2024-09-16T03:39:00.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054545+0000", 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.584e-06 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054608+0000", 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2953999999999994e-05 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054613+0000", 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6110000000000001e-06 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154852+0000", 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10023838 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154915+0000", 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3138000000000004e-05 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155001+0000", 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5730999999999996e-05 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156166+0000", 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001165485 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255909+0000", 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099742558999999995 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557800+0000", 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30189130600000003 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557836+0000", 2024-09-16T03:39:00.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6566000000000002e-05 2024-09-16T03:39:00.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557877+0000", 2024-09-16T03:39:00.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0089000000000002e-05 2024-09-16T03:39:00.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.672 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:35 4.e 4:773b934b:::smithi15736696-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.607398101, 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50344320499999995, 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:39:00.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054564+0000", 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5710000000000002e-06 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054569+0000", 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0490000000000003e-06 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054578+0000", 2024-09-16T03:39:00.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:39:00.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054639+0000", 2024-09-16T03:39:00.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1061000000000004e-05 2024-09-16T03:39:00.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:00.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054644+0000", 2024-09-16T03:39:00.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999996e-06 2024-09-16T03:39:00.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155607+0000", 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100963372 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155652+0000", 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155722+0000", 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0130999999999997e-05 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.157921+0000", 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021990159999999998 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255940+0000", 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098018569999999999 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557930+0000", 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.301990333 2024-09-16T03:39:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557963+0000", 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2348000000000003e-05 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558003+0000", 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:36 4.e 4:773b934b:::smithi15736696-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:39:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.60735941199999, 2024-09-16T03:39:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50353811800000003, 2024-09-16T03:39:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-16T03:39:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054599+0000", 2024-09-16T03:39:00.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054601+0000", 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4360000000000001e-06 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054606+0000", 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1019999999999996e-06 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054610+0000", 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.681e-06 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054662+0000", 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1545000000000002e-05 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054664+0000", 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1919999999999999e-06 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155903+0000", 2024-09-16T03:39:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101239068 2024-09-16T03:39:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155970+0000", 2024-09-16T03:39:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6934999999999995e-05 2024-09-16T03:39:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558029+0000", 2024-09-16T03:39:00.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40205888899999997 2024-09-16T03:39:00.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558078+0000", 2024-09-16T03:39:00.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9842999999999997e-05 2024-09-16T03:39:00.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558136+0000", 2024-09-16T03:39:00.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7985000000000001e-05 2024-09-16T03:39:00.685 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.685 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.685 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:799 4.e 4:765e9179:::smithi15736696-44:head [setxattr _header (56) in=7b,truncate 2857157] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.655507185000005, 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50104159800000003, 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 799 2024-09-16T03:39:00.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006451+0000", 2024-09-16T03:39:00.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5000000000000002e-07 2024-09-16T03:39:00.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006454+0000", 2024-09-16T03:39:00.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5739999999999998e-06 2024-09-16T03:39:00.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006458+0000", 2024-09-16T03:39:00.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6179999999999997e-06 2024-09-16T03:39:00.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006462+0000", 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4999999999999999e-06 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204504+0000", 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19804184999999999 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204702+0000", 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019868199999999999 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.204888+0000", 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185701 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.206549+0000", 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016612790000000001 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507371+0000", 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30082157399999998 2024-09-16T03:39:00.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507414+0000", 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3053999999999998e-05 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507451+0000", 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6847999999999997e-05 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507492+0000", 2024-09-16T03:39:00.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1267999999999999e-05 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:800 4.e 4:765e9179:::smithi15736696-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.655483849999996, 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50119258099999997, 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.692 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.693 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 800 2024-09-16T03:39:00.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.693 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:39:00.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:39:00.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006474+0000", 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1899999999999998e-07 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006478+0000", 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006484+0000", 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6620000000000002e-06 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.006488+0000", 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8700000000000002e-06 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205228+0000", 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19873985899999999 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:00.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.205283+0000", 2024-09-16T03:39:00.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5878999999999998e-05 2024-09-16T03:39:00.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507527+0000", 2024-09-16T03:39:00.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302243924 2024-09-16T03:39:00.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507593+0000", 2024-09-16T03:39:00.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6061000000000003e-05 2024-09-16T03:39:00.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.507666+0000", 2024-09-16T03:39:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3059000000000006e-05 2024-09-16T03:39:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:90 3.0 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.105346243, 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62288137399999999, 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556614+0000", 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3439999999999999e-06 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556621+0000", 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7440000000000001e-06 2024-09-16T03:39:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556632+0000", 2024-09-16T03:39:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1725e-05 2024-09-16T03:39:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556640+0000", 2024-09-16T03:39:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0239999999999997e-06 2024-09-16T03:39:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556709+0000", 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8787000000000004e-05 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556718+0000", 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6920000000000008e-06 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758133+0000", 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201414977 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758273+0000", 2024-09-16T03:39:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140684 2024-09-16T03:39:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758479+0000", 2024-09-16T03:39:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020508599999999999 2024-09-16T03:39:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862766+0000", 2024-09-16T03:39:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104287952 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.878363+0000", 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015596601999999999 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960000+0000", 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081636630000000002 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960162+0000", 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000162597 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179366+0000", 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21920357100000001 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179411+0000", 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5461999999999999e-05 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179493+0000", 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1496999999999996e-05 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:91 3.0 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.105301284999996, 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70460932600000004, 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556657+0000", 2024-09-16T03:39:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:39:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556663+0000", 2024-09-16T03:39:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6579999999999999e-06 2024-09-16T03:39:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556670+0000", 2024-09-16T03:39:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3189999999999997e-06 2024-09-16T03:39:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556675+0000", 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000002e-06 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656929+0000", 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10025371299999999 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656944+0000", 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5460000000000001e-05 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861436+0000", 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.204491969 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861676+0000", 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024051899999999999 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861828+0000", 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015110199999999999 2024-09-16T03:39:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862775+0000", 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094718600000000003 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079130+0000", 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21635549200000001 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179557+0000", 2024-09-16T03:39:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10042638399999999 2024-09-16T03:39:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179972+0000", 2024-09-16T03:39:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041574 2024-09-16T03:39:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261034+0000", 2024-09-16T03:39:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081061604999999995 2024-09-16T03:39:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261142+0000", 2024-09-16T03:39:00.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107688 2024-09-16T03:39:00.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261266+0000", 2024-09-16T03:39:00.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124082 2024-09-16T03:39:00.712 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.712 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.712 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.712 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:94 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.105275160000005, 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70479513699999996, 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:39:00.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:39:00.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-09-16T03:39:00.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556685+0000", 2024-09-16T03:39:00.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4279999999999999e-06 2024-09-16T03:39:00.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556689+0000", 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.709e-06 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556693+0000", 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2939999999999999e-06 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657036+0000", 2024-09-16T03:39:00.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10034209600000001 2024-09-16T03:39:00.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:00.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657054+0000", 2024-09-16T03:39:00.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8162999999999999e-05 2024-09-16T03:39:00.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862279+0000", 2024-09-16T03:39:00.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20522560200000001 2024-09-16T03:39:00.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862380+0000", 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101178 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862509+0000", 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128909 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.863063+0000", 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055327600000000001 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079172+0000", 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21610910899999999 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179932+0000", 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100760575 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179997+0000", 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5018999999999995e-05 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261318+0000", 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081320838000000006 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261385+0000", 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7115999999999999e-05 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261478+0000", 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2360000000000003e-05 2024-09-16T03:39:00.719 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.720 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.720 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.720 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1742 3.0 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head [write 0~1659392] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T03:39:00.720 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:39:00.720 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 65.10524676, 2024-09-16T03:39:00.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62317617400000003, 2024-09-16T03:39:00.720 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.720 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.720 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1742 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556713+0000", 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7209999999999999e-06 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558531+0000", 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018184480000000001 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558539+0000", 2024-09-16T03:39:00.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0830000000000004e-06 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558546+0000", 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3369999999999996e-06 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759148+0000", 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20060235500000001 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759230+0000", 2024-09-16T03:39:00.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2391000000000004e-05 2024-09-16T03:39:00.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:00.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759329+0000", 2024-09-16T03:39:00.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9147000000000005e-05 2024-09-16T03:39:00.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862771+0000", 2024-09-16T03:39:00.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103441085 2024-09-16T03:39:00.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960102+0000", 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097331259000000003 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960218+0000", 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011581100000000001 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079074+0000", 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118856436 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179758+0000", 2024-09-16T03:39:00.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100683884 2024-09-16T03:39:00.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179800+0000", 2024-09-16T03:39:00.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1928000000000002e-05 2024-09-16T03:39:00.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179887+0000", 2024-09-16T03:39:00.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7288999999999997e-05 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:89 3.0 3:1c07e18b:::d3cef63b99e391058a88d444a56984ba6fc25b22efbc98e24ba274246c542d3c:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.325844263999997, 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50388158999999999, 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:39:00.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:39:00.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336115+0000", 2024-09-16T03:39:00.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.866e-06 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336123+0000", 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6990000000000003e-06 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336136+0000", 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3342e-05 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336143+0000", 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2099999999999998e-06 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336214+0000", 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1538000000000005e-05 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336342+0000", 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127408 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:00.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336593+0000", 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025150500000000001 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337284+0000", 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069060900000000001 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338068+0000", 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078438900000000003 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338107+0000", 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8858e-05 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538164+0000", 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200056971 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839843+0000", 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30167921599999997 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839905+0000", 2024-09-16T03:39:00.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1542000000000001e-05 2024-09-16T03:39:00.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839995+0000", 2024-09-16T03:39:00.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0437000000000002e-05 2024-09-16T03:39:00.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.733 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.733 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.733 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:98 3.0 3:04dfaaf9:::3de53e1d584ced9b0f970855867dbc9a14b3fc90962aa20a81c1ed25ff4b961f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:00.734 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:39:00.734 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.324516363000001, 2024-09-16T03:39:00.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70310345299999999, 2024-09-16T03:39:00.734 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.734 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.734 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.734 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:39:00.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337442+0000", 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0729999999999999e-06 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337447+0000", 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3259999999999997e-06 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337457+0000", 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337462+0000", 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0780000000000003e-06 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337511+0000", 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337574+0000", 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3114999999999995e-05 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337723+0000", 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014882299999999999 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.339039+0000", 2024-09-16T03:39:00.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001315524 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538225+0000", 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19918607199999999 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638838+0000", 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100613466 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.739726+0000", 2024-09-16T03:39:00.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100887854 2024-09-16T03:39:00.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040319+0000", 2024-09-16T03:39:00.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30059265699999999 2024-09-16T03:39:00.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040398+0000", 2024-09-16T03:39:00.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8852999999999999e-05 2024-09-16T03:39:00.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040545+0000", 2024-09-16T03:39:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147236 2024-09-16T03:39:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:101 3.0 3:090a0a71:::0ecec3e4e81d718f460a9826fb4cef2398bf4c951ee653be84827f33946eef5f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:39:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.225345617000002, 2024-09-16T03:39:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60435370600000005, 2024-09-16T03:39:00.740 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.741 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.741 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.741 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:39:00.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.741 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.741 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:39:00.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436614+0000", 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.844e-06 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436623+0000", 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0739999999999999e-06 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436636+0000", 2024-09-16T03:39:00.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2629999999999999e-05 2024-09-16T03:39:00.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436643+0000", 2024-09-16T03:39:00.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7489999999999998e-06 2024-09-16T03:39:00.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536906+0000", 2024-09-16T03:39:00.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10026262599999999 2024-09-16T03:39:00.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536979+0000", 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3311000000000003e-05 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537109+0000", 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013015800000000001 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538241+0000", 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001131726 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638904+0000", 2024-09-16T03:39:00.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066240799999999 2024-09-16T03:39:00.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.940200+0000", 2024-09-16T03:39:00.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30129597299999999 2024-09-16T03:39:00.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040604+0000", 2024-09-16T03:39:00.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040478999999999 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040759+0000", 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015445500000000001 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040824+0000", 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5633e-05 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040966+0000", 2024-09-16T03:39:00.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014132899999999999 2024-09-16T03:39:00.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.747 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.747 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:102 3.0 3:17742417:::632266e231c4c385ace473a8f4464285cf620fbc8799f68869353e2aa4660567:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:00.747 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:39:00.747 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.225298791, 2024-09-16T03:39:00.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.604540823, 2024-09-16T03:39:00.747 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.747 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.747 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.748 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:39:00.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:39:00.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436660+0000", 2024-09-16T03:39:00.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:39:00.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436663+0000", 2024-09-16T03:39:00.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1599999999999998e-06 2024-09-16T03:39:00.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436670+0000", 2024-09-16T03:39:00.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.742e-06 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436675+0000", 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3839999999999996e-06 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537499+0000", 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100823987 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537602+0000", 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102837 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537793+0000", 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019123499999999999 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638962+0000", 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101168944 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.840035+0000", 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201073481 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040706+0000", 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200670606 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041009+0000", 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030271600000000002 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041087+0000", 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8310000000000001e-05 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041127+0000", 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041200+0000", 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2381999999999999e-05 2024-09-16T03:39:00.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.754 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.754 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.754 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:103 3.0 3:28447854:::c811732c201b76a96e1532c4d9a18d4f4f2a8ddec3bf79945689a1a62c8a5a25:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:00.754 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:39:00.754 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.225253541999997, 2024-09-16T03:39:00.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60467526100000002, 2024-09-16T03:39:00.754 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.754 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.754 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.754 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436705+0000", 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.23e-07 2024-09-16T03:39:00.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436719+0000", 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3957999999999999e-05 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436725+0000", 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6100000000000002e-06 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436730+0000", 2024-09-16T03:39:00.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0609999999999997e-06 2024-09-16T03:39:00.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538257+0000", 2024-09-16T03:39:00.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10152739500000001 2024-09-16T03:39:00.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538314+0000", 2024-09-16T03:39:00.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7355000000000002e-05 2024-09-16T03:39:00.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:00.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538435+0000", 2024-09-16T03:39:00.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120967 2024-09-16T03:39:00.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638968+0000", 2024-09-16T03:39:00.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100532994 2024-09-16T03:39:00.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040662+0000", 2024-09-16T03:39:00.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40169386299999998 2024-09-16T03:39:00.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041059+0000", 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039723900000000002 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041247+0000", 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018756900000000001 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041286+0000", 2024-09-16T03:39:00.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8971999999999999e-05 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041318+0000", 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1628999999999998e-05 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041379+0000", 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1926000000000003e-05 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.761 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(osd.11.23:98 3.0 e74/69 ondisk, result = 0)", 2024-09-16T03:39:00.761 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:39:00.761 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.123437174000003, 2024-09-16T03:39:00.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50203460700000002, 2024-09-16T03:39:00.761 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.761 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.761 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:00.761 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:39:00.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:39:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538521+0000", 2024-09-16T03:39:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0899999999999997e-07 2024-09-16T03:39:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538523+0000", 2024-09-16T03:39:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7859999999999999e-06 2024-09-16T03:39:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538527+0000", 2024-09-16T03:39:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0539999999999996e-06 2024-09-16T03:39:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538529+0000", 2024-09-16T03:39:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7420000000000002e-06 2024-09-16T03:39:00.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040301+0000", 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50177146399999994 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040312+0000", 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1039999999999999e-05 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040555+0000", 2024-09-16T03:39:00.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243212 2024-09-16T03:39:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2473 4.e 4:765e9179:::smithi15736696-44:head [write 1830877~681726] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:39:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.651268317000003, 2024-09-16T03:39:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50285070099999996, 2024-09-16T03:39:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.765 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2473 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010691+0000", 2024-09-16T03:39:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-06 2024-09-16T03:39:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011744+0000", 2024-09-16T03:39:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001052486 2024-09-16T03:39:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011755+0000", 2024-09-16T03:39:00.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1073000000000001e-05 2024-09-16T03:39:00.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011761+0000", 2024-09-16T03:39:00.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1600000000000003e-06 2024-09-16T03:39:00.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011804+0000", 2024-09-16T03:39:00.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2697999999999997e-05 2024-09-16T03:39:00.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011885+0000", 2024-09-16T03:39:00.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.119e-05 2024-09-16T03:39:00.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:39:00.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.012013+0000", 2024-09-16T03:39:00.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128066 2024-09-16T03:39:00.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.013923+0000", 2024-09-16T03:39:00.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019104739999999999 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.412920+0000", 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39899685200000001 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513347+0000", 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100427167 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513490+0000", 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142163 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513540+0000", 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0482000000000002e-05 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:135 3.0 3:03e4b4cc:::36686daa84b9b41d23533e7d0b5d4ef41f745a0756e58cc66c36c36f24851219:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.114458+0000", 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.547499541000001, 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.502137163, 2024-09-16T03:39:00.771 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114458+0000", 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114458+0000", 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114459+0000", 2024-09-16T03:39:00.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.26e-06 2024-09-16T03:39:00.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114464+0000", 2024-09-16T03:39:00.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8049999999999997e-06 2024-09-16T03:39:00.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114474+0000", 2024-09-16T03:39:00.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4369999999999996e-06 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114479+0000", 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3669999999999996e-06 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114539+0000", 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114652+0000", 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112948 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114886+0000", 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000233666 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116749+0000", 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001863314 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116765+0000", 2024-09-16T03:39:00.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5528000000000001e-05 2024-09-16T03:39:00.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.413659+0000", 2024-09-16T03:39:00.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.296894245 2024-09-16T03:39:00.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414169+0000", 2024-09-16T03:39:00.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050972500000000004 2024-09-16T03:39:00.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616394+0000", 2024-09-16T03:39:00.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20222523000000001 2024-09-16T03:39:00.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616467+0000", 2024-09-16T03:39:00.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2929000000000006e-05 2024-09-16T03:39:00.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616595+0000", 2024-09-16T03:39:00.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012867400000000001 2024-09-16T03:39:00.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.778 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.778 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:136 3.0 3:1b68d773:::374e1d62491a062bcb5530630b4c28c4696d618a79860b96215d227206cd62aa:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:00.778 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:39:00.778 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.547463778000001, 2024-09-16T03:39:00.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50236015199999995, 2024-09-16T03:39:00.778 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.778 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.778 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.778 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4499999999999997e-07 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114497+0000", 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114506+0000", 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5759999999999995e-06 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114511+0000", 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0649999999999998e-06 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115277+0000", 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076592800000000003 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115368+0000", 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1037000000000003e-05 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:00.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115573+0000", 2024-09-16T03:39:00.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000204594 2024-09-16T03:39:00.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116773+0000", 2024-09-16T03:39:00.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001200062 2024-09-16T03:39:00.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.121514+0000", 2024-09-16T03:39:00.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047407270000000001 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414056+0000", 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29254249700000001 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414192+0000", 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013572000000000001 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616663+0000", 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20247094800000001 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616728+0000", 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5217999999999998e-05 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616854+0000", 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126064 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:137 3.0 3:20b08a9e:::0ba23f501764ab3447cc323c825ee8bff84c3c6e340c68c08117252e1f0b39bb:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:39:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.547419832999999, 2024-09-16T03:39:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50256677599999999, 2024-09-16T03:39:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1200000000000003e-07 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114542+0000", 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7400000000000002e-06 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114547+0000", 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.42e-06 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114550+0000", 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4470000000000002e-06 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115951+0000", 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014005459999999999 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116047+0000", 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6899999999999997e-05 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116270+0000", 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022291699999999999 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.117236+0000", 2024-09-16T03:39:00.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096522299999999997 2024-09-16T03:39:00.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.218932+0000", 2024-09-16T03:39:00.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10169644799999999 2024-09-16T03:39:00.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414271+0000", 2024-09-16T03:39:00.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19533916300000001 2024-09-16T03:39:00.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.514717+0000", 2024-09-16T03:39:00.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100445879 2024-09-16T03:39:00.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616919+0000", 2024-09-16T03:39:00.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10220199100000001 2024-09-16T03:39:00.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616980+0000", 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0977999999999998e-05 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.617105+0000", 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012461199999999999 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3402 4.1b 4:dbb349e8:::smithi15736696-46:head [stat,delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T03:39:00.791 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:46.357580+0000", 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.304377902000001, 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.80350077600000003, 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3402 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:00.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357580+0000", 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357580+0000", 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357582+0000", 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1890000000000001e-06 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:00.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357588+0000", 2024-09-16T03:39:00.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.896e-06 2024-09-16T03:39:00.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:00.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357606+0000", 2024-09-16T03:39:00.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7841000000000001e-05 2024-09-16T03:39:00.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:00.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357617+0000", 2024-09-16T03:39:00.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1698999999999999e-05 2024-09-16T03:39:00.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357697+0000", 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9822999999999998e-05 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357708+0000", 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1119e-05 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.962316+0000", 2024-09-16T03:39:00.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60460744899999996 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.962430+0000", 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011423 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,6,8,9,15", 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.962530+0000", 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100005 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063659+0000", 2024-09-16T03:39:00.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101128564 2024-09-16T03:39:00.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063691+0000", 2024-09-16T03:39:00.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2413999999999998e-05 2024-09-16T03:39:00.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:00.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063701+0000", 2024-09-16T03:39:00.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9180000000000006e-06 2024-09-16T03:39:00.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063733+0000", 2024-09-16T03:39:00.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2416000000000003e-05 2024-09-16T03:39:00.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063758+0000", 2024-09-16T03:39:00.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.421e-05 2024-09-16T03:39:00.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.160869+0000", 2024-09-16T03:39:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097111571999999993 2024-09-16T03:39:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.160922+0000", 2024-09-16T03:39:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3025999999999997e-05 2024-09-16T03:39:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:00.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.161081+0000", 2024-09-16T03:39:00.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000158405 2024-09-16T03:39:00.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.800 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:00.800 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:00.800 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:00.802 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:39:00.803 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:00.799+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:00.803 INFO:tasks.rados.rados.0.smithi157.stdout:1630: finishing write tid 2 to smithi15736696-3 2024-09-16T03:39:00.803 INFO:tasks.rados.rados.0.smithi157.stdout:1630: finishing write tid 3 to smithi15736696-3 2024-09-16T03:39:00.804 INFO:tasks.rados.rados.0.smithi157.stdout:1630: finishing write tid 4 to smithi15736696-3 2024-09-16T03:39:00.804 INFO:tasks.rados.rados.0.smithi157.stdout:1630: finishing write tid 5 to smithi15736696-3 2024-09-16T03:39:00.804 INFO:tasks.rados.rados.0.smithi157.stdout:1630: oid 3 updating version 0 to 70 2024-09-16T03:39:00.804 INFO:tasks.rados.rados.0.smithi157.stdout:1630: oid 3 version 70 is already newer than 68 2024-09-16T03:39:00.804 INFO:tasks.rados.rados.0.smithi157.stdout:1630: oid 3 updating version 70 to 71 2024-09-16T03:39:00.804 INFO:tasks.rados.rados.0.smithi157.stdout:1630: oid 3 version 71 is already newer than 69 2024-09-16T03:39:00.804 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 71 (ObjNum 352 snap 0 seq_num 352) dirty exists 2024-09-16T03:39:00.804 INFO:tasks.rados.rados.0.smithi157.stdout:1630: left oid 3 (ObjNum 352 snap 0 seq_num 352) 2024-09-16T03:39:00.804 INFO:tasks.rados.rados.0.smithi157.stdout:1618: done (14 left) 2024-09-16T03:39:00.804 INFO:tasks.rados.rados.0.smithi157.stdout:1621: done (13 left) 2024-09-16T03:39:00.804 INFO:tasks.rados.rados.0.smithi157.stdout:1623: done (12 left) 2024-09-16T03:39:00.804 INFO:tasks.rados.rados.0.smithi157.stdout:1624: done (11 left) 2024-09-16T03:39:00.804 INFO:tasks.rados.rados.0.smithi157.stdout:1625: done (10 left) 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:1626: done (9 left) 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:1628: done (8 left) 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:1630: done (7 left) 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:1631: done (6 left) 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:1632: done (5 left) 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:1633: done (4 left) 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:1634: done (3 left) 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:1638: read oid 20 snap -1 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:1638: expect (ObjNum 346 snap 0 seq_num 346) 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:1639: read oid 4 snap -1 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:1639: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:1640: write (excl) oid 31 current snap is 0 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:1635: finishing set_chunk to oid 12 2024-09-16T03:39:00.805 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 12 set_chunk returned error code -95 offset: 1724416 length: 742912 2024-09-16T03:39:00.806 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 0 (ObjNum 297 snap 0 seq_num 297) dirty exists 2024-09-16T03:39:00.806 INFO:tasks.rados.rados.0.smithi157.stdout:1637: got (22) Invalid argument 2024-09-16T03:39:00.806 INFO:tasks.rados.rados.0.smithi157.stdout:1636: expect (ObjNum 338 snap 0 seq_num 338) 2024-09-16T03:39:00.806 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:00.902 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:00.901+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:00.910 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1958559 2024-09-16T03:39:00.950 INFO:tasks.rados.rados.0.smithi157.stdout:1638: expect (ObjNum 346 snap 0 seq_num 346) 2024-09-16T03:39:00.951 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:00.952 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:39:00.952 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:00.960 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:39:01.003 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:01.001+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:01.038 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:01.067 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:01.067 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:01.067 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:01.067 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:01.068 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:01.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:39:01.119 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:01.118+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:01.151 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1996135 2024-09-16T03:39:01.185 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:01.185 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:01.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:01.185 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:01.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:73 4.a 4:532814f2:::smithi15736696-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.870794424, 2024-09-16T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.362642662, 2024-09-16T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73 2024-09-16T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.186 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311261+0000", 2024-09-16T03:39:01.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.562e-06 2024-09-16T03:39:01.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311925+0000", 2024-09-16T03:39:01.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066385499999999998 2024-09-16T03:39:01.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311937+0000", 2024-09-16T03:39:01.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1881e-05 2024-09-16T03:39:01.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311943+0000", 2024-09-16T03:39:01.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6999999999999996e-06 2024-09-16T03:39:01.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311988+0000", 2024-09-16T03:39:01.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5358000000000002e-05 2024-09-16T03:39:01.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312070+0000", 2024-09-16T03:39:01.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1735000000000003e-05 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312167+0000", 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6685e-05 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.313661+0000", 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001494324 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.414759+0000", 2024-09-16T03:39:01.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10109850300000001 2024-09-16T03:39:01.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673821+0000", 2024-09-16T03:39:01.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.25906127400000001 2024-09-16T03:39:01.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673858+0000", 2024-09-16T03:39:01.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7620999999999998e-05 2024-09-16T03:39:01.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673902+0000", 2024-09-16T03:39:01.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4163999999999999e-05 2024-09-16T03:39:01.192 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.192 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.192 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.192 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:303 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:01.192 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:39:01.192 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.770335594, 2024-09-16T03:39:01.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50956986000000004, 2024-09-16T03:39:01.193 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.193 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.193 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.193 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.193 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.193 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.193 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 303 2024-09-16T03:39:01.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.193 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:39:01.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411721+0000", 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0200000000000001e-06 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411729+0000", 2024-09-16T03:39:01.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1240000000000005e-06 2024-09-16T03:39:01.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411744+0000", 2024-09-16T03:39:01.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099000000000001e-05 2024-09-16T03:39:01.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411751+0000", 2024-09-16T03:39:01.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.029e-06 2024-09-16T03:39:01.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511892+0000", 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100140879 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511961+0000", 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9379000000000001e-05 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.613647+0000", 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101685994 2024-09-16T03:39:01.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.614294+0000", 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064692799999999996 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.714962+0000", 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066765 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921157+0000", 2024-09-16T03:39:01.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206195463 2024-09-16T03:39:01.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921233+0000", 2024-09-16T03:39:01.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5446999999999998e-05 2024-09-16T03:39:01.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921288+0000", 2024-09-16T03:39:01.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5847999999999998e-05 2024-09-16T03:39:01.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.198 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.198 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.199 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:503 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:01.199 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:39:01.199 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 99.360261042000005, 2024-09-16T03:39:01.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60521621599999997, 2024-09-16T03:39:01.199 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.199 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.199 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.199 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.199 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.200 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.200 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 503 2024-09-16T03:39:01.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.200 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:39:01.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:39:01.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821794+0000", 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821798+0000", 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2030000000000002e-06 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821804+0000", 2024-09-16T03:39:01.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:39:01.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821810+0000", 2024-09-16T03:39:01.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4249999999999997e-06 2024-09-16T03:39:01.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922212+0000", 2024-09-16T03:39:01.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040228800000001 2024-09-16T03:39:01.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922307+0000", 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5089999999999999e-05 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.126007+0000", 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20369931499999999 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.127605+0000", 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001598213 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.327263+0000", 2024-09-16T03:39:01.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199657905 2024-09-16T03:39:01.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426900+0000", 2024-09-16T03:39:01.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099637217 2024-09-16T03:39:01.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426963+0000", 2024-09-16T03:39:01.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:39:01.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.427009+0000", 2024-09-16T03:39:01.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6533000000000003e-05 2024-09-16T03:39:01.205 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.205 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.205 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.205 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1227 4.a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v18,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:39:01.205 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:39:01.205 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.506084778000002, 2024-09-16T03:39:01.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.44160531400000003, 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1227 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675972+0000", 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7269999999999999e-06 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675981+0000", 2024-09-16T03:39:01.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.166e-06 2024-09-16T03:39:01.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675996+0000", 2024-09-16T03:39:01.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4901e-05 2024-09-16T03:39:01.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676005+0000", 2024-09-16T03:39:01.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4579999999999993e-06 2024-09-16T03:39:01.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676081+0000", 2024-09-16T03:39:01.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6157999999999998e-05 2024-09-16T03:39:01.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676188+0000", 2024-09-16T03:39:01.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106871 2024-09-16T03:39:01.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:01.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.016594+0000", 2024-09-16T03:39:01.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34040622300000001 2024-09-16T03:39:01.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017089+0000", 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049523899999999997 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017779+0000", 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069032200000000005 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117504+0000", 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099724207999999995 2024-09-16T03:39:01.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117543+0000", 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9208999999999997e-05 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117575+0000", 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1832000000000003e-05 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1523 4.a 4:50107449:::smithi15736696-27:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 75.819043452000003, 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.410522742, 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1523 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363012+0000", 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.269e-06 2024-09-16T03:39:01.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363016+0000", 2024-09-16T03:39:01.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8920000000000002e-06 2024-09-16T03:39:01.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363027+0000", 2024-09-16T03:39:01.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1079e-05 2024-09-16T03:39:01.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463160+0000", 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10013267100000001 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463297+0000", 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013717699999999999 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463381+0000", 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3701000000000004e-05 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.773533+0000", 2024-09-16T03:39:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31015295300000001 2024-09-16T03:39:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1690 4.a 4:51338165:::smithi15736696-4:head [assert-version v63,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:39:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.731770784999995, 2024-09-16T03:39:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52002349699999995, 2024-09-16T03:39:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1690 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450286+0000", 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450293+0000", 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5089999999999996e-06 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450308+0000", 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4955e-05 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450316+0000", 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7079999999999997e-06 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450386+0000", 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9715999999999995e-05 2024-09-16T03:39:01.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450478+0000", 2024-09-16T03:39:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2610999999999998e-05 2024-09-16T03:39:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.665802+0000", 2024-09-16T03:39:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21532342700000001 2024-09-16T03:39:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668724+0000", 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029220600000000002 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668758+0000", 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.434e-05 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970243+0000", 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30148501 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970278+0000", 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4965000000000002e-05 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970307+0000", 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8968000000000001e-05 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1654 4.12 4:4a9d235d:::smithi15736696-1:head [write 652138~461367 in=461367b,stat out=16b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:01.222 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.640017294000003, 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42790198699999998, 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1654 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:39:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:39:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542039+0000", 2024-09-16T03:39:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0250000000000001e-06 2024-09-16T03:39:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542682+0000", 2024-09-16T03:39:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064305099999999995 2024-09-16T03:39:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542698+0000", 2024-09-16T03:39:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6469000000000001e-05 2024-09-16T03:39:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542710+0000", 2024-09-16T03:39:01.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1173e-05 2024-09-16T03:39:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542767+0000", 2024-09-16T03:39:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7584000000000003e-05 2024-09-16T03:39:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542775+0000", 2024-09-16T03:39:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6149999999999999e-06 2024-09-16T03:39:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666597+0000", 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123821815 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666745+0000", 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148675 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666935+0000", 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018963899999999999 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969207+0000", 2024-09-16T03:39:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302272031 2024-09-16T03:39:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969374+0000", 2024-09-16T03:39:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016686699999999999 2024-09-16T03:39:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969533+0000", 2024-09-16T03:39:01.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015909299999999999 2024-09-16T03:39:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969560+0000", 2024-09-16T03:39:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7148000000000002e-05 2024-09-16T03:39:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969631+0000", 2024-09-16T03:39:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1174000000000007e-05 2024-09-16T03:39:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969685+0000", 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3613000000000003e-05 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969764+0000", 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9281000000000002e-05 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969939+0000", 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473400000000001 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.230 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1655 4.12 4:4a9d235d:::smithi15736696-1:head [write 1574940~677198 in=677198b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:39:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.639325176, 2024-09-16T03:39:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42773147700000003, 2024-09-16T03:39:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.231 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1655 2024-09-16T03:39:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:39:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:39:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542730+0000", 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.085e-06 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543338+0000", 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060826100000000002 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543351+0000", 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2789000000000001e-05 2024-09-16T03:39:01.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543360+0000", 2024-09-16T03:39:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1970000000000002e-06 2024-09-16T03:39:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543420+0000", 2024-09-16T03:39:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9562999999999999e-05 2024-09-16T03:39:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:01.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543431+0000", 2024-09-16T03:39:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0866999999999999e-05 2024-09-16T03:39:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.768963+0000", 2024-09-16T03:39:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.225532487 2024-09-16T03:39:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769029+0000", 2024-09-16T03:39:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5881000000000002e-05 2024-09-16T03:39:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:39:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769134+0000", 2024-09-16T03:39:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104923 2024-09-16T03:39:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969639+0000", 2024-09-16T03:39:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20050517900000001 2024-09-16T03:39:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969978+0000", 2024-09-16T03:39:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033848599999999998 2024-09-16T03:39:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970206+0000", 2024-09-16T03:39:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022867300000000001 2024-09-16T03:39:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970248+0000", 2024-09-16T03:39:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1433999999999997e-05 2024-09-16T03:39:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970299+0000", 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0677999999999998e-05 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970338+0000", 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9646e-05 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970377+0000", 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8439999999999998e-05 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970460+0000", 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3887999999999994e-05 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1656 4.12 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2252138] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.638679104000005, 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72904596899999996, 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1656 2024-09-16T03:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543376+0000", 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4200000000000004e-07 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543383+0000", 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6290000000000004e-06 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543390+0000", 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543396+0000", 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.02e-06 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543461+0000", 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4876999999999996e-05 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543465+0000", 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2139999999999998e-06 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970490+0000", 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42702478300000002 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970556+0000", 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6137999999999996e-05 2024-09-16T03:39:01.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970670+0000", 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114416 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971375+0000", 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070496000000000003 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071773+0000", 2024-09-16T03:39:01.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100397945 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071823+0000", 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.072067+0000", 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243856 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.171991+0000", 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099924052999999999 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272176+0000", 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100184963 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272251+0000", 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5902000000000006e-05 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272421+0000", 2024-09-16T03:39:01.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016962800000000001 2024-09-16T03:39:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1657 4.12 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:39:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.638647117999994, 2024-09-16T03:39:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72934181899999995, 2024-09-16T03:39:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:01.247 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1657 2024-09-16T03:39:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:39:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543408+0000", 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.13e-07 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543410+0000", 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7030000000000002e-06 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543417+0000", 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.567e-06 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543423+0000", 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0619999999999997e-06 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543489+0000", 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6315999999999993e-05 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543498+0000", 2024-09-16T03:39:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7239999999999998e-06 2024-09-16T03:39:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970952+0000", 2024-09-16T03:39:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42745424399999998 2024-09-16T03:39:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971012+0000", 2024-09-16T03:39:01.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9771000000000001e-05 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272463+0000", 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30145090800000002 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272558+0000", 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4863000000000003e-05 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272749+0000", 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191048 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2105 4.a 4:51338165:::smithi15736696-4:head [sparse-read 0~887916,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.133182924000003, 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40244934799999998, 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:01.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2105 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048881+0000", 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6080000000000002e-06 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048889+0000", 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7209999999999994e-06 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048903+0000", 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4538e-05 2024-09-16T03:39:01.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048912+0000", 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2390000000000008e-06 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149039+0000", 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100126563 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149044+0000", 2024-09-16T03:39:01.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0340000000000004e-06 2024-09-16T03:39:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451087+0000", 2024-09-16T03:39:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30204318000000002 2024-09-16T03:39:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451159+0000", 2024-09-16T03:39:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1627999999999999e-05 2024-09-16T03:39:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451321+0000", 2024-09-16T03:39:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016183699999999999 2024-09-16T03:39:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2106 4.a 4:51338165:::smithi15736696-4:head [sparse-read 0~887916] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:39:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:39:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.133122165000003, 2024-09-16T03:39:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40254453200000001, 2024-09-16T03:39:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2106 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:39:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048933+0000", 2024-09-16T03:39:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:39:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048936+0000", 2024-09-16T03:39:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3890000000000001e-06 2024-09-16T03:39:01.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048943+0000", 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1250000000000004e-06 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048953+0000", 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8190000000000003e-06 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149113+0000", 2024-09-16T03:39:01.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100160007 2024-09-16T03:39:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149118+0000", 2024-09-16T03:39:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7650000000000001e-06 2024-09-16T03:39:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451341+0000", 2024-09-16T03:39:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30222344699999998 2024-09-16T03:39:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451381+0000", 2024-09-16T03:39:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9517000000000001e-05 2024-09-16T03:39:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451477+0000", 2024-09-16T03:39:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5656999999999996e-05 2024-09-16T03:39:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.263 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2107 4.a 4:51338165:::smithi15736696-4:head [read 1698310~486600,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.133092101999999, 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40288642400000002, 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2107 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048963+0000", 2024-09-16T03:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048967+0000", 2024-09-16T03:39:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9060000000000004e-06 2024-09-16T03:39:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048972+0000", 2024-09-16T03:39:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5910000000000003e-06 2024-09-16T03:39:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048977+0000", 2024-09-16T03:39:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9629999999999997e-06 2024-09-16T03:39:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149135+0000", 2024-09-16T03:39:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015828 2024-09-16T03:39:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149138+0000", 2024-09-16T03:39:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.779e-06 2024-09-16T03:39:01.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451506+0000", 2024-09-16T03:39:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302367891 2024-09-16T03:39:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451558+0000", 2024-09-16T03:39:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2374999999999999e-05 2024-09-16T03:39:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451849+0000", 2024-09-16T03:39:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029009899999999999 2024-09-16T03:39:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2537 4.a 4:503187b3:::smithi15736696-34:head [assert-version v87,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:17.047079+0000", 2024-09-16T03:39:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.134975271999998, 2024-09-16T03:39:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.32976008000000001, 2024-09-16T03:39:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2537 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047079+0000", 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047079+0000", 2024-09-16T03:39:01.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047081+0000", 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6929999999999999e-06 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047086+0000", 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8680000000000004e-06 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047099+0000", 2024-09-16T03:39:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2309e-05 2024-09-16T03:39:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047105+0000", 2024-09-16T03:39:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7050000000000002e-06 2024-09-16T03:39:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047157+0000", 2024-09-16T03:39:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1057999999999998e-05 2024-09-16T03:39:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047219+0000", 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2553999999999998e-05 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.149371+0000", 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102152112 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.149846+0000", 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047467400000000002 2024-09-16T03:39:01.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.183065+0000", 2024-09-16T03:39:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033219374000000003 2024-09-16T03:39:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.376793+0000", 2024-09-16T03:39:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.193728023 2024-09-16T03:39:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.376819+0000", 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6142000000000001e-05 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.376839+0000", 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9568e-05 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2687 4.a 4:503187b3:::smithi15736696-34:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:21.869464+0000", 2024-09-16T03:39:01.275 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.312589828999997, 2024-09-16T03:39:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31565330200000002, 2024-09-16T03:39:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2687 2024-09-16T03:39:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869464+0000", 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869464+0000", 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869465+0000", 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.51e-07 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869467+0000", 2024-09-16T03:39:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:39:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869473+0000", 2024-09-16T03:39:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9560000000000002e-06 2024-09-16T03:39:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869488+0000", 2024-09-16T03:39:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5333e-05 2024-09-16T03:39:01.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869667+0000", 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017900699999999999 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869710+0000", 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2749e-05 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:22.185118+0000", 2024-09-16T03:39:01.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31540757600000002 2024-09-16T03:39:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2986 4.a 4:5a5e9bc2:::smithi15736696-36:head [write 487469~563838] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:39:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.912930+0000", 2024-09-16T03:39:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.269124652999999, 2024-09-16T03:39:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39506163799999999, 2024-09-16T03:39:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.280 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2986 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.912930+0000", 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.912930+0000", 2024-09-16T03:39:01.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.912932+0000", 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8169999999999999e-06 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913522+0000", 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058972099999999997 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913539+0000", 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7306999999999999e-05 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913550+0000", 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0356e-05 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913637+0000", 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7552999999999993e-05 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913699+0000", 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1991999999999999e-05 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913797+0000", 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7738000000000005e-05 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.915370+0000", 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015725719999999999 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.916260+0000", 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089050100000000003 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.307864+0000", 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39160404900000001 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.307936+0000", 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2025000000000002e-05 2024-09-16T03:39:01.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.307991+0000", 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5006999999999997e-05 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2987 4.a 4:5a5e9bc2:::smithi15736696-36:head [write 1607751~551752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.263106613000001, 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49279943700000001, 2024-09-16T03:39:01.286 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2987 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:39:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918949+0000", 2024-09-16T03:39:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.86e-06 2024-09-16T03:39:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919579+0000", 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062909000000000001 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919591+0000", 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2078e-05 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919596+0000", 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019705+0000", 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100108405 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019754+0000", 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9258999999999997e-05 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019847+0000", 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2954999999999993e-05 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.023003+0000", 2024-09-16T03:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031561110000000001 2024-09-16T03:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.122544+0000", 2024-09-16T03:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099540553000000004 2024-09-16T03:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411658+0000", 2024-09-16T03:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.28911426699999998 2024-09-16T03:39:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411696+0000", 2024-09-16T03:39:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8430000000000003e-05 2024-09-16T03:39:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411747+0000", 2024-09-16T03:39:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0745999999999998e-05 2024-09-16T03:39:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2988 4.a 4:5a5e9bc2:::smithi15736696-36:head [setxattr _header (56) in=7b,truncate 2487469] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:39:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:39:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.262444874, 2024-09-16T03:39:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59236646500000001, 2024-09-16T03:39:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.293 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2988 2024-09-16T03:39:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:39:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:39:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919610+0000", 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.048e-06 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919615+0000", 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8239999999999999e-06 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919621+0000", 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4380000000000003e-06 2024-09-16T03:39:01.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919626+0000", 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2000000000000002e-06 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.020922+0000", 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101295782 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.020995+0000", 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3256999999999993e-05 2024-09-16T03:39:01.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.021088+0000", 2024-09-16T03:39:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3419999999999997e-05 2024-09-16T03:39:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.023007+0000", 2024-09-16T03:39:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019186380000000001 2024-09-16T03:39:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.222530+0000", 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19952276899999999 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511887+0000", 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.289357378 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511927+0000", 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0357e-05 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511976+0000", 2024-09-16T03:39:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8353999999999998e-05 2024-09-16T03:39:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3258 4.1a 4:5e3c0760:::smithi15736696-24:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:39:01.299 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.751993999, 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3302292470000001, 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3258 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:39:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:39:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430061+0000", 2024-09-16T03:39:01.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.27e-07 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430062+0000", 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6780000000000001e-06 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430066+0000", 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5269999999999999e-06 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430069+0000", 2024-09-16T03:39:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:39:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430119+0000", 2024-09-16T03:39:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0212e-05 2024-09-16T03:39:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:39:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430122+0000", 2024-09-16T03:39:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7319999999999998e-06 2024-09-16T03:39:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.631493+0000", 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20137142499999999 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.631579+0000", 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5418000000000001e-05 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:41.760289+0000", 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1287106680000001 2024-09-16T03:39:01.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3701 4.a 4:51338165:::smithi15736696-4:head [assert-version v115,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e121)", 2024-09-16T03:39:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:39:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8111474599999999, 2024-09-16T03:39:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50414302200000005, 2024-09-16T03:39:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.305 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3701 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370910+0000", 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1200000000000002e-06 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370923+0000", 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3078000000000001e-05 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370940+0000", 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7461000000000001e-05 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370950+0000", 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0091999999999999e-05 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.371029+0000", 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8879000000000002e-05 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.371136+0000", 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106983 2024-09-16T03:39:01.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.573316+0000", 2024-09-16T03:39:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20217934500000001 2024-09-16T03:39:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.573914+0000", 2024-09-16T03:39:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059857900000000002 2024-09-16T03:39:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.763374+0000", 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18945932200000001 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.874936+0000", 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111562256 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.875002+0000", 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6050000000000006e-05 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.875050+0000", 2024-09-16T03:39:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7856999999999999e-05 2024-09-16T03:39:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.311 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.311 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:01.312 INFO:tasks.rados.rados.0.smithi157.stdout:1639: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-16T03:39:01.313 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:01.313 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:39:01.314 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:01.311+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:01.331 INFO:tasks.rados.rados.0.smithi157.stdout:1640: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-09-16T03:39:01.339 INFO:tasks.rados.rados.0.smithi157.stdout:1640: writing smithi15736696-31 from 732871 to 1448006 tid 1 2024-09-16T03:39:01.344 INFO:tasks.rados.rados.0.smithi157.stdout:1640: writing smithi15736696-31 from 1873673 to 2517780 tid 2 2024-09-16T03:39:01.353 INFO:tasks.rados.rados.0.smithi157.stdout:1640: writing smithi15736696-31 from 3244434 to 3932871 tid 3 2024-09-16T03:39:01.353 INFO:tasks.rados.rados.0.smithi157.stdout:1635: done (5 left) 2024-09-16T03:39:01.353 INFO:tasks.rados.rados.0.smithi157.stdout:1636: done (4 left) 2024-09-16T03:39:01.353 INFO:tasks.rados.rados.0.smithi157.stdout:1637: done (3 left) 2024-09-16T03:39:01.353 INFO:tasks.rados.rados.0.smithi157.stdout:1638: done (2 left) 2024-09-16T03:39:01.353 INFO:tasks.rados.rados.0.smithi157.stdout:1639: done (1 left) 2024-09-16T03:39:01.353 INFO:tasks.rados.rados.0.smithi157.stdout:1641: read oid 29 snap -1 2024-09-16T03:39:01.353 INFO:tasks.rados.rados.0.smithi157.stdout:1641: expect (ObjNum 344 snap 0 seq_num 344) 2024-09-16T03:39:01.353 INFO:tasks.rados.rados.0.smithi157.stdout:1642: read oid 36 snap -1 2024-09-16T03:39:01.353 INFO:tasks.rados.rados.0.smithi157.stdout:1642: expect (ObjNum 306 snap 0 seq_num 306) 2024-09-16T03:39:01.353 INFO:tasks.rados.rados.0.smithi157.stdout:1643: read oid 50 snap -1 2024-09-16T03:39:01.354 INFO:tasks.rados.rados.0.smithi157.stdout:1643: expect (ObjNum 350 snap 0 seq_num 350) 2024-09-16T03:39:01.354 INFO:tasks.rados.rados.0.smithi157.stdout:1644: delete oid 42 current snap is 0 2024-09-16T03:39:01.367 INFO:tasks.rados.rados.0.smithi157.stdout:1641: expect (ObjNum 344 snap 0 seq_num 344) 2024-09-16T03:39:01.367 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 314163 2024-09-16T03:39:01.413 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:01.412+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:01.441 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:01.461 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:01.461 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:01.461 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:39:01.461 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:01.470 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:39:01.494 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:01.493+0000 7f928a16a640 -1 osd.14 126 handle_osd_map injecting map crc failure 2024-09-16T03:39:01.495 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:39:01.495 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T03:39:01.514 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:01.512+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:01.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-16T03:39:01.520 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 213073 2024-09-16T03:39:01.583 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:01.583 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:01.583 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:01.583 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:01.583 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:01.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:39:01.596 INFO:tasks.rados.rados.0.smithi157.stdout:1642: expect (ObjNum 306 snap 0 seq_num 306) 2024-09-16T03:39:01.615 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:01.613+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:01.678 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:01.716 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:01.713+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:01.726 INFO:tasks.rados.rados.0.smithi157.stdout:1643: expect (ObjNum 350 snap 0 seq_num 350) 2024-09-16T03:39:01.746 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:432 4.c 4:34492f38:::smithi15736696-33:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:39:01.746 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 100.82963463900001, 2024-09-16T03:39:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.404460861, 2024-09-16T03:39:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 432 2024-09-16T03:39:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.747 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913833+0000", 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0610000000000001e-06 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913839+0000", 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9479999999999996e-06 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913848+0000", 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5629999999999992e-06 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913857+0000", 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0399999999999998e-06 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914022+0000", 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000165422 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914101+0000", 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8462999999999997e-05 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217171+0000", 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30307040000000002 2024-09-16T03:39:01.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217854+0000", 2024-09-16T03:39:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068278700000000002 2024-09-16T03:39:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.218098+0000", 2024-09-16T03:39:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243825 2024-09-16T03:39:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318164+0000", 2024-09-16T03:39:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100066451 2024-09-16T03:39:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318232+0000", 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8113000000000003e-05 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318293+0000", 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0788000000000001e-05 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:790 4.d 4:b3fe05b2:::smithi15736696-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:01.753 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:39:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 95.855494565000001, 2024-09-16T03:39:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41456747799999999, 2024-09-16T03:39:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 790 2024-09-16T03:39:01.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:39:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:39:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887975+0000", 2024-09-16T03:39:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0369999999999999e-06 2024-09-16T03:39:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887987+0000", 2024-09-16T03:39:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1606e-05 2024-09-16T03:39:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888016+0000", 2024-09-16T03:39:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.917e-05 2024-09-16T03:39:01.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888032+0000", 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5616000000000001e-05 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888178+0000", 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612100000000001 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888307+0000", 2024-09-16T03:39:01.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012953899999999999 2024-09-16T03:39:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.302540+0000", 2024-09-16T03:39:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41423238899999998 2024-09-16T03:39:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1418 4.c 4:3313e97c:::smithi15736696-21:head [write 615156~555834] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:39:01.758 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.517383393000003, 2024-09-16T03:39:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40891108799999998, 2024-09-16T03:39:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1418 2024-09-16T03:39:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226087+0000", 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8890000000000004e-06 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226827+0000", 2024-09-16T03:39:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000739298 2024-09-16T03:39:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226848+0000", 2024-09-16T03:39:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1418000000000001e-05 2024-09-16T03:39:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226858+0000", 2024-09-16T03:39:01.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0434e-05 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327036+0000", 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10017769 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327098+0000", 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1760999999999994e-05 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:01.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327230+0000", 2024-09-16T03:39:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131839 2024-09-16T03:39:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434369+0000", 2024-09-16T03:39:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107139788 2024-09-16T03:39:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434444+0000", 2024-09-16T03:39:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4171999999999994e-05 2024-09-16T03:39:01.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634830+0000", 2024-09-16T03:39:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20038660599999999 2024-09-16T03:39:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634937+0000", 2024-09-16T03:39:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010716499999999999 2024-09-16T03:39:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634994+0000", 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7027999999999997e-05 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1419 4.c 4:3313e97c:::smithi15736696-21:head [write 1603467~416361] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.512718004000007, 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40430432100000002, 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.765 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1419 2024-09-16T03:39:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:39:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230752+0000", 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.354e-06 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231164+0000", 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041208599999999998 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231179+0000", 2024-09-16T03:39:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4885999999999999e-05 2024-09-16T03:39:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231186+0000", 2024-09-16T03:39:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8759999999999999e-06 2024-09-16T03:39:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328726+0000", 2024-09-16T03:39:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097539855999999994 2024-09-16T03:39:01.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328786+0000", 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0059000000000002e-05 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328890+0000", 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104338 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434475+0000", 2024-09-16T03:39:01.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.105584709 2024-09-16T03:39:01.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.534594+0000", 2024-09-16T03:39:01.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10011914099999999 2024-09-16T03:39:01.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635003+0000", 2024-09-16T03:39:01.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100408834 2024-09-16T03:39:01.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635029+0000", 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6551999999999999e-05 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635053+0000", 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3629999999999999e-05 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.771 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1421 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.504035243999994, 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.497492088, 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1421 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239432+0000", 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5999999999999991e-07 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239436+0000", 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239443+0000", 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.968e-06 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239447+0000", 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6969999999999999e-06 2024-09-16T03:39:01.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635225+0000", 2024-09-16T03:39:01.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39577807399999998 2024-09-16T03:39:01.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635290+0000", 2024-09-16T03:39:01.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4583000000000002e-05 2024-09-16T03:39:01.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.775 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:01.775 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635410+0000", 2024-09-16T03:39:01.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120684 2024-09-16T03:39:01.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636180+0000", 2024-09-16T03:39:01.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076997899999999995 2024-09-16T03:39:01.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636799+0000", 2024-09-16T03:39:01.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061906600000000004 2024-09-16T03:39:01.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736742+0000", 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099943041999999996 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736843+0000", 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010076799999999999 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736924+0000", 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0419000000000004e-05 2024-09-16T03:39:01.777 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.778 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.778 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.778 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1422 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:01.778 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:01.778 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.504013963999995, 2024-09-16T03:39:01.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49764818700000002, 2024-09-16T03:39:01.778 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.778 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.778 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:01.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.778 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1422 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239455+0000", 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9860000000000001e-06 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239461+0000", 2024-09-16T03:39:01.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:39:01.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239464+0000", 2024-09-16T03:39:01.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2619999999999999e-06 2024-09-16T03:39:01.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635736+0000", 2024-09-16T03:39:01.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.396272287 2024-09-16T03:39:01.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635790+0000", 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4252000000000002e-05 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736964+0000", 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101173319 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737024+0000", 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9975000000000001e-05 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737101+0000", 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7303000000000002e-05 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2336 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.648426846, 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.407180869, 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.783 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.784 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:01.784 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.784 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.784 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.784 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2336 2024-09-16T03:39:01.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.784 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:39:01.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2399999999999999e-07 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095042+0000", 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.567e-06 2024-09-16T03:39:01.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095046+0000", 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.639e-06 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195114+0000", 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10006748 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195173+0000", 2024-09-16T03:39:01.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8727000000000002e-05 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195278+0000", 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010522900000000001 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.501699+0000", 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30642159299999999 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.501995+0000", 2024-09-16T03:39:01.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000295822 2024-09-16T03:39:01.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.502221+0000", 2024-09-16T03:39:01.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022548800000000001 2024-09-16T03:39:01.788 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.788 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.788 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.788 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2452 4.c 4:34492f38:::smithi15736696-33:head [setxattr _header (56) in=7b,truncate 3479049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:39:01.788 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:15.463009+0000", 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.280457533000003, 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40224235800000002, 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2452 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463009+0000", 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463009+0000", 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463010+0000", 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6000000000000004e-07 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463012+0000", 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5100000000000001e-06 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463017+0000", 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5959999999999997e-06 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463020+0000", 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2339999999999999e-06 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.465322+0000", 2024-09-16T03:39:01.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002301464 2024-09-16T03:39:01.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.465374+0000", 2024-09-16T03:39:01.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2945999999999998e-05 2024-09-16T03:39:01.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:01.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.465470+0000", 2024-09-16T03:39:01.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5445000000000005e-05 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.560795+0000", 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095324675999999997 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.566503+0000", 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0057087659999999997 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.566596+0000", 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3107000000000002e-05 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865113+0000", 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29851622100000003 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865160+0000", 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6863000000000002e-05 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865252+0000", 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1970000000000002e-05 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2453 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:15.463024+0000", 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.280442549, 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40260837500000002, 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.795 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2453 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463024+0000", 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463024+0000", 2024-09-16T03:39:01.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463025+0000", 2024-09-16T03:39:01.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4499999999999998e-07 2024-09-16T03:39:01.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463026+0000", 2024-09-16T03:39:01.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4500000000000001e-06 2024-09-16T03:39:01.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463030+0000", 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5449999999999997e-06 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.463034+0000", 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0309999999999999e-06 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.560713+0000", 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097679777999999995 2024-09-16T03:39:01.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.560765+0000", 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2021000000000002e-05 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865314+0000", 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30454909000000002 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865423+0000", 2024-09-16T03:39:01.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000108342 2024-09-16T03:39:01.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:15.865633+0000", 2024-09-16T03:39:01.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020977300000000001 2024-09-16T03:39:01.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.800 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.800 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.800 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2642 4.c 4:34492f38:::smithi15736696-33:head [write 665610~797860,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:01.800 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:39:01.800 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.867280100999999, 2024-09-16T03:39:01.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40488026700000002, 2024-09-16T03:39:01.801 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.801 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.801 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.801 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.801 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.801 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.801 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2642 2024-09-16T03:39:01.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.801 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876189+0000", 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0549999999999998e-06 2024-09-16T03:39:01.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877003+0000", 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081467900000000001 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877018+0000", 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.471e-05 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877025+0000", 2024-09-16T03:39:01.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.379e-06 2024-09-16T03:39:01.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977237+0000", 2024-09-16T03:39:01.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100211914 2024-09-16T03:39:01.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977354+0000", 2024-09-16T03:39:01.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116416 2024-09-16T03:39:01.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977480+0000", 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012629899999999999 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.984474+0000", 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069943230000000002 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084635+0000", 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100161018 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185809+0000", 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10117406700000001 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.280666+0000", 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094856491000000001 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.280857+0000", 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019133999999999999 2024-09-16T03:39:01.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.281067+0000", 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000209576 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2643 4.c 4:34492f38:::smithi15736696-33:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.862613215000003, 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50004631700000002, 2024-09-16T03:39:01.807 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.808 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.808 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.808 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.808 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.808 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.808 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2643 2024-09-16T03:39:01.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.808 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:39:01.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880854+0000", 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6599999999999994e-07 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881441+0000", 2024-09-16T03:39:01.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058613000000000003 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881450+0000", 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6260000000000007e-06 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881455+0000", 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.515e-06 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979409+0000", 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097954645000000007 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979512+0000", 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102701 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979684+0000", 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017176299999999999 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084699+0000", 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10501545599999999 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084714+0000", 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4738e-05 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185858+0000", 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101144074 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380720+0000", 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.194862072 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380783+0000", 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3262000000000005e-05 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380900+0000", 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116369 2024-09-16T03:39:01.813 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2644 4.c 4:34492f38:::smithi15736696-33:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.857936719999998, 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.495569859, 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.814 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2644 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885531+0000", 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885813+0000", 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028188399999999999 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885824+0000", 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-16T03:39:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885830+0000", 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6289999999999998e-06 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983163+0000", 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097333127000000005 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983273+0000", 2024-09-16T03:39:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109868 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983458+0000", 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018554899999999999 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084722+0000", 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101263244 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185717+0000", 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100995575 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185904+0000", 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186682 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380965+0000", 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.195060921 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381030+0000", 2024-09-16T03:39:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5174999999999999e-05 2024-09-16T03:39:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381100+0000", 2024-09-16T03:39:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9877999999999997e-05 2024-09-16T03:39:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2645 4.c 4:34492f38:::smithi15736696-33:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.857627219000001, 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59563304800000005, 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2645 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885840+0000", 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885843+0000", 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.785e-06 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885848+0000", 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3979999999999998e-06 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885852+0000", 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6509999999999999e-06 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084768+0000", 2024-09-16T03:39:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19891652000000001 2024-09-16T03:39:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084831+0000", 2024-09-16T03:39:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2717000000000003e-05 2024-09-16T03:39:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084964+0000", 2024-09-16T03:39:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132731 2024-09-16T03:39:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185772+0000", 2024-09-16T03:39:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10080797399999999 2024-09-16T03:39:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185790+0000", 2024-09-16T03:39:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7966999999999999e-05 2024-09-16T03:39:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185959+0000", 2024-09-16T03:39:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168777 2024-09-16T03:39:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481235+0000", 2024-09-16T03:39:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29527644800000002 2024-09-16T03:39:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481307+0000", 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1539999999999996e-05 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481473+0000", 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016595600000000001 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2646 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.857610530000002, 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59596245699999995, 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2646 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885857+0000", 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885860+0000", 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8940000000000003e-06 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885864+0000", 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.602e-06 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885867+0000", 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0089999999999999e-06 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085196+0000", 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19932923399999999 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085287+0000", 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0480000000000001e-05 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481521+0000", 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39623439799999999 2024-09-16T03:39:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481613+0000", 2024-09-16T03:39:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2276999999999998e-05 2024-09-16T03:39:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481819+0000", 2024-09-16T03:39:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020521499999999999 2024-09-16T03:39:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3554 4.c 4.3e97c8cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.293437505, 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2069473150000001, 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3554 2024-09-16T03:39:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450032+0000", 2024-09-16T03:39:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9639999999999999e-06 2024-09-16T03:39:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450041+0000", 2024-09-16T03:39:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-16T03:39:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450059+0000", 2024-09-16T03:39:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-16T03:39:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450069+0000", 2024-09-16T03:39:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8320000000000001e-06 2024-09-16T03:39:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.467620+0000", 2024-09-16T03:39:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017550676000000001 2024-09-16T03:39:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.467626+0000", 2024-09-16T03:39:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3430000000000001e-06 2024-09-16T03:39:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656941+0000", 2024-09-16T03:39:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1893151820000001 2024-09-16T03:39:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656977+0000", 2024-09-16T03:39:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5491999999999998e-05 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3581 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.717930691999999, 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.63145314900000005, 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3581 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025539+0000", 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4510000000000001e-06 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026168+0000", 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062808199999999997 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026186+0000", 2024-09-16T03:39:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8694000000000001e-05 2024-09-16T03:39:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026196+0000", 2024-09-16T03:39:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8800000000000003e-06 2024-09-16T03:39:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035407+0000", 2024-09-16T03:39:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092111020000000005 2024-09-16T03:39:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035416+0000", 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9870000000000005e-06 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.557740+0000", 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.522323918 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656989+0000", 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099249034999999999 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3582 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.710688847, 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62429066099999997, 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3582 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032779+0000", 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2720000000000001e-06 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035239+0000", 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024594180000000001 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035261+0000", 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2246e-05 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035273+0000", 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2289999999999999e-05 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038105+0000", 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028319780000000002 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038111+0000", 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3099999999999997e-06 2024-09-16T03:39:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657045+0000", 2024-09-16T03:39:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61893337199999998 2024-09-16T03:39:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657069+0000", 2024-09-16T03:39:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3774999999999999e-05 2024-09-16T03:39:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3583 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:39:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.705580717, 2024-09-16T03:39:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61919224299999998, 2024-09-16T03:39:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3583 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037888+0000", 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-06 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038032+0000", 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014443799999999999 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038045+0000", 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3417999999999999e-05 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038052+0000", 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3890000000000002e-06 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038133+0000", 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0856e-05 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038136+0000", 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7160000000000001e-06 2024-09-16T03:39:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657073+0000", 2024-09-16T03:39:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61893632899999995 2024-09-16T03:39:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657078+0000", 2024-09-16T03:39:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4770000000000002e-06 2024-09-16T03:39:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3584 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:39:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.705402423000001, 2024-09-16T03:39:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61903161699999998, 2024-09-16T03:39:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3584 2024-09-16T03:39:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038065+0000", 2024-09-16T03:39:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3499999999999996e-07 2024-09-16T03:39:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038069+0000", 2024-09-16T03:39:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:39:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038075+0000", 2024-09-16T03:39:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0549999999999996e-06 2024-09-16T03:39:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038078+0000", 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4939999999999999e-06 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.126379+0000", 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088300141999999998 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.126393+0000", 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4827999999999999e-05 2024-09-16T03:39:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657092+0000", 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.53069909500000001 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657096+0000", 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4139999999999998e-06 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3585 4.2c 4.1cf4922c (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.705381862999999, 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61902921799999999, 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3585 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1899999999999998e-07 2024-09-16T03:39:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038087+0000", 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038091+0000", 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1590000000000003e-06 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038095+0000", 2024-09-16T03:39:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5779999999999999e-06 2024-09-16T03:39:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.138249+0000", 2024-09-16T03:39:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100154101 2024-09-16T03:39:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.138256+0000", 2024-09-16T03:39:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4209999999999998e-06 2024-09-16T03:39:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657111+0000", 2024-09-16T03:39:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.51885431800000004 2024-09-16T03:39:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657114+0000", 2024-09-16T03:39:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5190000000000001e-06 2024-09-16T03:39:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:01.862 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:01.863 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:39:01.864 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:01.862+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:01.865 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:39:01.964 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:01.962+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:01.978 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:01.978 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:01.979 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:01.979 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:01.988 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:39:02.031 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:02.031 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:02.048 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:02.049 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:02.065 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:02.063+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:02.065 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:02.066 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:02.082 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:02.083 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:02.092 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:02.092 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:02.098 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:02.098 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:02.101 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:39:02.115 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:02.116 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:02.132 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:02.133 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:02.144 INFO:tasks.rados.rados.0.smithi157.stdout:1641: done (4 left) 2024-09-16T03:39:02.144 INFO:tasks.rados.rados.0.smithi157.stdout:1642: done (3 left) 2024-09-16T03:39:02.144 INFO:tasks.rados.rados.0.smithi157.stdout:1643: done (2 left) 2024-09-16T03:39:02.144 INFO:tasks.rados.rados.0.smithi157.stdout:1644: done (1 left) 2024-09-16T03:39:02.144 INFO:tasks.rados.rados.0.smithi157.stdout:1645: read oid 26 snap -1 2024-09-16T03:39:02.145 INFO:tasks.rados.rados.0.smithi157.stdout:1645: expect (ObjNum 336 snap 0 seq_num 336) 2024-09-16T03:39:02.145 INFO:tasks.rados.rados.0.smithi157.stdout:1646: write oid 50 current snap is 0 2024-09-16T03:39:02.145 INFO:tasks.rados.rados.0.smithi157.stdout:1646: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2024-09-16T03:39:02.150 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:02.150 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:02.151 INFO:tasks.rados.rados.0.smithi157.stdout:1646: writing smithi15736696-50 from 766485 to 1232118 tid 1 2024-09-16T03:39:02.158 INFO:tasks.rados.rados.0.smithi157.stdout:1646: writing smithi15736696-50 from 1990637 to 2638413 tid 2 2024-09-16T03:39:02.166 INFO:tasks.rados.rados.0.smithi157.stdout:1646: writing smithi15736696-50 from 3299280 to 3966485 tid 3 2024-09-16T03:39:02.166 INFO:tasks.rados.rados.0.smithi157.stdout:1647: read oid 36 snap -1 2024-09-16T03:39:02.166 INFO:tasks.rados.rados.0.smithi157.stdout:1647: expect (ObjNum 306 snap 0 seq_num 306) 2024-09-16T03:39:02.166 INFO:tasks.rados.rados.0.smithi157.stdout:1648: tier_evict oid 30 2024-09-16T03:39:02.166 INFO:tasks.rados.rados.0.smithi157.stdout:1648: tier_evict oid 30 snap -1 2024-09-16T03:39:02.166 INFO:tasks.rados.rados.0.smithi157.stdout:1649: write (excl) oid 37 current snap is 0 2024-09-16T03:39:02.167 INFO:tasks.rados.rados.0.smithi157.stdout:1649: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-09-16T03:39:02.167 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:02.167 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:02.173 INFO:tasks.rados.rados.0.smithi157.stdout:1649: writing smithi15736696-37 from 783292 to 1324174 tid 1 2024-09-16T03:39:02.179 INFO:tasks.rados.rados.0.smithi157.stdout:1649: writing smithi15736696-37 from 1932766 to 2440553 tid 2 2024-09-16T03:39:02.183 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:02.183 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:02.184 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:02.183+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:02.186 INFO:tasks.rados.rados.0.smithi157.stdout:1649: writing smithi15736696-37 from 3210350 to 3782363 tid 3 2024-09-16T03:39:02.186 INFO:tasks.rados.rados.0.smithi157.stdout:1650: write (excl) oid 1 current snap is 0 2024-09-16T03:39:02.186 INFO:tasks.rados.rados.0.smithi157.stdout:1650: seq_num 357 ranges {400099=616131,1591248=408851} 2024-09-16T03:39:02.191 INFO:tasks.rados.rados.0.smithi157.stdout:1650: writing smithi15736696-1 from 400099 to 1016230 tid 1 2024-09-16T03:39:02.197 INFO:tasks.rados.rados.0.smithi157.stdout:1650: writing smithi15736696-1 from 1591248 to 2000099 tid 2 2024-09-16T03:39:02.197 INFO:tasks.rados.rados.0.smithi157.stdout:1651: read oid 11 snap -1 2024-09-16T03:39:02.197 INFO:tasks.rados.rados.0.smithi157.stdout:1651: expect deleted 2024-09-16T03:39:02.197 INFO:tasks.rados.rados.0.smithi157.stdout:1652: read oid 6 snap -1 2024-09-16T03:39:02.197 INFO:tasks.rados.rados.0.smithi157.stdout:1652: expect (ObjNum 327 snap 0 seq_num 327) 2024-09-16T03:39:02.197 INFO:tasks.rados.rados.0.smithi157.stdout:1653: set_chunk oid 24 target oid 2024-09-16T03:39:02.199 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:02.200 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:02.219 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:02.235 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:02.245 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:02.245 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:02.254 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:39:02.254 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:02.255 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:02.266 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:02.264+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:02.270 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:02.270 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:02.286 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:02.287 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:02.353 INFO:tasks.rados.rados.0.smithi157.stdout:1653: set_chunk oid 24 offset: 1093120 length: 1340928 target oid 8fb6b06dd7f3bef147dafbed5016ad10a1b8fef579b1c769b68bf3848766f038 offset: 0 snap -1 2024-09-16T03:39:02.353 INFO:tasks.rados.rados.0.smithi157.stdout:1654: read oid 12 snap -1 2024-09-16T03:39:02.353 INFO:tasks.rados.rados.0.smithi157.stdout:1654: expect (ObjNum 297 snap 0 seq_num 297) 2024-09-16T03:39:02.353 INFO:tasks.rados.rados.0.smithi157.stdout:1655: read oid 20 snap -1 2024-09-16T03:39:02.354 INFO:tasks.rados.rados.0.smithi157.stdout:1655: expect (ObjNum 346 snap 0 seq_num 346) 2024-09-16T03:39:02.354 INFO:tasks.rados.rados.0.smithi157.stdout:1656: write oid 47 current snap is 0 2024-09-16T03:39:02.354 INFO:tasks.rados.rados.0.smithi157.stdout:1656: seq_num 358 ranges {416906=407733,1366083=511456} 2024-09-16T03:39:02.358 INFO:tasks.rados.rados.0.smithi157.stdout:1656: writing smithi15736696-47 from 416906 to 824639 tid 1 2024-09-16T03:39:02.363 INFO:tasks.rados.rados.0.smithi157.stdout:1656: writing smithi15736696-47 from 1366083 to 1877539 tid 2 2024-09-16T03:39:02.364 INFO:tasks.rados.rados.0.smithi157.stdout:1657: set_chunk oid 4 target oid 2024-09-16T03:39:02.367 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:02.366+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:02.398 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:02.399 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:02.408 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:39:02.468 INFO:tasks.rados.rados.0.smithi157.stdout:1657: set_chunk oid 4 offset: 607744 length: 101888 target oid d09d36d885ae3d54d38e03e74a0335a61d978a75804ba25b8bbd603f0716a2d0 offset: 0 snap -1 2024-09-16T03:39:02.468 INFO:tasks.rados.rados.0.smithi157.stdout:1658: write oid 8 current snap is 0 2024-09-16T03:39:02.468 INFO:tasks.rados.rados.0.smithi157.stdout:1658: seq_num 359 ranges {433713=482982,1424565=609148} 2024-09-16T03:39:02.469 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:02.465+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:02.473 INFO:tasks.rados.rados.0.smithi157.stdout:1658: writing smithi15736696-8 from 433713 to 916695 tid 1 2024-09-16T03:39:02.479 INFO:tasks.rados.rados.0.smithi157.stdout:1658: writing smithi15736696-8 from 1424565 to 2033713 tid 2 2024-09-16T03:39:02.479 INFO:tasks.rados.rados.0.smithi157.stdout:1659: delete oid 35 current snap is 0 2024-09-16T03:39:02.479 INFO:tasks.rados.rados.0.smithi157.stdout:1647: expect (ObjNum 306 snap 0 seq_num 306) 2024-09-16T03:39:02.505 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T03:39:02.512 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:02.512 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:02.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-16T03:39:02.521 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:39:02.527 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:02.569 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:02.567+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:02.570 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:02.568+0000 7f4212d00640 -1 osd.8 127 handle_osd_map injecting map crc failure 2024-09-16T03:39:02.624 INFO:tasks.rados.rados.0.smithi157.stdout:1652: expect (ObjNum 327 snap 0 seq_num 327) 2024-09-16T03:39:02.624 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:02.625 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:02.625 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:02.634 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:39:02.668 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 801345 2024-09-16T03:39:02.670 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:02.667+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:02.725 INFO:tasks.rados.rados.0.smithi157.stdout:1645: expect (ObjNum 336 snap 0 seq_num 336) 2024-09-16T03:39:02.725 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:02.771 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:02.769+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:02.778 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:02.778 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:02.787 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:39:02.841 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 763071 2024-09-16T03:39:02.871 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:02.870+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:02.890 INFO:tasks.rados.rados.0.smithi157.stdout:1648: got (22) Invalid argument 2024-09-16T03:39:02.890 INFO:tasks.rados.rados.0.smithi157.stdout:1649: finishing write tid 1 to smithi15736696-37 2024-09-16T03:39:02.890 INFO:tasks.rados.rados.0.smithi157.stdout:1640: finishing write tid 1 to smithi15736696-31 2024-09-16T03:39:02.890 INFO:tasks.rados.rados.0.smithi157.stdout:1640: finishing write tid 2 to smithi15736696-31 2024-09-16T03:39:02.890 INFO:tasks.rados.rados.0.smithi157.stdout:1640: finishing write tid 3 to smithi15736696-31 2024-09-16T03:39:02.890 INFO:tasks.rados.rados.0.smithi157.stdout:1646: finishing write tid 1 to smithi15736696-50 2024-09-16T03:39:02.890 INFO:tasks.rados.rados.0.smithi157.stdout:1655: expect (ObjNum 346 snap 0 seq_num 346) 2024-09-16T03:39:02.900 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 887956 2024-09-16T03:39:02.930 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:02.930 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:02.939 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:39:02.973 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:02.971+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:03.020 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2731548 2024-09-16T03:39:03.073 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:03.071+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:03.083 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:03.083 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:03.092 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:39:03.139 INFO:tasks.rados.rados.0.smithi157.stdout:1650: finishing write tid 1 to smithi15736696-1 2024-09-16T03:39:03.139 INFO:tasks.rados.rados.0.smithi157.stdout:1650: finishing write tid 2 to smithi15736696-1 2024-09-16T03:39:03.139 INFO:tasks.rados.rados.0.smithi157.stdout:1650: finishing write tid 3 to smithi15736696-1 2024-09-16T03:39:03.139 INFO:tasks.rados.rados.0.smithi157.stdout:1650: finishing write tid 4 to smithi15736696-1 2024-09-16T03:39:03.139 INFO:tasks.rados.rados.0.smithi157.stdout:1650: oid 1 updating version 0 to 19 2024-09-16T03:39:03.139 INFO:tasks.rados.rados.0.smithi157.stdout:1650: oid 1 version 19 is already newer than 18 2024-09-16T03:39:03.139 INFO:tasks.rados.rados.0.smithi157.stdout:1650: oid 1 updating version 19 to 20 2024-09-16T03:39:03.139 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 20 (ObjNum 357 snap 0 seq_num 357) dirty exists 2024-09-16T03:39:03.140 INFO:tasks.rados.rados.0.smithi157.stdout:1650: left oid 1 (ObjNum 357 snap 0 seq_num 357) 2024-09-16T03:39:03.140 INFO:tasks.rados.rados.0.smithi157.stdout:1640: finishing write tid 4 to smithi15736696-31 2024-09-16T03:39:03.140 INFO:tasks.rados.rados.0.smithi157.stdout:1645: done (15 left) 2024-09-16T03:39:03.140 INFO:tasks.rados.rados.0.smithi157.stdout:1647: done (14 left) 2024-09-16T03:39:03.140 INFO:tasks.rados.rados.0.smithi157.stdout:1648: done (13 left) 2024-09-16T03:39:03.140 INFO:tasks.rados.rados.0.smithi157.stdout:1650: done (12 left) 2024-09-16T03:39:03.140 INFO:tasks.rados.rados.0.smithi157.stdout:1652: done (11 left) 2024-09-16T03:39:03.140 INFO:tasks.rados.rados.0.smithi157.stdout:1655: done (10 left) 2024-09-16T03:39:03.140 INFO:tasks.rados.rados.0.smithi157.stdout:1659: done (9 left) 2024-09-16T03:39:03.140 INFO:tasks.rados.rados.0.smithi157.stdout:1660: write (excl) oid 30 current snap is 0 2024-09-16T03:39:03.140 INFO:tasks.rados.rados.0.smithi157.stdout:1649: finishing write tid 2 to smithi15736696-37 2024-09-16T03:39:03.140 INFO:tasks.rados.rados.0.smithi157.stdout:1660: seq_num 361 ranges {467327=633480,1825176=242151} 2024-09-16T03:39:03.144 INFO:tasks.rados.rados.0.smithi157.stdout:1660: writing smithi15736696-30 from 467327 to 1100807 tid 1 2024-09-16T03:39:03.149 INFO:tasks.rados.rados.0.smithi157.stdout:1660: writing smithi15736696-30 from 1825176 to 2067327 tid 2 2024-09-16T03:39:03.149 INFO:tasks.rados.rados.0.smithi157.stdout:1661: set_chunk oid 13 target oid 2024-09-16T03:39:03.174 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:03.173+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:03.195 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:03.195 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:03.203 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:39:03.275 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:03.274+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:03.307 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:03.307 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:03.316 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:39:03.376 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:03.375+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:03.417 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:03.418 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:03.426 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:39:03.477 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:03.476+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:03.515 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:03.514+0000 7f9288166640 -1 osd.14 128 handle_osd_map injecting map crc failure 2024-09-16T03:39:03.515 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T03:39:03.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:39:03.531 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-16T03:39:03.534 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:03.534 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:03.543 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:39:03.579 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:03.577+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:03.648 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:03.649 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:03.657 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:39:03.679 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:03.677+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:03.729 INFO:tasks.rados.rados.0.smithi157.stdout:1661: set_chunk oid 13 offset: 534016 length: 1922560 target oid c28523a08e83933e647e0d3d9aca883355ae6f10e8f59d40bbb26ddc7302adf0 offset: 0 snap -1 2024-09-16T03:39:03.729 INFO:tasks.rados.rados.0.smithi157.stdout:1662: read oid 5 snap -1 2024-09-16T03:39:03.729 INFO:tasks.rados.rados.0.smithi157.stdout:1662: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-16T03:39:03.729 INFO:tasks.rados.rados.0.smithi157.stdout:1654: expect (ObjNum 297 snap 0 seq_num 297) 2024-09-16T03:39:03.749 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2131086 2024-09-16T03:39:03.763 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:03.763 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:03.772 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:39:03.780 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:03.777+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:03.782 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1924536 2024-09-16T03:39:03.806 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1032281 2024-09-16T03:39:03.841 INFO:tasks.rados.rados.0.smithi157.stdout:1656: finishing write tid 1 to smithi15736696-47 2024-09-16T03:39:03.842 INFO:tasks.rados.rados.0.smithi157.stdout:1656: finishing write tid 2 to smithi15736696-47 2024-09-16T03:39:03.842 INFO:tasks.rados.rados.0.smithi157.stdout:1656: finishing write tid 3 to smithi15736696-47 2024-09-16T03:39:03.842 INFO:tasks.rados.rados.0.smithi157.stdout:1656: finishing write tid 4 to smithi15736696-47 2024-09-16T03:39:03.842 INFO:tasks.rados.rados.0.smithi157.stdout:1656: oid 47 updating version 0 to 127 2024-09-16T03:39:03.842 INFO:tasks.rados.rados.0.smithi157.stdout:1656: oid 47 updating version 127 to 128 2024-09-16T03:39:03.842 INFO:tasks.rados.rados.0.smithi157.stdout:1656: oid 47 updating version 128 to 129 2024-09-16T03:39:03.842 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 129 (ObjNum 358 snap 0 seq_num 358) dirty exists 2024-09-16T03:39:03.842 INFO:tasks.rados.rados.0.smithi157.stdout:1656: left oid 47 (ObjNum 358 snap 0 seq_num 358) 2024-09-16T03:39:03.842 INFO:tasks.rados.rados.0.smithi157.stdout:1649: finishing write tid 3 to smithi15736696-37 2024-09-16T03:39:03.843 INFO:tasks.rados.rados.0.smithi157.stdout:1663: read oid 27 snap -1 2024-09-16T03:39:03.843 INFO:tasks.rados.rados.0.smithi157.stdout:1663: expect (ObjNum 337 snap 0 seq_num 337) 2024-09-16T03:39:03.843 INFO:tasks.rados.rados.0.smithi157.stdout:1654: done (12 left) 2024-09-16T03:39:03.843 INFO:tasks.rados.rados.0.smithi157.stdout:1656: done (11 left) 2024-09-16T03:39:03.843 INFO:tasks.rados.rados.0.smithi157.stdout:1664: read oid 42 snap -1 2024-09-16T03:39:03.843 INFO:tasks.rados.rados.0.smithi157.stdout:1664: expect deleted 2024-09-16T03:39:03.843 INFO:tasks.rados.rados.0.smithi157.stdout:1665: read oid 19 snap -1 2024-09-16T03:39:03.843 INFO:tasks.rados.rados.0.smithi157.stdout:1665: expect (ObjNum 331 snap 0 seq_num 331) 2024-09-16T03:39:03.843 INFO:tasks.rados.rados.0.smithi157.stdout:1666: write (excl) oid 47 current snap is 0 2024-09-16T03:39:03.843 INFO:tasks.rados.rados.0.smithi157.stdout:1649: finishing write tid 4 to smithi15736696-37 2024-09-16T03:39:03.843 INFO:tasks.rados.rados.0.smithi157.stdout:1666: seq_num 362 ranges {484134=708729,1883658=200476} 2024-09-16T03:39:03.846 INFO:tasks.rados.rados.0.smithi157.stdout:1666: writing smithi15736696-47 from 484134 to 1192863 tid 1 2024-09-16T03:39:03.852 INFO:tasks.rados.rados.0.smithi157.stdout:1666: writing smithi15736696-47 from 1883658 to 2084134 tid 2 2024-09-16T03:39:03.852 INFO:tasks.rados.rados.0.smithi157.stdout:1667: read oid 10 snap -1 2024-09-16T03:39:03.852 INFO:tasks.rados.rados.0.smithi157.stdout:1667: expect (ObjNum 276 snap 0 seq_num 276) 2024-09-16T03:39:03.852 INFO:tasks.rados.rados.0.smithi157.stdout:1668: read oid 48 snap -1 2024-09-16T03:39:03.852 INFO:tasks.rados.rados.0.smithi157.stdout:1668: expect (ObjNum 349 snap 0 seq_num 349) 2024-09-16T03:39:03.852 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:39:03.852 INFO:tasks.rados.rados.0.smithi157.stdout:1640: finishing write tid 5 to smithi15736696-31 2024-09-16T03:39:03.852 INFO:tasks.rados.rados.0.smithi157.stdout:1640: oid 31 updating version 0 to 78 2024-09-16T03:39:03.852 INFO:tasks.rados.rados.0.smithi157.stdout:1640: oid 31 version 78 is already newer than 77 2024-09-16T03:39:03.852 INFO:tasks.rados.rados.0.smithi157.stdout:1640: oid 31 version 78 is already newer than 76 2024-09-16T03:39:03.852 INFO:tasks.rados.rados.0.smithi157.stdout:1640: oid 31 updating version 78 to 79 2024-09-16T03:39:03.852 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 79 (ObjNum 353 snap 0 seq_num 353) dirty exists 2024-09-16T03:39:03.853 INFO:tasks.rados.rados.0.smithi157.stdout:1640: left oid 31 (ObjNum 353 snap 0 seq_num 353) 2024-09-16T03:39:03.853 INFO:tasks.rados.rados.0.smithi157.stdout:1646: finishing write tid 2 to smithi15736696-50 2024-09-16T03:39:03.853 INFO:tasks.rados.rados.0.smithi157.stdout:1646: finishing write tid 3 to smithi15736696-50 2024-09-16T03:39:03.853 INFO:tasks.rados.rados.0.smithi157.stdout:1646: finishing write tid 4 to smithi15736696-50 2024-09-16T03:39:03.853 INFO:tasks.rados.rados.0.smithi157.stdout:1646: finishing write tid 5 to smithi15736696-50 2024-09-16T03:39:03.853 INFO:tasks.rados.rados.0.smithi157.stdout:1646: oid 50 updating version 0 to 143 2024-09-16T03:39:03.853 INFO:tasks.rados.rados.0.smithi157.stdout:1646: oid 50 version 143 is already newer than 140 2024-09-16T03:39:03.853 INFO:tasks.rados.rados.0.smithi157.stdout:1646: oid 50 version 143 is already newer than 141 2024-09-16T03:39:03.853 INFO:tasks.rados.rados.0.smithi157.stdout:1646: oid 50 version 143 is already newer than 142 2024-09-16T03:39:03.853 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 143 (ObjNum 355 snap 0 seq_num 355) dirty exists 2024-09-16T03:39:03.854 INFO:tasks.rados.rados.0.smithi157.stdout:1646: left oid 50 (ObjNum 355 snap 0 seq_num 355) 2024-09-16T03:39:03.854 INFO:tasks.rados.rados.0.smithi157.stdout:1649: finishing write tid 5 to smithi15736696-37 2024-09-16T03:39:03.854 INFO:tasks.rados.rados.0.smithi157.stdout:1649: oid 37 updating version 0 to 75 2024-09-16T03:39:03.854 INFO:tasks.rados.rados.0.smithi157.stdout:1649: oid 37 version 75 is already newer than 74 2024-09-16T03:39:03.854 INFO:tasks.rados.rados.0.smithi157.stdout:1649: oid 37 updating version 75 to 76 2024-09-16T03:39:03.854 INFO:tasks.rados.rados.0.smithi157.stdout:1649: oid 37 version 76 is already newer than 73 2024-09-16T03:39:03.854 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 76 (ObjNum 356 snap 0 seq_num 356) dirty exists 2024-09-16T03:39:03.854 INFO:tasks.rados.rados.0.smithi157.stdout:1649: left oid 37 (ObjNum 356 snap 0 seq_num 356) 2024-09-16T03:39:03.855 INFO:tasks.rados.rados.0.smithi157.stdout:1640: done (15 left) 2024-09-16T03:39:03.855 INFO:tasks.rados.rados.0.smithi157.stdout:1646: done (14 left) 2024-09-16T03:39:03.855 INFO:tasks.rados.rados.0.smithi157.stdout:1649: done (13 left) 2024-09-16T03:39:03.855 INFO:tasks.rados.rados.0.smithi157.stdout:1669: write oid 17 current snap is 0 2024-09-16T03:39:03.855 INFO:tasks.rados.rados.0.smithi157.stdout:1669: seq_num 363 ranges {500941=783978,1942140=158801} 2024-09-16T03:39:03.858 INFO:tasks.rados.rados.0.smithi157.stdout:1669: writing smithi15736696-17 from 500941 to 1284919 tid 1 2024-09-16T03:39:03.863 INFO:tasks.rados.rados.0.smithi157.stdout:1669: writing smithi15736696-17 from 1942140 to 2100941 tid 2 2024-09-16T03:39:03.863 INFO:tasks.rados.rados.0.smithi157.stdout:1670: set_chunk oid 23 target oid 2024-09-16T03:39:03.877 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:03.877 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:03.881 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:03.880+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:03.886 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:39:03.983 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:03.981+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:03.991 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:03.991 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:04.000 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:39:04.083 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:04.082+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:04.146 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:04.146 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:04.155 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:39:04.185 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:04.183+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:04.238 INFO:tasks.rados.rados.0.smithi157.stdout:1670: set_chunk oid 23 offset: 2356224 length: 302080 target oid c43c7ed3b3dd9903f63ac2779fd0279f077bc30cc15f09fe0693b68cea475eb3 offset: 0 snap -1 2024-09-16T03:39:04.238 INFO:tasks.rados.rados.0.smithi157.stdout:1671: read oid 46 snap -1 2024-09-16T03:39:04.238 INFO:tasks.rados.rados.0.smithi157.stdout:1671: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-16T03:39:04.238 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:39:04.238 INFO:tasks.rados.rados.0.smithi157.stdout:1653: finishing set_chunk to oid 24 2024-09-16T03:39:04.238 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 124 (ObjNum 333 snap 0 seq_num 333) dirty exists 2024-09-16T03:39:04.238 INFO:tasks.rados.rados.0.smithi157.stdout:1657: finishing set_chunk to oid 4 2024-09-16T03:39:04.238 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 115 (ObjNum 296 snap 0 seq_num 296) dirty exists 2024-09-16T03:39:04.238 INFO:tasks.rados.rados.0.smithi157.stdout:1660: finishing write tid 1 to smithi15736696-30 2024-09-16T03:39:04.238 INFO:tasks.rados.rados.0.smithi157.stdout:1658: finishing write tid 1 to smithi15736696-8 2024-09-16T03:39:04.238 INFO:tasks.rados.rados.0.smithi157.stdout:1658: finishing write tid 2 to smithi15736696-8 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1658: finishing write tid 3 to smithi15736696-8 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1658: finishing write tid 4 to smithi15736696-8 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1658: oid 8 updating version 0 to 97 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1658: oid 8 version 97 is already newer than 95 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1658: oid 8 version 97 is already newer than 96 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 97 (ObjNum 359 snap 0 seq_num 359) dirty exists 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1658: left oid 8 (ObjNum 359 snap 0 seq_num 359) 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1660: finishing write tid 2 to smithi15736696-30 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1653: done (15 left) 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1657: done (14 left) 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1658: done (13 left) 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1672: set_chunk oid 39 target oid 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1672: set_chunk oid 39 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:39:04.239 INFO:tasks.rados.rados.0.smithi157.stdout:1673: read oid 4 snap -1 2024-09-16T03:39:04.240 INFO:tasks.rados.rados.0.smithi157.stdout:1673: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-16T03:39:04.240 INFO:tasks.rados.rados.0.smithi157.stdout:1674: set_chunk oid 16 target oid 2024-09-16T03:39:04.260 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:04.260 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:04.269 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:39:04.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:39:04.312 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:04.311+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:04.342 INFO:tasks.rados.rados.0.smithi157.stdout:1674: set_chunk oid 16 offset: 272896 length: 386048 target oid 6997c8343ff0d4df2e079217f6dd2e519398e6e616cc2138c481a60e80d32adf offset: 0 snap -1 2024-09-16T03:39:04.342 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:39:04.343 INFO:tasks.rados.rados.0.smithi157.stdout:1660: finishing write tid 3 to smithi15736696-30 2024-09-16T03:39:04.343 INFO:tasks.rados.rados.0.smithi157.stdout:1660: finishing write tid 4 to smithi15736696-30 2024-09-16T03:39:04.343 INFO:tasks.rados.rados.0.smithi157.stdout:1660: oid 30 updating version 0 to 49 2024-09-16T03:39:04.343 INFO:tasks.rados.rados.0.smithi157.stdout:1660: oid 30 updating version 49 to 51 2024-09-16T03:39:04.343 INFO:tasks.rados.rados.0.smithi157.stdout:1660: oid 30 version 51 is already newer than 50 2024-09-16T03:39:04.343 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 51 (ObjNum 361 snap 0 seq_num 361) dirty exists 2024-09-16T03:39:04.343 INFO:tasks.rados.rados.0.smithi157.stdout:1660: left oid 30 (ObjNum 361 snap 0 seq_num 361) 2024-09-16T03:39:04.343 INFO:tasks.rados.rados.0.smithi157.stdout:1660: done (15 left) 2024-09-16T03:39:04.343 INFO:tasks.rados.rados.0.smithi157.stdout:1675: write (excl) oid 43 current snap is 0 2024-09-16T03:39:04.343 INFO:tasks.rados.rados.0.smithi157.stdout:1662: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-16T03:39:04.343 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:04.373 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:04.373 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:04.382 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:39:04.386 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:04.385+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:04.420 INFO:tasks.rados.rados.0.smithi157.stdout:1668: expect (ObjNum 349 snap 0 seq_num 349) 2024-09-16T03:39:04.423 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:04.423 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:04.440 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:04.441 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:04.457 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:04.457 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:04.474 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:04.474 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:04.488 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:04.486+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:04.490 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:04.490 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:04.501 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2537900 2024-09-16T03:39:04.515 INFO:tasks.rados.rados.0.smithi157.stdout:1661: finishing set_chunk to oid 13 2024-09-16T03:39:04.515 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 82 (ObjNum 348 snap 0 seq_num 348) dirty exists 2024-09-16T03:39:04.516 INFO:tasks.rados.rados.0.smithi157.stdout:1667: expect (ObjNum 276 snap 0 seq_num 276) 2024-09-16T03:39:04.516 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:39:04.521 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T03:39:04.531 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:04.531 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:04.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-16T03:39:04.540 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:39:04.588 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:04.587+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:04.605 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:04.605 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:04.622 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:04.622 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:04.637 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:04.645 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:04.645 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:04.653 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:39:04.656 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:04.656 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:04.673 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:04.674 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:04.689 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:04.687+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:04.690 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:04.690 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:04.696 INFO:tasks.rados.rados.0.smithi157.stdout:1666: finishing write tid 1 to smithi15736696-47 2024-09-16T03:39:04.696 INFO:tasks.rados.rados.0.smithi157.stdout:1666: finishing write tid 2 to smithi15736696-47 2024-09-16T03:39:04.696 INFO:tasks.rados.rados.0.smithi157.stdout:1666: finishing write tid 3 to smithi15736696-47 2024-09-16T03:39:04.697 INFO:tasks.rados.rados.0.smithi157.stdout:1666: finishing write tid 4 to smithi15736696-47 2024-09-16T03:39:04.697 INFO:tasks.rados.rados.0.smithi157.stdout:1666: oid 47 updating version 0 to 130 2024-09-16T03:39:04.697 INFO:tasks.rados.rados.0.smithi157.stdout:1666: oid 47 updating version 130 to 131 2024-09-16T03:39:04.697 INFO:tasks.rados.rados.0.smithi157.stdout:1666: oid 47 updating version 131 to 132 2024-09-16T03:39:04.697 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 132 (ObjNum 362 snap 0 seq_num 362) dirty exists 2024-09-16T03:39:04.697 INFO:tasks.rados.rados.0.smithi157.stdout:1666: left oid 47 (ObjNum 362 snap 0 seq_num 362) 2024-09-16T03:39:04.697 INFO:tasks.rados.rados.0.smithi157.stdout:1669: finishing write tid 1 to smithi15736696-17 2024-09-16T03:39:04.697 INFO:tasks.rados.rados.0.smithi157.stdout:1663: expect (ObjNum 337 snap 0 seq_num 337) 2024-09-16T03:39:04.708 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:04.709 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:04.713 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1483790 2024-09-16T03:39:04.725 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:04.725 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:04.741 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:04.741 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:04.757 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:04.757 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:04.775 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:04.775 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:04.789 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:04.788+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:04.791 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:04.792 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:04.798 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:04.798 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:04.798 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:04.807 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:39:04.887 INFO:tasks.rados.rados.0.smithi157.stdout:1665: expect (ObjNum 331 snap 0 seq_num 331) 2024-09-16T03:39:04.890 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:04.889+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:04.954 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:04.954 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:04.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:04.954 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:04.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.954 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:04.954 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:04.954 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 112.80784625299999, 2024-09-16T03:39:04.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:39:04.955 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:04.955 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:04.955 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:04.955 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:04.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:04.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:04.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:04.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:04.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:04.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:39:04.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:39:04.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:04.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:39:04.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:39:04.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.958 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:04.958 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:39:04.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:39:04.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.958 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.958 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:04.958 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:39:04.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:39:04.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.958 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.958 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:04.958 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 112.807492893, 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:04.959 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:04.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:04.960 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:04.960 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:04.960 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:04.960 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:04.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:39:04.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:39:04.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:04.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:39:04.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:39:04.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:04.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:39:04.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:39:04.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:04.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:39:04.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:39:04.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:04.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:39:04.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:39:04.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:39:04.964 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.965 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:04.965 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.965 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.965 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:39:04.965 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:04.965 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 112.806365095, 2024-09-16T03:39:04.965 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:39:04.965 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:04.965 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:04.965 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:04.965 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:04.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:04.966 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:04.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:04.966 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:04.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.966 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:04.966 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.969 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:04.969 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:39:04.969 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:39:04.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.969 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:04.969 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:39:04.969 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:39:04.969 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.969 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:04.969 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.970 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:39:04.970 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:04.970 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 112.805377654, 2024-09-16T03:39:04.970 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:39:04.970 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:04.970 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:04.970 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:04.970 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:04.970 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.970 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:04.970 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:04.971 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.971 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.971 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.971 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:04.971 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:04.971 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.971 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.971 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.971 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:04.971 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:39:04.971 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:39:04.971 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:04.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:39:04.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:39:04.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:04.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:39:04.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:39:04.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.974 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:39:04.975 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:04.975 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 112.80462091, 2024-09-16T03:39:04.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:39:04.975 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:04.975 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:04.975 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:04.975 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:04.975 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.975 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:04.975 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:04.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.975 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.976 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:04.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:04.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.976 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.976 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:04.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:39:04.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:39:04.976 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.976 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:04.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:39:04.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:39:04.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.977 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:04.977 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:39:04.977 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:39:04.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.977 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:04.977 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:39:04.977 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:39:04.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.977 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:04.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:39:04.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:39:04.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:04.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:39:04.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:39:04.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:04.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:39:04.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:39:04.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:86 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'8, mlcod=39'5)", 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 112.34310867400001, 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20388281899999999, 2024-09-16T03:39:04.979 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:04.980 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:04.980 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:04.980 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:04.980 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.980 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:04.980 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:39:04.980 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.980 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.980 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.980 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:04.980 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:39:04.980 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608472+0000", 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.227e-06 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608616+0000", 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000144177 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:04.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608638+0000", 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1543e-05 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608645+0000", 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5839999999999997e-06 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608740+0000", 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4716999999999995e-05 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:04.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608763+0000", 2024-09-16T03:39:04.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2671e-05 2024-09-16T03:39:04.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:04.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812264+0000", 2024-09-16T03:39:04.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203501194 2024-09-16T03:39:04.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:04.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812353+0000", 2024-09-16T03:39:04.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8706000000000001e-05 2024-09-16T03:39:04.983 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:87 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'9, mlcod=39'5)", 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 112.3428953, 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20372206400000001, 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608684+0000", 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6700000000000003e-07 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:04.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608689+0000", 2024-09-16T03:39:04.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.412e-06 2024-09-16T03:39:04.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:04.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608704+0000", 2024-09-16T03:39:04.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4397e-05 2024-09-16T03:39:04.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:04.987 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608713+0000", 2024-09-16T03:39:04.987 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:39:04.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.987 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.987 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:04.987 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710025+0000", 2024-09-16T03:39:04.987 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101312032 2024-09-16T03:39:04.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.987 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.987 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:04.987 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710042+0000", 2024-09-16T03:39:04.987 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6889e-05 2024-09-16T03:39:04.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.988 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:04.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812361+0000", 2024-09-16T03:39:04.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10231889600000001 2024-09-16T03:39:04.988 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.988 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:04.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812405+0000", 2024-09-16T03:39:04.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4101999999999999e-05 2024-09-16T03:39:04.988 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.988 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:04.988 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.988 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:101 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'10, mlcod=39'5)", 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 112.341881863, 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20275586900000001, 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609696+0000", 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609698+0000", 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.24e-06 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609892+0000", 2024-09-16T03:39:04.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019465899999999999 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609904+0000", 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2031000000000001e-05 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.609912+0000", 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7160000000000003e-06 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:04.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710228+0000", 2024-09-16T03:39:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100315682 2024-09-16T03:39:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710242+0000", 2024-09-16T03:39:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4201e-05 2024-09-16T03:39:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812411+0000", 2024-09-16T03:39:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10216881799999999 2024-09-16T03:39:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812452+0000", 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1522e-05 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:102 4.f e39/32 4:fcb75e43:::smithi15736696-26:head v 39'11, mlcod=39'5)", 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 112.33986583799999, 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20134195399999999, 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:39:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611713+0000", 2024-09-16T03:39:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611714+0000", 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.269e-06 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611926+0000", 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021252700000000001 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611938+0000", 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1437000000000001e-05 2024-09-16T03:39:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:04.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.611944+0000", 2024-09-16T03:39:04.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.993e-06 2024-09-16T03:39:04.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:04.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811266+0000", 2024-09-16T03:39:04.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.199322306 2024-09-16T03:39:04.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:04.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.811291+0000", 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4542999999999999e-05 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813021+0000", 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001730363 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.813054+0000", 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3516000000000002e-05 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:04.997 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:375 4.f e45/32 4:fcb75e43:::smithi15736696-26:head v 45'23, mlcod=41'20)", 2024-09-16T03:39:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:39:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 104.64353810999999, 2024-09-16T03:39:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200786568, 2024-09-16T03:39:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:04.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:39:04.999 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:04.999 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308040+0000", 2024-09-16T03:39:04.999 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:04.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:04.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:04.999 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308041+0000", 2024-09-16T03:39:04.999 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8299999999999995e-07 2024-09-16T03:39:04.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:04.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.000 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:05.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308046+0000", 2024-09-16T03:39:05.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.904e-06 2024-09-16T03:39:05.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.000 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.000 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:05.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.308056+0000", 2024-09-16T03:39:05.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9210000000000004e-06 2024-09-16T03:39:05.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.000 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.000 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.408122+0000", 2024-09-16T03:39:05.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100066302 2024-09-16T03:39:05.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.001 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:05.001 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508289+0000", 2024-09-16T03:39:05.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166531 2024-09-16T03:39:05.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.001 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:05.001 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508302+0000", 2024-09-16T03:39:05.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2641e-05 2024-09-16T03:39:05.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.002 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:05.002 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508794+0000", 2024-09-16T03:39:05.002 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00049272400000000005 2024-09-16T03:39:05.002 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.002 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.002 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:05.002 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.508827+0000", 2024-09-16T03:39:05.002 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2561999999999998e-05 2024-09-16T03:39:05.002 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.002 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:05.002 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.002 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.003 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:989 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'31, mlcod=46'28)", 2024-09-16T03:39:05.003 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:05.003 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 94.563769550000004, 2024-09-16T03:39:05.003 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40332266900000002, 2024-09-16T03:39:05.003 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:05.003 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:05.003 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.003 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:05.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:05.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:05.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:05.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:05.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:05.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387812+0000", 2024-09-16T03:39:05.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:39:05.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.005 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:05.005 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388467+0000", 2024-09-16T03:39:05.005 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065539500000000002 2024-09-16T03:39:05.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.005 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.005 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:05.005 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388481+0000", 2024-09-16T03:39:05.005 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3905e-05 2024-09-16T03:39:05.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.005 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.005 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.005 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488552+0000", 2024-09-16T03:39:05.005 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10007120899999999 2024-09-16T03:39:05.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:05.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688818+0000", 2024-09-16T03:39:05.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20026627599999999 2024-09-16T03:39:05.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:05.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688845+0000", 2024-09-16T03:39:05.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:39:05.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:05.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791091+0000", 2024-09-16T03:39:05.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10224588699999999 2024-09-16T03:39:05.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:05.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791131+0000", 2024-09-16T03:39:05.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0293000000000002e-05 2024-09-16T03:39:05.007 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.007 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:05.007 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.007 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:990 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'32, mlcod=46'28)", 2024-09-16T03:39:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 94.462985877999998, 2024-09-16T03:39:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30256419200000001, 2024-09-16T03:39:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:05.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:05.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:05.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488594+0000", 2024-09-16T03:39:05.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:39:05.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:05.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488616+0000", 2024-09-16T03:39:05.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2419e-05 2024-09-16T03:39:05.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:05.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488627+0000", 2024-09-16T03:39:05.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1251e-05 2024-09-16T03:39:05.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588753+0000", 2024-09-16T03:39:05.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012544399999999 2024-09-16T03:39:05.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:05.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690679+0000", 2024-09-16T03:39:05.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101926027 2024-09-16T03:39:05.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:05.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690695+0000", 2024-09-16T03:39:05.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6528000000000002e-05 2024-09-16T03:39:05.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:05.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791133+0000", 2024-09-16T03:39:05.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100438233 2024-09-16T03:39:05.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:05.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791157+0000", 2024-09-16T03:39:05.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-05 2024-09-16T03:39:05.012 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.012 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:05.012 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:991 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'33, mlcod=46'28)", 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 94.362788381000001, 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202786365, 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:05.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588790+0000", 2024-09-16T03:39:05.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:05.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588792+0000", 2024-09-16T03:39:05.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0530000000000001e-06 2024-09-16T03:39:05.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:05.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588807+0000", 2024-09-16T03:39:05.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5257e-05 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588832+0000", 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5123e-05 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588846+0000", 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3312999999999999e-05 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:05.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790810+0000", 2024-09-16T03:39:05.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20196378100000001 2024-09-16T03:39:05.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:05.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.790866+0000", 2024-09-16T03:39:05.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6168999999999997e-05 2024-09-16T03:39:05.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:05.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791542+0000", 2024-09-16T03:39:05.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067583699999999999 2024-09-16T03:39:05.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.017 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:05.017 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791576+0000", 2024-09-16T03:39:05.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4832000000000001e-05 2024-09-16T03:39:05.017 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.017 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:05.017 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.017 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.017 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1580 4.f e61/32 4:fa163101:::smithi15736696-10:head v 61'52, mlcod=58'50)", 2024-09-16T03:39:05.017 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:39:05.017 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 77.845956854999997, 2024-09-16T03:39:05.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.201787367, 2024-09-16T03:39:05.017 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:05.018 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:05.018 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.018 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:05.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:05.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:39:05.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:05.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:39:05.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.105622+0000", 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2600000000000001e-07 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106083+0000", 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00046163900000000002 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.106093+0000", 2024-09-16T03:39:05.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9320000000000008e-06 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206190+0000", 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100096716 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206286+0000", 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5982000000000004e-05 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.206312+0000", 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6279999999999999e-05 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307345+0000", 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10103240199999999 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:47.307409+0000", 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4189999999999994e-05 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.022 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1760 4.f e71/32 4:f101f67d:::smithi15736696-22:head v 71'55, mlcod=61'53)", 2024-09-16T03:39:05.022 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:39:05.022 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 68.570752889000005, 2024-09-16T03:39:05.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20118894600000001, 2024-09-16T03:39:05.022 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:05.022 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:05.022 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.022 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:05.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:05.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:39:05.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380825+0000", 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380826+0000", 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1899999999999998e-07 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:05.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380831+0000", 2024-09-16T03:39:05.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6049999999999997e-06 2024-09-16T03:39:05.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:05.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380851+0000", 2024-09-16T03:39:05.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0021e-05 2024-09-16T03:39:05.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.380862+0000", 2024-09-16T03:39:05.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0458e-05 2024-09-16T03:39:05.024 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.024 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:05.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581184+0000", 2024-09-16T03:39:05.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20032164299999999 2024-09-16T03:39:05.025 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:05.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581202+0000", 2024-09-16T03:39:05.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8410000000000002e-05 2024-09-16T03:39:05.025 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.025 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.025 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:05.025 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.581965+0000", 2024-09-16T03:39:05.025 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00076291200000000003 2024-09-16T03:39:05.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.026 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:05.026 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:56.582014+0000", 2024-09-16T03:39:05.026 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9577999999999999e-05 2024-09-16T03:39:05.026 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.026 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:05.026 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.026 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.026 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.026 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2729 4.12 4:4a9d235d:::smithi15736696-1:head [write 1578407~488887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:39:05.026 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:22.608814+0000", 2024-09-16T03:39:05.026 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.342764150999997, 2024-09-16T03:39:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203332443, 2024-09-16T03:39:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2729 2024-09-16T03:39:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.608814+0000", 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.608814+0000", 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.608816+0000", 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0480000000000001e-06 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609287+0000", 2024-09-16T03:39:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000470524 2024-09-16T03:39:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609298+0000", 2024-09-16T03:39:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1629e-05 2024-09-16T03:39:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609306+0000", 2024-09-16T03:39:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5480000000000004e-06 2024-09-16T03:39:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:05.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609354+0000", 2024-09-16T03:39:05.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7688999999999997e-05 2024-09-16T03:39:05.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:05.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609448+0000", 2024-09-16T03:39:05.030 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4060000000000004e-05 2024-09-16T03:39:05.030 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.030 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.030 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 2,11", 2024-09-16T03:39:05.030 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609571+0000", 2024-09-16T03:39:05.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012361699999999999 2024-09-16T03:39:05.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:05.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.611263+0000", 2024-09-16T03:39:05.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016921099999999999 2024-09-16T03:39:05.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.031 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.031 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.613981+0000", 2024-09-16T03:39:05.031 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027178340000000001 2024-09-16T03:39:05.031 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.031 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812075+0000", 2024-09-16T03:39:05.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19809358399999999 2024-09-16T03:39:05.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:05.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812114+0000", 2024-09-16T03:39:05.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9082000000000003e-05 2024-09-16T03:39:05.032 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.032 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.032 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:05.032 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812147+0000", 2024-09-16T03:39:05.032 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2718000000000001e-05 2024-09-16T03:39:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2730 4.12 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2067294] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:39:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:22.609319+0000", 2024-09-16T03:39:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.342259490000004, 2024-09-16T03:39:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202937274, 2024-09-16T03:39:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.033 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2730 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609319+0000", 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609319+0000", 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.034 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609320+0000", 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3099999999999997e-07 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609324+0000", 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.98e-06 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609330+0000", 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4640000000000003e-06 2024-09-16T03:39:05.035 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609337+0000", 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.374e-06 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.610575+0000", 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001237414 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.610663+0000", 2024-09-16T03:39:05.036 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8268000000000003e-05 2024-09-16T03:39:05.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 2,11", 2024-09-16T03:39:05.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.610822+0000", 2024-09-16T03:39:05.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015895199999999999 2024-09-16T03:39:05.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:05.037 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.613696+0000", 2024-09-16T03:39:05.037 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0028742239999999999 2024-09-16T03:39:05.037 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.037 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.037 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.614015+0000", 2024-09-16T03:39:05.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00031823499999999999 2024-09-16T03:39:05.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812188+0000", 2024-09-16T03:39:05.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.198173138 2024-09-16T03:39:05.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.038 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.038 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:05.038 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812222+0000", 2024-09-16T03:39:05.038 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4724999999999997e-05 2024-09-16T03:39:05.038 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812256+0000", 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3769000000000001e-05 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2731 4.12 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:22.609352+0000", 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.342226041000004, 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203034889, 2024-09-16T03:39:05.039 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2731 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609352+0000", 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.040 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609352+0000", 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609353+0000", 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4099999999999999e-07 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609355+0000", 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4439999999999998e-06 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.041 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609360+0000", 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609363+0000", 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3780000000000001e-06 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.611279+0000", 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019158140000000001 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.042 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.611318+0000", 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8312999999999997e-05 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812286+0000", 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20096847600000001 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812332+0000", 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6313000000000002e-05 2024-09-16T03:39:05.043 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.044 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:05.044 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812387+0000", 2024-09-16T03:39:05.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4857e-05 2024-09-16T03:39:05.044 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.044 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:05.044 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.044 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.044 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.044 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3081 4.1f e102/100 4:fa163101:::smithi15736696-10:head v 102'93, mlcod=102'91)", 2024-09-16T03:39:05.044 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:05.044 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 29.617863409999998, 2024-09-16T03:39:05.044 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20437049600000001, 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.045 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333718+0000", 2024-09-16T03:39:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3720000000000001e-06 2024-09-16T03:39:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334338+0000", 2024-09-16T03:39:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061959099999999996 2024-09-16T03:39:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:05.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334355+0000", 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.677e-05 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334364+0000", 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3780000000000007e-06 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334424+0000", 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0084000000000003e-05 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:05.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334447+0000", 2024-09-16T03:39:05.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3113e-05 2024-09-16T03:39:05.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:05.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538030+0000", 2024-09-16T03:39:05.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20358272399999999 2024-09-16T03:39:05.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:05.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538085+0000", 2024-09-16T03:39:05.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5464000000000003e-05 2024-09-16T03:39:05.048 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.049 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:05.049 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.049 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3082 4.1f e102/100 4:fa163101:::smithi15736696-10:head v 102'94, mlcod=102'91)", 2024-09-16T03:39:05.049 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:35.335460+0000", 2024-09-16T03:39:05.049 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 29.616118451999998, 2024-09-16T03:39:05.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264421699999999, 2024-09-16T03:39:05.049 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:05.049 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:05.049 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.049 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:05.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335460+0000", 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335460+0000", 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335462+0000", 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9929999999999998e-06 2024-09-16T03:39:05.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335889+0000", 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042691199999999999 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335898+0000", 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6660000000000003e-06 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335906+0000", 2024-09-16T03:39:05.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0720000000000002e-06 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335960+0000", 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4456000000000002e-05 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.335978+0000", 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8082000000000001e-05 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538089+0000", 2024-09-16T03:39:05.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202111078 2024-09-16T03:39:05.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:05.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:05.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:05.053 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538104+0000", 2024-09-16T03:39:05.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4958e-05 2024-09-16T03:39:05.053 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.053 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:05.053 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.053 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:05.053 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:05.053 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:05.055 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2560638 2024-09-16T03:39:05.055 INFO:tasks.rados.rados.0.smithi157.stdout:1673: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-16T03:39:05.055 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:39:05.056 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:05.053+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:05.057 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:05.105 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:05.153 INFO:tasks.rados.rados.0.smithi157.stdout:1669: finishing write tid 2 to smithi15736696-17 2024-09-16T03:39:05.153 INFO:tasks.rados.rados.0.smithi157.stdout:1669: finishing write tid 3 to smithi15736696-17 2024-09-16T03:39:05.153 INFO:tasks.rados.rados.0.smithi157.stdout:1669: finishing write tid 4 to smithi15736696-17 2024-09-16T03:39:05.153 INFO:tasks.rados.rados.0.smithi157.stdout:1669: oid 17 updating version 0 to 129 2024-09-16T03:39:05.154 INFO:tasks.rados.rados.0.smithi157.stdout:1669: oid 17 updating version 129 to 130 2024-09-16T03:39:05.154 INFO:tasks.rados.rados.0.smithi157.stdout:1669: oid 17 version 130 is already newer than 128 2024-09-16T03:39:05.154 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 130 (ObjNum 363 snap 0 seq_num 363) dirty exists 2024-09-16T03:39:05.154 INFO:tasks.rados.rados.0.smithi157.stdout:1669: left oid 17 (ObjNum 363 snap 0 seq_num 363) 2024-09-16T03:39:05.154 INFO:tasks.rados.rados.0.smithi157.stdout:1672: finishing set_chunk to oid 39 2024-09-16T03:39:05.154 INFO:tasks.rados.rados.0.smithi157.stdout:1672: got expected ENOENT (src dne) 2024-09-16T03:39:05.154 INFO:tasks.rados.rados.0.smithi157.stdout:1671: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-16T03:39:05.155 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:05.153+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:05.160 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:05.161 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:05.161 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:39:05.161 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:05.169 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:39:05.200 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:05.199+0000 7f6528e68640 -1 osd.1 129 handle_osd_map injecting map crc failure 2024-09-16T03:39:05.256 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:05.255+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:05.257 INFO:tasks.rados.rados.0.smithi157.stdout:1670: finishing set_chunk to oid 23 2024-09-16T03:39:05.257 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 23 v 61 (ObjNum 232 snap 0 seq_num 232) dirty exists 2024-09-16T03:39:05.257 INFO:tasks.rados.rados.0.smithi157.stdout:1675: seq_num 364 ranges {517748=459227,1600622=406110} 2024-09-16T03:39:05.262 INFO:tasks.rados.rados.0.smithi157.stdout:1675: writing smithi15736696-43 from 517748 to 976975 tid 1 2024-09-16T03:39:05.269 INFO:tasks.rados.rados.0.smithi157.stdout:1675: writing smithi15736696-43 from 1600622 to 2006732 tid 2 2024-09-16T03:39:05.269 INFO:tasks.rados.rados.0.smithi157.stdout:1651: done (15 left) 2024-09-16T03:39:05.269 INFO:tasks.rados.rados.0.smithi157.stdout:1661: done (14 left) 2024-09-16T03:39:05.269 INFO:tasks.rados.rados.0.smithi157.stdout:1662: done (13 left) 2024-09-16T03:39:05.269 INFO:tasks.rados.rados.0.smithi157.stdout:1663: done (12 left) 2024-09-16T03:39:05.269 INFO:tasks.rados.rados.0.smithi157.stdout:1664: done (11 left) 2024-09-16T03:39:05.269 INFO:tasks.rados.rados.0.smithi157.stdout:1665: done (10 left) 2024-09-16T03:39:05.269 INFO:tasks.rados.rados.0.smithi157.stdout:1666: done (9 left) 2024-09-16T03:39:05.269 INFO:tasks.rados.rados.0.smithi157.stdout:1667: done (8 left) 2024-09-16T03:39:05.270 INFO:tasks.rados.rados.0.smithi157.stdout:1668: done (7 left) 2024-09-16T03:39:05.270 INFO:tasks.rados.rados.0.smithi157.stdout:1669: done (6 left) 2024-09-16T03:39:05.270 INFO:tasks.rados.rados.0.smithi157.stdout:1670: done (5 left) 2024-09-16T03:39:05.270 INFO:tasks.rados.rados.0.smithi157.stdout:1671: done (4 left) 2024-09-16T03:39:05.270 INFO:tasks.rados.rados.0.smithi157.stdout:1672: done (3 left) 2024-09-16T03:39:05.270 INFO:tasks.rados.rados.0.smithi157.stdout:1673: done (2 left) 2024-09-16T03:39:05.270 INFO:tasks.rados.rados.0.smithi157.stdout:1676: write (excl) oid 39 current snap is 0 2024-09-16T03:39:05.270 INFO:tasks.rados.rados.0.smithi157.stdout:1676: seq_num 365 ranges {534555=650829,1659104=475451} 2024-09-16T03:39:05.275 INFO:tasks.rados.rados.0.smithi157.stdout:1676: writing smithi15736696-39 from 534555 to 1185384 tid 1 2024-09-16T03:39:05.275 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:05.275 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:05.275 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:05.275 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:05.275 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:05.280 INFO:tasks.rados.rados.0.smithi157.stdout:1676: writing smithi15736696-39 from 1659104 to 2134555 tid 2 2024-09-16T03:39:05.280 INFO:tasks.rados.rados.0.smithi157.stdout:1677: read oid 33 snap -1 2024-09-16T03:39:05.280 INFO:tasks.rados.rados.0.smithi157.stdout:1677: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-16T03:39:05.280 INFO:tasks.rados.rados.0.smithi157.stdout:1678: write (excl) oid 21 current snap is 0 2024-09-16T03:39:05.280 INFO:tasks.rados.rados.0.smithi157.stdout:1678: seq_num 366 ranges {551362=726078,1717586=409779} 2024-09-16T03:39:05.284 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:39:05.287 INFO:tasks.rados.rados.0.smithi157.stdout:1678: writing smithi15736696-21 from 551362 to 1277440 tid 1 2024-09-16T03:39:05.292 INFO:tasks.rados.rados.0.smithi157.stdout:1678: writing smithi15736696-21 from 1717586 to 2127365 tid 2 2024-09-16T03:39:05.292 INFO:tasks.rados.rados.0.smithi157.stdout:1679: write (excl) oid 48 current snap is 0 2024-09-16T03:39:05.292 INFO:tasks.rados.rados.0.smithi157.stdout:1679: seq_num 367 ranges {568169=401327,1376068=669790} 2024-09-16T03:39:05.297 INFO:tasks.rados.rados.0.smithi157.stdout:1679: writing smithi15736696-48 from 568169 to 969496 tid 1 2024-09-16T03:39:05.303 INFO:tasks.rados.rados.0.smithi157.stdout:1679: writing smithi15736696-48 from 1376068 to 2045858 tid 2 2024-09-16T03:39:05.303 INFO:tasks.rados.rados.0.smithi157.stdout:1680: tier_evict oid 2 2024-09-16T03:39:05.303 INFO:tasks.rados.rados.0.smithi157.stdout:1680: tier_evict oid 2 snap -1 2024-09-16T03:39:05.303 INFO:tasks.rados.rados.0.smithi157.stdout:1681: read oid 5 snap -1 2024-09-16T03:39:05.304 INFO:tasks.rados.rados.0.smithi157.stdout:1681: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-16T03:39:05.304 INFO:tasks.rados.rados.0.smithi157.stdout:1682: write (excl) oid 3 current snap is 0 2024-09-16T03:39:05.304 INFO:tasks.rados.rados.0.smithi157.stdout:1682: seq_num 368 ranges {584976=476576,1834550=350426} 2024-09-16T03:39:05.309 INFO:tasks.rados.rados.0.smithi157.stdout:1682: writing smithi15736696-3 from 584976 to 1061552 tid 1 2024-09-16T03:39:05.315 INFO:tasks.rados.rados.0.smithi157.stdout:1682: writing smithi15736696-3 from 1834550 to 2184976 tid 2 2024-09-16T03:39:05.315 INFO:tasks.rados.rados.0.smithi157.stdout:1683: read oid 9 snap -1 2024-09-16T03:39:05.315 INFO:tasks.rados.rados.0.smithi157.stdout:1683: expect deleted 2024-09-16T03:39:05.316 INFO:tasks.rados.rados.0.smithi157.stdout:1684: set_chunk oid 10 target oid 2024-09-16T03:39:05.357 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:05.356+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:05.395 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:05.395 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:05.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:05.396 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:05.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.396 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1515 3.1 e59/31 3:9c9e970c:::7249cb65b52cadf4fe8f4ec923c7e548d1de7d9628823acd2c8de19121f1c1ce:head v 59'69, mlcod=56'67)", 2024-09-16T03:39:05.396 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:39:05.396 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 80.441284474, 2024-09-16T03:39:05.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.203376156, 2024-09-16T03:39:05.396 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.396 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.396 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.396 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:39:05.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951650+0000", 2024-09-16T03:39:05.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.951651+0000", 2024-09-16T03:39:05.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8299999999999996e-07 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953610+0000", 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019588769999999999 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953622+0000", 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1846e-05 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:44.953632+0000", 2024-09-16T03:39:05.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0202e-05 2024-09-16T03:39:05.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.053808+0000", 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100175615 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.053846+0000", 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8383000000000002e-05 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.155006+0000", 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101160131 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.155027+0000", 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0319e-05 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1704 3.1 e65/31 3:b6a94245:::f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99:head v 65'77, mlcod=64'75)", 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.599149199999999, 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30251757499999998, 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.401 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793786+0000", 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.793788+0000", 2024-09-16T03:39:05.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:39:05.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794391+0000", 2024-09-16T03:39:05.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00060354199999999999 2024-09-16T03:39:05.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794406+0000", 2024-09-16T03:39:05.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5045000000000001e-05 2024-09-16T03:39:05.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.794432+0000", 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5996999999999999e-05 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.994704+0000", 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20027177700000001 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.994719+0000", 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5021000000000001e-05 2024-09-16T03:39:05.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096277+0000", 2024-09-16T03:39:05.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101557437 2024-09-16T03:39:05.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096303+0000", 2024-09-16T03:39:05.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6732000000000001e-05 2024-09-16T03:39:05.405 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.405 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.405 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:84 3.1 e65/31 3:b24f289d:::e84532c28f467d2f2a6e2cf44904fa0d4a555e1304bbe1b25db645af42152bd7:head v 65'78, mlcod=64'75)", 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.590134036999999, 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29353782099999998, 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.407 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.407 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802801+0000", 2024-09-16T03:39:05.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.407 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.407 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.802805+0000", 2024-09-16T03:39:05.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1200000000000004e-06 2024-09-16T03:39:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803910+0000", 2024-09-16T03:39:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011051660000000001 2024-09-16T03:39:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803928+0000", 2024-09-16T03:39:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7880000000000002e-05 2024-09-16T03:39:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.803938+0000", 2024-09-16T03:39:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6690000000000001e-06 2024-09-16T03:39:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095473+0000", 2024-09-16T03:39:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29153552399999999 2024-09-16T03:39:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095486+0000", 2024-09-16T03:39:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2455e-05 2024-09-16T03:39:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096306+0000", 2024-09-16T03:39:05.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00082031900000000002 2024-09-16T03:39:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096339+0000", 2024-09-16T03:39:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2688000000000003e-05 2024-09-16T03:39:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:85 3.1 e65/31 3:b380f269:::e65351e8a65a0b1a517e3b1a85e04b84c0ab7cfa6655f11fb0da31f56b4dd9af:head v 65'79, mlcod=64'75)", 2024-09-16T03:39:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:39:05.411 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.588853459000006, 2024-09-16T03:39:05.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29260815400000001, 2024-09-16T03:39:05.412 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.412 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.412 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.412 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:39:05.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804081+0000", 2024-09-16T03:39:05.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804083+0000", 2024-09-16T03:39:05.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:39:05.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804266+0000", 2024-09-16T03:39:05.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018268099999999999 2024-09-16T03:39:05.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.804275+0000", 2024-09-16T03:39:05.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4439999999999997e-06 2024-09-16T03:39:05.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904371+0000", 2024-09-16T03:39:05.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100095276 2024-09-16T03:39:05.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095883+0000", 2024-09-16T03:39:05.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.191512444 2024-09-16T03:39:05.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.095901+0000", 2024-09-16T03:39:05.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8462999999999999e-05 2024-09-16T03:39:05.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096658+0000", 2024-09-16T03:39:05.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00075622000000000005 2024-09-16T03:39:05.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.096690+0000", 2024-09-16T03:39:05.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1919999999999999e-05 2024-09-16T03:39:05.417 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.417 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.417 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.417 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:86 3.1 e65/31 3:b3acb53e:::e951eb69ce5151a3f54107fa22c70f80729eff845da1613447a91536a24046e8:head v 65'80, mlcod=64'75)", 2024-09-16T03:39:05.417 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:39:05.417 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 74.488550043000004, 2024-09-16T03:39:05.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29244589199999999, 2024-09-16T03:39:05.417 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.418 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.418 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.418 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:39:05.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904385+0000", 2024-09-16T03:39:05.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904386+0000", 2024-09-16T03:39:05.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3e-07 2024-09-16T03:39:05.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904537+0000", 2024-09-16T03:39:05.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015122700000000001 2024-09-16T03:39:05.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904544+0000", 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.024e-06 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:50.904547+0000", 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6540000000000001e-06 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196016+0000", 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29146812599999999 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196027+0000", 2024-09-16T03:39:05.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1172000000000001e-05 2024-09-16T03:39:05.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196791+0000", 2024-09-16T03:39:05.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00076418700000000005 2024-09-16T03:39:05.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:51.196831+0000", 2024-09-16T03:39:05.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9771999999999998e-05 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2543 4.a e86/32 4:5e3c0760:::smithi15736696-24:head v 86'88, mlcod=84'86)", 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:17.073635+0000", 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 48.319299540000003, 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20396895300000001, 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.073635+0000", 2024-09-16T03:39:05.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.073635+0000", 2024-09-16T03:39:05.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.073636+0000", 2024-09-16T03:39:05.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7099999999999997e-07 2024-09-16T03:39:05.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.074518+0000", 2024-09-16T03:39:05.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000882532 2024-09-16T03:39:05.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.074530+0000", 2024-09-16T03:39:05.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1799e-05 2024-09-16T03:39:05.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.174624+0000", 2024-09-16T03:39:05.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100094445 2024-09-16T03:39:05.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.274761+0000", 2024-09-16T03:39:05.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013633199999999 2024-09-16T03:39:05.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.274785+0000", 2024-09-16T03:39:05.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3731e-05 2024-09-16T03:39:05.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.277559+0000", 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0027741179999999999 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.277604+0000", 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5624999999999998e-05 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2664 4.f e91/32 4:f817def9:::smithi15736696-45:head v 91'71, mlcod=89'69)", 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.623500+0000", 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.769434494999999, 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20350020799999999, 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.623500+0000", 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.623500+0000", 2024-09-16T03:39:05.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.623501+0000", 2024-09-16T03:39:05.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.99e-07 2024-09-16T03:39:05.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624286+0000", 2024-09-16T03:39:05.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00078502800000000001 2024-09-16T03:39:05.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624295+0000", 2024-09-16T03:39:05.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8630000000000003e-06 2024-09-16T03:39:05.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624302+0000", 2024-09-16T03:39:05.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0589999999999999e-06 2024-09-16T03:39:05.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624347+0000", 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5531000000000002e-05 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.624362+0000", 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4571e-05 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.826968+0000", 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20260592199999999 2024-09-16T03:39:05.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.827001+0000", 2024-09-16T03:39:05.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3034999999999998e-05 2024-09-16T03:39:05.431 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.431 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.431 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.431 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:3081 4.1f e102/100 4:fa163101:::smithi15736696-10:head v 102'93, mlcod=102'91)", 2024-09-16T03:39:05.431 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:35.333773+0000", 2024-09-16T03:39:05.431 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.059161574000001, 2024-09-16T03:39:05.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20354451600000001, 2024-09-16T03:39:05.432 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.432 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.432 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.432 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.333773+0000", 2024-09-16T03:39:05.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.333773+0000", 2024-09-16T03:39:05.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.333774+0000", 2024-09-16T03:39:05.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1570000000000001e-06 2024-09-16T03:39:05.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.334419+0000", 2024-09-16T03:39:05.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00064475100000000005 2024-09-16T03:39:05.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.334444+0000", 2024-09-16T03:39:05.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5123e-05 2024-09-16T03:39:05.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.434544+0000", 2024-09-16T03:39:05.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10009976499999999 2024-09-16T03:39:05.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.434625+0000", 2024-09-16T03:39:05.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0538999999999996e-05 2024-09-16T03:39:05.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.434652+0000", 2024-09-16T03:39:05.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.779e-05 2024-09-16T03:39:05.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.537288+0000", 2024-09-16T03:39:05.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102635448 2024-09-16T03:39:05.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.537318+0000", 2024-09-16T03:39:05.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9943000000000001e-05 2024-09-16T03:39:05.436 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.436 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.436 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.436 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(osd.10.22:219 3.0 e103/101 3:21e0c4e8:::f1ac9f6b1092b2dbfccdb2f32bdf689fb55e405da82aefeec91d0d73cdce51e1:head v 103'198, mlcod=0'0)", 2024-09-16T03:39:05.436 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:35.987822+0000", 2024-09-16T03:39:05.436 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.405112681999999, 2024-09-16T03:39:05.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30141685800000001, 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.987822+0000", 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.987822+0000", 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.987823+0000", 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0900000000000002e-07 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.987826+0000", 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8199999999999998e-06 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:35.987831+0000", 2024-09-16T03:39:05.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0990000000000001e-06 2024-09-16T03:39:05.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.087895+0000", 2024-09-16T03:39:05.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100064234 2024-09-16T03:39:05.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.087948+0000", 2024-09-16T03:39:05.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2781999999999997e-05 2024-09-16T03:39:05.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.087971+0000", 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3108e-05 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.289201+0000", 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.201230722 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.289239+0000", 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7684e-05 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.440 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:3260 4.f e107/100 4:f0d14c71:::smithi15736696-8:head v 107'91, mlcod=0'0)", 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:40.646237+0000", 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 24.746697673, 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20284116699999999, 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.646237+0000", 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.646237+0000", 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.646238+0000", 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0300000000000001e-06 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.647035+0000", 2024-09-16T03:39:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00079685300000000004 2024-09-16T03:39:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.647045+0000", 2024-09-16T03:39:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7140000000000005e-06 2024-09-16T03:39:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.647050+0000", 2024-09-16T03:39:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0030000000000002e-06 2024-09-16T03:39:05.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.747166+0000", 2024-09-16T03:39:05.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100116127 2024-09-16T03:39:05.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.747190+0000", 2024-09-16T03:39:05.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4474e-05 2024-09-16T03:39:05.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.849053+0000", 2024-09-16T03:39:05.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101862482 2024-09-16T03:39:05.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:40.849078+0000", 2024-09-16T03:39:05.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5483999999999999e-05 2024-09-16T03:39:05.445 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.445 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.445 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.445 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3402 4.1b 4.1792cddb (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T03:39:05.445 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:44.983846+0000", 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 20.409089332000001, 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53764449299999995, 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3402 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983846+0000", 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983846+0000", 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983855+0000", 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2059999999999996e-06 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983860+0000", 2024-09-16T03:39:05.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3990000000000003e-06 2024-09-16T03:39:05.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983869+0000", 2024-09-16T03:39:05.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5609999999999999e-06 2024-09-16T03:39:05.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983875+0000", 2024-09-16T03:39:05.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9900000000000002e-06 2024-09-16T03:39:05.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983933+0000", 2024-09-16T03:39:05.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8519e-05 2024-09-16T03:39:05.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:05.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983940+0000", 2024-09-16T03:39:05.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1579999999999999e-06 2024-09-16T03:39:05.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:45.521480+0000", 2024-09-16T03:39:05.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53753933300000001 2024-09-16T03:39:05.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:45.521490+0000", 2024-09-16T03:39:05.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0327e-05 2024-09-16T03:39:05.450 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.450 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.450 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.450 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3567 4.1b 4:dbb349e8:::smithi15736696-46:head [setxattr _header (56) in=63b,truncate 3479082] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:05.450 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:50.999803+0000", 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.393132328, 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.257763882, 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3567 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999803+0000", 2024-09-16T03:39:05.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999803+0000", 2024-09-16T03:39:05.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.452 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.452 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999803+0000", 2024-09-16T03:39:05.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7199999999999998e-07 2024-09-16T03:39:05.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999807+0000", 2024-09-16T03:39:05.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.782e-06 2024-09-16T03:39:05.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999813+0000", 2024-09-16T03:39:05.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8300000000000001e-06 2024-09-16T03:39:05.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999816+0000", 2024-09-16T03:39:05.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3050000000000001e-06 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001310+0000", 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001494057 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001397+0000", 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6458000000000005e-05 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:05.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001551+0000", 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015404000000000001 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001975+0000", 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042458099999999999 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044734+0000", 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.042758530000000003 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257487+0000", 2024-09-16T03:39:05.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.212752841 2024-09-16T03:39:05.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257519+0000", 2024-09-16T03:39:05.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2113000000000002e-05 2024-09-16T03:39:05.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257566+0000", 2024-09-16T03:39:05.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7573e-05 2024-09-16T03:39:05.456 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3568 4.1b 4:dbb349e8:::smithi15736696-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:50.999832+0000", 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.393102907999999, 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25787906900000002, 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:05.457 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3568 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999832+0000", 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999832+0000", 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999833+0000", 2024-09-16T03:39:05.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.51e-07 2024-09-16T03:39:05.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999835+0000", 2024-09-16T03:39:05.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5330000000000001e-06 2024-09-16T03:39:05.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999841+0000", 2024-09-16T03:39:05.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9939999999999997e-06 2024-09-16T03:39:05.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:50.999846+0000", 2024-09-16T03:39:05.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.31e-06 2024-09-16T03:39:05.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.460 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001833+0000", 2024-09-16T03:39:05.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0019864890000000001 2024-09-16T03:39:05.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.460 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:05.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.001875+0000", 2024-09-16T03:39:05.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2113999999999997e-05 2024-09-16T03:39:05.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257607+0000", 2024-09-16T03:39:05.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25573209000000002 2024-09-16T03:39:05.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257651+0000", 2024-09-16T03:39:05.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4233000000000002e-05 2024-09-16T03:39:05.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257711+0000", 2024-09-16T03:39:05.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9755e-05 2024-09-16T03:39:05.462 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.462 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.462 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.462 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3589 4.1b 4:dcb6965e:::smithi15736696-28:head [write 729503~560100 in=560100b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:05.462 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.044505+0000", 2024-09-16T03:39:05.462 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.34843012, 2024-09-16T03:39:05.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.21331082500000001, 2024-09-16T03:39:05.463 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.463 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.463 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.463 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:05.463 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:05.463 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:05.463 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3589 2024-09-16T03:39:05.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.463 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044505+0000", 2024-09-16T03:39:05.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044505+0000", 2024-09-16T03:39:05.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044508+0000", 2024-09-16T03:39:05.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5190000000000001e-06 2024-09-16T03:39:05.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044633+0000", 2024-09-16T03:39:05.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000124447 2024-09-16T03:39:05.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044653+0000", 2024-09-16T03:39:05.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0032999999999999e-05 2024-09-16T03:39:05.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.044663+0000", 2024-09-16T03:39:05.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0487e-05 2024-09-16T03:39:05.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052467+0000", 2024-09-16T03:39:05.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0078037280000000002 2024-09-16T03:39:05.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052561+0000", 2024-09-16T03:39:05.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3879e-05 2024-09-16T03:39:05.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:05.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052665+0000", 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000104613 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.054365+0000", 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001699983 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156645+0000", 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10227989699999999 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257746+0000", 2024-09-16T03:39:05.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10110100800000001 2024-09-16T03:39:05.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257780+0000", 2024-09-16T03:39:05.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4001999999999997e-05 2024-09-16T03:39:05.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257816+0000", 2024-09-16T03:39:05.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5228999999999998e-05 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3590 4.1b 4:dcb6965e:::smithi15736696-28:head [write 2019164~716432 in=716432b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.052101+0000", 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.340833720999999, 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50534822099999999, 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:05.469 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3590 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052101+0000", 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052101+0000", 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052103+0000", 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.44e-06 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052404+0000", 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030129000000000001 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052416+0000", 2024-09-16T03:39:05.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1868999999999999e-05 2024-09-16T03:39:05.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052422+0000", 2024-09-16T03:39:05.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:39:05.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.103424+0000", 2024-09-16T03:39:05.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051001643999999999 2024-09-16T03:39:05.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.103503+0000", 2024-09-16T03:39:05.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9871999999999997e-05 2024-09-16T03:39:05.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:05.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.103590+0000", 2024-09-16T03:39:05.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6268999999999997e-05 2024-09-16T03:39:05.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:05.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.105959+0000", 2024-09-16T03:39:05.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023694179999999999 2024-09-16T03:39:05.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.356869+0000", 2024-09-16T03:39:05.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.250909726 2024-09-16T03:39:05.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.557321+0000", 2024-09-16T03:39:05.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200451935 2024-09-16T03:39:05.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.557381+0000", 2024-09-16T03:39:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9960000000000002e-05 2024-09-16T03:39:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.557449+0000", 2024-09-16T03:39:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8621000000000006e-05 2024-09-16T03:39:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.475 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3591 4.1b 4:dcb6965e:::smithi15736696-28:head [write 3335341~194162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:05.476 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:39:05.476 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.336529575, 2024-09-16T03:39:05.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60146457399999997, 2024-09-16T03:39:05.476 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.476 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.476 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.476 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:05.476 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:05.476 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:05.476 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3591 2024-09-16T03:39:05.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.476 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056407+0000", 2024-09-16T03:39:05.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8920000000000001e-06 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056502+0000", 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4838000000000002e-05 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056511+0000", 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3579999999999992e-06 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156581+0000", 2024-09-16T03:39:05.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100069254 2024-09-16T03:39:05.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256413+0000", 2024-09-16T03:39:05.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099832613000000001 2024-09-16T03:39:05.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256479+0000", 2024-09-16T03:39:05.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5388000000000005e-05 2024-09-16T03:39:05.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256582+0000", 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000102876 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257889+0000", 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001307119 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657701+0000", 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39981235999999998 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657769+0000", 2024-09-16T03:39:05.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7508000000000005e-05 2024-09-16T03:39:05.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657827+0000", 2024-09-16T03:39:05.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8878000000000001e-05 2024-09-16T03:39:05.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657870+0000", 2024-09-16T03:39:05.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2490000000000001e-05 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3592 4.1b 4:dcb6965e:::smithi15736696-28:head [setxattr _header (56) in=7b,truncate 3529503] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.236321423, 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60139765499999998, 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:05.482 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:05.483 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:05.483 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3592 2024-09-16T03:39:05.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.483 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:39:05.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156615+0000", 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.714e-06 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156623+0000", 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5939999999999996e-06 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156634+0000", 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.093e-05 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156640+0000", 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1890000000000004e-06 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257172+0000", 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100532525 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257233+0000", 2024-09-16T03:39:05.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0736999999999998e-05 2024-09-16T03:39:05.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:05.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257319+0000", 2024-09-16T03:39:05.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5870000000000003e-05 2024-09-16T03:39:05.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:05.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257891+0000", 2024-09-16T03:39:05.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057231100000000004 2024-09-16T03:39:05.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657732+0000", 2024-09-16T03:39:05.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39984067699999998 2024-09-16T03:39:05.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:05.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.757877+0000", 2024-09-16T03:39:05.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100145238 2024-09-16T03:39:05.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.757947+0000", 2024-09-16T03:39:05.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9821999999999997e-05 2024-09-16T03:39:05.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758011+0000", 2024-09-16T03:39:05.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4047999999999994e-05 2024-09-16T03:39:05.488 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.488 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.488 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.488 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3593 4.1b 4:dcb6965e:::smithi15736696-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:05.489 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:39:05.489 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.236286689, 2024-09-16T03:39:05.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60159266700000003, 2024-09-16T03:39:05.489 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.489 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.489 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:05.489 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:05.489 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:05.489 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:05.489 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3593 2024-09-16T03:39:05.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.489 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:39:05.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:39:05.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156649+0000", 2024-09-16T03:39:05.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6400000000000002e-07 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156652+0000", 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9579999999999999e-06 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156658+0000", 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4729999999999997e-06 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156667+0000", 2024-09-16T03:39:05.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1940000000000004e-06 2024-09-16T03:39:05.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257836+0000", 2024-09-16T03:39:05.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101168444 2024-09-16T03:39:05.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:05.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257881+0000", 2024-09-16T03:39:05.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4910999999999997e-05 2024-09-16T03:39:05.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758073+0000", 2024-09-16T03:39:05.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50019239699999996 2024-09-16T03:39:05.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758147+0000", 2024-09-16T03:39:05.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3360000000000002e-05 2024-09-16T03:39:05.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758241+0000", 2024-09-16T03:39:05.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4265999999999995e-05 2024-09-16T03:39:05.494 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.494 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.494 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.494 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop_reply(client.4628.0:3591 4.1b e116/115 ondisk, result = 0)", 2024-09-16T03:39:05.494 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.358678+0000", 2024-09-16T03:39:05.494 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.034256748000001, 2024-09-16T03:39:05.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.299035153, 2024-09-16T03:39:05.494 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.494 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.494 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:05.494 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.358678+0000", 2024-09-16T03:39:05.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.358678+0000", 2024-09-16T03:39:05.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.358679+0000", 2024-09-16T03:39:05.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0699999999999999e-06 2024-09-16T03:39:05.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.358685+0000", 2024-09-16T03:39:05.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2920000000000001e-06 2024-09-16T03:39:05.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.358697+0000", 2024-09-16T03:39:05.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1202e-05 2024-09-16T03:39:05.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.458791+0000", 2024-09-16T03:39:05.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100094378 2024-09-16T03:39:05.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657682+0000", 2024-09-16T03:39:05.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198890804 2024-09-16T03:39:05.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657693+0000", 2024-09-16T03:39:05.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1056999999999999e-05 2024-09-16T03:39:05.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657713+0000", 2024-09-16T03:39:05.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.035e-05 2024-09-16T03:39:05.498 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.498 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.498 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.498 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop_reply(client.4628.0:3592 4.1b e117/115 ondisk, result = 0)", 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.460884+0000", 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 13.932050474, 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.297134383, 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.460884+0000", 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.499 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.460884+0000", 2024-09-16T03:39:05.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.460885+0000", 2024-09-16T03:39:05.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1600000000000002e-07 2024-09-16T03:39:05.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.460887+0000", 2024-09-16T03:39:05.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9489999999999999e-06 2024-09-16T03:39:05.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.460893+0000", 2024-09-16T03:39:05.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5470000000000002e-06 2024-09-16T03:39:05.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.560963+0000", 2024-09-16T03:39:05.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100069639 2024-09-16T03:39:05.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.757856+0000", 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19689335499999999 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.757869+0000", 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3015e-05 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758019+0000", 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000149462 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.502 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.503 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.503 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.503 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:05.504 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:39:05.505 INFO:tasks.rados.rados.0.smithi157.stdout:1684: set_chunk oid 10 offset: 68096 length: 1150976 target oid 6febdaa2c830f247e7e6d8f1831a882bcd15b656e772d0f6b6c3a228a449d538 offset: 0 snap -1 2024-09-16T03:39:05.506 INFO:tasks.rados.rados.0.smithi157.stdout:1685: tier_promote oid 8 2024-09-16T03:39:05.506 INFO:tasks.rados.rados.0.smithi157.stdout:1680: got (2) No such file or directory 2024-09-16T03:39:05.506 INFO:tasks.rados.rados.0.smithi157.stdout:1680: got expected ENOENT (src dne) 2024-09-16T03:39:05.506 INFO:tasks.rados.rados.0.smithi157.stdout:1680: done (11 left) 2024-09-16T03:39:05.506 INFO:tasks.rados.rados.0.smithi157.stdout:1686: read oid 4 snap -1 2024-09-16T03:39:05.506 INFO:tasks.rados.rados.0.smithi157.stdout:1686: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-16T03:39:05.506 INFO:tasks.rados.rados.0.smithi157.stdout:1687: write (excl) oid 14 current snap is 0 2024-09-16T03:39:05.506 INFO:tasks.rados.rados.0.smithi157.stdout:1675: finishing write tid 1 to smithi15736696-43 2024-09-16T03:39:05.506 INFO:tasks.rados.rados.0.smithi157.stdout:1675: finishing write tid 2 to smithi15736696-43 2024-09-16T03:39:05.506 INFO:tasks.rados.rados.0.smithi157.stdout:1687: seq_num 369 ranges {601783=551825,1776679=425104} 2024-09-16T03:39:05.506 INFO:tasks.rados.rados.0.smithi157.stdout:1687: writing smithi15736696-14 from 601783 to 1153608 tid 1 2024-09-16T03:39:05.506 INFO:tasks.rados.rados.0.smithi157.stdout:1687: writing smithi15736696-14 from 1776679 to 2201783 tid 2 2024-09-16T03:39:05.506 INFO:tasks.rados.rados.0.smithi157.stdout:1688: set_chunk oid 26 target oid 2024-09-16T03:39:05.507 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:05.503+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:05.518 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:05.516+0000 7f3d4270c640 -1 osd.12 130 handle_osd_map injecting map crc failure 2024-09-16T03:39:05.518 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:05.517+0000 7fa38a6ea640 -1 osd.10 130 handle_osd_map injecting map crc failure 2024-09-16T03:39:05.520 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.12 to 1 (10000) 2024-09-16T03:39:05.534 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-16T03:39:05.541 INFO:tasks.rados.rados.0.smithi157.stdout:1688: set_chunk oid 26 offset: 2068480 length: 272384 target oid 76bf89f67373ccae7775e0150128f9585e848319ae2062c6468c19fbba69c322 offset: 0 snap -1 2024-09-16T03:39:05.541 INFO:tasks.rados.rados.0.smithi157.stdout:1689: write oid 28 current snap is 0 2024-09-16T03:39:05.541 INFO:tasks.rados.rados.0.smithi157.stdout:1677: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-16T03:39:05.573 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:05.605 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:05.604+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:05.636 INFO:tasks.rados.rados.0.smithi157.stdout:1684: finishing set_chunk to oid 10 2024-09-16T03:39:05.637 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 10 set_chunk returned error code -95 offset: 68096 length: 1150976 2024-09-16T03:39:05.637 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 0 (ObjNum 276 snap 0 seq_num 276) dirty exists 2024-09-16T03:39:05.637 INFO:tasks.rados.rados.0.smithi157.stdout:1686: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-16T03:39:05.649 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:05.650 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:05.650 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:39:05.650 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:05.659 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:39:05.684 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:05.706 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:05.704+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:05.775 INFO:tasks.rados.rados.0.smithi157.stdout:1675: finishing write tid 3 to smithi15736696-43 2024-09-16T03:39:05.775 INFO:tasks.rados.rados.0.smithi157.stdout:1681: expect (ObjNum 281 snap 0 seq_num 281) 2024-09-16T03:39:05.790 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:05.790 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:05.790 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:05.790 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:05.791 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:05.799 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:39:05.806 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:05.805+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:05.831 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 779623 2024-09-16T03:39:05.853 INFO:tasks.rados.rados.0.smithi157.stdout:1682: finishing write tid 1 to smithi15736696-3 2024-09-16T03:39:05.853 INFO:tasks.rados.rados.0.smithi157.stdout:1676: finishing write tid 1 to smithi15736696-39 2024-09-16T03:39:05.853 INFO:tasks.rados.rados.0.smithi157.stdout:1676: finishing write tid 2 to smithi15736696-39 2024-09-16T03:39:05.854 INFO:tasks.rados.rados.0.smithi157.stdout:1689: seq_num 370 ranges {618590=627074,1835161=383429} 2024-09-16T03:39:05.860 INFO:tasks.rados.rados.0.smithi157.stdout:1689: writing smithi15736696-28 from 618590 to 1245664 tid 1 2024-09-16T03:39:05.865 INFO:tasks.rados.rados.0.smithi157.stdout:1689: writing smithi15736696-28 from 1835161 to 2218590 tid 2 2024-09-16T03:39:05.865 INFO:tasks.rados.rados.0.smithi157.stdout:1677: done (14 left) 2024-09-16T03:39:05.866 INFO:tasks.rados.rados.0.smithi157.stdout:1681: done (13 left) 2024-09-16T03:39:05.866 INFO:tasks.rados.rados.0.smithi157.stdout:1683: done (12 left) 2024-09-16T03:39:05.866 INFO:tasks.rados.rados.0.smithi157.stdout:1684: done (11 left) 2024-09-16T03:39:05.866 INFO:tasks.rados.rados.0.smithi157.stdout:1686: done (10 left) 2024-09-16T03:39:05.866 INFO:tasks.rados.rados.0.smithi157.stdout:1690: set_chunk oid 20 target oid 2024-09-16T03:39:05.908 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:05.908 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:05.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:05.908 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:05.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.908 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:39:05.908 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:39:05.908 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.719340956, 2024-09-16T03:39:05.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102630163, 2024-09-16T03:39:05.908 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.908 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.909 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.909 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:39:05.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:39:05.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186081+0000", 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5600000000000005e-07 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186970+0000", 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00088889000000000004 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186979+0000", 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0620000000000007e-06 2024-09-16T03:39:05.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186986+0000", 2024-09-16T03:39:05.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:39:05.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287184+0000", 2024-09-16T03:39:05.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100198036 2024-09-16T03:39:05.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287199+0000", 2024-09-16T03:39:05.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5458999999999999e-05 2024-09-16T03:39:05.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288659+0000", 2024-09-16T03:39:05.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014601900000000001 2024-09-16T03:39:05.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288711+0000", 2024-09-16T03:39:05.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1088000000000003e-05 2024-09-16T03:39:05.912 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.912 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.913 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.913 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:05.913 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:39:05.913 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.718304523, 2024-09-16T03:39:05.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10205370900000001, 2024-09-16T03:39:05.913 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.913 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.913 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.913 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:39:05.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:39:05.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187118+0000", 2024-09-16T03:39:05.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8599999999999998e-07 2024-09-16T03:39:05.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188062+0000", 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094447300000000004 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188070+0000", 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.695e-06 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188074+0000", 2024-09-16T03:39:05.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0709999999999996e-06 2024-09-16T03:39:05.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287842+0000", 2024-09-16T03:39:05.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099768478999999993 2024-09-16T03:39:05.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287864+0000", 2024-09-16T03:39:05.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1679999999999999e-05 2024-09-16T03:39:05.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289137+0000", 2024-09-16T03:39:05.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012732430000000001 2024-09-16T03:39:05.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289171+0000", 2024-09-16T03:39:05.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3382e-05 2024-09-16T03:39:05.917 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.917 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.917 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.917 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:42 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:05.918 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:39:05.918 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.64822261499999, 2024-09-16T03:39:05.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10172858, 2024-09-16T03:39:05.918 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.918 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.918 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.918 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:39:05.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257201+0000", 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257536+0000", 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033513899999999998 2024-09-16T03:39:05.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257555+0000", 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8547999999999999e-05 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257566+0000", 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1379999999999999e-05 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357720+0000", 2024-09-16T03:39:05.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100153532 2024-09-16T03:39:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357734+0000", 2024-09-16T03:39:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4491000000000001e-05 2024-09-16T03:39:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358896+0000", 2024-09-16T03:39:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011620599999999999 2024-09-16T03:39:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358927+0000", 2024-09-16T03:39:05.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1158999999999998e-05 2024-09-16T03:39:05.922 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.922 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.922 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.922 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:43 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'3, mlcod=0'0)", 2024-09-16T03:39:05.922 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:39:05.922 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.646438703, 2024-09-16T03:39:05.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101764453, 2024-09-16T03:39:05.923 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.923 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.923 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.923 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:39:05.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:39:05.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258984+0000", 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5689999999999999e-06 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258994+0000", 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259008+0000", 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4151999999999999e-05 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359073+0000", 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100065628 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359110+0000", 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7045999999999997e-05 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359124+0000", 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.382e-05 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360716+0000", 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015919829999999999 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360747+0000", 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0874999999999998e-05 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.926 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:492 4.5 e45/32 4:a11e1d7c:::smithi15736696-43:head v 45'22, mlcod=45'20)", 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 104.08207041599999, 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100797151, 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:39:05.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:39:05.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823353+0000", 2024-09-16T03:39:05.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2450000000000001e-06 2024-09-16T03:39:05.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823363+0000", 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0074999999999999e-05 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823378+0000", 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4534e-05 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923461+0000", 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100082859 2024-09-16T03:39:05.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923525+0000", 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4616999999999995e-05 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923549+0000", 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.357e-05 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924106+0000", 2024-09-16T03:39:05.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000556753 2024-09-16T03:39:05.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.931 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924148+0000", 2024-09-16T03:39:05.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2497999999999999e-05 2024-09-16T03:39:05.931 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.931 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.931 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.931 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:53'30,epoch:57/50,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:39:05.931 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:39:05.931 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 85.506718800000002, 2024-09-16T03:39:05.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100230893, 2024-09-16T03:39:05.932 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.932 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.932 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:05.932 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.932 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:39:05.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.932 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:39:05.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398705+0000", 2024-09-16T03:39:05.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5689999999999999e-06 2024-09-16T03:39:05.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398722+0000", 2024-09-16T03:39:05.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6558e-05 2024-09-16T03:39:05.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398735+0000", 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3499999999999999e-05 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398746+0000", 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1182999999999999e-05 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498898+0000", 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015112599999999 2024-09-16T03:39:05.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498906+0000", 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3189999999999995e-06 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498933+0000", 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7637999999999999e-05 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1570 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'31, mlcod=48'29)", 2024-09-16T03:39:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 79.731111804999998, 2024-09-16T03:39:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10253683500000001, 2024-09-16T03:39:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:05.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:05.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8500000000000002e-07 2024-09-16T03:39:05.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174689+0000", 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037958600000000001 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174699+0000", 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0015e-05 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174706+0000", 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6250000000000001e-06 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274865+0000", 2024-09-16T03:39:05.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100159157 2024-09-16T03:39:05.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274879+0000", 2024-09-16T03:39:05.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3983e-05 2024-09-16T03:39:05.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276820+0000", 2024-09-16T03:39:05.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001941248 2024-09-16T03:39:05.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276846+0000", 2024-09-16T03:39:05.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5936e-05 2024-09-16T03:39:05.940 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.940 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.940 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1571 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'32, mlcod=48'29)", 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 79.730661050999998, 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10249543, 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:39:05.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174761+0000", 2024-09-16T03:39:05.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0800000000000004e-07 2024-09-16T03:39:05.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175053+0000", 2024-09-16T03:39:05.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029151600000000002 2024-09-16T03:39:05.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175063+0000", 2024-09-16T03:39:05.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0538e-05 2024-09-16T03:39:05.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175070+0000", 2024-09-16T03:39:05.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7819999999999996e-06 2024-09-16T03:39:05.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275895+0000", 2024-09-16T03:39:05.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10082516 2024-09-16T03:39:05.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275909+0000", 2024-09-16T03:39:05.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4197e-05 2024-09-16T03:39:05.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277232+0000", 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013231009999999999 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277256+0000", 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3428e-05 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1572 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'33, mlcod=48'29)", 2024-09-16T03:39:05.945 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 79.730319750999996, 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10216886, 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:05.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:05.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8400000000000004e-07 2024-09-16T03:39:05.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175107+0000", 2024-09-16T03:39:05.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.578e-06 2024-09-16T03:39:05.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175114+0000", 2024-09-16T03:39:05.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.888e-06 2024-09-16T03:39:05.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175119+0000", 2024-09-16T03:39:05.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.074e-06 2024-09-16T03:39:05.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276662+0000", 2024-09-16T03:39:05.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101542753 2024-09-16T03:39:05.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276674+0000", 2024-09-16T03:39:05.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1960000000000001e-05 2024-09-16T03:39:05.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277258+0000", 2024-09-16T03:39:05.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058431900000000003 2024-09-16T03:39:05.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277270+0000", 2024-09-16T03:39:05.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2604e-05 2024-09-16T03:39:05.951 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.951 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.951 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.951 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:60'33,epoch:61/50,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:39:05.951 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:39:05.951 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 79.228398497000001, 2024-09-16T03:39:05.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10023048699999999, 2024-09-16T03:39:05.952 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.952 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.952 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:05.952 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:39:05.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:39:05.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:39:05.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8599999999999999e-07 2024-09-16T03:39:05.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677027+0000", 2024-09-16T03:39:05.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7359999999999999e-06 2024-09-16T03:39:05.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677042+0000", 2024-09-16T03:39:05.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4888e-05 2024-09-16T03:39:05.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677052+0000", 2024-09-16T03:39:05.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7750000000000004e-06 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777182+0000", 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100130394 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777195+0000", 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3373999999999999e-05 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777253+0000", 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7933999999999998e-05 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2454 4.15 e84/50 4:aff4cc41:::smithi15736696-11:head v 84'41, mlcod=79'39)", 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.442182389000003, 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101244889, 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.956 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:39:05.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:39:05.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:39:05.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2899999999999999e-07 2024-09-16T03:39:05.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463249+0000", 2024-09-16T03:39:05.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8039999999999996e-06 2024-09-16T03:39:05.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463267+0000", 2024-09-16T03:39:05.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7955e-05 2024-09-16T03:39:05.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563340+0000", 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100072999 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563396+0000", 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6141999999999999e-05 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563414+0000", 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8088999999999999e-05 2024-09-16T03:39:05.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.564427+0000", 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010127230000000001 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.564484+0000", 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6747999999999999e-05 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2467 4.5 e85/50)", 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 49.754290263000001, 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102439455, 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:39:05.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151134+0000", 2024-09-16T03:39:05.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2030000000000001e-06 2024-09-16T03:39:05.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151480+0000", 2024-09-16T03:39:05.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034613999999999999 2024-09-16T03:39:05.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151500+0000", 2024-09-16T03:39:05.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9582999999999999e-05 2024-09-16T03:39:05.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151513+0000", 2024-09-16T03:39:05.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:39:05.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251653+0000", 2024-09-16T03:39:05.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10014007900000001 2024-09-16T03:39:05.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251679+0000", 2024-09-16T03:39:05.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5721e-05 2024-09-16T03:39:05.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253532+0000", 2024-09-16T03:39:05.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001852701 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253571+0000", 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8677999999999998e-05 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2468 4.5 e85/50)", 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 49.752821734000001, 2024-09-16T03:39:05.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101505239, 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152601+0000", 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6869999999999999e-06 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152719+0000", 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117508 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152732+0000", 2024-09-16T03:39:05.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:39:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152740+0000", 2024-09-16T03:39:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7090000000000002e-06 2024-09-16T03:39:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252704+0000", 2024-09-16T03:39:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099964259999999999 2024-09-16T03:39:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252727+0000", 2024-09-16T03:39:05.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3601e-05 2024-09-16T03:39:05.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254075+0000", 2024-09-16T03:39:05.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00134772 2024-09-16T03:39:05.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254105+0000", 2024-09-16T03:39:05.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9802e-05 2024-09-16T03:39:05.969 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2623 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'62, mlcod=85'60)", 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 45.556184104000003, 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.104590794, 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1899999999999999e-07 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349496+0000", 2024-09-16T03:39:05.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000258934 2024-09-16T03:39:05.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349506+0000", 2024-09-16T03:39:05.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9389999999999992e-06 2024-09-16T03:39:05.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349513+0000", 2024-09-16T03:39:05.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6209999999999998e-06 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449668+0000", 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015550099999999 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449695+0000", 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6253000000000001e-05 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453784+0000", 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040896609999999996 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453828+0000", 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3665999999999998e-05 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2624 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'63, mlcod=85'60)", 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:39:05.974 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 45.554836866000002, 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103286796, 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:39:05.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350586+0000", 2024-09-16T03:39:05.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0020000000000001e-06 2024-09-16T03:39:05.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350794+0000", 2024-09-16T03:39:05.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020824099999999999 2024-09-16T03:39:05.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350805+0000", 2024-09-16T03:39:05.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1443999999999999e-05 2024-09-16T03:39:05.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450876+0000", 2024-09-16T03:39:05.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100070612 2024-09-16T03:39:05.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450960+0000", 2024-09-16T03:39:05.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4444999999999995e-05 2024-09-16T03:39:05.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450980+0000", 2024-09-16T03:39:05.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9728000000000002e-05 2024-09-16T03:39:05.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453831+0000", 2024-09-16T03:39:05.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002850923 2024-09-16T03:39:05.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453871+0000", 2024-09-16T03:39:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0401000000000002e-05 2024-09-16T03:39:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2626 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'65, mlcod=85'60)", 2024-09-16T03:39:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:39:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 45.454132889, 2024-09-16T03:39:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100625902, 2024-09-16T03:39:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.979 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.980 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:39:05.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:39:05.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451289+0000", 2024-09-16T03:39:05.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7100000000000002e-07 2024-09-16T03:39:05.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451295+0000", 2024-09-16T03:39:05.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2890000000000003e-06 2024-09-16T03:39:05.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451300+0000", 2024-09-16T03:39:05.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0710000000000002e-06 2024-09-16T03:39:05.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551399+0000", 2024-09-16T03:39:05.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10009881 2024-09-16T03:39:05.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551482+0000", 2024-09-16T03:39:05.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2738000000000006e-05 2024-09-16T03:39:05.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551496+0000", 2024-09-16T03:39:05.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4394e-05 2024-09-16T03:39:05.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551881+0000", 2024-09-16T03:39:05.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038522300000000002 2024-09-16T03:39:05.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551914+0000", 2024-09-16T03:39:05.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2906e-05 2024-09-16T03:39:05.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.984 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.984 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2661 4.15 e91/50 4:aff4cc41:::smithi15736696-11:head v 91'52, mlcod=88'48)", 2024-09-16T03:39:05.984 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:39:05.984 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.496236635999999, 2024-09-16T03:39:05.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10164756699999999, 2024-09-16T03:39:05.984 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.984 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.985 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.985 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:39:05.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:39:05.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409186+0000", 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-06 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409568+0000", 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038196600000000002 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409578+0000", 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.781e-06 2024-09-16T03:39:05.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409583+0000", 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3070000000000002e-06 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409633+0000", 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0358000000000001e-05 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409649+0000", 2024-09-16T03:39:05.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5875999999999998e-05 2024-09-16T03:39:05.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510800+0000", 2024-09-16T03:39:05.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101150728 2024-09-16T03:39:05.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510832+0000", 2024-09-16T03:39:05.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2500999999999999e-05 2024-09-16T03:39:05.988 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.988 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.988 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.989 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2662 4.15 e91/50 4:aff4cc41:::smithi15736696-11:head v 91'53, mlcod=88'48)", 2024-09-16T03:39:05.989 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:39:05.989 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.495028685999998, 2024-09-16T03:39:05.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100650287, 2024-09-16T03:39:05.989 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.989 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.989 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.989 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410394+0000", 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6299999999999993e-07 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410399+0000", 2024-09-16T03:39:05.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9400000000000001e-06 2024-09-16T03:39:05.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410407+0000", 2024-09-16T03:39:05.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8859999999999993e-06 2024-09-16T03:39:05.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410412+0000", 2024-09-16T03:39:05.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9339999999999997e-06 2024-09-16T03:39:05.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510516+0000", 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100103159 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510536+0000", 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0072999999999999e-05 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.511018+0000", 2024-09-16T03:39:05.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00048191899999999998 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.511043+0000", 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5412999999999999e-05 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2914 4.5 e98/50 4:a4fbca73:::smithi15736696-39:head v 98'72, mlcod=96'69)", 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:30.471084+0000", 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.434337114000002, 2024-09-16T03:39:05.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10075181900000001, 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471084+0000", 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471084+0000", 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471087+0000", 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3580000000000001e-06 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471099+0000", 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.255e-05 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471116+0000", 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6867999999999998e-05 2024-09-16T03:39:05.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471128+0000", 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2313999999999999e-05 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571266+0000", 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013743999999999 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571286+0000", 2024-09-16T03:39:05.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0313999999999999e-05 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571787+0000", 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00050111600000000002 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571836+0000", 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8859000000000001e-05 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:05.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:3970 4.5 e130/110 4:a11e1d7c:::smithi15736696-43:head v 130'83, mlcod=113'81)", 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.64276505500000003, 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102913225, 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262657+0000", 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.72e-07 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262783+0000", 2024-09-16T03:39:05.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012681199999999999 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262792+0000", 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3839999999999997e-06 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262799+0000", 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4100000000000002e-06 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.362958+0000", 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100158395 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.362977+0000", 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9741e-05 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.365511+0000", 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0025332499999999999 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.365570+0000", 2024-09-16T03:39:06.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8961e-05 2024-09-16T03:39:06.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.002 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.002 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.002 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:06.003 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:05.906+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:06.004 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:39:06.030 INFO:tasks.rados.rados.0.smithi157.stdout:1690: set_chunk oid 20 offset: 3100160 length: 441344 target oid e91489bbab32b415a07cd30dcaac5e491006fbec8d40ddeeb87cc2b859b45747 offset: 0 snap -1 2024-09-16T03:39:06.030 INFO:tasks.rados.rados.0.smithi157.stdout:1691: read oid 23 snap -1 2024-09-16T03:39:06.031 INFO:tasks.rados.rados.0.smithi157.stdout:1691: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:39:06.031 INFO:tasks.rados.rados.0.smithi157.stdout:1675: finishing write tid 4 to smithi15736696-43 2024-09-16T03:39:06.031 INFO:tasks.rados.rados.0.smithi157.stdout:1675: oid 43 updating version 0 to 84 2024-09-16T03:39:06.031 INFO:tasks.rados.rados.0.smithi157.stdout:1675: oid 43 updating version 84 to 85 2024-09-16T03:39:06.031 INFO:tasks.rados.rados.0.smithi157.stdout:1675: oid 43 version 85 is already newer than 83 2024-09-16T03:39:06.031 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 85 (ObjNum 364 snap 0 seq_num 364) dirty exists 2024-09-16T03:39:06.031 INFO:tasks.rados.rados.0.smithi157.stdout:1675: left oid 43 (ObjNum 364 snap 0 seq_num 364) 2024-09-16T03:39:06.031 INFO:tasks.rados.rados.0.smithi157.stdout:1678: finishing write tid 1 to smithi15736696-21 2024-09-16T03:39:06.031 INFO:tasks.rados.rados.0.smithi157.stdout:1692: read oid 37 snap -1 2024-09-16T03:39:06.031 INFO:tasks.rados.rados.0.smithi157.stdout:1692: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-16T03:39:06.031 INFO:tasks.rados.rados.0.smithi157.stdout:1675: done (12 left) 2024-09-16T03:39:06.031 INFO:tasks.rados.rados.0.smithi157.stdout:1693: tier_evict oid 10 2024-09-16T03:39:06.031 INFO:tasks.rados.rados.0.smithi157.stdout:1693: tier_evict oid 10 snap -1 2024-09-16T03:39:06.032 INFO:tasks.rados.rados.0.smithi157.stdout:1678: finishing write tid 2 to smithi15736696-21 2024-09-16T03:39:06.032 INFO:tasks.rados.rados.0.smithi157.stdout:1694: read oid 29 snap -1 2024-09-16T03:39:06.032 INFO:tasks.rados.rados.0.smithi157.stdout:1694: expect (ObjNum 344 snap 0 seq_num 344) 2024-09-16T03:39:06.032 INFO:tasks.rados.rados.0.smithi157.stdout:1695: read oid 44 snap -1 2024-09-16T03:39:06.032 INFO:tasks.rados.rados.0.smithi157.stdout:1695: expect (ObjNum 287 snap 0 seq_num 287) 2024-09-16T03:39:06.032 INFO:tasks.rados.rados.0.smithi157.stdout:1696: write (excl) oid 41 current snap is 0 2024-09-16T03:39:06.032 INFO:tasks.rados.rados.0.smithi157.stdout:1676: finishing write tid 3 to smithi15736696-39 2024-09-16T03:39:06.032 INFO:tasks.rados.rados.0.smithi157.stdout:1676: finishing write tid 4 to smithi15736696-39 2024-09-16T03:39:06.032 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:06.028+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:06.033 INFO:tasks.rados.rados.0.smithi157.stdout:1676: oid 39 updating version 0 to 84 2024-09-16T03:39:06.033 INFO:tasks.rados.rados.0.smithi157.stdout:1676: oid 39 updating version 84 to 85 2024-09-16T03:39:06.033 INFO:tasks.rados.rados.0.smithi157.stdout:1676: oid 39 version 85 is already newer than 83 2024-09-16T03:39:06.033 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 85 (ObjNum 365 snap 0 seq_num 365) dirty exists 2024-09-16T03:39:06.033 INFO:tasks.rados.rados.0.smithi157.stdout:1676: left oid 39 (ObjNum 365 snap 0 seq_num 365) 2024-09-16T03:39:06.033 INFO:tasks.rados.rados.0.smithi157.stdout:1682: finishing write tid 2 to smithi15736696-3 2024-09-16T03:39:06.033 INFO:tasks.rados.rados.0.smithi157.stdout:1696: seq_num 371 ranges {635397=702323,1893643=341754} 2024-09-16T03:39:06.037 INFO:tasks.rados.rados.0.smithi157.stdout:1696: writing smithi15736696-41 from 635397 to 1337720 tid 1 2024-09-16T03:39:06.042 INFO:tasks.rados.rados.0.smithi157.stdout:1696: writing smithi15736696-41 from 1893643 to 2235397 tid 2 2024-09-16T03:39:06.043 INFO:tasks.rados.rados.0.smithi157.stdout:1676: done (15 left) 2024-09-16T03:39:06.043 INFO:tasks.rados.rados.0.smithi157.stdout:1682: finishing write tid 3 to smithi15736696-3 2024-09-16T03:39:06.043 INFO:tasks.rados.rados.0.smithi157.stdout:1682: finishing write tid 4 to smithi15736696-3 2024-09-16T03:39:06.043 INFO:tasks.rados.rados.0.smithi157.stdout:1682: oid 3 updating version 0 to 74 2024-09-16T03:39:06.043 INFO:tasks.rados.rados.0.smithi157.stdout:1682: oid 3 version 74 is already newer than 73 2024-09-16T03:39:06.043 INFO:tasks.rados.rados.0.smithi157.stdout:1682: oid 3 version 74 is already newer than 72 2024-09-16T03:39:06.043 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 74 (ObjNum 368 snap 0 seq_num 368) dirty exists 2024-09-16T03:39:06.043 INFO:tasks.rados.rados.0.smithi157.stdout:1682: left oid 3 (ObjNum 368 snap 0 seq_num 368) 2024-09-16T03:39:06.043 INFO:tasks.rados.rados.0.smithi157.stdout:1697: read oid 50 snap -1 2024-09-16T03:39:06.043 INFO:tasks.rados.rados.0.smithi157.stdout:1697: expect (ObjNum 355 snap 0 seq_num 355) 2024-09-16T03:39:06.043 INFO:tasks.rados.rados.0.smithi157.stdout:1682: done (15 left) 2024-09-16T03:39:06.043 INFO:tasks.rados.rados.0.smithi157.stdout:1698: write (excl) oid 7 current snap is 0 2024-09-16T03:39:06.043 INFO:tasks.rados.rados.0.smithi157.stdout:1679: finishing write tid 1 to smithi15736696-48 2024-09-16T03:39:06.044 INFO:tasks.rados.rados.0.smithi157.stdout:1679: finishing write tid 2 to smithi15736696-48 2024-09-16T03:39:06.044 INFO:tasks.rados.rados.0.smithi157.stdout:1678: finishing write tid 3 to smithi15736696-21 2024-09-16T03:39:06.044 INFO:tasks.rados.rados.0.smithi157.stdout:1678: finishing write tid 4 to smithi15736696-21 2024-09-16T03:39:06.044 INFO:tasks.rados.rados.0.smithi157.stdout:1678: oid 21 updating version 0 to 63 2024-09-16T03:39:06.044 INFO:tasks.rados.rados.0.smithi157.stdout:1678: oid 21 updating version 63 to 65 2024-09-16T03:39:06.044 INFO:tasks.rados.rados.0.smithi157.stdout:1678: oid 21 version 65 is already newer than 64 2024-09-16T03:39:06.044 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 65 (ObjNum 366 snap 0 seq_num 366) dirty exists 2024-09-16T03:39:06.044 INFO:tasks.rados.rados.0.smithi157.stdout:1678: left oid 21 (ObjNum 366 snap 0 seq_num 366) 2024-09-16T03:39:06.044 INFO:tasks.rados.rados.0.smithi157.stdout:1687: finishing write tid 1 to smithi15736696-14 2024-09-16T03:39:06.044 INFO:tasks.rados.rados.0.smithi157.stdout:1698: seq_num 372 ranges {652204=777572,1952125=300079} 2024-09-16T03:39:06.050 INFO:tasks.rados.rados.0.smithi157.stdout:1698: writing smithi15736696-7 from 652204 to 1429776 tid 1 2024-09-16T03:39:06.054 INFO:tasks.rados.rados.0.smithi157.stdout:1698: writing smithi15736696-7 from 1952125 to 2252204 tid 2 2024-09-16T03:39:06.054 INFO:tasks.rados.rados.0.smithi157.stdout:1678: done (15 left) 2024-09-16T03:39:06.055 INFO:tasks.rados.rados.0.smithi157.stdout:1699: set_chunk oid 25 target oid 2024-09-16T03:39:06.109 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:06.108+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:06.135 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:06.135 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:06.135 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:39:06.135 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:06.145 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:39:06.210 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:06.209+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:06.293 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:06.294 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:06.294 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:06.294 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:06.294 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:06.303 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:39:06.347 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:06.346+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:06.361 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.14 to 1 (10000) 2024-09-16T03:39:06.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-16T03:39:06.409 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:06.409 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:06.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:06.409 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:06.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.409 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T03:39:06.409 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:39:06.409 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 135.329388843, 2024-09-16T03:39:06.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.88203620400000005, 2024-09-16T03:39:06.410 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.410 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.410 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.410 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.410 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.410 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.410 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:39:06.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.410 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:39:06.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:39:06.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076767+0000", 2024-09-16T03:39:06.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9300000000000002e-06 2024-09-16T03:39:06.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076774+0000", 2024-09-16T03:39:06.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6630000000000004e-06 2024-09-16T03:39:06.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076785+0000", 2024-09-16T03:39:06.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1221e-05 2024-09-16T03:39:06.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076796+0000", 2024-09-16T03:39:06.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0428000000000001e-05 2024-09-16T03:39:06.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076856+0000", 2024-09-16T03:39:06.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:39:06.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:06.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076863+0000", 2024-09-16T03:39:06.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9840000000000004e-06 2024-09-16T03:39:06.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855770+0000", 2024-09-16T03:39:06.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.77890736999999999 2024-09-16T03:39:06.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855902+0000", 2024-09-16T03:39:06.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013207 2024-09-16T03:39:06.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.856054+0000", 2024-09-16T03:39:06.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015220899999999999 2024-09-16T03:39:06.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.858407+0000", 2024-09-16T03:39:06.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023530069999999998 2024-09-16T03:39:06.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958598+0000", 2024-09-16T03:39:06.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100190194 2024-09-16T03:39:06.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958716+0000", 2024-09-16T03:39:06.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118589 2024-09-16T03:39:06.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958761+0000", 2024-09-16T03:39:06.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5120000000000002e-05 2024-09-16T03:39:06.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958802+0000", 2024-09-16T03:39:06.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0383999999999998e-05 2024-09-16T03:39:06.417 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.417 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.417 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.417 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:39:06.417 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:06.417 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 131.810972666, 2024-09-16T03:39:06.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70508991300000001, 2024-09-16T03:39:06.417 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.417 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.417 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.417 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.418 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.418 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.418 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 52 2024-09-16T03:39:06.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.418 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:06.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:06.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:06.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6300000000000005e-07 2024-09-16T03:39:06.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595187+0000", 2024-09-16T03:39:06.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9790000000000001e-06 2024-09-16T03:39:06.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595194+0000", 2024-09-16T03:39:06.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6490000000000002e-06 2024-09-16T03:39:06.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595208+0000", 2024-09-16T03:39:06.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4139e-05 2024-09-16T03:39:06.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801931+0000", 2024-09-16T03:39:06.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.206723083 2024-09-16T03:39:06.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801990+0000", 2024-09-16T03:39:06.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8625999999999997e-05 2024-09-16T03:39:06.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802066+0000", 2024-09-16T03:39:06.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6553999999999999e-05 2024-09-16T03:39:06.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802504+0000", 2024-09-16T03:39:06.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043775200000000002 2024-09-16T03:39:06.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.199998+0000", 2024-09-16T03:39:06.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39749365599999997 2024-09-16T03:39:06.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300151+0000", 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015310099999999 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300210+0000", 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9376999999999997e-05 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300271+0000", 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1334000000000006e-05 2024-09-16T03:39:06.423 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.424 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.424 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.424 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.424 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:06.424 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 130.03608168299999, 2024-09-16T03:39:06.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60623559900000001, 2024-09-16T03:39:06.424 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.424 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.424 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.424 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.425 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.425 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.425 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 71 2024-09-16T03:39:06.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.425 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:06.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:06.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:06.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:39:06.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370077+0000", 2024-09-16T03:39:06.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9940000000000002e-06 2024-09-16T03:39:06.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370081+0000", 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5850000000000001e-06 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370087+0000", 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673522+0000", 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.303434969 2024-09-16T03:39:06.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673567+0000", 2024-09-16T03:39:06.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5080000000000002e-05 2024-09-16T03:39:06.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673651+0000", 2024-09-16T03:39:06.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3652999999999993e-05 2024-09-16T03:39:06.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775115+0000", 2024-09-16T03:39:06.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101464178 2024-09-16T03:39:06.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975918+0000", 2024-09-16T03:39:06.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20080318699999999 2024-09-16T03:39:06.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976254+0000", 2024-09-16T03:39:06.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033619200000000002 2024-09-16T03:39:06.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976285+0000", 2024-09-16T03:39:06.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0679000000000002e-05 2024-09-16T03:39:06.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976308+0000", 2024-09-16T03:39:06.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3051e-05 2024-09-16T03:39:06.432 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.432 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.432 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.432 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.432 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:06.432 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 130.03605832299999, 2024-09-16T03:39:06.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60632925100000001, 2024-09-16T03:39:06.432 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.432 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.433 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.433 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.433 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.433 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.433 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 72 2024-09-16T03:39:06.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.433 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:06.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:06.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:06.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:39:06.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370150+0000", 2024-09-16T03:39:06.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3906999999999997e-05 2024-09-16T03:39:06.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370157+0000", 2024-09-16T03:39:06.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:39:06.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370162+0000", 2024-09-16T03:39:06.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7249999999999997e-06 2024-09-16T03:39:06.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673969+0000", 2024-09-16T03:39:06.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30380713199999998 2024-09-16T03:39:06.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674030+0000", 2024-09-16T03:39:06.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1141000000000003e-05 2024-09-16T03:39:06.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674124+0000", 2024-09-16T03:39:06.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3859999999999999e-05 2024-09-16T03:39:06.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775687+0000", 2024-09-16T03:39:06.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10156293299999999 2024-09-16T03:39:06.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976103+0000", 2024-09-16T03:39:06.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20041631400000001 2024-09-16T03:39:06.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976355+0000", 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000251611 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976385+0000", 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0883999999999998e-05 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976425+0000", 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9666000000000003e-05 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.438 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.439 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.439 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:39:06.439 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 130.03598583900001, 2024-09-16T03:39:06.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60763126999999995, 2024-09-16T03:39:06.439 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.439 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.439 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.439 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.439 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.439 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.439 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 73 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370169+0000", 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6199999999999998e-07 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370171+0000", 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370175+0000", 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.596e-06 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470295+0000", 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100119967 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775148+0000", 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30485311799999998 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775203+0000", 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4592999999999997e-05 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775293+0000", 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0859999999999994e-05 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976223+0000", 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20092996599999999 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976463+0000", 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000239695 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977730+0000", 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012667690000000001 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977773+0000", 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2604000000000001e-05 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977800+0000", 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7137000000000001e-05 2024-09-16T03:39:06.444 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 129.93584698399999, 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60633810099999996, 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.445 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 74 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470308+0000", 2024-09-16T03:39:06.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0950000000000001e-06 2024-09-16T03:39:06.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470313+0000", 2024-09-16T03:39:06.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:39:06.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470320+0000", 2024-09-16T03:39:06.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5089999999999999e-06 2024-09-16T03:39:06.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470327+0000", 2024-09-16T03:39:06.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:39:06.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976483+0000", 2024-09-16T03:39:06.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50615579499999996 2024-09-16T03:39:06.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976533+0000", 2024-09-16T03:39:06.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0096000000000003e-05 2024-09-16T03:39:06.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.449 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976643+0000", 2024-09-16T03:39:06.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109481 2024-09-16T03:39:06.449 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977808+0000", 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011649869999999999 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.981699+0000", 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003891579 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076496+0000", 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.094797333999999997 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.450 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076583+0000", 2024-09-16T03:39:06.451 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6571999999999997e-05 2024-09-16T03:39:06.451 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.451 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.451 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.451 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076645+0000", 2024-09-16T03:39:06.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2331999999999999e-05 2024-09-16T03:39:06.452 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.452 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.452 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.452 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.452 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.452 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.452 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:39:06.452 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 129.93581684899999, 2024-09-16T03:39:06.452 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71251765099999997, 2024-09-16T03:39:06.452 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.452 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.452 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.453 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.453 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.453 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.453 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 75 2024-09-16T03:39:06.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.453 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:39:06.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:39:06.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470338+0000", 2024-09-16T03:39:06.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2500000000000005e-07 2024-09-16T03:39:06.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470389+0000", 2024-09-16T03:39:06.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.134e-05 2024-09-16T03:39:06.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470397+0000", 2024-09-16T03:39:06.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8140000000000001e-06 2024-09-16T03:39:06.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570537+0000", 2024-09-16T03:39:06.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100139425 2024-09-16T03:39:06.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977018+0000", 2024-09-16T03:39:06.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40648102899999999 2024-09-16T03:39:06.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977063+0000", 2024-09-16T03:39:06.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5247000000000002e-05 2024-09-16T03:39:06.456 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977143+0000", 2024-09-16T03:39:06.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0054000000000003e-05 2024-09-16T03:39:06.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.978132+0000", 2024-09-16T03:39:06.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00098896399999999995 2024-09-16T03:39:06.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.458 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.458 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182725+0000", 2024-09-16T03:39:06.458 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.204592626 2024-09-16T03:39:06.458 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.458 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182783+0000", 2024-09-16T03:39:06.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8285000000000002e-05 2024-09-16T03:39:06.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182825+0000", 2024-09-16T03:39:06.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2398999999999998e-05 2024-09-16T03:39:06.459 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.459 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.459 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.459 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182855+0000", 2024-09-16T03:39:06.459 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9842999999999999e-05 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 129.835590282, 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71263271800000005, 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.460 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 76 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.461 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570566+0000", 2024-09-16T03:39:06.462 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:39:06.462 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.462 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.462 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.462 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570577+0000", 2024-09-16T03:39:06.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0672999999999999e-05 2024-09-16T03:39:06.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570593+0000", 2024-09-16T03:39:06.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5979999999999999e-05 2024-09-16T03:39:06.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.463 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.463 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570604+0000", 2024-09-16T03:39:06.463 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.077e-05 2024-09-16T03:39:06.463 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.463 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979890+0000", 2024-09-16T03:39:06.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40928624200000002 2024-09-16T03:39:06.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979945+0000", 2024-09-16T03:39:06.464 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5651e-05 2024-09-16T03:39:06.464 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.464 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.464 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.464 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980038+0000", 2024-09-16T03:39:06.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2603999999999996e-05 2024-09-16T03:39:06.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980669+0000", 2024-09-16T03:39:06.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063121900000000003 2024-09-16T03:39:06.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.465 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.465 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.465 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283050+0000", 2024-09-16T03:39:06.465 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30238120800000001 2024-09-16T03:39:06.465 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283132+0000", 2024-09-16T03:39:06.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1841999999999993e-05 2024-09-16T03:39:06.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.466 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.466 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.466 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283168+0000", 2024-09-16T03:39:06.466 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5438999999999997e-05 2024-09-16T03:39:06.466 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283197+0000", 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8960000000000001e-05 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 128.02105518299999, 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70799990400000001, 2024-09-16T03:39:06.467 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.468 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.468 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.468 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.468 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.468 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.468 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 94 2024-09-16T03:39:06.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.468 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.468 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.468 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.468 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:39:06.468 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.468 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:39:06.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385100+0000", 2024-09-16T03:39:06.469 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9599999999999998e-07 2024-09-16T03:39:06.469 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.469 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.469 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.469 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385104+0000", 2024-09-16T03:39:06.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7579999999999998e-06 2024-09-16T03:39:06.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.470 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.470 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385128+0000", 2024-09-16T03:39:06.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4700999999999999e-05 2024-09-16T03:39:06.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.470 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.470 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385139+0000", 2024-09-16T03:39:06.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0587e-05 2024-09-16T03:39:06.470 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787499+0000", 2024-09-16T03:39:06.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40235976099999998 2024-09-16T03:39:06.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787545+0000", 2024-09-16T03:39:06.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6418999999999997e-05 2024-09-16T03:39:06.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787623+0000", 2024-09-16T03:39:06.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7897000000000003e-05 2024-09-16T03:39:06.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888605+0000", 2024-09-16T03:39:06.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100982321 2024-09-16T03:39:06.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990721+0000", 2024-09-16T03:39:06.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102115922 2024-09-16T03:39:06.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093019+0000", 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10229788500000001 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093051+0000", 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2255999999999997e-05 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093099+0000", 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7500999999999998e-05 2024-09-16T03:39:06.473 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.474 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.474 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.474 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.474 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:39:06.474 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 128.02100386000001, 2024-09-16T03:39:06.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80404594699999998, 2024-09-16T03:39:06.474 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.474 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.474 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.474 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.474 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.475 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.475 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 95 2024-09-16T03:39:06.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.475 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:39:06.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:39:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385151+0000", 2024-09-16T03:39:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:39:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385165+0000", 2024-09-16T03:39:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4175999999999999e-05 2024-09-16T03:39:06.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385175+0000", 2024-09-16T03:39:06.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:39:06.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385183+0000", 2024-09-16T03:39:06.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:39:06.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888643+0000", 2024-09-16T03:39:06.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50345988200000003 2024-09-16T03:39:06.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.479 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.479 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888695+0000", 2024-09-16T03:39:06.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2834999999999998e-05 2024-09-16T03:39:06.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888802+0000", 2024-09-16T03:39:06.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010697899999999999 2024-09-16T03:39:06.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990658+0000", 2024-09-16T03:39:06.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185585699999999 2024-09-16T03:39:06.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990917+0000", 2024-09-16T03:39:06.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025853600000000002 2024-09-16T03:39:06.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189054+0000", 2024-09-16T03:39:06.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198136812 2024-09-16T03:39:06.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189135+0000", 2024-09-16T03:39:06.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1482000000000004e-05 2024-09-16T03:39:06.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189196+0000", 2024-09-16T03:39:06.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1175999999999998e-05 2024-09-16T03:39:06.483 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.483 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.483 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.484 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.484 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:39:06.484 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 128.020963233, 2024-09-16T03:39:06.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80412488400000004, 2024-09-16T03:39:06.484 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.484 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.484 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.484 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.484 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.484 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.485 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 96 2024-09-16T03:39:06.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.485 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:39:06.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.486 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.486 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:39:06.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.486 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385192+0000", 2024-09-16T03:39:06.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:39:06.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385267+0000", 2024-09-16T03:39:06.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5535000000000001e-05 2024-09-16T03:39:06.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385277+0000", 2024-09-16T03:39:06.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6239999999999997e-06 2024-09-16T03:39:06.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385284+0000", 2024-09-16T03:39:06.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4800000000000004e-06 2024-09-16T03:39:06.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989283+0000", 2024-09-16T03:39:06.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60399849500000002 2024-09-16T03:39:06.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989410+0000", 2024-09-16T03:39:06.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127298 2024-09-16T03:39:06.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989615+0000", 2024-09-16T03:39:06.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000205045 2024-09-16T03:39:06.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991036+0000", 2024-09-16T03:39:06.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001421042 2024-09-16T03:39:06.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189228+0000", 2024-09-16T03:39:06.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19819188500000001 2024-09-16T03:39:06.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189249+0000", 2024-09-16T03:39:06.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0996999999999999e-05 2024-09-16T03:39:06.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189293+0000", 2024-09-16T03:39:06.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3912000000000002e-05 2024-09-16T03:39:06.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189316+0000", 2024-09-16T03:39:06.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2739e-05 2024-09-16T03:39:06.497 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.498 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.498 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.498 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.498 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:39:06.499 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 128.02085950200001, 2024-09-16T03:39:06.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80642264100000005, 2024-09-16T03:39:06.499 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.499 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.499 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.500 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.500 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.500 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.500 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 97 2024-09-16T03:39:06.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.501 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:39:06.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:39:06.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385296+0000", 2024-09-16T03:39:06.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.006e-06 2024-09-16T03:39:06.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385301+0000", 2024-09-16T03:39:06.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2630000000000003e-06 2024-09-16T03:39:06.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385308+0000", 2024-09-16T03:39:06.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6030000000000001e-06 2024-09-16T03:39:06.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485459+0000", 2024-09-16T03:39:06.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150929 2024-09-16T03:39:06.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990238+0000", 2024-09-16T03:39:06.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50477992000000005 2024-09-16T03:39:06.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990287+0000", 2024-09-16T03:39:06.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8863999999999998e-05 2024-09-16T03:39:06.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990377+0000", 2024-09-16T03:39:06.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9900000000000003e-05 2024-09-16T03:39:06.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093109+0000", 2024-09-16T03:39:06.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102732055 2024-09-16T03:39:06.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189351+0000", 2024-09-16T03:39:06.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096241766000000006 2024-09-16T03:39:06.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191570+0000", 2024-09-16T03:39:06.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00221915 2024-09-16T03:39:06.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.512 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.512 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191653+0000", 2024-09-16T03:39:06.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3046000000000004e-05 2024-09-16T03:39:06.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191717+0000", 2024-09-16T03:39:06.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4139000000000004e-05 2024-09-16T03:39:06.513 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.513 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.514 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.514 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.514 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:39:06.514 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 127.920669045, 2024-09-16T03:39:06.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90688294999999997, 2024-09-16T03:39:06.515 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.515 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.515 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.515 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.515 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.516 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.516 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 98 2024-09-16T03:39:06.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.516 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:39:06.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:39:06.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485487+0000", 2024-09-16T03:39:06.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1799999999999999e-06 2024-09-16T03:39:06.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485498+0000", 2024-09-16T03:39:06.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0648999999999999e-05 2024-09-16T03:39:06.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485514+0000", 2024-09-16T03:39:06.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5943999999999998e-05 2024-09-16T03:39:06.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485525+0000", 2024-09-16T03:39:06.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.065e-05 2024-09-16T03:39:06.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991102+0000", 2024-09-16T03:39:06.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50557748400000002 2024-09-16T03:39:06.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991161+0000", 2024-09-16T03:39:06.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8801999999999997e-05 2024-09-16T03:39:06.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991255+0000", 2024-09-16T03:39:06.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3936000000000003e-05 2024-09-16T03:39:06.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093112+0000", 2024-09-16T03:39:06.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185709499999999 2024-09-16T03:39:06.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291911+0000", 2024-09-16T03:39:06.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19879897699999999 2024-09-16T03:39:06.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392167+0000", 2024-09-16T03:39:06.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10025634999999999 2024-09-16T03:39:06.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392249+0000", 2024-09-16T03:39:06.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2175000000000005e-05 2024-09-16T03:39:06.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392368+0000", 2024-09-16T03:39:06.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118708 2024-09-16T03:39:06.529 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.530 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.530 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.530 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.530 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:39:06.531 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 127.920615422, 2024-09-16T03:39:06.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90707203199999997, 2024-09-16T03:39:06.531 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.531 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.531 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.532 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.532 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.532 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.532 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 99 2024-09-16T03:39:06.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.532 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.533 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.533 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:39:06.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:39:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485540+0000", 2024-09-16T03:39:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-06 2024-09-16T03:39:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485661+0000", 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120835 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485680+0000", 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8885999999999998e-05 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485693+0000", 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3297e-05 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991546+0000", 2024-09-16T03:39:06.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.505853098 2024-09-16T03:39:06.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991598+0000", 2024-09-16T03:39:06.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1921e-05 2024-09-16T03:39:06.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991681+0000", 2024-09-16T03:39:06.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2502000000000004e-05 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093115+0000", 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10143424299999999 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291977+0000", 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198862172 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392541+0000", 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100563758 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392579+0000", 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7796999999999997e-05 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392611+0000", 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2073999999999998e-05 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.538 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.539 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.539 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:06.539 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 127.920436529, 2024-09-16T03:39:06.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0073149939999999, 2024-09-16T03:39:06.539 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.539 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.539 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.539 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.539 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.539 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.539 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 100 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7400000000000003e-07 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485721+0000", 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4530000000000001e-06 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485726+0000", 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0590000000000002e-06 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485729+0000", 2024-09-16T03:39:06.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:39:06.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092342+0000", 2024-09-16T03:39:06.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60661343700000003 2024-09-16T03:39:06.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092469+0000", 2024-09-16T03:39:06.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012658799999999999 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092671+0000", 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020206 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392452+0000", 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29978163699999999 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392486+0000", 2024-09-16T03:39:06.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3736999999999999e-05 2024-09-16T03:39:06.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492812+0000", 2024-09-16T03:39:06.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100325621 2024-09-16T03:39:06.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492901+0000", 2024-09-16T03:39:06.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8702000000000006e-05 2024-09-16T03:39:06.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.493033+0000", 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000132185 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 125.74528600799999, 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60874581500000002, 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.545 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.546 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.546 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.546 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.546 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.546 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 123 2024-09-16T03:39:06.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.546 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.546 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.546 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:39:06.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660869+0000", 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9400000000000005e-07 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660921+0000", 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2312000000000003e-05 2024-09-16T03:39:06.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660927+0000", 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9329999999999998e-06 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660931+0000", 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.967991+0000", 2024-09-16T03:39:06.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30706032700000002 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968039+0000", 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7477e-05 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968117+0000", 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8589000000000003e-05 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067678+0000", 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099560406000000004 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170175+0000", 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102496927 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269491+0000", 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099316205000000005 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269574+0000", 2024-09-16T03:39:06.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2903000000000002e-05 2024-09-16T03:39:06.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269614+0000", 2024-09-16T03:39:06.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0188000000000002e-05 2024-09-16T03:39:06.551 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.551 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.551 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.551 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 125.74521731, 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60879245500000001, 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 124 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660938+0000", 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660940+0000", 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2529999999999999e-06 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660943+0000", 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.01e-06 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660946+0000", 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8159999999999998e-06 2024-09-16T03:39:06.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012417+0000", 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.35147120599999998 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012497+0000", 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9963000000000007e-05 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012585+0000", 2024-09-16T03:39:06.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:39:06.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067680+0000", 2024-09-16T03:39:06.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055095203000000002 2024-09-16T03:39:06.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170243+0000", 2024-09-16T03:39:06.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10256290799999999 2024-09-16T03:39:06.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269662+0000", 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099418896000000007 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269701+0000", 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8829999999999999e-05 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269729+0000", 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8569e-05 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.557 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.558 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.558 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1550 4.13 4:c8492bfc:::smithi15736696-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:39:06.558 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:06.558 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 80.739489169999999, 2024-09-16T03:39:06.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62706974999999998, 2024-09-16T03:39:06.558 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.558 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.558 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:06.558 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.558 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:06.558 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1550 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:06.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0699999999999998e-07 2024-09-16T03:39:06.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666669+0000", 2024-09-16T03:39:06.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.393e-06 2024-09-16T03:39:06.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.560 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.560 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666673+0000", 2024-09-16T03:39:06.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:39:06.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666682+0000", 2024-09-16T03:39:06.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6510000000000007e-06 2024-09-16T03:39:06.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.913940+0000", 2024-09-16T03:39:06.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.24725763000000001 2024-09-16T03:39:06.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:06.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.914034+0000", 2024-09-16T03:39:06.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3923000000000002e-05 2024-09-16T03:39:06.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293650+0000", 2024-09-16T03:39:06.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.37961662200000001 2024-09-16T03:39:06.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293688+0000", 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8096000000000003e-05 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293735+0000", 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6451e-05 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.564 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:77 3.2 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:39:06.564 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:39:06.564 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 66.327612548999994, 2024-09-16T03:39:06.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.61452436499999996, 2024-09-16T03:39:06.564 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.564 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.564 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:06.564 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:39:06.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:39:06.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078543+0000", 2024-09-16T03:39:06.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.553e-06 2024-09-16T03:39:06.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078551+0000", 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0790000000000001e-06 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078564+0000", 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2943e-05 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178635+0000", 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10007044599999999 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178684+0000", 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9592999999999998e-05 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178747+0000", 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2286000000000006e-05 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4,5", 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178857+0000", 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110807 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.567 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280364+0000", 2024-09-16T03:39:06.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101506518 2024-09-16T03:39:06.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:06.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280373+0000", 2024-09-16T03:39:06.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6899999999999998e-06 2024-09-16T03:39:06.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:06.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.369249+0000", 2024-09-16T03:39:06.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.088876172000000003 2024-09-16T03:39:06.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693044+0000", 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.32379486499999999 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693066+0000", 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2413e-05 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3119 4.13 4:c8492bfc:::smithi15736696-42:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.474715493000001, 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.64286237700000004, 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3119 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:06.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931441+0000", 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3599999999999999e-06 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:06.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931449+0000", 2024-09-16T03:39:06.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4080000000000001e-06 2024-09-16T03:39:06.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:06.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931463+0000", 2024-09-16T03:39:06.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.488e-05 2024-09-16T03:39:06.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:06.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931473+0000", 2024-09-16T03:39:06.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8109999999999997e-06 2024-09-16T03:39:06.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:06.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.031604+0000", 2024-09-16T03:39:06.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013079499999999 2024-09-16T03:39:06.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:06.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.031668+0000", 2024-09-16T03:39:06.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4226999999999993e-05 2024-09-16T03:39:06.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:06.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:06.574 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:06.574 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.574301+0000", 2024-09-16T03:39:06.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54263289599999998 2024-09-16T03:39:06.574 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.574 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.574 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.574 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:06.574 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:06.574 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:06.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:39:06.576 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:39:06.577 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:06.574+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:06.644 INFO:tasks.rados.rados.0.smithi157.stdout:1699: set_chunk oid 25 offset: 446464 length: 1201152 target oid dce542a79a00e18d86bcad9b7e35b22c010a7ca1928705dca52eb83cbed579e2 offset: 0 snap -1 2024-09-16T03:39:06.644 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:39:06.644 INFO:tasks.rados.rados.0.smithi157.stdout:1679: finishing write tid 3 to smithi15736696-48 2024-09-16T03:39:06.644 INFO:tasks.rados.rados.0.smithi157.stdout:1674: finishing set_chunk to oid 16 2024-09-16T03:39:06.644 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 65 (ObjNum 347 snap 0 seq_num 347) dirty exists 2024-09-16T03:39:06.644 INFO:tasks.rados.rados.0.smithi157.stdout:1685: got (0) Success 2024-09-16T03:39:06.644 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 97 (ObjNum 359 snap 0 seq_num 359) dirty exists 2024-09-16T03:39:06.645 INFO:tasks.rados.rados.0.smithi157.stdout:1687: finishing write tid 2 to smithi15736696-14 2024-09-16T03:39:06.645 INFO:tasks.rados.rados.0.smithi157.stdout:1687: finishing write tid 3 to smithi15736696-14 2024-09-16T03:39:06.645 INFO:tasks.rados.rados.0.smithi157.stdout:1689: finishing write tid 1 to smithi15736696-28 2024-09-16T03:39:06.645 INFO:tasks.rados.rados.0.smithi157.stdout:1688: finishing set_chunk to oid 26 2024-09-16T03:39:06.645 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 111 (ObjNum 336 snap 0 seq_num 336) dirty exists 2024-09-16T03:39:06.645 INFO:tasks.rados.rados.0.smithi157.stdout:1679: finishing write tid 4 to smithi15736696-48 2024-09-16T03:39:06.645 INFO:tasks.rados.rados.0.smithi157.stdout:1679: oid 48 updating version 0 to 89 2024-09-16T03:39:06.645 INFO:tasks.rados.rados.0.smithi157.stdout:1679: oid 48 updating version 89 to 90 2024-09-16T03:39:06.645 INFO:tasks.rados.rados.0.smithi157.stdout:1679: oid 48 version 90 is already newer than 88 2024-09-16T03:39:06.645 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 48 v 90 (ObjNum 367 snap 0 seq_num 367) dirty exists 2024-09-16T03:39:06.645 INFO:tasks.rados.rados.0.smithi157.stdout:1679: left oid 48 (ObjNum 367 snap 0 seq_num 367) 2024-09-16T03:39:06.645 INFO:tasks.rados.rados.0.smithi157.stdout:1687: finishing write tid 4 to smithi15736696-14 2024-09-16T03:39:06.645 INFO:tasks.rados.rados.0.smithi157.stdout:1687: oid 14 updating version 0 to 50 2024-09-16T03:39:06.646 INFO:tasks.rados.rados.0.smithi157.stdout:1687: oid 14 updating version 50 to 52 2024-09-16T03:39:06.646 INFO:tasks.rados.rados.0.smithi157.stdout:1687: oid 14 version 52 is already newer than 51 2024-09-16T03:39:06.646 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 52 (ObjNum 369 snap 0 seq_num 369) dirty exists 2024-09-16T03:39:06.646 INFO:tasks.rados.rados.0.smithi157.stdout:1687: left oid 14 (ObjNum 369 snap 0 seq_num 369) 2024-09-16T03:39:06.646 INFO:tasks.rados.rados.0.smithi157.stdout:1693: got (22) Invalid argument 2024-09-16T03:39:06.646 INFO:tasks.rados.rados.0.smithi157.stdout:1674: done (15 left) 2024-09-16T03:39:06.646 INFO:tasks.rados.rados.0.smithi157.stdout:1679: done (14 left) 2024-09-16T03:39:06.646 INFO:tasks.rados.rados.0.smithi157.stdout:1685: done (13 left) 2024-09-16T03:39:06.646 INFO:tasks.rados.rados.0.smithi157.stdout:1687: done (12 left) 2024-09-16T03:39:06.646 INFO:tasks.rados.rados.0.smithi157.stdout:1688: done (11 left) 2024-09-16T03:39:06.646 INFO:tasks.rados.rados.0.smithi157.stdout:1693: done (10 left) 2024-09-16T03:39:06.646 INFO:tasks.rados.rados.0.smithi157.stdout:1700: tier_evict oid 31 2024-09-16T03:39:06.647 INFO:tasks.rados.rados.0.smithi157.stdout:1689: finishing write tid 2 to smithi15736696-28 2024-09-16T03:39:06.647 INFO:tasks.rados.rados.0.smithi157.stdout:1691: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:39:06.676 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:06.674+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:06.722 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:06.722 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:06.722 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:39:06.722 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:06.730 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:39:06.777 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:06.776+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:06.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:39:06.808 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3801037 2024-09-16T03:39:06.812 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:06.875 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:06.875 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:06.875 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:06.875 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:06.875 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:06.878 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:06.877+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:06.884 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:39:06.906 INFO:tasks.rados.rados.0.smithi157.stdout:1689: finishing write tid 3 to smithi15736696-28 2024-09-16T03:39:06.906 INFO:tasks.rados.rados.0.smithi157.stdout:1698: finishing write tid 1 to smithi15736696-7 2024-09-16T03:39:06.906 INFO:tasks.rados.rados.0.smithi157.stdout:1698: finishing write tid 2 to smithi15736696-7 2024-09-16T03:39:06.906 INFO:tasks.rados.rados.0.smithi157.stdout:1690: finishing set_chunk to oid 20 2024-09-16T03:39:06.907 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 52 (ObjNum 346 snap 0 seq_num 346) dirty exists 2024-09-16T03:39:06.907 INFO:tasks.rados.rados.0.smithi157.stdout:1695: expect (ObjNum 287 snap 0 seq_num 287) 2024-09-16T03:39:06.907 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:06.928 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:06.928 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:06.946 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:06.946 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:06.963 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:06.963 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:06.979 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:06.980 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:06.980 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:06.978+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:06.995 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:06.995 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:07.012 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:07.013 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:07.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:07.013 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:07.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.013 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:64 4.5 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:07.013 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:39:07.013 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 114.75272504, 2024-09-16T03:39:07.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50484290399999998, 2024-09-16T03:39:07.013 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.013 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.013 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:07.013 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.014 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.014 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.014 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 64 2024-09-16T03:39:07.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.014 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:39:07.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:39:07.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257341+0000", 2024-09-16T03:39:07.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:39:07.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257343+0000", 2024-09-16T03:39:07.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3889999999999999e-06 2024-09-16T03:39:07.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257346+0000", 2024-09-16T03:39:07.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:39:07.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257349+0000", 2024-09-16T03:39:07.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8559999999999998e-06 2024-09-16T03:39:07.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357057+0000", 2024-09-16T03:39:07.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099708121999999996 2024-09-16T03:39:07.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:07.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357156+0000", 2024-09-16T03:39:07.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9162999999999999e-05 2024-09-16T03:39:07.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762060+0000", 2024-09-16T03:39:07.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40490364400000001 2024-09-16T03:39:07.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762112+0000", 2024-09-16T03:39:07.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1882000000000002e-05 2024-09-16T03:39:07.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762183+0000", 2024-09-16T03:39:07.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1297000000000005e-05 2024-09-16T03:39:07.018 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.018 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.018 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:465 4.5 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 105.994360267, 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60529649299999999, 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 465 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:39:07.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:39:07.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015706+0000", 2024-09-16T03:39:07.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.091e-06 2024-09-16T03:39:07.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015712+0000", 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015724+0000", 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1408e-05 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015731+0000", 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:39:07.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216049+0000", 2024-09-16T03:39:07.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200317895 2024-09-16T03:39:07.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216178+0000", 2024-09-16T03:39:07.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012848800000000001 2024-09-16T03:39:07.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:07.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620010+0000", 2024-09-16T03:39:07.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40383207199999999 2024-09-16T03:39:07.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:07.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620710+0000", 2024-09-16T03:39:07.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069988400000000001 2024-09-16T03:39:07.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620902+0000", 2024-09-16T03:39:07.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019197800000000001 2024-09-16T03:39:07.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620933+0000", 2024-09-16T03:39:07.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1513999999999997e-05 2024-09-16T03:39:07.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:07.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620968+0000", 2024-09-16T03:39:07.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4524999999999999e-05 2024-09-16T03:39:07.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.621002+0000", 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3955000000000003e-05 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:10 3.3 3:deac805c:::smithi15736696-44 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 100.602865291, 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53191748100000003, 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:07.026 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:39:07.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:39:07.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407202+0000", 2024-09-16T03:39:07.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.855e-06 2024-09-16T03:39:07.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407208+0000", 2024-09-16T03:39:07.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9819999999999996e-06 2024-09-16T03:39:07.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407220+0000", 2024-09-16T03:39:07.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2191999999999999e-05 2024-09-16T03:39:07.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407236+0000", 2024-09-16T03:39:07.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5698000000000001e-05 2024-09-16T03:39:07.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538167+0000", 2024-09-16T03:39:07.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.13093111499999999 2024-09-16T03:39:07.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538229+0000", 2024-09-16T03:39:07.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2441999999999996e-05 2024-09-16T03:39:07.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:39:07.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538346+0000", 2024-09-16T03:39:07.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011716400000000001 2024-09-16T03:39:07.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:07.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540757+0000", 2024-09-16T03:39:07.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024105089999999999 2024-09-16T03:39:07.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.639501+0000", 2024-09-16T03:39:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098743545000000002 2024-09-16T03:39:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939038+0000", 2024-09-16T03:39:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29953765700000001 2024-09-16T03:39:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939075+0000", 2024-09-16T03:39:07.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6600999999999997e-05 2024-09-16T03:39:07.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939118+0000", 2024-09-16T03:39:07.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2721e-05 2024-09-16T03:39:07.032 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.032 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.032 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.032 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:984 4.15 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:07.032 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:39:07.032 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 96.637945363, 2024-09-16T03:39:07.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.97836159599999994, 2024-09-16T03:39:07.033 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.033 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.033 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:07.033 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.033 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.033 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.033 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 984 2024-09-16T03:39:07.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.033 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:39:07.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:39:07.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372122+0000", 2024-09-16T03:39:07.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.934e-06 2024-09-16T03:39:07.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372129+0000", 2024-09-16T03:39:07.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8029999999999999e-06 2024-09-16T03:39:07.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372141+0000", 2024-09-16T03:39:07.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2006e-05 2024-09-16T03:39:07.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.035 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472234+0000", 2024-09-16T03:39:07.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100092985 2024-09-16T03:39:07.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472318+0000", 2024-09-16T03:39:07.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4257000000000003e-05 2024-09-16T03:39:07.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:07.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472326+0000", 2024-09-16T03:39:07.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8960000000000003e-06 2024-09-16T03:39:07.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350293+0000", 2024-09-16T03:39:07.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.87796699899999997 2024-09-16T03:39:07.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350433+0000", 2024-09-16T03:39:07.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013999600000000001 2024-09-16T03:39:07.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350482+0000", 2024-09-16T03:39:07.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8720000000000001e-05 2024-09-16T03:39:07.037 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1000 4.15 4:aff4cc41:::smithi15736696-11:head [read 0~2873964,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 96.419506278, 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.761707841, 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.038 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1000 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590560+0000", 2024-09-16T03:39:07.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:39:07.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590562+0000", 2024-09-16T03:39:07.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.17e-06 2024-09-16T03:39:07.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590568+0000", 2024-09-16T03:39:07.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7939999999999997e-06 2024-09-16T03:39:07.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590572+0000", 2024-09-16T03:39:07.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1300000000000001e-06 2024-09-16T03:39:07.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590619+0000", 2024-09-16T03:39:07.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7148999999999999e-05 2024-09-16T03:39:07.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590625+0000", 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5679999999999996e-06 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350858+0000", 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76023295000000002 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350973+0000", 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115023 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352267+0000", 2024-09-16T03:39:07.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012935970000000001 2024-09-16T03:39:07.043 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.043 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.043 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.043 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1001 4.15 4:aff4cc41:::smithi15736696-11:head [read 1254674~1619290,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:07.043 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:07.043 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 96.419487692999994, 2024-09-16T03:39:07.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76275827799999996, 2024-09-16T03:39:07.043 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.044 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.044 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:07.044 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.044 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.044 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.044 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1001 2024-09-16T03:39:07.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.044 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:07.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:07.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:07.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:39:07.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590580+0000", 2024-09-16T03:39:07.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:39:07.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590584+0000", 2024-09-16T03:39:07.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8800000000000001e-06 2024-09-16T03:39:07.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590587+0000", 2024-09-16T03:39:07.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6630000000000002e-06 2024-09-16T03:39:07.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590650+0000", 2024-09-16T03:39:07.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3950999999999998e-05 2024-09-16T03:39:07.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:07.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590654+0000", 2024-09-16T03:39:07.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.89e-06 2024-09-16T03:39:07.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352309+0000", 2024-09-16T03:39:07.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76165495699999997 2024-09-16T03:39:07.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352407+0000", 2024-09-16T03:39:07.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7188999999999994e-05 2024-09-16T03:39:07.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.353336+0000", 2024-09-16T03:39:07.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092939400000000005 2024-09-16T03:39:07.048 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.048 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.048 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1002 4.15 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~1527025] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 96.419458438999996, 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85963523600000002, 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1002 2024-09-16T03:39:07.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.050 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:07.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:07.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:07.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.15e-07 2024-09-16T03:39:07.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590609+0000", 2024-09-16T03:39:07.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.606e-06 2024-09-16T03:39:07.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590615+0000", 2024-09-16T03:39:07.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3040000000000001e-06 2024-09-16T03:39:07.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590618+0000", 2024-09-16T03:39:07.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.309e-06 2024-09-16T03:39:07.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590669+0000", 2024-09-16T03:39:07.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1048000000000003e-05 2024-09-16T03:39:07.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:07.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590673+0000", 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.061e-06 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.449932+0000", 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85925979699999999 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450032+0000", 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9858000000000005e-05 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450242+0000", 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020993800000000001 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2467 4.5 4:a11e1d7c:::smithi15736696-43:head [write 1772395~538068] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.018782068, 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.56288137400000005, 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.054 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:07.055 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.055 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.055 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.055 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2467 2024-09-16T03:39:07.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.055 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:39:07.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:39:07.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991285+0000", 2024-09-16T03:39:07.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0990000000000001e-06 2024-09-16T03:39:07.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991447+0000", 2024-09-16T03:39:07.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016170099999999999 2024-09-16T03:39:07.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991458+0000", 2024-09-16T03:39:07.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.128e-05 2024-09-16T03:39:07.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991464+0000", 2024-09-16T03:39:07.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3800000000000002e-06 2024-09-16T03:39:07.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150823+0000", 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15935943799999999 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150899+0000", 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5359999999999997e-05 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151007+0000", 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108317 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253992+0000", 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102985249 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.353797+0000", 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099805039999999998 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554039+0000", 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20024175899999999 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:07.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554129+0000", 2024-09-16T03:39:07.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0067000000000004e-05 2024-09-16T03:39:07.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554165+0000", 2024-09-16T03:39:07.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5683999999999999e-05 2024-09-16T03:39:07.060 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.060 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.060 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.061 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2468 4.5 4:a11e1d7c:::smithi15736696-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:07.061 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:39:07.061 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.013361289000002, 2024-09-16T03:39:07.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.55765803899999999, 2024-09-16T03:39:07.061 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.061 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.061 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:07.061 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.061 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.061 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.061 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2468 2024-09-16T03:39:07.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.061 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:39:07.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:39:07.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996708+0000", 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5149999999999998e-06 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996921+0000", 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000213692 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996937+0000", 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5699999999999999e-05 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996946+0000", 2024-09-16T03:39:07.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7159999999999992e-06 2024-09-16T03:39:07.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152362+0000", 2024-09-16T03:39:07.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15541682800000001 2024-09-16T03:39:07.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152421+0000", 2024-09-16T03:39:07.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-16T03:39:07.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:07.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152515+0000", 2024-09-16T03:39:07.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3659999999999994e-05 2024-09-16T03:39:07.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:07.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253995+0000", 2024-09-16T03:39:07.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10148022499999999 2024-09-16T03:39:07.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.453908+0000", 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19991299700000001 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554321+0000", 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100412977 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554344+0000", 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.262e-05 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554362+0000", 2024-09-16T03:39:07.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8539999999999999e-05 2024-09-16T03:39:07.067 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.067 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.067 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.067 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2469 4.5 4:a11e1d7c:::smithi15736696-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:07.067 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:39:07.067 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.013105926000001, 2024-09-16T03:39:07.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.75785074799999996, 2024-09-16T03:39:07.067 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.068 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.068 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:07.068 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.068 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.068 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.068 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2469 2024-09-16T03:39:07.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.068 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:39:07.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996960+0000", 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3000000000000001e-07 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996963+0000", 2024-09-16T03:39:07.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8679999999999999e-06 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996967+0000", 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.622e-06 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996971+0000", 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1429999999999998e-06 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253536+0000", 2024-09-16T03:39:07.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25656582900000002 2024-09-16T03:39:07.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253632+0000", 2024-09-16T03:39:07.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6067e-05 2024-09-16T03:39:07.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:07.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253745+0000", 2024-09-16T03:39:07.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011256699999999999 2024-09-16T03:39:07.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:07.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254218+0000", 2024-09-16T03:39:07.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00047321400000000001 2024-09-16T03:39:07.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554397+0000", 2024-09-16T03:39:07.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30017854399999999 2024-09-16T03:39:07.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754557+0000", 2024-09-16T03:39:07.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200159848 2024-09-16T03:39:07.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754646+0000", 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9370999999999995e-05 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754810+0000", 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016414500000000001 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.074 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2470 4.5 4:a11e1d7c:::smithi15736696-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:07.074 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:07.074 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 51.013089596999997, 2024-09-16T03:39:07.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76022711799999998, 2024-09-16T03:39:07.074 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.074 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.074 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:07.074 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.074 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.074 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.074 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2470 2024-09-16T03:39:07.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.074 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:07.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.65e-07 2024-09-16T03:39:07.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996977+0000", 2024-09-16T03:39:07.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4079999999999999e-06 2024-09-16T03:39:07.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996981+0000", 2024-09-16T03:39:07.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-16T03:39:07.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996988+0000", 2024-09-16T03:39:07.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0890000000000002e-06 2024-09-16T03:39:07.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253947+0000", 2024-09-16T03:39:07.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25695931100000002 2024-09-16T03:39:07.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.077 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.077 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:07.077 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253985+0000", 2024-09-16T03:39:07.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7939999999999999e-05 2024-09-16T03:39:07.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757090+0000", 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50310477499999995 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757140+0000", 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9567999999999997e-05 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757203+0000", 2024-09-16T03:39:07.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3215999999999999e-05 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2487 4.15 4:ad0b9852:::smithi15736696-16:head [write 1947841~685395] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.594045006999998, 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.74140762199999999, 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.079 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.080 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.080 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2487 2024-09-16T03:39:07.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.080 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:39:07.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:39:07.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416023+0000", 2024-09-16T03:39:07.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:39:07.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416341+0000", 2024-09-16T03:39:07.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031881699999999999 2024-09-16T03:39:07.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416353+0000", 2024-09-16T03:39:07.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.189e-05 2024-09-16T03:39:07.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416360+0000", 2024-09-16T03:39:07.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5719999999999999e-06 2024-09-16T03:39:07.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755578+0000", 2024-09-16T03:39:07.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33921859199999999 2024-09-16T03:39:07.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755639+0000", 2024-09-16T03:39:07.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0615000000000002e-05 2024-09-16T03:39:07.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:07.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755735+0000", 2024-09-16T03:39:07.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5798999999999995e-05 2024-09-16T03:39:07.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858633+0000", 2024-09-16T03:39:07.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10289859599999999 2024-09-16T03:39:07.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:07.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858695+0000", 2024-09-16T03:39:07.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2000000000000003e-05 2024-09-16T03:39:07.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.084 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.084 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157315+0000", 2024-09-16T03:39:07.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29861931600000002 2024-09-16T03:39:07.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:07.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157375+0000", 2024-09-16T03:39:07.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0268e-05 2024-09-16T03:39:07.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157428+0000", 2024-09-16T03:39:07.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2939999999999998e-05 2024-09-16T03:39:07.085 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.085 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.085 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2488 4.15 4:ad0b9852:::smithi15736696-16:head [write 3383215~196676] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.588372356999997, 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73589615900000005, 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2488 2024-09-16T03:39:07.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.087 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:39:07.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:39:07.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421695+0000", 2024-09-16T03:39:07.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1469999999999999e-06 2024-09-16T03:39:07.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421975+0000", 2024-09-16T03:39:07.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028008299999999998 2024-09-16T03:39:07.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421987+0000", 2024-09-16T03:39:07.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1294000000000001e-05 2024-09-16T03:39:07.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421992+0000", 2024-09-16T03:39:07.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9349999999999999e-06 2024-09-16T03:39:07.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757205+0000", 2024-09-16T03:39:07.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33521242800000001 2024-09-16T03:39:07.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757269+0000", 2024-09-16T03:39:07.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3761000000000002e-05 2024-09-16T03:39:07.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:07.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757376+0000", 2024-09-16T03:39:07.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010706599999999999 2024-09-16T03:39:07.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858684+0000", 2024-09-16T03:39:07.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101308255 2024-09-16T03:39:07.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:07.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858700+0000", 2024-09-16T03:39:07.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6138999999999999e-05 2024-09-16T03:39:07.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157491+0000", 2024-09-16T03:39:07.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29879056999999998 2024-09-16T03:39:07.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.092 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:07.092 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157546+0000", 2024-09-16T03:39:07.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5668999999999998e-05 2024-09-16T03:39:07.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.092 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.092 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157589+0000", 2024-09-16T03:39:07.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2812000000000003e-05 2024-09-16T03:39:07.092 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.092 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2489 4.15 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=7b,truncate 3579891] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.588063535000003, 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73575981700000004, 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.093 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2489 2024-09-16T03:39:07.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.094 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:07.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:07.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0200000000000002e-07 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422005+0000", 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7379999999999999e-06 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422009+0000", 2024-09-16T03:39:07.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6219999999999998e-06 2024-09-16T03:39:07.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422016+0000", 2024-09-16T03:39:07.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5789999999999998e-06 2024-09-16T03:39:07.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858071+0000", 2024-09-16T03:39:07.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.436054302 2024-09-16T03:39:07.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858130+0000", 2024-09-16T03:39:07.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9493000000000001e-05 2024-09-16T03:39:07.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:07.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858226+0000", 2024-09-16T03:39:07.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6174000000000004e-05 2024-09-16T03:39:07.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:39:07.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10106483199999999 2024-09-16T03:39:07.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:07.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959639+0000", 2024-09-16T03:39:07.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034802999999999999 2024-09-16T03:39:07.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157640+0000", 2024-09-16T03:39:07.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19800045299999999 2024-09-16T03:39:07.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:07.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157702+0000", 2024-09-16T03:39:07.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2218000000000006e-05 2024-09-16T03:39:07.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157762+0000", 2024-09-16T03:39:07.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9774000000000001e-05 2024-09-16T03:39:07.099 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.100 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.100 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.100 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2490 4.15 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:07.100 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:39:07.100 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.588044943, 2024-09-16T03:39:07.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73597824999999994, 2024-09-16T03:39:07.100 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.100 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.100 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:07.100 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.100 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.101 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.101 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2490 2024-09-16T03:39:07.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.101 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:39:07.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.101 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.101 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:39:07.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422021+0000", 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8099999999999999e-07 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422022+0000", 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.573e-06 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422027+0000", 2024-09-16T03:39:07.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3250000000000001e-06 2024-09-16T03:39:07.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422030+0000", 2024-09-16T03:39:07.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.948e-06 2024-09-16T03:39:07.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858510+0000", 2024-09-16T03:39:07.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.43648032599999997 2024-09-16T03:39:07.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858588+0000", 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7888999999999999e-05 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157803+0000", 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.299215642 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157893+0000", 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9565000000000001e-05 2024-09-16T03:39:07.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157999+0000", 2024-09-16T03:39:07.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105701 2024-09-16T03:39:07.105 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.105 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.105 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.105 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2492 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:07.106 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:07.106 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.588032433999999, 2024-09-16T03:39:07.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53721099400000005, 2024-09-16T03:39:07.106 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.106 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.106 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:07.106 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.106 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.106 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.106 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2492 2024-09-16T03:39:07.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.106 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:39:07.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422036+0000", 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5900000000000002e-06 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422043+0000", 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3069999999999997e-06 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422048+0000", 2024-09-16T03:39:07.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6619999999999996e-06 2024-09-16T03:39:07.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958774+0000", 2024-09-16T03:39:07.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53672593499999999 2024-09-16T03:39:07.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958868+0000", 2024-09-16T03:39:07.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4408999999999997e-05 2024-09-16T03:39:07.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959244+0000", 2024-09-16T03:39:07.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037581099999999998 2024-09-16T03:39:07.110 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.110 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.110 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.110 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2493 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:07.110 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:07.110 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.588012655, 2024-09-16T03:39:07.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53757571299999996, 2024-09-16T03:39:07.110 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.110 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2493 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3600000000000004e-07 2024-09-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422055+0000", 2024-09-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7460000000000001e-06 2024-09-16T03:39:07.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422058+0000", 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422061+0000", 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6759999999999999e-06 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:39:07.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53723057900000004 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959367+0000", 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5158000000000001e-05 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959628+0000", 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026175800000000001 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2494 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 1738052~782998] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.588001624999997, 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53787640199999998, 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2494 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.115 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:07.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:07.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6100000000000002e-07 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422065+0000", 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2839999999999999e-06 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422069+0000", 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7239999999999998e-06 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422072+0000", 2024-09-16T03:39:07.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6560000000000001e-06 2024-09-16T03:39:07.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959661+0000", 2024-09-16T03:39:07.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53758982600000005 2024-09-16T03:39:07.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959739+0000", 2024-09-16T03:39:07.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7581000000000001e-05 2024-09-16T03:39:07.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959940+0000", 2024-09-16T03:39:07.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020107 2024-09-16T03:39:07.119 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.119 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.119 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.119 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3987 4.36 4:6ef3b53d:::smithi15736696-48:head [setxattr _header (56) in=7b,truncate 2168169] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:07.119 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:39:07.119 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.63394774, 2024-09-16T03:39:07.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50651914200000003, 2024-09-16T03:39:07.120 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.120 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.120 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:07.120 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.120 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.120 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.120 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3987 2024-09-16T03:39:07.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.120 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:39:07.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.121 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.121 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:39:07.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376119+0000", 2024-09-16T03:39:07.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3799999999999996e-07 2024-09-16T03:39:07.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376122+0000", 2024-09-16T03:39:07.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.202e-06 2024-09-16T03:39:07.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376251+0000", 2024-09-16T03:39:07.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012896699999999999 2024-09-16T03:39:07.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376259+0000", 2024-09-16T03:39:07.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.32e-06 2024-09-16T03:39:07.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379368+0000", 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031084989999999998 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379441+0000", 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3331999999999995e-05 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,10", 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379559+0000", 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118068 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:07.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380350+0000", 2024-09-16T03:39:07.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00079103699999999997 2024-09-16T03:39:07.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.580657+0000", 2024-09-16T03:39:07.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20030706600000001 2024-09-16T03:39:07.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:07.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882497+0000", 2024-09-16T03:39:07.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30183998899999998 2024-09-16T03:39:07.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:07.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882560+0000", 2024-09-16T03:39:07.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3372000000000002e-05 2024-09-16T03:39:07.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882637+0000", 2024-09-16T03:39:07.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6352000000000004e-05 2024-09-16T03:39:07.126 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.126 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.126 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3988 4.36 4:6ef3b53d:::smithi15736696-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.6337938919999999, 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60662574199999997, 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3988 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.127 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376273+0000", 2024-09-16T03:39:07.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3170000000000001e-06 2024-09-16T03:39:07.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:07.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376278+0000", 2024-09-16T03:39:07.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7849999999999999e-06 2024-09-16T03:39:07.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.129 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:07.129 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376286+0000", 2024-09-16T03:39:07.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2009999999999999e-06 2024-09-16T03:39:07.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.129 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376291+0000", 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4029999999999998e-06 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380020+0000", 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037289630000000001 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380065+0000", 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4666999999999998e-05 2024-09-16T03:39:07.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:07.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982800+0000", 2024-09-16T03:39:07.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60273539099999995 2024-09-16T03:39:07.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:07.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982838+0000", 2024-09-16T03:39:07.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8155999999999999e-05 2024-09-16T03:39:07.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:07.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:07.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:07.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982897+0000", 2024-09-16T03:39:07.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8859e-05 2024-09-16T03:39:07.132 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.132 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.132 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.132 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:07.132 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:07.132 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:07.133 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:07.134 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:07.132+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:07.134 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:07.134 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:07.135 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:07.135 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:07.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:39:07.136 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:07.136 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:07.137 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:07.137 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:07.137 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:07.137 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:07.138 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:07.138 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:07.138 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:07.139 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:07.139 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:07.139 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:07.145 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:07.145 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:07.162 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:07.162 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:07.178 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:07.179 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:07.179 INFO:tasks.rados.rados.0.smithi157.stdout:1696: finishing write tid 1 to smithi15736696-41 2024-09-16T03:39:07.179 INFO:tasks.rados.rados.0.smithi157.stdout:1697: expect (ObjNum 355 snap 0 seq_num 355) 2024-09-16T03:39:07.234 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:07.232+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:07.247 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:07.247 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:07.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:39:07.270 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:07.268+0000 7f6528e68640 -1 osd.1 131 handle_osd_map injecting map crc failure 2024-09-16T03:39:07.275 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:07.334 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:07.333+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:07.403 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:07.403 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:07.411 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:39:07.436 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:07.435+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:07.500 INFO:tasks.rados.rados.0.smithi157.stdout:1698: finishing write tid 3 to smithi15736696-7 2024-09-16T03:39:07.500 INFO:tasks.rados.rados.0.smithi157.stdout:1698: finishing write tid 4 to smithi15736696-7 2024-09-16T03:39:07.500 INFO:tasks.rados.rados.0.smithi157.stdout:1698: oid 7 updating version 0 to 93 2024-09-16T03:39:07.500 INFO:tasks.rados.rados.0.smithi157.stdout:1698: oid 7 version 93 is already newer than 91 2024-09-16T03:39:07.500 INFO:tasks.rados.rados.0.smithi157.stdout:1698: oid 7 version 93 is already newer than 92 2024-09-16T03:39:07.500 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 7 v 93 (ObjNum 372 snap 0 seq_num 372) dirty exists 2024-09-16T03:39:07.500 INFO:tasks.rados.rados.0.smithi157.stdout:1698: left oid 7 (ObjNum 372 snap 0 seq_num 372) 2024-09-16T03:39:07.500 INFO:tasks.rados.rados.0.smithi157.stdout:1689: finishing write tid 4 to smithi15736696-28 2024-09-16T03:39:07.500 INFO:tasks.rados.rados.0.smithi157.stdout:1689: oid 28 updating version 0 to 126 2024-09-16T03:39:07.500 INFO:tasks.rados.rados.0.smithi157.stdout:1689: oid 28 version 126 is already newer than 125 2024-09-16T03:39:07.500 INFO:tasks.rados.rados.0.smithi157.stdout:1689: oid 28 updating version 126 to 127 2024-09-16T03:39:07.501 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 127 (ObjNum 370 snap 0 seq_num 370) dirty exists 2024-09-16T03:39:07.501 INFO:tasks.rados.rados.0.smithi157.stdout:1689: left oid 28 (ObjNum 370 snap 0 seq_num 370) 2024-09-16T03:39:07.501 INFO:tasks.rados.rados.0.smithi157.stdout:1692: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-16T03:39:07.518 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:07.519 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:07.527 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:39:07.531 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.15 to 1 (10000) 2024-09-16T03:39:07.531 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:39:07.536 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:07.534+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:07.547 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-16T03:39:07.638 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:07.638 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:07.638 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:07.636+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:07.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:39:07.739 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:07.737+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:07.756 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:07.756 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:07.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:39:07.776 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2975122 2024-09-16T03:39:07.796 INFO:tasks.rados.rados.0.smithi157.stdout:1694: expect (ObjNum 344 snap 0 seq_num 344) 2024-09-16T03:39:07.840 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:07.839+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:07.873 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:07.873 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:07.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:39:07.928 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:07.941 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:07.939+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:07.989 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:07.989 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:07.998 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:39:08.006 INFO:tasks.rados.rados.0.smithi157.stdout:1696: finishing write tid 2 to smithi15736696-41 2024-09-16T03:39:08.006 INFO:tasks.rados.rados.0.smithi157.stdout:1696: finishing write tid 3 to smithi15736696-41 2024-09-16T03:39:08.006 INFO:tasks.rados.rados.0.smithi157.stdout:1696: finishing write tid 4 to smithi15736696-41 2024-09-16T03:39:08.006 INFO:tasks.rados.rados.0.smithi157.stdout:1696: oid 41 updating version 0 to 50 2024-09-16T03:39:08.006 INFO:tasks.rados.rados.0.smithi157.stdout:1696: oid 41 updating version 50 to 51 2024-09-16T03:39:08.006 INFO:tasks.rados.rados.0.smithi157.stdout:1696: oid 41 updating version 51 to 52 2024-09-16T03:39:08.006 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 52 (ObjNum 371 snap 0 seq_num 371) dirty exists 2024-09-16T03:39:08.006 INFO:tasks.rados.rados.0.smithi157.stdout:1696: left oid 41 (ObjNum 371 snap 0 seq_num 371) 2024-09-16T03:39:08.006 INFO:tasks.rados.rados.0.smithi157.stdout:1700: tier_evict oid 31 snap -1 2024-09-16T03:39:08.006 INFO:tasks.rados.rados.0.smithi157.stdout:1699: finishing set_chunk to oid 25 2024-09-16T03:39:08.006 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 122 (ObjNum 295 snap 0 seq_num 295) dirty exists 2024-09-16T03:39:08.006 INFO:tasks.rados.rados.0.smithi157.stdout:1689: done (10 left) 2024-09-16T03:39:08.007 INFO:tasks.rados.rados.0.smithi157.stdout:1690: done (9 left) 2024-09-16T03:39:08.007 INFO:tasks.rados.rados.0.smithi157.stdout:1691: done (8 left) 2024-09-16T03:39:08.007 INFO:tasks.rados.rados.0.smithi157.stdout:1692: done (7 left) 2024-09-16T03:39:08.007 INFO:tasks.rados.rados.0.smithi157.stdout:1694: done (6 left) 2024-09-16T03:39:08.007 INFO:tasks.rados.rados.0.smithi157.stdout:1695: done (5 left) 2024-09-16T03:39:08.007 INFO:tasks.rados.rados.0.smithi157.stdout:1696: done (4 left) 2024-09-16T03:39:08.007 INFO:tasks.rados.rados.0.smithi157.stdout:1697: done (3 left) 2024-09-16T03:39:08.007 INFO:tasks.rados.rados.0.smithi157.stdout:1698: done (2 left) 2024-09-16T03:39:08.007 INFO:tasks.rados.rados.0.smithi157.stdout:1699: done (1 left) 2024-09-16T03:39:08.007 INFO:tasks.rados.rados.0.smithi157.stdout:1701: tier_evict oid 36 2024-09-16T03:39:08.007 INFO:tasks.rados.rados.0.smithi157.stdout:1701: tier_evict oid 36 snap -1 2024-09-16T03:39:08.007 INFO:tasks.rados.rados.0.smithi157.stdout:1702: set_chunk oid 10 target oid 2024-09-16T03:39:08.042 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:08.041+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:08.105 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:08.105 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:08.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:39:08.157 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:08.156+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:08.221 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:08.222 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:08.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:39:08.272 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:08.270+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:08.314 INFO:tasks.rados.rados.0.smithi157.stdout:1702: set_chunk oid 10 offset: 1886208 length: 628224 target oid ba32fe3bb387087d04554cac21317fa22464e7a823baadddd710b9e5290e0961 offset: 0 snap -1 2024-09-16T03:39:08.314 INFO:tasks.rados.rados.0.smithi157.stdout:1700: got (22) Invalid argument 2024-09-16T03:39:08.314 INFO:tasks.rados.rados.0.smithi157.stdout:1701: got (22) Invalid argument 2024-09-16T03:39:08.314 INFO:tasks.rados.rados.0.smithi157.stdout:1703: read oid 42 snap -1 2024-09-16T03:39:08.314 INFO:tasks.rados.rados.0.smithi157.stdout:1703: expect deleted 2024-09-16T03:39:08.314 INFO:tasks.rados.rados.0.smithi157.stdout:1700: done (3 left) 2024-09-16T03:39:08.314 INFO:tasks.rados.rados.0.smithi157.stdout:1701: done (2 left) 2024-09-16T03:39:08.314 INFO:tasks.rados.rados.0.smithi157.stdout:1704: read oid 29 snap -1 2024-09-16T03:39:08.315 INFO:tasks.rados.rados.0.smithi157.stdout:1704: expect (ObjNum 344 snap 0 seq_num 344) 2024-09-16T03:39:08.315 INFO:tasks.rados.rados.0.smithi157.stdout:1705: set_chunk oid 34 target oid 2024-09-16T03:39:08.345 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:08.344+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:08.371 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:08.369+0000 7f6528e68640 -1 osd.1 132 handle_osd_map injecting map crc failure 2024-09-16T03:39:08.380 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:08.380 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:08.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:39:08.446 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:08.444+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:08.496 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:08.496 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:08.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:39:08.528 INFO:tasks.rados.rados.0.smithi157.stdout:1705: set_chunk oid 34 offset: 869376 length: 1458688 target oid 16a36d133f9f2b003f497df01d55330c8d3302d4ff647000f6aefc973327fc05 offset: 0 snap -1 2024-09-16T03:39:08.528 INFO:tasks.rados.rados.0.smithi157.stdout:1706: read oid 16 snap -1 2024-09-16T03:39:08.528 INFO:tasks.rados.rados.0.smithi157.stdout:1706: expect (ObjNum 347 snap 0 seq_num 347) 2024-09-16T03:39:08.528 INFO:tasks.rados.rados.0.smithi157.stdout:1707: delete oid 20 current snap is 0 2024-09-16T03:39:08.528 INFO:tasks.rados.rados.0.smithi157.stdout:1702: finishing set_chunk to oid 10 2024-09-16T03:39:08.528 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 10 set_chunk returned error code -95 offset: 1886208 length: 628224 2024-09-16T03:39:08.528 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 0 (ObjNum 276 snap 0 seq_num 276) dirty exists 2024-09-16T03:39:08.528 INFO:tasks.rados.rados.0.smithi157.stdout:1705: finishing set_chunk to oid 34 2024-09-16T03:39:08.528 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 34 set_chunk returned error code -95 offset: 869376 length: 1458688 2024-09-16T03:39:08.528 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 0 (ObjNum 342 snap 0 seq_num 342) dirty exists 2024-09-16T03:39:08.531 INFO:tasks.rados.rados.0.smithi157.stdout:1706: expect (ObjNum 347 snap 0 seq_num 347) 2024-09-16T03:39:08.537 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:08.536+0000 7f8be76d1640 -1 osd.7 133 handle_osd_map injecting map crc failure 2024-09-16T03:39:08.540 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T03:39:08.543 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:08.542+0000 7fd101e6f640 -1 osd.2 133 handle_osd_map injecting map crc failure 2024-09-16T03:39:08.557 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T03:39:08.558 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:39:08.558 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:08.555+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:08.647 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:08.645+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:08.651 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:08.651 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:08.658 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:08.661 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:39:08.731 INFO:tasks.rados.rados.0.smithi157.stdout:1704: expect (ObjNum 344 snap 0 seq_num 344) 2024-09-16T03:39:08.731 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:08.748 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:08.746+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:08.776 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:08.776 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:08.785 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:39:08.808 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 359653 2024-09-16T03:39:08.848 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:08.847+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:08.900 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:08.901 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:39:08.912 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-16T03:39:08.536923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215256347,"num_objects":137,"num_object_clones":0,"num_object_copies":411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":1834,"num_read_kb":800924,"num_write":1711,"num_write_kb":370607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":61872448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":187,"ondisk_log_size":187,"up":204,"acting":204,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":220,"num_osds":13,"num_per_pool_osds":13,"num_per_pool_omap_osds":13,"kb":1226833920,"kb_used":837432,"kb_used_data":401572,"kb_used_omap":20,"kb_used_meta":405163,"kb_avail":1225996488,"statfs":{"total":1256277934080,"available":1255420403712,"internally_reserved":0,"allocated":411209728,"data_stored":418089987,"data_compressed":10329280,"data_compressed_allocated":11165696,"data_compressed_original":24604672,"omap_allocated":20735,"internal_metadata":414887681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7439868,"num_objects":-31,"num_object_clones":0,"num_object_copies":-93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-309,"num_read_kb":-304189,"num_write":-80,"num_write_kb":-32977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-15741625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018136"},"pg_stats":[{"pgid":"4.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:39:00.487335+0000","last_change":"2024-09-16T03:39:00.487335+0000","last_active":"2024-09-16T03:39:00.487335+0000","last_peered":"2024-09-16T03:39:00.487335+0000","last_clean":"2024-09-16T03:39:00.487335+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:39:00.487335+0000","last_undegraded":"2024-09-16T03:39:00.487335+0000","last_fullsized":"2024-09-16T03:39:00.487335+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:00.487335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:00.487335+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.487335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.36","version":"131'93","reported_seq":460,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.251987+0000","last_change":"2024-09-16T03:39:03.716194+0000","last_active":"2024-09-16T03:39:06.251987+0000","last_peered":"2024-09-16T03:39:06.251987+0000","last_clean":"2024-09-16T03:39:06.251987+0000","last_became_active":"2024-09-16T03:39:02.099320+0000","last_became_peered":"2024-09-16T03:39:02.099320+0000","last_unstale":"2024-09-16T03:39:06.251987+0000","last_undegraded":"2024-09-16T03:39:06.251987+0000","last_fullsized":"2024-09-16T03:39:06.251987+0000","mapping_epoch":126,"log_start":"130'90","ondisk_log_start":"130'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.716125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.716125+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":88,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.328700+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269209,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":4665,"num_write":8,"num_write_kb":2101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.35","version":"123'65","reported_seq":429,"reported_epoch":129,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:39:04.236629+0000","last_change":"2024-09-16T03:39:04.117063+0000","last_active":"2024-09-16T03:39:04.236629+0000","last_peered":"2024-09-16T03:39:04.236629+0000","last_clean":"2024-09-16T03:38:49.553429+0000","last_became_active":"2024-09-16T03:39:03.915711+0000","last_became_peered":"2024-09-16T03:39:03.915711+0000","last_unstale":"2024-09-16T03:39:04.236629+0000","last_undegraded":"2024-09-16T03:39:04.236629+0000","last_fullsized":"2024-09-16T03:39:04.236629+0000","mapping_epoch":128,"log_start":"123'62","ondisk_log_start":"123'62","created":126,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916015,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84364,"num_write":19,"num_write_kb":5459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3131048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,6],"acting":[3,10,9],"avail_no_missing":["3","6","9","10","11","15"],"object_location_counts":[{"shards":"3,9,10","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"104'20","reported_seq":317,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622626+0000","last_change":"2024-09-16T03:39:02.919356+0000","last_active":"2024-09-16T03:39:05.622626+0000","last_peered":"2024-09-16T03:39:05.622626+0000","last_clean":"2024-09-16T03:39:05.622626+0000","last_became_active":"2024-09-16T03:39:01.699286+0000","last_became_peered":"2024-09-16T03:39:01.699286+0000","last_unstale":"2024-09-16T03:39:05.622626+0000","last_undegraded":"2024-09-16T03:39:05.622626+0000","last_fullsized":"2024-09-16T03:39:05.622626+0000","mapping_epoch":126,"log_start":"98'17","ondisk_log_start":"98'17","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:02.919259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.919259+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.209096+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2705960,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"108'72","reported_seq":449,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.730259+0000","last_change":"2024-09-16T03:39:02.143407+0000","last_active":"2024-09-16T03:39:03.730259+0000","last_peered":"2024-09-16T03:39:03.730259+0000","last_clean":"2024-09-16T03:39:03.730259+0000","last_became_active":"2024-09-16T03:39:01.642310+0000","last_became_peered":"2024-09-16T03:39:01.642310+0000","last_unstale":"2024-09-16T03:39:03.730259+0000","last_undegraded":"2024-09-16T03:39:03.730259+0000","last_fullsized":"2024-09-16T03:39:03.730259+0000","mapping_epoch":126,"log_start":"108'70","ondisk_log_start":"108'70","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'72","last_scrub_stamp":"2024-09-16T03:39:02.143339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.143339+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":68,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:29.596134+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2722767,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":6432,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,14],"acting":[4,3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.32","version":"106'17","reported_seq":332,"reported_epoch":129,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:39:04.023713+0000","last_change":"2024-09-16T03:39:04.023713+0000","last_active":"2024-09-16T03:39:04.023713+0000","last_peered":"2024-09-16T03:39:04.023713+0000","last_clean":"2024-09-16T03:38:49.359852+0000","last_became_active":"2024-09-16T03:39:03.822701+0000","last_became_peered":"2024-09-16T03:39:03.822701+0000","last_unstale":"2024-09-16T03:39:04.023713+0000","last_undegraded":"2024-09-16T03:39:04.023713+0000","last_fullsized":"2024-09-16T03:39:04.023713+0000","mapping_epoch":128,"log_start":"82'13","ondisk_log_start":"82'13","created":126,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1033647,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":26745,"num_write":6,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1331059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,11],"acting":[14,2,9],"avail_no_missing":["14","2","9"],"object_location_counts":[{"shards":"2,9,14","objects":1}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.31","version":"114'17","reported_seq":287,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622532+0000","last_change":"2024-09-16T03:39:05.622532+0000","last_active":"2024-09-16T03:39:05.622532+0000","last_peered":"2024-09-16T03:39:05.622532+0000","last_clean":"2024-09-16T03:39:05.622532+0000","last_became_active":"2024-09-16T03:39:01.997491+0000","last_became_peered":"2024-09-16T03:39:01.997491+0000","last_unstale":"2024-09-16T03:39:05.622532+0000","last_undegraded":"2024-09-16T03:39:05.622532+0000","last_fullsized":"2024-09-16T03:39:05.622532+0000","mapping_epoch":126,"log_start":"107'14","ondisk_log_start":"107'14","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'9","last_scrub_stamp":"2024-09-16T03:37:50.756120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:50.756120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.630555+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1495840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":28490,"num_write":4,"num_write_kb":1066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"118'64","reported_seq":380,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517735+0000","last_change":"2024-09-16T03:39:01.901719+0000","last_active":"2024-09-16T03:39:03.517735+0000","last_peered":"2024-09-16T03:39:03.517735+0000","last_clean":"2024-09-16T03:39:03.517735+0000","last_became_active":"2024-09-16T03:39:01.901037+0000","last_became_peered":"2024-09-16T03:39:01.901037+0000","last_unstale":"2024-09-16T03:39:03.517735+0000","last_undegraded":"2024-09-16T03:39:03.517735+0000","last_fullsized":"2024-09-16T03:39:03.517735+0000","mapping_epoch":126,"log_start":"113'61","ondisk_log_start":"113'61","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1680717,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":23268,"num_write":10,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":918091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2f","version":"113'94","reported_seq":452,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.519844+0000","last_change":"2024-09-16T03:39:03.319075+0000","last_active":"2024-09-16T03:39:03.519844+0000","last_peered":"2024-09-16T03:39:03.519844+0000","last_clean":"2024-09-16T03:39:03.519844+0000","last_became_active":"2024-09-16T03:39:01.700988+0000","last_became_peered":"2024-09-16T03:39:01.700988+0000","last_unstale":"2024-09-16T03:39:03.519844+0000","last_undegraded":"2024-09-16T03:39:03.519844+0000","last_fullsized":"2024-09-16T03:39:03.519844+0000","mapping_epoch":126,"log_start":"107'90","ondisk_log_start":"107'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"113'94","last_scrub_stamp":"2024-09-16T03:39:03.319012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.319012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":87,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:13.773817+0000","scrub_duration":1038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2e","version":"123'75","reported_seq":432,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:04.235301+0000","last_change":"2024-09-16T03:39:04.235234+0000","last_active":"2024-09-16T03:39:04.235301+0000","last_peered":"2024-09-16T03:39:04.235301+0000","last_clean":"2024-09-16T03:39:04.235301+0000","last_became_active":"2024-09-16T03:39:01.999291+0000","last_became_peered":"2024-09-16T03:39:01.999291+0000","last_unstale":"2024-09-16T03:39:04.235301+0000","last_undegraded":"2024-09-16T03:39:04.235301+0000","last_fullsized":"2024-09-16T03:39:04.235301+0000","mapping_epoch":126,"log_start":"114'72","ondisk_log_start":"114'72","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"123'75","last_scrub_stamp":"2024-09-16T03:39:04.235141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.235141+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":72,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.010761+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"128'97","reported_seq":457,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.917422+0000","last_change":"2024-09-16T03:39:03.917422+0000","last_active":"2024-09-16T03:39:03.917422+0000","last_peered":"2024-09-16T03:39:03.917422+0000","last_clean":"2024-09-16T03:39:03.917422+0000","last_became_active":"2024-09-16T03:39:01.801112+0000","last_became_peered":"2024-09-16T03:39:01.801112+0000","last_unstale":"2024-09-16T03:39:03.917422+0000","last_undegraded":"2024-09-16T03:39:03.917422+0000","last_fullsized":"2024-09-16T03:39:03.917422+0000","mapping_epoch":32,"log_start":"113'94","ondisk_log_start":"113'94","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'97","last_scrub_stamp":"2024-09-16T03:39:03.917355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.917355+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":91,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.686164+0000","scrub_duration":771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2033713,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.e","version":"127'79","reported_seq":428,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.629943+0000","last_change":"2024-09-16T03:39:01.999648+0000","last_active":"2024-09-16T03:39:03.629943+0000","last_peered":"2024-09-16T03:39:03.629943+0000","last_clean":"2024-09-16T03:39:03.629943+0000","last_became_active":"2024-09-16T03:39:01.999282+0000","last_became_peered":"2024-09-16T03:39:01.999282+0000","last_unstale":"2024-09-16T03:39:03.629943+0000","last_undegraded":"2024-09-16T03:39:03.629943+0000","last_fullsized":"2024-09-16T03:39:03.629943+0000","mapping_epoch":32,"log_start":"123'75","ondisk_log_start":"123'75","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:39.194237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.194237+0000","objects_scrubbed":3,"log_size":4,"log_dups_size":72,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3705976,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":28930,"num_write":10,"num_write_kb":3507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"124'19","reported_seq":336,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.614753+0000","last_change":"2024-09-16T03:39:01.149616+0000","last_active":"2024-09-16T03:39:03.614753+0000","last_peered":"2024-09-16T03:39:03.614753+0000","last_clean":"2024-09-16T03:39:03.614753+0000","last_became_active":"2024-09-16T03:38:45.622965+0000","last_became_peered":"2024-09-16T03:38:45.622965+0000","last_unstale":"2024-09-16T03:39:03.614753+0000","last_undegraded":"2024-09-16T03:39:03.614753+0000","last_fullsized":"2024-09-16T03:39:03.614753+0000","mapping_epoch":72,"log_start":"118'16","ondisk_log_start":"118'16","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'19","last_scrub_stamp":"2024-09-16T03:39:01.149486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.149486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":16,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.574017+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613538,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"122'62","reported_seq":396,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.714704+0000","last_change":"2024-09-16T03:38:54.765354+0000","last_active":"2024-09-16T03:39:03.714704+0000","last_peered":"2024-09-16T03:39:03.714704+0000","last_clean":"2024-09-16T03:39:03.714704+0000","last_became_active":"2024-09-16T03:38:54.765107+0000","last_became_peered":"2024-09-16T03:38:54.765107+0000","last_unstale":"2024-09-16T03:39:03.714704+0000","last_undegraded":"2024-09-16T03:39:03.714704+0000","last_fullsized":"2024-09-16T03:39:03.714704+0000","mapping_epoch":119,"log_start":"122'59","ondisk_log_start":"122'59","created":32,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'57","last_scrub_stamp":"2024-09-16T03:38:48.923184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.923184+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":57,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:59.751869+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731187,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":2797,"num_write":5,"num_write_kb":1777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,11],"acting":[12,15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"129'132","reported_seq":585,"reported_epoch":131,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:05.585816+0000","last_change":"2024-09-16T03:38:52.963503+0000","last_active":"2024-09-16T03:39:05.585816+0000","last_peered":"2024-09-16T03:39:05.585816+0000","last_clean":"2024-09-16T03:38:49.362875+0000","last_became_active":"2024-09-16T03:38:52.661315+0000","last_became_peered":"2024-09-16T03:38:52.661315+0000","last_unstale":"2024-09-16T03:39:05.585816+0000","last_undegraded":"2024-09-16T03:39:05.585816+0000","last_fullsized":"2024-09-16T03:39:05.585816+0000","mapping_epoch":117,"log_start":"127'129","ondisk_log_start":"127'129","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:46.855462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.855462+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":125,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2084134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":16397,"num_write":31,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,12],"acting":[15,4,9],"avail_no_missing":["15","4","9"],"object_location_counts":[{"shards":"4,9,15","objects":1}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.a","version":"128'148","reported_seq":668,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:04.240043+0000","last_change":"2024-09-16T03:39:01.504751+0000","last_active":"2024-09-16T03:39:04.240043+0000","last_peered":"2024-09-16T03:39:04.240043+0000","last_clean":"2024-09-16T03:39:04.240043+0000","last_became_active":"2024-09-16T03:38:46.021805+0000","last_became_peered":"2024-09-16T03:38:46.021805+0000","last_unstale":"2024-09-16T03:39:04.240043+0000","last_undegraded":"2024-09-16T03:39:04.240043+0000","last_fullsized":"2024-09-16T03:39:04.240043+0000","mapping_epoch":32,"log_start":"123'145","ondisk_log_start":"123'145","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'147","last_scrub_stamp":"2024-09-16T03:39:01.504663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.504663+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":142,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:18.060284+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13949942,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":33116,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517135+0000","last_change":"2024-09-16T03:38:46.881357+0000","last_active":"2024-09-16T03:39:03.517135+0000","last_peered":"2024-09-16T03:39:03.517135+0000","last_clean":"2024-09-16T03:39:03.517135+0000","last_became_active":"2024-09-16T03:38:45.728494+0000","last_became_peered":"2024-09-16T03:38:45.728494+0000","last_unstale":"2024-09-16T03:39:03.517135+0000","last_undegraded":"2024-09-16T03:39:03.517135+0000","last_fullsized":"2024-09-16T03:39:03.517135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:46.881248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.881248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:04.991122+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.8","version":"124'71","reported_seq":441,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.515749+0000","last_change":"2024-09-16T03:39:02.002321+0000","last_active":"2024-09-16T03:39:03.515749+0000","last_peered":"2024-09-16T03:39:03.515749+0000","last_clean":"2024-09-16T03:39:03.515749+0000","last_became_active":"2024-09-16T03:38:55.664848+0000","last_became_peered":"2024-09-16T03:38:55.664848+0000","last_unstale":"2024-09-16T03:39:03.515749+0000","last_undegraded":"2024-09-16T03:39:03.515749+0000","last_fullsized":"2024-09-16T03:39:03.515749+0000","mapping_epoch":120,"log_start":"123'67","ondisk_log_start":"123'67","created":32,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'71","last_scrub_stamp":"2024-09-16T03:39:02.002247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.002247+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":65,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:16.545628+0000","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3916064,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"130'85","reported_seq":424,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.585970+0000","last_change":"2024-09-16T03:39:00.900732+0000","last_active":"2024-09-16T03:39:05.585970+0000","last_peered":"2024-09-16T03:39:05.585970+0000","last_clean":"2024-09-16T03:39:05.585970+0000","last_became_active":"2024-09-16T03:38:45.523605+0000","last_became_peered":"2024-09-16T03:38:45.523605+0000","last_unstale":"2024-09-16T03:39:05.585970+0000","last_undegraded":"2024-09-16T03:39:05.585970+0000","last_fullsized":"2024-09-16T03:39:05.585970+0000","mapping_epoch":32,"log_start":"116'82","ondisk_log_start":"116'82","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'82","last_scrub_stamp":"2024-09-16T03:39:00.900619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.900619+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":80,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.385837+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2117748,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":417,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.722771+0000","last_change":"2024-09-16T03:38:36.757156+0000","last_active":"2024-09-16T03:39:05.722771+0000","last_peered":"2024-09-16T03:39:05.722771+0000","last_clean":"2024-09-16T03:39:05.722771+0000","last_became_active":"2024-09-16T03:38:36.756855+0000","last_became_peered":"2024-09-16T03:38:36.756855+0000","last_unstale":"2024-09-16T03:39:05.722771+0000","last_undegraded":"2024-09-16T03:39:05.722771+0000","last_fullsized":"2024-09-16T03:39:05.722771+0000","mapping_epoch":103,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:38:28.063690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:38:28.063690+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:28.682469+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":116,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.515733+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:39:03.515733+0000","last_peered":"2024-09-16T03:39:03.515733+0000","last_clean":"2024-09-16T03:39:03.515733+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:39:03.515733+0000","last_undegraded":"2024-09-16T03:39:03.515733+0000","last_fullsized":"2024-09-16T03:39:03.515733+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"131'260","reported_seq":558,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.723185+0000","last_change":"2024-09-16T03:38:01.603352+0000","last_active":"2024-09-16T03:39:05.723185+0000","last_peered":"2024-09-16T03:39:05.723185+0000","last_clean":"2024-09-16T03:39:05.723185+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:39:05.723185+0000","last_undegraded":"2024-09-16T03:39:05.723185+0000","last_fullsized":"2024-09-16T03:39:05.723185+0000","mapping_epoch":74,"log_start":"129'257","ondisk_log_start":"129'257","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":266,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.902901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22624199,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":247,"num_read_kb":8660,"num_write":324,"num_write_kb":70574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.6","version":"108'44","reported_seq":372,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.618253+0000","last_change":"2024-09-16T03:38:48.279289+0000","last_active":"2024-09-16T03:39:03.618253+0000","last_peered":"2024-09-16T03:39:03.618253+0000","last_clean":"2024-09-16T03:39:03.618253+0000","last_became_active":"2024-09-16T03:38:45.924286+0000","last_became_peered":"2024-09-16T03:38:45.924286+0000","last_unstale":"2024-09-16T03:39:03.618253+0000","last_undegraded":"2024-09-16T03:39:03.618253+0000","last_fullsized":"2024-09-16T03:39:03.618253+0000","mapping_epoch":70,"log_start":"105'41","ondisk_log_start":"105'41","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:48.279224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.279224+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.718165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":249,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.798927+0000","last_change":"2024-09-16T03:38:51.562198+0000","last_active":"2024-09-16T03:39:03.798927+0000","last_peered":"2024-09-16T03:39:03.798927+0000","last_clean":"2024-09-16T03:39:03.798927+0000","last_became_active":"2024-09-16T03:38:51.561887+0000","last_became_peered":"2024-09-16T03:38:51.561887+0000","last_unstale":"2024-09-16T03:39:03.798927+0000","last_undegraded":"2024-09-16T03:39:03.798927+0000","last_fullsized":"2024-09-16T03:39:03.798927+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:24.358339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:24.358339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:30.852736+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,10],"acting":[8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"131'218","reported_seq":447,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.336591+0000","last_change":"2024-09-16T03:38:15.890094+0000","last_active":"2024-09-16T03:39:06.336591+0000","last_peered":"2024-09-16T03:39:06.336591+0000","last_clean":"2024-09-16T03:39:06.336591+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:39:06.336591+0000","last_undegraded":"2024-09-16T03:39:06.336591+0000","last_fullsized":"2024-09-16T03:39:06.336591+0000","mapping_epoch":31,"log_start":"130'215","ondisk_log_start":"130'215","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'125","last_scrub_stamp":"2024-09-16T03:38:15.889996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.889996+0000","objects_scrubbed":18,"log_size":3,"log_dups_size":226,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:35.350472+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15702938,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":193,"num_read_kb":2025,"num_write":275,"num_write_kb":58342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"127'76","reported_seq":407,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.618566+0000","last_change":"2024-09-16T03:39:01.297447+0000","last_active":"2024-09-16T03:39:03.618566+0000","last_peered":"2024-09-16T03:39:03.618566+0000","last_clean":"2024-09-16T03:39:03.618566+0000","last_became_active":"2024-09-16T03:38:45.723754+0000","last_became_peered":"2024-09-16T03:38:45.723754+0000","last_unstale":"2024-09-16T03:39:03.618566+0000","last_undegraded":"2024-09-16T03:39:03.618566+0000","last_fullsized":"2024-09-16T03:39:03.618566+0000","mapping_epoch":32,"log_start":"127'73","ondisk_log_start":"127'73","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'72","last_scrub_stamp":"2024-09-16T03:39:01.297324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.297324+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":71,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.355718+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7294304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":232,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.585505+0000","last_change":"2024-09-16T03:38:15.949627+0000","last_active":"2024-09-16T03:39:05.585505+0000","last_peered":"2024-09-16T03:39:05.585505+0000","last_clean":"2024-09-16T03:39:05.585505+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:39:05.585505+0000","last_undegraded":"2024-09-16T03:39:05.585505+0000","last_fullsized":"2024-09-16T03:39:05.585505+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:15.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:24.286324+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"129'252","reported_seq":620,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.630052+0000","last_change":"2024-09-16T03:38:59.856507+0000","last_active":"2024-09-16T03:39:03.630052+0000","last_peered":"2024-09-16T03:39:03.630052+0000","last_clean":"2024-09-16T03:39:03.630052+0000","last_became_active":"2024-09-16T03:38:59.854999+0000","last_became_peered":"2024-09-16T03:38:59.854999+0000","last_unstale":"2024-09-16T03:39:03.630052+0000","last_undegraded":"2024-09-16T03:39:03.630052+0000","last_fullsized":"2024-09-16T03:39:03.630052+0000","mapping_epoch":124,"log_start":"128'249","ondisk_log_start":"128'249","created":31,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":270,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17434102,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":231,"num_read_kb":12817,"num_write":311,"num_write_kb":78387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":27250603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"105'32","reported_seq":327,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.722978+0000","last_change":"2024-09-16T03:38:47.042020+0000","last_active":"2024-09-16T03:39:05.722978+0000","last_peered":"2024-09-16T03:39:05.722978+0000","last_clean":"2024-09-16T03:39:05.722978+0000","last_became_active":"2024-09-16T03:38:45.820169+0000","last_became_peered":"2024-09-16T03:38:45.820169+0000","last_unstale":"2024-09-16T03:39:05.722978+0000","last_undegraded":"2024-09-16T03:39:05.722978+0000","last_fullsized":"2024-09-16T03:39:05.722978+0000","mapping_epoch":32,"log_start":"83'29","ondisk_log_start":"83'29","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.041851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.041851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.311785+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":232,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.799153+0000","last_change":"2024-09-16T03:38:12.795978+0000","last_active":"2024-09-16T03:39:03.799153+0000","last_peered":"2024-09-16T03:39:03.799153+0000","last_clean":"2024-09-16T03:39:03.799153+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:39:03.799153+0000","last_undegraded":"2024-09-16T03:39:03.799153+0000","last_fullsized":"2024-09-16T03:39:03.799153+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:38:12.795855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.795855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:13.088740+0000","scrub_duration":501,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"131'274","reported_seq":506,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.028683+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:39:06.028683+0000","last_peered":"2024-09-16T03:39:06.028683+0000","last_clean":"2024-09-16T03:39:06.028683+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:39:06.028683+0000","last_undegraded":"2024-09-16T03:39:06.028683+0000","last_fullsized":"2024-09-16T03:39:06.028683+0000","mapping_epoch":31,"log_start":"130'271","ondisk_log_start":"130'271","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.679811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24909264,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":249,"num_read_kb":6268,"num_write":343,"num_write_kb":73467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":265,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.585584+0000","last_change":"2024-09-16T03:38:35.828277+0000","last_active":"2024-09-16T03:39:05.585584+0000","last_peered":"2024-09-16T03:39:05.585584+0000","last_clean":"2024-09-16T03:39:05.585584+0000","last_became_active":"2024-09-16T03:38:33.778361+0000","last_became_peered":"2024-09-16T03:38:33.778361+0000","last_unstale":"2024-09-16T03:39:05.585584+0000","last_undegraded":"2024-09-16T03:39:05.585584+0000","last_fullsized":"2024-09-16T03:39:05.585584+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:35.828203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.828203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:49.130136+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":237,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.542820+0000","last_change":"2024-09-16T03:37:52.652365+0000","last_active":"2024-09-16T03:39:06.542820+0000","last_peered":"2024-09-16T03:39:06.542820+0000","last_clean":"2024-09-16T03:39:06.542820+0000","last_became_active":"2024-09-16T03:37:52.651907+0000","last_became_peered":"2024-09-16T03:37:52.651907+0000","last_unstale":"2024-09-16T03:39:06.542820+0000","last_undegraded":"2024-09-16T03:39:06.542820+0000","last_fullsized":"2024-09-16T03:39:06.542820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:07.210820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"94'34","reported_seq":336,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.680919+0000","last_change":"2024-09-16T03:38:39.081942+0000","last_active":"2024-09-16T03:39:05.680919+0000","last_peered":"2024-09-16T03:39:05.680919+0000","last_clean":"2024-09-16T03:39:05.680919+0000","last_became_active":"2024-09-16T03:38:33.580022+0000","last_became_peered":"2024-09-16T03:38:33.580022+0000","last_unstale":"2024-09-16T03:39:05.680919+0000","last_undegraded":"2024-09-16T03:39:05.680919+0000","last_fullsized":"2024-09-16T03:39:05.680919+0000","mapping_epoch":96,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:39.081810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.081810+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:48.167935+0000","scrub_duration":473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":120,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622686+0000","last_change":"2024-09-16T03:38:51.363599+0000","last_active":"2024-09-16T03:39:05.622686+0000","last_peered":"2024-09-16T03:39:05.622686+0000","last_clean":"2024-09-16T03:39:05.622686+0000","last_became_active":"2024-09-16T03:38:51.363288+0000","last_became_peered":"2024-09-16T03:38:51.363288+0000","last_unstale":"2024-09-16T03:39:05.622686+0000","last_undegraded":"2024-09-16T03:39:05.622686+0000","last_fullsized":"2024-09-16T03:39:05.622686+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:52.581979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"129'130","reported_seq":553,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:04.265574+0000","last_change":"2024-09-16T03:38:39.704994+0000","last_active":"2024-09-16T03:39:04.265574+0000","last_peered":"2024-09-16T03:39:04.265574+0000","last_clean":"2024-09-16T03:39:04.265574+0000","last_became_active":"2024-09-16T03:38:33.581141+0000","last_became_peered":"2024-09-16T03:38:33.581141+0000","last_unstale":"2024-09-16T03:39:04.265574+0000","last_undegraded":"2024-09-16T03:39:04.265574+0000","last_fullsized":"2024-09-16T03:39:04.265574+0000","mapping_epoch":32,"log_start":"123'127","ondisk_log_start":"123'127","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'118","last_scrub_stamp":"2024-09-16T03:38:39.704919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.704919+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":124,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.098482+0000","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882549,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":53012,"num_write":14,"num_write_kb":4474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517581+0000","last_change":"2024-09-16T03:38:12.041048+0000","last_active":"2024-09-16T03:39:03.517581+0000","last_peered":"2024-09-16T03:39:03.517581+0000","last_clean":"2024-09-16T03:39:03.517581+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:39:03.517581+0000","last_undegraded":"2024-09-16T03:39:03.517581+0000","last_fullsized":"2024-09-16T03:39:03.517581+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:12.040917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.040917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.877949+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":248,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.629634+0000","last_change":"2024-09-16T03:38:34.928371+0000","last_active":"2024-09-16T03:39:03.629634+0000","last_peered":"2024-09-16T03:39:03.629634+0000","last_clean":"2024-09-16T03:39:03.629634+0000","last_became_active":"2024-09-16T03:38:33.484091+0000","last_became_peered":"2024-09-16T03:38:33.484091+0000","last_unstale":"2024-09-16T03:39:03.629634+0000","last_undegraded":"2024-09-16T03:39:03.629634+0000","last_fullsized":"2024-09-16T03:39:03.629634+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:34.928298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.928298+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:43.935101+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":231,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.899166+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:39:03.899166+0000","last_peered":"2024-09-16T03:39:03.899166+0000","last_clean":"2024-09-16T03:39:03.899166+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:39:03.899166+0000","last_undegraded":"2024-09-16T03:39:03.899166+0000","last_fullsized":"2024-09-16T03:39:03.899166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.232558+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.10","version":"118'64","reported_seq":385,"reported_epoch":129,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:39:04.265681+0000","last_change":"2024-09-16T03:39:04.188556+0000","last_active":"2024-09-16T03:39:04.265681+0000","last_peered":"2024-09-16T03:39:04.265681+0000","last_clean":"2024-09-16T03:39:04.265681+0000","last_became_active":"2024-09-16T03:39:01.900921+0000","last_became_peered":"2024-09-16T03:39:01.900921+0000","last_unstale":"2024-09-16T03:39:04.265681+0000","last_undegraded":"2024-09-16T03:39:04.265681+0000","last_fullsized":"2024-09-16T03:39:04.265681+0000","mapping_epoch":72,"log_start":"113'61","ondisk_log_start":"113'61","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1680716,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":23267,"num_write":9,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":918090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"114'17","reported_seq":296,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.722729+0000","last_change":"2024-09-16T03:39:03.929961+0000","last_active":"2024-09-16T03:39:05.722729+0000","last_peered":"2024-09-16T03:39:05.722729+0000","last_clean":"2024-09-16T03:39:05.722729+0000","last_became_active":"2024-09-16T03:39:01.696690+0000","last_became_peered":"2024-09-16T03:39:01.696690+0000","last_unstale":"2024-09-16T03:39:05.722729+0000","last_undegraded":"2024-09-16T03:39:05.722729+0000","last_fullsized":"2024-09-16T03:39:05.722729+0000","mapping_epoch":63,"log_start":"107'14","ondisk_log_start":"107'14","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:03.929863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.929863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:26.527714+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2991679,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"127'20","reported_seq":352,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.923159+0000","last_change":"2024-09-16T03:39:03.923060+0000","last_active":"2024-09-16T03:39:03.923159+0000","last_peered":"2024-09-16T03:39:03.923159+0000","last_clean":"2024-09-16T03:39:03.923159+0000","last_became_active":"2024-09-16T03:39:03.822624+0000","last_became_peered":"2024-09-16T03:39:03.822624+0000","last_unstale":"2024-09-16T03:39:03.923159+0000","last_undegraded":"2024-09-16T03:39:03.923159+0000","last_fullsized":"2024-09-16T03:39:03.923159+0000","mapping_epoch":128,"log_start":"106'17","ondisk_log_start":"106'17","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966452,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":26744,"num_write":10,"num_write_kb":2629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2569696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,11],"acting":[14,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.13","version":"127'73","reported_seq":448,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.629829+0000","last_change":"2024-09-16T03:39:03.124648+0000","last_active":"2024-09-16T03:39:03.629829+0000","last_peered":"2024-09-16T03:39:03.629829+0000","last_clean":"2024-09-16T03:39:03.629829+0000","last_became_active":"2024-09-16T03:39:01.843090+0000","last_became_peered":"2024-09-16T03:39:01.843090+0000","last_unstale":"2024-09-16T03:39:03.629829+0000","last_undegraded":"2024-09-16T03:39:03.629829+0000","last_fullsized":"2024-09-16T03:39:03.629829+0000","mapping_epoch":119,"log_start":"108'70","ondisk_log_start":"108'70","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'73","last_scrub_stamp":"2024-09-16T03:39:03.124583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.124583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":68,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:24.286892+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,14],"acting":[4,3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"104'20","reported_seq":316,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622809+0000","last_change":"2024-09-16T03:39:01.898409+0000","last_active":"2024-09-16T03:39:05.622809+0000","last_peered":"2024-09-16T03:39:05.622809+0000","last_clean":"2024-09-16T03:39:05.622809+0000","last_became_active":"2024-09-16T03:39:01.596287+0000","last_became_peered":"2024-09-16T03:39:01.596287+0000","last_unstale":"2024-09-16T03:39:05.622809+0000","last_undegraded":"2024-09-16T03:39:05.622809+0000","last_fullsized":"2024-09-16T03:39:05.622809+0000","mapping_epoch":114,"log_start":"98'17","ondisk_log_start":"98'17","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:01.898331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.898331+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.472424+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"123'65","reported_seq":438,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.916162+0000","last_change":"2024-09-16T03:39:03.916087+0000","last_active":"2024-09-16T03:39:03.916162+0000","last_peered":"2024-09-16T03:39:03.916162+0000","last_clean":"2024-09-16T03:39:03.916162+0000","last_became_active":"2024-09-16T03:39:03.915707+0000","last_became_peered":"2024-09-16T03:39:03.915707+0000","last_unstale":"2024-09-16T03:39:03.916162+0000","last_undegraded":"2024-09-16T03:39:03.916162+0000","last_fullsized":"2024-09-16T03:39:03.916162+0000","mapping_epoch":128,"log_start":"123'62","ondisk_log_start":"123'62","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916014,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":83986,"num_write":19,"num_write_kb":5459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":826342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,6],"acting":[3,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"124'86","reported_seq":446,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.782232+0000","last_change":"2024-09-16T03:39:03.073530+0000","last_active":"2024-09-16T03:39:05.782232+0000","last_peered":"2024-09-16T03:39:05.782232+0000","last_clean":"2024-09-16T03:39:05.782232+0000","last_became_active":"2024-09-16T03:39:01.797805+0000","last_became_peered":"2024-09-16T03:39:01.797805+0000","last_unstale":"2024-09-16T03:39:05.782232+0000","last_undegraded":"2024-09-16T03:39:05.782232+0000","last_fullsized":"2024-09-16T03:39:05.782232+0000","mapping_epoch":115,"log_start":"124'82","ondisk_log_start":"124'82","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.073459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.073459+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":80,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:32.204501+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.17","version":"117'49","reported_seq":385,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.719166+0000","last_change":"2024-09-16T03:39:03.717673+0000","last_active":"2024-09-16T03:39:03.719166+0000","last_peered":"2024-09-16T03:39:03.719166+0000","last_clean":"2024-09-16T03:39:03.719166+0000","last_became_active":"2024-09-16T03:39:03.717420+0000","last_became_peered":"2024-09-16T03:39:03.717420+0000","last_unstale":"2024-09-16T03:39:03.719166+0000","last_undegraded":"2024-09-16T03:39:03.719166+0000","last_fullsized":"2024-09-16T03:39:03.719166+0000","mapping_epoch":128,"log_start":"104'45","ondisk_log_start":"104'45","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":44,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2890820,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":44515,"num_write":7,"num_write_kb":2114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2516795,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"128'51","reported_seq":363,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.798775+0000","last_change":"2024-09-16T03:39:02.668794+0000","last_active":"2024-09-16T03:39:03.798775+0000","last_peered":"2024-09-16T03:39:03.798775+0000","last_clean":"2024-09-16T03:39:03.798775+0000","last_became_active":"2024-09-16T03:38:47.562641+0000","last_became_peered":"2024-09-16T03:38:47.562641+0000","last_unstale":"2024-09-16T03:39:03.798775+0000","last_undegraded":"2024-09-16T03:39:03.798775+0000","last_fullsized":"2024-09-16T03:39:03.798775+0000","mapping_epoch":112,"log_start":"124'48","ondisk_log_start":"124'48","created":50,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'48","last_scrub_stamp":"2024-09-16T03:39:02.668681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.668681+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":45,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.451476+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067327,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":21991,"num_write":23,"num_write_kb":6049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":51,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622532+0000","last_change":"2024-09-16T03:39:01.106936+0000","last_active":"2024-09-16T03:39:05.622532+0000","last_peered":"2024-09-16T03:39:05.622532+0000","last_clean":"2024-09-16T03:39:05.622532+0000","last_became_active":"2024-09-16T03:38:51.558378+0000","last_became_peered":"2024-09-16T03:38:51.558378+0000","last_unstale":"2024-09-16T03:39:05.622532+0000","last_undegraded":"2024-09-16T03:39:05.622532+0000","last_fullsized":"2024-09-16T03:39:05.622532+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:01.106840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.106840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.164091+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,6],"acting":[2,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"128'125","reported_seq":573,"reported_epoch":129,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:03.515341+0000","last_change":"2024-09-16T03:38:47.159152+0000","last_active":"2024-09-16T03:39:03.515341+0000","last_peered":"2024-09-16T03:39:03.515341+0000","last_clean":"2024-09-16T03:38:43.521485+0000","last_became_active":"2024-09-16T03:38:46.957471+0000","last_became_peered":"2024-09-16T03:38:46.957471+0000","last_unstale":"2024-09-16T03:39:03.515341+0000","last_undegraded":"2024-09-16T03:39:03.515341+0000","last_fullsized":"2024-09-16T03:39:03.515341+0000","mapping_epoch":111,"log_start":"117'122","ondisk_log_start":"117'122","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'115","last_scrub_stamp":"2024-09-16T03:38:35.254488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.254488+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":122,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6739673,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":31161,"num_write":9,"num_write_kb":2915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6,11],"acting":[3,8,10],"avail_no_missing":["3","8","10"],"object_location_counts":[{"shards":"3,8,10","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"119'124","reported_seq":569,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:04.237300+0000","last_change":"2024-09-16T03:38:50.465770+0000","last_active":"2024-09-16T03:39:04.237300+0000","last_peered":"2024-09-16T03:39:04.237300+0000","last_clean":"2024-09-16T03:39:04.237300+0000","last_became_active":"2024-09-16T03:38:50.465463+0000","last_became_peered":"2024-09-16T03:38:50.465463+0000","last_unstale":"2024-09-16T03:39:04.237300+0000","last_undegraded":"2024-09-16T03:39:04.237300+0000","last_fullsized":"2024-09-16T03:39:04.237300+0000","mapping_epoch":115,"log_start":"116'120","ondisk_log_start":"116'120","created":100,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'107","last_scrub_stamp":"2024-09-16T03:38:39.905697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.905697+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":114,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.828899+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7159848,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":74651,"num_write":20,"num_write_kb":6207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2376197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"99'56","reported_seq":356,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.542809+0000","last_change":"2024-09-16T03:39:00.988586+0000","last_active":"2024-09-16T03:39:06.542809+0000","last_peered":"2024-09-16T03:39:06.542809+0000","last_clean":"2024-09-16T03:39:06.542809+0000","last_became_active":"2024-09-16T03:38:53.479253+0000","last_became_peered":"2024-09-16T03:38:53.479253+0000","last_unstale":"2024-09-16T03:39:06.542809+0000","last_undegraded":"2024-09-16T03:39:06.542809+0000","last_fullsized":"2024-09-16T03:39:06.542809+0000","mapping_epoch":118,"log_start":"90'53","ondisk_log_start":"90'53","created":100,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":5,"last_scrub":"99'56","last_scrub_stamp":"2024-09-16T03:39:00.988517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.988517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.840710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"92'14","reported_seq":269,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.614582+0000","last_change":"2024-09-16T03:38:49.092747+0000","last_active":"2024-09-16T03:39:03.614582+0000","last_peered":"2024-09-16T03:39:03.614582+0000","last_clean":"2024-09-16T03:39:03.614582+0000","last_became_active":"2024-09-16T03:38:48.661213+0000","last_became_peered":"2024-09-16T03:38:48.661213+0000","last_unstale":"2024-09-16T03:39:03.614582+0000","last_undegraded":"2024-09-16T03:39:03.614582+0000","last_fullsized":"2024-09-16T03:39:03.614582+0000","mapping_epoch":113,"log_start":"91'11","ondisk_log_start":"91'11","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:49.092687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.092687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.372348+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,12],"acting":[3,11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"92'59","reported_seq":378,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517440+0000","last_change":"2024-09-16T03:38:48.561603+0000","last_active":"2024-09-16T03:39:03.517440+0000","last_peered":"2024-09-16T03:39:03.517440+0000","last_clean":"2024-09-16T03:39:03.517440+0000","last_became_active":"2024-09-16T03:38:48.561170+0000","last_became_peered":"2024-09-16T03:38:48.561170+0000","last_unstale":"2024-09-16T03:39:03.517440+0000","last_undegraded":"2024-09-16T03:39:03.517440+0000","last_fullsized":"2024-09-16T03:39:03.517440+0000","mapping_epoch":113,"log_start":"89'56","ondisk_log_start":"89'56","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'59","last_scrub_stamp":"2024-09-16T03:38:35.977646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.977646+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:38.632635+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,6],"acting":[14,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1f","version":"118'111","reported_seq":481,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.951101+0000","last_change":"2024-09-16T03:38:48.762617+0000","last_active":"2024-09-16T03:39:03.951101+0000","last_peered":"2024-09-16T03:39:03.951101+0000","last_clean":"2024-09-16T03:39:03.951101+0000","last_became_active":"2024-09-16T03:38:48.761213+0000","last_became_peered":"2024-09-16T03:38:48.761213+0000","last_unstale":"2024-09-16T03:39:03.951101+0000","last_undegraded":"2024-09-16T03:39:03.951101+0000","last_fullsized":"2024-09-16T03:39:03.951101+0000","mapping_epoch":113,"log_start":"115'107","ondisk_log_start":"115'107","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'95","last_scrub_stamp":"2024-09-16T03:38:37.201959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:37.201959+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":107,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.307408+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9748159,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":35115,"num_write":27,"num_write_kb":6985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1704821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,12],"acting":[3,10,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"127'143","reported_seq":589,"reported_epoch":129,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:03.629583+0000","last_change":"2024-09-16T03:38:46.862171+0000","last_active":"2024-09-16T03:39:03.629583+0000","last_peered":"2024-09-16T03:39:03.629583+0000","last_clean":"2024-09-16T03:38:44.034266+0000","last_became_active":"2024-09-16T03:38:46.560752+0000","last_became_peered":"2024-09-16T03:38:46.560752+0000","last_unstale":"2024-09-16T03:39:03.629583+0000","last_undegraded":"2024-09-16T03:39:03.629583+0000","last_fullsized":"2024-09-16T03:39:03.629583+0000","mapping_epoch":111,"log_start":"127'140","ondisk_log_start":"127'140","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'118","last_scrub_stamp":"2024-09-16T03:38:36.698150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:36.698150+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":140,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":93,"num_read_kb":75680,"num_write":30,"num_write_kb":9656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[4,10,15],"avail_no_missing":["4","10","15"],"object_location_counts":[{"shards":"4,10,15","objects":3}],"blocked_by":[],"up_primary":8,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.21","version":"48'7","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517018+0000","last_change":"2024-09-16T03:38:49.064947+0000","last_active":"2024-09-16T03:39:03.517018+0000","last_peered":"2024-09-16T03:39:03.517018+0000","last_clean":"2024-09-16T03:39:03.517018+0000","last_became_active":"2024-09-16T03:38:48.559431+0000","last_became_peered":"2024-09-16T03:38:48.559431+0000","last_unstale":"2024-09-16T03:39:03.517018+0000","last_undegraded":"2024-09-16T03:39:03.517018+0000","last_fullsized":"2024-09-16T03:39:03.517018+0000","mapping_epoch":113,"log_start":"40'4","ondisk_log_start":"40'4","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":6,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:49.064870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064870+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.310170+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,10],"acting":[14,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.22","version":"45'6","reported_seq":121,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.629609+0000","last_change":"2024-09-16T03:39:00.968576+0000","last_active":"2024-09-16T03:39:03.629609+0000","last_peered":"2024-09-16T03:39:03.629609+0000","last_clean":"2024-09-16T03:39:03.629609+0000","last_became_active":"2024-09-16T03:38:54.462316+0000","last_became_peered":"2024-09-16T03:38:54.462316+0000","last_unstale":"2024-09-16T03:39:03.629609+0000","last_undegraded":"2024-09-16T03:39:03.629609+0000","last_fullsized":"2024-09-16T03:39:03.629609+0000","mapping_epoch":119,"log_start":"39'3","ondisk_log_start":"39'3","created":100,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:39:00.968448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.968448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:06.219174+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,12],"acting":[4,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"108'38","reported_seq":345,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517421+0000","last_change":"2024-09-16T03:38:49.464614+0000","last_active":"2024-09-16T03:39:03.517421+0000","last_peered":"2024-09-16T03:39:03.517421+0000","last_clean":"2024-09-16T03:39:03.517421+0000","last_became_active":"2024-09-16T03:38:49.463797+0000","last_became_peered":"2024-09-16T03:38:49.463797+0000","last_unstale":"2024-09-16T03:39:03.517421+0000","last_undegraded":"2024-09-16T03:39:03.517421+0000","last_fullsized":"2024-09-16T03:39:03.517421+0000","mapping_epoch":114,"log_start":"105'36","ondisk_log_start":"105'36","created":100,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:34.893524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.893524+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":35,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:52.829073+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,2],"acting":[14,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.24","version":"105'32","reported_seq":325,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.823050+0000","last_change":"2024-09-16T03:38:47.978956+0000","last_active":"2024-09-16T03:39:05.823050+0000","last_peered":"2024-09-16T03:39:05.823050+0000","last_clean":"2024-09-16T03:39:05.823050+0000","last_became_active":"2024-09-16T03:38:45.820201+0000","last_became_peered":"2024-09-16T03:38:45.820201+0000","last_unstale":"2024-09-16T03:39:05.823050+0000","last_undegraded":"2024-09-16T03:39:05.823050+0000","last_fullsized":"2024-09-16T03:39:05.823050+0000","mapping_epoch":110,"log_start":"83'29","ondisk_log_start":"83'29","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.978888+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:56.881865+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.25","version":"130'85","reported_seq":414,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.679497+0000","last_change":"2024-09-16T03:38:48.088833+0000","last_active":"2024-09-16T03:39:05.679497+0000","last_peered":"2024-09-16T03:39:05.679497+0000","last_clean":"2024-09-16T03:39:05.679497+0000","last_became_active":"2024-09-16T03:38:45.724757+0000","last_became_peered":"2024-09-16T03:38:45.724757+0000","last_unstale":"2024-09-16T03:39:05.679497+0000","last_undegraded":"2024-09-16T03:39:05.679497+0000","last_fullsized":"2024-09-16T03:39:05.679497+0000","mapping_epoch":110,"log_start":"116'82","ondisk_log_start":"116'82","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'77","last_scrub_stamp":"2024-09-16T03:38:48.088768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.088768+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":79,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:53.112635+0000","scrub_duration":388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2134555,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":10,"num_write_kb":2500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.26","version":"123'52","reported_seq":389,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517824+0000","last_change":"2024-09-16T03:38:49.477393+0000","last_active":"2024-09-16T03:39:03.517824+0000","last_peered":"2024-09-16T03:39:03.517824+0000","last_clean":"2024-09-16T03:39:03.517824+0000","last_became_active":"2024-09-16T03:38:45.926029+0000","last_became_peered":"2024-09-16T03:38:45.926029+0000","last_unstale":"2024-09-16T03:39:03.517824+0000","last_undegraded":"2024-09-16T03:39:03.517824+0000","last_fullsized":"2024-09-16T03:39:03.517824+0000","mapping_epoch":110,"log_start":"123'49","ondisk_log_start":"123'49","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:49.477297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.477297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":48,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.242430+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3815222,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":23772,"num_write":10,"num_write_kb":3140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.27","version":"109'66","reported_seq":379,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.519880+0000","last_change":"2024-09-16T03:38:47.511343+0000","last_active":"2024-09-16T03:39:03.519880+0000","last_peered":"2024-09-16T03:39:03.519880+0000","last_clean":"2024-09-16T03:39:03.519880+0000","last_became_active":"2024-09-16T03:38:45.824884+0000","last_became_peered":"2024-09-16T03:38:45.824884+0000","last_unstale":"2024-09-16T03:39:03.519880+0000","last_undegraded":"2024-09-16T03:39:03.519880+0000","last_fullsized":"2024-09-16T03:39:03.519880+0000","mapping_epoch":110,"log_start":"105'63","ondisk_log_start":"105'63","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'66","last_scrub_stamp":"2024-09-16T03:38:47.511252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.511252+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.884916+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.28","version":"118'67","reported_seq":441,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.515839+0000","last_change":"2024-09-16T03:38:55.727806+0000","last_active":"2024-09-16T03:39:03.515839+0000","last_peered":"2024-09-16T03:39:03.515839+0000","last_clean":"2024-09-16T03:39:03.515839+0000","last_became_active":"2024-09-16T03:38:55.727094+0000","last_became_peered":"2024-09-16T03:38:55.727094+0000","last_unstale":"2024-09-16T03:39:03.515839+0000","last_undegraded":"2024-09-16T03:39:03.515839+0000","last_fullsized":"2024-09-16T03:39:03.515839+0000","mapping_epoch":120,"log_start":"105'63","ondisk_log_start":"105'63","created":110,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:48.551929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.551929+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":62,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:00.496373+0000","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395113,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":21976,"num_write":5,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3610944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":241,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517045+0000","last_change":"2024-09-16T03:39:00.834877+0000","last_active":"2024-09-16T03:39:03.517045+0000","last_peered":"2024-09-16T03:39:03.517045+0000","last_clean":"2024-09-16T03:39:03.517045+0000","last_became_active":"2024-09-16T03:38:45.627369+0000","last_became_peered":"2024-09-16T03:38:45.627369+0000","last_unstale":"2024-09-16T03:39:03.517045+0000","last_undegraded":"2024-09-16T03:39:03.517045+0000","last_fullsized":"2024-09-16T03:39:03.517045+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:00.834742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.834742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.318226+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.2a","version":"108'122","reported_seq":594,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.798898+0000","last_change":"2024-09-16T03:38:47.964525+0000","last_active":"2024-09-16T03:39:03.798898+0000","last_peered":"2024-09-16T03:39:03.798898+0000","last_clean":"2024-09-16T03:39:03.798898+0000","last_became_active":"2024-09-16T03:38:46.022108+0000","last_became_peered":"2024-09-16T03:38:46.022108+0000","last_unstale":"2024-09-16T03:39:03.798898+0000","last_undegraded":"2024-09-16T03:39:03.798898+0000","last_fullsized":"2024-09-16T03:39:03.798898+0000","mapping_epoch":110,"log_start":"107'115","ondisk_log_start":"107'115","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'122","last_scrub_stamp":"2024-09-16T03:38:47.964439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.964439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":112,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.917123+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.2b","version":"127'124","reported_seq":571,"reported_epoch":131,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:05.585588+0000","last_change":"2024-09-16T03:38:53.063728+0000","last_active":"2024-09-16T03:39:05.585588+0000","last_peered":"2024-09-16T03:39:05.585588+0000","last_clean":"2024-09-16T03:38:49.362783+0000","last_became_active":"2024-09-16T03:38:52.761638+0000","last_became_peered":"2024-09-16T03:38:52.761638+0000","last_unstale":"2024-09-16T03:39:05.585588+0000","last_undegraded":"2024-09-16T03:39:05.585588+0000","last_fullsized":"2024-09-16T03:39:05.585588+0000","mapping_epoch":117,"log_start":"119'121","ondisk_log_start":"119'121","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:49.064061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064061+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":121,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,12],"acting":[15,4,9],"avail_no_missing":["15","4","9","12"],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2c","version":"121'63","reported_seq":396,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.562891+0000","last_change":"2024-09-16T03:38:55.563264+0000","last_active":"2024-09-16T03:39:03.562891+0000","last_peered":"2024-09-16T03:39:03.562891+0000","last_clean":"2024-09-16T03:39:03.562891+0000","last_became_active":"2024-09-16T03:38:55.562810+0000","last_became_peered":"2024-09-16T03:38:55.562810+0000","last_unstale":"2024-09-16T03:39:03.562891+0000","last_undegraded":"2024-09-16T03:39:03.562891+0000","last_fullsized":"2024-09-16T03:39:03.562891+0000","mapping_epoch":120,"log_start":"118'60","ondisk_log_start":"118'60","created":110,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'58","last_scrub_stamp":"2024-09-16T03:38:47.889276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.889276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.138029+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1262403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,11],"acting":[12,15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.2d","version":"92'14","reported_seq":322,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.614665+0000","last_change":"2024-09-16T03:38:47.115739+0000","last_active":"2024-09-16T03:39:03.614665+0000","last_peered":"2024-09-16T03:39:03.614665+0000","last_clean":"2024-09-16T03:39:03.614665+0000","last_became_active":"2024-09-16T03:38:45.723438+0000","last_became_peered":"2024-09-16T03:38:45.723438+0000","last_unstale":"2024-09-16T03:39:03.614665+0000","last_undegraded":"2024-09-16T03:39:03.614665+0000","last_fullsized":"2024-09-16T03:39:03.614665+0000","mapping_epoch":110,"log_start":"91'11","ondisk_log_start":"91'11","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:47.115651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.115651+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:00.107802+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":56,"stat_sum":{"num_bytes":133995457,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":828,"num_read_kb":771081,"num_write":311,"num_write_kb":87163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":21165486,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":251191296,"data_stored":250753849,"data_compressed":606603,"data_compressed_allocated":663552,"data_compressed_original":1568768,"omap_allocated":0,"internal_metadata":0},"log_size":169,"ondisk_log_size":169,"up":165,"acting":165,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":80670503,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":920,"num_read_kb":29770,"num_write":1253,"num_write_kb":280770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":39649106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183762944,"data_stored":196582908,"data_compressed":11492126,"data_compressed_allocated":12234752,"data_compressed_original":25473024,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":2644640,"data_compressed":159066,"data_compressed_allocated":204800,"data_compressed_original":2584576,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247837,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83256,"kb_used_data":43372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94288584,"statfs":{"total":96636764160,"available":96551510016,"internally_reserved":0,"allocated":44412928,"data_stored":45041141,"data_compressed":1329713,"data_compressed_allocated":1417216,"data_compressed_original":2596864,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247838,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60108,"kb_used_data":25512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311732,"statfs":{"total":96636764160,"available":96575213568,"internally_reserved":0,"allocated":26124288,"data_stored":27258963,"data_compressed":896583,"data_compressed_allocated":978944,"data_compressed_original":2617344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247837,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73204,"kb_used_data":38960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94298636,"statfs":{"total":96636764160,"available":96561803264,"internally_reserved":0,"allocated":39895040,"data_stored":40905705,"data_compressed":1172110,"data_compressed_allocated":1261568,"data_compressed_original":2797568,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280542,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53928,"kb_used_data":22736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94317912,"statfs":{"total":96636764160,"available":96581541888,"internally_reserved":0,"allocated":23281664,"data_stored":24303845,"data_compressed":842009,"data_compressed_allocated":921600,"data_compressed_original":2420736,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280542,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280541,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313246,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36664,"kb_used_data":7296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335176,"statfs":{"total":96636764160,"available":96599220224,"internally_reserved":0,"allocated":7471104,"data_stored":7083983,"data_compressed":25616,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280542,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88900,"kb_used_data":52272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94282940,"statfs":{"total":96636764160,"available":96545730560,"internally_reserved":0,"allocated":53526528,"data_stored":54527195,"data_compressed":1186238,"data_compressed_allocated":1277952,"data_compressed_original":2863104,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280541,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100932,"kb_used_data":65448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94270908,"statfs":{"total":96636764160,"available":96533409792,"internally_reserved":0,"allocated":67018752,"data_stored":68966584,"data_compressed":1952237,"data_compressed_allocated":2097152,"data_compressed_original":4653056,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313246,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280541,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37192,"kb_used_data":5420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334648,"statfs":{"total":96636764160,"available":96598679552,"internally_reserved":0,"allocated":5550080,"data_stored":5211330,"data_compressed":25616,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71344,"kb_used_data":40160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94300496,"statfs":{"total":96636764160,"available":96563707904,"internally_reserved":0,"allocated":41123840,"data_stored":41803559,"data_compressed":1229872,"data_compressed_allocated":1314816,"data_compressed_original":2510848,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280541,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38652,"kb_used_data":5076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333188,"statfs":{"total":96636764160,"available":96597184512,"internally_reserved":0,"allocated":5197824,"data_stored":5332821,"data_compressed":69039,"data_compressed_allocated":90112,"data_compressed_original":618496,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,3,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280542,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91884,"kb_used_data":57568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94279956,"statfs":{"total":96636764160,"available":96542674944,"internally_reserved":0,"allocated":58949632,"data_stored":58629208,"data_compressed":220712,"data_compressed_allocated":249856,"data_compressed_original":598016,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,3,4,5,6,7,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313245,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43672,"kb_used_data":12484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328168,"statfs":{"total":96636764160,"available":96592044032,"internally_reserved":0,"allocated":12783616,"data_stored":12453761,"data_compressed":109264,"data_compressed_allocated":126976,"data_compressed_original":270336,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247838,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57696,"kb_used_data":25268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94314144,"statfs":{"total":96636764160,"available":96577683456,"internally_reserved":0,"allocated":25874432,"data_stored":26571892,"data_compressed":1270271,"data_compressed_allocated":1355776,"data_compressed_original":2510848,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13885440,"data_stored":14937543,"data_compressed":829441,"data_compressed_allocated":888832,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13422592,"data_stored":14113910,"data_compressed":606276,"data_compressed_allocated":643072,"data_compressed_original":1376256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14032896,"data_stored":15086444,"data_compressed":1224396,"data_compressed_allocated":1294336,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20283392,"data_stored":21623696,"data_compressed":1067426,"data_compressed_allocated":1138688,"data_compressed_original":2527232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":34295808,"data_stored":36685143,"data_compressed":1896867,"data_compressed_allocated":2027520,"data_compressed_original":4493312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21327872,"data_stored":22801296,"data_compressed":1184415,"data_compressed_allocated":1265664,"data_compressed_original":2789376,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14458880,"data_stored":15539052,"data_compressed":1194708,"data_compressed_allocated":1265664,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":12505088,"data_stored":13249430,"data_compressed":646795,"data_compressed_allocated":684032,"data_compressed_original":1470464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14012416,"data_stored":15061447,"data_compressed":829441,"data_compressed_allocated":888832,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14032896,"data_stored":15086444,"data_compressed":1224396,"data_compressed_allocated":1294336,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10883072,"data_stored":10816365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10244096,"data_stored":10207520,"data_compressed":49947,"data_compressed_allocated":53248,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31731712,"data_stored":31727597,"data_compressed":93196,"data_compressed_allocated":102400,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30445568,"data_stored":30338139,"data_compressed":29754,"data_compressed_allocated":32768,"data_compressed_original":86016,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16941056,"data_stored":16931882,"data_compressed":44673,"data_compressed_allocated":49152,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3395584,"data_stored":3372648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":25522176,"data_stored":25455397,"data_compressed":9548,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4005888,"data_stored":3995623,"data_compressed":10762,"data_compressed_allocated":12288,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":57798656,"data_stored":57819101,"data_compressed":195096,"data_compressed_allocated":212992,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":10862592,"data_stored":10841283,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":11612160,"data_stored":11641798,"data_compressed":83648,"data_compressed_allocated":90112,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":7446528,"data_stored":7397291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":29212672,"data_stored":29129165,"data_compressed":79701,"data_compressed_allocated":86016,"data_compressed_original":151552,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:39:08.916 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:39:08.929 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:08.929 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:08.938 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:39:08.943 INFO:tasks.rados.rados.0.smithi157.stdout:1702: done (5 left) 2024-09-16T03:39:08.943 INFO:tasks.rados.rados.0.smithi157.stdout:1703: done (4 left) 2024-09-16T03:39:08.943 INFO:tasks.rados.rados.0.smithi157.stdout:1704: done (3 left) 2024-09-16T03:39:08.943 INFO:tasks.rados.rados.0.smithi157.stdout:1705: done (2 left) 2024-09-16T03:39:08.943 INFO:tasks.rados.rados.0.smithi157.stdout:1706: done (1 left) 2024-09-16T03:39:08.943 INFO:tasks.rados.rados.0.smithi157.stdout:1707: done (0 left) 2024-09-16T03:39:08.943 INFO:tasks.rados.rados.0.smithi157.stdout:1708: set_chunk oid 45 target oid 2024-09-16T03:39:08.980 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:08.979+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:09.041 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:09.041 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:09.049 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:39:09.051 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:09.050+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:09.147 INFO:tasks.rados.rados.0.smithi157.stdout:1708: set_chunk oid 45 offset: 509440 length: 261120 target oid 1807d37f793626ab04b10f5eff1303ee336a5b29a278c6d7f912449e30cb5768 offset: 0 snap -1 2024-09-16T03:39:09.148 INFO:tasks.rados.rados.0.smithi157.stdout:1709: read oid 11 snap -1 2024-09-16T03:39:09.148 INFO:tasks.rados.rados.0.smithi157.stdout:1709: expect deleted 2024-09-16T03:39:09.148 INFO:tasks.rados.rados.0.smithi157.stdout:1710: read oid 22 snap -1 2024-09-16T03:39:09.148 INFO:tasks.rados.rados.0.smithi157.stdout:1710: expect deleted 2024-09-16T03:39:09.148 INFO:tasks.rados.rados.0.smithi157.stdout:1711: read oid 34 snap -1 2024-09-16T03:39:09.148 INFO:tasks.rados.rados.0.smithi157.stdout:1711: expect (ObjNum 342 snap 0 seq_num 342) 2024-09-16T03:39:09.148 INFO:tasks.rados.rados.0.smithi157.stdout:1712: set_chunk oid 28 target oid 2024-09-16T03:39:09.151 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:09.150+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:09.190 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:39:09.192 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:09.192 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:09.200 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:39:09.217 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:09.218 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:39:09.229 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-16T03:39:08.536923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":215256347,"num_objects":137,"num_object_clones":0,"num_object_copies":411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":1834,"num_read_kb":800924,"num_write":1711,"num_write_kb":370607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":61872448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":187,"ondisk_log_size":187,"up":204,"acting":204,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":220,"num_osds":13,"num_per_pool_osds":13,"num_per_pool_omap_osds":13,"kb":1226833920,"kb_used":837432,"kb_used_data":401572,"kb_used_omap":20,"kb_used_meta":405163,"kb_avail":1225996488,"statfs":{"total":1256277934080,"available":1255420403712,"internally_reserved":0,"allocated":411209728,"data_stored":418089987,"data_compressed":10329280,"data_compressed_allocated":11165696,"data_compressed_original":24604672,"omap_allocated":20735,"internal_metadata":414887681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7439868,"num_objects":-31,"num_object_clones":0,"num_object_copies":-93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-309,"num_read_kb":-304189,"num_write":-80,"num_write_kb":-32977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-15741625,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018136"},"pg_stats":[{"pgid":"4.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-16T03:39:00.487335+0000","last_change":"2024-09-16T03:39:00.487335+0000","last_active":"2024-09-16T03:39:00.487335+0000","last_peered":"2024-09-16T03:39:00.487335+0000","last_clean":"2024-09-16T03:39:00.487335+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:39:00.487335+0000","last_undegraded":"2024-09-16T03:39:00.487335+0000","last_fullsized":"2024-09-16T03:39:00.487335+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:00.487335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:39:00.487335+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.487335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.36","version":"131'93","reported_seq":460,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.251987+0000","last_change":"2024-09-16T03:39:03.716194+0000","last_active":"2024-09-16T03:39:06.251987+0000","last_peered":"2024-09-16T03:39:06.251987+0000","last_clean":"2024-09-16T03:39:06.251987+0000","last_became_active":"2024-09-16T03:39:02.099320+0000","last_became_peered":"2024-09-16T03:39:02.099320+0000","last_unstale":"2024-09-16T03:39:06.251987+0000","last_undegraded":"2024-09-16T03:39:06.251987+0000","last_fullsized":"2024-09-16T03:39:06.251987+0000","mapping_epoch":126,"log_start":"130'90","ondisk_log_start":"130'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.716125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.716125+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":88,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.328700+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269209,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":4665,"num_write":8,"num_write_kb":2101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.35","version":"123'65","reported_seq":429,"reported_epoch":129,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:39:04.236629+0000","last_change":"2024-09-16T03:39:04.117063+0000","last_active":"2024-09-16T03:39:04.236629+0000","last_peered":"2024-09-16T03:39:04.236629+0000","last_clean":"2024-09-16T03:38:49.553429+0000","last_became_active":"2024-09-16T03:39:03.915711+0000","last_became_peered":"2024-09-16T03:39:03.915711+0000","last_unstale":"2024-09-16T03:39:04.236629+0000","last_undegraded":"2024-09-16T03:39:04.236629+0000","last_fullsized":"2024-09-16T03:39:04.236629+0000","mapping_epoch":128,"log_start":"123'62","ondisk_log_start":"123'62","created":126,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916015,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84364,"num_write":19,"num_write_kb":5459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3131048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,6],"acting":[3,10,9],"avail_no_missing":["3","6","9","10","11","15"],"object_location_counts":[{"shards":"3,9,10","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"104'20","reported_seq":317,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622626+0000","last_change":"2024-09-16T03:39:02.919356+0000","last_active":"2024-09-16T03:39:05.622626+0000","last_peered":"2024-09-16T03:39:05.622626+0000","last_clean":"2024-09-16T03:39:05.622626+0000","last_became_active":"2024-09-16T03:39:01.699286+0000","last_became_peered":"2024-09-16T03:39:01.699286+0000","last_unstale":"2024-09-16T03:39:05.622626+0000","last_undegraded":"2024-09-16T03:39:05.622626+0000","last_fullsized":"2024-09-16T03:39:05.622626+0000","mapping_epoch":126,"log_start":"98'17","ondisk_log_start":"98'17","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:02.919259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.919259+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.209096+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2705960,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"108'72","reported_seq":449,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.730259+0000","last_change":"2024-09-16T03:39:02.143407+0000","last_active":"2024-09-16T03:39:03.730259+0000","last_peered":"2024-09-16T03:39:03.730259+0000","last_clean":"2024-09-16T03:39:03.730259+0000","last_became_active":"2024-09-16T03:39:01.642310+0000","last_became_peered":"2024-09-16T03:39:01.642310+0000","last_unstale":"2024-09-16T03:39:03.730259+0000","last_undegraded":"2024-09-16T03:39:03.730259+0000","last_fullsized":"2024-09-16T03:39:03.730259+0000","mapping_epoch":126,"log_start":"108'70","ondisk_log_start":"108'70","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'72","last_scrub_stamp":"2024-09-16T03:39:02.143339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.143339+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":68,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:29.596134+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2722767,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":6432,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,14],"acting":[4,3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.32","version":"106'17","reported_seq":332,"reported_epoch":129,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:39:04.023713+0000","last_change":"2024-09-16T03:39:04.023713+0000","last_active":"2024-09-16T03:39:04.023713+0000","last_peered":"2024-09-16T03:39:04.023713+0000","last_clean":"2024-09-16T03:38:49.359852+0000","last_became_active":"2024-09-16T03:39:03.822701+0000","last_became_peered":"2024-09-16T03:39:03.822701+0000","last_unstale":"2024-09-16T03:39:04.023713+0000","last_undegraded":"2024-09-16T03:39:04.023713+0000","last_fullsized":"2024-09-16T03:39:04.023713+0000","mapping_epoch":128,"log_start":"82'13","ondisk_log_start":"82'13","created":126,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1033647,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":26745,"num_write":6,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1331059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,11],"acting":[14,2,9],"avail_no_missing":["14","2","9"],"object_location_counts":[{"shards":"2,9,14","objects":1}],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.31","version":"114'17","reported_seq":287,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622532+0000","last_change":"2024-09-16T03:39:05.622532+0000","last_active":"2024-09-16T03:39:05.622532+0000","last_peered":"2024-09-16T03:39:05.622532+0000","last_clean":"2024-09-16T03:39:05.622532+0000","last_became_active":"2024-09-16T03:39:01.997491+0000","last_became_peered":"2024-09-16T03:39:01.997491+0000","last_unstale":"2024-09-16T03:39:05.622532+0000","last_undegraded":"2024-09-16T03:39:05.622532+0000","last_fullsized":"2024-09-16T03:39:05.622532+0000","mapping_epoch":126,"log_start":"107'14","ondisk_log_start":"107'14","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'9","last_scrub_stamp":"2024-09-16T03:37:50.756120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:50.756120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.630555+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1495840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":28490,"num_write":4,"num_write_kb":1066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"118'64","reported_seq":380,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517735+0000","last_change":"2024-09-16T03:39:01.901719+0000","last_active":"2024-09-16T03:39:03.517735+0000","last_peered":"2024-09-16T03:39:03.517735+0000","last_clean":"2024-09-16T03:39:03.517735+0000","last_became_active":"2024-09-16T03:39:01.901037+0000","last_became_peered":"2024-09-16T03:39:01.901037+0000","last_unstale":"2024-09-16T03:39:03.517735+0000","last_undegraded":"2024-09-16T03:39:03.517735+0000","last_fullsized":"2024-09-16T03:39:03.517735+0000","mapping_epoch":126,"log_start":"113'61","ondisk_log_start":"113'61","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1680717,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":23268,"num_write":10,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":918091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2f","version":"113'94","reported_seq":452,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.519844+0000","last_change":"2024-09-16T03:39:03.319075+0000","last_active":"2024-09-16T03:39:03.519844+0000","last_peered":"2024-09-16T03:39:03.519844+0000","last_clean":"2024-09-16T03:39:03.519844+0000","last_became_active":"2024-09-16T03:39:01.700988+0000","last_became_peered":"2024-09-16T03:39:01.700988+0000","last_unstale":"2024-09-16T03:39:03.519844+0000","last_undegraded":"2024-09-16T03:39:03.519844+0000","last_fullsized":"2024-09-16T03:39:03.519844+0000","mapping_epoch":126,"log_start":"107'90","ondisk_log_start":"107'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"113'94","last_scrub_stamp":"2024-09-16T03:39:03.319012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.319012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":87,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:13.773817+0000","scrub_duration":1038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2e","version":"123'75","reported_seq":432,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:04.235301+0000","last_change":"2024-09-16T03:39:04.235234+0000","last_active":"2024-09-16T03:39:04.235301+0000","last_peered":"2024-09-16T03:39:04.235301+0000","last_clean":"2024-09-16T03:39:04.235301+0000","last_became_active":"2024-09-16T03:39:01.999291+0000","last_became_peered":"2024-09-16T03:39:01.999291+0000","last_unstale":"2024-09-16T03:39:04.235301+0000","last_undegraded":"2024-09-16T03:39:04.235301+0000","last_fullsized":"2024-09-16T03:39:04.235301+0000","mapping_epoch":126,"log_start":"114'72","ondisk_log_start":"114'72","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"123'75","last_scrub_stamp":"2024-09-16T03:39:04.235141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.235141+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":72,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.010761+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"128'97","reported_seq":457,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.917422+0000","last_change":"2024-09-16T03:39:03.917422+0000","last_active":"2024-09-16T03:39:03.917422+0000","last_peered":"2024-09-16T03:39:03.917422+0000","last_clean":"2024-09-16T03:39:03.917422+0000","last_became_active":"2024-09-16T03:39:01.801112+0000","last_became_peered":"2024-09-16T03:39:01.801112+0000","last_unstale":"2024-09-16T03:39:03.917422+0000","last_undegraded":"2024-09-16T03:39:03.917422+0000","last_fullsized":"2024-09-16T03:39:03.917422+0000","mapping_epoch":32,"log_start":"113'94","ondisk_log_start":"113'94","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'97","last_scrub_stamp":"2024-09-16T03:39:03.917355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.917355+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":91,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.686164+0000","scrub_duration":771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2033713,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.e","version":"127'79","reported_seq":428,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.629943+0000","last_change":"2024-09-16T03:39:01.999648+0000","last_active":"2024-09-16T03:39:03.629943+0000","last_peered":"2024-09-16T03:39:03.629943+0000","last_clean":"2024-09-16T03:39:03.629943+0000","last_became_active":"2024-09-16T03:39:01.999282+0000","last_became_peered":"2024-09-16T03:39:01.999282+0000","last_unstale":"2024-09-16T03:39:03.629943+0000","last_undegraded":"2024-09-16T03:39:03.629943+0000","last_fullsized":"2024-09-16T03:39:03.629943+0000","mapping_epoch":32,"log_start":"123'75","ondisk_log_start":"123'75","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:39.194237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.194237+0000","objects_scrubbed":3,"log_size":4,"log_dups_size":72,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3705976,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":28930,"num_write":10,"num_write_kb":3507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"124'19","reported_seq":336,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.614753+0000","last_change":"2024-09-16T03:39:01.149616+0000","last_active":"2024-09-16T03:39:03.614753+0000","last_peered":"2024-09-16T03:39:03.614753+0000","last_clean":"2024-09-16T03:39:03.614753+0000","last_became_active":"2024-09-16T03:38:45.622965+0000","last_became_peered":"2024-09-16T03:38:45.622965+0000","last_unstale":"2024-09-16T03:39:03.614753+0000","last_undegraded":"2024-09-16T03:39:03.614753+0000","last_fullsized":"2024-09-16T03:39:03.614753+0000","mapping_epoch":72,"log_start":"118'16","ondisk_log_start":"118'16","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'19","last_scrub_stamp":"2024-09-16T03:39:01.149486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.149486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":16,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.574017+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613538,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"122'62","reported_seq":396,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.714704+0000","last_change":"2024-09-16T03:38:54.765354+0000","last_active":"2024-09-16T03:39:03.714704+0000","last_peered":"2024-09-16T03:39:03.714704+0000","last_clean":"2024-09-16T03:39:03.714704+0000","last_became_active":"2024-09-16T03:38:54.765107+0000","last_became_peered":"2024-09-16T03:38:54.765107+0000","last_unstale":"2024-09-16T03:39:03.714704+0000","last_undegraded":"2024-09-16T03:39:03.714704+0000","last_fullsized":"2024-09-16T03:39:03.714704+0000","mapping_epoch":119,"log_start":"122'59","ondisk_log_start":"122'59","created":32,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'57","last_scrub_stamp":"2024-09-16T03:38:48.923184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.923184+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":57,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:59.751869+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731187,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":2797,"num_write":5,"num_write_kb":1777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,11],"acting":[12,15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"129'132","reported_seq":585,"reported_epoch":131,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:05.585816+0000","last_change":"2024-09-16T03:38:52.963503+0000","last_active":"2024-09-16T03:39:05.585816+0000","last_peered":"2024-09-16T03:39:05.585816+0000","last_clean":"2024-09-16T03:38:49.362875+0000","last_became_active":"2024-09-16T03:38:52.661315+0000","last_became_peered":"2024-09-16T03:38:52.661315+0000","last_unstale":"2024-09-16T03:39:05.585816+0000","last_undegraded":"2024-09-16T03:39:05.585816+0000","last_fullsized":"2024-09-16T03:39:05.585816+0000","mapping_epoch":117,"log_start":"127'129","ondisk_log_start":"127'129","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:46.855462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.855462+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":125,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2084134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":16397,"num_write":31,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,12],"acting":[15,4,9],"avail_no_missing":["15","4","9"],"object_location_counts":[{"shards":"4,9,15","objects":1}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.a","version":"128'148","reported_seq":668,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:04.240043+0000","last_change":"2024-09-16T03:39:01.504751+0000","last_active":"2024-09-16T03:39:04.240043+0000","last_peered":"2024-09-16T03:39:04.240043+0000","last_clean":"2024-09-16T03:39:04.240043+0000","last_became_active":"2024-09-16T03:38:46.021805+0000","last_became_peered":"2024-09-16T03:38:46.021805+0000","last_unstale":"2024-09-16T03:39:04.240043+0000","last_undegraded":"2024-09-16T03:39:04.240043+0000","last_fullsized":"2024-09-16T03:39:04.240043+0000","mapping_epoch":32,"log_start":"123'145","ondisk_log_start":"123'145","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'147","last_scrub_stamp":"2024-09-16T03:39:01.504663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.504663+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":142,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:18.060284+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13949942,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":33116,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517135+0000","last_change":"2024-09-16T03:38:46.881357+0000","last_active":"2024-09-16T03:39:03.517135+0000","last_peered":"2024-09-16T03:39:03.517135+0000","last_clean":"2024-09-16T03:39:03.517135+0000","last_became_active":"2024-09-16T03:38:45.728494+0000","last_became_peered":"2024-09-16T03:38:45.728494+0000","last_unstale":"2024-09-16T03:39:03.517135+0000","last_undegraded":"2024-09-16T03:39:03.517135+0000","last_fullsized":"2024-09-16T03:39:03.517135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:46.881248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.881248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:04.991122+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.8","version":"124'71","reported_seq":441,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.515749+0000","last_change":"2024-09-16T03:39:02.002321+0000","last_active":"2024-09-16T03:39:03.515749+0000","last_peered":"2024-09-16T03:39:03.515749+0000","last_clean":"2024-09-16T03:39:03.515749+0000","last_became_active":"2024-09-16T03:38:55.664848+0000","last_became_peered":"2024-09-16T03:38:55.664848+0000","last_unstale":"2024-09-16T03:39:03.515749+0000","last_undegraded":"2024-09-16T03:39:03.515749+0000","last_fullsized":"2024-09-16T03:39:03.515749+0000","mapping_epoch":120,"log_start":"123'67","ondisk_log_start":"123'67","created":32,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'71","last_scrub_stamp":"2024-09-16T03:39:02.002247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.002247+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":65,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:16.545628+0000","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3916064,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"130'85","reported_seq":424,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.585970+0000","last_change":"2024-09-16T03:39:00.900732+0000","last_active":"2024-09-16T03:39:05.585970+0000","last_peered":"2024-09-16T03:39:05.585970+0000","last_clean":"2024-09-16T03:39:05.585970+0000","last_became_active":"2024-09-16T03:38:45.523605+0000","last_became_peered":"2024-09-16T03:38:45.523605+0000","last_unstale":"2024-09-16T03:39:05.585970+0000","last_undegraded":"2024-09-16T03:39:05.585970+0000","last_fullsized":"2024-09-16T03:39:05.585970+0000","mapping_epoch":32,"log_start":"116'82","ondisk_log_start":"116'82","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'82","last_scrub_stamp":"2024-09-16T03:39:00.900619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.900619+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":80,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.385837+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2117748,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":417,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.722771+0000","last_change":"2024-09-16T03:38:36.757156+0000","last_active":"2024-09-16T03:39:05.722771+0000","last_peered":"2024-09-16T03:39:05.722771+0000","last_clean":"2024-09-16T03:39:05.722771+0000","last_became_active":"2024-09-16T03:38:36.756855+0000","last_became_peered":"2024-09-16T03:38:36.756855+0000","last_unstale":"2024-09-16T03:39:05.722771+0000","last_undegraded":"2024-09-16T03:39:05.722771+0000","last_fullsized":"2024-09-16T03:39:05.722771+0000","mapping_epoch":103,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:38:28.063690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:38:28.063690+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:28.682469+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":116,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.515733+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:39:03.515733+0000","last_peered":"2024-09-16T03:39:03.515733+0000","last_clean":"2024-09-16T03:39:03.515733+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:39:03.515733+0000","last_undegraded":"2024-09-16T03:39:03.515733+0000","last_fullsized":"2024-09-16T03:39:03.515733+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"131'260","reported_seq":558,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.723185+0000","last_change":"2024-09-16T03:38:01.603352+0000","last_active":"2024-09-16T03:39:05.723185+0000","last_peered":"2024-09-16T03:39:05.723185+0000","last_clean":"2024-09-16T03:39:05.723185+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:39:05.723185+0000","last_undegraded":"2024-09-16T03:39:05.723185+0000","last_fullsized":"2024-09-16T03:39:05.723185+0000","mapping_epoch":74,"log_start":"129'257","ondisk_log_start":"129'257","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":266,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.902901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22624199,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":247,"num_read_kb":8660,"num_write":324,"num_write_kb":70574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.6","version":"108'44","reported_seq":372,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.618253+0000","last_change":"2024-09-16T03:38:48.279289+0000","last_active":"2024-09-16T03:39:03.618253+0000","last_peered":"2024-09-16T03:39:03.618253+0000","last_clean":"2024-09-16T03:39:03.618253+0000","last_became_active":"2024-09-16T03:38:45.924286+0000","last_became_peered":"2024-09-16T03:38:45.924286+0000","last_unstale":"2024-09-16T03:39:03.618253+0000","last_undegraded":"2024-09-16T03:39:03.618253+0000","last_fullsized":"2024-09-16T03:39:03.618253+0000","mapping_epoch":70,"log_start":"105'41","ondisk_log_start":"105'41","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:48.279224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.279224+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.718165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":249,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.798927+0000","last_change":"2024-09-16T03:38:51.562198+0000","last_active":"2024-09-16T03:39:03.798927+0000","last_peered":"2024-09-16T03:39:03.798927+0000","last_clean":"2024-09-16T03:39:03.798927+0000","last_became_active":"2024-09-16T03:38:51.561887+0000","last_became_peered":"2024-09-16T03:38:51.561887+0000","last_unstale":"2024-09-16T03:39:03.798927+0000","last_undegraded":"2024-09-16T03:39:03.798927+0000","last_fullsized":"2024-09-16T03:39:03.798927+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:24.358339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:24.358339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:30.852736+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,10],"acting":[8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"131'218","reported_seq":447,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.336591+0000","last_change":"2024-09-16T03:38:15.890094+0000","last_active":"2024-09-16T03:39:06.336591+0000","last_peered":"2024-09-16T03:39:06.336591+0000","last_clean":"2024-09-16T03:39:06.336591+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:39:06.336591+0000","last_undegraded":"2024-09-16T03:39:06.336591+0000","last_fullsized":"2024-09-16T03:39:06.336591+0000","mapping_epoch":31,"log_start":"130'215","ondisk_log_start":"130'215","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'125","last_scrub_stamp":"2024-09-16T03:38:15.889996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.889996+0000","objects_scrubbed":18,"log_size":3,"log_dups_size":226,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:35.350472+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15702938,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":193,"num_read_kb":2025,"num_write":275,"num_write_kb":58342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"127'76","reported_seq":407,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.618566+0000","last_change":"2024-09-16T03:39:01.297447+0000","last_active":"2024-09-16T03:39:03.618566+0000","last_peered":"2024-09-16T03:39:03.618566+0000","last_clean":"2024-09-16T03:39:03.618566+0000","last_became_active":"2024-09-16T03:38:45.723754+0000","last_became_peered":"2024-09-16T03:38:45.723754+0000","last_unstale":"2024-09-16T03:39:03.618566+0000","last_undegraded":"2024-09-16T03:39:03.618566+0000","last_fullsized":"2024-09-16T03:39:03.618566+0000","mapping_epoch":32,"log_start":"127'73","ondisk_log_start":"127'73","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'72","last_scrub_stamp":"2024-09-16T03:39:01.297324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.297324+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":71,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.355718+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7294304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":232,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.585505+0000","last_change":"2024-09-16T03:38:15.949627+0000","last_active":"2024-09-16T03:39:05.585505+0000","last_peered":"2024-09-16T03:39:05.585505+0000","last_clean":"2024-09-16T03:39:05.585505+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:39:05.585505+0000","last_undegraded":"2024-09-16T03:39:05.585505+0000","last_fullsized":"2024-09-16T03:39:05.585505+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:15.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:24.286324+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"129'252","reported_seq":620,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.630052+0000","last_change":"2024-09-16T03:38:59.856507+0000","last_active":"2024-09-16T03:39:03.630052+0000","last_peered":"2024-09-16T03:39:03.630052+0000","last_clean":"2024-09-16T03:39:03.630052+0000","last_became_active":"2024-09-16T03:38:59.854999+0000","last_became_peered":"2024-09-16T03:38:59.854999+0000","last_unstale":"2024-09-16T03:39:03.630052+0000","last_undegraded":"2024-09-16T03:39:03.630052+0000","last_fullsized":"2024-09-16T03:39:03.630052+0000","mapping_epoch":124,"log_start":"128'249","ondisk_log_start":"128'249","created":31,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":270,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17434102,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":231,"num_read_kb":12817,"num_write":311,"num_write_kb":78387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":27250603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"105'32","reported_seq":327,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.722978+0000","last_change":"2024-09-16T03:38:47.042020+0000","last_active":"2024-09-16T03:39:05.722978+0000","last_peered":"2024-09-16T03:39:05.722978+0000","last_clean":"2024-09-16T03:39:05.722978+0000","last_became_active":"2024-09-16T03:38:45.820169+0000","last_became_peered":"2024-09-16T03:38:45.820169+0000","last_unstale":"2024-09-16T03:39:05.722978+0000","last_undegraded":"2024-09-16T03:39:05.722978+0000","last_fullsized":"2024-09-16T03:39:05.722978+0000","mapping_epoch":32,"log_start":"83'29","ondisk_log_start":"83'29","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.041851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.041851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.311785+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":232,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.799153+0000","last_change":"2024-09-16T03:38:12.795978+0000","last_active":"2024-09-16T03:39:03.799153+0000","last_peered":"2024-09-16T03:39:03.799153+0000","last_clean":"2024-09-16T03:39:03.799153+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:39:03.799153+0000","last_undegraded":"2024-09-16T03:39:03.799153+0000","last_fullsized":"2024-09-16T03:39:03.799153+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:38:12.795855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.795855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:13.088740+0000","scrub_duration":501,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"131'274","reported_seq":506,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.028683+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:39:06.028683+0000","last_peered":"2024-09-16T03:39:06.028683+0000","last_clean":"2024-09-16T03:39:06.028683+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:39:06.028683+0000","last_undegraded":"2024-09-16T03:39:06.028683+0000","last_fullsized":"2024-09-16T03:39:06.028683+0000","mapping_epoch":31,"log_start":"130'271","ondisk_log_start":"130'271","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.679811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24909264,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":249,"num_read_kb":6268,"num_write":343,"num_write_kb":73467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":265,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.585584+0000","last_change":"2024-09-16T03:38:35.828277+0000","last_active":"2024-09-16T03:39:05.585584+0000","last_peered":"2024-09-16T03:39:05.585584+0000","last_clean":"2024-09-16T03:39:05.585584+0000","last_became_active":"2024-09-16T03:38:33.778361+0000","last_became_peered":"2024-09-16T03:38:33.778361+0000","last_unstale":"2024-09-16T03:39:05.585584+0000","last_undegraded":"2024-09-16T03:39:05.585584+0000","last_fullsized":"2024-09-16T03:39:05.585584+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:35.828203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.828203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:49.130136+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":237,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.542820+0000","last_change":"2024-09-16T03:37:52.652365+0000","last_active":"2024-09-16T03:39:06.542820+0000","last_peered":"2024-09-16T03:39:06.542820+0000","last_clean":"2024-09-16T03:39:06.542820+0000","last_became_active":"2024-09-16T03:37:52.651907+0000","last_became_peered":"2024-09-16T03:37:52.651907+0000","last_unstale":"2024-09-16T03:39:06.542820+0000","last_undegraded":"2024-09-16T03:39:06.542820+0000","last_fullsized":"2024-09-16T03:39:06.542820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:07.210820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"94'34","reported_seq":336,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.680919+0000","last_change":"2024-09-16T03:38:39.081942+0000","last_active":"2024-09-16T03:39:05.680919+0000","last_peered":"2024-09-16T03:39:05.680919+0000","last_clean":"2024-09-16T03:39:05.680919+0000","last_became_active":"2024-09-16T03:38:33.580022+0000","last_became_peered":"2024-09-16T03:38:33.580022+0000","last_unstale":"2024-09-16T03:39:05.680919+0000","last_undegraded":"2024-09-16T03:39:05.680919+0000","last_fullsized":"2024-09-16T03:39:05.680919+0000","mapping_epoch":96,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:39.081810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.081810+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:48.167935+0000","scrub_duration":473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":120,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622686+0000","last_change":"2024-09-16T03:38:51.363599+0000","last_active":"2024-09-16T03:39:05.622686+0000","last_peered":"2024-09-16T03:39:05.622686+0000","last_clean":"2024-09-16T03:39:05.622686+0000","last_became_active":"2024-09-16T03:38:51.363288+0000","last_became_peered":"2024-09-16T03:38:51.363288+0000","last_unstale":"2024-09-16T03:39:05.622686+0000","last_undegraded":"2024-09-16T03:39:05.622686+0000","last_fullsized":"2024-09-16T03:39:05.622686+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:52.581979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"129'130","reported_seq":553,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:04.265574+0000","last_change":"2024-09-16T03:38:39.704994+0000","last_active":"2024-09-16T03:39:04.265574+0000","last_peered":"2024-09-16T03:39:04.265574+0000","last_clean":"2024-09-16T03:39:04.265574+0000","last_became_active":"2024-09-16T03:38:33.581141+0000","last_became_peered":"2024-09-16T03:38:33.581141+0000","last_unstale":"2024-09-16T03:39:04.265574+0000","last_undegraded":"2024-09-16T03:39:04.265574+0000","last_fullsized":"2024-09-16T03:39:04.265574+0000","mapping_epoch":32,"log_start":"123'127","ondisk_log_start":"123'127","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'118","last_scrub_stamp":"2024-09-16T03:38:39.704919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.704919+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":124,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.098482+0000","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882549,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":53012,"num_write":14,"num_write_kb":4474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":238,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517581+0000","last_change":"2024-09-16T03:38:12.041048+0000","last_active":"2024-09-16T03:39:03.517581+0000","last_peered":"2024-09-16T03:39:03.517581+0000","last_clean":"2024-09-16T03:39:03.517581+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:39:03.517581+0000","last_undegraded":"2024-09-16T03:39:03.517581+0000","last_fullsized":"2024-09-16T03:39:03.517581+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:12.040917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.040917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.877949+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":248,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.629634+0000","last_change":"2024-09-16T03:38:34.928371+0000","last_active":"2024-09-16T03:39:03.629634+0000","last_peered":"2024-09-16T03:39:03.629634+0000","last_clean":"2024-09-16T03:39:03.629634+0000","last_became_active":"2024-09-16T03:38:33.484091+0000","last_became_peered":"2024-09-16T03:38:33.484091+0000","last_unstale":"2024-09-16T03:39:03.629634+0000","last_undegraded":"2024-09-16T03:39:03.629634+0000","last_fullsized":"2024-09-16T03:39:03.629634+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:34.928298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.928298+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:43.935101+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":231,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.899166+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:39:03.899166+0000","last_peered":"2024-09-16T03:39:03.899166+0000","last_clean":"2024-09-16T03:39:03.899166+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:39:03.899166+0000","last_undegraded":"2024-09-16T03:39:03.899166+0000","last_fullsized":"2024-09-16T03:39:03.899166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.232558+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.10","version":"118'64","reported_seq":385,"reported_epoch":129,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:39:04.265681+0000","last_change":"2024-09-16T03:39:04.188556+0000","last_active":"2024-09-16T03:39:04.265681+0000","last_peered":"2024-09-16T03:39:04.265681+0000","last_clean":"2024-09-16T03:39:04.265681+0000","last_became_active":"2024-09-16T03:39:01.900921+0000","last_became_peered":"2024-09-16T03:39:01.900921+0000","last_unstale":"2024-09-16T03:39:04.265681+0000","last_undegraded":"2024-09-16T03:39:04.265681+0000","last_fullsized":"2024-09-16T03:39:04.265681+0000","mapping_epoch":72,"log_start":"113'61","ondisk_log_start":"113'61","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1680716,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":23267,"num_write":9,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":918090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"114'17","reported_seq":296,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.722729+0000","last_change":"2024-09-16T03:39:03.929961+0000","last_active":"2024-09-16T03:39:05.722729+0000","last_peered":"2024-09-16T03:39:05.722729+0000","last_clean":"2024-09-16T03:39:05.722729+0000","last_became_active":"2024-09-16T03:39:01.696690+0000","last_became_peered":"2024-09-16T03:39:01.696690+0000","last_unstale":"2024-09-16T03:39:05.722729+0000","last_undegraded":"2024-09-16T03:39:05.722729+0000","last_fullsized":"2024-09-16T03:39:05.722729+0000","mapping_epoch":63,"log_start":"107'14","ondisk_log_start":"107'14","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:03.929863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.929863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:26.527714+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2991679,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"127'20","reported_seq":352,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.923159+0000","last_change":"2024-09-16T03:39:03.923060+0000","last_active":"2024-09-16T03:39:03.923159+0000","last_peered":"2024-09-16T03:39:03.923159+0000","last_clean":"2024-09-16T03:39:03.923159+0000","last_became_active":"2024-09-16T03:39:03.822624+0000","last_became_peered":"2024-09-16T03:39:03.822624+0000","last_unstale":"2024-09-16T03:39:03.923159+0000","last_undegraded":"2024-09-16T03:39:03.923159+0000","last_fullsized":"2024-09-16T03:39:03.923159+0000","mapping_epoch":128,"log_start":"106'17","ondisk_log_start":"106'17","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966452,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":26744,"num_write":10,"num_write_kb":2629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2569696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,11],"acting":[14,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.13","version":"127'73","reported_seq":448,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.629829+0000","last_change":"2024-09-16T03:39:03.124648+0000","last_active":"2024-09-16T03:39:03.629829+0000","last_peered":"2024-09-16T03:39:03.629829+0000","last_clean":"2024-09-16T03:39:03.629829+0000","last_became_active":"2024-09-16T03:39:01.843090+0000","last_became_peered":"2024-09-16T03:39:01.843090+0000","last_unstale":"2024-09-16T03:39:03.629829+0000","last_undegraded":"2024-09-16T03:39:03.629829+0000","last_fullsized":"2024-09-16T03:39:03.629829+0000","mapping_epoch":119,"log_start":"108'70","ondisk_log_start":"108'70","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'73","last_scrub_stamp":"2024-09-16T03:39:03.124583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.124583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":68,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:24.286892+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,14],"acting":[4,3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"104'20","reported_seq":316,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622809+0000","last_change":"2024-09-16T03:39:01.898409+0000","last_active":"2024-09-16T03:39:05.622809+0000","last_peered":"2024-09-16T03:39:05.622809+0000","last_clean":"2024-09-16T03:39:05.622809+0000","last_became_active":"2024-09-16T03:39:01.596287+0000","last_became_peered":"2024-09-16T03:39:01.596287+0000","last_unstale":"2024-09-16T03:39:05.622809+0000","last_undegraded":"2024-09-16T03:39:05.622809+0000","last_fullsized":"2024-09-16T03:39:05.622809+0000","mapping_epoch":114,"log_start":"98'17","ondisk_log_start":"98'17","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:01.898331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.898331+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.472424+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"123'65","reported_seq":438,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.916162+0000","last_change":"2024-09-16T03:39:03.916087+0000","last_active":"2024-09-16T03:39:03.916162+0000","last_peered":"2024-09-16T03:39:03.916162+0000","last_clean":"2024-09-16T03:39:03.916162+0000","last_became_active":"2024-09-16T03:39:03.915707+0000","last_became_peered":"2024-09-16T03:39:03.915707+0000","last_unstale":"2024-09-16T03:39:03.916162+0000","last_undegraded":"2024-09-16T03:39:03.916162+0000","last_fullsized":"2024-09-16T03:39:03.916162+0000","mapping_epoch":128,"log_start":"123'62","ondisk_log_start":"123'62","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916014,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":83986,"num_write":19,"num_write_kb":5459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":826342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,6],"acting":[3,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"124'86","reported_seq":446,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.782232+0000","last_change":"2024-09-16T03:39:03.073530+0000","last_active":"2024-09-16T03:39:05.782232+0000","last_peered":"2024-09-16T03:39:05.782232+0000","last_clean":"2024-09-16T03:39:05.782232+0000","last_became_active":"2024-09-16T03:39:01.797805+0000","last_became_peered":"2024-09-16T03:39:01.797805+0000","last_unstale":"2024-09-16T03:39:05.782232+0000","last_undegraded":"2024-09-16T03:39:05.782232+0000","last_fullsized":"2024-09-16T03:39:05.782232+0000","mapping_epoch":115,"log_start":"124'82","ondisk_log_start":"124'82","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.073459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.073459+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":80,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:32.204501+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.17","version":"117'49","reported_seq":385,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.719166+0000","last_change":"2024-09-16T03:39:03.717673+0000","last_active":"2024-09-16T03:39:03.719166+0000","last_peered":"2024-09-16T03:39:03.719166+0000","last_clean":"2024-09-16T03:39:03.719166+0000","last_became_active":"2024-09-16T03:39:03.717420+0000","last_became_peered":"2024-09-16T03:39:03.717420+0000","last_unstale":"2024-09-16T03:39:03.719166+0000","last_undegraded":"2024-09-16T03:39:03.719166+0000","last_fullsized":"2024-09-16T03:39:03.719166+0000","mapping_epoch":128,"log_start":"104'45","ondisk_log_start":"104'45","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":44,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2890820,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":44515,"num_write":7,"num_write_kb":2114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2516795,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"128'51","reported_seq":363,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.798775+0000","last_change":"2024-09-16T03:39:02.668794+0000","last_active":"2024-09-16T03:39:03.798775+0000","last_peered":"2024-09-16T03:39:03.798775+0000","last_clean":"2024-09-16T03:39:03.798775+0000","last_became_active":"2024-09-16T03:38:47.562641+0000","last_became_peered":"2024-09-16T03:38:47.562641+0000","last_unstale":"2024-09-16T03:39:03.798775+0000","last_undegraded":"2024-09-16T03:39:03.798775+0000","last_fullsized":"2024-09-16T03:39:03.798775+0000","mapping_epoch":112,"log_start":"124'48","ondisk_log_start":"124'48","created":50,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'48","last_scrub_stamp":"2024-09-16T03:39:02.668681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.668681+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":45,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.451476+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067327,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":21991,"num_write":23,"num_write_kb":6049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":51,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622532+0000","last_change":"2024-09-16T03:39:01.106936+0000","last_active":"2024-09-16T03:39:05.622532+0000","last_peered":"2024-09-16T03:39:05.622532+0000","last_clean":"2024-09-16T03:39:05.622532+0000","last_became_active":"2024-09-16T03:38:51.558378+0000","last_became_peered":"2024-09-16T03:38:51.558378+0000","last_unstale":"2024-09-16T03:39:05.622532+0000","last_undegraded":"2024-09-16T03:39:05.622532+0000","last_fullsized":"2024-09-16T03:39:05.622532+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:01.106840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.106840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.164091+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,6],"acting":[2,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"128'125","reported_seq":573,"reported_epoch":129,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:03.515341+0000","last_change":"2024-09-16T03:38:47.159152+0000","last_active":"2024-09-16T03:39:03.515341+0000","last_peered":"2024-09-16T03:39:03.515341+0000","last_clean":"2024-09-16T03:38:43.521485+0000","last_became_active":"2024-09-16T03:38:46.957471+0000","last_became_peered":"2024-09-16T03:38:46.957471+0000","last_unstale":"2024-09-16T03:39:03.515341+0000","last_undegraded":"2024-09-16T03:39:03.515341+0000","last_fullsized":"2024-09-16T03:39:03.515341+0000","mapping_epoch":111,"log_start":"117'122","ondisk_log_start":"117'122","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'115","last_scrub_stamp":"2024-09-16T03:38:35.254488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.254488+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":122,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6739673,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":31161,"num_write":9,"num_write_kb":2915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6,11],"acting":[3,8,10],"avail_no_missing":["3","8","10"],"object_location_counts":[{"shards":"3,8,10","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"119'124","reported_seq":569,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:04.237300+0000","last_change":"2024-09-16T03:38:50.465770+0000","last_active":"2024-09-16T03:39:04.237300+0000","last_peered":"2024-09-16T03:39:04.237300+0000","last_clean":"2024-09-16T03:39:04.237300+0000","last_became_active":"2024-09-16T03:38:50.465463+0000","last_became_peered":"2024-09-16T03:38:50.465463+0000","last_unstale":"2024-09-16T03:39:04.237300+0000","last_undegraded":"2024-09-16T03:39:04.237300+0000","last_fullsized":"2024-09-16T03:39:04.237300+0000","mapping_epoch":115,"log_start":"116'120","ondisk_log_start":"116'120","created":100,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'107","last_scrub_stamp":"2024-09-16T03:38:39.905697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.905697+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":114,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.828899+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7159848,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":74651,"num_write":20,"num_write_kb":6207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2376197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"99'56","reported_seq":356,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.542809+0000","last_change":"2024-09-16T03:39:00.988586+0000","last_active":"2024-09-16T03:39:06.542809+0000","last_peered":"2024-09-16T03:39:06.542809+0000","last_clean":"2024-09-16T03:39:06.542809+0000","last_became_active":"2024-09-16T03:38:53.479253+0000","last_became_peered":"2024-09-16T03:38:53.479253+0000","last_unstale":"2024-09-16T03:39:06.542809+0000","last_undegraded":"2024-09-16T03:39:06.542809+0000","last_fullsized":"2024-09-16T03:39:06.542809+0000","mapping_epoch":118,"log_start":"90'53","ondisk_log_start":"90'53","created":100,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":5,"last_scrub":"99'56","last_scrub_stamp":"2024-09-16T03:39:00.988517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.988517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.840710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"92'14","reported_seq":269,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.614582+0000","last_change":"2024-09-16T03:38:49.092747+0000","last_active":"2024-09-16T03:39:03.614582+0000","last_peered":"2024-09-16T03:39:03.614582+0000","last_clean":"2024-09-16T03:39:03.614582+0000","last_became_active":"2024-09-16T03:38:48.661213+0000","last_became_peered":"2024-09-16T03:38:48.661213+0000","last_unstale":"2024-09-16T03:39:03.614582+0000","last_undegraded":"2024-09-16T03:39:03.614582+0000","last_fullsized":"2024-09-16T03:39:03.614582+0000","mapping_epoch":113,"log_start":"91'11","ondisk_log_start":"91'11","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:49.092687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.092687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.372348+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,12],"acting":[3,11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"92'59","reported_seq":378,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517440+0000","last_change":"2024-09-16T03:38:48.561603+0000","last_active":"2024-09-16T03:39:03.517440+0000","last_peered":"2024-09-16T03:39:03.517440+0000","last_clean":"2024-09-16T03:39:03.517440+0000","last_became_active":"2024-09-16T03:38:48.561170+0000","last_became_peered":"2024-09-16T03:38:48.561170+0000","last_unstale":"2024-09-16T03:39:03.517440+0000","last_undegraded":"2024-09-16T03:39:03.517440+0000","last_fullsized":"2024-09-16T03:39:03.517440+0000","mapping_epoch":113,"log_start":"89'56","ondisk_log_start":"89'56","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'59","last_scrub_stamp":"2024-09-16T03:38:35.977646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.977646+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:38.632635+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,6],"acting":[14,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1f","version":"118'111","reported_seq":481,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.951101+0000","last_change":"2024-09-16T03:38:48.762617+0000","last_active":"2024-09-16T03:39:03.951101+0000","last_peered":"2024-09-16T03:39:03.951101+0000","last_clean":"2024-09-16T03:39:03.951101+0000","last_became_active":"2024-09-16T03:38:48.761213+0000","last_became_peered":"2024-09-16T03:38:48.761213+0000","last_unstale":"2024-09-16T03:39:03.951101+0000","last_undegraded":"2024-09-16T03:39:03.951101+0000","last_fullsized":"2024-09-16T03:39:03.951101+0000","mapping_epoch":113,"log_start":"115'107","ondisk_log_start":"115'107","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'95","last_scrub_stamp":"2024-09-16T03:38:37.201959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:37.201959+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":107,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.307408+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9748159,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":35115,"num_write":27,"num_write_kb":6985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1704821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,12],"acting":[3,10,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"127'143","reported_seq":589,"reported_epoch":129,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:03.629583+0000","last_change":"2024-09-16T03:38:46.862171+0000","last_active":"2024-09-16T03:39:03.629583+0000","last_peered":"2024-09-16T03:39:03.629583+0000","last_clean":"2024-09-16T03:38:44.034266+0000","last_became_active":"2024-09-16T03:38:46.560752+0000","last_became_peered":"2024-09-16T03:38:46.560752+0000","last_unstale":"2024-09-16T03:39:03.629583+0000","last_undegraded":"2024-09-16T03:39:03.629583+0000","last_fullsized":"2024-09-16T03:39:03.629583+0000","mapping_epoch":111,"log_start":"127'140","ondisk_log_start":"127'140","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'118","last_scrub_stamp":"2024-09-16T03:38:36.698150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:36.698150+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":140,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":93,"num_read_kb":75680,"num_write":30,"num_write_kb":9656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[4,10,15],"avail_no_missing":["4","10","15"],"object_location_counts":[{"shards":"4,10,15","objects":3}],"blocked_by":[],"up_primary":8,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.21","version":"48'7","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517018+0000","last_change":"2024-09-16T03:38:49.064947+0000","last_active":"2024-09-16T03:39:03.517018+0000","last_peered":"2024-09-16T03:39:03.517018+0000","last_clean":"2024-09-16T03:39:03.517018+0000","last_became_active":"2024-09-16T03:38:48.559431+0000","last_became_peered":"2024-09-16T03:38:48.559431+0000","last_unstale":"2024-09-16T03:39:03.517018+0000","last_undegraded":"2024-09-16T03:39:03.517018+0000","last_fullsized":"2024-09-16T03:39:03.517018+0000","mapping_epoch":113,"log_start":"40'4","ondisk_log_start":"40'4","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":6,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:49.064870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064870+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.310170+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,10],"acting":[14,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.22","version":"45'6","reported_seq":121,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.629609+0000","last_change":"2024-09-16T03:39:00.968576+0000","last_active":"2024-09-16T03:39:03.629609+0000","last_peered":"2024-09-16T03:39:03.629609+0000","last_clean":"2024-09-16T03:39:03.629609+0000","last_became_active":"2024-09-16T03:38:54.462316+0000","last_became_peered":"2024-09-16T03:38:54.462316+0000","last_unstale":"2024-09-16T03:39:03.629609+0000","last_undegraded":"2024-09-16T03:39:03.629609+0000","last_fullsized":"2024-09-16T03:39:03.629609+0000","mapping_epoch":119,"log_start":"39'3","ondisk_log_start":"39'3","created":100,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:39:00.968448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.968448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:06.219174+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,12],"acting":[4,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"108'38","reported_seq":345,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517421+0000","last_change":"2024-09-16T03:38:49.464614+0000","last_active":"2024-09-16T03:39:03.517421+0000","last_peered":"2024-09-16T03:39:03.517421+0000","last_clean":"2024-09-16T03:39:03.517421+0000","last_became_active":"2024-09-16T03:38:49.463797+0000","last_became_peered":"2024-09-16T03:38:49.463797+0000","last_unstale":"2024-09-16T03:39:03.517421+0000","last_undegraded":"2024-09-16T03:39:03.517421+0000","last_fullsized":"2024-09-16T03:39:03.517421+0000","mapping_epoch":114,"log_start":"105'36","ondisk_log_start":"105'36","created":100,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:34.893524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.893524+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":35,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:52.829073+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,2],"acting":[14,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.24","version":"105'32","reported_seq":325,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.823050+0000","last_change":"2024-09-16T03:38:47.978956+0000","last_active":"2024-09-16T03:39:05.823050+0000","last_peered":"2024-09-16T03:39:05.823050+0000","last_clean":"2024-09-16T03:39:05.823050+0000","last_became_active":"2024-09-16T03:38:45.820201+0000","last_became_peered":"2024-09-16T03:38:45.820201+0000","last_unstale":"2024-09-16T03:39:05.823050+0000","last_undegraded":"2024-09-16T03:39:05.823050+0000","last_fullsized":"2024-09-16T03:39:05.823050+0000","mapping_epoch":110,"log_start":"83'29","ondisk_log_start":"83'29","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.978888+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:56.881865+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.25","version":"130'85","reported_seq":414,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.679497+0000","last_change":"2024-09-16T03:38:48.088833+0000","last_active":"2024-09-16T03:39:05.679497+0000","last_peered":"2024-09-16T03:39:05.679497+0000","last_clean":"2024-09-16T03:39:05.679497+0000","last_became_active":"2024-09-16T03:38:45.724757+0000","last_became_peered":"2024-09-16T03:38:45.724757+0000","last_unstale":"2024-09-16T03:39:05.679497+0000","last_undegraded":"2024-09-16T03:39:05.679497+0000","last_fullsized":"2024-09-16T03:39:05.679497+0000","mapping_epoch":110,"log_start":"116'82","ondisk_log_start":"116'82","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'77","last_scrub_stamp":"2024-09-16T03:38:48.088768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.088768+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":79,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:53.112635+0000","scrub_duration":388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2134555,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":10,"num_write_kb":2500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.26","version":"123'52","reported_seq":389,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517824+0000","last_change":"2024-09-16T03:38:49.477393+0000","last_active":"2024-09-16T03:39:03.517824+0000","last_peered":"2024-09-16T03:39:03.517824+0000","last_clean":"2024-09-16T03:39:03.517824+0000","last_became_active":"2024-09-16T03:38:45.926029+0000","last_became_peered":"2024-09-16T03:38:45.926029+0000","last_unstale":"2024-09-16T03:39:03.517824+0000","last_undegraded":"2024-09-16T03:39:03.517824+0000","last_fullsized":"2024-09-16T03:39:03.517824+0000","mapping_epoch":110,"log_start":"123'49","ondisk_log_start":"123'49","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:49.477297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.477297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":48,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.242430+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3815222,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":23772,"num_write":10,"num_write_kb":3140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.27","version":"109'66","reported_seq":379,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.519880+0000","last_change":"2024-09-16T03:38:47.511343+0000","last_active":"2024-09-16T03:39:03.519880+0000","last_peered":"2024-09-16T03:39:03.519880+0000","last_clean":"2024-09-16T03:39:03.519880+0000","last_became_active":"2024-09-16T03:38:45.824884+0000","last_became_peered":"2024-09-16T03:38:45.824884+0000","last_unstale":"2024-09-16T03:39:03.519880+0000","last_undegraded":"2024-09-16T03:39:03.519880+0000","last_fullsized":"2024-09-16T03:39:03.519880+0000","mapping_epoch":110,"log_start":"105'63","ondisk_log_start":"105'63","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'66","last_scrub_stamp":"2024-09-16T03:38:47.511252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.511252+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.884916+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.28","version":"118'67","reported_seq":441,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.515839+0000","last_change":"2024-09-16T03:38:55.727806+0000","last_active":"2024-09-16T03:39:03.515839+0000","last_peered":"2024-09-16T03:39:03.515839+0000","last_clean":"2024-09-16T03:39:03.515839+0000","last_became_active":"2024-09-16T03:38:55.727094+0000","last_became_peered":"2024-09-16T03:38:55.727094+0000","last_unstale":"2024-09-16T03:39:03.515839+0000","last_undegraded":"2024-09-16T03:39:03.515839+0000","last_fullsized":"2024-09-16T03:39:03.515839+0000","mapping_epoch":120,"log_start":"105'63","ondisk_log_start":"105'63","created":110,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:48.551929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.551929+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":62,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:00.496373+0000","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395113,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":21976,"num_write":5,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3610944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,10],"acting":[0,4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":241,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.517045+0000","last_change":"2024-09-16T03:39:00.834877+0000","last_active":"2024-09-16T03:39:03.517045+0000","last_peered":"2024-09-16T03:39:03.517045+0000","last_clean":"2024-09-16T03:39:03.517045+0000","last_became_active":"2024-09-16T03:38:45.627369+0000","last_became_peered":"2024-09-16T03:38:45.627369+0000","last_unstale":"2024-09-16T03:39:03.517045+0000","last_undegraded":"2024-09-16T03:39:03.517045+0000","last_fullsized":"2024-09-16T03:39:03.517045+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:00.834742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.834742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.318226+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.2a","version":"108'122","reported_seq":594,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.798898+0000","last_change":"2024-09-16T03:38:47.964525+0000","last_active":"2024-09-16T03:39:03.798898+0000","last_peered":"2024-09-16T03:39:03.798898+0000","last_clean":"2024-09-16T03:39:03.798898+0000","last_became_active":"2024-09-16T03:38:46.022108+0000","last_became_peered":"2024-09-16T03:38:46.022108+0000","last_unstale":"2024-09-16T03:39:03.798898+0000","last_undegraded":"2024-09-16T03:39:03.798898+0000","last_fullsized":"2024-09-16T03:39:03.798898+0000","mapping_epoch":110,"log_start":"107'115","ondisk_log_start":"107'115","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'122","last_scrub_stamp":"2024-09-16T03:38:47.964439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.964439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":112,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.917123+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.2b","version":"127'124","reported_seq":571,"reported_epoch":131,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:05.585588+0000","last_change":"2024-09-16T03:38:53.063728+0000","last_active":"2024-09-16T03:39:05.585588+0000","last_peered":"2024-09-16T03:39:05.585588+0000","last_clean":"2024-09-16T03:38:49.362783+0000","last_became_active":"2024-09-16T03:38:52.761638+0000","last_became_peered":"2024-09-16T03:38:52.761638+0000","last_unstale":"2024-09-16T03:39:05.585588+0000","last_undegraded":"2024-09-16T03:39:05.585588+0000","last_fullsized":"2024-09-16T03:39:05.585588+0000","mapping_epoch":117,"log_start":"119'121","ondisk_log_start":"119'121","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:49.064061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064061+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":121,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,12],"acting":[15,4,9],"avail_no_missing":["15","4","9","12"],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2c","version":"121'63","reported_seq":396,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.562891+0000","last_change":"2024-09-16T03:38:55.563264+0000","last_active":"2024-09-16T03:39:03.562891+0000","last_peered":"2024-09-16T03:39:03.562891+0000","last_clean":"2024-09-16T03:39:03.562891+0000","last_became_active":"2024-09-16T03:38:55.562810+0000","last_became_peered":"2024-09-16T03:38:55.562810+0000","last_unstale":"2024-09-16T03:39:03.562891+0000","last_undegraded":"2024-09-16T03:39:03.562891+0000","last_fullsized":"2024-09-16T03:39:03.562891+0000","mapping_epoch":120,"log_start":"118'60","ondisk_log_start":"118'60","created":110,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'58","last_scrub_stamp":"2024-09-16T03:38:47.889276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.889276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.138029+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1262403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,11],"acting":[12,15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.2d","version":"92'14","reported_seq":322,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.614665+0000","last_change":"2024-09-16T03:38:47.115739+0000","last_active":"2024-09-16T03:39:03.614665+0000","last_peered":"2024-09-16T03:39:03.614665+0000","last_clean":"2024-09-16T03:39:03.614665+0000","last_became_active":"2024-09-16T03:38:45.723438+0000","last_became_peered":"2024-09-16T03:38:45.723438+0000","last_unstale":"2024-09-16T03:39:03.614665+0000","last_undegraded":"2024-09-16T03:39:03.614665+0000","last_fullsized":"2024-09-16T03:39:03.614665+0000","mapping_epoch":110,"log_start":"91'11","ondisk_log_start":"91'11","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:47.115651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.115651+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:00.107802+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":56,"stat_sum":{"num_bytes":133995457,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":828,"num_read_kb":771081,"num_write":311,"num_write_kb":87163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":21165486,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":251191296,"data_stored":250753849,"data_compressed":606603,"data_compressed_allocated":663552,"data_compressed_original":1568768,"omap_allocated":0,"internal_metadata":0},"log_size":169,"ondisk_log_size":169,"up":165,"acting":165,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":80670503,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":920,"num_read_kb":29770,"num_write":1253,"num_write_kb":280770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":39649106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183762944,"data_stored":196582908,"data_compressed":11492126,"data_compressed_allocated":12234752,"data_compressed_original":25473024,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":2644640,"data_compressed":159066,"data_compressed_allocated":204800,"data_compressed_original":2584576,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247837,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83256,"kb_used_data":43372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94288584,"statfs":{"total":96636764160,"available":96551510016,"internally_reserved":0,"allocated":44412928,"data_stored":45041141,"data_compressed":1329713,"data_compressed_allocated":1417216,"data_compressed_original":2596864,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247838,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60108,"kb_used_data":25512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311732,"statfs":{"total":96636764160,"available":96575213568,"internally_reserved":0,"allocated":26124288,"data_stored":27258963,"data_compressed":896583,"data_compressed_allocated":978944,"data_compressed_original":2617344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247837,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73204,"kb_used_data":38960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94298636,"statfs":{"total":96636764160,"available":96561803264,"internally_reserved":0,"allocated":39895040,"data_stored":40905705,"data_compressed":1172110,"data_compressed_allocated":1261568,"data_compressed_original":2797568,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280542,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53928,"kb_used_data":22736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94317912,"statfs":{"total":96636764160,"available":96581541888,"internally_reserved":0,"allocated":23281664,"data_stored":24303845,"data_compressed":842009,"data_compressed_allocated":921600,"data_compressed_original":2420736,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280542,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280541,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313246,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36664,"kb_used_data":7296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335176,"statfs":{"total":96636764160,"available":96599220224,"internally_reserved":0,"allocated":7471104,"data_stored":7083983,"data_compressed":25616,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280542,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88900,"kb_used_data":52272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94282940,"statfs":{"total":96636764160,"available":96545730560,"internally_reserved":0,"allocated":53526528,"data_stored":54527195,"data_compressed":1186238,"data_compressed_allocated":1277952,"data_compressed_original":2863104,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280541,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100932,"kb_used_data":65448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94270908,"statfs":{"total":96636764160,"available":96533409792,"internally_reserved":0,"allocated":67018752,"data_stored":68966584,"data_compressed":1952237,"data_compressed_allocated":2097152,"data_compressed_original":4653056,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313246,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280541,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37192,"kb_used_data":5420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334648,"statfs":{"total":96636764160,"available":96598679552,"internally_reserved":0,"allocated":5550080,"data_stored":5211330,"data_compressed":25616,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280541,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71344,"kb_used_data":40160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94300496,"statfs":{"total":96636764160,"available":96563707904,"internally_reserved":0,"allocated":41123840,"data_stored":41803559,"data_compressed":1229872,"data_compressed_allocated":1314816,"data_compressed_original":2510848,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280541,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38652,"kb_used_data":5076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333188,"statfs":{"total":96636764160,"available":96597184512,"internally_reserved":0,"allocated":5197824,"data_stored":5332821,"data_compressed":69039,"data_compressed_allocated":90112,"data_compressed_original":618496,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,3,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280542,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91884,"kb_used_data":57568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94279956,"statfs":{"total":96636764160,"available":96542674944,"internally_reserved":0,"allocated":58949632,"data_stored":58629208,"data_compressed":220712,"data_compressed_allocated":249856,"data_compressed_original":598016,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,3,4,5,6,7,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313245,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43672,"kb_used_data":12484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328168,"statfs":{"total":96636764160,"available":96592044032,"internally_reserved":0,"allocated":12783616,"data_stored":12453761,"data_compressed":109264,"data_compressed_allocated":126976,"data_compressed_original":270336,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247838,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57696,"kb_used_data":25268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94314144,"statfs":{"total":96636764160,"available":96577683456,"internally_reserved":0,"allocated":25874432,"data_stored":26571892,"data_compressed":1270271,"data_compressed_allocated":1355776,"data_compressed_original":2510848,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13885440,"data_stored":14937543,"data_compressed":829441,"data_compressed_allocated":888832,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13422592,"data_stored":14113910,"data_compressed":606276,"data_compressed_allocated":643072,"data_compressed_original":1376256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14032896,"data_stored":15086444,"data_compressed":1224396,"data_compressed_allocated":1294336,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20283392,"data_stored":21623696,"data_compressed":1067426,"data_compressed_allocated":1138688,"data_compressed_original":2527232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":34295808,"data_stored":36685143,"data_compressed":1896867,"data_compressed_allocated":2027520,"data_compressed_original":4493312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21327872,"data_stored":22801296,"data_compressed":1184415,"data_compressed_allocated":1265664,"data_compressed_original":2789376,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14458880,"data_stored":15539052,"data_compressed":1194708,"data_compressed_allocated":1265664,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":12505088,"data_stored":13249430,"data_compressed":646795,"data_compressed_allocated":684032,"data_compressed_original":1470464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14012416,"data_stored":15061447,"data_compressed":829441,"data_compressed_allocated":888832,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14032896,"data_stored":15086444,"data_compressed":1224396,"data_compressed_allocated":1294336,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10883072,"data_stored":10816365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10244096,"data_stored":10207520,"data_compressed":49947,"data_compressed_allocated":53248,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":31731712,"data_stored":31727597,"data_compressed":93196,"data_compressed_allocated":102400,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30445568,"data_stored":30338139,"data_compressed":29754,"data_compressed_allocated":32768,"data_compressed_original":86016,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16941056,"data_stored":16931882,"data_compressed":44673,"data_compressed_allocated":49152,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3395584,"data_stored":3372648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":25522176,"data_stored":25455397,"data_compressed":9548,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4005888,"data_stored":3995623,"data_compressed":10762,"data_compressed_allocated":12288,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":57798656,"data_stored":57819101,"data_compressed":195096,"data_compressed_allocated":212992,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":10862592,"data_stored":10841283,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":11612160,"data_stored":11641798,"data_compressed":83648,"data_compressed_allocated":90112,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":7446528,"data_stored":7397291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":29212672,"data_stored":29129165,"data_compressed":79701,"data_compressed_allocated":86016,"data_compressed_original":151552,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:39:09.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T03:39:09.252 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:09.251+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:09.303 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:09.303 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:09.312 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:39:09.317 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:09.317 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:09.335 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:09.335 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:09.352 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:09.351+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:09.353 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:09.353 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:09.415 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:09.415 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:09.423 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:39:09.453 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:09.451+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:09.525 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:09.525 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:09.534 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:39:09.545 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:09.544+0000 7f66f5209640 -1 osd.15 134 handle_osd_map injecting map crc failure 2024-09-16T03:39:09.547 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:09.545+0000 7eff7884b640 -1 osd.3 134 handle_osd_map injecting map crc failure 2024-09-16T03:39:09.549 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:39:09.592 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:09.591+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:09.638 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:09.638 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:09.647 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:39:09.647 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:09.647 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull","count":4},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":152,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":135,"num_osds":16,"num_up_osds":16,"osd_up_since":1726457810,"num_in_osds":13,"osd_in_since":1726457912,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":61},{"state_name":"active+remapped+backfill_toofull","count":4},{"state_name":"active+remapped+backfilling","count":2},{"state_name":"unknown","count":1},{"state_name":"active+clean+scrubbing","count":1}],"num_pgs":69,"num_pools":4,"num_objects":137,"data_bytes":215256347,"bytes_used":857530368,"bytes_avail":1255420403712,"bytes_total":1256277934080,"unknown_pgs_ratio":0.014492753893136978,"misplaced_objects":17,"misplaced_total":411,"misplaced_ratio":0.041362530413625302},"fsmap":{"epoch":1,"btime":"2024-09-16T03:36:37:310011+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T03:37:51.688536+0000","services":{}},"progress_events":{}} 2024-09-16T03:39:09.654 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:09.652+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:09.661 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-16T03:39:09.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:39:09.748 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:09.748 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:09.751 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:09.751 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:09.755 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:09.754+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:09.760 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:39:09.784 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:09.784 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:09.807 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:09.807 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:09.824 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:09.825 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:09.841 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:09.842 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:09.856 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:09.855+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:09.858 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:09.858 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:09.865 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:09.865 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:09.874 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:39:09.875 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:09.875 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:09.891 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:09.891 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:09.908 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:09.909 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:09.924 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:09.925 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:09.941 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:09.941 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:09.957 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:09.956+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:09.958 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:09.959 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:09.975 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:09.976 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:09.978 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:09.979 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:09.979 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:09.988 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:39:10.011 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:10.012 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:39:10.023 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-16T03:39:09.705194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":209895113,"num_objects":138,"num_object_clones":0,"num_object_copies":414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":1936,"num_read_kb":893114,"num_write":1732,"num_write_kb":375394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":62102448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":207,"acting":207,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":218,"num_osds":13,"num_per_pool_osds":13,"num_per_pool_omap_osds":13,"kb":1226833920,"kb_used":829196,"kb_used_data":396016,"kb_used_omap":20,"kb_used_meta":405163,"kb_avail":1226004724,"statfs":{"total":1256277934080,"available":1255428837376,"internally_reserved":0,"allocated":405520384,"data_stored":412551301,"data_compressed":10602046,"data_compressed_allocated":11452416,"data_compressed_original":25128960,"omap_allocated":20735,"internal_metadata":414887681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11423092,"num_objects":-22,"num_object_clones":0,"num_object_copies":-66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2,"num_objects_unfound":0,"num_objects_dirty":-21,"num_whiteouts":0,"num_read":-31,"num_read_kb":-55539,"num_write":30,"num_write_kb":-4382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":-7553389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000690"},"pg_stats":[{"pgid":"4.37","version":"131'52","reported_seq":397,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641567+0000","last_change":"2024-09-16T03:39:08.371859+0000","last_active":"2024-09-16T03:39:08.641567+0000","last_peered":"2024-09-16T03:39:08.641567+0000","last_clean":"2024-09-16T03:39:08.641567+0000","last_became_active":"2024-09-16T03:39:05.721626+0000","last_became_peered":"2024-09-16T03:39:05.721626+0000","last_unstale":"2024-09-16T03:39:08.641567+0000","last_undegraded":"2024-09-16T03:39:08.641567+0000","last_fullsized":"2024-09-16T03:39:08.641567+0000","mapping_epoch":130,"log_start":"104'45","ondisk_log_start":"104'45","created":126,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":44,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:36.963537+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2890887,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":44517,"num_write":12,"num_write_kb":3069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1148090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.36","version":"131'93","reported_seq":460,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.251987+0000","last_change":"2024-09-16T03:39:03.716194+0000","last_active":"2024-09-16T03:39:06.251987+0000","last_peered":"2024-09-16T03:39:06.251987+0000","last_clean":"2024-09-16T03:39:06.251987+0000","last_became_active":"2024-09-16T03:39:02.099320+0000","last_became_peered":"2024-09-16T03:39:02.099320+0000","last_unstale":"2024-09-16T03:39:06.251987+0000","last_undegraded":"2024-09-16T03:39:06.251987+0000","last_fullsized":"2024-09-16T03:39:06.251987+0000","mapping_epoch":126,"log_start":"130'90","ondisk_log_start":"130'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.716125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.716125+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":88,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.328700+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269209,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":4665,"num_write":8,"num_write_kb":2101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.35","version":"131'66","reported_seq":456,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.529794+0000","last_change":"2024-09-16T03:39:08.541599+0000","last_active":"2024-09-16T03:39:08.541599+0000","last_peered":"2024-09-16T03:39:08.529794+0000","last_clean":"2024-09-16T03:39:08.529794+0000","last_became_active":"2024-09-16T03:39:05.722545+0000","last_became_peered":"2024-09-16T03:39:05.722545+0000","last_unstale":"2024-09-16T03:39:08.529794+0000","last_undegraded":"2024-09-16T03:39:08.529794+0000","last_fullsized":"2024-09-16T03:39:08.529794+0000","mapping_epoch":134,"log_start":"123'63","ondisk_log_start":"123'63","created":126,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916015,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84364,"num_write":19,"num_write_kb":5459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3131048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,9,10],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"104'20","reported_seq":317,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622626+0000","last_change":"2024-09-16T03:39:02.919356+0000","last_active":"2024-09-16T03:39:05.622626+0000","last_peered":"2024-09-16T03:39:05.622626+0000","last_clean":"2024-09-16T03:39:05.622626+0000","last_became_active":"2024-09-16T03:39:01.699286+0000","last_became_peered":"2024-09-16T03:39:01.699286+0000","last_unstale":"2024-09-16T03:39:05.622626+0000","last_undegraded":"2024-09-16T03:39:05.622626+0000","last_fullsized":"2024-09-16T03:39:05.622626+0000","mapping_epoch":126,"log_start":"98'17","ondisk_log_start":"98'17","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:02.919259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.919259+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.209096+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2705960,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"108'72","reported_seq":449,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.730259+0000","last_change":"2024-09-16T03:39:02.143407+0000","last_active":"2024-09-16T03:39:03.730259+0000","last_peered":"2024-09-16T03:39:03.730259+0000","last_clean":"2024-09-16T03:39:03.730259+0000","last_became_active":"2024-09-16T03:39:01.642310+0000","last_became_peered":"2024-09-16T03:39:01.642310+0000","last_unstale":"2024-09-16T03:39:03.730259+0000","last_undegraded":"2024-09-16T03:39:03.730259+0000","last_fullsized":"2024-09-16T03:39:03.730259+0000","mapping_epoch":126,"log_start":"108'70","ondisk_log_start":"108'70","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'72","last_scrub_stamp":"2024-09-16T03:39:02.143339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.143339+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":68,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:29.596134+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2722767,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":6432,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,14],"acting":[4,3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.32","version":"106'17","reported_seq":355,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.471820+0000","last_change":"2024-09-16T03:39:08.648217+0000","last_active":"2024-09-16T03:39:08.648217+0000","last_peered":"2024-09-16T03:39:08.471820+0000","last_clean":"2024-09-16T03:39:08.471820+0000","last_became_active":"2024-09-16T03:39:05.823393+0000","last_became_peered":"2024-09-16T03:39:05.823393+0000","last_unstale":"2024-09-16T03:39:08.471820+0000","last_undegraded":"2024-09-16T03:39:08.471820+0000","last_fullsized":"2024-09-16T03:39:08.471820+0000","mapping_epoch":134,"log_start":"82'13","ondisk_log_start":"82'13","created":126,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1033647,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":26745,"num_write":6,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1331059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,9,11],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.31","version":"114'17","reported_seq":287,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622532+0000","last_change":"2024-09-16T03:39:05.622532+0000","last_active":"2024-09-16T03:39:05.622532+0000","last_peered":"2024-09-16T03:39:05.622532+0000","last_clean":"2024-09-16T03:39:05.622532+0000","last_became_active":"2024-09-16T03:39:01.997491+0000","last_became_peered":"2024-09-16T03:39:01.997491+0000","last_unstale":"2024-09-16T03:39:05.622532+0000","last_undegraded":"2024-09-16T03:39:05.622532+0000","last_fullsized":"2024-09-16T03:39:05.622532+0000","mapping_epoch":126,"log_start":"107'14","ondisk_log_start":"107'14","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'9","last_scrub_stamp":"2024-09-16T03:37:50.756120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:50.756120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.630555+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1495840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":28490,"num_write":4,"num_write_kb":1066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"118'64","reported_seq":392,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.615533+0000","last_change":"2024-09-16T03:39:05.621064+0000","last_active":"2024-09-16T03:39:08.615533+0000","last_peered":"2024-09-16T03:39:08.615533+0000","last_clean":"2024-09-16T03:39:08.615533+0000","last_became_active":"2024-09-16T03:39:01.901037+0000","last_became_peered":"2024-09-16T03:39:01.901037+0000","last_unstale":"2024-09-16T03:39:08.615533+0000","last_undegraded":"2024-09-16T03:39:08.615533+0000","last_fullsized":"2024-09-16T03:39:08.615533+0000","mapping_epoch":126,"log_start":"113'61","ondisk_log_start":"113'61","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:35.157846+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1680717,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":23268,"num_write":10,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":918091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2f","version":"113'94","reported_seq":462,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.552454+0000","last_change":"2024-09-16T03:39:03.319075+0000","last_active":"2024-09-16T03:39:08.552454+0000","last_peered":"2024-09-16T03:39:08.552454+0000","last_clean":"2024-09-16T03:39:08.552454+0000","last_became_active":"2024-09-16T03:39:01.700988+0000","last_became_peered":"2024-09-16T03:39:01.700988+0000","last_unstale":"2024-09-16T03:39:08.552454+0000","last_undegraded":"2024-09-16T03:39:08.552454+0000","last_fullsized":"2024-09-16T03:39:08.552454+0000","mapping_epoch":126,"log_start":"107'90","ondisk_log_start":"107'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"113'94","last_scrub_stamp":"2024-09-16T03:39:03.319012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.319012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":87,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:13.773817+0000","scrub_duration":1038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2e","version":"123'75","reported_seq":432,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:04.235301+0000","last_change":"2024-09-16T03:39:04.235234+0000","last_active":"2024-09-16T03:39:04.235301+0000","last_peered":"2024-09-16T03:39:04.235301+0000","last_clean":"2024-09-16T03:39:04.235301+0000","last_became_active":"2024-09-16T03:39:01.999291+0000","last_became_peered":"2024-09-16T03:39:01.999291+0000","last_unstale":"2024-09-16T03:39:04.235301+0000","last_undegraded":"2024-09-16T03:39:04.235301+0000","last_fullsized":"2024-09-16T03:39:04.235301+0000","mapping_epoch":126,"log_start":"114'72","ondisk_log_start":"114'72","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"123'75","last_scrub_stamp":"2024-09-16T03:39:04.235141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.235141+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":72,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.010761+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"131'99","reported_seq":469,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.652681+0000","last_change":"2024-09-16T03:39:03.917422+0000","last_active":"2024-09-16T03:39:08.652681+0000","last_peered":"2024-09-16T03:39:08.652681+0000","last_clean":"2024-09-16T03:39:08.652681+0000","last_became_active":"2024-09-16T03:39:01.801112+0000","last_became_peered":"2024-09-16T03:39:01.801112+0000","last_unstale":"2024-09-16T03:39:08.652681+0000","last_undegraded":"2024-09-16T03:39:08.652681+0000","last_fullsized":"2024-09-16T03:39:08.652681+0000","mapping_epoch":32,"log_start":"128'96","ondisk_log_start":"128'96","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'97","last_scrub_stamp":"2024-09-16T03:39:03.917355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.917355+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":93,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.686164+0000","scrub_duration":771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2033713,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.e","version":"127'79","reported_seq":428,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.629943+0000","last_change":"2024-09-16T03:39:01.999648+0000","last_active":"2024-09-16T03:39:03.629943+0000","last_peered":"2024-09-16T03:39:03.629943+0000","last_clean":"2024-09-16T03:39:03.629943+0000","last_became_active":"2024-09-16T03:39:01.999282+0000","last_became_peered":"2024-09-16T03:39:01.999282+0000","last_unstale":"2024-09-16T03:39:03.629943+0000","last_undegraded":"2024-09-16T03:39:03.629943+0000","last_fullsized":"2024-09-16T03:39:03.629943+0000","mapping_epoch":32,"log_start":"123'75","ondisk_log_start":"123'75","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:39.194237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.194237+0000","objects_scrubbed":3,"log_size":4,"log_dups_size":72,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3705976,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":28930,"num_write":10,"num_write_kb":3507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"124'19","reported_seq":346,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.548749+0000","last_change":"2024-09-16T03:39:01.149616+0000","last_active":"2024-09-16T03:39:08.548749+0000","last_peered":"2024-09-16T03:39:08.548749+0000","last_clean":"2024-09-16T03:39:08.548749+0000","last_became_active":"2024-09-16T03:38:45.622965+0000","last_became_peered":"2024-09-16T03:38:45.622965+0000","last_unstale":"2024-09-16T03:39:08.548749+0000","last_undegraded":"2024-09-16T03:39:08.548749+0000","last_fullsized":"2024-09-16T03:39:08.548749+0000","mapping_epoch":72,"log_start":"118'16","ondisk_log_start":"118'16","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'19","last_scrub_stamp":"2024-09-16T03:39:01.149486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.149486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":16,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.574017+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613538,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"130'65","reported_seq":409,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:07.529674+0000","last_change":"2024-09-16T03:39:08.548464+0000","last_active":"2024-09-16T03:39:08.548464+0000","last_peered":"2024-09-16T03:39:07.529674+0000","last_clean":"2024-09-16T03:39:07.529674+0000","last_became_active":"2024-09-16T03:38:54.765107+0000","last_became_peered":"2024-09-16T03:38:54.765107+0000","last_unstale":"2024-09-16T03:39:07.529674+0000","last_undegraded":"2024-09-16T03:39:07.529674+0000","last_fullsized":"2024-09-16T03:39:07.529674+0000","mapping_epoch":134,"log_start":"122'62","ondisk_log_start":"122'62","created":32,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'57","last_scrub_stamp":"2024-09-16T03:38:48.923184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.923184+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":2798,"num_write":9,"num_write_kb":2888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[9,11,15],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"129'132","reported_seq":585,"reported_epoch":131,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:05.585816+0000","last_change":"2024-09-16T03:38:52.963503+0000","last_active":"2024-09-16T03:39:05.585816+0000","last_peered":"2024-09-16T03:39:05.585816+0000","last_clean":"2024-09-16T03:38:49.362875+0000","last_became_active":"2024-09-16T03:38:52.661315+0000","last_became_peered":"2024-09-16T03:38:52.661315+0000","last_unstale":"2024-09-16T03:39:05.585816+0000","last_undegraded":"2024-09-16T03:39:05.585816+0000","last_fullsized":"2024-09-16T03:39:05.585816+0000","mapping_epoch":117,"log_start":"127'129","ondisk_log_start":"127'129","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:46.855462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.855462+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":125,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2084134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":16397,"num_write":31,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,12],"acting":[15,4,9],"avail_no_missing":["15","4","9"],"object_location_counts":[{"shards":"4,9,15","objects":1}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.a","version":"128'148","reported_seq":687,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.547162+0000","last_change":"2024-09-16T03:39:01.504751+0000","last_active":"2024-09-16T03:39:09.547162+0000","last_peered":"2024-09-16T03:39:09.547162+0000","last_clean":"2024-09-16T03:39:09.547162+0000","last_became_active":"2024-09-16T03:38:46.021805+0000","last_became_peered":"2024-09-16T03:38:46.021805+0000","last_unstale":"2024-09-16T03:39:09.547162+0000","last_undegraded":"2024-09-16T03:39:09.547162+0000","last_fullsized":"2024-09-16T03:39:09.547162+0000","mapping_epoch":32,"log_start":"123'145","ondisk_log_start":"123'145","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'147","last_scrub_stamp":"2024-09-16T03:39:01.504663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.504663+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":142,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:18.060284+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13949942,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":53,"num_read_kb":53989,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":248,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641520+0000","last_change":"2024-09-16T03:38:46.881357+0000","last_active":"2024-09-16T03:39:08.641520+0000","last_peered":"2024-09-16T03:39:08.641520+0000","last_clean":"2024-09-16T03:39:08.641520+0000","last_became_active":"2024-09-16T03:38:45.728494+0000","last_became_peered":"2024-09-16T03:38:45.728494+0000","last_unstale":"2024-09-16T03:39:08.641520+0000","last_undegraded":"2024-09-16T03:39:08.641520+0000","last_fullsized":"2024-09-16T03:39:08.641520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:46.881248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.881248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:04.991122+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.8","version":"130'74","reported_seq":454,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.372137+0000","last_change":"2024-09-16T03:39:08.542925+0000","last_active":"2024-09-16T03:39:08.542925+0000","last_peered":"2024-09-16T03:39:08.372137+0000","last_clean":"2024-09-16T03:39:08.372137+0000","last_became_active":"2024-09-16T03:38:55.664848+0000","last_became_peered":"2024-09-16T03:38:55.664848+0000","last_unstale":"2024-09-16T03:39:08.372137+0000","last_undegraded":"2024-09-16T03:39:08.372137+0000","last_fullsized":"2024-09-16T03:39:08.372137+0000","mapping_epoch":134,"log_start":"124'71","ondisk_log_start":"124'71","created":32,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'71","last_scrub_stamp":"2024-09-16T03:39:02.002247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.002247+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":69,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2184976,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,9,10],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"130'85","reported_seq":424,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.585970+0000","last_change":"2024-09-16T03:39:00.900732+0000","last_active":"2024-09-16T03:39:05.585970+0000","last_peered":"2024-09-16T03:39:05.585970+0000","last_clean":"2024-09-16T03:39:05.585970+0000","last_became_active":"2024-09-16T03:38:45.523605+0000","last_became_peered":"2024-09-16T03:38:45.523605+0000","last_unstale":"2024-09-16T03:39:05.585970+0000","last_undegraded":"2024-09-16T03:39:05.585970+0000","last_fullsized":"2024-09-16T03:39:05.585970+0000","mapping_epoch":32,"log_start":"116'82","ondisk_log_start":"116'82","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'82","last_scrub_stamp":"2024-09-16T03:39:00.900619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.900619+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":80,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.385837+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2117748,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":417,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.722771+0000","last_change":"2024-09-16T03:38:36.757156+0000","last_active":"2024-09-16T03:39:05.722771+0000","last_peered":"2024-09-16T03:39:05.722771+0000","last_clean":"2024-09-16T03:39:05.722771+0000","last_became_active":"2024-09-16T03:38:36.756855+0000","last_became_peered":"2024-09-16T03:38:36.756855+0000","last_unstale":"2024-09-16T03:39:05.722771+0000","last_undegraded":"2024-09-16T03:39:05.722771+0000","last_fullsized":"2024-09-16T03:39:05.722771+0000","mapping_epoch":103,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:38:28.063690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:38:28.063690+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:28.682469+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":126,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.541136+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:39:08.541136+0000","last_peered":"2024-09-16T03:39:08.541136+0000","last_clean":"2024-09-16T03:39:08.541136+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:39:08.541136+0000","last_undegraded":"2024-09-16T03:39:08.541136+0000","last_fullsized":"2024-09-16T03:39:08.541136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"131'260","reported_seq":558,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.723185+0000","last_change":"2024-09-16T03:38:01.603352+0000","last_active":"2024-09-16T03:39:05.723185+0000","last_peered":"2024-09-16T03:39:05.723185+0000","last_clean":"2024-09-16T03:39:05.723185+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:39:05.723185+0000","last_undegraded":"2024-09-16T03:39:05.723185+0000","last_fullsized":"2024-09-16T03:39:05.723185+0000","mapping_epoch":74,"log_start":"129'257","ondisk_log_start":"129'257","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":266,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.902901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22624199,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":247,"num_read_kb":8660,"num_write":324,"num_write_kb":70574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.6","version":"108'44","reported_seq":382,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.552545+0000","last_change":"2024-09-16T03:38:48.279289+0000","last_active":"2024-09-16T03:39:08.552545+0000","last_peered":"2024-09-16T03:39:08.552545+0000","last_clean":"2024-09-16T03:39:08.552545+0000","last_became_active":"2024-09-16T03:38:45.924286+0000","last_became_peered":"2024-09-16T03:38:45.924286+0000","last_unstale":"2024-09-16T03:39:08.552545+0000","last_undegraded":"2024-09-16T03:39:08.552545+0000","last_fullsized":"2024-09-16T03:39:08.552545+0000","mapping_epoch":70,"log_start":"105'41","ondisk_log_start":"105'41","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:48.279224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.279224+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.718165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":264,"reported_epoch":135,"state":"active","last_fresh":"2024-09-16T03:39:09.649028+0000","last_change":"2024-09-16T03:39:09.649028+0000","last_active":"2024-09-16T03:39:09.649028+0000","last_peered":"2024-09-16T03:39:09.649028+0000","last_clean":"2024-09-16T03:39:08.025961+0000","last_became_active":"2024-09-16T03:39:09.649028+0000","last_became_peered":"2024-09-16T03:39:09.649028+0000","last_unstale":"2024-09-16T03:39:09.649028+0000","last_undegraded":"2024-09-16T03:39:09.649028+0000","last_fullsized":"2024-09-16T03:39:09.649028+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:24.358339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:24.358339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":["8","6","9","10"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"131'218","reported_seq":447,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.336591+0000","last_change":"2024-09-16T03:38:15.890094+0000","last_active":"2024-09-16T03:39:06.336591+0000","last_peered":"2024-09-16T03:39:06.336591+0000","last_clean":"2024-09-16T03:39:06.336591+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:39:06.336591+0000","last_undegraded":"2024-09-16T03:39:06.336591+0000","last_fullsized":"2024-09-16T03:39:06.336591+0000","mapping_epoch":31,"log_start":"130'215","ondisk_log_start":"130'215","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'125","last_scrub_stamp":"2024-09-16T03:38:15.889996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.889996+0000","objects_scrubbed":18,"log_size":3,"log_dups_size":226,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:35.350472+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15702938,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":193,"num_read_kb":2025,"num_write":275,"num_write_kb":58342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"127'76","reported_seq":420,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.752847+0000","last_change":"2024-09-16T03:39:01.297447+0000","last_active":"2024-09-16T03:39:08.752847+0000","last_peered":"2024-09-16T03:39:08.752847+0000","last_clean":"2024-09-16T03:39:08.752847+0000","last_became_active":"2024-09-16T03:38:45.723754+0000","last_became_peered":"2024-09-16T03:38:45.723754+0000","last_unstale":"2024-09-16T03:39:08.752847+0000","last_undegraded":"2024-09-16T03:39:08.752847+0000","last_fullsized":"2024-09-16T03:39:08.752847+0000","mapping_epoch":32,"log_start":"127'73","ondisk_log_start":"127'73","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'72","last_scrub_stamp":"2024-09-16T03:39:01.297324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.297324+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":71,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.355718+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7294304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":4176,"num_write":5,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":232,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.585505+0000","last_change":"2024-09-16T03:38:15.949627+0000","last_active":"2024-09-16T03:39:05.585505+0000","last_peered":"2024-09-16T03:39:05.585505+0000","last_clean":"2024-09-16T03:39:05.585505+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:39:05.585505+0000","last_undegraded":"2024-09-16T03:39:05.585505+0000","last_fullsized":"2024-09-16T03:39:05.585505+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:15.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:24.286324+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"129'252","reported_seq":620,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.630052+0000","last_change":"2024-09-16T03:38:59.856507+0000","last_active":"2024-09-16T03:39:03.630052+0000","last_peered":"2024-09-16T03:39:03.630052+0000","last_clean":"2024-09-16T03:39:03.630052+0000","last_became_active":"2024-09-16T03:38:59.854999+0000","last_became_peered":"2024-09-16T03:38:59.854999+0000","last_unstale":"2024-09-16T03:39:03.630052+0000","last_undegraded":"2024-09-16T03:39:03.630052+0000","last_fullsized":"2024-09-16T03:39:03.630052+0000","mapping_epoch":124,"log_start":"128'249","ondisk_log_start":"128'249","created":31,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":270,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17434102,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":231,"num_read_kb":12817,"num_write":311,"num_write_kb":78387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":27250603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"105'32","reported_seq":327,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.722978+0000","last_change":"2024-09-16T03:38:47.042020+0000","last_active":"2024-09-16T03:39:05.722978+0000","last_peered":"2024-09-16T03:39:05.722978+0000","last_clean":"2024-09-16T03:39:05.722978+0000","last_became_active":"2024-09-16T03:38:45.820169+0000","last_became_peered":"2024-09-16T03:38:45.820169+0000","last_unstale":"2024-09-16T03:39:05.722978+0000","last_undegraded":"2024-09-16T03:39:05.722978+0000","last_fullsized":"2024-09-16T03:39:05.722978+0000","mapping_epoch":32,"log_start":"83'29","ondisk_log_start":"83'29","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.041851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.041851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.311785+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":244,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.647284+0000","last_change":"2024-09-16T03:38:12.795978+0000","last_active":"2024-09-16T03:39:09.647284+0000","last_peered":"2024-09-16T03:39:09.647284+0000","last_clean":"2024-09-16T03:39:09.647284+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:39:09.647284+0000","last_undegraded":"2024-09-16T03:39:09.647284+0000","last_fullsized":"2024-09-16T03:39:09.647284+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:38:12.795855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.795855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:13.088740+0000","scrub_duration":501,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"131'274","reported_seq":506,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.028683+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:39:06.028683+0000","last_peered":"2024-09-16T03:39:06.028683+0000","last_clean":"2024-09-16T03:39:06.028683+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:39:06.028683+0000","last_undegraded":"2024-09-16T03:39:06.028683+0000","last_fullsized":"2024-09-16T03:39:06.028683+0000","mapping_epoch":31,"log_start":"130'271","ondisk_log_start":"130'271","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":283,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.679811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24909264,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":249,"num_read_kb":6268,"num_write":343,"num_write_kb":73467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":265,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.585584+0000","last_change":"2024-09-16T03:38:35.828277+0000","last_active":"2024-09-16T03:39:05.585584+0000","last_peered":"2024-09-16T03:39:05.585584+0000","last_clean":"2024-09-16T03:39:05.585584+0000","last_became_active":"2024-09-16T03:38:33.778361+0000","last_became_peered":"2024-09-16T03:38:33.778361+0000","last_unstale":"2024-09-16T03:39:05.585584+0000","last_undegraded":"2024-09-16T03:39:05.585584+0000","last_fullsized":"2024-09-16T03:39:05.585584+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:35.828203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.828203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:49.130136+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":237,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.542820+0000","last_change":"2024-09-16T03:37:52.652365+0000","last_active":"2024-09-16T03:39:06.542820+0000","last_peered":"2024-09-16T03:39:06.542820+0000","last_clean":"2024-09-16T03:39:06.542820+0000","last_became_active":"2024-09-16T03:37:52.651907+0000","last_became_peered":"2024-09-16T03:37:52.651907+0000","last_unstale":"2024-09-16T03:39:06.542820+0000","last_undegraded":"2024-09-16T03:39:06.542820+0000","last_fullsized":"2024-09-16T03:39:06.542820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:07.210820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"94'34","reported_seq":336,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.680919+0000","last_change":"2024-09-16T03:38:39.081942+0000","last_active":"2024-09-16T03:39:05.680919+0000","last_peered":"2024-09-16T03:39:05.680919+0000","last_clean":"2024-09-16T03:39:05.680919+0000","last_became_active":"2024-09-16T03:38:33.580022+0000","last_became_peered":"2024-09-16T03:38:33.580022+0000","last_unstale":"2024-09-16T03:39:05.680919+0000","last_undegraded":"2024-09-16T03:39:05.680919+0000","last_fullsized":"2024-09-16T03:39:05.680919+0000","mapping_epoch":96,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:39.081810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.081810+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:48.167935+0000","scrub_duration":473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":120,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622686+0000","last_change":"2024-09-16T03:38:51.363599+0000","last_active":"2024-09-16T03:39:05.622686+0000","last_peered":"2024-09-16T03:39:05.622686+0000","last_clean":"2024-09-16T03:39:05.622686+0000","last_became_active":"2024-09-16T03:38:51.363288+0000","last_became_peered":"2024-09-16T03:38:51.363288+0000","last_unstale":"2024-09-16T03:39:05.622686+0000","last_undegraded":"2024-09-16T03:39:05.622686+0000","last_fullsized":"2024-09-16T03:39:05.622686+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:52.581979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"129'130","reported_seq":568,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.719379+0000","last_change":"2024-09-16T03:38:39.704994+0000","last_active":"2024-09-16T03:39:08.719379+0000","last_peered":"2024-09-16T03:39:08.719379+0000","last_clean":"2024-09-16T03:39:08.719379+0000","last_became_active":"2024-09-16T03:38:33.581141+0000","last_became_peered":"2024-09-16T03:38:33.581141+0000","last_unstale":"2024-09-16T03:39:08.719379+0000","last_undegraded":"2024-09-16T03:39:08.719379+0000","last_fullsized":"2024-09-16T03:39:08.719379+0000","mapping_epoch":32,"log_start":"123'127","ondisk_log_start":"123'127","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'118","last_scrub_stamp":"2024-09-16T03:38:39.704919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.704919+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":124,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.098482+0000","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882549,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":77034,"num_write":14,"num_write_kb":4474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":248,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641493+0000","last_change":"2024-09-16T03:38:12.041048+0000","last_active":"2024-09-16T03:39:08.641493+0000","last_peered":"2024-09-16T03:39:08.641493+0000","last_clean":"2024-09-16T03:39:08.641493+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:39:08.641493+0000","last_undegraded":"2024-09-16T03:39:08.641493+0000","last_fullsized":"2024-09-16T03:39:08.641493+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:12.040917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.040917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.877949+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":248,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.629634+0000","last_change":"2024-09-16T03:38:34.928371+0000","last_active":"2024-09-16T03:39:03.629634+0000","last_peered":"2024-09-16T03:39:03.629634+0000","last_clean":"2024-09-16T03:39:03.629634+0000","last_became_active":"2024-09-16T03:38:33.484091+0000","last_became_peered":"2024-09-16T03:38:33.484091+0000","last_unstale":"2024-09-16T03:39:03.629634+0000","last_undegraded":"2024-09-16T03:39:03.629634+0000","last_fullsized":"2024-09-16T03:39:03.629634+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:34.928298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.928298+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:43.935101+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":243,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.546720+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:39:09.546720+0000","last_peered":"2024-09-16T03:39:09.546720+0000","last_clean":"2024-09-16T03:39:09.546720+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:39:09.546720+0000","last_undegraded":"2024-09-16T03:39:09.546720+0000","last_fullsized":"2024-09-16T03:39:09.546720+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.232558+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.10","version":"118'64","reported_seq":399,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.616521+0000","last_change":"2024-09-16T03:39:04.442655+0000","last_active":"2024-09-16T03:39:08.616521+0000","last_peered":"2024-09-16T03:39:08.616521+0000","last_clean":"2024-09-16T03:39:08.616521+0000","last_became_active":"2024-09-16T03:39:01.900921+0000","last_became_peered":"2024-09-16T03:39:01.900921+0000","last_unstale":"2024-09-16T03:39:08.616521+0000","last_undegraded":"2024-09-16T03:39:08.616521+0000","last_fullsized":"2024-09-16T03:39:08.616521+0000","mapping_epoch":72,"log_start":"113'61","ondisk_log_start":"113'61","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"118'64","last_scrub_stamp":"2024-09-16T03:39:04.442556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.442556+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.917777+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361433,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"114'17","reported_seq":296,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.722729+0000","last_change":"2024-09-16T03:39:03.929961+0000","last_active":"2024-09-16T03:39:05.722729+0000","last_peered":"2024-09-16T03:39:05.722729+0000","last_clean":"2024-09-16T03:39:05.722729+0000","last_became_active":"2024-09-16T03:39:01.696690+0000","last_became_peered":"2024-09-16T03:39:01.696690+0000","last_unstale":"2024-09-16T03:39:05.722729+0000","last_undegraded":"2024-09-16T03:39:05.722729+0000","last_fullsized":"2024-09-16T03:39:05.722729+0000","mapping_epoch":63,"log_start":"107'14","ondisk_log_start":"107'14","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:03.929863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.929863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:26.527714+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2991679,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"127'20","reported_seq":364,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.471860+0000","last_change":"2024-09-16T03:39:08.648919+0000","last_active":"2024-09-16T03:39:08.648919+0000","last_peered":"2024-09-16T03:39:08.471860+0000","last_clean":"2024-09-16T03:39:08.471860+0000","last_became_active":"2024-09-16T03:39:03.822624+0000","last_became_peered":"2024-09-16T03:39:03.822624+0000","last_unstale":"2024-09-16T03:39:08.471860+0000","last_undegraded":"2024-09-16T03:39:08.471860+0000","last_fullsized":"2024-09-16T03:39:08.471860+0000","mapping_epoch":134,"log_start":"106'17","ondisk_log_start":"106'17","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966452,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":26744,"num_write":10,"num_write_kb":2629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2569696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,9,11],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.13","version":"127'73","reported_seq":448,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.629829+0000","last_change":"2024-09-16T03:39:03.124648+0000","last_active":"2024-09-16T03:39:03.629829+0000","last_peered":"2024-09-16T03:39:03.629829+0000","last_clean":"2024-09-16T03:39:03.629829+0000","last_became_active":"2024-09-16T03:39:01.843090+0000","last_became_peered":"2024-09-16T03:39:01.843090+0000","last_unstale":"2024-09-16T03:39:03.629829+0000","last_undegraded":"2024-09-16T03:39:03.629829+0000","last_fullsized":"2024-09-16T03:39:03.629829+0000","mapping_epoch":119,"log_start":"108'70","ondisk_log_start":"108'70","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'73","last_scrub_stamp":"2024-09-16T03:39:03.124583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.124583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":68,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:24.286892+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,14],"acting":[4,3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"104'20","reported_seq":316,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622809+0000","last_change":"2024-09-16T03:39:01.898409+0000","last_active":"2024-09-16T03:39:05.622809+0000","last_peered":"2024-09-16T03:39:05.622809+0000","last_clean":"2024-09-16T03:39:05.622809+0000","last_became_active":"2024-09-16T03:39:01.596287+0000","last_became_peered":"2024-09-16T03:39:01.596287+0000","last_unstale":"2024-09-16T03:39:05.622809+0000","last_undegraded":"2024-09-16T03:39:05.622809+0000","last_fullsized":"2024-09-16T03:39:05.622809+0000","mapping_epoch":114,"log_start":"98'17","ondisk_log_start":"98'17","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:01.898331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.898331+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.472424+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"123'65","reported_seq":450,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.110040+0000","last_change":"2024-09-16T03:39:08.541769+0000","last_active":"2024-09-16T03:39:08.541769+0000","last_peered":"2024-09-16T03:39:08.110040+0000","last_clean":"2024-09-16T03:39:08.110040+0000","last_became_active":"2024-09-16T03:39:03.915707+0000","last_became_peered":"2024-09-16T03:39:03.915707+0000","last_unstale":"2024-09-16T03:39:08.110040+0000","last_undegraded":"2024-09-16T03:39:08.110040+0000","last_fullsized":"2024-09-16T03:39:08.110040+0000","mapping_epoch":134,"log_start":"123'62","ondisk_log_start":"123'62","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916014,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":83986,"num_write":19,"num_write_kb":5459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":826342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,9,10],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"124'86","reported_seq":446,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.782232+0000","last_change":"2024-09-16T03:39:03.073530+0000","last_active":"2024-09-16T03:39:05.782232+0000","last_peered":"2024-09-16T03:39:05.782232+0000","last_clean":"2024-09-16T03:39:05.782232+0000","last_became_active":"2024-09-16T03:39:01.797805+0000","last_became_peered":"2024-09-16T03:39:01.797805+0000","last_unstale":"2024-09-16T03:39:05.782232+0000","last_undegraded":"2024-09-16T03:39:05.782232+0000","last_fullsized":"2024-09-16T03:39:05.782232+0000","mapping_epoch":115,"log_start":"124'82","ondisk_log_start":"124'82","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.073459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.073459+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":80,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:32.204501+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.17","version":"131'52","reported_seq":402,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.741683+0000","last_change":"2024-09-16T03:39:05.721286+0000","last_active":"2024-09-16T03:39:08.741683+0000","last_peered":"2024-09-16T03:39:08.741683+0000","last_clean":"2024-09-16T03:39:08.741683+0000","last_became_active":"2024-09-16T03:39:03.717420+0000","last_became_peered":"2024-09-16T03:39:03.717420+0000","last_unstale":"2024-09-16T03:39:08.741683+0000","last_undegraded":"2024-09-16T03:39:08.741683+0000","last_fullsized":"2024-09-16T03:39:08.741683+0000","mapping_epoch":128,"log_start":"117'49","ondisk_log_start":"117'49","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":48,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.790793+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1546293,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":52,"num_read_kb":44516,"num_write":11,"num_write_kb":3134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2516795,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"128'51","reported_seq":375,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.546548+0000","last_change":"2024-09-16T03:39:02.668794+0000","last_active":"2024-09-16T03:39:09.546548+0000","last_peered":"2024-09-16T03:39:09.546548+0000","last_clean":"2024-09-16T03:39:09.546548+0000","last_became_active":"2024-09-16T03:38:47.562641+0000","last_became_peered":"2024-09-16T03:38:47.562641+0000","last_unstale":"2024-09-16T03:39:09.546548+0000","last_undegraded":"2024-09-16T03:39:09.546548+0000","last_fullsized":"2024-09-16T03:39:09.546548+0000","mapping_epoch":112,"log_start":"124'48","ondisk_log_start":"124'48","created":50,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'48","last_scrub_stamp":"2024-09-16T03:39:02.668681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.668681+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":45,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.451476+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067327,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":21991,"num_write":23,"num_write_kb":6049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":51,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.622532+0000","last_change":"2024-09-16T03:39:01.106936+0000","last_active":"2024-09-16T03:39:05.622532+0000","last_peered":"2024-09-16T03:39:05.622532+0000","last_clean":"2024-09-16T03:39:05.622532+0000","last_became_active":"2024-09-16T03:38:51.558378+0000","last_became_peered":"2024-09-16T03:38:51.558378+0000","last_unstale":"2024-09-16T03:39:05.622532+0000","last_undegraded":"2024-09-16T03:39:05.622532+0000","last_fullsized":"2024-09-16T03:39:05.622532+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:01.106840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.106840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.164091+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,6],"acting":[2,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1a","version":"128'125","reported_seq":583,"reported_epoch":134,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:08.641112+0000","last_change":"2024-09-16T03:38:47.159152+0000","last_active":"2024-09-16T03:39:08.641112+0000","last_peered":"2024-09-16T03:39:08.641112+0000","last_clean":"2024-09-16T03:38:43.521485+0000","last_became_active":"2024-09-16T03:38:46.957471+0000","last_became_peered":"2024-09-16T03:38:46.957471+0000","last_unstale":"2024-09-16T03:39:08.641112+0000","last_undegraded":"2024-09-16T03:39:08.641112+0000","last_fullsized":"2024-09-16T03:39:08.641112+0000","mapping_epoch":111,"log_start":"117'122","ondisk_log_start":"117'122","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'115","last_scrub_stamp":"2024-09-16T03:38:35.254488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.254488+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":122,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6739673,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":31161,"num_write":9,"num_write_kb":2915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6,11],"acting":[3,8,10],"avail_no_missing":["3","8","10"],"object_location_counts":[{"shards":"3,8,10","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"131'127","reported_seq":584,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.742125+0000","last_change":"2024-09-16T03:38:50.465770+0000","last_active":"2024-09-16T03:39:08.742125+0000","last_peered":"2024-09-16T03:39:08.742125+0000","last_clean":"2024-09-16T03:39:08.742125+0000","last_became_active":"2024-09-16T03:38:50.465463+0000","last_became_peered":"2024-09-16T03:38:50.465463+0000","last_unstale":"2024-09-16T03:39:08.742125+0000","last_undegraded":"2024-09-16T03:39:08.742125+0000","last_fullsized":"2024-09-16T03:39:08.742125+0000","mapping_epoch":115,"log_start":"119'124","ondisk_log_start":"119'124","created":100,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'107","last_scrub_stamp":"2024-09-16T03:38:39.905697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.905697+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":118,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.828899+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5697672,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":62,"num_read_kb":76423,"num_write":24,"num_write_kb":7195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2376197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"99'56","reported_seq":356,"reported_epoch":132,"state":"active+clean","last_fresh":"2024-09-16T03:39:06.542809+0000","last_change":"2024-09-16T03:39:00.988586+0000","last_active":"2024-09-16T03:39:06.542809+0000","last_peered":"2024-09-16T03:39:06.542809+0000","last_clean":"2024-09-16T03:39:06.542809+0000","last_became_active":"2024-09-16T03:38:53.479253+0000","last_became_peered":"2024-09-16T03:38:53.479253+0000","last_unstale":"2024-09-16T03:39:06.542809+0000","last_undegraded":"2024-09-16T03:39:06.542809+0000","last_fullsized":"2024-09-16T03:39:06.542809+0000","mapping_epoch":118,"log_start":"90'53","ondisk_log_start":"90'53","created":100,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":5,"last_scrub":"99'56","last_scrub_stamp":"2024-09-16T03:39:00.988517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.988517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.840710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"92'14","reported_seq":279,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.642046+0000","last_change":"2024-09-16T03:38:49.092747+0000","last_active":"2024-09-16T03:39:08.642046+0000","last_peered":"2024-09-16T03:39:08.642046+0000","last_clean":"2024-09-16T03:39:08.642046+0000","last_became_active":"2024-09-16T03:38:48.661213+0000","last_became_peered":"2024-09-16T03:38:48.661213+0000","last_unstale":"2024-09-16T03:39:08.642046+0000","last_undegraded":"2024-09-16T03:39:08.642046+0000","last_fullsized":"2024-09-16T03:39:08.642046+0000","mapping_epoch":113,"log_start":"91'11","ondisk_log_start":"91'11","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:49.092687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.092687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.372348+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,12],"acting":[3,11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"92'59","reported_seq":388,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641478+0000","last_change":"2024-09-16T03:38:48.561603+0000","last_active":"2024-09-16T03:39:08.641478+0000","last_peered":"2024-09-16T03:39:08.641478+0000","last_clean":"2024-09-16T03:39:08.641478+0000","last_became_active":"2024-09-16T03:38:48.561170+0000","last_became_peered":"2024-09-16T03:38:48.561170+0000","last_unstale":"2024-09-16T03:39:08.641478+0000","last_undegraded":"2024-09-16T03:39:08.641478+0000","last_fullsized":"2024-09-16T03:39:08.641478+0000","mapping_epoch":113,"log_start":"89'56","ondisk_log_start":"89'56","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'59","last_scrub_stamp":"2024-09-16T03:38:35.977646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.977646+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:38.632635+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,6],"acting":[14,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1f","version":"131'112","reported_seq":496,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.541483+0000","last_change":"2024-09-16T03:38:48.762617+0000","last_active":"2024-09-16T03:39:08.541483+0000","last_peered":"2024-09-16T03:39:08.541483+0000","last_clean":"2024-09-16T03:39:08.541483+0000","last_became_active":"2024-09-16T03:38:48.761213+0000","last_became_peered":"2024-09-16T03:38:48.761213+0000","last_unstale":"2024-09-16T03:39:08.541483+0000","last_undegraded":"2024-09-16T03:39:08.541483+0000","last_fullsized":"2024-09-16T03:39:08.541483+0000","mapping_epoch":113,"log_start":"118'109","ondisk_log_start":"118'109","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'95","last_scrub_stamp":"2024-09-16T03:38:37.201959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:37.201959+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.307408+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9748159,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":37374,"num_write":27,"num_write_kb":6985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1704821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,12],"acting":[3,10,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"127'143","reported_seq":589,"reported_epoch":129,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:03.629583+0000","last_change":"2024-09-16T03:38:46.862171+0000","last_active":"2024-09-16T03:39:03.629583+0000","last_peered":"2024-09-16T03:39:03.629583+0000","last_clean":"2024-09-16T03:38:44.034266+0000","last_became_active":"2024-09-16T03:38:46.560752+0000","last_became_peered":"2024-09-16T03:38:46.560752+0000","last_unstale":"2024-09-16T03:39:03.629583+0000","last_undegraded":"2024-09-16T03:39:03.629583+0000","last_fullsized":"2024-09-16T03:39:03.629583+0000","mapping_epoch":111,"log_start":"127'140","ondisk_log_start":"127'140","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'118","last_scrub_stamp":"2024-09-16T03:38:36.698150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:36.698150+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":140,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":93,"num_read_kb":75680,"num_write":30,"num_write_kb":9656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[4,10,15],"avail_no_missing":["4","10","15"],"object_location_counts":[{"shards":"4,10,15","objects":3}],"blocked_by":[],"up_primary":8,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.21","version":"48'7","reported_seq":141,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641376+0000","last_change":"2024-09-16T03:38:49.064947+0000","last_active":"2024-09-16T03:39:08.641376+0000","last_peered":"2024-09-16T03:39:08.641376+0000","last_clean":"2024-09-16T03:39:08.641376+0000","last_became_active":"2024-09-16T03:38:48.559431+0000","last_became_peered":"2024-09-16T03:38:48.559431+0000","last_unstale":"2024-09-16T03:39:08.641376+0000","last_undegraded":"2024-09-16T03:39:08.641376+0000","last_fullsized":"2024-09-16T03:39:08.641376+0000","mapping_epoch":113,"log_start":"40'4","ondisk_log_start":"40'4","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":6,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:49.064870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064870+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.310170+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,10],"acting":[14,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.22","version":"45'6","reported_seq":121,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.629609+0000","last_change":"2024-09-16T03:39:00.968576+0000","last_active":"2024-09-16T03:39:03.629609+0000","last_peered":"2024-09-16T03:39:03.629609+0000","last_clean":"2024-09-16T03:39:03.629609+0000","last_became_active":"2024-09-16T03:38:54.462316+0000","last_became_peered":"2024-09-16T03:38:54.462316+0000","last_unstale":"2024-09-16T03:39:03.629609+0000","last_undegraded":"2024-09-16T03:39:03.629609+0000","last_fullsized":"2024-09-16T03:39:03.629609+0000","mapping_epoch":119,"log_start":"39'3","ondisk_log_start":"39'3","created":100,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:39:00.968448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.968448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:06.219174+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,12],"acting":[4,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"108'38","reported_seq":355,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.741629+0000","last_change":"2024-09-16T03:38:49.464614+0000","last_active":"2024-09-16T03:39:08.741629+0000","last_peered":"2024-09-16T03:39:08.741629+0000","last_clean":"2024-09-16T03:39:08.741629+0000","last_became_active":"2024-09-16T03:38:49.463797+0000","last_became_peered":"2024-09-16T03:38:49.463797+0000","last_unstale":"2024-09-16T03:39:08.741629+0000","last_undegraded":"2024-09-16T03:39:08.741629+0000","last_fullsized":"2024-09-16T03:39:08.741629+0000","mapping_epoch":114,"log_start":"105'36","ondisk_log_start":"105'36","created":100,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:34.893524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.893524+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":35,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:52.829073+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,2],"acting":[14,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.24","version":"105'32","reported_seq":325,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.823050+0000","last_change":"2024-09-16T03:38:47.978956+0000","last_active":"2024-09-16T03:39:05.823050+0000","last_peered":"2024-09-16T03:39:05.823050+0000","last_clean":"2024-09-16T03:39:05.823050+0000","last_became_active":"2024-09-16T03:38:45.820201+0000","last_became_peered":"2024-09-16T03:38:45.820201+0000","last_unstale":"2024-09-16T03:39:05.823050+0000","last_undegraded":"2024-09-16T03:39:05.823050+0000","last_fullsized":"2024-09-16T03:39:05.823050+0000","mapping_epoch":110,"log_start":"83'29","ondisk_log_start":"83'29","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.978888+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:56.881865+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.25","version":"130'85","reported_seq":414,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-16T03:39:05.679497+0000","last_change":"2024-09-16T03:38:48.088833+0000","last_active":"2024-09-16T03:39:05.679497+0000","last_peered":"2024-09-16T03:39:05.679497+0000","last_clean":"2024-09-16T03:39:05.679497+0000","last_became_active":"2024-09-16T03:38:45.724757+0000","last_became_peered":"2024-09-16T03:38:45.724757+0000","last_unstale":"2024-09-16T03:39:05.679497+0000","last_undegraded":"2024-09-16T03:39:05.679497+0000","last_fullsized":"2024-09-16T03:39:05.679497+0000","mapping_epoch":110,"log_start":"116'82","ondisk_log_start":"116'82","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'77","last_scrub_stamp":"2024-09-16T03:38:48.088768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.088768+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":79,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:53.112635+0000","scrub_duration":388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2134555,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":10,"num_write_kb":2500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.26","version":"133'54","reported_seq":402,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.552465+0000","last_change":"2024-09-16T03:38:49.477393+0000","last_active":"2024-09-16T03:39:08.552465+0000","last_peered":"2024-09-16T03:39:08.552465+0000","last_clean":"2024-09-16T03:39:08.552465+0000","last_became_active":"2024-09-16T03:38:45.926029+0000","last_became_peered":"2024-09-16T03:38:45.926029+0000","last_unstale":"2024-09-16T03:39:08.552465+0000","last_undegraded":"2024-09-16T03:39:08.552465+0000","last_fullsized":"2024-09-16T03:39:08.552465+0000","mapping_epoch":110,"log_start":"123'51","ondisk_log_start":"123'51","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:49.477297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.477297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":50,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.242430+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":24203,"num_write":11,"num_write_kb":3140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.27","version":"109'66","reported_seq":389,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.552485+0000","last_change":"2024-09-16T03:38:47.511343+0000","last_active":"2024-09-16T03:39:08.552485+0000","last_peered":"2024-09-16T03:39:08.552485+0000","last_clean":"2024-09-16T03:39:08.552485+0000","last_became_active":"2024-09-16T03:38:45.824884+0000","last_became_peered":"2024-09-16T03:38:45.824884+0000","last_unstale":"2024-09-16T03:39:08.552485+0000","last_undegraded":"2024-09-16T03:39:08.552485+0000","last_fullsized":"2024-09-16T03:39:08.552485+0000","mapping_epoch":110,"log_start":"105'63","ondisk_log_start":"105'63","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'66","last_scrub_stamp":"2024-09-16T03:38:47.511252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.511252+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.884916+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.28","version":"129'68","reported_seq":454,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.372306+0000","last_change":"2024-09-16T03:39:08.541570+0000","last_active":"2024-09-16T03:39:08.541570+0000","last_peered":"2024-09-16T03:39:08.372306+0000","last_clean":"2024-09-16T03:39:08.372306+0000","last_became_active":"2024-09-16T03:38:55.727094+0000","last_became_peered":"2024-09-16T03:38:55.727094+0000","last_unstale":"2024-09-16T03:39:08.372306+0000","last_undegraded":"2024-09-16T03:39:08.372306+0000","last_fullsized":"2024-09-16T03:39:08.372306+0000","mapping_epoch":134,"log_start":"118'65","ondisk_log_start":"118'65","created":110,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:48.551929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.551929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":64,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395113,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":31731,"num_write":5,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3610944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,9,10],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":251,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641395+0000","last_change":"2024-09-16T03:39:00.834877+0000","last_active":"2024-09-16T03:39:08.641395+0000","last_peered":"2024-09-16T03:39:08.641395+0000","last_clean":"2024-09-16T03:39:08.641395+0000","last_became_active":"2024-09-16T03:38:45.627369+0000","last_became_peered":"2024-09-16T03:38:45.627369+0000","last_unstale":"2024-09-16T03:39:08.641395+0000","last_undegraded":"2024-09-16T03:39:08.641395+0000","last_fullsized":"2024-09-16T03:39:08.641395+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:00.834742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.834742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.318226+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.2a","version":"108'122","reported_seq":606,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.546800+0000","last_change":"2024-09-16T03:38:47.964525+0000","last_active":"2024-09-16T03:39:09.546800+0000","last_peered":"2024-09-16T03:39:09.546800+0000","last_clean":"2024-09-16T03:39:09.546800+0000","last_became_active":"2024-09-16T03:38:46.022108+0000","last_became_peered":"2024-09-16T03:38:46.022108+0000","last_unstale":"2024-09-16T03:39:09.546800+0000","last_undegraded":"2024-09-16T03:39:09.546800+0000","last_fullsized":"2024-09-16T03:39:09.546800+0000","mapping_epoch":110,"log_start":"107'115","ondisk_log_start":"107'115","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'122","last_scrub_stamp":"2024-09-16T03:38:47.964439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.964439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":112,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.917123+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.2b","version":"127'124","reported_seq":571,"reported_epoch":131,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:05.585588+0000","last_change":"2024-09-16T03:38:53.063728+0000","last_active":"2024-09-16T03:39:05.585588+0000","last_peered":"2024-09-16T03:39:05.585588+0000","last_clean":"2024-09-16T03:38:49.362783+0000","last_became_active":"2024-09-16T03:38:52.761638+0000","last_became_peered":"2024-09-16T03:38:52.761638+0000","last_unstale":"2024-09-16T03:39:05.585588+0000","last_undegraded":"2024-09-16T03:39:05.585588+0000","last_fullsized":"2024-09-16T03:39:05.585588+0000","mapping_epoch":117,"log_start":"119'121","ondisk_log_start":"119'121","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:49.064061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064061+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":121,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,12],"acting":[15,4,9],"avail_no_missing":["15","4","9","12"],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2c","version":"121'63","reported_seq":408,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:07.529785+0000","last_change":"2024-09-16T03:39:08.548352+0000","last_active":"2024-09-16T03:39:08.548352+0000","last_peered":"2024-09-16T03:39:07.529785+0000","last_clean":"2024-09-16T03:39:07.529785+0000","last_became_active":"2024-09-16T03:38:55.562810+0000","last_became_peered":"2024-09-16T03:38:55.562810+0000","last_unstale":"2024-09-16T03:39:07.529785+0000","last_undegraded":"2024-09-16T03:39:07.529785+0000","last_fullsized":"2024-09-16T03:39:07.529785+0000","mapping_epoch":134,"log_start":"118'60","ondisk_log_start":"118'60","created":110,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'58","last_scrub_stamp":"2024-09-16T03:38:47.889276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.889276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":7651,"num_write":5,"num_write_kb":1234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1262403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[9,11,15],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.2d","version":"92'14","reported_seq":332,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.548592+0000","last_change":"2024-09-16T03:38:47.115739+0000","last_active":"2024-09-16T03:39:08.548592+0000","last_peered":"2024-09-16T03:39:08.548592+0000","last_clean":"2024-09-16T03:39:08.548592+0000","last_became_active":"2024-09-16T03:38:45.723438+0000","last_became_peered":"2024-09-16T03:38:45.723438+0000","last_unstale":"2024-09-16T03:39:08.548592+0000","last_undegraded":"2024-09-16T03:39:08.548592+0000","last_fullsized":"2024-09-16T03:39:08.548592+0000","mapping_epoch":110,"log_start":"91'11","ondisk_log_start":"91'11","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:47.115651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.115651+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:00.107802+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":56,"stat_sum":{"num_bytes":128634223,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":930,"num_read_kb":863271,"num_write":332,"num_write_kb":91950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":21395486,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":240472064,"data_stored":240061123,"data_compressed":572208,"data_compressed_allocated":626688,"data_compressed_original":1503232,"omap_allocated":0,"internal_metadata":0},"log_size":171,"ondisk_log_size":171,"up":168,"acting":168,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":80670503,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":920,"num_read_kb":29770,"num_write":1253,"num_write_kb":280770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":39649106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":185962496,"data_stored":198955004,"data_compressed":11754991,"data_compressed_allocated":12513280,"data_compressed_original":25931776,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":2644640,"data_compressed":159066,"data_compressed_allocated":204800,"data_compressed_original":2584576,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247837,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83256,"kb_used_data":43372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94288584,"statfs":{"total":96636764160,"available":96551510016,"internally_reserved":0,"allocated":44412928,"data_stored":45041141,"data_compressed":1329713,"data_compressed_allocated":1417216,"data_compressed_original":2596864,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247838,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60108,"kb_used_data":25512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94311732,"statfs":{"total":96636764160,"available":96575213568,"internally_reserved":0,"allocated":26124288,"data_stored":27258963,"data_compressed":896583,"data_compressed_allocated":978944,"data_compressed_original":2617344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247837,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73204,"kb_used_data":38960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94298636,"statfs":{"total":96636764160,"available":96561803264,"internally_reserved":0,"allocated":39895040,"data_stored":40905705,"data_compressed":1172110,"data_compressed_allocated":1261568,"data_compressed_original":2797568,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280543,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59508,"kb_used_data":23688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94312332,"statfs":{"total":96636764160,"available":96575827968,"internally_reserved":0,"allocated":24256512,"data_stored":25354703,"data_compressed":886305,"data_compressed_allocated":966656,"data_compressed_original":2551808,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280543,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280542,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313247,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34820,"kb_used_data":7788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337020,"statfs":{"total":96636764160,"available":96601108480,"internally_reserved":0,"allocated":7974912,"data_stored":7570474,"data_compressed":25616,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280543,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88796,"kb_used_data":53444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94283044,"statfs":{"total":96636764160,"available":96545837056,"internally_reserved":0,"allocated":54726656,"data_stored":55922960,"data_compressed":1410985,"data_compressed_allocated":1519616,"data_compressed_original":3321856,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280541,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100932,"kb_used_data":65448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94270908,"statfs":{"total":96636764160,"available":96533409792,"internally_reserved":0,"allocated":67018752,"data_stored":68966584,"data_compressed":1952237,"data_compressed_allocated":2097152,"data_compressed_original":4653056,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313247,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280542,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35892,"kb_used_data":4704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335948,"statfs":{"total":96636764160,"available":96600010752,"internally_reserved":0,"allocated":4816896,"data_stored":4453365,"data_compressed":25616,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280542,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69660,"kb_used_data":38468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302180,"statfs":{"total":96636764160,"available":96565432320,"internally_reserved":0,"allocated":39391232,"data_stored":40017276,"data_compressed":1267990,"data_compressed_allocated":1351680,"data_compressed_original":2510848,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280541,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38652,"kb_used_data":5076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333188,"statfs":{"total":96636764160,"available":96597184512,"internally_reserved":0,"allocated":5197824,"data_stored":5332821,"data_compressed":69039,"data_compressed_allocated":90112,"data_compressed_original":618496,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,3,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280543,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83532,"kb_used_data":52344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94288308,"statfs":{"total":96636764160,"available":96551227392,"internally_reserved":0,"allocated":53600256,"data_stored":53264385,"data_compressed":186317,"data_compressed_allocated":212992,"data_compressed_original":532480,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,3,4,5,6,7,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313246,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43140,"kb_used_data":11944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328700,"statfs":{"total":96636764160,"available":96592588800,"internally_reserved":0,"allocated":12230656,"data_stored":11891032,"data_compressed":109264,"data_compressed_allocated":126976,"data_compressed_original":270336,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247838,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57696,"kb_used_data":25268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94314144,"statfs":{"total":96636764160,"available":96577683456,"internally_reserved":0,"allocated":25874432,"data_stored":26571892,"data_compressed":1270271,"data_compressed_allocated":1355776,"data_compressed_original":2510848,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14012416,"data_stored":15061447,"data_compressed":829441,"data_compressed_allocated":888832,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13422592,"data_stored":14113910,"data_compressed":606276,"data_compressed_allocated":643072,"data_compressed_original":1376256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14032896,"data_stored":15086444,"data_compressed":1224396,"data_compressed_allocated":1294336,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22970368,"data_stored":24522128,"data_compressed":1292173,"data_compressed_allocated":1380352,"data_compressed_original":2985984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":34295808,"data_stored":36685143,"data_compressed":1896867,"data_compressed_allocated":2027520,"data_compressed_original":4493312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21327872,"data_stored":22801296,"data_compressed":1184415,"data_compressed_allocated":1265664,"data_compressed_original":2789376,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":13844480,"data_stored":14888812,"data_compressed":1232826,"data_compressed_allocated":1302528,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":12505088,"data_stored":13249430,"data_compressed":646795,"data_compressed_allocated":684032,"data_compressed_original":1470464,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14012416,"data_stored":15061447,"data_compressed":829441,"data_compressed_allocated":888832,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14032896,"data_stored":15086444,"data_compressed":1224396,"data_compressed_allocated":1294336,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8982528,"data_stored":8928361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10244096,"data_stored":10207520,"data_compressed":49947,"data_compressed_allocated":53248,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30552064,"data_stored":30549819,"data_compressed":93196,"data_compressed_allocated":102400,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30445568,"data_stored":30338139,"data_compressed":29754,"data_compressed_allocated":32768,"data_compressed_original":86016,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16941056,"data_stored":16931882,"data_compressed":44673,"data_compressed_allocated":49152,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3641344,"data_stored":3612626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24342528,"data_stored":24277619,"data_compressed":9548,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4005888,"data_stored":3995623,"data_compressed":10762,"data_compressed_allocated":12288,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":52391936,"data_stored":52412861,"data_compressed":160701,"data_compressed_allocated":176128,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":10862592,"data_stored":10841283,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":11001856,"data_stored":11037652,"data_compressed":83648,"data_compressed_allocated":90112,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":6758400,"data_stored":6718533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":29212672,"data_stored":29129165,"data_compressed":79701,"data_compressed_allocated":86016,"data_compressed_original":151552,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:39:10.063 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:10.062+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:10.092 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:57 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'1, mlcod=0'0)", 2024-09-16T03:39:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:39:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.965060417, 2024-09-16T03:39:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20372090900000001, 2024-09-16T03:39:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:39:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125313+0000", 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7180000000000001e-06 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126730+0000", 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00141731 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126746+0000", 2024-09-16T03:39:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6455000000000001e-05 2024-09-16T03:39:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126756+0000", 2024-09-16T03:39:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:39:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227068+0000", 2024-09-16T03:39:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100312308 2024-09-16T03:39:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227085+0000", 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6898e-05 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328995+0000", 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101909903 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329031+0000", 2024-09-16T03:39:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5717999999999999e-05 2024-09-16T03:39:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:58 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:39:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.961727287, 2024-09-16T03:39:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084524500000001, 2024-09-16T03:39:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:39:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:39:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128644+0000", 2024-09-16T03:39:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.251e-06 2024-09-16T03:39:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128913+0000", 2024-09-16T03:39:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026856200000000002 2024-09-16T03:39:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128925+0000", 2024-09-16T03:39:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2109999999999999e-05 2024-09-16T03:39:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128933+0000", 2024-09-16T03:39:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6260000000000003e-06 2024-09-16T03:39:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328196+0000", 2024-09-16T03:39:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19926303200000001 2024-09-16T03:39:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328253+0000", 2024-09-16T03:39:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7262999999999997e-05 2024-09-16T03:39:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329458+0000", 2024-09-16T03:39:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012051939999999999 2024-09-16T03:39:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329488+0000", 2024-09-16T03:39:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0207000000000001e-05 2024-09-16T03:39:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:59 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'3, mlcod=0'0)", 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 117.961231854, 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036954900000001, 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129139+0000", 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6099999999999999e-07 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129158+0000", 2024-09-16T03:39:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088e-05 2024-09-16T03:39:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129168+0000", 2024-09-16T03:39:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.048e-05 2024-09-16T03:39:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.229233+0000", 2024-09-16T03:39:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100065168 2024-09-16T03:39:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328815+0000", 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099581694999999998 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328845+0000", 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0193e-05 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329492+0000", 2024-09-16T03:39:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064671800000000001 2024-09-16T03:39:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329508+0000", 2024-09-16T03:39:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6246e-05 2024-09-16T03:39:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.0:15 4.d e47/32 4:b3fe05b2:::smithi15736696-15:head v 47'5, mlcod=39'3)", 2024-09-16T03:39:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:39:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 103.787937128, 2024-09-16T03:39:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100920751, 2024-09-16T03:39:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:39:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:39:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302435+0000", 2024-09-16T03:39:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2340000000000001e-06 2024-09-16T03:39:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302446+0000", 2024-09-16T03:39:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0596e-05 2024-09-16T03:39:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302461+0000", 2024-09-16T03:39:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5082000000000001e-05 2024-09-16T03:39:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402592+0000", 2024-09-16T03:39:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100130488 2024-09-16T03:39:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402712+0000", 2024-09-16T03:39:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012013100000000001 2024-09-16T03:39:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402741+0000", 2024-09-16T03:39:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9717999999999999e-05 2024-09-16T03:39:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403309+0000", 2024-09-16T03:39:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056725700000000005 2024-09-16T03:39:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403354+0000", 2024-09-16T03:39:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5244999999999998e-05 2024-09-16T03:39:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:39:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:39:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 79.940438419000003, 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032593800000000001, 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:39:10.112 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149943+0000", 2024-09-16T03:39:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0817e-05 2024-09-16T03:39:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151026+0000", 2024-09-16T03:39:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010835350000000001 2024-09-16T03:39:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151040+0000", 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3884e-05 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151049+0000", 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151107+0000", 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8199000000000003e-05 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151124+0000", 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7756000000000001e-05 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153142+0000", 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020177569999999998 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153191+0000", 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9051999999999997e-05 2024-09-16T03:39:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1682 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'7, mlcod=40'4)", 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 79.836841229000001, 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10169502499999999, 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253540+0000", 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0888e-05 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254015+0000", 2024-09-16T03:39:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047485499999999999 2024-09-16T03:39:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:10.119 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254030+0000", 2024-09-16T03:39:10.119 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4899000000000001e-05 2024-09-16T03:39:10.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254039+0000", 2024-09-16T03:39:10.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5000000000000005e-06 2024-09-16T03:39:10.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.120 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:10.120 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254107+0000", 2024-09-16T03:39:10.120 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7298000000000005e-05 2024-09-16T03:39:10.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254122+0000", 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5928999999999999e-05 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355184+0000", 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101061703 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355224+0000", 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9953000000000002e-05 2024-09-16T03:39:10.121 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.122 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:10.122 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.122 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1683 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'8, mlcod=40'4)", 2024-09-16T03:39:10.122 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:39:10.122 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 79.836303681000004, 2024-09-16T03:39:10.122 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101630885, 2024-09-16T03:39:10.122 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:10.122 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:10.122 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.122 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:10.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:10.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:39:10.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:10.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:39:10.123 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.123 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:10.123 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254068+0000", 2024-09-16T03:39:10.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5700000000000001e-07 2024-09-16T03:39:10.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:10.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254075+0000", 2024-09-16T03:39:10.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2710000000000003e-06 2024-09-16T03:39:10.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.124 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:10.124 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254083+0000", 2024-09-16T03:39:10.124 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0220000000000004e-06 2024-09-16T03:39:10.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254089+0000", 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8699999999999997e-06 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354968+0000", 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100879625 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354992+0000", 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-16T03:39:10.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:10.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355665+0000", 2024-09-16T03:39:10.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00067371100000000001 2024-09-16T03:39:10.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.126 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:10.126 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355698+0000", 2024-09-16T03:39:10.126 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2291e-05 2024-09-16T03:39:10.126 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.126 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:10.126 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.0:177 4.10 e65/57 4:0a55bbf2:::smithi15736696-18:head v 65'47, mlcod=0'0)", 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 78.883357093000001, 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074461099999999995, 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:10.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:10.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:10.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:10.128 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5300000000000001e-07 2024-09-16T03:39:10.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.128 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:10.128 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207018+0000", 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7210000000000002e-06 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207031+0000", 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2767e-05 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207040+0000", 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5400000000000001e-06 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.129 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207107+0000", 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6625999999999995e-05 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207131+0000", 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4128e-05 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207712+0000", 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00058124700000000001 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.131 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:10.131 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207758+0000", 2024-09-16T03:39:10.131 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5429000000000002e-05 2024-09-16T03:39:10.131 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.131 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:10.131 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.131 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1723 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'48, mlcod=53'46)", 2024-09-16T03:39:10.131 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:39:10.131 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 78.441289912000002, 2024-09-16T03:39:10.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00088284599999999996, 2024-09-16T03:39:10.132 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:10.132 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:10.132 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.132 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:10.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.132 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:10.132 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:39:10.132 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:10.133 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:39:10.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:10.133 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649081+0000", 2024-09-16T03:39:10.133 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8299999999999995e-07 2024-09-16T03:39:10.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.133 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:10.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649101+0000", 2024-09-16T03:39:10.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0126e-05 2024-09-16T03:39:10.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:10.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649141+0000", 2024-09-16T03:39:10.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9609e-05 2024-09-16T03:39:10.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.134 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.134 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649161+0000", 2024-09-16T03:39:10.134 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9599999999999999e-05 2024-09-16T03:39:10.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:10.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649299+0000", 2024-09-16T03:39:10.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013822199999999999 2024-09-16T03:39:10.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:10.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649318+0000", 2024-09-16T03:39:10.135 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9273999999999999e-05 2024-09-16T03:39:10.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.135 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:10.135 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649919+0000", 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060074100000000001 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649963+0000", 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4391000000000002e-05 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1724 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'49, mlcod=53'46)", 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:39:10.136 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 78.437624947000003, 2024-09-16T03:39:10.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064536300000000003, 2024-09-16T03:39:10.137 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:10.137 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:10.137 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.137 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:10.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:10.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:39:10.137 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.137 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:10.137 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:39:10.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:10.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652747+0000", 2024-09-16T03:39:10.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.291e-06 2024-09-16T03:39:10.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.138 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:10.138 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652753+0000", 2024-09-16T03:39:10.138 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4450000000000001e-06 2024-09-16T03:39:10.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652782+0000", 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9094999999999998e-05 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652797+0000", 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5021999999999999e-05 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652854+0000", 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6586999999999999e-05 2024-09-16T03:39:10.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:10.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652885+0000", 2024-09-16T03:39:10.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.146e-05 2024-09-16T03:39:10.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:10.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653350+0000", 2024-09-16T03:39:10.140 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046443599999999998 2024-09-16T03:39:10.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.140 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:10.140 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653391+0000", 2024-09-16T03:39:10.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1026999999999999e-05 2024-09-16T03:39:10.141 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.141 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:10.141 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.141 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1725 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'50, mlcod=53'46)", 2024-09-16T03:39:10.141 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:39:10.141 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 78.436832124000006, 2024-09-16T03:39:10.141 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056324100000000002, 2024-09-16T03:39:10.141 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:10.141 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.142 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653539+0000", 2024-09-16T03:39:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9699999999999994e-07 2024-09-16T03:39:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653545+0000", 2024-09-16T03:39:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9309999999999996e-06 2024-09-16T03:39:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653557+0000", 2024-09-16T03:39:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2340999999999999e-05 2024-09-16T03:39:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653564+0000", 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2779999999999999e-06 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653605+0000", 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1301000000000003e-05 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653621+0000", 2024-09-16T03:39:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5951e-05 2024-09-16T03:39:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654064+0000", 2024-09-16T03:39:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000442843 2024-09-16T03:39:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654101+0000", 2024-09-16T03:39:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7598999999999997e-05 2024-09-16T03:39:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.146 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:10.146 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:10.146 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:10.147 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:39:10.191 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:10.190+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:10.260 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:10.259+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:10.264 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:10.264 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:10.264 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:39:10.264 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:10.273 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:39:10.374 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:10.373+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:10.436 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:10.436 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:10.436 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:10.436 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:10.436 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:10.444 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:39:10.462 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:10.461+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:10.546 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:10.544+0000 7f3a21722640 -1 osd.6 135 handle_osd_map injecting map crc failure 2024-09-16T03:39:10.546 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:39:10.563 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:10.561+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:10.592 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:10.592 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:10.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:10.593 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:10.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.593 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:5 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'1, mlcod=0'0)", 2024-09-16T03:39:10.593 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:39:10.593 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 118.621970926, 2024-09-16T03:39:10.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.105742802, 2024-09-16T03:39:10.593 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.593 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.593 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.593 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968147+0000", 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4e-07 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968752+0000", 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060437900000000005 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968762+0000", 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0284e-05 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968778+0000", 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6124e-05 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069797+0000", 2024-09-16T03:39:10.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10101895700000001 2024-09-16T03:39:10.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069833+0000", 2024-09-16T03:39:10.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6013000000000002e-05 2024-09-16T03:39:10.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073838+0000", 2024-09-16T03:39:10.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040049049999999996 2024-09-16T03:39:10.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073889+0000", 2024-09-16T03:39:10.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.13e-05 2024-09-16T03:39:10.597 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.597 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.597 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:7 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'3, mlcod=0'0)", 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 118.615078427, 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20061241399999999, 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975042+0000", 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5289999999999998e-06 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975079+0000", 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7626999999999997e-05 2024-09-16T03:39:10.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975104+0000", 2024-09-16T03:39:10.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4958999999999999e-05 2024-09-16T03:39:10.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.075211+0000", 2024-09-16T03:39:10.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100106814 2024-09-16T03:39:10.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080573+0000", 2024-09-16T03:39:10.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0053618650000000004 2024-09-16T03:39:10.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080592+0000", 2024-09-16T03:39:10.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9079000000000001e-05 2024-09-16T03:39:10.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175497+0000", 2024-09-16T03:39:10.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.094904832999999994 2024-09-16T03:39:10.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175651+0000", 2024-09-16T03:39:10.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015470800000000001 2024-09-16T03:39:10.602 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.602 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.602 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.602 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:44 3.3 e49/31 3:dc284e8d:::smithi15736696-28 low_tier:head v 49'55, mlcod=49'53)", 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 100.46715403100001, 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.149341897, 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122964+0000", 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.91e-07 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122968+0000", 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1470000000000003e-06 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122979+0000", 2024-09-16T03:39:10.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1090000000000001e-05 2024-09-16T03:39:10.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.223110+0000", 2024-09-16T03:39:10.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100131357 2024-09-16T03:39:10.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271688+0000", 2024-09-16T03:39:10.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048578192999999999 2024-09-16T03:39:10.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271712+0000", 2024-09-16T03:39:10.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4066e-05 2024-09-16T03:39:10.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272260+0000", 2024-09-16T03:39:10.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054737000000000004 2024-09-16T03:39:10.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272305+0000", 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5482999999999998e-05 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1007 3.3 e50/31 3:fda66a0d:::920f29fe85d0745d4cd24a887714ac53f1b9a978ec2208ff473c99287fdf8feb:head v 50'57, mlcod=49'55)", 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 99.995527476999996, 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20236485200000001, 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.607 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.608 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:39:10.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:39:10.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594592+0000", 2024-09-16T03:39:10.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0700000000000001e-06 2024-09-16T03:39:10.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595508+0000", 2024-09-16T03:39:10.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00091599499999999996 2024-09-16T03:39:10.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595521+0000", 2024-09-16T03:39:10.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:39:10.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595531+0000", 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2499999999999995e-06 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795762+0000", 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200230977 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795778+0000", 2024-09-16T03:39:10.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:39:10.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796912+0000", 2024-09-16T03:39:10.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011338520000000001 2024-09-16T03:39:10.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796955+0000", 2024-09-16T03:39:10.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2768000000000001e-05 2024-09-16T03:39:10.611 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.611 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.611 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1247 4.11 4:89839b7e:::smithi15736696-12:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 93.634540295999997, 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202294525, 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1247 2024-09-16T03:39:10.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.613 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:39:10.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:39:10.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955579+0000", 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3240000000000001e-06 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955587+0000", 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955600+0000", 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2723e-05 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955612+0000", 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2422999999999999e-05 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055837+0000", 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10022539699999999 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055974+0000", 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000136995 2024-09-16T03:39:10.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,7,8,11,12", 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.056092+0000", 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011738 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057209+0000", 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011170329999999999 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057262+0000", 2024-09-16T03:39:10.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2859000000000003e-05 2024-09-16T03:39:10.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:10.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157594+0000", 2024-09-16T03:39:10.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10033241699999999 2024-09-16T03:39:10.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:10.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157648+0000", 2024-09-16T03:39:10.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3863999999999998e-05 2024-09-16T03:39:10.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157692+0000", 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4406999999999997e-05 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157732+0000", 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9156000000000003e-05 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157780+0000", 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8801999999999998e-05 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157872+0000", 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1311999999999996e-05 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1518 3.3 e59/31 3:df1be05b:::e06092e68680905b5745d5057e93f6c7581518aed07ca5db2802546e9c29aeef:head v 59'95, mlcod=58'93)", 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 85.332096622999998, 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104523454, 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.619 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.620 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:39:10.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:39:10.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258022+0000", 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.372e-06 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258462+0000", 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00043970100000000002 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258476+0000", 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.376e-05 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258483+0000", 2024-09-16T03:39:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.294e-06 2024-09-16T03:39:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358679+0000", 2024-09-16T03:39:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100196177 2024-09-16T03:39:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358718+0000", 2024-09-16T03:39:10.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9295999999999998e-05 2024-09-16T03:39:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362496+0000", 2024-09-16T03:39:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037772069999999999 2024-09-16T03:39:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362544+0000", 2024-09-16T03:39:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8646999999999997e-05 2024-09-16T03:39:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.623 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1923 4.4 e73/50 4:239dbd85:::smithi15736696-2:head v 73'19, mlcod=62'17)", 2024-09-16T03:39:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:39:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.626235801000007, 2024-09-16T03:39:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104144848, 2024-09-16T03:39:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963884+0000", 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964597+0000", 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071314700000000002 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964615+0000", 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8209000000000001e-05 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964626+0000", 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0511e-05 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064779+0000", 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100153495 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064794+0000", 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4946000000000001e-05 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.067999+0000", 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032050709999999999 2024-09-16T03:39:10.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.628 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068026+0000", 2024-09-16T03:39:10.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7066000000000001e-05 2024-09-16T03:39:10.628 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.628 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.628 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.628 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:106 3.3 e73/31 3:dbb0d6f3:::b417679517d63919e63cf420e20217e40570d74b9322ef8b960ef314107040f0:head v 73'120, mlcod=73'116)", 2024-09-16T03:39:10.628 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:39:10.628 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 70.312014875000003, 2024-09-16T03:39:10.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.1035437, 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4900000000000004e-07 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278120+0000", 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6214000000000001e-05 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278129+0000", 2024-09-16T03:39:10.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.115e-06 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278138+0000", 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.329e-06 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.378298+0000", 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100159945 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.378320+0000", 2024-09-16T03:39:10.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1810999999999998e-05 2024-09-16T03:39:10.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.381607+0000", 2024-09-16T03:39:10.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032868210000000001 2024-09-16T03:39:10.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.381646+0000", 2024-09-16T03:39:10.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9515999999999999e-05 2024-09-16T03:39:10.632 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.632 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2145 4.4 e77/50 4:239dbd85:::smithi15736696-2:head v 77'22, mlcod=73'20)", 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.410542+0000", 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 64.179575077999999, 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10336585099999999, 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.410542+0000", 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.410542+0000", 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.410543+0000", 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0599999999999999e-07 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.411160+0000", 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061733000000000003 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.411172+0000", 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2153e-05 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511308+0000", 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10013572900000001 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511607+0000", 2024-09-16T03:39:10.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029879100000000002 2024-09-16T03:39:10.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511647+0000", 2024-09-16T03:39:10.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0241999999999999e-05 2024-09-16T03:39:10.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.513870+0000", 2024-09-16T03:39:10.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022230499999999999 2024-09-16T03:39:10.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.513908+0000", 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8349999999999997e-05 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2152 4.11 4:89839b7e:::smithi15736696-12:head [read 0~2487436,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 64.078543394999997, 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10362692900000001, 2024-09-16T03:39:10.637 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2152 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511576+0000", 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511584+0000", 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5939999999999994e-06 2024-09-16T03:39:10.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511612+0000", 2024-09-16T03:39:10.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7855e-05 2024-09-16T03:39:10.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.611754+0000", 2024-09-16T03:39:10.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100141641 2024-09-16T03:39:10.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613255+0000", 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001500744 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613366+0000", 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000111088 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.615201+0000", 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0018353009999999999 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.641 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2445 4.11 4:89839b7e:::smithi15736696-12:head [sparse-read 1818507~599510,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 55.145953972999997, 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200613392, 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2445 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.642 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444167+0000", 2024-09-16T03:39:10.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6650000000000001e-06 2024-09-16T03:39:10.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444182+0000", 2024-09-16T03:39:10.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4678000000000001e-05 2024-09-16T03:39:10.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444207+0000", 2024-09-16T03:39:10.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4768000000000001e-05 2024-09-16T03:39:10.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.544359+0000", 2024-09-16T03:39:10.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100152804 2024-09-16T03:39:10.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644561+0000", 2024-09-16T03:39:10.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10020214500000001 2024-09-16T03:39:10.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644624+0000", 2024-09-16T03:39:10.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2444e-05 2024-09-16T03:39:10.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644777+0000", 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000152888 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2629 3.3 e89/31 3:e23364e2:::e447860b11e9f9fde68b75cb1d2f204a74952889ec51a2417ef18c1427ff9a70:head v 89'173, mlcod=87'171)", 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.13821128, 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10848095100000001, 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.646 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.647 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.647 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.647 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:39:10.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.647 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:39:10.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451907+0000", 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6599999999999996e-07 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456443+0000", 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0045365509999999998 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456470+0000", 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6897e-05 2024-09-16T03:39:10.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456496+0000", 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.618e-05 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556666+0000", 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100169678 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556686+0000", 2024-09-16T03:39:10.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9740000000000001e-05 2024-09-16T03:39:10.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560345+0000", 2024-09-16T03:39:10.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003658821 2024-09-16T03:39:10.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560387+0000", 2024-09-16T03:39:10.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2518000000000002e-05 2024-09-16T03:39:10.650 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.650 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.650 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:209 3.3 e99/31 3:f2e0c7d0:::6bf49184c86dbc717b1f919afcf6cb40a4ca9c2465eb100f7ba59687540c40ed:head v 99'201, mlcod=98'194)", 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 38.384081082999998, 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200559603, 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:39:10.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:39:10.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206037+0000", 2024-09-16T03:39:10.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9999999999999999e-07 2024-09-16T03:39:10.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206039+0000", 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2579999999999998e-06 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206041+0000", 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.542e-06 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306162+0000", 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100120348 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306250+0000", 2024-09-16T03:39:10.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:39:10.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306268+0000", 2024-09-16T03:39:10.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8127000000000001e-05 2024-09-16T03:39:10.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.406549+0000", 2024-09-16T03:39:10.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10028121199999999 2024-09-16T03:39:10.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.406596+0000", 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6700999999999999e-05 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3274 4.11 4:89839b7e:::smithi15736696-12:head [write 1813152~490434] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 29.392317912999999, 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.28154463400000002, 2024-09-16T03:39:10.655 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.656 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.656 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.656 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:10.656 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:10.656 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:10.656 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3274 2024-09-16T03:39:10.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.656 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:39:10.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197801+0000", 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9539999999999998e-06 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.198205+0000", 2024-09-16T03:39:10.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00040328199999999998 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.198218+0000", 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3176000000000001e-05 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298318+0000", 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100100562 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298419+0000", 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100667 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298489+0000", 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0023000000000004e-05 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,12", 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298583+0000", 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3634000000000005e-05 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:10.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300283+0000", 2024-09-16T03:39:10.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00170049 2024-09-16T03:39:10.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:10.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.307072+0000", 2024-09-16T03:39:10.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0067889209999999998 2024-09-16T03:39:10.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:10.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479243+0000", 2024-09-16T03:39:10.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.17217031799999999 2024-09-16T03:39:10.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479300+0000", 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7837999999999997e-05 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479344+0000", 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3769000000000001e-05 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3275 4.11 4:89839b7e:::smithi15736696-12:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 29.2917725, 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.276974636, 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3275 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298364+0000", 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9273e-05 2024-09-16T03:39:10.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.664 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298373+0000", 2024-09-16T03:39:10.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5809999999999997e-06 2024-09-16T03:39:10.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.664 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298382+0000", 2024-09-16T03:39:10.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6269999999999995e-06 2024-09-16T03:39:10.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.664 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298390+0000", 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2259999999999999e-06 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299770+0000", 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013803159999999999 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299852+0000", 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.174e-05 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,12", 2024-09-16T03:39:10.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299948+0000", 2024-09-16T03:39:10.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6669000000000006e-05 2024-09-16T03:39:10.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.666 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:10.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300496+0000", 2024-09-16T03:39:10.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054762799999999998 2024-09-16T03:39:10.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.666 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:10.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.401930+0000", 2024-09-16T03:39:10.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101434495 2024-09-16T03:39:10.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575189+0000", 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.173258996 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575247+0000", 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7651000000000001e-05 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575320+0000", 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2434000000000005e-05 2024-09-16T03:39:10.667 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.668 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.668 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.668 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3276 4.11 4:89839b7e:::smithi15736696-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:39:10.668 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:39:10.668 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 29.291703687999998, 2024-09-16T03:39:10.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.37724456499999998, 2024-09-16T03:39:10.668 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.668 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.668 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:10.668 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:10.668 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3276 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298415+0000", 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298418+0000", 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.766e-06 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298426+0000", 2024-09-16T03:39:10.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1359999999999999e-06 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298433+0000", 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0899999999999999e-06 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300168+0000", 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017353869999999999 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:10.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300215+0000", 2024-09-16T03:39:10.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6702000000000001e-05 2024-09-16T03:39:10.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675524+0000", 2024-09-16T03:39:10.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.375309585 2024-09-16T03:39:10.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675596+0000", 2024-09-16T03:39:10.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1921000000000005e-05 2024-09-16T03:39:10.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675658+0000", 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2052999999999997e-05 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3985 4.36 e130/126 4:6ef3b53d:::smithi15736696-48:head v 130'88, mlcod=124'86)", 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.376369+0000", 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.2137480070000004, 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20392002300000001, 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.673 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376369+0000", 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376369+0000", 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376370+0000", 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1299999999999999e-07 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376877+0000", 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050726199999999997 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376888+0000", 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0798999999999999e-05 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376896+0000", 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1380000000000007e-06 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376950+0000", 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4580000000000003e-05 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376970+0000", 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9607e-05 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580253+0000", 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20328323600000001 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580289+0000", 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6188e-05 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.677 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3986 4.36 e130/126 4:6ef3b53d:::smithi15736696-48:head v 130'89, mlcod=124'86)", 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.377664+0000", 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.2124532769999998, 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264148100000001, 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.377664+0000", 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.377664+0000", 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.377665+0000", 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.540000000000001e-07 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378852+0000", 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001187097 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378863+0000", 2024-09-16T03:39:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0721e-05 2024-09-16T03:39:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378868+0000", 2024-09-16T03:39:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5650000000000001e-06 2024-09-16T03:39:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378913+0000", 2024-09-16T03:39:10.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4867999999999998e-05 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378929+0000", 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5752000000000001e-05 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580291+0000", 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20136209599999999 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580306+0000", 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4428e-05 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3987 4.36 e130/126 4:6ef3b53d:::smithi15736696-48:head v 130'90, mlcod=124'86)", 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.379666+0000", 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.2104518149999999, 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20094095300000001, 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.682 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379666+0000", 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379666+0000", 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379667+0000", 2024-09-16T03:39:10.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.11e-06 2024-09-16T03:39:10.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379675+0000", 2024-09-16T03:39:10.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4489999999999999e-06 2024-09-16T03:39:10.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379683+0000", 2024-09-16T03:39:10.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0660000000000004e-06 2024-09-16T03:39:10.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379690+0000", 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4010000000000002e-06 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.579898+0000", 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20020810999999999 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.579914+0000", 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6198e-05 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580572+0000", 2024-09-16T03:39:10.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065845699999999999 2024-09-16T03:39:10.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580607+0000", 2024-09-16T03:39:10.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4162000000000003e-05 2024-09-16T03:39:10.686 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.686 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.686 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:4043 4.36 e131/126 4:6ece7830:::smithi15736696-7:head v 131'93, mlcod=130'89)", 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.055626+0000", 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.5344910340000002, 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19612945100000001, 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055626+0000", 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.687 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055626+0000", 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055627+0000", 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0869999999999999e-06 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055633+0000", 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.101e-06 2024-09-16T03:39:10.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055641+0000", 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9759999999999995e-06 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055646+0000", 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4360000000000001e-06 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251121+0000", 2024-09-16T03:39:10.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.195475068 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251137+0000", 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6271999999999999e-05 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251719+0000", 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058167799999999997 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:10.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251756+0000", 2024-09-16T03:39:10.691 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6832999999999998e-05 2024-09-16T03:39:10.691 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.691 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.691 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.691 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:10.691 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:10.691 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:10.692 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:39:10.693 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:10.690+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:10.793 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:10.792+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:10.797 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:10.797 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:39:10.797 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:10.806 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:39:10.894 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:10.892+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:10.950 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:10.950 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:10.950 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:10.950 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:10.950 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:10.958 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:39:11.002 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:11.001+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:11.065 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:39 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:39:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 104.222662993, 2024-09-16T03:39:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40113680400000001, 2024-09-16T03:39:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839700+0000", 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.571e-06 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839704+0000", 2024-09-16T03:39:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8399999999999997e-06 2024-09-16T03:39:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839712+0000", 2024-09-16T03:39:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9990000000000008e-06 2024-09-16T03:39:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839720+0000", 2024-09-16T03:39:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0099999999999995e-06 2024-09-16T03:39:11.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839783+0000", 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3014000000000003e-05 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839870+0000", 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7392999999999995e-05 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840049+0000", 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017825800000000001 2024-09-16T03:39:11.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840874+0000", 2024-09-16T03:39:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082532099999999995 2024-09-16T03:39:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.040285+0000", 2024-09-16T03:39:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19941088400000001 2024-09-16T03:39:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240748+0000", 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20046322699999999 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240786+0000", 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8158000000000003e-05 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240834+0000", 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8128999999999999e-05 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:983 3.1 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head [write 0~2138112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:39:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:39:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 100.998760053, 2024-09-16T03:39:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30819799799999997, 2024-09-16T03:39:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 983 2024-09-16T03:39:11.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063610+0000", 2024-09-16T03:39:11.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2990000000000002e-06 2024-09-16T03:39:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065735+0000", 2024-09-16T03:39:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021257799999999999 2024-09-16T03:39:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065752+0000", 2024-09-16T03:39:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6895999999999999e-05 2024-09-16T03:39:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065762+0000", 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6830000000000003e-06 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065871+0000", 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010904900000000001 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066005+0000", 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013369799999999999 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066135+0000", 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000130481 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.069822+0000", 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036863759999999999 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272370+0000", 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20254802599999999 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371671+0000", 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099300893000000001 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371754+0000", 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3075e-05 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371798+0000", 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4742e-05 2024-09-16T03:39:11.077 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1515 3.1 3:9c9e970c:::7249cb65b52cadf4fe8f4ec923c7e548d1de7d9628823acd2c8de19121f1c1ce:head [write 0~1094656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:39:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:39:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 86.112634315999998, 2024-09-16T03:39:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.305958065, 2024-09-16T03:39:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.078 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1515 2024-09-16T03:39:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:39:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:39:11.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949729+0000", 2024-09-16T03:39:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.785e-06 2024-09-16T03:39:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950933+0000", 2024-09-16T03:39:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001204341 2024-09-16T03:39:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950950+0000", 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6838e-05 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950960+0000", 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0316000000000001e-05 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951065+0000", 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104128 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951202+0000", 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013706600000000001 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951350+0000", 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000148197 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.953857+0000", 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0025067150000000001 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.958049+0000", 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0041924650000000003 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255547+0000", 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29749841900000001 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255649+0000", 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101513 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255684+0000", 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5281999999999999e-05 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1704 3.1 3:b6a94245:::f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99:head [write 0~453120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 80.269522961000007, 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40403313699999999, 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.084 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1704 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:39:11.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792841+0000", 2024-09-16T03:39:11.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.433e-06 2024-09-16T03:39:11.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793329+0000", 2024-09-16T03:39:11.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048839200000000004 2024-09-16T03:39:11.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793345+0000", 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6079e-05 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793354+0000", 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.32e-06 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793418+0000", 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3944999999999998e-05 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793549+0000", 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013159900000000001 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793666+0000", 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116532 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.797707+0000", 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040409629999999998 2024-09-16T03:39:11.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897288+0000", 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099581449000000002 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196775+0000", 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948696800000002 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196814+0000", 2024-09-16T03:39:11.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8336e-05 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196871+0000", 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7120999999999997e-05 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:84 3.1 3:b24f289d:::e84532c28f467d2f2a6e2cf44904fa0d4a555e1304bbe1b25db645af42152bd7:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 80.260751021999994, 2024-09-16T03:39:11.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.395455581, 2024-09-16T03:39:11.091 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.091 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.091 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.091 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:39:11.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:39:11.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801613+0000", 2024-09-16T03:39:11.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9559999999999999e-06 2024-09-16T03:39:11.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802117+0000", 2024-09-16T03:39:11.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050401600000000003 2024-09-16T03:39:11.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802141+0000", 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3618e-05 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802155+0000", 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4358e-05 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802229+0000", 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3930999999999994e-05 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802355+0000", 2024-09-16T03:39:11.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012527299999999999 2024-09-16T03:39:11.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802567+0000", 2024-09-16T03:39:11.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021250199999999999 2024-09-16T03:39:11.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806218+0000", 2024-09-16T03:39:11.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036512189999999998 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897323+0000", 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091104892000000007 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196959+0000", 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299635971 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197021+0000", 2024-09-16T03:39:11.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1938000000000002e-05 2024-09-16T03:39:11.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197065+0000", 2024-09-16T03:39:11.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3906999999999998e-05 2024-09-16T03:39:11.096 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.096 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.096 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.096 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:85 3.1 3:b380f269:::e65351e8a65a0b1a517e3b1a85e04b84c0ab7cfa6655f11fb0da31f56b4dd9af:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:11.096 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 80.260172124999997, 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39502727300000001, 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:39:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802190+0000", 2024-09-16T03:39:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2419999999999999e-06 2024-09-16T03:39:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802464+0000", 2024-09-16T03:39:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027463699999999999 2024-09-16T03:39:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802478+0000", 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3375e-05 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802484+0000", 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0990000000000004e-06 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803475+0000", 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00099113099999999991 2024-09-16T03:39:11.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803651+0000", 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017654300000000001 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803923+0000", 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000271842 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806228+0000", 2024-09-16T03:39:11.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023049590000000001 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897563+0000", 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091334724000000006 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197123+0000", 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299559992 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197177+0000", 2024-09-16T03:39:11.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4447999999999998e-05 2024-09-16T03:39:11.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197216+0000", 2024-09-16T03:39:11.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8281000000000002e-05 2024-09-16T03:39:11.102 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.102 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.102 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.102 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:86 3.1 3:b3acb53e:::e951eb69ce5151a3f54107fa22c70f80729eff845da1613447a91536a24046e8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:11.102 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 80.25986503, 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.394851071, 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802496+0000", 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802782+0000", 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00028523600000000002 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802796+0000", 2024-09-16T03:39:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4594e-05 2024-09-16T03:39:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802804+0000", 2024-09-16T03:39:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3790000000000006e-06 2024-09-16T03:39:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804787+0000", 2024-09-16T03:39:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019825060000000002 2024-09-16T03:39:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804895+0000", 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010813499999999999 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.805046+0000", 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150972 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806681+0000", 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016353839999999999 2024-09-16T03:39:11.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.005712+0000", 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19903093199999999 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197278+0000", 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.191565295 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197315+0000", 2024-09-16T03:39:11.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6946000000000002e-05 2024-09-16T03:39:11.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197347+0000", 2024-09-16T03:39:11.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1863000000000003e-05 2024-09-16T03:39:11.108 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.108 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.108 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.108 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:140 3.1 3:845e3b2c:::21d7b947791b34f7e3fcd39ad119c9f923ad2f3cc7a703501eac7a68189d5437:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:11.108 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:39:11.108 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.947685776, 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39914108999999998, 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:39:11.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114677+0000", 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1610000000000001e-06 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114684+0000", 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3150000000000003e-06 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114699+0000", 2024-09-16T03:39:11.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4996e-05 2024-09-16T03:39:11.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114705+0000", 2024-09-16T03:39:11.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0580000000000003e-06 2024-09-16T03:39:11.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.214862+0000", 2024-09-16T03:39:11.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015700199999999 2024-09-16T03:39:11.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.214922+0000", 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0053999999999998e-05 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215023+0000", 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100432 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.216100+0000", 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010776819999999999 2024-09-16T03:39:11.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.499919+0000", 2024-09-16T03:39:11.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.28381845900000002 2024-09-16T03:39:11.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513755+0000", 2024-09-16T03:39:11.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013836477 2024-09-16T03:39:11.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513790+0000", 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4295999999999998e-05 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513816+0000", 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6157999999999999e-05 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:141 3.1 3:9387c678:::99a87d3b555b50e905a1ff55d99eb4aa025bd0070f07a4910d03cf1af95bbc44:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:39:11.114 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.947635079000001, 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50160030300000003, 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114726+0000", 2024-09-16T03:39:11.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4399999999999999e-07 2024-09-16T03:39:11.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114730+0000", 2024-09-16T03:39:11.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1479999999999999e-06 2024-09-16T03:39:11.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114738+0000", 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3200000000000002e-06 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114743+0000", 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.891e-06 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215320+0000", 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100576463 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215376+0000", 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6310999999999997e-05 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215473+0000", 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7103999999999997e-05 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.216108+0000", 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00063422800000000003 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513851+0000", 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29774376400000002 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616086+0000", 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10223441 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616240+0000", 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015453799999999999 2024-09-16T03:39:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616326+0000", 2024-09-16T03:39:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5382000000000003e-05 2024-09-16T03:39:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:142 3.1 3:a93b7b74:::70cd8f47898d416018aded7de9c29e0fe002b2336cb6cfabbf3eb6238766c3b1:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:39:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.947605692000003, 2024-09-16T03:39:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50166088799999997, 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.121 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3600000000000003e-07 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114762+0000", 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6039999999999998e-06 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114769+0000", 2024-09-16T03:39:11.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5939999999999999e-06 2024-09-16T03:39:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114774+0000", 2024-09-16T03:39:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2179999999999998e-06 2024-09-16T03:39:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215687+0000", 2024-09-16T03:39:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100912974 2024-09-16T03:39:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215733+0000", 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6419999999999999e-05 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215836+0000", 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102981 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316766+0000", 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100929693 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513877+0000", 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19711067800000001 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616358+0000", 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10248180699999999 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616391+0000", 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2551000000000001e-05 2024-09-16T03:39:11.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616416+0000", 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4732e-05 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:143 3.1 3:ac848b49:::adc4f0c8a4d0bc7e52f9d5c3b4981447737a7dbe58b8313a369b04383e10893f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.947575686, 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60333394699999998, 2024-09-16T03:39:11.126 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:39:11.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7899999999999999e-07 2024-09-16T03:39:11.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114788+0000", 2024-09-16T03:39:11.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4569999999999999e-06 2024-09-16T03:39:11.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114792+0000", 2024-09-16T03:39:11.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5730000000000001e-06 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114796+0000", 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5449999999999999e-06 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316237+0000", 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20144099300000001 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316417+0000", 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017981099999999999 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316564+0000", 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014693700000000001 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316993+0000", 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042985499999999998 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.717965+0000", 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40097123699999998 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718005+0000", 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0083000000000002e-05 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718064+0000", 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9020000000000001e-05 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718119+0000", 2024-09-16T03:39:11.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4957000000000003e-05 2024-09-16T03:39:11.132 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.132 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.132 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.132 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:149 3.1 3:8eb88b02:::fa73f0ab3a7395fcfc19219899ff8c0a4a3c96caf8327f9da9656e2450435b27:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:11.132 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:39:11.132 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 53.933667038000003, 2024-09-16T03:39:11.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.348655887, 2024-09-16T03:39:11.132 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128698+0000", 2024-09-16T03:39:11.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.527e-06 2024-09-16T03:39:11.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128714+0000", 2024-09-16T03:39:11.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6019e-05 2024-09-16T03:39:11.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128757+0000", 2024-09-16T03:39:11.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3318000000000001e-05 2024-09-16T03:39:11.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128764+0000", 2024-09-16T03:39:11.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.28e-06 2024-09-16T03:39:11.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.228916+0000", 2024-09-16T03:39:11.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015252199999999 2024-09-16T03:39:11.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.228976+0000", 2024-09-16T03:39:11.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0390999999999998e-05 2024-09-16T03:39:11.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.229077+0000", 2024-09-16T03:39:11.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010053 2024-09-16T03:39:11.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.274759+0000", 2024-09-16T03:39:11.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.045682404000000003 2024-09-16T03:39:11.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.330314+0000", 2024-09-16T03:39:11.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.055554916000000003 2024-09-16T03:39:11.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477196+0000", 2024-09-16T03:39:11.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14688130099999999 2024-09-16T03:39:11.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477291+0000", 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5144999999999998e-05 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477349+0000", 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8533999999999999e-05 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:168 3.1 3:a655680f:::65b183ceded8ac2b61dbe9ba8fb4fd4f9f3a4854c219da4322bb1e81ebb7e2e5:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.648756+0000", 2024-09-16T03:39:11.138 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.413604016000001, 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30591469500000001, 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648756+0000", 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648756+0000", 2024-09-16T03:39:11.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648757+0000", 2024-09-16T03:39:11.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7299999999999995e-07 2024-09-16T03:39:11.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648799+0000", 2024-09-16T03:39:11.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.227e-05 2024-09-16T03:39:11.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648805+0000", 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4790000000000004e-06 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648809+0000", 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7890000000000001e-06 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.650787+0000", 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019778700000000001 2024-09-16T03:39:11.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.650914+0000", 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012707899999999999 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651123+0000", 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020929100000000001 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.652287+0000", 2024-09-16T03:39:11.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011644859999999999 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.653716+0000", 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014290889999999999 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954577+0000", 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30086063800000001 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954621+0000", 2024-09-16T03:39:11.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3820999999999999e-05 2024-09-16T03:39:11.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954671+0000", 2024-09-16T03:39:11.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0210000000000002e-05 2024-09-16T03:39:11.144 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.144 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.144 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.144 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:169 3.1 3:8ca52973:::737e1241b430663b0c460fc4c0e6bf0a2c97ca3f584ee4e3541536e0014b01da:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:39:11.144 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.648814+0000", 2024-09-16T03:39:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.413546191000002, 2024-09-16T03:39:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30599283399999999, 2024-09-16T03:39:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648814+0000", 2024-09-16T03:39:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648814+0000", 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648815+0000", 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9900000000000001e-07 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648861+0000", 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6786999999999998e-05 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648867+0000", 2024-09-16T03:39:11.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1830000000000002e-06 2024-09-16T03:39:11.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648872+0000", 2024-09-16T03:39:11.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9169999999999997e-06 2024-09-16T03:39:11.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651624+0000", 2024-09-16T03:39:11.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027522150000000001 2024-09-16T03:39:11.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651689+0000", 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5065000000000001e-05 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651814+0000", 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012493199999999999 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.653881+0000", 2024-09-16T03:39:11.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020671830000000002 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.752238+0000", 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098357336000000004 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954726+0000", 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20248775199999999 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954780+0000", 2024-09-16T03:39:11.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3859000000000001e-05 2024-09-16T03:39:11.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954807+0000", 2024-09-16T03:39:11.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7206e-05 2024-09-16T03:39:11.150 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.150 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.150 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.150 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:171 3.1 3:93178f8f:::777eb2bf7c36938cba5735a17f90bb453157d7a9a42a0a6f52227fe7530ec353:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:39:11.150 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.648876+0000", 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.413484146000002, 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30603207999999998, 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648876+0000", 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648876+0000", 2024-09-16T03:39:11.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648877+0000", 2024-09-16T03:39:11.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8799999999999998e-07 2024-09-16T03:39:11.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648918+0000", 2024-09-16T03:39:11.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1488e-05 2024-09-16T03:39:11.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648923+0000", 2024-09-16T03:39:11.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0320000000000003e-06 2024-09-16T03:39:11.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648926+0000", 2024-09-16T03:39:11.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1690000000000001e-06 2024-09-16T03:39:11.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.652727+0000", 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038005909999999999 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.652854+0000", 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012661700000000001 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.653069+0000", 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021573799999999999 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.654043+0000", 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00097351999999999996 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.854187+0000", 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20014437400000001 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954834+0000", 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100646293 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954875+0000", 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1637999999999997e-05 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954908+0000", 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3232000000000003e-05 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.156 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:139 3.1 3:bd600e93:::b81929c6ed6df2fe181809e10b331bcd3c2df1a1077e06024548babcdcd8f550:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.020786925000003, 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.43180016300000001, 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2300000000000001e-07 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041700+0000", 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000126367 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041707+0000", 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1519999999999997e-06 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041710+0000", 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7869999999999999e-06 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043365+0000", 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001654914 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043453+0000", 2024-09-16T03:39:11.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7954000000000005e-05 2024-09-16T03:39:11.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043596+0000", 2024-09-16T03:39:11.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014304500000000001 2024-09-16T03:39:11.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.145131+0000", 2024-09-16T03:39:11.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10153440599999999 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.344273+0000", 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19914232200000001 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473290+0000", 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12901701900000001 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473325+0000", 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4656000000000002e-05 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473374+0000", 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9017999999999997e-05 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:144 3.1 3:91c40fc0:::f2d8823355177e50718038dafd7a8313c1a25b0231461a8672e684a7d4e587b4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:39:11.162 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.020642668000001, 2024-09-16T03:39:11.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.431801096, 2024-09-16T03:39:11.163 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.163 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.163 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.163 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:39:11.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.75e-07 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042077+0000", 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035865099999999998 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042084+0000", 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2400000000000001e-06 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042088+0000", 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1570000000000002e-06 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144179+0000", 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10209033300000001 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144427+0000", 2024-09-16T03:39:11.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000247879 2024-09-16T03:39:11.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144656+0000", 2024-09-16T03:39:11.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022919800000000001 2024-09-16T03:39:11.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.145866+0000", 2024-09-16T03:39:11.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001210046 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.344293+0000", 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.198427353 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473432+0000", 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12913908800000001 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473478+0000", 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6165000000000003e-05 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473519+0000", 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0510999999999999e-05 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:165 3.1 3:8166b007:::3555453b0ab6ec5b2f29a1e76cc26fdaf7f55d077a227627cdf2200eb9ca342d:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:39:11.168 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 29.235125866000001, 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40346431100000002, 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:39:11.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5999999999999999e-07 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827239+0000", 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8920000000000002e-06 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827243+0000", 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2760000000000002e-06 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827246+0000", 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1990000000000002e-06 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.029284+0000", 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20203769499999999 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.029375+0000", 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1419e-05 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.029542+0000", 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016613199999999999 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.130201+0000", 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10065989 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230566+0000", 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100364563 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230623+0000", 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6848000000000002e-05 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230670+0000", 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7117999999999998e-05 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230699+0000", 2024-09-16T03:39:11.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8918999999999999e-05 2024-09-16T03:39:11.174 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.174 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.174 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.174 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:166 3.1 3:91c40fc0:::f2d8823355177e50718038dafd7a8313c1a25b0231461a8672e684a7d4e587b4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:11.174 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.827250+0000", 2024-09-16T03:39:11.174 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 29.235110155000001, 2024-09-16T03:39:11.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40362257099999999, 2024-09-16T03:39:11.174 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827250+0000", 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827250+0000", 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827251+0000", 2024-09-16T03:39:11.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:39:11.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827253+0000", 2024-09-16T03:39:11.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.928e-06 2024-09-16T03:39:11.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827257+0000", 2024-09-16T03:39:11.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0249999999999996e-06 2024-09-16T03:39:11.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827260+0000", 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3359999999999999e-06 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.090467+0000", 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.26320751100000001 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.090543+0000", 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5740000000000003e-05 2024-09-16T03:39:11.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.090651+0000", 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000107415 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.130204+0000", 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.039553455000000001 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230753+0000", 2024-09-16T03:39:11.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100549096 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230809+0000", 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6243999999999999e-05 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230848+0000", 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9045999999999999e-05 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230873+0000", 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4433000000000001e-05 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4046 3.1 3:a08385ac:::dce542a79a00e18d86bcad9b7e35b22c010a7ca1928705dca52eb83cbed579e2:head [write 0~1201152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.336463+0000", 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.7258970920000003, 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30533954099999999, 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.180 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4046 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.336463+0000", 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.336463+0000", 2024-09-16T03:39:11.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.336468+0000", 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4000000000000002e-06 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.337369+0000", 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00090117699999999997 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.182 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.337379+0000", 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0496000000000001e-05 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.337390+0000", 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0604e-05 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.337432+0000", 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1854000000000003e-05 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.183 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.337542+0000", 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011011 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.337642+0000", 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010011 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.339464+0000", 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0018221330000000001 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.184 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.341640+0000", 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021757109999999999 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.641710+0000", 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30006966499999999 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.641744+0000", 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4121000000000001e-05 2024-09-16T03:39:11.185 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.186 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.186 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.186 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.641803+0000", 2024-09-16T03:39:11.186 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9160000000000003e-05 2024-09-16T03:39:11.186 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.186 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.186 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.186 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.186 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.186 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:11.188 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:39:11.188 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:11.185+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:11.285 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:11.283+0000 7f4214d04640 -1 osd.8 135 handle_osd_map injecting map crc failure 2024-09-16T03:39:11.288 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:11.286+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:11.333 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:11.333 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:11.334 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:39:11.334 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:11.342 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:39:11.390 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:11.387+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:11.446 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:11.446 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:11.446 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:11.446 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:11.447 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:11.455 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:39:11.490 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:11.487+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:11.562 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:11.563 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:11.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:11.563 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:11.563 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.563 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:416 4.0 4:023abada:::smithi15736696-17:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:11.563 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:39:11.563 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 111.25046414400001, 2024-09-16T03:39:11.563 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60401983699999995, 2024-09-16T03:39:11.563 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.563 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.563 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.563 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.563 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.564 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.564 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 416 2024-09-16T03:39:11.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.564 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.564 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.564 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:39:11.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.564 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.564 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.564 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.564 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:39:11.564 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.565 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.565 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309243+0000", 2024-09-16T03:39:11.565 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.178e-06 2024-09-16T03:39:11.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.565 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.565 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.565 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309248+0000", 2024-09-16T03:39:11.565 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:39:11.565 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309257+0000", 2024-09-16T03:39:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4720000000000001e-06 2024-09-16T03:39:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309262+0000", 2024-09-16T03:39:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0839999999999999e-06 2024-09-16T03:39:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.566 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309297+0000", 2024-09-16T03:39:11.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4997999999999999e-05 2024-09-16T03:39:11.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309367+0000", 2024-09-16T03:39:11.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0124999999999997e-05 2024-09-16T03:39:11.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.567 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.567 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:11.567 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812055+0000", 2024-09-16T03:39:11.567 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50268796900000001 2024-09-16T03:39:11.567 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.568 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.568 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.568 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812614+0000", 2024-09-16T03:39:11.568 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000558309 2024-09-16T03:39:11.568 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.568 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.568 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.568 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.813040+0000", 2024-09-16T03:39:11.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042652800000000002 2024-09-16T03:39:11.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.569 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.569 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913189+0000", 2024-09-16T03:39:11.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10014880399999999 2024-09-16T03:39:11.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.569 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.569 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913220+0000", 2024-09-16T03:39:11.569 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1353000000000003e-05 2024-09-16T03:39:11.569 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.569 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913262+0000", 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1284e-05 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:494 4.f 4:f817def9:::smithi15736696-45:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 109.837524409, 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60451760399999999, 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.570 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 494 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.571 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:39:11.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.572 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722183+0000", 2024-09-16T03:39:11.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:39:11.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.572 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.572 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.572 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722189+0000", 2024-09-16T03:39:11.572 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9429999999999997e-06 2024-09-16T03:39:11.572 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722203+0000", 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4367e-05 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722210+0000", 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822367+0000", 2024-09-16T03:39:11.573 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100156933 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822461+0000", 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3216999999999999e-05 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125593+0000", 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30313253699999998 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.574 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126307+0000", 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071350000000000005 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126825+0000", 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051846100000000001 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326612+0000", 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19978721999999999 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.575 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326663+0000", 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0460000000000001e-05 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326699+0000", 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6340000000000001e-05 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:784 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=63b,truncate 2840350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:11.576 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:39:11.577 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 105.753116533, 2024-09-16T03:39:11.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63358405200000001, 2024-09-16T03:39:11.577 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.577 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.577 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.577 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.577 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.577 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.577 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 784 2024-09-16T03:39:11.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.577 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806590+0000", 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6000000000000003e-07 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.579 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.579 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806593+0000", 2024-09-16T03:39:11.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6649999999999999e-06 2024-09-16T03:39:11.579 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.579 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.579 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.579 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806597+0000", 2024-09-16T03:39:11.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.245e-06 2024-09-16T03:39:11.579 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.579 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.579 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.579 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806600+0000", 2024-09-16T03:39:11.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.23e-06 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099381+0000", 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29278119400000002 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099438+0000", 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6198e-05 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:11.580 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099527+0000", 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9505000000000004e-05 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200775+0000", 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101247904 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200889+0000", 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114301 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.581 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440066+0000", 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.239176367 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440109+0000", 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2941000000000001e-05 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440173+0000", 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4842e-05 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.582 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:785 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 105.753100608, 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63376373200000002, 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.583 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 785 2024-09-16T03:39:11.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.584 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:39:11.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:39:11.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806606+0000", 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806607+0000", 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.424e-06 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806611+0000", 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1080000000000003e-06 2024-09-16T03:39:11.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806614+0000", 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.067e-06 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099813+0000", 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29319853699999998 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099858+0000", 2024-09-16T03:39:11.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5667999999999997e-05 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440236+0000", 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34037791499999998 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440297+0000", 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0988e-05 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440369+0000", 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1682999999999998e-05 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:801 4.0 4:07ad0560:::smithi15736696-25:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 105.55362271, 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.534510445, 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:11.588 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 801 2024-09-16T03:39:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:39:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:39:11.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006084+0000", 2024-09-16T03:39:11.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.322e-06 2024-09-16T03:39:11.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.590 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.590 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006088+0000", 2024-09-16T03:39:11.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000001e-06 2024-09-16T03:39:11.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006098+0000", 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0186999999999999e-05 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006104+0000", 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.271e-06 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.199952+0000", 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.193847935 2024-09-16T03:39:11.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200027+0000", 2024-09-16T03:39:11.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4987000000000006e-05 2024-09-16T03:39:11.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.540594+0000", 2024-09-16T03:39:11.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34056708299999999 2024-09-16T03:39:11.592 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.592 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.592 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.593 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1023 4.0 4:06b6d050:::smithi15736696-50:head [write 2246563~579356] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:11.593 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:39:11.593 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 100.24847364, 2024-09-16T03:39:11.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60727534699999997, 2024-09-16T03:39:11.593 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.593 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.593 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.593 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.593 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.593 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.593 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1023 2024-09-16T03:39:11.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311233+0000", 2024-09-16T03:39:11.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1039999999999999e-06 2024-09-16T03:39:11.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311692+0000", 2024-09-16T03:39:11.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045908800000000002 2024-09-16T03:39:11.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.595 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.595 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311704+0000", 2024-09-16T03:39:11.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1844e-05 2024-09-16T03:39:11.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311713+0000", 2024-09-16T03:39:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2500000000000006e-06 2024-09-16T03:39:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359204+0000", 2024-09-16T03:39:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.047491646999999998 2024-09-16T03:39:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359258+0000", 2024-09-16T03:39:11.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4082000000000002e-05 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359354+0000", 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5686999999999994e-05 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512245+0000", 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15289135700000001 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512956+0000", 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071110300000000004 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918361+0000", 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40540489400000002 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918455+0000", 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3412999999999995e-05 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918508+0000", 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1024 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3176490] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 100.24798065900001, 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60701227000000002, 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.599 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1024 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:39:11.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311726+0000", 2024-09-16T03:39:11.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.02e-07 2024-09-16T03:39:11.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311730+0000", 2024-09-16T03:39:11.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1099999999999996e-06 2024-09-16T03:39:11.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311735+0000", 2024-09-16T03:39:11.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4680000000000003e-06 2024-09-16T03:39:11.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311739+0000", 2024-09-16T03:39:11.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9380000000000002e-06 2024-09-16T03:39:11.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.602 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.602 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361468+0000", 2024-09-16T03:39:11.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.049729450000000001 2024-09-16T03:39:11.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361539+0000", 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0896999999999996e-05 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361661+0000", 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000122404 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512943+0000", 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.151281481 2024-09-16T03:39:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512959+0000", 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918588+0000", 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40562815499999999 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918646+0000", 2024-09-16T03:39:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8566999999999997e-05 2024-09-16T03:39:11.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918737+0000", 2024-09-16T03:39:11.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1308000000000001e-05 2024-09-16T03:39:11.605 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.605 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.605 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.605 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1025 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:11.605 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:11.605 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 100.247962161, 2024-09-16T03:39:11.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60723649599999996, 2024-09-16T03:39:11.606 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.606 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.606 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:11.606 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.606 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.606 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.606 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1025 2024-09-16T03:39:11.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.606 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9400000000000001e-07 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311746+0000", 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2670000000000001e-06 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311752+0000", 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6389999999999997e-06 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311757+0000", 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3639999999999998e-06 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362018+0000", 2024-09-16T03:39:11.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.050260969000000003 2024-09-16T03:39:11.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:11.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362089+0000", 2024-09-16T03:39:11.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0735000000000006e-05 2024-09-16T03:39:11.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918781+0000", 2024-09-16T03:39:11.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55669215400000005 2024-09-16T03:39:11.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918867+0000", 2024-09-16T03:39:11.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6019000000000004e-05 2024-09-16T03:39:11.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918980+0000", 2024-09-16T03:39:11.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011295500000000001 2024-09-16T03:39:11.610 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.610 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.610 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1054 4.10 4:0a55bbf2:::smithi15736696-18:head [write 2633066~610652] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:39:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 100.046468953, 2024-09-16T03:39:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51115411600000005, 2024-09-16T03:39:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1054 2024-09-16T03:39:11.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.612 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:39:11.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:39:11.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513238+0000", 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1999999999999998e-07 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513695+0000", 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045695099999999998 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513720+0000", 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.567e-05 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513725+0000", 2024-09-16T03:39:11.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9899999999999997e-06 2024-09-16T03:39:11.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613892+0000", 2024-09-16T03:39:11.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166253 2024-09-16T03:39:11.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:39:11.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613898+0000", 2024-09-16T03:39:11.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8710000000000002e-06 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916870+0000", 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30297253800000001 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916934+0000", 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3723e-05 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:11.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.917033+0000", 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8908999999999998e-05 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921278+0000", 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0042456999999999998 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019617+0000", 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098338910000000002 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024269+0000", 2024-09-16T03:39:11.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046515519999999998 2024-09-16T03:39:11.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024358+0000", 2024-09-16T03:39:11.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9096000000000002e-05 2024-09-16T03:39:11.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024391+0000", 2024-09-16T03:39:11.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3033e-05 2024-09-16T03:39:11.617 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1055 4.10 4:0a55bbf2:::smithi15736696-18:head [setxattr _header (56) in=7b,truncate 3243718] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 100.045969453, 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61098759599999997, 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.618 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1055 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513737+0000", 2024-09-16T03:39:11.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9399999999999993e-07 2024-09-16T03:39:11.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513759+0000", 2024-09-16T03:39:11.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1903999999999999e-05 2024-09-16T03:39:11.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513773+0000", 2024-09-16T03:39:11.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3244000000000001e-05 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513780+0000", 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999999e-06 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714016+0000", 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20023635000000001 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:39:11.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714024+0000", 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8359999999999992e-06 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918736+0000", 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471236700000001 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918830+0000", 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3769000000000003e-05 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919003+0000", 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000173198 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921589+0000", 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002586419 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019655+0000", 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098065801999999994 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124538+0000", 2024-09-16T03:39:11.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10488272899999999 2024-09-16T03:39:11.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124635+0000", 2024-09-16T03:39:11.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7125000000000003e-05 2024-09-16T03:39:11.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124724+0000", 2024-09-16T03:39:11.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8858999999999997e-05 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1056 4.10 4:0a55bbf2:::smithi15736696-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 100.045916882, 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61111632800000004, 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.625 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1056 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2499999999999995e-07 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513792+0000", 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513797+0000", 2024-09-16T03:39:11.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4940000000000003e-06 2024-09-16T03:39:11.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.628 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513802+0000", 2024-09-16T03:39:11.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5390000000000001e-06 2024-09-16T03:39:11.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.628 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919221+0000", 2024-09-16T03:39:11.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40541913899999998 2024-09-16T03:39:11.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919268+0000", 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7027000000000002e-05 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124766+0000", 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20549842099999999 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124828+0000", 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1803000000000004e-05 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124905+0000", 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7163000000000006e-05 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1702 4.10 4:0a55bbf2:::smithi15736696-18:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 80.870561891999998, 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51800317600000001, 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.630 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1702 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689147+0000", 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689158+0000", 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0426e-05 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689185+0000", 2024-09-16T03:39:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7299000000000001e-05 2024-09-16T03:39:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789328+0000", 2024-09-16T03:39:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100143102 2024-09-16T03:39:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789532+0000", 2024-09-16T03:39:11.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000204317 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789671+0000", 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013852600000000001 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.207147+0000", 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.41747625599999999 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.635 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2160 4.17 4:ed874e32:::smithi15736696-14:head [assert-version v36,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-09-16T03:39:11.635 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:39:11.635 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 64.729560472000003, 2024-09-16T03:39:11.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55377054800000003, 2024-09-16T03:39:11.635 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.635 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.635 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.635 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.635 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.635 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.635 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2160 2024-09-16T03:39:11.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830149+0000", 2024-09-16T03:39:11.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.743e-06 2024-09-16T03:39:11.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830162+0000", 2024-09-16T03:39:11.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3283e-05 2024-09-16T03:39:11.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830191+0000", 2024-09-16T03:39:11.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8466000000000002e-05 2024-09-16T03:39:11.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830207+0000", 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6453e-05 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830362+0000", 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015450699999999999 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830608+0000", 2024-09-16T03:39:11.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024587399999999998 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.233387+0000", 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40277970099999999 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234208+0000", 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082053900000000001 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234370+0000", 2024-09-16T03:39:11.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000161645 2024-09-16T03:39:11.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383821+0000", 2024-09-16T03:39:11.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14945145300000001 2024-09-16T03:39:11.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383867+0000", 2024-09-16T03:39:11.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5498999999999999e-05 2024-09-16T03:39:11.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383916+0000", 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9385000000000002e-05 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2532 4.0 4:06b6d050:::smithi15736696-50:head [write 2603977~725887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.965413083000001, 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53439557900000001, 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.641 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2532 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594298+0000", 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6030000000000004e-06 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594876+0000", 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00057757800000000003 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594885+0000", 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4150000000000005e-06 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594892+0000", 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7229999999999998e-06 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628138+0000", 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033246407999999998 2024-09-16T03:39:11.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628252+0000", 2024-09-16T03:39:11.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113755 2024-09-16T03:39:11.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:11.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628344+0000", 2024-09-16T03:39:11.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2028999999999996e-05 2024-09-16T03:39:11.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730215+0000", 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101870761 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.732105+0000", 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001889951 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128411+0000", 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.396305764 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128611+0000", 2024-09-16T03:39:11.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019990599999999999 2024-09-16T03:39:11.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.647 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.647 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128688+0000", 2024-09-16T03:39:11.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7686000000000001e-05 2024-09-16T03:39:11.647 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.647 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.647 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.647 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2533 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3663926] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:11.648 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:39:11.648 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.964805730000002, 2024-09-16T03:39:11.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53394911599999995, 2024-09-16T03:39:11.648 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.648 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.648 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.648 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.648 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.648 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.648 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2533 2024-09-16T03:39:11.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.648 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:39:11.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:39:11.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594901+0000", 2024-09-16T03:39:11.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6700000000000003e-07 2024-09-16T03:39:11.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594904+0000", 2024-09-16T03:39:11.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1099999999999999e-06 2024-09-16T03:39:11.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594907+0000", 2024-09-16T03:39:11.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1020000000000001e-06 2024-09-16T03:39:11.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594910+0000", 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7889999999999999e-06 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729537+0000", 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.134627725 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729660+0000", 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012283800000000001 2024-09-16T03:39:11.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:11.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729786+0000", 2024-09-16T03:39:11.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012572000000000001 2024-09-16T03:39:11.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730385+0000", 2024-09-16T03:39:11.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059949700000000005 2024-09-16T03:39:11.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.028223+0000", 2024-09-16T03:39:11.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29783727999999998 2024-09-16T03:39:11.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128774+0000", 2024-09-16T03:39:11.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100551027 2024-09-16T03:39:11.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128816+0000", 2024-09-16T03:39:11.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1946000000000001e-05 2024-09-16T03:39:11.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128849+0000", 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3515e-05 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2534 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.964791308000002, 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53406219200000005, 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.654 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2534 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8999999999999998e-07 2024-09-16T03:39:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594916+0000", 2024-09-16T03:39:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.344e-06 2024-09-16T03:39:11.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594919+0000", 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0580000000000002e-06 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594922+0000", 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4389999999999999e-06 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729999+0000", 2024-09-16T03:39:11.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.13507707399999999 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730046+0000", 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7382999999999997e-05 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128879+0000", 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39883298499999997 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128924+0000", 2024-09-16T03:39:11.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:39:11.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128977+0000", 2024-09-16T03:39:11.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2648999999999997e-05 2024-09-16T03:39:11.659 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.659 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.659 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.659 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2621 4.f 4:fa163101:::smithi15736696-10:head [setxattr _header (56) in=7b,truncate 3815189] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:39:11.659 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 51.317014929999999, 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55430782499999998, 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2621 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:39:11.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.661 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:39:11.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.661 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242692+0000", 2024-09-16T03:39:11.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7599999999999996e-07 2024-09-16T03:39:11.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242695+0000", 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.529e-06 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242699+0000", 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1010000000000002e-06 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242703+0000", 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.518e-06 2024-09-16T03:39:11.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.494935+0000", 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25223231200000001 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495011+0000", 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5692999999999995e-05 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495117+0000", 2024-09-16T03:39:11.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000105944 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.596308+0000", 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101190825 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796852+0000", 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20054443199999999 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796893+0000", 2024-09-16T03:39:11.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1221999999999999e-05 2024-09-16T03:39:11.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796946+0000", 2024-09-16T03:39:11.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2435000000000002e-05 2024-09-16T03:39:11.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796999+0000", 2024-09-16T03:39:11.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3038000000000003e-05 2024-09-16T03:39:11.665 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.665 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2622 4.f 4:fa163101:::smithi15736696-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 51.316999074999998, 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55448354399999999, 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.666 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2622 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9999999999999999e-07 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242709+0000", 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.429e-06 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242712+0000", 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6090000000000001e-06 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242716+0000", 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6480000000000001e-06 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495371+0000", 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25265485700000001 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495424+0000", 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2927999999999999e-05 2024-09-16T03:39:11.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.670 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.670 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797042+0000", 2024-09-16T03:39:11.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30161821900000002 2024-09-16T03:39:11.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.670 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.670 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797114+0000", 2024-09-16T03:39:11.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2445000000000002e-05 2024-09-16T03:39:11.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.670 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.670 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797190+0000", 2024-09-16T03:39:11.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6108999999999999e-05 2024-09-16T03:39:11.671 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.671 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.671 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.671 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2941 4.17 4:eb60a28d:::smithi15736696-41:head [setxattr _header (56) in=7b,truncate 2420241] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:39:11.671 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:30.762823+0000", 2024-09-16T03:39:11.671 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 40.796882770000003, 2024-09-16T03:39:11.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51643088500000001, 2024-09-16T03:39:11.671 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.671 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.671 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:11.672 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.672 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.672 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.672 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2941 2024-09-16T03:39:11.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.672 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762823+0000", 2024-09-16T03:39:11.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762823+0000", 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762824+0000", 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3200000000000005e-07 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762826+0000", 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7590000000000002e-06 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762830+0000", 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4879999999999999e-06 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762834+0000", 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7249999999999999e-06 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.763900+0000", 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001066851 2024-09-16T03:39:11.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.763982+0000", 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1225999999999998e-05 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.764130+0000", 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014830999999999999 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.767207+0000", 2024-09-16T03:39:11.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0030773189999999998 2024-09-16T03:39:11.676 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.062538+0000", 2024-09-16T03:39:11.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29533109499999999 2024-09-16T03:39:11.676 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:11.676 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279028+0000", 2024-09-16T03:39:11.676 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.21648964100000001 2024-09-16T03:39:11.676 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279214+0000", 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000185606 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279254+0000", 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0333000000000002e-05 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2942 4.17 4:eb60a28d:::smithi15736696-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:39:11.677 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:30.762852+0000", 2024-09-16T03:39:11.678 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 40.796853577999997, 2024-09-16T03:39:11.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51654379900000003, 2024-09-16T03:39:11.678 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.678 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.678 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:11.678 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.678 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.678 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.678 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2942 2024-09-16T03:39:11.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.678 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762852+0000", 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762852+0000", 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762853+0000", 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762855+0000", 2024-09-16T03:39:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0810000000000001e-06 2024-09-16T03:39:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762860+0000", 2024-09-16T03:39:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8720000000000001e-06 2024-09-16T03:39:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762863+0000", 2024-09-16T03:39:11.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2930000000000001e-06 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.764919+0000", 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020556979999999999 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.764964+0000", 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5788999999999998e-05 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:11.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279291+0000", 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51432677999999998 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279337+0000", 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5892e-05 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279396+0000", 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9018000000000003e-05 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.682 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.683 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.683 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.683 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:11.684 INFO:tasks.rados.rados.0.smithi157.stdout:1712: set_chunk oid 28 offset: 619008 length: 1183232 target oid 13e91a6c9fb63c85209a4d9065cc1e1225929f8ddc4955a7e63272d6c7055a80 offset: 0 snap -1 2024-09-16T03:39:11.685 INFO:tasks.rados.rados.0.smithi157.stdout:1713: delete oid 2 current snap is 0 2024-09-16T03:39:11.685 INFO:tasks.rados.rados.0.smithi157.stdout:1711: expect (ObjNum 342 snap 0 seq_num 342) 2024-09-16T03:39:11.685 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:11.685 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2863756 2024-09-16T03:39:11.685 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:39:11.686 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:11.682+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:11.730 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2671703 2024-09-16T03:39:11.746 INFO:tasks.rados.rados.0.smithi157.stdout:1708: finishing set_chunk to oid 45 2024-09-16T03:39:11.746 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 107 (ObjNum 325 snap 0 seq_num 325) dirty exists 2024-09-16T03:39:11.782 INFO:tasks.rados.rados.0.smithi157.stdout:1708: done (5 left) 2024-09-16T03:39:11.782 INFO:tasks.rados.rados.0.smithi157.stdout:1709: done (4 left) 2024-09-16T03:39:11.782 INFO:tasks.rados.rados.0.smithi157.stdout:1710: done (3 left) 2024-09-16T03:39:11.782 INFO:tasks.rados.rados.0.smithi157.stdout:1711: done (2 left) 2024-09-16T03:39:11.782 INFO:tasks.rados.rados.0.smithi157.stdout:1713: done (1 left) 2024-09-16T03:39:11.782 INFO:tasks.rados.rados.0.smithi157.stdout:1714: read oid 38 snap -1 2024-09-16T03:39:11.782 INFO:tasks.rados.rados.0.smithi157.stdout:1714: expect deleted 2024-09-16T03:39:11.782 INFO:tasks.rados.rados.0.smithi157.stdout:1715: read oid 33 snap -1 2024-09-16T03:39:11.782 INFO:tasks.rados.rados.0.smithi157.stdout:1715: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-16T03:39:11.783 INFO:tasks.rados.rados.0.smithi157.stdout:1716: set_chunk oid 25 target oid 2024-09-16T03:39:11.783 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:11.780+0000 7f49439d4640 -1 osd.4 136 handle_osd_map injecting map crc failure 2024-09-16T03:39:11.785 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:11.784+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:11.831 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:11.831 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:11.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.831 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4083 4.23 4.ce1b48a3 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e137)", 2024-09-16T03:39:11.831 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:11.780716+0000", 2024-09-16T03:39:11.831 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.049263410000000001, 2024-09-16T03:39:11.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.049279539999999997, 2024-09-16T03:39:11.831 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.831 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.831 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "queued for pg", 2024-09-16T03:39:11.832 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.832 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.832 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.832 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4083 2024-09-16T03:39:11.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.832 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.832 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.780716+0000", 2024-09-16T03:39:11.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.780716+0000", 2024-09-16T03:39:11.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.780717+0000", 2024-09-16T03:39:11.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2500000000000005e-07 2024-09-16T03:39:11.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.780737+0000", 2024-09-16T03:39:11.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0055999999999999e-05 2024-09-16T03:39:11.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.834 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.780743+0000", 2024-09-16T03:39:11.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8669999999999999e-06 2024-09-16T03:39:11.834 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.834 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.780748+0000", 2024-09-16T03:39:11.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3240000000000002e-06 2024-09-16T03:39:11.834 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.834 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4084 4.23 4.ce1b48a3 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e137)", 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:11.780770+0000", 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.049209432999999997, 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.049257297999999998, 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "queued for pg", 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:11.835 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4084 2024-09-16T03:39:11.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.836 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:11.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.836 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:11.836 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.780770+0000", 2024-09-16T03:39:11.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.836 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:11.836 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.780770+0000", 2024-09-16T03:39:11.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:11.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.780771+0000", 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1900000000000002e-07 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.780777+0000", 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0430000000000004e-06 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.780784+0000", 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5400000000000001e-06 2024-09-16T03:39:11.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:11.838 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:11.838 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:11.838 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.780798+0000", 2024-09-16T03:39:11.838 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4486e-05 2024-09-16T03:39:11.838 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.838 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:11.839 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.839 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:11.839 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2024-09-16T03:39:11.839 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 2 2024-09-16T03:39:11.839 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:11.840 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:39:11.887 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:11.885+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:11.909 INFO:tasks.rados.rados.0.smithi157.stdout:1716: set_chunk oid 25 offset: 1297920 length: 483840 target oid 76570743831c19cf233e5332003abe4c42a92c02de58717be3bf390d6bb12c48 offset: 0 snap -1 2024-09-16T03:39:11.909 INFO:tasks.rados.rados.0.smithi157.stdout:1717: read oid 1 snap -1 2024-09-16T03:39:11.909 INFO:tasks.rados.rados.0.smithi157.stdout:1717: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-16T03:39:11.909 INFO:tasks.rados.rados.0.smithi157.stdout:1718: write (excl) oid 10 current snap is 0 2024-09-16T03:39:11.909 INFO:tasks.rados.rados.0.smithi157.stdout:1718: seq_num 375 ranges {702625=719672,2127571=175054} 2024-09-16T03:39:11.916 INFO:tasks.rados.rados.0.smithi157.stdout:1718: writing smithi15736696-10 from 702625 to 1422297 tid 1 2024-09-16T03:39:11.922 INFO:tasks.rados.rados.0.smithi157.stdout:1718: writing smithi15736696-10 from 2127571 to 2302625 tid 2 2024-09-16T03:39:11.922 INFO:tasks.rados.rados.0.smithi157.stdout:1719: read oid 7 snap -1 2024-09-16T03:39:11.922 INFO:tasks.rados.rados.0.smithi157.stdout:1719: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-16T03:39:11.922 INFO:tasks.rados.rados.0.smithi157.stdout:1720: write oid 41 current snap is 0 2024-09-16T03:39:11.922 INFO:tasks.rados.rados.0.smithi157.stdout:1720: seq_num 376 ranges {719432=794921,2186053=133379} 2024-09-16T03:39:11.922 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:11.920+0000 7f0ae754b640 -1 osd.0 136 handle_osd_map injecting map crc failure 2024-09-16T03:39:11.930 INFO:tasks.rados.rados.0.smithi157.stdout:1720: writing smithi15736696-41 from 719432 to 1514353 tid 1 2024-09-16T03:39:11.934 INFO:tasks.rados.rados.0.smithi157.stdout:1720: writing smithi15736696-41 from 2186053 to 2319432 tid 2 2024-09-16T03:39:11.935 INFO:tasks.rados.rados.0.smithi157.stdout:1721: read oid 23 snap -1 2024-09-16T03:39:11.935 INFO:tasks.rados.rados.0.smithi157.stdout:1721: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:39:11.935 INFO:tasks.rados.rados.0.smithi157.stdout:1722: write oid 20 current snap is 0 2024-09-16T03:39:11.935 INFO:tasks.rados.rados.0.smithi157.stdout:1722: seq_num 377 ranges {736239=470170,1728182=571782} 2024-09-16T03:39:11.941 INFO:tasks.rados.rados.0.smithi157.stdout:1722: writing smithi15736696-20 from 736239 to 1206409 tid 1 2024-09-16T03:39:11.947 INFO:tasks.rados.rados.0.smithi157.stdout:1722: writing smithi15736696-20 from 1728182 to 2299964 tid 2 2024-09-16T03:39:11.947 INFO:tasks.rados.rados.0.smithi157.stdout:1723: write (excl) oid 5 current snap is 0 2024-09-16T03:39:11.947 INFO:tasks.rados.rados.0.smithi157.stdout:1723: seq_num 378 ranges {753046=545419,1786664=431793} 2024-09-16T03:39:11.948 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:11.948 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:11.948 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:11.948 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:11.948 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:11.954 INFO:tasks.rados.rados.0.smithi157.stdout:1723: writing smithi15736696-5 from 753046 to 1298465 tid 1 2024-09-16T03:39:11.957 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:39:11.959 INFO:tasks.rados.rados.0.smithi157.stdout:1723: writing smithi15736696-5 from 1786664 to 2218457 tid 2 2024-09-16T03:39:11.959 INFO:tasks.rados.rados.0.smithi157.stdout:1724: write (excl) oid 26 current snap is 0 2024-09-16T03:39:11.959 INFO:tasks.rados.rados.0.smithi157.stdout:1724: seq_num 379 ranges {769853=620668,1845146=524707} 2024-09-16T03:39:11.966 INFO:tasks.rados.rados.0.smithi157.stdout:1724: writing smithi15736696-26 from 769853 to 1390521 tid 1 2024-09-16T03:39:11.973 INFO:tasks.rados.rados.0.smithi157.stdout:1724: writing smithi15736696-26 from 1845146 to 2369853 tid 2 2024-09-16T03:39:11.973 INFO:tasks.rados.rados.0.smithi157.stdout:1725: read oid 46 snap -1 2024-09-16T03:39:11.973 INFO:tasks.rados.rados.0.smithi157.stdout:1725: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-16T03:39:11.973 INFO:tasks.rados.rados.0.smithi157.stdout:1726: read oid 43 snap -1 2024-09-16T03:39:11.973 INFO:tasks.rados.rados.0.smithi157.stdout:1726: expect (ObjNum 364 snap 0 seq_num 364) 2024-09-16T03:39:11.973 INFO:tasks.rados.rados.0.smithi157.stdout:1727: read oid 15 snap -1 2024-09-16T03:39:11.973 INFO:tasks.rados.rados.0.smithi157.stdout:1727: expect (ObjNum 334 snap 0 seq_num 334) 2024-09-16T03:39:11.973 INFO:tasks.rados.rados.0.smithi157.stdout:1728: write oid 31 current snap is 0 2024-09-16T03:39:11.973 INFO:tasks.rados.rados.0.smithi157.stdout:1712: finishing set_chunk to oid 28 2024-09-16T03:39:11.973 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 127 (ObjNum 370 snap 0 seq_num 370) dirty exists 2024-09-16T03:39:11.973 INFO:tasks.rados.rados.0.smithi157.stdout:1715: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-16T03:39:11.983 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2682890 2024-09-16T03:39:11.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:39:11.988 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:11.987+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:12.057 INFO:tasks.rados.rados.0.smithi157.stdout:1716: finishing set_chunk to oid 25 2024-09-16T03:39:12.057 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk returned error code -95 offset: 1297920 length: 483840 2024-09-16T03:39:12.057 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 295 snap 0 seq_num 295) dirty exists 2024-09-16T03:39:12.057 INFO:tasks.rados.rados.0.smithi157.stdout:1718: finishing write tid 1 to smithi15736696-10 2024-09-16T03:39:12.058 INFO:tasks.rados.rados.0.smithi157.stdout:1719: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-16T03:39:12.065 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:12.065 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:12.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:12.065 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:12.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.065 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:12.065 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:12.066 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 119.918968664, 2024-09-16T03:39:12.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:39:12.066 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.066 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.066 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.066 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.066 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.066 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:12.066 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:39:12.067 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:39:12.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:39:12.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:39:12.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:39:12.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:39:12.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:39:12.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:39:12.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:39:12.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:39:12.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:39:12.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:39:12.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:39:12.070 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.070 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.070 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.070 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:39:12.070 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:12.070 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 119.918615304, 2024-09-16T03:39:12.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:39:12.070 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:39:12.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:39:12.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:39:12.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:39:12.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:39:12.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:39:12.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:39:12.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:39:12.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:39:12.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:39:12.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:39:12.075 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.075 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.075 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.075 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:39:12.075 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:12.075 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 119.917487506, 2024-09-16T03:39:12.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:39:12.075 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:39:12.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:39:12.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:39:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 119.91650006499999, 2024-09-16T03:39:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:39:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:12.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:39:12.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:39:12.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:39:12.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:39:12.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.082 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.082 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:39:12.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:39:12.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:39:12.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:39:12.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:39:12.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:39:12.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:39:12.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:39:12.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:39:12.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:39:12.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:39:12.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:39:12.084 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.084 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.084 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.085 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:39:12.085 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:12.085 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 119.91574332099999, 2024-09-16T03:39:12.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:39:12.085 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.085 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.085 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.085 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:39:12.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:39:12.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:39:12.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:39:12.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:39:12.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:39:12.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:39:12.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:39:12.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:39:12.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:39:12.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:39:12.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:39:12.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:39:12.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:39:12.089 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.089 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.089 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.090 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:86 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'8, mlcod=39'5)", 2024-09-16T03:39:12.090 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:39:12.090 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 119.454231085, 2024-09-16T03:39:12.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20388281899999999, 2024-09-16T03:39:12.090 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.090 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.090 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.090 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608472+0000", 2024-09-16T03:39:12.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.227e-06 2024-09-16T03:39:12.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608616+0000", 2024-09-16T03:39:12.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000144177 2024-09-16T03:39:12.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608638+0000", 2024-09-16T03:39:12.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1543e-05 2024-09-16T03:39:12.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608645+0000", 2024-09-16T03:39:12.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5839999999999997e-06 2024-09-16T03:39:12.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608740+0000", 2024-09-16T03:39:12.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4716999999999995e-05 2024-09-16T03:39:12.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608763+0000", 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2671e-05 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812264+0000", 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203501194 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812353+0000", 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8706000000000001e-05 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.094 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:87 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'9, mlcod=39'5)", 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 119.45401771100001, 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20372206400000001, 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:39:12.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:39:12.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608684+0000", 2024-09-16T03:39:12.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6700000000000003e-07 2024-09-16T03:39:12.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608689+0000", 2024-09-16T03:39:12.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.412e-06 2024-09-16T03:39:12.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608704+0000", 2024-09-16T03:39:12.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4397e-05 2024-09-16T03:39:12.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608713+0000", 2024-09-16T03:39:12.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:39:12.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710025+0000", 2024-09-16T03:39:12.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101312032 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710042+0000", 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6889e-05 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812361+0000", 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10231889600000001 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812405+0000", 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4101999999999999e-05 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:989 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'31, mlcod=46'28)", 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 101.674891961, 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40332266900000002, 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:12.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:12.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387812+0000", 2024-09-16T03:39:12.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:39:12.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388467+0000", 2024-09-16T03:39:12.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065539500000000002 2024-09-16T03:39:12.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388481+0000", 2024-09-16T03:39:12.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3905e-05 2024-09-16T03:39:12.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488552+0000", 2024-09-16T03:39:12.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10007120899999999 2024-09-16T03:39:12.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688818+0000", 2024-09-16T03:39:12.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20026627599999999 2024-09-16T03:39:12.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688845+0000", 2024-09-16T03:39:12.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:39:12.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791091+0000", 2024-09-16T03:39:12.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10224588699999999 2024-09-16T03:39:12.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791131+0000", 2024-09-16T03:39:12.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0293000000000002e-05 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:990 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'32, mlcod=46'28)", 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 101.57410828899999, 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30256419200000001, 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488594+0000", 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:39:12.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488616+0000", 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2419e-05 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488627+0000", 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1251e-05 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588753+0000", 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012544399999999 2024-09-16T03:39:12.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690679+0000", 2024-09-16T03:39:12.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101926027 2024-09-16T03:39:12.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690695+0000", 2024-09-16T03:39:12.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6528000000000002e-05 2024-09-16T03:39:12.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791133+0000", 2024-09-16T03:39:12.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100438233 2024-09-16T03:39:12.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791157+0000", 2024-09-16T03:39:12.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-05 2024-09-16T03:39:12.109 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.109 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.109 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.109 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2729 4.12 4:4a9d235d:::smithi15736696-1:head [write 1578407~488887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:22.608814+0000", 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.453886562000001, 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203332443, 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2729 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.608814+0000", 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.608814+0000", 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.608816+0000", 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0480000000000001e-06 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609287+0000", 2024-09-16T03:39:12.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000470524 2024-09-16T03:39:12.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609298+0000", 2024-09-16T03:39:12.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1629e-05 2024-09-16T03:39:12.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609306+0000", 2024-09-16T03:39:12.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5480000000000004e-06 2024-09-16T03:39:12.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609354+0000", 2024-09-16T03:39:12.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7688999999999997e-05 2024-09-16T03:39:12.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609448+0000", 2024-09-16T03:39:12.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4060000000000004e-05 2024-09-16T03:39:12.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 2,11", 2024-09-16T03:39:12.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609571+0000", 2024-09-16T03:39:12.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012361699999999999 2024-09-16T03:39:12.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:12.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.611263+0000", 2024-09-16T03:39:12.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016921099999999999 2024-09-16T03:39:12.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.613981+0000", 2024-09-16T03:39:12.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027178340000000001 2024-09-16T03:39:12.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812075+0000", 2024-09-16T03:39:12.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19809358399999999 2024-09-16T03:39:12.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812114+0000", 2024-09-16T03:39:12.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9082000000000003e-05 2024-09-16T03:39:12.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812147+0000", 2024-09-16T03:39:12.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2718000000000001e-05 2024-09-16T03:39:12.116 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.116 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.116 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.117 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2731 4.12 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-09-16T03:39:12.117 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:22.609352+0000", 2024-09-16T03:39:12.117 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 49.453348452, 2024-09-16T03:39:12.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203034889, 2024-09-16T03:39:12.117 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.117 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.117 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:12.117 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:12.117 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:12.117 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:12.117 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2731 2024-09-16T03:39:12.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.117 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609352+0000", 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609352+0000", 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609353+0000", 2024-09-16T03:39:12.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4099999999999999e-07 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609355+0000", 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4439999999999998e-06 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609360+0000", 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.609363+0000", 2024-09-16T03:39:12.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3780000000000001e-06 2024-09-16T03:39:12.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.611279+0000", 2024-09-16T03:39:12.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019158140000000001 2024-09-16T03:39:12.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:12.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.611318+0000", 2024-09-16T03:39:12.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8312999999999997e-05 2024-09-16T03:39:12.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812286+0000", 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20096847600000001 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812332+0000", 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6313000000000002e-05 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:22.812387+0000", 2024-09-16T03:39:12.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4857e-05 2024-09-16T03:39:12.122 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.122 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.122 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.122 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3081 4.1f e102/100 4:fa163101:::smithi15736696-10:head v 102'93, mlcod=102'91)", 2024-09-16T03:39:12.122 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:12.122 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 36.728985821000002, 2024-09-16T03:39:12.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20437049600000001, 2024-09-16T03:39:12.122 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.122 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.122 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.122 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:12.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:12.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333718+0000", 2024-09-16T03:39:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3720000000000001e-06 2024-09-16T03:39:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334338+0000", 2024-09-16T03:39:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061959099999999996 2024-09-16T03:39:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334355+0000", 2024-09-16T03:39:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.677e-05 2024-09-16T03:39:12.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334364+0000", 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3780000000000007e-06 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334424+0000", 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0084000000000003e-05 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334447+0000", 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3113e-05 2024-09-16T03:39:12.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538030+0000", 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20358272399999999 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538085+0000", 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5464000000000003e-05 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.127 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4007 4.37 4:ed874e32:::smithi15736696-14:head [write 601783~551825,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:12.127 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:39:12.127 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.5295019749999996, 2024-09-16T03:39:12.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29242759600000001, 2024-09-16T03:39:12.127 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.127 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.127 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.127 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:12.127 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:12.127 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:12.127 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4007 2024-09-16T03:39:12.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533200+0000", 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3600000000000002e-07 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533841+0000", 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00064141100000000004 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533862+0000", 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0687e-05 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634001+0000", 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100139413 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721351+0000", 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.087349914000000001 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721357+0000", 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7100000000000004e-06 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721891+0000", 2024-09-16T03:39:12.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053461299999999997 2024-09-16T03:39:12.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.722024+0000", 2024-09-16T03:39:12.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013274599999999999 2024-09-16T03:39:12.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:12.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.722158+0000", 2024-09-16T03:39:12.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013362399999999999 2024-09-16T03:39:12.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:12.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725271+0000", 2024-09-16T03:39:12.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003113345 2024-09-16T03:39:12.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825258+0000", 2024-09-16T03:39:12.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099987260999999994 2024-09-16T03:39:12.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825453+0000", 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019453999999999999 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825591+0000", 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013849700000000001 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825626+0000", 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4999000000000002e-05 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.133 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.134 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.134 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4008 4.37 4:ed874e32:::smithi15736696-14:head [write 1776679~425104] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:12.134 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:39:12.134 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.4286818520000004, 2024-09-16T03:39:12.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29371618199999999, 2024-09-16T03:39:12.134 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.134 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.134 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.134 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:12.135 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:12.135 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:12.135 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4008 2024-09-16T03:39:12.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.135 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:39:12.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634021+0000", 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9989999999999998e-06 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634187+0000", 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016655199999999999 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634197+0000", 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7520000000000008e-06 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634205+0000", 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8210000000000002e-06 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725407+0000", 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091202249999999999 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725506+0000", 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8493999999999996e-05 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725648+0000", 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014187299999999999 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.727142+0000", 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001494478 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825761+0000", 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098619340999999999 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927671+0000", 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10190922700000001 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927705+0000", 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4523000000000001e-05 2024-09-16T03:39:12.139 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927735+0000", 2024-09-16T03:39:12.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9872000000000001e-05 2024-09-16T03:39:12.140 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.140 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.140 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.140 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4009 4.37 4:ed874e32:::smithi15736696-14:head [setxattr _header (56) in=7b,truncate 2201783] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:12.141 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:39:12.141 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.4284849250000002, 2024-09-16T03:39:12.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29364004500000002, 2024-09-16T03:39:12.141 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.141 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.141 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.141 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:12.141 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:12.141 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:12.141 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4009 2024-09-16T03:39:12.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.141 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634217+0000", 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7199999999999998e-07 2024-09-16T03:39:12.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634221+0000", 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9650000000000002e-06 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634226+0000", 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7830000000000004e-06 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634232+0000", 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2909999999999998e-06 2024-09-16T03:39:12.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726750+0000", 2024-09-16T03:39:12.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.09251827 2024-09-16T03:39:12.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726830+0000", 2024-09-16T03:39:12.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9808999999999994e-05 2024-09-16T03:39:12.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:12.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726916+0000", 2024-09-16T03:39:12.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6055000000000002e-05 2024-09-16T03:39:12.145 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:12.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.727785+0000", 2024-09-16T03:39:12.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086898599999999998 2024-09-16T03:39:12.145 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.926315+0000", 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19852985000000001 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927777+0000", 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001462138 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927818+0000", 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1006e-05 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927856+0000", 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8120000000000001e-05 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4010 4.37 4:ed874e32:::smithi15736696-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.4284612320000001, 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39386449699999998, 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.147 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:12.148 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:12.148 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:12.148 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:12.148 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4010 2024-09-16T03:39:12.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.148 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:12.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634243+0000", 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.869e-06 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634249+0000", 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9449999999999998e-06 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634255+0000", 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1979999999999998e-06 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.734247+0000", 2024-09-16T03:39:12.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099991570000000002 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.734386+0000", 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013961600000000001 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.027978+0000", 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29359164599999998 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.028036+0000", 2024-09-16T03:39:12.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7802e-05 2024-09-16T03:39:12.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.028104+0000", 2024-09-16T03:39:12.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8264999999999998e-05 2024-09-16T03:39:12.152 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.152 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.152 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.152 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4034 4.17 4:eb60a28d:::smithi15736696-41:head [write 635397~702323,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:39:12.152 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.0260397870000002, 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471451500000001, 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4034 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:39:12.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:39:12.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036663+0000", 2024-09-16T03:39:12.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.855e-06 2024-09-16T03:39:12.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037227+0000", 2024-09-16T03:39:12.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00056451099999999996 2024-09-16T03:39:12.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037240+0000", 2024-09-16T03:39:12.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2208999999999999e-05 2024-09-16T03:39:12.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037245+0000", 2024-09-16T03:39:12.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6319999999999996e-06 2024-09-16T03:39:12.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037305+0000", 2024-09-16T03:39:12.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0272000000000002e-05 2024-09-16T03:39:12.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037384+0000", 2024-09-16T03:39:12.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8330000000000004e-05 2024-09-16T03:39:12.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037491+0000", 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010730699999999999 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.042284+0000", 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0047932649999999997 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.140922+0000", 2024-09-16T03:39:12.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098637552000000003 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241267+0000", 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100344866 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241330+0000", 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2953000000000005e-05 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241375+0000", 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5763000000000002e-05 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4035 4.17 4:eb60a28d:::smithi15736696-41:head [write 1893643~341754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.140882+0000", 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.9218192399999996, 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20336865200000001, 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.159 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4035 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.140882+0000", 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.140882+0000", 2024-09-16T03:39:12.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.140884+0000", 2024-09-16T03:39:12.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1490000000000001e-06 2024-09-16T03:39:12.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141010+0000", 2024-09-16T03:39:12.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000126165 2024-09-16T03:39:12.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141020+0000", 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0030999999999999e-05 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141027+0000", 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2910000000000002e-06 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141063+0000", 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6310000000000003e-05 2024-09-16T03:39:12.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141123+0000", 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9308000000000002e-05 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141224+0000", 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101517 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.143157+0000", 2024-09-16T03:39:12.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001932416 2024-09-16T03:39:12.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.243447+0000", 2024-09-16T03:39:12.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100290165 2024-09-16T03:39:12.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.344011+0000", 2024-09-16T03:39:12.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100563955 2024-09-16T03:39:12.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.344173+0000", 2024-09-16T03:39:12.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016259600000000001 2024-09-16T03:39:12.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.344250+0000", 2024-09-16T03:39:12.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6749000000000007e-05 2024-09-16T03:39:12.165 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.165 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.165 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4036 4.17 4:eb60a28d:::smithi15736696-41:head [setxattr _header (56) in=7b,truncate 2235397] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.141036+0000", 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.9216642860000004, 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20335434499999999, 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4036 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.166 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141036+0000", 2024-09-16T03:39:12.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141036+0000", 2024-09-16T03:39:12.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141037+0000", 2024-09-16T03:39:12.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:39:12.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141040+0000", 2024-09-16T03:39:12.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5840000000000002e-06 2024-09-16T03:39:12.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141043+0000", 2024-09-16T03:39:12.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4850000000000001e-06 2024-09-16T03:39:12.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141046+0000", 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9249999999999999e-06 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.142275+0000", 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001229214 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.142354+0000", 2024-09-16T03:39:12.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9133000000000003e-05 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.142445+0000", 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0727000000000001e-05 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.143417+0000", 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00097157299999999999 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.343863+0000", 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20044679200000001 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.344319+0000", 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045507099999999998 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.344353+0000", 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4940000000000001e-05 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.344391+0000", 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7360999999999997e-05 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4037 4.17 4:eb60a28d:::smithi15736696-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.141061+0000", 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.9216397829999998, 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20354841700000001, 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:12.172 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:12.173 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:12.173 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:12.173 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:12.173 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4037 2024-09-16T03:39:12.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.173 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:12.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:12.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141061+0000", 2024-09-16T03:39:12.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141061+0000", 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141061+0000", 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1199999999999998e-07 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141063+0000", 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9039999999999999e-06 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:12.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141068+0000", 2024-09-16T03:39:12.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3730000000000003e-06 2024-09-16T03:39:12.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:12.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141071+0000", 2024-09-16T03:39:12.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.501e-06 2024-09-16T03:39:12.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.142770+0000", 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016985769999999999 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.142821+0000", 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1394999999999998e-05 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.344429+0000", 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.201608341 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.176 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:12.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.344514+0000", 2024-09-16T03:39:12.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4165999999999993e-05 2024-09-16T03:39:12.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:12.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:12.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:12.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.344609+0000", 2024-09-16T03:39:12.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5748000000000006e-05 2024-09-16T03:39:12.177 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.177 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.177 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.177 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:12.177 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:12.177 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:12.179 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:12.179 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:12.179 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1124360 2024-09-16T03:39:12.179 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 353621 2024-09-16T03:39:12.180 INFO:tasks.rados.rados.0.smithi157.stdout:1721: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:39:12.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:12.180 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:12.181 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:39:12.181 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:12.181 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:12.182 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:12.177+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:12.192 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:12.193 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:12.209 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:12.209 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:12.225 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:12.225 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:12.242 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:12.243 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:12.260 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:12.260 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:12.276 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:12.276 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:12.279 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:12.278+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:12.289 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:12.289 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:12.298 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:39:12.324 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:12.324 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:12.341 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:12.341 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:12.358 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:12.358 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:12.374 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:12.374 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:12.380 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:12.378+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:12.391 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:12.392 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:12.408 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:12.408 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:12.425 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:12.425 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:12.438 INFO:tasks.rados.rados.0.smithi157.stdout:1727: expect (ObjNum 334 snap 0 seq_num 334) 2024-09-16T03:39:12.443 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:12.443 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:12.452 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:39:12.481 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:12.479+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:12.544 INFO:tasks.rados.rados.0.smithi157.stdout:1722: finishing write tid 1 to smithi15736696-20 2024-09-16T03:39:12.544 INFO:tasks.rados.rados.0.smithi157.stdout:1722: finishing write tid 2 to smithi15736696-20 2024-09-16T03:39:12.544 INFO:tasks.rados.rados.0.smithi157.stdout:1722: finishing write tid 3 to smithi15736696-20 2024-09-16T03:39:12.544 INFO:tasks.rados.rados.0.smithi157.stdout:1722: finishing write tid 4 to smithi15736696-20 2024-09-16T03:39:12.544 INFO:tasks.rados.rados.0.smithi157.stdout:1722: oid 20 updating version 0 to 56 2024-09-16T03:39:12.544 INFO:tasks.rados.rados.0.smithi157.stdout:1722: oid 20 version 56 is already newer than 55 2024-09-16T03:39:12.544 INFO:tasks.rados.rados.0.smithi157.stdout:1722: oid 20 updating version 56 to 57 2024-09-16T03:39:12.545 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 57 (ObjNum 377 snap 0 seq_num 377) dirty exists 2024-09-16T03:39:12.545 INFO:tasks.rados.rados.0.smithi157.stdout:1722: left oid 20 (ObjNum 377 snap 0 seq_num 377) 2024-09-16T03:39:12.545 INFO:tasks.rados.rados.0.smithi157.stdout:1728: seq_num 380 ranges {786660=695917,1903628=435462} 2024-09-16T03:39:12.552 INFO:tasks.rados.rados.0.smithi157.stdout:1728: writing smithi15736696-31 from 786660 to 1482577 tid 1 2024-09-16T03:39:12.557 INFO:tasks.rados.rados.0.smithi157.stdout:1728: writing smithi15736696-31 from 1903628 to 2339090 tid 2 2024-09-16T03:39:12.558 INFO:tasks.rados.rados.0.smithi157.stdout:1712: done (15 left) 2024-09-16T03:39:12.558 INFO:tasks.rados.rados.0.smithi157.stdout:1714: done (14 left) 2024-09-16T03:39:12.558 INFO:tasks.rados.rados.0.smithi157.stdout:1715: done (13 left) 2024-09-16T03:39:12.558 INFO:tasks.rados.rados.0.smithi157.stdout:1716: done (12 left) 2024-09-16T03:39:12.558 INFO:tasks.rados.rados.0.smithi157.stdout:1719: done (11 left) 2024-09-16T03:39:12.558 INFO:tasks.rados.rados.0.smithi157.stdout:1721: done (10 left) 2024-09-16T03:39:12.558 INFO:tasks.rados.rados.0.smithi157.stdout:1722: done (9 left) 2024-09-16T03:39:12.558 INFO:tasks.rados.rados.0.smithi157.stdout:1727: done (8 left) 2024-09-16T03:39:12.558 INFO:tasks.rados.rados.0.smithi157.stdout:1729: set_chunk oid 6 target oid 2024-09-16T03:39:12.559 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:12.559 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:12.563 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:39:12.566 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:39:12.569 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:12.568+0000 7f8be76d1640 -1 osd.7 137 handle_osd_map injecting map crc failure 2024-09-16T03:39:12.582 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:12.580+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:12.672 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:12.672 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:12.681 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:39:12.682 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:12.681+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:12.783 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:12.782+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:12.826 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:12.826 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:12.827 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:12.824+0000 7f6528e68640 -1 osd.1 137 handle_osd_map injecting map crc failure 2024-09-16T03:39:12.835 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:39:12.861 INFO:tasks.rados.rados.0.smithi157.stdout:1729: set_chunk oid 6 offset: 2620928 length: 19968 target oid 3c7182533c6c380fb6c7cd19ec3d024deccbd91945ee879a30216fd3c326c6b2 offset: 0 snap -1 2024-09-16T03:39:12.861 INFO:tasks.rados.rados.0.smithi157.stdout:1730: write (excl) oid 3 current snap is 0 2024-09-16T03:39:12.861 INFO:tasks.rados.rados.0.smithi157.stdout:1726: expect (ObjNum 364 snap 0 seq_num 364) 2024-09-16T03:39:12.868 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 792894 2024-09-16T03:39:12.884 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:12.882+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:12.939 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:12.939 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:12.948 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:39:12.985 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:12.983+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:12.991 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1206225 2024-09-16T03:39:13.026 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:39:13.027 INFO:tasks.rados.rados.0.smithi157.stdout:1717: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-16T03:39:13.082 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 546864 2024-09-16T03:39:13.086 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:13.084+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:13.094 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:13.094 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:13.102 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:39:13.128 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 621522 2024-09-16T03:39:13.169 INFO:tasks.rados.rados.0.smithi157.stdout:1725: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-16T03:39:13.186 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:13.184+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:13.232 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:13.233 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:13.241 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:39:13.248 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1950078 2024-09-16T03:39:13.265 INFO:tasks.rados.rados.0.smithi157.stdout:1720: finishing write tid 1 to smithi15736696-41 2024-09-16T03:39:13.265 INFO:tasks.rados.rados.0.smithi157.stdout:1718: finishing write tid 2 to smithi15736696-10 2024-09-16T03:39:13.265 INFO:tasks.rados.rados.0.smithi157.stdout:1718: finishing write tid 3 to smithi15736696-10 2024-09-16T03:39:13.265 INFO:tasks.rados.rados.0.smithi157.stdout:1718: finishing write tid 4 to smithi15736696-10 2024-09-16T03:39:13.265 INFO:tasks.rados.rados.0.smithi157.stdout:1718: oid 10 updating version 0 to 116 2024-09-16T03:39:13.265 INFO:tasks.rados.rados.0.smithi157.stdout:1718: oid 10 version 116 is already newer than 114 2024-09-16T03:39:13.265 INFO:tasks.rados.rados.0.smithi157.stdout:1718: oid 10 version 116 is already newer than 115 2024-09-16T03:39:13.265 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 116 (ObjNum 375 snap 0 seq_num 375) dirty exists 2024-09-16T03:39:13.265 INFO:tasks.rados.rados.0.smithi157.stdout:1718: left oid 10 (ObjNum 375 snap 0 seq_num 375) 2024-09-16T03:39:13.265 INFO:tasks.rados.rados.0.smithi157.stdout:1724: finishing write tid 1 to smithi15736696-26 2024-09-16T03:39:13.265 INFO:tasks.rados.rados.0.smithi157.stdout:1724: finishing write tid 2 to smithi15736696-26 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:1724: finishing write tid 3 to smithi15736696-26 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:1724: finishing write tid 4 to smithi15736696-26 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:1724: oid 26 updating version 0 to 117 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:1724: oid 26 updating version 117 to 118 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:1724: oid 26 updating version 118 to 119 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 119 (ObjNum 379 snap 0 seq_num 379) dirty exists 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:1724: left oid 26 (ObjNum 379 snap 0 seq_num 379) 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:1723: finishing write tid 1 to smithi15736696-5 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:1723: finishing write tid 2 to smithi15736696-5 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:1723: finishing write tid 3 to smithi15736696-5 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:1723: finishing write tid 4 to smithi15736696-5 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:1723: oid 5 updating version 0 to 74 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:1723: oid 5 version 74 is already newer than 73 2024-09-16T03:39:13.266 INFO:tasks.rados.rados.0.smithi157.stdout:1723: oid 5 updating version 74 to 75 2024-09-16T03:39:13.267 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 75 (ObjNum 378 snap 0 seq_num 378) dirty exists 2024-09-16T03:39:13.267 INFO:tasks.rados.rados.0.smithi157.stdout:1723: left oid 5 (ObjNum 378 snap 0 seq_num 378) 2024-09-16T03:39:13.267 INFO:tasks.rados.rados.0.smithi157.stdout:1720: finishing write tid 2 to smithi15736696-41 2024-09-16T03:39:13.267 INFO:tasks.rados.rados.0.smithi157.stdout:1730: seq_num 381 ranges {403467=487519,1562110=579120} 2024-09-16T03:39:13.269 INFO:tasks.rados.rados.0.smithi157.stdout:1730: writing smithi15736696-3 from 403467 to 890986 tid 1 2024-09-16T03:39:13.277 INFO:tasks.rados.rados.0.smithi157.stdout:1730: writing smithi15736696-3 from 1562110 to 2141230 tid 2 2024-09-16T03:39:13.277 INFO:tasks.rados.rados.0.smithi157.stdout:1717: done (9 left) 2024-09-16T03:39:13.277 INFO:tasks.rados.rados.0.smithi157.stdout:1718: done (8 left) 2024-09-16T03:39:13.277 INFO:tasks.rados.rados.0.smithi157.stdout:1723: done (7 left) 2024-09-16T03:39:13.277 INFO:tasks.rados.rados.0.smithi157.stdout:1724: done (6 left) 2024-09-16T03:39:13.277 INFO:tasks.rados.rados.0.smithi157.stdout:1725: done (5 left) 2024-09-16T03:39:13.277 INFO:tasks.rados.rados.0.smithi157.stdout:1726: done (4 left) 2024-09-16T03:39:13.278 INFO:tasks.rados.rados.0.smithi157.stdout:1731: write (excl) oid 43 current snap is 0 2024-09-16T03:39:13.278 INFO:tasks.rados.rados.0.smithi157.stdout:1720: finishing write tid 3 to smithi15736696-41 2024-09-16T03:39:13.278 INFO:tasks.rados.rados.0.smithi157.stdout:1731: seq_num 382 ranges {420274=562768,1620592=439131} 2024-09-16T03:39:13.282 INFO:tasks.rados.rados.0.smithi157.stdout:1731: writing smithi15736696-43 from 420274 to 983042 tid 1 2024-09-16T03:39:13.287 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:13.285+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:13.289 INFO:tasks.rados.rados.0.smithi157.stdout:1731: writing smithi15736696-43 from 1620592 to 2059723 tid 2 2024-09-16T03:39:13.289 INFO:tasks.rados.rados.0.smithi157.stdout:1732: read oid 12 snap -1 2024-09-16T03:39:13.289 INFO:tasks.rados.rados.0.smithi157.stdout:1732: expect (ObjNum 297 snap 0 seq_num 297) 2024-09-16T03:39:13.289 INFO:tasks.rados.rados.0.smithi157.stdout:1720: finishing write tid 4 to smithi15736696-41 2024-09-16T03:39:13.289 INFO:tasks.rados.rados.0.smithi157.stdout:1720: oid 41 updating version 0 to 54 2024-09-16T03:39:13.289 INFO:tasks.rados.rados.0.smithi157.stdout:1720: oid 41 updating version 54 to 55 2024-09-16T03:39:13.289 INFO:tasks.rados.rados.0.smithi157.stdout:1720: oid 41 version 55 is already newer than 53 2024-09-16T03:39:13.289 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 55 (ObjNum 376 snap 0 seq_num 376) dirty exists 2024-09-16T03:39:13.289 INFO:tasks.rados.rados.0.smithi157.stdout:1720: left oid 41 (ObjNum 376 snap 0 seq_num 376) 2024-09-16T03:39:13.290 INFO:tasks.rados.rados.0.smithi157.stdout:1728: finishing write tid 1 to smithi15736696-31 2024-09-16T03:39:13.290 INFO:tasks.rados.rados.0.smithi157.stdout:1728: finishing write tid 2 to smithi15736696-31 2024-09-16T03:39:13.290 INFO:tasks.rados.rados.0.smithi157.stdout:1728: finishing write tid 3 to smithi15736696-31 2024-09-16T03:39:13.290 INFO:tasks.rados.rados.0.smithi157.stdout:1733: read oid 26 snap -1 2024-09-16T03:39:13.290 INFO:tasks.rados.rados.0.smithi157.stdout:1733: expect (ObjNum 379 snap 0 seq_num 379) 2024-09-16T03:39:13.290 INFO:tasks.rados.rados.0.smithi157.stdout:1720: done (6 left) 2024-09-16T03:39:13.290 INFO:tasks.rados.rados.0.smithi157.stdout:1734: read oid 21 snap -1 2024-09-16T03:39:13.290 INFO:tasks.rados.rados.0.smithi157.stdout:1734: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-16T03:39:13.290 INFO:tasks.rados.rados.0.smithi157.stdout:1735: write oid 46 current snap is 0 2024-09-16T03:39:13.290 INFO:tasks.rados.rados.0.smithi157.stdout:1728: finishing write tid 4 to smithi15736696-31 2024-09-16T03:39:13.290 INFO:tasks.rados.rados.0.smithi157.stdout:1728: oid 31 updating version 0 to 80 2024-09-16T03:39:13.290 INFO:tasks.rados.rados.0.smithi157.stdout:1728: oid 31 updating version 80 to 82 2024-09-16T03:39:13.290 INFO:tasks.rados.rados.0.smithi157.stdout:1728: oid 31 version 82 is already newer than 81 2024-09-16T03:39:13.291 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 82 (ObjNum 380 snap 0 seq_num 380) dirty exists 2024-09-16T03:39:13.291 INFO:tasks.rados.rados.0.smithi157.stdout:1728: left oid 31 (ObjNum 380 snap 0 seq_num 380) 2024-09-16T03:39:13.291 INFO:tasks.rados.rados.0.smithi157.stdout:1735: seq_num 383 ranges {437081=638017,1679074=582789} 2024-09-16T03:39:13.294 INFO:tasks.rados.rados.0.smithi157.stdout:1735: writing smithi15736696-46 from 437081 to 1075098 tid 1 2024-09-16T03:39:13.301 INFO:tasks.rados.rados.0.smithi157.stdout:1735: writing smithi15736696-46 from 1679074 to 2261863 tid 2 2024-09-16T03:39:13.301 INFO:tasks.rados.rados.0.smithi157.stdout:1728: done (7 left) 2024-09-16T03:39:13.301 INFO:tasks.rados.rados.0.smithi157.stdout:1736: tier_evict oid 4 2024-09-16T03:39:13.301 INFO:tasks.rados.rados.0.smithi157.stdout:1736: tier_evict oid 4 snap -1 2024-09-16T03:39:13.301 INFO:tasks.rados.rados.0.smithi157.stdout:1733: expect (ObjNum 379 snap 0 seq_num 379) 2024-09-16T03:39:13.309 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1182298 2024-09-16T03:39:13.327 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:13.328 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:39:13.336 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:13.338 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-16T03:39:12.569206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":210343113,"num_objects":139,"num_object_clones":0,"num_object_copies":417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15,"num_objects_unfound":0,"num_objects_dirty":138,"num_whiteouts":0,"num_read":1965,"num_read_kb":923329,"num_write":1737,"num_write_kb":376263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":62102448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":207,"acting":207,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":218,"num_osds":13,"num_per_pool_osds":13,"num_per_pool_omap_osds":13,"kb":1226833920,"kb_used":799652,"kb_used_data":389528,"kb_used_omap":20,"kb_used_meta":405163,"kb_avail":1226034268,"statfs":{"total":1256277934080,"available":1255459090432,"internally_reserved":0,"allocated":398876672,"data_stored":406217072,"data_compressed":11089660,"data_compressed_allocated":11980800,"data_compressed_original":26054656,"omap_allocated":20735,"internal_metadata":414887681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4913234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":131,"num_read_kb":122405,"num_write":26,"num_write_kb":5656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":230000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032282"},"pg_stats":[{"pgid":"4.37","version":"131'52","reported_seq":397,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641567+0000","last_change":"2024-09-16T03:39:08.371859+0000","last_active":"2024-09-16T03:39:08.641567+0000","last_peered":"2024-09-16T03:39:08.641567+0000","last_clean":"2024-09-16T03:39:08.641567+0000","last_became_active":"2024-09-16T03:39:05.721626+0000","last_became_peered":"2024-09-16T03:39:05.721626+0000","last_unstale":"2024-09-16T03:39:08.641567+0000","last_undegraded":"2024-09-16T03:39:08.641567+0000","last_fullsized":"2024-09-16T03:39:08.641567+0000","mapping_epoch":130,"log_start":"104'45","ondisk_log_start":"104'45","created":126,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":44,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:36.963537+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2890887,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":44517,"num_write":12,"num_write_kb":3069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1148090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.36","version":"131'93","reported_seq":472,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680959+0000","last_change":"2024-09-16T03:39:03.716194+0000","last_active":"2024-09-16T03:39:11.680959+0000","last_peered":"2024-09-16T03:39:11.680959+0000","last_clean":"2024-09-16T03:39:11.680959+0000","last_became_active":"2024-09-16T03:39:02.099320+0000","last_became_peered":"2024-09-16T03:39:02.099320+0000","last_unstale":"2024-09-16T03:39:11.680959+0000","last_undegraded":"2024-09-16T03:39:11.680959+0000","last_fullsized":"2024-09-16T03:39:11.680959+0000","mapping_epoch":126,"log_start":"130'90","ondisk_log_start":"130'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.716125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.716125+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":88,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.328700+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269209,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":4665,"num_write":8,"num_write_kb":2101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.35","version":"131'66","reported_seq":456,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.529794+0000","last_change":"2024-09-16T03:39:08.541599+0000","last_active":"2024-09-16T03:39:08.541599+0000","last_peered":"2024-09-16T03:39:08.529794+0000","last_clean":"2024-09-16T03:39:08.529794+0000","last_became_active":"2024-09-16T03:39:05.722545+0000","last_became_peered":"2024-09-16T03:39:05.722545+0000","last_unstale":"2024-09-16T03:39:08.529794+0000","last_undegraded":"2024-09-16T03:39:08.529794+0000","last_fullsized":"2024-09-16T03:39:08.529794+0000","mapping_epoch":134,"log_start":"123'63","ondisk_log_start":"123'63","created":126,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916015,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84364,"num_write":19,"num_write_kb":5459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3131048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,9,10],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"104'20","reported_seq":327,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T03:39:10.547004+0000","last_change":"2024-09-16T03:39:02.919356+0000","last_active":"2024-09-16T03:39:10.547004+0000","last_peered":"2024-09-16T03:39:10.547004+0000","last_clean":"2024-09-16T03:39:10.547004+0000","last_became_active":"2024-09-16T03:39:01.699286+0000","last_became_peered":"2024-09-16T03:39:01.699286+0000","last_unstale":"2024-09-16T03:39:10.547004+0000","last_undegraded":"2024-09-16T03:39:10.547004+0000","last_fullsized":"2024-09-16T03:39:10.547004+0000","mapping_epoch":126,"log_start":"98'17","ondisk_log_start":"98'17","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:02.919259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.919259+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.209096+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2705960,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"108'72","reported_seq":462,"reported_epoch":134,"state":"remapped+peering","last_fresh":"2024-09-16T03:39:08.538848+0000","last_change":"2024-09-16T03:39:09.647095+0000","last_active":"2024-09-16T03:39:08.538848+0000","last_peered":"2024-09-16T03:39:08.110144+0000","last_clean":"2024-09-16T03:39:08.110144+0000","last_became_active":"2024-09-16T03:39:01.642310+0000","last_became_peered":"2024-09-16T03:39:01.642310+0000","last_unstale":"2024-09-16T03:39:08.538848+0000","last_undegraded":"2024-09-16T03:39:08.538848+0000","last_fullsized":"2024-09-16T03:39:08.538848+0000","mapping_epoch":135,"log_start":"108'70","ondisk_log_start":"108'70","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'72","last_scrub_stamp":"2024-09-16T03:39:02.143339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.143339+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":68,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2722767,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":11293,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,9,14],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.32","version":"106'17","reported_seq":355,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.471820+0000","last_change":"2024-09-16T03:39:08.648217+0000","last_active":"2024-09-16T03:39:08.648217+0000","last_peered":"2024-09-16T03:39:08.471820+0000","last_clean":"2024-09-16T03:39:08.471820+0000","last_became_active":"2024-09-16T03:39:05.823393+0000","last_became_peered":"2024-09-16T03:39:05.823393+0000","last_unstale":"2024-09-16T03:39:08.471820+0000","last_undegraded":"2024-09-16T03:39:08.471820+0000","last_fullsized":"2024-09-16T03:39:08.471820+0000","mapping_epoch":134,"log_start":"82'13","ondisk_log_start":"82'13","created":126,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1033647,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":26745,"num_write":6,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1331059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,9,11],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.31","version":"114'17","reported_seq":297,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T03:39:10.646986+0000","last_change":"2024-09-16T03:39:05.622532+0000","last_active":"2024-09-16T03:39:10.646986+0000","last_peered":"2024-09-16T03:39:10.646986+0000","last_clean":"2024-09-16T03:39:10.646986+0000","last_became_active":"2024-09-16T03:39:01.997491+0000","last_became_peered":"2024-09-16T03:39:01.997491+0000","last_unstale":"2024-09-16T03:39:10.646986+0000","last_undegraded":"2024-09-16T03:39:10.646986+0000","last_fullsized":"2024-09-16T03:39:10.646986+0000","mapping_epoch":126,"log_start":"107'14","ondisk_log_start":"107'14","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'9","last_scrub_stamp":"2024-09-16T03:37:50.756120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:50.756120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.630555+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1495840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":28490,"num_write":4,"num_write_kb":1066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"118'64","reported_seq":392,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.615533+0000","last_change":"2024-09-16T03:39:05.621064+0000","last_active":"2024-09-16T03:39:08.615533+0000","last_peered":"2024-09-16T03:39:08.615533+0000","last_clean":"2024-09-16T03:39:08.615533+0000","last_became_active":"2024-09-16T03:39:01.901037+0000","last_became_peered":"2024-09-16T03:39:01.901037+0000","last_unstale":"2024-09-16T03:39:08.615533+0000","last_undegraded":"2024-09-16T03:39:08.615533+0000","last_fullsized":"2024-09-16T03:39:08.615533+0000","mapping_epoch":126,"log_start":"113'61","ondisk_log_start":"113'61","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:35.157846+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1680717,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":23268,"num_write":10,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":918091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2f","version":"113'94","reported_seq":462,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.552454+0000","last_change":"2024-09-16T03:39:03.319075+0000","last_active":"2024-09-16T03:39:08.552454+0000","last_peered":"2024-09-16T03:39:08.552454+0000","last_clean":"2024-09-16T03:39:08.552454+0000","last_became_active":"2024-09-16T03:39:01.700988+0000","last_became_peered":"2024-09-16T03:39:01.700988+0000","last_unstale":"2024-09-16T03:39:08.552454+0000","last_undegraded":"2024-09-16T03:39:08.552454+0000","last_fullsized":"2024-09-16T03:39:08.552454+0000","mapping_epoch":126,"log_start":"107'90","ondisk_log_start":"107'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"113'94","last_scrub_stamp":"2024-09-16T03:39:03.319012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.319012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":87,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:13.773817+0000","scrub_duration":1038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2e","version":"123'75","reported_seq":447,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.546559+0000","last_change":"2024-09-16T03:39:04.235234+0000","last_active":"2024-09-16T03:39:09.546559+0000","last_peered":"2024-09-16T03:39:09.546559+0000","last_clean":"2024-09-16T03:39:09.546559+0000","last_became_active":"2024-09-16T03:39:01.999291+0000","last_became_peered":"2024-09-16T03:39:01.999291+0000","last_unstale":"2024-09-16T03:39:09.546559+0000","last_undegraded":"2024-09-16T03:39:09.546559+0000","last_fullsized":"2024-09-16T03:39:09.546559+0000","mapping_epoch":126,"log_start":"114'72","ondisk_log_start":"114'72","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"123'75","last_scrub_stamp":"2024-09-16T03:39:04.235141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.235141+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":72,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.010761+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":13153,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"131'99","reported_seq":469,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.652681+0000","last_change":"2024-09-16T03:39:03.917422+0000","last_active":"2024-09-16T03:39:08.652681+0000","last_peered":"2024-09-16T03:39:08.652681+0000","last_clean":"2024-09-16T03:39:08.652681+0000","last_became_active":"2024-09-16T03:39:01.801112+0000","last_became_peered":"2024-09-16T03:39:01.801112+0000","last_unstale":"2024-09-16T03:39:08.652681+0000","last_undegraded":"2024-09-16T03:39:08.652681+0000","last_fullsized":"2024-09-16T03:39:08.652681+0000","mapping_epoch":32,"log_start":"128'96","ondisk_log_start":"128'96","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'97","last_scrub_stamp":"2024-09-16T03:39:03.917355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.917355+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":93,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.686164+0000","scrub_duration":771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2033713,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.e","version":"127'79","reported_seq":443,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.547110+0000","last_change":"2024-09-16T03:39:05.725455+0000","last_active":"2024-09-16T03:39:09.547110+0000","last_peered":"2024-09-16T03:39:09.547110+0000","last_clean":"2024-09-16T03:39:09.547110+0000","last_became_active":"2024-09-16T03:39:01.999282+0000","last_became_peered":"2024-09-16T03:39:01.999282+0000","last_unstale":"2024-09-16T03:39:09.547110+0000","last_undegraded":"2024-09-16T03:39:09.547110+0000","last_fullsized":"2024-09-16T03:39:09.547110+0000","mapping_epoch":32,"log_start":"123'75","ondisk_log_start":"123'75","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:39.194237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.194237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":72,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.946501+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3705976,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":28930,"num_write":10,"num_write_kb":3507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"124'19","reported_seq":346,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.548749+0000","last_change":"2024-09-16T03:39:01.149616+0000","last_active":"2024-09-16T03:39:08.548749+0000","last_peered":"2024-09-16T03:39:08.548749+0000","last_clean":"2024-09-16T03:39:08.548749+0000","last_became_active":"2024-09-16T03:38:45.622965+0000","last_became_peered":"2024-09-16T03:38:45.622965+0000","last_unstale":"2024-09-16T03:39:08.548749+0000","last_undegraded":"2024-09-16T03:39:08.548749+0000","last_fullsized":"2024-09-16T03:39:08.548749+0000","mapping_epoch":72,"log_start":"118'16","ondisk_log_start":"118'16","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'19","last_scrub_stamp":"2024-09-16T03:39:01.149486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.149486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":16,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.574017+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613538,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"130'65","reported_seq":409,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:07.529674+0000","last_change":"2024-09-16T03:39:08.548464+0000","last_active":"2024-09-16T03:39:08.548464+0000","last_peered":"2024-09-16T03:39:07.529674+0000","last_clean":"2024-09-16T03:39:07.529674+0000","last_became_active":"2024-09-16T03:38:54.765107+0000","last_became_peered":"2024-09-16T03:38:54.765107+0000","last_unstale":"2024-09-16T03:39:07.529674+0000","last_undegraded":"2024-09-16T03:39:07.529674+0000","last_fullsized":"2024-09-16T03:39:07.529674+0000","mapping_epoch":134,"log_start":"122'62","ondisk_log_start":"122'62","created":32,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'57","last_scrub_stamp":"2024-09-16T03:38:48.923184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.923184+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":2798,"num_write":9,"num_write_kb":2888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[9,11,15],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"129'132","reported_seq":604,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680864+0000","last_change":"2024-09-16T03:39:09.748835+0000","last_active":"2024-09-16T03:39:11.680864+0000","last_peered":"2024-09-16T03:39:11.680864+0000","last_clean":"2024-09-16T03:39:11.680864+0000","last_became_active":"2024-09-16T03:39:09.748391+0000","last_became_peered":"2024-09-16T03:39:09.748391+0000","last_unstale":"2024-09-16T03:39:11.680864+0000","last_undegraded":"2024-09-16T03:39:11.680864+0000","last_fullsized":"2024-09-16T03:39:11.680864+0000","mapping_epoch":134,"log_start":"127'129","ondisk_log_start":"127'129","created":32,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:46.855462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.855462+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":125,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:12.773343+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2084134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":16397,"num_write":31,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.a","version":"128'148","reported_seq":687,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.547162+0000","last_change":"2024-09-16T03:39:01.504751+0000","last_active":"2024-09-16T03:39:09.547162+0000","last_peered":"2024-09-16T03:39:09.547162+0000","last_clean":"2024-09-16T03:39:09.547162+0000","last_became_active":"2024-09-16T03:38:46.021805+0000","last_became_peered":"2024-09-16T03:38:46.021805+0000","last_unstale":"2024-09-16T03:39:09.547162+0000","last_undegraded":"2024-09-16T03:39:09.547162+0000","last_fullsized":"2024-09-16T03:39:09.547162+0000","mapping_epoch":32,"log_start":"123'145","ondisk_log_start":"123'145","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'147","last_scrub_stamp":"2024-09-16T03:39:01.504663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.504663+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":142,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:18.060284+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13949942,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":53,"num_read_kb":53989,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":248,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641520+0000","last_change":"2024-09-16T03:38:46.881357+0000","last_active":"2024-09-16T03:39:08.641520+0000","last_peered":"2024-09-16T03:39:08.641520+0000","last_clean":"2024-09-16T03:39:08.641520+0000","last_became_active":"2024-09-16T03:38:45.728494+0000","last_became_peered":"2024-09-16T03:38:45.728494+0000","last_unstale":"2024-09-16T03:39:08.641520+0000","last_undegraded":"2024-09-16T03:39:08.641520+0000","last_fullsized":"2024-09-16T03:39:08.641520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:46.881248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.881248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:04.991122+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.8","version":"130'74","reported_seq":454,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.372137+0000","last_change":"2024-09-16T03:39:08.542925+0000","last_active":"2024-09-16T03:39:08.542925+0000","last_peered":"2024-09-16T03:39:08.372137+0000","last_clean":"2024-09-16T03:39:08.372137+0000","last_became_active":"2024-09-16T03:38:55.664848+0000","last_became_peered":"2024-09-16T03:38:55.664848+0000","last_unstale":"2024-09-16T03:39:08.372137+0000","last_undegraded":"2024-09-16T03:39:08.372137+0000","last_fullsized":"2024-09-16T03:39:08.372137+0000","mapping_epoch":134,"log_start":"124'71","ondisk_log_start":"124'71","created":32,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'71","last_scrub_stamp":"2024-09-16T03:39:02.002247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.002247+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":69,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2184976,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,9,10],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"130'85","reported_seq":437,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.681102+0000","last_change":"2024-09-16T03:39:00.900732+0000","last_active":"2024-09-16T03:39:11.681102+0000","last_peered":"2024-09-16T03:39:11.681102+0000","last_clean":"2024-09-16T03:39:11.681102+0000","last_became_active":"2024-09-16T03:38:45.523605+0000","last_became_peered":"2024-09-16T03:38:45.523605+0000","last_unstale":"2024-09-16T03:39:11.681102+0000","last_undegraded":"2024-09-16T03:39:11.681102+0000","last_fullsized":"2024-09-16T03:39:11.681102+0000","mapping_epoch":32,"log_start":"116'82","ondisk_log_start":"116'82","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'82","last_scrub_stamp":"2024-09-16T03:39:00.900619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.900619+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":80,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.385837+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2117748,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":429,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.760857+0000","last_change":"2024-09-16T03:38:36.757156+0000","last_active":"2024-09-16T03:39:11.760857+0000","last_peered":"2024-09-16T03:39:11.760857+0000","last_clean":"2024-09-16T03:39:11.760857+0000","last_became_active":"2024-09-16T03:38:36.756855+0000","last_became_peered":"2024-09-16T03:38:36.756855+0000","last_unstale":"2024-09-16T03:39:11.760857+0000","last_undegraded":"2024-09-16T03:39:11.760857+0000","last_fullsized":"2024-09-16T03:39:11.760857+0000","mapping_epoch":103,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:38:28.063690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:38:28.063690+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:28.682469+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":126,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.541136+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:39:08.541136+0000","last_peered":"2024-09-16T03:39:08.541136+0000","last_clean":"2024-09-16T03:39:08.541136+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:39:08.541136+0000","last_undegraded":"2024-09-16T03:39:08.541136+0000","last_fullsized":"2024-09-16T03:39:08.541136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"131'260","reported_seq":571,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.660977+0000","last_change":"2024-09-16T03:38:01.603352+0000","last_active":"2024-09-16T03:39:11.660977+0000","last_peered":"2024-09-16T03:39:11.660977+0000","last_clean":"2024-09-16T03:39:11.660977+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:39:11.660977+0000","last_undegraded":"2024-09-16T03:39:11.660977+0000","last_fullsized":"2024-09-16T03:39:11.660977+0000","mapping_epoch":74,"log_start":"129'257","ondisk_log_start":"129'257","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":266,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.902901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22624199,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":247,"num_read_kb":8660,"num_write":324,"num_write_kb":70574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.6","version":"108'44","reported_seq":382,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.552545+0000","last_change":"2024-09-16T03:38:48.279289+0000","last_active":"2024-09-16T03:39:08.552545+0000","last_peered":"2024-09-16T03:39:08.552545+0000","last_clean":"2024-09-16T03:39:08.552545+0000","last_became_active":"2024-09-16T03:38:45.924286+0000","last_became_peered":"2024-09-16T03:38:45.924286+0000","last_unstale":"2024-09-16T03:39:08.552545+0000","last_undegraded":"2024-09-16T03:39:08.552545+0000","last_fullsized":"2024-09-16T03:39:08.552545+0000","mapping_epoch":70,"log_start":"105'41","ondisk_log_start":"105'41","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:48.279224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.279224+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.718165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":264,"reported_epoch":135,"state":"active","last_fresh":"2024-09-16T03:39:09.649028+0000","last_change":"2024-09-16T03:39:09.649028+0000","last_active":"2024-09-16T03:39:09.649028+0000","last_peered":"2024-09-16T03:39:09.649028+0000","last_clean":"2024-09-16T03:39:08.025961+0000","last_became_active":"2024-09-16T03:39:09.649028+0000","last_became_peered":"2024-09-16T03:39:09.649028+0000","last_unstale":"2024-09-16T03:39:09.649028+0000","last_undegraded":"2024-09-16T03:39:09.649028+0000","last_fullsized":"2024-09-16T03:39:09.649028+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:24.358339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:24.358339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":["8","6","9","10"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"133'220","reported_seq":462,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.562283+0000","last_change":"2024-09-16T03:38:15.890094+0000","last_active":"2024-09-16T03:39:11.562283+0000","last_peered":"2024-09-16T03:39:11.562283+0000","last_clean":"2024-09-16T03:39:11.562283+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:39:11.562283+0000","last_undegraded":"2024-09-16T03:39:11.562283+0000","last_fullsized":"2024-09-16T03:39:11.562283+0000","mapping_epoch":31,"log_start":"131'217","ondisk_log_start":"131'217","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'125","last_scrub_stamp":"2024-09-16T03:38:15.889996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.889996+0000","objects_scrubbed":18,"log_size":3,"log_dups_size":228,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:35.350472+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16331162,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":194,"num_read_kb":2025,"num_write":277,"num_write_kb":58956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"127'76","reported_seq":420,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.752847+0000","last_change":"2024-09-16T03:39:01.297447+0000","last_active":"2024-09-16T03:39:08.752847+0000","last_peered":"2024-09-16T03:39:08.752847+0000","last_clean":"2024-09-16T03:39:08.752847+0000","last_became_active":"2024-09-16T03:38:45.723754+0000","last_became_peered":"2024-09-16T03:38:45.723754+0000","last_unstale":"2024-09-16T03:39:08.752847+0000","last_undegraded":"2024-09-16T03:39:08.752847+0000","last_fullsized":"2024-09-16T03:39:08.752847+0000","mapping_epoch":32,"log_start":"127'73","ondisk_log_start":"127'73","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'72","last_scrub_stamp":"2024-09-16T03:39:01.297324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.297324+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":71,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.355718+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7294304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":4176,"num_write":5,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":244,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680762+0000","last_change":"2024-09-16T03:38:15.949627+0000","last_active":"2024-09-16T03:39:11.680762+0000","last_peered":"2024-09-16T03:39:11.680762+0000","last_clean":"2024-09-16T03:39:11.680762+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:39:11.680762+0000","last_undegraded":"2024-09-16T03:39:11.680762+0000","last_fullsized":"2024-09-16T03:39:11.680762+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:15.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:24.286324+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"129'252","reported_seq":620,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.630052+0000","last_change":"2024-09-16T03:38:59.856507+0000","last_active":"2024-09-16T03:39:03.630052+0000","last_peered":"2024-09-16T03:39:03.630052+0000","last_clean":"2024-09-16T03:39:03.630052+0000","last_became_active":"2024-09-16T03:38:59.854999+0000","last_became_peered":"2024-09-16T03:38:59.854999+0000","last_unstale":"2024-09-16T03:39:03.630052+0000","last_undegraded":"2024-09-16T03:39:03.630052+0000","last_fullsized":"2024-09-16T03:39:03.630052+0000","mapping_epoch":124,"log_start":"128'249","ondisk_log_start":"128'249","created":31,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":270,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17434102,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":231,"num_read_kb":12817,"num_write":311,"num_write_kb":78387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":27250603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"105'32","reported_seq":339,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.660984+0000","last_change":"2024-09-16T03:38:47.042020+0000","last_active":"2024-09-16T03:39:11.660984+0000","last_peered":"2024-09-16T03:39:11.660984+0000","last_clean":"2024-09-16T03:39:11.660984+0000","last_became_active":"2024-09-16T03:38:45.820169+0000","last_became_peered":"2024-09-16T03:38:45.820169+0000","last_unstale":"2024-09-16T03:39:11.660984+0000","last_undegraded":"2024-09-16T03:39:11.660984+0000","last_fullsized":"2024-09-16T03:39:11.660984+0000","mapping_epoch":32,"log_start":"83'29","ondisk_log_start":"83'29","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.041851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.041851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.311785+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":244,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.647284+0000","last_change":"2024-09-16T03:38:12.795978+0000","last_active":"2024-09-16T03:39:09.647284+0000","last_peered":"2024-09-16T03:39:09.647284+0000","last_clean":"2024-09-16T03:39:09.647284+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:39:09.647284+0000","last_undegraded":"2024-09-16T03:39:09.647284+0000","last_fullsized":"2024-09-16T03:39:09.647284+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:38:12.795855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.795855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:13.088740+0000","scrub_duration":501,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"134'277","reported_seq":522,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.681013+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:39:11.681013+0000","last_peered":"2024-09-16T03:39:11.681013+0000","last_clean":"2024-09-16T03:39:11.681013+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:39:11.681013+0000","last_undegraded":"2024-09-16T03:39:11.681013+0000","last_fullsized":"2024-09-16T03:39:11.681013+0000","mapping_epoch":31,"log_start":"131'274","ondisk_log_start":"131'274","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":286,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.679811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24729040,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":251,"num_read_kb":6269,"num_write":346,"num_write_kb":73722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":278,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680880+0000","last_change":"2024-09-16T03:38:35.828277+0000","last_active":"2024-09-16T03:39:11.680880+0000","last_peered":"2024-09-16T03:39:11.680880+0000","last_clean":"2024-09-16T03:39:11.680880+0000","last_became_active":"2024-09-16T03:38:33.778361+0000","last_became_peered":"2024-09-16T03:38:33.778361+0000","last_unstale":"2024-09-16T03:39:11.680880+0000","last_undegraded":"2024-09-16T03:39:11.680880+0000","last_fullsized":"2024-09-16T03:39:11.680880+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:35.828203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.828203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:49.130136+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":247,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.662354+0000","last_change":"2024-09-16T03:37:52.652365+0000","last_active":"2024-09-16T03:39:11.662354+0000","last_peered":"2024-09-16T03:39:11.662354+0000","last_clean":"2024-09-16T03:39:11.662354+0000","last_became_active":"2024-09-16T03:37:52.651907+0000","last_became_peered":"2024-09-16T03:37:52.651907+0000","last_unstale":"2024-09-16T03:39:11.662354+0000","last_undegraded":"2024-09-16T03:39:11.662354+0000","last_fullsized":"2024-09-16T03:39:11.662354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.985111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"94'34","reported_seq":349,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.660973+0000","last_change":"2024-09-16T03:38:39.081942+0000","last_active":"2024-09-16T03:39:11.660973+0000","last_peered":"2024-09-16T03:39:11.660973+0000","last_clean":"2024-09-16T03:39:11.660973+0000","last_became_active":"2024-09-16T03:38:33.580022+0000","last_became_peered":"2024-09-16T03:38:33.580022+0000","last_unstale":"2024-09-16T03:39:11.660973+0000","last_undegraded":"2024-09-16T03:39:11.660973+0000","last_fullsized":"2024-09-16T03:39:11.660973+0000","mapping_epoch":96,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:39.081810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.081810+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:48.167935+0000","scrub_duration":473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":5,"reported_epoch":135,"state":"activating","last_fresh":"2024-09-16T03:39:09.748033+0000","last_change":"2024-09-16T03:39:09.546055+0000","last_active":"2024-09-16T03:36:52.069777+0000","last_peered":"2024-09-16T03:36:52.069777+0000","last_clean":"2024-09-16T03:36:52.069777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:39:09.748033+0000","last_undegraded":"2024-09-16T03:39:09.748033+0000","last_fullsized":"2024-09-16T03:39:09.748033+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":["9","0","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"129'130","reported_seq":568,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.719379+0000","last_change":"2024-09-16T03:38:39.704994+0000","last_active":"2024-09-16T03:39:08.719379+0000","last_peered":"2024-09-16T03:39:08.719379+0000","last_clean":"2024-09-16T03:39:08.719379+0000","last_became_active":"2024-09-16T03:38:33.581141+0000","last_became_peered":"2024-09-16T03:38:33.581141+0000","last_unstale":"2024-09-16T03:39:08.719379+0000","last_undegraded":"2024-09-16T03:39:08.719379+0000","last_fullsized":"2024-09-16T03:39:08.719379+0000","mapping_epoch":32,"log_start":"123'127","ondisk_log_start":"123'127","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'118","last_scrub_stamp":"2024-09-16T03:38:39.704919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.704919+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":124,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.098482+0000","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882549,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":77034,"num_write":14,"num_write_kb":4474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":248,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641493+0000","last_change":"2024-09-16T03:38:12.041048+0000","last_active":"2024-09-16T03:39:08.641493+0000","last_peered":"2024-09-16T03:39:08.641493+0000","last_clean":"2024-09-16T03:39:08.641493+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:39:08.641493+0000","last_undegraded":"2024-09-16T03:39:08.641493+0000","last_fullsized":"2024-09-16T03:39:08.641493+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:12.040917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.040917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.877949+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":260,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.546220+0000","last_change":"2024-09-16T03:38:34.928371+0000","last_active":"2024-09-16T03:39:09.546220+0000","last_peered":"2024-09-16T03:39:09.546220+0000","last_clean":"2024-09-16T03:39:09.546220+0000","last_became_active":"2024-09-16T03:38:33.484091+0000","last_became_peered":"2024-09-16T03:38:33.484091+0000","last_unstale":"2024-09-16T03:39:09.546220+0000","last_undegraded":"2024-09-16T03:39:09.546220+0000","last_fullsized":"2024-09-16T03:39:09.546220+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:34.928298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.928298+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:43.935101+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":243,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.546720+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:39:09.546720+0000","last_peered":"2024-09-16T03:39:09.546720+0000","last_clean":"2024-09-16T03:39:09.546720+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:39:09.546720+0000","last_undegraded":"2024-09-16T03:39:09.546720+0000","last_fullsized":"2024-09-16T03:39:09.546720+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.232558+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.10","version":"118'64","reported_seq":399,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.616521+0000","last_change":"2024-09-16T03:39:04.442655+0000","last_active":"2024-09-16T03:39:08.616521+0000","last_peered":"2024-09-16T03:39:08.616521+0000","last_clean":"2024-09-16T03:39:08.616521+0000","last_became_active":"2024-09-16T03:39:01.900921+0000","last_became_peered":"2024-09-16T03:39:01.900921+0000","last_unstale":"2024-09-16T03:39:08.616521+0000","last_undegraded":"2024-09-16T03:39:08.616521+0000","last_fullsized":"2024-09-16T03:39:08.616521+0000","mapping_epoch":72,"log_start":"113'61","ondisk_log_start":"113'61","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"118'64","last_scrub_stamp":"2024-09-16T03:39:04.442556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.442556+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.917777+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361433,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"114'17","reported_seq":306,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T03:39:10.647396+0000","last_change":"2024-09-16T03:39:03.929961+0000","last_active":"2024-09-16T03:39:10.647396+0000","last_peered":"2024-09-16T03:39:10.647396+0000","last_clean":"2024-09-16T03:39:10.647396+0000","last_became_active":"2024-09-16T03:39:01.696690+0000","last_became_peered":"2024-09-16T03:39:01.696690+0000","last_unstale":"2024-09-16T03:39:10.647396+0000","last_undegraded":"2024-09-16T03:39:10.647396+0000","last_fullsized":"2024-09-16T03:39:10.647396+0000","mapping_epoch":63,"log_start":"107'14","ondisk_log_start":"107'14","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:03.929863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.929863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:26.527714+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2991679,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"127'20","reported_seq":364,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.471860+0000","last_change":"2024-09-16T03:39:08.648919+0000","last_active":"2024-09-16T03:39:08.648919+0000","last_peered":"2024-09-16T03:39:08.471860+0000","last_clean":"2024-09-16T03:39:08.471860+0000","last_became_active":"2024-09-16T03:39:03.822624+0000","last_became_peered":"2024-09-16T03:39:03.822624+0000","last_unstale":"2024-09-16T03:39:08.471860+0000","last_undegraded":"2024-09-16T03:39:08.471860+0000","last_fullsized":"2024-09-16T03:39:08.471860+0000","mapping_epoch":134,"log_start":"106'17","ondisk_log_start":"106'17","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966452,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":26744,"num_write":10,"num_write_kb":2629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2569696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,9,11],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.13","version":"127'73","reported_seq":458,"reported_epoch":134,"state":"remapped+peering","last_fresh":"2024-09-16T03:39:08.539422+0000","last_change":"2024-09-16T03:39:09.647087+0000","last_active":"2024-09-16T03:39:08.539422+0000","last_peered":"2024-09-16T03:39:08.110262+0000","last_clean":"2024-09-16T03:39:08.110262+0000","last_became_active":"2024-09-16T03:39:01.843090+0000","last_became_peered":"2024-09-16T03:39:01.843090+0000","last_unstale":"2024-09-16T03:39:08.539422+0000","last_undegraded":"2024-09-16T03:39:08.539422+0000","last_fullsized":"2024-09-16T03:39:08.539422+0000","mapping_epoch":135,"log_start":"108'70","ondisk_log_start":"108'70","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'73","last_scrub_stamp":"2024-09-16T03:39:03.124583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.124583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":68,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,9,14],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"104'20","reported_seq":326,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T03:39:10.547215+0000","last_change":"2024-09-16T03:39:01.898409+0000","last_active":"2024-09-16T03:39:10.547215+0000","last_peered":"2024-09-16T03:39:10.547215+0000","last_clean":"2024-09-16T03:39:10.547215+0000","last_became_active":"2024-09-16T03:39:01.596287+0000","last_became_peered":"2024-09-16T03:39:01.596287+0000","last_unstale":"2024-09-16T03:39:10.547215+0000","last_undegraded":"2024-09-16T03:39:10.547215+0000","last_fullsized":"2024-09-16T03:39:10.547215+0000","mapping_epoch":114,"log_start":"98'17","ondisk_log_start":"98'17","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:01.898331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.898331+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.472424+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"123'65","reported_seq":450,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.110040+0000","last_change":"2024-09-16T03:39:08.541769+0000","last_active":"2024-09-16T03:39:08.541769+0000","last_peered":"2024-09-16T03:39:08.110040+0000","last_clean":"2024-09-16T03:39:08.110040+0000","last_became_active":"2024-09-16T03:39:03.915707+0000","last_became_peered":"2024-09-16T03:39:03.915707+0000","last_unstale":"2024-09-16T03:39:08.110040+0000","last_undegraded":"2024-09-16T03:39:08.110040+0000","last_fullsized":"2024-09-16T03:39:08.110040+0000","mapping_epoch":134,"log_start":"123'62","ondisk_log_start":"123'62","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916014,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":83986,"num_write":19,"num_write_kb":5459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":826342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,9,10],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"124'86","reported_seq":458,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.681074+0000","last_change":"2024-09-16T03:39:03.073530+0000","last_active":"2024-09-16T03:39:11.681074+0000","last_peered":"2024-09-16T03:39:11.681074+0000","last_clean":"2024-09-16T03:39:11.681074+0000","last_became_active":"2024-09-16T03:39:01.797805+0000","last_became_peered":"2024-09-16T03:39:01.797805+0000","last_unstale":"2024-09-16T03:39:11.681074+0000","last_undegraded":"2024-09-16T03:39:11.681074+0000","last_fullsized":"2024-09-16T03:39:11.681074+0000","mapping_epoch":115,"log_start":"124'82","ondisk_log_start":"124'82","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.073459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.073459+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":80,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:32.204501+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.17","version":"131'52","reported_seq":402,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.741683+0000","last_change":"2024-09-16T03:39:05.721286+0000","last_active":"2024-09-16T03:39:08.741683+0000","last_peered":"2024-09-16T03:39:08.741683+0000","last_clean":"2024-09-16T03:39:08.741683+0000","last_became_active":"2024-09-16T03:39:03.717420+0000","last_became_peered":"2024-09-16T03:39:03.717420+0000","last_unstale":"2024-09-16T03:39:08.741683+0000","last_undegraded":"2024-09-16T03:39:08.741683+0000","last_fullsized":"2024-09-16T03:39:08.741683+0000","mapping_epoch":128,"log_start":"117'49","ondisk_log_start":"117'49","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":48,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.790793+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1546293,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":52,"num_read_kb":44516,"num_write":11,"num_write_kb":3134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2516795,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"128'51","reported_seq":375,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.546548+0000","last_change":"2024-09-16T03:39:02.668794+0000","last_active":"2024-09-16T03:39:09.546548+0000","last_peered":"2024-09-16T03:39:09.546548+0000","last_clean":"2024-09-16T03:39:09.546548+0000","last_became_active":"2024-09-16T03:38:47.562641+0000","last_became_peered":"2024-09-16T03:38:47.562641+0000","last_unstale":"2024-09-16T03:39:09.546548+0000","last_undegraded":"2024-09-16T03:39:09.546548+0000","last_fullsized":"2024-09-16T03:39:09.546548+0000","mapping_epoch":112,"log_start":"124'48","ondisk_log_start":"124'48","created":50,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'48","last_scrub_stamp":"2024-09-16T03:39:02.668681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.668681+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":45,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.451476+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067327,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":21991,"num_write":23,"num_write_kb":6049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":5,"reported_epoch":135,"state":"activating","last_fresh":"2024-09-16T03:39:09.746323+0000","last_change":"2024-09-16T03:39:09.646000+0000","last_active":"2024-09-16T03:39:09.645854+0000","last_peered":"2024-09-16T03:39:07.528715+0000","last_clean":"2024-09-16T03:39:07.528715+0000","last_became_active":"2024-09-16T03:38:51.558378+0000","last_became_peered":"2024-09-16T03:38:51.558378+0000","last_unstale":"2024-09-16T03:39:09.746323+0000","last_undegraded":"2024-09-16T03:39:09.746323+0000","last_fullsized":"2024-09-16T03:39:09.746323+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:01.106840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.106840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":["9","2","6","11"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"128'125","reported_seq":583,"reported_epoch":134,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:08.641112+0000","last_change":"2024-09-16T03:38:47.159152+0000","last_active":"2024-09-16T03:39:08.641112+0000","last_peered":"2024-09-16T03:39:08.641112+0000","last_clean":"2024-09-16T03:38:43.521485+0000","last_became_active":"2024-09-16T03:38:46.957471+0000","last_became_peered":"2024-09-16T03:38:46.957471+0000","last_unstale":"2024-09-16T03:39:08.641112+0000","last_undegraded":"2024-09-16T03:39:08.641112+0000","last_fullsized":"2024-09-16T03:39:08.641112+0000","mapping_epoch":111,"log_start":"117'122","ondisk_log_start":"117'122","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'115","last_scrub_stamp":"2024-09-16T03:38:35.254488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.254488+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":122,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6739673,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":31161,"num_write":9,"num_write_kb":2915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6,11],"acting":[3,8,10],"avail_no_missing":["3","8","10"],"object_location_counts":[{"shards":"3,8,10","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"131'127","reported_seq":584,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.742125+0000","last_change":"2024-09-16T03:38:50.465770+0000","last_active":"2024-09-16T03:39:08.742125+0000","last_peered":"2024-09-16T03:39:08.742125+0000","last_clean":"2024-09-16T03:39:08.742125+0000","last_became_active":"2024-09-16T03:38:50.465463+0000","last_became_peered":"2024-09-16T03:38:50.465463+0000","last_unstale":"2024-09-16T03:39:08.742125+0000","last_undegraded":"2024-09-16T03:39:08.742125+0000","last_fullsized":"2024-09-16T03:39:08.742125+0000","mapping_epoch":115,"log_start":"119'124","ondisk_log_start":"119'124","created":100,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'107","last_scrub_stamp":"2024-09-16T03:38:39.905697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.905697+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":118,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.828899+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5697672,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":62,"num_read_kb":76423,"num_write":24,"num_write_kb":7195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2376197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"99'56","reported_seq":375,"reported_epoch":136,"state":"peering","last_fresh":"2024-09-16T03:39:11.493056+0000","last_change":"2024-09-16T03:39:11.662898+0000","last_active":"2024-09-16T03:39:11.662898+0000","last_peered":"2024-09-16T03:39:11.493056+0000","last_clean":"2024-09-16T03:39:11.493056+0000","last_became_active":"2024-09-16T03:39:11.177251+0000","last_became_peered":"2024-09-16T03:39:11.177251+0000","last_unstale":"2024-09-16T03:39:11.493056+0000","last_undegraded":"2024-09-16T03:39:11.493056+0000","last_fullsized":"2024-09-16T03:39:11.493056+0000","mapping_epoch":137,"log_start":"90'53","ondisk_log_start":"90'53","created":100,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"99'56","last_scrub_stamp":"2024-09-16T03:39:00.988517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.988517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,9,10],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"92'14","reported_seq":279,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.642046+0000","last_change":"2024-09-16T03:38:49.092747+0000","last_active":"2024-09-16T03:39:08.642046+0000","last_peered":"2024-09-16T03:39:08.642046+0000","last_clean":"2024-09-16T03:39:08.642046+0000","last_became_active":"2024-09-16T03:38:48.661213+0000","last_became_peered":"2024-09-16T03:38:48.661213+0000","last_unstale":"2024-09-16T03:39:08.642046+0000","last_undegraded":"2024-09-16T03:39:08.642046+0000","last_fullsized":"2024-09-16T03:39:08.642046+0000","mapping_epoch":113,"log_start":"91'11","ondisk_log_start":"91'11","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:49.092687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.092687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.372348+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,12],"acting":[3,11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"92'59","reported_seq":388,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641478+0000","last_change":"2024-09-16T03:38:48.561603+0000","last_active":"2024-09-16T03:39:08.641478+0000","last_peered":"2024-09-16T03:39:08.641478+0000","last_clean":"2024-09-16T03:39:08.641478+0000","last_became_active":"2024-09-16T03:38:48.561170+0000","last_became_peered":"2024-09-16T03:38:48.561170+0000","last_unstale":"2024-09-16T03:39:08.641478+0000","last_undegraded":"2024-09-16T03:39:08.641478+0000","last_fullsized":"2024-09-16T03:39:08.641478+0000","mapping_epoch":113,"log_start":"89'56","ondisk_log_start":"89'56","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'59","last_scrub_stamp":"2024-09-16T03:38:35.977646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.977646+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:38.632635+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,6],"acting":[14,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1f","version":"131'112","reported_seq":496,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.541483+0000","last_change":"2024-09-16T03:38:48.762617+0000","last_active":"2024-09-16T03:39:08.541483+0000","last_peered":"2024-09-16T03:39:08.541483+0000","last_clean":"2024-09-16T03:39:08.541483+0000","last_became_active":"2024-09-16T03:38:48.761213+0000","last_became_peered":"2024-09-16T03:38:48.761213+0000","last_unstale":"2024-09-16T03:39:08.541483+0000","last_undegraded":"2024-09-16T03:39:08.541483+0000","last_fullsized":"2024-09-16T03:39:08.541483+0000","mapping_epoch":113,"log_start":"118'109","ondisk_log_start":"118'109","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'95","last_scrub_stamp":"2024-09-16T03:38:37.201959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:37.201959+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.307408+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9748159,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":37374,"num_write":27,"num_write_kb":6985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1704821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,12],"acting":[3,10,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"132'144","reported_seq":606,"reported_epoch":135,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:09.547333+0000","last_change":"2024-09-16T03:38:46.862171+0000","last_active":"2024-09-16T03:39:09.547333+0000","last_peered":"2024-09-16T03:39:09.547333+0000","last_clean":"2024-09-16T03:38:44.034266+0000","last_became_active":"2024-09-16T03:38:46.560752+0000","last_became_peered":"2024-09-16T03:38:46.560752+0000","last_unstale":"2024-09-16T03:39:09.547333+0000","last_undegraded":"2024-09-16T03:39:09.547333+0000","last_fullsized":"2024-09-16T03:39:09.547333+0000","mapping_epoch":111,"log_start":"127'141","ondisk_log_start":"127'141","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'118","last_scrub_stamp":"2024-09-16T03:38:36.698150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:36.698150+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":141,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":87880,"num_write":30,"num_write_kb":9656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[4,10,15],"avail_no_missing":["4","10","15"],"object_location_counts":[{"shards":"4,10,15","objects":3}],"blocked_by":[],"up_primary":8,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.21","version":"48'7","reported_seq":141,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641376+0000","last_change":"2024-09-16T03:38:49.064947+0000","last_active":"2024-09-16T03:39:08.641376+0000","last_peered":"2024-09-16T03:39:08.641376+0000","last_clean":"2024-09-16T03:39:08.641376+0000","last_became_active":"2024-09-16T03:38:48.559431+0000","last_became_peered":"2024-09-16T03:38:48.559431+0000","last_unstale":"2024-09-16T03:39:08.641376+0000","last_undegraded":"2024-09-16T03:39:08.641376+0000","last_fullsized":"2024-09-16T03:39:08.641376+0000","mapping_epoch":113,"log_start":"40'4","ondisk_log_start":"40'4","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":6,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:49.064870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064870+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.310170+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,10],"acting":[14,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.22","version":"45'6","reported_seq":130,"reported_epoch":133,"state":"remapped+peering","last_fresh":"2024-09-16T03:39:08.110164+0000","last_change":"2024-09-16T03:39:09.546581+0000","last_active":"2024-09-16T03:39:09.546581+0000","last_peered":"2024-09-16T03:39:08.110164+0000","last_clean":"2024-09-16T03:39:08.110164+0000","last_became_active":"2024-09-16T03:38:54.462316+0000","last_became_peered":"2024-09-16T03:38:54.462316+0000","last_unstale":"2024-09-16T03:39:08.110164+0000","last_undegraded":"2024-09-16T03:39:08.110164+0000","last_fullsized":"2024-09-16T03:39:08.110164+0000","mapping_epoch":135,"log_start":"39'3","ondisk_log_start":"39'3","created":100,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:39:00.968448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.968448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,6],"acting":[4,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,9,12],"up_primary":9,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"108'38","reported_seq":355,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.741629+0000","last_change":"2024-09-16T03:38:49.464614+0000","last_active":"2024-09-16T03:39:08.741629+0000","last_peered":"2024-09-16T03:39:08.741629+0000","last_clean":"2024-09-16T03:39:08.741629+0000","last_became_active":"2024-09-16T03:38:49.463797+0000","last_became_peered":"2024-09-16T03:38:49.463797+0000","last_unstale":"2024-09-16T03:39:08.741629+0000","last_undegraded":"2024-09-16T03:39:08.741629+0000","last_fullsized":"2024-09-16T03:39:08.741629+0000","mapping_epoch":114,"log_start":"105'36","ondisk_log_start":"105'36","created":100,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:34.893524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.893524+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":35,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:52.829073+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,2],"acting":[14,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.24","version":"105'32","reported_seq":337,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.761038+0000","last_change":"2024-09-16T03:38:47.978956+0000","last_active":"2024-09-16T03:39:11.761038+0000","last_peered":"2024-09-16T03:39:11.761038+0000","last_clean":"2024-09-16T03:39:11.761038+0000","last_became_active":"2024-09-16T03:38:45.820201+0000","last_became_peered":"2024-09-16T03:38:45.820201+0000","last_unstale":"2024-09-16T03:39:11.761038+0000","last_undegraded":"2024-09-16T03:39:11.761038+0000","last_fullsized":"2024-09-16T03:39:11.761038+0000","mapping_epoch":110,"log_start":"83'29","ondisk_log_start":"83'29","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.978888+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:56.881865+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.25","version":"130'85","reported_seq":426,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680968+0000","last_change":"2024-09-16T03:38:48.088833+0000","last_active":"2024-09-16T03:39:11.680968+0000","last_peered":"2024-09-16T03:39:11.680968+0000","last_clean":"2024-09-16T03:39:11.680968+0000","last_became_active":"2024-09-16T03:38:45.724757+0000","last_became_peered":"2024-09-16T03:38:45.724757+0000","last_unstale":"2024-09-16T03:39:11.680968+0000","last_undegraded":"2024-09-16T03:39:11.680968+0000","last_fullsized":"2024-09-16T03:39:11.680968+0000","mapping_epoch":110,"log_start":"116'82","ondisk_log_start":"116'82","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'77","last_scrub_stamp":"2024-09-16T03:38:48.088768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.088768+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":79,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:53.112635+0000","scrub_duration":388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2134555,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":10,"num_write_kb":2500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.26","version":"133'54","reported_seq":402,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.552465+0000","last_change":"2024-09-16T03:38:49.477393+0000","last_active":"2024-09-16T03:39:08.552465+0000","last_peered":"2024-09-16T03:39:08.552465+0000","last_clean":"2024-09-16T03:39:08.552465+0000","last_became_active":"2024-09-16T03:38:45.926029+0000","last_became_peered":"2024-09-16T03:38:45.926029+0000","last_unstale":"2024-09-16T03:39:08.552465+0000","last_undegraded":"2024-09-16T03:39:08.552465+0000","last_fullsized":"2024-09-16T03:39:08.552465+0000","mapping_epoch":110,"log_start":"123'51","ondisk_log_start":"123'51","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:49.477297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.477297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":50,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.242430+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":24203,"num_write":11,"num_write_kb":3140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.27","version":"109'66","reported_seq":389,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.552485+0000","last_change":"2024-09-16T03:38:47.511343+0000","last_active":"2024-09-16T03:39:08.552485+0000","last_peered":"2024-09-16T03:39:08.552485+0000","last_clean":"2024-09-16T03:39:08.552485+0000","last_became_active":"2024-09-16T03:38:45.824884+0000","last_became_peered":"2024-09-16T03:38:45.824884+0000","last_unstale":"2024-09-16T03:39:08.552485+0000","last_undegraded":"2024-09-16T03:39:08.552485+0000","last_fullsized":"2024-09-16T03:39:08.552485+0000","mapping_epoch":110,"log_start":"105'63","ondisk_log_start":"105'63","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'66","last_scrub_stamp":"2024-09-16T03:38:47.511252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.511252+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.884916+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.28","version":"129'68","reported_seq":454,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.372306+0000","last_change":"2024-09-16T03:39:08.541570+0000","last_active":"2024-09-16T03:39:08.541570+0000","last_peered":"2024-09-16T03:39:08.372306+0000","last_clean":"2024-09-16T03:39:08.372306+0000","last_became_active":"2024-09-16T03:38:55.727094+0000","last_became_peered":"2024-09-16T03:38:55.727094+0000","last_unstale":"2024-09-16T03:39:08.372306+0000","last_undegraded":"2024-09-16T03:39:08.372306+0000","last_fullsized":"2024-09-16T03:39:08.372306+0000","mapping_epoch":134,"log_start":"118'65","ondisk_log_start":"118'65","created":110,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:48.551929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.551929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":64,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395113,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":31731,"num_write":5,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3610944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,9,10],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":251,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641395+0000","last_change":"2024-09-16T03:39:00.834877+0000","last_active":"2024-09-16T03:39:08.641395+0000","last_peered":"2024-09-16T03:39:08.641395+0000","last_clean":"2024-09-16T03:39:08.641395+0000","last_became_active":"2024-09-16T03:38:45.627369+0000","last_became_peered":"2024-09-16T03:38:45.627369+0000","last_unstale":"2024-09-16T03:39:08.641395+0000","last_undegraded":"2024-09-16T03:39:08.641395+0000","last_fullsized":"2024-09-16T03:39:08.641395+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:00.834742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.834742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.318226+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.2a","version":"108'122","reported_seq":606,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.546800+0000","last_change":"2024-09-16T03:38:47.964525+0000","last_active":"2024-09-16T03:39:09.546800+0000","last_peered":"2024-09-16T03:39:09.546800+0000","last_clean":"2024-09-16T03:39:09.546800+0000","last_became_active":"2024-09-16T03:38:46.022108+0000","last_became_peered":"2024-09-16T03:38:46.022108+0000","last_unstale":"2024-09-16T03:39:09.546800+0000","last_undegraded":"2024-09-16T03:39:09.546800+0000","last_fullsized":"2024-09-16T03:39:09.546800+0000","mapping_epoch":110,"log_start":"107'115","ondisk_log_start":"107'115","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'122","last_scrub_stamp":"2024-09-16T03:38:47.964439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.964439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":112,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.917123+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.2b","version":"127'124","reported_seq":590,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680858+0000","last_change":"2024-09-16T03:39:09.949533+0000","last_active":"2024-09-16T03:39:11.680858+0000","last_peered":"2024-09-16T03:39:11.680858+0000","last_clean":"2024-09-16T03:39:11.680858+0000","last_became_active":"2024-09-16T03:39:09.949146+0000","last_became_peered":"2024-09-16T03:39:09.949146+0000","last_unstale":"2024-09-16T03:39:11.680858+0000","last_undegraded":"2024-09-16T03:39:11.680858+0000","last_fullsized":"2024-09-16T03:39:11.680858+0000","mapping_epoch":134,"log_start":"119'121","ondisk_log_start":"119'121","created":110,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:49.064061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064061+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":121,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:16.396765+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2c","version":"121'63","reported_seq":408,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:07.529785+0000","last_change":"2024-09-16T03:39:08.548352+0000","last_active":"2024-09-16T03:39:08.548352+0000","last_peered":"2024-09-16T03:39:07.529785+0000","last_clean":"2024-09-16T03:39:07.529785+0000","last_became_active":"2024-09-16T03:38:55.562810+0000","last_became_peered":"2024-09-16T03:38:55.562810+0000","last_unstale":"2024-09-16T03:39:07.529785+0000","last_undegraded":"2024-09-16T03:39:07.529785+0000","last_fullsized":"2024-09-16T03:39:07.529785+0000","mapping_epoch":134,"log_start":"118'60","ondisk_log_start":"118'60","created":110,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'58","last_scrub_stamp":"2024-09-16T03:38:47.889276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.889276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":7651,"num_write":5,"num_write_kb":1234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1262403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[9,11,15],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.2d","version":"92'14","reported_seq":332,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.548592+0000","last_change":"2024-09-16T03:38:47.115739+0000","last_active":"2024-09-16T03:39:08.548592+0000","last_peered":"2024-09-16T03:39:08.548592+0000","last_clean":"2024-09-16T03:39:08.548592+0000","last_became_active":"2024-09-16T03:38:45.723438+0000","last_became_peered":"2024-09-16T03:38:45.723438+0000","last_unstale":"2024-09-16T03:39:08.548592+0000","last_undegraded":"2024-09-16T03:39:08.548592+0000","last_fullsized":"2024-09-16T03:39:08.548592+0000","mapping_epoch":110,"log_start":"91'11","ondisk_log_start":"91'11","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:47.115651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.115651+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:00.107802+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":56,"stat_sum":{"num_bytes":128634223,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":956,"num_read_kb":893485,"num_write":332,"num_write_kb":91950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":21395486,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":231534592,"data_stored":231174352,"data_compressed":572208,"data_compressed_allocated":626688,"data_compressed_original":1503232,"omap_allocated":0,"internal_metadata":0},"log_size":171,"ondisk_log_size":171,"up":168,"acting":168,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":81118503,"num_objects":91,"num_object_clones":0,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":923,"num_read_kb":29771,"num_write":1258,"num_write_kb":281639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":39649106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":178487296,"data_stored":191047270,"data_compressed":11544299,"data_compressed_allocated":12292096,"data_compressed_original":25247744,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":2644640,"data_compressed":159066,"data_compressed_allocated":204800,"data_compressed_original":2584576,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247838,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74436,"kb_used_data":43248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94297404,"statfs":{"total":96636764160,"available":96560541696,"internally_reserved":0,"allocated":44285952,"data_stored":44886400,"data_compressed":1343912,"data_compressed_allocated":1433600,"data_compressed_original":2613248,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247839,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56776,"kb_used_data":25584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94315064,"statfs":{"total":96636764160,"available":96578625536,"internally_reserved":0,"allocated":26198016,"data_stored":27310439,"data_compressed":905243,"data_compressed_allocated":991232,"data_compressed_original":2641920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247838,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68900,"kb_used_data":37700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302940,"statfs":{"total":96636764160,"available":96566210560,"internally_reserved":0,"allocated":38604800,"data_stored":39803729,"data_compressed":1368231,"data_compressed_allocated":1474560,"data_compressed_original":3207168,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280543,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59508,"kb_used_data":23688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94312332,"statfs":{"total":96636764160,"available":96575827968,"internally_reserved":0,"allocated":24256512,"data_stored":25354703,"data_compressed":886305,"data_compressed_allocated":966656,"data_compressed_original":2551808,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280543,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280542,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313247,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34820,"kb_used_data":7788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337020,"statfs":{"total":96636764160,"available":96601108480,"internally_reserved":0,"allocated":7974912,"data_stored":7570474,"data_compressed":25616,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280543,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88796,"kb_used_data":53444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94283044,"statfs":{"total":96636764160,"available":96545837056,"internally_reserved":0,"allocated":54726656,"data_stored":55922960,"data_compressed":1410985,"data_compressed_allocated":1519616,"data_compressed_original":3321856,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280542,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94552,"kb_used_data":63360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94277288,"statfs":{"total":96636764160,"available":96539942912,"internally_reserved":0,"allocated":64880640,"data_stored":67044955,"data_compressed":2176984,"data_compressed_allocated":2338816,"data_compressed_original":5111808,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313247,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280542,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35892,"kb_used_data":4704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335948,"statfs":{"total":96636764160,"available":96600010752,"internally_reserved":0,"allocated":4816896,"data_stored":4453365,"data_compressed":25616,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280542,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69660,"kb_used_data":38468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302180,"statfs":{"total":96636764160,"available":96565432320,"internally_reserved":0,"allocated":39391232,"data_stored":40017276,"data_compressed":1267990,"data_compressed_allocated":1351680,"data_compressed_original":2510848,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280542,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34052,"kb_used_data":2864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337788,"statfs":{"total":96636764160,"available":96601894912,"internally_reserved":0,"allocated":2932736,"data_stored":3069533,"data_compressed":69039,"data_compressed_allocated":90112,"data_compressed_original":618496,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280543,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83532,"kb_used_data":52344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94288308,"statfs":{"total":96636764160,"available":96551227392,"internally_reserved":0,"allocated":53600256,"data_stored":53264385,"data_compressed":186317,"data_compressed_allocated":212992,"data_compressed_original":532480,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,3,4,5,6,7,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313246,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43140,"kb_used_data":11944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328700,"statfs":{"total":96636764160,"available":96592588800,"internally_reserved":0,"allocated":12230656,"data_stored":11891032,"data_compressed":109264,"data_compressed_allocated":126976,"data_compressed_original":270336,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247839,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55588,"kb_used_data":24392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316252,"statfs":{"total":96636764160,"available":96579842048,"internally_reserved":0,"allocated":24977408,"data_stored":25627821,"data_compressed":1314158,"data_compressed_allocated":1400832,"data_compressed_original":2527232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14012416,"data_stored":15061447,"data_compressed":829441,"data_compressed_allocated":888832,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13422592,"data_stored":14113910,"data_compressed":606276,"data_compressed_allocated":643072,"data_compressed_original":1376256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13844480,"data_stored":14888812,"data_compressed":1232826,"data_compressed_allocated":1302528,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22970368,"data_stored":24522128,"data_compressed":1292173,"data_compressed_allocated":1380352,"data_compressed_original":2985984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36982784,"data_stored":39583575,"data_compressed":2121614,"data_compressed_allocated":2269184,"data_compressed_original":4952064,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24047616,"data_stored":25639824,"data_compressed":1378911,"data_compressed_allocated":1470464,"data_compressed_original":3117056,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":13844480,"data_stored":14888812,"data_compressed":1232826,"data_compressed_allocated":1302528,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14012416,"data_stored":15061447,"data_compressed":829441,"data_compressed_allocated":888832,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":13844480,"data_stored":14888812,"data_compressed":1232826,"data_compressed_allocated":1302528,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8982528,"data_stored":8928361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9904128,"data_stored":9871428,"data_compressed":49947,"data_compressed_allocated":53248,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30552064,"data_stored":30549819,"data_compressed":93196,"data_compressed_allocated":102400,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":26701824,"data_stored":26610529,"data_compressed":29754,"data_compressed_allocated":32768,"data_compressed_original":86016,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":14409728,"data_stored":14413518,"data_compressed":44673,"data_compressed_allocated":49152,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3641344,"data_stored":3612626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24342528,"data_stored":24277619,"data_compressed":9548,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1683456,"data_stored":1690918,"data_compressed":10762,"data_compressed_allocated":12288,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":52391936,"data_stored":52412861,"data_compressed":160701,"data_compressed_allocated":176128,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":10862592,"data_stored":10841283,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":11001856,"data_stored":11037652,"data_compressed":83648,"data_compressed_allocated":90112,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":6758400,"data_stored":6718533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":29212672,"data_stored":29129165,"data_compressed":79701,"data_compressed_allocated":86016,"data_compressed_original":151552,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:39:13.342 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T03:39:13.354 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:13.354 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:13.363 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:39:13.387 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:13.385+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:1733: done (7 left) 2024-09-16T03:39:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:1737: tier_promote oid 33 2024-09-16T03:39:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:1736: got (22) Invalid argument 2024-09-16T03:39:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:1730: finishing write tid 1 to smithi15736696-3 2024-09-16T03:39:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:1730: finishing write tid 2 to smithi15736696-3 2024-09-16T03:39:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:1730: finishing write tid 3 to smithi15736696-3 2024-09-16T03:39:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:1730: finishing write tid 4 to smithi15736696-3 2024-09-16T03:39:13.423 INFO:tasks.rados.rados.0.smithi157.stdout:1730: oid 3 updating version 0 to 75 2024-09-16T03:39:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:1730: oid 3 updating version 75 to 76 2024-09-16T03:39:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:1730: oid 3 updating version 76 to 77 2024-09-16T03:39:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 77 (ObjNum 381 snap 0 seq_num 381) dirty exists 2024-09-16T03:39:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:1730: left oid 3 (ObjNum 381 snap 0 seq_num 381) 2024-09-16T03:39:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:1731: finishing write tid 1 to smithi15736696-43 2024-09-16T03:39:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:1732: expect (ObjNum 297 snap 0 seq_num 297) 2024-09-16T03:39:13.424 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 337657 2024-09-16T03:39:13.488 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:13.486+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:13.510 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:13.510 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:13.519 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:39:13.566 INFO:tasks.rados.rados.0.smithi157.stdout:1735: finishing write tid 1 to smithi15736696-46 2024-09-16T03:39:13.566 INFO:tasks.rados.rados.0.smithi157.stdout:1735: finishing write tid 2 to smithi15736696-46 2024-09-16T03:39:13.566 INFO:tasks.rados.rados.0.smithi157.stdout:1737: got (0) Success 2024-09-16T03:39:13.566 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 62 (ObjNum 328 snap 0 seq_num 328) dirty exists 2024-09-16T03:39:13.566 INFO:tasks.rados.rados.0.smithi157.stdout:1735: finishing write tid 3 to smithi15736696-46 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:1735: finishing write tid 4 to smithi15736696-46 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:1735: oid 46 updating version 0 to 131 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:1735: oid 46 version 131 is already newer than 130 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:1735: oid 46 version 131 is already newer than 129 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 131 (ObjNum 383 snap 0 seq_num 383) dirty exists 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:1735: left oid 46 (ObjNum 383 snap 0 seq_num 383) 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:1730: done (7 left) 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:1732: done (6 left) 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:1735: done (5 left) 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:1736: done (4 left) 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:1737: done (3 left) 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:1738: read oid 30 snap -1 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:1738: expect (ObjNum 361 snap 0 seq_num 361) 2024-09-16T03:39:13.567 INFO:tasks.rados.rados.0.smithi157.stdout:1739: read oid 42 snap -1 2024-09-16T03:39:13.568 INFO:tasks.rados.rados.0.smithi157.stdout:1739: expect deleted 2024-09-16T03:39:13.568 INFO:tasks.rados.rados.0.smithi157.stdout:1740: set_chunk oid 37 target oid 2024-09-16T03:39:13.572 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:39:13.588 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:13.586+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:13.668 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:13.668 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:13.677 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:39:13.689 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:13.687+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:13.760 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:13.760 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg inactive","count":1},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull","count":4},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":156,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":139,"num_osds":16,"num_up_osds":16,"osd_up_since":1726457810,"num_in_osds":13,"osd_in_since":1726457912,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":51},{"state_name":"peering","count":8},{"state_name":"active+remapped+backfill_toofull","count":4},{"state_name":"remapped+peering","count":3},{"state_name":"activating","count":2},{"state_name":"active","count":1}],"num_pgs":69,"num_pools":4,"num_objects":138,"data_bytes":209895113,"bytes_used":835694592,"bytes_avail":1255442239488,"bytes_total":1256277934080,"inactive_pgs_ratio":0.18840579688549042,"misplaced_objects":16,"misplaced_total":414,"misplaced_ratio":0.03864734299516908,"recovering_objects_per_sec":0,"recovering_bytes_per_sec":57487,"recovering_keys_per_sec":0,"num_objects_recovered":1,"num_bytes_recovered":230000,"num_keys_recovered":0,"read_bytes_sec":31328483,"write_bytes_sec":1225200,"read_op_per_sec":31,"write_op_per_sec":5},"fsmap":{"epoch":1,"btime":"2024-09-16T03:36:37:310011+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T03:37:51.688536+0000","services":{}},"progress_events":{}} 2024-09-16T03:39:13.772 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-16T03:39:13.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:39:13.781 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:13.782 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:13.787 INFO:tasks.rados.rados.0.smithi157.stdout:1740: set_chunk oid 37 offset: 2162176 length: 1261568 target oid bf9ea504c451aec47a829e6ce08253236900c52611c9b291783d49a96a9019cf offset: 0 snap -1 2024-09-16T03:39:13.787 INFO:tasks.rados.rados.0.smithi157.stdout:1741: read oid 33 snap -1 2024-09-16T03:39:13.787 INFO:tasks.rados.rados.0.smithi157.stdout:1741: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-16T03:39:13.787 INFO:tasks.rados.rados.0.smithi157.stdout:1742: read oid 23 snap -1 2024-09-16T03:39:13.788 INFO:tasks.rados.rados.0.smithi157.stdout:1742: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:39:13.788 INFO:tasks.rados.rados.0.smithi157.stdout:1743: read oid 3 snap -1 2024-09-16T03:39:13.788 INFO:tasks.rados.rados.0.smithi157.stdout:1743: expect (ObjNum 381 snap 0 seq_num 381) 2024-09-16T03:39:13.788 INFO:tasks.rados.rados.0.smithi157.stdout:1744: write (excl) oid 29 current snap is 0 2024-09-16T03:39:13.788 INFO:tasks.rados.rados.0.smithi157.stdout:1744: seq_num 384 ranges {453888=713266,1737556=442800} 2024-09-16T03:39:13.789 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:13.787+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:13.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:39:13.793 INFO:tasks.rados.rados.0.smithi157.stdout:1744: writing smithi15736696-29 from 453888 to 1167154 tid 1 2024-09-16T03:39:13.799 INFO:tasks.rados.rados.0.smithi157.stdout:1744: writing smithi15736696-29 from 1737556 to 2180356 tid 2 2024-09-16T03:39:13.799 INFO:tasks.rados.rados.0.smithi157.stdout:1745: read oid 35 snap -1 2024-09-16T03:39:13.799 INFO:tasks.rados.rados.0.smithi157.stdout:1745: expect deleted 2024-09-16T03:39:13.799 INFO:tasks.rados.rados.0.smithi157.stdout:1746: read oid 8 snap -1 2024-09-16T03:39:13.800 INFO:tasks.rados.rados.0.smithi157.stdout:1746: expect (ObjNum 359 snap 0 seq_num 359) 2024-09-16T03:39:13.800 INFO:tasks.rados.rados.0.smithi157.stdout:1747: delete oid 11 current snap is 0 2024-09-16T03:39:13.800 INFO:tasks.rados.rados.0.smithi157.stdout:1731: finishing write tid 2 to smithi15736696-43 2024-09-16T03:39:13.800 INFO:tasks.rados.rados.0.smithi157.stdout:1731: finishing write tid 3 to smithi15736696-43 2024-09-16T03:39:13.800 INFO:tasks.rados.rados.0.smithi157.stdout:1731: finishing write tid 4 to smithi15736696-43 2024-09-16T03:39:13.800 INFO:tasks.rados.rados.0.smithi157.stdout:1731: oid 43 updating version 0 to 87 2024-09-16T03:39:13.800 INFO:tasks.rados.rados.0.smithi157.stdout:1731: oid 43 updating version 87 to 88 2024-09-16T03:39:13.800 INFO:tasks.rados.rados.0.smithi157.stdout:1731: oid 43 version 88 is already newer than 86 2024-09-16T03:39:13.800 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 88 (ObjNum 382 snap 0 seq_num 382) dirty exists 2024-09-16T03:39:13.800 INFO:tasks.rados.rados.0.smithi157.stdout:1731: left oid 43 (ObjNum 382 snap 0 seq_num 382) 2024-09-16T03:39:13.800 INFO:tasks.rados.rados.0.smithi157.stdout:1738: expect (ObjNum 361 snap 0 seq_num 361) 2024-09-16T03:39:13.885 INFO:tasks.rados.rados.0.smithi157.stdout:1734: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-16T03:39:13.886 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:13.890 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:13.888+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:13.926 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:13.927 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:13.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:39:13.938 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2004911 2024-09-16T03:39:13.961 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2072802 2024-09-16T03:39:13.966 INFO:tasks.rados.rados.0.smithi157.stdout:1741: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-16T03:39:13.966 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:13.991 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:13.989+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:14.044 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:14.044 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:14.053 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:39:14.066 INFO:tasks.rados.rados.0.smithi157.stdout:1746: expect (ObjNum 359 snap 0 seq_num 359) 2024-09-16T03:39:14.071 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 608386 2024-09-16T03:39:14.079 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:14.080 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:39:14.091 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:14.090+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:14.093 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-16T03:39:13.706236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":210343113,"num_objects":139,"num_object_clones":0,"num_object_copies":417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15,"num_objects_unfound":0,"num_objects_dirty":138,"num_whiteouts":0,"num_read":1965,"num_read_kb":923329,"num_write":1737,"num_write_kb":376263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":62102448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":207,"acting":207,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":218,"num_osds":13,"num_per_pool_osds":13,"num_per_pool_omap_osds":13,"kb":1226833920,"kb_used":799652,"kb_used_data":389528,"kb_used_omap":20,"kb_used_meta":405163,"kb_avail":1226034268,"statfs":{"total":1256277934080,"available":1255459090432,"internally_reserved":0,"allocated":398876672,"data_stored":406217072,"data_compressed":11089660,"data_compressed_allocated":11980800,"data_compressed_original":26054656,"omap_allocated":20735,"internal_metadata":414887681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":448000,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":30215,"num_write":5,"num_write_kb":869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001042"},"pg_stats":[{"pgid":"4.37","version":"131'52","reported_seq":397,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641567+0000","last_change":"2024-09-16T03:39:08.371859+0000","last_active":"2024-09-16T03:39:08.641567+0000","last_peered":"2024-09-16T03:39:08.641567+0000","last_clean":"2024-09-16T03:39:08.641567+0000","last_became_active":"2024-09-16T03:39:05.721626+0000","last_became_peered":"2024-09-16T03:39:05.721626+0000","last_unstale":"2024-09-16T03:39:08.641567+0000","last_undegraded":"2024-09-16T03:39:08.641567+0000","last_fullsized":"2024-09-16T03:39:08.641567+0000","mapping_epoch":130,"log_start":"104'45","ondisk_log_start":"104'45","created":126,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":44,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:36.963537+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2890887,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":44517,"num_write":12,"num_write_kb":3069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1148090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.36","version":"131'93","reported_seq":472,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680959+0000","last_change":"2024-09-16T03:39:03.716194+0000","last_active":"2024-09-16T03:39:11.680959+0000","last_peered":"2024-09-16T03:39:11.680959+0000","last_clean":"2024-09-16T03:39:11.680959+0000","last_became_active":"2024-09-16T03:39:02.099320+0000","last_became_peered":"2024-09-16T03:39:02.099320+0000","last_unstale":"2024-09-16T03:39:11.680959+0000","last_undegraded":"2024-09-16T03:39:11.680959+0000","last_fullsized":"2024-09-16T03:39:11.680959+0000","mapping_epoch":126,"log_start":"130'90","ondisk_log_start":"130'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.716125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.716125+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":88,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.328700+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269209,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":4665,"num_write":8,"num_write_kb":2101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.35","version":"131'66","reported_seq":456,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.529794+0000","last_change":"2024-09-16T03:39:08.541599+0000","last_active":"2024-09-16T03:39:08.541599+0000","last_peered":"2024-09-16T03:39:08.529794+0000","last_clean":"2024-09-16T03:39:08.529794+0000","last_became_active":"2024-09-16T03:39:05.722545+0000","last_became_peered":"2024-09-16T03:39:05.722545+0000","last_unstale":"2024-09-16T03:39:08.529794+0000","last_undegraded":"2024-09-16T03:39:08.529794+0000","last_fullsized":"2024-09-16T03:39:08.529794+0000","mapping_epoch":134,"log_start":"123'63","ondisk_log_start":"123'63","created":126,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916015,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84364,"num_write":19,"num_write_kb":5459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3131048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,9,10],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"104'20","reported_seq":327,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T03:39:10.547004+0000","last_change":"2024-09-16T03:39:02.919356+0000","last_active":"2024-09-16T03:39:10.547004+0000","last_peered":"2024-09-16T03:39:10.547004+0000","last_clean":"2024-09-16T03:39:10.547004+0000","last_became_active":"2024-09-16T03:39:01.699286+0000","last_became_peered":"2024-09-16T03:39:01.699286+0000","last_unstale":"2024-09-16T03:39:10.547004+0000","last_undegraded":"2024-09-16T03:39:10.547004+0000","last_fullsized":"2024-09-16T03:39:10.547004+0000","mapping_epoch":126,"log_start":"98'17","ondisk_log_start":"98'17","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:02.919259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.919259+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.209096+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2705960,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"108'72","reported_seq":462,"reported_epoch":134,"state":"remapped+peering","last_fresh":"2024-09-16T03:39:08.538848+0000","last_change":"2024-09-16T03:39:09.647095+0000","last_active":"2024-09-16T03:39:08.538848+0000","last_peered":"2024-09-16T03:39:08.110144+0000","last_clean":"2024-09-16T03:39:08.110144+0000","last_became_active":"2024-09-16T03:39:01.642310+0000","last_became_peered":"2024-09-16T03:39:01.642310+0000","last_unstale":"2024-09-16T03:39:08.538848+0000","last_undegraded":"2024-09-16T03:39:08.538848+0000","last_fullsized":"2024-09-16T03:39:08.538848+0000","mapping_epoch":135,"log_start":"108'70","ondisk_log_start":"108'70","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'72","last_scrub_stamp":"2024-09-16T03:39:02.143339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.143339+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":68,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2722767,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":11293,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,9,14],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.32","version":"106'17","reported_seq":355,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.471820+0000","last_change":"2024-09-16T03:39:08.648217+0000","last_active":"2024-09-16T03:39:08.648217+0000","last_peered":"2024-09-16T03:39:08.471820+0000","last_clean":"2024-09-16T03:39:08.471820+0000","last_became_active":"2024-09-16T03:39:05.823393+0000","last_became_peered":"2024-09-16T03:39:05.823393+0000","last_unstale":"2024-09-16T03:39:08.471820+0000","last_undegraded":"2024-09-16T03:39:08.471820+0000","last_fullsized":"2024-09-16T03:39:08.471820+0000","mapping_epoch":134,"log_start":"82'13","ondisk_log_start":"82'13","created":126,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1033647,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":26745,"num_write":6,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1331059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,9,11],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.31","version":"114'17","reported_seq":297,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T03:39:10.646986+0000","last_change":"2024-09-16T03:39:05.622532+0000","last_active":"2024-09-16T03:39:10.646986+0000","last_peered":"2024-09-16T03:39:10.646986+0000","last_clean":"2024-09-16T03:39:10.646986+0000","last_became_active":"2024-09-16T03:39:01.997491+0000","last_became_peered":"2024-09-16T03:39:01.997491+0000","last_unstale":"2024-09-16T03:39:10.646986+0000","last_undegraded":"2024-09-16T03:39:10.646986+0000","last_fullsized":"2024-09-16T03:39:10.646986+0000","mapping_epoch":126,"log_start":"107'14","ondisk_log_start":"107'14","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'9","last_scrub_stamp":"2024-09-16T03:37:50.756120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:50.756120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.630555+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1495840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":28490,"num_write":4,"num_write_kb":1066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"118'64","reported_seq":392,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.615533+0000","last_change":"2024-09-16T03:39:05.621064+0000","last_active":"2024-09-16T03:39:08.615533+0000","last_peered":"2024-09-16T03:39:08.615533+0000","last_clean":"2024-09-16T03:39:08.615533+0000","last_became_active":"2024-09-16T03:39:01.901037+0000","last_became_peered":"2024-09-16T03:39:01.901037+0000","last_unstale":"2024-09-16T03:39:08.615533+0000","last_undegraded":"2024-09-16T03:39:08.615533+0000","last_fullsized":"2024-09-16T03:39:08.615533+0000","mapping_epoch":126,"log_start":"113'61","ondisk_log_start":"113'61","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:35.157846+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1680717,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":23268,"num_write":10,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":918091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2f","version":"113'94","reported_seq":462,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.552454+0000","last_change":"2024-09-16T03:39:03.319075+0000","last_active":"2024-09-16T03:39:08.552454+0000","last_peered":"2024-09-16T03:39:08.552454+0000","last_clean":"2024-09-16T03:39:08.552454+0000","last_became_active":"2024-09-16T03:39:01.700988+0000","last_became_peered":"2024-09-16T03:39:01.700988+0000","last_unstale":"2024-09-16T03:39:08.552454+0000","last_undegraded":"2024-09-16T03:39:08.552454+0000","last_fullsized":"2024-09-16T03:39:08.552454+0000","mapping_epoch":126,"log_start":"107'90","ondisk_log_start":"107'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"113'94","last_scrub_stamp":"2024-09-16T03:39:03.319012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.319012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":87,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:13.773817+0000","scrub_duration":1038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2e","version":"123'75","reported_seq":447,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.546559+0000","last_change":"2024-09-16T03:39:04.235234+0000","last_active":"2024-09-16T03:39:09.546559+0000","last_peered":"2024-09-16T03:39:09.546559+0000","last_clean":"2024-09-16T03:39:09.546559+0000","last_became_active":"2024-09-16T03:39:01.999291+0000","last_became_peered":"2024-09-16T03:39:01.999291+0000","last_unstale":"2024-09-16T03:39:09.546559+0000","last_undegraded":"2024-09-16T03:39:09.546559+0000","last_fullsized":"2024-09-16T03:39:09.546559+0000","mapping_epoch":126,"log_start":"114'72","ondisk_log_start":"114'72","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"123'75","last_scrub_stamp":"2024-09-16T03:39:04.235141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.235141+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":72,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.010761+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":13153,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"131'99","reported_seq":469,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.652681+0000","last_change":"2024-09-16T03:39:03.917422+0000","last_active":"2024-09-16T03:39:08.652681+0000","last_peered":"2024-09-16T03:39:08.652681+0000","last_clean":"2024-09-16T03:39:08.652681+0000","last_became_active":"2024-09-16T03:39:01.801112+0000","last_became_peered":"2024-09-16T03:39:01.801112+0000","last_unstale":"2024-09-16T03:39:08.652681+0000","last_undegraded":"2024-09-16T03:39:08.652681+0000","last_fullsized":"2024-09-16T03:39:08.652681+0000","mapping_epoch":32,"log_start":"128'96","ondisk_log_start":"128'96","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'97","last_scrub_stamp":"2024-09-16T03:39:03.917355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.917355+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":93,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.686164+0000","scrub_duration":771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2033713,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.e","version":"127'79","reported_seq":443,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.547110+0000","last_change":"2024-09-16T03:39:05.725455+0000","last_active":"2024-09-16T03:39:09.547110+0000","last_peered":"2024-09-16T03:39:09.547110+0000","last_clean":"2024-09-16T03:39:09.547110+0000","last_became_active":"2024-09-16T03:39:01.999282+0000","last_became_peered":"2024-09-16T03:39:01.999282+0000","last_unstale":"2024-09-16T03:39:09.547110+0000","last_undegraded":"2024-09-16T03:39:09.547110+0000","last_fullsized":"2024-09-16T03:39:09.547110+0000","mapping_epoch":32,"log_start":"123'75","ondisk_log_start":"123'75","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:39.194237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.194237+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":72,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.946501+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3705976,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":28930,"num_write":10,"num_write_kb":3507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"124'19","reported_seq":346,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.548749+0000","last_change":"2024-09-16T03:39:01.149616+0000","last_active":"2024-09-16T03:39:08.548749+0000","last_peered":"2024-09-16T03:39:08.548749+0000","last_clean":"2024-09-16T03:39:08.548749+0000","last_became_active":"2024-09-16T03:38:45.622965+0000","last_became_peered":"2024-09-16T03:38:45.622965+0000","last_unstale":"2024-09-16T03:39:08.548749+0000","last_undegraded":"2024-09-16T03:39:08.548749+0000","last_fullsized":"2024-09-16T03:39:08.548749+0000","mapping_epoch":72,"log_start":"118'16","ondisk_log_start":"118'16","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'19","last_scrub_stamp":"2024-09-16T03:39:01.149486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.149486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":16,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.574017+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613538,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"130'65","reported_seq":409,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:07.529674+0000","last_change":"2024-09-16T03:39:08.548464+0000","last_active":"2024-09-16T03:39:08.548464+0000","last_peered":"2024-09-16T03:39:07.529674+0000","last_clean":"2024-09-16T03:39:07.529674+0000","last_became_active":"2024-09-16T03:38:54.765107+0000","last_became_peered":"2024-09-16T03:38:54.765107+0000","last_unstale":"2024-09-16T03:39:07.529674+0000","last_undegraded":"2024-09-16T03:39:07.529674+0000","last_fullsized":"2024-09-16T03:39:07.529674+0000","mapping_epoch":134,"log_start":"122'62","ondisk_log_start":"122'62","created":32,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'57","last_scrub_stamp":"2024-09-16T03:38:48.923184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.923184+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":2798,"num_write":9,"num_write_kb":2888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[9,11,15],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"129'132","reported_seq":604,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680864+0000","last_change":"2024-09-16T03:39:09.748835+0000","last_active":"2024-09-16T03:39:11.680864+0000","last_peered":"2024-09-16T03:39:11.680864+0000","last_clean":"2024-09-16T03:39:11.680864+0000","last_became_active":"2024-09-16T03:39:09.748391+0000","last_became_peered":"2024-09-16T03:39:09.748391+0000","last_unstale":"2024-09-16T03:39:11.680864+0000","last_undegraded":"2024-09-16T03:39:11.680864+0000","last_fullsized":"2024-09-16T03:39:11.680864+0000","mapping_epoch":134,"log_start":"127'129","ondisk_log_start":"127'129","created":32,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:46.855462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.855462+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":125,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:12.773343+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2084134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":16397,"num_write":31,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.a","version":"128'148","reported_seq":687,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.547162+0000","last_change":"2024-09-16T03:39:01.504751+0000","last_active":"2024-09-16T03:39:09.547162+0000","last_peered":"2024-09-16T03:39:09.547162+0000","last_clean":"2024-09-16T03:39:09.547162+0000","last_became_active":"2024-09-16T03:38:46.021805+0000","last_became_peered":"2024-09-16T03:38:46.021805+0000","last_unstale":"2024-09-16T03:39:09.547162+0000","last_undegraded":"2024-09-16T03:39:09.547162+0000","last_fullsized":"2024-09-16T03:39:09.547162+0000","mapping_epoch":32,"log_start":"123'145","ondisk_log_start":"123'145","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'147","last_scrub_stamp":"2024-09-16T03:39:01.504663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.504663+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":142,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:18.060284+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13949942,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":53,"num_read_kb":53989,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":248,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641520+0000","last_change":"2024-09-16T03:38:46.881357+0000","last_active":"2024-09-16T03:39:08.641520+0000","last_peered":"2024-09-16T03:39:08.641520+0000","last_clean":"2024-09-16T03:39:08.641520+0000","last_became_active":"2024-09-16T03:38:45.728494+0000","last_became_peered":"2024-09-16T03:38:45.728494+0000","last_unstale":"2024-09-16T03:39:08.641520+0000","last_undegraded":"2024-09-16T03:39:08.641520+0000","last_fullsized":"2024-09-16T03:39:08.641520+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:46.881248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.881248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:04.991122+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.8","version":"130'74","reported_seq":454,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.372137+0000","last_change":"2024-09-16T03:39:08.542925+0000","last_active":"2024-09-16T03:39:08.542925+0000","last_peered":"2024-09-16T03:39:08.372137+0000","last_clean":"2024-09-16T03:39:08.372137+0000","last_became_active":"2024-09-16T03:38:55.664848+0000","last_became_peered":"2024-09-16T03:38:55.664848+0000","last_unstale":"2024-09-16T03:39:08.372137+0000","last_undegraded":"2024-09-16T03:39:08.372137+0000","last_fullsized":"2024-09-16T03:39:08.372137+0000","mapping_epoch":134,"log_start":"124'71","ondisk_log_start":"124'71","created":32,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'71","last_scrub_stamp":"2024-09-16T03:39:02.002247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.002247+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":69,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2184976,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,9,10],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"130'85","reported_seq":437,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.681102+0000","last_change":"2024-09-16T03:39:00.900732+0000","last_active":"2024-09-16T03:39:11.681102+0000","last_peered":"2024-09-16T03:39:11.681102+0000","last_clean":"2024-09-16T03:39:11.681102+0000","last_became_active":"2024-09-16T03:38:45.523605+0000","last_became_peered":"2024-09-16T03:38:45.523605+0000","last_unstale":"2024-09-16T03:39:11.681102+0000","last_undegraded":"2024-09-16T03:39:11.681102+0000","last_fullsized":"2024-09-16T03:39:11.681102+0000","mapping_epoch":32,"log_start":"116'82","ondisk_log_start":"116'82","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'82","last_scrub_stamp":"2024-09-16T03:39:00.900619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.900619+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":80,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.385837+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2117748,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":429,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.760857+0000","last_change":"2024-09-16T03:38:36.757156+0000","last_active":"2024-09-16T03:39:11.760857+0000","last_peered":"2024-09-16T03:39:11.760857+0000","last_clean":"2024-09-16T03:39:11.760857+0000","last_became_active":"2024-09-16T03:38:36.756855+0000","last_became_peered":"2024-09-16T03:38:36.756855+0000","last_unstale":"2024-09-16T03:39:11.760857+0000","last_undegraded":"2024-09-16T03:39:11.760857+0000","last_fullsized":"2024-09-16T03:39:11.760857+0000","mapping_epoch":103,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:38:28.063690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:38:28.063690+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:28.682469+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":126,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.541136+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:39:08.541136+0000","last_peered":"2024-09-16T03:39:08.541136+0000","last_clean":"2024-09-16T03:39:08.541136+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:39:08.541136+0000","last_undegraded":"2024-09-16T03:39:08.541136+0000","last_fullsized":"2024-09-16T03:39:08.541136+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"131'260","reported_seq":571,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.660977+0000","last_change":"2024-09-16T03:38:01.603352+0000","last_active":"2024-09-16T03:39:11.660977+0000","last_peered":"2024-09-16T03:39:11.660977+0000","last_clean":"2024-09-16T03:39:11.660977+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:39:11.660977+0000","last_undegraded":"2024-09-16T03:39:11.660977+0000","last_fullsized":"2024-09-16T03:39:11.660977+0000","mapping_epoch":74,"log_start":"129'257","ondisk_log_start":"129'257","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":266,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.902901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22624199,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":247,"num_read_kb":8660,"num_write":324,"num_write_kb":70574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.6","version":"108'44","reported_seq":382,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.552545+0000","last_change":"2024-09-16T03:38:48.279289+0000","last_active":"2024-09-16T03:39:08.552545+0000","last_peered":"2024-09-16T03:39:08.552545+0000","last_clean":"2024-09-16T03:39:08.552545+0000","last_became_active":"2024-09-16T03:38:45.924286+0000","last_became_peered":"2024-09-16T03:38:45.924286+0000","last_unstale":"2024-09-16T03:39:08.552545+0000","last_undegraded":"2024-09-16T03:39:08.552545+0000","last_fullsized":"2024-09-16T03:39:08.552545+0000","mapping_epoch":70,"log_start":"105'41","ondisk_log_start":"105'41","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:48.279224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.279224+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.718165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":264,"reported_epoch":135,"state":"active","last_fresh":"2024-09-16T03:39:09.649028+0000","last_change":"2024-09-16T03:39:09.649028+0000","last_active":"2024-09-16T03:39:09.649028+0000","last_peered":"2024-09-16T03:39:09.649028+0000","last_clean":"2024-09-16T03:39:08.025961+0000","last_became_active":"2024-09-16T03:39:09.649028+0000","last_became_peered":"2024-09-16T03:39:09.649028+0000","last_unstale":"2024-09-16T03:39:09.649028+0000","last_undegraded":"2024-09-16T03:39:09.649028+0000","last_fullsized":"2024-09-16T03:39:09.649028+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:24.358339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:24.358339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":["8","6","9","10"],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"133'220","reported_seq":462,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.562283+0000","last_change":"2024-09-16T03:38:15.890094+0000","last_active":"2024-09-16T03:39:11.562283+0000","last_peered":"2024-09-16T03:39:11.562283+0000","last_clean":"2024-09-16T03:39:11.562283+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:39:11.562283+0000","last_undegraded":"2024-09-16T03:39:11.562283+0000","last_fullsized":"2024-09-16T03:39:11.562283+0000","mapping_epoch":31,"log_start":"131'217","ondisk_log_start":"131'217","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'125","last_scrub_stamp":"2024-09-16T03:38:15.889996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.889996+0000","objects_scrubbed":18,"log_size":3,"log_dups_size":228,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:35.350472+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16331162,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":194,"num_read_kb":2025,"num_write":277,"num_write_kb":58956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"127'76","reported_seq":420,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.752847+0000","last_change":"2024-09-16T03:39:01.297447+0000","last_active":"2024-09-16T03:39:08.752847+0000","last_peered":"2024-09-16T03:39:08.752847+0000","last_clean":"2024-09-16T03:39:08.752847+0000","last_became_active":"2024-09-16T03:38:45.723754+0000","last_became_peered":"2024-09-16T03:38:45.723754+0000","last_unstale":"2024-09-16T03:39:08.752847+0000","last_undegraded":"2024-09-16T03:39:08.752847+0000","last_fullsized":"2024-09-16T03:39:08.752847+0000","mapping_epoch":32,"log_start":"127'73","ondisk_log_start":"127'73","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'72","last_scrub_stamp":"2024-09-16T03:39:01.297324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.297324+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":71,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.355718+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7294304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":4176,"num_write":5,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":244,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680762+0000","last_change":"2024-09-16T03:38:15.949627+0000","last_active":"2024-09-16T03:39:11.680762+0000","last_peered":"2024-09-16T03:39:11.680762+0000","last_clean":"2024-09-16T03:39:11.680762+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:39:11.680762+0000","last_undegraded":"2024-09-16T03:39:11.680762+0000","last_fullsized":"2024-09-16T03:39:11.680762+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:15.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:24.286324+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"129'252","reported_seq":620,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-16T03:39:03.630052+0000","last_change":"2024-09-16T03:38:59.856507+0000","last_active":"2024-09-16T03:39:03.630052+0000","last_peered":"2024-09-16T03:39:03.630052+0000","last_clean":"2024-09-16T03:39:03.630052+0000","last_became_active":"2024-09-16T03:38:59.854999+0000","last_became_peered":"2024-09-16T03:38:59.854999+0000","last_unstale":"2024-09-16T03:39:03.630052+0000","last_undegraded":"2024-09-16T03:39:03.630052+0000","last_fullsized":"2024-09-16T03:39:03.630052+0000","mapping_epoch":124,"log_start":"128'249","ondisk_log_start":"128'249","created":31,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":270,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17434102,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":231,"num_read_kb":12817,"num_write":311,"num_write_kb":78387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":27250603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"105'32","reported_seq":339,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.660984+0000","last_change":"2024-09-16T03:38:47.042020+0000","last_active":"2024-09-16T03:39:11.660984+0000","last_peered":"2024-09-16T03:39:11.660984+0000","last_clean":"2024-09-16T03:39:11.660984+0000","last_became_active":"2024-09-16T03:38:45.820169+0000","last_became_peered":"2024-09-16T03:38:45.820169+0000","last_unstale":"2024-09-16T03:39:11.660984+0000","last_undegraded":"2024-09-16T03:39:11.660984+0000","last_fullsized":"2024-09-16T03:39:11.660984+0000","mapping_epoch":32,"log_start":"83'29","ondisk_log_start":"83'29","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.041851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.041851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.311785+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":244,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.647284+0000","last_change":"2024-09-16T03:38:12.795978+0000","last_active":"2024-09-16T03:39:09.647284+0000","last_peered":"2024-09-16T03:39:09.647284+0000","last_clean":"2024-09-16T03:39:09.647284+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:39:09.647284+0000","last_undegraded":"2024-09-16T03:39:09.647284+0000","last_fullsized":"2024-09-16T03:39:09.647284+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:38:12.795855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.795855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:13.088740+0000","scrub_duration":501,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"134'277","reported_seq":522,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.681013+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:39:11.681013+0000","last_peered":"2024-09-16T03:39:11.681013+0000","last_clean":"2024-09-16T03:39:11.681013+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:39:11.681013+0000","last_undegraded":"2024-09-16T03:39:11.681013+0000","last_fullsized":"2024-09-16T03:39:11.681013+0000","mapping_epoch":31,"log_start":"131'274","ondisk_log_start":"131'274","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":286,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.679811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24729040,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":251,"num_read_kb":6269,"num_write":346,"num_write_kb":73722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":278,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680880+0000","last_change":"2024-09-16T03:38:35.828277+0000","last_active":"2024-09-16T03:39:11.680880+0000","last_peered":"2024-09-16T03:39:11.680880+0000","last_clean":"2024-09-16T03:39:11.680880+0000","last_became_active":"2024-09-16T03:38:33.778361+0000","last_became_peered":"2024-09-16T03:38:33.778361+0000","last_unstale":"2024-09-16T03:39:11.680880+0000","last_undegraded":"2024-09-16T03:39:11.680880+0000","last_fullsized":"2024-09-16T03:39:11.680880+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:35.828203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.828203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:49.130136+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":247,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.662354+0000","last_change":"2024-09-16T03:37:52.652365+0000","last_active":"2024-09-16T03:39:11.662354+0000","last_peered":"2024-09-16T03:39:11.662354+0000","last_clean":"2024-09-16T03:39:11.662354+0000","last_became_active":"2024-09-16T03:37:52.651907+0000","last_became_peered":"2024-09-16T03:37:52.651907+0000","last_unstale":"2024-09-16T03:39:11.662354+0000","last_undegraded":"2024-09-16T03:39:11.662354+0000","last_fullsized":"2024-09-16T03:39:11.662354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.985111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"94'34","reported_seq":349,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.660973+0000","last_change":"2024-09-16T03:38:39.081942+0000","last_active":"2024-09-16T03:39:11.660973+0000","last_peered":"2024-09-16T03:39:11.660973+0000","last_clean":"2024-09-16T03:39:11.660973+0000","last_became_active":"2024-09-16T03:38:33.580022+0000","last_became_peered":"2024-09-16T03:38:33.580022+0000","last_unstale":"2024-09-16T03:39:11.660973+0000","last_undegraded":"2024-09-16T03:39:11.660973+0000","last_fullsized":"2024-09-16T03:39:11.660973+0000","mapping_epoch":96,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:39.081810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.081810+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:48.167935+0000","scrub_duration":473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":5,"reported_epoch":135,"state":"activating","last_fresh":"2024-09-16T03:39:09.748033+0000","last_change":"2024-09-16T03:39:09.546055+0000","last_active":"2024-09-16T03:36:52.069777+0000","last_peered":"2024-09-16T03:36:52.069777+0000","last_clean":"2024-09-16T03:36:52.069777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T03:39:09.748033+0000","last_undegraded":"2024-09-16T03:39:09.748033+0000","last_fullsized":"2024-09-16T03:39:09.748033+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":["9","0","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"129'130","reported_seq":568,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.719379+0000","last_change":"2024-09-16T03:38:39.704994+0000","last_active":"2024-09-16T03:39:08.719379+0000","last_peered":"2024-09-16T03:39:08.719379+0000","last_clean":"2024-09-16T03:39:08.719379+0000","last_became_active":"2024-09-16T03:38:33.581141+0000","last_became_peered":"2024-09-16T03:38:33.581141+0000","last_unstale":"2024-09-16T03:39:08.719379+0000","last_undegraded":"2024-09-16T03:39:08.719379+0000","last_fullsized":"2024-09-16T03:39:08.719379+0000","mapping_epoch":32,"log_start":"123'127","ondisk_log_start":"123'127","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'118","last_scrub_stamp":"2024-09-16T03:38:39.704919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.704919+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":124,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.098482+0000","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5882549,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":77034,"num_write":14,"num_write_kb":4474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":248,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641493+0000","last_change":"2024-09-16T03:38:12.041048+0000","last_active":"2024-09-16T03:39:08.641493+0000","last_peered":"2024-09-16T03:39:08.641493+0000","last_clean":"2024-09-16T03:39:08.641493+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:39:08.641493+0000","last_undegraded":"2024-09-16T03:39:08.641493+0000","last_fullsized":"2024-09-16T03:39:08.641493+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:12.040917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.040917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.877949+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":260,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.546220+0000","last_change":"2024-09-16T03:38:34.928371+0000","last_active":"2024-09-16T03:39:09.546220+0000","last_peered":"2024-09-16T03:39:09.546220+0000","last_clean":"2024-09-16T03:39:09.546220+0000","last_became_active":"2024-09-16T03:38:33.484091+0000","last_became_peered":"2024-09-16T03:38:33.484091+0000","last_unstale":"2024-09-16T03:39:09.546220+0000","last_undegraded":"2024-09-16T03:39:09.546220+0000","last_fullsized":"2024-09-16T03:39:09.546220+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:34.928298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.928298+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:43.935101+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":243,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.546720+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:39:09.546720+0000","last_peered":"2024-09-16T03:39:09.546720+0000","last_clean":"2024-09-16T03:39:09.546720+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:39:09.546720+0000","last_undegraded":"2024-09-16T03:39:09.546720+0000","last_fullsized":"2024-09-16T03:39:09.546720+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.232558+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.10","version":"118'64","reported_seq":399,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.616521+0000","last_change":"2024-09-16T03:39:04.442655+0000","last_active":"2024-09-16T03:39:08.616521+0000","last_peered":"2024-09-16T03:39:08.616521+0000","last_clean":"2024-09-16T03:39:08.616521+0000","last_became_active":"2024-09-16T03:39:01.900921+0000","last_became_peered":"2024-09-16T03:39:01.900921+0000","last_unstale":"2024-09-16T03:39:08.616521+0000","last_undegraded":"2024-09-16T03:39:08.616521+0000","last_fullsized":"2024-09-16T03:39:08.616521+0000","mapping_epoch":72,"log_start":"113'61","ondisk_log_start":"113'61","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"118'64","last_scrub_stamp":"2024-09-16T03:39:04.442556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.442556+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.917777+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361433,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"114'17","reported_seq":306,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T03:39:10.647396+0000","last_change":"2024-09-16T03:39:03.929961+0000","last_active":"2024-09-16T03:39:10.647396+0000","last_peered":"2024-09-16T03:39:10.647396+0000","last_clean":"2024-09-16T03:39:10.647396+0000","last_became_active":"2024-09-16T03:39:01.696690+0000","last_became_peered":"2024-09-16T03:39:01.696690+0000","last_unstale":"2024-09-16T03:39:10.647396+0000","last_undegraded":"2024-09-16T03:39:10.647396+0000","last_fullsized":"2024-09-16T03:39:10.647396+0000","mapping_epoch":63,"log_start":"107'14","ondisk_log_start":"107'14","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:03.929863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.929863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:26.527714+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2991679,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"127'20","reported_seq":364,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.471860+0000","last_change":"2024-09-16T03:39:08.648919+0000","last_active":"2024-09-16T03:39:08.648919+0000","last_peered":"2024-09-16T03:39:08.471860+0000","last_clean":"2024-09-16T03:39:08.471860+0000","last_became_active":"2024-09-16T03:39:03.822624+0000","last_became_peered":"2024-09-16T03:39:03.822624+0000","last_unstale":"2024-09-16T03:39:08.471860+0000","last_undegraded":"2024-09-16T03:39:08.471860+0000","last_fullsized":"2024-09-16T03:39:08.471860+0000","mapping_epoch":134,"log_start":"106'17","ondisk_log_start":"106'17","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966452,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":26744,"num_write":10,"num_write_kb":2629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2569696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,9,11],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.13","version":"127'73","reported_seq":458,"reported_epoch":134,"state":"remapped+peering","last_fresh":"2024-09-16T03:39:08.539422+0000","last_change":"2024-09-16T03:39:09.647087+0000","last_active":"2024-09-16T03:39:08.539422+0000","last_peered":"2024-09-16T03:39:08.110262+0000","last_clean":"2024-09-16T03:39:08.110262+0000","last_became_active":"2024-09-16T03:39:01.843090+0000","last_became_peered":"2024-09-16T03:39:01.843090+0000","last_unstale":"2024-09-16T03:39:08.539422+0000","last_undegraded":"2024-09-16T03:39:08.539422+0000","last_fullsized":"2024-09-16T03:39:08.539422+0000","mapping_epoch":135,"log_start":"108'70","ondisk_log_start":"108'70","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'73","last_scrub_stamp":"2024-09-16T03:39:03.124583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.124583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":68,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,9,14],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"104'20","reported_seq":326,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-16T03:39:10.547215+0000","last_change":"2024-09-16T03:39:01.898409+0000","last_active":"2024-09-16T03:39:10.547215+0000","last_peered":"2024-09-16T03:39:10.547215+0000","last_clean":"2024-09-16T03:39:10.547215+0000","last_became_active":"2024-09-16T03:39:01.596287+0000","last_became_peered":"2024-09-16T03:39:01.596287+0000","last_unstale":"2024-09-16T03:39:10.547215+0000","last_undegraded":"2024-09-16T03:39:10.547215+0000","last_fullsized":"2024-09-16T03:39:10.547215+0000","mapping_epoch":114,"log_start":"98'17","ondisk_log_start":"98'17","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:01.898331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.898331+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.472424+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"123'65","reported_seq":450,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.110040+0000","last_change":"2024-09-16T03:39:08.541769+0000","last_active":"2024-09-16T03:39:08.541769+0000","last_peered":"2024-09-16T03:39:08.110040+0000","last_clean":"2024-09-16T03:39:08.110040+0000","last_became_active":"2024-09-16T03:39:03.915707+0000","last_became_peered":"2024-09-16T03:39:03.915707+0000","last_unstale":"2024-09-16T03:39:08.110040+0000","last_undegraded":"2024-09-16T03:39:08.110040+0000","last_fullsized":"2024-09-16T03:39:08.110040+0000","mapping_epoch":134,"log_start":"123'62","ondisk_log_start":"123'62","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916014,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":83986,"num_write":19,"num_write_kb":5459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":826342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,9,10],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"124'86","reported_seq":458,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.681074+0000","last_change":"2024-09-16T03:39:03.073530+0000","last_active":"2024-09-16T03:39:11.681074+0000","last_peered":"2024-09-16T03:39:11.681074+0000","last_clean":"2024-09-16T03:39:11.681074+0000","last_became_active":"2024-09-16T03:39:01.797805+0000","last_became_peered":"2024-09-16T03:39:01.797805+0000","last_unstale":"2024-09-16T03:39:11.681074+0000","last_undegraded":"2024-09-16T03:39:11.681074+0000","last_fullsized":"2024-09-16T03:39:11.681074+0000","mapping_epoch":115,"log_start":"124'82","ondisk_log_start":"124'82","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.073459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.073459+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":80,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:32.204501+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.17","version":"131'52","reported_seq":402,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.741683+0000","last_change":"2024-09-16T03:39:05.721286+0000","last_active":"2024-09-16T03:39:08.741683+0000","last_peered":"2024-09-16T03:39:08.741683+0000","last_clean":"2024-09-16T03:39:08.741683+0000","last_became_active":"2024-09-16T03:39:03.717420+0000","last_became_peered":"2024-09-16T03:39:03.717420+0000","last_unstale":"2024-09-16T03:39:08.741683+0000","last_undegraded":"2024-09-16T03:39:08.741683+0000","last_fullsized":"2024-09-16T03:39:08.741683+0000","mapping_epoch":128,"log_start":"117'49","ondisk_log_start":"117'49","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":48,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.790793+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1546293,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":52,"num_read_kb":44516,"num_write":11,"num_write_kb":3134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2516795,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"128'51","reported_seq":375,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.546548+0000","last_change":"2024-09-16T03:39:02.668794+0000","last_active":"2024-09-16T03:39:09.546548+0000","last_peered":"2024-09-16T03:39:09.546548+0000","last_clean":"2024-09-16T03:39:09.546548+0000","last_became_active":"2024-09-16T03:38:47.562641+0000","last_became_peered":"2024-09-16T03:38:47.562641+0000","last_unstale":"2024-09-16T03:39:09.546548+0000","last_undegraded":"2024-09-16T03:39:09.546548+0000","last_fullsized":"2024-09-16T03:39:09.546548+0000","mapping_epoch":112,"log_start":"124'48","ondisk_log_start":"124'48","created":50,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'48","last_scrub_stamp":"2024-09-16T03:39:02.668681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.668681+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":45,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.451476+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067327,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":21991,"num_write":23,"num_write_kb":6049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":5,"reported_epoch":135,"state":"activating","last_fresh":"2024-09-16T03:39:09.746323+0000","last_change":"2024-09-16T03:39:09.646000+0000","last_active":"2024-09-16T03:39:09.645854+0000","last_peered":"2024-09-16T03:39:07.528715+0000","last_clean":"2024-09-16T03:39:07.528715+0000","last_became_active":"2024-09-16T03:38:51.558378+0000","last_became_peered":"2024-09-16T03:38:51.558378+0000","last_unstale":"2024-09-16T03:39:09.746323+0000","last_undegraded":"2024-09-16T03:39:09.746323+0000","last_fullsized":"2024-09-16T03:39:09.746323+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:01.106840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.106840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":["9","2","6","11"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"128'125","reported_seq":583,"reported_epoch":134,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:08.641112+0000","last_change":"2024-09-16T03:38:47.159152+0000","last_active":"2024-09-16T03:39:08.641112+0000","last_peered":"2024-09-16T03:39:08.641112+0000","last_clean":"2024-09-16T03:38:43.521485+0000","last_became_active":"2024-09-16T03:38:46.957471+0000","last_became_peered":"2024-09-16T03:38:46.957471+0000","last_unstale":"2024-09-16T03:39:08.641112+0000","last_undegraded":"2024-09-16T03:39:08.641112+0000","last_fullsized":"2024-09-16T03:39:08.641112+0000","mapping_epoch":111,"log_start":"117'122","ondisk_log_start":"117'122","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'115","last_scrub_stamp":"2024-09-16T03:38:35.254488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.254488+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":122,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6739673,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":31161,"num_write":9,"num_write_kb":2915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6,11],"acting":[3,8,10],"avail_no_missing":["3","8","10"],"object_location_counts":[{"shards":"3,8,10","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"131'127","reported_seq":584,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.742125+0000","last_change":"2024-09-16T03:38:50.465770+0000","last_active":"2024-09-16T03:39:08.742125+0000","last_peered":"2024-09-16T03:39:08.742125+0000","last_clean":"2024-09-16T03:39:08.742125+0000","last_became_active":"2024-09-16T03:38:50.465463+0000","last_became_peered":"2024-09-16T03:38:50.465463+0000","last_unstale":"2024-09-16T03:39:08.742125+0000","last_undegraded":"2024-09-16T03:39:08.742125+0000","last_fullsized":"2024-09-16T03:39:08.742125+0000","mapping_epoch":115,"log_start":"119'124","ondisk_log_start":"119'124","created":100,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'107","last_scrub_stamp":"2024-09-16T03:38:39.905697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.905697+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":118,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.828899+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5697672,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":62,"num_read_kb":76423,"num_write":24,"num_write_kb":7195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2376197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"99'56","reported_seq":375,"reported_epoch":136,"state":"peering","last_fresh":"2024-09-16T03:39:11.493056+0000","last_change":"2024-09-16T03:39:11.662898+0000","last_active":"2024-09-16T03:39:11.662898+0000","last_peered":"2024-09-16T03:39:11.493056+0000","last_clean":"2024-09-16T03:39:11.493056+0000","last_became_active":"2024-09-16T03:39:11.177251+0000","last_became_peered":"2024-09-16T03:39:11.177251+0000","last_unstale":"2024-09-16T03:39:11.493056+0000","last_undegraded":"2024-09-16T03:39:11.493056+0000","last_fullsized":"2024-09-16T03:39:11.493056+0000","mapping_epoch":137,"log_start":"90'53","ondisk_log_start":"90'53","created":100,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"99'56","last_scrub_stamp":"2024-09-16T03:39:00.988517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.988517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,9,10],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"92'14","reported_seq":279,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.642046+0000","last_change":"2024-09-16T03:38:49.092747+0000","last_active":"2024-09-16T03:39:08.642046+0000","last_peered":"2024-09-16T03:39:08.642046+0000","last_clean":"2024-09-16T03:39:08.642046+0000","last_became_active":"2024-09-16T03:38:48.661213+0000","last_became_peered":"2024-09-16T03:38:48.661213+0000","last_unstale":"2024-09-16T03:39:08.642046+0000","last_undegraded":"2024-09-16T03:39:08.642046+0000","last_fullsized":"2024-09-16T03:39:08.642046+0000","mapping_epoch":113,"log_start":"91'11","ondisk_log_start":"91'11","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:49.092687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.092687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.372348+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,12],"acting":[3,11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"92'59","reported_seq":388,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641478+0000","last_change":"2024-09-16T03:38:48.561603+0000","last_active":"2024-09-16T03:39:08.641478+0000","last_peered":"2024-09-16T03:39:08.641478+0000","last_clean":"2024-09-16T03:39:08.641478+0000","last_became_active":"2024-09-16T03:38:48.561170+0000","last_became_peered":"2024-09-16T03:38:48.561170+0000","last_unstale":"2024-09-16T03:39:08.641478+0000","last_undegraded":"2024-09-16T03:39:08.641478+0000","last_fullsized":"2024-09-16T03:39:08.641478+0000","mapping_epoch":113,"log_start":"89'56","ondisk_log_start":"89'56","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'59","last_scrub_stamp":"2024-09-16T03:38:35.977646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.977646+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:38.632635+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,6],"acting":[14,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1f","version":"131'112","reported_seq":496,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.541483+0000","last_change":"2024-09-16T03:38:48.762617+0000","last_active":"2024-09-16T03:39:08.541483+0000","last_peered":"2024-09-16T03:39:08.541483+0000","last_clean":"2024-09-16T03:39:08.541483+0000","last_became_active":"2024-09-16T03:38:48.761213+0000","last_became_peered":"2024-09-16T03:38:48.761213+0000","last_unstale":"2024-09-16T03:39:08.541483+0000","last_undegraded":"2024-09-16T03:39:08.541483+0000","last_fullsized":"2024-09-16T03:39:08.541483+0000","mapping_epoch":113,"log_start":"118'109","ondisk_log_start":"118'109","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'95","last_scrub_stamp":"2024-09-16T03:38:37.201959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:37.201959+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.307408+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9748159,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":37374,"num_write":27,"num_write_kb":6985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1704821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,12],"acting":[3,10,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"132'144","reported_seq":606,"reported_epoch":135,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:09.547333+0000","last_change":"2024-09-16T03:38:46.862171+0000","last_active":"2024-09-16T03:39:09.547333+0000","last_peered":"2024-09-16T03:39:09.547333+0000","last_clean":"2024-09-16T03:38:44.034266+0000","last_became_active":"2024-09-16T03:38:46.560752+0000","last_became_peered":"2024-09-16T03:38:46.560752+0000","last_unstale":"2024-09-16T03:39:09.547333+0000","last_undegraded":"2024-09-16T03:39:09.547333+0000","last_fullsized":"2024-09-16T03:39:09.547333+0000","mapping_epoch":111,"log_start":"127'141","ondisk_log_start":"127'141","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'118","last_scrub_stamp":"2024-09-16T03:38:36.698150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:36.698150+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":141,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":87880,"num_write":30,"num_write_kb":9656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[4,10,15],"avail_no_missing":["4","10","15"],"object_location_counts":[{"shards":"4,10,15","objects":3}],"blocked_by":[],"up_primary":8,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.21","version":"48'7","reported_seq":141,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641376+0000","last_change":"2024-09-16T03:38:49.064947+0000","last_active":"2024-09-16T03:39:08.641376+0000","last_peered":"2024-09-16T03:39:08.641376+0000","last_clean":"2024-09-16T03:39:08.641376+0000","last_became_active":"2024-09-16T03:38:48.559431+0000","last_became_peered":"2024-09-16T03:38:48.559431+0000","last_unstale":"2024-09-16T03:39:08.641376+0000","last_undegraded":"2024-09-16T03:39:08.641376+0000","last_fullsized":"2024-09-16T03:39:08.641376+0000","mapping_epoch":113,"log_start":"40'4","ondisk_log_start":"40'4","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":6,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:49.064870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064870+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.310170+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,10],"acting":[14,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.22","version":"45'6","reported_seq":130,"reported_epoch":133,"state":"remapped+peering","last_fresh":"2024-09-16T03:39:08.110164+0000","last_change":"2024-09-16T03:39:09.546581+0000","last_active":"2024-09-16T03:39:09.546581+0000","last_peered":"2024-09-16T03:39:08.110164+0000","last_clean":"2024-09-16T03:39:08.110164+0000","last_became_active":"2024-09-16T03:38:54.462316+0000","last_became_peered":"2024-09-16T03:38:54.462316+0000","last_unstale":"2024-09-16T03:39:08.110164+0000","last_undegraded":"2024-09-16T03:39:08.110164+0000","last_fullsized":"2024-09-16T03:39:08.110164+0000","mapping_epoch":135,"log_start":"39'3","ondisk_log_start":"39'3","created":100,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:39:00.968448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.968448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,6],"acting":[4,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,9,12],"up_primary":9,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"108'38","reported_seq":355,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.741629+0000","last_change":"2024-09-16T03:38:49.464614+0000","last_active":"2024-09-16T03:39:08.741629+0000","last_peered":"2024-09-16T03:39:08.741629+0000","last_clean":"2024-09-16T03:39:08.741629+0000","last_became_active":"2024-09-16T03:38:49.463797+0000","last_became_peered":"2024-09-16T03:38:49.463797+0000","last_unstale":"2024-09-16T03:39:08.741629+0000","last_undegraded":"2024-09-16T03:39:08.741629+0000","last_fullsized":"2024-09-16T03:39:08.741629+0000","mapping_epoch":114,"log_start":"105'36","ondisk_log_start":"105'36","created":100,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:34.893524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.893524+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":35,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:52.829073+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,2],"acting":[14,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.24","version":"105'32","reported_seq":337,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.761038+0000","last_change":"2024-09-16T03:38:47.978956+0000","last_active":"2024-09-16T03:39:11.761038+0000","last_peered":"2024-09-16T03:39:11.761038+0000","last_clean":"2024-09-16T03:39:11.761038+0000","last_became_active":"2024-09-16T03:38:45.820201+0000","last_became_peered":"2024-09-16T03:38:45.820201+0000","last_unstale":"2024-09-16T03:39:11.761038+0000","last_undegraded":"2024-09-16T03:39:11.761038+0000","last_fullsized":"2024-09-16T03:39:11.761038+0000","mapping_epoch":110,"log_start":"83'29","ondisk_log_start":"83'29","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.978888+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:56.881865+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.25","version":"130'85","reported_seq":426,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680968+0000","last_change":"2024-09-16T03:38:48.088833+0000","last_active":"2024-09-16T03:39:11.680968+0000","last_peered":"2024-09-16T03:39:11.680968+0000","last_clean":"2024-09-16T03:39:11.680968+0000","last_became_active":"2024-09-16T03:38:45.724757+0000","last_became_peered":"2024-09-16T03:38:45.724757+0000","last_unstale":"2024-09-16T03:39:11.680968+0000","last_undegraded":"2024-09-16T03:39:11.680968+0000","last_fullsized":"2024-09-16T03:39:11.680968+0000","mapping_epoch":110,"log_start":"116'82","ondisk_log_start":"116'82","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'77","last_scrub_stamp":"2024-09-16T03:38:48.088768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.088768+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":79,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:53.112635+0000","scrub_duration":388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2134555,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":10,"num_write_kb":2500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.26","version":"133'54","reported_seq":402,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.552465+0000","last_change":"2024-09-16T03:38:49.477393+0000","last_active":"2024-09-16T03:39:08.552465+0000","last_peered":"2024-09-16T03:39:08.552465+0000","last_clean":"2024-09-16T03:39:08.552465+0000","last_became_active":"2024-09-16T03:38:45.926029+0000","last_became_peered":"2024-09-16T03:38:45.926029+0000","last_unstale":"2024-09-16T03:39:08.552465+0000","last_undegraded":"2024-09-16T03:39:08.552465+0000","last_fullsized":"2024-09-16T03:39:08.552465+0000","mapping_epoch":110,"log_start":"123'51","ondisk_log_start":"123'51","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:49.477297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.477297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":50,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.242430+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":22,"num_read_kb":24203,"num_write":11,"num_write_kb":3140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.27","version":"109'66","reported_seq":389,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.552485+0000","last_change":"2024-09-16T03:38:47.511343+0000","last_active":"2024-09-16T03:39:08.552485+0000","last_peered":"2024-09-16T03:39:08.552485+0000","last_clean":"2024-09-16T03:39:08.552485+0000","last_became_active":"2024-09-16T03:38:45.824884+0000","last_became_peered":"2024-09-16T03:38:45.824884+0000","last_unstale":"2024-09-16T03:39:08.552485+0000","last_undegraded":"2024-09-16T03:39:08.552485+0000","last_fullsized":"2024-09-16T03:39:08.552485+0000","mapping_epoch":110,"log_start":"105'63","ondisk_log_start":"105'63","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'66","last_scrub_stamp":"2024-09-16T03:38:47.511252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.511252+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.884916+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.28","version":"129'68","reported_seq":454,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:08.372306+0000","last_change":"2024-09-16T03:39:08.541570+0000","last_active":"2024-09-16T03:39:08.541570+0000","last_peered":"2024-09-16T03:39:08.372306+0000","last_clean":"2024-09-16T03:39:08.372306+0000","last_became_active":"2024-09-16T03:38:55.727094+0000","last_became_peered":"2024-09-16T03:38:55.727094+0000","last_unstale":"2024-09-16T03:39:08.372306+0000","last_undegraded":"2024-09-16T03:39:08.372306+0000","last_fullsized":"2024-09-16T03:39:08.372306+0000","mapping_epoch":134,"log_start":"118'65","ondisk_log_start":"118'65","created":110,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:48.551929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.551929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":64,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395113,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":31731,"num_write":5,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3610944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,9,10],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":251,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.641395+0000","last_change":"2024-09-16T03:39:00.834877+0000","last_active":"2024-09-16T03:39:08.641395+0000","last_peered":"2024-09-16T03:39:08.641395+0000","last_clean":"2024-09-16T03:39:08.641395+0000","last_became_active":"2024-09-16T03:38:45.627369+0000","last_became_peered":"2024-09-16T03:38:45.627369+0000","last_unstale":"2024-09-16T03:39:08.641395+0000","last_undegraded":"2024-09-16T03:39:08.641395+0000","last_fullsized":"2024-09-16T03:39:08.641395+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:00.834742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.834742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.318226+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.2a","version":"108'122","reported_seq":606,"reported_epoch":135,"state":"active+clean","last_fresh":"2024-09-16T03:39:09.546800+0000","last_change":"2024-09-16T03:38:47.964525+0000","last_active":"2024-09-16T03:39:09.546800+0000","last_peered":"2024-09-16T03:39:09.546800+0000","last_clean":"2024-09-16T03:39:09.546800+0000","last_became_active":"2024-09-16T03:38:46.022108+0000","last_became_peered":"2024-09-16T03:38:46.022108+0000","last_unstale":"2024-09-16T03:39:09.546800+0000","last_undegraded":"2024-09-16T03:39:09.546800+0000","last_fullsized":"2024-09-16T03:39:09.546800+0000","mapping_epoch":110,"log_start":"107'115","ondisk_log_start":"107'115","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'122","last_scrub_stamp":"2024-09-16T03:38:47.964439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.964439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":112,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.917123+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.2b","version":"127'124","reported_seq":590,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680858+0000","last_change":"2024-09-16T03:39:09.949533+0000","last_active":"2024-09-16T03:39:11.680858+0000","last_peered":"2024-09-16T03:39:11.680858+0000","last_clean":"2024-09-16T03:39:11.680858+0000","last_became_active":"2024-09-16T03:39:09.949146+0000","last_became_peered":"2024-09-16T03:39:09.949146+0000","last_unstale":"2024-09-16T03:39:11.680858+0000","last_undegraded":"2024-09-16T03:39:11.680858+0000","last_fullsized":"2024-09-16T03:39:11.680858+0000","mapping_epoch":134,"log_start":"119'121","ondisk_log_start":"119'121","created":110,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:49.064061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064061+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":121,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:16.396765+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2c","version":"121'63","reported_seq":408,"reported_epoch":133,"state":"peering","last_fresh":"2024-09-16T03:39:07.529785+0000","last_change":"2024-09-16T03:39:08.548352+0000","last_active":"2024-09-16T03:39:08.548352+0000","last_peered":"2024-09-16T03:39:07.529785+0000","last_clean":"2024-09-16T03:39:07.529785+0000","last_became_active":"2024-09-16T03:38:55.562810+0000","last_became_peered":"2024-09-16T03:38:55.562810+0000","last_unstale":"2024-09-16T03:39:07.529785+0000","last_undegraded":"2024-09-16T03:39:07.529785+0000","last_fullsized":"2024-09-16T03:39:07.529785+0000","mapping_epoch":134,"log_start":"118'60","ondisk_log_start":"118'60","created":110,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'58","last_scrub_stamp":"2024-09-16T03:38:47.889276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.889276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":59,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":7651,"num_write":5,"num_write_kb":1234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1262403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[9,11,15],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.2d","version":"92'14","reported_seq":332,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-09-16T03:39:08.548592+0000","last_change":"2024-09-16T03:38:47.115739+0000","last_active":"2024-09-16T03:39:08.548592+0000","last_peered":"2024-09-16T03:39:08.548592+0000","last_clean":"2024-09-16T03:39:08.548592+0000","last_became_active":"2024-09-16T03:38:45.723438+0000","last_became_peered":"2024-09-16T03:38:45.723438+0000","last_unstale":"2024-09-16T03:39:08.548592+0000","last_undegraded":"2024-09-16T03:39:08.548592+0000","last_fullsized":"2024-09-16T03:39:08.548592+0000","mapping_epoch":110,"log_start":"91'11","ondisk_log_start":"91'11","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:47.115651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.115651+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:00.107802+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":56,"stat_sum":{"num_bytes":128634223,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":956,"num_read_kb":893485,"num_write":332,"num_write_kb":91950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":21395486,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":231534592,"data_stored":231174352,"data_compressed":572208,"data_compressed_allocated":626688,"data_compressed_original":1503232,"omap_allocated":0,"internal_metadata":0},"log_size":171,"ondisk_log_size":171,"up":168,"acting":168,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":81118503,"num_objects":91,"num_object_clones":0,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":923,"num_read_kb":29771,"num_write":1258,"num_write_kb":281639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":39649106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":178487296,"data_stored":191047270,"data_compressed":11544299,"data_compressed_allocated":12292096,"data_compressed_original":25247744,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":2644640,"data_compressed":159066,"data_compressed_allocated":204800,"data_compressed_original":2584576,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247838,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74436,"kb_used_data":43248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94297404,"statfs":{"total":96636764160,"available":96560541696,"internally_reserved":0,"allocated":44285952,"data_stored":44886400,"data_compressed":1343912,"data_compressed_allocated":1433600,"data_compressed_original":2613248,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247839,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56776,"kb_used_data":25584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94315064,"statfs":{"total":96636764160,"available":96578625536,"internally_reserved":0,"allocated":26198016,"data_stored":27310439,"data_compressed":905243,"data_compressed_allocated":991232,"data_compressed_original":2641920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247838,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68900,"kb_used_data":37700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302940,"statfs":{"total":96636764160,"available":96566210560,"internally_reserved":0,"allocated":38604800,"data_stored":39803729,"data_compressed":1368231,"data_compressed_allocated":1474560,"data_compressed_original":3207168,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280543,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59508,"kb_used_data":23688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94312332,"statfs":{"total":96636764160,"available":96575827968,"internally_reserved":0,"allocated":24256512,"data_stored":25354703,"data_compressed":886305,"data_compressed_allocated":966656,"data_compressed_original":2551808,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280543,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280542,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313247,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34820,"kb_used_data":7788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337020,"statfs":{"total":96636764160,"available":96601108480,"internally_reserved":0,"allocated":7974912,"data_stored":7570474,"data_compressed":25616,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280543,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88796,"kb_used_data":53444,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94283044,"statfs":{"total":96636764160,"available":96545837056,"internally_reserved":0,"allocated":54726656,"data_stored":55922960,"data_compressed":1410985,"data_compressed_allocated":1519616,"data_compressed_original":3321856,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280542,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94552,"kb_used_data":63360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94277288,"statfs":{"total":96636764160,"available":96539942912,"internally_reserved":0,"allocated":64880640,"data_stored":67044955,"data_compressed":2176984,"data_compressed_allocated":2338816,"data_compressed_original":5111808,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313247,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280542,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35892,"kb_used_data":4704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335948,"statfs":{"total":96636764160,"available":96600010752,"internally_reserved":0,"allocated":4816896,"data_stored":4453365,"data_compressed":25616,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280542,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69660,"kb_used_data":38468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302180,"statfs":{"total":96636764160,"available":96565432320,"internally_reserved":0,"allocated":39391232,"data_stored":40017276,"data_compressed":1267990,"data_compressed_allocated":1351680,"data_compressed_original":2510848,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280542,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34052,"kb_used_data":2864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337788,"statfs":{"total":96636764160,"available":96601894912,"internally_reserved":0,"allocated":2932736,"data_stored":3069533,"data_compressed":69039,"data_compressed_allocated":90112,"data_compressed_original":618496,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280543,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83532,"kb_used_data":52344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94288308,"statfs":{"total":96636764160,"available":96551227392,"internally_reserved":0,"allocated":53600256,"data_stored":53264385,"data_compressed":186317,"data_compressed_allocated":212992,"data_compressed_original":532480,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,3,4,5,6,7,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313246,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43140,"kb_used_data":11944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328700,"statfs":{"total":96636764160,"available":96592588800,"internally_reserved":0,"allocated":12230656,"data_stored":11891032,"data_compressed":109264,"data_compressed_allocated":126976,"data_compressed_original":270336,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247839,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55588,"kb_used_data":24392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316252,"statfs":{"total":96636764160,"available":96579842048,"internally_reserved":0,"allocated":24977408,"data_stored":25627821,"data_compressed":1314158,"data_compressed_allocated":1400832,"data_compressed_original":2527232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14012416,"data_stored":15061447,"data_compressed":829441,"data_compressed_allocated":888832,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13422592,"data_stored":14113910,"data_compressed":606276,"data_compressed_allocated":643072,"data_compressed_original":1376256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13844480,"data_stored":14888812,"data_compressed":1232826,"data_compressed_allocated":1302528,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22970368,"data_stored":24522128,"data_compressed":1292173,"data_compressed_allocated":1380352,"data_compressed_original":2985984,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":36982784,"data_stored":39583575,"data_compressed":2121614,"data_compressed_allocated":2269184,"data_compressed_original":4952064,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24047616,"data_stored":25639824,"data_compressed":1378911,"data_compressed_allocated":1470464,"data_compressed_original":3117056,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":13844480,"data_stored":14888812,"data_compressed":1232826,"data_compressed_allocated":1302528,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14012416,"data_stored":15061447,"data_compressed":829441,"data_compressed_allocated":888832,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":13844480,"data_stored":14888812,"data_compressed":1232826,"data_compressed_allocated":1302528,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8982528,"data_stored":8928361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9904128,"data_stored":9871428,"data_compressed":49947,"data_compressed_allocated":53248,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30552064,"data_stored":30549819,"data_compressed":93196,"data_compressed_allocated":102400,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":26701824,"data_stored":26610529,"data_compressed":29754,"data_compressed_allocated":32768,"data_compressed_original":86016,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":14409728,"data_stored":14413518,"data_compressed":44673,"data_compressed_allocated":49152,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3641344,"data_stored":3612626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24342528,"data_stored":24277619,"data_compressed":9548,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1683456,"data_stored":1690918,"data_compressed":10762,"data_compressed_allocated":12288,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":52391936,"data_stored":52412861,"data_compressed":160701,"data_compressed_allocated":176128,"data_compressed_original":458752,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":10862592,"data_stored":10841283,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":11001856,"data_stored":11037652,"data_compressed":83648,"data_compressed_allocated":90112,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":6758400,"data_stored":6718533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":29212672,"data_stored":29129165,"data_compressed":79701,"data_compressed_allocated":86016,"data_compressed_original":151552,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:39:14.098 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:14.162 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:14.162 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:14.167 INFO:tasks.rados.rados.0.smithi157.stdout:1744: finishing write tid 1 to smithi15736696-29 2024-09-16T03:39:14.167 INFO:tasks.rados.rados.0.smithi157.stdout:1744: finishing write tid 2 to smithi15736696-29 2024-09-16T03:39:14.167 INFO:tasks.rados.rados.0.smithi157.stdout:1740: finishing set_chunk to oid 37 2024-09-16T03:39:14.168 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 76 (ObjNum 356 snap 0 seq_num 356) dirty exists 2024-09-16T03:39:14.168 INFO:tasks.rados.rados.0.smithi157.stdout:1744: finishing write tid 3 to smithi15736696-29 2024-09-16T03:39:14.168 INFO:tasks.rados.rados.0.smithi157.stdout:1744: finishing write tid 4 to smithi15736696-29 2024-09-16T03:39:14.168 INFO:tasks.rados.rados.0.smithi157.stdout:1744: oid 29 updating version 0 to 133 2024-09-16T03:39:14.168 INFO:tasks.rados.rados.0.smithi157.stdout:1744: oid 29 version 133 is already newer than 131 2024-09-16T03:39:14.168 INFO:tasks.rados.rados.0.smithi157.stdout:1744: oid 29 version 133 is already newer than 132 2024-09-16T03:39:14.168 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 133 (ObjNum 384 snap 0 seq_num 384) dirty exists 2024-09-16T03:39:14.169 INFO:tasks.rados.rados.0.smithi157.stdout:1744: left oid 29 (ObjNum 384 snap 0 seq_num 384) 2024-09-16T03:39:14.171 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:39:14.176 INFO:tasks.rados.rados.0.smithi157.stdout:1731: done (12 left) 2024-09-16T03:39:14.176 INFO:tasks.rados.rados.0.smithi157.stdout:1734: done (11 left) 2024-09-16T03:39:14.176 INFO:tasks.rados.rados.0.smithi157.stdout:1738: done (10 left) 2024-09-16T03:39:14.176 INFO:tasks.rados.rados.0.smithi157.stdout:1740: done (9 left) 2024-09-16T03:39:14.176 INFO:tasks.rados.rados.0.smithi157.stdout:1741: done (8 left) 2024-09-16T03:39:14.176 INFO:tasks.rados.rados.0.smithi157.stdout:1744: done (7 left) 2024-09-16T03:39:14.176 INFO:tasks.rados.rados.0.smithi157.stdout:1745: done (6 left) 2024-09-16T03:39:14.176 INFO:tasks.rados.rados.0.smithi157.stdout:1746: done (5 left) 2024-09-16T03:39:14.176 INFO:tasks.rados.rados.0.smithi157.stdout:1747: done (4 left) 2024-09-16T03:39:14.176 INFO:tasks.rados.rados.0.smithi157.stdout:1748: read oid 37 snap -1 2024-09-16T03:39:14.176 INFO:tasks.rados.rados.0.smithi157.stdout:1748: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-16T03:39:14.176 INFO:tasks.rados.rados.0.smithi157.stdout:1749: write oid 15 current snap is 0 2024-09-16T03:39:14.177 INFO:tasks.rados.rados.0.smithi157.stdout:1749: seq_num 386 ranges {487502=463764,1738167=446469} 2024-09-16T03:39:14.180 INFO:tasks.rados.rados.0.smithi157.stdout:1749: writing smithi15736696-15 from 487502 to 951266 tid 1 2024-09-16T03:39:14.188 INFO:tasks.rados.rados.0.smithi157.stdout:1749: writing smithi15736696-15 from 1738167 to 2184636 tid 2 2024-09-16T03:39:14.188 INFO:tasks.rados.rados.0.smithi157.stdout:1750: set_chunk oid 45 target oid 2024-09-16T03:39:14.192 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:14.190+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:14.281 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:14.282 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:14.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:39:14.293 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:14.291+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:14.395 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:14.392+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:14.399 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:14.399 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:14.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:39:14.409 INFO:tasks.rados.rados.0.smithi157.stdout:1750: set_chunk oid 45 offset: 378880 length: 2840576 target oid ad5fc4f9b019d1796f9e0b5472c45e7119501595b62d92cfdd1f37f8ccd5c7b3 offset: 0 snap -1 2024-09-16T03:39:14.409 INFO:tasks.rados.rados.0.smithi157.stdout:1751: write (excl) oid 28 current snap is 0 2024-09-16T03:39:14.409 INFO:tasks.rados.rados.0.smithi157.stdout:1751: seq_num 387 ranges {504309=539013,1796649=590127} 2024-09-16T03:39:14.414 INFO:tasks.rados.rados.0.smithi157.stdout:1751: writing smithi15736696-28 from 504309 to 1043322 tid 1 2024-09-16T03:39:14.421 INFO:tasks.rados.rados.0.smithi157.stdout:1751: writing smithi15736696-28 from 1796649 to 2386776 tid 2 2024-09-16T03:39:14.421 INFO:tasks.rados.rados.0.smithi157.stdout:1752: read oid 46 snap -1 2024-09-16T03:39:14.421 INFO:tasks.rados.rados.0.smithi157.stdout:1752: expect (ObjNum 383 snap 0 seq_num 383) 2024-09-16T03:39:14.421 INFO:tasks.rados.rados.0.smithi157.stdout:1753: read oid 1 snap -1 2024-09-16T03:39:14.421 INFO:tasks.rados.rados.0.smithi157.stdout:1753: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-16T03:39:14.422 INFO:tasks.rados.rados.0.smithi157.stdout:1754: tier_promote oid 50 2024-09-16T03:39:14.422 INFO:tasks.rados.rados.0.smithi157.stdout:1742: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:39:14.422 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:14.436 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:39:14.495 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:14.493+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:14.517 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:14.518 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:14.528 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:39:14.581 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:14.578+0000 7fedb5739640 -1 osd.9 139 handle_osd_map injecting map crc failure 2024-09-16T03:39:14.583 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:14.584 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:14.596 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:14.594+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:14.602 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:14.602 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:14.620 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:14.621 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1325278 2024-09-16T03:39:14.621 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:14.637 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:14.637 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:14.641 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:14.641 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:14.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:39:14.654 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:14.654 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:14.671 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:14.671 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:14.689 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:14.689 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:14.690 INFO:tasks.rados.rados.0.smithi157.stdout:1743: expect (ObjNum 381 snap 0 seq_num 381) 2024-09-16T03:39:14.697 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:14.694+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:14.706 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:14.706 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:14.723 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:14.724 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:14.741 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:14.741 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:14.757 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:14.758 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:14.759 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:14.759 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:14.759 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:14.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:39:14.774 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:14.775 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:14.797 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:14.795+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:14.808 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:14.809 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:14.811 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1821350 2024-09-16T03:39:14.824 INFO:tasks.rados.rados.0.smithi157.stdout:1748: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-16T03:39:14.824 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:14.826 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:14.826 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:14.857 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:14.857 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:14.873 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:14.874 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:14.897 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:14.897 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:14.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:14.897 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:14.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.897 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:73 4.a 4:532814f2:::smithi15736696-19:head [write 702526~787188,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:14.897 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:39:14.897 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.58287801500001, 2024-09-16T03:39:14.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.362642662, 2024-09-16T03:39:14.897 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 73 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311260+0000", 2024-09-16T03:39:14.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311261+0000", 2024-09-16T03:39:14.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.562e-06 2024-09-16T03:39:14.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311925+0000", 2024-09-16T03:39:14.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00066385499999999998 2024-09-16T03:39:14.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311937+0000", 2024-09-16T03:39:14.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1881e-05 2024-09-16T03:39:14.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311943+0000", 2024-09-16T03:39:14.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6999999999999996e-06 2024-09-16T03:39:14.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.311988+0000", 2024-09-16T03:39:14.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5358000000000002e-05 2024-09-16T03:39:14.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312070+0000", 2024-09-16T03:39:14.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1735000000000003e-05 2024-09-16T03:39:14.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:14.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.312167+0000", 2024-09-16T03:39:14.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6685e-05 2024-09-16T03:39:14.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:14.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.313661+0000", 2024-09-16T03:39:14.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001494324 2024-09-16T03:39:14.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.414759+0000", 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10109850300000001 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673821+0000", 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.25906127400000001 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673858+0000", 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7620999999999998e-05 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.673902+0000", 2024-09-16T03:39:14.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4163999999999999e-05 2024-09-16T03:39:14.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.904 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.904 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:303 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:14.904 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:39:14.904 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 118.482419185, 2024-09-16T03:39:14.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50956986000000004, 2024-09-16T03:39:14.904 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.904 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:14.905 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.905 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.905 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.905 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 303 2024-09-16T03:39:14.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.905 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:39:14.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411721+0000", 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0200000000000001e-06 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411729+0000", 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1240000000000005e-06 2024-09-16T03:39:14.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411744+0000", 2024-09-16T03:39:14.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099000000000001e-05 2024-09-16T03:39:14.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411751+0000", 2024-09-16T03:39:14.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.029e-06 2024-09-16T03:39:14.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511892+0000", 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100140879 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511961+0000", 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9379000000000001e-05 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.613647+0000", 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101685994 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:14.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.614294+0000", 2024-09-16T03:39:14.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064692799999999996 2024-09-16T03:39:14.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.714962+0000", 2024-09-16T03:39:14.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066765 2024-09-16T03:39:14.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921157+0000", 2024-09-16T03:39:14.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206195463 2024-09-16T03:39:14.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:14.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921233+0000", 2024-09-16T03:39:14.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5446999999999998e-05 2024-09-16T03:39:14.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921288+0000", 2024-09-16T03:39:14.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5847999999999998e-05 2024-09-16T03:39:14.910 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.910 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.910 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:503 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:39:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.072344633, 2024-09-16T03:39:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60521621599999997, 2024-09-16T03:39:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 503 2024-09-16T03:39:14.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.912 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:39:14.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:39:14.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821794+0000", 2024-09-16T03:39:14.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-09-16T03:39:14.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821798+0000", 2024-09-16T03:39:14.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2030000000000002e-06 2024-09-16T03:39:14.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821804+0000", 2024-09-16T03:39:14.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:39:14.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821810+0000", 2024-09-16T03:39:14.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4249999999999997e-06 2024-09-16T03:39:14.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922212+0000", 2024-09-16T03:39:14.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040228800000001 2024-09-16T03:39:14.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922307+0000", 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5089999999999999e-05 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.126007+0000", 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20369931499999999 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.127605+0000", 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001598213 2024-09-16T03:39:14.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.327263+0000", 2024-09-16T03:39:14.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199657905 2024-09-16T03:39:14.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426900+0000", 2024-09-16T03:39:14.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099637217 2024-09-16T03:39:14.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:14.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426963+0000", 2024-09-16T03:39:14.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:39:14.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.427009+0000", 2024-09-16T03:39:14.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6533000000000003e-05 2024-09-16T03:39:14.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.917 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.917 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1227 4.a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v18,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 98.218168368999997, 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.44160531400000003, 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1227 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:39:14.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:39:14.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675972+0000", 2024-09-16T03:39:14.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7269999999999999e-06 2024-09-16T03:39:14.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675981+0000", 2024-09-16T03:39:14.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.166e-06 2024-09-16T03:39:14.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675996+0000", 2024-09-16T03:39:14.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4901e-05 2024-09-16T03:39:14.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676005+0000", 2024-09-16T03:39:14.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4579999999999993e-06 2024-09-16T03:39:14.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676081+0000", 2024-09-16T03:39:14.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6157999999999998e-05 2024-09-16T03:39:14.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676188+0000", 2024-09-16T03:39:14.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106871 2024-09-16T03:39:14.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:14.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.016594+0000", 2024-09-16T03:39:14.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34040622300000001 2024-09-16T03:39:14.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:14.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017089+0000", 2024-09-16T03:39:14.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049523899999999997 2024-09-16T03:39:14.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017779+0000", 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069032200000000005 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117504+0000", 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099724207999999995 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117543+0000", 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9208999999999997e-05 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117575+0000", 2024-09-16T03:39:14.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1832000000000003e-05 2024-09-16T03:39:14.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.924 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.924 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1523 4.a 4:50107449:::smithi15736696-27:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:39:14.924 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:39:14.924 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 89.531127042999998, 2024-09-16T03:39:14.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.410522742, 2024-09-16T03:39:14.924 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.924 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.924 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1523 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363012+0000", 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.269e-06 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363016+0000", 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8920000000000002e-06 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363027+0000", 2024-09-16T03:39:14.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1079e-05 2024-09-16T03:39:14.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463160+0000", 2024-09-16T03:39:14.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10013267100000001 2024-09-16T03:39:14.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463297+0000", 2024-09-16T03:39:14.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013717699999999999 2024-09-16T03:39:14.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463381+0000", 2024-09-16T03:39:14.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3701000000000004e-05 2024-09-16T03:39:14.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.773533+0000", 2024-09-16T03:39:14.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31015295300000001 2024-09-16T03:39:14.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.929 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.929 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1690 4.a 4:51338165:::smithi15736696-4:head [assert-version v63,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:14.929 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:39:14.929 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.443854376000004, 2024-09-16T03:39:14.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52002349699999995, 2024-09-16T03:39:14.929 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.929 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:14.929 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.930 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.930 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.930 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1690 2024-09-16T03:39:14.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.930 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:39:14.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:39:14.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450286+0000", 2024-09-16T03:39:14.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-16T03:39:14.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450293+0000", 2024-09-16T03:39:14.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5089999999999996e-06 2024-09-16T03:39:14.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450308+0000", 2024-09-16T03:39:14.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4955e-05 2024-09-16T03:39:14.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450316+0000", 2024-09-16T03:39:14.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7079999999999997e-06 2024-09-16T03:39:14.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450386+0000", 2024-09-16T03:39:14.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9715999999999995e-05 2024-09-16T03:39:14.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450478+0000", 2024-09-16T03:39:14.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2610999999999998e-05 2024-09-16T03:39:14.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:14.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.665802+0000", 2024-09-16T03:39:14.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21532342700000001 2024-09-16T03:39:14.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:14.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668724+0000", 2024-09-16T03:39:14.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029220600000000002 2024-09-16T03:39:14.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668758+0000", 2024-09-16T03:39:14.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.434e-05 2024-09-16T03:39:14.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970243+0000", 2024-09-16T03:39:14.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30148501 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970278+0000", 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4965000000000002e-05 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970307+0000", 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8968000000000001e-05 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1654 4.12 4:4a9d235d:::smithi15736696-1:head [write 652138~461367 in=461367b,stat out=16b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.352100884999999, 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42790198699999998, 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1654 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.936 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542039+0000", 2024-09-16T03:39:14.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0250000000000001e-06 2024-09-16T03:39:14.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542682+0000", 2024-09-16T03:39:14.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064305099999999995 2024-09-16T03:39:14.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542698+0000", 2024-09-16T03:39:14.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6469000000000001e-05 2024-09-16T03:39:14.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542710+0000", 2024-09-16T03:39:14.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1173e-05 2024-09-16T03:39:14.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542767+0000", 2024-09-16T03:39:14.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7584000000000003e-05 2024-09-16T03:39:14.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:14.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542775+0000", 2024-09-16T03:39:14.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6149999999999999e-06 2024-09-16T03:39:14.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666597+0000", 2024-09-16T03:39:14.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123821815 2024-09-16T03:39:14.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666745+0000", 2024-09-16T03:39:14.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148675 2024-09-16T03:39:14.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:39:14.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666935+0000", 2024-09-16T03:39:14.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018963899999999999 2024-09-16T03:39:14.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969207+0000", 2024-09-16T03:39:14.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302272031 2024-09-16T03:39:14.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969374+0000", 2024-09-16T03:39:14.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016686699999999999 2024-09-16T03:39:14.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969533+0000", 2024-09-16T03:39:14.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015909299999999999 2024-09-16T03:39:14.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:14.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969560+0000", 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7148000000000002e-05 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969631+0000", 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1174000000000007e-05 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969685+0000", 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3613000000000003e-05 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:14.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969764+0000", 2024-09-16T03:39:14.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9281000000000002e-05 2024-09-16T03:39:14.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969939+0000", 2024-09-16T03:39:14.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473400000000001 2024-09-16T03:39:14.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.944 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.944 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.945 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1655 4.12 4:4a9d235d:::smithi15736696-1:head [write 1574940~677198 in=677198b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:14.945 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:39:14.945 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.351408766999995, 2024-09-16T03:39:14.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42773147700000003, 2024-09-16T03:39:14.945 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.945 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.945 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:14.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.946 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1655 2024-09-16T03:39:14.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.946 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:39:14.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542730+0000", 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.085e-06 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543338+0000", 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060826100000000002 2024-09-16T03:39:14.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543351+0000", 2024-09-16T03:39:14.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2789000000000001e-05 2024-09-16T03:39:14.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543360+0000", 2024-09-16T03:39:14.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1970000000000002e-06 2024-09-16T03:39:14.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543420+0000", 2024-09-16T03:39:14.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9562999999999999e-05 2024-09-16T03:39:14.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:14.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543431+0000", 2024-09-16T03:39:14.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0866999999999999e-05 2024-09-16T03:39:14.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.768963+0000", 2024-09-16T03:39:14.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.225532487 2024-09-16T03:39:14.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769029+0000", 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5881000000000002e-05 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769134+0000", 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104923 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969639+0000", 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20050517900000001 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969978+0000", 2024-09-16T03:39:14.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033848599999999998 2024-09-16T03:39:14.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970206+0000", 2024-09-16T03:39:14.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022867300000000001 2024-09-16T03:39:14.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970248+0000", 2024-09-16T03:39:14.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1433999999999997e-05 2024-09-16T03:39:14.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970299+0000", 2024-09-16T03:39:14.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0677999999999998e-05 2024-09-16T03:39:14.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970338+0000", 2024-09-16T03:39:14.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9646e-05 2024-09-16T03:39:14.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:14.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970377+0000", 2024-09-16T03:39:14.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8439999999999998e-05 2024-09-16T03:39:14.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970460+0000", 2024-09-16T03:39:14.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3887999999999994e-05 2024-09-16T03:39:14.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.953 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.953 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1656 4.12 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2252138] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:14.954 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:39:14.954 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.350762695, 2024-09-16T03:39:14.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72904596899999996, 2024-09-16T03:39:14.954 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.954 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.954 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:14.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.954 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1656 2024-09-16T03:39:14.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.954 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:39:14.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:39:14.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543376+0000", 2024-09-16T03:39:14.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4200000000000004e-07 2024-09-16T03:39:14.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543383+0000", 2024-09-16T03:39:14.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6290000000000004e-06 2024-09-16T03:39:14.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543390+0000", 2024-09-16T03:39:14.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:39:14.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543396+0000", 2024-09-16T03:39:14.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.02e-06 2024-09-16T03:39:14.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543461+0000", 2024-09-16T03:39:14.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4876999999999996e-05 2024-09-16T03:39:14.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:14.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543465+0000", 2024-09-16T03:39:14.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2139999999999998e-06 2024-09-16T03:39:14.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970490+0000", 2024-09-16T03:39:14.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42702478300000002 2024-09-16T03:39:14.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970556+0000", 2024-09-16T03:39:14.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6137999999999996e-05 2024-09-16T03:39:14.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:39:14.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970670+0000", 2024-09-16T03:39:14.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114416 2024-09-16T03:39:14.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:14.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971375+0000", 2024-09-16T03:39:14.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070496000000000003 2024-09-16T03:39:14.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071773+0000", 2024-09-16T03:39:14.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100397945 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071823+0000", 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.072067+0000", 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243856 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.171991+0000", 2024-09-16T03:39:14.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099924052999999999 2024-09-16T03:39:14.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272176+0000", 2024-09-16T03:39:14.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100184963 2024-09-16T03:39:14.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:14.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272251+0000", 2024-09-16T03:39:14.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5902000000000006e-05 2024-09-16T03:39:14.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272421+0000", 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016962800000000001 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1657 4.12 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.350730709000004, 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72934181899999995, 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.962 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1657 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:39:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543408+0000", 2024-09-16T03:39:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.13e-07 2024-09-16T03:39:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543410+0000", 2024-09-16T03:39:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7030000000000002e-06 2024-09-16T03:39:14.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543417+0000", 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.567e-06 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543423+0000", 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0619999999999997e-06 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543489+0000", 2024-09-16T03:39:14.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6315999999999993e-05 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543498+0000", 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7239999999999998e-06 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970952+0000", 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42745424399999998 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971012+0000", 2024-09-16T03:39:14.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9771000000000001e-05 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272463+0000", 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30145090800000002 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272558+0000", 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4863000000000003e-05 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272749+0000", 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191048 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2105 4.a 4:51338165:::smithi15736696-4:head [sparse-read 0~887916,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.845266515000006, 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40244934799999998, 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:14.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2105 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048871+0000", 2024-09-16T03:39:14.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048881+0000", 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6080000000000002e-06 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048889+0000", 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7209999999999994e-06 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048903+0000", 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4538e-05 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048912+0000", 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2390000000000008e-06 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149039+0000", 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100126563 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:39:14.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149044+0000", 2024-09-16T03:39:14.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0340000000000004e-06 2024-09-16T03:39:14.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451087+0000", 2024-09-16T03:39:14.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30204318000000002 2024-09-16T03:39:14.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451159+0000", 2024-09-16T03:39:14.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1627999999999999e-05 2024-09-16T03:39:14.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451321+0000", 2024-09-16T03:39:14.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016183699999999999 2024-09-16T03:39:14.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.973 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.973 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.973 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2106 4.a 4:51338165:::smithi15736696-4:head [sparse-read 0~887916] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:39:14.973 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:39:14.973 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.845205755999999, 2024-09-16T03:39:14.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40254453200000001, 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2106 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048933+0000", 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048936+0000", 2024-09-16T03:39:14.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3890000000000001e-06 2024-09-16T03:39:14.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048943+0000", 2024-09-16T03:39:14.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1250000000000004e-06 2024-09-16T03:39:14.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048953+0000", 2024-09-16T03:39:14.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8190000000000003e-06 2024-09-16T03:39:14.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149113+0000", 2024-09-16T03:39:14.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100160007 2024-09-16T03:39:14.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:14.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149118+0000", 2024-09-16T03:39:14.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7650000000000001e-06 2024-09-16T03:39:14.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451341+0000", 2024-09-16T03:39:14.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30222344699999998 2024-09-16T03:39:14.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451381+0000", 2024-09-16T03:39:14.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9517000000000001e-05 2024-09-16T03:39:14.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451477+0000", 2024-09-16T03:39:14.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5656999999999996e-05 2024-09-16T03:39:14.978 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2107 4.a 4:51338165:::smithi15736696-4:head [read 1698310~486600,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.845175693000002, 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40288642400000002, 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2107 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048963+0000", 2024-09-16T03:39:14.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:39:14.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048967+0000", 2024-09-16T03:39:14.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9060000000000004e-06 2024-09-16T03:39:14.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048972+0000", 2024-09-16T03:39:14.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5910000000000003e-06 2024-09-16T03:39:14.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048977+0000", 2024-09-16T03:39:14.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9629999999999997e-06 2024-09-16T03:39:14.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149135+0000", 2024-09-16T03:39:14.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015828 2024-09-16T03:39:14.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:14.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149138+0000", 2024-09-16T03:39:14.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.779e-06 2024-09-16T03:39:14.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451506+0000", 2024-09-16T03:39:14.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302367891 2024-09-16T03:39:14.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451558+0000", 2024-09-16T03:39:14.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2374999999999999e-05 2024-09-16T03:39:14.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451849+0000", 2024-09-16T03:39:14.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029009899999999999 2024-09-16T03:39:14.984 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.984 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.984 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.984 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2537 4.a 4:503187b3:::smithi15736696-34:head [assert-version v87,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:14.984 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:17.047079+0000", 2024-09-16T03:39:14.984 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.847058863000001, 2024-09-16T03:39:14.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.32976008000000001, 2024-09-16T03:39:14.985 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.985 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.985 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:14.985 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.985 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.985 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.985 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2537 2024-09-16T03:39:14.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.985 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047079+0000", 2024-09-16T03:39:14.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047079+0000", 2024-09-16T03:39:14.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047081+0000", 2024-09-16T03:39:14.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6929999999999999e-06 2024-09-16T03:39:14.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047086+0000", 2024-09-16T03:39:14.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8680000000000004e-06 2024-09-16T03:39:14.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047099+0000", 2024-09-16T03:39:14.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2309e-05 2024-09-16T03:39:14.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047105+0000", 2024-09-16T03:39:14.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7050000000000002e-06 2024-09-16T03:39:14.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047157+0000", 2024-09-16T03:39:14.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1057999999999998e-05 2024-09-16T03:39:14.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.047219+0000", 2024-09-16T03:39:14.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2553999999999998e-05 2024-09-16T03:39:14.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:14.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.149371+0000", 2024-09-16T03:39:14.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.102152112 2024-09-16T03:39:14.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:14.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.149846+0000", 2024-09-16T03:39:14.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047467400000000002 2024-09-16T03:39:14.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.183065+0000", 2024-09-16T03:39:14.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033219374000000003 2024-09-16T03:39:14.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:14.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.376793+0000", 2024-09-16T03:39:14.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.193728023 2024-09-16T03:39:14.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:14.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.376819+0000", 2024-09-16T03:39:14.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6142000000000001e-05 2024-09-16T03:39:14.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:17.376839+0000", 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9568e-05 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2687 4.a 4:503187b3:::smithi15736696-34:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:21.869464+0000", 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 53.024673419999999, 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31565330200000002, 2024-09-16T03:39:14.991 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2687 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869464+0000", 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869464+0000", 2024-09-16T03:39:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869465+0000", 2024-09-16T03:39:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.51e-07 2024-09-16T03:39:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869467+0000", 2024-09-16T03:39:14.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:39:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869473+0000", 2024-09-16T03:39:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9560000000000002e-06 2024-09-16T03:39:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869488+0000", 2024-09-16T03:39:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5333e-05 2024-09-16T03:39:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869667+0000", 2024-09-16T03:39:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017900699999999999 2024-09-16T03:39:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.869710+0000", 2024-09-16T03:39:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2749e-05 2024-09-16T03:39:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:22.185118+0000", 2024-09-16T03:39:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31540757600000002 2024-09-16T03:39:14.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2986 4.a 4:5a5e9bc2:::smithi15736696-36:head [write 487469~563838] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:39:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.912930+0000", 2024-09-16T03:39:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.981208244000001, 2024-09-16T03:39:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39506163799999999, 2024-09-16T03:39:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:14.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2986 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.912930+0000", 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.912930+0000", 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:14.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.912932+0000", 2024-09-16T03:39:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8169999999999999e-06 2024-09-16T03:39:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913522+0000", 2024-09-16T03:39:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058972099999999997 2024-09-16T03:39:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913539+0000", 2024-09-16T03:39:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7306999999999999e-05 2024-09-16T03:39:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913550+0000", 2024-09-16T03:39:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0356e-05 2024-09-16T03:39:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:14.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913637+0000", 2024-09-16T03:39:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7552999999999993e-05 2024-09-16T03:39:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913699+0000", 2024-09-16T03:39:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1991999999999999e-05 2024-09-16T03:39:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.913797+0000", 2024-09-16T03:39:15.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7738000000000005e-05 2024-09-16T03:39:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.915370+0000", 2024-09-16T03:39:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015725719999999999 2024-09-16T03:39:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.916260+0000", 2024-09-16T03:39:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089050100000000003 2024-09-16T03:39:15.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.307864+0000", 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39160404900000001 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.307936+0000", 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2025000000000002e-05 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.307991+0000", 2024-09-16T03:39:15.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5006999999999997e-05 2024-09-16T03:39:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2987 4.a 4:5a5e9bc2:::smithi15736696-36:head [write 1607751~551752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:39:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:39:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.975190204, 2024-09-16T03:39:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49279943700000001, 2024-09-16T03:39:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.003 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2987 2024-09-16T03:39:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:39:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:39:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918949+0000", 2024-09-16T03:39:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.86e-06 2024-09-16T03:39:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919579+0000", 2024-09-16T03:39:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062909000000000001 2024-09-16T03:39:15.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919591+0000", 2024-09-16T03:39:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2078e-05 2024-09-16T03:39:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919596+0000", 2024-09-16T03:39:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:39:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019705+0000", 2024-09-16T03:39:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100108405 2024-09-16T03:39:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019754+0000", 2024-09-16T03:39:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9258999999999997e-05 2024-09-16T03:39:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019847+0000", 2024-09-16T03:39:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2954999999999993e-05 2024-09-16T03:39:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.023003+0000", 2024-09-16T03:39:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031561110000000001 2024-09-16T03:39:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.122544+0000", 2024-09-16T03:39:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099540553000000004 2024-09-16T03:39:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411658+0000", 2024-09-16T03:39:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.28911426699999998 2024-09-16T03:39:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411696+0000", 2024-09-16T03:39:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8430000000000003e-05 2024-09-16T03:39:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411747+0000", 2024-09-16T03:39:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0745999999999998e-05 2024-09-16T03:39:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2988 4.a 4:5a5e9bc2:::smithi15736696-36:head [setxattr _header (56) in=7b,truncate 2487469] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:39:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:39:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.974528464999999, 2024-09-16T03:39:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59236646500000001, 2024-09-16T03:39:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2988 2024-09-16T03:39:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:39:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:39:15.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919610+0000", 2024-09-16T03:39:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.048e-06 2024-09-16T03:39:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919615+0000", 2024-09-16T03:39:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8239999999999999e-06 2024-09-16T03:39:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919621+0000", 2024-09-16T03:39:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4380000000000003e-06 2024-09-16T03:39:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919626+0000", 2024-09-16T03:39:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2000000000000002e-06 2024-09-16T03:39:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.020922+0000", 2024-09-16T03:39:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101295782 2024-09-16T03:39:15.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.020995+0000", 2024-09-16T03:39:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3256999999999993e-05 2024-09-16T03:39:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.021088+0000", 2024-09-16T03:39:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3419999999999997e-05 2024-09-16T03:39:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:15.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.023007+0000", 2024-09-16T03:39:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019186380000000001 2024-09-16T03:39:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.222530+0000", 2024-09-16T03:39:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19952276899999999 2024-09-16T03:39:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511887+0000", 2024-09-16T03:39:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.289357378 2024-09-16T03:39:15.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511927+0000", 2024-09-16T03:39:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0357e-05 2024-09-16T03:39:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511976+0000", 2024-09-16T03:39:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8353999999999998e-05 2024-09-16T03:39:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3258 4.1a 4:5e3c0760:::smithi15736696-24:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:39:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:39:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.464077590000002, 2024-09-16T03:39:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3302292470000001, 2024-09-16T03:39:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3258 2024-09-16T03:39:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:39:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:39:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430061+0000", 2024-09-16T03:39:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.27e-07 2024-09-16T03:39:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430062+0000", 2024-09-16T03:39:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6780000000000001e-06 2024-09-16T03:39:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430066+0000", 2024-09-16T03:39:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5269999999999999e-06 2024-09-16T03:39:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430069+0000", 2024-09-16T03:39:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:39:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430119+0000", 2024-09-16T03:39:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0212e-05 2024-09-16T03:39:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:39:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430122+0000", 2024-09-16T03:39:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7319999999999998e-06 2024-09-16T03:39:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.631493+0000", 2024-09-16T03:39:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20137142499999999 2024-09-16T03:39:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.631579+0000", 2024-09-16T03:39:15.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5418000000000001e-05 2024-09-16T03:39:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:41.760289+0000", 2024-09-16T03:39:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1287106680000001 2024-09-16T03:39:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3701 4.a 4:51338165:::smithi15736696-4:head [assert-version v115,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e121)", 2024-09-16T03:39:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:39:15.022 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.523231051, 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50414302200000005, 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3701 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:39:15.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:39:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370910+0000", 2024-09-16T03:39:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1200000000000002e-06 2024-09-16T03:39:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370923+0000", 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3078000000000001e-05 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370940+0000", 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7461000000000001e-05 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370950+0000", 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0091999999999999e-05 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.371029+0000", 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8879000000000002e-05 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.371136+0000", 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106983 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.573316+0000", 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20217934500000001 2024-09-16T03:39:15.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.573914+0000", 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059857900000000002 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.763374+0000", 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18945932200000001 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.874936+0000", 2024-09-16T03:39:15.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111562256 2024-09-16T03:39:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.875002+0000", 2024-09-16T03:39:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6050000000000006e-05 2024-09-16T03:39:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.875050+0000", 2024-09-16T03:39:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7856999999999999e-05 2024-09-16T03:39:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.029 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.029 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:15.030 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2364503 2024-09-16T03:39:15.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:39:15.033 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:15.028+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:15.131 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:15.129+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:15.155 INFO:tasks.rados.rados.0.smithi157.stdout:1749: finishing write tid 1 to smithi15736696-15 2024-09-16T03:39:15.155 INFO:tasks.rados.rados.0.smithi157.stdout:1729: finishing set_chunk to oid 6 2024-09-16T03:39:15.155 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 67 (ObjNum 327 snap 0 seq_num 327) dirty exists 2024-09-16T03:39:15.155 INFO:tasks.rados.rados.0.smithi157.stdout:1750: finishing set_chunk to oid 45 2024-09-16T03:39:15.156 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 45 set_chunk returned error code -95 offset: 378880 length: 2840576 2024-09-16T03:39:15.156 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 0 (ObjNum 325 snap 0 seq_num 325) dirty exists 2024-09-16T03:39:15.156 INFO:tasks.rados.rados.0.smithi157.stdout:1753: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-16T03:39:15.160 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 543277 2024-09-16T03:39:15.180 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:15.180 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:15.180 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:39:15.180 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:15.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:39:15.233 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:15.230+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:15.295 INFO:tasks.rados.rados.0.smithi157.stdout:1749: finishing write tid 2 to smithi15736696-15 2024-09-16T03:39:15.295 INFO:tasks.rados.rados.0.smithi157.stdout:1754: got (0) Success 2024-09-16T03:39:15.295 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 143 (ObjNum 355 snap 0 seq_num 355) dirty exists 2024-09-16T03:39:15.296 INFO:tasks.rados.rados.0.smithi157.stdout:1749: finishing write tid 3 to smithi15736696-15 2024-09-16T03:39:15.296 INFO:tasks.rados.rados.0.smithi157.stdout:1729: done (10 left) 2024-09-16T03:39:15.296 INFO:tasks.rados.rados.0.smithi157.stdout:1742: done (9 left) 2024-09-16T03:39:15.296 INFO:tasks.rados.rados.0.smithi157.stdout:1743: done (8 left) 2024-09-16T03:39:15.296 INFO:tasks.rados.rados.0.smithi157.stdout:1748: done (7 left) 2024-09-16T03:39:15.296 INFO:tasks.rados.rados.0.smithi157.stdout:1750: done (6 left) 2024-09-16T03:39:15.296 INFO:tasks.rados.rados.0.smithi157.stdout:1753: done (5 left) 2024-09-16T03:39:15.296 INFO:tasks.rados.rados.0.smithi157.stdout:1754: done (4 left) 2024-09-16T03:39:15.296 INFO:tasks.rados.rados.0.smithi157.stdout:1755: write (excl) oid 16 current snap is 0 2024-09-16T03:39:15.296 INFO:tasks.rados.rados.0.smithi157.stdout:1755: seq_num 388 ranges {521116=730615,1971484=549632} 2024-09-16T03:39:15.301 INFO:tasks.rados.rados.0.smithi157.stdout:1755: writing smithi15736696-16 from 521116 to 1251731 tid 1 2024-09-16T03:39:15.309 INFO:tasks.rados.rados.0.smithi157.stdout:1755: writing smithi15736696-16 from 1971484 to 2521116 tid 2 2024-09-16T03:39:15.310 INFO:tasks.rados.rados.0.smithi157.stdout:1756: read oid 39 snap -1 2024-09-16T03:39:15.310 INFO:tasks.rados.rados.0.smithi157.stdout:1756: expect (ObjNum 365 snap 0 seq_num 365) 2024-09-16T03:39:15.310 INFO:tasks.rados.rados.0.smithi157.stdout:1749: finishing write tid 4 to smithi15736696-15 2024-09-16T03:39:15.310 INFO:tasks.rados.rados.0.smithi157.stdout:1749: oid 15 updating version 0 to 21 2024-09-16T03:39:15.310 INFO:tasks.rados.rados.0.smithi157.stdout:1749: oid 15 version 21 is already newer than 20 2024-09-16T03:39:15.310 INFO:tasks.rados.rados.0.smithi157.stdout:1749: oid 15 updating version 21 to 22 2024-09-16T03:39:15.310 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 22 (ObjNum 386 snap 0 seq_num 386) dirty exists 2024-09-16T03:39:15.310 INFO:tasks.rados.rados.0.smithi157.stdout:1749: left oid 15 (ObjNum 386 snap 0 seq_num 386) 2024-09-16T03:39:15.310 INFO:tasks.rados.rados.0.smithi157.stdout:1751: finishing write tid 1 to smithi15736696-28 2024-09-16T03:39:15.310 INFO:tasks.rados.rados.0.smithi157.stdout:1751: finishing write tid 2 to smithi15736696-28 2024-09-16T03:39:15.310 INFO:tasks.rados.rados.0.smithi157.stdout:1751: finishing write tid 3 to smithi15736696-28 2024-09-16T03:39:15.310 INFO:tasks.rados.rados.0.smithi157.stdout:1751: finishing write tid 4 to smithi15736696-28 2024-09-16T03:39:15.310 INFO:tasks.rados.rados.0.smithi157.stdout:1751: oid 28 updating version 0 to 134 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:1751: oid 28 version 134 is already newer than 132 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:1751: oid 28 version 134 is already newer than 133 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 134 (ObjNum 387 snap 0 seq_num 387) dirty exists 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:1751: left oid 28 (ObjNum 387 snap 0 seq_num 387) 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:1757: read oid 32 snap -1 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:1757: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:1749: done (6 left) 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:1751: done (5 left) 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:1758: read oid 26 snap -1 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:1758: expect (ObjNum 379 snap 0 seq_num 379) 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:1759: read oid 48 snap -1 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:1759: expect (ObjNum 367 snap 0 seq_num 367) 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:1760: read oid 4 snap -1 2024-09-16T03:39:15.311 INFO:tasks.rados.rados.0.smithi157.stdout:1760: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-16T03:39:15.312 INFO:tasks.rados.rados.0.smithi157.stdout:1761: read oid 45 snap -1 2024-09-16T03:39:15.312 INFO:tasks.rados.rados.0.smithi157.stdout:1761: expect (ObjNum 325 snap 0 seq_num 325) 2024-09-16T03:39:15.312 INFO:tasks.rados.rados.0.smithi157.stdout:1762: read oid 5 snap -1 2024-09-16T03:39:15.312 INFO:tasks.rados.rados.0.smithi157.stdout:1762: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-16T03:39:15.312 INFO:tasks.rados.rados.0.smithi157.stdout:1763: write (excl) oid 29 current snap is 0 2024-09-16T03:39:15.312 INFO:tasks.rados.rados.0.smithi157.stdout:1763: seq_num 389 ranges {537923=405864,1513613=593796} 2024-09-16T03:39:15.314 INFO:tasks.rados.rados.0.smithi157.stdout:1763: writing smithi15736696-29 from 537923 to 943787 tid 1 2024-09-16T03:39:15.321 INFO:tasks.rados.rados.0.smithi157.stdout:1763: writing smithi15736696-29 from 1513613 to 2107409 tid 2 2024-09-16T03:39:15.321 INFO:tasks.rados.rados.0.smithi157.stdout:1764: write (excl) oid 44 current snap is 0 2024-09-16T03:39:15.321 INFO:tasks.rados.rados.0.smithi157.stdout:1752: expect (ObjNum 383 snap 0 seq_num 383) 2024-09-16T03:39:15.321 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:15.334 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:15.331+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:15.339 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:15.339 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:15.339 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:15.339 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:15.339 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:15.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:39:15.428 INFO:tasks.rados.rados.0.smithi157.stdout:1762: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-16T03:39:15.434 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:15.432+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:15.458 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:432 4.c 4:34492f38:::smithi15736696-33:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:39:15.458 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 114.541585517, 2024-09-16T03:39:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.404460861, 2024-09-16T03:39:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 432 2024-09-16T03:39:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:39:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:39:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913833+0000", 2024-09-16T03:39:15.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0610000000000001e-06 2024-09-16T03:39:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913839+0000", 2024-09-16T03:39:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9479999999999996e-06 2024-09-16T03:39:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913848+0000", 2024-09-16T03:39:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5629999999999992e-06 2024-09-16T03:39:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913857+0000", 2024-09-16T03:39:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0399999999999998e-06 2024-09-16T03:39:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914022+0000", 2024-09-16T03:39:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000165422 2024-09-16T03:39:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914101+0000", 2024-09-16T03:39:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8462999999999997e-05 2024-09-16T03:39:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217171+0000", 2024-09-16T03:39:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30307040000000002 2024-09-16T03:39:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217854+0000", 2024-09-16T03:39:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068278700000000002 2024-09-16T03:39:15.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.218098+0000", 2024-09-16T03:39:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243825 2024-09-16T03:39:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318164+0000", 2024-09-16T03:39:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100066451 2024-09-16T03:39:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318232+0000", 2024-09-16T03:39:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8113000000000003e-05 2024-09-16T03:39:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318293+0000", 2024-09-16T03:39:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0788000000000001e-05 2024-09-16T03:39:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:790 4.d 4:b3fe05b2:::smithi15736696-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 109.567445443, 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41456747799999999, 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 790 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:39:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:39:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887975+0000", 2024-09-16T03:39:15.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0369999999999999e-06 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887987+0000", 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1606e-05 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888016+0000", 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.917e-05 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888032+0000", 2024-09-16T03:39:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5616000000000001e-05 2024-09-16T03:39:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888178+0000", 2024-09-16T03:39:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612100000000001 2024-09-16T03:39:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888307+0000", 2024-09-16T03:39:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012953899999999999 2024-09-16T03:39:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.302540+0000", 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41423238899999998 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1418 4.c 4:3313e97c:::smithi15736696-21:head [write 615156~555834] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.229334270999999, 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40891108799999998, 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.470 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1418 2024-09-16T03:39:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:39:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:39:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226087+0000", 2024-09-16T03:39:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8890000000000004e-06 2024-09-16T03:39:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226827+0000", 2024-09-16T03:39:15.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000739298 2024-09-16T03:39:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226848+0000", 2024-09-16T03:39:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1418000000000001e-05 2024-09-16T03:39:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226858+0000", 2024-09-16T03:39:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0434e-05 2024-09-16T03:39:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327036+0000", 2024-09-16T03:39:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10017769 2024-09-16T03:39:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327098+0000", 2024-09-16T03:39:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1760999999999994e-05 2024-09-16T03:39:15.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327230+0000", 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131839 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434369+0000", 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107139788 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434444+0000", 2024-09-16T03:39:15.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4171999999999994e-05 2024-09-16T03:39:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634830+0000", 2024-09-16T03:39:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20038660599999999 2024-09-16T03:39:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634937+0000", 2024-09-16T03:39:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010716499999999999 2024-09-16T03:39:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634994+0000", 2024-09-16T03:39:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7027999999999997e-05 2024-09-16T03:39:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1419 4.c 4:3313e97c:::smithi15736696-21:head [write 1603467~416361] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:39:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.224668882000003, 2024-09-16T03:39:15.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40430432100000002, 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1419 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:39:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230752+0000", 2024-09-16T03:39:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.354e-06 2024-09-16T03:39:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231164+0000", 2024-09-16T03:39:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041208599999999998 2024-09-16T03:39:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231179+0000", 2024-09-16T03:39:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4885999999999999e-05 2024-09-16T03:39:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231186+0000", 2024-09-16T03:39:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8759999999999999e-06 2024-09-16T03:39:15.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328726+0000", 2024-09-16T03:39:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097539855999999994 2024-09-16T03:39:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328786+0000", 2024-09-16T03:39:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0059000000000002e-05 2024-09-16T03:39:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328890+0000", 2024-09-16T03:39:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104338 2024-09-16T03:39:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434475+0000", 2024-09-16T03:39:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.105584709 2024-09-16T03:39:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.534594+0000", 2024-09-16T03:39:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10011914099999999 2024-09-16T03:39:15.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635003+0000", 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100408834 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635029+0000", 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6551999999999999e-05 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635053+0000", 2024-09-16T03:39:15.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3629999999999999e-05 2024-09-16T03:39:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1421 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:39:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.215986122000004, 2024-09-16T03:39:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.497492088, 2024-09-16T03:39:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.484 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1421 2024-09-16T03:39:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:39:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:39:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239432+0000", 2024-09-16T03:39:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5999999999999991e-07 2024-09-16T03:39:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239436+0000", 2024-09-16T03:39:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:39:15.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239443+0000", 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.968e-06 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239447+0000", 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6969999999999999e-06 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635225+0000", 2024-09-16T03:39:15.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39577807399999998 2024-09-16T03:39:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635290+0000", 2024-09-16T03:39:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4583000000000002e-05 2024-09-16T03:39:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635410+0000", 2024-09-16T03:39:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120684 2024-09-16T03:39:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636180+0000", 2024-09-16T03:39:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076997899999999995 2024-09-16T03:39:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636799+0000", 2024-09-16T03:39:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061906600000000004 2024-09-16T03:39:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736742+0000", 2024-09-16T03:39:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099943041999999996 2024-09-16T03:39:15.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736843+0000", 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010076799999999999 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736924+0000", 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0419000000000004e-05 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1422 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.215964842000005, 2024-09-16T03:39:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49764818700000002, 2024-09-16T03:39:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1422 2024-09-16T03:39:15.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:39:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239455+0000", 2024-09-16T03:39:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9860000000000001e-06 2024-09-16T03:39:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239461+0000", 2024-09-16T03:39:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:39:15.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239464+0000", 2024-09-16T03:39:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2619999999999999e-06 2024-09-16T03:39:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635736+0000", 2024-09-16T03:39:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.396272287 2024-09-16T03:39:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:15.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635790+0000", 2024-09-16T03:39:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4252000000000002e-05 2024-09-16T03:39:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736964+0000", 2024-09-16T03:39:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101173319 2024-09-16T03:39:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737024+0000", 2024-09-16T03:39:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9975000000000001e-05 2024-09-16T03:39:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737101+0000", 2024-09-16T03:39:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7303000000000002e-05 2024-09-16T03:39:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2336 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:39:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:39:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.360377724000003, 2024-09-16T03:39:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.407180869, 2024-09-16T03:39:15.496 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2336 2024-09-16T03:39:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:39:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:39:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:39:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2399999999999999e-07 2024-09-16T03:39:15.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095042+0000", 2024-09-16T03:39:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.567e-06 2024-09-16T03:39:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095046+0000", 2024-09-16T03:39:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.639e-06 2024-09-16T03:39:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195114+0000", 2024-09-16T03:39:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10006748 2024-09-16T03:39:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195173+0000", 2024-09-16T03:39:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8727000000000002e-05 2024-09-16T03:39:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195278+0000", 2024-09-16T03:39:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010522900000000001 2024-09-16T03:39:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.501699+0000", 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30642159299999999 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.501995+0000", 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000295822 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.502221+0000", 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022548800000000001 2024-09-16T03:39:15.501 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2642 4.c 4:34492f38:::smithi15736696-33:head [write 665610~797860,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:39:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.579230979000002, 2024-09-16T03:39:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40488026700000002, 2024-09-16T03:39:15.502 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2642 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:39:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876189+0000", 2024-09-16T03:39:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0549999999999998e-06 2024-09-16T03:39:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877003+0000", 2024-09-16T03:39:15.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081467900000000001 2024-09-16T03:39:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877018+0000", 2024-09-16T03:39:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.471e-05 2024-09-16T03:39:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877025+0000", 2024-09-16T03:39:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.379e-06 2024-09-16T03:39:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977237+0000", 2024-09-16T03:39:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100211914 2024-09-16T03:39:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977354+0000", 2024-09-16T03:39:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116416 2024-09-16T03:39:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977480+0000", 2024-09-16T03:39:15.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012629899999999999 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.984474+0000", 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069943230000000002 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084635+0000", 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100161018 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185809+0000", 2024-09-16T03:39:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10117406700000001 2024-09-16T03:39:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.280666+0000", 2024-09-16T03:39:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094856491000000001 2024-09-16T03:39:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.280857+0000", 2024-09-16T03:39:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019133999999999999 2024-09-16T03:39:15.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.281067+0000", 2024-09-16T03:39:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000209576 2024-09-16T03:39:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2643 4.c 4:34492f38:::smithi15736696-33:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:39:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.574564092999999, 2024-09-16T03:39:15.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50004631700000002, 2024-09-16T03:39:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2643 2024-09-16T03:39:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880854+0000", 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6599999999999994e-07 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881441+0000", 2024-09-16T03:39:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058613000000000003 2024-09-16T03:39:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881450+0000", 2024-09-16T03:39:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6260000000000007e-06 2024-09-16T03:39:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881455+0000", 2024-09-16T03:39:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.515e-06 2024-09-16T03:39:15.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979409+0000", 2024-09-16T03:39:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097954645000000007 2024-09-16T03:39:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979512+0000", 2024-09-16T03:39:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102701 2024-09-16T03:39:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:15.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979684+0000", 2024-09-16T03:39:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017176299999999999 2024-09-16T03:39:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084699+0000", 2024-09-16T03:39:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10501545599999999 2024-09-16T03:39:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084714+0000", 2024-09-16T03:39:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4738e-05 2024-09-16T03:39:15.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185858+0000", 2024-09-16T03:39:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101144074 2024-09-16T03:39:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380720+0000", 2024-09-16T03:39:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.194862072 2024-09-16T03:39:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380783+0000", 2024-09-16T03:39:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3262000000000005e-05 2024-09-16T03:39:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380900+0000", 2024-09-16T03:39:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116369 2024-09-16T03:39:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2644 4.c 4:34492f38:::smithi15736696-33:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:39:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.569887598000001, 2024-09-16T03:39:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.495569859, 2024-09-16T03:39:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2644 2024-09-16T03:39:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:39:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:39:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885531+0000", 2024-09-16T03:39:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-16T03:39:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885813+0000", 2024-09-16T03:39:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028188399999999999 2024-09-16T03:39:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885824+0000", 2024-09-16T03:39:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-16T03:39:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885830+0000", 2024-09-16T03:39:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6289999999999998e-06 2024-09-16T03:39:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983163+0000", 2024-09-16T03:39:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097333127000000005 2024-09-16T03:39:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983273+0000", 2024-09-16T03:39:15.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109868 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983458+0000", 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018554899999999999 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084722+0000", 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101263244 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185717+0000", 2024-09-16T03:39:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100995575 2024-09-16T03:39:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185904+0000", 2024-09-16T03:39:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186682 2024-09-16T03:39:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380965+0000", 2024-09-16T03:39:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.195060921 2024-09-16T03:39:15.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381030+0000", 2024-09-16T03:39:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5174999999999999e-05 2024-09-16T03:39:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381100+0000", 2024-09-16T03:39:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9877999999999997e-05 2024-09-16T03:39:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2645 4.c 4:34492f38:::smithi15736696-33:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:39:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.569578096999997, 2024-09-16T03:39:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59563304800000005, 2024-09-16T03:39:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.524 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2645 2024-09-16T03:39:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:39:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:39:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885840+0000", 2024-09-16T03:39:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-09-16T03:39:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885843+0000", 2024-09-16T03:39:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.785e-06 2024-09-16T03:39:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885848+0000", 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3979999999999998e-06 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885852+0000", 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6509999999999999e-06 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084768+0000", 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19891652000000001 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084831+0000", 2024-09-16T03:39:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2717000000000003e-05 2024-09-16T03:39:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084964+0000", 2024-09-16T03:39:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132731 2024-09-16T03:39:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185772+0000", 2024-09-16T03:39:15.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10080797399999999 2024-09-16T03:39:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185790+0000", 2024-09-16T03:39:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7966999999999999e-05 2024-09-16T03:39:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185959+0000", 2024-09-16T03:39:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168777 2024-09-16T03:39:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481235+0000", 2024-09-16T03:39:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29527644800000002 2024-09-16T03:39:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481307+0000", 2024-09-16T03:39:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1539999999999996e-05 2024-09-16T03:39:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481473+0000", 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016595600000000001 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2646 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.569561407999998, 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59596245699999995, 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.531 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2646 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:39:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885857+0000", 2024-09-16T03:39:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:39:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885860+0000", 2024-09-16T03:39:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8940000000000003e-06 2024-09-16T03:39:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885864+0000", 2024-09-16T03:39:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.602e-06 2024-09-16T03:39:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885867+0000", 2024-09-16T03:39:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0089999999999999e-06 2024-09-16T03:39:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085196+0000", 2024-09-16T03:39:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19932923399999999 2024-09-16T03:39:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085287+0000", 2024-09-16T03:39:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0480000000000001e-05 2024-09-16T03:39:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481521+0000", 2024-09-16T03:39:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39623439799999999 2024-09-16T03:39:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481613+0000", 2024-09-16T03:39:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2276999999999998e-05 2024-09-16T03:39:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481819+0000", 2024-09-16T03:39:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020521499999999999 2024-09-16T03:39:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3554 4.c 4.3e97c8cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:39:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.005388383, 2024-09-16T03:39:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2069473150000001, 2024-09-16T03:39:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3554 2024-09-16T03:39:15.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.538 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:39:15.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:39:15.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450032+0000", 2024-09-16T03:39:15.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9639999999999999e-06 2024-09-16T03:39:15.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450041+0000", 2024-09-16T03:39:15.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-16T03:39:15.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450059+0000", 2024-09-16T03:39:15.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-16T03:39:15.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450069+0000", 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8320000000000001e-06 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.467620+0000", 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017550676000000001 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.467626+0000", 2024-09-16T03:39:15.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3430000000000001e-06 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656941+0000", 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1893151820000001 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656977+0000", 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5491999999999998e-05 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3581 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.429881569999999, 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.63145314900000005, 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3581 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.542 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:39:15.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:39:15.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025539+0000", 2024-09-16T03:39:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4510000000000001e-06 2024-09-16T03:39:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026168+0000", 2024-09-16T03:39:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062808199999999997 2024-09-16T03:39:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026186+0000", 2024-09-16T03:39:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8694000000000001e-05 2024-09-16T03:39:15.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026196+0000", 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8800000000000003e-06 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035407+0000", 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092111020000000005 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035416+0000", 2024-09-16T03:39:15.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9870000000000005e-06 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.557740+0000", 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.522323918 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656989+0000", 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099249034999999999 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3582 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.422639725, 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62429066099999997, 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3582 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.547 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:39:15.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:39:15.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032779+0000", 2024-09-16T03:39:15.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2720000000000001e-06 2024-09-16T03:39:15.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035239+0000", 2024-09-16T03:39:15.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024594180000000001 2024-09-16T03:39:15.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035261+0000", 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2246e-05 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035273+0000", 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2289999999999999e-05 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038105+0000", 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028319780000000002 2024-09-16T03:39:15.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038111+0000", 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3099999999999997e-06 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657045+0000", 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61893337199999998 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657069+0000", 2024-09-16T03:39:15.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3774999999999999e-05 2024-09-16T03:39:15.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.552 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.552 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.552 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3583 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:15.552 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:39:15.552 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.417531595, 2024-09-16T03:39:15.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61919224299999998, 2024-09-16T03:39:15.552 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.552 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.552 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:15.553 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.553 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.553 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.553 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3583 2024-09-16T03:39:15.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.553 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:39:15.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:39:15.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037888+0000", 2024-09-16T03:39:15.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-06 2024-09-16T03:39:15.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038032+0000", 2024-09-16T03:39:15.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014443799999999999 2024-09-16T03:39:15.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038045+0000", 2024-09-16T03:39:15.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3417999999999999e-05 2024-09-16T03:39:15.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038052+0000", 2024-09-16T03:39:15.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3890000000000002e-06 2024-09-16T03:39:15.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038133+0000", 2024-09-16T03:39:15.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0856e-05 2024-09-16T03:39:15.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:15.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038136+0000", 2024-09-16T03:39:15.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7160000000000001e-06 2024-09-16T03:39:15.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657073+0000", 2024-09-16T03:39:15.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61893632899999995 2024-09-16T03:39:15.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657078+0000", 2024-09-16T03:39:15.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4770000000000002e-06 2024-09-16T03:39:15.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.557 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.557 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3584 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:15.557 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:39:15.557 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.417353300999999, 2024-09-16T03:39:15.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61903161699999998, 2024-09-16T03:39:15.558 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.558 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.558 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:15.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.558 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.558 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3584 2024-09-16T03:39:15.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.558 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038065+0000", 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3499999999999996e-07 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038069+0000", 2024-09-16T03:39:15.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:39:15.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038075+0000", 2024-09-16T03:39:15.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0549999999999996e-06 2024-09-16T03:39:15.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038078+0000", 2024-09-16T03:39:15.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4939999999999999e-06 2024-09-16T03:39:15.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.126379+0000", 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088300141999999998 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.126393+0000", 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4827999999999999e-05 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657092+0000", 2024-09-16T03:39:15.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.53069909500000001 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657096+0000", 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4139999999999998e-06 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3585 4.2c 4.1cf4922c (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.417332740999999, 2024-09-16T03:39:15.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61902921799999999, 2024-09-16T03:39:15.563 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.563 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:15.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.563 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3585 2024-09-16T03:39:15.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.563 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:15.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:15.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:15.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1899999999999998e-07 2024-09-16T03:39:15.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038087+0000", 2024-09-16T03:39:15.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-16T03:39:15.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038091+0000", 2024-09-16T03:39:15.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1590000000000003e-06 2024-09-16T03:39:15.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038095+0000", 2024-09-16T03:39:15.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5779999999999999e-06 2024-09-16T03:39:15.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.138249+0000", 2024-09-16T03:39:15.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100154101 2024-09-16T03:39:15.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:15.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.138256+0000", 2024-09-16T03:39:15.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4209999999999998e-06 2024-09-16T03:39:15.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657111+0000", 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.51885431800000004 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657114+0000", 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5190000000000001e-06 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.567 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3983 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 2151362] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:15.568 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:39:15.568 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.163423977000001, 2024-09-16T03:39:15.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49254613699999999, 2024-09-16T03:39:15.568 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.568 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.568 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.568 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3983 2024-09-16T03:39:15.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.568 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9500000000000002e-07 2024-09-16T03:39:15.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291997+0000", 2024-09-16T03:39:15.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9679999999999998e-06 2024-09-16T03:39:15.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292002+0000", 2024-09-16T03:39:15.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7700000000000001e-06 2024-09-16T03:39:15.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292005+0000", 2024-09-16T03:39:15.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3019999999999999e-06 2024-09-16T03:39:15.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.382133+0000", 2024-09-16T03:39:15.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.090127600000000002 2024-09-16T03:39:15.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.382184+0000", 2024-09-16T03:39:15.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1304999999999998e-05 2024-09-16T03:39:15.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 11,15", 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.382301+0000", 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116673 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.383337+0000", 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00103619 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.583745+0000", 2024-09-16T03:39:15.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20040787600000001 2024-09-16T03:39:15.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:15.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784456+0000", 2024-09-16T03:39:15.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20071103700000001 2024-09-16T03:39:15.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:15.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784496+0000", 2024-09-16T03:39:15.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9740999999999998e-05 2024-09-16T03:39:15.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784540+0000", 2024-09-16T03:39:15.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4079999999999998e-05 2024-09-16T03:39:15.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.574 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.574 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3984 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:15.574 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:39:15.574 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.163407796, 2024-09-16T03:39:15.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49270909200000002, 2024-09-16T03:39:15.574 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:15.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3984 2024-09-16T03:39:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:39:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4900000000000001e-07 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292012+0000", 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9250000000000002e-06 2024-09-16T03:39:15.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292015+0000", 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4199999999999999e-06 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292018+0000", 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.994e-06 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.388062+0000", 2024-09-16T03:39:15.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096043327999999997 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.388110+0000", 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8272999999999997e-05 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784615+0000", 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39650470300000001 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:15.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784664+0000", 2024-09-16T03:39:15.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8841000000000002e-05 2024-09-16T03:39:15.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:15.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:15.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:15.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784719+0000", 2024-09-16T03:39:15.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5259000000000001e-05 2024-09-16T03:39:15.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.579 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:15.579 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:15.579 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:15.581 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2029227 2024-09-16T03:39:15.581 INFO:tasks.rados.rados.0.smithi157.stdout:1756: expect (ObjNum 365 snap 0 seq_num 365) 2024-09-16T03:39:15.581 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:15.579+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:15.582 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:39:15.584 INFO:tasks.rados.rados.0.smithi157.stdout:1757: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-16T03:39:15.584 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:15.592 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:39:15.614 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:15.656 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2254132 2024-09-16T03:39:15.661 INFO:tasks.rados.rados.0.smithi157.stdout:1760: expect (ObjNum 296 snap 0 seq_num 296) 2024-09-16T03:39:15.681 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:15.680+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:15.696 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:15.696 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:15.696 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:15.696 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:15.705 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:39:15.729 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2332138 2024-09-16T03:39:15.744 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 576005 2024-09-16T03:39:15.782 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:15.780+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:15.782 INFO:tasks.rados.rados.0.smithi157.stdout:1755: finishing write tid 1 to smithi15736696-16 2024-09-16T03:39:15.783 INFO:tasks.rados.rados.0.smithi157.stdout:1755: finishing write tid 2 to smithi15736696-16 2024-09-16T03:39:15.783 INFO:tasks.rados.rados.0.smithi157.stdout:1759: expect (ObjNum 367 snap 0 seq_num 367) 2024-09-16T03:39:15.783 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:15.809 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:15.809 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:15.809 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:15.809 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:15.809 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:15.817 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:39:15.819 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 778008 2024-09-16T03:39:15.843 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:15.876 INFO:tasks.rados.rados.0.smithi157.stdout:1758: expect (ObjNum 379 snap 0 seq_num 379) 2024-09-16T03:39:15.876 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:15.882 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:15.880+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:15.962 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:15.962 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:15.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:15.962 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:15.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.962 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:21 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'4, mlcod=0'0)", 2024-09-16T03:39:15.962 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:39:15.962 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 123.77243470099999, 2024-09-16T03:39:15.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10231100799999999, 2024-09-16T03:39:15.963 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:15.963 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:15.963 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.963 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:15.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:15.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:39:15.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:15.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:39:15.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:15.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187300+0000", 2024-09-16T03:39:15.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0360000000000001e-06 2024-09-16T03:39:15.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:15.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187905+0000", 2024-09-16T03:39:15.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060508100000000004 2024-09-16T03:39:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187920+0000", 2024-09-16T03:39:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4314000000000001e-05 2024-09-16T03:39:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187928+0000", 2024-09-16T03:39:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7150000000000004e-06 2024-09-16T03:39:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:15.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187984+0000", 2024-09-16T03:39:15.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5872999999999999e-05 2024-09-16T03:39:15.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:15.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188016+0000", 2024-09-16T03:39:15.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2045000000000002e-05 2024-09-16T03:39:15.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:15.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289567+0000", 2024-09-16T03:39:15.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101550875 2024-09-16T03:39:15.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:15.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289609+0000", 2024-09-16T03:39:15.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2069e-05 2024-09-16T03:39:15.967 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.967 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:15.967 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.967 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.21:5 3.0 e45/44 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'39, mlcod=45'37)", 2024-09-16T03:39:15.967 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:39:15.967 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 115.04513766700001, 2024-09-16T03:39:15.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20119541799999999, 2024-09-16T03:39:15.968 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:15.968 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:15.968 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.968 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:15.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:15.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:39:15.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:15.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:39:15.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:15.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914597+0000", 2024-09-16T03:39:15.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.494e-06 2024-09-16T03:39:15.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:15.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914603+0000", 2024-09-16T03:39:15.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7349999999999999e-06 2024-09-16T03:39:15.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:15.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914614+0000", 2024-09-16T03:39:15.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0995999999999999e-05 2024-09-16T03:39:15.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914623+0000", 2024-09-16T03:39:15.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3899999999999993e-06 2024-09-16T03:39:15.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:15.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.114941+0000", 2024-09-16T03:39:15.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20031812399999999 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115001+0000", 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0161999999999998e-05 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115750+0000", 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074851699999999998 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:15.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115791+0000", 2024-09-16T03:39:15.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1e-05 2024-09-16T03:39:15.972 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.972 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:15.972 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.972 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.5.21:36 3.0 e52/44 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 52'62, mlcod=51'60)", 2024-09-16T03:39:15.972 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:39:15.972 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.075107008, 2024-09-16T03:39:15.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20104840199999999, 2024-09-16T03:39:15.972 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:15.972 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:15.972 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.973 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:15.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:15.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:39:15.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:15.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:39:15.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:15.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884627+0000", 2024-09-16T03:39:15.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.322e-06 2024-09-16T03:39:15.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:15.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884634+0000", 2024-09-16T03:39:15.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:39:15.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:15.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884643+0000", 2024-09-16T03:39:15.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7680000000000003e-06 2024-09-16T03:39:15.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984740+0000", 2024-09-16T03:39:15.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100096066 2024-09-16T03:39:15.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:15.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084931+0000", 2024-09-16T03:39:15.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100191408 2024-09-16T03:39:15.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084952+0000", 2024-09-16T03:39:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.128e-05 2024-09-16T03:39:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085645+0000", 2024-09-16T03:39:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00069309500000000002 2024-09-16T03:39:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085674+0000", 2024-09-16T03:39:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9025000000000001e-05 2024-09-16T03:39:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1537 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'35, mlcod=58'33)", 2024-09-16T03:39:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:39:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 90.239353714000003, 2024-09-16T03:39:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102969106, 2024-09-16T03:39:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:39:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:39:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720381+0000", 2024-09-16T03:39:15.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.767e-06 2024-09-16T03:39:15.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:15.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721134+0000", 2024-09-16T03:39:15.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00075281999999999996 2024-09-16T03:39:15.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:15.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721148+0000", 2024-09-16T03:39:15.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4534e-05 2024-09-16T03:39:15.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721160+0000", 2024-09-16T03:39:15.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1595e-05 2024-09-16T03:39:15.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:15.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721256+0000", 2024-09-16T03:39:15.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6136000000000002e-05 2024-09-16T03:39:15.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:15.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721274+0000", 2024-09-16T03:39:15.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8006e-05 2024-09-16T03:39:15.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:15.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823312+0000", 2024-09-16T03:39:15.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102038064 2024-09-16T03:39:15.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:15.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823348+0000", 2024-09-16T03:39:15.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6183999999999998e-05 2024-09-16T03:39:15.981 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.981 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:15.982 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.982 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1538 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'36, mlcod=58'33)", 2024-09-16T03:39:15.982 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:39:15.982 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 90.238199377000001, 2024-09-16T03:39:15.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102397932, 2024-09-16T03:39:15.982 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:15.982 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:15.982 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.982 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:15.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:15.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:39:15.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:15.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:39:15.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:15.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:39:15.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5199999999999996e-07 2024-09-16T03:39:15.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:15.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722162+0000", 2024-09-16T03:39:15.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062758599999999996 2024-09-16T03:39:15.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:15.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722171+0000", 2024-09-16T03:39:15.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7139999999999999e-06 2024-09-16T03:39:15.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722177+0000", 2024-09-16T03:39:15.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.866e-06 2024-09-16T03:39:15.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:15.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822283+0000", 2024-09-16T03:39:15.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10010508899999999 2024-09-16T03:39:15.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:15.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822298+0000", 2024-09-16T03:39:15.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5497000000000001e-05 2024-09-16T03:39:15.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:15.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823907+0000", 2024-09-16T03:39:15.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001609166 2024-09-16T03:39:15.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:15.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823931+0000", 2024-09-16T03:39:15.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4261999999999999e-05 2024-09-16T03:39:15.986 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.986 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1539 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'37, mlcod=58'33)", 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 90.236999315999995, 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103545683, 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:15.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:39:15.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:15.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:39:15.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:15.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722735+0000", 2024-09-16T03:39:15.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8400000000000002e-07 2024-09-16T03:39:15.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722740+0000", 2024-09-16T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3129999999999998e-06 2024-09-16T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722747+0000", 2024-09-16T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0600000000000002e-06 2024-09-16T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822841+0000", 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100094477 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823376+0000", 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00053497199999999999 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823399+0000", 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2680999999999998e-05 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.826259+0000", 2024-09-16T03:39:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028598500000000002 2024-09-16T03:39:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.826279+0000", 2024-09-16T03:39:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0346000000000001e-05 2024-09-16T03:39:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:39:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:39:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 85.809808411000006, 2024-09-16T03:39:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20169899299999999, 2024-09-16T03:39:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:39:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:39:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149927+0000", 2024-09-16T03:39:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9299999999999999e-06 2024-09-16T03:39:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151090+0000", 2024-09-16T03:39:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011624459999999999 2024-09-16T03:39:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151105+0000", 2024-09-16T03:39:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5238999999999999e-05 2024-09-16T03:39:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251193+0000", 2024-09-16T03:39:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100088025 2024-09-16T03:39:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251556+0000", 2024-09-16T03:39:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036245599999999999 2024-09-16T03:39:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251578+0000", 2024-09-16T03:39:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2651e-05 2024-09-16T03:39:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351582+0000", 2024-09-16T03:39:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100003811 2024-09-16T03:39:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351624+0000", 2024-09-16T03:39:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1434999999999999e-05 2024-09-16T03:39:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:90 3.0 e70/69 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head v 70'103, mlcod=0'0)", 2024-09-16T03:39:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:39:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.183970127999999, 2024-09-16T03:39:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102185333, 2024-09-16T03:39:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:15.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:15.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:39:15.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:15.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:39:15.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:15.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:15.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:39:15.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-16T03:39:15.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:15.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775769+0000", 2024-09-16T03:39:15.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7379999999999999e-06 2024-09-16T03:39:15.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:15.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:15.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:15.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775782+0000", 2024-09-16T03:39:15.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2989e-05 2024-09-16T03:39:16.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775791+0000", 2024-09-16T03:39:16.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5369999999999997e-06 2024-09-16T03:39:16.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876295+0000", 2024-09-16T03:39:16.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100504817 2024-09-16T03:39:16.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876316+0000", 2024-09-16T03:39:16.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0409999999999999e-05 2024-09-16T03:39:16.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877916+0000", 2024-09-16T03:39:16.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016003790000000001 2024-09-16T03:39:16.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877948+0000", 2024-09-16T03:39:16.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1949999999999997e-05 2024-09-16T03:39:16.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.002 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.002 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.002 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1742 3.0 e70/69 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head v 70'104, mlcod=0'0)", 2024-09-16T03:39:16.002 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:16.002 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.183933432000003, 2024-09-16T03:39:16.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20271830699999999, 2024-09-16T03:39:16.002 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.002 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.002 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.002 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:39:16.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777455+0000", 2024-09-16T03:39:16.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001655524 2024-09-16T03:39:16.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777468+0000", 2024-09-16T03:39:16.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2486999999999999e-05 2024-09-16T03:39:16.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777474+0000", 2024-09-16T03:39:16.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:39:16.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876577+0000", 2024-09-16T03:39:16.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099103360000000001 2024-09-16T03:39:16.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876594+0000", 2024-09-16T03:39:16.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6449000000000001e-05 2024-09-16T03:39:16.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978469+0000", 2024-09-16T03:39:16.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101875521 2024-09-16T03:39:16.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978518+0000", 2024-09-16T03:39:16.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8488000000000001e-05 2024-09-16T03:39:16.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.006 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.007 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:91 3.0 e70/69 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head v 70'105, mlcod=0'0)", 2024-09-16T03:39:16.007 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:39:16.007 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.097721562999993, 2024-09-16T03:39:16.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.116532025, 2024-09-16T03:39:16.007 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.007 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.007 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.007 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:39:16.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:39:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862015+0000", 2024-09-16T03:39:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3569999999999998e-06 2024-09-16T03:39:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.008 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862029+0000", 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4082e-05 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862055+0000", 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6570000000000001e-05 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862070+0000", 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4473e-05 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877631+0000", 2024-09-16T03:39:16.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.015560744 2024-09-16T03:39:16.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877674+0000", 2024-09-16T03:39:16.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3615999999999997e-05 2024-09-16T03:39:16.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978525+0000", 2024-09-16T03:39:16.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100850696 2024-09-16T03:39:16.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978543+0000", 2024-09-16T03:39:16.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8487000000000001e-05 2024-09-16T03:39:16.011 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.011 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.011 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.011 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:94 3.0 e70/69 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head v 70'106, mlcod=0'0)", 2024-09-16T03:39:16.011 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:39:16.011 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 80.097185343999996, 2024-09-16T03:39:16.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11627603, 2024-09-16T03:39:16.012 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.012 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.012 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.012 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:39:16.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:39:16.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862549+0000", 2024-09-16T03:39:16.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1650000000000001e-06 2024-09-16T03:39:16.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862556+0000", 2024-09-16T03:39:16.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8709999999999999e-06 2024-09-16T03:39:16.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862566+0000", 2024-09-16T03:39:16.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0271e-05 2024-09-16T03:39:16.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862573+0000", 2024-09-16T03:39:16.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.861e-06 2024-09-16T03:39:16.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978123+0000", 2024-09-16T03:39:16.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11554993199999999 2024-09-16T03:39:16.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978163+0000", 2024-09-16T03:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0201999999999999e-05 2024-09-16T03:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978796+0000", 2024-09-16T03:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063316700000000002 2024-09-16T03:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978824+0000", 2024-09-16T03:39:16.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7560999999999999e-05 2024-09-16T03:39:16.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.016 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.016 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1947 4.8 e74/68 4:146f84f0:::smithi15736696-6:head v 74'47, mlcod=73'45)", 2024-09-16T03:39:16.016 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:39:16.016 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 74.834569205999998, 2024-09-16T03:39:16.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103866244, 2024-09-16T03:39:16.016 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.016 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.017 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.017 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:39:16.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:39:16.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125165+0000", 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.513e-06 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126855+0000", 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016901010000000001 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126870+0000", 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4487999999999999e-05 2024-09-16T03:39:16.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.019 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227009+0000", 2024-09-16T03:39:16.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100138888 2024-09-16T03:39:16.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.019 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227117+0000", 2024-09-16T03:39:16.019 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107984 2024-09-16T03:39:16.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.019 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227136+0000", 2024-09-16T03:39:16.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9247999999999999e-05 2024-09-16T03:39:16.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229002+0000", 2024-09-16T03:39:16.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001866143 2024-09-16T03:39:16.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.020 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.020 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229030+0000", 2024-09-16T03:39:16.020 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7878999999999999e-05 2024-09-16T03:39:16.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1979 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'32, mlcod=0'0)", 2024-09-16T03:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 74.121994658999995, 2024-09-16T03:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20429768500000001, 2024-09-16T03:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837740+0000", 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.779e-06 2024-09-16T03:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.023 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.023 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838356+0000", 2024-09-16T03:39:16.023 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061634500000000002 2024-09-16T03:39:16.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.023 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.023 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838371+0000", 2024-09-16T03:39:16.023 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5007e-05 2024-09-16T03:39:16.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.023 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.024 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838380+0000", 2024-09-16T03:39:16.024 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1009999999999992e-06 2024-09-16T03:39:16.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.024 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.024 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038674+0000", 2024-09-16T03:39:16.024 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20029450500000001 2024-09-16T03:39:16.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.024 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.024 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038710+0000", 2024-09-16T03:39:16.024 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6100999999999999e-05 2024-09-16T03:39:16.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042001+0000", 2024-09-16T03:39:16.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032909430000000002 2024-09-16T03:39:16.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.025 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.025 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042036+0000", 2024-09-16T03:39:16.025 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4904000000000003e-05 2024-09-16T03:39:16.025 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.025 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.025 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.026 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1980 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'33, mlcod=0'0)", 2024-09-16T03:39:16.026 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:39:16.026 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 74.120118063000007, 2024-09-16T03:39:16.026 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20296795100000001, 2024-09-16T03:39:16.026 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.026 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.026 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.026 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.026 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.026 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839616+0000", 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3099999999999996e-07 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.027 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.028 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840706+0000", 2024-09-16T03:39:16.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010900980000000001 2024-09-16T03:39:16.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.028 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.028 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840720+0000", 2024-09-16T03:39:16.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.378e-05 2024-09-16T03:39:16.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.028 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.028 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840727+0000", 2024-09-16T03:39:16.028 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.525e-06 2024-09-16T03:39:16.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.029 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.029 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039940+0000", 2024-09-16T03:39:16.029 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19921313199999999 2024-09-16T03:39:16.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.029 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.029 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039967+0000", 2024-09-16T03:39:16.029 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6393999999999998e-05 2024-09-16T03:39:16.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.029 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.029 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042552+0000", 2024-09-16T03:39:16.030 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025850999999999999 2024-09-16T03:39:16.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.030 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.030 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042583+0000", 2024-09-16T03:39:16.030 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0991000000000002e-05 2024-09-16T03:39:16.030 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.030 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.030 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.030 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1981 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'34, mlcod=0'0)", 2024-09-16T03:39:16.030 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:39:16.031 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 74.118095707999998, 2024-09-16T03:39:16.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201270175, 2024-09-16T03:39:16.031 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.031 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.031 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.031 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.031 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.031 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:39:16.031 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841638+0000", 2024-09-16T03:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-16T03:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841644+0000", 2024-09-16T03:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8050000000000003e-06 2024-09-16T03:39:16.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.033 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.033 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841653+0000", 2024-09-16T03:39:16.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6580000000000008e-06 2024-09-16T03:39:16.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.033 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.033 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841660+0000", 2024-09-16T03:39:16.033 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9290000000000001e-06 2024-09-16T03:39:16.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.033 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041673+0000", 2024-09-16T03:39:16.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20001311499999999 2024-09-16T03:39:16.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.034 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041696+0000", 2024-09-16T03:39:16.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3365000000000001e-05 2024-09-16T03:39:16.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.034 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.034 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042878+0000", 2024-09-16T03:39:16.034 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011814499999999999 2024-09-16T03:39:16.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.035 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.035 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042907+0000", 2024-09-16T03:39:16.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9655999999999999e-05 2024-09-16T03:39:16.035 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.035 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.035 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.035 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2117 3.0 e76/69 3:3c8eb54b:::593314a3a723c40dec2b25351adb282089b02c0264cb0891ee3f17edde05dfe9:head v 76'138, mlcod=75'136)", 2024-09-16T03:39:16.035 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:39:16.035 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 70.307688020000001, 2024-09-16T03:39:16.035 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10374235, 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8099999999999999e-07 2024-09-16T03:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654214+0000", 2024-09-16T03:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002169191 2024-09-16T03:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.037 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654230+0000", 2024-09-16T03:39:16.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6144e-05 2024-09-16T03:39:16.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654242+0000", 2024-09-16T03:39:16.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1748e-05 2024-09-16T03:39:16.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.038 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.038 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.754387+0000", 2024-09-16T03:39:16.038 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10014505899999999 2024-09-16T03:39:16.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.754409+0000", 2024-09-16T03:39:16.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2181999999999999e-05 2024-09-16T03:39:16.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.755736+0000", 2024-09-16T03:39:16.039 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013261690000000001 2024-09-16T03:39:16.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.039 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.039 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.755787+0000", 2024-09-16T03:39:16.040 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1675999999999998e-05 2024-09-16T03:39:16.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.040 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.040 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2278 4.18 e80/68 4:1f63731d:::smithi15736696-30:head v 80'37, mlcod=75'34)", 2024-09-16T03:39:16.040 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:39:16.040 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 65.434277820999995, 2024-09-16T03:39:16.040 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10326089300000001, 2024-09-16T03:39:16.040 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.040 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:39:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:39:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525456+0000", 2024-09-16T03:39:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.248e-06 2024-09-16T03:39:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525936+0000", 2024-09-16T03:39:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047982899999999998 2024-09-16T03:39:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.042 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525945+0000", 2024-09-16T03:39:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8489999999999995e-06 2024-09-16T03:39:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525955+0000", 2024-09-16T03:39:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.043e-05 2024-09-16T03:39:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.526339+0000", 2024-09-16T03:39:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038385200000000001 2024-09-16T03:39:16.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.526358+0000", 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8802000000000001e-05 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.628679+0000", 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102321304 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.628716+0000", 2024-09-16T03:39:16.044 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6579000000000003e-05 2024-09-16T03:39:16.045 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.045 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.045 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.045 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.045 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2536 3.0 e86/69 3:10a17b1b:::13dc2fab1f287170fabf96291b912bf80e0b081f291f2c6d41d494566f2d1f8c:head v 86'176, mlcod=86'174)", 2024-09-16T03:39:16.045 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:39:16.045 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 59.12022133, 2024-09-16T03:39:16.045 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20664157499999999, 2024-09-16T03:39:16.045 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.045 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.045 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.046 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:39:16.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.046 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:39:16.046 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.046 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839515+0000", 2024-09-16T03:39:16.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3330000000000001e-06 2024-09-16T03:39:16.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.047 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843392+0000", 2024-09-16T03:39:16.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038769329999999999 2024-09-16T03:39:16.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.047 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.047 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843412+0000", 2024-09-16T03:39:16.047 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0608e-05 2024-09-16T03:39:16.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.943546+0000", 2024-09-16T03:39:16.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100133176 2024-09-16T03:39:16.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043824+0000", 2024-09-16T03:39:16.048 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100278785 2024-09-16T03:39:16.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.048 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.048 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043869+0000", 2024-09-16T03:39:16.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4845999999999997e-05 2024-09-16T03:39:16.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046096+0000", 2024-09-16T03:39:16.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022262559999999998 2024-09-16T03:39:16.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.049 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.049 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046153+0000", 2024-09-16T03:39:16.049 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7637999999999999e-05 2024-09-16T03:39:16.049 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.049 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.050 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.050 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2654 4.8 e90/68 4:148e627d:::smithi15736696-23:head v 90'58, mlcod=80'56)", 2024-09-16T03:39:16.050 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:39:16.050 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 54.584833971999998, 2024-09-16T03:39:16.050 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.106604212, 2024-09-16T03:39:16.050 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.050 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.050 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.050 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:39:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:39:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374901+0000", 2024-09-16T03:39:16.051 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1749999999999999e-06 2024-09-16T03:39:16.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375810+0000", 2024-09-16T03:39:16.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00090911400000000004 2024-09-16T03:39:16.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.052 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.052 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.052 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375827+0000", 2024-09-16T03:39:16.052 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7010000000000001e-05 2024-09-16T03:39:16.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476010+0000", 2024-09-16T03:39:16.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100182879 2024-09-16T03:39:16.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476184+0000", 2024-09-16T03:39:16.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017408899999999999 2024-09-16T03:39:16.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.053 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.053 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476206+0000", 2024-09-16T03:39:16.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2013000000000001e-05 2024-09-16T03:39:16.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481458+0000", 2024-09-16T03:39:16.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052519120000000001 2024-09-16T03:39:16.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481503+0000", 2024-09-16T03:39:16.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5019999999999999e-05 2024-09-16T03:39:16.054 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.054 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.055 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.055 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "pg_update_log_missing(4.c epoch 99/68 rep_tid 69 entries 99'56 (0'0) error 4:34492f38:::smithi15736696-33:head by client.4628.0:3000 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 90'53 roll_forward_to 90'54)", 2024-09-16T03:39:16.055 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:39:16.055 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 44.039875821000003, 2024-09-16T03:39:16.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201155684, 2024-09-16T03:39:16.055 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.055 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.055 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:16.055 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:39:16.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:39:16.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919858+0000", 2024-09-16T03:39:16.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:39:16.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919871+0000", 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3433e-05 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919929+0000", 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7543000000000001e-05 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919942+0000", 2024-09-16T03:39:16.057 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3122000000000001e-05 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.120302+0000", 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036052800000001 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.121013+0000", 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071046800000000002 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.058 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.059 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.059 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:16.060 INFO:tasks.rados.rados.0.smithi157.stdout:1761: expect (ObjNum 325 snap 0 seq_num 325) 2024-09-16T03:39:16.060 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:16.058+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:16.061 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:39:16.092 INFO:tasks.rados.rados.0.smithi157.stdout:1755: finishing write tid 3 to smithi15736696-16 2024-09-16T03:39:16.093 INFO:tasks.rados.rados.0.smithi157.stdout:1763: finishing write tid 1 to smithi15736696-29 2024-09-16T03:39:16.093 INFO:tasks.rados.rados.0.smithi157.stdout:1755: finishing write tid 4 to smithi15736696-16 2024-09-16T03:39:16.093 INFO:tasks.rados.rados.0.smithi157.stdout:1755: oid 16 updating version 0 to 68 2024-09-16T03:39:16.093 INFO:tasks.rados.rados.0.smithi157.stdout:1755: oid 16 updating version 68 to 69 2024-09-16T03:39:16.093 INFO:tasks.rados.rados.0.smithi157.stdout:1755: oid 16 updating version 69 to 70 2024-09-16T03:39:16.093 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 70 (ObjNum 388 snap 0 seq_num 388) dirty exists 2024-09-16T03:39:16.093 INFO:tasks.rados.rados.0.smithi157.stdout:1755: left oid 16 (ObjNum 388 snap 0 seq_num 388) 2024-09-16T03:39:16.093 INFO:tasks.rados.rados.0.smithi157.stdout:1763: finishing write tid 2 to smithi15736696-29 2024-09-16T03:39:16.093 INFO:tasks.rados.rados.0.smithi157.stdout:1763: finishing write tid 3 to smithi15736696-29 2024-09-16T03:39:16.093 INFO:tasks.rados.rados.0.smithi157.stdout:1763: finishing write tid 4 to smithi15736696-29 2024-09-16T03:39:16.093 INFO:tasks.rados.rados.0.smithi157.stdout:1763: oid 29 updating version 0 to 134 2024-09-16T03:39:16.093 INFO:tasks.rados.rados.0.smithi157.stdout:1763: oid 29 updating version 134 to 135 2024-09-16T03:39:16.094 INFO:tasks.rados.rados.0.smithi157.stdout:1763: oid 29 updating version 135 to 136 2024-09-16T03:39:16.094 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 136 (ObjNum 389 snap 0 seq_num 389) dirty exists 2024-09-16T03:39:16.094 INFO:tasks.rados.rados.0.smithi157.stdout:1763: left oid 29 (ObjNum 389 snap 0 seq_num 389) 2024-09-16T03:39:16.094 INFO:tasks.rados.rados.0.smithi157.stdout:1764: seq_num 390 ranges {554730=481113,1572095=737454} 2024-09-16T03:39:16.098 INFO:tasks.rados.rados.0.smithi157.stdout:1764: writing smithi15736696-44 from 554730 to 1035843 tid 1 2024-09-16T03:39:16.105 INFO:tasks.rados.rados.0.smithi157.stdout:1764: writing smithi15736696-44 from 1572095 to 2309549 tid 2 2024-09-16T03:39:16.106 INFO:tasks.rados.rados.0.smithi157.stdout:1739: done (11 left) 2024-09-16T03:39:16.106 INFO:tasks.rados.rados.0.smithi157.stdout:1752: done (10 left) 2024-09-16T03:39:16.106 INFO:tasks.rados.rados.0.smithi157.stdout:1755: done (9 left) 2024-09-16T03:39:16.106 INFO:tasks.rados.rados.0.smithi157.stdout:1756: done (8 left) 2024-09-16T03:39:16.106 INFO:tasks.rados.rados.0.smithi157.stdout:1757: done (7 left) 2024-09-16T03:39:16.106 INFO:tasks.rados.rados.0.smithi157.stdout:1758: done (6 left) 2024-09-16T03:39:16.106 INFO:tasks.rados.rados.0.smithi157.stdout:1759: done (5 left) 2024-09-16T03:39:16.106 INFO:tasks.rados.rados.0.smithi157.stdout:1760: done (4 left) 2024-09-16T03:39:16.106 INFO:tasks.rados.rados.0.smithi157.stdout:1761: done (3 left) 2024-09-16T03:39:16.106 INFO:tasks.rados.rados.0.smithi157.stdout:1762: done (2 left) 2024-09-16T03:39:16.106 INFO:tasks.rados.rados.0.smithi157.stdout:1763: done (1 left) 2024-09-16T03:39:16.106 INFO:tasks.rados.rados.0.smithi157.stdout:1765: set_chunk oid 47 target oid 2024-09-16T03:39:16.161 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:16.159+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:16.164 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:16.164 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:16.164 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:16.164 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:16.173 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:39:16.261 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:16.260+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:16.276 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:16.276 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:16.276 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:16.276 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:16.276 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:16.285 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:39:16.362 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:16.361+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:16.388 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:25 4.6 4:6ece7830:::smithi15736696-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:39:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 124.26625900400001, 2024-09-16T03:39:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40559452699999998, 2024-09-16T03:39:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.389 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 25 2024-09-16T03:39:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:39:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119830+0000", 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9200000000000001e-07 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120013+0000", 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018367200000000001 2024-09-16T03:39:16.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120021+0000", 2024-09-16T03:39:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9179999999999994e-06 2024-09-16T03:39:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120027+0000", 2024-09-16T03:39:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6710000000000004e-06 2024-09-16T03:39:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220194+0000", 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10016689500000001 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220399+0000", 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020502899999999999 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220613+0000", 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021383900000000001 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224640+0000", 2024-09-16T03:39:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040268099999999996 2024-09-16T03:39:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423348+0000", 2024-09-16T03:39:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19870853399999999 2024-09-16T03:39:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525343+0000", 2024-09-16T03:39:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101995081 2024-09-16T03:39:16.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525381+0000", 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7842000000000001e-05 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525424+0000", 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2644e-05 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:26 4.6 4:6ece7830:::smithi15736696-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:39:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 124.266046777, 2024-09-16T03:39:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550111300000002, 2024-09-16T03:39:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 26 2024-09-16T03:39:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.396 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:39:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:39:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120042+0000", 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7999999999999995e-07 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120210+0000", 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016779499999999999 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120219+0000", 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6859999999999995e-06 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120223+0000", 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0330000000000002e-06 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221494+0000", 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101271812 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221572+0000", 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7733999999999998e-05 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221697+0000", 2024-09-16T03:39:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012474900000000001 2024-09-16T03:39:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224670+0000", 2024-09-16T03:39:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029729589999999998 2024-09-16T03:39:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423398+0000", 2024-09-16T03:39:16.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19872784900000001 2024-09-16T03:39:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525465+0000", 2024-09-16T03:39:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102067557 2024-09-16T03:39:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525494+0000", 2024-09-16T03:39:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9113999999999999e-05 2024-09-16T03:39:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525542+0000", 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8145000000000001e-05 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:27 4.6 4:6ece7830:::smithi15736696-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 124.26585919199999, 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40541507399999999, 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.402 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.403 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.403 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.403 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.403 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27 2024-09-16T03:39:16.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.403 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:39:16.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120230+0000", 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5799999999999999e-07 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120233+0000", 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.055e-06 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120237+0000", 2024-09-16T03:39:16.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7099999999999998e-06 2024-09-16T03:39:16.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120244+0000", 2024-09-16T03:39:16.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4810000000000003e-06 2024-09-16T03:39:16.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222660+0000", 2024-09-16T03:39:16.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102416089 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222705+0000", 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5368999999999999e-05 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222782+0000", 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6623000000000001e-05 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224696+0000", 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019143929999999999 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423402+0000", 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198705305 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525582+0000", 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102180178 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525615+0000", 2024-09-16T03:39:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3303000000000003e-05 2024-09-16T03:39:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525644+0000", 2024-09-16T03:39:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8909999999999999e-05 2024-09-16T03:39:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.408 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:28 4.6 4:6ece7830:::smithi15736696-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 124.265838444, 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550787799999999, 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 28 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7999999999999996e-07 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120252+0000", 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.249e-06 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120257+0000", 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1969999999999998e-06 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120261+0000", 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1130000000000003e-06 2024-09-16T03:39:16.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222976+0000", 2024-09-16T03:39:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102715397 2024-09-16T03:39:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223049+0000", 2024-09-16T03:39:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2846e-05 2024-09-16T03:39:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525666+0000", 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30261668800000002 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525704+0000", 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8025999999999998e-05 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525758+0000", 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3881999999999997e-05 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.413 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:49 4.6 4:6df3bbbe:::smithi15736696-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 124.26582338, 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40556222800000002, 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49 2024-09-16T03:39:16.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8700000000000002e-07 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120454+0000", 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018909299999999999 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120465+0000", 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.083e-05 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120468+0000", 2024-09-16T03:39:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3869999999999999e-06 2024-09-16T03:39:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223082+0000", 2024-09-16T03:39:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102613379 2024-09-16T03:39:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223189+0000", 2024-09-16T03:39:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107728 2024-09-16T03:39:16.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223368+0000", 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179011 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.226087+0000", 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027183390000000002 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423408+0000", 2024-09-16T03:39:16.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19732106499999999 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525779+0000", 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10237133 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525805+0000", 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5967000000000001e-05 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525827+0000", 2024-09-16T03:39:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1812000000000001e-05 2024-09-16T03:39:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:50 4.6 4:6df3bbbe:::smithi15736696-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:39:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 124.265615071, 2024-09-16T03:39:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50526462999999999, 2024-09-16T03:39:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.420 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 50 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:39:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120474+0000", 2024-09-16T03:39:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4600000000000005e-07 2024-09-16T03:39:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120901+0000", 2024-09-16T03:39:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00042690000000000002 2024-09-16T03:39:16.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120907+0000", 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7710000000000001e-06 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120911+0000", 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0500000000000002e-06 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224717+0000", 2024-09-16T03:39:16.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103806175 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224763+0000", 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5629e-05 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224838+0000", 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4548999999999994e-05 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423390+0000", 2024-09-16T03:39:16.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19855242400000001 2024-09-16T03:39:16.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423411+0000", 2024-09-16T03:39:16.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0888000000000001e-05 2024-09-16T03:39:16.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625680+0000", 2024-09-16T03:39:16.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20226923099999999 2024-09-16T03:39:16.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625709+0000", 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9232000000000001e-05 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625738+0000", 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8235e-05 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.426 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:51 4.6 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 124.265170489, 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50490003699999997, 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 51 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8699999999999995e-07 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120920+0000", 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0820000000000001e-06 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120924+0000", 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120927+0000", 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5809999999999999e-06 2024-09-16T03:39:16.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225783+0000", 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10485588699999999 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225832+0000", 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9750000000000003e-05 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225910+0000", 2024-09-16T03:39:16.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7084999999999998e-05 2024-09-16T03:39:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423414+0000", 2024-09-16T03:39:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19750438200000001 2024-09-16T03:39:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423456+0000", 2024-09-16T03:39:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2386999999999999e-05 2024-09-16T03:39:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625766+0000", 2024-09-16T03:39:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20231011300000001 2024-09-16T03:39:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625790+0000", 2024-09-16T03:39:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3672999999999998e-05 2024-09-16T03:39:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625818+0000", 2024-09-16T03:39:16.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7576000000000002e-05 2024-09-16T03:39:16.433 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.433 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.433 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:52 4.6 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 124.265158293, 2024-09-16T03:39:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50502012900000004, 2024-09-16T03:39:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.434 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 52 2024-09-16T03:39:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:39:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120932+0000", 2024-09-16T03:39:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.181e-06 2024-09-16T03:39:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120935+0000", 2024-09-16T03:39:16.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7690000000000001e-06 2024-09-16T03:39:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120938+0000", 2024-09-16T03:39:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6639999999999998e-06 2024-09-16T03:39:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423188+0000", 2024-09-16T03:39:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30224989200000002 2024-09-16T03:39:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423318+0000", 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013045999999999999 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625844+0000", 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20252547600000001 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625895+0000", 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1242000000000002e-05 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625950+0000", 2024-09-16T03:39:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5164999999999998e-05 2024-09-16T03:39:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:279 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:39:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.189206506, 2024-09-16T03:39:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50446570800000001, 2024-09-16T03:39:16.439 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 279 2024-09-16T03:39:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:39:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:39:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196883+0000", 2024-09-16T03:39:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.48e-06 2024-09-16T03:39:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196889+0000", 2024-09-16T03:39:16.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6239999999999999e-06 2024-09-16T03:39:16.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.442 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.442 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196902+0000", 2024-09-16T03:39:16.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2804e-05 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196909+0000", 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9379999999999992e-06 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196963+0000", 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3437999999999999e-05 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op must wait for map", 2024-09-16T03:39:16.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196969+0000", 2024-09-16T03:39:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1709999999999999e-06 2024-09-16T03:39:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302538+0000", 2024-09-16T03:39:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10556889899999999 2024-09-16T03:39:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302659+0000", 2024-09-16T03:39:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121441 2024-09-16T03:39:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500040+0000", 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19738082900000001 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500589+0000", 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00054827000000000001 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701118+0000", 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20052917100000001 2024-09-16T03:39:16.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701205+0000", 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7120999999999995e-05 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701289+0000", 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4457999999999997e-05 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701347+0000", 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8063999999999998e-05 2024-09-16T03:39:16.446 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.447 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.447 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:306 4.6 4:6ece7830:::smithi15736696-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:16.447 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:39:16.447 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 119.873204191, 2024-09-16T03:39:16.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40927214699999998, 2024-09-16T03:39:16.447 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.447 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.447 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.447 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.447 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.448 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.448 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 306 2024-09-16T03:39:16.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.448 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:39:16.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:39:16.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512885+0000", 2024-09-16T03:39:16.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.277e-06 2024-09-16T03:39:16.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.449 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512890+0000", 2024-09-16T03:39:16.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6689999999999997e-06 2024-09-16T03:39:16.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.449 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512900+0000", 2024-09-16T03:39:16.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0393e-05 2024-09-16T03:39:16.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512906+0000", 2024-09-16T03:39:16.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4569999999999996e-06 2024-09-16T03:39:16.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512951+0000", 2024-09-16T03:39:16.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4891000000000001e-05 2024-09-16T03:39:16.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.513000+0000", 2024-09-16T03:39:16.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9187999999999997e-05 2024-09-16T03:39:16.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:16.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.720758+0000", 2024-09-16T03:39:16.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20775775699999999 2024-09-16T03:39:16.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.721960+0000", 2024-09-16T03:39:16.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001202107 2024-09-16T03:39:16.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922010+0000", 2024-09-16T03:39:16.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20004993900000001 2024-09-16T03:39:16.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922055+0000", 2024-09-16T03:39:16.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4981000000000002e-05 2024-09-16T03:39:16.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922103+0000", 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8097999999999999e-05 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922156+0000", 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3390000000000002e-05 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.453 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:419 4.6 4:64ce87bc:::smithi15736696-20:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:16.454 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:39:16.454 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 115.47240457700001, 2024-09-16T03:39:16.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40437787200000003, 2024-09-16T03:39:16.454 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.454 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.454 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.454 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.454 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.454 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.454 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 419 2024-09-16T03:39:16.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.454 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913687+0000", 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3299999999999999e-06 2024-09-16T03:39:16.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913700+0000", 2024-09-16T03:39:16.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2830000000000001e-05 2024-09-16T03:39:16.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913733+0000", 2024-09-16T03:39:16.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2815999999999999e-05 2024-09-16T03:39:16.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.456 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.456 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913750+0000", 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7073000000000001e-05 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913920+0000", 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000170015 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913981+0000", 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1631999999999995e-05 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216580+0000", 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30259888200000001 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217205+0000", 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062461099999999996 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317915+0000", 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10071053000000001 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317996+0000", 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0440999999999998e-05 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318032+0000", 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5944e-05 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318061+0000", 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9768e-05 2024-09-16T03:39:16.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:479 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:39:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 114.664548431, 2024-09-16T03:39:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.504011706, 2024-09-16T03:39:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 479 2024-09-16T03:39:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:39:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:39:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721542+0000", 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3199999999999998e-06 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721553+0000", 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1413000000000001e-05 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721570+0000", 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.685e-05 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721596+0000", 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5647999999999999e-05 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.921950+0000", 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20035402899999999 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.922009+0000", 2024-09-16T03:39:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9046999999999999e-05 2024-09-16T03:39:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:39:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.023850+0000", 2024-09-16T03:39:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10184130199999999 2024-09-16T03:39:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.024503+0000", 2024-09-16T03:39:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065257399999999995 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.125105+0000", 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100602046 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225461+0000", 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100355835 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225512+0000", 2024-09-16T03:39:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0862999999999997e-05 2024-09-16T03:39:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225551+0000", 2024-09-16T03:39:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9779e-05 2024-09-16T03:39:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:996 4.8 4:148e627d:::smithi15736696-23:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 105.338745524, 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.73588397999999999, 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 996 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:39:16.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:39:16.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047344+0000", 2024-09-16T03:39:16.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.846e-06 2024-09-16T03:39:16.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047351+0000", 2024-09-16T03:39:16.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.973e-06 2024-09-16T03:39:16.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047365+0000", 2024-09-16T03:39:16.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3556e-05 2024-09-16T03:39:16.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047382+0000", 2024-09-16T03:39:16.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6994e-05 2024-09-16T03:39:16.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047453+0000", 2024-09-16T03:39:16.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1458000000000006e-05 2024-09-16T03:39:16.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.470 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:16.470 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047465+0000", 2024-09-16T03:39:16.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1119e-05 2024-09-16T03:39:16.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449411+0000", 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40194680399999999 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449535+0000", 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124043 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783227+0000", 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.333691187 2024-09-16T03:39:16.471 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1048 4.16 4:6df3bbbe:::smithi15736696-13:head [write 1622009~726683] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.810301407, 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40682847599999999, 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.472 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.473 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1048 2024-09-16T03:39:16.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.473 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:16.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:16.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:16.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-16T03:39:16.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576941+0000", 2024-09-16T03:39:16.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011532230000000001 2024-09-16T03:39:16.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576949+0000", 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1089999999999998e-06 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576956+0000", 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000001e-06 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878710+0000", 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30175434299999998 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878765+0000", 2024-09-16T03:39:16.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4988000000000003e-05 2024-09-16T03:39:16.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:16.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878852+0000", 2024-09-16T03:39:16.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7201999999999996e-05 2024-09-16T03:39:16.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881535+0000", 2024-09-16T03:39:16.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026828310000000001 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883968+0000", 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024330659999999998 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982541+0000", 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098572876000000004 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982580+0000", 2024-09-16T03:39:16.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8714000000000002e-05 2024-09-16T03:39:16.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982615+0000", 2024-09-16T03:39:16.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5438000000000002e-05 2024-09-16T03:39:16.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.478 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.479 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1049 4.16 4:6df3bbbe:::smithi15736696-13:head [write 3121996~104915] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:16.479 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:39:16.479 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.809123258, 2024-09-16T03:39:16.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40579890600000001, 2024-09-16T03:39:16.479 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.479 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.479 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.479 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.479 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.479 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.479 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1049 2024-09-16T03:39:16.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.480 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:39:16.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:39:16.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576966+0000", 2024-09-16T03:39:16.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8199999999999999e-07 2024-09-16T03:39:16.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577069+0000", 2024-09-16T03:39:16.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103583 2024-09-16T03:39:16.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577076+0000", 2024-09-16T03:39:16.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8190000000000003e-06 2024-09-16T03:39:16.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577082+0000", 2024-09-16T03:39:16.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5130000000000004e-06 2024-09-16T03:39:16.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880130+0000", 2024-09-16T03:39:16.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.303048127 2024-09-16T03:39:16.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880183+0000", 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3127999999999997e-05 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880282+0000", 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8881000000000004e-05 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883101+0000", 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028194180000000002 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.979716+0000", 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.096614899000000004 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982707+0000", 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029914249999999998 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982739+0000", 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1250000000000001e-05 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.485 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.485 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982764+0000", 2024-09-16T03:39:16.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5080999999999999e-05 2024-09-16T03:39:16.485 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.485 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.485 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.485 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1050 4.16 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 3226911] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:16.485 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:39:16.485 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.808998663, 2024-09-16T03:39:16.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40784258099999998, 2024-09-16T03:39:16.485 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.486 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.486 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.486 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.486 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.486 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.486 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1050 2024-09-16T03:39:16.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.486 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:39:16.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577090+0000", 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000003e-07 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577094+0000", 2024-09-16T03:39:16.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4340000000000001e-06 2024-09-16T03:39:16.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577099+0000", 2024-09-16T03:39:16.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8599999999999998e-06 2024-09-16T03:39:16.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577104+0000", 2024-09-16T03:39:16.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0010000000000001e-06 2024-09-16T03:39:16.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881242+0000", 2024-09-16T03:39:16.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30413779000000002 2024-09-16T03:39:16.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881290+0000", 2024-09-16T03:39:16.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7298e-05 2024-09-16T03:39:16.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:16.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881367+0000", 2024-09-16T03:39:16.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7750000000000006e-05 2024-09-16T03:39:16.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883117+0000", 2024-09-16T03:39:16.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001750096 2024-09-16T03:39:16.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.980218+0000", 2024-09-16T03:39:16.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097100724999999999 2024-09-16T03:39:16.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984859+0000", 2024-09-16T03:39:16.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046412950000000001 2024-09-16T03:39:16.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984900+0000", 2024-09-16T03:39:16.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0300999999999999e-05 2024-09-16T03:39:16.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984932+0000", 2024-09-16T03:39:16.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2332999999999997e-05 2024-09-16T03:39:16.491 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.491 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.492 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.492 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1051 4.16 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:16.492 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:16.492 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.808976383, 2024-09-16T03:39:16.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50807376999999998, 2024-09-16T03:39:16.492 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.492 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.492 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:16.492 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.492 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.492 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1051 2024-09-16T03:39:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:39:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577115+0000", 2024-09-16T03:39:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3980000000000002e-06 2024-09-16T03:39:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577120+0000", 2024-09-16T03:39:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5589999999999996e-06 2024-09-16T03:39:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577125+0000", 2024-09-16T03:39:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9910000000000002e-06 2024-09-16T03:39:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882780+0000", 2024-09-16T03:39:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30565438499999997 2024-09-16T03:39:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882825+0000", 2024-09-16T03:39:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5739999999999999e-05 2024-09-16T03:39:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085039+0000", 2024-09-16T03:39:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20221376499999999 2024-09-16T03:39:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085096+0000", 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6666000000000003e-05 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085186+0000", 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9716000000000006e-05 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1359 4.6 4:64ce87bc:::smithi15736696-20:head [write 1768928~405354] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:39:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 96.219953244999999, 2024-09-16T03:39:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40239613499999999, 2024-09-16T03:39:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1359 2024-09-16T03:39:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166136+0000", 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:39:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166923+0000", 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00078683500000000005 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166933+0000", 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0445000000000001e-05 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166938+0000", 2024-09-16T03:39:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0119999999999996e-06 2024-09-16T03:39:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264901+0000", 2024-09-16T03:39:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097962831 2024-09-16T03:39:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264963+0000", 2024-09-16T03:39:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1812999999999999e-05 2024-09-16T03:39:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.265085+0000", 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121914 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.269430+0000", 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0043455000000000004 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.372259+0000", 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102828645 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.503 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.503 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568289+0000", 2024-09-16T03:39:16.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19603034999999999 2024-09-16T03:39:16.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.503 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.503 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568475+0000", 2024-09-16T03:39:16.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018613599999999999 2024-09-16T03:39:16.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.503 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.503 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568531+0000", 2024-09-16T03:39:16.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5729000000000001e-05 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1676 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v29,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 86.256124381999996, 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50424139300000004, 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.504 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1676 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129968+0000", 2024-09-16T03:39:16.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8430000000000003e-06 2024-09-16T03:39:16.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129982+0000", 2024-09-16T03:39:16.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4514e-05 2024-09-16T03:39:16.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130013+0000", 2024-09-16T03:39:16.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1081999999999998e-05 2024-09-16T03:39:16.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130029+0000", 2024-09-16T03:39:16.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5849999999999999e-05 2024-09-16T03:39:16.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130169+0000", 2024-09-16T03:39:16.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000139973 2024-09-16T03:39:16.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.508 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130240+0000", 2024-09-16T03:39:16.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0638999999999999e-05 2024-09-16T03:39:16.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.508 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:39:16.508 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.532892+0000", 2024-09-16T03:39:16.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402652654 2024-09-16T03:39:16.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.508 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.508 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533546+0000", 2024-09-16T03:39:16.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065390700000000001 2024-09-16T03:39:16.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.509 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.509 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533943+0000", 2024-09-16T03:39:16.509 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039703999999999998 2024-09-16T03:39:16.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.509 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.509 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634089+0000", 2024-09-16T03:39:16.509 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100146098 2024-09-16T03:39:16.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.509 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634153+0000", 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3581e-05 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634205+0000", 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2212000000000001e-05 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:77 3.2 e73/68 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head v 73'117, mlcod=66'115)", 2024-09-16T03:39:16.510 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:39:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 76.207086103999998, 2024-09-16T03:39:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.401393467, 2024-09-16T03:39:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:39:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:39:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179005+0000", 2024-09-16T03:39:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.553e-06 2024-09-16T03:39:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179029+0000", 2024-09-16T03:39:16.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4039000000000001e-05 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179046+0000", 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7139999999999999e-05 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.279144+0000", 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100098245 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379323+0000", 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100178621 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379348+0000", 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5315000000000001e-05 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580315+0000", 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20096734599999999 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580395+0000", 2024-09-16T03:39:16.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0208000000000002e-05 2024-09-16T03:39:16.515 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.515 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.515 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.515 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.515 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.515 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:16.516 INFO:tasks.rados.rados.0.smithi157.stdout:1765: set_chunk oid 47 offset: 463360 length: 636928 target oid 8a777defa75e38fc371085a7cd11b86a5f577618baee3efc60aad962871bb246 offset: 0 snap -1 2024-09-16T03:39:16.516 INFO:tasks.rados.rados.0.smithi157.stdout:1766: set_chunk oid 12 target oid 2024-09-16T03:39:16.517 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:16.514+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:16.517 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:39:16.530 INFO:tasks.rados.rados.0.smithi157.stdout:1766: set_chunk oid 12 offset: 706560 length: 1960960 target oid f0eac3272ef85932293696a6df41f27eb72727e4d8cdcb86ca36f91e206bebd5 offset: 0 snap -1 2024-09-16T03:39:16.530 INFO:tasks.rados.rados.0.smithi157.stdout:1767: read oid 48 snap -1 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:1767: expect (ObjNum 367 snap 0 seq_num 367) 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:1768: tier_flush oid 9 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:1768: tier_flush oid 9 snap -1 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:1764: finishing write tid 1 to smithi15736696-44 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:1769: read oid 41 snap -1 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:1769: expect (ObjNum 376 snap 0 seq_num 376) 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:1770: read oid 37 snap -1 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:1770: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:1771: tier_promote oid 27 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:1764: finishing write tid 2 to smithi15736696-44 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:1764: finishing write tid 3 to smithi15736696-44 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:1765: finishing set_chunk to oid 47 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 132 (ObjNum 362 snap 0 seq_num 362) dirty exists 2024-09-16T03:39:16.531 INFO:tasks.rados.rados.0.smithi157.stdout:1765: done (7 left) 2024-09-16T03:39:16.532 INFO:tasks.rados.rados.0.smithi157.stdout:1772: set_chunk oid 22 target oid 2024-09-16T03:39:16.532 INFO:tasks.rados.rados.0.smithi157.stdout:1772: set_chunk oid 22 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:39:16.532 INFO:tasks.rados.rados.0.smithi157.stdout:1773: read oid 19 snap -1 2024-09-16T03:39:16.532 INFO:tasks.rados.rados.0.smithi157.stdout:1773: expect (ObjNum 331 snap 0 seq_num 331) 2024-09-16T03:39:16.532 INFO:tasks.rados.rados.0.smithi157.stdout:1774: read oid 50 snap -1 2024-09-16T03:39:16.532 INFO:tasks.rados.rados.0.smithi157.stdout:1774: expect (ObjNum 355 snap 0 seq_num 355) 2024-09-16T03:39:16.532 INFO:tasks.rados.rados.0.smithi157.stdout:1775: read oid 5 snap -1 2024-09-16T03:39:16.532 INFO:tasks.rados.rados.0.smithi157.stdout:1775: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-16T03:39:16.532 INFO:tasks.rados.rados.0.smithi157.stdout:1776: read oid 10 snap -1 2024-09-16T03:39:16.532 INFO:tasks.rados.rados.0.smithi157.stdout:1776: expect (ObjNum 375 snap 0 seq_num 375) 2024-09-16T03:39:16.532 INFO:tasks.rados.rados.0.smithi157.stdout:1777: delete oid 3 current snap is 0 2024-09-16T03:39:16.533 INFO:tasks.rados.rados.0.smithi157.stdout:1777: done (12 left) 2024-09-16T03:39:16.533 INFO:tasks.rados.rados.0.smithi157.stdout:1778: write oid 35 current snap is 0 2024-09-16T03:39:16.533 INFO:tasks.rados.rados.0.smithi157.stdout:1778: seq_num 392 ranges {588344=631611,1689059=741123} 2024-09-16T03:39:16.540 INFO:tasks.rados.rados.0.smithi157.stdout:1778: writing smithi15736696-35 from 588344 to 1219955 tid 1 2024-09-16T03:39:16.547 INFO:tasks.rados.rados.0.smithi157.stdout:1778: writing smithi15736696-35 from 1689059 to 2430182 tid 2 2024-09-16T03:39:16.547 INFO:tasks.rados.rados.0.smithi157.stdout:1779: read oid 3 snap -1 2024-09-16T03:39:16.547 INFO:tasks.rados.rados.0.smithi157.stdout:1779: expect deleted 2024-09-16T03:39:16.547 INFO:tasks.rados.rados.0.smithi157.stdout:1780: tier_evict oid 23 2024-09-16T03:39:16.547 INFO:tasks.rados.rados.0.smithi157.stdout:1780: tier_evict oid 23 snap -1 2024-09-16T03:39:16.547 INFO:tasks.rados.rados.0.smithi157.stdout:1769: expect (ObjNum 376 snap 0 seq_num 376) 2024-09-16T03:39:16.579 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:16.576+0000 7f3a21722640 -1 osd.6 140 handle_osd_map injecting map crc failure 2024-09-16T03:39:16.583 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:39:16.596 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 420464 2024-09-16T03:39:16.617 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:16.615+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:16.621 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:16.621 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:16.621 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:16.622 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:16.629 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:39:16.644 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:16.696 INFO:tasks.rados.rados.0.smithi157.stdout:1774: expect (ObjNum 355 snap 0 seq_num 355) 2024-09-16T03:39:16.704 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:16.702+0000 7f66f3205640 -1 osd.15 140 handle_osd_map injecting map crc failure 2024-09-16T03:39:16.718 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:16.716+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:16.733 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:16.733 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:16.733 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:16.733 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:16.733 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:16.734 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:16.731+0000 7f49419d0640 -1 osd.4 140 handle_osd_map injecting map crc failure 2024-09-16T03:39:16.742 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:39:16.819 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:16.817+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:16.847 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:16.847 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:16.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:16.847 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:16.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.847 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:3 3.0 3.f934d18 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:16.848 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:39:16.848 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.445115415, 2024-09-16T03:39:16.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78381636700000001, 2024-09-16T03:39:16.848 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.848 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.848 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:16.848 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:39:16.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:39:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399429+0000", 2024-09-16T03:39:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:39:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399434+0000", 2024-09-16T03:39:16.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1050000000000003e-06 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399442+0000", 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0160000000000008e-06 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399448+0000", 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.879e-06 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399489+0000", 2024-09-16T03:39:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0815999999999997e-05 2024-09-16T03:39:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399495+0000", 2024-09-16T03:39:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3439999999999997e-06 2024-09-16T03:39:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183226+0000", 2024-09-16T03:39:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78373077400000002 2024-09-16T03:39:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183244+0000", 2024-09-16T03:39:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8128999999999999e-05 2024-09-16T03:39:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.852 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:5 3.0 3.aefbbec0 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:39:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.345347953, 2024-09-16T03:39:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68447923499999996, 2024-09-16T03:39:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:39:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499196+0000", 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3480000000000001e-06 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499202+0000", 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4219999999999999e-06 2024-09-16T03:39:16.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499212+0000", 2024-09-16T03:39:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:39:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499218+0000", 2024-09-16T03:39:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4920000000000001e-06 2024-09-16T03:39:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499283+0000", 2024-09-16T03:39:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4906000000000005e-05 2024-09-16T03:39:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499293+0000", 2024-09-16T03:39:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0513000000000001e-05 2024-09-16T03:39:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183665+0000", 2024-09-16T03:39:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68437196600000005 2024-09-16T03:39:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183674+0000", 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9700000000000005e-06 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:6 3.0 3.3bcda120 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.345287344, 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78474037699999999, 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.857 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0500000000000002e-07 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499260+0000", 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6669999999999998e-06 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499266+0000", 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499274+0000", 2024-09-16T03:39:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2250000000000003e-06 2024-09-16T03:39:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499316+0000", 2024-09-16T03:39:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2670000000000003e-05 2024-09-16T03:39:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499323+0000", 2024-09-16T03:39:16.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.37e-06 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283987+0000", 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78466433700000005 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283996+0000", 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8650000000000007e-06 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.862 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:15 3.0 3.7ec93288 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:16.862 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:39:16.862 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.236095334, 2024-09-16T03:39:16.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57518172599999995, 2024-09-16T03:39:16.862 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.862 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.863 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608449+0000", 2024-09-16T03:39:16.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.184e-06 2024-09-16T03:39:16.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608459+0000", 2024-09-16T03:39:16.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0553999999999999e-05 2024-09-16T03:39:16.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608470+0000", 2024-09-16T03:39:16.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0524e-05 2024-09-16T03:39:16.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608493+0000", 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2980999999999999e-05 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608537+0000", 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3850000000000002e-05 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608542+0000", 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:39:16.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183623+0000", 2024-09-16T03:39:16.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.575080649 2024-09-16T03:39:16.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183629+0000", 2024-09-16T03:39:16.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6950000000000003e-06 2024-09-16T03:39:16.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.867 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.868 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:11 3.0 3.77005160 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:16.868 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:39:16.868 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 121.142595024, 2024-09-16T03:39:16.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.48153733399999998, 2024-09-16T03:39:16.868 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.868 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.868 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:16.868 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701949+0000", 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.314e-06 2024-09-16T03:39:16.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701954+0000", 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701964+0000", 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9550000000000004e-06 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701986+0000", 2024-09-16T03:39:16.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1931999999999999e-05 2024-09-16T03:39:16.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802129+0000", 2024-09-16T03:39:16.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100142986 2024-09-16T03:39:16.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:16.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802135+0000", 2024-09-16T03:39:16.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6350000000000003e-06 2024-09-16T03:39:16.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183475+0000", 2024-09-16T03:39:16.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38133958099999998 2024-09-16T03:39:16.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183485+0000", 2024-09-16T03:39:16.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0642000000000001e-05 2024-09-16T03:39:16.872 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.872 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.872 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.872 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:6 3.0 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:16.873 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:39:16.873 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 119.861691111, 2024-09-16T03:39:16.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60469085300000003, 2024-09-16T03:39:16.873 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.873 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.873 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.873 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:39:16.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:39:16.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982853+0000", 2024-09-16T03:39:16.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.393e-06 2024-09-16T03:39:16.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982858+0000", 2024-09-16T03:39:16.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.062e-06 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982867+0000", 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982879+0000", 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2241e-05 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982996+0000", 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116996 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983007+0000", 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0694e-05 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284499+0000", 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30149171400000002 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.876 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284690+0000", 2024-09-16T03:39:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000191032 2024-09-16T03:39:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284908+0000", 2024-09-16T03:39:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021827399999999999 2024-09-16T03:39:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487092+0000", 2024-09-16T03:39:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.202184156 2024-09-16T03:39:16.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487110+0000", 2024-09-16T03:39:16.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7856999999999999e-05 2024-09-16T03:39:16.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587335+0000", 2024-09-16T03:39:16.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100224802 2024-09-16T03:39:16.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587410+0000", 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5343e-05 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587543+0000", 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000132492 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:2 3.0 3:16550cfd:::smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:39:16.879 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 119.86161891899999, 2024-09-16T03:39:16.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40289696000000003, 2024-09-16T03:39:16.880 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.880 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.880 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:16.880 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:39:16.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982925+0000", 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982928+0000", 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4130000000000002e-06 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982936+0000", 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.875e-06 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982942+0000", 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983093+0000", 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015144400000000001 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:16.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983098+0000", 2024-09-16T03:39:16.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4909999999999999e-06 2024-09-16T03:39:16.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385695+0000", 2024-09-16T03:39:16.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402597539 2024-09-16T03:39:16.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.883 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385821+0000", 2024-09-16T03:39:16.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125418 2024-09-16T03:39:16.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.884 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.884 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:9 3.0 3:2c9b9e7c:::smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:39:16.884 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:39:16.884 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 119.86160273, 2024-09-16T03:39:16.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60615655800000001, 2024-09-16T03:39:16.884 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.884 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.884 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.884 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982941+0000", 2024-09-16T03:39:16.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0860000000000001e-06 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982946+0000", 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982954+0000", 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8520000000000004e-06 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982958+0000", 2024-09-16T03:39:16.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4870000000000004e-06 2024-09-16T03:39:16.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983116+0000", 2024-09-16T03:39:16.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015774699999999999 2024-09-16T03:39:16.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:16.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983120+0000", 2024-09-16T03:39:16.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:39:16.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486023+0000", 2024-09-16T03:39:16.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50290281400000003 2024-09-16T03:39:16.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486423+0000", 2024-09-16T03:39:16.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040079299999999998 2024-09-16T03:39:16.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:16.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486707+0000", 2024-09-16T03:39:16.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028336600000000001 2024-09-16T03:39:16.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588861+0000", 2024-09-16T03:39:16.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102154756 2024-09-16T03:39:16.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588952+0000", 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0630000000000005e-05 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588984+0000", 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2116999999999998e-05 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589022+0000", 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7910000000000001e-05 2024-09-16T03:39:16.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589097+0000", 2024-09-16T03:39:16.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4536999999999995e-05 2024-09-16T03:39:16.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.891 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.891 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:12 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:16.891 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:39:16.891 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 119.86152738200001, 2024-09-16T03:39:16.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60616102900000002, 2024-09-16T03:39:16.892 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.892 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.892 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:16.892 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:39:16.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:39:16.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983017+0000", 2024-09-16T03:39:16.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.483e-06 2024-09-16T03:39:16.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983022+0000", 2024-09-16T03:39:16.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983032+0000", 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0036e-05 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983038+0000", 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6800000000000004e-06 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983155+0000", 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116401 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983163+0000", 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6219999999999998e-06 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589120+0000", 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60595653199999999 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589176+0000", 2024-09-16T03:39:16.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6542000000000002e-05 2024-09-16T03:39:16.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.896 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.896 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:30 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:16.896 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:39:16.896 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 115.628208876, 2024-09-16T03:39:16.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40312788999999999, 2024-09-16T03:39:16.896 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.896 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216338+0000", 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0799999999999999e-06 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216354+0000", 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6439e-05 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216392+0000", 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7162000000000001e-05 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216444+0000", 2024-09-16T03:39:16.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2082e-05 2024-09-16T03:39:16.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417447+0000", 2024-09-16T03:39:16.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20100358400000001 2024-09-16T03:39:16.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417583+0000", 2024-09-16T03:39:16.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135537 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417777+0000", 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019431200000000001 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418479+0000", 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00070148499999999996 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418755+0000", 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027627100000000003 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619285+0000", 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200530138 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619370+0000", 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4567999999999994e-05 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619462+0000", 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2231999999999994e-05 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:34 3.0 3:08307d25:::2b3cb797ae2d3abc24b2cae4a012f26325debe2fc38ccb08bd4503e949d305ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 105.369752819, 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30798127600000003, 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.902 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474791+0000", 2024-09-16T03:39:16.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6000000000000003e-07 2024-09-16T03:39:16.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474865+0000", 2024-09-16T03:39:16.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4504000000000004e-05 2024-09-16T03:39:16.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474882+0000", 2024-09-16T03:39:16.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6765999999999998e-05 2024-09-16T03:39:16.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474890+0000", 2024-09-16T03:39:16.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.14e-06 2024-09-16T03:39:16.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681242+0000", 2024-09-16T03:39:16.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20635221200000001 2024-09-16T03:39:16.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681355+0000", 2024-09-16T03:39:16.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000112211 2024-09-16T03:39:16.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:16.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681505+0000", 2024-09-16T03:39:16.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015040100000000001 2024-09-16T03:39:16.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682352+0000", 2024-09-16T03:39:16.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084686400000000002 2024-09-16T03:39:16.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682989+0000", 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063749800000000001 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782695+0000", 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099706063999999997 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782735+0000", 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9304000000000002e-05 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782771+0000", 2024-09-16T03:39:16.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6551999999999998e-05 2024-09-16T03:39:16.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.908 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.908 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:87 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:16.908 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:39:16.908 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 86.040558094999994, 2024-09-16T03:39:16.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402368742, 2024-09-16T03:39:16.908 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.908 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.909 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.909 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:39:16.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:39:16.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803986+0000", 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9400000000000004e-07 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804061+0000", 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5230999999999999e-05 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804069+0000", 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7530000000000001e-06 2024-09-16T03:39:16.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804073+0000", 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9660000000000004e-06 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907236+0000", 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103162566 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907324+0000", 2024-09-16T03:39:16.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7894999999999998e-05 2024-09-16T03:39:16.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:16.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907474+0000", 2024-09-16T03:39:16.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014983999999999999 2024-09-16T03:39:16.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.908330+0000", 2024-09-16T03:39:16.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085600100000000001 2024-09-16T03:39:16.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107220+0000", 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198889812 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206269+0000", 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099049611999999995 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206305+0000", 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5679000000000002e-05 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206353+0000", 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8593e-05 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:3086 3.0 3:1e42685c:::6a9007270aa32b63b7f7e5355768bf791fa2d1832cb8adf59666d35d1b037cad:head [write 0~1679360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 41.432720568000001, 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57631983099999995, 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.914 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 3086 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411824+0000", 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.849e-06 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414291+0000", 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024666900000000001 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414304+0000", 2024-09-16T03:39:16.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2886e-05 2024-09-16T03:39:16.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414310+0000", 2024-09-16T03:39:16.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4219999999999997e-06 2024-09-16T03:39:16.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783484+0000", 2024-09-16T03:39:16.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.36917337700000002 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783601+0000", 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011796899999999999 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783711+0000", 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110012 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887652+0000", 2024-09-16T03:39:16.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103940373 2024-09-16T03:39:16.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887669+0000", 2024-09-16T03:39:16.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6864e-05 2024-09-16T03:39:16.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942330+0000", 2024-09-16T03:39:16.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.054661333999999999 2024-09-16T03:39:16.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987927+0000", 2024-09-16T03:39:16.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045596722999999999 2024-09-16T03:39:16.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987993+0000", 2024-09-16T03:39:16.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6141000000000003e-05 2024-09-16T03:39:16.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.988142+0000", 2024-09-16T03:39:16.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014919099999999999 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:211 3.0 3:09d857e3:::85925bf7ff4b61513b387a03c5bd27d81ce6bd960cfaf104686b99338ce39220:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 41.206541459, 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.55082720799999996, 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:39:16.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:39:16.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638003+0000", 2024-09-16T03:39:16.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8250000000000001e-06 2024-09-16T03:39:16.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638010+0000", 2024-09-16T03:39:16.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4130000000000003e-06 2024-09-16T03:39:16.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638022+0000", 2024-09-16T03:39:16.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2016e-05 2024-09-16T03:39:16.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638028+0000", 2024-09-16T03:39:16.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0390000000000001e-06 2024-09-16T03:39:16.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886321+0000", 2024-09-16T03:39:16.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.248293081 2024-09-16T03:39:16.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886428+0000", 2024-09-16T03:39:16.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000106791 2024-09-16T03:39:16.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:16.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886630+0000", 2024-09-16T03:39:16.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020280499999999999 2024-09-16T03:39:16.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889701+0000", 2024-09-16T03:39:16.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030706650000000002 2024-09-16T03:39:16.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942392+0000", 2024-09-16T03:39:16.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.052691156000000003 2024-09-16T03:39:16.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987675+0000", 2024-09-16T03:39:16.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045283229000000001 2024-09-16T03:39:16.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188710+0000", 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20103479899999999 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188753+0000", 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2762000000000002e-05 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188829+0000", 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5626999999999999e-05 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.927 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.928 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.928 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:214 3.0 3:2fb3d035:::5dba7feb0009e61b5e6a50f88cc7ba99127287f7eeb210342a4b40e996304a6b:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:16.928 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:39:16.928 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 41.206427384999998, 2024-09-16T03:39:16.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.55085424699999996, 2024-09-16T03:39:16.928 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.928 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.928 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.928 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:39:16.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:39:16.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638116+0000", 2024-09-16T03:39:16.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1200000000000002e-07 2024-09-16T03:39:16.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638120+0000", 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3400000000000002e-06 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638126+0000", 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2339999999999999e-06 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638130+0000", 2024-09-16T03:39:16.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6349999999999997e-06 2024-09-16T03:39:16.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887101+0000", 2024-09-16T03:39:16.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.248970462 2024-09-16T03:39:16.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887191+0000", 2024-09-16T03:39:16.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0087000000000007e-05 2024-09-16T03:39:16.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:16.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887375+0000", 2024-09-16T03:39:16.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018401600000000001 2024-09-16T03:39:16.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889706+0000", 2024-09-16T03:39:16.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023305869999999998 2024-09-16T03:39:16.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942851+0000", 2024-09-16T03:39:16.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.053145489999999997 2024-09-16T03:39:16.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.088369+0000", 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.14551832200000001 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188856+0000", 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100486244 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188902+0000", 2024-09-16T03:39:16.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6134999999999998e-05 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188970+0000", 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7983000000000003e-05 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:219 3.0 3:21e0c4e8:::f1ac9f6b1092b2dbfccdb2f32bdf689fb55e405da82aefeec91d0d73cdce51e1:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 41.106621103000002, 2024-09-16T03:39:16.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.71756203500000004, 2024-09-16T03:39:16.935 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.935 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.935 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.935 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:39:16.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:39:16.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737924+0000", 2024-09-16T03:39:16.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8500000000000001e-06 2024-09-16T03:39:16.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737931+0000", 2024-09-16T03:39:16.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9879999999999998e-06 2024-09-16T03:39:16.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737942+0000", 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1837e-05 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.838013+0000", 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100070438 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887695+0000", 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.049682624000000002 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887831+0000", 2024-09-16T03:39:16.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013565499999999999 2024-09-16T03:39:16.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:16.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887948+0000", 2024-09-16T03:39:16.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000117353 2024-09-16T03:39:16.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889709+0000", 2024-09-16T03:39:16.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017605629999999999 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.043145+0000", 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.153436136 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188664+0000", 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.14551857000000001 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455373+0000", 2024-09-16T03:39:16.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.26670951199999998 2024-09-16T03:39:16.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455413+0000", 2024-09-16T03:39:16.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9517999999999997e-05 2024-09-16T03:39:16.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455484+0000", 2024-09-16T03:39:16.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0990999999999998e-05 2024-09-16T03:39:16.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.940 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:3100 3.0 3:2db88641:::a63ac782fa4bce1e96397c8301f2734ce3d5850a2dc454a54c42fc5451a8d72d:head [write 0~1584128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:39:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 40.573040845999998, 2024-09-16T03:39:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38491842399999998, 2024-09-16T03:39:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:16.941 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 3100 2024-09-16T03:39:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:39:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:39:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271523+0000", 2024-09-16T03:39:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0649000000000002e-05 2024-09-16T03:39:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273460+0000", 2024-09-16T03:39:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019373820000000001 2024-09-16T03:39:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273488+0000", 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7537000000000001e-05 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273503+0000", 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5438999999999999e-05 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273645+0000", 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000142013 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273820+0000", 2024-09-16T03:39:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017546200000000001 2024-09-16T03:39:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273943+0000", 2024-09-16T03:39:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012269099999999999 2024-09-16T03:39:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.276532+0000", 2024-09-16T03:39:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025884010000000002 2024-09-16T03:39:16.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455527+0000", 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.17899536199999999 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.556259+0000", 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100731821 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656291+0000", 2024-09-16T03:39:16.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100032144 2024-09-16T03:39:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656330+0000", 2024-09-16T03:39:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8600000000000003e-05 2024-09-16T03:39:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656420+0000", 2024-09-16T03:39:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0922999999999997e-05 2024-09-16T03:39:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.947 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:54 3.0 3:04086e73:::a30f14ebc2b326cba32a5aae93ab4cf09bd2a972b24cfcba49e65d7da3ac52de:head [call cas.chunk_create_or_get_ref in=131071b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 35.489234471000003, 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30746864499999998, 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:39:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:39:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355311+0000", 2024-09-16T03:39:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3250000000000002e-06 2024-09-16T03:39:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355502+0000", 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019106 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355515+0000", 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3730000000000001e-05 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355526+0000", 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0473e-05 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355567+0000", 2024-09-16T03:39:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1455000000000002e-05 2024-09-16T03:39:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355576+0000", 2024-09-16T03:39:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2840000000000006e-06 2024-09-16T03:39:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.457986+0000", 2024-09-16T03:39:16.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10241070200000001 2024-09-16T03:39:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.458087+0000", 2024-09-16T03:39:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000100715 2024-09-16T03:39:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.458252+0000", 2024-09-16T03:39:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016499 2024-09-16T03:39:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560797+0000", 2024-09-16T03:39:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10254485300000001 2024-09-16T03:39:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662591+0000", 2024-09-16T03:39:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10179377100000001 2024-09-16T03:39:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662693+0000", 2024-09-16T03:39:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000102174 2024-09-16T03:39:16.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662745+0000", 2024-09-16T03:39:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1887e-05 2024-09-16T03:39:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662777+0000", 2024-09-16T03:39:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2225999999999999e-05 2024-09-16T03:39:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:58 3.0 3:07e63342:::0385a732ea8ffd270f01137d752687ebb6ebb56bc894ea96f00cc7a27fcd2563:head [call cas.chunk_create_or_get_ref in=284238b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 35.489001637000001, 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30738563099999999, 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355542+0000", 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1699999999999997e-07 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355761+0000", 2024-09-16T03:39:16.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021889699999999999 2024-09-16T03:39:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355769+0000", 2024-09-16T03:39:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.695e-06 2024-09-16T03:39:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355775+0000", 2024-09-16T03:39:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5239999999999997e-06 2024-09-16T03:39:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355851+0000", 2024-09-16T03:39:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6103e-05 2024-09-16T03:39:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355860+0000", 2024-09-16T03:39:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0710000000000001e-06 2024-09-16T03:39:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459143+0000", 2024-09-16T03:39:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10328306399999999 2024-09-16T03:39:16.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459292+0000", 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000148221 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459540+0000", 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024846100000000001 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662661+0000", 2024-09-16T03:39:16.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.203120936 2024-09-16T03:39:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662669+0000", 2024-09-16T03:39:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6520000000000006e-06 2024-09-16T03:39:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662824+0000", 2024-09-16T03:39:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015512699999999999 2024-09-16T03:39:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662885+0000", 2024-09-16T03:39:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0789999999999999e-05 2024-09-16T03:39:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662927+0000", 2024-09-16T03:39:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2172999999999998e-05 2024-09-16T03:39:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.961 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:60 3.0 3:16d0c248:::a2b7d44d389d3ac8752762fa1d216869a8025d7e5ae15e8811d7e7f2543095f6:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 35.488756565999999, 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50513552500000003, 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:16.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355787+0000", 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355791+0000", 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.631e-06 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:16.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355797+0000", 2024-09-16T03:39:16.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7379999999999999e-06 2024-09-16T03:39:16.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:16.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355802+0000", 2024-09-16T03:39:16.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.322e-06 2024-09-16T03:39:16.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.356898+0000", 2024-09-16T03:39:16.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010963710000000001 2024-09-16T03:39:16.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.356902+0000", 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0110000000000002e-06 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560272+0000", 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20336954199999999 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560376+0000", 2024-09-16T03:39:16.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103885 2024-09-16T03:39:16.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:16.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560508+0000", 2024-09-16T03:39:16.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013254599999999999 2024-09-16T03:39:16.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:16.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662786+0000", 2024-09-16T03:39:16.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10227757699999999 2024-09-16T03:39:16.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860745+0000", 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19795950700000001 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860843+0000", 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7829000000000001e-05 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860890+0000", 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7005999999999997e-05 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:16.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:16.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860922+0000", 2024-09-16T03:39:16.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1680000000000002e-05 2024-09-16T03:39:16.968 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.968 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.968 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.968 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:16.968 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:16.968 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:16.968 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:39:16.970 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:16.971 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:39:16.971 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:16.968+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:17.035 INFO:tasks.rados.rados.0.smithi157.stdout:1780: got (22) Invalid argument 2024-09-16T03:39:17.035 INFO:tasks.rados.rados.0.smithi157.stdout:1764: finishing write tid 4 to smithi15736696-44 2024-09-16T03:39:17.035 INFO:tasks.rados.rados.0.smithi157.stdout:1764: oid 44 updating version 0 to 77 2024-09-16T03:39:17.035 INFO:tasks.rados.rados.0.smithi157.stdout:1764: oid 44 version 77 is already newer than 76 2024-09-16T03:39:17.035 INFO:tasks.rados.rados.0.smithi157.stdout:1764: oid 44 updating version 77 to 78 2024-09-16T03:39:17.036 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 78 (ObjNum 390 snap 0 seq_num 390) dirty exists 2024-09-16T03:39:17.036 INFO:tasks.rados.rados.0.smithi157.stdout:1764: left oid 44 (ObjNum 390 snap 0 seq_num 390) 2024-09-16T03:39:17.036 INFO:tasks.rados.rados.0.smithi157.stdout:1766: finishing set_chunk to oid 12 2024-09-16T03:39:17.036 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 12 set_chunk returned error code -95 offset: 706560 length: 1960960 2024-09-16T03:39:17.036 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 0 (ObjNum 297 snap 0 seq_num 297) dirty exists 2024-09-16T03:39:17.036 INFO:tasks.rados.rados.0.smithi157.stdout:1776: expect (ObjNum 375 snap 0 seq_num 375) 2024-09-16T03:39:17.071 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:17.070+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:17.091 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1241026 2024-09-16T03:39:17.095 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:17.096 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:17.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:39:17.113 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:17.113 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:17.115 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:17.115 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:17.115 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:17.124 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:39:17.127 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1820327 2024-09-16T03:39:17.130 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:17.131 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:17.139 INFO:tasks.rados.rados.0.smithi157.stdout:1772: finishing set_chunk to oid 22 2024-09-16T03:39:17.140 INFO:tasks.rados.rados.0.smithi157.stdout:1772: got expected ENOENT (src dne) 2024-09-16T03:39:17.140 INFO:tasks.rados.rados.0.smithi157.stdout:1770: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-16T03:39:17.147 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:17.147 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:17.163 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:17.163 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:17.171 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:17.170+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:17.180 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:17.180 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:17.197 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:17.197 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:17.214 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:17.215 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:17.227 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:17.227 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:17.228 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:17.230 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:17.231 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:17.236 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:39:17.246 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:17.247 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:17.264 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:17.264 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:17.273 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:17.271+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:17.280 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:17.280 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:17.298 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:17.298 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:17.299 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2237406 2024-09-16T03:39:17.314 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:17.315 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:17.331 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:17.331 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:17.339 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:17.339 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:17.348 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:17.348 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:17.349 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:39:17.356 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:17.393 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:17.391+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:17.401 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:17.402 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:39:17.413 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-16T03:39:16.580263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":206510015,"num_objects":140,"num_object_clones":0,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":35,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":1969,"num_read_kb":911500,"num_write":1754,"num_write_kb":381235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":69240114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":21,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":193,"ondisk_log_size":193,"up":207,"acting":207,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":225,"num_osds":13,"num_per_pool_osds":13,"num_per_pool_omap_osds":13,"kb":1226833920,"kb_used":811960,"kb_used_data":397420,"kb_used_omap":20,"kb_used_meta":405163,"kb_avail":1226021960,"statfs":{"total":1256277934080,"available":1255446487040,"internally_reserved":0,"allocated":406958080,"data_stored":413770951,"data_compressed":11180902,"data_compressed_allocated":12079104,"data_compressed_original":25956352,"omap_allocated":20735,"internal_metadata":414887681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1,1,2,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3385098,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":-11828,"num_write":22,"num_write_kb":5841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7137666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.874528"},"pg_stats":[{"pgid":"4.37","version":"131'52","reported_seq":407,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.573095+0000","last_change":"2024-09-16T03:39:08.371859+0000","last_active":"2024-09-16T03:39:13.573095+0000","last_peered":"2024-09-16T03:39:13.573095+0000","last_clean":"2024-09-16T03:39:13.573095+0000","last_became_active":"2024-09-16T03:39:05.721626+0000","last_became_peered":"2024-09-16T03:39:05.721626+0000","last_unstale":"2024-09-16T03:39:13.573095+0000","last_undegraded":"2024-09-16T03:39:13.573095+0000","last_fullsized":"2024-09-16T03:39:13.573095+0000","mapping_epoch":130,"log_start":"104'45","ondisk_log_start":"104'45","created":126,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":44,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:36.963537+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2890887,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":44517,"num_write":12,"num_write_kb":3069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1148090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.36","version":"131'93","reported_seq":472,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680959+0000","last_change":"2024-09-16T03:39:03.716194+0000","last_active":"2024-09-16T03:39:11.680959+0000","last_peered":"2024-09-16T03:39:11.680959+0000","last_clean":"2024-09-16T03:39:11.680959+0000","last_became_active":"2024-09-16T03:39:02.099320+0000","last_became_peered":"2024-09-16T03:39:02.099320+0000","last_unstale":"2024-09-16T03:39:11.680959+0000","last_undegraded":"2024-09-16T03:39:11.680959+0000","last_fullsized":"2024-09-16T03:39:11.680959+0000","mapping_epoch":126,"log_start":"130'90","ondisk_log_start":"130'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.716125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.716125+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":88,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.328700+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269209,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":4665,"num_write":8,"num_write_kb":2101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.35","version":"139'67","reported_seq":487,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.973991+0000","last_change":"2024-09-16T03:39:13.972817+0000","last_active":"2024-09-16T03:39:13.973991+0000","last_peered":"2024-09-16T03:39:13.973991+0000","last_clean":"2024-09-16T03:39:13.973991+0000","last_became_active":"2024-09-16T03:39:13.773139+0000","last_became_peered":"2024-09-16T03:39:13.773139+0000","last_unstale":"2024-09-16T03:39:13.973991+0000","last_undegraded":"2024-09-16T03:39:13.973991+0000","last_fullsized":"2024-09-16T03:39:13.973991+0000","mapping_epoch":138,"log_start":"123'64","ondisk_log_start":"123'64","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":62,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916015,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84364,"num_write":19,"num_write_kb":5459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5435753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"104'20","reported_seq":338,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:15.312712+0000","last_change":"2024-09-16T03:39:02.919356+0000","last_active":"2024-09-16T03:39:15.312712+0000","last_peered":"2024-09-16T03:39:15.312712+0000","last_clean":"2024-09-16T03:39:15.312712+0000","last_became_active":"2024-09-16T03:39:01.699286+0000","last_became_peered":"2024-09-16T03:39:01.699286+0000","last_unstale":"2024-09-16T03:39:15.312712+0000","last_undegraded":"2024-09-16T03:39:15.312712+0000","last_fullsized":"2024-09-16T03:39:15.312712+0000","mapping_epoch":126,"log_start":"98'17","ondisk_log_start":"98'17","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:02.919259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.919259+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.209096+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2705960,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":11460,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"137'75","reported_seq":497,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.579280+0000","last_change":"2024-09-16T03:39:13.974391+0000","last_active":"2024-09-16T03:39:14.579280+0000","last_peered":"2024-09-16T03:39:14.579280+0000","last_clean":"2024-09-16T03:39:14.579280+0000","last_became_active":"2024-09-16T03:39:13.873951+0000","last_became_peered":"2024-09-16T03:39:13.873951+0000","last_unstale":"2024-09-16T03:39:14.579280+0000","last_undegraded":"2024-09-16T03:39:14.579280+0000","last_fullsized":"2024-09-16T03:39:14.579280+0000","mapping_epoch":138,"log_start":"108'70","ondisk_log_start":"108'70","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'72","last_scrub_stamp":"2024-09-16T03:39:02.143339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.143339+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":68,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.465359+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353046,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":11293,"num_write":4,"num_write_kb":955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":994759,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.32","version":"106'17","reported_seq":391,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.075288+0000","last_change":"2024-09-16T03:39:14.075213+0000","last_active":"2024-09-16T03:39:14.075288+0000","last_peered":"2024-09-16T03:39:14.075288+0000","last_clean":"2024-09-16T03:39:14.075288+0000","last_became_active":"2024-09-16T03:39:14.074797+0000","last_became_peered":"2024-09-16T03:39:14.074797+0000","last_unstale":"2024-09-16T03:39:14.075288+0000","last_undegraded":"2024-09-16T03:39:14.075288+0000","last_fullsized":"2024-09-16T03:39:14.075288+0000","mapping_epoch":138,"log_start":"82'13","ondisk_log_start":"82'13","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"106'17","last_scrub_stamp":"2024-09-16T03:39:12.108317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:12.108317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:16.313362+0000","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.31","version":"114'17","reported_seq":305,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.723994+0000","last_change":"2024-09-16T03:39:05.622532+0000","last_active":"2024-09-16T03:39:14.723994+0000","last_peered":"2024-09-16T03:39:14.723994+0000","last_clean":"2024-09-16T03:39:14.723994+0000","last_became_active":"2024-09-16T03:39:01.997491+0000","last_became_peered":"2024-09-16T03:39:01.997491+0000","last_unstale":"2024-09-16T03:39:14.723994+0000","last_undegraded":"2024-09-16T03:39:14.723994+0000","last_fullsized":"2024-09-16T03:39:14.723994+0000","mapping_epoch":126,"log_start":"107'14","ondisk_log_start":"107'14","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'9","last_scrub_stamp":"2024-09-16T03:37:50.756120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:50.756120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.630555+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1495840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":28490,"num_write":4,"num_write_kb":1066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"118'64","reported_seq":402,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.570873+0000","last_change":"2024-09-16T03:39:05.621064+0000","last_active":"2024-09-16T03:39:13.570873+0000","last_peered":"2024-09-16T03:39:13.570873+0000","last_clean":"2024-09-16T03:39:13.570873+0000","last_became_active":"2024-09-16T03:39:01.901037+0000","last_became_peered":"2024-09-16T03:39:01.901037+0000","last_unstale":"2024-09-16T03:39:13.570873+0000","last_undegraded":"2024-09-16T03:39:13.570873+0000","last_fullsized":"2024-09-16T03:39:13.570873+0000","mapping_epoch":126,"log_start":"113'61","ondisk_log_start":"113'61","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:35.157846+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1680717,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":23268,"num_write":10,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":918091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2f","version":"113'94","reported_seq":472,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.571296+0000","last_change":"2024-09-16T03:39:03.319075+0000","last_active":"2024-09-16T03:39:13.571296+0000","last_peered":"2024-09-16T03:39:13.571296+0000","last_clean":"2024-09-16T03:39:13.571296+0000","last_became_active":"2024-09-16T03:39:01.700988+0000","last_became_peered":"2024-09-16T03:39:01.700988+0000","last_unstale":"2024-09-16T03:39:13.571296+0000","last_undegraded":"2024-09-16T03:39:13.571296+0000","last_fullsized":"2024-09-16T03:39:13.571296+0000","mapping_epoch":126,"log_start":"107'90","ondisk_log_start":"107'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"113'94","last_scrub_stamp":"2024-09-16T03:39:03.319012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.319012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":87,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:13.773817+0000","scrub_duration":1038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2e","version":"123'75","reported_seq":457,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.679398+0000","last_change":"2024-09-16T03:39:04.235234+0000","last_active":"2024-09-16T03:39:14.679398+0000","last_peered":"2024-09-16T03:39:14.679398+0000","last_clean":"2024-09-16T03:39:14.679398+0000","last_became_active":"2024-09-16T03:39:01.999291+0000","last_became_peered":"2024-09-16T03:39:01.999291+0000","last_unstale":"2024-09-16T03:39:14.679398+0000","last_undegraded":"2024-09-16T03:39:14.679398+0000","last_fullsized":"2024-09-16T03:39:14.679398+0000","mapping_epoch":126,"log_start":"114'72","ondisk_log_start":"114'72","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"123'75","last_scrub_stamp":"2024-09-16T03:39:04.235141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.235141+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":72,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.010761+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":13153,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"131'99","reported_seq":482,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.902179+0000","last_change":"2024-09-16T03:39:03.917422+0000","last_active":"2024-09-16T03:39:13.902179+0000","last_peered":"2024-09-16T03:39:13.902179+0000","last_clean":"2024-09-16T03:39:13.902179+0000","last_became_active":"2024-09-16T03:39:01.801112+0000","last_became_peered":"2024-09-16T03:39:01.801112+0000","last_unstale":"2024-09-16T03:39:13.902179+0000","last_undegraded":"2024-09-16T03:39:13.902179+0000","last_fullsized":"2024-09-16T03:39:13.902179+0000","mapping_epoch":32,"log_start":"128'96","ondisk_log_start":"128'96","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'97","last_scrub_stamp":"2024-09-16T03:39:03.917355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.917355+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":93,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.686164+0000","scrub_duration":771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2033713,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8316,"num_write":1,"num_write_kb":214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.e","version":"138'82","reported_seq":457,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.579549+0000","last_change":"2024-09-16T03:39:05.725455+0000","last_active":"2024-09-16T03:39:14.579549+0000","last_peered":"2024-09-16T03:39:14.579549+0000","last_clean":"2024-09-16T03:39:14.579549+0000","last_became_active":"2024-09-16T03:39:01.999282+0000","last_became_peered":"2024-09-16T03:39:01.999282+0000","last_unstale":"2024-09-16T03:39:14.579549+0000","last_undegraded":"2024-09-16T03:39:14.579549+0000","last_fullsized":"2024-09-16T03:39:14.579549+0000","mapping_epoch":32,"log_start":"127'79","ondisk_log_start":"127'79","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:39.194237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.194237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":76,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.946501+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2159765,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":28931,"num_write":14,"num_write_kb":4613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"124'19","reported_seq":359,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.671128+0000","last_change":"2024-09-16T03:39:01.149616+0000","last_active":"2024-09-16T03:39:13.671128+0000","last_peered":"2024-09-16T03:39:13.671128+0000","last_clean":"2024-09-16T03:39:13.671128+0000","last_became_active":"2024-09-16T03:38:45.622965+0000","last_became_peered":"2024-09-16T03:38:45.622965+0000","last_unstale":"2024-09-16T03:39:13.671128+0000","last_undegraded":"2024-09-16T03:39:13.671128+0000","last_fullsized":"2024-09-16T03:39:13.671128+0000","mapping_epoch":72,"log_start":"118'16","ondisk_log_start":"118'16","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'19","last_scrub_stamp":"2024-09-16T03:39:01.149486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.149486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":16,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.574017+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613538,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":3908,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"130'65","reported_seq":442,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.777397+0000","last_change":"2024-09-16T03:39:13.775457+0000","last_active":"2024-09-16T03:39:13.777397+0000","last_peered":"2024-09-16T03:39:13.777397+0000","last_clean":"2024-09-16T03:39:13.777397+0000","last_became_active":"2024-09-16T03:39:13.775138+0000","last_became_peered":"2024-09-16T03:39:13.775138+0000","last_unstale":"2024-09-16T03:39:13.777397+0000","last_undegraded":"2024-09-16T03:39:13.777397+0000","last_fullsized":"2024-09-16T03:39:13.777397+0000","mapping_epoch":138,"log_start":"122'62","ondisk_log_start":"122'62","created":32,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'57","last_scrub_stamp":"2024-09-16T03:38:48.923184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.923184+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:53.322016+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":7446,"num_write":9,"num_write_kb":2888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1213217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"129'132","reported_seq":604,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680864+0000","last_change":"2024-09-16T03:39:09.748835+0000","last_active":"2024-09-16T03:39:11.680864+0000","last_peered":"2024-09-16T03:39:11.680864+0000","last_clean":"2024-09-16T03:39:11.680864+0000","last_became_active":"2024-09-16T03:39:09.748391+0000","last_became_peered":"2024-09-16T03:39:09.748391+0000","last_unstale":"2024-09-16T03:39:11.680864+0000","last_undegraded":"2024-09-16T03:39:11.680864+0000","last_fullsized":"2024-09-16T03:39:11.680864+0000","mapping_epoch":134,"log_start":"127'129","ondisk_log_start":"127'129","created":32,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:46.855462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.855462+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":125,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:12.773343+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2084134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":16397,"num_write":31,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.a","version":"128'148","reported_seq":695,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.981860+0000","last_change":"2024-09-16T03:39:01.504751+0000","last_active":"2024-09-16T03:39:13.981860+0000","last_peered":"2024-09-16T03:39:13.981860+0000","last_clean":"2024-09-16T03:39:13.981860+0000","last_became_active":"2024-09-16T03:38:46.021805+0000","last_became_peered":"2024-09-16T03:38:46.021805+0000","last_unstale":"2024-09-16T03:39:13.981860+0000","last_undegraded":"2024-09-16T03:39:13.981860+0000","last_fullsized":"2024-09-16T03:39:13.981860+0000","mapping_epoch":32,"log_start":"123'145","ondisk_log_start":"123'145","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'147","last_scrub_stamp":"2024-09-16T03:39:01.504663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.504663+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":142,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:18.060284+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13949942,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":53,"num_read_kb":53989,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":258,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.573212+0000","last_change":"2024-09-16T03:38:46.881357+0000","last_active":"2024-09-16T03:39:13.573212+0000","last_peered":"2024-09-16T03:39:13.573212+0000","last_clean":"2024-09-16T03:39:13.573212+0000","last_became_active":"2024-09-16T03:38:45.728494+0000","last_became_peered":"2024-09-16T03:38:45.728494+0000","last_unstale":"2024-09-16T03:39:13.573212+0000","last_undegraded":"2024-09-16T03:39:13.573212+0000","last_fullsized":"2024-09-16T03:39:13.573212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:46.881248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.881248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:04.991122+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.8","version":"138'77","reported_seq":478,"reported_epoch":139,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:14.180135+0000","last_change":"2024-09-16T03:39:11.576416+0000","last_active":"2024-09-16T03:39:14.180135+0000","last_peered":"2024-09-16T03:39:14.180135+0000","last_clean":"2024-09-16T03:39:08.372137+0000","last_became_active":"2024-09-16T03:39:11.177629+0000","last_became_peered":"2024-09-16T03:39:11.177629+0000","last_unstale":"2024-09-16T03:39:14.180135+0000","last_undegraded":"2024-09-16T03:39:14.180135+0000","last_fullsized":"2024-09-16T03:39:14.180135+0000","mapping_epoch":135,"log_start":"130'74","ondisk_log_start":"130'74","created":32,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'71","last_scrub_stamp":"2024-09-16T03:39:02.002247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.002247+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":72,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403467,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":2299,"num_write":8,"num_write_kb":1852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,10],"avail_no_missing":["0","4","10"],"object_location_counts":[{"shards":"0,4,10","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"130'85","reported_seq":437,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.681102+0000","last_change":"2024-09-16T03:39:00.900732+0000","last_active":"2024-09-16T03:39:11.681102+0000","last_peered":"2024-09-16T03:39:11.681102+0000","last_clean":"2024-09-16T03:39:11.681102+0000","last_became_active":"2024-09-16T03:38:45.523605+0000","last_became_peered":"2024-09-16T03:38:45.523605+0000","last_unstale":"2024-09-16T03:39:11.681102+0000","last_undegraded":"2024-09-16T03:39:11.681102+0000","last_fullsized":"2024-09-16T03:39:11.681102+0000","mapping_epoch":32,"log_start":"116'82","ondisk_log_start":"116'82","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'82","last_scrub_stamp":"2024-09-16T03:39:00.900619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.900619+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":80,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.385837+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2117748,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":429,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.760857+0000","last_change":"2024-09-16T03:38:36.757156+0000","last_active":"2024-09-16T03:39:11.760857+0000","last_peered":"2024-09-16T03:39:11.760857+0000","last_clean":"2024-09-16T03:39:11.760857+0000","last_became_active":"2024-09-16T03:38:36.756855+0000","last_became_peered":"2024-09-16T03:38:36.756855+0000","last_unstale":"2024-09-16T03:39:11.760857+0000","last_undegraded":"2024-09-16T03:39:11.760857+0000","last_fullsized":"2024-09-16T03:39:11.760857+0000","mapping_epoch":103,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:38:28.063690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:38:28.063690+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:28.682469+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":136,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.772525+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:39:13.772525+0000","last_peered":"2024-09-16T03:39:13.772525+0000","last_clean":"2024-09-16T03:39:13.772525+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:39:13.772525+0000","last_undegraded":"2024-09-16T03:39:13.772525+0000","last_fullsized":"2024-09-16T03:39:13.772525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"131'260","reported_seq":571,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.660977+0000","last_change":"2024-09-16T03:38:01.603352+0000","last_active":"2024-09-16T03:39:11.660977+0000","last_peered":"2024-09-16T03:39:11.660977+0000","last_clean":"2024-09-16T03:39:11.660977+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:39:11.660977+0000","last_undegraded":"2024-09-16T03:39:11.660977+0000","last_fullsized":"2024-09-16T03:39:11.660977+0000","mapping_epoch":74,"log_start":"129'257","ondisk_log_start":"129'257","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":266,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.902901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22624199,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":247,"num_read_kb":8660,"num_write":324,"num_write_kb":70574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.6","version":"108'44","reported_seq":392,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.671865+0000","last_change":"2024-09-16T03:38:48.279289+0000","last_active":"2024-09-16T03:39:13.671865+0000","last_peered":"2024-09-16T03:39:13.671865+0000","last_clean":"2024-09-16T03:39:13.671865+0000","last_became_active":"2024-09-16T03:38:45.924286+0000","last_became_peered":"2024-09-16T03:38:45.924286+0000","last_unstale":"2024-09-16T03:39:13.671865+0000","last_undegraded":"2024-09-16T03:39:13.671865+0000","last_fullsized":"2024-09-16T03:39:13.671865+0000","mapping_epoch":70,"log_start":"105'41","ondisk_log_start":"105'41","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:48.279224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.279224+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.718165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":276,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.782710+0000","last_change":"2024-09-16T03:39:09.749531+0000","last_active":"2024-09-16T03:39:13.782710+0000","last_peered":"2024-09-16T03:39:13.782710+0000","last_clean":"2024-09-16T03:39:13.782710+0000","last_became_active":"2024-09-16T03:39:09.649028+0000","last_became_peered":"2024-09-16T03:39:09.649028+0000","last_unstale":"2024-09-16T03:39:13.782710+0000","last_undegraded":"2024-09-16T03:39:13.782710+0000","last_fullsized":"2024-09-16T03:39:13.782710+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:24.358339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:24.358339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:34.729230+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"133'220","reported_seq":462,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.562283+0000","last_change":"2024-09-16T03:38:15.890094+0000","last_active":"2024-09-16T03:39:11.562283+0000","last_peered":"2024-09-16T03:39:11.562283+0000","last_clean":"2024-09-16T03:39:11.562283+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:39:11.562283+0000","last_undegraded":"2024-09-16T03:39:11.562283+0000","last_fullsized":"2024-09-16T03:39:11.562283+0000","mapping_epoch":31,"log_start":"131'217","ondisk_log_start":"131'217","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'125","last_scrub_stamp":"2024-09-16T03:38:15.889996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.889996+0000","objects_scrubbed":18,"log_size":3,"log_dups_size":228,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:35.350472+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16331162,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":194,"num_read_kb":2025,"num_write":277,"num_write_kb":58956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"139'77","reported_seq":434,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.176785+0000","last_change":"2024-09-16T03:39:01.297447+0000","last_active":"2024-09-16T03:39:14.176785+0000","last_peered":"2024-09-16T03:39:14.176785+0000","last_clean":"2024-09-16T03:39:14.176785+0000","last_became_active":"2024-09-16T03:38:45.723754+0000","last_became_peered":"2024-09-16T03:38:45.723754+0000","last_unstale":"2024-09-16T03:39:14.176785+0000","last_undegraded":"2024-09-16T03:39:14.176785+0000","last_fullsized":"2024-09-16T03:39:14.176785+0000","mapping_epoch":32,"log_start":"127'74","ondisk_log_start":"127'74","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'72","last_scrub_stamp":"2024-09-16T03:39:01.297324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.297324+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":72,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.355718+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7294304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":16354,"num_write":5,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":244,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680762+0000","last_change":"2024-09-16T03:38:15.949627+0000","last_active":"2024-09-16T03:39:11.680762+0000","last_peered":"2024-09-16T03:39:11.680762+0000","last_clean":"2024-09-16T03:39:11.680762+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:39:11.680762+0000","last_undegraded":"2024-09-16T03:39:11.680762+0000","last_fullsized":"2024-09-16T03:39:11.680762+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:15.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:24.286324+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"137'256","reported_seq":650,"reported_epoch":139,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:13.772977+0000","last_change":"2024-09-16T03:39:11.578535+0000","last_active":"2024-09-16T03:39:13.772977+0000","last_peered":"2024-09-16T03:39:13.772977+0000","last_clean":"2024-09-16T03:39:08.210670+0000","last_became_active":"2024-09-16T03:39:11.274646+0000","last_became_peered":"2024-09-16T03:39:11.274646+0000","last_unstale":"2024-09-16T03:39:13.772977+0000","last_undegraded":"2024-09-16T03:39:13.772977+0000","last_fullsized":"2024-09-16T03:39:13.772977+0000","mapping_epoch":135,"log_start":"133'253","ondisk_log_start":"133'253","created":31,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":274,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19810959,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":233,"num_read_kb":12818,"num_write":315,"num_write_kb":80968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":27250603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[3,4,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":18}],"blocked_by":[],"up_primary":9,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"105'32","reported_seq":339,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.660984+0000","last_change":"2024-09-16T03:38:47.042020+0000","last_active":"2024-09-16T03:39:11.660984+0000","last_peered":"2024-09-16T03:39:11.660984+0000","last_clean":"2024-09-16T03:39:11.660984+0000","last_became_active":"2024-09-16T03:38:45.820169+0000","last_became_peered":"2024-09-16T03:38:45.820169+0000","last_unstale":"2024-09-16T03:39:11.660984+0000","last_undegraded":"2024-09-16T03:39:11.660984+0000","last_fullsized":"2024-09-16T03:39:11.660984+0000","mapping_epoch":32,"log_start":"83'29","ondisk_log_start":"83'29","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.041851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.041851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.311785+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":260,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.082589+0000","last_change":"2024-09-16T03:39:14.082589+0000","last_active":"2024-09-16T03:39:14.082589+0000","last_peered":"2024-09-16T03:39:14.082589+0000","last_clean":"2024-09-16T03:39:14.082589+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:39:14.082589+0000","last_undegraded":"2024-09-16T03:39:14.082589+0000","last_fullsized":"2024-09-16T03:39:14.082589+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:39:14.082477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.082477+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.989780+0000","scrub_duration":733,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"134'277","reported_seq":522,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.681013+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:39:11.681013+0000","last_peered":"2024-09-16T03:39:11.681013+0000","last_clean":"2024-09-16T03:39:11.681013+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:39:11.681013+0000","last_undegraded":"2024-09-16T03:39:11.681013+0000","last_fullsized":"2024-09-16T03:39:11.681013+0000","mapping_epoch":31,"log_start":"131'274","ondisk_log_start":"131'274","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":286,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.679811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24729040,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":251,"num_read_kb":6269,"num_write":346,"num_write_kb":73722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":278,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680880+0000","last_change":"2024-09-16T03:38:35.828277+0000","last_active":"2024-09-16T03:39:11.680880+0000","last_peered":"2024-09-16T03:39:11.680880+0000","last_clean":"2024-09-16T03:39:11.680880+0000","last_became_active":"2024-09-16T03:38:33.778361+0000","last_became_peered":"2024-09-16T03:38:33.778361+0000","last_unstale":"2024-09-16T03:39:11.680880+0000","last_undegraded":"2024-09-16T03:39:11.680880+0000","last_fullsized":"2024-09-16T03:39:11.680880+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:35.828203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.828203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:49.130136+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":247,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.662354+0000","last_change":"2024-09-16T03:37:52.652365+0000","last_active":"2024-09-16T03:39:11.662354+0000","last_peered":"2024-09-16T03:39:11.662354+0000","last_clean":"2024-09-16T03:39:11.662354+0000","last_became_active":"2024-09-16T03:37:52.651907+0000","last_became_peered":"2024-09-16T03:37:52.651907+0000","last_unstale":"2024-09-16T03:39:11.662354+0000","last_undegraded":"2024-09-16T03:39:11.662354+0000","last_fullsized":"2024-09-16T03:39:11.662354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.985111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"94'34","reported_seq":349,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.660973+0000","last_change":"2024-09-16T03:38:39.081942+0000","last_active":"2024-09-16T03:39:11.660973+0000","last_peered":"2024-09-16T03:39:11.660973+0000","last_clean":"2024-09-16T03:39:11.660973+0000","last_became_active":"2024-09-16T03:38:33.580022+0000","last_became_peered":"2024-09-16T03:38:33.580022+0000","last_unstale":"2024-09-16T03:39:11.660973+0000","last_undegraded":"2024-09-16T03:39:11.660973+0000","last_fullsized":"2024-09-16T03:39:11.660973+0000","mapping_epoch":96,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:39.081810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.081810+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:48.167935+0000","scrub_duration":473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.582720+0000","last_change":"2024-09-16T03:39:09.848905+0000","last_active":"2024-09-16T03:39:14.582720+0000","last_peered":"2024-09-16T03:39:14.582720+0000","last_clean":"2024-09-16T03:39:14.582720+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:39:14.582720+0000","last_undegraded":"2024-09-16T03:39:14.582720+0000","last_fullsized":"2024-09-16T03:39:14.582720+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:48.949444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"139'133","reported_seq":582,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.003402+0000","last_change":"2024-09-16T03:38:39.704994+0000","last_active":"2024-09-16T03:39:14.003402+0000","last_peered":"2024-09-16T03:39:14.003402+0000","last_clean":"2024-09-16T03:39:14.003402+0000","last_became_active":"2024-09-16T03:38:33.581141+0000","last_became_peered":"2024-09-16T03:38:33.581141+0000","last_unstale":"2024-09-16T03:39:14.003402+0000","last_undegraded":"2024-09-16T03:39:14.003402+0000","last_fullsized":"2024-09-16T03:39:14.003402+0000","mapping_epoch":32,"log_start":"129'130","ondisk_log_start":"129'130","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'118","last_scrub_stamp":"2024-09-16T03:38:39.704919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.704919+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":127,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.098482+0000","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4554829,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":62,"num_read_kb":77035,"num_write":18,"num_write_kb":5604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":258,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.573121+0000","last_change":"2024-09-16T03:38:12.041048+0000","last_active":"2024-09-16T03:39:13.573121+0000","last_peered":"2024-09-16T03:39:13.573121+0000","last_clean":"2024-09-16T03:39:13.573121+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:39:13.573121+0000","last_undegraded":"2024-09-16T03:39:13.573121+0000","last_fullsized":"2024-09-16T03:39:13.573121+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:12.040917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.040917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.877949+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":270,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.679338+0000","last_change":"2024-09-16T03:38:34.928371+0000","last_active":"2024-09-16T03:39:14.679338+0000","last_peered":"2024-09-16T03:39:14.679338+0000","last_clean":"2024-09-16T03:39:14.679338+0000","last_became_active":"2024-09-16T03:38:33.484091+0000","last_became_peered":"2024-09-16T03:38:33.484091+0000","last_unstale":"2024-09-16T03:39:14.679338+0000","last_undegraded":"2024-09-16T03:39:14.679338+0000","last_fullsized":"2024-09-16T03:39:14.679338+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:34.928298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.928298+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:43.935101+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":251,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.782769+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:39:13.782769+0000","last_peered":"2024-09-16T03:39:13.782769+0000","last_clean":"2024-09-16T03:39:13.782769+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:39:13.782769+0000","last_undegraded":"2024-09-16T03:39:13.782769+0000","last_fullsized":"2024-09-16T03:39:13.782769+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.232558+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.10","version":"118'64","reported_seq":409,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.571104+0000","last_change":"2024-09-16T03:39:04.442655+0000","last_active":"2024-09-16T03:39:13.571104+0000","last_peered":"2024-09-16T03:39:13.571104+0000","last_clean":"2024-09-16T03:39:13.571104+0000","last_became_active":"2024-09-16T03:39:01.900921+0000","last_became_peered":"2024-09-16T03:39:01.900921+0000","last_unstale":"2024-09-16T03:39:13.571104+0000","last_undegraded":"2024-09-16T03:39:13.571104+0000","last_fullsized":"2024-09-16T03:39:13.571104+0000","mapping_epoch":72,"log_start":"113'61","ondisk_log_start":"113'61","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"118'64","last_scrub_stamp":"2024-09-16T03:39:04.442556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.442556+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.917777+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361433,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"114'17","reported_seq":317,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.724049+0000","last_change":"2024-09-16T03:39:03.929961+0000","last_active":"2024-09-16T03:39:14.724049+0000","last_peered":"2024-09-16T03:39:14.724049+0000","last_clean":"2024-09-16T03:39:14.724049+0000","last_became_active":"2024-09-16T03:39:01.696690+0000","last_became_peered":"2024-09-16T03:39:01.696690+0000","last_unstale":"2024-09-16T03:39:14.724049+0000","last_undegraded":"2024-09-16T03:39:14.724049+0000","last_fullsized":"2024-09-16T03:39:14.724049+0000","mapping_epoch":63,"log_start":"107'14","ondisk_log_start":"107'14","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:03.929863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.929863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:26.527714+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2991679,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":7626,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"127'20","reported_seq":397,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.974721+0000","last_change":"2024-09-16T03:39:13.974599+0000","last_active":"2024-09-16T03:39:13.974721+0000","last_peered":"2024-09-16T03:39:13.974721+0000","last_clean":"2024-09-16T03:39:13.974721+0000","last_became_active":"2024-09-16T03:39:13.974170+0000","last_became_peered":"2024-09-16T03:39:13.974170+0000","last_unstale":"2024-09-16T03:39:13.974721+0000","last_undegraded":"2024-09-16T03:39:13.974721+0000","last_fullsized":"2024-09-16T03:39:13.974721+0000","mapping_epoch":138,"log_start":"106'17","ondisk_log_start":"106'17","created":50,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966452,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":26744,"num_write":10,"num_write_kb":2629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3741138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.13","version":"127'73","reported_seq":480,"reported_epoch":140,"state":"activating","last_fresh":"2024-09-16T03:39:14.579589+0000","last_change":"2024-09-16T03:39:14.579589+0000","last_active":"2024-09-16T03:39:13.573601+0000","last_peered":"2024-09-16T03:39:12.672413+0000","last_clean":"2024-09-16T03:39:12.672413+0000","last_became_active":"2024-09-16T03:39:11.176806+0000","last_became_peered":"2024-09-16T03:39:11.176806+0000","last_unstale":"2024-09-16T03:39:14.579589+0000","last_undegraded":"2024-09-16T03:39:14.579589+0000","last_fullsized":"2024-09-16T03:39:14.579589+0000","mapping_epoch":139,"log_start":"108'70","ondisk_log_start":"108'70","created":50,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'73","last_scrub_stamp":"2024-09-16T03:39:03.124583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.124583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":68,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":["4","3","9","14"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"104'20","reported_seq":334,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.824514+0000","last_change":"2024-09-16T03:39:01.898409+0000","last_active":"2024-09-16T03:39:14.824514+0000","last_peered":"2024-09-16T03:39:14.824514+0000","last_clean":"2024-09-16T03:39:14.824514+0000","last_became_active":"2024-09-16T03:39:01.596287+0000","last_became_peered":"2024-09-16T03:39:01.596287+0000","last_unstale":"2024-09-16T03:39:14.824514+0000","last_undegraded":"2024-09-16T03:39:14.824514+0000","last_fullsized":"2024-09-16T03:39:14.824514+0000","mapping_epoch":114,"log_start":"98'17","ondisk_log_start":"98'17","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:01.898331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.898331+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.472424+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"123'65","reported_seq":486,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.175562+0000","last_change":"2024-09-16T03:39:14.175562+0000","last_active":"2024-09-16T03:39:14.175562+0000","last_peered":"2024-09-16T03:39:14.175562+0000","last_clean":"2024-09-16T03:39:14.175562+0000","last_became_active":"2024-09-16T03:39:13.671769+0000","last_became_peered":"2024-09-16T03:39:13.671769+0000","last_unstale":"2024-09-16T03:39:14.175562+0000","last_undegraded":"2024-09-16T03:39:14.175562+0000","last_fullsized":"2024-09-16T03:39:14.175562+0000","mapping_epoch":138,"log_start":"123'62","ondisk_log_start":"123'62","created":50,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"123'65","last_scrub_stamp":"2024-09-16T03:39:14.175493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.175493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:31.666781+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"124'86","reported_seq":458,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.681074+0000","last_change":"2024-09-16T03:39:03.073530+0000","last_active":"2024-09-16T03:39:11.681074+0000","last_peered":"2024-09-16T03:39:11.681074+0000","last_clean":"2024-09-16T03:39:11.681074+0000","last_became_active":"2024-09-16T03:39:01.797805+0000","last_became_peered":"2024-09-16T03:39:01.797805+0000","last_unstale":"2024-09-16T03:39:11.681074+0000","last_undegraded":"2024-09-16T03:39:11.681074+0000","last_fullsized":"2024-09-16T03:39:11.681074+0000","mapping_epoch":115,"log_start":"124'82","ondisk_log_start":"124'82","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.073459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.073459+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":80,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:32.204501+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.17","version":"137'55","reported_seq":416,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.573387+0000","last_change":"2024-09-16T03:39:05.721286+0000","last_active":"2024-09-16T03:39:13.573387+0000","last_peered":"2024-09-16T03:39:13.573387+0000","last_clean":"2024-09-16T03:39:13.573387+0000","last_became_active":"2024-09-16T03:39:03.717420+0000","last_became_peered":"2024-09-16T03:39:03.717420+0000","last_unstale":"2024-09-16T03:39:13.573387+0000","last_undegraded":"2024-09-16T03:39:13.573387+0000","last_fullsized":"2024-09-16T03:39:13.573387+0000","mapping_epoch":128,"log_start":"131'52","ondisk_log_start":"131'52","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":51,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.790793+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1630328,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":44517,"num_write":15,"num_write_kb":4042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2516795,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"128'51","reported_seq":386,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.782591+0000","last_change":"2024-09-16T03:39:02.668794+0000","last_active":"2024-09-16T03:39:13.782591+0000","last_peered":"2024-09-16T03:39:13.782591+0000","last_clean":"2024-09-16T03:39:13.782591+0000","last_became_active":"2024-09-16T03:38:47.562641+0000","last_became_peered":"2024-09-16T03:38:47.562641+0000","last_unstale":"2024-09-16T03:39:13.782591+0000","last_undegraded":"2024-09-16T03:39:13.782591+0000","last_fullsized":"2024-09-16T03:39:13.782591+0000","mapping_epoch":112,"log_start":"124'48","ondisk_log_start":"124'48","created":50,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'48","last_scrub_stamp":"2024-09-16T03:39:02.668681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.668681+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":45,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.451476+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067327,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":25040,"num_write":23,"num_write_kb":6049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.582522+0000","last_change":"2024-09-16T03:39:09.847101+0000","last_active":"2024-09-16T03:39:14.582522+0000","last_peered":"2024-09-16T03:39:14.582522+0000","last_clean":"2024-09-16T03:39:14.582522+0000","last_became_active":"2024-09-16T03:39:09.846691+0000","last_became_peered":"2024-09-16T03:39:09.846691+0000","last_unstale":"2024-09-16T03:39:14.582522+0000","last_undegraded":"2024-09-16T03:39:14.582522+0000","last_fullsized":"2024-09-16T03:39:14.582522+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:01.106840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.106840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.495875+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"128'125","reported_seq":593,"reported_epoch":139,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:13.872959+0000","last_change":"2024-09-16T03:38:47.159152+0000","last_active":"2024-09-16T03:39:13.872959+0000","last_peered":"2024-09-16T03:39:13.872959+0000","last_clean":"2024-09-16T03:38:43.521485+0000","last_became_active":"2024-09-16T03:38:46.957471+0000","last_became_peered":"2024-09-16T03:38:46.957471+0000","last_unstale":"2024-09-16T03:39:13.872959+0000","last_undegraded":"2024-09-16T03:39:13.872959+0000","last_fullsized":"2024-09-16T03:39:13.872959+0000","mapping_epoch":111,"log_start":"117'122","ondisk_log_start":"117'122","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'115","last_scrub_stamp":"2024-09-16T03:38:35.254488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.254488+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":122,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6739673,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":31161,"num_write":9,"num_write_kb":2915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6,11],"acting":[3,8,10],"avail_no_missing":["3","8","10"],"object_location_counts":[{"shards":"3,8,10","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"138'131","reported_seq":603,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.773024+0000","last_change":"2024-09-16T03:38:50.465770+0000","last_active":"2024-09-16T03:39:13.773024+0000","last_peered":"2024-09-16T03:39:13.773024+0000","last_clean":"2024-09-16T03:39:13.773024+0000","last_became_active":"2024-09-16T03:38:50.465463+0000","last_became_peered":"2024-09-16T03:38:50.465463+0000","last_unstale":"2024-09-16T03:39:13.773024+0000","last_undegraded":"2024-09-16T03:39:13.773024+0000","last_fullsized":"2024-09-16T03:39:13.773024+0000","mapping_epoch":115,"log_start":"136'128","ondisk_log_start":"136'128","created":100,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'107","last_scrub_stamp":"2024-09-16T03:38:39.905697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.905697+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":122,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.828899+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4655671,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":83952,"num_write":28,"num_write_kb":8389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2376197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"99'56","reported_seq":375,"reported_epoch":136,"state":"peering","last_fresh":"2024-09-16T03:39:11.493056+0000","last_change":"2024-09-16T03:39:11.662898+0000","last_active":"2024-09-16T03:39:11.662898+0000","last_peered":"2024-09-16T03:39:11.493056+0000","last_clean":"2024-09-16T03:39:11.493056+0000","last_became_active":"2024-09-16T03:39:11.177251+0000","last_became_peered":"2024-09-16T03:39:11.177251+0000","last_unstale":"2024-09-16T03:39:11.493056+0000","last_undegraded":"2024-09-16T03:39:11.493056+0000","last_fullsized":"2024-09-16T03:39:11.493056+0000","mapping_epoch":137,"log_start":"90'53","ondisk_log_start":"90'53","created":100,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":5,"last_scrub":"99'56","last_scrub_stamp":"2024-09-16T03:39:00.988517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.988517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,9,10],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"92'14","reported_seq":289,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.873443+0000","last_change":"2024-09-16T03:38:49.092747+0000","last_active":"2024-09-16T03:39:13.873443+0000","last_peered":"2024-09-16T03:39:13.873443+0000","last_clean":"2024-09-16T03:39:13.873443+0000","last_became_active":"2024-09-16T03:38:48.661213+0000","last_became_peered":"2024-09-16T03:38:48.661213+0000","last_unstale":"2024-09-16T03:39:13.873443+0000","last_undegraded":"2024-09-16T03:39:13.873443+0000","last_fullsized":"2024-09-16T03:39:13.873443+0000","mapping_epoch":113,"log_start":"91'11","ondisk_log_start":"91'11","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:49.092687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.092687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.372348+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,12],"acting":[3,11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"92'59","reported_seq":398,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.673168+0000","last_change":"2024-09-16T03:38:48.561603+0000","last_active":"2024-09-16T03:39:13.673168+0000","last_peered":"2024-09-16T03:39:13.673168+0000","last_clean":"2024-09-16T03:39:13.673168+0000","last_became_active":"2024-09-16T03:38:48.561170+0000","last_became_peered":"2024-09-16T03:38:48.561170+0000","last_unstale":"2024-09-16T03:39:13.673168+0000","last_undegraded":"2024-09-16T03:39:13.673168+0000","last_fullsized":"2024-09-16T03:39:13.673168+0000","mapping_epoch":113,"log_start":"89'56","ondisk_log_start":"89'56","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'59","last_scrub_stamp":"2024-09-16T03:38:35.977646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.977646+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:38.632635+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,6],"acting":[14,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1f","version":"137'119","reported_seq":519,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.773051+0000","last_change":"2024-09-16T03:38:48.762617+0000","last_active":"2024-09-16T03:39:13.773051+0000","last_peered":"2024-09-16T03:39:13.773051+0000","last_clean":"2024-09-16T03:39:13.773051+0000","last_became_active":"2024-09-16T03:38:48.761213+0000","last_became_peered":"2024-09-16T03:38:48.761213+0000","last_unstale":"2024-09-16T03:39:13.773051+0000","last_undegraded":"2024-09-16T03:39:13.773051+0000","last_fullsized":"2024-09-16T03:39:13.773051+0000","mapping_epoch":113,"log_start":"137'114","ondisk_log_start":"137'114","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'95","last_scrub_stamp":"2024-09-16T03:38:37.201959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:37.201959+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":114,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.307408+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8134753,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":48941,"num_write":35,"num_write_kb":8979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1704821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,12],"acting":[3,10,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"139'146","reported_seq":619,"reported_epoch":140,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:14.723059+0000","last_change":"2024-09-16T03:38:46.862171+0000","last_active":"2024-09-16T03:39:14.723059+0000","last_peered":"2024-09-16T03:39:14.723059+0000","last_clean":"2024-09-16T03:38:44.034266+0000","last_became_active":"2024-09-16T03:38:46.560752+0000","last_became_peered":"2024-09-16T03:38:46.560752+0000","last_unstale":"2024-09-16T03:39:14.723059+0000","last_undegraded":"2024-09-16T03:39:14.723059+0000","last_fullsized":"2024-09-16T03:39:14.723059+0000","mapping_epoch":111,"log_start":"127'143","ondisk_log_start":"127'143","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'118","last_scrub_stamp":"2024-09-16T03:38:36.698150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:36.698150+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":143,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":88353,"num_write":32,"num_write_kb":9785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[4,10,15],"avail_no_missing":["4","10","15"],"object_location_counts":[{"shards":"4,10,15","objects":3}],"blocked_by":[],"up_primary":8,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.21","version":"48'7","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.572953+0000","last_change":"2024-09-16T03:38:49.064947+0000","last_active":"2024-09-16T03:39:13.572953+0000","last_peered":"2024-09-16T03:39:13.572953+0000","last_clean":"2024-09-16T03:39:13.572953+0000","last_became_active":"2024-09-16T03:38:48.559431+0000","last_became_peered":"2024-09-16T03:38:48.559431+0000","last_unstale":"2024-09-16T03:39:13.572953+0000","last_undegraded":"2024-09-16T03:39:13.572953+0000","last_fullsized":"2024-09-16T03:39:13.572953+0000","mapping_epoch":113,"log_start":"40'4","ondisk_log_start":"40'4","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":6,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:49.064870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064870+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.310170+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,10],"acting":[14,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.22","version":"45'6","reported_seq":156,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.582720+0000","last_change":"2024-09-16T03:39:13.974174+0000","last_active":"2024-09-16T03:39:14.582720+0000","last_peered":"2024-09-16T03:39:14.582720+0000","last_clean":"2024-09-16T03:39:14.582720+0000","last_became_active":"2024-09-16T03:39:13.973813+0000","last_became_peered":"2024-09-16T03:39:13.973813+0000","last_unstale":"2024-09-16T03:39:14.582720+0000","last_undegraded":"2024-09-16T03:39:14.582720+0000","last_fullsized":"2024-09-16T03:39:14.582720+0000","mapping_epoch":138,"log_start":"39'3","ondisk_log_start":"39'3","created":100,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:39:00.968448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.968448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.929675+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,6],"acting":[9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.23","version":"108'38","reported_seq":365,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.573224+0000","last_change":"2024-09-16T03:38:49.464614+0000","last_active":"2024-09-16T03:39:13.573224+0000","last_peered":"2024-09-16T03:39:13.573224+0000","last_clean":"2024-09-16T03:39:13.573224+0000","last_became_active":"2024-09-16T03:38:49.463797+0000","last_became_peered":"2024-09-16T03:38:49.463797+0000","last_unstale":"2024-09-16T03:39:13.573224+0000","last_undegraded":"2024-09-16T03:39:13.573224+0000","last_fullsized":"2024-09-16T03:39:13.573224+0000","mapping_epoch":114,"log_start":"105'36","ondisk_log_start":"105'36","created":100,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:34.893524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.893524+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":35,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:52.829073+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,2],"acting":[14,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.24","version":"105'32","reported_seq":337,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.761038+0000","last_change":"2024-09-16T03:38:47.978956+0000","last_active":"2024-09-16T03:39:11.761038+0000","last_peered":"2024-09-16T03:39:11.761038+0000","last_clean":"2024-09-16T03:39:11.761038+0000","last_became_active":"2024-09-16T03:38:45.820201+0000","last_became_peered":"2024-09-16T03:38:45.820201+0000","last_unstale":"2024-09-16T03:39:11.761038+0000","last_undegraded":"2024-09-16T03:39:11.761038+0000","last_fullsized":"2024-09-16T03:39:11.761038+0000","mapping_epoch":110,"log_start":"83'29","ondisk_log_start":"83'29","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.978888+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:56.881865+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.25","version":"130'85","reported_seq":426,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680968+0000","last_change":"2024-09-16T03:38:48.088833+0000","last_active":"2024-09-16T03:39:11.680968+0000","last_peered":"2024-09-16T03:39:11.680968+0000","last_clean":"2024-09-16T03:39:11.680968+0000","last_became_active":"2024-09-16T03:38:45.724757+0000","last_became_peered":"2024-09-16T03:38:45.724757+0000","last_unstale":"2024-09-16T03:39:11.680968+0000","last_undegraded":"2024-09-16T03:39:11.680968+0000","last_fullsized":"2024-09-16T03:39:11.680968+0000","mapping_epoch":110,"log_start":"116'82","ondisk_log_start":"116'82","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'77","last_scrub_stamp":"2024-09-16T03:38:48.088768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.088768+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":79,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:53.112635+0000","scrub_duration":388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2134555,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":10,"num_write_kb":2500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.26","version":"137'57","reported_seq":416,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.571298+0000","last_change":"2024-09-16T03:38:49.477393+0000","last_active":"2024-09-16T03:39:13.571298+0000","last_peered":"2024-09-16T03:39:13.571298+0000","last_clean":"2024-09-16T03:39:13.571298+0000","last_became_active":"2024-09-16T03:38:45.926029+0000","last_became_peered":"2024-09-16T03:38:45.926029+0000","last_unstale":"2024-09-16T03:39:13.571298+0000","last_undegraded":"2024-09-16T03:39:13.571298+0000","last_fullsized":"2024-09-16T03:39:13.571298+0000","mapping_epoch":110,"log_start":"133'54","ondisk_log_start":"133'54","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:49.477297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.477297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.242430+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336239,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":24204,"num_write":15,"num_write_kb":4159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.27","version":"109'66","reported_seq":399,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.571408+0000","last_change":"2024-09-16T03:38:47.511343+0000","last_active":"2024-09-16T03:39:13.571408+0000","last_peered":"2024-09-16T03:39:13.571408+0000","last_clean":"2024-09-16T03:39:13.571408+0000","last_became_active":"2024-09-16T03:38:45.824884+0000","last_became_peered":"2024-09-16T03:38:45.824884+0000","last_unstale":"2024-09-16T03:39:13.571408+0000","last_undegraded":"2024-09-16T03:39:13.571408+0000","last_fullsized":"2024-09-16T03:39:13.571408+0000","mapping_epoch":110,"log_start":"105'63","ondisk_log_start":"105'63","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'66","last_scrub_stamp":"2024-09-16T03:38:47.511252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.511252+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.884916+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.28","version":"129'68","reported_seq":493,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.178559+0000","last_change":"2024-09-16T03:39:13.974976+0000","last_active":"2024-09-16T03:39:14.178559+0000","last_peered":"2024-09-16T03:39:14.178559+0000","last_clean":"2024-09-16T03:39:14.178559+0000","last_became_active":"2024-09-16T03:39:13.974464+0000","last_became_peered":"2024-09-16T03:39:13.974464+0000","last_unstale":"2024-09-16T03:39:14.178559+0000","last_undegraded":"2024-09-16T03:39:14.178559+0000","last_fullsized":"2024-09-16T03:39:14.178559+0000","mapping_epoch":138,"log_start":"118'65","ondisk_log_start":"118'65","created":110,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:48.551929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.551929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":64,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:10.916309+0000","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395113,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":46322,"num_write":5,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7221888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":261,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.572963+0000","last_change":"2024-09-16T03:39:00.834877+0000","last_active":"2024-09-16T03:39:13.572963+0000","last_peered":"2024-09-16T03:39:13.572963+0000","last_clean":"2024-09-16T03:39:13.572963+0000","last_became_active":"2024-09-16T03:38:45.627369+0000","last_became_peered":"2024-09-16T03:38:45.627369+0000","last_unstale":"2024-09-16T03:39:13.572963+0000","last_undegraded":"2024-09-16T03:39:13.572963+0000","last_fullsized":"2024-09-16T03:39:13.572963+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:00.834742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.834742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.318226+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.2a","version":"108'122","reported_seq":614,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.982814+0000","last_change":"2024-09-16T03:38:47.964525+0000","last_active":"2024-09-16T03:39:13.982814+0000","last_peered":"2024-09-16T03:39:13.982814+0000","last_clean":"2024-09-16T03:39:13.982814+0000","last_became_active":"2024-09-16T03:38:46.022108+0000","last_became_peered":"2024-09-16T03:38:46.022108+0000","last_unstale":"2024-09-16T03:39:13.982814+0000","last_undegraded":"2024-09-16T03:39:13.982814+0000","last_fullsized":"2024-09-16T03:39:13.982814+0000","mapping_epoch":110,"log_start":"107'115","ondisk_log_start":"107'115","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'122","last_scrub_stamp":"2024-09-16T03:38:47.964439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.964439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":112,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.917123+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.2b","version":"127'124","reported_seq":590,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-16T03:39:11.680858+0000","last_change":"2024-09-16T03:39:09.949533+0000","last_active":"2024-09-16T03:39:11.680858+0000","last_peered":"2024-09-16T03:39:11.680858+0000","last_clean":"2024-09-16T03:39:11.680858+0000","last_became_active":"2024-09-16T03:39:09.949146+0000","last_became_peered":"2024-09-16T03:39:09.949146+0000","last_unstale":"2024-09-16T03:39:11.680858+0000","last_undegraded":"2024-09-16T03:39:11.680858+0000","last_fullsized":"2024-09-16T03:39:11.680858+0000","mapping_epoch":134,"log_start":"119'121","ondisk_log_start":"119'121","created":110,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:49.064061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064061+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":121,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:16.396765+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2c","version":"138'64","reported_seq":433,"reported_epoch":139,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:13.877237+0000","last_change":"2024-09-16T03:39:11.380628+0000","last_active":"2024-09-16T03:39:13.877237+0000","last_peered":"2024-09-16T03:39:13.877237+0000","last_clean":"2024-09-16T03:39:07.529785+0000","last_became_active":"2024-09-16T03:39:11.279455+0000","last_became_peered":"2024-09-16T03:39:11.279455+0000","last_unstale":"2024-09-16T03:39:13.877237+0000","last_undegraded":"2024-09-16T03:39:13.877237+0000","last_fullsized":"2024-09-16T03:39:13.877237+0000","mapping_epoch":135,"log_start":"118'61","ondisk_log_start":"118'61","created":110,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'58","last_scrub_stamp":"2024-09-16T03:38:47.889276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.889276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":18905,"num_write":5,"num_write_kb":1234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1262403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,11],"avail_no_missing":["12","11","15"],"object_location_counts":[{"shards":"11,12,15","objects":1}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.2d","version":"92'14","reported_seq":342,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.771105+0000","last_change":"2024-09-16T03:38:47.115739+0000","last_active":"2024-09-16T03:39:13.771105+0000","last_peered":"2024-09-16T03:39:13.771105+0000","last_clean":"2024-09-16T03:39:13.771105+0000","last_became_active":"2024-09-16T03:38:45.723438+0000","last_became_peered":"2024-09-16T03:38:45.723438+0000","last_unstale":"2024-09-16T03:39:13.771105+0000","last_undegraded":"2024-09-16T03:39:13.771105+0000","last_fullsized":"2024-09-16T03:39:13.771105+0000","mapping_epoch":110,"log_start":"91'11","ondisk_log_start":"91'11","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:47.115651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.115651+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:00.107802+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":56,"stat_sum":{"num_bytes":122424268,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":958,"num_read_kb":881655,"num_write":345,"num_write_kb":94341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":28533152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":21,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":234151936,"data_stored":233554255,"data_compressed":389631,"data_compressed_allocated":430080,"data_compressed_original":1110016,"omap_allocated":0,"internal_metadata":0},"log_size":175,"ondisk_log_size":175,"up":168,"acting":168,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":83495360,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":925,"num_read_kb":29772,"num_write":1262,"num_write_kb":284220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":39649106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180535296,"data_stored":193017454,"data_compressed":11968527,"data_compressed_allocated":12718080,"data_compressed_original":25575424,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":2644640,"data_compressed":159066,"data_compressed_allocated":204800,"data_compressed_original":2584576,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247838,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74436,"kb_used_data":43248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94297404,"statfs":{"total":96636764160,"available":96560541696,"internally_reserved":0,"allocated":44285952,"data_stored":44886400,"data_compressed":1343912,"data_compressed_allocated":1433600,"data_compressed_original":2613248,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247839,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56776,"kb_used_data":25584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94315064,"statfs":{"total":96636764160,"available":96578625536,"internally_reserved":0,"allocated":26198016,"data_stored":27310439,"data_compressed":905243,"data_compressed_allocated":991232,"data_compressed_original":2641920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247838,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68900,"kb_used_data":37700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302940,"statfs":{"total":96636764160,"available":96566210560,"internally_reserved":0,"allocated":38604800,"data_stored":39803729,"data_compressed":1368231,"data_compressed_allocated":1474560,"data_compressed_original":3207168,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280544,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55464,"kb_used_data":23664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316376,"statfs":{"total":96636764160,"available":96579969024,"internally_reserved":0,"allocated":24231936,"data_stored":25264169,"data_compressed":880190,"data_compressed_allocated":962560,"data_compressed_original":2510848,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280544,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280543,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313248,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35008,"kb_used_data":7980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336832,"statfs":{"total":96636764160,"available":96600915968,"internally_reserved":0,"allocated":8171520,"data_stored":7746857,"data_compressed":34276,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1,2,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280544,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90972,"kb_used_data":54212,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94280868,"statfs":{"total":96636764160,"available":96543608832,"internally_reserved":0,"allocated":55513088,"data_stored":56739541,"data_compressed":1505705,"data_compressed_allocated":1617920,"data_compressed_original":3477504,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280543,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94780,"kb_used_data":63596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94277060,"statfs":{"total":96636764160,"available":96539709440,"internally_reserved":0,"allocated":65122304,"data_stored":67341692,"data_compressed":2317788,"data_compressed_allocated":2486272,"data_compressed_original":5332992,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313248,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280543,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35952,"kb_used_data":4756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335888,"statfs":{"total":96636764160,"available":96599949312,"internally_reserved":0,"allocated":4870144,"data_stored":4495691,"data_compressed":34276,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280543,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69156,"kb_used_data":36556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302684,"statfs":{"total":96636764160,"available":96565948416,"internally_reserved":0,"allocated":37433344,"data_stored":37994437,"data_compressed":1250690,"data_compressed_allocated":1331200,"data_compressed_original":2437120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280543,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44220,"kb_used_data":12116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327620,"statfs":{"total":96636764160,"available":96591482880,"internally_reserved":0,"allocated":12406784,"data_stored":12456948,"data_compressed":77699,"data_compressed_allocated":102400,"data_compressed_original":643072,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,3,4,6,7,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280544,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88784,"kb_used_data":52876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94283056,"statfs":{"total":96636764160,"available":96545849344,"internally_reserved":0,"allocated":54145024,"data_stored":53713305,"data_compressed":134118,"data_compressed_allocated":159744,"data_compressed_original":425984,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,3,4,5,6,7,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313247,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42800,"kb_used_data":11608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329040,"statfs":{"total":96636764160,"available":96592936960,"internally_reserved":0,"allocated":11886592,"data_stored":11460816,"data_compressed":57065,"data_compressed_allocated":73728,"data_compressed_original":163840,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247840,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54712,"kb_used_data":23524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94317128,"statfs":{"total":96636764160,"available":96580739072,"internally_reserved":0,"allocated":24088576,"data_stored":24556927,"data_compressed":1271709,"data_compressed_allocated":1347584,"data_compressed_original":2306048,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13398016,"data_stored":14400861,"data_compressed":814666,"data_compressed_allocated":872448,"data_compressed_original":1900544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13422592,"data_stored":14113910,"data_compressed":606276,"data_compressed_allocated":643072,"data_compressed_original":1376256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12906496,"data_stored":13784230,"data_compressed":1187486,"data_compressed_allocated":1245184,"data_compressed_original":2142208,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25546752,"data_stored":27272388,"data_compressed":1564402,"data_compressed_allocated":1667072,"data_compressed_original":3444736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38944768,"data_stored":41673249,"data_compressed":2379068,"data_compressed_allocated":2539520,"data_compressed_original":5345280,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24047616,"data_stored":25639824,"data_compressed":1378911,"data_compressed_allocated":1470464,"data_compressed_original":3117056,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12906496,"data_stored":13784230,"data_compressed":1187486,"data_compressed_allocated":1245184,"data_compressed_original":2142208,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14012416,"data_stored":15061447,"data_compressed":829441,"data_compressed_allocated":888832,"data_compressed_original":1966080,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":13844480,"data_stored":14888812,"data_compressed":1232826,"data_compressed_allocated":1302528,"data_compressed_original":2371584,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9502720,"data_stored":9446986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9904128,"data_stored":9871428,"data_compressed":49947,"data_compressed_allocated":53248,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29691904,"data_stored":29622776,"data_compressed":32337,"data_compressed_allocated":36864,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":26198016,"data_stored":26107915,"data_compressed":29754,"data_compressed_allocated":32768,"data_compressed_original":86016,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":14409728,"data_stored":14413518,"data_compressed":44673,"data_compressed_allocated":49152,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3657728,"data_stored":3629400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":22847488,"data_stored":22795540,"data_compressed":9548,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":11096064,"data_stored":11035135,"data_compressed":10762,"data_compressed_allocated":12288,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49242112,"data_stored":49209022,"data_compressed":99842,"data_compressed_allocated":110592,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":10862592,"data_stored":10841283,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":10596352,"data_stored":10564238,"data_compressed":22789,"data_compressed_allocated":24576,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5840896,"data_stored":5807809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":29212672,"data_stored":29129165,"data_compressed":79701,"data_compressed_allocated":86016,"data_compressed_original":151552,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:39:17.416 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T03:39:17.465 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:17.465 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:17.471 INFO:tasks.rados.rados.0.smithi157.stdout:1773: expect (ObjNum 331 snap 0 seq_num 331) 2024-09-16T03:39:17.471 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:17.474 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:39:17.475 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:17.472+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:17.524 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:17.575 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:17.573+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:17.599 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2462563 2024-09-16T03:39:17.618 INFO:tasks.rados.rados.0.smithi157.stdout:1767: expect (ObjNum 367 snap 0 seq_num 367) 2024-09-16T03:39:17.623 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:17.623 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:17.626 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1062519 2024-09-16T03:39:17.632 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:39:17.676 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:17.675+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:17.711 INFO:tasks.rados.rados.0.smithi157.stdout:1775: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-16T03:39:17.769 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:17.775 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:17.775 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:17.777 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:17.776+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:17.783 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:39:17.810 INFO:tasks.rados.rados.0.smithi157.stdout:1768: got (2) No such file or directory 2024-09-16T03:39:17.810 INFO:tasks.rados.rados.0.smithi157.stdout:1768: got expected ENOENT (src dne) 2024-09-16T03:39:17.810 INFO:tasks.rados.rados.0.smithi157.stdout:1778: finishing write tid 1 to smithi15736696-35 2024-09-16T03:39:17.810 INFO:tasks.rados.rados.0.smithi157.stdout:1778: finishing write tid 2 to smithi15736696-35 2024-09-16T03:39:17.811 INFO:tasks.rados.rados.0.smithi157.stdout:1778: finishing write tid 3 to smithi15736696-35 2024-09-16T03:39:17.811 INFO:tasks.rados.rados.0.smithi157.stdout:1778: finishing write tid 4 to smithi15736696-35 2024-09-16T03:39:17.811 INFO:tasks.rados.rados.0.smithi157.stdout:1778: oid 35 updating version 0 to 127 2024-09-16T03:39:17.811 INFO:tasks.rados.rados.0.smithi157.stdout:1778: oid 35 version 127 is already newer than 126 2024-09-16T03:39:17.811 INFO:tasks.rados.rados.0.smithi157.stdout:1778: oid 35 version 127 is already newer than 125 2024-09-16T03:39:17.811 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 127 (ObjNum 392 snap 0 seq_num 392) dirty exists 2024-09-16T03:39:17.812 INFO:tasks.rados.rados.0.smithi157.stdout:1778: left oid 35 (ObjNum 392 snap 0 seq_num 392) 2024-09-16T03:39:17.812 INFO:tasks.rados.rados.0.smithi157.stdout:1771: got (0) Success 2024-09-16T03:39:17.812 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 140 (ObjNum 337 snap 0 seq_num 337) dirty exists 2024-09-16T03:39:17.812 INFO:tasks.rados.rados.0.smithi157.stdout:1764: done (14 left) 2024-09-16T03:39:17.812 INFO:tasks.rados.rados.0.smithi157.stdout:1766: done (13 left) 2024-09-16T03:39:17.812 INFO:tasks.rados.rados.0.smithi157.stdout:1767: done (12 left) 2024-09-16T03:39:17.812 INFO:tasks.rados.rados.0.smithi157.stdout:1768: done (11 left) 2024-09-16T03:39:17.812 INFO:tasks.rados.rados.0.smithi157.stdout:1769: done (10 left) 2024-09-16T03:39:17.812 INFO:tasks.rados.rados.0.smithi157.stdout:1770: done (9 left) 2024-09-16T03:39:17.812 INFO:tasks.rados.rados.0.smithi157.stdout:1771: done (8 left) 2024-09-16T03:39:17.812 INFO:tasks.rados.rados.0.smithi157.stdout:1772: done (7 left) 2024-09-16T03:39:17.813 INFO:tasks.rados.rados.0.smithi157.stdout:1773: done (6 left) 2024-09-16T03:39:17.813 INFO:tasks.rados.rados.0.smithi157.stdout:1774: done (5 left) 2024-09-16T03:39:17.813 INFO:tasks.rados.rados.0.smithi157.stdout:1775: done (4 left) 2024-09-16T03:39:17.813 INFO:tasks.rados.rados.0.smithi157.stdout:1776: done (3 left) 2024-09-16T03:39:17.813 INFO:tasks.rados.rados.0.smithi157.stdout:1778: done (2 left) 2024-09-16T03:39:17.813 INFO:tasks.rados.rados.0.smithi157.stdout:1779: done (1 left) 2024-09-16T03:39:17.813 INFO:tasks.rados.rados.0.smithi157.stdout:1780: done (0 left) 2024-09-16T03:39:17.813 INFO:tasks.rados.rados.0.smithi157.stdout:1781: read oid 32 snap -1 2024-09-16T03:39:17.813 INFO:tasks.rados.rados.0.smithi157.stdout:1781: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-16T03:39:17.813 INFO:tasks.rados.rados.0.smithi157.stdout:1782: read oid 40 snap -1 2024-09-16T03:39:17.813 INFO:tasks.rados.rados.0.smithi157.stdout:1782: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:39:17.813 INFO:tasks.rados.rados.0.smithi157.stdout:1783: set_chunk oid 34 target oid 2024-09-16T03:39:17.814 INFO:tasks.rados.rados.0.smithi157.stdout:1783: set_chunk oid 34 offset: 1708032 length: 227328 target oid cf898f6f3e098f301b9ea267a5ce2ae9da8afef304d5dd8969a2d548f0ed23a7 offset: 0 snap -1 2024-09-16T03:39:17.814 INFO:tasks.rados.rados.0.smithi157.stdout:1784: read oid 8 snap -1 2024-09-16T03:39:17.814 INFO:tasks.rados.rados.0.smithi157.stdout:1784: expect (ObjNum 359 snap 0 seq_num 359) 2024-09-16T03:39:17.815 INFO:tasks.rados.rados.0.smithi157.stdout:1785: set_chunk oid 9 target oid 2024-09-16T03:39:17.815 INFO:tasks.rados.rados.0.smithi157.stdout:1785: set_chunk oid 9 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:39:17.815 INFO:tasks.rados.rados.0.smithi157.stdout:1786: read oid 36 snap -1 2024-09-16T03:39:17.815 INFO:tasks.rados.rados.0.smithi157.stdout:1786: expect (ObjNum 306 snap 0 seq_num 306) 2024-09-16T03:39:17.815 INFO:tasks.rados.rados.0.smithi157.stdout:1787: write (excl) oid 18 current snap is 0 2024-09-16T03:39:17.815 INFO:tasks.rados.rados.0.smithi157.stdout:1787: seq_num 393 ranges {605151=706860,2031188=573963} 2024-09-16T03:39:17.820 INFO:tasks.rados.rados.0.smithi157.stdout:1787: writing smithi15736696-18 from 605151 to 1312011 tid 1 2024-09-16T03:39:17.826 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:17.826 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull","count":5},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":160,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":141,"num_osds":16,"num_up_osds":16,"osd_up_since":1726457810,"num_in_osds":13,"osd_in_since":1726457912,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":62},{"state_name":"active+remapped+backfill_toofull","count":5},{"state_name":"peering","count":1},{"state_name":"activating","count":1}],"num_pgs":69,"num_pools":4,"num_objects":140,"data_bytes":206510015,"bytes_used":832344064,"bytes_avail":1255445590016,"bytes_total":1256277934080,"inactive_pgs_ratio":0.028985507786273956,"misplaced_objects":35,"misplaced_total":420,"misplaced_ratio":0.083333333333333315,"recovering_objects_per_sec":1,"recovering_bytes_per_sec":1721278,"recovering_keys_per_sec":0,"num_objects_recovered":5,"num_bytes_recovered":7137666,"num_keys_recovered":0,"write_bytes_sec":1442387,"read_op_per_sec":0,"write_op_per_sec":5},"fsmap":{"epoch":1,"btime":"2024-09-16T03:36:37:310011+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T03:37:51.688536+0000","services":{}},"progress_events":{}} 2024-09-16T03:39:17.827 INFO:tasks.rados.rados.0.smithi157.stdout:1787: writing smithi15736696-18 from 2031188 to 2605151 tid 2 2024-09-16T03:39:17.827 INFO:tasks.rados.rados.0.smithi157.stdout:1788: write (excl) oid 10 current snap is 0 2024-09-16T03:39:17.827 INFO:tasks.rados.rados.0.smithi157.stdout:1788: seq_num 394 ranges {621958=782109,2089670=532288} 2024-09-16T03:39:17.835 INFO:tasks.rados.rados.0.smithi157.stdout:1788: writing smithi15736696-10 from 621958 to 1404067 tid 1 2024-09-16T03:39:17.840 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-16T03:39:17.840 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:39:17.842 INFO:tasks.rados.rados.0.smithi157.stdout:1788: writing smithi15736696-10 from 2089670 to 2621958 tid 2 2024-09-16T03:39:17.842 INFO:tasks.rados.rados.0.smithi157.stdout:1789: write (excl) oid 27 current snap is 0 2024-09-16T03:39:17.842 INFO:tasks.rados.rados.0.smithi157.stdout:1789: seq_num 395 ranges {638765=457358,1748152=604803} 2024-09-16T03:39:17.848 INFO:tasks.rados.rados.0.smithi157.stdout:1789: writing smithi15736696-27 from 638765 to 1096123 tid 1 2024-09-16T03:39:17.855 INFO:tasks.rados.rados.0.smithi157.stdout:1789: writing smithi15736696-27 from 1748152 to 2352955 tid 2 2024-09-16T03:39:17.855 INFO:tasks.rados.rados.0.smithi157.stdout:1790: read oid 7 snap -1 2024-09-16T03:39:17.855 INFO:tasks.rados.rados.0.smithi157.stdout:1790: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-16T03:39:17.855 INFO:tasks.rados.rados.0.smithi157.stdout:1791: delete oid 19 current snap is 0 2024-09-16T03:39:17.855 INFO:tasks.rados.rados.0.smithi157.stdout:1783: finishing set_chunk to oid 34 2024-09-16T03:39:17.855 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 34 set_chunk returned error code -95 offset: 1708032 length: 227328 2024-09-16T03:39:17.855 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 0 (ObjNum 342 snap 0 seq_num 342) dirty exists 2024-09-16T03:39:17.855 INFO:tasks.rados.rados.0.smithi157.stdout:1785: finishing set_chunk to oid 9 2024-09-16T03:39:17.855 INFO:tasks.rados.rados.0.smithi157.stdout:1785: got expected ENOENT (src dne) 2024-09-16T03:39:17.856 INFO:tasks.rados.rados.0.smithi157.stdout:1790: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-16T03:39:17.878 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:17.877+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:17.887 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 126369 2024-09-16T03:39:17.888 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:17.889 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:17.897 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:39:17.948 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1574559 2024-09-16T03:39:17.960 INFO:tasks.rados.rados.0.smithi157.stdout:1782: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:39:17.960 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:17.979 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:17.977+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:18.005 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:18.005 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:18.014 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:39:18.017 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:18.080 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:18.078+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:18.118 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:18.118 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:18.127 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:39:18.128 INFO:tasks.rados.rados.0.smithi157.stdout:1781: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-16T03:39:18.144 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:18.180 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:18.178+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:18.187 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:18.188 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:39:18.199 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-16T03:39:17.707177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":212543577,"num_objects":135,"num_object_clones":0,"num_object_copies":405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":35,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":2032,"num_read_kb":944443,"num_write":1781,"num_write_kb":389969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":69240114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":193,"ondisk_log_size":193,"up":207,"acting":207,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":220,"num_osds":13,"num_per_pool_osds":13,"num_per_pool_omap_osds":13,"kb":1226833920,"kb_used":808596,"kb_used_data":391740,"kb_used_omap":20,"kb_used_meta":405163,"kb_avail":1226025324,"statfs":{"total":1256277934080,"available":1255449931776,"internally_reserved":0,"allocated":401141760,"data_stored":407869533,"data_compressed":11312060,"data_compressed_allocated":12210176,"data_compressed_original":26005504,"omap_allocated":20735,"internal_metadata":414887681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1,2,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2200464,"num_objects":-4,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":67,"num_read_kb":21114,"num_write":44,"num_write_kb":13706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7137666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.137970"},"pg_stats":[{"pgid":"4.37","version":"131'52","reported_seq":407,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.573095+0000","last_change":"2024-09-16T03:39:08.371859+0000","last_active":"2024-09-16T03:39:13.573095+0000","last_peered":"2024-09-16T03:39:13.573095+0000","last_clean":"2024-09-16T03:39:13.573095+0000","last_became_active":"2024-09-16T03:39:05.721626+0000","last_became_peered":"2024-09-16T03:39:05.721626+0000","last_unstale":"2024-09-16T03:39:13.573095+0000","last_undegraded":"2024-09-16T03:39:13.573095+0000","last_fullsized":"2024-09-16T03:39:13.573095+0000","mapping_epoch":130,"log_start":"104'45","ondisk_log_start":"104'45","created":126,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":44,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:36.963537+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2890887,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":44517,"num_write":12,"num_write_kb":3069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1148090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.36","version":"131'93","reported_seq":489,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.729722+0000","last_change":"2024-09-16T03:39:03.716194+0000","last_active":"2024-09-16T03:39:16.729722+0000","last_peered":"2024-09-16T03:39:16.729722+0000","last_clean":"2024-09-16T03:39:16.729722+0000","last_became_active":"2024-09-16T03:39:02.099320+0000","last_became_peered":"2024-09-16T03:39:02.099320+0000","last_unstale":"2024-09-16T03:39:16.729722+0000","last_undegraded":"2024-09-16T03:39:16.729722+0000","last_fullsized":"2024-09-16T03:39:16.729722+0000","mapping_epoch":126,"log_start":"130'90","ondisk_log_start":"130'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.716125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.716125+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":88,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.328700+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269209,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":39,"num_read_kb":28771,"num_write":8,"num_write_kb":2101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.35","version":"139'67","reported_seq":487,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.973991+0000","last_change":"2024-09-16T03:39:13.972817+0000","last_active":"2024-09-16T03:39:13.973991+0000","last_peered":"2024-09-16T03:39:13.973991+0000","last_clean":"2024-09-16T03:39:13.973991+0000","last_became_active":"2024-09-16T03:39:13.773139+0000","last_became_peered":"2024-09-16T03:39:13.773139+0000","last_unstale":"2024-09-16T03:39:13.973991+0000","last_undegraded":"2024-09-16T03:39:13.973991+0000","last_fullsized":"2024-09-16T03:39:13.973991+0000","mapping_epoch":138,"log_start":"123'64","ondisk_log_start":"123'64","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"60'33","last_scrub_stamp":"2024-09-16T03:37:47.078574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.078574+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":62,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916015,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84364,"num_write":19,"num_write_kb":5459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5435753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"104'20","reported_seq":338,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:15.312712+0000","last_change":"2024-09-16T03:39:02.919356+0000","last_active":"2024-09-16T03:39:15.312712+0000","last_peered":"2024-09-16T03:39:15.312712+0000","last_clean":"2024-09-16T03:39:15.312712+0000","last_became_active":"2024-09-16T03:39:01.699286+0000","last_became_peered":"2024-09-16T03:39:01.699286+0000","last_unstale":"2024-09-16T03:39:15.312712+0000","last_undegraded":"2024-09-16T03:39:15.312712+0000","last_fullsized":"2024-09-16T03:39:15.312712+0000","mapping_epoch":126,"log_start":"98'17","ondisk_log_start":"98'17","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:02.919259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.919259+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.209096+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2705960,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":11460,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"137'75","reported_seq":497,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.579280+0000","last_change":"2024-09-16T03:39:13.974391+0000","last_active":"2024-09-16T03:39:14.579280+0000","last_peered":"2024-09-16T03:39:14.579280+0000","last_clean":"2024-09-16T03:39:14.579280+0000","last_became_active":"2024-09-16T03:39:13.873951+0000","last_became_peered":"2024-09-16T03:39:13.873951+0000","last_unstale":"2024-09-16T03:39:14.579280+0000","last_undegraded":"2024-09-16T03:39:14.579280+0000","last_fullsized":"2024-09-16T03:39:14.579280+0000","mapping_epoch":138,"log_start":"108'70","ondisk_log_start":"108'70","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'72","last_scrub_stamp":"2024-09-16T03:39:02.143339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.143339+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":68,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.465359+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353046,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":11293,"num_write":4,"num_write_kb":955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":994759,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.32","version":"106'17","reported_seq":391,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.075288+0000","last_change":"2024-09-16T03:39:14.075213+0000","last_active":"2024-09-16T03:39:14.075288+0000","last_peered":"2024-09-16T03:39:14.075288+0000","last_clean":"2024-09-16T03:39:14.075288+0000","last_became_active":"2024-09-16T03:39:14.074797+0000","last_became_peered":"2024-09-16T03:39:14.074797+0000","last_unstale":"2024-09-16T03:39:14.075288+0000","last_undegraded":"2024-09-16T03:39:14.075288+0000","last_fullsized":"2024-09-16T03:39:14.075288+0000","mapping_epoch":138,"log_start":"82'13","ondisk_log_start":"82'13","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"106'17","last_scrub_stamp":"2024-09-16T03:39:12.108317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:12.108317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:16.313362+0000","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.31","version":"114'17","reported_seq":305,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.723994+0000","last_change":"2024-09-16T03:39:05.622532+0000","last_active":"2024-09-16T03:39:14.723994+0000","last_peered":"2024-09-16T03:39:14.723994+0000","last_clean":"2024-09-16T03:39:14.723994+0000","last_became_active":"2024-09-16T03:39:01.997491+0000","last_became_peered":"2024-09-16T03:39:01.997491+0000","last_unstale":"2024-09-16T03:39:14.723994+0000","last_undegraded":"2024-09-16T03:39:14.723994+0000","last_fullsized":"2024-09-16T03:39:14.723994+0000","mapping_epoch":126,"log_start":"107'14","ondisk_log_start":"107'14","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'9","last_scrub_stamp":"2024-09-16T03:37:50.756120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:50.756120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.630555+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1495840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":28490,"num_write":4,"num_write_kb":1066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"118'64","reported_seq":402,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.570873+0000","last_change":"2024-09-16T03:39:05.621064+0000","last_active":"2024-09-16T03:39:13.570873+0000","last_peered":"2024-09-16T03:39:13.570873+0000","last_clean":"2024-09-16T03:39:13.570873+0000","last_became_active":"2024-09-16T03:39:01.901037+0000","last_became_peered":"2024-09-16T03:39:01.901037+0000","last_unstale":"2024-09-16T03:39:13.570873+0000","last_undegraded":"2024-09-16T03:39:13.570873+0000","last_fullsized":"2024-09-16T03:39:13.570873+0000","mapping_epoch":126,"log_start":"113'61","ondisk_log_start":"113'61","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:35.157846+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1680717,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":23268,"num_write":10,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":918091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2f","version":"113'94","reported_seq":472,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.571296+0000","last_change":"2024-09-16T03:39:03.319075+0000","last_active":"2024-09-16T03:39:13.571296+0000","last_peered":"2024-09-16T03:39:13.571296+0000","last_clean":"2024-09-16T03:39:13.571296+0000","last_became_active":"2024-09-16T03:39:01.700988+0000","last_became_peered":"2024-09-16T03:39:01.700988+0000","last_unstale":"2024-09-16T03:39:13.571296+0000","last_undegraded":"2024-09-16T03:39:13.571296+0000","last_fullsized":"2024-09-16T03:39:13.571296+0000","mapping_epoch":126,"log_start":"107'90","ondisk_log_start":"107'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"113'94","last_scrub_stamp":"2024-09-16T03:39:03.319012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.319012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":87,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:13.773817+0000","scrub_duration":1038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2e","version":"123'75","reported_seq":457,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.679398+0000","last_change":"2024-09-16T03:39:04.235234+0000","last_active":"2024-09-16T03:39:14.679398+0000","last_peered":"2024-09-16T03:39:14.679398+0000","last_clean":"2024-09-16T03:39:14.679398+0000","last_became_active":"2024-09-16T03:39:01.999291+0000","last_became_peered":"2024-09-16T03:39:01.999291+0000","last_unstale":"2024-09-16T03:39:14.679398+0000","last_undegraded":"2024-09-16T03:39:14.679398+0000","last_fullsized":"2024-09-16T03:39:14.679398+0000","mapping_epoch":126,"log_start":"114'72","ondisk_log_start":"114'72","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"123'75","last_scrub_stamp":"2024-09-16T03:39:04.235141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.235141+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":72,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.010761+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":13153,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"131'99","reported_seq":482,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.902179+0000","last_change":"2024-09-16T03:39:03.917422+0000","last_active":"2024-09-16T03:39:13.902179+0000","last_peered":"2024-09-16T03:39:13.902179+0000","last_clean":"2024-09-16T03:39:13.902179+0000","last_became_active":"2024-09-16T03:39:01.801112+0000","last_became_peered":"2024-09-16T03:39:01.801112+0000","last_unstale":"2024-09-16T03:39:13.902179+0000","last_undegraded":"2024-09-16T03:39:13.902179+0000","last_fullsized":"2024-09-16T03:39:13.902179+0000","mapping_epoch":32,"log_start":"128'96","ondisk_log_start":"128'96","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'97","last_scrub_stamp":"2024-09-16T03:39:03.917355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.917355+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":93,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.686164+0000","scrub_duration":771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2033713,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8316,"num_write":1,"num_write_kb":214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.e","version":"138'82","reported_seq":457,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.579549+0000","last_change":"2024-09-16T03:39:05.725455+0000","last_active":"2024-09-16T03:39:14.579549+0000","last_peered":"2024-09-16T03:39:14.579549+0000","last_clean":"2024-09-16T03:39:14.579549+0000","last_became_active":"2024-09-16T03:39:01.999282+0000","last_became_peered":"2024-09-16T03:39:01.999282+0000","last_unstale":"2024-09-16T03:39:14.579549+0000","last_undegraded":"2024-09-16T03:39:14.579549+0000","last_fullsized":"2024-09-16T03:39:14.579549+0000","mapping_epoch":32,"log_start":"127'79","ondisk_log_start":"127'79","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:39.194237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.194237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":76,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.946501+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2159765,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":28931,"num_write":14,"num_write_kb":4613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"124'19","reported_seq":359,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.671128+0000","last_change":"2024-09-16T03:39:01.149616+0000","last_active":"2024-09-16T03:39:13.671128+0000","last_peered":"2024-09-16T03:39:13.671128+0000","last_clean":"2024-09-16T03:39:13.671128+0000","last_became_active":"2024-09-16T03:38:45.622965+0000","last_became_peered":"2024-09-16T03:38:45.622965+0000","last_unstale":"2024-09-16T03:39:13.671128+0000","last_undegraded":"2024-09-16T03:39:13.671128+0000","last_fullsized":"2024-09-16T03:39:13.671128+0000","mapping_epoch":72,"log_start":"118'16","ondisk_log_start":"118'16","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'19","last_scrub_stamp":"2024-09-16T03:39:01.149486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.149486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":16,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.574017+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3613538,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":3908,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"130'65","reported_seq":442,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.777397+0000","last_change":"2024-09-16T03:39:13.775457+0000","last_active":"2024-09-16T03:39:13.777397+0000","last_peered":"2024-09-16T03:39:13.777397+0000","last_clean":"2024-09-16T03:39:13.777397+0000","last_became_active":"2024-09-16T03:39:13.775138+0000","last_became_peered":"2024-09-16T03:39:13.775138+0000","last_unstale":"2024-09-16T03:39:13.777397+0000","last_undegraded":"2024-09-16T03:39:13.777397+0000","last_fullsized":"2024-09-16T03:39:13.777397+0000","mapping_epoch":138,"log_start":"122'62","ondisk_log_start":"122'62","created":32,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'57","last_scrub_stamp":"2024-09-16T03:38:48.923184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.923184+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:53.322016+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":7446,"num_write":9,"num_write_kb":2888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1213217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"140'133","reported_seq":614,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.706028+0000","last_change":"2024-09-16T03:39:09.748835+0000","last_active":"2024-09-16T03:39:16.706028+0000","last_peered":"2024-09-16T03:39:16.706028+0000","last_clean":"2024-09-16T03:39:16.706028+0000","last_became_active":"2024-09-16T03:39:09.748391+0000","last_became_peered":"2024-09-16T03:39:09.748391+0000","last_unstale":"2024-09-16T03:39:16.706028+0000","last_undegraded":"2024-09-16T03:39:16.706028+0000","last_fullsized":"2024-09-16T03:39:16.706028+0000","mapping_epoch":134,"log_start":"129'130","ondisk_log_start":"129'130","created":32,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:46.855462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.855462+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":126,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:12.773343+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2084134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":17019,"num_write":31,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.a","version":"128'148","reported_seq":695,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.981860+0000","last_change":"2024-09-16T03:39:01.504751+0000","last_active":"2024-09-16T03:39:13.981860+0000","last_peered":"2024-09-16T03:39:13.981860+0000","last_clean":"2024-09-16T03:39:13.981860+0000","last_became_active":"2024-09-16T03:38:46.021805+0000","last_became_peered":"2024-09-16T03:38:46.021805+0000","last_unstale":"2024-09-16T03:39:13.981860+0000","last_undegraded":"2024-09-16T03:39:13.981860+0000","last_fullsized":"2024-09-16T03:39:13.981860+0000","mapping_epoch":32,"log_start":"123'145","ondisk_log_start":"123'145","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'147","last_scrub_stamp":"2024-09-16T03:39:01.504663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.504663+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":142,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:18.060284+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13949942,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":53,"num_read_kb":53989,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":258,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.573212+0000","last_change":"2024-09-16T03:38:46.881357+0000","last_active":"2024-09-16T03:39:13.573212+0000","last_peered":"2024-09-16T03:39:13.573212+0000","last_clean":"2024-09-16T03:39:13.573212+0000","last_became_active":"2024-09-16T03:38:45.728494+0000","last_became_peered":"2024-09-16T03:38:45.728494+0000","last_unstale":"2024-09-16T03:39:13.573212+0000","last_undegraded":"2024-09-16T03:39:13.573212+0000","last_fullsized":"2024-09-16T03:39:13.573212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:46.881248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.881248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:04.991122+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.8","version":"138'77","reported_seq":478,"reported_epoch":139,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:14.180135+0000","last_change":"2024-09-16T03:39:11.576416+0000","last_active":"2024-09-16T03:39:14.180135+0000","last_peered":"2024-09-16T03:39:14.180135+0000","last_clean":"2024-09-16T03:39:08.372137+0000","last_became_active":"2024-09-16T03:39:11.177629+0000","last_became_peered":"2024-09-16T03:39:11.177629+0000","last_unstale":"2024-09-16T03:39:14.180135+0000","last_undegraded":"2024-09-16T03:39:14.180135+0000","last_fullsized":"2024-09-16T03:39:14.180135+0000","mapping_epoch":135,"log_start":"130'74","ondisk_log_start":"130'74","created":32,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'71","last_scrub_stamp":"2024-09-16T03:39:02.002247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.002247+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":72,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2403467,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":2299,"num_write":8,"num_write_kb":1852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,10],"avail_no_missing":["0","4","10"],"object_location_counts":[{"shards":"0,4,10","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"138'88","reported_seq":452,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.706103+0000","last_change":"2024-09-16T03:39:00.900732+0000","last_active":"2024-09-16T03:39:16.706103+0000","last_peered":"2024-09-16T03:39:16.706103+0000","last_clean":"2024-09-16T03:39:16.706103+0000","last_became_active":"2024-09-16T03:38:45.523605+0000","last_became_peered":"2024-09-16T03:38:45.523605+0000","last_unstale":"2024-09-16T03:39:16.706103+0000","last_undegraded":"2024-09-16T03:39:16.706103+0000","last_fullsized":"2024-09-16T03:39:16.706103+0000","mapping_epoch":32,"log_start":"130'85","ondisk_log_start":"130'85","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'82","last_scrub_stamp":"2024-09-16T03:39:00.900619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.900619+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":83,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.385837+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2420274,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":5790,"num_write":8,"num_write_kb":1825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":437,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.704186+0000","last_change":"2024-09-16T03:38:36.757156+0000","last_active":"2024-09-16T03:39:16.704186+0000","last_peered":"2024-09-16T03:39:16.704186+0000","last_clean":"2024-09-16T03:39:16.704186+0000","last_became_active":"2024-09-16T03:38:36.756855+0000","last_became_peered":"2024-09-16T03:38:36.756855+0000","last_unstale":"2024-09-16T03:39:16.704186+0000","last_undegraded":"2024-09-16T03:39:16.704186+0000","last_fullsized":"2024-09-16T03:39:16.704186+0000","mapping_epoch":103,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:38:28.063690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:38:28.063690+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:28.682469+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":136,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.772525+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:39:13.772525+0000","last_peered":"2024-09-16T03:39:13.772525+0000","last_clean":"2024-09-16T03:39:13.772525+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:39:13.772525+0000","last_undegraded":"2024-09-16T03:39:13.772525+0000","last_fullsized":"2024-09-16T03:39:13.772525+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"140'265","reported_seq":586,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.704408+0000","last_change":"2024-09-16T03:38:01.603352+0000","last_active":"2024-09-16T03:39:16.704408+0000","last_peered":"2024-09-16T03:39:16.704408+0000","last_clean":"2024-09-16T03:39:16.704408+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:39:16.704408+0000","last_undegraded":"2024-09-16T03:39:16.704408+0000","last_fullsized":"2024-09-16T03:39:16.704408+0000","mapping_epoch":74,"log_start":"137'262","ondisk_log_start":"137'262","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":271,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.902901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21682013,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":254,"num_read_kb":8762,"num_write":329,"num_write_kb":70574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.6","version":"108'44","reported_seq":392,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.671865+0000","last_change":"2024-09-16T03:38:48.279289+0000","last_active":"2024-09-16T03:39:13.671865+0000","last_peered":"2024-09-16T03:39:13.671865+0000","last_clean":"2024-09-16T03:39:13.671865+0000","last_became_active":"2024-09-16T03:38:45.924286+0000","last_became_peered":"2024-09-16T03:38:45.924286+0000","last_unstale":"2024-09-16T03:39:13.671865+0000","last_undegraded":"2024-09-16T03:39:13.671865+0000","last_fullsized":"2024-09-16T03:39:13.671865+0000","mapping_epoch":70,"log_start":"105'41","ondisk_log_start":"105'41","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:48.279224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.279224+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.718165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":276,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.782710+0000","last_change":"2024-09-16T03:39:09.749531+0000","last_active":"2024-09-16T03:39:13.782710+0000","last_peered":"2024-09-16T03:39:13.782710+0000","last_clean":"2024-09-16T03:39:13.782710+0000","last_became_active":"2024-09-16T03:39:09.649028+0000","last_became_peered":"2024-09-16T03:39:09.649028+0000","last_unstale":"2024-09-16T03:39:13.782710+0000","last_undegraded":"2024-09-16T03:39:13.782710+0000","last_fullsized":"2024-09-16T03:39:13.782710+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:24.358339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:24.358339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:34.729230+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"140'228","reported_seq":479,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.682092+0000","last_change":"2024-09-16T03:38:15.890094+0000","last_active":"2024-09-16T03:39:16.682092+0000","last_peered":"2024-09-16T03:39:16.682092+0000","last_clean":"2024-09-16T03:39:16.682092+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:39:16.682092+0000","last_undegraded":"2024-09-16T03:39:16.682092+0000","last_fullsized":"2024-09-16T03:39:16.682092+0000","mapping_epoch":31,"log_start":"138'225","ondisk_log_start":"138'225","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'125","last_scrub_stamp":"2024-09-16T03:38:15.889996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.889996+0000","objects_scrubbed":18,"log_size":3,"log_dups_size":236,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:35.350472+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22211125,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":199,"num_read_kb":2083,"num_write":285,"num_write_kb":65350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"139'77","reported_seq":434,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.176785+0000","last_change":"2024-09-16T03:39:01.297447+0000","last_active":"2024-09-16T03:39:14.176785+0000","last_peered":"2024-09-16T03:39:14.176785+0000","last_clean":"2024-09-16T03:39:14.176785+0000","last_became_active":"2024-09-16T03:38:45.723754+0000","last_became_peered":"2024-09-16T03:38:45.723754+0000","last_unstale":"2024-09-16T03:39:14.176785+0000","last_undegraded":"2024-09-16T03:39:14.176785+0000","last_fullsized":"2024-09-16T03:39:14.176785+0000","mapping_epoch":32,"log_start":"127'74","ondisk_log_start":"127'74","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'72","last_scrub_stamp":"2024-09-16T03:39:01.297324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.297324+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":72,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.355718+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7294304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":16354,"num_write":5,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":252,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.705946+0000","last_change":"2024-09-16T03:38:15.949627+0000","last_active":"2024-09-16T03:39:16.705946+0000","last_peered":"2024-09-16T03:39:16.705946+0000","last_clean":"2024-09-16T03:39:16.705946+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:39:16.705946+0000","last_undegraded":"2024-09-16T03:39:16.705946+0000","last_fullsized":"2024-09-16T03:39:16.705946+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:15.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:24.286324+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"137'256","reported_seq":650,"reported_epoch":139,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:13.772977+0000","last_change":"2024-09-16T03:39:11.578535+0000","last_active":"2024-09-16T03:39:13.772977+0000","last_peered":"2024-09-16T03:39:13.772977+0000","last_clean":"2024-09-16T03:39:08.210670+0000","last_became_active":"2024-09-16T03:39:11.274646+0000","last_became_peered":"2024-09-16T03:39:11.274646+0000","last_unstale":"2024-09-16T03:39:13.772977+0000","last_undegraded":"2024-09-16T03:39:13.772977+0000","last_fullsized":"2024-09-16T03:39:13.772977+0000","mapping_epoch":135,"log_start":"133'253","ondisk_log_start":"133'253","created":31,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":274,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19810959,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":233,"num_read_kb":12818,"num_write":315,"num_write_kb":80968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":27250603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[3,4,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":18}],"blocked_by":[],"up_primary":9,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"137'33","reported_seq":348,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.704550+0000","last_change":"2024-09-16T03:38:47.042020+0000","last_active":"2024-09-16T03:39:16.704550+0000","last_peered":"2024-09-16T03:39:16.704550+0000","last_clean":"2024-09-16T03:39:16.704550+0000","last_became_active":"2024-09-16T03:38:45.820169+0000","last_became_peered":"2024-09-16T03:38:45.820169+0000","last_unstale":"2024-09-16T03:39:16.704550+0000","last_undegraded":"2024-09-16T03:39:16.704550+0000","last_fullsized":"2024-09-16T03:39:16.704550+0000","mapping_epoch":32,"log_start":"87'30","ondisk_log_start":"87'30","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.041851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.041851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":30,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.311785+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":260,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.082589+0000","last_change":"2024-09-16T03:39:14.082589+0000","last_active":"2024-09-16T03:39:14.082589+0000","last_peered":"2024-09-16T03:39:14.082589+0000","last_clean":"2024-09-16T03:39:14.082589+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:39:14.082589+0000","last_undegraded":"2024-09-16T03:39:14.082589+0000","last_fullsized":"2024-09-16T03:39:14.082589+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:39:14.082477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.082477+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.989780+0000","scrub_duration":733,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"139'283","reported_seq":536,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.806157+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:39:16.806157+0000","last_peered":"2024-09-16T03:39:16.806157+0000","last_clean":"2024-09-16T03:39:16.806157+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:39:16.806157+0000","last_undegraded":"2024-09-16T03:39:16.806157+0000","last_fullsized":"2024-09-16T03:39:16.806157+0000","mapping_epoch":31,"log_start":"137'280","ondisk_log_start":"137'280","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":292,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.679811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22933955,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":256,"num_read_kb":6273,"num_write":352,"num_write_kb":73742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":286,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.806053+0000","last_change":"2024-09-16T03:38:35.828277+0000","last_active":"2024-09-16T03:39:16.806053+0000","last_peered":"2024-09-16T03:39:16.806053+0000","last_clean":"2024-09-16T03:39:16.806053+0000","last_became_active":"2024-09-16T03:38:33.778361+0000","last_became_peered":"2024-09-16T03:38:33.778361+0000","last_unstale":"2024-09-16T03:39:16.806053+0000","last_undegraded":"2024-09-16T03:39:16.806053+0000","last_fullsized":"2024-09-16T03:39:16.806053+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:35.828203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.828203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:49.130136+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":255,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.682145+0000","last_change":"2024-09-16T03:37:52.652365+0000","last_active":"2024-09-16T03:39:16.682145+0000","last_peered":"2024-09-16T03:39:16.682145+0000","last_clean":"2024-09-16T03:39:16.682145+0000","last_became_active":"2024-09-16T03:37:52.651907+0000","last_became_peered":"2024-09-16T03:37:52.651907+0000","last_unstale":"2024-09-16T03:39:16.682145+0000","last_undegraded":"2024-09-16T03:39:16.682145+0000","last_fullsized":"2024-09-16T03:39:16.682145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.985111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"94'34","reported_seq":357,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.704280+0000","last_change":"2024-09-16T03:38:39.081942+0000","last_active":"2024-09-16T03:39:16.704280+0000","last_peered":"2024-09-16T03:39:16.704280+0000","last_clean":"2024-09-16T03:39:16.704280+0000","last_became_active":"2024-09-16T03:38:33.580022+0000","last_became_peered":"2024-09-16T03:38:33.580022+0000","last_unstale":"2024-09-16T03:39:16.704280+0000","last_undegraded":"2024-09-16T03:39:16.704280+0000","last_fullsized":"2024-09-16T03:39:16.704280+0000","mapping_epoch":96,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:39.081810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.081810+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:48.167935+0000","scrub_duration":473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.582720+0000","last_change":"2024-09-16T03:39:09.848905+0000","last_active":"2024-09-16T03:39:14.582720+0000","last_peered":"2024-09-16T03:39:14.582720+0000","last_clean":"2024-09-16T03:39:14.582720+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:39:14.582720+0000","last_undegraded":"2024-09-16T03:39:14.582720+0000","last_fullsized":"2024-09-16T03:39:14.582720+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:48.949444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"139'133","reported_seq":582,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.003402+0000","last_change":"2024-09-16T03:38:39.704994+0000","last_active":"2024-09-16T03:39:14.003402+0000","last_peered":"2024-09-16T03:39:14.003402+0000","last_clean":"2024-09-16T03:39:14.003402+0000","last_became_active":"2024-09-16T03:38:33.581141+0000","last_became_peered":"2024-09-16T03:38:33.581141+0000","last_unstale":"2024-09-16T03:39:14.003402+0000","last_undegraded":"2024-09-16T03:39:14.003402+0000","last_fullsized":"2024-09-16T03:39:14.003402+0000","mapping_epoch":32,"log_start":"129'130","ondisk_log_start":"129'130","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'118","last_scrub_stamp":"2024-09-16T03:38:39.704919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.704919+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":127,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.098482+0000","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4554829,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":62,"num_read_kb":77035,"num_write":18,"num_write_kb":5604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":258,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.573121+0000","last_change":"2024-09-16T03:38:12.041048+0000","last_active":"2024-09-16T03:39:13.573121+0000","last_peered":"2024-09-16T03:39:13.573121+0000","last_clean":"2024-09-16T03:39:13.573121+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:39:13.573121+0000","last_undegraded":"2024-09-16T03:39:13.573121+0000","last_fullsized":"2024-09-16T03:39:13.573121+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:12.040917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.040917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.877949+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":270,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.679338+0000","last_change":"2024-09-16T03:38:34.928371+0000","last_active":"2024-09-16T03:39:14.679338+0000","last_peered":"2024-09-16T03:39:14.679338+0000","last_clean":"2024-09-16T03:39:14.679338+0000","last_became_active":"2024-09-16T03:38:33.484091+0000","last_became_peered":"2024-09-16T03:38:33.484091+0000","last_unstale":"2024-09-16T03:39:14.679338+0000","last_undegraded":"2024-09-16T03:39:14.679338+0000","last_fullsized":"2024-09-16T03:39:14.679338+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:34.928298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.928298+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:43.935101+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":251,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.782769+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:39:13.782769+0000","last_peered":"2024-09-16T03:39:13.782769+0000","last_clean":"2024-09-16T03:39:13.782769+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:39:13.782769+0000","last_undegraded":"2024-09-16T03:39:13.782769+0000","last_fullsized":"2024-09-16T03:39:13.782769+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.232558+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.10","version":"118'64","reported_seq":409,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.571104+0000","last_change":"2024-09-16T03:39:04.442655+0000","last_active":"2024-09-16T03:39:13.571104+0000","last_peered":"2024-09-16T03:39:13.571104+0000","last_clean":"2024-09-16T03:39:13.571104+0000","last_became_active":"2024-09-16T03:39:01.900921+0000","last_became_peered":"2024-09-16T03:39:01.900921+0000","last_unstale":"2024-09-16T03:39:13.571104+0000","last_undegraded":"2024-09-16T03:39:13.571104+0000","last_fullsized":"2024-09-16T03:39:13.571104+0000","mapping_epoch":72,"log_start":"113'61","ondisk_log_start":"113'61","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"118'64","last_scrub_stamp":"2024-09-16T03:39:04.442556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.442556+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.917777+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3361433,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"114'17","reported_seq":317,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.724049+0000","last_change":"2024-09-16T03:39:03.929961+0000","last_active":"2024-09-16T03:39:14.724049+0000","last_peered":"2024-09-16T03:39:14.724049+0000","last_clean":"2024-09-16T03:39:14.724049+0000","last_became_active":"2024-09-16T03:39:01.696690+0000","last_became_peered":"2024-09-16T03:39:01.696690+0000","last_unstale":"2024-09-16T03:39:14.724049+0000","last_undegraded":"2024-09-16T03:39:14.724049+0000","last_fullsized":"2024-09-16T03:39:14.724049+0000","mapping_epoch":63,"log_start":"107'14","ondisk_log_start":"107'14","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:03.929863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.929863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:26.527714+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2991679,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":7626,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"127'20","reported_seq":397,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.974721+0000","last_change":"2024-09-16T03:39:13.974599+0000","last_active":"2024-09-16T03:39:13.974721+0000","last_peered":"2024-09-16T03:39:13.974721+0000","last_clean":"2024-09-16T03:39:13.974721+0000","last_became_active":"2024-09-16T03:39:13.974170+0000","last_became_peered":"2024-09-16T03:39:13.974170+0000","last_unstale":"2024-09-16T03:39:13.974721+0000","last_undegraded":"2024-09-16T03:39:13.974721+0000","last_fullsized":"2024-09-16T03:39:13.974721+0000","mapping_epoch":138,"log_start":"106'17","ondisk_log_start":"106'17","created":50,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:37:34.761920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:34.761920+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966452,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":26744,"num_write":10,"num_write_kb":2629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3741138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.13","version":"127'73","reported_seq":480,"reported_epoch":140,"state":"activating","last_fresh":"2024-09-16T03:39:14.579589+0000","last_change":"2024-09-16T03:39:14.579589+0000","last_active":"2024-09-16T03:39:13.573601+0000","last_peered":"2024-09-16T03:39:12.672413+0000","last_clean":"2024-09-16T03:39:12.672413+0000","last_became_active":"2024-09-16T03:39:11.176806+0000","last_became_peered":"2024-09-16T03:39:11.176806+0000","last_unstale":"2024-09-16T03:39:14.579589+0000","last_undegraded":"2024-09-16T03:39:14.579589+0000","last_fullsized":"2024-09-16T03:39:14.579589+0000","mapping_epoch":139,"log_start":"108'70","ondisk_log_start":"108'70","created":50,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'73","last_scrub_stamp":"2024-09-16T03:39:03.124583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.124583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":68,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":["4","3","9","14"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"104'20","reported_seq":334,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.824514+0000","last_change":"2024-09-16T03:39:01.898409+0000","last_active":"2024-09-16T03:39:14.824514+0000","last_peered":"2024-09-16T03:39:14.824514+0000","last_clean":"2024-09-16T03:39:14.824514+0000","last_became_active":"2024-09-16T03:39:01.596287+0000","last_became_peered":"2024-09-16T03:39:01.596287+0000","last_unstale":"2024-09-16T03:39:14.824514+0000","last_undegraded":"2024-09-16T03:39:14.824514+0000","last_fullsized":"2024-09-16T03:39:14.824514+0000","mapping_epoch":114,"log_start":"98'17","ondisk_log_start":"98'17","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:01.898331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.898331+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.472424+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"123'65","reported_seq":486,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.175562+0000","last_change":"2024-09-16T03:39:14.175562+0000","last_active":"2024-09-16T03:39:14.175562+0000","last_peered":"2024-09-16T03:39:14.175562+0000","last_clean":"2024-09-16T03:39:14.175562+0000","last_became_active":"2024-09-16T03:39:13.671769+0000","last_became_peered":"2024-09-16T03:39:13.671769+0000","last_unstale":"2024-09-16T03:39:14.175562+0000","last_undegraded":"2024-09-16T03:39:14.175562+0000","last_fullsized":"2024-09-16T03:39:14.175562+0000","mapping_epoch":138,"log_start":"123'62","ondisk_log_start":"123'62","created":50,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"123'65","last_scrub_stamp":"2024-09-16T03:39:14.175493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.175493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:31.666781+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"124'86","reported_seq":466,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.729748+0000","last_change":"2024-09-16T03:39:03.073530+0000","last_active":"2024-09-16T03:39:16.729748+0000","last_peered":"2024-09-16T03:39:16.729748+0000","last_clean":"2024-09-16T03:39:16.729748+0000","last_became_active":"2024-09-16T03:39:01.797805+0000","last_became_peered":"2024-09-16T03:39:01.797805+0000","last_unstale":"2024-09-16T03:39:16.729748+0000","last_undegraded":"2024-09-16T03:39:16.729748+0000","last_fullsized":"2024-09-16T03:39:16.729748+0000","mapping_epoch":115,"log_start":"124'82","ondisk_log_start":"124'82","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.073459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.073459+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":80,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:32.204501+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.17","version":"137'55","reported_seq":416,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.573387+0000","last_change":"2024-09-16T03:39:05.721286+0000","last_active":"2024-09-16T03:39:13.573387+0000","last_peered":"2024-09-16T03:39:13.573387+0000","last_clean":"2024-09-16T03:39:13.573387+0000","last_became_active":"2024-09-16T03:39:03.717420+0000","last_became_peered":"2024-09-16T03:39:03.717420+0000","last_unstale":"2024-09-16T03:39:13.573387+0000","last_undegraded":"2024-09-16T03:39:13.573387+0000","last_fullsized":"2024-09-16T03:39:13.573387+0000","mapping_epoch":128,"log_start":"131'52","ondisk_log_start":"131'52","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":51,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.790793+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1630328,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":44517,"num_write":15,"num_write_kb":4042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2516795,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"128'51","reported_seq":386,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.782591+0000","last_change":"2024-09-16T03:39:02.668794+0000","last_active":"2024-09-16T03:39:13.782591+0000","last_peered":"2024-09-16T03:39:13.782591+0000","last_clean":"2024-09-16T03:39:13.782591+0000","last_became_active":"2024-09-16T03:38:47.562641+0000","last_became_peered":"2024-09-16T03:38:47.562641+0000","last_unstale":"2024-09-16T03:39:13.782591+0000","last_undegraded":"2024-09-16T03:39:13.782591+0000","last_fullsized":"2024-09-16T03:39:13.782591+0000","mapping_epoch":112,"log_start":"124'48","ondisk_log_start":"124'48","created":50,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'48","last_scrub_stamp":"2024-09-16T03:39:02.668681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.668681+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":45,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.451476+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067327,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":25040,"num_write":23,"num_write_kb":6049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.582522+0000","last_change":"2024-09-16T03:39:09.847101+0000","last_active":"2024-09-16T03:39:14.582522+0000","last_peered":"2024-09-16T03:39:14.582522+0000","last_clean":"2024-09-16T03:39:14.582522+0000","last_became_active":"2024-09-16T03:39:09.846691+0000","last_became_peered":"2024-09-16T03:39:09.846691+0000","last_unstale":"2024-09-16T03:39:14.582522+0000","last_undegraded":"2024-09-16T03:39:14.582522+0000","last_fullsized":"2024-09-16T03:39:14.582522+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:01.106840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.106840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.495875+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"128'125","reported_seq":593,"reported_epoch":139,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:13.872959+0000","last_change":"2024-09-16T03:38:47.159152+0000","last_active":"2024-09-16T03:39:13.872959+0000","last_peered":"2024-09-16T03:39:13.872959+0000","last_clean":"2024-09-16T03:38:43.521485+0000","last_became_active":"2024-09-16T03:38:46.957471+0000","last_became_peered":"2024-09-16T03:38:46.957471+0000","last_unstale":"2024-09-16T03:39:13.872959+0000","last_undegraded":"2024-09-16T03:39:13.872959+0000","last_fullsized":"2024-09-16T03:39:13.872959+0000","mapping_epoch":111,"log_start":"117'122","ondisk_log_start":"117'122","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'115","last_scrub_stamp":"2024-09-16T03:38:35.254488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.254488+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":122,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6739673,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":31161,"num_write":9,"num_write_kb":2915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6,11],"acting":[3,8,10],"avail_no_missing":["3","8","10"],"object_location_counts":[{"shards":"3,8,10","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"138'131","reported_seq":603,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.773024+0000","last_change":"2024-09-16T03:38:50.465770+0000","last_active":"2024-09-16T03:39:13.773024+0000","last_peered":"2024-09-16T03:39:13.773024+0000","last_clean":"2024-09-16T03:39:13.773024+0000","last_became_active":"2024-09-16T03:38:50.465463+0000","last_became_peered":"2024-09-16T03:38:50.465463+0000","last_unstale":"2024-09-16T03:39:13.773024+0000","last_undegraded":"2024-09-16T03:39:13.773024+0000","last_fullsized":"2024-09-16T03:39:13.773024+0000","mapping_epoch":115,"log_start":"136'128","ondisk_log_start":"136'128","created":100,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'107","last_scrub_stamp":"2024-09-16T03:38:39.905697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.905697+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":122,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.828899+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4655671,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":83952,"num_write":28,"num_write_kb":8389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2376197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"99'56","reported_seq":391,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.682077+0000","last_change":"2024-09-16T03:39:12.670900+0000","last_active":"2024-09-16T03:39:16.682077+0000","last_peered":"2024-09-16T03:39:16.682077+0000","last_clean":"2024-09-16T03:39:16.682077+0000","last_became_active":"2024-09-16T03:39:12.670496+0000","last_became_peered":"2024-09-16T03:39:12.670496+0000","last_unstale":"2024-09-16T03:39:16.682077+0000","last_undegraded":"2024-09-16T03:39:16.682077+0000","last_fullsized":"2024-09-16T03:39:16.682077+0000","mapping_epoch":137,"log_start":"90'53","ondisk_log_start":"90'53","created":100,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"99'56","last_scrub_stamp":"2024-09-16T03:39:00.988517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.988517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:21.733247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"92'14","reported_seq":289,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.873443+0000","last_change":"2024-09-16T03:38:49.092747+0000","last_active":"2024-09-16T03:39:13.873443+0000","last_peered":"2024-09-16T03:39:13.873443+0000","last_clean":"2024-09-16T03:39:13.873443+0000","last_became_active":"2024-09-16T03:38:48.661213+0000","last_became_peered":"2024-09-16T03:38:48.661213+0000","last_unstale":"2024-09-16T03:39:13.873443+0000","last_undegraded":"2024-09-16T03:39:13.873443+0000","last_fullsized":"2024-09-16T03:39:13.873443+0000","mapping_epoch":113,"log_start":"91'11","ondisk_log_start":"91'11","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:49.092687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.092687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.372348+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,12],"acting":[3,11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"92'59","reported_seq":398,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.673168+0000","last_change":"2024-09-16T03:38:48.561603+0000","last_active":"2024-09-16T03:39:13.673168+0000","last_peered":"2024-09-16T03:39:13.673168+0000","last_clean":"2024-09-16T03:39:13.673168+0000","last_became_active":"2024-09-16T03:38:48.561170+0000","last_became_peered":"2024-09-16T03:38:48.561170+0000","last_unstale":"2024-09-16T03:39:13.673168+0000","last_undegraded":"2024-09-16T03:39:13.673168+0000","last_fullsized":"2024-09-16T03:39:13.673168+0000","mapping_epoch":113,"log_start":"89'56","ondisk_log_start":"89'56","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'59","last_scrub_stamp":"2024-09-16T03:38:35.977646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.977646+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:38.632635+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,6],"acting":[14,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1f","version":"137'119","reported_seq":519,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.773051+0000","last_change":"2024-09-16T03:38:48.762617+0000","last_active":"2024-09-16T03:39:13.773051+0000","last_peered":"2024-09-16T03:39:13.773051+0000","last_clean":"2024-09-16T03:39:13.773051+0000","last_became_active":"2024-09-16T03:38:48.761213+0000","last_became_peered":"2024-09-16T03:38:48.761213+0000","last_unstale":"2024-09-16T03:39:13.773051+0000","last_undegraded":"2024-09-16T03:39:13.773051+0000","last_fullsized":"2024-09-16T03:39:13.773051+0000","mapping_epoch":113,"log_start":"137'114","ondisk_log_start":"137'114","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'95","last_scrub_stamp":"2024-09-16T03:38:37.201959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:37.201959+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":114,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.307408+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8134753,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":48941,"num_write":35,"num_write_kb":8979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1704821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,12],"acting":[3,10,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"139'146","reported_seq":619,"reported_epoch":140,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:14.723059+0000","last_change":"2024-09-16T03:38:46.862171+0000","last_active":"2024-09-16T03:39:14.723059+0000","last_peered":"2024-09-16T03:39:14.723059+0000","last_clean":"2024-09-16T03:38:44.034266+0000","last_became_active":"2024-09-16T03:38:46.560752+0000","last_became_peered":"2024-09-16T03:38:46.560752+0000","last_unstale":"2024-09-16T03:39:14.723059+0000","last_undegraded":"2024-09-16T03:39:14.723059+0000","last_fullsized":"2024-09-16T03:39:14.723059+0000","mapping_epoch":111,"log_start":"127'143","ondisk_log_start":"127'143","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'118","last_scrub_stamp":"2024-09-16T03:38:36.698150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:36.698150+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":143,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":103,"num_read_kb":88353,"num_write":32,"num_write_kb":9785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[4,10,15],"avail_no_missing":["4","10","15"],"object_location_counts":[{"shards":"4,10,15","objects":3}],"blocked_by":[],"up_primary":8,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.21","version":"48'7","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.572953+0000","last_change":"2024-09-16T03:38:49.064947+0000","last_active":"2024-09-16T03:39:13.572953+0000","last_peered":"2024-09-16T03:39:13.572953+0000","last_clean":"2024-09-16T03:39:13.572953+0000","last_became_active":"2024-09-16T03:38:48.559431+0000","last_became_peered":"2024-09-16T03:38:48.559431+0000","last_unstale":"2024-09-16T03:39:13.572953+0000","last_undegraded":"2024-09-16T03:39:13.572953+0000","last_fullsized":"2024-09-16T03:39:13.572953+0000","mapping_epoch":113,"log_start":"40'4","ondisk_log_start":"40'4","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":6,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:49.064870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064870+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.310170+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,10],"acting":[14,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.22","version":"45'6","reported_seq":156,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.582720+0000","last_change":"2024-09-16T03:39:13.974174+0000","last_active":"2024-09-16T03:39:14.582720+0000","last_peered":"2024-09-16T03:39:14.582720+0000","last_clean":"2024-09-16T03:39:14.582720+0000","last_became_active":"2024-09-16T03:39:13.973813+0000","last_became_peered":"2024-09-16T03:39:13.973813+0000","last_unstale":"2024-09-16T03:39:14.582720+0000","last_undegraded":"2024-09-16T03:39:14.582720+0000","last_fullsized":"2024-09-16T03:39:14.582720+0000","mapping_epoch":138,"log_start":"39'3","ondisk_log_start":"39'3","created":100,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:39:00.968448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.968448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.929675+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,6],"acting":[9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.23","version":"108'38","reported_seq":365,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.573224+0000","last_change":"2024-09-16T03:38:49.464614+0000","last_active":"2024-09-16T03:39:13.573224+0000","last_peered":"2024-09-16T03:39:13.573224+0000","last_clean":"2024-09-16T03:39:13.573224+0000","last_became_active":"2024-09-16T03:38:49.463797+0000","last_became_peered":"2024-09-16T03:38:49.463797+0000","last_unstale":"2024-09-16T03:39:13.573224+0000","last_undegraded":"2024-09-16T03:39:13.573224+0000","last_fullsized":"2024-09-16T03:39:13.573224+0000","mapping_epoch":114,"log_start":"105'36","ondisk_log_start":"105'36","created":100,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:34.893524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.893524+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":35,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:52.829073+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,2],"acting":[14,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.24","version":"105'32","reported_seq":345,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.704456+0000","last_change":"2024-09-16T03:38:47.978956+0000","last_active":"2024-09-16T03:39:16.704456+0000","last_peered":"2024-09-16T03:39:16.704456+0000","last_clean":"2024-09-16T03:39:16.704456+0000","last_became_active":"2024-09-16T03:38:45.820201+0000","last_became_peered":"2024-09-16T03:38:45.820201+0000","last_unstale":"2024-09-16T03:39:16.704456+0000","last_undegraded":"2024-09-16T03:39:16.704456+0000","last_fullsized":"2024-09-16T03:39:16.704456+0000","mapping_epoch":110,"log_start":"83'29","ondisk_log_start":"83'29","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.978888+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:56.881865+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.25","version":"130'85","reported_seq":437,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.706102+0000","last_change":"2024-09-16T03:38:48.088833+0000","last_active":"2024-09-16T03:39:16.706102+0000","last_peered":"2024-09-16T03:39:16.706102+0000","last_clean":"2024-09-16T03:39:16.706102+0000","last_became_active":"2024-09-16T03:38:45.724757+0000","last_became_peered":"2024-09-16T03:38:45.724757+0000","last_unstale":"2024-09-16T03:39:16.706102+0000","last_undegraded":"2024-09-16T03:39:16.706102+0000","last_fullsized":"2024-09-16T03:39:16.706102+0000","mapping_epoch":110,"log_start":"116'82","ondisk_log_start":"116'82","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'77","last_scrub_stamp":"2024-09-16T03:38:48.088768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.088768+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":79,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:53.112635+0000","scrub_duration":388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2134555,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":2264,"num_write":10,"num_write_kb":2500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.26","version":"137'57","reported_seq":416,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.571298+0000","last_change":"2024-09-16T03:38:49.477393+0000","last_active":"2024-09-16T03:39:13.571298+0000","last_peered":"2024-09-16T03:39:13.571298+0000","last_clean":"2024-09-16T03:39:13.571298+0000","last_became_active":"2024-09-16T03:38:45.926029+0000","last_became_peered":"2024-09-16T03:38:45.926029+0000","last_unstale":"2024-09-16T03:39:13.571298+0000","last_undegraded":"2024-09-16T03:39:13.571298+0000","last_fullsized":"2024-09-16T03:39:13.571298+0000","mapping_epoch":110,"log_start":"133'54","ondisk_log_start":"133'54","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:49.477297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.477297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.242430+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336239,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":24204,"num_write":15,"num_write_kb":4159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.27","version":"109'66","reported_seq":399,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.571408+0000","last_change":"2024-09-16T03:38:47.511343+0000","last_active":"2024-09-16T03:39:13.571408+0000","last_peered":"2024-09-16T03:39:13.571408+0000","last_clean":"2024-09-16T03:39:13.571408+0000","last_became_active":"2024-09-16T03:38:45.824884+0000","last_became_peered":"2024-09-16T03:38:45.824884+0000","last_unstale":"2024-09-16T03:39:13.571408+0000","last_undegraded":"2024-09-16T03:39:13.571408+0000","last_fullsized":"2024-09-16T03:39:13.571408+0000","mapping_epoch":110,"log_start":"105'63","ondisk_log_start":"105'63","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'66","last_scrub_stamp":"2024-09-16T03:38:47.511252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.511252+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.884916+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.28","version":"129'68","reported_seq":493,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.178559+0000","last_change":"2024-09-16T03:39:13.974976+0000","last_active":"2024-09-16T03:39:14.178559+0000","last_peered":"2024-09-16T03:39:14.178559+0000","last_clean":"2024-09-16T03:39:14.178559+0000","last_became_active":"2024-09-16T03:39:13.974464+0000","last_became_peered":"2024-09-16T03:39:13.974464+0000","last_unstale":"2024-09-16T03:39:14.178559+0000","last_undegraded":"2024-09-16T03:39:14.178559+0000","last_fullsized":"2024-09-16T03:39:14.178559+0000","mapping_epoch":138,"log_start":"118'65","ondisk_log_start":"118'65","created":110,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:48.551929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.551929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":64,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:10.916309+0000","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395113,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":46322,"num_write":5,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7221888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":261,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.572963+0000","last_change":"2024-09-16T03:39:00.834877+0000","last_active":"2024-09-16T03:39:13.572963+0000","last_peered":"2024-09-16T03:39:13.572963+0000","last_clean":"2024-09-16T03:39:13.572963+0000","last_became_active":"2024-09-16T03:38:45.627369+0000","last_became_peered":"2024-09-16T03:38:45.627369+0000","last_unstale":"2024-09-16T03:39:13.572963+0000","last_undegraded":"2024-09-16T03:39:13.572963+0000","last_fullsized":"2024-09-16T03:39:13.572963+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:00.834742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.834742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.318226+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.2a","version":"108'122","reported_seq":614,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.982814+0000","last_change":"2024-09-16T03:38:47.964525+0000","last_active":"2024-09-16T03:39:13.982814+0000","last_peered":"2024-09-16T03:39:13.982814+0000","last_clean":"2024-09-16T03:39:13.982814+0000","last_became_active":"2024-09-16T03:38:46.022108+0000","last_became_peered":"2024-09-16T03:38:46.022108+0000","last_unstale":"2024-09-16T03:39:13.982814+0000","last_undegraded":"2024-09-16T03:39:13.982814+0000","last_fullsized":"2024-09-16T03:39:13.982814+0000","mapping_epoch":110,"log_start":"107'115","ondisk_log_start":"107'115","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'122","last_scrub_stamp":"2024-09-16T03:38:47.964439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.964439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":112,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.917123+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.2b","version":"140'127","reported_seq":600,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.733610+0000","last_change":"2024-09-16T03:39:09.949533+0000","last_active":"2024-09-16T03:39:16.733610+0000","last_peered":"2024-09-16T03:39:16.733610+0000","last_clean":"2024-09-16T03:39:16.733610+0000","last_became_active":"2024-09-16T03:39:09.949146+0000","last_became_peered":"2024-09-16T03:39:09.949146+0000","last_unstale":"2024-09-16T03:39:16.733610+0000","last_undegraded":"2024-09-16T03:39:16.733610+0000","last_fullsized":"2024-09-16T03:39:16.733610+0000","mapping_epoch":134,"log_start":"127'124","ondisk_log_start":"127'124","created":110,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:49.064061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064061+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":124,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:16.396765+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588344,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":3265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2c","version":"138'64","reported_seq":433,"reported_epoch":139,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:13.877237+0000","last_change":"2024-09-16T03:39:11.380628+0000","last_active":"2024-09-16T03:39:13.877237+0000","last_peered":"2024-09-16T03:39:13.877237+0000","last_clean":"2024-09-16T03:39:07.529785+0000","last_became_active":"2024-09-16T03:39:11.279455+0000","last_became_peered":"2024-09-16T03:39:11.279455+0000","last_unstale":"2024-09-16T03:39:13.877237+0000","last_undegraded":"2024-09-16T03:39:13.877237+0000","last_fullsized":"2024-09-16T03:39:13.877237+0000","mapping_epoch":135,"log_start":"118'61","ondisk_log_start":"118'61","created":110,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'58","last_scrub_stamp":"2024-09-16T03:38:47.889276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.889276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":18905,"num_write":5,"num_write_kb":1234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1262403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,11],"avail_no_missing":["12","11","15"],"object_location_counts":[{"shards":"11,12,15","objects":1}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.2d","version":"92'14","reported_seq":342,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-16T03:39:13.771105+0000","last_change":"2024-09-16T03:38:47.115739+0000","last_active":"2024-09-16T03:39:13.771105+0000","last_peered":"2024-09-16T03:39:13.771105+0000","last_clean":"2024-09-16T03:39:13.771105+0000","last_became_active":"2024-09-16T03:38:45.723438+0000","last_became_peered":"2024-09-16T03:38:45.723438+0000","last_unstale":"2024-09-16T03:39:13.771105+0000","last_undegraded":"2024-09-16T03:39:13.771105+0000","last_fullsized":"2024-09-16T03:39:13.771105+0000","mapping_epoch":110,"log_start":"91'11","ondisk_log_start":"91'11","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:47.115651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.115651+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:00.107802+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":56,"stat_sum":{"num_bytes":125315138,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":17,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1004,"num_read_kb":914434,"num_write":353,"num_write_kb":96661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":28533152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":229715968,"data_stored":229151645,"data_compressed":389631,"data_compressed_allocated":430080,"data_compressed_original":1110016,"omap_allocated":0,"internal_metadata":0},"log_size":175,"ondisk_log_size":175,"up":168,"acting":168,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":86638052,"num_objects":86,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":942,"num_read_kb":29936,"num_write":1281,"num_write_kb":290634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":39649106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180994048,"data_stored":193423986,"data_compressed":12064856,"data_compressed_allocated":12812288,"data_compressed_original":25608192,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":2644640,"data_compressed":159066,"data_compressed_allocated":204800,"data_compressed_original":2584576,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247839,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72824,"kb_used_data":40236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94299016,"statfs":{"total":96636764160,"available":96562192384,"internally_reserved":0,"allocated":41201664,"data_stored":41637944,"data_compressed":1301463,"data_compressed_allocated":1380352,"data_compressed_original":2392064,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247840,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53800,"kb_used_data":22612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318040,"statfs":{"total":96636764160,"available":96581672960,"internally_reserved":0,"allocated":23154688,"data_stored":24221122,"data_compressed":890468,"data_compressed_allocated":974848,"data_compressed_original":2576384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247839,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70124,"kb_used_data":38004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94301716,"statfs":{"total":96636764160,"available":96564957184,"internally_reserved":0,"allocated":38916096,"data_stored":40240084,"data_compressed":1556613,"data_compressed_allocated":1675264,"data_compressed_original":3543040,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280544,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55464,"kb_used_data":23664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94316376,"statfs":{"total":96636764160,"available":96579969024,"internally_reserved":0,"allocated":24231936,"data_stored":25264169,"data_compressed":880190,"data_compressed_allocated":962560,"data_compressed_original":2510848,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280544,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280543,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313248,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35008,"kb_used_data":7980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336832,"statfs":{"total":96636764160,"available":96600915968,"internally_reserved":0,"allocated":8171520,"data_stored":7746857,"data_compressed":34276,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1,2,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280544,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":90972,"kb_used_data":54212,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94280868,"statfs":{"total":96636764160,"available":96543608832,"internally_reserved":0,"allocated":55513088,"data_stored":56739541,"data_compressed":1505705,"data_compressed_allocated":1617920,"data_compressed_original":3477504,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280543,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94780,"kb_used_data":63596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94277060,"statfs":{"total":96636764160,"available":96539709440,"internally_reserved":0,"allocated":65122304,"data_stored":67341692,"data_compressed":2317788,"data_compressed_allocated":2486272,"data_compressed_original":5332992,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313248,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280543,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35952,"kb_used_data":4756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335888,"statfs":{"total":96636764160,"available":96599949312,"internally_reserved":0,"allocated":4870144,"data_stored":4495691,"data_compressed":34276,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280543,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69156,"kb_used_data":36556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302684,"statfs":{"total":96636764160,"available":96565948416,"internally_reserved":0,"allocated":37433344,"data_stored":37994437,"data_compressed":1250690,"data_compressed_allocated":1331200,"data_compressed_original":2437120,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280543,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44220,"kb_used_data":12116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327620,"statfs":{"total":96636764160,"available":96591482880,"internally_reserved":0,"allocated":12406784,"data_stored":12456948,"data_compressed":77699,"data_compressed_allocated":102400,"data_compressed_original":643072,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,3,4,6,7,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280544,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88784,"kb_used_data":52876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94283056,"statfs":{"total":96636764160,"available":96545849344,"internally_reserved":0,"allocated":54145024,"data_stored":53713305,"data_compressed":134118,"data_compressed_allocated":159744,"data_compressed_original":425984,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,3,4,5,6,7,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313247,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42800,"kb_used_data":11608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329040,"statfs":{"total":96636764160,"available":96592936960,"internally_reserved":0,"allocated":11886592,"data_stored":11460816,"data_compressed":57065,"data_compressed_allocated":73728,"data_compressed_original":163840,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247840,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54712,"kb_used_data":23524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94317128,"statfs":{"total":96636764160,"available":96580739072,"internally_reserved":0,"allocated":24088576,"data_stored":24556927,"data_compressed":1271709,"data_compressed_allocated":1347584,"data_compressed_original":2306048,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13398016,"data_stored":14400861,"data_compressed":814666,"data_compressed_allocated":872448,"data_compressed_original":1900544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13422592,"data_stored":14113910,"data_compressed":606276,"data_compressed_allocated":643072,"data_compressed_original":1376256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12906496,"data_stored":13784230,"data_compressed":1187486,"data_compressed_allocated":1245184,"data_compressed_original":2142208,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25546752,"data_stored":27272388,"data_compressed":1564402,"data_compressed_allocated":1667072,"data_compressed_original":3444736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38944768,"data_stored":41673249,"data_compressed":2379068,"data_compressed_allocated":2539520,"data_compressed_original":5345280,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":26152960,"data_stored":27905732,"data_compressed":1535355,"data_compressed_allocated":1638400,"data_compressed_original":3444736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12906496,"data_stored":13784230,"data_compressed":1187486,"data_compressed_allocated":1245184,"data_compressed_original":2142208,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":13303808,"data_stored":14306653,"data_compressed":814666,"data_compressed_allocated":872448,"data_compressed_original":1900544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":12906496,"data_stored":13784230,"data_compressed":1187486,"data_compressed_allocated":1245184,"data_compressed_original":2142208,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9502720,"data_stored":9446986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9904128,"data_stored":9871428,"data_compressed":49947,"data_compressed_allocated":53248,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":29691904,"data_stored":29622776,"data_compressed":32337,"data_compressed_allocated":36864,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":26198016,"data_stored":26107915,"data_compressed":29754,"data_compressed_allocated":32768,"data_compressed_original":86016,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13172736,"data_stored":13183621,"data_compressed":44673,"data_compressed_allocated":49152,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3657728,"data_stored":3629400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":22847488,"data_stored":22795540,"data_compressed":9548,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":11096064,"data_stored":11035135,"data_compressed":10762,"data_compressed_allocated":12288,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49242112,"data_stored":49209022,"data_compressed":99842,"data_compressed_allocated":110592,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8478720,"data_stored":8473398,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":10596352,"data_stored":10564238,"data_compressed":22789,"data_compressed_allocated":24576,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5840896,"data_stored":5807809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":28397568,"data_stored":28324337,"data_compressed":79701,"data_compressed_allocated":86016,"data_compressed_original":151552,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:39:18.206 INFO:tasks.rados.rados.0.smithi157.stdout:1786: expect (ObjNum 306 snap 0 seq_num 306) 2024-09-16T03:39:18.233 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:18.233 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:18.241 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:39:18.272 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2466246 2024-09-16T03:39:18.282 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:18.280+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:18.331 INFO:tasks.rados.rados.0.smithi157.stdout:1787: finishing write tid 1 to smithi15736696-18 2024-09-16T03:39:18.331 INFO:tasks.rados.rados.0.smithi157.stdout:1784: expect (ObjNum 359 snap 0 seq_num 359) 2024-09-16T03:39:18.365 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 402646 2024-09-16T03:39:18.381 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:18.380+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:18.384 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:18.384 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:18.392 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:39:18.431 INFO:tasks.rados.rados.0.smithi157.stdout:1789: finishing write tid 1 to smithi15736696-27 2024-09-16T03:39:18.431 INFO:tasks.rados.rados.0.smithi157.stdout:1789: finishing write tid 2 to smithi15736696-27 2024-09-16T03:39:18.431 INFO:tasks.rados.rados.0.smithi157.stdout:1789: finishing write tid 3 to smithi15736696-27 2024-09-16T03:39:18.431 INFO:tasks.rados.rados.0.smithi157.stdout:1789: finishing write tid 4 to smithi15736696-27 2024-09-16T03:39:18.431 INFO:tasks.rados.rados.0.smithi157.stdout:1789: oid 27 updating version 0 to 152 2024-09-16T03:39:18.431 INFO:tasks.rados.rados.0.smithi157.stdout:1789: oid 27 version 152 is already newer than 151 2024-09-16T03:39:18.431 INFO:tasks.rados.rados.0.smithi157.stdout:1789: oid 27 updating version 152 to 153 2024-09-16T03:39:18.432 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 27 v 153 (ObjNum 395 snap 0 seq_num 395) dirty exists 2024-09-16T03:39:18.432 INFO:tasks.rados.rados.0.smithi157.stdout:1789: left oid 27 (ObjNum 395 snap 0 seq_num 395) 2024-09-16T03:39:18.432 INFO:tasks.rados.rados.0.smithi157.stdout:1781: done (10 left) 2024-09-16T03:39:18.432 INFO:tasks.rados.rados.0.smithi157.stdout:1782: done (9 left) 2024-09-16T03:39:18.432 INFO:tasks.rados.rados.0.smithi157.stdout:1783: done (8 left) 2024-09-16T03:39:18.432 INFO:tasks.rados.rados.0.smithi157.stdout:1784: done (7 left) 2024-09-16T03:39:18.432 INFO:tasks.rados.rados.0.smithi157.stdout:1785: done (6 left) 2024-09-16T03:39:18.432 INFO:tasks.rados.rados.0.smithi157.stdout:1786: done (5 left) 2024-09-16T03:39:18.432 INFO:tasks.rados.rados.0.smithi157.stdout:1789: done (4 left) 2024-09-16T03:39:18.432 INFO:tasks.rados.rados.0.smithi157.stdout:1790: done (3 left) 2024-09-16T03:39:18.432 INFO:tasks.rados.rados.0.smithi157.stdout:1791: done (2 left) 2024-09-16T03:39:18.432 INFO:tasks.rados.rados.0.smithi157.stdout:1792: set_chunk oid 47 target oid 2024-09-16T03:39:18.483 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:18.481+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:18.496 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:18.497 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:18.505 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:39:18.584 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:18.582+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:18.594 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:18.592+0000 7f12e4a1e640 -1 osd.5 141 handle_osd_map injecting map crc failure 2024-09-16T03:39:18.598 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:39:18.610 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:18.610 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:18.619 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:39:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1792: set_chunk oid 47 offset: 1159168 length: 760832 target oid 2d288ab35083a152cee70fb2e3ea8c11f18c8e0dfe039ff85754d17c33306c04 offset: 0 snap -1 2024-09-16T03:39:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1793: read oid 36 snap -1 2024-09-16T03:39:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1793: expect (ObjNum 306 snap 0 seq_num 306) 2024-09-16T03:39:18.639 INFO:tasks.rados.rados.0.smithi157.stdout:1794: set_chunk oid 2 target oid 2024-09-16T03:39:18.640 INFO:tasks.rados.rados.0.smithi157.stdout:1794: set_chunk oid 2 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:39:18.640 INFO:tasks.rados.rados.0.smithi157.stdout:1795: read oid 3 snap -1 2024-09-16T03:39:18.640 INFO:tasks.rados.rados.0.smithi157.stdout:1795: expect deleted 2024-09-16T03:39:18.640 INFO:tasks.rados.rados.0.smithi157.stdout:1796: tier_flush oid 8 2024-09-16T03:39:18.640 INFO:tasks.rados.rados.0.smithi157.stdout:1796: tier_flush oid 8 snap -1 2024-09-16T03:39:18.640 INFO:tasks.rados.rados.0.smithi157.stdout:1797: read oid 27 snap -1 2024-09-16T03:39:18.640 INFO:tasks.rados.rados.0.smithi157.stdout:1797: expect (ObjNum 395 snap 0 seq_num 395) 2024-09-16T03:39:18.640 INFO:tasks.rados.rados.0.smithi157.stdout:1798: write oid 38 current snap is 0 2024-09-16T03:39:18.640 INFO:tasks.rados.rados.0.smithi157.stdout:1798: seq_num 397 ranges {672379=724209,1865116=608472} 2024-09-16T03:39:18.647 INFO:tasks.rados.rados.0.smithi157.stdout:1798: writing smithi15736696-38 from 672379 to 1396588 tid 1 2024-09-16T03:39:18.655 INFO:tasks.rados.rados.0.smithi157.stdout:1798: writing smithi15736696-38 from 1865116 to 2473588 tid 2 2024-09-16T03:39:18.655 INFO:tasks.rados.rados.0.smithi157.stdout:1799: read oid 28 snap -1 2024-09-16T03:39:18.655 INFO:tasks.rados.rados.0.smithi157.stdout:1799: expect (ObjNum 387 snap 0 seq_num 387) 2024-09-16T03:39:18.655 INFO:tasks.rados.rados.0.smithi157.stdout:1800: write (excl) oid 14 current snap is 0 2024-09-16T03:39:18.655 INFO:tasks.rados.rados.0.smithi157.stdout:1787: finishing write tid 2 to smithi15736696-18 2024-09-16T03:39:18.655 INFO:tasks.rados.rados.0.smithi157.stdout:1787: finishing write tid 3 to smithi15736696-18 2024-09-16T03:39:18.655 INFO:tasks.rados.rados.0.smithi157.stdout:1787: finishing write tid 4 to smithi15736696-18 2024-09-16T03:39:18.655 INFO:tasks.rados.rados.0.smithi157.stdout:1787: oid 18 updating version 0 to 67 2024-09-16T03:39:18.655 INFO:tasks.rados.rados.0.smithi157.stdout:1787: oid 18 version 67 is already newer than 65 2024-09-16T03:39:18.656 INFO:tasks.rados.rados.0.smithi157.stdout:1787: oid 18 version 67 is already newer than 66 2024-09-16T03:39:18.656 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 67 (ObjNum 393 snap 0 seq_num 393) dirty exists 2024-09-16T03:39:18.656 INFO:tasks.rados.rados.0.smithi157.stdout:1787: left oid 18 (ObjNum 393 snap 0 seq_num 393) 2024-09-16T03:39:18.656 INFO:tasks.rados.rados.0.smithi157.stdout:1788: finishing write tid 1 to smithi15736696-10 2024-09-16T03:39:18.656 INFO:tasks.rados.rados.0.smithi157.stdout:1788: finishing write tid 2 to smithi15736696-10 2024-09-16T03:39:18.656 INFO:tasks.rados.rados.0.smithi157.stdout:1788: finishing write tid 3 to smithi15736696-10 2024-09-16T03:39:18.656 INFO:tasks.rados.rados.0.smithi157.stdout:1788: finishing write tid 4 to smithi15736696-10 2024-09-16T03:39:18.656 INFO:tasks.rados.rados.0.smithi157.stdout:1788: oid 10 updating version 0 to 120 2024-09-16T03:39:18.656 INFO:tasks.rados.rados.0.smithi157.stdout:1788: oid 10 updating version 120 to 121 2024-09-16T03:39:18.656 INFO:tasks.rados.rados.0.smithi157.stdout:1788: oid 10 updating version 121 to 122 2024-09-16T03:39:18.656 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 122 (ObjNum 394 snap 0 seq_num 394) dirty exists 2024-09-16T03:39:18.656 INFO:tasks.rados.rados.0.smithi157.stdout:1788: left oid 10 (ObjNum 394 snap 0 seq_num 394) 2024-09-16T03:39:18.656 INFO:tasks.rados.rados.0.smithi157.stdout:1800: seq_num 398 ranges {689186=799458,1923598=752130} 2024-09-16T03:39:18.662 INFO:tasks.rados.rados.0.smithi157.stdout:1800: writing smithi15736696-14 from 689186 to 1488644 tid 1 2024-09-16T03:39:18.669 INFO:tasks.rados.rados.0.smithi157.stdout:1800: writing smithi15736696-14 from 1923598 to 2675728 tid 2 2024-09-16T03:39:18.669 INFO:tasks.rados.rados.0.smithi157.stdout:1787: done (10 left) 2024-09-16T03:39:18.669 INFO:tasks.rados.rados.0.smithi157.stdout:1788: done (9 left) 2024-09-16T03:39:18.669 INFO:tasks.rados.rados.0.smithi157.stdout:1801: read oid 13 snap -1 2024-09-16T03:39:18.669 INFO:tasks.rados.rados.0.smithi157.stdout:1801: expect (ObjNum 348 snap 0 seq_num 348) 2024-09-16T03:39:18.669 INFO:tasks.rados.rados.0.smithi157.stdout:1802: write (excl) oid 20 current snap is 0 2024-09-16T03:39:18.669 INFO:tasks.rados.rados.0.smithi157.stdout:1794: finishing set_chunk to oid 2 2024-09-16T03:39:18.669 INFO:tasks.rados.rados.0.smithi157.stdout:1794: got expected ENOENT (src dne) 2024-09-16T03:39:18.669 INFO:tasks.rados.rados.0.smithi157.stdout:1797: expect (ObjNum 395 snap 0 seq_num 395) 2024-09-16T03:39:18.704 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:18.702+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:18.765 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:18.765 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:18.774 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:39:18.785 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:18.784+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:18.851 INFO:tasks.rados.rados.0.smithi157.stdout:1801: expect (ObjNum 348 snap 0 seq_num 348) 2024-09-16T03:39:18.886 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:18.884+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:18.921 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:18.921 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:18.929 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:39:18.948 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 967101 2024-09-16T03:39:18.987 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:18.986+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:19.035 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:19.035 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:19.044 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:39:19.088 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:19.087+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:19.139 INFO:tasks.rados.rados.0.smithi157.stdout:1799: expect (ObjNum 387 snap 0 seq_num 387) 2024-09-16T03:39:19.150 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 977929 2024-09-16T03:39:19.187 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:19.188 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:19.193 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:19.187+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:19.197 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:39:19.289 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:19.288+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:19.326 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2092718 2024-09-16T03:39:19.338 INFO:tasks.rados.rados.0.smithi157.stdout:1798: finishing write tid 1 to smithi15736696-38 2024-09-16T03:39:19.338 INFO:tasks.rados.rados.0.smithi157.stdout:1800: finishing write tid 1 to smithi15736696-14 2024-09-16T03:39:19.338 INFO:tasks.rados.rados.0.smithi157.stdout:1802: seq_num 399 ranges {705993=474707,1582080=612141} 2024-09-16T03:39:19.341 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:19.341 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:19.344 INFO:tasks.rados.rados.0.smithi157.stdout:1802: writing smithi15736696-20 from 705993 to 1180700 tid 1 2024-09-16T03:39:19.351 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:39:19.353 INFO:tasks.rados.rados.0.smithi157.stdout:1802: writing smithi15736696-20 from 1582080 to 2194221 tid 2 2024-09-16T03:39:19.353 INFO:tasks.rados.rados.0.smithi157.stdout:1794: done (10 left) 2024-09-16T03:39:19.353 INFO:tasks.rados.rados.0.smithi157.stdout:1795: done (9 left) 2024-09-16T03:39:19.353 INFO:tasks.rados.rados.0.smithi157.stdout:1797: done (8 left) 2024-09-16T03:39:19.354 INFO:tasks.rados.rados.0.smithi157.stdout:1799: done (7 left) 2024-09-16T03:39:19.354 INFO:tasks.rados.rados.0.smithi157.stdout:1801: done (6 left) 2024-09-16T03:39:19.354 INFO:tasks.rados.rados.0.smithi157.stdout:1803: read oid 7 snap -1 2024-09-16T03:39:19.354 INFO:tasks.rados.rados.0.smithi157.stdout:1803: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-16T03:39:19.354 INFO:tasks.rados.rados.0.smithi157.stdout:1804: read oid 10 snap -1 2024-09-16T03:39:19.354 INFO:tasks.rados.rados.0.smithi157.stdout:1804: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-16T03:39:19.354 INFO:tasks.rados.rados.0.smithi157.stdout:1805: read oid 16 snap -1 2024-09-16T03:39:19.354 INFO:tasks.rados.rados.0.smithi157.stdout:1805: expect (ObjNum 388 snap 0 seq_num 388) 2024-09-16T03:39:19.354 INFO:tasks.rados.rados.0.smithi157.stdout:1806: write (excl) oid 42 current snap is 0 2024-09-16T03:39:19.354 INFO:tasks.rados.rados.0.smithi157.stdout:1800: finishing write tid 2 to smithi15736696-14 2024-09-16T03:39:19.354 INFO:tasks.rados.rados.0.smithi157.stdout:1798: finishing write tid 2 to smithi15736696-38 2024-09-16T03:39:19.354 INFO:tasks.rados.rados.0.smithi157.stdout:1800: finishing write tid 3 to smithi15736696-14 2024-09-16T03:39:19.354 INFO:tasks.rados.rados.0.smithi157.stdout:1806: seq_num 400 ranges {722800=549956,2040562=682238} 2024-09-16T03:39:19.357 INFO:tasks.rados.rados.0.smithi157.stdout:1806: writing smithi15736696-42 from 722800 to 1272756 tid 1 2024-09-16T03:39:19.359 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:39:19.366 INFO:tasks.rados.rados.0.smithi157.stdout:1806: writing smithi15736696-42 from 2040562 to 2722800 tid 2 2024-09-16T03:39:19.366 INFO:tasks.rados.rados.0.smithi157.stdout:1807: write oid 13 current snap is 0 2024-09-16T03:39:19.366 INFO:tasks.rados.rados.0.smithi157.stdout:1798: finishing write tid 3 to smithi15736696-38 2024-09-16T03:39:19.366 INFO:tasks.rados.rados.0.smithi157.stdout:1798: finishing write tid 4 to smithi15736696-38 2024-09-16T03:39:19.366 INFO:tasks.rados.rados.0.smithi157.stdout:1798: oid 38 updating version 0 to 41 2024-09-16T03:39:19.366 INFO:tasks.rados.rados.0.smithi157.stdout:1798: oid 38 version 41 is already newer than 40 2024-09-16T03:39:19.366 INFO:tasks.rados.rados.0.smithi157.stdout:1798: oid 38 version 41 is already newer than 39 2024-09-16T03:39:19.366 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 41 (ObjNum 397 snap 0 seq_num 397) dirty exists 2024-09-16T03:39:19.366 INFO:tasks.rados.rados.0.smithi157.stdout:1798: left oid 38 (ObjNum 397 snap 0 seq_num 397) 2024-09-16T03:39:19.366 INFO:tasks.rados.rados.0.smithi157.stdout:1800: finishing write tid 4 to smithi15736696-14 2024-09-16T03:39:19.366 INFO:tasks.rados.rados.0.smithi157.stdout:1800: oid 14 updating version 0 to 53 2024-09-16T03:39:19.367 INFO:tasks.rados.rados.0.smithi157.stdout:1800: oid 14 updating version 53 to 55 2024-09-16T03:39:19.367 INFO:tasks.rados.rados.0.smithi157.stdout:1800: oid 14 version 55 is already newer than 54 2024-09-16T03:39:19.367 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 55 (ObjNum 398 snap 0 seq_num 398) dirty exists 2024-09-16T03:39:19.367 INFO:tasks.rados.rados.0.smithi157.stdout:1800: left oid 14 (ObjNum 398 snap 0 seq_num 398) 2024-09-16T03:39:19.367 INFO:tasks.rados.rados.0.smithi157.stdout:1807: seq_num 401 ranges {739607=625205,1982691=499457} 2024-09-16T03:39:19.373 INFO:tasks.rados.rados.0.smithi157.stdout:1807: writing smithi15736696-13 from 739607 to 1364812 tid 1 2024-09-16T03:39:19.379 INFO:tasks.rados.rados.0.smithi157.stdout:1807: writing smithi15736696-13 from 1982691 to 2482148 tid 2 2024-09-16T03:39:19.379 INFO:tasks.rados.rados.0.smithi157.stdout:1798: done (10 left) 2024-09-16T03:39:19.379 INFO:tasks.rados.rados.0.smithi157.stdout:1800: done (9 left) 2024-09-16T03:39:19.379 INFO:tasks.rados.rados.0.smithi157.stdout:1808: delete oid 18 current snap is 0 2024-09-16T03:39:19.379 INFO:tasks.rados.rados.0.smithi157.stdout:1796: got (0) Success 2024-09-16T03:39:19.379 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 359 snap 0 seq_num 359) dirty exists 2024-09-16T03:39:19.391 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:19.389+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:19.451 INFO:tasks.rados.rados.0.smithi157.stdout:1805: expect (ObjNum 388 snap 0 seq_num 388) 2024-09-16T03:39:19.454 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 605268 2024-09-16T03:39:19.462 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:19.462 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:19.472 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:39:19.487 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:19.487 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:19.491 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:19.489+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:19.492 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:19.505 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:19.505 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:19.522 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:19.522 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:19.540 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:19.540 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:19.546 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1576265 2024-09-16T03:39:19.556 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:19.556 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:19.572 INFO:tasks.rados.rados.0.smithi157.stdout:1804: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-16T03:39:19.572 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:19.573 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:19.573 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:19.590 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:19.590 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:19.591 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:19.590+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:19.597 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:19.597 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:19.601 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:39:19.607 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:39:19.608 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:19.608 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:19.624 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:19.625 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:19.641 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:19.641 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:19.652 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1926660 2024-09-16T03:39:19.658 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:19.658 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:19.676 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:19.676 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:19.692 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:19.692 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:19.693 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:19.690+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:19.709 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:19.709 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:19.721 INFO:tasks.rados.rados.0.smithi157.stdout:1802: finishing write tid 1 to smithi15736696-20 2024-09-16T03:39:19.721 INFO:tasks.rados.rados.0.smithi157.stdout:1792: finishing set_chunk to oid 47 2024-09-16T03:39:19.721 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 47 v 132 (ObjNum 362 snap 0 seq_num 362) dirty exists 2024-09-16T03:39:19.721 INFO:tasks.rados.rados.0.smithi157.stdout:1802: finishing write tid 2 to smithi15736696-20 2024-09-16T03:39:19.721 INFO:tasks.rados.rados.0.smithi157.stdout:1802: finishing write tid 3 to smithi15736696-20 2024-09-16T03:39:19.721 INFO:tasks.rados.rados.0.smithi157.stdout:1806: finishing write tid 1 to smithi15736696-42 2024-09-16T03:39:19.721 INFO:tasks.rados.rados.0.smithi157.stdout:1806: finishing write tid 2 to smithi15736696-42 2024-09-16T03:39:19.721 INFO:tasks.rados.rados.0.smithi157.stdout:1806: finishing write tid 3 to smithi15736696-42 2024-09-16T03:39:19.721 INFO:tasks.rados.rados.0.smithi157.stdout:1806: finishing write tid 4 to smithi15736696-42 2024-09-16T03:39:19.721 INFO:tasks.rados.rados.0.smithi157.stdout:1806: oid 42 updating version 0 to 75 2024-09-16T03:39:19.722 INFO:tasks.rados.rados.0.smithi157.stdout:1806: oid 42 version 75 is already newer than 74 2024-09-16T03:39:19.722 INFO:tasks.rados.rados.0.smithi157.stdout:1806: oid 42 updating version 75 to 76 2024-09-16T03:39:19.722 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 76 (ObjNum 400 snap 0 seq_num 400) dirty exists 2024-09-16T03:39:19.722 INFO:tasks.rados.rados.0.smithi157.stdout:1806: left oid 42 (ObjNum 400 snap 0 seq_num 400) 2024-09-16T03:39:19.722 INFO:tasks.rados.rados.0.smithi157.stdout:1803: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-16T03:39:19.726 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:19.726 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:19.742 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:19.743 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:19.754 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:19.755 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:19.755 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:19.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:19.755 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:19.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.756 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:39:19.756 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:39:19.756 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 127.56663462500001, 2024-09-16T03:39:19.756 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102630163, 2024-09-16T03:39:19.756 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.756 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.756 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.756 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:39:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:39:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186081+0000", 2024-09-16T03:39:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5600000000000005e-07 2024-09-16T03:39:19.757 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.758 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.758 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.758 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186970+0000", 2024-09-16T03:39:19.758 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00088889000000000004 2024-09-16T03:39:19.758 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.758 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.758 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.758 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186979+0000", 2024-09-16T03:39:19.758 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0620000000000007e-06 2024-09-16T03:39:19.758 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.758 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.758 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.759 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186986+0000", 2024-09-16T03:39:19.759 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:39:19.759 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.759 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.759 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.759 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287184+0000", 2024-09-16T03:39:19.759 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100198036 2024-09-16T03:39:19.759 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.759 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.759 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.759 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287199+0000", 2024-09-16T03:39:19.759 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5458999999999999e-05 2024-09-16T03:39:19.759 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.760 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.760 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288659+0000", 2024-09-16T03:39:19.760 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014601900000000001 2024-09-16T03:39:19.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.760 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.760 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288711+0000", 2024-09-16T03:39:19.760 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1088000000000003e-05 2024-09-16T03:39:19.760 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.760 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.760 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.761 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.761 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:19.761 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:39:19.761 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 127.565598192, 2024-09-16T03:39:19.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10205370900000001, 2024-09-16T03:39:19.761 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.761 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.761 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.761 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.761 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.761 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.761 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:39:19.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:39:19.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187118+0000", 2024-09-16T03:39:19.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8599999999999998e-07 2024-09-16T03:39:19.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188062+0000", 2024-09-16T03:39:19.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094447300000000004 2024-09-16T03:39:19.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.763 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188070+0000", 2024-09-16T03:39:19.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.695e-06 2024-09-16T03:39:19.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.764 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188074+0000", 2024-09-16T03:39:19.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0709999999999996e-06 2024-09-16T03:39:19.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.764 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287842+0000", 2024-09-16T03:39:19.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099768478999999993 2024-09-16T03:39:19.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.765 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.765 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287864+0000", 2024-09-16T03:39:19.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1679999999999999e-05 2024-09-16T03:39:19.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289137+0000", 2024-09-16T03:39:19.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012732430000000001 2024-09-16T03:39:19.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.767 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.767 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289171+0000", 2024-09-16T03:39:19.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3382e-05 2024-09-16T03:39:19.767 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.767 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.767 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.767 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:42 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:19.767 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:39:19.768 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 127.495516284, 2024-09-16T03:39:19.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10172858, 2024-09-16T03:39:19.768 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.768 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.768 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.768 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:39:19.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:39:19.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257201+0000", 2024-09-16T03:39:19.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:39:19.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.770 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257536+0000", 2024-09-16T03:39:19.770 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033513899999999998 2024-09-16T03:39:19.770 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.770 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.770 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.770 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257555+0000", 2024-09-16T03:39:19.770 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8547999999999999e-05 2024-09-16T03:39:19.770 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.770 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.770 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.770 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257566+0000", 2024-09-16T03:39:19.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1379999999999999e-05 2024-09-16T03:39:19.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.771 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.771 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357720+0000", 2024-09-16T03:39:19.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100153532 2024-09-16T03:39:19.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.771 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.771 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357734+0000", 2024-09-16T03:39:19.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4491000000000001e-05 2024-09-16T03:39:19.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.772 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358896+0000", 2024-09-16T03:39:19.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011620599999999999 2024-09-16T03:39:19.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.772 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.772 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358927+0000", 2024-09-16T03:39:19.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1158999999999998e-05 2024-09-16T03:39:19.772 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.772 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.772 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.773 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.773 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:43 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'3, mlcod=0'0)", 2024-09-16T03:39:19.773 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:39:19.773 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 127.493732372, 2024-09-16T03:39:19.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101764453, 2024-09-16T03:39:19.773 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.773 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.773 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.773 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.773 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.773 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:39:19.774 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.774 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.774 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:39:19.774 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.774 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.774 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258984+0000", 2024-09-16T03:39:19.774 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5689999999999999e-06 2024-09-16T03:39:19.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.775 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.775 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258994+0000", 2024-09-16T03:39:19.775 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-16T03:39:19.775 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.775 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.775 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.775 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259008+0000", 2024-09-16T03:39:19.775 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4151999999999999e-05 2024-09-16T03:39:19.775 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.775 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.775 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.775 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359073+0000", 2024-09-16T03:39:19.775 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100065628 2024-09-16T03:39:19.776 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.776 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.776 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.776 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359110+0000", 2024-09-16T03:39:19.776 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7045999999999997e-05 2024-09-16T03:39:19.776 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.776 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.776 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.776 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359124+0000", 2024-09-16T03:39:19.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.382e-05 2024-09-16T03:39:19.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360716+0000", 2024-09-16T03:39:19.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015919829999999999 2024-09-16T03:39:19.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360747+0000", 2024-09-16T03:39:19.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0874999999999998e-05 2024-09-16T03:39:19.778 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.778 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.778 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.778 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:492 4.5 e45/32 4:a11e1d7c:::smithi15736696-43:head v 45'22, mlcod=45'20)", 2024-09-16T03:39:19.779 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:39:19.779 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 117.929364085, 2024-09-16T03:39:19.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100797151, 2024-09-16T03:39:19.779 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.779 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.779 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.779 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:39:19.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.780 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.780 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.780 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.780 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:39:19.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.780 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.780 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.780 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.780 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823353+0000", 2024-09-16T03:39:19.781 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2450000000000001e-06 2024-09-16T03:39:19.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.781 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.781 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.781 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823363+0000", 2024-09-16T03:39:19.781 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0074999999999999e-05 2024-09-16T03:39:19.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.781 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.781 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.781 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823378+0000", 2024-09-16T03:39:19.781 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4534e-05 2024-09-16T03:39:19.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923461+0000", 2024-09-16T03:39:19.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100082859 2024-09-16T03:39:19.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923525+0000", 2024-09-16T03:39:19.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4616999999999995e-05 2024-09-16T03:39:19.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923549+0000", 2024-09-16T03:39:19.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.357e-05 2024-09-16T03:39:19.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924106+0000", 2024-09-16T03:39:19.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000556753 2024-09-16T03:39:19.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924148+0000", 2024-09-16T03:39:19.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2497999999999999e-05 2024-09-16T03:39:19.783 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.783 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.784 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.784 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:53'30,epoch:57/50,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:39:19.784 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:39:19.784 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 99.354012468999997, 2024-09-16T03:39:19.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100230893, 2024-09-16T03:39:19.784 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.784 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.784 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:19.784 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.785 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.785 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:39:19.785 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.785 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.785 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.785 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398703+0000", 2024-09-16T03:39:19.785 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.785 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.785 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.785 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398705+0000", 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5689999999999999e-06 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398722+0000", 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6558e-05 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398735+0000", 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3499999999999999e-05 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.398746+0000", 2024-09-16T03:39:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1182999999999999e-05 2024-09-16T03:39:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498898+0000", 2024-09-16T03:39:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015112599999999 2024-09-16T03:39:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498906+0000", 2024-09-16T03:39:19.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3189999999999995e-06 2024-09-16T03:39:19.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:40.498933+0000", 2024-09-16T03:39:19.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7637999999999999e-05 2024-09-16T03:39:19.788 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.789 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.789 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.789 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1570 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'31, mlcod=48'29)", 2024-09-16T03:39:19.789 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:19.790 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 93.578405473999993, 2024-09-16T03:39:19.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10253683500000001, 2024-09-16T03:39:19.790 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.790 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.790 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.790 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:19.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:19.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:19.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8500000000000002e-07 2024-09-16T03:39:19.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174689+0000", 2024-09-16T03:39:19.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037958600000000001 2024-09-16T03:39:19.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174699+0000", 2024-09-16T03:39:19.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0015e-05 2024-09-16T03:39:19.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174706+0000", 2024-09-16T03:39:19.793 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6250000000000001e-06 2024-09-16T03:39:19.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.793 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.793 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.793 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274865+0000", 2024-09-16T03:39:19.793 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100159157 2024-09-16T03:39:19.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.793 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.793 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274879+0000", 2024-09-16T03:39:19.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3983e-05 2024-09-16T03:39:19.794 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.794 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276820+0000", 2024-09-16T03:39:19.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001941248 2024-09-16T03:39:19.794 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.794 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276846+0000", 2024-09-16T03:39:19.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5936e-05 2024-09-16T03:39:19.794 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.795 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.795 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.795 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1571 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'32, mlcod=48'29)", 2024-09-16T03:39:19.795 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:39:19.795 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 93.577954719999994, 2024-09-16T03:39:19.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10249543, 2024-09-16T03:39:19.795 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.795 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.795 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.795 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:39:19.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:39:19.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174761+0000", 2024-09-16T03:39:19.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0800000000000004e-07 2024-09-16T03:39:19.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175053+0000", 2024-09-16T03:39:19.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029151600000000002 2024-09-16T03:39:19.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175063+0000", 2024-09-16T03:39:19.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0538e-05 2024-09-16T03:39:19.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.798 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.798 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.798 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175070+0000", 2024-09-16T03:39:19.798 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7819999999999996e-06 2024-09-16T03:39:19.798 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.798 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.798 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.798 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275895+0000", 2024-09-16T03:39:19.798 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10082516 2024-09-16T03:39:19.798 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.798 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.798 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275909+0000", 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4197e-05 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277232+0000", 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013231009999999999 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277256+0000", 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3428e-05 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.799 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1572 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'33, mlcod=48'29)", 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 93.577613420000006, 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10216886, 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.800 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:19.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:19.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8400000000000004e-07 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175107+0000", 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.578e-06 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175114+0000", 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.888e-06 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175119+0000", 2024-09-16T03:39:19.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.074e-06 2024-09-16T03:39:19.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276662+0000", 2024-09-16T03:39:19.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101542753 2024-09-16T03:39:19.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276674+0000", 2024-09-16T03:39:19.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1960000000000001e-05 2024-09-16T03:39:19.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277258+0000", 2024-09-16T03:39:19.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058431900000000003 2024-09-16T03:39:19.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277270+0000", 2024-09-16T03:39:19.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2604e-05 2024-09-16T03:39:19.804 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.804 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.804 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.805 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.805 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "replica_scrub(pg: 4.15,from:0'0,to:60'33,epoch:61/50,start:4:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:39:19.805 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:39:19.805 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 93.075692165999996, 2024-09-16T03:39:19.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10023048699999999, 2024-09-16T03:39:19.805 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.805 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.805 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:19.805 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.805 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.805 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:39:19.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:39:19.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677023+0000", 2024-09-16T03:39:19.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8599999999999999e-07 2024-09-16T03:39:19.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.807 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677027+0000", 2024-09-16T03:39:19.807 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7359999999999999e-06 2024-09-16T03:39:19.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.807 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677042+0000", 2024-09-16T03:39:19.807 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4888e-05 2024-09-16T03:39:19.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.677052+0000", 2024-09-16T03:39:19.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7750000000000004e-06 2024-09-16T03:39:19.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777182+0000", 2024-09-16T03:39:19.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100130394 2024-09-16T03:39:19.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777195+0000", 2024-09-16T03:39:19.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3373999999999999e-05 2024-09-16T03:39:19.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.777253+0000", 2024-09-16T03:39:19.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7933999999999998e-05 2024-09-16T03:39:19.809 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.809 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.809 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.810 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2454 4.15 e84/50 4:aff4cc41:::smithi15736696-11:head v 84'41, mlcod=79'39)", 2024-09-16T03:39:19.810 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:39:19.810 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.289476058000005, 2024-09-16T03:39:19.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101244889, 2024-09-16T03:39:19.810 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.810 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.810 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.810 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:39:19.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:39:19.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:39:19.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2899999999999999e-07 2024-09-16T03:39:19.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.812 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463249+0000", 2024-09-16T03:39:19.812 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8039999999999996e-06 2024-09-16T03:39:19.812 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.812 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.812 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.812 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463267+0000", 2024-09-16T03:39:19.812 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7955e-05 2024-09-16T03:39:19.812 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.812 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.812 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.812 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563340+0000", 2024-09-16T03:39:19.812 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100072999 2024-09-16T03:39:19.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.813 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563396+0000", 2024-09-16T03:39:19.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6141999999999999e-05 2024-09-16T03:39:19.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.813 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563414+0000", 2024-09-16T03:39:19.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8088999999999999e-05 2024-09-16T03:39:19.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.564427+0000", 2024-09-16T03:39:19.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010127230000000001 2024-09-16T03:39:19.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.564484+0000", 2024-09-16T03:39:19.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6747999999999999e-05 2024-09-16T03:39:19.814 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.814 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.814 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.815 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2467 4.5 e85/50)", 2024-09-16T03:39:19.815 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:39:19.815 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.601583931999997, 2024-09-16T03:39:19.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102439455, 2024-09-16T03:39:19.815 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.815 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.815 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.815 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:39:19.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.816 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:39:19.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.816 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151134+0000", 2024-09-16T03:39:19.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2030000000000001e-06 2024-09-16T03:39:19.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151480+0000", 2024-09-16T03:39:19.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034613999999999999 2024-09-16T03:39:19.817 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151500+0000", 2024-09-16T03:39:19.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9582999999999999e-05 2024-09-16T03:39:19.817 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151513+0000", 2024-09-16T03:39:19.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251653+0000", 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10014007900000001 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251679+0000", 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5721e-05 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253532+0000", 2024-09-16T03:39:19.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001852701 2024-09-16T03:39:19.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.819 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.819 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.819 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253571+0000", 2024-09-16T03:39:19.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8677999999999998e-05 2024-09-16T03:39:19.819 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.819 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.819 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.819 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.819 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2468 4.5 e85/50)", 2024-09-16T03:39:19.819 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:39:19.819 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.600115402999997, 2024-09-16T03:39:19.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101505239, 2024-09-16T03:39:19.820 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.820 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.820 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.820 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:39:19.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:39:19.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.821 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152601+0000", 2024-09-16T03:39:19.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6869999999999999e-06 2024-09-16T03:39:19.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.821 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152719+0000", 2024-09-16T03:39:19.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117508 2024-09-16T03:39:19.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.822 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.822 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.822 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152732+0000", 2024-09-16T03:39:19.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:39:19.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.822 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.822 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.822 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152740+0000", 2024-09-16T03:39:19.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7090000000000002e-06 2024-09-16T03:39:19.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252704+0000", 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099964259999999999 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252727+0000", 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3601e-05 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254075+0000", 2024-09-16T03:39:19.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00134772 2024-09-16T03:39:19.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254105+0000", 2024-09-16T03:39:19.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9802e-05 2024-09-16T03:39:19.824 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.824 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.824 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.824 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2623 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'62, mlcod=85'60)", 2024-09-16T03:39:19.825 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:19.825 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.403477772999999, 2024-09-16T03:39:19.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.104590794, 2024-09-16T03:39:19.825 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.825 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.825 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.825 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.825 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.825 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.825 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:19.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.826 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.826 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.826 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:19.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.826 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.826 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.826 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:19.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1899999999999999e-07 2024-09-16T03:39:19.827 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.827 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.827 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.827 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349496+0000", 2024-09-16T03:39:19.827 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000258934 2024-09-16T03:39:19.827 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.827 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.827 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.827 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349506+0000", 2024-09-16T03:39:19.827 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9389999999999992e-06 2024-09-16T03:39:19.827 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.827 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349513+0000", 2024-09-16T03:39:19.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6209999999999998e-06 2024-09-16T03:39:19.828 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.828 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449668+0000", 2024-09-16T03:39:19.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015550099999999 2024-09-16T03:39:19.828 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.828 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449695+0000", 2024-09-16T03:39:19.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6253000000000001e-05 2024-09-16T03:39:19.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453784+0000", 2024-09-16T03:39:19.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040896609999999996 2024-09-16T03:39:19.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453828+0000", 2024-09-16T03:39:19.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3665999999999998e-05 2024-09-16T03:39:19.829 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.830 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.830 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.830 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2624 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'63, mlcod=85'60)", 2024-09-16T03:39:19.830 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:39:19.830 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.402130534999998, 2024-09-16T03:39:19.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103286796, 2024-09-16T03:39:19.830 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.830 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.830 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.830 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.831 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.831 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.831 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:39:19.831 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.831 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.831 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.831 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.831 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:39:19.831 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.831 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.831 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.831 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.832 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350586+0000", 2024-09-16T03:39:19.832 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0020000000000001e-06 2024-09-16T03:39:19.832 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.832 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.832 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.832 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350794+0000", 2024-09-16T03:39:19.832 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020824099999999999 2024-09-16T03:39:19.832 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.832 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350805+0000", 2024-09-16T03:39:19.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1443999999999999e-05 2024-09-16T03:39:19.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450876+0000", 2024-09-16T03:39:19.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100070612 2024-09-16T03:39:19.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450960+0000", 2024-09-16T03:39:19.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4444999999999995e-05 2024-09-16T03:39:19.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450980+0000", 2024-09-16T03:39:19.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9728000000000002e-05 2024-09-16T03:39:19.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453831+0000", 2024-09-16T03:39:19.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002850923 2024-09-16T03:39:19.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453871+0000", 2024-09-16T03:39:19.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0401000000000002e-05 2024-09-16T03:39:19.835 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.835 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.835 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.835 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2626 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'65, mlcod=85'60)", 2024-09-16T03:39:19.835 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:39:19.836 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 59.301426558000003, 2024-09-16T03:39:19.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100625902, 2024-09-16T03:39:19.836 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.836 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.836 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.836 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.836 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.836 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.836 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:39:19.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:39:19.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451289+0000", 2024-09-16T03:39:19.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7100000000000002e-07 2024-09-16T03:39:19.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451295+0000", 2024-09-16T03:39:19.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2890000000000003e-06 2024-09-16T03:39:19.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451300+0000", 2024-09-16T03:39:19.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0710000000000002e-06 2024-09-16T03:39:19.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551399+0000", 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10009881 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551482+0000", 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2738000000000006e-05 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551496+0000", 2024-09-16T03:39:19.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4394e-05 2024-09-16T03:39:19.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551881+0000", 2024-09-16T03:39:19.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038522300000000002 2024-09-16T03:39:19.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551914+0000", 2024-09-16T03:39:19.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2906e-05 2024-09-16T03:39:19.840 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.841 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.841 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.841 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.841 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2661 4.15 e91/50 4:aff4cc41:::smithi15736696-11:head v 91'52, mlcod=88'48)", 2024-09-16T03:39:19.841 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:39:19.841 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.343530305000002, 2024-09-16T03:39:19.841 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10164756699999999, 2024-09-16T03:39:19.841 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.841 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.841 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.841 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:39:19.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:39:19.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409186+0000", 2024-09-16T03:39:19.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-06 2024-09-16T03:39:19.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409568+0000", 2024-09-16T03:39:19.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038196600000000002 2024-09-16T03:39:19.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409578+0000", 2024-09-16T03:39:19.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.781e-06 2024-09-16T03:39:19.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409583+0000", 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3070000000000002e-06 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409633+0000", 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0358000000000001e-05 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409649+0000", 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5875999999999998e-05 2024-09-16T03:39:19.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510800+0000", 2024-09-16T03:39:19.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101150728 2024-09-16T03:39:19.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510832+0000", 2024-09-16T03:39:19.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2500999999999999e-05 2024-09-16T03:39:19.845 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.845 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.845 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2662 4.15 e91/50 4:aff4cc41:::smithi15736696-11:head v 91'53, mlcod=88'48)", 2024-09-16T03:39:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:39:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.342322355, 2024-09-16T03:39:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100650287, 2024-09-16T03:39:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:39:19.846 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:39:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410394+0000", 2024-09-16T03:39:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6299999999999993e-07 2024-09-16T03:39:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410399+0000", 2024-09-16T03:39:19.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9400000000000001e-06 2024-09-16T03:39:19.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410407+0000", 2024-09-16T03:39:19.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8859999999999993e-06 2024-09-16T03:39:19.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410412+0000", 2024-09-16T03:39:19.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9339999999999997e-06 2024-09-16T03:39:19.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510516+0000", 2024-09-16T03:39:19.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100103159 2024-09-16T03:39:19.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510536+0000", 2024-09-16T03:39:19.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0072999999999999e-05 2024-09-16T03:39:19.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.511018+0000", 2024-09-16T03:39:19.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00048191899999999998 2024-09-16T03:39:19.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.511043+0000", 2024-09-16T03:39:19.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5412999999999999e-05 2024-09-16T03:39:19.850 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.850 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.851 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.851 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.851 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.851 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2914 4.5 e98/50 4:a4fbca73:::smithi15736696-39:head v 98'72, mlcod=96'69)", 2024-09-16T03:39:19.851 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:30.471084+0000", 2024-09-16T03:39:19.851 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 49.281630782999997, 2024-09-16T03:39:19.851 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10075181900000001, 2024-09-16T03:39:19.851 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.851 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.852 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.852 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471084+0000", 2024-09-16T03:39:19.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471084+0000", 2024-09-16T03:39:19.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471087+0000", 2024-09-16T03:39:19.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3580000000000001e-06 2024-09-16T03:39:19.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471099+0000", 2024-09-16T03:39:19.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.255e-05 2024-09-16T03:39:19.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471116+0000", 2024-09-16T03:39:19.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6867999999999998e-05 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471128+0000", 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2313999999999999e-05 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571266+0000", 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013743999999999 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571286+0000", 2024-09-16T03:39:19.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0313999999999999e-05 2024-09-16T03:39:19.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571787+0000", 2024-09-16T03:39:19.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00050111600000000002 2024-09-16T03:39:19.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571836+0000", 2024-09-16T03:39:19.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8859000000000001e-05 2024-09-16T03:39:19.855 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.856 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.856 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.856 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:3970 4.5 e130/110 4:a11e1d7c:::smithi15736696-43:head v 130'83, mlcod=113'81)", 2024-09-16T03:39:19.856 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:39:19.856 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.490058724000001, 2024-09-16T03:39:19.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102913225, 2024-09-16T03:39:19.856 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:19.856 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:19.856 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:19.856 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:19.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:19.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:39:19.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:19.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:39:19.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:19.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:19.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262657+0000", 2024-09-16T03:39:19.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.72e-07 2024-09-16T03:39:19.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:19.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262783+0000", 2024-09-16T03:39:19.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012681199999999999 2024-09-16T03:39:19.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:19.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262792+0000", 2024-09-16T03:39:19.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3839999999999997e-06 2024-09-16T03:39:19.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262799+0000", 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4100000000000002e-06 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.362958+0000", 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100158395 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.362977+0000", 2024-09-16T03:39:19.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9741e-05 2024-09-16T03:39:19.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:19.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.365511+0000", 2024-09-16T03:39:19.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0025332499999999999 2024-09-16T03:39:19.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:19.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:19.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:19.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.365570+0000", 2024-09-16T03:39:19.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8961e-05 2024-09-16T03:39:19.860 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.860 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.860 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.861 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:19.861 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:19.861 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:19.862 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:39:19.863 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:19.864 INFO:tasks.rados.rados.0.smithi157.stdout:1802: finishing write tid 4 to smithi15736696-20 2024-09-16T03:39:19.864 INFO:tasks.rados.rados.0.smithi157.stdout:1802: oid 20 updating version 0 to 59 2024-09-16T03:39:19.864 INFO:tasks.rados.rados.0.smithi157.stdout:1802: oid 20 version 59 is already newer than 58 2024-09-16T03:39:19.864 INFO:tasks.rados.rados.0.smithi157.stdout:1802: oid 20 updating version 59 to 60 2024-09-16T03:39:19.864 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 60 (ObjNum 399 snap 0 seq_num 399) dirty exists 2024-09-16T03:39:19.864 INFO:tasks.rados.rados.0.smithi157.stdout:1802: left oid 20 (ObjNum 399 snap 0 seq_num 399) 2024-09-16T03:39:19.864 INFO:tasks.rados.rados.0.smithi157.stdout:1807: finishing write tid 1 to smithi15736696-13 2024-09-16T03:39:19.864 INFO:tasks.rados.rados.0.smithi157.stdout:1792: done (9 left) 2024-09-16T03:39:19.864 INFO:tasks.rados.rados.0.smithi157.stdout:1796: done (8 left) 2024-09-16T03:39:19.864 INFO:tasks.rados.rados.0.smithi157.stdout:1802: done (7 left) 2024-09-16T03:39:19.864 INFO:tasks.rados.rados.0.smithi157.stdout:1803: done (6 left) 2024-09-16T03:39:19.864 INFO:tasks.rados.rados.0.smithi157.stdout:1804: done (5 left) 2024-09-16T03:39:19.865 INFO:tasks.rados.rados.0.smithi157.stdout:1805: done (4 left) 2024-09-16T03:39:19.865 INFO:tasks.rados.rados.0.smithi157.stdout:1806: done (3 left) 2024-09-16T03:39:19.865 INFO:tasks.rados.rados.0.smithi157.stdout:1808: done (2 left) 2024-09-16T03:39:19.865 INFO:tasks.rados.rados.0.smithi157.stdout:1809: tier_evict oid 7 2024-09-16T03:39:19.865 INFO:tasks.rados.rados.0.smithi157.stdout:1809: tier_evict oid 7 snap -1 2024-09-16T03:39:19.865 INFO:tasks.rados.rados.0.smithi157.stdout:1810: set_chunk oid 38 target oid 2024-09-16T03:39:19.865 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:19.860+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:19.962 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:19.961+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:20.008 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:20.008 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:20.008 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:39:20.008 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:20.017 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:39:20.101 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:20.100+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:20.160 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:20.160 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:20.160 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:20.160 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:20.160 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:20.164 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:20.163+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:20.168 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:39:20.265 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:20.263+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:20.314 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:20.314 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:20.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:20.314 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:20.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.314 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T03:39:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:39:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 149.234630342, 2024-09-16T03:39:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.88203620400000005, 2024-09-16T03:39:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:39:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:39:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:39:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076767+0000", 2024-09-16T03:39:20.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9300000000000002e-06 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076774+0000", 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6630000000000004e-06 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076785+0000", 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1221e-05 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076796+0000", 2024-09-16T03:39:20.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0428000000000001e-05 2024-09-16T03:39:20.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076856+0000", 2024-09-16T03:39:20.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:39:20.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:20.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076863+0000", 2024-09-16T03:39:20.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9840000000000004e-06 2024-09-16T03:39:20.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855770+0000", 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.77890736999999999 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855902+0000", 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013207 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.856054+0000", 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015220899999999999 2024-09-16T03:39:20.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.858407+0000", 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023530069999999998 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958598+0000", 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100190194 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958716+0000", 2024-09-16T03:39:20.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118589 2024-09-16T03:39:20.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958761+0000", 2024-09-16T03:39:20.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5120000000000002e-05 2024-09-16T03:39:20.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958802+0000", 2024-09-16T03:39:20.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0383999999999998e-05 2024-09-16T03:39:20.321 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.321 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.321 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 145.716214165, 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70508991300000001, 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 52 2024-09-16T03:39:20.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6300000000000005e-07 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595187+0000", 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9790000000000001e-06 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595194+0000", 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6490000000000002e-06 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595208+0000", 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4139e-05 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801931+0000", 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.206723083 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801990+0000", 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8625999999999997e-05 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802066+0000", 2024-09-16T03:39:20.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6553999999999999e-05 2024-09-16T03:39:20.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802504+0000", 2024-09-16T03:39:20.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043775200000000002 2024-09-16T03:39:20.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.199998+0000", 2024-09-16T03:39:20.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39749365599999997 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300151+0000", 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015310099999999 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300210+0000", 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9376999999999997e-05 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300271+0000", 2024-09-16T03:39:20.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1334000000000006e-05 2024-09-16T03:39:20.328 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.328 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.328 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.328 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.328 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:20.328 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 143.94132318199999, 2024-09-16T03:39:20.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60623559900000001, 2024-09-16T03:39:20.328 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.328 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.328 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 71 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:20.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:20.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:39:20.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370077+0000", 2024-09-16T03:39:20.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9940000000000002e-06 2024-09-16T03:39:20.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370081+0000", 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5850000000000001e-06 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370087+0000", 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673522+0000", 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.303434969 2024-09-16T03:39:20.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673567+0000", 2024-09-16T03:39:20.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5080000000000002e-05 2024-09-16T03:39:20.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673651+0000", 2024-09-16T03:39:20.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3652999999999993e-05 2024-09-16T03:39:20.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775115+0000", 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101464178 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975918+0000", 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20080318699999999 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976254+0000", 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033619200000000002 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976285+0000", 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0679000000000002e-05 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976308+0000", 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3051e-05 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.334 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 143.94129982199999, 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60632925100000001, 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 72 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:20.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:20.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:20.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:39:20.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370150+0000", 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3906999999999997e-05 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370157+0000", 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370162+0000", 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7249999999999997e-06 2024-09-16T03:39:20.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673969+0000", 2024-09-16T03:39:20.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30380713199999998 2024-09-16T03:39:20.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674030+0000", 2024-09-16T03:39:20.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1141000000000003e-05 2024-09-16T03:39:20.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674124+0000", 2024-09-16T03:39:20.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3859999999999999e-05 2024-09-16T03:39:20.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775687+0000", 2024-09-16T03:39:20.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10156293299999999 2024-09-16T03:39:20.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976103+0000", 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20041631400000001 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976355+0000", 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000251611 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976385+0000", 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0883999999999998e-05 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976425+0000", 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9666000000000003e-05 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 143.941227338, 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60763126999999995, 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.342 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.343 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.343 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.343 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.343 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.343 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 73 2024-09-16T03:39:20.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.343 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:39:20.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:39:20.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370169+0000", 2024-09-16T03:39:20.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6199999999999998e-07 2024-09-16T03:39:20.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370171+0000", 2024-09-16T03:39:20.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:39:20.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370175+0000", 2024-09-16T03:39:20.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.596e-06 2024-09-16T03:39:20.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470295+0000", 2024-09-16T03:39:20.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100119967 2024-09-16T03:39:20.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775148+0000", 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30485311799999998 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775203+0000", 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4592999999999997e-05 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775293+0000", 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0859999999999994e-05 2024-09-16T03:39:20.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976223+0000", 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20092996599999999 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976463+0000", 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000239695 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977730+0000", 2024-09-16T03:39:20.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012667690000000001 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977773+0000", 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2604000000000001e-05 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977800+0000", 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7137000000000001e-05 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 143.84108848299999, 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60633810099999996, 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 74 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470308+0000", 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0950000000000001e-06 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470313+0000", 2024-09-16T03:39:20.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:39:20.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470320+0000", 2024-09-16T03:39:20.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5089999999999999e-06 2024-09-16T03:39:20.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470327+0000", 2024-09-16T03:39:20.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976483+0000", 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50615579499999996 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976533+0000", 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0096000000000003e-05 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976643+0000", 2024-09-16T03:39:20.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109481 2024-09-16T03:39:20.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977808+0000", 2024-09-16T03:39:20.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011649869999999999 2024-09-16T03:39:20.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.981699+0000", 2024-09-16T03:39:20.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003891579 2024-09-16T03:39:20.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076496+0000", 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.094797333999999997 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076583+0000", 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6571999999999997e-05 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076645+0000", 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2331999999999999e-05 2024-09-16T03:39:20.354 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 143.84105834799999, 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71251765099999997, 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.355 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 75 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470338+0000", 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2500000000000005e-07 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470389+0000", 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.134e-05 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470397+0000", 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8140000000000001e-06 2024-09-16T03:39:20.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570537+0000", 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100139425 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977018+0000", 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40648102899999999 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977063+0000", 2024-09-16T03:39:20.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5247000000000002e-05 2024-09-16T03:39:20.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977143+0000", 2024-09-16T03:39:20.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0054000000000003e-05 2024-09-16T03:39:20.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.978132+0000", 2024-09-16T03:39:20.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00098896399999999995 2024-09-16T03:39:20.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182725+0000", 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.204592626 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182783+0000", 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8285000000000002e-05 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182825+0000", 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2398999999999998e-05 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182855+0000", 2024-09-16T03:39:20.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9842999999999999e-05 2024-09-16T03:39:20.361 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.361 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.361 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.361 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.361 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:39:20.361 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 143.740831781, 2024-09-16T03:39:20.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71263271800000005, 2024-09-16T03:39:20.361 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.362 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.362 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.362 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.362 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.362 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.362 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 76 2024-09-16T03:39:20.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.362 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:39:20.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570566+0000", 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570577+0000", 2024-09-16T03:39:20.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0672999999999999e-05 2024-09-16T03:39:20.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570593+0000", 2024-09-16T03:39:20.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5979999999999999e-05 2024-09-16T03:39:20.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570604+0000", 2024-09-16T03:39:20.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.077e-05 2024-09-16T03:39:20.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979890+0000", 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40928624200000002 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979945+0000", 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5651e-05 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980038+0000", 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2603999999999996e-05 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980669+0000", 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063121900000000003 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283050+0000", 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30238120800000001 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283132+0000", 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1841999999999993e-05 2024-09-16T03:39:20.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283168+0000", 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5438999999999997e-05 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283197+0000", 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8960000000000001e-05 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.368 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.368 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:39:20.368 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 141.92629668199999, 2024-09-16T03:39:20.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70799990400000001, 2024-09-16T03:39:20.368 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.368 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.368 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.368 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.368 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.368 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.368 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 94 2024-09-16T03:39:20.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.368 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385100+0000", 2024-09-16T03:39:20.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9599999999999998e-07 2024-09-16T03:39:20.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385104+0000", 2024-09-16T03:39:20.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7579999999999998e-06 2024-09-16T03:39:20.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385128+0000", 2024-09-16T03:39:20.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4700999999999999e-05 2024-09-16T03:39:20.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385139+0000", 2024-09-16T03:39:20.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0587e-05 2024-09-16T03:39:20.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787499+0000", 2024-09-16T03:39:20.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40235976099999998 2024-09-16T03:39:20.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787545+0000", 2024-09-16T03:39:20.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6418999999999997e-05 2024-09-16T03:39:20.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787623+0000", 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7897000000000003e-05 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888605+0000", 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100982321 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990721+0000", 2024-09-16T03:39:20.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102115922 2024-09-16T03:39:20.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093019+0000", 2024-09-16T03:39:20.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10229788500000001 2024-09-16T03:39:20.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093051+0000", 2024-09-16T03:39:20.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2255999999999997e-05 2024-09-16T03:39:20.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093099+0000", 2024-09-16T03:39:20.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7500999999999998e-05 2024-09-16T03:39:20.374 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.374 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.374 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.374 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.374 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 141.92624535900001, 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80404594699999998, 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 95 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:39:20.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:39:20.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385151+0000", 2024-09-16T03:39:20.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:39:20.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385165+0000", 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4175999999999999e-05 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385175+0000", 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385183+0000", 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:39:20.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888643+0000", 2024-09-16T03:39:20.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50345988200000003 2024-09-16T03:39:20.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888695+0000", 2024-09-16T03:39:20.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2834999999999998e-05 2024-09-16T03:39:20.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888802+0000", 2024-09-16T03:39:20.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010697899999999999 2024-09-16T03:39:20.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990658+0000", 2024-09-16T03:39:20.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185585699999999 2024-09-16T03:39:20.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990917+0000", 2024-09-16T03:39:20.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025853600000000002 2024-09-16T03:39:20.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189054+0000", 2024-09-16T03:39:20.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198136812 2024-09-16T03:39:20.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189135+0000", 2024-09-16T03:39:20.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1482000000000004e-05 2024-09-16T03:39:20.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189196+0000", 2024-09-16T03:39:20.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1175999999999998e-05 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 141.926204732, 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80412488400000004, 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.381 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 96 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385192+0000", 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385267+0000", 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5535000000000001e-05 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385277+0000", 2024-09-16T03:39:20.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6239999999999997e-06 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385284+0000", 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4800000000000004e-06 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989283+0000", 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60399849500000002 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989410+0000", 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127298 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989615+0000", 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000205045 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991036+0000", 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001421042 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189228+0000", 2024-09-16T03:39:20.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19819188500000001 2024-09-16T03:39:20.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189249+0000", 2024-09-16T03:39:20.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0996999999999999e-05 2024-09-16T03:39:20.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189293+0000", 2024-09-16T03:39:20.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3912000000000002e-05 2024-09-16T03:39:20.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189316+0000", 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2739e-05 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 141.92610100100001, 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80642264100000005, 2024-09-16T03:39:20.387 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.388 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.388 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.388 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.388 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.388 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.388 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 97 2024-09-16T03:39:20.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.388 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:39:20.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:39:20.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385296+0000", 2024-09-16T03:39:20.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.006e-06 2024-09-16T03:39:20.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385301+0000", 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2630000000000003e-06 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385308+0000", 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6030000000000001e-06 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485459+0000", 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150929 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990238+0000", 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50477992000000005 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990287+0000", 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8863999999999998e-05 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990377+0000", 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9900000000000003e-05 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093109+0000", 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102732055 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189351+0000", 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096241766000000006 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191570+0000", 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00221915 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191653+0000", 2024-09-16T03:39:20.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3046000000000004e-05 2024-09-16T03:39:20.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191717+0000", 2024-09-16T03:39:20.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4139000000000004e-05 2024-09-16T03:39:20.393 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.393 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.393 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.394 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.394 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:39:20.394 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 141.82591054400001, 2024-09-16T03:39:20.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90688294999999997, 2024-09-16T03:39:20.394 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.394 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.394 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.394 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.394 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.394 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.394 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 98 2024-09-16T03:39:20.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.394 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485487+0000", 2024-09-16T03:39:20.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1799999999999999e-06 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485498+0000", 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0648999999999999e-05 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485514+0000", 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5943999999999998e-05 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485525+0000", 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.065e-05 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991102+0000", 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50557748400000002 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991161+0000", 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8801999999999997e-05 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991255+0000", 2024-09-16T03:39:20.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3936000000000003e-05 2024-09-16T03:39:20.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093112+0000", 2024-09-16T03:39:20.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185709499999999 2024-09-16T03:39:20.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291911+0000", 2024-09-16T03:39:20.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19879897699999999 2024-09-16T03:39:20.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392167+0000", 2024-09-16T03:39:20.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10025634999999999 2024-09-16T03:39:20.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392249+0000", 2024-09-16T03:39:20.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2175000000000005e-05 2024-09-16T03:39:20.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392368+0000", 2024-09-16T03:39:20.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118708 2024-09-16T03:39:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:39:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 141.825856921, 2024-09-16T03:39:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90707203199999997, 2024-09-16T03:39:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.400 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.401 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.401 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.401 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 99 2024-09-16T03:39:20.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.401 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:39:20.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485540+0000", 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-06 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485661+0000", 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120835 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485680+0000", 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8885999999999998e-05 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485693+0000", 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3297e-05 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991546+0000", 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.505853098 2024-09-16T03:39:20.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991598+0000", 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1921e-05 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991681+0000", 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2502000000000004e-05 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093115+0000", 2024-09-16T03:39:20.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10143424299999999 2024-09-16T03:39:20.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291977+0000", 2024-09-16T03:39:20.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198862172 2024-09-16T03:39:20.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392541+0000", 2024-09-16T03:39:20.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100563758 2024-09-16T03:39:20.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392579+0000", 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7796999999999997e-05 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392611+0000", 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2073999999999998e-05 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.406 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:20.407 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 141.825678028, 2024-09-16T03:39:20.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0073149939999999, 2024-09-16T03:39:20.407 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.407 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.407 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.407 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.407 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.407 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.407 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 100 2024-09-16T03:39:20.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.408 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:20.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:20.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:20.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7400000000000003e-07 2024-09-16T03:39:20.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485721+0000", 2024-09-16T03:39:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4530000000000001e-06 2024-09-16T03:39:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485726+0000", 2024-09-16T03:39:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0590000000000002e-06 2024-09-16T03:39:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485729+0000", 2024-09-16T03:39:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:39:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092342+0000", 2024-09-16T03:39:20.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60661343700000003 2024-09-16T03:39:20.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092469+0000", 2024-09-16T03:39:20.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012658799999999999 2024-09-16T03:39:20.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092671+0000", 2024-09-16T03:39:20.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020206 2024-09-16T03:39:20.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392452+0000", 2024-09-16T03:39:20.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29978163699999999 2024-09-16T03:39:20.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392486+0000", 2024-09-16T03:39:20.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3736999999999999e-05 2024-09-16T03:39:20.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492812+0000", 2024-09-16T03:39:20.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100325621 2024-09-16T03:39:20.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492901+0000", 2024-09-16T03:39:20.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8702000000000006e-05 2024-09-16T03:39:20.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.493033+0000", 2024-09-16T03:39:20.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000132185 2024-09-16T03:39:20.414 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.414 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.414 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.415 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.415 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:39:20.415 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 139.65052750699999, 2024-09-16T03:39:20.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60874581500000002, 2024-09-16T03:39:20.415 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.415 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.415 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.415 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.415 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.415 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.415 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 123 2024-09-16T03:39:20.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.416 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:39:20.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:39:20.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660869+0000", 2024-09-16T03:39:20.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9400000000000005e-07 2024-09-16T03:39:20.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660921+0000", 2024-09-16T03:39:20.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2312000000000003e-05 2024-09-16T03:39:20.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660927+0000", 2024-09-16T03:39:20.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9329999999999998e-06 2024-09-16T03:39:20.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660931+0000", 2024-09-16T03:39:20.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:39:20.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.967991+0000", 2024-09-16T03:39:20.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30706032700000002 2024-09-16T03:39:20.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968039+0000", 2024-09-16T03:39:20.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7477e-05 2024-09-16T03:39:20.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968117+0000", 2024-09-16T03:39:20.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8589000000000003e-05 2024-09-16T03:39:20.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067678+0000", 2024-09-16T03:39:20.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099560406000000004 2024-09-16T03:39:20.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170175+0000", 2024-09-16T03:39:20.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102496927 2024-09-16T03:39:20.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269491+0000", 2024-09-16T03:39:20.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099316205000000005 2024-09-16T03:39:20.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269574+0000", 2024-09-16T03:39:20.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2903000000000002e-05 2024-09-16T03:39:20.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269614+0000", 2024-09-16T03:39:20.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0188000000000002e-05 2024-09-16T03:39:20.422 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.422 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.422 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.422 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:20.422 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:39:20.422 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 139.65045880900001, 2024-09-16T03:39:20.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60879245500000001, 2024-09-16T03:39:20.423 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.423 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.423 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.423 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.423 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:20.423 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:20.423 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 124 2024-09-16T03:39:20.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.423 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:39:20.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:39:20.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660938+0000", 2024-09-16T03:39:20.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:39:20.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660940+0000", 2024-09-16T03:39:20.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2529999999999999e-06 2024-09-16T03:39:20.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660943+0000", 2024-09-16T03:39:20.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.01e-06 2024-09-16T03:39:20.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660946+0000", 2024-09-16T03:39:20.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8159999999999998e-06 2024-09-16T03:39:20.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012417+0000", 2024-09-16T03:39:20.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.35147120599999998 2024-09-16T03:39:20.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012497+0000", 2024-09-16T03:39:20.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9963000000000007e-05 2024-09-16T03:39:20.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:20.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012585+0000", 2024-09-16T03:39:20.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:39:20.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067680+0000", 2024-09-16T03:39:20.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055095203000000002 2024-09-16T03:39:20.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170243+0000", 2024-09-16T03:39:20.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10256290799999999 2024-09-16T03:39:20.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269662+0000", 2024-09-16T03:39:20.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099418896000000007 2024-09-16T03:39:20.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269701+0000", 2024-09-16T03:39:20.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8829999999999999e-05 2024-09-16T03:39:20.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269729+0000", 2024-09-16T03:39:20.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8569e-05 2024-09-16T03:39:20.430 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.430 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.430 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.430 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1550 4.13 4:c8492bfc:::smithi15736696-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:39:20.430 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:20.430 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 94.644730668999998, 2024-09-16T03:39:20.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62706974999999998, 2024-09-16T03:39:20.431 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.431 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.431 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:20.431 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.431 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.431 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.431 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1550 2024-09-16T03:39:20.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.431 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:20.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:20.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:20.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0699999999999998e-07 2024-09-16T03:39:20.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666669+0000", 2024-09-16T03:39:20.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.393e-06 2024-09-16T03:39:20.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666673+0000", 2024-09-16T03:39:20.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:39:20.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666682+0000", 2024-09-16T03:39:20.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6510000000000007e-06 2024-09-16T03:39:20.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.913940+0000", 2024-09-16T03:39:20.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.24725763000000001 2024-09-16T03:39:20.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:20.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.914034+0000", 2024-09-16T03:39:20.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3923000000000002e-05 2024-09-16T03:39:20.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293650+0000", 2024-09-16T03:39:20.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.37961662200000001 2024-09-16T03:39:20.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293688+0000", 2024-09-16T03:39:20.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8096000000000003e-05 2024-09-16T03:39:20.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293735+0000", 2024-09-16T03:39:20.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6451e-05 2024-09-16T03:39:20.436 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.436 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.436 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.437 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:77 3.2 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:39:20.437 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:39:20.437 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 80.232854047999993, 2024-09-16T03:39:20.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.61452436499999996, 2024-09-16T03:39:20.437 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.437 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.437 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:20.437 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:39:20.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:39:20.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078543+0000", 2024-09-16T03:39:20.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.553e-06 2024-09-16T03:39:20.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078551+0000", 2024-09-16T03:39:20.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0790000000000001e-06 2024-09-16T03:39:20.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078564+0000", 2024-09-16T03:39:20.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2943e-05 2024-09-16T03:39:20.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178635+0000", 2024-09-16T03:39:20.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10007044599999999 2024-09-16T03:39:20.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178684+0000", 2024-09-16T03:39:20.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9592999999999998e-05 2024-09-16T03:39:20.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178747+0000", 2024-09-16T03:39:20.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2286000000000006e-05 2024-09-16T03:39:20.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4,5", 2024-09-16T03:39:20.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178857+0000", 2024-09-16T03:39:20.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110807 2024-09-16T03:39:20.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280364+0000", 2024-09-16T03:39:20.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101506518 2024-09-16T03:39:20.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280373+0000", 2024-09-16T03:39:20.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6899999999999998e-06 2024-09-16T03:39:20.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.369249+0000", 2024-09-16T03:39:20.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.088876172000000003 2024-09-16T03:39:20.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693044+0000", 2024-09-16T03:39:20.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.32379486499999999 2024-09-16T03:39:20.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693066+0000", 2024-09-16T03:39:20.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2413e-05 2024-09-16T03:39:20.443 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.443 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.444 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.444 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3119 4.13 4:c8492bfc:::smithi15736696-42:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:39:20.444 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:39:20.444 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.379956991999997, 2024-09-16T03:39:20.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.64286237700000004, 2024-09-16T03:39:20.444 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.444 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.444 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:20.444 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.445 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.445 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.445 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3119 2024-09-16T03:39:20.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.445 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:39:20.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:39:20.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931441+0000", 2024-09-16T03:39:20.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3599999999999999e-06 2024-09-16T03:39:20.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931449+0000", 2024-09-16T03:39:20.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4080000000000001e-06 2024-09-16T03:39:20.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931463+0000", 2024-09-16T03:39:20.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.488e-05 2024-09-16T03:39:20.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931473+0000", 2024-09-16T03:39:20.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8109999999999997e-06 2024-09-16T03:39:20.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.031604+0000", 2024-09-16T03:39:20.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013079499999999 2024-09-16T03:39:20.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.031668+0000", 2024-09-16T03:39:20.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4226999999999993e-05 2024-09-16T03:39:20.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.574301+0000", 2024-09-16T03:39:20.449 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54263289599999998 2024-09-16T03:39:20.449 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.449 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.449 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.449 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.449 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.449 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:20.450 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:20.373+0000 7f4214d04640 -1 osd.8 142 handle_osd_map injecting map crc failure 2024-09-16T03:39:20.451 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:39:20.452 INFO:tasks.rados.rados.0.smithi157.stdout:1810: set_chunk oid 38 offset: 1218560 length: 243200 target oid b595f19538cd60ceade5ba11d0db1f07eff23820ff1cafbf439b875b0bc21add offset: 0 snap -1 2024-09-16T03:39:20.453 INFO:tasks.rados.rados.0.smithi157.stdout:1811: tier_promote oid 11 2024-09-16T03:39:20.453 INFO:tasks.rados.rados.0.smithi157.stdout:1809: got (22) Invalid argument 2024-09-16T03:39:20.453 INFO:tasks.rados.rados.0.smithi157.stdout:1809: done (4 left) 2024-09-16T03:39:20.453 INFO:tasks.rados.rados.0.smithi157.stdout:1812: write oid 4 current snap is 0 2024-09-16T03:39:20.453 INFO:tasks.rados.rados.0.smithi157.stdout:1807: finishing write tid 2 to smithi15736696-13 2024-09-16T03:39:20.453 INFO:tasks.rados.rados.0.smithi157.stdout:1807: finishing write tid 3 to smithi15736696-13 2024-09-16T03:39:20.453 INFO:tasks.rados.rados.0.smithi157.stdout:1812: seq_num 403 ranges {773221=492056,1816008=503126} 2024-09-16T03:39:20.453 INFO:tasks.rados.rados.0.smithi157.stdout:1812: writing smithi15736696-4 from 773221 to 1265277 tid 1 2024-09-16T03:39:20.453 INFO:tasks.rados.rados.0.smithi157.stdout:1812: writing smithi15736696-4 from 1816008 to 2319134 tid 2 2024-09-16T03:39:20.453 INFO:tasks.rados.rados.0.smithi157.stdout:1813: set_chunk oid 25 target oid 2024-09-16T03:39:20.454 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:20.449+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:20.551 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:20.549+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:20.597 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:20.598 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:20.598 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:39:20.598 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:20.603 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:20.600+0000 7fd103e73640 -1 osd.2 143 handle_osd_map injecting map crc failure 2024-09-16T03:39:20.606 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:39:20.652 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:20.650+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:20.687 INFO:tasks.rados.rados.0.smithi157.stdout:1813: set_chunk oid 25 offset: 1469440 length: 1226752 target oid e78062fe11bded5b0dfbe3bf79dfdc9dee286cfe94341e7d51857de1af7a4ac6 offset: 0 snap -1 2024-09-16T03:39:20.687 INFO:tasks.rados.rados.0.smithi157.stdout:1814: write oid 22 current snap is 0 2024-09-16T03:39:20.687 INFO:tasks.rados.rados.0.smithi157.stdout:1807: finishing write tid 4 to smithi15736696-13 2024-09-16T03:39:20.687 INFO:tasks.rados.rados.0.smithi157.stdout:1807: oid 13 updating version 0 to 96 2024-09-16T03:39:20.687 INFO:tasks.rados.rados.0.smithi157.stdout:1807: oid 13 version 96 is already newer than 95 2024-09-16T03:39:20.687 INFO:tasks.rados.rados.0.smithi157.stdout:1807: oid 13 version 96 is already newer than 94 2024-09-16T03:39:20.688 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 13 v 96 (ObjNum 401 snap 0 seq_num 401) dirty exists 2024-09-16T03:39:20.688 INFO:tasks.rados.rados.0.smithi157.stdout:1807: left oid 13 (ObjNum 401 snap 0 seq_num 401) 2024-09-16T03:39:20.688 INFO:tasks.rados.rados.0.smithi157.stdout:1793: expect (ObjNum 306 snap 0 seq_num 306) 2024-09-16T03:39:20.713 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:20.713 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:20.713 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:20.713 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:20.713 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:20.722 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:39:20.762 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:20.761+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:20.791 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2887236 2024-09-16T03:39:20.801 INFO:tasks.rados.rados.0.smithi157.stdout:1811: got (2) No such file or directory 2024-09-16T03:39:20.801 INFO:tasks.rados.rados.0.smithi157.stdout:1811: got expected ENOENT (src dne) 2024-09-16T03:39:20.801 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 70 (ObjNum 113 snap 0 seq_num 2307889553) clean dne 2024-09-16T03:39:20.801 INFO:tasks.rados.rados.0.smithi157.stdout:1810: finishing set_chunk to oid 38 2024-09-16T03:39:20.801 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 41 (ObjNum 397 snap 0 seq_num 397) dirty exists 2024-09-16T03:39:20.801 INFO:tasks.rados.rados.0.smithi157.stdout:1813: finishing set_chunk to oid 25 2024-09-16T03:39:20.801 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 25 set_chunk returned error code -95 offset: 1469440 length: 1226752 2024-09-16T03:39:20.801 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 0 (ObjNum 295 snap 0 seq_num 295) dirty exists 2024-09-16T03:39:20.801 INFO:tasks.rados.rados.0.smithi157.stdout:1812: finishing write tid 1 to smithi15736696-4 2024-09-16T03:39:20.801 INFO:tasks.rados.rados.0.smithi157.stdout:1814: seq_num 404 ranges {790028=567305,1874490=763137} 2024-09-16T03:39:20.808 INFO:tasks.rados.rados.0.smithi157.stdout:1814: writing smithi15736696-22 from 790028 to 1357333 tid 1 2024-09-16T03:39:20.816 INFO:tasks.rados.rados.0.smithi157.stdout:1814: writing smithi15736696-22 from 1874490 to 2637627 tid 2 2024-09-16T03:39:20.816 INFO:tasks.rados.rados.0.smithi157.stdout:1793: done (6 left) 2024-09-16T03:39:20.816 INFO:tasks.rados.rados.0.smithi157.stdout:1807: done (5 left) 2024-09-16T03:39:20.816 INFO:tasks.rados.rados.0.smithi157.stdout:1810: done (4 left) 2024-09-16T03:39:20.817 INFO:tasks.rados.rados.0.smithi157.stdout:1811: done (3 left) 2024-09-16T03:39:20.817 INFO:tasks.rados.rados.0.smithi157.stdout:1813: done (2 left) 2024-09-16T03:39:20.817 INFO:tasks.rados.rados.0.smithi157.stdout:1815: write oid 44 current snap is 0 2024-09-16T03:39:20.817 INFO:tasks.rados.rados.0.smithi157.stdout:1815: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-09-16T03:39:20.821 INFO:tasks.rados.rados.0.smithi157.stdout:1815: writing smithi15736696-44 from 406835 to 1049389 tid 1 2024-09-16T03:39:20.823 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:20.821+0000 7f4212d00640 -1 osd.8 143 handle_osd_map injecting map crc failure 2024-09-16T03:39:20.826 INFO:tasks.rados.rados.0.smithi157.stdout:1815: writing smithi15736696-44 from 1532972 to 2039767 tid 2 2024-09-16T03:39:20.831 INFO:tasks.rados.rados.0.smithi157.stdout:1815: writing smithi15736696-44 from 2772312 to 2806835 tid 3 2024-09-16T03:39:20.831 INFO:tasks.rados.rados.0.smithi157.stdout:1816: read oid 19 snap -1 2024-09-16T03:39:20.831 INFO:tasks.rados.rados.0.smithi157.stdout:1816: expect deleted 2024-09-16T03:39:20.831 INFO:tasks.rados.rados.0.smithi157.stdout:1817: tier_evict oid 11 2024-09-16T03:39:20.831 INFO:tasks.rados.rados.0.smithi157.stdout:1817: tier_evict oid 11 snap -1 2024-09-16T03:39:20.831 INFO:tasks.rados.rados.0.smithi157.stdout:1818: read oid 29 snap -1 2024-09-16T03:39:20.831 INFO:tasks.rados.rados.0.smithi157.stdout:1818: expect (ObjNum 389 snap 0 seq_num 389) 2024-09-16T03:39:20.831 INFO:tasks.rados.rados.0.smithi157.stdout:1819: write (excl) oid 17 current snap is 0 2024-09-16T03:39:20.832 INFO:tasks.rados.rados.0.smithi157.stdout:1819: seq_num 406 ranges {423642=717803,1875101=766806} 2024-09-16T03:39:20.837 INFO:tasks.rados.rados.0.smithi157.stdout:1819: writing smithi15736696-17 from 423642 to 1141445 tid 1 2024-09-16T03:39:20.845 INFO:tasks.rados.rados.0.smithi157.stdout:1819: writing smithi15736696-17 from 1875101 to 2641907 tid 2 2024-09-16T03:39:20.845 INFO:tasks.rados.rados.0.smithi157.stdout:1820: read oid 14 snap -1 2024-09-16T03:39:20.845 INFO:tasks.rados.rados.0.smithi157.stdout:1820: expect (ObjNum 398 snap 0 seq_num 398) 2024-09-16T03:39:20.845 INFO:tasks.rados.rados.0.smithi157.stdout:1821: set_chunk oid 8 target oid 2024-09-16T03:39:20.853 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:20.851+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:20.869 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:20.869 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:20.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:20.869 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:20.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.870 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:64 4.5 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:20.870 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:39:20.870 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 128.60938246699999, 2024-09-16T03:39:20.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50484290399999998, 2024-09-16T03:39:20.870 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.870 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.870 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:20.870 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.870 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.870 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.870 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 64 2024-09-16T03:39:20.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.871 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:39:20.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257340+0000", 2024-09-16T03:39:20.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257341+0000", 2024-09-16T03:39:20.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:39:20.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257343+0000", 2024-09-16T03:39:20.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3889999999999999e-06 2024-09-16T03:39:20.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257346+0000", 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257349+0000", 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8559999999999998e-06 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357057+0000", 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099708121999999996 2024-09-16T03:39:20.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:20.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357156+0000", 2024-09-16T03:39:20.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9162999999999999e-05 2024-09-16T03:39:20.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762060+0000", 2024-09-16T03:39:20.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40490364400000001 2024-09-16T03:39:20.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762112+0000", 2024-09-16T03:39:20.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1882000000000002e-05 2024-09-16T03:39:20.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.762183+0000", 2024-09-16T03:39:20.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1297000000000005e-05 2024-09-16T03:39:20.875 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.875 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.875 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.875 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:465 4.5 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:20.875 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 119.85101769400001, 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60529649299999999, 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 465 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:39:20.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:39:20.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015706+0000", 2024-09-16T03:39:20.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.091e-06 2024-09-16T03:39:20.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015712+0000", 2024-09-16T03:39:20.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-16T03:39:20.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015724+0000", 2024-09-16T03:39:20.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1408e-05 2024-09-16T03:39:20.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.879 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015731+0000", 2024-09-16T03:39:20.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:39:20.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.879 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216049+0000", 2024-09-16T03:39:20.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200317895 2024-09-16T03:39:20.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.879 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216178+0000", 2024-09-16T03:39:20.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012848800000000001 2024-09-16T03:39:20.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:20.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620010+0000", 2024-09-16T03:39:20.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40383207199999999 2024-09-16T03:39:20.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620710+0000", 2024-09-16T03:39:20.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069988400000000001 2024-09-16T03:39:20.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620902+0000", 2024-09-16T03:39:20.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019197800000000001 2024-09-16T03:39:20.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620933+0000", 2024-09-16T03:39:20.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1513999999999997e-05 2024-09-16T03:39:20.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620968+0000", 2024-09-16T03:39:20.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4524999999999999e-05 2024-09-16T03:39:20.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.621002+0000", 2024-09-16T03:39:20.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3955000000000003e-05 2024-09-16T03:39:20.883 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.883 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.883 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.883 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.883 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:10 3.3 3:deac805c:::smithi15736696-44 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:20.883 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:39:20.883 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 114.459522718, 2024-09-16T03:39:20.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53191748100000003, 2024-09-16T03:39:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:39:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407202+0000", 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.855e-06 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407208+0000", 2024-09-16T03:39:20.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9819999999999996e-06 2024-09-16T03:39:20.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407220+0000", 2024-09-16T03:39:20.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2191999999999999e-05 2024-09-16T03:39:20.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407236+0000", 2024-09-16T03:39:20.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5698000000000001e-05 2024-09-16T03:39:20.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538167+0000", 2024-09-16T03:39:20.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.13093111499999999 2024-09-16T03:39:20.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538229+0000", 2024-09-16T03:39:20.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2441999999999996e-05 2024-09-16T03:39:20.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:39:20.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538346+0000", 2024-09-16T03:39:20.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011716400000000001 2024-09-16T03:39:20.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540757+0000", 2024-09-16T03:39:20.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024105089999999999 2024-09-16T03:39:20.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.639501+0000", 2024-09-16T03:39:20.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098743545000000002 2024-09-16T03:39:20.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939038+0000", 2024-09-16T03:39:20.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29953765700000001 2024-09-16T03:39:20.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939075+0000", 2024-09-16T03:39:20.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6600999999999997e-05 2024-09-16T03:39:20.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939118+0000", 2024-09-16T03:39:20.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2721e-05 2024-09-16T03:39:20.890 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.890 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.890 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.890 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:984 4.15 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:20.890 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:39:20.891 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 110.49460279, 2024-09-16T03:39:20.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.97836159599999994, 2024-09-16T03:39:20.891 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.891 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.891 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:20.891 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.891 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.891 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.891 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 984 2024-09-16T03:39:20.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.891 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:39:20.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:39:20.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372122+0000", 2024-09-16T03:39:20.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.934e-06 2024-09-16T03:39:20.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372129+0000", 2024-09-16T03:39:20.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8029999999999999e-06 2024-09-16T03:39:20.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372141+0000", 2024-09-16T03:39:20.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2006e-05 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472234+0000", 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100092985 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472318+0000", 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4257000000000003e-05 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:20.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472326+0000", 2024-09-16T03:39:20.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8960000000000003e-06 2024-09-16T03:39:20.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350293+0000", 2024-09-16T03:39:20.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.87796699899999997 2024-09-16T03:39:20.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350433+0000", 2024-09-16T03:39:20.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013999600000000001 2024-09-16T03:39:20.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350482+0000", 2024-09-16T03:39:20.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8720000000000001e-05 2024-09-16T03:39:20.896 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.896 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.896 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.896 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1000 4.15 4:aff4cc41:::smithi15736696-11:head [read 0~2873964,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:20.896 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:39:20.896 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 110.276163705, 2024-09-16T03:39:20.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.761707841, 2024-09-16T03:39:20.897 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.897 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.897 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:20.897 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.897 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.897 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.897 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1000 2024-09-16T03:39:20.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.897 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:39:20.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:39:20.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590560+0000", 2024-09-16T03:39:20.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:39:20.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590562+0000", 2024-09-16T03:39:20.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.17e-06 2024-09-16T03:39:20.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590568+0000", 2024-09-16T03:39:20.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7939999999999997e-06 2024-09-16T03:39:20.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590572+0000", 2024-09-16T03:39:20.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1300000000000001e-06 2024-09-16T03:39:20.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590619+0000", 2024-09-16T03:39:20.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7148999999999999e-05 2024-09-16T03:39:20.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:20.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590625+0000", 2024-09-16T03:39:20.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5679999999999996e-06 2024-09-16T03:39:20.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350858+0000", 2024-09-16T03:39:20.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76023295000000002 2024-09-16T03:39:20.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350973+0000", 2024-09-16T03:39:20.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115023 2024-09-16T03:39:20.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352267+0000", 2024-09-16T03:39:20.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012935970000000001 2024-09-16T03:39:20.902 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.902 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.902 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.902 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1001 4.15 4:aff4cc41:::smithi15736696-11:head [read 1254674~1619290,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:20.902 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:20.902 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 110.27614512, 2024-09-16T03:39:20.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76275827799999996, 2024-09-16T03:39:20.903 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.903 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.903 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:20.903 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.903 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.903 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.903 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1001 2024-09-16T03:39:20.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.903 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:20.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:20.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:20.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:39:20.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590580+0000", 2024-09-16T03:39:20.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:39:20.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590584+0000", 2024-09-16T03:39:20.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8800000000000001e-06 2024-09-16T03:39:20.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590587+0000", 2024-09-16T03:39:20.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6630000000000002e-06 2024-09-16T03:39:20.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590650+0000", 2024-09-16T03:39:20.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3950999999999998e-05 2024-09-16T03:39:20.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:20.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590654+0000", 2024-09-16T03:39:20.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.89e-06 2024-09-16T03:39:20.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.908 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.908 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352309+0000", 2024-09-16T03:39:20.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76165495699999997 2024-09-16T03:39:20.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.908 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.908 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352407+0000", 2024-09-16T03:39:20.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7188999999999994e-05 2024-09-16T03:39:20.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.353336+0000", 2024-09-16T03:39:20.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092939400000000005 2024-09-16T03:39:20.909 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.909 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.909 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.909 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1002 4.15 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~1527025] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:20.909 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:20.909 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 110.276115866, 2024-09-16T03:39:20.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85963523600000002, 2024-09-16T03:39:20.910 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.910 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.910 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:20.910 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.910 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.910 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.910 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1002 2024-09-16T03:39:20.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.910 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:20.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:20.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:20.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.15e-07 2024-09-16T03:39:20.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590609+0000", 2024-09-16T03:39:20.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.606e-06 2024-09-16T03:39:20.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590615+0000", 2024-09-16T03:39:20.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3040000000000001e-06 2024-09-16T03:39:20.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590618+0000", 2024-09-16T03:39:20.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.309e-06 2024-09-16T03:39:20.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590669+0000", 2024-09-16T03:39:20.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1048000000000003e-05 2024-09-16T03:39:20.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:20.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590673+0000", 2024-09-16T03:39:20.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.061e-06 2024-09-16T03:39:20.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.449932+0000", 2024-09-16T03:39:20.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85925979699999999 2024-09-16T03:39:20.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450032+0000", 2024-09-16T03:39:20.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9858000000000005e-05 2024-09-16T03:39:20.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450242+0000", 2024-09-16T03:39:20.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020993800000000001 2024-09-16T03:39:20.915 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.915 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.915 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.916 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2467 4.5 4:a11e1d7c:::smithi15736696-43:head [write 1772395~538068] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:20.916 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:39:20.916 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.875439494999995, 2024-09-16T03:39:20.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.56288137400000005, 2024-09-16T03:39:20.916 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.916 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.916 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.916 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.916 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.916 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.916 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2467 2024-09-16T03:39:20.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.917 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:39:20.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:39:20.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991285+0000", 2024-09-16T03:39:20.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0990000000000001e-06 2024-09-16T03:39:20.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991447+0000", 2024-09-16T03:39:20.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016170099999999999 2024-09-16T03:39:20.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991458+0000", 2024-09-16T03:39:20.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.128e-05 2024-09-16T03:39:20.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991464+0000", 2024-09-16T03:39:20.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3800000000000002e-06 2024-09-16T03:39:20.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150823+0000", 2024-09-16T03:39:20.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15935943799999999 2024-09-16T03:39:20.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150899+0000", 2024-09-16T03:39:20.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5359999999999997e-05 2024-09-16T03:39:20.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:20.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151007+0000", 2024-09-16T03:39:20.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108317 2024-09-16T03:39:20.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253992+0000", 2024-09-16T03:39:20.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102985249 2024-09-16T03:39:20.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.353797+0000", 2024-09-16T03:39:20.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099805039999999998 2024-09-16T03:39:20.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554039+0000", 2024-09-16T03:39:20.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20024175899999999 2024-09-16T03:39:20.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554129+0000", 2024-09-16T03:39:20.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0067000000000004e-05 2024-09-16T03:39:20.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554165+0000", 2024-09-16T03:39:20.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5683999999999999e-05 2024-09-16T03:39:20.923 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.923 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.923 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.923 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2468 4.5 4:a11e1d7c:::smithi15736696-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:20.923 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:39:20.923 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.870018716000004, 2024-09-16T03:39:20.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.55765803899999999, 2024-09-16T03:39:20.924 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.924 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.924 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.924 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.924 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.924 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.924 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2468 2024-09-16T03:39:20.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.924 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:39:20.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:39:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996708+0000", 2024-09-16T03:39:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5149999999999998e-06 2024-09-16T03:39:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996921+0000", 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000213692 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996937+0000", 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5699999999999999e-05 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996946+0000", 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7159999999999992e-06 2024-09-16T03:39:20.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152362+0000", 2024-09-16T03:39:20.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15541682800000001 2024-09-16T03:39:20.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152421+0000", 2024-09-16T03:39:20.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-16T03:39:20.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:20.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152515+0000", 2024-09-16T03:39:20.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3659999999999994e-05 2024-09-16T03:39:20.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253995+0000", 2024-09-16T03:39:20.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10148022499999999 2024-09-16T03:39:20.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.453908+0000", 2024-09-16T03:39:20.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19991299700000001 2024-09-16T03:39:20.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554321+0000", 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100412977 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554344+0000", 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.262e-05 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554362+0000", 2024-09-16T03:39:20.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8539999999999999e-05 2024-09-16T03:39:20.930 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.930 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.930 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.930 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2469 4.5 4:a11e1d7c:::smithi15736696-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:20.930 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:39:20.931 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.869763352999996, 2024-09-16T03:39:20.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.75785074799999996, 2024-09-16T03:39:20.931 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.931 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.931 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.931 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.931 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.931 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.931 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2469 2024-09-16T03:39:20.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.931 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:39:20.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.932 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:39:20.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.932 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996960+0000", 2024-09-16T03:39:20.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3000000000000001e-07 2024-09-16T03:39:20.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996963+0000", 2024-09-16T03:39:20.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8679999999999999e-06 2024-09-16T03:39:20.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996967+0000", 2024-09-16T03:39:20.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.622e-06 2024-09-16T03:39:20.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996971+0000", 2024-09-16T03:39:20.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1429999999999998e-06 2024-09-16T03:39:20.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253536+0000", 2024-09-16T03:39:20.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25656582900000002 2024-09-16T03:39:20.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253632+0000", 2024-09-16T03:39:20.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6067e-05 2024-09-16T03:39:20.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:20.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253745+0000", 2024-09-16T03:39:20.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011256699999999999 2024-09-16T03:39:20.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254218+0000", 2024-09-16T03:39:20.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00047321400000000001 2024-09-16T03:39:20.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554397+0000", 2024-09-16T03:39:20.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30017854399999999 2024-09-16T03:39:20.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754557+0000", 2024-09-16T03:39:20.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200159848 2024-09-16T03:39:20.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754646+0000", 2024-09-16T03:39:20.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9370999999999995e-05 2024-09-16T03:39:20.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754810+0000", 2024-09-16T03:39:20.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016414500000000001 2024-09-16T03:39:20.939 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.939 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.939 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.939 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2470 4.5 4:a11e1d7c:::smithi15736696-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:20.940 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:20.940 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.869747024000006, 2024-09-16T03:39:20.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76022711799999998, 2024-09-16T03:39:20.940 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.940 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.940 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:20.940 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.940 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.940 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.940 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2470 2024-09-16T03:39:20.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.940 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:20.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:20.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.65e-07 2024-09-16T03:39:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996977+0000", 2024-09-16T03:39:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4079999999999999e-06 2024-09-16T03:39:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996981+0000", 2024-09-16T03:39:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-16T03:39:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996988+0000", 2024-09-16T03:39:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0890000000000002e-06 2024-09-16T03:39:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253947+0000", 2024-09-16T03:39:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25695931100000002 2024-09-16T03:39:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253985+0000", 2024-09-16T03:39:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7939999999999999e-05 2024-09-16T03:39:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757090+0000", 2024-09-16T03:39:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50310477499999995 2024-09-16T03:39:20.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757140+0000", 2024-09-16T03:39:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9567999999999997e-05 2024-09-16T03:39:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757203+0000", 2024-09-16T03:39:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3215999999999999e-05 2024-09-16T03:39:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.946 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2487 4.15 4:ad0b9852:::smithi15736696-16:head [write 1947841~685395] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:20.946 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:39:20.946 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.450702433999993, 2024-09-16T03:39:20.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.74140762199999999, 2024-09-16T03:39:20.946 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.946 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.946 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.946 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.946 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.946 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.946 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2487 2024-09-16T03:39:20.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416023+0000", 2024-09-16T03:39:20.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:39:20.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416341+0000", 2024-09-16T03:39:20.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031881699999999999 2024-09-16T03:39:20.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416353+0000", 2024-09-16T03:39:20.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.189e-05 2024-09-16T03:39:20.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416360+0000", 2024-09-16T03:39:20.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5719999999999999e-06 2024-09-16T03:39:20.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755578+0000", 2024-09-16T03:39:20.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33921859199999999 2024-09-16T03:39:20.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755639+0000", 2024-09-16T03:39:20.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0615000000000002e-05 2024-09-16T03:39:20.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:20.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755735+0000", 2024-09-16T03:39:20.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5798999999999995e-05 2024-09-16T03:39:20.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858633+0000", 2024-09-16T03:39:20.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10289859599999999 2024-09-16T03:39:20.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858695+0000", 2024-09-16T03:39:20.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2000000000000003e-05 2024-09-16T03:39:20.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157315+0000", 2024-09-16T03:39:20.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29861931600000002 2024-09-16T03:39:20.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157375+0000", 2024-09-16T03:39:20.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0268e-05 2024-09-16T03:39:20.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157428+0000", 2024-09-16T03:39:20.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2939999999999998e-05 2024-09-16T03:39:20.953 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.953 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.953 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.953 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2488 4.15 4:ad0b9852:::smithi15736696-16:head [write 3383215~196676] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:20.953 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:39:20.953 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.445029783999999, 2024-09-16T03:39:20.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73589615900000005, 2024-09-16T03:39:20.953 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.956 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.956 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.956 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.956 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.956 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.956 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2488 2024-09-16T03:39:20.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.956 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:39:20.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:39:20.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421695+0000", 2024-09-16T03:39:20.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1469999999999999e-06 2024-09-16T03:39:20.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421975+0000", 2024-09-16T03:39:20.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028008299999999998 2024-09-16T03:39:20.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421987+0000", 2024-09-16T03:39:20.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1294000000000001e-05 2024-09-16T03:39:20.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421992+0000", 2024-09-16T03:39:20.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9349999999999999e-06 2024-09-16T03:39:20.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757205+0000", 2024-09-16T03:39:20.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33521242800000001 2024-09-16T03:39:20.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757269+0000", 2024-09-16T03:39:20.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3761000000000002e-05 2024-09-16T03:39:20.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:20.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757376+0000", 2024-09-16T03:39:20.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010706599999999999 2024-09-16T03:39:20.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858684+0000", 2024-09-16T03:39:20.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101308255 2024-09-16T03:39:20.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858700+0000", 2024-09-16T03:39:20.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6138999999999999e-05 2024-09-16T03:39:20.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157491+0000", 2024-09-16T03:39:20.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29879056999999998 2024-09-16T03:39:20.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157546+0000", 2024-09-16T03:39:20.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5668999999999998e-05 2024-09-16T03:39:20.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157589+0000", 2024-09-16T03:39:20.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2812000000000003e-05 2024-09-16T03:39:20.962 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.963 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.963 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.963 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2489 4.15 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=7b,truncate 3579891] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:20.963 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:20.963 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.444720962000005, 2024-09-16T03:39:20.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73575981700000004, 2024-09-16T03:39:20.963 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.963 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.963 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.963 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.964 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.964 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.964 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2489 2024-09-16T03:39:20.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.964 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:20.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:20.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:20.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0200000000000002e-07 2024-09-16T03:39:20.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422005+0000", 2024-09-16T03:39:20.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7379999999999999e-06 2024-09-16T03:39:20.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422009+0000", 2024-09-16T03:39:20.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6219999999999998e-06 2024-09-16T03:39:20.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422016+0000", 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5789999999999998e-06 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858071+0000", 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.436054302 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858130+0000", 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9493000000000001e-05 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:20.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858226+0000", 2024-09-16T03:39:20.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6174000000000004e-05 2024-09-16T03:39:20.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:39:20.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10106483199999999 2024-09-16T03:39:20.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959639+0000", 2024-09-16T03:39:20.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034802999999999999 2024-09-16T03:39:20.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157640+0000", 2024-09-16T03:39:20.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19800045299999999 2024-09-16T03:39:20.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:20.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157702+0000", 2024-09-16T03:39:20.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2218000000000006e-05 2024-09-16T03:39:20.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157762+0000", 2024-09-16T03:39:20.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9774000000000001e-05 2024-09-16T03:39:20.970 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.970 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.970 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.970 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2490 4.15 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:20.970 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:39:20.970 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.444702370000002, 2024-09-16T03:39:20.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73597824999999994, 2024-09-16T03:39:20.970 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.971 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.971 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:20.971 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.971 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.971 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.971 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2490 2024-09-16T03:39:20.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.972 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:39:20.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:39:20.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422021+0000", 2024-09-16T03:39:20.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8099999999999999e-07 2024-09-16T03:39:20.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422022+0000", 2024-09-16T03:39:20.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.573e-06 2024-09-16T03:39:20.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422027+0000", 2024-09-16T03:39:20.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3250000000000001e-06 2024-09-16T03:39:20.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422030+0000", 2024-09-16T03:39:20.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.948e-06 2024-09-16T03:39:20.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858510+0000", 2024-09-16T03:39:20.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.43648032599999997 2024-09-16T03:39:20.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:20.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858588+0000", 2024-09-16T03:39:20.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7888999999999999e-05 2024-09-16T03:39:20.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157803+0000", 2024-09-16T03:39:20.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.299215642 2024-09-16T03:39:20.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157893+0000", 2024-09-16T03:39:20.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9565000000000001e-05 2024-09-16T03:39:20.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157999+0000", 2024-09-16T03:39:20.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105701 2024-09-16T03:39:20.978 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.978 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.978 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.978 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2492 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:20.978 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:20.978 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.444689861000001, 2024-09-16T03:39:20.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53721099400000005, 2024-09-16T03:39:20.979 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.979 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.979 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:20.979 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.979 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.979 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.979 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2492 2024-09-16T03:39:20.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.979 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:20.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:20.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:20.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:39:20.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422036+0000", 2024-09-16T03:39:20.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5900000000000002e-06 2024-09-16T03:39:20.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422043+0000", 2024-09-16T03:39:20.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3069999999999997e-06 2024-09-16T03:39:20.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422048+0000", 2024-09-16T03:39:20.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6619999999999996e-06 2024-09-16T03:39:20.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958774+0000", 2024-09-16T03:39:20.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53672593499999999 2024-09-16T03:39:20.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958868+0000", 2024-09-16T03:39:20.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4408999999999997e-05 2024-09-16T03:39:20.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959244+0000", 2024-09-16T03:39:20.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037581099999999998 2024-09-16T03:39:20.983 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.983 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.984 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.984 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2493 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:20.984 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:20.984 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.444670082000002, 2024-09-16T03:39:20.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53757571299999996, 2024-09-16T03:39:20.984 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.984 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.985 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:20.985 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.985 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.985 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.985 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2493 2024-09-16T03:39:20.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.985 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:20.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3600000000000004e-07 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422055+0000", 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7460000000000001e-06 2024-09-16T03:39:20.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422058+0000", 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422061+0000", 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6759999999999999e-06 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:39:20.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53723057900000004 2024-09-16T03:39:20.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959367+0000", 2024-09-16T03:39:20.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5158000000000001e-05 2024-09-16T03:39:20.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959628+0000", 2024-09-16T03:39:20.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026175800000000001 2024-09-16T03:39:20.989 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.989 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.989 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.989 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2494 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 1738052~782998] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:20.989 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:20.989 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.444659052000006, 2024-09-16T03:39:20.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53787640199999998, 2024-09-16T03:39:20.989 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.989 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2494 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:20.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:20.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6100000000000002e-07 2024-09-16T03:39:20.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422065+0000", 2024-09-16T03:39:20.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2839999999999999e-06 2024-09-16T03:39:20.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422069+0000", 2024-09-16T03:39:20.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7239999999999998e-06 2024-09-16T03:39:20.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422072+0000", 2024-09-16T03:39:20.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6560000000000001e-06 2024-09-16T03:39:20.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959661+0000", 2024-09-16T03:39:20.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53758982600000005 2024-09-16T03:39:20.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959739+0000", 2024-09-16T03:39:20.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7581000000000001e-05 2024-09-16T03:39:20.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:20.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959940+0000", 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020107 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3987 4.36 4:6ef3b53d:::smithi15736696-48:head [setxattr _header (56) in=7b,truncate 2168169] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.490605167, 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50651914200000003, 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:20.994 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:20.995 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:20.995 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:20.995 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3987 2024-09-16T03:39:20.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.995 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:20.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:20.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:39:20.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:20.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:39:20.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:20.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:20.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376119+0000", 2024-09-16T03:39:20.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3799999999999996e-07 2024-09-16T03:39:20.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:20.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376122+0000", 2024-09-16T03:39:20.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.202e-06 2024-09-16T03:39:20.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:20.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376251+0000", 2024-09-16T03:39:20.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012896699999999999 2024-09-16T03:39:20.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:20.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376259+0000", 2024-09-16T03:39:20.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.32e-06 2024-09-16T03:39:20.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:20.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379368+0000", 2024-09-16T03:39:20.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031084989999999998 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379441+0000", 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3331999999999995e-05 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,10", 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379559+0000", 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118068 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:20.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380350+0000", 2024-09-16T03:39:20.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00079103699999999997 2024-09-16T03:39:20.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.580657+0000", 2024-09-16T03:39:20.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20030706600000001 2024-09-16T03:39:20.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:20.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:20.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882497+0000", 2024-09-16T03:39:20.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30183998899999998 2024-09-16T03:39:20.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:20.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:21.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882560+0000", 2024-09-16T03:39:21.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3372000000000002e-05 2024-09-16T03:39:21.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:21.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:21.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882637+0000", 2024-09-16T03:39:21.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6352000000000004e-05 2024-09-16T03:39:21.000 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:21.000 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:21.000 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:21.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3988 4.36 4:6ef3b53d:::smithi15736696-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.490451319, 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60662574199999997, 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3988 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:21.001 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:21.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:21.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:39:21.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:21.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:21.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:21.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:39:21.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:21.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:21.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376273+0000", 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3170000000000001e-06 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376278+0000", 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7849999999999999e-06 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376286+0000", 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2009999999999999e-06 2024-09-16T03:39:21.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:21.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376291+0000", 2024-09-16T03:39:21.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4029999999999998e-06 2024-09-16T03:39:21.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:21.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:21.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380020+0000", 2024-09-16T03:39:21.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037289630000000001 2024-09-16T03:39:21.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:21.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:21.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380065+0000", 2024-09-16T03:39:21.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4666999999999998e-05 2024-09-16T03:39:21.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:21.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:21.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982800+0000", 2024-09-16T03:39:21.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60273539099999995 2024-09-16T03:39:21.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:21.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:21.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982838+0000", 2024-09-16T03:39:21.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8155999999999999e-05 2024-09-16T03:39:21.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:21.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:21.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:21.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982897+0000", 2024-09-16T03:39:21.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8859e-05 2024-09-16T03:39:21.007 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:21.007 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:21.007 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:21.007 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:21.007 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:21.007 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:21.009 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:39:21.009 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:21.007+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:21.109 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:21.107+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:21.130 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:21.130 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:21.130 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:39:21.130 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:21.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:39:21.207 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:39:21.209 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:21.207+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:21.236 INFO:tasks.rados.rados.0.smithi157.stdout:1821: set_chunk oid 8 offset: 947712 length: 256512 target oid b4995a936b657520af06e731b938af26fb8df085ea04abf2caacf15a805fc64f offset: 0 snap -1 2024-09-16T03:39:21.236 INFO:tasks.rados.rados.0.smithi157.stdout:1822: write (excl) oid 31 current snap is 0 2024-09-16T03:39:21.236 INFO:tasks.rados.rados.0.smithi157.stdout:1817: got (22) Invalid argument 2024-09-16T03:39:21.236 INFO:tasks.rados.rados.0.smithi157.stdout:1820: expect (ObjNum 398 snap 0 seq_num 398) 2024-09-16T03:39:21.236 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:21.291 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:21.291 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:21.291 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:21.291 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:21.291 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:21.301 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:39:21.310 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:21.308+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:21.334 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2134306 2024-09-16T03:39:21.343 INFO:tasks.rados.rados.0.smithi157.stdout:1815: finishing write tid 1 to smithi15736696-44 2024-09-16T03:39:21.344 INFO:tasks.rados.rados.0.smithi157.stdout:1815: finishing write tid 2 to smithi15736696-44 2024-09-16T03:39:21.344 INFO:tasks.rados.rados.0.smithi157.stdout:1818: expect (ObjNum 389 snap 0 seq_num 389) 2024-09-16T03:39:21.374 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2194333 2024-09-16T03:39:21.410 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:21.409+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:21.414 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:21.414 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:21.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:21.414 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:21.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.414 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:39 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'21, mlcod=26'19)", 2024-09-16T03:39:21.414 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:39:21.414 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 146.81562330200001, 2024-09-16T03:39:21.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30197182500000003, 2024-09-16T03:39:21.414 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.414 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:21.415 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:39:21.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:39:21.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595320+0000", 2024-09-16T03:39:21.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.962e-06 2024-09-16T03:39:21.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595336+0000", 2024-09-16T03:39:21.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5568000000000001e-05 2024-09-16T03:39:21.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595349+0000", 2024-09-16T03:39:21.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3203e-05 2024-09-16T03:39:21.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695424+0000", 2024-09-16T03:39:21.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007542899999999 2024-09-16T03:39:21.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695495+0000", 2024-09-16T03:39:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0450999999999993e-05 2024-09-16T03:39:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695538+0000", 2024-09-16T03:39:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3433000000000002e-05 2024-09-16T03:39:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897256+0000", 2024-09-16T03:39:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20171840699999999 2024-09-16T03:39:21.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897290+0000", 2024-09-16T03:39:21.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3371999999999998e-05 2024-09-16T03:39:21.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.419 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.419 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.419 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1948 4.8 4:146f84f0:::smithi15736696-6:head [setxattr _header (55) in=7b,truncate 2756348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:21.419 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:39:21.419 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.286134496000003, 2024-09-16T03:39:21.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40326984799999999, 2024-09-16T03:39:21.420 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:21.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.420 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1948 2024-09-16T03:39:21.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.420 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:39:21.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:39:21.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124808+0000", 2024-09-16T03:39:21.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9099999999999991e-07 2024-09-16T03:39:21.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124813+0000", 2024-09-16T03:39:21.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9599999999999999e-06 2024-09-16T03:39:21.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124821+0000", 2024-09-16T03:39:21.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0490000000000004e-06 2024-09-16T03:39:21.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124827+0000", 2024-09-16T03:39:21.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3339999999999998e-06 2024-09-16T03:39:21.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126739+0000", 2024-09-16T03:39:21.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00191147 2024-09-16T03:39:21.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126830+0000", 2024-09-16T03:39:21.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1751999999999998e-05 2024-09-16T03:39:21.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:21.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127005+0000", 2024-09-16T03:39:21.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174227 2024-09-16T03:39:21.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127981+0000", 2024-09-16T03:39:21.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097622799999999999 2024-09-16T03:39:21.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:21.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.128193+0000", 2024-09-16T03:39:21.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021176999999999999 2024-09-16T03:39:21.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.135281+0000", 2024-09-16T03:39:21.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070882150000000001 2024-09-16T03:39:21.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.427785+0000", 2024-09-16T03:39:21.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29250390599999998 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527932+0000", 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10014759500000001 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527980+0000", 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7725000000000001e-05 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528077+0000", 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6626000000000006e-05 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1949 4.8 4:146f84f0:::smithi15736696-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 80.286103752000002, 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40346921899999999, 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:21.427 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.428 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.428 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.428 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1949 2024-09-16T03:39:21.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.428 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:21.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:21.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:21.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:39:21.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124841+0000", 2024-09-16T03:39:21.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5890000000000001e-06 2024-09-16T03:39:21.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124847+0000", 2024-09-16T03:39:21.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0820000000000004e-06 2024-09-16T03:39:21.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124852+0000", 2024-09-16T03:39:21.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1070000000000004e-06 2024-09-16T03:39:21.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127536+0000", 2024-09-16T03:39:21.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026844619999999999 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127585+0000", 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8915999999999997e-05 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528126+0000", 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40054061899999999 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528193+0000", 2024-09-16T03:39:21.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6717999999999994e-05 2024-09-16T03:39:21.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528307+0000", 2024-09-16T03:39:21.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011413600000000001 2024-09-16T03:39:21.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.432 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.432 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.432 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1979 4.18 4:1f63731d:::smithi15736696-30:head [write 789962~534747] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:21.432 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:39:21.433 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.775096439999999, 2024-09-16T03:39:21.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.506754182, 2024-09-16T03:39:21.433 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.433 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.433 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:21.433 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.433 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.433 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.434 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1979 2024-09-16T03:39:21.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.434 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:39:21.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:39:21.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635850+0000", 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6670000000000004e-06 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636429+0000", 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000579745 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636441+0000", 2024-09-16T03:39:21.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1965e-05 2024-09-16T03:39:21.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736594+0000", 2024-09-16T03:39:21.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100152291 2024-09-16T03:39:21.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837326+0000", 2024-09-16T03:39:21.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10073201399999999 2024-09-16T03:39:21.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837437+0000", 2024-09-16T03:39:21.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111914 2024-09-16T03:39:21.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:21.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837625+0000", 2024-09-16T03:39:21.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000187391 2024-09-16T03:39:21.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939449+0000", 2024-09-16T03:39:21.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10182371599999999 2024-09-16T03:39:21.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:21.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939476+0000", 2024-09-16T03:39:21.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7117000000000001e-05 2024-09-16T03:39:21.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142297+0000", 2024-09-16T03:39:21.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20282091499999999 2024-09-16T03:39:21.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142347+0000", 2024-09-16T03:39:21.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0800000000000002e-05 2024-09-16T03:39:21.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142485+0000", 2024-09-16T03:39:21.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013799999999999999 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142526+0000", 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0269999999999999e-05 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142599+0000", 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3376999999999999e-05 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.441 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1980 4.18 4:1f63731d:::smithi15736696-30:head [write 1780952~690056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:21.441 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:39:21.441 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.674295279000006, 2024-09-16T03:39:21.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40615011400000001, 2024-09-16T03:39:21.441 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.441 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.441 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:21.441 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.441 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.441 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.441 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1980 2024-09-16T03:39:21.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736656+0000", 2024-09-16T03:39:21.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0339999999999999e-05 2024-09-16T03:39:21.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737092+0000", 2024-09-16T03:39:21.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435957 2024-09-16T03:39:21.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737102+0000", 2024-09-16T03:39:21.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7860000000000008e-06 2024-09-16T03:39:21.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737107+0000", 2024-09-16T03:39:21.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:39:21.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839282+0000", 2024-09-16T03:39:21.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10217523100000001 2024-09-16T03:39:21.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839389+0000", 2024-09-16T03:39:21.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010619399999999999 2024-09-16T03:39:21.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:21.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839567+0000", 2024-09-16T03:39:21.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017886200000000001 2024-09-16T03:39:21.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:21.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939485+0000", 2024-09-16T03:39:21.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099917168000000001 2024-09-16T03:39:21.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943655+0000", 2024-09-16T03:39:21.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004170287 2024-09-16T03:39:21.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142373+0000", 2024-09-16T03:39:21.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19871839599999999 2024-09-16T03:39:21.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142431+0000", 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7880999999999997e-05 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142627+0000", 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019580500000000001 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142723+0000", 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6209e-05 2024-09-16T03:39:21.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142796+0000", 2024-09-16T03:39:21.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2978000000000005e-05 2024-09-16T03:39:21.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.448 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.448 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2277 4.18 4:1f63731d:::smithi15736696-30:head [write 460558~707131] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:21.448 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:39:21.448 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.988159578999998, 2024-09-16T03:39:21.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39421539100000003, 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2277 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:39:21.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422785+0000", 2024-09-16T03:39:21.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9809999999999999e-06 2024-09-16T03:39:21.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423586+0000", 2024-09-16T03:39:21.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080133600000000004 2024-09-16T03:39:21.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423607+0000", 2024-09-16T03:39:21.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1039e-05 2024-09-16T03:39:21.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423616+0000", 2024-09-16T03:39:21.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0189999999999995e-06 2024-09-16T03:39:21.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523776+0000", 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015945399999999 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523851+0000", 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5549999999999993e-05 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523972+0000", 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120683 2024-09-16T03:39:21.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527436+0000", 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034638870000000001 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636588+0000", 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109152601 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636665+0000", 2024-09-16T03:39:21.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6661000000000003e-05 2024-09-16T03:39:21.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637128+0000", 2024-09-16T03:39:21.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046316199999999998 2024-09-16T03:39:21.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816886+0000", 2024-09-16T03:39:21.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17975745200000001 2024-09-16T03:39:21.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:21.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816927+0000", 2024-09-16T03:39:21.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1529000000000002e-05 2024-09-16T03:39:21.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816997+0000", 2024-09-16T03:39:21.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0036999999999994e-05 2024-09-16T03:39:21.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.457 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.457 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2278 4.18 4:1f63731d:::smithi15736696-30:head [write 1869389~508716] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:21.457 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:39:21.457 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.981590577999995, 2024-09-16T03:39:21.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48788153400000001, 2024-09-16T03:39:21.457 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.457 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.457 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:21.458 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.458 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.458 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.458 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2278 2024-09-16T03:39:21.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.458 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:39:21.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429352+0000", 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.604e-06 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429895+0000", 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054308899999999996 2024-09-16T03:39:21.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429909+0000", 2024-09-16T03:39:21.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3518e-05 2024-09-16T03:39:21.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429917+0000", 2024-09-16T03:39:21.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-09-16T03:39:21.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525208+0000", 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095291736000000002 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525303+0000", 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4304999999999999e-05 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525410+0000", 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010755399999999999 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636785+0000", 2024-09-16T03:39:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111375037 2024-09-16T03:39:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637092+0000", 2024-09-16T03:39:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030685599999999999 2024-09-16T03:39:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637157+0000", 2024-09-16T03:39:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4593999999999999e-05 2024-09-16T03:39:21.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637236+0000", 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9563999999999999e-05 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917119+0000", 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.279882931 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917158+0000", 2024-09-16T03:39:21.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9096e-05 2024-09-16T03:39:21.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917232+0000", 2024-09-16T03:39:21.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4091000000000006e-05 2024-09-16T03:39:21.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.464 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.464 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2279 4.18 4:1f63731d:::smithi15736696-30:head [write 3094884~165674] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:21.464 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:39:21.464 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.976901783000002, 2024-09-16T03:39:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483372209, 2024-09-16T03:39:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2279 2024-09-16T03:39:21.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:39:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:39:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434041+0000", 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.195e-06 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434191+0000", 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000150488 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434202+0000", 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0978000000000001e-05 2024-09-16T03:39:21.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434208+0000", 2024-09-16T03:39:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-09-16T03:39:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526820+0000", 2024-09-16T03:39:21.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092612229000000004 2024-09-16T03:39:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526894+0000", 2024-09-16T03:39:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3435000000000004e-05 2024-09-16T03:39:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526990+0000", 2024-09-16T03:39:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6112000000000004e-05 2024-09-16T03:39:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637050+0000", 2024-09-16T03:39:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110059859 2024-09-16T03:39:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637097+0000", 2024-09-16T03:39:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6848000000000003e-05 2024-09-16T03:39:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637183+0000", 2024-09-16T03:39:21.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6798000000000005e-05 2024-09-16T03:39:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637262+0000", 2024-09-16T03:39:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9049999999999997e-05 2024-09-16T03:39:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917287+0000", 2024-09-16T03:39:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280025036 2024-09-16T03:39:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917326+0000", 2024-09-16T03:39:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.871e-05 2024-09-16T03:39:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917412+0000", 2024-09-16T03:39:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5638999999999998e-05 2024-09-16T03:39:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.472 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2280 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 3260558] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.976723501999999, 2024-09-16T03:39:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48353995599999999, 2024-09-16T03:39:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2280 2024-09-16T03:39:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:21.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:21.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:21.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-09-16T03:39:21.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434221+0000", 2024-09-16T03:39:21.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8329999999999998e-06 2024-09-16T03:39:21.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434225+0000", 2024-09-16T03:39:21.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7589999999999999e-06 2024-09-16T03:39:21.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434228+0000", 2024-09-16T03:39:21.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.971e-06 2024-09-16T03:39:21.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527459+0000", 2024-09-16T03:39:21.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093231249000000002 2024-09-16T03:39:21.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527519+0000", 2024-09-16T03:39:21.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9471e-05 2024-09-16T03:39:21.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:21.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527606+0000", 2024-09-16T03:39:21.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6776999999999999e-05 2024-09-16T03:39:21.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637084+0000", 2024-09-16T03:39:21.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109478228 2024-09-16T03:39:21.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:21.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637100+0000", 2024-09-16T03:39:21.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6711999999999999e-05 2024-09-16T03:39:21.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637210+0000", 2024-09-16T03:39:21.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010909200000000001 2024-09-16T03:39:21.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637296+0000", 2024-09-16T03:39:21.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6718000000000006e-05 2024-09-16T03:39:21.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917637+0000", 2024-09-16T03:39:21.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280340537 2024-09-16T03:39:21.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917671+0000", 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3983999999999999e-05 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917758+0000", 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7017999999999999e-05 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.481 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2281 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:21.481 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:21.481 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.976709206999999, 2024-09-16T03:39:21.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483761683, 2024-09-16T03:39:21.481 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.481 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.481 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:21.481 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.481 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.481 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.481 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2281 2024-09-16T03:39:21.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.481 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:21.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.96e-07 2024-09-16T03:39:21.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434234+0000", 2024-09-16T03:39:21.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4050000000000001e-06 2024-09-16T03:39:21.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434237+0000", 2024-09-16T03:39:21.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1939999999999998e-06 2024-09-16T03:39:21.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434240+0000", 2024-09-16T03:39:21.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.74e-06 2024-09-16T03:39:21.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530037+0000", 2024-09-16T03:39:21.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095796830999999999 2024-09-16T03:39:21.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:21.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530089+0000", 2024-09-16T03:39:21.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:39:21.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917823+0000", 2024-09-16T03:39:21.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38773380000000002 2024-09-16T03:39:21.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917891+0000", 2024-09-16T03:39:21.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7676999999999996e-05 2024-09-16T03:39:21.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917994+0000", 2024-09-16T03:39:21.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010286200000000001 2024-09-16T03:39:21.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.487 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.487 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.487 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2790 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v39,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-09-16T03:39:21.487 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:39:21.487 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.006861751999999, 2024-09-16T03:39:21.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42581759000000002, 2024-09-16T03:39:21.487 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.487 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.487 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:21.487 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.488 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.488 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2790 2024-09-16T03:39:21.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.488 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:39:21.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:39:21.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404083+0000", 2024-09-16T03:39:21.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2770000000000001e-06 2024-09-16T03:39:21.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404095+0000", 2024-09-16T03:39:21.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1780000000000001e-05 2024-09-16T03:39:21.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404116+0000", 2024-09-16T03:39:21.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0882000000000001e-05 2024-09-16T03:39:21.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404127+0000", 2024-09-16T03:39:21.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1078e-05 2024-09-16T03:39:21.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489837+0000", 2024-09-16T03:39:21.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085709977000000007 2024-09-16T03:39:21.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489956+0000", 2024-09-16T03:39:21.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119856 2024-09-16T03:39:21.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:21.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.592758+0000", 2024-09-16T03:39:21.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10280165400000001 2024-09-16T03:39:21.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:21.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.593693+0000", 2024-09-16T03:39:21.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000935285 2024-09-16T03:39:21.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.594164+0000", 2024-09-16T03:39:21.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047097000000000003 2024-09-16T03:39:21.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.628426+0000", 2024-09-16T03:39:21.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034262070999999998 2024-09-16T03:39:21.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.732623+0000", 2024-09-16T03:39:21.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10419628 2024-09-16T03:39:21.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829726+0000", 2024-09-16T03:39:21.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097103098999999998 2024-09-16T03:39:21.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:21.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829790+0000", 2024-09-16T03:39:21.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4677000000000004e-05 2024-09-16T03:39:21.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829897+0000", 2024-09-16T03:39:21.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106704 2024-09-16T03:39:21.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.495 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.495 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.495 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2850 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 2252171] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:39:21.495 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:39:21.495 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.490321469999998, 2024-09-16T03:39:21.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30323033799999999, 2024-09-16T03:39:21.495 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.495 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.496 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:21.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.496 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.496 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2850 2024-09-16T03:39:21.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.496 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:39:21.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:39:21.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920621+0000", 2024-09-16T03:39:21.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.47e-07 2024-09-16T03:39:21.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920625+0000", 2024-09-16T03:39:21.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1620000000000001e-06 2024-09-16T03:39:21.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920632+0000", 2024-09-16T03:39:21.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0180000000000004e-06 2024-09-16T03:39:21.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920638+0000", 2024-09-16T03:39:21.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0950000000000001e-06 2024-09-16T03:39:21.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922161+0000", 2024-09-16T03:39:21.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015229880000000001 2024-09-16T03:39:21.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922222+0000", 2024-09-16T03:39:21.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0903999999999998e-05 2024-09-16T03:39:21.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:21.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922323+0000", 2024-09-16T03:39:21.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101425 2024-09-16T03:39:21.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:21.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.023371+0000", 2024-09-16T03:39:21.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101047493 2024-09-16T03:39:21.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123227+0000", 2024-09-16T03:39:21.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099856057999999998 2024-09-16T03:39:21.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123840+0000", 2024-09-16T03:39:21.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061324999999999997 2024-09-16T03:39:21.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123902+0000", 2024-09-16T03:39:21.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1447999999999998e-05 2024-09-16T03:39:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223497+0000", 2024-09-16T03:39:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099595499000000004 2024-09-16T03:39:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223665+0000", 2024-09-16T03:39:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016807399999999999 2024-09-16T03:39:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223850+0000", 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018507699999999999 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2851 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.490285374000003, 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30354833399999998, 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.503 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:21.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.504 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.504 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2851 2024-09-16T03:39:21.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.504 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:39:21.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920657+0000", 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0100000000000004e-07 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920660+0000", 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6519999999999999e-06 2024-09-16T03:39:21.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920668+0000", 2024-09-16T03:39:21.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1919999999999999e-06 2024-09-16T03:39:21.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920674+0000", 2024-09-16T03:39:21.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-16T03:39:21.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126299+0000", 2024-09-16T03:39:21.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20562534700000001 2024-09-16T03:39:21.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:21.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126358+0000", 2024-09-16T03:39:21.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8459999999999999e-05 2024-09-16T03:39:21.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223913+0000", 2024-09-16T03:39:21.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097555263000000003 2024-09-16T03:39:21.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224009+0000", 2024-09-16T03:39:21.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5723999999999994e-05 2024-09-16T03:39:21.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224204+0000", 2024-09-16T03:39:21.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195498 2024-09-16T03:39:21.509 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.509 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.509 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.509 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3570 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:21.509 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:39:21.509 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.405152050000002, 2024-09-16T03:39:21.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.353157053, 2024-09-16T03:39:21.509 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.510 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.510 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:21.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.510 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.510 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3570 2024-09-16T03:39:21.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.510 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:39:21.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:39:21.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005796+0000", 2024-09-16T03:39:21.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9229999999999996e-06 2024-09-16T03:39:21.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006222+0000", 2024-09-16T03:39:21.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000426282 2024-09-16T03:39:21.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006240+0000", 2024-09-16T03:39:21.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7858000000000001e-05 2024-09-16T03:39:21.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006249+0000", 2024-09-16T03:39:21.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7120000000000006e-06 2024-09-16T03:39:21.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006323+0000", 2024-09-16T03:39:21.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3808999999999998e-05 2024-09-16T03:39:21.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:21.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006333+0000", 2024-09-16T03:39:21.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0488e-05 2024-09-16T03:39:21.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358910+0000", 2024-09-16T03:39:21.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.35257616400000003 2024-09-16T03:39:21.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358946+0000", 2024-09-16T03:39:21.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6816999999999997e-05 2024-09-16T03:39:21.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.515 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.515 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.515 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3571 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:21.515 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:39:21.515 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.398646893999999, 2024-09-16T03:39:21.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34670431699999998, 2024-09-16T03:39:21.516 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.516 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.516 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:21.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.516 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.516 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3571 2024-09-16T03:39:21.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.516 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:39:21.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:39:21.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012298+0000", 2024-09-16T03:39:21.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.18e-06 2024-09-16T03:39:21.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012947+0000", 2024-09-16T03:39:21.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064929600000000005 2024-09-16T03:39:21.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012961+0000", 2024-09-16T03:39:21.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4341e-05 2024-09-16T03:39:21.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012970+0000", 2024-09-16T03:39:21.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2740000000000007e-06 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.013025+0000", 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5353999999999999e-05 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.013033+0000", 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4619999999999996e-06 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358989+0000", 2024-09-16T03:39:21.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34595562699999999 2024-09-16T03:39:21.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.521 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.521 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358999+0000", 2024-09-16T03:39:21.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7829999999999993e-06 2024-09-16T03:39:21.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.521 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.521 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.521 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3572 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:21.521 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:39:21.522 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.392626138000001, 2024-09-16T03:39:21.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34073019300000001, 2024-09-16T03:39:21.522 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.522 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.522 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:21.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.522 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.522 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3572 2024-09-16T03:39:21.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.522 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018317+0000", 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4610000000000001e-06 2024-09-16T03:39:21.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018779+0000", 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046202500000000002 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018793+0000", 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4629e-05 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018799+0000", 2024-09-16T03:39:21.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1589999999999998e-06 2024-09-16T03:39:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018867+0000", 2024-09-16T03:39:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7879000000000005e-05 2024-09-16T03:39:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018888+0000", 2024-09-16T03:39:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0747999999999999e-05 2024-09-16T03:39:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359036+0000", 2024-09-16T03:39:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34014758 2024-09-16T03:39:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359045+0000", 2024-09-16T03:39:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7119999999999995e-06 2024-09-16T03:39:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.526 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3573 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:39:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.392126443999999, 2024-09-16T03:39:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34027060599999998, 2024-09-16T03:39:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3573 2024-09-16T03:39:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:39:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:39:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018816+0000", 2024-09-16T03:39:21.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1400000000000001e-06 2024-09-16T03:39:21.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018821+0000", 2024-09-16T03:39:21.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4179999999999996e-06 2024-09-16T03:39:21.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018829+0000", 2024-09-16T03:39:21.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6320000000000008e-06 2024-09-16T03:39:21.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018835+0000", 2024-09-16T03:39:21.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0399999999999998e-06 2024-09-16T03:39:21.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018953+0000", 2024-09-16T03:39:21.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117713 2024-09-16T03:39:21.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018967+0000", 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4334e-05 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359078+0000", 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34011044499999998 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359086+0000", 2024-09-16T03:39:21.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8839999999999994e-06 2024-09-16T03:39:21.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.532 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.532 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.532 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3574 4.28 4.f21f628 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:21.532 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:39:21.532 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.392093289000002, 2024-09-16T03:39:21.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.340272882, 2024-09-16T03:39:21.532 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.532 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.533 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:21.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.533 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3574 2024-09-16T03:39:21.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.533 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:39:21.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:39:21.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018849+0000", 2024-09-16T03:39:21.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9599999999999998e-07 2024-09-16T03:39:21.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018852+0000", 2024-09-16T03:39:21.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9749999999999999e-06 2024-09-16T03:39:21.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018867+0000", 2024-09-16T03:39:21.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5102e-05 2024-09-16T03:39:21.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018882+0000", 2024-09-16T03:39:21.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5319e-05 2024-09-16T03:39:21.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018993+0000", 2024-09-16T03:39:21.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110457 2024-09-16T03:39:21.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:21.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018998+0000", 2024-09-16T03:39:21.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.028e-06 2024-09-16T03:39:21.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359114+0000", 2024-09-16T03:39:21.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34011593099999998 2024-09-16T03:39:21.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359121+0000", 2024-09-16T03:39:21.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2740000000000002e-06 2024-09-16T03:39:21.537 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.537 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.538 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3995 4.8 4:13b03fc9:::smithi15736696-3:head [setxattr _header (55) in=7b,truncate 2184976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:21.538 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:05.315015+0000", 2024-09-16T03:39:21.538 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.095926775999999, 2024-09-16T03:39:21.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30168798099999999, 2024-09-16T03:39:21.538 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.538 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.538 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:21.538 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.538 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.538 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.539 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3995 2024-09-16T03:39:21.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.539 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315015+0000", 2024-09-16T03:39:21.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315015+0000", 2024-09-16T03:39:21.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315016+0000", 2024-09-16T03:39:21.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4e-07 2024-09-16T03:39:21.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315020+0000", 2024-09-16T03:39:21.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0980000000000004e-06 2024-09-16T03:39:21.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315028+0000", 2024-09-16T03:39:21.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2579999999999995e-06 2024-09-16T03:39:21.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315034+0000", 2024-09-16T03:39:21.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4930000000000003e-06 2024-09-16T03:39:21.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.316207+0000", 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001172867 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.316268+0000", 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0729000000000001e-05 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4,10", 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.316353+0000", 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.543e-05 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.515813+0000", 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19945929000000001 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.616015+0000", 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100202772 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.616558+0000", 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054254800000000001 2024-09-16T03:39:21.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:21.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.616621+0000", 2024-09-16T03:39:21.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3341000000000002e-05 2024-09-16T03:39:21.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.616703+0000", 2024-09-16T03:39:21.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1215000000000001e-05 2024-09-16T03:39:21.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.544 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.544 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3996 4.8 4:13b03fc9:::smithi15736696-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:05.315045+0000", 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.09589588, 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30191814700000003, 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3996 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315045+0000", 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315045+0000", 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315046+0000", 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9100000000000003e-07 2024-09-16T03:39:21.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315049+0000", 2024-09-16T03:39:21.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1020000000000001e-06 2024-09-16T03:39:21.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315056+0000", 2024-09-16T03:39:21.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5400000000000001e-06 2024-09-16T03:39:21.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315062+0000", 2024-09-16T03:39:21.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.908e-06 2024-09-16T03:39:21.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.515314+0000", 2024-09-16T03:39:21.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20025267999999999 2024-09-16T03:39:21.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:21.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.515437+0000", 2024-09-16T03:39:21.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122955 2024-09-16T03:39:21.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.616750+0000", 2024-09-16T03:39:21.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10131272199999999 2024-09-16T03:39:21.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.616842+0000", 2024-09-16T03:39:21.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2408e-05 2024-09-16T03:39:21.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:21.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.616964+0000", 2024-09-16T03:39:21.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121141 2024-09-16T03:39:21.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.550 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.550 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.550 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.550 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:21.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:39:21.556 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:21.557 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:39:21.559 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-16T03:39:20.602747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":210924521,"num_objects":136,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":2083,"num_read_kb":977243,"num_write":1805,"num_write_kb":395717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":60063223,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":19,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":189,"ondisk_log_size":189,"up":207,"acting":207,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":216,"num_osds":13,"num_per_pool_osds":13,"num_per_pool_omap_osds":13,"kb":1226833920,"kb_used":801992,"kb_used_data":376960,"kb_used_omap":20,"kb_used_meta":405163,"kb_avail":1226031928,"statfs":{"total":1256277934080,"available":1255456694272,"internally_reserved":0,"allocated":386007040,"data_stored":392661496,"data_compressed":11331300,"data_compressed_allocated":12238848,"data_compressed_original":25944064,"omap_allocated":20735,"internal_metadata":414887681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4414506,"num_objects":-4,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":132,"num_read_kb":84829,"num_write":51,"num_write_kb":14482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-9176891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.896117"},"pg_stats":[{"pgid":"4.37","version":"142'55","reported_seq":417,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:19.067173+0000","last_change":"2024-09-16T03:39:08.371859+0000","last_active":"2024-09-16T03:39:19.067173+0000","last_peered":"2024-09-16T03:39:19.067173+0000","last_clean":"2024-09-16T03:39:19.067173+0000","last_became_active":"2024-09-16T03:39:05.721626+0000","last_became_peered":"2024-09-16T03:39:05.721626+0000","last_unstale":"2024-09-16T03:39:19.067173+0000","last_undegraded":"2024-09-16T03:39:19.067173+0000","last_fullsized":"2024-09-16T03:39:19.067173+0000","mapping_epoch":130,"log_start":"131'52","ondisk_log_start":"131'52","created":126,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":6,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":47,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:36.963537+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3378290,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":55,"num_read_kb":44518,"num_write":16,"num_write_kb":4585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1148090,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.36","version":"131'93","reported_seq":489,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.729722+0000","last_change":"2024-09-16T03:39:03.716194+0000","last_active":"2024-09-16T03:39:16.729722+0000","last_peered":"2024-09-16T03:39:16.729722+0000","last_clean":"2024-09-16T03:39:16.729722+0000","last_became_active":"2024-09-16T03:39:02.099320+0000","last_became_peered":"2024-09-16T03:39:02.099320+0000","last_unstale":"2024-09-16T03:39:16.729722+0000","last_undegraded":"2024-09-16T03:39:16.729722+0000","last_fullsized":"2024-09-16T03:39:16.729722+0000","mapping_epoch":126,"log_start":"130'90","ondisk_log_start":"130'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.716125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.716125+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":88,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.328700+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8269209,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":39,"num_read_kb":28771,"num_write":8,"num_write_kb":2101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.35","version":"140'70","reported_seq":508,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:19.352437+0000","last_change":"2024-09-16T03:39:15.080084+0000","last_active":"2024-09-16T03:39:19.352437+0000","last_peered":"2024-09-16T03:39:19.352437+0000","last_clean":"2024-09-16T03:39:19.352437+0000","last_became_active":"2024-09-16T03:39:13.773139+0000","last_became_peered":"2024-09-16T03:39:13.773139+0000","last_unstale":"2024-09-16T03:39:19.352437+0000","last_undegraded":"2024-09-16T03:39:19.352437+0000","last_fullsized":"2024-09-16T03:39:19.352437+0000","mapping_epoch":138,"log_start":"139'67","ondisk_log_start":"139'67","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'67","last_scrub_stamp":"2024-09-16T03:39:15.079987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:15.079987+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":65,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.045373+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521116,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10519,"num_write":4,"num_write_kb":1251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"104'20","reported_seq":338,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:15.312712+0000","last_change":"2024-09-16T03:39:02.919356+0000","last_active":"2024-09-16T03:39:15.312712+0000","last_peered":"2024-09-16T03:39:15.312712+0000","last_clean":"2024-09-16T03:39:15.312712+0000","last_became_active":"2024-09-16T03:39:01.699286+0000","last_became_peered":"2024-09-16T03:39:01.699286+0000","last_unstale":"2024-09-16T03:39:15.312712+0000","last_undegraded":"2024-09-16T03:39:15.312712+0000","last_fullsized":"2024-09-16T03:39:15.312712+0000","mapping_epoch":126,"log_start":"98'17","ondisk_log_start":"98'17","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:02.919259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.919259+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:28.209096+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2705960,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":11460,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"137'75","reported_seq":509,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-16T03:39:19.599729+0000","last_change":"2024-09-16T03:39:13.974391+0000","last_active":"2024-09-16T03:39:19.599729+0000","last_peered":"2024-09-16T03:39:19.599729+0000","last_clean":"2024-09-16T03:39:19.599729+0000","last_became_active":"2024-09-16T03:39:13.873951+0000","last_became_peered":"2024-09-16T03:39:13.873951+0000","last_unstale":"2024-09-16T03:39:19.599729+0000","last_undegraded":"2024-09-16T03:39:19.599729+0000","last_fullsized":"2024-09-16T03:39:19.599729+0000","mapping_epoch":138,"log_start":"108'70","ondisk_log_start":"108'70","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'72","last_scrub_stamp":"2024-09-16T03:39:02.143339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.143339+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":68,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.465359+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353046,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":19591,"num_write":4,"num_write_kb":955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":994759,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.32","version":"106'17","reported_seq":397,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.651398+0000","last_change":"2024-09-16T03:39:14.075213+0000","last_active":"2024-09-16T03:39:18.651398+0000","last_peered":"2024-09-16T03:39:18.651398+0000","last_clean":"2024-09-16T03:39:18.651398+0000","last_became_active":"2024-09-16T03:39:14.074797+0000","last_became_peered":"2024-09-16T03:39:14.074797+0000","last_unstale":"2024-09-16T03:39:18.651398+0000","last_undegraded":"2024-09-16T03:39:18.651398+0000","last_fullsized":"2024-09-16T03:39:18.651398+0000","mapping_epoch":138,"log_start":"82'13","ondisk_log_start":"82'13","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"106'17","last_scrub_stamp":"2024-09-16T03:39:12.108317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:12.108317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:16.313362+0000","scrub_duration":367,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.31","version":"114'17","reported_seq":305,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.723994+0000","last_change":"2024-09-16T03:39:05.622532+0000","last_active":"2024-09-16T03:39:14.723994+0000","last_peered":"2024-09-16T03:39:14.723994+0000","last_clean":"2024-09-16T03:39:14.723994+0000","last_became_active":"2024-09-16T03:39:01.997491+0000","last_became_peered":"2024-09-16T03:39:01.997491+0000","last_unstale":"2024-09-16T03:39:14.723994+0000","last_undegraded":"2024-09-16T03:39:14.723994+0000","last_fullsized":"2024-09-16T03:39:14.723994+0000","mapping_epoch":126,"log_start":"107'14","ondisk_log_start":"107'14","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"62'9","last_scrub_stamp":"2024-09-16T03:37:50.756120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:50.756120+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.630555+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1495840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":28490,"num_write":4,"num_write_kb":1066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.30","version":"118'64","reported_seq":408,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.639964+0000","last_change":"2024-09-16T03:39:05.621064+0000","last_active":"2024-09-16T03:39:18.639964+0000","last_peered":"2024-09-16T03:39:18.639964+0000","last_clean":"2024-09-16T03:39:18.639964+0000","last_became_active":"2024-09-16T03:39:01.901037+0000","last_became_peered":"2024-09-16T03:39:01.901037+0000","last_unstale":"2024-09-16T03:39:18.639964+0000","last_undegraded":"2024-09-16T03:39:18.639964+0000","last_fullsized":"2024-09-16T03:39:18.639964+0000","mapping_epoch":126,"log_start":"113'61","ondisk_log_start":"113'61","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"53'46","last_scrub_stamp":"2024-09-16T03:37:35.198817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:35.198817+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:35.157846+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1680717,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":23268,"num_write":10,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":918091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2f","version":"113'94","reported_seq":478,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.740578+0000","last_change":"2024-09-16T03:39:03.319075+0000","last_active":"2024-09-16T03:39:18.740578+0000","last_peered":"2024-09-16T03:39:18.740578+0000","last_clean":"2024-09-16T03:39:18.740578+0000","last_became_active":"2024-09-16T03:39:01.700988+0000","last_became_peered":"2024-09-16T03:39:01.700988+0000","last_unstale":"2024-09-16T03:39:18.740578+0000","last_undegraded":"2024-09-16T03:39:18.740578+0000","last_fullsized":"2024-09-16T03:39:18.740578+0000","mapping_epoch":126,"log_start":"107'90","ondisk_log_start":"107'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"113'94","last_scrub_stamp":"2024-09-16T03:39:03.319012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.319012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":87,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:13.773817+0000","scrub_duration":1038,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2e","version":"140'78","reported_seq":467,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-16T03:39:19.599897+0000","last_change":"2024-09-16T03:39:04.235234+0000","last_active":"2024-09-16T03:39:19.599897+0000","last_peered":"2024-09-16T03:39:19.599897+0000","last_clean":"2024-09-16T03:39:19.599897+0000","last_became_active":"2024-09-16T03:39:01.999291+0000","last_became_peered":"2024-09-16T03:39:01.999291+0000","last_unstale":"2024-09-16T03:39:19.599897+0000","last_undegraded":"2024-09-16T03:39:19.599897+0000","last_fullsized":"2024-09-16T03:39:19.599897+0000","mapping_epoch":126,"log_start":"123'75","ondisk_log_start":"123'75","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"123'75","last_scrub_stamp":"2024-09-16T03:39:04.235141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.235141+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":75,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.010761+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2554730,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":13154,"num_write":4,"num_write_kb":1191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"131'99","reported_seq":491,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.746195+0000","last_change":"2024-09-16T03:39:03.917422+0000","last_active":"2024-09-16T03:39:18.746195+0000","last_peered":"2024-09-16T03:39:18.746195+0000","last_clean":"2024-09-16T03:39:18.746195+0000","last_became_active":"2024-09-16T03:39:01.801112+0000","last_became_peered":"2024-09-16T03:39:01.801112+0000","last_unstale":"2024-09-16T03:39:18.746195+0000","last_undegraded":"2024-09-16T03:39:18.746195+0000","last_fullsized":"2024-09-16T03:39:18.746195+0000","mapping_epoch":32,"log_start":"128'96","ondisk_log_start":"128'96","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'97","last_scrub_stamp":"2024-09-16T03:39:03.917355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.917355+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":93,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.686164+0000","scrub_duration":771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2033713,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":14614,"num_write":1,"num_write_kb":214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.e","version":"138'82","reported_seq":463,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-16T03:39:19.600140+0000","last_change":"2024-09-16T03:39:05.725455+0000","last_active":"2024-09-16T03:39:19.600140+0000","last_peered":"2024-09-16T03:39:19.600140+0000","last_clean":"2024-09-16T03:39:19.600140+0000","last_became_active":"2024-09-16T03:39:01.999282+0000","last_became_peered":"2024-09-16T03:39:01.999282+0000","last_unstale":"2024-09-16T03:39:19.600140+0000","last_undegraded":"2024-09-16T03:39:19.600140+0000","last_fullsized":"2024-09-16T03:39:19.600140+0000","mapping_epoch":32,"log_start":"127'79","ondisk_log_start":"127'79","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:39.194237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.194237+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":76,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.946501+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2159765,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":28931,"num_write":14,"num_write_kb":4613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,15,8],"acting":[4,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"139'22","reported_seq":369,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.593442+0000","last_change":"2024-09-16T03:39:01.149616+0000","last_active":"2024-09-16T03:39:18.593442+0000","last_peered":"2024-09-16T03:39:18.593442+0000","last_clean":"2024-09-16T03:39:18.593442+0000","last_became_active":"2024-09-16T03:38:45.622965+0000","last_became_peered":"2024-09-16T03:38:45.622965+0000","last_unstale":"2024-09-16T03:39:18.593442+0000","last_undegraded":"2024-09-16T03:39:18.593442+0000","last_fullsized":"2024-09-16T03:39:18.593442+0000","mapping_epoch":72,"log_start":"124'19","ondisk_log_start":"124'19","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'19","last_scrub_stamp":"2024-09-16T03:39:01.149486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.149486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":19,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.574017+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487502,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":3909,"num_write":4,"num_write_kb":890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"130'65","reported_seq":448,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.593375+0000","last_change":"2024-09-16T03:39:13.775457+0000","last_active":"2024-09-16T03:39:18.593375+0000","last_peered":"2024-09-16T03:39:18.593375+0000","last_clean":"2024-09-16T03:39:18.593375+0000","last_became_active":"2024-09-16T03:39:13.775138+0000","last_became_peered":"2024-09-16T03:39:13.775138+0000","last_unstale":"2024-09-16T03:39:18.593375+0000","last_undegraded":"2024-09-16T03:39:18.593375+0000","last_fullsized":"2024-09-16T03:39:18.593375+0000","mapping_epoch":138,"log_start":"122'62","ondisk_log_start":"122'62","created":32,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'57","last_scrub_stamp":"2024-09-16T03:38:48.923184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.923184+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:53.322016+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":7446,"num_write":9,"num_write_kb":2888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1213217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"140'133","reported_seq":614,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.706028+0000","last_change":"2024-09-16T03:39:09.748835+0000","last_active":"2024-09-16T03:39:16.706028+0000","last_peered":"2024-09-16T03:39:16.706028+0000","last_clean":"2024-09-16T03:39:16.706028+0000","last_became_active":"2024-09-16T03:39:09.748391+0000","last_became_peered":"2024-09-16T03:39:09.748391+0000","last_unstale":"2024-09-16T03:39:16.706028+0000","last_undegraded":"2024-09-16T03:39:16.706028+0000","last_fullsized":"2024-09-16T03:39:16.706028+0000","mapping_epoch":134,"log_start":"129'130","ondisk_log_start":"129'130","created":32,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:46.855462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.855462+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":126,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:12.773343+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2084134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":17019,"num_write":31,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.a","version":"141'154","reported_seq":719,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.643690+0000","last_change":"2024-09-16T03:39:01.504751+0000","last_active":"2024-09-16T03:39:18.643690+0000","last_peered":"2024-09-16T03:39:18.643690+0000","last_clean":"2024-09-16T03:39:18.643690+0000","last_became_active":"2024-09-16T03:38:46.021805+0000","last_became_peered":"2024-09-16T03:38:46.021805+0000","last_unstale":"2024-09-16T03:39:18.643690+0000","last_undegraded":"2024-09-16T03:39:18.643690+0000","last_fullsized":"2024-09-16T03:39:18.643690+0000","mapping_epoch":32,"log_start":"140'150","ondisk_log_start":"140'150","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'147","last_scrub_stamp":"2024-09-16T03:39:01.504663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.504663+0000","objects_scrubbed":4,"log_size":4,"log_dups_size":147,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:18.060284+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9361631,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":82900,"num_write":6,"num_write_kb":1061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":264,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.651460+0000","last_change":"2024-09-16T03:38:46.881357+0000","last_active":"2024-09-16T03:39:18.651460+0000","last_peered":"2024-09-16T03:39:18.651460+0000","last_clean":"2024-09-16T03:39:18.651460+0000","last_became_active":"2024-09-16T03:38:45.728494+0000","last_became_peered":"2024-09-16T03:38:45.728494+0000","last_unstale":"2024-09-16T03:39:18.651460+0000","last_undegraded":"2024-09-16T03:39:18.651460+0000","last_fullsized":"2024-09-16T03:39:18.651460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:46.881248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.881248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:04.991122+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.8","version":"140'78","reported_seq":486,"reported_epoch":142,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:18.637905+0000","last_change":"2024-09-16T03:39:11.576416+0000","last_active":"2024-09-16T03:39:18.637905+0000","last_peered":"2024-09-16T03:39:18.637905+0000","last_clean":"2024-09-16T03:39:08.372137+0000","last_became_active":"2024-09-16T03:39:11.177629+0000","last_became_peered":"2024-09-16T03:39:11.177629+0000","last_unstale":"2024-09-16T03:39:18.637905+0000","last_undegraded":"2024-09-16T03:39:18.637905+0000","last_fullsized":"2024-09-16T03:39:18.637905+0000","mapping_epoch":135,"log_start":"138'75","ondisk_log_start":"138'75","created":32,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'71","last_scrub_stamp":"2024-09-16T03:39:02.002247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.002247+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":73,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":3437,"num_write":9,"num_write_kb":1852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,10],"avail_no_missing":["0","4","9","10"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"138'88","reported_seq":452,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.706103+0000","last_change":"2024-09-16T03:39:00.900732+0000","last_active":"2024-09-16T03:39:16.706103+0000","last_peered":"2024-09-16T03:39:16.706103+0000","last_clean":"2024-09-16T03:39:16.706103+0000","last_became_active":"2024-09-16T03:38:45.523605+0000","last_became_peered":"2024-09-16T03:38:45.523605+0000","last_unstale":"2024-09-16T03:39:16.706103+0000","last_undegraded":"2024-09-16T03:39:16.706103+0000","last_fullsized":"2024-09-16T03:39:16.706103+0000","mapping_epoch":32,"log_start":"130'85","ondisk_log_start":"130'85","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'82","last_scrub_stamp":"2024-09-16T03:39:00.900619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.900619+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":83,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.385837+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2420274,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":5790,"num_write":8,"num_write_kb":1825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":437,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.704186+0000","last_change":"2024-09-16T03:38:36.757156+0000","last_active":"2024-09-16T03:39:16.704186+0000","last_peered":"2024-09-16T03:39:16.704186+0000","last_clean":"2024-09-16T03:39:16.704186+0000","last_became_active":"2024-09-16T03:38:36.756855+0000","last_became_peered":"2024-09-16T03:38:36.756855+0000","last_unstale":"2024-09-16T03:39:16.704186+0000","last_undegraded":"2024-09-16T03:39:16.704186+0000","last_fullsized":"2024-09-16T03:39:16.704186+0000","mapping_epoch":103,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:38:28.063690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:38:28.063690+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:28.682469+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":142,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.637792+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:39:18.637792+0000","last_peered":"2024-09-16T03:39:18.637792+0000","last_clean":"2024-09-16T03:39:18.637792+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:39:18.637792+0000","last_undegraded":"2024-09-16T03:39:18.637792+0000","last_fullsized":"2024-09-16T03:39:18.637792+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"140'265","reported_seq":586,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.704408+0000","last_change":"2024-09-16T03:38:01.603352+0000","last_active":"2024-09-16T03:39:16.704408+0000","last_peered":"2024-09-16T03:39:16.704408+0000","last_clean":"2024-09-16T03:39:16.704408+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:39:16.704408+0000","last_undegraded":"2024-09-16T03:39:16.704408+0000","last_fullsized":"2024-09-16T03:39:16.704408+0000","mapping_epoch":74,"log_start":"137'262","ondisk_log_start":"137'262","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":271,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.902901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21682013,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":254,"num_read_kb":8762,"num_write":329,"num_write_kb":70574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.6","version":"108'44","reported_seq":398,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.640646+0000","last_change":"2024-09-16T03:38:48.279289+0000","last_active":"2024-09-16T03:39:18.640646+0000","last_peered":"2024-09-16T03:39:18.640646+0000","last_clean":"2024-09-16T03:39:18.640646+0000","last_became_active":"2024-09-16T03:38:45.924286+0000","last_became_peered":"2024-09-16T03:38:45.924286+0000","last_unstale":"2024-09-16T03:39:18.640646+0000","last_undegraded":"2024-09-16T03:39:18.640646+0000","last_fullsized":"2024-09-16T03:39:18.640646+0000","mapping_epoch":70,"log_start":"105'41","ondisk_log_start":"105'41","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:48.279224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.279224+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.718165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":282,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.638327+0000","last_change":"2024-09-16T03:39:09.749531+0000","last_active":"2024-09-16T03:39:18.638327+0000","last_peered":"2024-09-16T03:39:18.638327+0000","last_clean":"2024-09-16T03:39:18.638327+0000","last_became_active":"2024-09-16T03:39:09.649028+0000","last_became_peered":"2024-09-16T03:39:09.649028+0000","last_unstale":"2024-09-16T03:39:18.638327+0000","last_undegraded":"2024-09-16T03:39:18.638327+0000","last_fullsized":"2024-09-16T03:39:18.638327+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:24.358339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:24.358339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:34.729230+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"140'228","reported_seq":479,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.682092+0000","last_change":"2024-09-16T03:38:15.890094+0000","last_active":"2024-09-16T03:39:16.682092+0000","last_peered":"2024-09-16T03:39:16.682092+0000","last_clean":"2024-09-16T03:39:16.682092+0000","last_became_active":"2024-09-16T03:37:05.033762+0000","last_became_peered":"2024-09-16T03:37:05.033762+0000","last_unstale":"2024-09-16T03:39:16.682092+0000","last_undegraded":"2024-09-16T03:39:16.682092+0000","last_fullsized":"2024-09-16T03:39:16.682092+0000","mapping_epoch":31,"log_start":"138'225","ondisk_log_start":"138'225","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'125","last_scrub_stamp":"2024-09-16T03:38:15.889996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.889996+0000","objects_scrubbed":18,"log_size":3,"log_dups_size":236,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:35.350472+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22211125,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":199,"num_read_kb":2083,"num_write":285,"num_write_kb":65350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"139'77","reported_seq":445,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.840889+0000","last_change":"2024-09-16T03:39:01.297447+0000","last_active":"2024-09-16T03:39:18.840889+0000","last_peered":"2024-09-16T03:39:18.840889+0000","last_clean":"2024-09-16T03:39:18.840889+0000","last_became_active":"2024-09-16T03:38:45.723754+0000","last_became_peered":"2024-09-16T03:38:45.723754+0000","last_unstale":"2024-09-16T03:39:18.840889+0000","last_undegraded":"2024-09-16T03:39:18.840889+0000","last_fullsized":"2024-09-16T03:39:18.840889+0000","mapping_epoch":32,"log_start":"127'74","ondisk_log_start":"127'74","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'72","last_scrub_stamp":"2024-09-16T03:39:01.297324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.297324+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":72,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.355718+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7294304,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":32694,"num_write":5,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":252,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.705946+0000","last_change":"2024-09-16T03:38:15.949627+0000","last_active":"2024-09-16T03:39:16.705946+0000","last_peered":"2024-09-16T03:39:16.705946+0000","last_clean":"2024-09-16T03:39:16.705946+0000","last_became_active":"2024-09-16T03:36:53.383588+0000","last_became_peered":"2024-09-16T03:36:53.383588+0000","last_unstale":"2024-09-16T03:39:16.705946+0000","last_undegraded":"2024-09-16T03:39:16.705946+0000","last_fullsized":"2024-09-16T03:39:16.705946+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:15.949523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:15.949523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:24.286324+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00049487299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"142'261","reported_seq":661,"reported_epoch":142,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:18.797678+0000","last_change":"2024-09-16T03:39:11.578535+0000","last_active":"2024-09-16T03:39:18.797678+0000","last_peered":"2024-09-16T03:39:18.797678+0000","last_clean":"2024-09-16T03:39:08.210670+0000","last_became_active":"2024-09-16T03:39:11.274646+0000","last_became_peered":"2024-09-16T03:39:11.274646+0000","last_unstale":"2024-09-16T03:39:18.797678+0000","last_undegraded":"2024-09-16T03:39:18.797678+0000","last_fullsized":"2024-09-16T03:39:18.797678+0000","mapping_epoch":135,"log_start":"140'258","ondisk_log_start":"140'258","created":31,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":279,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19124461,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":237,"num_read_kb":12820,"num_write":321,"num_write_kb":81770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":27250603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[3,4,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":18}],"blocked_by":[],"up_primary":9,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"137'33","reported_seq":348,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.704550+0000","last_change":"2024-09-16T03:38:47.042020+0000","last_active":"2024-09-16T03:39:16.704550+0000","last_peered":"2024-09-16T03:39:16.704550+0000","last_clean":"2024-09-16T03:39:16.704550+0000","last_became_active":"2024-09-16T03:38:45.820169+0000","last_became_peered":"2024-09-16T03:38:45.820169+0000","last_unstale":"2024-09-16T03:39:16.704550+0000","last_undegraded":"2024-09-16T03:39:16.704550+0000","last_fullsized":"2024-09-16T03:39:16.704550+0000","mapping_epoch":32,"log_start":"87'30","ondisk_log_start":"87'30","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.041851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.041851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":30,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.311785+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":266,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.638604+0000","last_change":"2024-09-16T03:39:14.082589+0000","last_active":"2024-09-16T03:39:18.638604+0000","last_peered":"2024-09-16T03:39:18.638604+0000","last_clean":"2024-09-16T03:39:18.638604+0000","last_became_active":"2024-09-16T03:36:53.177666+0000","last_became_peered":"2024-09-16T03:36:53.177666+0000","last_unstale":"2024-09-16T03:39:18.638604+0000","last_undegraded":"2024-09-16T03:39:18.638604+0000","last_fullsized":"2024-09-16T03:39:18.638604+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:39:14.082477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.082477+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.989780+0000","scrub_duration":733,"objects_trimmed":0,"snaptrim_duration":0.0011455720000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"139'283","reported_seq":536,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.806157+0000","last_change":"2024-09-16T03:37:05.035031+0000","last_active":"2024-09-16T03:39:16.806157+0000","last_peered":"2024-09-16T03:39:16.806157+0000","last_clean":"2024-09-16T03:39:16.806157+0000","last_became_active":"2024-09-16T03:37:05.034666+0000","last_became_peered":"2024-09-16T03:37:05.034666+0000","last_unstale":"2024-09-16T03:39:16.806157+0000","last_undegraded":"2024-09-16T03:39:16.806157+0000","last_fullsized":"2024-09-16T03:39:16.806157+0000","mapping_epoch":31,"log_start":"137'280","ondisk_log_start":"137'280","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":292,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:20.679811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22933955,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":256,"num_read_kb":6273,"num_write":352,"num_write_kb":73742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":286,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.806053+0000","last_change":"2024-09-16T03:38:35.828277+0000","last_active":"2024-09-16T03:39:16.806053+0000","last_peered":"2024-09-16T03:39:16.806053+0000","last_clean":"2024-09-16T03:39:16.806053+0000","last_became_active":"2024-09-16T03:38:33.778361+0000","last_became_peered":"2024-09-16T03:38:33.778361+0000","last_unstale":"2024-09-16T03:39:16.806053+0000","last_undegraded":"2024-09-16T03:39:16.806053+0000","last_fullsized":"2024-09-16T03:39:16.806053+0000","mapping_epoch":32,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:35.828203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.828203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:49.130136+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,11],"acting":[15,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":255,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.682145+0000","last_change":"2024-09-16T03:37:52.652365+0000","last_active":"2024-09-16T03:39:16.682145+0000","last_peered":"2024-09-16T03:39:16.682145+0000","last_clean":"2024-09-16T03:39:16.682145+0000","last_became_active":"2024-09-16T03:37:52.651907+0000","last_became_peered":"2024-09-16T03:37:52.651907+0000","last_unstale":"2024-09-16T03:39:16.682145+0000","last_undegraded":"2024-09-16T03:39:16.682145+0000","last_fullsized":"2024-09-16T03:39:16.682145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.985111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"94'34","reported_seq":357,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.704280+0000","last_change":"2024-09-16T03:38:39.081942+0000","last_active":"2024-09-16T03:39:16.704280+0000","last_peered":"2024-09-16T03:39:16.704280+0000","last_clean":"2024-09-16T03:39:16.704280+0000","last_became_active":"2024-09-16T03:38:33.580022+0000","last_became_peered":"2024-09-16T03:38:33.580022+0000","last_unstale":"2024-09-16T03:39:16.704280+0000","last_undegraded":"2024-09-16T03:39:16.704280+0000","last_fullsized":"2024-09-16T03:39:16.704280+0000","mapping_epoch":96,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:39.081810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.081810+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:48.167935+0000","scrub_duration":473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.697779+0000","last_change":"2024-09-16T03:39:09.848905+0000","last_active":"2024-09-16T03:39:18.697779+0000","last_peered":"2024-09-16T03:39:18.697779+0000","last_clean":"2024-09-16T03:39:18.697779+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:39:18.697779+0000","last_undegraded":"2024-09-16T03:39:18.697779+0000","last_fullsized":"2024-09-16T03:39:18.697779+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:48.949444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"140'136","reported_seq":592,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.640462+0000","last_change":"2024-09-16T03:38:39.704994+0000","last_active":"2024-09-16T03:39:18.640462+0000","last_peered":"2024-09-16T03:39:18.640462+0000","last_clean":"2024-09-16T03:39:18.640462+0000","last_became_active":"2024-09-16T03:38:33.581141+0000","last_became_peered":"2024-09-16T03:38:33.581141+0000","last_unstale":"2024-09-16T03:39:18.640462+0000","last_undegraded":"2024-09-16T03:39:18.640462+0000","last_fullsized":"2024-09-16T03:39:18.640462+0000","mapping_epoch":32,"log_start":"139'133","ondisk_log_start":"139'133","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'118","last_scrub_stamp":"2024-09-16T03:38:39.704919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.704919+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":130,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:50.098482+0000","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4638864,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":64,"num_read_kb":77036,"num_write":22,"num_write_kb":6581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,15],"acting":[10,4,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":264,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.651445+0000","last_change":"2024-09-16T03:38:12.041048+0000","last_active":"2024-09-16T03:39:18.651445+0000","last_peered":"2024-09-16T03:39:18.651445+0000","last_clean":"2024-09-16T03:39:18.651445+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:39:18.651445+0000","last_undegraded":"2024-09-16T03:39:18.651445+0000","last_fullsized":"2024-09-16T03:39:18.651445+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:12.040917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:12.040917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:36.877949+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":276,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-16T03:39:19.599656+0000","last_change":"2024-09-16T03:38:34.928371+0000","last_active":"2024-09-16T03:39:19.599656+0000","last_peered":"2024-09-16T03:39:19.599656+0000","last_clean":"2024-09-16T03:39:19.599656+0000","last_became_active":"2024-09-16T03:38:33.484091+0000","last_became_peered":"2024-09-16T03:38:33.484091+0000","last_unstale":"2024-09-16T03:39:19.599656+0000","last_undegraded":"2024-09-16T03:39:19.599656+0000","last_fullsized":"2024-09-16T03:39:19.599656+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:34.928298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.928298+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:43.935101+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":257,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.638153+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:39:18.638153+0000","last_peered":"2024-09-16T03:39:18.638153+0000","last_clean":"2024-09-16T03:39:18.638153+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:39:18.638153+0000","last_undegraded":"2024-09-16T03:39:18.638153+0000","last_fullsized":"2024-09-16T03:39:18.638153+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:29.232558+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.10","version":"141'67","reported_seq":419,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.640351+0000","last_change":"2024-09-16T03:39:04.442655+0000","last_active":"2024-09-16T03:39:18.640351+0000","last_peered":"2024-09-16T03:39:18.640351+0000","last_clean":"2024-09-16T03:39:18.640351+0000","last_became_active":"2024-09-16T03:39:01.900921+0000","last_became_peered":"2024-09-16T03:39:01.900921+0000","last_unstale":"2024-09-16T03:39:18.640351+0000","last_undegraded":"2024-09-16T03:39:18.640351+0000","last_fullsized":"2024-09-16T03:39:18.640351+0000","mapping_epoch":72,"log_start":"118'64","ondisk_log_start":"118'64","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"118'64","last_scrub_stamp":"2024-09-16T03:39:04.442556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.442556+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.917777+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605151,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,4],"acting":[10,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"114'17","reported_seq":317,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.724049+0000","last_change":"2024-09-16T03:39:03.929961+0000","last_active":"2024-09-16T03:39:14.724049+0000","last_peered":"2024-09-16T03:39:14.724049+0000","last_clean":"2024-09-16T03:39:14.724049+0000","last_became_active":"2024-09-16T03:39:01.696690+0000","last_became_peered":"2024-09-16T03:39:01.696690+0000","last_unstale":"2024-09-16T03:39:14.724049+0000","last_undegraded":"2024-09-16T03:39:14.724049+0000","last_fullsized":"2024-09-16T03:39:14.724049+0000","mapping_epoch":63,"log_start":"107'14","ondisk_log_start":"107'14","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:03.929863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.929863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:26.527714+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2991679,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":7626,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"127'20","reported_seq":414,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.651397+0000","last_change":"2024-09-16T03:39:14.925442+0000","last_active":"2024-09-16T03:39:18.651397+0000","last_peered":"2024-09-16T03:39:18.651397+0000","last_clean":"2024-09-16T03:39:18.651397+0000","last_became_active":"2024-09-16T03:39:13.974170+0000","last_became_peered":"2024-09-16T03:39:13.974170+0000","last_unstale":"2024-09-16T03:39:18.651397+0000","last_undegraded":"2024-09-16T03:39:18.651397+0000","last_fullsized":"2024-09-16T03:39:18.651397+0000","mapping_epoch":138,"log_start":"106'17","ondisk_log_start":"106'17","created":50,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'20","last_scrub_stamp":"2024-09-16T03:39:14.925356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.925356+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":17,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:32.989972+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2000099,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.13","version":"142'76","reported_seq":497,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-16T03:39:19.599811+0000","last_change":"2024-09-16T03:39:15.224056+0000","last_active":"2024-09-16T03:39:19.599811+0000","last_peered":"2024-09-16T03:39:19.599811+0000","last_clean":"2024-09-16T03:39:19.599811+0000","last_became_active":"2024-09-16T03:39:15.223614+0000","last_became_peered":"2024-09-16T03:39:15.223614+0000","last_unstale":"2024-09-16T03:39:19.599811+0000","last_undegraded":"2024-09-16T03:39:19.599811+0000","last_fullsized":"2024-09-16T03:39:19.599811+0000","mapping_epoch":139,"log_start":"127'73","ondisk_log_start":"127'73","created":50,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'73","last_scrub_stamp":"2024-09-16T03:39:03.124583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.124583+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":71,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:09.327891+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2722800,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"104'20","reported_seq":334,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-16T03:39:14.824514+0000","last_change":"2024-09-16T03:39:01.898409+0000","last_active":"2024-09-16T03:39:14.824514+0000","last_peered":"2024-09-16T03:39:14.824514+0000","last_clean":"2024-09-16T03:39:14.824514+0000","last_became_active":"2024-09-16T03:39:01.596287+0000","last_became_peered":"2024-09-16T03:39:01.596287+0000","last_unstale":"2024-09-16T03:39:14.824514+0000","last_undegraded":"2024-09-16T03:39:14.824514+0000","last_fullsized":"2024-09-16T03:39:14.824514+0000","mapping_epoch":114,"log_start":"98'17","ondisk_log_start":"98'17","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:01.898331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.898331+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.472424+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"123'65","reported_seq":492,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.695424+0000","last_change":"2024-09-16T03:39:14.175562+0000","last_active":"2024-09-16T03:39:18.695424+0000","last_peered":"2024-09-16T03:39:18.695424+0000","last_clean":"2024-09-16T03:39:18.695424+0000","last_became_active":"2024-09-16T03:39:13.671769+0000","last_became_peered":"2024-09-16T03:39:13.671769+0000","last_unstale":"2024-09-16T03:39:18.695424+0000","last_undegraded":"2024-09-16T03:39:18.695424+0000","last_fullsized":"2024-09-16T03:39:18.695424+0000","mapping_epoch":138,"log_start":"123'62","ondisk_log_start":"123'62","created":50,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"123'65","last_scrub_stamp":"2024-09-16T03:39:14.175493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.175493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:31.666781+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"124'86","reported_seq":466,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.729748+0000","last_change":"2024-09-16T03:39:03.073530+0000","last_active":"2024-09-16T03:39:16.729748+0000","last_peered":"2024-09-16T03:39:16.729748+0000","last_clean":"2024-09-16T03:39:16.729748+0000","last_became_active":"2024-09-16T03:39:01.797805+0000","last_became_peered":"2024-09-16T03:39:01.797805+0000","last_unstale":"2024-09-16T03:39:16.729748+0000","last_undegraded":"2024-09-16T03:39:16.729748+0000","last_fullsized":"2024-09-16T03:39:16.729748+0000","mapping_epoch":115,"log_start":"124'82","ondisk_log_start":"124'82","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.073459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.073459+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":80,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:32.204501+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.17","version":"137'55","reported_seq":425,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.651551+0000","last_change":"2024-09-16T03:39:05.721286+0000","last_active":"2024-09-16T03:39:18.651551+0000","last_peered":"2024-09-16T03:39:18.651551+0000","last_clean":"2024-09-16T03:39:18.651551+0000","last_became_active":"2024-09-16T03:39:03.717420+0000","last_became_peered":"2024-09-16T03:39:03.717420+0000","last_unstale":"2024-09-16T03:39:18.651551+0000","last_undegraded":"2024-09-16T03:39:18.651551+0000","last_fullsized":"2024-09-16T03:39:18.651551+0000","mapping_epoch":128,"log_start":"131'52","ondisk_log_start":"131'52","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":51,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:39:34.790793+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1630328,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":62,"num_read_kb":54076,"num_write":15,"num_write_kb":4042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2516795,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"128'51","reported_seq":392,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.638291+0000","last_change":"2024-09-16T03:39:02.668794+0000","last_active":"2024-09-16T03:39:18.638291+0000","last_peered":"2024-09-16T03:39:18.638291+0000","last_clean":"2024-09-16T03:39:18.638291+0000","last_became_active":"2024-09-16T03:38:47.562641+0000","last_became_peered":"2024-09-16T03:38:47.562641+0000","last_unstale":"2024-09-16T03:39:18.638291+0000","last_undegraded":"2024-09-16T03:39:18.638291+0000","last_fullsized":"2024-09-16T03:39:18.638291+0000","mapping_epoch":112,"log_start":"124'48","ondisk_log_start":"124'48","created":50,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'48","last_scrub_stamp":"2024-09-16T03:39:02.668681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.668681+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":45,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.451476+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2067327,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":25040,"num_write":23,"num_write_kb":6049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.697534+0000","last_change":"2024-09-16T03:39:09.847101+0000","last_active":"2024-09-16T03:39:18.697534+0000","last_peered":"2024-09-16T03:39:18.697534+0000","last_clean":"2024-09-16T03:39:18.697534+0000","last_became_active":"2024-09-16T03:39:09.846691+0000","last_became_peered":"2024-09-16T03:39:09.846691+0000","last_unstale":"2024-09-16T03:39:18.697534+0000","last_undegraded":"2024-09-16T03:39:18.697534+0000","last_fullsized":"2024-09-16T03:39:18.697534+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:01.106840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.106840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.495875+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"128'125","reported_seq":593,"reported_epoch":139,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:13.872959+0000","last_change":"2024-09-16T03:38:47.159152+0000","last_active":"2024-09-16T03:39:13.872959+0000","last_peered":"2024-09-16T03:39:13.872959+0000","last_clean":"2024-09-16T03:38:43.521485+0000","last_became_active":"2024-09-16T03:38:46.957471+0000","last_became_peered":"2024-09-16T03:38:46.957471+0000","last_unstale":"2024-09-16T03:39:13.872959+0000","last_undegraded":"2024-09-16T03:39:13.872959+0000","last_fullsized":"2024-09-16T03:39:13.872959+0000","mapping_epoch":111,"log_start":"117'122","ondisk_log_start":"117'122","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'115","last_scrub_stamp":"2024-09-16T03:38:35.254488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.254488+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":122,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6739673,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":31161,"num_write":9,"num_write_kb":2915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,6,11],"acting":[3,8,10],"avail_no_missing":["3","8","10"],"object_location_counts":[{"shards":"3,8,10","objects":2}],"blocked_by":[],"up_primary":15,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"139'134","reported_seq":619,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.797290+0000","last_change":"2024-09-16T03:38:50.465770+0000","last_active":"2024-09-16T03:39:18.797290+0000","last_peered":"2024-09-16T03:39:18.797290+0000","last_clean":"2024-09-16T03:39:18.797290+0000","last_became_active":"2024-09-16T03:38:50.465463+0000","last_became_peered":"2024-09-16T03:38:50.465463+0000","last_unstale":"2024-09-16T03:39:18.797290+0000","last_undegraded":"2024-09-16T03:39:18.797290+0000","last_fullsized":"2024-09-16T03:39:18.797290+0000","mapping_epoch":115,"log_start":"138'131","ondisk_log_start":"138'131","created":100,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'107","last_scrub_stamp":"2024-09-16T03:38:39.905697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.905697+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":125,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.828899+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4941390,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":96001,"num_write":32,"num_write_kb":9493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2376197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"99'56","reported_seq":391,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.682077+0000","last_change":"2024-09-16T03:39:12.670900+0000","last_active":"2024-09-16T03:39:16.682077+0000","last_peered":"2024-09-16T03:39:16.682077+0000","last_clean":"2024-09-16T03:39:16.682077+0000","last_became_active":"2024-09-16T03:39:12.670496+0000","last_became_peered":"2024-09-16T03:39:12.670496+0000","last_unstale":"2024-09-16T03:39:16.682077+0000","last_undegraded":"2024-09-16T03:39:16.682077+0000","last_fullsized":"2024-09-16T03:39:16.682077+0000","mapping_epoch":137,"log_start":"90'53","ondisk_log_start":"90'53","created":100,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"99'56","last_scrub_stamp":"2024-09-16T03:39:00.988517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.988517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:21.733247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"92'14","reported_seq":295,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.695548+0000","last_change":"2024-09-16T03:38:49.092747+0000","last_active":"2024-09-16T03:39:18.695548+0000","last_peered":"2024-09-16T03:39:18.695548+0000","last_clean":"2024-09-16T03:39:18.695548+0000","last_became_active":"2024-09-16T03:38:48.661213+0000","last_became_peered":"2024-09-16T03:38:48.661213+0000","last_unstale":"2024-09-16T03:39:18.695548+0000","last_undegraded":"2024-09-16T03:39:18.695548+0000","last_fullsized":"2024-09-16T03:39:18.695548+0000","mapping_epoch":113,"log_start":"91'11","ondisk_log_start":"91'11","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:49.092687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.092687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.372348+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,12],"acting":[3,11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"92'59","reported_seq":404,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.651386+0000","last_change":"2024-09-16T03:38:48.561603+0000","last_active":"2024-09-16T03:39:18.651386+0000","last_peered":"2024-09-16T03:39:18.651386+0000","last_clean":"2024-09-16T03:39:18.651386+0000","last_became_active":"2024-09-16T03:38:48.561170+0000","last_became_peered":"2024-09-16T03:38:48.561170+0000","last_unstale":"2024-09-16T03:39:18.651386+0000","last_undegraded":"2024-09-16T03:39:18.651386+0000","last_fullsized":"2024-09-16T03:39:18.651386+0000","mapping_epoch":113,"log_start":"89'56","ondisk_log_start":"89'56","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'59","last_scrub_stamp":"2024-09-16T03:38:35.977646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.977646+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:38.632635+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,6],"acting":[14,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1f","version":"141'122","reported_seq":541,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:19.451409+0000","last_change":"2024-09-16T03:38:48.762617+0000","last_active":"2024-09-16T03:39:19.451409+0000","last_peered":"2024-09-16T03:39:19.451409+0000","last_clean":"2024-09-16T03:39:19.451409+0000","last_became_active":"2024-09-16T03:38:48.761213+0000","last_became_peered":"2024-09-16T03:38:48.761213+0000","last_unstale":"2024-09-16T03:39:19.451409+0000","last_undegraded":"2024-09-16T03:39:19.451409+0000","last_fullsized":"2024-09-16T03:39:19.451409+0000","mapping_epoch":113,"log_start":"137'119","ondisk_log_start":"137'119","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'95","last_scrub_stamp":"2024-09-16T03:38:37.201959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:37.201959+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":119,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.307408+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454086,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":72192,"num_write":39,"num_write_kb":10263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1704821,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,12],"acting":[3,10,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.20","version":"139'146","reported_seq":635,"reported_epoch":143,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:19.599597+0000","last_change":"2024-09-16T03:39:16.963754+0000","last_active":"2024-09-16T03:39:19.599597+0000","last_peered":"2024-09-16T03:39:19.599597+0000","last_clean":"2024-09-16T03:38:44.034266+0000","last_became_active":"2024-09-16T03:38:46.560752+0000","last_became_peered":"2024-09-16T03:38:46.560752+0000","last_unstale":"2024-09-16T03:39:19.599597+0000","last_undegraded":"2024-09-16T03:39:19.599597+0000","last_fullsized":"2024-09-16T03:39:19.599597+0000","mapping_epoch":111,"log_start":"127'143","ondisk_log_start":"127'143","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'118","last_scrub_stamp":"2024-09-16T03:38:36.698150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:36.698150+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":143,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10000264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":115889,"num_write":32,"num_write_kb":9785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[4,10,15],"avail_no_missing":["4","10","15"],"object_location_counts":[{"shards":"4,10,15","objects":3}],"blocked_by":[],"up_primary":8,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.21","version":"48'7","reported_seq":157,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.651338+0000","last_change":"2024-09-16T03:38:49.064947+0000","last_active":"2024-09-16T03:39:18.651338+0000","last_peered":"2024-09-16T03:39:18.651338+0000","last_clean":"2024-09-16T03:39:18.651338+0000","last_became_active":"2024-09-16T03:38:48.559431+0000","last_became_peered":"2024-09-16T03:38:48.559431+0000","last_unstale":"2024-09-16T03:39:18.651338+0000","last_undegraded":"2024-09-16T03:39:18.651338+0000","last_fullsized":"2024-09-16T03:39:18.651338+0000","mapping_epoch":113,"log_start":"40'4","ondisk_log_start":"40'4","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":6,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:49.064870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064870+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.310170+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,10],"acting":[14,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.22","version":"45'6","reported_seq":160,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.797791+0000","last_change":"2024-09-16T03:39:13.974174+0000","last_active":"2024-09-16T03:39:18.797791+0000","last_peered":"2024-09-16T03:39:18.797791+0000","last_clean":"2024-09-16T03:39:18.797791+0000","last_became_active":"2024-09-16T03:39:13.973813+0000","last_became_peered":"2024-09-16T03:39:13.973813+0000","last_unstale":"2024-09-16T03:39:18.797791+0000","last_undegraded":"2024-09-16T03:39:18.797791+0000","last_fullsized":"2024-09-16T03:39:18.797791+0000","mapping_epoch":138,"log_start":"39'3","ondisk_log_start":"39'3","created":100,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:39:00.968448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.968448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.929675+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,6],"acting":[9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.23","version":"142'41","reported_seq":375,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:19.157312+0000","last_change":"2024-09-16T03:38:49.464614+0000","last_active":"2024-09-16T03:39:19.157312+0000","last_peered":"2024-09-16T03:39:19.157312+0000","last_clean":"2024-09-16T03:39:19.157312+0000","last_became_active":"2024-09-16T03:38:49.463797+0000","last_became_peered":"2024-09-16T03:38:49.463797+0000","last_unstale":"2024-09-16T03:39:19.157312+0000","last_undegraded":"2024-09-16T03:39:19.157312+0000","last_fullsized":"2024-09-16T03:39:19.157312+0000","mapping_epoch":114,"log_start":"108'38","ondisk_log_start":"108'38","created":100,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:34.893524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.893524+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":37,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:52.829073+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672379,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":2300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,2],"acting":[14,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.24","version":"105'32","reported_seq":345,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.704456+0000","last_change":"2024-09-16T03:38:47.978956+0000","last_active":"2024-09-16T03:39:16.704456+0000","last_peered":"2024-09-16T03:39:16.704456+0000","last_clean":"2024-09-16T03:39:16.704456+0000","last_became_active":"2024-09-16T03:38:45.820201+0000","last_became_peered":"2024-09-16T03:38:45.820201+0000","last_unstale":"2024-09-16T03:39:16.704456+0000","last_undegraded":"2024-09-16T03:39:16.704456+0000","last_fullsized":"2024-09-16T03:39:16.704456+0000","mapping_epoch":110,"log_start":"83'29","ondisk_log_start":"83'29","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.978888+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:56.881865+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.25","version":"130'85","reported_seq":437,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.706102+0000","last_change":"2024-09-16T03:38:48.088833+0000","last_active":"2024-09-16T03:39:16.706102+0000","last_peered":"2024-09-16T03:39:16.706102+0000","last_clean":"2024-09-16T03:39:16.706102+0000","last_became_active":"2024-09-16T03:38:45.724757+0000","last_became_peered":"2024-09-16T03:38:45.724757+0000","last_unstale":"2024-09-16T03:39:16.706102+0000","last_undegraded":"2024-09-16T03:39:16.706102+0000","last_fullsized":"2024-09-16T03:39:16.706102+0000","mapping_epoch":110,"log_start":"116'82","ondisk_log_start":"116'82","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'77","last_scrub_stamp":"2024-09-16T03:38:48.088768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.088768+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":79,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:53.112635+0000","scrub_duration":388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2134555,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":2264,"num_write":10,"num_write_kb":2500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,7,11],"acting":[15,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.26","version":"137'57","reported_seq":422,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.740578+0000","last_change":"2024-09-16T03:38:49.477393+0000","last_active":"2024-09-16T03:39:18.740578+0000","last_peered":"2024-09-16T03:39:18.740578+0000","last_clean":"2024-09-16T03:39:18.740578+0000","last_became_active":"2024-09-16T03:38:45.926029+0000","last_became_peered":"2024-09-16T03:38:45.926029+0000","last_unstale":"2024-09-16T03:39:18.740578+0000","last_undegraded":"2024-09-16T03:39:18.740578+0000","last_fullsized":"2024-09-16T03:39:18.740578+0000","mapping_epoch":110,"log_start":"133'54","ondisk_log_start":"133'54","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:49.477297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.477297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.242430+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336239,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":24204,"num_write":15,"num_write_kb":4159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,4],"acting":[10,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.27","version":"109'66","reported_seq":405,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.645987+0000","last_change":"2024-09-16T03:38:47.511343+0000","last_active":"2024-09-16T03:39:18.645987+0000","last_peered":"2024-09-16T03:39:18.645987+0000","last_clean":"2024-09-16T03:39:18.645987+0000","last_became_active":"2024-09-16T03:38:45.824884+0000","last_became_peered":"2024-09-16T03:38:45.824884+0000","last_unstale":"2024-09-16T03:39:18.645987+0000","last_undegraded":"2024-09-16T03:39:18.645987+0000","last_fullsized":"2024-09-16T03:39:18.645987+0000","mapping_epoch":110,"log_start":"105'63","ondisk_log_start":"105'63","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'66","last_scrub_stamp":"2024-09-16T03:38:47.511252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.511252+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:55.884916+0000","scrub_duration":304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.28","version":"139'69","reported_seq":500,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.637766+0000","last_change":"2024-09-16T03:39:13.974976+0000","last_active":"2024-09-16T03:39:18.637766+0000","last_peered":"2024-09-16T03:39:18.637766+0000","last_clean":"2024-09-16T03:39:18.637766+0000","last_became_active":"2024-09-16T03:39:13.974464+0000","last_became_peered":"2024-09-16T03:39:13.974464+0000","last_unstale":"2024-09-16T03:39:18.637766+0000","last_undegraded":"2024-09-16T03:39:18.637766+0000","last_fullsized":"2024-09-16T03:39:18.637766+0000","mapping_epoch":138,"log_start":"118'66","ondisk_log_start":"118'66","created":110,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:48.551929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.551929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":65,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:10.916309+0000","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395113,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":46322,"num_write":5,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7221888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,9],"acting":[0,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":267,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.651342+0000","last_change":"2024-09-16T03:39:00.834877+0000","last_active":"2024-09-16T03:39:18.651342+0000","last_peered":"2024-09-16T03:39:18.651342+0000","last_clean":"2024-09-16T03:39:18.651342+0000","last_became_active":"2024-09-16T03:38:45.627369+0000","last_became_peered":"2024-09-16T03:38:45.627369+0000","last_unstale":"2024-09-16T03:39:18.651342+0000","last_undegraded":"2024-09-16T03:39:18.651342+0000","last_fullsized":"2024-09-16T03:39:18.651342+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:00.834742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.834742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.318226+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,6],"acting":[14,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.2a","version":"108'122","reported_seq":620,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.638282+0000","last_change":"2024-09-16T03:38:47.964525+0000","last_active":"2024-09-16T03:39:18.638282+0000","last_peered":"2024-09-16T03:39:18.638282+0000","last_clean":"2024-09-16T03:39:18.638282+0000","last_became_active":"2024-09-16T03:38:46.022108+0000","last_became_peered":"2024-09-16T03:38:46.022108+0000","last_unstale":"2024-09-16T03:39:18.638282+0000","last_undegraded":"2024-09-16T03:39:18.638282+0000","last_fullsized":"2024-09-16T03:39:18.638282+0000","mapping_epoch":110,"log_start":"107'115","ondisk_log_start":"107'115","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'122","last_scrub_stamp":"2024-09-16T03:38:47.964439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.964439+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":112,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:54.917123+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.2b","version":"140'127","reported_seq":600,"reported_epoch":141,"state":"active+clean","last_fresh":"2024-09-16T03:39:16.733610+0000","last_change":"2024-09-16T03:39:09.949533+0000","last_active":"2024-09-16T03:39:16.733610+0000","last_peered":"2024-09-16T03:39:16.733610+0000","last_clean":"2024-09-16T03:39:16.733610+0000","last_became_active":"2024-09-16T03:39:09.949146+0000","last_became_peered":"2024-09-16T03:39:09.949146+0000","last_unstale":"2024-09-16T03:39:16.733610+0000","last_undegraded":"2024-09-16T03:39:16.733610+0000","last_fullsized":"2024-09-16T03:39:16.733610+0000","mapping_epoch":134,"log_start":"127'124","ondisk_log_start":"127'124","created":110,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:49.064061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064061+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":124,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:16.396765+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2588344,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":3265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2c","version":"138'64","reported_seq":439,"reported_epoch":142,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:18.593357+0000","last_change":"2024-09-16T03:39:11.380628+0000","last_active":"2024-09-16T03:39:18.593357+0000","last_peered":"2024-09-16T03:39:18.593357+0000","last_clean":"2024-09-16T03:39:07.529785+0000","last_became_active":"2024-09-16T03:39:11.279455+0000","last_became_peered":"2024-09-16T03:39:11.279455+0000","last_unstale":"2024-09-16T03:39:18.593357+0000","last_undegraded":"2024-09-16T03:39:18.593357+0000","last_fullsized":"2024-09-16T03:39:18.593357+0000","mapping_epoch":135,"log_start":"118'61","ondisk_log_start":"118'61","created":110,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'58","last_scrub_stamp":"2024-09-16T03:38:47.889276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.889276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512696,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":18905,"num_write":5,"num_write_kb":1234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1262403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,9],"acting":[12,15,11],"avail_no_missing":["12","11","15"],"object_location_counts":[{"shards":"11,12,15","objects":1}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.2d","version":"92'14","reported_seq":348,"reported_epoch":142,"state":"active+clean","last_fresh":"2024-09-16T03:39:18.693444+0000","last_change":"2024-09-16T03:38:47.115739+0000","last_active":"2024-09-16T03:39:18.693444+0000","last_peered":"2024-09-16T03:39:18.693444+0000","last_clean":"2024-09-16T03:39:18.693444+0000","last_became_active":"2024-09-16T03:38:45.723438+0000","last_became_peered":"2024-09-16T03:38:45.723438+0000","last_unstale":"2024-09-16T03:39:18.693444+0000","last_undegraded":"2024-09-16T03:39:18.693444+0000","last_fullsized":"2024-09-16T03:39:18.693444+0000","mapping_epoch":110,"log_start":"91'11","ondisk_log_start":"91'11","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:47.115651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.115651+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:00.107802+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,10],"acting":[12,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":56,"stat_sum":{"num_bytes":124382580,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":1051,"num_read_kb":947232,"num_write":371,"num_write_kb":101607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":19356261,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":19,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221298688,"data_stored":220810667,"data_compressed":389631,"data_compressed_allocated":430080,"data_compressed_original":1110016,"omap_allocated":0,"internal_metadata":0},"log_size":171,"ondisk_log_size":171,"up":168,"acting":168,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":85951554,"num_objects":86,"num_object_clones":0,"num_object_copies":258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":946,"num_read_kb":29938,"num_write":1287,"num_write_kb":291436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":39649106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":182398976,"data_stored":194670125,"data_compressed":11815431,"data_compressed_allocated":12554240,"data_compressed_original":25219072,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":2644640,"data_compressed":159066,"data_compressed_allocated":204800,"data_compressed_original":2584576,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247839,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72824,"kb_used_data":40236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94299016,"statfs":{"total":96636764160,"available":96562192384,"internally_reserved":0,"allocated":41201664,"data_stored":41637944,"data_compressed":1301463,"data_compressed_allocated":1380352,"data_compressed_original":2392064,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247840,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53800,"kb_used_data":22612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318040,"statfs":{"total":96636764160,"available":96581672960,"internally_reserved":0,"allocated":23154688,"data_stored":24221122,"data_compressed":890468,"data_compressed_allocated":974848,"data_compressed_original":2576384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,7,8,9,10,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247839,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70124,"kb_used_data":38004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94301716,"statfs":{"total":96636764160,"available":96564957184,"internally_reserved":0,"allocated":38916096,"data_stored":40240084,"data_compressed":1556613,"data_compressed_allocated":1675264,"data_compressed_original":3543040,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280545,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53208,"kb_used_data":22020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318632,"statfs":{"total":96636764160,"available":96582279168,"internally_reserved":0,"allocated":22548480,"data_stored":23579967,"data_compressed":880190,"data_compressed_allocated":962560,"data_compressed_original":2510848,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,8,9,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280545,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":94489280544,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313249,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34000,"kb_used_data":6972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337840,"statfs":{"total":96636764160,"available":96601948160,"internally_reserved":0,"allocated":7139328,"data_stored":6719222,"data_compressed":34276,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280545,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94808,"kb_used_data":52772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94277032,"statfs":{"total":96636764160,"available":96539680768,"internally_reserved":0,"allocated":54038528,"data_stored":55357603,"data_compressed":1601968,"data_compressed_allocated":1724416,"data_compressed_original":3674112,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280544,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98944,"kb_used_data":63868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94272896,"statfs":{"total":96636764160,"available":96535445504,"internally_reserved":0,"allocated":65400832,"data_stored":67590908,"data_compressed":2332330,"data_compressed_allocated":2506752,"data_compressed_original":5332992,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313249,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280544,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35992,"kb_used_data":4804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335848,"statfs":{"total":96636764160,"available":96599908352,"internally_reserved":0,"allocated":4919296,"data_stored":4537075,"data_compressed":34276,"data_compressed_allocated":49152,"data_compressed_original":98304,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,3,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280544,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65272,"kb_used_data":34088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94306568,"statfs":{"total":96636764160,"available":96569925632,"internally_reserved":0,"allocated":34906112,"data_stored":35307937,"data_compressed":1159125,"data_compressed_allocated":1232896,"data_compressed_original":2179072,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280544,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44072,"kb_used_data":12884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327768,"statfs":{"total":96636764160,"available":96591634432,"internally_reserved":0,"allocated":13193216,"data_stored":13237897,"data_compressed":77699,"data_compressed_allocated":102400,"data_compressed_original":643072,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[2,3,4,6,7,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280545,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81532,"kb_used_data":43660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94290308,"statfs":{"total":96636764160,"available":96553275392,"internally_reserved":0,"allocated":44707840,"data_stored":44321430,"data_compressed":134118,"data_compressed_allocated":159744,"data_compressed_original":425984,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,3,4,5,6,7,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313248,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42704,"kb_used_data":11516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329136,"statfs":{"total":96636764160,"available":96593035264,"internally_reserved":0,"allocated":11792384,"data_stored":11353380,"data_compressed":57065,"data_compressed_allocated":73728,"data_compressed_original":163840,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,3,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247840,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54712,"kb_used_data":23524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94317128,"statfs":{"total":96636764160,"available":96580739072,"internally_reserved":0,"allocated":24088576,"data_stored":24556927,"data_compressed":1271709,"data_compressed_allocated":1347584,"data_compressed_original":2306048,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13860864,"data_stored":14856794,"data_compressed":814666,"data_compressed_allocated":872448,"data_compressed_original":1900544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":13422592,"data_stored":14113910,"data_compressed":606276,"data_compressed_allocated":643072,"data_compressed_original":1376256,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12906496,"data_stored":13784230,"data_compressed":1187486,"data_compressed_allocated":1245184,"data_compressed_original":2142208,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26824704,"data_stored":28548278,"data_compressed":1497930,"data_compressed_allocated":1597440,"data_compressed_original":3379200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":38690816,"data_stored":41329067,"data_compressed":2268300,"data_compressed_allocated":2424832,"data_compressed_original":5148672,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":26152960,"data_stored":27905732,"data_compressed":1535355,"data_compressed_allocated":1638400,"data_compressed_original":3444736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12824576,"data_stored":13642728,"data_compressed":1115301,"data_compressed_allocated":1171456,"data_compressed_original":2015232,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":13303808,"data_stored":14306653,"data_compressed":814666,"data_compressed_allocated":872448,"data_compressed_original":1900544,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":12906496,"data_stored":13784230,"data_compressed":1187486,"data_compressed_allocated":1245184,"data_compressed_original":2142208,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8523776,"data_stored":8471963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9904128,"data_stored":9871428,"data_compressed":49947,"data_compressed_allocated":53248,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":27107328,"data_stored":27056941,"data_compressed":32337,"data_compressed_allocated":36864,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25411584,"data_stored":25335244,"data_compressed":29754,"data_compressed_allocated":32768,"data_compressed_original":86016,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13172736,"data_stored":13183621,"data_compressed":44673,"data_compressed_allocated":49152,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3657728,"data_stored":3629400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20783104,"data_stored":20746677,"data_compressed":9548,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":13086720,"data_stored":13023699,"data_compressed":10762,"data_compressed_allocated":12288,"data_compressed_original":20480,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":43401216,"data_stored":43401877,"data_compressed":99842,"data_compressed_allocated":110592,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8478720,"data_stored":8473398,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":10465280,"data_stored":10431971,"data_compressed":22789,"data_compressed_allocated":24576,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":7819264,"data_stored":7780071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":28397568,"data_stored":28324337,"data_compressed":79701,"data_compressed_allocated":86016,"data_compressed_original":151552,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:39:21.560 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:21.550+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:21.563 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T03:39:21.563 INFO:tasks.rados.rados.0.smithi157.stdout:1814: finishing write tid 1 to smithi15736696-22 2024-09-16T03:39:21.563 INFO:tasks.rados.rados.0.smithi157.stdout:1815: finishing write tid 3 to smithi15736696-44 2024-09-16T03:39:21.563 INFO:tasks.rados.rados.0.smithi157.stdout:1812: finishing write tid 2 to smithi15736696-4 2024-09-16T03:39:21.563 INFO:tasks.rados.rados.0.smithi157.stdout:1812: finishing write tid 3 to smithi15736696-4 2024-09-16T03:39:21.563 INFO:tasks.rados.rados.0.smithi157.stdout:1812: finishing write tid 4 to smithi15736696-4 2024-09-16T03:39:21.563 INFO:tasks.rados.rados.0.smithi157.stdout:1812: oid 4 updating version 0 to 157 2024-09-16T03:39:21.563 INFO:tasks.rados.rados.0.smithi157.stdout:1812: oid 4 version 157 is already newer than 155 2024-09-16T03:39:21.563 INFO:tasks.rados.rados.0.smithi157.stdout:1812: oid 4 version 157 is already newer than 156 2024-09-16T03:39:21.564 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 157 (ObjNum 403 snap 0 seq_num 403) dirty exists 2024-09-16T03:39:21.564 INFO:tasks.rados.rados.0.smithi157.stdout:1812: left oid 4 (ObjNum 403 snap 0 seq_num 403) 2024-09-16T03:39:21.564 INFO:tasks.rados.rados.0.smithi157.stdout:1815: finishing write tid 4 to smithi15736696-44 2024-09-16T03:39:21.564 INFO:tasks.rados.rados.0.smithi157.stdout:1822: seq_num 407 ranges {440449=793052,1933583=510464} 2024-09-16T03:39:21.564 INFO:tasks.rados.rados.0.smithi157.stdout:1822: writing smithi15736696-31 from 440449 to 1233501 tid 1 2024-09-16T03:39:21.564 INFO:tasks.rados.rados.0.smithi157.stdout:1822: writing smithi15736696-31 from 1933583 to 2444047 tid 2 2024-09-16T03:39:21.564 INFO:tasks.rados.rados.0.smithi157.stdout:1812: done (9 left) 2024-09-16T03:39:21.564 INFO:tasks.rados.rados.0.smithi157.stdout:1816: done (8 left) 2024-09-16T03:39:21.564 INFO:tasks.rados.rados.0.smithi157.stdout:1817: done (7 left) 2024-09-16T03:39:21.564 INFO:tasks.rados.rados.0.smithi157.stdout:1818: done (6 left) 2024-09-16T03:39:21.564 INFO:tasks.rados.rados.0.smithi157.stdout:1820: done (5 left) 2024-09-16T03:39:21.564 INFO:tasks.rados.rados.0.smithi157.stdout:1823: read oid 15 snap -1 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:1823: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:1814: finishing write tid 2 to smithi15736696-22 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:1819: finishing write tid 1 to smithi15736696-17 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:1824: tier_promote oid 32 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:1815: finishing write tid 5 to smithi15736696-44 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:1815: oid 44 updating version 0 to 81 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:1815: oid 44 updating version 81 to 82 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:1815: oid 44 version 82 is already newer than 79 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:1815: oid 44 version 82 is already newer than 80 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 82 (ObjNum 405 snap 0 seq_num 405) dirty exists 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:1815: left oid 44 (ObjNum 405 snap 0 seq_num 405) 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:1815: done (6 left) 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:1825: read oid 33 snap -1 2024-09-16T03:39:21.565 INFO:tasks.rados.rados.0.smithi157.stdout:1825: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-16T03:39:21.566 INFO:tasks.rados.rados.0.smithi157.stdout:1814: finishing write tid 3 to smithi15736696-22 2024-09-16T03:39:21.566 INFO:tasks.rados.rados.0.smithi157.stdout:1814: finishing write tid 4 to smithi15736696-22 2024-09-16T03:39:21.566 INFO:tasks.rados.rados.0.smithi157.stdout:1814: oid 22 updating version 0 to 102 2024-09-16T03:39:21.566 INFO:tasks.rados.rados.0.smithi157.stdout:1814: oid 22 updating version 102 to 103 2024-09-16T03:39:21.566 INFO:tasks.rados.rados.0.smithi157.stdout:1814: oid 22 version 103 is already newer than 101 2024-09-16T03:39:21.566 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 103 (ObjNum 404 snap 0 seq_num 404) dirty exists 2024-09-16T03:39:21.566 INFO:tasks.rados.rados.0.smithi157.stdout:1814: left oid 22 (ObjNum 404 snap 0 seq_num 404) 2024-09-16T03:39:21.566 INFO:tasks.rados.rados.0.smithi157.stdout:1819: finishing write tid 2 to smithi15736696-17 2024-09-16T03:39:21.566 INFO:tasks.rados.rados.0.smithi157.stdout:1819: finishing write tid 3 to smithi15736696-17 2024-09-16T03:39:21.566 INFO:tasks.rados.rados.0.smithi157.stdout:1819: finishing write tid 4 to smithi15736696-17 2024-09-16T03:39:21.566 INFO:tasks.rados.rados.0.smithi157.stdout:1819: oid 17 updating version 0 to 137 2024-09-16T03:39:21.566 INFO:tasks.rados.rados.0.smithi157.stdout:1819: oid 17 updating version 137 to 139 2024-09-16T03:39:21.566 INFO:tasks.rados.rados.0.smithi157.stdout:1819: oid 17 version 139 is already newer than 138 2024-09-16T03:39:21.567 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 139 (ObjNum 406 snap 0 seq_num 406) dirty exists 2024-09-16T03:39:21.567 INFO:tasks.rados.rados.0.smithi157.stdout:1819: left oid 17 (ObjNum 406 snap 0 seq_num 406) 2024-09-16T03:39:21.567 INFO:tasks.rados.rados.0.smithi157.stdout:1821: finishing set_chunk to oid 8 2024-09-16T03:39:21.567 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 8 set_chunk returned error code -95 offset: 947712 length: 256512 2024-09-16T03:39:21.567 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 359 snap 0 seq_num 359) dirty exists 2024-09-16T03:39:21.567 INFO:tasks.rados.rados.0.smithi157.stdout:1826: read oid 11 snap -1 2024-09-16T03:39:21.567 INFO:tasks.rados.rados.0.smithi157.stdout:1826: expect deleted 2024-09-16T03:39:21.568 INFO:tasks.rados.rados.0.smithi157.stdout:1814: done (7 left) 2024-09-16T03:39:21.568 INFO:tasks.rados.rados.0.smithi157.stdout:1819: done (6 left) 2024-09-16T03:39:21.568 INFO:tasks.rados.rados.0.smithi157.stdout:1821: done (5 left) 2024-09-16T03:39:21.568 INFO:tasks.rados.rados.0.smithi157.stdout:1827: tier_evict oid 7 2024-09-16T03:39:21.568 INFO:tasks.rados.rados.0.smithi157.stdout:1827: tier_evict oid 7 snap -1 2024-09-16T03:39:21.568 INFO:tasks.rados.rados.0.smithi157.stdout:1828: read oid 14 snap -1 2024-09-16T03:39:21.568 INFO:tasks.rados.rados.0.smithi157.stdout:1828: expect (ObjNum 398 snap 0 seq_num 398) 2024-09-16T03:39:21.568 INFO:tasks.rados.rados.0.smithi157.stdout:1829: write oid 36 current snap is 0 2024-09-16T03:39:21.568 INFO:tasks.rados.rados.0.smithi157.stdout:1829: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2024-09-16T03:39:21.568 INFO:tasks.rados.rados.0.smithi157.stdout:1829: writing smithi15736696-36 from 457256 to 925557 tid 1 2024-09-16T03:39:21.568 INFO:tasks.rados.rados.0.smithi157.stdout:1829: writing smithi15736696-36 from 1592065 to 2362540 tid 2 2024-09-16T03:39:21.568 INFO:tasks.rados.rados.0.smithi157.stdout:1829: writing smithi15736696-36 from 2854581 to 2857256 tid 3 2024-09-16T03:39:21.568 INFO:tasks.rados.rados.0.smithi157.stdout:1830: set_chunk oid 46 target oid 2024-09-16T03:39:21.569 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:39:21.569 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:39:21.569 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T03:39:21.569 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.975222 2024-09-16T03:39:21.569 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 0.9752221562504411 2024-09-16T03:39:21.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:39:21.687 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:21.685+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:21.714 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:21.714 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:21.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.714 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4401 4.e 4:70297eed:::smithi15736696-31:head [write 440449~793052 in=793052b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-16T03:39:21.715 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:21.512984+0000", 2024-09-16T03:39:21.715 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.20011242800000001, 2024-09-16T03:39:21.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20013953700000001, 2024-09-16T03:39:21.715 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.715 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.715 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:39:21.715 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.715 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.715 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.715 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4401 2024-09-16T03:39:21.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.715 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.512984+0000", 2024-09-16T03:39:21.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.512984+0000", 2024-09-16T03:39:21.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.512986+0000", 2024-09-16T03:39:21.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0090000000000002e-06 2024-09-16T03:39:21.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.513888+0000", 2024-09-16T03:39:21.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090152500000000003 2024-09-16T03:39:21.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.513903+0000", 2024-09-16T03:39:21.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4712000000000001e-05 2024-09-16T03:39:21.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.513910+0000", 2024-09-16T03:39:21.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3749999999999997e-06 2024-09-16T03:39:21.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:21.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.620243+0000", 2024-09-16T03:39:21.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106332564 2024-09-16T03:39:21.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:21.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.620337+0000", 2024-09-16T03:39:21.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4155999999999998e-05 2024-09-16T03:39:21.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:39:21.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.620529+0000", 2024-09-16T03:39:21.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191817 2024-09-16T03:39:21.719 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.719 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.719 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4402 4.e 4.b77e940e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:21.519523+0000", 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.193574255, 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19364279400000001, 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4402 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.720 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.519523+0000", 2024-09-16T03:39:21.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.519523+0000", 2024-09-16T03:39:21.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.519524+0000", 2024-09-16T03:39:21.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.328e-06 2024-09-16T03:39:21.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520088+0000", 2024-09-16T03:39:21.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056399800000000002 2024-09-16T03:39:21.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520101+0000", 2024-09-16T03:39:21.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2763e-05 2024-09-16T03:39:21.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520106+0000", 2024-09-16T03:39:21.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5879999999999997e-06 2024-09-16T03:39:21.723 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.723 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.723 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.723 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4403 4.e 4.b77e940e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-16T03:39:21.724 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:21.520118+0000", 2024-09-16T03:39:21.724 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.19297931500000001, 2024-09-16T03:39:21.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19307426599999999, 2024-09-16T03:39:21.724 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.724 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.724 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:39:21.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.724 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4403 2024-09-16T03:39:21.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.724 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520118+0000", 2024-09-16T03:39:21.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520118+0000", 2024-09-16T03:39:21.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520118+0000", 2024-09-16T03:39:21.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7199999999999998e-07 2024-09-16T03:39:21.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520122+0000", 2024-09-16T03:39:21.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8689999999999997e-06 2024-09-16T03:39:21.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520130+0000", 2024-09-16T03:39:21.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7770000000000002e-06 2024-09-16T03:39:21.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520136+0000", 2024-09-16T03:39:21.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0209999999999996e-06 2024-09-16T03:39:21.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.727 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.727 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.727 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4404 4.e 4.b77e940e (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e144)", 2024-09-16T03:39:21.727 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:21.520143+0000", 2024-09-16T03:39:21.727 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.19295345799999999, 2024-09-16T03:39:21.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.193073157, 2024-09-16T03:39:21.727 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.727 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.728 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:39:21.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:21.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:21.728 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:21.728 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4404 2024-09-16T03:39:21.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.728 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520143+0000", 2024-09-16T03:39:21.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520143+0000", 2024-09-16T03:39:21.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520144+0000", 2024-09-16T03:39:21.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.75e-07 2024-09-16T03:39:21.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520146+0000", 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.939e-06 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.520151+0000", 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3739999999999997e-06 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.620233+0000", 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100081508 2024-09-16T03:39:21.730 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4628.0:4401 4.e e144/126)", 2024-09-16T03:39:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:21.626279+0000", 2024-09-16T03:39:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.086817722, 2024-09-16T03:39:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086962854000000006, 2024-09-16T03:39:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "no flag points reached", 2024-09-16T03:39:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:21.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.626279+0000", 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.626279+0000", 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.626281+0000", 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.756e-06 2024-09-16T03:39:21.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:21.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.626288+0000", 2024-09-16T03:39:21.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1860000000000003e-06 2024-09-16T03:39:21.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:21.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:21.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:21.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.626300+0000", 2024-09-16T03:39:21.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1625e-05 2024-09-16T03:39:21.734 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.734 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:21.734 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.734 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:21.734 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:39:21.734 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 5 2024-09-16T03:39:21.734 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:21.735 INFO:tasks.rados.rados.0.smithi157.stdout:1830: set_chunk oid 46 offset: 1169920 length: 882688 target oid 6f384269f2e00ce220e1ec79cd11144a182a7126eccc6f5c3255d33140c160ac offset: 0 snap -1 2024-09-16T03:39:21.736 INFO:tasks.rados.rados.0.smithi157.stdout:1831: read oid 3 snap -1 2024-09-16T03:39:21.736 INFO:tasks.rados.rados.0.smithi157.stdout:1831: expect deleted 2024-09-16T03:39:21.736 INFO:tasks.rados.rados.0.smithi157.stdout:1832: read oid 49 snap -1 2024-09-16T03:39:21.736 INFO:tasks.rados.rados.0.smithi157.stdout:1832: expect (ObjNum 316 snap 0 seq_num 316) 2024-09-16T03:39:21.736 INFO:tasks.rados.rados.0.smithi157.stdout:1833: set_chunk oid 43 target oid 2024-09-16T03:39:21.736 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:39:21.753 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:21.750+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:21.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:39:21.854 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:21.852+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:21.890 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:21.890 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:21.890 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:21.891 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:21.891 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:21.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:39:21.909 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:21.910 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:21.929 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:21.929 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:21.947 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:21.947 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:21.954 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:21.952+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:21.963 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:21.964 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:21.980 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:21.980 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:22.013 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:22.013 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:22.034 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:22.034 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:22.035 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:22.036 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:22.044 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:39:22.053 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:22.053 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:22.055 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:22.053+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:22.069 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:22.069 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:22.078 INFO:tasks.rados.rados.0.smithi157.stdout:1833: set_chunk oid 43 offset: 324608 length: 375296 target oid 87f4d84dd7d5642c7a0c2b0da9d6b3891747f2805421a5e7b963a310b9d349dd offset: 0 snap -1 2024-09-16T03:39:22.078 INFO:tasks.rados.rados.0.smithi157.stdout:1834: tier_flush oid 35 2024-09-16T03:39:22.078 INFO:tasks.rados.rados.0.smithi157.stdout:1834: tier_flush oid 35 snap -1 2024-09-16T03:39:22.078 INFO:tasks.rados.rados.0.smithi157.stdout:1824: got (0) Success 2024-09-16T03:39:22.078 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 20 (ObjNum 280 snap 0 seq_num 280) dirty exists 2024-09-16T03:39:22.079 INFO:tasks.rados.rados.0.smithi157.stdout:1825: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-16T03:39:22.079 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:22.086 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:22.086 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:22.103 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:22.103 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:22.121 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:22.121 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:22.130 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2048067 2024-09-16T03:39:22.137 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:22.137 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:22.154 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:22.155 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:22.156 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:22.154+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:22.160 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:22.160 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:22.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:39:22.171 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:22.171 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:22.187 INFO:tasks.rados.rados.0.smithi157.stdout:1823: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-16T03:39:22.195 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:22.196 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:22.257 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:22.255+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:22.277 INFO:tasks.rados.rados.0.smithi157.stdout:1828: expect (ObjNum 398 snap 0 seq_num 398) 2024-09-16T03:39:22.277 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 51847 2024-09-16T03:39:22.278 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:22.279 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:22.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:39:22.321 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:22.357 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:22.356+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:22.378 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1152203 2024-09-16T03:39:22.405 INFO:tasks.rados.rados.0.smithi157.stdout:1822: finishing write tid 1 to smithi15736696-31 2024-09-16T03:39:22.405 INFO:tasks.rados.rados.0.smithi157.stdout:1822: finishing write tid 2 to smithi15736696-31 2024-09-16T03:39:22.405 INFO:tasks.rados.rados.0.smithi157.stdout:1827: got (22) Invalid argument 2024-09-16T03:39:22.405 INFO:tasks.rados.rados.0.smithi157.stdout:1823: done (12 left) 2024-09-16T03:39:22.405 INFO:tasks.rados.rados.0.smithi157.stdout:1824: done (11 left) 2024-09-16T03:39:22.405 INFO:tasks.rados.rados.0.smithi157.stdout:1825: done (10 left) 2024-09-16T03:39:22.405 INFO:tasks.rados.rados.0.smithi157.stdout:1826: done (9 left) 2024-09-16T03:39:22.405 INFO:tasks.rados.rados.0.smithi157.stdout:1827: done (8 left) 2024-09-16T03:39:22.405 INFO:tasks.rados.rados.0.smithi157.stdout:1828: done (7 left) 2024-09-16T03:39:22.405 INFO:tasks.rados.rados.0.smithi157.stdout:1835: read oid 18 snap -1 2024-09-16T03:39:22.406 INFO:tasks.rados.rados.0.smithi157.stdout:1835: expect deleted 2024-09-16T03:39:22.406 INFO:tasks.rados.rados.0.smithi157.stdout:1836: read oid 5 snap -1 2024-09-16T03:39:22.406 INFO:tasks.rados.rados.0.smithi157.stdout:1836: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-16T03:39:22.406 INFO:tasks.rados.rados.0.smithi157.stdout:1829: finishing write tid 1 to smithi15736696-36 2024-09-16T03:39:22.406 INFO:tasks.rados.rados.0.smithi157.stdout:1832: expect (ObjNum 316 snap 0 seq_num 316) 2024-09-16T03:39:22.435 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:22.435 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:22.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:39:22.458 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:22.456+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:22.462 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:22.552 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:22.552 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:22.558 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:22.557+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:22.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:39:22.581 INFO:tasks.rados.rados.0.smithi157.stdout:1830: finishing set_chunk to oid 46 2024-09-16T03:39:22.581 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 46 v 131 (ObjNum 383 snap 0 seq_num 383) dirty exists 2024-09-16T03:39:22.581 INFO:tasks.rados.rados.0.smithi157.stdout:1822: finishing write tid 3 to smithi15736696-31 2024-09-16T03:39:22.581 INFO:tasks.rados.rados.0.smithi157.stdout:1822: finishing write tid 4 to smithi15736696-31 2024-09-16T03:39:22.581 INFO:tasks.rados.rados.0.smithi157.stdout:1822: oid 31 updating version 0 to 85 2024-09-16T03:39:22.581 INFO:tasks.rados.rados.0.smithi157.stdout:1822: oid 31 version 85 is already newer than 84 2024-09-16T03:39:22.581 INFO:tasks.rados.rados.0.smithi157.stdout:1822: oid 31 version 85 is already newer than 83 2024-09-16T03:39:22.581 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 85 (ObjNum 407 snap 0 seq_num 407) dirty exists 2024-09-16T03:39:22.581 INFO:tasks.rados.rados.0.smithi157.stdout:1822: left oid 31 (ObjNum 407 snap 0 seq_num 407) 2024-09-16T03:39:22.581 INFO:tasks.rados.rados.0.smithi157.stdout:1837: read oid 24 snap -1 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1837: expect (ObjNum 333 snap 0 seq_num 333) 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1822: done (9 left) 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1830: done (8 left) 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1831: done (7 left) 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1832: done (6 left) 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1838: read oid 21 snap -1 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1838: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1839: read oid 2 snap -1 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1839: expect deleted 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1840: tier_evict oid 19 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1840: tier_evict oid 19 snap -1 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1829: finishing write tid 2 to smithi15736696-36 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1829: finishing write tid 3 to smithi15736696-36 2024-09-16T03:39:22.582 INFO:tasks.rados.rados.0.smithi157.stdout:1829: finishing write tid 4 to smithi15736696-36 2024-09-16T03:39:22.583 INFO:tasks.rados.rados.0.smithi157.stdout:1829: finishing write tid 5 to smithi15736696-36 2024-09-16T03:39:22.583 INFO:tasks.rados.rados.0.smithi157.stdout:1829: oid 36 updating version 0 to 127 2024-09-16T03:39:22.583 INFO:tasks.rados.rados.0.smithi157.stdout:1829: oid 36 version 127 is already newer than 126 2024-09-16T03:39:22.583 INFO:tasks.rados.rados.0.smithi157.stdout:1829: oid 36 updating version 127 to 128 2024-09-16T03:39:22.583 INFO:tasks.rados.rados.0.smithi157.stdout:1829: oid 36 updating version 128 to 129 2024-09-16T03:39:22.583 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 129 (ObjNum 408 snap 0 seq_num 408) dirty exists 2024-09-16T03:39:22.583 INFO:tasks.rados.rados.0.smithi157.stdout:1829: left oid 36 (ObjNum 408 snap 0 seq_num 408) 2024-09-16T03:39:22.583 INFO:tasks.rados.rados.0.smithi157.stdout:1833: finishing set_chunk to oid 43 2024-09-16T03:39:22.583 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 88 (ObjNum 382 snap 0 seq_num 382) dirty exists 2024-09-16T03:39:22.583 INFO:tasks.rados.rados.0.smithi157.stdout:1836: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-16T03:39:22.583 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 381604 2024-09-16T03:39:22.608 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:22.606+0000 7f49439d4640 -1 osd.4 144 handle_osd_map injecting map crc failure 2024-09-16T03:39:22.610 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.4 primary-affinity to 0.975222 (8639122) 2024-09-16T03:39:22.610 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:39:22.660 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:22.658+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:22.688 INFO:tasks.rados.rados.0.smithi157.stdout:1840: got (22) Invalid argument 2024-09-16T03:39:22.689 INFO:tasks.rados.rados.0.smithi157.stdout:1834: got (0) Success 2024-09-16T03:39:22.689 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 0 (ObjNum 392 snap 0 seq_num 392) dirty exists 2024-09-16T03:39:22.689 INFO:tasks.rados.rados.0.smithi157.stdout:1837: expect (ObjNum 333 snap 0 seq_num 333) 2024-09-16T03:39:22.700 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1239530 2024-09-16T03:39:22.712 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:22.712 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:22.721 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:39:22.761 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:22.759+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:22.818 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1356818 2024-09-16T03:39:22.829 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:22.829 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:22.837 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:39:22.862 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:22.860+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:22.866 INFO:tasks.rados.rados.0.smithi157.stdout:1829: done (8 left) 2024-09-16T03:39:22.866 INFO:tasks.rados.rados.0.smithi157.stdout:1833: done (7 left) 2024-09-16T03:39:22.866 INFO:tasks.rados.rados.0.smithi157.stdout:1834: done (6 left) 2024-09-16T03:39:22.867 INFO:tasks.rados.rados.0.smithi157.stdout:1835: done (5 left) 2024-09-16T03:39:22.867 INFO:tasks.rados.rados.0.smithi157.stdout:1836: done (4 left) 2024-09-16T03:39:22.867 INFO:tasks.rados.rados.0.smithi157.stdout:1837: done (3 left) 2024-09-16T03:39:22.867 INFO:tasks.rados.rados.0.smithi157.stdout:1839: done (2 left) 2024-09-16T03:39:22.867 INFO:tasks.rados.rados.0.smithi157.stdout:1840: done (1 left) 2024-09-16T03:39:22.867 INFO:tasks.rados.rados.0.smithi157.stdout:1841: delete oid 25 current snap is 0 2024-09-16T03:39:22.872 INFO:tasks.rados.rados.0.smithi157.stdout:1838: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-16T03:39:22.936 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:22.953 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:22.953 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:22.962 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:39:22.963 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:22.960+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:23.063 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:23.061+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:23.065 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:23.065 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:23.074 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:39:23.164 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:23.162+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:23.169 INFO:tasks.rados.rados.0.smithi157.stdout:1838: done (1 left) 2024-09-16T03:39:23.169 INFO:tasks.rados.rados.0.smithi157.stdout:1841: done (0 left) 2024-09-16T03:39:23.169 INFO:tasks.rados.rados.0.smithi157.stdout:1842: read oid 2 snap -1 2024-09-16T03:39:23.169 INFO:tasks.rados.rados.0.smithi157.stdout:1842: expect deleted 2024-09-16T03:39:23.170 INFO:tasks.rados.rados.0.smithi157.stdout:1843: read oid 42 snap -1 2024-09-16T03:39:23.170 INFO:tasks.rados.rados.0.smithi157.stdout:1843: expect (ObjNum 400 snap 0 seq_num 400) 2024-09-16T03:39:23.170 INFO:tasks.rados.rados.0.smithi157.stdout:1844: read oid 22 snap -1 2024-09-16T03:39:23.170 INFO:tasks.rados.rados.0.smithi157.stdout:1844: expect (ObjNum 404 snap 0 seq_num 404) 2024-09-16T03:39:23.170 INFO:tasks.rados.rados.0.smithi157.stdout:1845: write oid 44 current snap is 0 2024-09-16T03:39:23.170 INFO:tasks.rados.rados.0.smithi157.stdout:1845: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2024-09-16T03:39:23.175 INFO:tasks.rados.rados.0.smithi157.stdout:1845: writing smithi15736696-44 from 490870 to 1109669 tid 1 2024-09-16T03:39:23.177 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:23.177 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:23.182 INFO:tasks.rados.rados.0.smithi157.stdout:1845: writing smithi15736696-44 from 1592676 to 2366820 tid 2 2024-09-16T03:39:23.186 INFO:tasks.rados.rados.0.smithi157.stdout:1845: writing smithi15736696-44 from 2793074 to 2890870 tid 3 2024-09-16T03:39:23.186 INFO:tasks.rados.rados.0.smithi157.stdout:1846: read oid 48 snap -1 2024-09-16T03:39:23.186 INFO:tasks.rados.rados.0.smithi157.stdout:1846: expect (ObjNum 367 snap 0 seq_num 367) 2024-09-16T03:39:23.186 INFO:tasks.rados.rados.0.smithi157.stdout:1847: read oid 50 snap -1 2024-09-16T03:39:23.186 INFO:tasks.rados.rados.0.smithi157.stdout:1847: expect (ObjNum 355 snap 0 seq_num 355) 2024-09-16T03:39:23.187 INFO:tasks.rados.rados.0.smithi157.stdout:1848: read oid 16 snap -1 2024-09-16T03:39:23.187 INFO:tasks.rados.rados.0.smithi157.stdout:1848: expect (ObjNum 388 snap 0 seq_num 388) 2024-09-16T03:39:23.187 INFO:tasks.rados.rados.0.smithi157.stdout:1849: read oid 15 snap -1 2024-09-16T03:39:23.187 INFO:tasks.rados.rados.0.smithi157.stdout:1849: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-16T03:39:23.187 INFO:tasks.rados.rados.0.smithi157.stdout:1850: read oid 49 snap -1 2024-09-16T03:39:23.187 INFO:tasks.rados.rados.0.smithi157.stdout:1850: expect (ObjNum 316 snap 0 seq_num 316) 2024-09-16T03:39:23.187 INFO:tasks.rados.rados.0.smithi157.stdout:1851: read oid 13 snap -1 2024-09-16T03:39:23.187 INFO:tasks.rados.rados.0.smithi157.stdout:1851: expect (ObjNum 401 snap 0 seq_num 401) 2024-09-16T03:39:23.187 INFO:tasks.rados.rados.0.smithi157.stdout:1852: read oid 45 snap -1 2024-09-16T03:39:23.187 INFO:tasks.rados.rados.0.smithi157.stdout:1852: expect (ObjNum 325 snap 0 seq_num 325) 2024-09-16T03:39:23.187 INFO:tasks.rados.rados.0.smithi157.stdout:1853: set_chunk oid 11 target oid 2024-09-16T03:39:23.187 INFO:tasks.rados.rados.0.smithi157.stdout:1853: set_chunk oid 11 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:39:23.187 INFO:tasks.rados.rados.0.smithi157.stdout:1854: read oid 40 snap -1 2024-09-16T03:39:23.188 INFO:tasks.rados.rados.0.smithi157.stdout:1854: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:39:23.188 INFO:tasks.rados.rados.0.smithi157.stdout:1855: write (excl) oid 4 current snap is 0 2024-09-16T03:39:23.188 INFO:tasks.rados.rados.0.smithi157.stdout:1855: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2024-09-16T03:39:23.189 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:39:23.190 INFO:tasks.rados.rados.0.smithi157.stdout:1855: writing smithi15736696-4 from 507677 to 918078 tid 1 2024-09-16T03:39:23.196 INFO:tasks.rados.rados.0.smithi157.stdout:1855: writing smithi15736696-4 from 1367511 to 1885313 tid 2 2024-09-16T03:39:23.201 INFO:tasks.rados.rados.0.smithi157.stdout:1855: writing smithi15736696-4 from 2420497 to 2907677 tid 3 2024-09-16T03:39:23.201 INFO:tasks.rados.rados.0.smithi157.stdout:1856: set_chunk oid 38 target oid 2024-09-16T03:39:23.265 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:23.263+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:23.291 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:23.291 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:23.300 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:39:23.366 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:23.364+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:23.413 INFO:tasks.rados.rados.0.smithi157.stdout:1856: set_chunk oid 38 offset: 719872 length: 1018880 target oid ac25e6702e2d7551dc7447ef0aa706cc750ba5f6c08f1bfddd0c8b886dcebb35 offset: 0 snap -1 2024-09-16T03:39:23.413 INFO:tasks.rados.rados.0.smithi157.stdout:1857: read oid 21 snap -1 2024-09-16T03:39:23.413 INFO:tasks.rados.rados.0.smithi157.stdout:1857: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-16T03:39:23.413 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:39:23.413 INFO:tasks.rados.rados.0.smithi157.stdout:1843: expect (ObjNum 400 snap 0 seq_num 400) 2024-09-16T03:39:23.414 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:23.442 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:23.443 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:23.443 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:23.451 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:39:23.467 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:23.466+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:23.503 INFO:tasks.rados.rados.0.smithi157.stdout:1849: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-16T03:39:23.552 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1903588 2024-09-16T03:39:23.554 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:23.554 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:23.563 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:39:23.567 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:23.565+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:23.574 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1813751 2024-09-16T03:39:23.582 INFO:tasks.rados.rados.0.smithi157.stdout:1844: expect (ObjNum 404 snap 0 seq_num 404) 2024-09-16T03:39:23.622 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2431914 2024-09-16T03:39:23.626 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:23.657 INFO:tasks.rados.rados.0.smithi157.stdout:1850: expect (ObjNum 316 snap 0 seq_num 316) 2024-09-16T03:39:23.657 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:23.668 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:23.668 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:23.668 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:23.667+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:23.676 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:39:23.705 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:23.703+0000 7fa3886e6640 -1 osd.10 145 handle_osd_map injecting map crc failure 2024-09-16T03:39:23.721 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:23.769 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:23.768+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:23.779 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:23.779 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:23.787 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:39:23.794 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 864594 2024-09-16T03:39:23.845 INFO:tasks.rados.rados.0.smithi157.stdout:1847: expect (ObjNum 355 snap 0 seq_num 355) 2024-09-16T03:39:23.869 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:23.868+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:23.889 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:23.889 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:23.897 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:39:23.970 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:23.969+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:24.000 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:24.001 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:24.009 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:39:24.013 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2531069 2024-09-16T03:39:24.061 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:24.071 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:24.070+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:24.112 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:24.112 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:24.121 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:39:24.172 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:24.171+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:24.175 INFO:tasks.rados.rados.0.smithi157.stdout:1846: expect (ObjNum 367 snap 0 seq_num 367) 2024-09-16T03:39:24.208 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:39:24.224 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:24.224 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:24.232 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:39:24.266 INFO:tasks.rados.rados.0.smithi157.stdout:1855: finishing write tid 1 to smithi15736696-4 2024-09-16T03:39:24.266 INFO:tasks.rados.rados.0.smithi157.stdout:1854: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:39:24.273 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:24.271+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:24.346 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:24.346 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:24.347 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:24.347 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:24.356 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:39:24.364 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:24.364 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:24.374 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 234209 2024-09-16T03:39:24.374 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:24.372+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:24.381 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:24.381 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:24.397 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:24.398 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:24.414 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:24.414 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:24.426 INFO:tasks.rados.rados.0.smithi157.stdout:1856: finishing set_chunk to oid 38 2024-09-16T03:39:24.426 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 38 set_chunk returned error code -95 offset: 719872 length: 1018880 2024-09-16T03:39:24.426 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 38 v 0 (ObjNum 397 snap 0 seq_num 397) dirty exists 2024-09-16T03:39:24.426 INFO:tasks.rados.rados.0.smithi157.stdout:1845: finishing write tid 1 to smithi15736696-44 2024-09-16T03:39:24.426 INFO:tasks.rados.rados.0.smithi157.stdout:1845: finishing write tid 2 to smithi15736696-44 2024-09-16T03:39:24.427 INFO:tasks.rados.rados.0.smithi157.stdout:1845: finishing write tid 3 to smithi15736696-44 2024-09-16T03:39:24.427 INFO:tasks.rados.rados.0.smithi157.stdout:1851: expect (ObjNum 401 snap 0 seq_num 401) 2024-09-16T03:39:24.431 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:24.431 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:24.441 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1331066 2024-09-16T03:39:24.447 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:24.447 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:24.461 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:24.461 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:24.461 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:24.461 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:24.461 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:24.464 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:24.464 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:24.469 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:39:24.479 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:24.480 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:24.496 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:24.496 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:24.512 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:24.513 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:24.513 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:24.511+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:24.518 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 128692 2024-09-16T03:39:24.529 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:24.529 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:24.545 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:24.546 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:24.562 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:24.562 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:24.576 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:24.576 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:24.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:24.576 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:24.576 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.576 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:5 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'1, mlcod=0'0)", 2024-09-16T03:39:24.576 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:39:24.576 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 132.60544176499999, 2024-09-16T03:39:24.576 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.105742802, 2024-09-16T03:39:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:39:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:39:24.577 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968147+0000", 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4e-07 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968752+0000", 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060437900000000005 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968762+0000", 2024-09-16T03:39:24.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0284e-05 2024-09-16T03:39:24.579 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.579 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.579 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.579 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968778+0000", 2024-09-16T03:39:24.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6124e-05 2024-09-16T03:39:24.579 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.579 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.579 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.579 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069797+0000", 2024-09-16T03:39:24.579 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10101895700000001 2024-09-16T03:39:24.579 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069833+0000", 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6013000000000002e-05 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073838+0000", 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040049049999999996 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073889+0000", 2024-09-16T03:39:24.580 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.13e-05 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:7 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'3, mlcod=0'0)", 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 132.59854926599999, 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20061241399999999, 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.581 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975042+0000", 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5289999999999998e-06 2024-09-16T03:39:24.582 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975079+0000", 2024-09-16T03:39:24.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7626999999999997e-05 2024-09-16T03:39:24.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975104+0000", 2024-09-16T03:39:24.583 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4958999999999999e-05 2024-09-16T03:39:24.583 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.583 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.583 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.583 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.075211+0000", 2024-09-16T03:39:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100106814 2024-09-16T03:39:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080573+0000", 2024-09-16T03:39:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0053618650000000004 2024-09-16T03:39:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080592+0000", 2024-09-16T03:39:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9079000000000001e-05 2024-09-16T03:39:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.584 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175497+0000", 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.094904832999999994 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175651+0000", 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015470800000000001 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.585 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:44 3.3 e49/31 3:dc284e8d:::smithi15736696-28 low_tier:head v 49'55, mlcod=49'53)", 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 114.45062487, 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.149341897, 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.586 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:39:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122964+0000", 2024-09-16T03:39:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.91e-07 2024-09-16T03:39:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122968+0000", 2024-09-16T03:39:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1470000000000003e-06 2024-09-16T03:39:24.587 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122979+0000", 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1090000000000001e-05 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.223110+0000", 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100131357 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271688+0000", 2024-09-16T03:39:24.588 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048578192999999999 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271712+0000", 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4066e-05 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272260+0000", 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054737000000000004 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.589 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272305+0000", 2024-09-16T03:39:24.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5482999999999998e-05 2024-09-16T03:39:24.590 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.590 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.590 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.590 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.590 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.590 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1007 3.3 e50/31 3:fda66a0d:::920f29fe85d0745d4cd24a887714ac53f1b9a978ec2208ff473c99287fdf8feb:head v 50'57, mlcod=49'55)", 2024-09-16T03:39:24.590 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:39:24.590 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 113.978998316, 2024-09-16T03:39:24.590 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20236485200000001, 2024-09-16T03:39:24.590 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.590 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.590 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.591 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594592+0000", 2024-09-16T03:39:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0700000000000001e-06 2024-09-16T03:39:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595508+0000", 2024-09-16T03:39:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00091599499999999996 2024-09-16T03:39:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595521+0000", 2024-09-16T03:39:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:39:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.592 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595531+0000", 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2499999999999995e-06 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795762+0000", 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200230977 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795778+0000", 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:39:24.593 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796912+0000", 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011338520000000001 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796955+0000", 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2768000000000001e-05 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.594 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1247 4.11 4:89839b7e:::smithi15736696-12:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 107.618011135, 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202294525, 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1247 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.595 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:39:24.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:39:24.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.596 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.596 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.596 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.596 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955579+0000", 2024-09-16T03:39:24.596 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3240000000000001e-06 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955587+0000", 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955600+0000", 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2723e-05 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.597 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955612+0000", 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2422999999999999e-05 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055837+0000", 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10022539699999999 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055974+0000", 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000136995 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.598 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,7,8,11,12", 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.056092+0000", 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011738 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057209+0000", 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011170329999999999 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057262+0000", 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2859000000000003e-05 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.599 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157594+0000", 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10033241699999999 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157648+0000", 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3863999999999998e-05 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157692+0000", 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4406999999999997e-05 2024-09-16T03:39:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157732+0000", 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9156000000000003e-05 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157780+0000", 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8801999999999998e-05 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157872+0000", 2024-09-16T03:39:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1311999999999996e-05 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1518 3.3 e59/31 3:df1be05b:::e06092e68680905b5745d5057e93f6c7581518aed07ca5db2802546e9c29aeef:head v 59'95, mlcod=58'93)", 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 99.315567462000004, 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104523454, 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258022+0000", 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.372e-06 2024-09-16T03:39:24.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258462+0000", 2024-09-16T03:39:24.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00043970100000000002 2024-09-16T03:39:24.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258476+0000", 2024-09-16T03:39:24.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.376e-05 2024-09-16T03:39:24.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258483+0000", 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.294e-06 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358679+0000", 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100196177 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358718+0000", 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9295999999999998e-05 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362496+0000", 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037772069999999999 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362544+0000", 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8646999999999997e-05 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1923 4.4 e73/50 4:239dbd85:::smithi15736696-2:head v 73'19, mlcod=62'17)", 2024-09-16T03:39:24.606 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.609706639999999, 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104144848, 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:39:24.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963884+0000", 2024-09-16T03:39:24.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:39:24.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964597+0000", 2024-09-16T03:39:24.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071314700000000002 2024-09-16T03:39:24.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964615+0000", 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8209000000000001e-05 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964626+0000", 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0511e-05 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064779+0000", 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100153495 2024-09-16T03:39:24.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064794+0000", 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4946000000000001e-05 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.067999+0000", 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032050709999999999 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068026+0000", 2024-09-16T03:39:24.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7066000000000001e-05 2024-09-16T03:39:24.611 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.611 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.611 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.611 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:106 3.3 e73/31 3:dbb0d6f3:::b417679517d63919e63cf420e20217e40570d74b9322ef8b960ef314107040f0:head v 73'120, mlcod=73'116)", 2024-09-16T03:39:24.611 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:39:24.611 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 84.295485713999994, 2024-09-16T03:39:24.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.1035437, 2024-09-16T03:39:24.611 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.611 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.611 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.611 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278103+0000", 2024-09-16T03:39:24.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4900000000000004e-07 2024-09-16T03:39:24.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278120+0000", 2024-09-16T03:39:24.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6214000000000001e-05 2024-09-16T03:39:24.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278129+0000", 2024-09-16T03:39:24.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.115e-06 2024-09-16T03:39:24.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.278138+0000", 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.329e-06 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.378298+0000", 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100159945 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.378320+0000", 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1810999999999998e-05 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.381607+0000", 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032868210000000001 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.381646+0000", 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9515999999999999e-05 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.615 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2152 4.11 4:89839b7e:::smithi15736696-12:head [read 0~2487436,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-09-16T03:39:24.616 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:39:24.616 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 78.062014234000003, 2024-09-16T03:39:24.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10362692900000001, 2024-09-16T03:39:24.616 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.616 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.616 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:24.616 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:24.616 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:24.616 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:24.616 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2152 2024-09-16T03:39:24.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.616 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511574+0000", 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511576+0000", 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7060000000000001e-06 2024-09-16T03:39:24.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511584+0000", 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5939999999999994e-06 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.511612+0000", 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7855e-05 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.611754+0000", 2024-09-16T03:39:24.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100141641 2024-09-16T03:39:24.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613255+0000", 2024-09-16T03:39:24.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001500744 2024-09-16T03:39:24.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.613366+0000", 2024-09-16T03:39:24.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000111088 2024-09-16T03:39:24.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.615201+0000", 2024-09-16T03:39:24.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0018353009999999999 2024-09-16T03:39:24.620 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.620 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.620 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.620 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2445 4.11 4:89839b7e:::smithi15736696-12:head [sparse-read 1818507~599510,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:39:24.620 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:39:24.620 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.129424811999996, 2024-09-16T03:39:24.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200613392, 2024-09-16T03:39:24.620 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.620 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2445 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:39:24.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444167+0000", 2024-09-16T03:39:24.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6650000000000001e-06 2024-09-16T03:39:24.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.622 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.622 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444182+0000", 2024-09-16T03:39:24.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4678000000000001e-05 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444207+0000", 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4768000000000001e-05 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.544359+0000", 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100152804 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644561+0000", 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10020214500000001 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644624+0000", 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2444e-05 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644777+0000", 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000152888 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2629 3.3 e89/31 3:e23364e2:::e447860b11e9f9fde68b75cb1d2f204a74952889ec51a2417ef18c1427ff9a70:head v 89'173, mlcod=87'171)", 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 64.121682118999999, 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10848095100000001, 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:39:24.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:39:24.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451907+0000", 2024-09-16T03:39:24.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6599999999999996e-07 2024-09-16T03:39:24.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456443+0000", 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0045365509999999998 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456470+0000", 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6897e-05 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456496+0000", 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.618e-05 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556666+0000", 2024-09-16T03:39:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100169678 2024-09-16T03:39:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556686+0000", 2024-09-16T03:39:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9740000000000001e-05 2024-09-16T03:39:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560345+0000", 2024-09-16T03:39:24.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003658821 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560387+0000", 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2518000000000002e-05 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:209 3.3 e99/31 3:f2e0c7d0:::6bf49184c86dbc717b1f919afcf6cb40a4ca9c2465eb100f7ba59687540c40ed:head v 99'201, mlcod=98'194)", 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 52.367551921999997, 2024-09-16T03:39:24.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200559603, 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206037+0000", 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9999999999999999e-07 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206039+0000", 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2579999999999998e-06 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206041+0000", 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.542e-06 2024-09-16T03:39:24.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306162+0000", 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100120348 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306250+0000", 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306268+0000", 2024-09-16T03:39:24.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8127000000000001e-05 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.406549+0000", 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10028121199999999 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.406596+0000", 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6700999999999999e-05 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.634 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3274 4.11 4:89839b7e:::smithi15736696-12:head [write 1813152~490434] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:39:24.634 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:39:24.634 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 43.375788751999998, 2024-09-16T03:39:24.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.28154463400000002, 2024-09-16T03:39:24.634 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.634 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.634 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.634 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:24.634 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3274 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197801+0000", 2024-09-16T03:39:24.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9539999999999998e-06 2024-09-16T03:39:24.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.198205+0000", 2024-09-16T03:39:24.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00040328199999999998 2024-09-16T03:39:24.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.198218+0000", 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3176000000000001e-05 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298318+0000", 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100100562 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298419+0000", 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100667 2024-09-16T03:39:24.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298489+0000", 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0023000000000004e-05 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,12", 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298583+0000", 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3634000000000005e-05 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300283+0000", 2024-09-16T03:39:24.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00170049 2024-09-16T03:39:24.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:24.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.307072+0000", 2024-09-16T03:39:24.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0067889209999999998 2024-09-16T03:39:24.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:24.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479243+0000", 2024-09-16T03:39:24.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.17217031799999999 2024-09-16T03:39:24.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479300+0000", 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7837999999999997e-05 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479344+0000", 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3769000000000001e-05 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.641 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3275 4.11 4:89839b7e:::smithi15736696-12:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:39:24.641 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:39:24.641 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 43.275243338999999, 2024-09-16T03:39:24.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.276974636, 2024-09-16T03:39:24.641 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.641 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.641 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.641 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:24.641 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:24.641 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:24.641 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3275 2024-09-16T03:39:24.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.641 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298364+0000", 2024-09-16T03:39:24.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9273e-05 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298373+0000", 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5809999999999997e-06 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298382+0000", 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6269999999999995e-06 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298390+0000", 2024-09-16T03:39:24.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2259999999999999e-06 2024-09-16T03:39:24.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299770+0000", 2024-09-16T03:39:24.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013803159999999999 2024-09-16T03:39:24.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299852+0000", 2024-09-16T03:39:24.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.174e-05 2024-09-16T03:39:24.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,12", 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299948+0000", 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6669000000000006e-05 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300496+0000", 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054762799999999998 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.401930+0000", 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101434495 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575189+0000", 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.173258996 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575247+0000", 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7651000000000001e-05 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575320+0000", 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2434000000000005e-05 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.646 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3276 4.11 4:89839b7e:::smithi15736696-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 43.275174526999997, 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.37724456499999998, 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:24.647 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3276 2024-09-16T03:39:24.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.648 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:39:24.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:39:24.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298415+0000", 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298418+0000", 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.766e-06 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298426+0000", 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1359999999999999e-06 2024-09-16T03:39:24.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298433+0000", 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0899999999999999e-06 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300168+0000", 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017353869999999999 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300215+0000", 2024-09-16T03:39:24.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6702000000000001e-05 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675524+0000", 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.375309585 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675596+0000", 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1921000000000005e-05 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675658+0000", 2024-09-16T03:39:24.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2052999999999997e-05 2024-09-16T03:39:24.652 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.652 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.652 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.652 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3985 4.36 e130/126 4:6ef3b53d:::smithi15736696-48:head v 130'88, mlcod=124'86)", 2024-09-16T03:39:24.652 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.376369+0000", 2024-09-16T03:39:24.652 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.197218845999998, 2024-09-16T03:39:24.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20392002300000001, 2024-09-16T03:39:24.652 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.652 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.652 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376369+0000", 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376369+0000", 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376370+0000", 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1299999999999999e-07 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376877+0000", 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050726199999999997 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376888+0000", 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0798999999999999e-05 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376896+0000", 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1380000000000007e-06 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376950+0000", 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4580000000000003e-05 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376970+0000", 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9607e-05 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.655 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580253+0000", 2024-09-16T03:39:24.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20328323600000001 2024-09-16T03:39:24.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580289+0000", 2024-09-16T03:39:24.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6188e-05 2024-09-16T03:39:24.656 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.656 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.656 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3986 4.36 e130/126 4:6ef3b53d:::smithi15736696-48:head v 130'89, mlcod=124'86)", 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.377664+0000", 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.195924116, 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264148100000001, 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.377664+0000", 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.377664+0000", 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.377665+0000", 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.540000000000001e-07 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378852+0000", 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001187097 2024-09-16T03:39:24.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378863+0000", 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0721e-05 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378868+0000", 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5650000000000001e-06 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378913+0000", 2024-09-16T03:39:24.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4867999999999998e-05 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378929+0000", 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5752000000000001e-05 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580291+0000", 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20136209599999999 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580306+0000", 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4428e-05 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3987 4.36 e130/126 4:6ef3b53d:::smithi15736696-48:head v 130'90, mlcod=124'86)", 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.379666+0000", 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.193922654000001, 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20094095300000001, 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.661 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.662 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379666+0000", 2024-09-16T03:39:24.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.662 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379666+0000", 2024-09-16T03:39:24.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.662 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379667+0000", 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.11e-06 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379675+0000", 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4489999999999999e-06 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379683+0000", 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0660000000000004e-06 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379690+0000", 2024-09-16T03:39:24.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4010000000000002e-06 2024-09-16T03:39:24.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.664 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.579898+0000", 2024-09-16T03:39:24.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20020810999999999 2024-09-16T03:39:24.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.664 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.579914+0000", 2024-09-16T03:39:24.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6198e-05 2024-09-16T03:39:24.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580572+0000", 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065845699999999999 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580607+0000", 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4162000000000003e-05 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:4043 4.36 e131/126 4:6ece7830:::smithi15736696-7:head v 131'93, mlcod=130'89)", 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.055626+0000", 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 18.517961873000001, 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19612945100000001, 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055626+0000", 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055626+0000", 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055627+0000", 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0869999999999999e-06 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055633+0000", 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.101e-06 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055641+0000", 2024-09-16T03:39:24.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9759999999999995e-06 2024-09-16T03:39:24.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055646+0000", 2024-09-16T03:39:24.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4360000000000001e-06 2024-09-16T03:39:24.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251121+0000", 2024-09-16T03:39:24.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.195475068 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251137+0000", 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6271999999999999e-05 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251719+0000", 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058167799999999997 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251756+0000", 2024-09-16T03:39:24.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6832999999999998e-05 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "replica_scrub(pg: 4.12,from:0'0,to:127'20,epoch:139/138,start:4:48000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:14.698983+0000", 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 9.8746048260000006, 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12536847100000001, 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:24.670 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.698983+0000", 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.698983+0000", 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.698985+0000", 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4899999999999999e-06 2024-09-16T03:39:24.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.698991+0000", 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9440000000000002e-06 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.699003+0000", 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1708999999999999e-05 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.699009+0000", 2024-09-16T03:39:24.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4849999999999997e-06 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.824228+0000", 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12521855800000001 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.824262+0000", 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4513e-05 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:24.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.824352+0000", 2024-09-16T03:39:24.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9771999999999994e-05 2024-09-16T03:39:24.674 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.674 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.674 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.674 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:24.674 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:24.674 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:24.674 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:24.573+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:24.675 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:39:24.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:39:24.677 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 527447 2024-09-16T03:39:24.677 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:24.677 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:24.677 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:24.678 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:24.678 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:24.675+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:24.710 INFO:tasks.rados.rados.0.smithi157.stdout:1848: expect (ObjNum 388 snap 0 seq_num 388) 2024-09-16T03:39:24.711 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:24.777 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:24.776+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:24.818 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:24.818 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:24.818 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:39:24.818 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:24.827 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:39:24.838 INFO:tasks.rados.rados.0.smithi157.stdout:1853: finishing set_chunk to oid 11 2024-09-16T03:39:24.839 INFO:tasks.rados.rados.0.smithi157.stdout:1853: got expected ENOENT (src dne) 2024-09-16T03:39:24.839 INFO:tasks.rados.rados.0.smithi157.stdout:1852: expect (ObjNum 325 snap 0 seq_num 325) 2024-09-16T03:39:24.848 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2840177 2024-09-16T03:39:24.859 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:24.878 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:24.876+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:24.897 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:24.930 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:24.930 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:24.930 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:24.930 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:24.930 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:24.936 INFO:tasks.rados.rados.0.smithi157.stdout:1857: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-16T03:39:24.938 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:39:24.945 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1268913 2024-09-16T03:39:24.963 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:24.978 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:24.977+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:24.998 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:25.032 INFO:tasks.rados.rados.0.smithi157.stdout:1845: finishing write tid 4 to smithi15736696-44 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:1855: finishing write tid 2 to smithi15736696-4 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:1855: finishing write tid 3 to smithi15736696-4 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:1855: finishing write tid 4 to smithi15736696-4 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:1855: finishing write tid 5 to smithi15736696-4 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:1855: oid 4 updating version 0 to 158 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:1855: oid 4 updating version 158 to 160 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:1855: oid 4 updating version 160 to 161 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:1855: oid 4 version 161 is already newer than 159 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 161 (ObjNum 411 snap 0 seq_num 411) dirty exists 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:1855: left oid 4 (ObjNum 411 snap 0 seq_num 411) 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:1845: finishing write tid 5 to smithi15736696-44 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:1845: oid 44 updating version 0 to 83 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:1845: oid 44 updating version 83 to 86 2024-09-16T03:39:25.033 INFO:tasks.rados.rados.0.smithi157.stdout:1845: oid 44 version 86 is already newer than 85 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:1845: oid 44 version 86 is already newer than 84 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 44 v 86 (ObjNum 410 snap 0 seq_num 410) dirty exists 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:1845: left oid 44 (ObjNum 410 snap 0 seq_num 410) 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:1842: done (15 left) 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:1843: done (14 left) 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:1844: done (13 left) 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:1845: done (12 left) 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:1846: done (11 left) 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:1847: done (10 left) 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:1848: done (9 left) 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:1849: done (8 left) 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:1850: done (7 left) 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:1851: done (6 left) 2024-09-16T03:39:25.034 INFO:tasks.rados.rados.0.smithi157.stdout:1852: done (5 left) 2024-09-16T03:39:25.035 INFO:tasks.rados.rados.0.smithi157.stdout:1853: done (4 left) 2024-09-16T03:39:25.035 INFO:tasks.rados.rados.0.smithi157.stdout:1854: done (3 left) 2024-09-16T03:39:25.035 INFO:tasks.rados.rados.0.smithi157.stdout:1855: done (2 left) 2024-09-16T03:39:25.035 INFO:tasks.rados.rados.0.smithi157.stdout:1856: done (1 left) 2024-09-16T03:39:25.035 INFO:tasks.rados.rados.0.smithi157.stdout:1857: done (0 left) 2024-09-16T03:39:25.035 INFO:tasks.rados.rados.0.smithi157.stdout:1858: set_chunk oid 36 target oid 2024-09-16T03:39:25.044 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:25.044 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:25.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:25.045 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:25.045 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.045 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:39 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:25.045 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:39:25.045 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 118.201981809, 2024-09-16T03:39:25.045 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40113680400000001, 2024-09-16T03:39:25.045 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.045 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.046 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.046 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:39:25.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.046 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.046 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.046 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.046 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:39:25.046 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839700+0000", 2024-09-16T03:39:25.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.571e-06 2024-09-16T03:39:25.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.047 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839704+0000", 2024-09-16T03:39:25.047 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8399999999999997e-06 2024-09-16T03:39:25.047 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.047 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.047 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839712+0000", 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9990000000000008e-06 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839720+0000", 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0099999999999995e-06 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839783+0000", 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3014000000000003e-05 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.048 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839870+0000", 2024-09-16T03:39:25.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7392999999999995e-05 2024-09-16T03:39:25.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840049+0000", 2024-09-16T03:39:25.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017825800000000001 2024-09-16T03:39:25.049 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.049 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.049 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.049 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840874+0000", 2024-09-16T03:39:25.049 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082532099999999995 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.040285+0000", 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19941088400000001 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240748+0000", 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20046322699999999 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.050 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240786+0000", 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8158000000000003e-05 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240834+0000", 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8128999999999999e-05 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:983 3.1 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head [write 0~2138112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:39:25.051 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 114.978078869, 2024-09-16T03:39:25.052 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30819799799999997, 2024-09-16T03:39:25.052 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.052 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.052 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.052 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.052 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.052 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.052 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 983 2024-09-16T03:39:25.052 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.052 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.052 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.052 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.052 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063610+0000", 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2990000000000002e-06 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.053 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065735+0000", 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021257799999999999 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065752+0000", 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6895999999999999e-05 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065762+0000", 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6830000000000003e-06 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.054 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065871+0000", 2024-09-16T03:39:25.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010904900000000001 2024-09-16T03:39:25.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.055 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.055 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066005+0000", 2024-09-16T03:39:25.055 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013369799999999999 2024-09-16T03:39:25.055 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.055 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066135+0000", 2024-09-16T03:39:25.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000130481 2024-09-16T03:39:25.056 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.056 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.056 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.056 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.069822+0000", 2024-09-16T03:39:25.056 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036863759999999999 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272370+0000", 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20254802599999999 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371671+0000", 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099300893000000001 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.057 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371754+0000", 2024-09-16T03:39:25.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3075e-05 2024-09-16T03:39:25.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.058 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.058 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371798+0000", 2024-09-16T03:39:25.058 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4742e-05 2024-09-16T03:39:25.058 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.058 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.058 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.058 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.058 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.058 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1515 3.1 3:9c9e970c:::7249cb65b52cadf4fe8f4ec923c7e548d1de7d9628823acd2c8de19121f1c1ce:head [write 0~1094656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:39:25.058 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:39:25.059 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 100.091953132, 2024-09-16T03:39:25.059 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.305958065, 2024-09-16T03:39:25.059 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.059 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.059 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.059 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.059 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.059 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.059 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1515 2024-09-16T03:39:25.059 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.059 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:39:25.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.060 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949726+0000", 2024-09-16T03:39:25.060 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.060 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.060 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.060 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.949729+0000", 2024-09-16T03:39:25.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.785e-06 2024-09-16T03:39:25.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.061 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950933+0000", 2024-09-16T03:39:25.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001204341 2024-09-16T03:39:25.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.061 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.061 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.061 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950950+0000", 2024-09-16T03:39:25.061 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6838e-05 2024-09-16T03:39:25.061 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.950960+0000", 2024-09-16T03:39:25.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0316000000000001e-05 2024-09-16T03:39:25.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.062 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951065+0000", 2024-09-16T03:39:25.062 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104128 2024-09-16T03:39:25.062 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.062 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.062 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951202+0000", 2024-09-16T03:39:25.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013706600000000001 2024-09-16T03:39:25.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.951350+0000", 2024-09-16T03:39:25.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000148197 2024-09-16T03:39:25.063 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.063 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.063 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.063 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.953857+0000", 2024-09-16T03:39:25.063 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0025067150000000001 2024-09-16T03:39:25.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:44.958049+0000", 2024-09-16T03:39:25.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0041924650000000003 2024-09-16T03:39:25.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.064 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.064 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.064 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255547+0000", 2024-09-16T03:39:25.064 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29749841900000001 2024-09-16T03:39:25.064 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.065 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.065 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255649+0000", 2024-09-16T03:39:25.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000101513 2024-09-16T03:39:25.065 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.065 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.065 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.065 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.255684+0000", 2024-09-16T03:39:25.065 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5281999999999999e-05 2024-09-16T03:39:25.066 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.066 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.066 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.066 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.066 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.066 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1704 3.1 3:b6a94245:::f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99:head [write 0~453120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:25.066 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:39:25.067 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 94.248841776999996, 2024-09-16T03:39:25.067 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40403313699999999, 2024-09-16T03:39:25.067 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.067 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.067 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.067 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.067 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.067 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.067 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1704 2024-09-16T03:39:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:39:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:39:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792841+0000", 2024-09-16T03:39:25.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.433e-06 2024-09-16T03:39:25.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793329+0000", 2024-09-16T03:39:25.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048839200000000004 2024-09-16T03:39:25.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793345+0000", 2024-09-16T03:39:25.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6079e-05 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793354+0000", 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.32e-06 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793418+0000", 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3944999999999998e-05 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793549+0000", 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013159900000000001 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793666+0000", 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116532 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.797707+0000", 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040409629999999998 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897288+0000", 2024-09-16T03:39:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099581449000000002 2024-09-16T03:39:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196775+0000", 2024-09-16T03:39:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948696800000002 2024-09-16T03:39:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196814+0000", 2024-09-16T03:39:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8336e-05 2024-09-16T03:39:25.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.073 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.073 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196871+0000", 2024-09-16T03:39:25.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7120999999999997e-05 2024-09-16T03:39:25.073 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.073 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.073 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.073 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:84 3.1 3:b24f289d:::e84532c28f467d2f2a6e2cf44904fa0d4a555e1304bbe1b25db645af42152bd7:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:25.073 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:39:25.073 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 94.240069837999997, 2024-09-16T03:39:25.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.395455581, 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801613+0000", 2024-09-16T03:39:25.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9559999999999999e-06 2024-09-16T03:39:25.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802117+0000", 2024-09-16T03:39:25.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050401600000000003 2024-09-16T03:39:25.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802141+0000", 2024-09-16T03:39:25.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3618e-05 2024-09-16T03:39:25.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802155+0000", 2024-09-16T03:39:25.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4358e-05 2024-09-16T03:39:25.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802229+0000", 2024-09-16T03:39:25.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3930999999999994e-05 2024-09-16T03:39:25.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802355+0000", 2024-09-16T03:39:25.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012527299999999999 2024-09-16T03:39:25.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802567+0000", 2024-09-16T03:39:25.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021250199999999999 2024-09-16T03:39:25.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806218+0000", 2024-09-16T03:39:25.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036512189999999998 2024-09-16T03:39:25.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897323+0000", 2024-09-16T03:39:25.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091104892000000007 2024-09-16T03:39:25.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196959+0000", 2024-09-16T03:39:25.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299635971 2024-09-16T03:39:25.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197021+0000", 2024-09-16T03:39:25.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1938000000000002e-05 2024-09-16T03:39:25.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197065+0000", 2024-09-16T03:39:25.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3906999999999998e-05 2024-09-16T03:39:25.082 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.082 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.082 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.082 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:85 3.1 3:b380f269:::e65351e8a65a0b1a517e3b1a85e04b84c0ab7cfa6655f11fb0da31f56b4dd9af:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:25.082 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:39:25.082 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 94.239490941, 2024-09-16T03:39:25.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39502727300000001, 2024-09-16T03:39:25.082 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.082 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.082 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802190+0000", 2024-09-16T03:39:25.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2419999999999999e-06 2024-09-16T03:39:25.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802464+0000", 2024-09-16T03:39:25.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027463699999999999 2024-09-16T03:39:25.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802478+0000", 2024-09-16T03:39:25.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3375e-05 2024-09-16T03:39:25.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802484+0000", 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0990000000000004e-06 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803475+0000", 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00099113099999999991 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803651+0000", 2024-09-16T03:39:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017654300000000001 2024-09-16T03:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803923+0000", 2024-09-16T03:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000271842 2024-09-16T03:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806228+0000", 2024-09-16T03:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023049590000000001 2024-09-16T03:39:25.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897563+0000", 2024-09-16T03:39:25.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091334724000000006 2024-09-16T03:39:25.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197123+0000", 2024-09-16T03:39:25.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299559992 2024-09-16T03:39:25.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197177+0000", 2024-09-16T03:39:25.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4447999999999998e-05 2024-09-16T03:39:25.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197216+0000", 2024-09-16T03:39:25.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8281000000000002e-05 2024-09-16T03:39:25.088 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:86 3.1 3:b3acb53e:::e951eb69ce5151a3f54107fa22c70f80729eff845da1613447a91536a24046e8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:39:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 94.239183846000003, 2024-09-16T03:39:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.394851071, 2024-09-16T03:39:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:39:25.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:39:25.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802496+0000", 2024-09-16T03:39:25.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:39:25.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802782+0000", 2024-09-16T03:39:25.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00028523600000000002 2024-09-16T03:39:25.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802796+0000", 2024-09-16T03:39:25.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4594e-05 2024-09-16T03:39:25.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802804+0000", 2024-09-16T03:39:25.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3790000000000006e-06 2024-09-16T03:39:25.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804787+0000", 2024-09-16T03:39:25.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019825060000000002 2024-09-16T03:39:25.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804895+0000", 2024-09-16T03:39:25.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010813499999999999 2024-09-16T03:39:25.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.805046+0000", 2024-09-16T03:39:25.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150972 2024-09-16T03:39:25.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806681+0000", 2024-09-16T03:39:25.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016353839999999999 2024-09-16T03:39:25.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.005712+0000", 2024-09-16T03:39:25.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19903093199999999 2024-09-16T03:39:25.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197278+0000", 2024-09-16T03:39:25.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.191565295 2024-09-16T03:39:25.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197315+0000", 2024-09-16T03:39:25.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6946000000000002e-05 2024-09-16T03:39:25.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197347+0000", 2024-09-16T03:39:25.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1863000000000003e-05 2024-09-16T03:39:25.096 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.096 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.096 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.096 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:140 3.1 3:845e3b2c:::21d7b947791b34f7e3fcd39ad119c9f923ad2f3cc7a703501eac7a68189d5437:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:25.097 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:39:25.097 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 68.927004592000003, 2024-09-16T03:39:25.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39914108999999998, 2024-09-16T03:39:25.097 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.097 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.097 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.097 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:39:25.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:39:25.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114677+0000", 2024-09-16T03:39:25.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1610000000000001e-06 2024-09-16T03:39:25.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114684+0000", 2024-09-16T03:39:25.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3150000000000003e-06 2024-09-16T03:39:25.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.101 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114699+0000", 2024-09-16T03:39:25.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4996e-05 2024-09-16T03:39:25.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114705+0000", 2024-09-16T03:39:25.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0580000000000003e-06 2024-09-16T03:39:25.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.214862+0000", 2024-09-16T03:39:25.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015700199999999 2024-09-16T03:39:25.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.214922+0000", 2024-09-16T03:39:25.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0053999999999998e-05 2024-09-16T03:39:25.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215023+0000", 2024-09-16T03:39:25.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100432 2024-09-16T03:39:25.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.216100+0000", 2024-09-16T03:39:25.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010776819999999999 2024-09-16T03:39:25.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.499919+0000", 2024-09-16T03:39:25.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.28381845900000002 2024-09-16T03:39:25.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513755+0000", 2024-09-16T03:39:25.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013836477 2024-09-16T03:39:25.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513790+0000", 2024-09-16T03:39:25.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4295999999999998e-05 2024-09-16T03:39:25.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513816+0000", 2024-09-16T03:39:25.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6157999999999999e-05 2024-09-16T03:39:25.107 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.107 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.107 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.108 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:141 3.1 3:9387c678:::99a87d3b555b50e905a1ff55d99eb4aa025bd0070f07a4910d03cf1af95bbc44:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:25.108 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:39:25.108 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 68.926953894999997, 2024-09-16T03:39:25.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50160030300000003, 2024-09-16T03:39:25.108 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.108 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.108 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.108 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:39:25.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:39:25.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114726+0000", 2024-09-16T03:39:25.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4399999999999999e-07 2024-09-16T03:39:25.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114730+0000", 2024-09-16T03:39:25.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1479999999999999e-06 2024-09-16T03:39:25.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.110 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.110 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114738+0000", 2024-09-16T03:39:25.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3200000000000002e-06 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114743+0000", 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.891e-06 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215320+0000", 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100576463 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215376+0000", 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6310999999999997e-05 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215473+0000", 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7103999999999997e-05 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.216108+0000", 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00063422800000000003 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513851+0000", 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29774376400000002 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616086+0000", 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10223441 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616240+0000", 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015453799999999999 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616326+0000", 2024-09-16T03:39:25.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5382000000000003e-05 2024-09-16T03:39:25.114 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.114 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.114 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.114 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:142 3.1 3:a93b7b74:::70cd8f47898d416018aded7de9c29e0fe002b2336cb6cfabbf3eb6238766c3b1:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:25.114 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:39:25.114 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 68.926924507999999, 2024-09-16T03:39:25.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50166088799999997, 2024-09-16T03:39:25.114 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:39:25.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3600000000000003e-07 2024-09-16T03:39:25.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114762+0000", 2024-09-16T03:39:25.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6039999999999998e-06 2024-09-16T03:39:25.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114769+0000", 2024-09-16T03:39:25.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5939999999999999e-06 2024-09-16T03:39:25.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114774+0000", 2024-09-16T03:39:25.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2179999999999998e-06 2024-09-16T03:39:25.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215687+0000", 2024-09-16T03:39:25.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100912974 2024-09-16T03:39:25.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215733+0000", 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6419999999999999e-05 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215836+0000", 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102981 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316766+0000", 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100929693 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513877+0000", 2024-09-16T03:39:25.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19711067800000001 2024-09-16T03:39:25.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616358+0000", 2024-09-16T03:39:25.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10248180699999999 2024-09-16T03:39:25.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616391+0000", 2024-09-16T03:39:25.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2551000000000001e-05 2024-09-16T03:39:25.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616416+0000", 2024-09-16T03:39:25.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4732e-05 2024-09-16T03:39:25.120 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.120 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.120 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.120 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:143 3.1 3:ac848b49:::adc4f0c8a4d0bc7e52f9d5c3b4981447737a7dbe58b8313a369b04383e10893f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:25.120 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:39:25.120 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 68.926894501999996, 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60333394699999998, 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:39:25.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7899999999999999e-07 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114788+0000", 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4569999999999999e-06 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114792+0000", 2024-09-16T03:39:25.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5730000000000001e-06 2024-09-16T03:39:25.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114796+0000", 2024-09-16T03:39:25.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5449999999999999e-06 2024-09-16T03:39:25.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316237+0000", 2024-09-16T03:39:25.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20144099300000001 2024-09-16T03:39:25.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316417+0000", 2024-09-16T03:39:25.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017981099999999999 2024-09-16T03:39:25.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316564+0000", 2024-09-16T03:39:25.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014693700000000001 2024-09-16T03:39:25.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316993+0000", 2024-09-16T03:39:25.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042985499999999998 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.717965+0000", 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40097123699999998 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718005+0000", 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0083000000000002e-05 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718064+0000", 2024-09-16T03:39:25.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9020000000000001e-05 2024-09-16T03:39:25.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718119+0000", 2024-09-16T03:39:25.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4957000000000003e-05 2024-09-16T03:39:25.126 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.126 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.126 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.126 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:149 3.1 3:8eb88b02:::fa73f0ab3a7395fcfc19219899ff8c0a4a3c96caf8327f9da9656e2450435b27:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:25.126 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:39:25.126 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 67.912985853999999, 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.348655887, 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:39:25.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128698+0000", 2024-09-16T03:39:25.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.527e-06 2024-09-16T03:39:25.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.128 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128714+0000", 2024-09-16T03:39:25.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6019e-05 2024-09-16T03:39:25.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.128 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128757+0000", 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3318000000000001e-05 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128764+0000", 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.28e-06 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.228916+0000", 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015252199999999 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.228976+0000", 2024-09-16T03:39:25.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0390999999999998e-05 2024-09-16T03:39:25.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.229077+0000", 2024-09-16T03:39:25.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010053 2024-09-16T03:39:25.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.274759+0000", 2024-09-16T03:39:25.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.045682404000000003 2024-09-16T03:39:25.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.330314+0000", 2024-09-16T03:39:25.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.055554916000000003 2024-09-16T03:39:25.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477196+0000", 2024-09-16T03:39:25.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14688130099999999 2024-09-16T03:39:25.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477291+0000", 2024-09-16T03:39:25.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5144999999999998e-05 2024-09-16T03:39:25.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477349+0000", 2024-09-16T03:39:25.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8533999999999999e-05 2024-09-16T03:39:25.132 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.132 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.132 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.132 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:169 3.1 3:8ca52973:::737e1241b430663b0c460fc4c0e6bf0a2c97ca3f584ee4e3541536e0014b01da:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.648814+0000", 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 63.392865006999997, 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30599283399999999, 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648814+0000", 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648814+0000", 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648815+0000", 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9900000000000001e-07 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648861+0000", 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6786999999999998e-05 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648867+0000", 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1830000000000002e-06 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648872+0000", 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9169999999999997e-06 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651624+0000", 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027522150000000001 2024-09-16T03:39:25.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651689+0000", 2024-09-16T03:39:25.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5065000000000001e-05 2024-09-16T03:39:25.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.651814+0000", 2024-09-16T03:39:25.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012493199999999999 2024-09-16T03:39:25.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.653881+0000", 2024-09-16T03:39:25.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020671830000000002 2024-09-16T03:39:25.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.752238+0000", 2024-09-16T03:39:25.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098357336000000004 2024-09-16T03:39:25.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954726+0000", 2024-09-16T03:39:25.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20248775199999999 2024-09-16T03:39:25.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954780+0000", 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3859000000000001e-05 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954807+0000", 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7206e-05 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:171 3.1 3:93178f8f:::777eb2bf7c36938cba5735a17f90bb453157d7a9a42a0a6f52227fe7530ec353:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e91)", 2024-09-16T03:39:25.138 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:21.648876+0000", 2024-09-16T03:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 63.392802961999998, 2024-09-16T03:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30603207999999998, 2024-09-16T03:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648876+0000", 2024-09-16T03:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.139 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648876+0000", 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648877+0000", 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8799999999999998e-07 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648918+0000", 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1488e-05 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648923+0000", 2024-09-16T03:39:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0320000000000003e-06 2024-09-16T03:39:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.648926+0000", 2024-09-16T03:39:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1690000000000001e-06 2024-09-16T03:39:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.652727+0000", 2024-09-16T03:39:25.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0038005909999999999 2024-09-16T03:39:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.652854+0000", 2024-09-16T03:39:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012661700000000001 2024-09-16T03:39:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.653069+0000", 2024-09-16T03:39:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021573799999999999 2024-09-16T03:39:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.654043+0000", 2024-09-16T03:39:25.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00097351999999999996 2024-09-16T03:39:25.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.854187+0000", 2024-09-16T03:39:25.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20014437400000001 2024-09-16T03:39:25.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954834+0000", 2024-09-16T03:39:25.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100646293 2024-09-16T03:39:25.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954875+0000", 2024-09-16T03:39:25.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1637999999999997e-05 2024-09-16T03:39:25.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:21.954908+0000", 2024-09-16T03:39:25.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3232000000000003e-05 2024-09-16T03:39:25.144 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.144 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.144 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.145 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:139 3.1 3:bd600e93:::b81929c6ed6df2fe181809e10b331bcd3c2df1a1077e06024548babcdcd8f550:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:39:25.145 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:39:25.145 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 48.000105740999999, 2024-09-16T03:39:25.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.43180016300000001, 2024-09-16T03:39:25.145 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.145 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.145 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.145 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:39:25.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:39:25.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:39:25.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2300000000000001e-07 2024-09-16T03:39:25.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041700+0000", 2024-09-16T03:39:25.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000126367 2024-09-16T03:39:25.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041707+0000", 2024-09-16T03:39:25.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1519999999999997e-06 2024-09-16T03:39:25.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.147 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.147 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.147 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041710+0000", 2024-09-16T03:39:25.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7869999999999999e-06 2024-09-16T03:39:25.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043365+0000", 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001654914 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043453+0000", 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7954000000000005e-05 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043596+0000", 2024-09-16T03:39:25.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014304500000000001 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.145131+0000", 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10153440599999999 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.344273+0000", 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19914232200000001 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473290+0000", 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12901701900000001 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473325+0000", 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4656000000000002e-05 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473374+0000", 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9017999999999997e-05 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.150 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:144 3.1 3:91c40fc0:::f2d8823355177e50718038dafd7a8313c1a25b0231461a8672e684a7d4e587b4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 47.999961483999996, 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.431801096, 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:39:25.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:39:25.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:39:25.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.75e-07 2024-09-16T03:39:25.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042077+0000", 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035865099999999998 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042084+0000", 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2400000000000001e-06 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042088+0000", 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1570000000000002e-06 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144179+0000", 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10209033300000001 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144427+0000", 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000247879 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144656+0000", 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022919800000000001 2024-09-16T03:39:25.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.145866+0000", 2024-09-16T03:39:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001210046 2024-09-16T03:39:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.344293+0000", 2024-09-16T03:39:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.198427353 2024-09-16T03:39:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473432+0000", 2024-09-16T03:39:25.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12913908800000001 2024-09-16T03:39:25.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473478+0000", 2024-09-16T03:39:25.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6165000000000003e-05 2024-09-16T03:39:25.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.156 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.156 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473519+0000", 2024-09-16T03:39:25.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0510999999999999e-05 2024-09-16T03:39:25.156 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.156 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.157 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.157 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:165 3.1 3:8166b007:::3555453b0ab6ec5b2f29a1e76cc26fdaf7f55d077a227627cdf2200eb9ca342d:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:25.157 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:39:25.157 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 43.214444682, 2024-09-16T03:39:25.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40346431100000002, 2024-09-16T03:39:25.157 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.157 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.157 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.157 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5999999999999999e-07 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827239+0000", 2024-09-16T03:39:25.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8920000000000002e-06 2024-09-16T03:39:25.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827243+0000", 2024-09-16T03:39:25.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2760000000000002e-06 2024-09-16T03:39:25.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827246+0000", 2024-09-16T03:39:25.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1990000000000002e-06 2024-09-16T03:39:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.029284+0000", 2024-09-16T03:39:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20203769499999999 2024-09-16T03:39:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.029375+0000", 2024-09-16T03:39:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1419e-05 2024-09-16T03:39:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.029542+0000", 2024-09-16T03:39:25.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016613199999999999 2024-09-16T03:39:25.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.130201+0000", 2024-09-16T03:39:25.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10065989 2024-09-16T03:39:25.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230566+0000", 2024-09-16T03:39:25.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100364563 2024-09-16T03:39:25.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230623+0000", 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6848000000000002e-05 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230670+0000", 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7117999999999998e-05 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230699+0000", 2024-09-16T03:39:25.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8918999999999999e-05 2024-09-16T03:39:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:166 3.1 3:91c40fc0:::f2d8823355177e50718038dafd7a8313c1a25b0231461a8672e684a7d4e587b4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.827250+0000", 2024-09-16T03:39:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 43.214428970999997, 2024-09-16T03:39:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40362257099999999, 2024-09-16T03:39:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.163 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827250+0000", 2024-09-16T03:39:25.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827250+0000", 2024-09-16T03:39:25.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827251+0000", 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827253+0000", 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.928e-06 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827257+0000", 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0249999999999996e-06 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827260+0000", 2024-09-16T03:39:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3359999999999999e-06 2024-09-16T03:39:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.090467+0000", 2024-09-16T03:39:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.26320751100000001 2024-09-16T03:39:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.090543+0000", 2024-09-16T03:39:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5740000000000003e-05 2024-09-16T03:39:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.090651+0000", 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000107415 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.130204+0000", 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.039553455000000001 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230753+0000", 2024-09-16T03:39:25.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100549096 2024-09-16T03:39:25.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230809+0000", 2024-09-16T03:39:25.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6243999999999999e-05 2024-09-16T03:39:25.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230848+0000", 2024-09-16T03:39:25.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9045999999999999e-05 2024-09-16T03:39:25.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230873+0000", 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4433000000000001e-05 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4365 3.1 3:8dceab4f:::b595f19538cd60ceade5ba11d0db1f07eff23820ff1cafbf439b875b0bc21add:head [write 0~243200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:19.957638+0000", 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.0840414430000003, 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40324459000000001, 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.169 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4365 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:19.957638+0000", 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:19.957638+0000", 2024-09-16T03:39:25.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:19.957640+0000", 2024-09-16T03:39:25.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.967e-06 2024-09-16T03:39:25.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:19.957896+0000", 2024-09-16T03:39:25.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000256641 2024-09-16T03:39:25.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:19.957908+0000", 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1402e-05 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.058035+0000", 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100127323 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.158314+0000", 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100278943 2024-09-16T03:39:25.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.158401+0000", 2024-09-16T03:39:25.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7225000000000006e-05 2024-09-16T03:39:25.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.158516+0000", 2024-09-16T03:39:25.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000115127 2024-09-16T03:39:25.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.159492+0000", 2024-09-16T03:39:25.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00097529099999999998 2024-09-16T03:39:25.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.159994+0000", 2024-09-16T03:39:25.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050216400000000004 2024-09-16T03:39:25.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.174 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.174 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.360804+0000", 2024-09-16T03:39:25.174 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20081049200000001 2024-09-16T03:39:25.174 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.174 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.360836+0000", 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1967000000000001e-05 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.360882+0000", 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6047999999999997e-05 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.175 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4373 3.1 3:86e0c90a:::e78062fe11bded5b0dfbe3bf79dfdc9dee286cfe94341e7d51857de1af7a4ac6:head [write 0~1226752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:20.379803+0000", 2024-09-16T03:39:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 4.6618765690000004, 2024-09-16T03:39:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.306323646, 2024-09-16T03:39:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4373 2024-09-16T03:39:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.176 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.379803+0000", 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.379803+0000", 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.379813+0000", 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0329999999999999e-05 2024-09-16T03:39:25.177 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.381263+0000", 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014503610000000001 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.381275+0000", 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1182999999999999e-05 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.481404+0000", 2024-09-16T03:39:25.178 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012908399999999 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.481488+0000", 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4145000000000002e-05 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.481582+0000", 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3883999999999998e-05 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:25.179 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.481673+0000", 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1337999999999999e-05 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.483542+0000", 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001869227 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.685985+0000", 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202442873 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.180 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.686056+0000", 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1166000000000003e-05 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.686096+0000", 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9249999999999999e-05 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.686126+0000", 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0805000000000001e-05 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.181 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.182 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.182 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.182 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.182 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:25.183 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:39:25.184 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:25.181+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:25.238 INFO:tasks.rados.rados.0.smithi157.stdout:1858: set_chunk oid 36 offset: 603648 length: 513536 target oid df21b27e4940b754b9c0285d87bcbc8dbfcdd0059bba94745a59fbc37acf1ebe offset: 0 snap -1 2024-09-16T03:39:25.238 INFO:tasks.rados.rados.0.smithi157.stdout:1859: write (excl) oid 25 current snap is 0 2024-09-16T03:39:25.238 INFO:tasks.rados.rados.0.smithi157.stdout:1859: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2024-09-16T03:39:25.243 INFO:tasks.rados.rados.0.smithi157.stdout:1859: writing smithi15736696-25 from 524484 to 1010134 tid 1 2024-09-16T03:39:25.249 INFO:tasks.rados.rados.0.smithi157.stdout:1859: writing smithi15736696-25 from 1425993 to 2087453 tid 2 2024-09-16T03:39:25.254 INFO:tasks.rados.rados.0.smithi157.stdout:1859: writing smithi15736696-25 from 2847920 to 2924484 tid 3 2024-09-16T03:39:25.254 INFO:tasks.rados.rados.0.smithi157.stdout:1860: set_chunk oid 35 target oid 2024-09-16T03:39:25.284 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:25.282+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:25.330 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:25.330 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:25.330 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:39:25.330 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:25.339 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:39:25.385 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:25.383+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:25.482 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:25.483 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:25.483 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:25.483 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:25.483 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:25.486 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:25.484+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:25.491 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:39:25.559 INFO:tasks.rados.rados.0.smithi157.stdout:1860: set_chunk oid 35 offset: 279552 length: 1852928 target oid e31fc22a49b8d3bb85bad4c444eb8bfcc9846be61fdb9baad4934d3b0ed41fae offset: 0 snap -1 2024-09-16T03:39:25.559 INFO:tasks.rados.rados.0.smithi157.stdout:1861: tier_evict oid 16 2024-09-16T03:39:25.559 INFO:tasks.rados.rados.0.smithi157.stdout:1861: tier_evict oid 16 snap -1 2024-09-16T03:39:25.559 INFO:tasks.rados.rados.0.smithi157.stdout:1859: finishing write tid 1 to smithi15736696-25 2024-09-16T03:39:25.559 INFO:tasks.rados.rados.0.smithi157.stdout:1862: read oid 20 snap -1 2024-09-16T03:39:25.559 INFO:tasks.rados.rados.0.smithi157.stdout:1862: expect (ObjNum 399 snap 0 seq_num 399) 2024-09-16T03:39:25.559 INFO:tasks.rados.rados.0.smithi157.stdout:1863: read oid 44 snap -1 2024-09-16T03:39:25.560 INFO:tasks.rados.rados.0.smithi157.stdout:1863: expect (ObjNum 410 snap 0 seq_num 410) 2024-09-16T03:39:25.560 INFO:tasks.rados.rados.0.smithi157.stdout:1864: write (excl) oid 19 current snap is 0 2024-09-16T03:39:25.560 INFO:tasks.rados.rados.0.smithi157.stdout:1864: seq_num 413 ranges {541291=560899,1884475=521471} 2024-09-16T03:39:25.565 INFO:tasks.rados.rados.0.smithi157.stdout:1864: writing smithi15736696-19 from 541291 to 1102190 tid 1 2024-09-16T03:39:25.572 INFO:tasks.rados.rados.0.smithi157.stdout:1864: writing smithi15736696-19 from 1884475 to 2405946 tid 2 2024-09-16T03:39:25.572 INFO:tasks.rados.rados.0.smithi157.stdout:1865: tier_promote oid 1 2024-09-16T03:39:25.572 INFO:tasks.rados.rados.0.smithi157.stdout:1860: finishing set_chunk to oid 35 2024-09-16T03:39:25.572 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 35 set_chunk returned error code -95 offset: 279552 length: 1852928 2024-09-16T03:39:25.573 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 0 (ObjNum 392 snap 0 seq_num 392) dirty exists 2024-09-16T03:39:25.573 INFO:tasks.rados.rados.0.smithi157.stdout:1862: expect (ObjNum 399 snap 0 seq_num 399) 2024-09-16T03:39:25.578 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1166480 2024-09-16T03:39:25.586 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:25.584+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:25.601 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:416 4.0 4:023abada:::smithi15736696-17:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:25.601 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:39:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 125.28906611799999, 2024-09-16T03:39:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60401983699999995, 2024-09-16T03:39:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 416 2024-09-16T03:39:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.602 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309243+0000", 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.178e-06 2024-09-16T03:39:25.603 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309248+0000", 2024-09-16T03:39:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:39:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309257+0000", 2024-09-16T03:39:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4720000000000001e-06 2024-09-16T03:39:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.604 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309262+0000", 2024-09-16T03:39:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0839999999999999e-06 2024-09-16T03:39:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309297+0000", 2024-09-16T03:39:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4997999999999999e-05 2024-09-16T03:39:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.605 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309367+0000", 2024-09-16T03:39:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0124999999999997e-05 2024-09-16T03:39:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812055+0000", 2024-09-16T03:39:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50268796900000001 2024-09-16T03:39:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812614+0000", 2024-09-16T03:39:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000558309 2024-09-16T03:39:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.606 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.813040+0000", 2024-09-16T03:39:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042652800000000002 2024-09-16T03:39:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913189+0000", 2024-09-16T03:39:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10014880399999999 2024-09-16T03:39:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.607 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913220+0000", 2024-09-16T03:39:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1353000000000003e-05 2024-09-16T03:39:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913262+0000", 2024-09-16T03:39:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1284e-05 2024-09-16T03:39:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:494 4.f 4:f817def9:::smithi15736696-45:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:25.608 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:39:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 123.876126383, 2024-09-16T03:39:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60451760399999999, 2024-09-16T03:39:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 494 2024-09-16T03:39:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.609 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722183+0000", 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.610 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722189+0000", 2024-09-16T03:39:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9429999999999997e-06 2024-09-16T03:39:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722203+0000", 2024-09-16T03:39:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4367e-05 2024-09-16T03:39:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722210+0000", 2024-09-16T03:39:25.611 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:39:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822367+0000", 2024-09-16T03:39:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100156933 2024-09-16T03:39:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822461+0000", 2024-09-16T03:39:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3216999999999999e-05 2024-09-16T03:39:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.612 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125593+0000", 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30313253699999998 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126307+0000", 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071350000000000005 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126825+0000", 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051846100000000001 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.613 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326612+0000", 2024-09-16T03:39:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19978721999999999 2024-09-16T03:39:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326663+0000", 2024-09-16T03:39:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0460000000000001e-05 2024-09-16T03:39:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326699+0000", 2024-09-16T03:39:25.614 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6340000000000001e-05 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:784 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=63b,truncate 2840350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 119.791718507, 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63358405200000001, 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.615 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 784 2024-09-16T03:39:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:39:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.616 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:39:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806590+0000", 2024-09-16T03:39:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6000000000000003e-07 2024-09-16T03:39:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806593+0000", 2024-09-16T03:39:25.617 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6649999999999999e-06 2024-09-16T03:39:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806597+0000", 2024-09-16T03:39:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.245e-06 2024-09-16T03:39:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806600+0000", 2024-09-16T03:39:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.23e-06 2024-09-16T03:39:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.618 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099381+0000", 2024-09-16T03:39:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29278119400000002 2024-09-16T03:39:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099438+0000", 2024-09-16T03:39:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6198e-05 2024-09-16T03:39:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099527+0000", 2024-09-16T03:39:25.619 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9505000000000004e-05 2024-09-16T03:39:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200775+0000", 2024-09-16T03:39:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101247904 2024-09-16T03:39:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200889+0000", 2024-09-16T03:39:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114301 2024-09-16T03:39:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.620 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440066+0000", 2024-09-16T03:39:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.239176367 2024-09-16T03:39:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440109+0000", 2024-09-16T03:39:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2941000000000001e-05 2024-09-16T03:39:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440173+0000", 2024-09-16T03:39:25.621 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4842e-05 2024-09-16T03:39:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:785 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:39:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 119.791702582, 2024-09-16T03:39:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63376373200000002, 2024-09-16T03:39:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:25.622 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 785 2024-09-16T03:39:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:39:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.623 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:39:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806606+0000", 2024-09-16T03:39:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:39:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.624 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806607+0000", 2024-09-16T03:39:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.424e-06 2024-09-16T03:39:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806611+0000", 2024-09-16T03:39:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1080000000000003e-06 2024-09-16T03:39:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806614+0000", 2024-09-16T03:39:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.067e-06 2024-09-16T03:39:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.625 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099813+0000", 2024-09-16T03:39:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29319853699999998 2024-09-16T03:39:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099858+0000", 2024-09-16T03:39:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5667999999999997e-05 2024-09-16T03:39:25.626 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440236+0000", 2024-09-16T03:39:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34037791499999998 2024-09-16T03:39:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.627 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440297+0000", 2024-09-16T03:39:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0988e-05 2024-09-16T03:39:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440369+0000", 2024-09-16T03:39:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1682999999999998e-05 2024-09-16T03:39:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:801 4.0 4:07ad0560:::smithi15736696-25:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:25.628 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:39:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 119.592224684, 2024-09-16T03:39:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.534510445, 2024-09-16T03:39:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 801 2024-09-16T03:39:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.629 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:39:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:39:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006084+0000", 2024-09-16T03:39:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.322e-06 2024-09-16T03:39:25.630 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006088+0000", 2024-09-16T03:39:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000001e-06 2024-09-16T03:39:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006098+0000", 2024-09-16T03:39:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0186999999999999e-05 2024-09-16T03:39:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.631 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006104+0000", 2024-09-16T03:39:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.271e-06 2024-09-16T03:39:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.199952+0000", 2024-09-16T03:39:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.193847935 2024-09-16T03:39:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200027+0000", 2024-09-16T03:39:25.632 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4987000000000006e-05 2024-09-16T03:39:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.540594+0000", 2024-09-16T03:39:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34056708299999999 2024-09-16T03:39:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1023 4.0 4:06b6d050:::smithi15736696-50:head [write 2246563~579356] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:39:25.633 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 114.287075614, 2024-09-16T03:39:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60727534699999997, 2024-09-16T03:39:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1023 2024-09-16T03:39:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.634 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311233+0000", 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1039999999999999e-06 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311692+0000", 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045908800000000002 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311704+0000", 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1844e-05 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311713+0000", 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2500000000000006e-06 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.636 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359204+0000", 2024-09-16T03:39:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.047491646999999998 2024-09-16T03:39:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359258+0000", 2024-09-16T03:39:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4082000000000002e-05 2024-09-16T03:39:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359354+0000", 2024-09-16T03:39:25.637 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5686999999999994e-05 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512245+0000", 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15289135700000001 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512956+0000", 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071110300000000004 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.638 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918361+0000", 2024-09-16T03:39:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40540489400000002 2024-09-16T03:39:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918455+0000", 2024-09-16T03:39:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3412999999999995e-05 2024-09-16T03:39:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918508+0000", 2024-09-16T03:39:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:39:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.639 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1024 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3176490] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:39:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 114.28658263299999, 2024-09-16T03:39:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60701227000000002, 2024-09-16T03:39:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.640 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1024 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.641 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311726+0000", 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.02e-07 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311730+0000", 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1099999999999996e-06 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311735+0000", 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4680000000000003e-06 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.642 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311739+0000", 2024-09-16T03:39:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9380000000000002e-06 2024-09-16T03:39:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361468+0000", 2024-09-16T03:39:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.049729450000000001 2024-09-16T03:39:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361539+0000", 2024-09-16T03:39:25.643 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0896999999999996e-05 2024-09-16T03:39:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361661+0000", 2024-09-16T03:39:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000122404 2024-09-16T03:39:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512943+0000", 2024-09-16T03:39:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.151281481 2024-09-16T03:39:25.644 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512959+0000", 2024-09-16T03:39:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:39:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918588+0000", 2024-09-16T03:39:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40562815499999999 2024-09-16T03:39:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.645 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918646+0000", 2024-09-16T03:39:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8566999999999997e-05 2024-09-16T03:39:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918737+0000", 2024-09-16T03:39:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1308000000000001e-05 2024-09-16T03:39:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.646 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1025 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 114.28656413500001, 2024-09-16T03:39:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60723649599999996, 2024-09-16T03:39:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.647 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1025 2024-09-16T03:39:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.648 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9400000000000001e-07 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311746+0000", 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2670000000000001e-06 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311752+0000", 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6389999999999997e-06 2024-09-16T03:39:25.649 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311757+0000", 2024-09-16T03:39:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3639999999999998e-06 2024-09-16T03:39:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362018+0000", 2024-09-16T03:39:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.050260969000000003 2024-09-16T03:39:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:25.650 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362089+0000", 2024-09-16T03:39:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0735000000000006e-05 2024-09-16T03:39:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918781+0000", 2024-09-16T03:39:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55669215400000005 2024-09-16T03:39:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918867+0000", 2024-09-16T03:39:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6019000000000004e-05 2024-09-16T03:39:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.651 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918980+0000", 2024-09-16T03:39:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011295500000000001 2024-09-16T03:39:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1054 4.10 4:0a55bbf2:::smithi15736696-18:head [write 2633066~610652] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:39:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 114.085070927, 2024-09-16T03:39:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51115411600000005, 2024-09-16T03:39:25.652 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1054 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.653 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513237+0000", 2024-09-16T03:39:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.654 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513238+0000", 2024-09-16T03:39:25.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1999999999999998e-07 2024-09-16T03:39:25.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.656 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513695+0000", 2024-09-16T03:39:25.656 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045695099999999998 2024-09-16T03:39:25.656 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.656 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.656 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513720+0000", 2024-09-16T03:39:25.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.567e-05 2024-09-16T03:39:25.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513725+0000", 2024-09-16T03:39:25.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9899999999999997e-06 2024-09-16T03:39:25.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.657 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.657 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.657 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613892+0000", 2024-09-16T03:39:25.657 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100166253 2024-09-16T03:39:25.657 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:39:25.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.613898+0000", 2024-09-16T03:39:25.658 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8710000000000002e-06 2024-09-16T03:39:25.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916870+0000", 2024-09-16T03:39:25.658 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30297253800000001 2024-09-16T03:39:25.658 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.658 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.658 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.658 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.916934+0000", 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3723e-05 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.917033+0000", 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8908999999999998e-05 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921278+0000", 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0042456999999999998 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019617+0000", 2024-09-16T03:39:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098338910000000002 2024-09-16T03:39:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024269+0000", 2024-09-16T03:39:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0046515519999999998 2024-09-16T03:39:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024358+0000", 2024-09-16T03:39:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9096000000000002e-05 2024-09-16T03:39:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.660 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.024391+0000", 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3033e-05 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1055 4.10 4:0a55bbf2:::smithi15736696-18:head [setxattr _header (56) in=7b,truncate 3243718] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 114.084571427, 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61098759599999997, 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.661 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.662 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.662 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.662 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.662 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.662 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1055 2024-09-16T03:39:25.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.662 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.662 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.662 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:39:25.662 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.662 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.662 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:39:25.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513737+0000", 2024-09-16T03:39:25.663 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9399999999999993e-07 2024-09-16T03:39:25.663 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.663 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.663 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.663 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513759+0000", 2024-09-16T03:39:25.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1903999999999999e-05 2024-09-16T03:39:25.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.664 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513773+0000", 2024-09-16T03:39:25.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3244000000000001e-05 2024-09-16T03:39:25.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.664 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.664 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513780+0000", 2024-09-16T03:39:25.664 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999999e-06 2024-09-16T03:39:25.664 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.664 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714016+0000", 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20023635000000001 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714024+0000", 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8359999999999992e-06 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918736+0000", 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471236700000001 2024-09-16T03:39:25.665 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918830+0000", 2024-09-16T03:39:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3769000000000003e-05 2024-09-16T03:39:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919003+0000", 2024-09-16T03:39:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000173198 2024-09-16T03:39:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.666 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921589+0000", 2024-09-16T03:39:25.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002586419 2024-09-16T03:39:25.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019655+0000", 2024-09-16T03:39:25.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098065801999999994 2024-09-16T03:39:25.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.667 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.667 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.667 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124538+0000", 2024-09-16T03:39:25.667 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10488272899999999 2024-09-16T03:39:25.667 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.668 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.668 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.668 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124635+0000", 2024-09-16T03:39:25.668 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7125000000000003e-05 2024-09-16T03:39:25.668 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.669 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.669 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124724+0000", 2024-09-16T03:39:25.669 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8858999999999997e-05 2024-09-16T03:39:25.669 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.669 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.669 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.669 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.669 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.670 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1056 4.10 4:0a55bbf2:::smithi15736696-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:25.670 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:25.670 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 114.084518856, 2024-09-16T03:39:25.670 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61111632800000004, 2024-09-16T03:39:25.670 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.670 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.670 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:25.670 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.670 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.670 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.670 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1056 2024-09-16T03:39:25.670 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.670 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:25.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.671 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.671 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.671 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:25.671 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.671 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:25.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2499999999999995e-07 2024-09-16T03:39:25.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513792+0000", 2024-09-16T03:39:25.672 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:39:25.672 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.672 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.672 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.672 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513797+0000", 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4940000000000003e-06 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513802+0000", 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5390000000000001e-06 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919221+0000", 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40541913899999998 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.673 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919268+0000", 2024-09-16T03:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7027000000000002e-05 2024-09-16T03:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124766+0000", 2024-09-16T03:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20549842099999999 2024-09-16T03:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124828+0000", 2024-09-16T03:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1803000000000004e-05 2024-09-16T03:39:25.674 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124905+0000", 2024-09-16T03:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7163000000000006e-05 2024-09-16T03:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1702 4.10 4:0a55bbf2:::smithi15736696-18:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 94.909163866, 2024-09-16T03:39:25.675 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51800317600000001, 2024-09-16T03:39:25.676 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.676 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.676 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:25.676 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.676 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.676 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.676 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1702 2024-09-16T03:39:25.676 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.676 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.676 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.676 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:39:25.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689144+0000", 2024-09-16T03:39:25.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.677 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.677 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.677 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.677 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689147+0000", 2024-09-16T03:39:25.677 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2499999999999998e-06 2024-09-16T03:39:25.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.678 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689158+0000", 2024-09-16T03:39:25.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0426e-05 2024-09-16T03:39:25.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.678 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.689185+0000", 2024-09-16T03:39:25.678 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7299000000000001e-05 2024-09-16T03:39:25.678 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.678 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.678 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789328+0000", 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100143102 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789532+0000", 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000204317 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.789671+0000", 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013852600000000001 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.679 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.680 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.680 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.207147+0000", 2024-09-16T03:39:25.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.41747625599999999 2024-09-16T03:39:25.680 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.680 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.680 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.680 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.680 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.680 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2160 4.17 4:ed874e32:::smithi15736696-14:head [assert-version v36,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-09-16T03:39:25.680 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:39:25.680 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 78.768162446000005, 2024-09-16T03:39:25.680 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55377054800000003, 2024-09-16T03:39:25.680 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2160 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.681 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830149+0000", 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.743e-06 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830162+0000", 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3283e-05 2024-09-16T03:39:25.682 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830191+0000", 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8466000000000002e-05 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830207+0000", 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6453e-05 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830362+0000", 2024-09-16T03:39:25.683 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015450699999999999 2024-09-16T03:39:25.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830608+0000", 2024-09-16T03:39:25.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024587399999999998 2024-09-16T03:39:25.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:39:25.684 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.233387+0000", 2024-09-16T03:39:25.684 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40277970099999999 2024-09-16T03:39:25.684 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.684 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.684 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234208+0000", 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082053900000000001 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234370+0000", 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000161645 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383821+0000", 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14945145300000001 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.685 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383867+0000", 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5498999999999999e-05 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383916+0000", 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9385000000000002e-05 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.686 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2532 4.0 4:06b6d050:::smithi15736696-50:head [write 2603977~725887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:25.687 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:39:25.687 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.004015057000004, 2024-09-16T03:39:25.687 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53439557900000001, 2024-09-16T03:39:25.687 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.687 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.687 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.687 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.687 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.687 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.687 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2532 2024-09-16T03:39:25.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.688 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:39:25.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.688 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.688 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:39:25.688 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.688 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.688 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594298+0000", 2024-09-16T03:39:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6030000000000004e-06 2024-09-16T03:39:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594876+0000", 2024-09-16T03:39:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00057757800000000003 2024-09-16T03:39:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594885+0000", 2024-09-16T03:39:25.689 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4150000000000005e-06 2024-09-16T03:39:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594892+0000", 2024-09-16T03:39:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7229999999999998e-06 2024-09-16T03:39:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628138+0000", 2024-09-16T03:39:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033246407999999998 2024-09-16T03:39:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.690 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628252+0000", 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113755 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628344+0000", 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2028999999999996e-05 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730215+0000", 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101870761 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.691 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.692 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.692 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.732105+0000", 2024-09-16T03:39:25.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001889951 2024-09-16T03:39:25.692 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.692 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.692 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.692 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128411+0000", 2024-09-16T03:39:25.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.396305764 2024-09-16T03:39:25.692 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.692 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.692 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.692 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128611+0000", 2024-09-16T03:39:25.692 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019990599999999999 2024-09-16T03:39:25.693 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.693 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.693 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.693 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128688+0000", 2024-09-16T03:39:25.693 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7686000000000001e-05 2024-09-16T03:39:25.693 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.693 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.693 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.693 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.693 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.693 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2533 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3663926] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:25.693 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:39:25.693 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.003407703999997, 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53394911599999995, 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2533 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:39:25.694 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.695 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.695 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.695 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594900+0000", 2024-09-16T03:39:25.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.695 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.695 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.695 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594901+0000", 2024-09-16T03:39:25.695 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6700000000000003e-07 2024-09-16T03:39:25.695 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.695 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594904+0000", 2024-09-16T03:39:25.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1099999999999999e-06 2024-09-16T03:39:25.696 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594907+0000", 2024-09-16T03:39:25.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1020000000000001e-06 2024-09-16T03:39:25.696 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.696 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.696 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.696 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594910+0000", 2024-09-16T03:39:25.696 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7889999999999999e-06 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729537+0000", 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.134627725 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729660+0000", 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012283800000000001 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:25.697 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729786+0000", 2024-09-16T03:39:25.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012572000000000001 2024-09-16T03:39:25.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730385+0000", 2024-09-16T03:39:25.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00059949700000000005 2024-09-16T03:39:25.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.698 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.698 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.028223+0000", 2024-09-16T03:39:25.698 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29783727999999998 2024-09-16T03:39:25.698 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.698 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128774+0000", 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100551027 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128816+0000", 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1946000000000001e-05 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128849+0000", 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3515e-05 2024-09-16T03:39:25.699 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.700 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.700 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.700 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.700 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.700 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2534 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:25.700 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:25.700 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 69.003393282000005, 2024-09-16T03:39:25.700 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53406219200000005, 2024-09-16T03:39:25.700 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.700 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.700 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:25.700 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.700 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.701 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.701 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2534 2024-09-16T03:39:25.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.701 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.701 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:25.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.701 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.701 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.701 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.701 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:25.701 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:25.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8999999999999998e-07 2024-09-16T03:39:25.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.702 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594916+0000", 2024-09-16T03:39:25.702 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.344e-06 2024-09-16T03:39:25.702 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.702 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.702 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594919+0000", 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0580000000000002e-06 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594922+0000", 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4389999999999999e-06 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729999+0000", 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.13507707399999999 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.703 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:25.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730046+0000", 2024-09-16T03:39:25.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7382999999999997e-05 2024-09-16T03:39:25.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128879+0000", 2024-09-16T03:39:25.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39883298499999997 2024-09-16T03:39:25.704 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.704 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.704 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.704 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128924+0000", 2024-09-16T03:39:25.704 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:39:25.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.705 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.705 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128977+0000", 2024-09-16T03:39:25.705 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2648999999999997e-05 2024-09-16T03:39:25.705 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.705 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.705 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.705 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.705 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.705 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2621 4.f 4:fa163101:::smithi15736696-10:head [setxattr _header (56) in=7b,truncate 3815189] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:39:25.705 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:39:25.706 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 65.355616904000001, 2024-09-16T03:39:25.706 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55430782499999998, 2024-09-16T03:39:25.706 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.706 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.706 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.706 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.706 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.706 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.706 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2621 2024-09-16T03:39:25.706 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.706 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.706 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.707 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:39:25.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.707 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:39:25.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.707 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.707 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.707 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.707 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242692+0000", 2024-09-16T03:39:25.707 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7599999999999996e-07 2024-09-16T03:39:25.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242695+0000", 2024-09-16T03:39:25.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.529e-06 2024-09-16T03:39:25.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.708 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242699+0000", 2024-09-16T03:39:25.708 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1010000000000002e-06 2024-09-16T03:39:25.708 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.708 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.708 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242703+0000", 2024-09-16T03:39:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.518e-06 2024-09-16T03:39:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.494935+0000", 2024-09-16T03:39:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25223231200000001 2024-09-16T03:39:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495011+0000", 2024-09-16T03:39:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5692999999999995e-05 2024-09-16T03:39:25.709 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495117+0000", 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000105944 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.596308+0000", 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101190825 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796852+0000", 2024-09-16T03:39:25.710 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20054443199999999 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796893+0000", 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1221999999999999e-05 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796946+0000", 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2435000000000002e-05 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796999+0000", 2024-09-16T03:39:25.711 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3038000000000003e-05 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2622 4.f 4:fa163101:::smithi15736696-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 65.355601049000001, 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55448354399999999, 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.712 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.713 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.713 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2622 2024-09-16T03:39:25.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.713 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:25.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.713 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.713 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.713 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.713 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:25.713 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9999999999999999e-07 2024-09-16T03:39:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242709+0000", 2024-09-16T03:39:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.429e-06 2024-09-16T03:39:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.714 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242712+0000", 2024-09-16T03:39:25.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6090000000000001e-06 2024-09-16T03:39:25.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242716+0000", 2024-09-16T03:39:25.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6480000000000001e-06 2024-09-16T03:39:25.715 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.715 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.715 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.715 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495371+0000", 2024-09-16T03:39:25.715 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25265485700000001 2024-09-16T03:39:25.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:25.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495424+0000", 2024-09-16T03:39:25.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2927999999999999e-05 2024-09-16T03:39:25.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.716 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.716 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797042+0000", 2024-09-16T03:39:25.716 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30161821900000002 2024-09-16T03:39:25.716 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.716 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797114+0000", 2024-09-16T03:39:25.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2445000000000002e-05 2024-09-16T03:39:25.717 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.717 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.717 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.717 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797190+0000", 2024-09-16T03:39:25.717 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6108999999999999e-05 2024-09-16T03:39:25.717 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.717 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.717 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2941 4.17 4:eb60a28d:::smithi15736696-41:head [setxattr _header (56) in=7b,truncate 2420241] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:39:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:30.762823+0000", 2024-09-16T03:39:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.835484743999999, 2024-09-16T03:39:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51643088500000001, 2024-09-16T03:39:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2941 2024-09-16T03:39:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762823+0000", 2024-09-16T03:39:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762823+0000", 2024-09-16T03:39:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762824+0000", 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3200000000000005e-07 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762826+0000", 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7590000000000002e-06 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762830+0000", 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4879999999999999e-06 2024-09-16T03:39:25.720 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762834+0000", 2024-09-16T03:39:25.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7249999999999999e-06 2024-09-16T03:39:25.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.763900+0000", 2024-09-16T03:39:25.721 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001066851 2024-09-16T03:39:25.721 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.721 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.721 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.721 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.763982+0000", 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1225999999999998e-05 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.764130+0000", 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014830999999999999 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.767207+0000", 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0030773189999999998 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.722 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.062538+0000", 2024-09-16T03:39:25.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29533109499999999 2024-09-16T03:39:25.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:25.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279028+0000", 2024-09-16T03:39:25.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.21648964100000001 2024-09-16T03:39:25.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.723 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.723 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:25.723 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279214+0000", 2024-09-16T03:39:25.723 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000185606 2024-09-16T03:39:25.723 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.724 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.724 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279254+0000", 2024-09-16T03:39:25.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0333000000000002e-05 2024-09-16T03:39:25.724 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.724 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.724 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.724 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.724 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.724 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2942 4.17 4:eb60a28d:::smithi15736696-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-09-16T03:39:25.724 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:30.762852+0000", 2024-09-16T03:39:25.724 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.835455551999999, 2024-09-16T03:39:25.724 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51654379900000003, 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2942 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762852+0000", 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.725 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762852+0000", 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762853+0000", 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762855+0000", 2024-09-16T03:39:25.726 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0810000000000001e-06 2024-09-16T03:39:25.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762860+0000", 2024-09-16T03:39:25.727 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8720000000000001e-06 2024-09-16T03:39:25.727 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.727 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.727 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.727 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.762863+0000", 2024-09-16T03:39:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2930000000000001e-06 2024-09-16T03:39:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.764919+0000", 2024-09-16T03:39:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0020556979999999999 2024-09-16T03:39:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:30.764964+0000", 2024-09-16T03:39:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5788999999999998e-05 2024-09-16T03:39:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.728 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279291+0000", 2024-09-16T03:39:25.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.51432677999999998 2024-09-16T03:39:25.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279337+0000", 2024-09-16T03:39:25.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5892e-05 2024-09-16T03:39:25.729 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.729 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.729 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:25.729 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:31.279396+0000", 2024-09-16T03:39:25.729 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9018000000000003e-05 2024-09-16T03:39:25.730 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.730 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.730 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.730 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.730 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.730 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:25.732 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:25.673+0000 7fedb5739640 -1 osd.9 146 handle_osd_map injecting map crc failure 2024-09-16T03:39:25.732 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:25.729+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:25.732 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:25.732 INFO:tasks.rados.rados.0.smithi157.stdout:1861: got (22) Invalid argument 2024-09-16T03:39:25.733 INFO:tasks.rados.rados.0.smithi157.stdout:1859: finishing write tid 2 to smithi15736696-25 2024-09-16T03:39:25.733 INFO:tasks.rados.rados.0.smithi157.stdout:1863: expect (ObjNum 410 snap 0 seq_num 410) 2024-09-16T03:39:25.733 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:25.733 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:39:25.734 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:39:25.808 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 384470 2024-09-16T03:39:25.832 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:25.831+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:25.838 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:25.839 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:25.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.839 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4526 4.12 4:4a9d235d:::smithi15736696-1:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T03:39:25.839 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:39:25.839 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.26612533599999999, 2024-09-16T03:39:25.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.26614348399999999, 2024-09-16T03:39:25.839 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.839 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.839 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:25.839 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:25.839 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:25.839 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:25.839 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4526 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571773+0000", 2024-09-16T03:39:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3750000000000001e-06 2024-09-16T03:39:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571780+0000", 2024-09-16T03:39:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5480000000000004e-06 2024-09-16T03:39:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571797+0000", 2024-09-16T03:39:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6320999999999998e-05 2024-09-16T03:39:25.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.842 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.842 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571821+0000", 2024-09-16T03:39:25.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4111e-05 2024-09-16T03:39:25.842 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.842 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:25.842 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571884+0000", 2024-09-16T03:39:25.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3608000000000005e-05 2024-09-16T03:39:25.842 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.843 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:25.843 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571981+0000", 2024-09-16T03:39:25.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6255999999999995e-05 2024-09-16T03:39:25.843 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.843 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.843 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.843 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.843 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "pg_update_log_missing_reply(4.12 epoch 146/138 rep_tid 25 lcod 146'22)", 2024-09-16T03:39:25.843 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:25.774919+0000", 2024-09-16T03:39:25.843 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.062976355999999997, 2024-09-16T03:39:25.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.063028985999999995, 2024-09-16T03:39:25.843 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "queued for pg", 2024-09-16T03:39:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.774919+0000", 2024-09-16T03:39:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.774919+0000", 2024-09-16T03:39:25.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.774920+0000", 2024-09-16T03:39:25.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8500000000000002e-07 2024-09-16T03:39:25.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.774922+0000", 2024-09-16T03:39:25.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4250000000000001e-06 2024-09-16T03:39:25.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.846 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.774928+0000", 2024-09-16T03:39:25.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0630000000000002e-06 2024-09-16T03:39:25.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.846 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.846 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:25.846 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.774933+0000", 2024-09-16T03:39:25.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2669999999999997e-06 2024-09-16T03:39:25.846 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.846 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.846 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.847 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.847 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:147,from:15,reservation_nonce:0)", 2024-09-16T03:39:25.847 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:25.836673+0000", 2024-09-16T03:39:25.847 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.0012226800000000001, 2024-09-16T03:39:25.847 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001295987, 2024-09-16T03:39:25.847 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:25.847 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:25.847 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "no flag points reached", 2024-09-16T03:39:25.847 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:25.847 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.847 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:25.847 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.836673+0000", 2024-09-16T03:39:25.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:25.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.836673+0000", 2024-09-16T03:39:25.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:25.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:25.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.836673+0000", 2024-09-16T03:39:25.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.03e-07 2024-09-16T03:39:25.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.836676+0000", 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2450000000000001e-06 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.836680+0000", 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3959999999999999e-06 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2024-09-16T03:39:25.849 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 3 2024-09-16T03:39:25.850 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:25.851 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:39:25.933 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:25.931+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:25.954 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:25.954 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:25.954 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:25.954 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:25.955 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:25.963 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:39:26.033 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:26.032+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:26.037 INFO:tasks.rados.rados.0.smithi157.stdout:1864: finishing write tid 1 to smithi15736696-19 2024-09-16T03:39:26.037 INFO:tasks.rados.rados.0.smithi157.stdout:1864: finishing write tid 2 to smithi15736696-19 2024-09-16T03:39:26.037 INFO:tasks.rados.rados.0.smithi157.stdout:1859: finishing write tid 3 to smithi15736696-25 2024-09-16T03:39:26.037 INFO:tasks.rados.rados.0.smithi157.stdout:1859: finishing write tid 4 to smithi15736696-25 2024-09-16T03:39:26.037 INFO:tasks.rados.rados.0.smithi157.stdout:1858: finishing set_chunk to oid 36 2024-09-16T03:39:26.037 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 129 (ObjNum 408 snap 0 seq_num 408) dirty exists 2024-09-16T03:39:26.037 INFO:tasks.rados.rados.0.smithi157.stdout:1864: finishing write tid 3 to smithi15736696-19 2024-09-16T03:39:26.037 INFO:tasks.rados.rados.0.smithi157.stdout:1864: finishing write tid 4 to smithi15736696-19 2024-09-16T03:39:26.037 INFO:tasks.rados.rados.0.smithi157.stdout:1864: oid 19 updating version 0 to 162 2024-09-16T03:39:26.037 INFO:tasks.rados.rados.0.smithi157.stdout:1864: oid 19 updating version 162 to 164 2024-09-16T03:39:26.037 INFO:tasks.rados.rados.0.smithi157.stdout:1864: oid 19 version 164 is already newer than 163 2024-09-16T03:39:26.037 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 164 (ObjNum 413 snap 0 seq_num 413) dirty exists 2024-09-16T03:39:26.038 INFO:tasks.rados.rados.0.smithi157.stdout:1864: left oid 19 (ObjNum 413 snap 0 seq_num 413) 2024-09-16T03:39:26.038 INFO:tasks.rados.rados.0.smithi157.stdout:1859: finishing write tid 5 to smithi15736696-25 2024-09-16T03:39:26.038 INFO:tasks.rados.rados.0.smithi157.stdout:1859: oid 25 updating version 0 to 149 2024-09-16T03:39:26.038 INFO:tasks.rados.rados.0.smithi157.stdout:1859: oid 25 updating version 149 to 151 2024-09-16T03:39:26.038 INFO:tasks.rados.rados.0.smithi157.stdout:1859: oid 25 version 151 is already newer than 150 2024-09-16T03:39:26.038 INFO:tasks.rados.rados.0.smithi157.stdout:1859: oid 25 version 151 is already newer than 148 2024-09-16T03:39:26.038 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 151 (ObjNum 412 snap 0 seq_num 412) dirty exists 2024-09-16T03:39:26.038 INFO:tasks.rados.rados.0.smithi157.stdout:1859: left oid 25 (ObjNum 412 snap 0 seq_num 412) 2024-09-16T03:39:26.038 INFO:tasks.rados.rados.0.smithi157.stdout:1865: got (0) Success 2024-09-16T03:39:26.038 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 20 (ObjNum 357 snap 0 seq_num 357) dirty exists 2024-09-16T03:39:26.038 INFO:tasks.rados.rados.0.smithi157.stdout:1858: done (7 left) 2024-09-16T03:39:26.038 INFO:tasks.rados.rados.0.smithi157.stdout:1859: done (6 left) 2024-09-16T03:39:26.038 INFO:tasks.rados.rados.0.smithi157.stdout:1860: done (5 left) 2024-09-16T03:39:26.039 INFO:tasks.rados.rados.0.smithi157.stdout:1861: done (4 left) 2024-09-16T03:39:26.039 INFO:tasks.rados.rados.0.smithi157.stdout:1862: done (3 left) 2024-09-16T03:39:26.039 INFO:tasks.rados.rados.0.smithi157.stdout:1863: done (2 left) 2024-09-16T03:39:26.039 INFO:tasks.rados.rados.0.smithi157.stdout:1864: done (1 left) 2024-09-16T03:39:26.039 INFO:tasks.rados.rados.0.smithi157.stdout:1865: done (0 left) 2024-09-16T03:39:26.039 INFO:tasks.rados.rados.0.smithi157.stdout:1866: delete oid 42 current snap is 0 2024-09-16T03:39:26.050 INFO:tasks.rados.rados.0.smithi157.stdout:1866: done (0 left) 2024-09-16T03:39:26.050 INFO:tasks.rados.rados.0.smithi157.stdout:1867: set_chunk oid 28 target oid 2024-09-16T03:39:26.054 INFO:tasks.rados.rados.0.smithi157.stdout:1867: set_chunk oid 28 offset: 1542656 length: 286208 target oid 0b7641b421978209b0b20830501e9af30b0194060ec909b7b8d81cc4541c040b offset: 0 snap -1 2024-09-16T03:39:26.054 INFO:tasks.rados.rados.0.smithi157.stdout:1868: read oid 34 snap -1 2024-09-16T03:39:26.054 INFO:tasks.rados.rados.0.smithi157.stdout:1868: expect (ObjNum 342 snap 0 seq_num 342) 2024-09-16T03:39:26.054 INFO:tasks.rados.rados.0.smithi157.stdout:1869: write oid 16 current snap is 0 2024-09-16T03:39:26.054 INFO:tasks.rados.rados.0.smithi157.stdout:1869: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-09-16T03:39:26.064 INFO:tasks.rados.rados.0.smithi157.stdout:1869: writing smithi15736696-16 from 574905 to 1286302 tid 1 2024-09-16T03:39:26.067 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:26.068 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:26.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:26.068 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:26.068 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.068 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:26.068 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:26.068 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 133.92161674499999, 2024-09-16T03:39:26.068 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:39:26.068 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.068 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.069 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.069 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:26.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.069 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.069 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.069 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:26.069 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.069 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:39:26.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:39:26.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.070 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:39:26.070 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:39:26.070 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.070 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.070 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.071 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:39:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:39:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:39:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:39:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:39:26.072 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 133.921263385, 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.073 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.074 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.075 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:39:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:39:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:39:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:39:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:39:26.076 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:39:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:39:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:39:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:39:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:26.077 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 133.920135587, 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:26.078 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:39:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:39:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:39:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:39:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.079 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:39:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:39:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:39:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:39:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:39:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:39:26.080 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:39:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:39:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:39:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:39:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.081 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:39:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:39:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:39:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 133.919148146, 2024-09-16T03:39:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:39:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.082 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:39:26.083 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:39:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:39:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:39:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:39:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:39:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.084 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:39:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:39:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:39:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:39:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:39:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:39:26.085 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:39:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:39:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:39:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:39:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.086 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 133.918391402, 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.087 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:39:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:39:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.088 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:39:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:39:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:39:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:39:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:39:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:39:26.089 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:39:26.090 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:86 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'8, mlcod=39'5)", 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 133.45687916599999, 2024-09-16T03:39:26.091 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20388281899999999, 2024-09-16T03:39:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:39:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608470+0000", 2024-09-16T03:39:26.092 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608472+0000", 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.227e-06 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608616+0000", 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000144177 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.093 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608638+0000", 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1543e-05 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608645+0000", 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5839999999999997e-06 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608740+0000", 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4716999999999995e-05 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.094 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608763+0000", 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2671e-05 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812264+0000", 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.203501194 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812353+0000", 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8706000000000001e-05 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.095 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:87 4.f e39/32 4:f101f67d:::smithi15736696-22:head v 39'9, mlcod=39'5)", 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 133.456665792, 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20372206400000001, 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:39:26.096 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608683+0000", 2024-09-16T03:39:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608684+0000", 2024-09-16T03:39:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6700000000000003e-07 2024-09-16T03:39:26.097 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608689+0000", 2024-09-16T03:39:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.412e-06 2024-09-16T03:39:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608704+0000", 2024-09-16T03:39:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4397e-05 2024-09-16T03:39:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.098 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.608713+0000", 2024-09-16T03:39:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:39:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710025+0000", 2024-09-16T03:39:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101312032 2024-09-16T03:39:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.710042+0000", 2024-09-16T03:39:26.099 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6889e-05 2024-09-16T03:39:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812361+0000", 2024-09-16T03:39:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10231889600000001 2024-09-16T03:39:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.812405+0000", 2024-09-16T03:39:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4101999999999999e-05 2024-09-16T03:39:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.100 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:989 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'31, mlcod=46'28)", 2024-09-16T03:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 115.677540042, 2024-09-16T03:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40332266900000002, 2024-09-16T03:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.101 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387812+0000", 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.102 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388467+0000", 2024-09-16T03:39:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065539500000000002 2024-09-16T03:39:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388481+0000", 2024-09-16T03:39:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3905e-05 2024-09-16T03:39:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488552+0000", 2024-09-16T03:39:26.103 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10007120899999999 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688818+0000", 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20026627599999999 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688845+0000", 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.104 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791091+0000", 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10224588699999999 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791131+0000", 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0293000000000002e-05 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:990 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'32, mlcod=46'28)", 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:26.105 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 115.57675637, 2024-09-16T03:39:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30256419200000001, 2024-09-16T03:39:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.106 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488594+0000", 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488616+0000", 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2419e-05 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.107 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488627+0000", 2024-09-16T03:39:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1251e-05 2024-09-16T03:39:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588753+0000", 2024-09-16T03:39:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012544399999999 2024-09-16T03:39:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690679+0000", 2024-09-16T03:39:26.108 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101926027 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690695+0000", 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6528000000000002e-05 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791133+0000", 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100438233 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791157+0000", 2024-09-16T03:39:26.109 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-05 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3081 4.1f e102/100 4:fa163101:::smithi15736696-10:head v 102'93, mlcod=102'91)", 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 50.731633901999999, 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20437049600000001, 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.110 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333718+0000", 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3720000000000001e-06 2024-09-16T03:39:26.111 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334338+0000", 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061959099999999996 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334355+0000", 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.677e-05 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334364+0000", 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3780000000000007e-06 2024-09-16T03:39:26.112 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334424+0000", 2024-09-16T03:39:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0084000000000003e-05 2024-09-16T03:39:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334447+0000", 2024-09-16T03:39:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3113e-05 2024-09-16T03:39:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.113 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538030+0000", 2024-09-16T03:39:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20358272399999999 2024-09-16T03:39:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538085+0000", 2024-09-16T03:39:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5464000000000003e-05 2024-09-16T03:39:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.114 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4007 4.37 4:ed874e32:::smithi15736696-14:head [write 601783~551825,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.532150055999999, 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29242759600000001, 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4007 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.115 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533200+0000", 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3600000000000002e-07 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.116 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533841+0000", 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00064141100000000004 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533862+0000", 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0687e-05 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634001+0000", 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100139413 2024-09-16T03:39:26.117 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721351+0000", 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.087349914000000001 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721357+0000", 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7100000000000004e-06 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721891+0000", 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053461299999999997 2024-09-16T03:39:26.118 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.722024+0000", 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013274599999999999 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.722158+0000", 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013362399999999999 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725271+0000", 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003113345 2024-09-16T03:39:26.119 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825258+0000", 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099987260999999994 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825453+0000", 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019453999999999999 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825591+0000", 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013849700000000001 2024-09-16T03:39:26.120 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825626+0000", 2024-09-16T03:39:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4999000000000002e-05 2024-09-16T03:39:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4008 4.37 4:ed874e32:::smithi15736696-14:head [write 1776679~425104] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:39:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.431329933000001, 2024-09-16T03:39:26.121 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29371618199999999, 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4008 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.122 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634021+0000", 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9989999999999998e-06 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634187+0000", 2024-09-16T03:39:26.123 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016655199999999999 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634197+0000", 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7520000000000008e-06 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634205+0000", 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8210000000000002e-06 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.124 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725407+0000", 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091202249999999999 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725506+0000", 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8493999999999996e-05 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725648+0000", 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014187299999999999 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:26.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.727142+0000", 2024-09-16T03:39:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001494478 2024-09-16T03:39:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825761+0000", 2024-09-16T03:39:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098619340999999999 2024-09-16T03:39:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927671+0000", 2024-09-16T03:39:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10190922700000001 2024-09-16T03:39:26.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927705+0000", 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4523000000000001e-05 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927735+0000", 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9872000000000001e-05 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4009 4.37 4:ed874e32:::smithi15736696-14:head [setxattr _header (56) in=7b,truncate 2201783] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.431133006, 2024-09-16T03:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29364004500000002, 2024-09-16T03:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4009 2024-09-16T03:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.128 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:39:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:39:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634217+0000", 2024-09-16T03:39:26.129 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7199999999999998e-07 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634221+0000", 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9650000000000002e-06 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634226+0000", 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7830000000000004e-06 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634232+0000", 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2909999999999998e-06 2024-09-16T03:39:26.130 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726750+0000", 2024-09-16T03:39:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.09251827 2024-09-16T03:39:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726830+0000", 2024-09-16T03:39:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9808999999999994e-05 2024-09-16T03:39:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:26.131 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726916+0000", 2024-09-16T03:39:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6055000000000002e-05 2024-09-16T03:39:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.727785+0000", 2024-09-16T03:39:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086898599999999998 2024-09-16T03:39:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.926315+0000", 2024-09-16T03:39:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19852985000000001 2024-09-16T03:39:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.132 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927777+0000", 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001462138 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927818+0000", 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1006e-05 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927856+0000", 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8120000000000001e-05 2024-09-16T03:39:26.133 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4010 4.37 4:ed874e32:::smithi15736696-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.431109313, 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39386449699999998, 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:26.134 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4010 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.135 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634243+0000", 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.869e-06 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634249+0000", 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9449999999999998e-06 2024-09-16T03:39:26.136 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634255+0000", 2024-09-16T03:39:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1979999999999998e-06 2024-09-16T03:39:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.734247+0000", 2024-09-16T03:39:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099991570000000002 2024-09-16T03:39:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.137 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.734386+0000", 2024-09-16T03:39:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013961600000000001 2024-09-16T03:39:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.027978+0000", 2024-09-16T03:39:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29359164599999998 2024-09-16T03:39:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.028036+0000", 2024-09-16T03:39:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7802e-05 2024-09-16T03:39:26.138 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.028104+0000", 2024-09-16T03:39:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8264999999999998e-05 2024-09-16T03:39:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4034 4.17 4:eb60a28d:::smithi15736696-41:head [write 635397~702323,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:39:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:39:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.028687867999999, 2024-09-16T03:39:26.139 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471451500000001, 2024-09-16T03:39:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4034 2024-09-16T03:39:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:39:26.140 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:39:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036663+0000", 2024-09-16T03:39:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.855e-06 2024-09-16T03:39:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.141 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037227+0000", 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00056451099999999996 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037240+0000", 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2208999999999999e-05 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037245+0000", 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6319999999999996e-06 2024-09-16T03:39:26.142 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037305+0000", 2024-09-16T03:39:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0272000000000002e-05 2024-09-16T03:39:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037384+0000", 2024-09-16T03:39:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8330000000000004e-05 2024-09-16T03:39:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.143 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037491+0000", 2024-09-16T03:39:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010730699999999999 2024-09-16T03:39:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.042284+0000", 2024-09-16T03:39:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0047932649999999997 2024-09-16T03:39:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.144 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.140922+0000", 2024-09-16T03:39:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098637552000000003 2024-09-16T03:39:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241267+0000", 2024-09-16T03:39:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100344866 2024-09-16T03:39:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241330+0000", 2024-09-16T03:39:26.145 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2953000000000005e-05 2024-09-16T03:39:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241375+0000", 2024-09-16T03:39:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5763000000000002e-05 2024-09-16T03:39:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4037 4.17 4:eb60a28d:::smithi15736696-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:39:26.146 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.141061+0000", 2024-09-16T03:39:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.924287864, 2024-09-16T03:39:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20354841700000001, 2024-09-16T03:39:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4037 2024-09-16T03:39:26.147 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141061+0000", 2024-09-16T03:39:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141061+0000", 2024-09-16T03:39:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.148 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141061+0000", 2024-09-16T03:39:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1199999999999998e-07 2024-09-16T03:39:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141063+0000", 2024-09-16T03:39:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9039999999999999e-06 2024-09-16T03:39:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.149 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141068+0000", 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3730000000000003e-06 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.141071+0000", 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.501e-06 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.142770+0000", 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016985769999999999 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:26.150 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.142821+0000", 2024-09-16T03:39:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1394999999999998e-05 2024-09-16T03:39:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.344429+0000", 2024-09-16T03:39:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.201608341 2024-09-16T03:39:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.344514+0000", 2024-09-16T03:39:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4165999999999993e-05 2024-09-16T03:39:26.151 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.344609+0000", 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5748000000000006e-05 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "rep_scrubmap(4.32 e135 from shard 11)", 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:11.840602+0000", 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.224746637000001, 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.26655261099999999, 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.152 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840602+0000", 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840602+0000", 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.153 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840605+0000", 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840616+0000", 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0672e-05 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840632+0000", 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6268000000000001e-05 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.154 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.940701+0000", 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10006937 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.107074+0000", 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.16637296600000001 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.107101+0000", 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6225999999999999e-05 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.107155+0000", 2024-09-16T03:39:26.155 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4311999999999998e-05 2024-09-16T03:39:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4101 4.17 4:eb60a28d:::smithi15736696-41:head [write 719432~794921] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T03:39:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:11.929701+0000", 2024-09-16T03:39:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.135647513, 2024-09-16T03:39:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30975548400000003, 2024-09-16T03:39:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.156 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4101 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.929701+0000", 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.157 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.929701+0000", 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.929704+0000", 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.21e-06 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.930544+0000", 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00084025300000000001 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.158 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.930561+0000", 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7601999999999999e-05 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.930571+0000", 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1260000000000004e-06 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.030702+0000", 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10013184899999999 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.159 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.030769+0000", 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6742000000000006e-05 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.030873+0000", 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104005 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.033373+0000", 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0025000489999999998 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.160 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.138540+0000", 2024-09-16T03:39:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10516709 2024-09-16T03:39:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.239289+0000", 2024-09-16T03:39:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10074846599999999 2024-09-16T03:39:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.161 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.239382+0000", 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3195000000000005e-05 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.239457+0000", 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4896999999999998e-05 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4366 4.23 4:c512d873:::smithi15736696-38:head [assert-version v41,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:20.361285+0000", 2024-09-16T03:39:26.162 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.7040635999999996, 2024-09-16T03:39:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.303627655, 2024-09-16T03:39:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4366 2024-09-16T03:39:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.163 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361285+0000", 2024-09-16T03:39:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361285+0000", 2024-09-16T03:39:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361287+0000", 2024-09-16T03:39:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.626e-06 2024-09-16T03:39:26.164 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361292+0000", 2024-09-16T03:39:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3279999999999996e-06 2024-09-16T03:39:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361304+0000", 2024-09-16T03:39:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1574e-05 2024-09-16T03:39:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.165 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361310+0000", 2024-09-16T03:39:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1990000000000003e-06 2024-09-16T03:39:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361362+0000", 2024-09-16T03:39:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2459e-05 2024-09-16T03:39:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361448+0000", 2024-09-16T03:39:26.166 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5928999999999997e-05 2024-09-16T03:39:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:39:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.463126+0000", 2024-09-16T03:39:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10167792 2024-09-16T03:39:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.463778+0000", 2024-09-16T03:39:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000652123 2024-09-16T03:39:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.167 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.564004+0000", 2024-09-16T03:39:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100225143 2024-09-16T03:39:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.664718+0000", 2024-09-16T03:39:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100714673 2024-09-16T03:39:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.664865+0000", 2024-09-16T03:39:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014715099999999999 2024-09-16T03:39:26.168 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.664913+0000", 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7530000000000001e-05 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4526 4.12 4:4a9d235d:::smithi15736696-1:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.49357853899999998, 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30344452999999999, 2024-09-16T03:39:26.169 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4526 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.170 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571773+0000", 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3750000000000001e-06 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571780+0000", 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5480000000000004e-06 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.171 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571797+0000", 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6320999999999998e-05 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571821+0000", 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4111e-05 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571884+0000", 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3608000000000005e-05 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.172 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571981+0000", 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6255999999999995e-05 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.875215+0000", 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30323431099999998 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:26.173 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:26.175 INFO:tasks.rados.rados.0.smithi157.stdout:1869: writing smithi15736696-16 from 1885086 to 2410226 tid 2 2024-09-16T03:39:26.175 INFO:tasks.rados.rados.0.smithi157.stdout:1869: writing smithi15736696-16 from 2930189 to 2974905 tid 3 2024-09-16T03:39:26.175 INFO:tasks.rados.rados.0.smithi157.stdout:1870: tier_flush oid 1 2024-09-16T03:39:26.176 INFO:tasks.rados.rados.0.smithi157.stdout:1870: tier_flush oid 1 snap -1 2024-09-16T03:39:26.176 INFO:tasks.rados.rados.0.smithi157.stdout:1871: read oid 18 snap -1 2024-09-16T03:39:26.176 INFO:tasks.rados.rados.0.smithi157.stdout:1871: expect deleted 2024-09-16T03:39:26.176 INFO:tasks.rados.rados.0.smithi157.stdout:1872: read oid 3 snap -1 2024-09-16T03:39:26.176 INFO:tasks.rados.rados.0.smithi157.stdout:1872: expect deleted 2024-09-16T03:39:26.176 INFO:tasks.rados.rados.0.smithi157.stdout:1873: set_chunk oid 5 target oid 2024-09-16T03:39:26.176 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:39:26.177 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:26.172+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:26.275 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:26.274+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:26.283 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:26.283 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:26.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.283 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4530 4.1b 4:dcb6965e:::smithi15736696-28:head [assert-version v134,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:26.283 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.053527+0000", 2024-09-16T03:39:26.283 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.22827625300000001, 2024-09-16T03:39:26.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.22829897499999999, 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4530 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.053527+0000", 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.053527+0000", 2024-09-16T03:39:26.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.053528+0000", 2024-09-16T03:39:26.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.282e-06 2024-09-16T03:39:26.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.053533+0000", 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7129999999999996e-06 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.053543+0000", 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0098000000000001e-05 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.053548+0000", 2024-09-16T03:39:26.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4070000000000001e-06 2024-09-16T03:39:26.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.053594+0000", 2024-09-16T03:39:26.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6109999999999997e-05 2024-09-16T03:39:26.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.053655+0000", 2024-09-16T03:39:26.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1029999999999997e-05 2024-09-16T03:39:26.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.288 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:26.288 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.157239+0000", 2024-09-16T03:39:26.288 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103583235 2024-09-16T03:39:26.288 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.288 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.288 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.288 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.157728+0000", 2024-09-16T03:39:26.288 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00048884199999999997 2024-09-16T03:39:26.288 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.288 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.288 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.288 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.183869+0000", 2024-09-16T03:39:26.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.026141863000000001 2024-09-16T03:39:26.289 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.289 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.289 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.289 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4534 4.35 4:ad0b9852:::smithi15736696-16:head [write 574905~711397 in=711397b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:26.289 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.060664+0000", 2024-09-16T03:39:26.289 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.22113930900000001, 2024-09-16T03:39:26.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.22120959400000001, 2024-09-16T03:39:26.289 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.289 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.289 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:39:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4534 2024-09-16T03:39:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060664+0000", 2024-09-16T03:39:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060664+0000", 2024-09-16T03:39:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060665+0000", 2024-09-16T03:39:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2330000000000001e-06 2024-09-16T03:39:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060808+0000", 2024-09-16T03:39:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014335500000000001 2024-09-16T03:39:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060819+0000", 2024-09-16T03:39:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0828e-05 2024-09-16T03:39:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060826+0000", 2024-09-16T03:39:26.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7209999999999997e-06 2024-09-16T03:39:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.237917+0000", 2024-09-16T03:39:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17709092100000001 2024-09-16T03:39:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.237985+0000", 2024-09-16T03:39:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8270999999999997e-05 2024-09-16T03:39:26.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.238092+0000", 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000107324 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241761+0000", 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036689190000000001 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243578+0000", 2024-09-16T03:39:26.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018165029999999999 2024-09-16T03:39:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4535 4.35 4:ad0b9852:::smithi15736696-16:head [write 1885086~525140 in=525140b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.066720+0000", 2024-09-16T03:39:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.215083107, 2024-09-16T03:39:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.21519205499999999, 2024-09-16T03:39:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:39:26.295 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4535 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066720+0000", 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066720+0000", 2024-09-16T03:39:26.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066721+0000", 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.111e-06 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066954+0000", 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023295100000000001 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066965+0000", 2024-09-16T03:39:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0845e-05 2024-09-16T03:39:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066970+0000", 2024-09-16T03:39:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5459999999999999e-06 2024-09-16T03:39:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239646+0000", 2024-09-16T03:39:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17267560400000001 2024-09-16T03:39:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239715+0000", 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8566999999999996e-05 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239843+0000", 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000128861 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241769+0000", 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001925544 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243609+0000", 2024-09-16T03:39:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018399340000000001 2024-09-16T03:39:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4536 4.35 4:ad0b9852:::smithi15736696-16:head [write 2930189~44716 in=44716b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.069712+0000", 2024-09-16T03:39:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.21209149699999999, 2024-09-16T03:39:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.212249144, 2024-09-16T03:39:26.300 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4536 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069712+0000", 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069712+0000", 2024-09-16T03:39:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069713+0000", 2024-09-16T03:39:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1209999999999999e-06 2024-09-16T03:39:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069748+0000", 2024-09-16T03:39:26.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5626000000000001e-05 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069765+0000", 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7104000000000001e-05 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069771+0000", 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4310000000000002e-06 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241008+0000", 2024-09-16T03:39:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17123754899999999 2024-09-16T03:39:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241070+0000", 2024-09-16T03:39:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1351000000000003e-05 2024-09-16T03:39:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:39:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241165+0000", 2024-09-16T03:39:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5372999999999996e-05 2024-09-16T03:39:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241774+0000", 2024-09-16T03:39:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000609258 2024-09-16T03:39:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243890+0000", 2024-09-16T03:39:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021156669999999999 2024-09-16T03:39:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4537 4.35 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=63b,truncate 2974905] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.069780+0000", 2024-09-16T03:39:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.21202300399999999, 2024-09-16T03:39:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.21223305000000001, 2024-09-16T03:39:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:39:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4537 2024-09-16T03:39:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.306 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069780+0000", 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069780+0000", 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069781+0000", 2024-09-16T03:39:26.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:39:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069784+0000", 2024-09-16T03:39:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9170000000000002e-06 2024-09-16T03:39:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069788+0000", 2024-09-16T03:39:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7230000000000004e-06 2024-09-16T03:39:26.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069791+0000", 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0060000000000001e-06 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241479+0000", 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17168776899999999 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241523+0000", 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4270000000000001e-05 2024-09-16T03:39:26.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241598+0000", 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4264000000000006e-05 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241973+0000", 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037565999999999999 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.310 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4538 4.35 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.212007477, 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.21228623799999999, 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for rw locks", 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4538 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.311 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.03e-07 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069797+0000", 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4500000000000001e-06 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069801+0000", 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8850000000000001e-06 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.169935+0000", 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013362100000001 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241791+0000", 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.071856512999999997 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241834+0000", 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2824999999999997e-05 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.314 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.14.21:26 3.0 3:05b9a3f2:::8210651ae676482c60ad6cc4383d48ecbd475ad9aff381c69b540894c39afb3d:head [call cas.chunk_create_or_get_ref in=468073b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.078285+0000", 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.203517855, 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.203857806, 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.078285+0000", 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.078285+0000", 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.078287+0000", 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.415e-06 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.079363+0000", 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001076137 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.079376+0000", 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.277e-05 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.079382+0000", 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0680000000000002e-06 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.137900+0000", 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.058518344 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.137963+0000", 2024-09-16T03:39:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2743000000000006e-05 2024-09-16T03:39:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,6,9", 2024-09-16T03:39:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.138091+0000", 2024-09-16T03:39:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012787399999999999 2024-09-16T03:39:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.138647+0000", 2024-09-16T03:39:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00055655899999999998 2024-09-16T03:39:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.181521+0000", 2024-09-16T03:39:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.042874250000000003 2024-09-16T03:39:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239519+0000", 2024-09-16T03:39:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.057997914999999997 2024-09-16T03:39:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4547 3.0 3.45ea2a34 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.185010+0000", 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.096793566999999997, 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.097168901000000002, 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "no flag points reached", 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4547 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.185010+0000", 2024-09-16T03:39:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.185010+0000", 2024-09-16T03:39:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.185011+0000", 2024-09-16T03:39:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.624e-06 2024-09-16T03:39:26.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.185596+0000", 2024-09-16T03:39:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058486699999999998 2024-09-16T03:39:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.185609+0000", 2024-09-16T03:39:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.305e-05 2024-09-16T03:39:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop_reply(client.4628.0:4537 4.35 e147/138)", 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.243855+0000", 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.037947638999999998, 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.038344333000000001, 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243855+0000", 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243855+0000", 2024-09-16T03:39:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243856+0000", 2024-09-16T03:39:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.36e-07 2024-09-16T03:39:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243857+0000", 2024-09-16T03:39:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3519999999999999e-06 2024-09-16T03:39:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243860+0000", 2024-09-16T03:39:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0410000000000002e-06 2024-09-16T03:39:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243862+0000", 2024-09-16T03:39:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3609999999999999e-06 2024-09-16T03:39:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.325 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-16T03:39:26.326 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 9 2024-09-16T03:39:26.326 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:26.327 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:39:26.376 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:26.374+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:26.471 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:26.471 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:26.471 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:26.471 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:26.471 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:26.477 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:26.475+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:26.479 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:39:26.489 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:26.486+0000 7fa38a6ea640 -1 osd.10 147 handle_osd_map injecting map crc failure 2024-09-16T03:39:26.490 INFO:tasks.rados.rados.0.smithi157.stdout:1873: set_chunk oid 5 offset: 460800 length: 1353216 target oid 72cf844d3cc51ec170d834580a83ac32680729a157079fa0a50fa691e9d443c5 offset: 0 snap -1 2024-09-16T03:39:26.490 INFO:tasks.rados.rados.0.smithi157.stdout:1874: read oid 36 snap -1 2024-09-16T03:39:26.490 INFO:tasks.rados.rados.0.smithi157.stdout:1874: expect (ObjNum 408 snap 0 seq_num 408) 2024-09-16T03:39:26.490 INFO:tasks.rados.rados.0.smithi157.stdout:1875: read oid 40 snap -1 2024-09-16T03:39:26.490 INFO:tasks.rados.rados.0.smithi157.stdout:1875: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:39:26.490 INFO:tasks.rados.rados.0.smithi157.stdout:1876: read oid 20 snap -1 2024-09-16T03:39:26.491 INFO:tasks.rados.rados.0.smithi157.stdout:1876: expect (ObjNum 399 snap 0 seq_num 399) 2024-09-16T03:39:26.491 INFO:tasks.rados.rados.0.smithi157.stdout:1877: set_chunk oid 42 target oid 2024-09-16T03:39:26.491 INFO:tasks.rados.rados.0.smithi157.stdout:1877: set_chunk oid 42 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:39:26.491 INFO:tasks.rados.rados.0.smithi157.stdout:1878: tier_promote oid 9 2024-09-16T03:39:26.491 INFO:tasks.rados.rados.0.smithi157.stdout:1868: expect (ObjNum 342 snap 0 seq_num 342) 2024-09-16T03:39:26.578 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:26.576+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:26.597 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:26.597 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:26.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:26.597 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:26.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3402 4.1b 4.1792cddb (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:44.983846+0000", 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 41.610629996, 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53764449299999995, 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3402 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983846+0000", 2024-09-16T03:39:26.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983846+0000", 2024-09-16T03:39:26.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983855+0000", 2024-09-16T03:39:26.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2059999999999996e-06 2024-09-16T03:39:26.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983860+0000", 2024-09-16T03:39:26.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3990000000000003e-06 2024-09-16T03:39:26.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983869+0000", 2024-09-16T03:39:26.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5609999999999999e-06 2024-09-16T03:39:26.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983875+0000", 2024-09-16T03:39:26.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9900000000000002e-06 2024-09-16T03:39:26.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983933+0000", 2024-09-16T03:39:26.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8519e-05 2024-09-16T03:39:26.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:26.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983940+0000", 2024-09-16T03:39:26.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1579999999999999e-06 2024-09-16T03:39:26.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:45.521480+0000", 2024-09-16T03:39:26.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53753933300000001 2024-09-16T03:39:26.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:45.521490+0000", 2024-09-16T03:39:26.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0327e-05 2024-09-16T03:39:26.602 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3590 4.1b 4:dcb6965e:::smithi15736696-28:head [write 2019164~716432 in=716432b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.052101+0000", 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.542374385000002, 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50534822099999999, 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.603 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3590 2024-09-16T03:39:26.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.604 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052101+0000", 2024-09-16T03:39:26.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.604 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.604 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052101+0000", 2024-09-16T03:39:26.604 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.604 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.604 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052103+0000", 2024-09-16T03:39:26.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.44e-06 2024-09-16T03:39:26.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.605 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052404+0000", 2024-09-16T03:39:26.605 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030129000000000001 2024-09-16T03:39:26.605 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.605 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.605 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052416+0000", 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1868999999999999e-05 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052422+0000", 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.103424+0000", 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051001643999999999 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.606 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.103503+0000", 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9871999999999997e-05 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.103590+0000", 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6268999999999997e-05 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.105959+0000", 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023694179999999999 2024-09-16T03:39:26.607 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.608 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.356869+0000", 2024-09-16T03:39:26.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.250909726 2024-09-16T03:39:26.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.608 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.557321+0000", 2024-09-16T03:39:26.608 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200451935 2024-09-16T03:39:26.608 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.608 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.608 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:26.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.557381+0000", 2024-09-16T03:39:26.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9960000000000002e-05 2024-09-16T03:39:26.609 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.609 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.609 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.609 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.557449+0000", 2024-09-16T03:39:26.609 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8621000000000006e-05 2024-09-16T03:39:26.609 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.609 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.610 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.610 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.610 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.610 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3591 4.1b 4:dcb6965e:::smithi15736696-28:head [write 3335341~194162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:26.610 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:39:26.610 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.538070239, 2024-09-16T03:39:26.610 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60146457399999997, 2024-09-16T03:39:26.610 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.610 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.610 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.610 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.610 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.610 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.611 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3591 2024-09-16T03:39:26.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.611 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:39:26.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.611 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.611 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.611 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:39:26.611 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.611 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056407+0000", 2024-09-16T03:39:26.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8920000000000001e-06 2024-09-16T03:39:26.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056502+0000", 2024-09-16T03:39:26.612 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4838000000000002e-05 2024-09-16T03:39:26.612 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.612 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.612 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.612 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056511+0000", 2024-09-16T03:39:26.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3579999999999992e-06 2024-09-16T03:39:26.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.613 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.613 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156581+0000", 2024-09-16T03:39:26.613 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100069254 2024-09-16T03:39:26.613 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.613 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256413+0000", 2024-09-16T03:39:26.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099832613000000001 2024-09-16T03:39:26.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.614 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256479+0000", 2024-09-16T03:39:26.614 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5388000000000005e-05 2024-09-16T03:39:26.614 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.614 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.614 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256582+0000", 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000102876 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257889+0000", 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001307119 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657701+0000", 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39981235999999998 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.615 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657769+0000", 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7508000000000005e-05 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657827+0000", 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8878000000000001e-05 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657870+0000", 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2490000000000001e-05 2024-09-16T03:39:26.616 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3592 4.1b 4:dcb6965e:::smithi15736696-28:head [setxattr _header (56) in=7b,truncate 3529503] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.437862086999999, 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60139765499999998, 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.617 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.618 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3592 2024-09-16T03:39:26.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.618 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:39:26.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.618 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.618 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.618 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.618 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:39:26.618 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156615+0000", 2024-09-16T03:39:26.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.714e-06 2024-09-16T03:39:26.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.619 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.619 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.619 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156623+0000", 2024-09-16T03:39:26.619 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5939999999999996e-06 2024-09-16T03:39:26.619 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156634+0000", 2024-09-16T03:39:26.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.093e-05 2024-09-16T03:39:26.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.620 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.620 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.620 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156640+0000", 2024-09-16T03:39:26.620 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1890000000000004e-06 2024-09-16T03:39:26.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257172+0000", 2024-09-16T03:39:26.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100532525 2024-09-16T03:39:26.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.621 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257233+0000", 2024-09-16T03:39:26.621 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0736999999999998e-05 2024-09-16T03:39:26.621 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.621 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.621 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:26.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257319+0000", 2024-09-16T03:39:26.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5870000000000003e-05 2024-09-16T03:39:26.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257891+0000", 2024-09-16T03:39:26.622 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057231100000000004 2024-09-16T03:39:26.622 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.622 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.622 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.622 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657732+0000", 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39984067699999998 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.757877+0000", 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100145238 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.757947+0000", 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9821999999999997e-05 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.623 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.624 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758011+0000", 2024-09-16T03:39:26.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4047999999999994e-05 2024-09-16T03:39:26.624 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.624 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.624 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.624 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.624 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.624 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3593 4.1b 4:dcb6965e:::smithi15736696-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:26.624 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:39:26.624 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.437827353000003, 2024-09-16T03:39:26.624 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60159266700000003, 2024-09-16T03:39:26.624 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.625 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.625 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:26.625 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.625 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.625 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.625 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3593 2024-09-16T03:39:26.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.625 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.625 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.625 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.625 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:39:26.625 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.625 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156649+0000", 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6400000000000002e-07 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156652+0000", 2024-09-16T03:39:26.626 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9579999999999999e-06 2024-09-16T03:39:26.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156658+0000", 2024-09-16T03:39:26.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4729999999999997e-06 2024-09-16T03:39:26.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.627 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.627 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156667+0000", 2024-09-16T03:39:26.627 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1940000000000004e-06 2024-09-16T03:39:26.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.627 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257836+0000", 2024-09-16T03:39:26.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101168444 2024-09-16T03:39:26.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:26.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257881+0000", 2024-09-16T03:39:26.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4910999999999997e-05 2024-09-16T03:39:26.628 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.628 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.628 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.628 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758073+0000", 2024-09-16T03:39:26.628 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50019239699999996 2024-09-16T03:39:26.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758147+0000", 2024-09-16T03:39:26.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3360000000000002e-05 2024-09-16T03:39:26.629 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.629 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.629 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.629 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758241+0000", 2024-09-16T03:39:26.629 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4265999999999995e-05 2024-09-16T03:39:26.629 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.629 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.630 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.630 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.630 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.630 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4013 4.1f 4:fcb75e43:::smithi15736696-26:head [assert-version v111,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:39:26.630 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.540779+0000", 2024-09-16T03:39:26.630 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 21.053696500000001, 2024-09-16T03:39:26.630 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40351445699999999, 2024-09-16T03:39:26.631 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.631 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.631 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.631 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.631 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.631 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.631 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4013 2024-09-16T03:39:26.631 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.631 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.631 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.631 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.631 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.540779+0000", 2024-09-16T03:39:26.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.632 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.632 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.632 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.632 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.540779+0000", 2024-09-16T03:39:26.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.632 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.632 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.632 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.632 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.540781+0000", 2024-09-16T03:39:26.632 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.787e-06 2024-09-16T03:39:26.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.633 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.633 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.540787+0000", 2024-09-16T03:39:26.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2709999999999998e-06 2024-09-16T03:39:26.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.633 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.633 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.540798+0000", 2024-09-16T03:39:26.633 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1185e-05 2024-09-16T03:39:26.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.633 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.540804+0000", 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1090000000000003e-06 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.640931+0000", 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10012628599999999 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.641041+0000", 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011045299999999999 2024-09-16T03:39:26.634 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 10,12", 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.743114+0000", 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102072786 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.743478+0000", 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00036393900000000003 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.743878+0000", 2024-09-16T03:39:26.635 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040044100000000002 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.944223+0000", 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200344627 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.944256+0000", 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3222999999999997e-05 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.636 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.944294+0000", 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.735e-05 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3969 4.35 4:ad0b9852:::smithi15736696-16:head [assert-version v65,set-chunk] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.621330+0000", 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 20.973145577, 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30476898600000002, 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.637 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3969 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.621330+0000", 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.621330+0000", 2024-09-16T03:39:26.638 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.621333+0000", 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7939999999999998e-06 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.621341+0000", 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0260000000000007e-06 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.621352+0000", 2024-09-16T03:39:26.639 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0835999999999999e-05 2024-09-16T03:39:26.640 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.640 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.640 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.640 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.621364+0000", 2024-09-16T03:39:26.640 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2544000000000001e-05 2024-09-16T03:39:26.640 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.640 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.640 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.640 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.621546+0000", 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000181714 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.621557+0000", 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0917000000000001e-05 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.722855+0000", 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101297731 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.641 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.722988+0000", 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013333900000000001 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.824835+0000", 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10184663300000001 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.825255+0000", 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042076400000000001 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.642 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.825717+0000", 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00046141299999999998 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.926019+0000", 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100302326 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.926061+0000", 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1786999999999998e-05 2024-09-16T03:39:26.643 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.644 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.644 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.644 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.926099+0000", 2024-09-16T03:39:26.644 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8161999999999998e-05 2024-09-16T03:39:26.644 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.644 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.644 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.644 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.644 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.644 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4017 4.1b 4:dcb6965e:::smithi15736696-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:39:26.644 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.865086+0000", 2024-09-16T03:39:26.644 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 20.729389601000001, 2024-09-16T03:39:26.644 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40078993499999999, 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4017 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.865086+0000", 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.865086+0000", 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.865086+0000", 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3200000000000001e-07 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.865088+0000", 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5060000000000001e-06 2024-09-16T03:39:26.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.865092+0000", 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3789999999999999e-06 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.865095+0000", 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9780000000000001e-06 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.928273+0000", 2024-09-16T03:39:26.647 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.063177997 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.928328+0000", 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5276999999999999e-05 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:06.265640+0000", 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33731141599999998 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.648 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:06.265732+0000", 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1739999999999999e-05 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:06.265876+0000", 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014431000000000001 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4069 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548281+0000", 2024-09-16T03:39:26.649 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.046194635999999, 2024-09-16T03:39:26.650 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62975188199999999, 2024-09-16T03:39:26.650 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.650 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.650 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:26.650 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.650 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.650 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.650 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4069 2024-09-16T03:39:26.650 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.650 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.650 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548281+0000", 2024-09-16T03:39:26.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.651 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.651 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548281+0000", 2024-09-16T03:39:26.651 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.651 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.651 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.651 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.651 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548284+0000", 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548292+0000", 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2169999999999994e-06 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548308+0000", 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5602000000000001e-05 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.652 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548322+0000", 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-05 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548706+0000", 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038430199999999999 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548721+0000", 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4440999999999999e-05 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.653 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.177929+0000", 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62920806500000004 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.178033+0000", 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103824 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.654 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4070 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548345+0000", 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.046130778999999, 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.729858275, 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4070 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.655 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548345+0000", 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548345+0000", 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548346+0000", 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.64e-07 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.656 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548354+0000", 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.875e-06 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548377+0000", 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3612999999999999e-05 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548390+0000", 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2836999999999999e-05 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.657 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548761+0000", 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037075600000000003 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548769+0000", 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5110000000000003e-06 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278121+0000", 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72935201800000005 2024-09-16T03:39:26.658 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278203+0000", 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1700999999999996e-05 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4071 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548403+0000", 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.046072788, 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73003053100000004, 2024-09-16T03:39:26.659 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.660 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.660 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:26.660 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.660 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.660 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.660 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4071 2024-09-16T03:39:26.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.660 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.660 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.660 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.660 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548403+0000", 2024-09-16T03:39:26.660 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548403+0000", 2024-09-16T03:39:26.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.661 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.661 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.661 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548404+0000", 2024-09-16T03:39:26.661 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0500000000000002e-07 2024-09-16T03:39:26.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548408+0000", 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2699999999999998e-06 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548415+0000", 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7340000000000002e-06 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548421+0000", 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.84e-06 2024-09-16T03:39:26.662 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548798+0000", 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000377916 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548806+0000", 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5599999999999996e-06 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278371+0000", 2024-09-16T03:39:26.663 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.729564616 2024-09-16T03:39:26.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.664 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.664 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278433+0000", 2024-09-16T03:39:26.664 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2689999999999998e-05 2024-09-16T03:39:26.664 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.664 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.664 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.664 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.664 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4079 3.0 3:18759fb9:::13e91a6c9fb63c85209a4d9065cc1e1225929f8ddc4955a7e63272d6c7055a80:head [write 0~1183232] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:39:26.665 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548428+0000", 2024-09-16T03:39:26.665 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 16.046047503, 2024-09-16T03:39:26.665 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0303502819999999, 2024-09-16T03:39:26.665 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.665 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.665 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.665 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.665 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.665 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4079 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548428+0000", 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548428+0000", 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.666 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.667 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.667 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548429+0000", 2024-09-16T03:39:26.667 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2399999999999998e-07 2024-09-16T03:39:26.667 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.667 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.667 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.667 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549249+0000", 2024-09-16T03:39:26.667 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00081996300000000005 2024-09-16T03:39:26.667 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.667 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.667 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.667 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549263+0000", 2024-09-16T03:39:26.668 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4319e-05 2024-09-16T03:39:26.668 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.668 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.668 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.668 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549271+0000", 2024-09-16T03:39:26.668 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1640000000000001e-06 2024-09-16T03:39:26.668 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.668 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.668 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.668 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549342+0000", 2024-09-16T03:39:26.668 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0638999999999999e-05 2024-09-16T03:39:26.668 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.668 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549353+0000", 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1653e-05 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.274940+0000", 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72558676 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.275056+0000", 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011581 2024-09-16T03:39:26.669 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,6,9", 2024-09-16T03:39:26.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.275176+0000", 2024-09-16T03:39:26.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119963 2024-09-16T03:39:26.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.670 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.377903+0000", 2024-09-16T03:39:26.670 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10272743199999999 2024-09-16T03:39:26.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.670 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.670 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578407+0000", 2024-09-16T03:39:26.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200503758 2024-09-16T03:39:26.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.671 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.671 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578475+0000", 2024-09-16T03:39:26.671 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7718000000000005e-05 2024-09-16T03:39:26.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.671 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578663+0000", 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018870599999999999 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578706+0000", 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2459000000000001e-05 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578778+0000", 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2414000000000002e-05 2024-09-16T03:39:26.672 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop_reply(client.4628.0:4199 4.1b e139/115 ondisk, result = 0)", 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:14.720724+0000", 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 11.873751929999999, 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40205568400000002, 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.673 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:14.720724+0000", 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:14.720724+0000", 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:14.720725+0000", 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0160000000000001e-06 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.674 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:14.720730+0000", 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3090000000000003e-06 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:14.720737+0000", 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8909999999999998e-06 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:14.720742+0000", 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7199999999999997e-06 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.675 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122680+0000", 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40193894600000002 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122686+0000", 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3360000000000002e-06 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122779+0000", 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3466000000000003e-05 2024-09-16T03:39:26.676 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop_reply(client.4628.0:4200 4.1b e139/115 ondisk, result = 0)", 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:14.720746+0000", 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 11.873729642000001, 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40218438000000001, 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.677 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:14.720746+0000", 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:14.720746+0000", 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:14.720746+0000", 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2499999999999999e-07 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.678 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:14.720747+0000", 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.296e-06 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:14.720757+0000", 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7410000000000004e-06 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:14.720777+0000", 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0103e-05 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.679 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.680 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.680 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122806+0000", 2024-09-16T03:39:26.680 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.402028516 2024-09-16T03:39:26.680 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.680 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.680 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.680 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122814+0000", 2024-09-16T03:39:26.680 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6600000000000001e-06 2024-09-16T03:39:26.680 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.681 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.681 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.681 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122930+0000", 2024-09-16T03:39:26.681 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115839 2024-09-16T03:39:26.681 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.681 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.681 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.681 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.681 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.681 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.3.22:33 3.0 3:18759fb9:::13e91a6c9fb63c85209a4d9065cc1e1225929f8ddc4955a7e63272d6c7055a80:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e140)", 2024-09-16T03:39:26.682 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:15.122660+0000", 2024-09-16T03:39:26.682 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 11.471815646, 2024-09-16T03:39:26.682 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.38719735599999999, 2024-09-16T03:39:26.682 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.682 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.682 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.682 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.683 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.683 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.683 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122660+0000", 2024-09-16T03:39:26.683 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.683 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.683 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.683 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.683 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122660+0000", 2024-09-16T03:39:26.683 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.683 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.684 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122662+0000", 2024-09-16T03:39:26.684 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.846e-06 2024-09-16T03:39:26.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.684 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122668+0000", 2024-09-16T03:39:26.684 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8810000000000001e-06 2024-09-16T03:39:26.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.684 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.684 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122680+0000", 2024-09-16T03:39:26.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2363e-05 2024-09-16T03:39:26.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.685 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122686+0000", 2024-09-16T03:39:26.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6840000000000001e-06 2024-09-16T03:39:26.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.685 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.685 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.685 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122731+0000", 2024-09-16T03:39:26.685 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5420000000000002e-05 2024-09-16T03:39:26.685 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122817+0000", 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5859000000000006e-05 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,6,9", 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.122984+0000", 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016696899999999999 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.123506+0000", 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00052237500000000005 2024-09-16T03:39:26.686 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.124260+0000", 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00075374199999999995 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.224447+0000", 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10018708799999999 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.509755+0000", 2024-09-16T03:39:26.687 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.28530834599999999 2024-09-16T03:39:26.688 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.688 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.688 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:26.688 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.509794+0000", 2024-09-16T03:39:26.688 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8056000000000003e-05 2024-09-16T03:39:26.688 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.688 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.688 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.688 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.509857+0000", 2024-09-16T03:39:26.688 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3726999999999995e-05 2024-09-16T03:39:26.688 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.688 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.688 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4211 4.35 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=7b,truncate 2521116] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:15.308738+0000", 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 11.285737934, 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39722028999999998, 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4211 2024-09-16T03:39:26.689 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308738+0000", 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308738+0000", 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.690 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308739+0000", 2024-09-16T03:39:26.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.068e-06 2024-09-16T03:39:26.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.691 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.691 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.691 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308744+0000", 2024-09-16T03:39:26.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0509999999999996e-06 2024-09-16T03:39:26.691 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.691 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.691 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.691 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308762+0000", 2024-09-16T03:39:26.691 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8657000000000001e-05 2024-09-16T03:39:26.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.692 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.692 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308775+0000", 2024-09-16T03:39:26.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2768999999999999e-05 2024-09-16T03:39:26.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.692 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.692 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.692 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.309792+0000", 2024-09-16T03:39:26.692 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001017186 2024-09-16T03:39:26.692 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.309868+0000", 2024-09-16T03:39:26.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5468000000000003e-05 2024-09-16T03:39:26.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.693 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:39:26.693 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.309975+0000", 2024-09-16T03:39:26.693 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000106908 2024-09-16T03:39:26.693 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.693 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.311122+0000", 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011467980000000001 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.410371+0000", 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099249765000000004 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.705870+0000", 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29549829500000002 2024-09-16T03:39:26.694 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:26.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.705908+0000", 2024-09-16T03:39:26.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8612999999999998e-05 2024-09-16T03:39:26.695 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.695 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.695 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.695 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.705958+0000", 2024-09-16T03:39:26.695 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9712000000000001e-05 2024-09-16T03:39:26.695 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.695 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.696 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.696 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.696 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.696 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4212 4.35 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e140)", 2024-09-16T03:39:26.696 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:15.308792+0000", 2024-09-16T03:39:26.696 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 11.285683800999999, 2024-09-16T03:39:26.696 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.49743623399999998, 2024-09-16T03:39:26.696 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.696 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.696 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4212 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308792+0000", 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.697 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308792+0000", 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308792+0000", 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.75e-07 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308796+0000", 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3160000000000001e-06 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.698 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308805+0000", 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2560000000000008e-06 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308812+0000", 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.635e-06 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.310568+0000", 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001756075 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.699 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.310639+0000", 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0964999999999996e-05 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.806106+0000", 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.49546743700000001 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.806160+0000", 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3767000000000002e-05 2024-09-16T03:39:26.700 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.806228+0000", 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8108000000000006e-05 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4243 3.0 3:2bcb1f44:::8a777defa75e38fc371085a7cd11b86a5f577618baee3efc60aad962871bb246:head [write 0~636928] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:16.107165+0000", 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 10.487310438, 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30437911400000001, 2024-09-16T03:39:26.701 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.702 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.702 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.702 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:26.702 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:26.702 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:26.702 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4243 2024-09-16T03:39:26.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.702 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.702 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.702 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.702 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.107165+0000", 2024-09-16T03:39:26.702 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.702 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.107165+0000", 2024-09-16T03:39:26.703 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.107167+0000", 2024-09-16T03:39:26.703 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9410000000000001e-06 2024-09-16T03:39:26.703 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.703 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.703 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.703 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.107439+0000", 2024-09-16T03:39:26.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00027169800000000001 2024-09-16T03:39:26.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.107451+0000", 2024-09-16T03:39:26.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1977000000000001e-05 2024-09-16T03:39:26.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.704 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.704 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.704 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.207544+0000", 2024-09-16T03:39:26.704 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10009367399999999 2024-09-16T03:39:26.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.207616+0000", 2024-09-16T03:39:26.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1416999999999997e-05 2024-09-16T03:39:26.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.705 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.207724+0000", 2024-09-16T03:39:26.705 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010835 2024-09-16T03:39:26.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.705 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.705 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,6,9", 2024-09-16T03:39:26.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.207847+0000", 2024-09-16T03:39:26.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012327599999999999 2024-09-16T03:39:26.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.208999+0000", 2024-09-16T03:39:26.706 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001151413 2024-09-16T03:39:26.706 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.706 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.706 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.706 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.209381+0000", 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038246100000000001 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.310366+0000", 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100984979 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.411424+0000", 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101057851 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.707 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:26.708 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.411462+0000", 2024-09-16T03:39:26.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8016999999999999e-05 2024-09-16T03:39:26.708 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.708 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.708 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:16.411544+0000", 2024-09-16T03:39:26.708 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2059999999999997e-05 2024-09-16T03:39:26.708 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.708 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.708 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.708 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.708 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.709 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:173 3.0 3:3f1f54cb:::df21b27e4940b754b9c0285d87bcbc8dbfcdd0059bba94745a59fbc37acf1ebe:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e146)", 2024-09-16T03:39:26.709 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:25.238052+0000", 2024-09-16T03:39:26.709 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.3564233299999999, 2024-09-16T03:39:26.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.41153400800000001, 2024-09-16T03:39:26.709 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.709 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.709 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.709 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.709 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.709 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.709 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238052+0000", 2024-09-16T03:39:26.709 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238052+0000", 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238054+0000", 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0600000000000002e-06 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.710 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238060+0000", 2024-09-16T03:39:26.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5169999999999999e-06 2024-09-16T03:39:26.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238071+0000", 2024-09-16T03:39:26.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1182999999999999e-05 2024-09-16T03:39:26.711 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.711 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.711 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.711 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238079+0000", 2024-09-16T03:39:26.711 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0499999999999992e-06 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238125+0000", 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6146000000000002e-05 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238185+0000", 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9463000000000003e-05 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,6,9", 2024-09-16T03:39:26.712 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238324+0000", 2024-09-16T03:39:26.713 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013895100000000001 2024-09-16T03:39:26.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.713 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.713 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238802+0000", 2024-09-16T03:39:26.713 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00047803899999999998 2024-09-16T03:39:26.713 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.713 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.239099+0000", 2024-09-16T03:39:26.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029756100000000001 2024-09-16T03:39:26.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.714 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.439494+0000", 2024-09-16T03:39:26.714 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200395028 2024-09-16T03:39:26.714 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.714 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.714 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.649394+0000", 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20989983100000001 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.649465+0000", 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1077999999999999e-05 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.649586+0000", 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000121101 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.715 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.14.21:26 3.0 3:05b9a3f2:::8210651ae676482c60ad6cc4383d48ecbd475ad9aff381c69b540894c39afb3d:head [call cas.chunk_create_or_get_ref in=468073b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.078285+0000", 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.51619032300000001, 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.310005429, 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:26.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.078285+0000", 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.078285+0000", 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.078287+0000", 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.415e-06 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.079363+0000", 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001076137 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.079376+0000", 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.277e-05 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.079382+0000", 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0680000000000002e-06 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.137900+0000", 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.058518344 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.137963+0000", 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2743000000000006e-05 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,6,9", 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.138091+0000", 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012787399999999999 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.719 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.138647+0000", 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00055655899999999998 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.181521+0000", 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.042874250000000003 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239519+0000", 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.057997914999999997 2024-09-16T03:39:26.720 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.388131+0000", 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.148612144 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.388181+0000", 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9370000000000003e-05 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.388291+0000", 2024-09-16T03:39:26.721 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010984000000000001 2024-09-16T03:39:26.722 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.722 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.722 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.722 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:26.722 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:26.722 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:26.722 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:39:26.724 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:26.721+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:26.725 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:39:26.726 INFO:tasks.rados.rados.0.smithi157.stdout:1867: finishing set_chunk to oid 28 2024-09-16T03:39:26.726 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 28 v 134 (ObjNum 387 snap 0 seq_num 387) dirty exists 2024-09-16T03:39:26.726 INFO:tasks.rados.rados.0.smithi157.stdout:1870: got (0) Success 2024-09-16T03:39:26.726 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 1 v 0 (ObjNum 357 snap 0 seq_num 357) dirty exists 2024-09-16T03:39:26.726 INFO:tasks.rados.rados.0.smithi157.stdout:1867: done (11 left) 2024-09-16T03:39:26.726 INFO:tasks.rados.rados.0.smithi157.stdout:1868: done (10 left) 2024-09-16T03:39:26.726 INFO:tasks.rados.rados.0.smithi157.stdout:1870: done (9 left) 2024-09-16T03:39:26.726 INFO:tasks.rados.rados.0.smithi157.stdout:1871: done (8 left) 2024-09-16T03:39:26.726 INFO:tasks.rados.rados.0.smithi157.stdout:1872: done (7 left) 2024-09-16T03:39:26.726 INFO:tasks.rados.rados.0.smithi157.stdout:1879: read oid 38 snap -1 2024-09-16T03:39:26.726 INFO:tasks.rados.rados.0.smithi157.stdout:1879: expect (ObjNum 397 snap 0 seq_num 397) 2024-09-16T03:39:26.726 INFO:tasks.rados.rados.0.smithi157.stdout:1880: delete oid 22 current snap is 0 2024-09-16T03:39:26.727 INFO:tasks.rados.rados.0.smithi157.stdout:1874: expect (ObjNum 408 snap 0 seq_num 408) 2024-09-16T03:39:26.727 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:26.825 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:26.822+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:26.832 INFO:tasks.rados.rados.0.smithi157.stdout:1876: expect (ObjNum 399 snap 0 seq_num 399) 2024-09-16T03:39:26.861 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:26.862 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:26.862 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:26.869 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:26.870 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:26.870 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:39:26.870 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:26.878 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:39:26.880 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:26.880 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:26.897 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:26.897 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:26.915 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:26.915 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:26.925 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:26.923+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:26.931 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:26.931 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:26.943 INFO:tasks.rados.rados.0.smithi157.stdout:1877: finishing set_chunk to oid 42 2024-09-16T03:39:26.943 INFO:tasks.rados.rados.0.smithi157.stdout:1877: got expected ENOENT (src dne) 2024-09-16T03:39:26.943 INFO:tasks.rados.rados.0.smithi157.stdout:1875: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:39:26.943 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:26.948 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:26.949 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:26.965 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:26.965 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:26.983 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:26.983 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:26.985 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:26.985 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:26.994 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:39:26.999 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:26.999 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:27.000 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:27.017 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:27.017 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:27.026 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:27.023+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:27.033 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:27.033 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:27.050 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:27.050 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:27.065 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:27.066 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:27.066 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 698701 2024-09-16T03:39:27.083 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:27.083 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:27.099 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:27.099 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:27.100 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:27.100 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:27.108 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:39:27.110 INFO:tasks.rados.rados.0.smithi157.stdout:1869: finishing write tid 1 to smithi15736696-16 2024-09-16T03:39:27.110 INFO:tasks.rados.rados.0.smithi157.stdout:1869: finishing write tid 2 to smithi15736696-16 2024-09-16T03:39:27.110 INFO:tasks.rados.rados.0.smithi157.stdout:1878: got (2) No such file or directory 2024-09-16T03:39:27.110 INFO:tasks.rados.rados.0.smithi157.stdout:1878: got expected ENOENT (src dne) 2024-09-16T03:39:27.110 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 86 (ObjNum 33 snap 0 seq_num 1503462679) clean dne 2024-09-16T03:39:27.110 INFO:tasks.rados.rados.0.smithi157.stdout:1869: finishing write tid 3 to smithi15736696-16 2024-09-16T03:39:27.110 INFO:tasks.rados.rados.0.smithi157.stdout:1874: done (8 left) 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1875: done (7 left) 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1876: done (6 left) 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1877: done (5 left) 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1878: done (4 left) 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1880: done (3 left) 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1881: read oid 39 snap -1 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1881: expect (ObjNum 365 snap 0 seq_num 365) 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1882: write oid 18 current snap is 0 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1869: finishing write tid 4 to smithi15736696-16 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1869: finishing write tid 5 to smithi15736696-16 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1869: oid 16 updating version 0 to 72 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1869: oid 16 updating version 72 to 74 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1869: oid 16 version 74 is already newer than 73 2024-09-16T03:39:27.111 INFO:tasks.rados.rados.0.smithi157.stdout:1869: oid 16 updating version 74 to 75 2024-09-16T03:39:27.112 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 75 (ObjNum 415 snap 0 seq_num 415) dirty exists 2024-09-16T03:39:27.112 INFO:tasks.rados.rados.0.smithi157.stdout:1869: left oid 16 (ObjNum 415 snap 0 seq_num 415) 2024-09-16T03:39:27.112 INFO:tasks.rados.rados.0.smithi157.stdout:1879: expect (ObjNum 397 snap 0 seq_num 397) 2024-09-16T03:39:27.116 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:27.117 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:27.120 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2041892 2024-09-16T03:39:27.127 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:27.125+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:27.127 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:27.157 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:27.187 INFO:tasks.rados.rados.0.smithi157.stdout:1882: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-09-16T03:39:27.193 INFO:tasks.rados.rados.0.smithi157.stdout:1882: writing smithi15736696-18 from 608519 to 1070414 tid 1 2024-09-16T03:39:27.199 INFO:tasks.rados.rados.0.smithi157.stdout:1882: writing smithi15736696-18 from 1602050 to 2130859 tid 2 2024-09-16T03:39:27.204 INFO:tasks.rados.rados.0.smithi157.stdout:1882: writing smithi15736696-18 from 2585035 to 3008519 tid 3 2024-09-16T03:39:27.205 INFO:tasks.rados.rados.0.smithi157.stdout:1869: done (4 left) 2024-09-16T03:39:27.205 INFO:tasks.rados.rados.0.smithi157.stdout:1879: done (3 left) 2024-09-16T03:39:27.205 INFO:tasks.rados.rados.0.smithi157.stdout:1883: read oid 12 snap -1 2024-09-16T03:39:27.205 INFO:tasks.rados.rados.0.smithi157.stdout:1883: expect (ObjNum 297 snap 0 seq_num 297) 2024-09-16T03:39:27.205 INFO:tasks.rados.rados.0.smithi157.stdout:1884: read oid 40 snap -1 2024-09-16T03:39:27.205 INFO:tasks.rados.rados.0.smithi157.stdout:1884: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:39:27.205 INFO:tasks.rados.rados.0.smithi157.stdout:1885: read oid 14 snap -1 2024-09-16T03:39:27.205 INFO:tasks.rados.rados.0.smithi157.stdout:1885: expect (ObjNum 398 snap 0 seq_num 398) 2024-09-16T03:39:27.205 INFO:tasks.rados.rados.0.smithi157.stdout:1886: tier_evict oid 31 2024-09-16T03:39:27.205 INFO:tasks.rados.rados.0.smithi157.stdout:1886: tier_evict oid 31 snap -1 2024-09-16T03:39:27.205 INFO:tasks.rados.rados.0.smithi157.stdout:1881: expect (ObjNum 365 snap 0 seq_num 365) 2024-09-16T03:39:27.219 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:27.219 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:27.228 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:39:27.229 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:27.226+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:27.232 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 630505 2024-09-16T03:39:27.272 INFO:tasks.rados.rados.0.smithi157.stdout:1882: finishing write tid 1 to smithi15736696-18 2024-09-16T03:39:27.272 INFO:tasks.rados.rados.0.smithi157.stdout:1885: expect (ObjNum 398 snap 0 seq_num 398) 2024-09-16T03:39:27.272 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 240899 2024-09-16T03:39:27.319 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:27.329 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:27.326+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:27.338 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:27.338 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:27.347 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:39:27.412 INFO:tasks.rados.rados.0.smithi157.stdout:1882: finishing write tid 2 to smithi15736696-18 2024-09-16T03:39:27.413 INFO:tasks.rados.rados.0.smithi157.stdout:1873: finishing set_chunk to oid 5 2024-09-16T03:39:27.413 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 75 (ObjNum 378 snap 0 seq_num 378) dirty exists 2024-09-16T03:39:27.413 INFO:tasks.rados.rados.0.smithi157.stdout:1886: got (22) Invalid argument 2024-09-16T03:39:27.413 INFO:tasks.rados.rados.0.smithi157.stdout:1882: finishing write tid 3 to smithi15736696-18 2024-09-16T03:39:27.413 INFO:tasks.rados.rados.0.smithi157.stdout:1873: done (6 left) 2024-09-16T03:39:27.413 INFO:tasks.rados.rados.0.smithi157.stdout:1881: done (5 left) 2024-09-16T03:39:27.413 INFO:tasks.rados.rados.0.smithi157.stdout:1885: done (4 left) 2024-09-16T03:39:27.413 INFO:tasks.rados.rados.0.smithi157.stdout:1886: done (3 left) 2024-09-16T03:39:27.414 INFO:tasks.rados.rados.0.smithi157.stdout:1887: read oid 13 snap -1 2024-09-16T03:39:27.414 INFO:tasks.rados.rados.0.smithi157.stdout:1887: expect (ObjNum 401 snap 0 seq_num 401) 2024-09-16T03:39:27.414 INFO:tasks.rados.rados.0.smithi157.stdout:1888: read oid 17 snap -1 2024-09-16T03:39:27.414 INFO:tasks.rados.rados.0.smithi157.stdout:1888: expect (ObjNum 406 snap 0 seq_num 406) 2024-09-16T03:39:27.414 INFO:tasks.rados.rados.0.smithi157.stdout:1889: write (excl) oid 43 current snap is 0 2024-09-16T03:39:27.414 INFO:tasks.rados.rados.0.smithi157.stdout:1882: finishing write tid 4 to smithi15736696-18 2024-09-16T03:39:27.414 INFO:tasks.rados.rados.0.smithi157.stdout:1883: expect (ObjNum 297 snap 0 seq_num 297) 2024-09-16T03:39:27.415 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:27.430 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:27.428+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:27.482 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2210206 2024-09-16T03:39:27.491 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:27.492 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:27.495 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:27.500 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:39:27.541 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:27.540+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:27.544 INFO:tasks.rados.rados.0.smithi157.stdout:1884: expect (ObjNum 302 snap 0 seq_num 302) 2024-09-16T03:39:27.559 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1166282 2024-09-16T03:39:27.597 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 459392 2024-09-16T03:39:27.627 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 9] out_osds: [1, 5, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:39:27.627 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:39:27.627 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:39:27.628 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-16T03:39:27.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-09-16T03:39:27.632 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:27.629+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:27.645 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:27.646 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:27.654 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:39:27.663 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1226950 2024-09-16T03:39:27.697 INFO:tasks.rados.rados.0.smithi157.stdout:1882: finishing write tid 5 to smithi15736696-18 2024-09-16T03:39:27.697 INFO:tasks.rados.rados.0.smithi157.stdout:1882: oid 18 updating version 0 to 71 2024-09-16T03:39:27.697 INFO:tasks.rados.rados.0.smithi157.stdout:1882: oid 18 updating version 71 to 72 2024-09-16T03:39:27.698 INFO:tasks.rados.rados.0.smithi157.stdout:1882: oid 18 version 72 is already newer than 70 2024-09-16T03:39:27.698 INFO:tasks.rados.rados.0.smithi157.stdout:1882: oid 18 version 72 is already newer than 69 2024-09-16T03:39:27.698 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 72 (ObjNum 417 snap 0 seq_num 417) dirty exists 2024-09-16T03:39:27.698 INFO:tasks.rados.rados.0.smithi157.stdout:1882: left oid 18 (ObjNum 417 snap 0 seq_num 417) 2024-09-16T03:39:27.698 INFO:tasks.rados.rados.0.smithi157.stdout:1889: seq_num 418 ranges {625326=537144,1944179=672467} 2024-09-16T03:39:27.704 INFO:tasks.rados.rados.0.smithi157.stdout:1889: writing smithi15736696-43 from 625326 to 1162470 tid 1 2024-09-16T03:39:27.714 INFO:tasks.rados.rados.0.smithi157.stdout:1889: writing smithi15736696-43 from 1944179 to 2616646 tid 2 2024-09-16T03:39:27.714 INFO:tasks.rados.rados.0.smithi157.stdout:1882: done (5 left) 2024-09-16T03:39:27.714 INFO:tasks.rados.rados.0.smithi157.stdout:1883: done (4 left) 2024-09-16T03:39:27.714 INFO:tasks.rados.rados.0.smithi157.stdout:1884: done (3 left) 2024-09-16T03:39:27.714 INFO:tasks.rados.rados.0.smithi157.stdout:1890: read oid 33 snap -1 2024-09-16T03:39:27.715 INFO:tasks.rados.rados.0.smithi157.stdout:1890: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-16T03:39:27.715 INFO:tasks.rados.rados.0.smithi157.stdout:1891: tier_promote oid 9 2024-09-16T03:39:27.715 INFO:tasks.rados.rados.0.smithi157.stdout:1887: expect (ObjNum 401 snap 0 seq_num 401) 2024-09-16T03:39:27.732 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:27.730+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:27.735 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:39:27.773 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:27.773 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:27.782 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:39:27.811 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:27.854 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:27.853+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:27.916 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:27.916 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:27.925 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:39:27.934 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:27.933+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:27.946 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-09-16T03:39:27.944+0000 7fdb04aef640 -1 mon.c@2(leader).osd e148 definitely_dead 0 2024-09-16T03:39:28.011 INFO:tasks.rados.rados.0.smithi157.stdout:1888: expect (ObjNum 406 snap 0 seq_num 406) 2024-09-16T03:39:28.033 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:28.034 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:28.035 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:28.033+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:28.042 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:39:28.053 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2542417 2024-09-16T03:39:28.069 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1077163 2024-09-16T03:39:28.136 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:28.134+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:28.149 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:28.150 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:28.158 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:39:28.193 INFO:tasks.rados.rados.0.smithi157.stdout:1891: got (2) No such file or directory 2024-09-16T03:39:28.193 INFO:tasks.rados.rados.0.smithi157.stdout:1891: got expected ENOENT (src dne) 2024-09-16T03:39:28.193 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 86 (ObjNum 33 snap 0 seq_num 1503462679) clean dne 2024-09-16T03:39:28.193 INFO:tasks.rados.rados.0.smithi157.stdout:1887: done (4 left) 2024-09-16T03:39:28.194 INFO:tasks.rados.rados.0.smithi157.stdout:1888: done (3 left) 2024-09-16T03:39:28.194 INFO:tasks.rados.rados.0.smithi157.stdout:1891: done (2 left) 2024-09-16T03:39:28.194 INFO:tasks.rados.rados.0.smithi157.stdout:1892: read oid 15 snap -1 2024-09-16T03:39:28.194 INFO:tasks.rados.rados.0.smithi157.stdout:1892: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-16T03:39:28.194 INFO:tasks.rados.rados.0.smithi157.stdout:1893: delete oid 41 current snap is 0 2024-09-16T03:39:28.194 INFO:tasks.rados.rados.0.smithi157.stdout:1889: finishing write tid 1 to smithi15736696-43 2024-09-16T03:39:28.194 INFO:tasks.rados.rados.0.smithi157.stdout:1889: finishing write tid 2 to smithi15736696-43 2024-09-16T03:39:28.194 INFO:tasks.rados.rados.0.smithi157.stdout:1889: finishing write tid 3 to smithi15736696-43 2024-09-16T03:39:28.194 INFO:tasks.rados.rados.0.smithi157.stdout:1890: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-16T03:39:28.233 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:28.236 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:28.233+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:28.266 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:28.266 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:28.275 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:39:28.278 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1026857 2024-09-16T03:39:28.307 INFO:tasks.rados.rados.0.smithi157.stdout:1889: finishing write tid 4 to smithi15736696-43 2024-09-16T03:39:28.307 INFO:tasks.rados.rados.0.smithi157.stdout:1889: oid 43 updating version 0 to 92 2024-09-16T03:39:28.307 INFO:tasks.rados.rados.0.smithi157.stdout:1889: oid 43 version 92 is already newer than 90 2024-09-16T03:39:28.307 INFO:tasks.rados.rados.0.smithi157.stdout:1889: oid 43 version 92 is already newer than 91 2024-09-16T03:39:28.307 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 92 (ObjNum 418 snap 0 seq_num 418) dirty exists 2024-09-16T03:39:28.307 INFO:tasks.rados.rados.0.smithi157.stdout:1889: left oid 43 (ObjNum 418 snap 0 seq_num 418) 2024-09-16T03:39:28.307 INFO:tasks.rados.rados.0.smithi157.stdout:1892: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-16T03:39:28.312 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 617937 2024-09-16T03:39:28.337 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:28.336+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:28.391 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2154494 2024-09-16T03:39:28.395 INFO:tasks.rados.rados.0.smithi157.stdout:1889: done (3 left) 2024-09-16T03:39:28.395 INFO:tasks.rados.rados.0.smithi157.stdout:1890: done (2 left) 2024-09-16T03:39:28.395 INFO:tasks.rados.rados.0.smithi157.stdout:1892: done (1 left) 2024-09-16T03:39:28.395 INFO:tasks.rados.rados.0.smithi157.stdout:1893: done (0 left) 2024-09-16T03:39:28.395 INFO:tasks.rados.rados.0.smithi157.stdout:1894: read oid 15 snap -1 2024-09-16T03:39:28.395 INFO:tasks.rados.rados.0.smithi157.stdout:1894: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-16T03:39:28.395 INFO:tasks.rados.rados.0.smithi157.stdout:1895: write oid 36 current snap is 0 2024-09-16T03:39:28.395 INFO:tasks.rados.rados.0.smithi157.stdout:1895: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-09-16T03:39:28.400 INFO:tasks.rados.rados.0.smithi157.stdout:1895: writing smithi15736696-36 from 658940 to 1062935 tid 1 2024-09-16T03:39:28.408 INFO:tasks.rados.rados.0.smithi157.stdout:1895: writing smithi15736696-36 from 1777496 to 2453632 tid 2 2024-09-16T03:39:28.411 INFO:tasks.rados.rados.0.smithi157.stdout:1895: writing smithi15736696-36 from 2950951 to 3058940 tid 3 2024-09-16T03:39:28.411 INFO:tasks.rados.rados.0.smithi157.stdout:1896: set_chunk oid 17 target oid 2024-09-16T03:39:28.425 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:28.425 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:28.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:39:28.439 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:28.437+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:28.540 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:28.537+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:28.541 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:28.542 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:28.550 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:39:28.614 INFO:tasks.rados.rados.0.smithi157.stdout:1896: set_chunk oid 17 offset: 2113536 length: 381952 target oid a24fa0526eb5140f90697be189e885c43bfae206d688f606b170a795c713f2ac offset: 0 snap -1 2024-09-16T03:39:28.614 INFO:tasks.rados.rados.0.smithi157.stdout:1897: set_chunk oid 9 target oid 2024-09-16T03:39:28.614 INFO:tasks.rados.rados.0.smithi157.stdout:1897: set_chunk oid 9 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:39:28.614 INFO:tasks.rados.rados.0.smithi157.stdout:1898: read oid 39 snap -1 2024-09-16T03:39:28.614 INFO:tasks.rados.rados.0.smithi157.stdout:1898: expect (ObjNum 365 snap 0 seq_num 365) 2024-09-16T03:39:28.614 INFO:tasks.rados.rados.0.smithi157.stdout:1899: write oid 49 current snap is 0 2024-09-16T03:39:28.614 INFO:tasks.rados.rados.0.smithi157.stdout:1899: seq_num 421 ranges {675747=479244,1835978=536147} 2024-09-16T03:39:28.620 INFO:tasks.rados.rados.0.smithi157.stdout:1899: writing smithi15736696-49 from 675747 to 1154991 tid 1 2024-09-16T03:39:28.626 INFO:tasks.rados.rados.0.smithi157.stdout:1899: writing smithi15736696-49 from 1835978 to 2372125 tid 2 2024-09-16T03:39:28.626 INFO:tasks.rados.rados.0.smithi157.stdout:1900: write (excl) oid 35 current snap is 0 2024-09-16T03:39:28.626 INFO:tasks.rados.rados.0.smithi157.stdout:1894: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-16T03:39:28.640 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:28.639+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:28.643 INFO:teuthology.orchestra.run.smithi064.stderr:marked in osd.1. 2024-09-16T03:39:28.644 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:39:28.663 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:28.665 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-16T03:39:28.699 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:28.702 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:28.703 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:28.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:39:28.736 INFO:tasks.rados.rados.0.smithi157.stdout:1895: finishing write tid 1 to smithi15736696-36 2024-09-16T03:39:28.736 INFO:tasks.rados.rados.0.smithi157.stdout:1897: finishing set_chunk to oid 9 2024-09-16T03:39:28.736 INFO:tasks.rados.rados.0.smithi157.stdout:1897: got expected ENOENT (src dne) 2024-09-16T03:39:28.736 INFO:tasks.rados.rados.0.smithi157.stdout:1895: finishing write tid 2 to smithi15736696-36 2024-09-16T03:39:28.736 INFO:tasks.rados.rados.0.smithi157.stdout:1895: finishing write tid 3 to smithi15736696-36 2024-09-16T03:39:28.736 INFO:tasks.rados.rados.0.smithi157.stdout:1895: finishing write tid 4 to smithi15736696-36 2024-09-16T03:39:28.736 INFO:tasks.rados.rados.0.smithi157.stdout:1895: finishing write tid 5 to smithi15736696-36 2024-09-16T03:39:28.736 INFO:tasks.rados.rados.0.smithi157.stdout:1895: oid 36 updating version 0 to 134 2024-09-16T03:39:28.736 INFO:tasks.rados.rados.0.smithi157.stdout:1895: oid 36 version 134 is already newer than 131 2024-09-16T03:39:28.737 INFO:tasks.rados.rados.0.smithi157.stdout:1895: oid 36 version 134 is already newer than 133 2024-09-16T03:39:28.737 INFO:tasks.rados.rados.0.smithi157.stdout:1895: oid 36 version 134 is already newer than 132 2024-09-16T03:39:28.737 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 36 v 134 (ObjNum 420 snap 0 seq_num 420) dirty exists 2024-09-16T03:39:28.737 INFO:tasks.rados.rados.0.smithi157.stdout:1895: left oid 36 (ObjNum 420 snap 0 seq_num 420) 2024-09-16T03:39:28.737 INFO:tasks.rados.rados.0.smithi157.stdout:1900: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-09-16T03:39:28.741 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:28.739+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:28.742 INFO:tasks.rados.rados.0.smithi157.stdout:1900: writing smithi15736696-35 from 692554 to 1247047 tid 1 2024-09-16T03:39:28.750 INFO:tasks.rados.rados.0.smithi157.stdout:1900: writing smithi15736696-35 from 1778107 to 2457912 tid 2 2024-09-16T03:39:28.753 INFO:tasks.rados.rados.0.smithi157.stdout:1900: writing smithi15736696-35 from 2889444 to 3092554 tid 3 2024-09-16T03:39:28.754 INFO:tasks.rados.rados.0.smithi157.stdout:1894: done (6 left) 2024-09-16T03:39:28.754 INFO:tasks.rados.rados.0.smithi157.stdout:1895: done (5 left) 2024-09-16T03:39:28.754 INFO:tasks.rados.rados.0.smithi157.stdout:1897: done (4 left) 2024-09-16T03:39:28.754 INFO:tasks.rados.rados.0.smithi157.stdout:1901: tier_flush oid 15 2024-09-16T03:39:28.754 INFO:tasks.rados.rados.0.smithi157.stdout:1901: tier_flush oid 15 snap -1 2024-09-16T03:39:28.754 INFO:tasks.rados.rados.0.smithi157.stdout:1902: set_chunk oid 41 target oid 2024-09-16T03:39:28.754 INFO:tasks.rados.rados.0.smithi157.stdout:1902: set_chunk oid 41 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:39:28.754 INFO:tasks.rados.rados.0.smithi157.stdout:1903: read oid 9 snap -1 2024-09-16T03:39:28.755 INFO:tasks.rados.rados.0.smithi157.stdout:1903: expect deleted 2024-09-16T03:39:28.755 INFO:tasks.rados.rados.0.smithi157.stdout:1904: set_chunk oid 29 target oid 2024-09-16T03:39:28.842 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:28.839+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:28.860 INFO:tasks.rados.rados.0.smithi157.stdout:1904: set_chunk oid 29 offset: 233984 length: 511488 target oid c4da79229a302ffbba551ac2caeaf38636a4f77c5177619fa2bcce33192cc43d offset: 0 snap -1 2024-09-16T03:39:28.860 INFO:tasks.rados.rados.0.smithi157.stdout:1905: read oid 43 snap -1 2024-09-16T03:39:28.861 INFO:tasks.rados.rados.0.smithi157.stdout:1905: expect (ObjNum 418 snap 0 seq_num 418) 2024-09-16T03:39:28.861 INFO:tasks.rados.rados.0.smithi157.stdout:1906: write oid 32 current snap is 0 2024-09-16T03:39:28.861 INFO:tasks.rados.rados.0.smithi157.stdout:1902: finishing set_chunk to oid 41 2024-09-16T03:39:28.861 INFO:tasks.rados.rados.0.smithi157.stdout:1902: got expected ENOENT (src dne) 2024-09-16T03:39:28.861 INFO:tasks.rados.rados.0.smithi157.stdout:1906: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2024-09-16T03:39:28.862 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:28.862 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:28.866 INFO:tasks.rados.rados.0.smithi157.stdout:1906: writing smithi15736696-32 from 709361 to 1339103 tid 1 2024-09-16T03:39:28.871 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:39:28.872 INFO:tasks.rados.rados.0.smithi157.stdout:1906: writing smithi15736696-32 from 1836589 to 2376405 tid 2 2024-09-16T03:39:28.875 INFO:tasks.rados.rados.0.smithi157.stdout:1906: writing smithi15736696-32 from 3033220 to 3109361 tid 3 2024-09-16T03:39:28.876 INFO:tasks.rados.rados.0.smithi157.stdout:1902: done (9 left) 2024-09-16T03:39:28.876 INFO:tasks.rados.rados.0.smithi157.stdout:1907: read oid 34 snap -1 2024-09-16T03:39:28.876 INFO:tasks.rados.rados.0.smithi157.stdout:1907: expect (ObjNum 342 snap 0 seq_num 342) 2024-09-16T03:39:28.876 INFO:tasks.rados.rados.0.smithi157.stdout:1908: set_chunk oid 24 target oid 2024-09-16T03:39:28.943 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:28.941+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:29.020 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:29.020 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:29.029 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:39:29.072 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:29.070+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:29.128 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:39:29.138 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:29.138 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:29.145 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:29.143+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:29.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:39:29.246 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:29.244+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:29.255 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:29.256 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:29.273 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:29.273 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:29.291 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:29.291 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:29.298 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:29.298 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:29.307 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:39:29.308 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:29.309 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:29.324 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:29.324 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:29.342 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:29.342 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:29.346 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:29.345+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:29.359 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:29.359 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:29.376 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:29.376 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:29.393 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:29.393 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:29.410 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:29.410 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:29.417 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:29.417 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:29.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:39:29.428 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:29.428 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:29.445 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:29.445 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:29.448 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:29.446+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:29.461 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:29.462 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:29.495 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:29.495 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:29.512 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:29.512 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:29.529 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:29.529 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:29.549 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:29.550 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:29.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:29.550 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:29.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.550 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:432 4.c 4:34492f38:::smithi15736696-33:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:29.550 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:39:29.550 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 128.63302014800001, 2024-09-16T03:39:29.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.404460861, 2024-09-16T03:39:29.550 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.550 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.550 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:29.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.550 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 432 2024-09-16T03:39:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:39:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913832+0000", 2024-09-16T03:39:29.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913833+0000", 2024-09-16T03:39:29.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0610000000000001e-06 2024-09-16T03:39:29.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913839+0000", 2024-09-16T03:39:29.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9479999999999996e-06 2024-09-16T03:39:29.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913848+0000", 2024-09-16T03:39:29.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5629999999999992e-06 2024-09-16T03:39:29.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.913857+0000", 2024-09-16T03:39:29.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0399999999999998e-06 2024-09-16T03:39:29.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914022+0000", 2024-09-16T03:39:29.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000165422 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:20.914101+0000", 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8462999999999997e-05 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217171+0000", 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30307040000000002 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:29.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.217854+0000", 2024-09-16T03:39:29.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00068278700000000002 2024-09-16T03:39:29.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.218098+0000", 2024-09-16T03:39:29.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243825 2024-09-16T03:39:29.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318164+0000", 2024-09-16T03:39:29.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100066451 2024-09-16T03:39:29.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318232+0000", 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8113000000000003e-05 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.318293+0000", 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0788000000000001e-05 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.556 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:790 4.d 4:b3fe05b2:::smithi15736696-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:29.557 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:39:29.557 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 123.658880074, 2024-09-16T03:39:29.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41456747799999999, 2024-09-16T03:39:29.557 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.557 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.557 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:29.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.557 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 790 2024-09-16T03:39:29.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.557 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:39:29.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:39:29.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887975+0000", 2024-09-16T03:39:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0369999999999999e-06 2024-09-16T03:39:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887987+0000", 2024-09-16T03:39:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1606e-05 2024-09-16T03:39:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888016+0000", 2024-09-16T03:39:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.917e-05 2024-09-16T03:39:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888032+0000", 2024-09-16T03:39:29.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5616000000000001e-05 2024-09-16T03:39:29.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888178+0000", 2024-09-16T03:39:29.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612100000000001 2024-09-16T03:39:29.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888307+0000", 2024-09-16T03:39:29.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012953899999999999 2024-09-16T03:39:29.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.302540+0000", 2024-09-16T03:39:29.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41423238899999998 2024-09-16T03:39:29.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.561 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.561 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1418 4.c 4:3313e97c:::smithi15736696-21:head [write 615156~555834] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:29.561 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:39:29.561 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.320768902, 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40891108799999998, 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1418 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:39:29.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:39:29.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226087+0000", 2024-09-16T03:39:29.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8890000000000004e-06 2024-09-16T03:39:29.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226827+0000", 2024-09-16T03:39:29.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000739298 2024-09-16T03:39:29.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226848+0000", 2024-09-16T03:39:29.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1418000000000001e-05 2024-09-16T03:39:29.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226858+0000", 2024-09-16T03:39:29.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0434e-05 2024-09-16T03:39:29.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327036+0000", 2024-09-16T03:39:29.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10017769 2024-09-16T03:39:29.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327098+0000", 2024-09-16T03:39:29.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1760999999999994e-05 2024-09-16T03:39:29.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327230+0000", 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131839 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434369+0000", 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107139788 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434444+0000", 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4171999999999994e-05 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:29.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634830+0000", 2024-09-16T03:39:29.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20038660599999999 2024-09-16T03:39:29.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:29.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634937+0000", 2024-09-16T03:39:29.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010716499999999999 2024-09-16T03:39:29.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634994+0000", 2024-09-16T03:39:29.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7027999999999997e-05 2024-09-16T03:39:29.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.568 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.568 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1419 4.c 4:3313e97c:::smithi15736696-21:head [write 1603467~416361] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:29.568 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:39:29.568 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.316103513, 2024-09-16T03:39:29.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40430432100000002, 2024-09-16T03:39:29.568 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.568 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.568 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1419 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230749+0000", 2024-09-16T03:39:29.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.230752+0000", 2024-09-16T03:39:29.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.354e-06 2024-09-16T03:39:29.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231164+0000", 2024-09-16T03:39:29.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041208599999999998 2024-09-16T03:39:29.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231179+0000", 2024-09-16T03:39:29.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4885999999999999e-05 2024-09-16T03:39:29.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.231186+0000", 2024-09-16T03:39:29.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8759999999999999e-06 2024-09-16T03:39:29.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328726+0000", 2024-09-16T03:39:29.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097539855999999994 2024-09-16T03:39:29.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328786+0000", 2024-09-16T03:39:29.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0059000000000002e-05 2024-09-16T03:39:29.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:29.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.328890+0000", 2024-09-16T03:39:29.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104338 2024-09-16T03:39:29.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434475+0000", 2024-09-16T03:39:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.105584709 2024-09-16T03:39:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.534594+0000", 2024-09-16T03:39:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10011914099999999 2024-09-16T03:39:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635003+0000", 2024-09-16T03:39:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100408834 2024-09-16T03:39:29.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635029+0000", 2024-09-16T03:39:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6551999999999999e-05 2024-09-16T03:39:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635053+0000", 2024-09-16T03:39:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3629999999999999e-05 2024-09-16T03:39:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1421 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.307420753, 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.497492088, 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1421 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.575 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:39:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:39:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239432+0000", 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5999999999999991e-07 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239436+0000", 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239443+0000", 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.968e-06 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239447+0000", 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6969999999999999e-06 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635225+0000", 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39577807399999998 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635290+0000", 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4583000000000002e-05 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635410+0000", 2024-09-16T03:39:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120684 2024-09-16T03:39:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636180+0000", 2024-09-16T03:39:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076997899999999995 2024-09-16T03:39:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636799+0000", 2024-09-16T03:39:29.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061906600000000004 2024-09-16T03:39:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736742+0000", 2024-09-16T03:39:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099943041999999996 2024-09-16T03:39:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736843+0000", 2024-09-16T03:39:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010076799999999999 2024-09-16T03:39:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736924+0000", 2024-09-16T03:39:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0419000000000004e-05 2024-09-16T03:39:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1422 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.307399473, 2024-09-16T03:39:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49764818700000002, 2024-09-16T03:39:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1422 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:39:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239455+0000", 2024-09-16T03:39:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9860000000000001e-06 2024-09-16T03:39:29.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239461+0000", 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239464+0000", 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2619999999999999e-06 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635736+0000", 2024-09-16T03:39:29.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.396272287 2024-09-16T03:39:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635790+0000", 2024-09-16T03:39:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4252000000000002e-05 2024-09-16T03:39:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736964+0000", 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101173319 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737024+0000", 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9975000000000001e-05 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737101+0000", 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7303000000000002e-05 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.586 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2336 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e82)", 2024-09-16T03:39:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:39:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.451812355000001, 2024-09-16T03:39:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.407180869, 2024-09-16T03:39:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.587 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2336 2024-09-16T03:39:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:39:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:39:29.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095040+0000", 2024-09-16T03:39:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2399999999999999e-07 2024-09-16T03:39:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095042+0000", 2024-09-16T03:39:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.567e-06 2024-09-16T03:39:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.095046+0000", 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.639e-06 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195114+0000", 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10006748 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195173+0000", 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8727000000000002e-05 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.195278+0000", 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010522900000000001 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.501699+0000", 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30642159299999999 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.501995+0000", 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000295822 2024-09-16T03:39:29.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:12.502221+0000", 2024-09-16T03:39:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022548800000000001 2024-09-16T03:39:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2642 4.c 4:34492f38:::smithi15736696-33:head [write 665610~797860,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:39:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.67066561, 2024-09-16T03:39:29.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40488026700000002, 2024-09-16T03:39:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2642 2024-09-16T03:39:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:39:29.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876187+0000", 2024-09-16T03:39:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.876189+0000", 2024-09-16T03:39:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0549999999999998e-06 2024-09-16T03:39:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877003+0000", 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081467900000000001 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877018+0000", 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.471e-05 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.877025+0000", 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.379e-06 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977237+0000", 2024-09-16T03:39:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100211914 2024-09-16T03:39:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977354+0000", 2024-09-16T03:39:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116416 2024-09-16T03:39:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.977480+0000", 2024-09-16T03:39:29.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012629899999999999 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.984474+0000", 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069943230000000002 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084635+0000", 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100161018 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185809+0000", 2024-09-16T03:39:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10117406700000001 2024-09-16T03:39:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.280666+0000", 2024-09-16T03:39:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094856491000000001 2024-09-16T03:39:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.280857+0000", 2024-09-16T03:39:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019133999999999999 2024-09-16T03:39:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.281067+0000", 2024-09-16T03:39:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000209576 2024-09-16T03:39:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2643 4.c 4:34492f38:::smithi15736696-33:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:39:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.665998724000005, 2024-09-16T03:39:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50004631700000002, 2024-09-16T03:39:29.599 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2643 2024-09-16T03:39:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:39:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:39:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880854+0000", 2024-09-16T03:39:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6599999999999994e-07 2024-09-16T03:39:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881441+0000", 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058613000000000003 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881450+0000", 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6260000000000007e-06 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881455+0000", 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.515e-06 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979409+0000", 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097954645000000007 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979512+0000", 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102701 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979684+0000", 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017176299999999999 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084699+0000", 2024-09-16T03:39:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10501545599999999 2024-09-16T03:39:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084714+0000", 2024-09-16T03:39:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4738e-05 2024-09-16T03:39:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185858+0000", 2024-09-16T03:39:29.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101144074 2024-09-16T03:39:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380720+0000", 2024-09-16T03:39:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.194862072 2024-09-16T03:39:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380783+0000", 2024-09-16T03:39:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3262000000000005e-05 2024-09-16T03:39:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380900+0000", 2024-09-16T03:39:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116369 2024-09-16T03:39:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2644 4.c 4:34492f38:::smithi15736696-33:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:39:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.661322229000007, 2024-09-16T03:39:29.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.495569859, 2024-09-16T03:39:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2644 2024-09-16T03:39:29.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:39:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:39:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885531+0000", 2024-09-16T03:39:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-16T03:39:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885813+0000", 2024-09-16T03:39:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028188399999999999 2024-09-16T03:39:29.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885824+0000", 2024-09-16T03:39:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-16T03:39:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885830+0000", 2024-09-16T03:39:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6289999999999998e-06 2024-09-16T03:39:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983163+0000", 2024-09-16T03:39:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097333127000000005 2024-09-16T03:39:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983273+0000", 2024-09-16T03:39:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109868 2024-09-16T03:39:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983458+0000", 2024-09-16T03:39:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018554899999999999 2024-09-16T03:39:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084722+0000", 2024-09-16T03:39:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101263244 2024-09-16T03:39:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185717+0000", 2024-09-16T03:39:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100995575 2024-09-16T03:39:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185904+0000", 2024-09-16T03:39:29.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186682 2024-09-16T03:39:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380965+0000", 2024-09-16T03:39:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.195060921 2024-09-16T03:39:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381030+0000", 2024-09-16T03:39:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5174999999999999e-05 2024-09-16T03:39:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381100+0000", 2024-09-16T03:39:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9877999999999997e-05 2024-09-16T03:39:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2645 4.c 4:34492f38:::smithi15736696-33:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:39:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.661012728000003, 2024-09-16T03:39:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59563304800000005, 2024-09-16T03:39:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.614 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2645 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:39:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885840+0000", 2024-09-16T03:39:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-09-16T03:39:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885843+0000", 2024-09-16T03:39:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.785e-06 2024-09-16T03:39:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885848+0000", 2024-09-16T03:39:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3979999999999998e-06 2024-09-16T03:39:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885852+0000", 2024-09-16T03:39:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6509999999999999e-06 2024-09-16T03:39:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084768+0000", 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19891652000000001 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084831+0000", 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2717000000000003e-05 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084964+0000", 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132731 2024-09-16T03:39:29.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185772+0000", 2024-09-16T03:39:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10080797399999999 2024-09-16T03:39:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185790+0000", 2024-09-16T03:39:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7966999999999999e-05 2024-09-16T03:39:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185959+0000", 2024-09-16T03:39:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168777 2024-09-16T03:39:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481235+0000", 2024-09-16T03:39:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29527644800000002 2024-09-16T03:39:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481307+0000", 2024-09-16T03:39:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1539999999999996e-05 2024-09-16T03:39:29.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481473+0000", 2024-09-16T03:39:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016595600000000001 2024-09-16T03:39:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2646 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:39:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.660996038999997, 2024-09-16T03:39:29.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59596245699999995, 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2646 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885857+0000", 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885860+0000", 2024-09-16T03:39:29.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8940000000000003e-06 2024-09-16T03:39:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885864+0000", 2024-09-16T03:39:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.602e-06 2024-09-16T03:39:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885867+0000", 2024-09-16T03:39:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0089999999999999e-06 2024-09-16T03:39:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085196+0000", 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19932923399999999 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085287+0000", 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0480000000000001e-05 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481521+0000", 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39623439799999999 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481613+0000", 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2276999999999998e-05 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481819+0000", 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020521499999999999 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.626 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3554 4.c 4.3e97c8cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 39.096823014000002, 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2069473150000001, 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3554 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:39:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:39:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450032+0000", 2024-09-16T03:39:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9639999999999999e-06 2024-09-16T03:39:29.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450041+0000", 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450059+0000", 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450069+0000", 2024-09-16T03:39:29.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8320000000000001e-06 2024-09-16T03:39:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.467620+0000", 2024-09-16T03:39:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017550676000000001 2024-09-16T03:39:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.467626+0000", 2024-09-16T03:39:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3430000000000001e-06 2024-09-16T03:39:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656941+0000", 2024-09-16T03:39:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1893151820000001 2024-09-16T03:39:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656977+0000", 2024-09-16T03:39:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5491999999999998e-05 2024-09-16T03:39:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.631 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3581 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:39:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.521316200999998, 2024-09-16T03:39:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.63145314900000005, 2024-09-16T03:39:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3581 2024-09-16T03:39:29.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:39:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:39:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025539+0000", 2024-09-16T03:39:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4510000000000001e-06 2024-09-16T03:39:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026168+0000", 2024-09-16T03:39:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062808199999999997 2024-09-16T03:39:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026186+0000", 2024-09-16T03:39:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8694000000000001e-05 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026196+0000", 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8800000000000003e-06 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035407+0000", 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092111020000000005 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:29.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035416+0000", 2024-09-16T03:39:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9870000000000005e-06 2024-09-16T03:39:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.557740+0000", 2024-09-16T03:39:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.522323918 2024-09-16T03:39:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656989+0000", 2024-09-16T03:39:29.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099249034999999999 2024-09-16T03:39:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3582 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:39:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.514074356000002, 2024-09-16T03:39:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62429066099999997, 2024-09-16T03:39:29.637 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3582 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:39:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032779+0000", 2024-09-16T03:39:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2720000000000001e-06 2024-09-16T03:39:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035239+0000", 2024-09-16T03:39:29.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024594180000000001 2024-09-16T03:39:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035261+0000", 2024-09-16T03:39:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2246e-05 2024-09-16T03:39:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035273+0000", 2024-09-16T03:39:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2289999999999999e-05 2024-09-16T03:39:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038105+0000", 2024-09-16T03:39:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028319780000000002 2024-09-16T03:39:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038111+0000", 2024-09-16T03:39:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3099999999999997e-06 2024-09-16T03:39:29.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657045+0000", 2024-09-16T03:39:29.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61893337199999998 2024-09-16T03:39:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657069+0000", 2024-09-16T03:39:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3774999999999999e-05 2024-09-16T03:39:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3583 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:29.643 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:39:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.508966225999998, 2024-09-16T03:39:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61919224299999998, 2024-09-16T03:39:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3583 2024-09-16T03:39:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:39:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:39:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037888+0000", 2024-09-16T03:39:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-06 2024-09-16T03:39:29.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038032+0000", 2024-09-16T03:39:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014443799999999999 2024-09-16T03:39:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038045+0000", 2024-09-16T03:39:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3417999999999999e-05 2024-09-16T03:39:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038052+0000", 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3890000000000002e-06 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038133+0000", 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0856e-05 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038136+0000", 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7160000000000001e-06 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657073+0000", 2024-09-16T03:39:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61893632899999995 2024-09-16T03:39:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657078+0000", 2024-09-16T03:39:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4770000000000002e-06 2024-09-16T03:39:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.648 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3584 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:39:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.508787931999997, 2024-09-16T03:39:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61903161699999998, 2024-09-16T03:39:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3584 2024-09-16T03:39:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:39:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:39:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038065+0000", 2024-09-16T03:39:29.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3499999999999996e-07 2024-09-16T03:39:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038069+0000", 2024-09-16T03:39:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:39:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038075+0000", 2024-09-16T03:39:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0549999999999996e-06 2024-09-16T03:39:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038078+0000", 2024-09-16T03:39:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4939999999999999e-06 2024-09-16T03:39:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.126379+0000", 2024-09-16T03:39:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088300141999999998 2024-09-16T03:39:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:29.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.126393+0000", 2024-09-16T03:39:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4827999999999999e-05 2024-09-16T03:39:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657092+0000", 2024-09-16T03:39:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.53069909500000001 2024-09-16T03:39:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657096+0000", 2024-09-16T03:39:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4139999999999998e-06 2024-09-16T03:39:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.653 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3585 4.2c 4.1cf4922c (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.508767372000001, 2024-09-16T03:39:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61902921799999999, 2024-09-16T03:39:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:29.654 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3585 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:29.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1899999999999998e-07 2024-09-16T03:39:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038087+0000", 2024-09-16T03:39:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-16T03:39:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038091+0000", 2024-09-16T03:39:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1590000000000003e-06 2024-09-16T03:39:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038095+0000", 2024-09-16T03:39:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5779999999999999e-06 2024-09-16T03:39:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.138249+0000", 2024-09-16T03:39:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100154101 2024-09-16T03:39:29.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.138256+0000", 2024-09-16T03:39:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4209999999999998e-06 2024-09-16T03:39:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657111+0000", 2024-09-16T03:39:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.51885431800000004 2024-09-16T03:39:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657114+0000", 2024-09-16T03:39:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5190000000000001e-06 2024-09-16T03:39:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3983 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 2151362] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:39:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.254858607999999, 2024-09-16T03:39:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49254613699999999, 2024-09-16T03:39:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.659 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3983 2024-09-16T03:39:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:39:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9500000000000002e-07 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291997+0000", 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9679999999999998e-06 2024-09-16T03:39:29.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292002+0000", 2024-09-16T03:39:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7700000000000001e-06 2024-09-16T03:39:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292005+0000", 2024-09-16T03:39:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3019999999999999e-06 2024-09-16T03:39:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.382133+0000", 2024-09-16T03:39:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.090127600000000002 2024-09-16T03:39:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.382184+0000", 2024-09-16T03:39:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1304999999999998e-05 2024-09-16T03:39:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 11,15", 2024-09-16T03:39:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.382301+0000", 2024-09-16T03:39:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116673 2024-09-16T03:39:29.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.383337+0000", 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00103619 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.583745+0000", 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20040787600000001 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784456+0000", 2024-09-16T03:39:29.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20071103700000001 2024-09-16T03:39:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784496+0000", 2024-09-16T03:39:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9740999999999998e-05 2024-09-16T03:39:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784540+0000", 2024-09-16T03:39:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4079999999999998e-05 2024-09-16T03:39:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.665 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3984 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:39:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.254842427, 2024-09-16T03:39:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49270909200000002, 2024-09-16T03:39:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:29.666 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3984 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:39:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4900000000000001e-07 2024-09-16T03:39:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292012+0000", 2024-09-16T03:39:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9250000000000002e-06 2024-09-16T03:39:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292015+0000", 2024-09-16T03:39:29.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4199999999999999e-06 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292018+0000", 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.994e-06 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.388062+0000", 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096043327999999997 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:29.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.388110+0000", 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8272999999999997e-05 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784615+0000", 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39650470300000001 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784664+0000", 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8841000000000002e-05 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:29.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784719+0000", 2024-09-16T03:39:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5259000000000001e-05 2024-09-16T03:39:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:29.671 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:29.671 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:29.671 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:29.546+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:29.672 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:39:29.674 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:29.644+0000 7eff7a84f640 -1 osd.3 149 handle_osd_map injecting map crc failure 2024-09-16T03:39:29.674 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:29.641+0000 7fa38a6ea640 -1 osd.10 149 handle_osd_map injecting map crc failure 2024-09-16T03:39:29.674 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:29.645+0000 7ff266366640 -1 osd.11 149 handle_osd_map injecting map crc failure 2024-09-16T03:39:29.675 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:29.643+0000 7f3a21722640 -1 osd.6 149 handle_osd_map injecting map crc failure 2024-09-16T03:39:29.675 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:29.672+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:29.774 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:29.772+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:29.788 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:29.789 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:29.789 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:29.789 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:29.797 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:39:29.875 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:29.873+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:29.900 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:29.900 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:29.900 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:29.900 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:29.901 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:29.909 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:39:29.976 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:29.974+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:30.052 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:30.053 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:30.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:30.053 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:30.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.053 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:21 4.8 e39/32 4:146f84f0:::smithi15736696-6:head v 39'4, mlcod=0'0)", 2024-09-16T03:39:30.053 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:39:30.053 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 137.863003685, 2024-09-16T03:39:30.053 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10231100799999999, 2024-09-16T03:39:30.053 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.053 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.053 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.054 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:39:30.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.054 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187298+0000", 2024-09-16T03:39:30.054 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.054 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187300+0000", 2024-09-16T03:39:30.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0360000000000001e-06 2024-09-16T03:39:30.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187905+0000", 2024-09-16T03:39:30.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060508100000000004 2024-09-16T03:39:30.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.055 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.055 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187920+0000", 2024-09-16T03:39:30.055 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4314000000000001e-05 2024-09-16T03:39:30.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.056 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.056 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187928+0000", 2024-09-16T03:39:30.056 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7150000000000004e-06 2024-09-16T03:39:30.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.057 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.057 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.187984+0000", 2024-09-16T03:39:30.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5872999999999999e-05 2024-09-16T03:39:30.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.188016+0000", 2024-09-16T03:39:30.058 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2045000000000002e-05 2024-09-16T03:39:30.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.058 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.058 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289567+0000", 2024-09-16T03:39:30.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101550875 2024-09-16T03:39:30.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.059 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.059 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.289609+0000", 2024-09-16T03:39:30.059 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2069e-05 2024-09-16T03:39:30.059 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.059 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.059 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.059 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.21:5 3.0 e45/44 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'39, mlcod=45'37)", 2024-09-16T03:39:30.060 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:39:30.060 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 129.13570665099999, 2024-09-16T03:39:30.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20119541799999999, 2024-09-16T03:39:30.060 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.060 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.060 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.060 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.060 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.060 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:39:30.060 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:39:30.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914597+0000", 2024-09-16T03:39:30.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.494e-06 2024-09-16T03:39:30.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.061 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.061 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914603+0000", 2024-09-16T03:39:30.061 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7349999999999999e-06 2024-09-16T03:39:30.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914614+0000", 2024-09-16T03:39:30.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0995999999999999e-05 2024-09-16T03:39:30.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.062 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.062 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914623+0000", 2024-09-16T03:39:30.062 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3899999999999993e-06 2024-09-16T03:39:30.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.114941+0000", 2024-09-16T03:39:30.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20031812399999999 2024-09-16T03:39:30.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.063 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.063 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115001+0000", 2024-09-16T03:39:30.063 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0161999999999998e-05 2024-09-16T03:39:30.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115750+0000", 2024-09-16T03:39:30.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074851699999999998 2024-09-16T03:39:30.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.064 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.064 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115791+0000", 2024-09-16T03:39:30.064 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1e-05 2024-09-16T03:39:30.064 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.064 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.064 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.065 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.5.21:36 3.0 e52/44 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 52'62, mlcod=51'60)", 2024-09-16T03:39:30.065 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:39:30.065 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 118.165675992, 2024-09-16T03:39:30.065 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20104840199999999, 2024-09-16T03:39:30.065 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.066 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.066 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.066 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.066 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.066 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:39:30.066 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.066 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884627+0000", 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.322e-06 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884634+0000", 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:39:30.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884643+0000", 2024-09-16T03:39:30.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7680000000000003e-06 2024-09-16T03:39:30.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.068 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.068 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984740+0000", 2024-09-16T03:39:30.068 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100096066 2024-09-16T03:39:30.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084931+0000", 2024-09-16T03:39:30.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100191408 2024-09-16T03:39:30.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084952+0000", 2024-09-16T03:39:30.069 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.128e-05 2024-09-16T03:39:30.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.069 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.069 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085645+0000", 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00069309500000000002 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085674+0000", 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9025000000000001e-05 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1537 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'35, mlcod=58'33)", 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:39:30.070 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.329922698, 2024-09-16T03:39:30.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102969106, 2024-09-16T03:39:30.071 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.071 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.071 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.071 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:39:30.071 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.071 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.071 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720379+0000", 2024-09-16T03:39:30.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.720381+0000", 2024-09-16T03:39:30.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.767e-06 2024-09-16T03:39:30.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.072 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.072 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721134+0000", 2024-09-16T03:39:30.072 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00075281999999999996 2024-09-16T03:39:30.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721148+0000", 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4534e-05 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721160+0000", 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1595e-05 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721256+0000", 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6136000000000002e-05 2024-09-16T03:39:30.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721274+0000", 2024-09-16T03:39:30.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8006e-05 2024-09-16T03:39:30.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.074 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.074 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823312+0000", 2024-09-16T03:39:30.074 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102038064 2024-09-16T03:39:30.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.075 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.075 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823348+0000", 2024-09-16T03:39:30.075 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6183999999999998e-05 2024-09-16T03:39:30.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.075 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.076 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1538 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'36, mlcod=58'33)", 2024-09-16T03:39:30.076 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:39:30.076 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.328768361, 2024-09-16T03:39:30.076 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102397932, 2024-09-16T03:39:30.076 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.077 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.077 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.077 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.077 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:39:30.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.077 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.077 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:39:30.077 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.721534+0000", 2024-09-16T03:39:30.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5199999999999996e-07 2024-09-16T03:39:30.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722162+0000", 2024-09-16T03:39:30.078 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062758599999999996 2024-09-16T03:39:30.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.078 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.078 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722171+0000", 2024-09-16T03:39:30.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7139999999999999e-06 2024-09-16T03:39:30.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722177+0000", 2024-09-16T03:39:30.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.866e-06 2024-09-16T03:39:30.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.079 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.079 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822283+0000", 2024-09-16T03:39:30.079 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10010508899999999 2024-09-16T03:39:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822298+0000", 2024-09-16T03:39:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5497000000000001e-05 2024-09-16T03:39:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823907+0000", 2024-09-16T03:39:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001609166 2024-09-16T03:39:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.081 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.081 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823931+0000", 2024-09-16T03:39:30.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4261999999999999e-05 2024-09-16T03:39:30.081 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.081 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.081 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.081 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1539 4.c e60/32 4:34492f38:::smithi15736696-33:head v 60'37, mlcod=58'33)", 2024-09-16T03:39:30.081 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:39:30.081 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.3275683, 2024-09-16T03:39:30.081 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103545683, 2024-09-16T03:39:30.081 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.082 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.082 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.082 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.082 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:39:30.082 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.082 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722734+0000", 2024-09-16T03:39:30.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.083 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722735+0000", 2024-09-16T03:39:30.083 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8400000000000002e-07 2024-09-16T03:39:30.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.083 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722740+0000", 2024-09-16T03:39:30.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3129999999999998e-06 2024-09-16T03:39:30.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.722747+0000", 2024-09-16T03:39:30.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0600000000000002e-06 2024-09-16T03:39:30.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.084 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.084 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.822841+0000", 2024-09-16T03:39:30.084 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100094477 2024-09-16T03:39:30.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823376+0000", 2024-09-16T03:39:30.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00053497199999999999 2024-09-16T03:39:30.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.085 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.085 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.823399+0000", 2024-09-16T03:39:30.085 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2680999999999998e-05 2024-09-16T03:39:30.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.086 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.826259+0000", 2024-09-16T03:39:30.086 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028598500000000002 2024-09-16T03:39:30.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.086 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.087 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:45.826279+0000", 2024-09-16T03:39:30.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0346000000000001e-05 2024-09-16T03:39:30.087 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.087 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.087 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.087 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:39:30.087 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:39:30.087 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 99.900377395000007, 2024-09-16T03:39:30.087 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20169899299999999, 2024-09-16T03:39:30.087 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.088 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.088 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.088 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.088 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.088 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:39:30.088 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:39:30.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.089 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149927+0000", 2024-09-16T03:39:30.089 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9299999999999999e-06 2024-09-16T03:39:30.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.089 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.089 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.090 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151090+0000", 2024-09-16T03:39:30.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011624459999999999 2024-09-16T03:39:30.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.090 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151105+0000", 2024-09-16T03:39:30.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5238999999999999e-05 2024-09-16T03:39:30.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.090 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.090 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251193+0000", 2024-09-16T03:39:30.090 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100088025 2024-09-16T03:39:30.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251556+0000", 2024-09-16T03:39:30.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036245599999999999 2024-09-16T03:39:30.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.091 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251578+0000", 2024-09-16T03:39:30.091 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2651e-05 2024-09-16T03:39:30.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.091 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351582+0000", 2024-09-16T03:39:30.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100003811 2024-09-16T03:39:30.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.092 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.092 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351624+0000", 2024-09-16T03:39:30.092 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1434999999999999e-05 2024-09-16T03:39:30.092 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.092 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.092 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.092 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:90 3.0 e70/69 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head v 70'103, mlcod=0'0)", 2024-09-16T03:39:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:39:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 94.274539111999999, 2024-09-16T03:39:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102185333, 2024-09-16T03:39:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:39:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:39:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775763+0000", 2024-09-16T03:39:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.13e-07 2024-09-16T03:39:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775769+0000", 2024-09-16T03:39:30.094 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7379999999999999e-06 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775782+0000", 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2989e-05 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775791+0000", 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5369999999999997e-06 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.095 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876295+0000", 2024-09-16T03:39:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100504817 2024-09-16T03:39:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876316+0000", 2024-09-16T03:39:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0409999999999999e-05 2024-09-16T03:39:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877916+0000", 2024-09-16T03:39:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016003790000000001 2024-09-16T03:39:30.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877948+0000", 2024-09-16T03:39:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1949999999999997e-05 2024-09-16T03:39:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1742 3.0 e70/69 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head v 70'104, mlcod=0'0)", 2024-09-16T03:39:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 94.274502416000004, 2024-09-16T03:39:30.097 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20271830699999999, 2024-09-16T03:39:30.098 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.098 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.098 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.098 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.098 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.098 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:30.098 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777455+0000", 2024-09-16T03:39:30.099 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001655524 2024-09-16T03:39:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777468+0000", 2024-09-16T03:39:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2486999999999999e-05 2024-09-16T03:39:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777474+0000", 2024-09-16T03:39:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:39:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.100 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876577+0000", 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099103360000000001 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876594+0000", 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6449000000000001e-05 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978469+0000", 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101875521 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978518+0000", 2024-09-16T03:39:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8488000000000001e-05 2024-09-16T03:39:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:91 3.0 e70/69 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head v 70'105, mlcod=0'0)", 2024-09-16T03:39:30.102 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:39:30.150 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 94.188290546999994, 2024-09-16T03:39:30.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.116532025, 2024-09-16T03:39:30.150 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.150 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.150 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.150 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:39:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:39:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862015+0000", 2024-09-16T03:39:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3569999999999998e-06 2024-09-16T03:39:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862029+0000", 2024-09-16T03:39:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4082e-05 2024-09-16T03:39:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862055+0000", 2024-09-16T03:39:30.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6570000000000001e-05 2024-09-16T03:39:30.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862070+0000", 2024-09-16T03:39:30.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4473e-05 2024-09-16T03:39:30.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877631+0000", 2024-09-16T03:39:30.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.015560744 2024-09-16T03:39:30.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877674+0000", 2024-09-16T03:39:30.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3615999999999997e-05 2024-09-16T03:39:30.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978525+0000", 2024-09-16T03:39:30.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100850696 2024-09-16T03:39:30.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978543+0000", 2024-09-16T03:39:30.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8487000000000001e-05 2024-09-16T03:39:30.154 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.155 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.155 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.155 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:94 3.0 e70/69 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head v 70'106, mlcod=0'0)", 2024-09-16T03:39:30.155 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:39:30.155 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 94.187754327999997, 2024-09-16T03:39:30.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11627603, 2024-09-16T03:39:30.155 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.155 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.155 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.156 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:39:30.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:39:30.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862549+0000", 2024-09-16T03:39:30.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1650000000000001e-06 2024-09-16T03:39:30.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862556+0000", 2024-09-16T03:39:30.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8709999999999999e-06 2024-09-16T03:39:30.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862566+0000", 2024-09-16T03:39:30.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0271e-05 2024-09-16T03:39:30.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862573+0000", 2024-09-16T03:39:30.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.861e-06 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978123+0000", 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11554993199999999 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978163+0000", 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0201999999999999e-05 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978796+0000", 2024-09-16T03:39:30.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063316700000000002 2024-09-16T03:39:30.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978824+0000", 2024-09-16T03:39:30.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7560999999999999e-05 2024-09-16T03:39:30.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.162 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1947 4.8 e74/68 4:146f84f0:::smithi15736696-6:head v 74'47, mlcod=73'45)", 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.925138189999998, 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103866244, 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:39:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125165+0000", 2024-09-16T03:39:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.513e-06 2024-09-16T03:39:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126855+0000", 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016901010000000001 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126870+0000", 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4487999999999999e-05 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227009+0000", 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100138888 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227117+0000", 2024-09-16T03:39:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107984 2024-09-16T03:39:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227136+0000", 2024-09-16T03:39:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9247999999999999e-05 2024-09-16T03:39:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229002+0000", 2024-09-16T03:39:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001866143 2024-09-16T03:39:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229030+0000", 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7878999999999999e-05 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1979 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'32, mlcod=0'0)", 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.212563642999996, 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20429768500000001, 2024-09-16T03:39:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:39:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:39:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837740+0000", 2024-09-16T03:39:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.779e-06 2024-09-16T03:39:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838356+0000", 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061634500000000002 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838371+0000", 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5007e-05 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838380+0000", 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1009999999999992e-06 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038674+0000", 2024-09-16T03:39:30.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20029450500000001 2024-09-16T03:39:30.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038710+0000", 2024-09-16T03:39:30.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6100999999999999e-05 2024-09-16T03:39:30.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.172 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042001+0000", 2024-09-16T03:39:30.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032909430000000002 2024-09-16T03:39:30.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.172 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042036+0000", 2024-09-16T03:39:30.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4904000000000003e-05 2024-09-16T03:39:30.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.172 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1980 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'33, mlcod=0'0)", 2024-09-16T03:39:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:39:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.210687046999993, 2024-09-16T03:39:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20296795100000001, 2024-09-16T03:39:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:39:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:39:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839616+0000", 2024-09-16T03:39:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3099999999999996e-07 2024-09-16T03:39:30.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840706+0000", 2024-09-16T03:39:30.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010900980000000001 2024-09-16T03:39:30.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840720+0000", 2024-09-16T03:39:30.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.378e-05 2024-09-16T03:39:30.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840727+0000", 2024-09-16T03:39:30.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.525e-06 2024-09-16T03:39:30.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039940+0000", 2024-09-16T03:39:30.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19921313199999999 2024-09-16T03:39:30.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039967+0000", 2024-09-16T03:39:30.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6393999999999998e-05 2024-09-16T03:39:30.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042552+0000", 2024-09-16T03:39:30.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025850999999999999 2024-09-16T03:39:30.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042583+0000", 2024-09-16T03:39:30.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0991000000000002e-05 2024-09-16T03:39:30.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.177 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.178 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1981 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'34, mlcod=0'0)", 2024-09-16T03:39:30.178 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:39:30.178 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 88.208664691999999, 2024-09-16T03:39:30.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201270175, 2024-09-16T03:39:30.178 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.178 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.178 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.178 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:39:30.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:39:30.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841638+0000", 2024-09-16T03:39:30.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-16T03:39:30.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841644+0000", 2024-09-16T03:39:30.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8050000000000003e-06 2024-09-16T03:39:30.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841653+0000", 2024-09-16T03:39:30.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6580000000000008e-06 2024-09-16T03:39:30.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841660+0000", 2024-09-16T03:39:30.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9290000000000001e-06 2024-09-16T03:39:30.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041673+0000", 2024-09-16T03:39:30.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20001311499999999 2024-09-16T03:39:30.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041696+0000", 2024-09-16T03:39:30.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3365000000000001e-05 2024-09-16T03:39:30.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042878+0000", 2024-09-16T03:39:30.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011814499999999999 2024-09-16T03:39:30.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042907+0000", 2024-09-16T03:39:30.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9655999999999999e-05 2024-09-16T03:39:30.182 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.182 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.182 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.182 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2117 3.0 e76/69 3:3c8eb54b:::593314a3a723c40dec2b25351adb282089b02c0264cb0891ee3f17edde05dfe9:head v 76'138, mlcod=75'136)", 2024-09-16T03:39:30.183 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:39:30.183 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.398257004000001, 2024-09-16T03:39:30.183 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10374235, 2024-09-16T03:39:30.183 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.183 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.183 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.183 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.183 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:39:30.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:39:30.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:39:30.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8099999999999999e-07 2024-09-16T03:39:30.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654214+0000", 2024-09-16T03:39:30.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002169191 2024-09-16T03:39:30.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.185 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654230+0000", 2024-09-16T03:39:30.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6144e-05 2024-09-16T03:39:30.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.185 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654242+0000", 2024-09-16T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1748e-05 2024-09-16T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.754387+0000", 2024-09-16T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10014505899999999 2024-09-16T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.754409+0000", 2024-09-16T03:39:30.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2181999999999999e-05 2024-09-16T03:39:30.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.755736+0000", 2024-09-16T03:39:30.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013261690000000001 2024-09-16T03:39:30.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.755787+0000", 2024-09-16T03:39:30.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1675999999999998e-05 2024-09-16T03:39:30.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.187 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.188 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2278 4.18 e80/68 4:1f63731d:::smithi15736696-30:head v 80'37, mlcod=75'34)", 2024-09-16T03:39:30.188 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:39:30.188 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 79.524846804999996, 2024-09-16T03:39:30.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10326089300000001, 2024-09-16T03:39:30.188 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.188 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.188 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.188 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525455+0000", 2024-09-16T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525456+0000", 2024-09-16T03:39:30.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.248e-06 2024-09-16T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525936+0000", 2024-09-16T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047982899999999998 2024-09-16T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525945+0000", 2024-09-16T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8489999999999995e-06 2024-09-16T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.525955+0000", 2024-09-16T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.043e-05 2024-09-16T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.526339+0000", 2024-09-16T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00038385200000000001 2024-09-16T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.526358+0000", 2024-09-16T03:39:30.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8802000000000001e-05 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.628679+0000", 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102321304 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:10.628716+0000", 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6579000000000003e-05 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2536 3.0 e86/69 3:10a17b1b:::13dc2fab1f287170fabf96291b912bf80e0b081f291f2c6d41d494566f2d1f8c:head v 86'176, mlcod=86'174)", 2024-09-16T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 73.210790313999993, 2024-09-16T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20664157499999999, 2024-09-16T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839515+0000", 2024-09-16T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3330000000000001e-06 2024-09-16T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843392+0000", 2024-09-16T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038769329999999999 2024-09-16T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843412+0000", 2024-09-16T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0608e-05 2024-09-16T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.943546+0000", 2024-09-16T03:39:30.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100133176 2024-09-16T03:39:30.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043824+0000", 2024-09-16T03:39:30.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100278785 2024-09-16T03:39:30.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043869+0000", 2024-09-16T03:39:30.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4845999999999997e-05 2024-09-16T03:39:30.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046096+0000", 2024-09-16T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022262559999999998 2024-09-16T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046153+0000", 2024-09-16T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7637999999999999e-05 2024-09-16T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2654 4.8 e90/68 4:148e627d:::smithi15736696-23:head v 90'58, mlcod=80'56)", 2024-09-16T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 68.675402955999999, 2024-09-16T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.106604212, 2024-09-16T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374901+0000", 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1749999999999999e-06 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375810+0000", 2024-09-16T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00090911400000000004 2024-09-16T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375827+0000", 2024-09-16T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7010000000000001e-05 2024-09-16T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476010+0000", 2024-09-16T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100182879 2024-09-16T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476184+0000", 2024-09-16T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017408899999999999 2024-09-16T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476206+0000", 2024-09-16T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2013000000000001e-05 2024-09-16T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481458+0000", 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052519120000000001 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481503+0000", 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5019999999999999e-05 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "pg_update_log_missing(4.c epoch 99/68 rep_tid 69 entries 99'56 (0'0) error 4:34492f38:::smithi15736696-33:head by client.4628.0:3000 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 90'53 roll_forward_to 90'54)", 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:39:30.202 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 58.130444805000003, 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201155684, 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:39:30.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919858+0000", 2024-09-16T03:39:30.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:39:30.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919871+0000", 2024-09-16T03:39:30.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3433e-05 2024-09-16T03:39:30.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.205 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919929+0000", 2024-09-16T03:39:30.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7543000000000001e-05 2024-09-16T03:39:30.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.205 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919942+0000", 2024-09-16T03:39:30.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3122000000000001e-05 2024-09-16T03:39:30.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.205 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.120302+0000", 2024-09-16T03:39:30.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036052800000001 2024-09-16T03:39:30.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.121013+0000", 2024-09-16T03:39:30.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071046800000000002 2024-09-16T03:39:30.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.206 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.206 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.206 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:30.208 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:39:30.208 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:30.206+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:30.309 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:30.307+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:30.312 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:30.312 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:30.320 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:39:30.356 INFO:tasks.rados.rados.0.smithi157.stdout:1908: set_chunk oid 24 offset: 768512 length: 1316352 target oid a9957f29a7f946349b562c449f43fa1bc2e3c48b00925943f7b23963f3ffcd03 offset: 0 snap -1 2024-09-16T03:39:30.356 INFO:tasks.rados.rados.0.smithi157.stdout:1909: read oid 28 snap -1 2024-09-16T03:39:30.356 INFO:tasks.rados.rados.0.smithi157.stdout:1909: expect (ObjNum 387 snap 0 seq_num 387) 2024-09-16T03:39:30.356 INFO:tasks.rados.rados.0.smithi157.stdout:1910: read oid 12 snap -1 2024-09-16T03:39:30.356 INFO:tasks.rados.rados.0.smithi157.stdout:1910: expect (ObjNum 297 snap 0 seq_num 297) 2024-09-16T03:39:30.356 INFO:tasks.rados.rados.0.smithi157.stdout:1911: read oid 10 snap -1 2024-09-16T03:39:30.357 INFO:tasks.rados.rados.0.smithi157.stdout:1911: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-16T03:39:30.357 INFO:tasks.rados.rados.0.smithi157.stdout:1912: set_chunk oid 8 target oid 2024-09-16T03:39:30.409 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:30.408+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:30.465 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:30.465 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:30.465 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:30.465 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:30.465 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:30.474 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:39:30.518 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:30.516+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:30.570 INFO:tasks.rados.rados.0.smithi157.stdout:1912: set_chunk oid 8 offset: 636928 length: 1108480 target oid 69b727e581e4742a32472ba833d288487f1f9e4a6859d1a7472bc34c11c4d9b5 offset: 0 snap -1 2024-09-16T03:39:30.570 INFO:tasks.rados.rados.0.smithi157.stdout:1913: tier_evict oid 6 2024-09-16T03:39:30.570 INFO:tasks.rados.rados.0.smithi157.stdout:1913: tier_evict oid 6 snap -1 2024-09-16T03:39:30.570 INFO:tasks.rados.rados.0.smithi157.stdout:1898: expect (ObjNum 365 snap 0 seq_num 365) 2024-09-16T03:39:30.570 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:30.579 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:25 4.6 4:6ece7830:::smithi15736696-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:39:30.579 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 138.45634382700001, 2024-09-16T03:39:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40559452699999998, 2024-09-16T03:39:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 25 2024-09-16T03:39:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.580 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:39:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:39:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119830+0000", 2024-09-16T03:39:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9200000000000001e-07 2024-09-16T03:39:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120013+0000", 2024-09-16T03:39:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018367200000000001 2024-09-16T03:39:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120021+0000", 2024-09-16T03:39:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9179999999999994e-06 2024-09-16T03:39:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120027+0000", 2024-09-16T03:39:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6710000000000004e-06 2024-09-16T03:39:30.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220194+0000", 2024-09-16T03:39:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10016689500000001 2024-09-16T03:39:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220399+0000", 2024-09-16T03:39:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020502899999999999 2024-09-16T03:39:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220613+0000", 2024-09-16T03:39:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021383900000000001 2024-09-16T03:39:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224640+0000", 2024-09-16T03:39:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040268099999999996 2024-09-16T03:39:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423348+0000", 2024-09-16T03:39:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19870853399999999 2024-09-16T03:39:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525343+0000", 2024-09-16T03:39:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101995081 2024-09-16T03:39:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525381+0000", 2024-09-16T03:39:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7842000000000001e-05 2024-09-16T03:39:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525424+0000", 2024-09-16T03:39:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2644e-05 2024-09-16T03:39:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:26 4.6 4:6ece7830:::smithi15736696-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 138.45613159999999, 2024-09-16T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550111300000002, 2024-09-16T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.588 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 26 2024-09-16T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120042+0000", 2024-09-16T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7999999999999995e-07 2024-09-16T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.590 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120210+0000", 2024-09-16T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016779499999999999 2024-09-16T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120219+0000", 2024-09-16T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6859999999999995e-06 2024-09-16T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120223+0000", 2024-09-16T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0330000000000002e-06 2024-09-16T03:39:30.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221494+0000", 2024-09-16T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101271812 2024-09-16T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221572+0000", 2024-09-16T03:39:30.592 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7733999999999998e-05 2024-09-16T03:39:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221697+0000", 2024-09-16T03:39:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012474900000000001 2024-09-16T03:39:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224670+0000", 2024-09-16T03:39:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029729589999999998 2024-09-16T03:39:30.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423398+0000", 2024-09-16T03:39:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19872784900000001 2024-09-16T03:39:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525465+0000", 2024-09-16T03:39:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102067557 2024-09-16T03:39:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.594 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525494+0000", 2024-09-16T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9113999999999999e-05 2024-09-16T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525542+0000", 2024-09-16T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8145000000000001e-05 2024-09-16T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:27 4.6 4:6ece7830:::smithi15736696-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 138.455944015, 2024-09-16T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40541507399999999, 2024-09-16T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27 2024-09-16T03:39:30.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120230+0000", 2024-09-16T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5799999999999999e-07 2024-09-16T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120233+0000", 2024-09-16T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.055e-06 2024-09-16T03:39:30.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120237+0000", 2024-09-16T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7099999999999998e-06 2024-09-16T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120244+0000", 2024-09-16T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4810000000000003e-06 2024-09-16T03:39:30.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222660+0000", 2024-09-16T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102416089 2024-09-16T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222705+0000", 2024-09-16T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5368999999999999e-05 2024-09-16T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222782+0000", 2024-09-16T03:39:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6623000000000001e-05 2024-09-16T03:39:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224696+0000", 2024-09-16T03:39:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019143929999999999 2024-09-16T03:39:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.601 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423402+0000", 2024-09-16T03:39:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198705305 2024-09-16T03:39:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525582+0000", 2024-09-16T03:39:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102180178 2024-09-16T03:39:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525615+0000", 2024-09-16T03:39:30.602 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3303000000000003e-05 2024-09-16T03:39:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525644+0000", 2024-09-16T03:39:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8909999999999999e-05 2024-09-16T03:39:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:28 4.6 4:6ece7830:::smithi15736696-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:30.603 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 138.455923267, 2024-09-16T03:39:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550787799999999, 2024-09-16T03:39:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 28 2024-09-16T03:39:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.604 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.605 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7999999999999996e-07 2024-09-16T03:39:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120252+0000", 2024-09-16T03:39:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.249e-06 2024-09-16T03:39:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120257+0000", 2024-09-16T03:39:30.606 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1969999999999998e-06 2024-09-16T03:39:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120261+0000", 2024-09-16T03:39:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1130000000000003e-06 2024-09-16T03:39:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222976+0000", 2024-09-16T03:39:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102715397 2024-09-16T03:39:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223049+0000", 2024-09-16T03:39:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2846e-05 2024-09-16T03:39:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525666+0000", 2024-09-16T03:39:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30261668800000002 2024-09-16T03:39:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.608 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525704+0000", 2024-09-16T03:39:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8025999999999998e-05 2024-09-16T03:39:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525758+0000", 2024-09-16T03:39:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3881999999999997e-05 2024-09-16T03:39:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:49 4.6 4:6df3bbbe:::smithi15736696-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 138.45590820300001, 2024-09-16T03:39:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40556222800000002, 2024-09-16T03:39:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49 2024-09-16T03:39:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.610 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8700000000000002e-07 2024-09-16T03:39:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120454+0000", 2024-09-16T03:39:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018909299999999999 2024-09-16T03:39:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.612 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120465+0000", 2024-09-16T03:39:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.083e-05 2024-09-16T03:39:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120468+0000", 2024-09-16T03:39:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3869999999999999e-06 2024-09-16T03:39:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223082+0000", 2024-09-16T03:39:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102613379 2024-09-16T03:39:30.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223189+0000", 2024-09-16T03:39:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107728 2024-09-16T03:39:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223368+0000", 2024-09-16T03:39:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179011 2024-09-16T03:39:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.614 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.226087+0000", 2024-09-16T03:39:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027183390000000002 2024-09-16T03:39:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423408+0000", 2024-09-16T03:39:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19732106499999999 2024-09-16T03:39:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.615 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525779+0000", 2024-09-16T03:39:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10237133 2024-09-16T03:39:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525805+0000", 2024-09-16T03:39:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5967000000000001e-05 2024-09-16T03:39:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525827+0000", 2024-09-16T03:39:30.616 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1812000000000001e-05 2024-09-16T03:39:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:50 4.6 4:6df3bbbe:::smithi15736696-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:39:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 138.45569989399999, 2024-09-16T03:39:30.617 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50526462999999999, 2024-09-16T03:39:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 50 2024-09-16T03:39:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:39:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:39:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.619 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120474+0000", 2024-09-16T03:39:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4600000000000005e-07 2024-09-16T03:39:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120901+0000", 2024-09-16T03:39:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00042690000000000002 2024-09-16T03:39:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.620 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120907+0000", 2024-09-16T03:39:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7710000000000001e-06 2024-09-16T03:39:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120911+0000", 2024-09-16T03:39:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0500000000000002e-06 2024-09-16T03:39:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224717+0000", 2024-09-16T03:39:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103806175 2024-09-16T03:39:30.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224763+0000", 2024-09-16T03:39:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5629e-05 2024-09-16T03:39:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224838+0000", 2024-09-16T03:39:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4548999999999994e-05 2024-09-16T03:39:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.622 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423390+0000", 2024-09-16T03:39:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19855242400000001 2024-09-16T03:39:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423411+0000", 2024-09-16T03:39:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0888000000000001e-05 2024-09-16T03:39:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625680+0000", 2024-09-16T03:39:30.623 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20226923099999999 2024-09-16T03:39:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625709+0000", 2024-09-16T03:39:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9232000000000001e-05 2024-09-16T03:39:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625738+0000", 2024-09-16T03:39:30.624 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8235e-05 2024-09-16T03:39:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:51 4.6 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 138.45525531199999, 2024-09-16T03:39:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50490003699999997, 2024-09-16T03:39:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.625 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 51 2024-09-16T03:39:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.626 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.627 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8699999999999995e-07 2024-09-16T03:39:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120920+0000", 2024-09-16T03:39:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0820000000000001e-06 2024-09-16T03:39:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.628 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120924+0000", 2024-09-16T03:39:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:39:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120927+0000", 2024-09-16T03:39:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5809999999999999e-06 2024-09-16T03:39:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225783+0000", 2024-09-16T03:39:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10485588699999999 2024-09-16T03:39:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225832+0000", 2024-09-16T03:39:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9750000000000003e-05 2024-09-16T03:39:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225910+0000", 2024-09-16T03:39:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7084999999999998e-05 2024-09-16T03:39:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423414+0000", 2024-09-16T03:39:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19750438200000001 2024-09-16T03:39:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.631 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423456+0000", 2024-09-16T03:39:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2386999999999999e-05 2024-09-16T03:39:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625766+0000", 2024-09-16T03:39:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20231011300000001 2024-09-16T03:39:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.632 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625790+0000", 2024-09-16T03:39:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3672999999999998e-05 2024-09-16T03:39:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625818+0000", 2024-09-16T03:39:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7576000000000002e-05 2024-09-16T03:39:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:52 4.6 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 138.45524311599999, 2024-09-16T03:39:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50502012900000004, 2024-09-16T03:39:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.634 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 52 2024-09-16T03:39:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.635 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:39:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.636 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120932+0000", 2024-09-16T03:39:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.181e-06 2024-09-16T03:39:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120935+0000", 2024-09-16T03:39:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7690000000000001e-06 2024-09-16T03:39:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.637 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120938+0000", 2024-09-16T03:39:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6639999999999998e-06 2024-09-16T03:39:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423188+0000", 2024-09-16T03:39:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30224989200000002 2024-09-16T03:39:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:30.638 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423318+0000", 2024-09-16T03:39:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013045999999999999 2024-09-16T03:39:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625844+0000", 2024-09-16T03:39:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20252547600000001 2024-09-16T03:39:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625895+0000", 2024-09-16T03:39:30.639 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1242000000000002e-05 2024-09-16T03:39:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625950+0000", 2024-09-16T03:39:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5164999999999998e-05 2024-09-16T03:39:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.640 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:279 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:39:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 135.37929132900001, 2024-09-16T03:39:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50446570800000001, 2024-09-16T03:39:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.641 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 279 2024-09-16T03:39:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:39:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:39:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196883+0000", 2024-09-16T03:39:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.48e-06 2024-09-16T03:39:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196889+0000", 2024-09-16T03:39:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6239999999999999e-06 2024-09-16T03:39:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.643 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196902+0000", 2024-09-16T03:39:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2804e-05 2024-09-16T03:39:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196909+0000", 2024-09-16T03:39:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9379999999999992e-06 2024-09-16T03:39:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.644 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196963+0000", 2024-09-16T03:39:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3437999999999999e-05 2024-09-16T03:39:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op must wait for map", 2024-09-16T03:39:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196969+0000", 2024-09-16T03:39:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1709999999999999e-06 2024-09-16T03:39:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302538+0000", 2024-09-16T03:39:30.645 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10556889899999999 2024-09-16T03:39:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302659+0000", 2024-09-16T03:39:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121441 2024-09-16T03:39:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:39:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500040+0000", 2024-09-16T03:39:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19738082900000001 2024-09-16T03:39:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500589+0000", 2024-09-16T03:39:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00054827000000000001 2024-09-16T03:39:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701118+0000", 2024-09-16T03:39:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20052917100000001 2024-09-16T03:39:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701205+0000", 2024-09-16T03:39:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7120999999999995e-05 2024-09-16T03:39:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701289+0000", 2024-09-16T03:39:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4457999999999997e-05 2024-09-16T03:39:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701347+0000", 2024-09-16T03:39:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8063999999999998e-05 2024-09-16T03:39:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:306 4.6 4:6ece7830:::smithi15736696-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:39:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 134.06328901399999, 2024-09-16T03:39:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40927214699999998, 2024-09-16T03:39:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 306 2024-09-16T03:39:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.650 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:39:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:39:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.651 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512885+0000", 2024-09-16T03:39:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.277e-06 2024-09-16T03:39:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512890+0000", 2024-09-16T03:39:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6689999999999997e-06 2024-09-16T03:39:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512900+0000", 2024-09-16T03:39:30.652 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0393e-05 2024-09-16T03:39:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512906+0000", 2024-09-16T03:39:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4569999999999996e-06 2024-09-16T03:39:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512951+0000", 2024-09-16T03:39:30.653 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4891000000000001e-05 2024-09-16T03:39:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.513000+0000", 2024-09-16T03:39:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9187999999999997e-05 2024-09-16T03:39:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.720758+0000", 2024-09-16T03:39:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20775775699999999 2024-09-16T03:39:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.721960+0000", 2024-09-16T03:39:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001202107 2024-09-16T03:39:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922010+0000", 2024-09-16T03:39:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20004993900000001 2024-09-16T03:39:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922055+0000", 2024-09-16T03:39:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4981000000000002e-05 2024-09-16T03:39:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922103+0000", 2024-09-16T03:39:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8097999999999999e-05 2024-09-16T03:39:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.656 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922156+0000", 2024-09-16T03:39:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3390000000000002e-05 2024-09-16T03:39:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:419 4.6 4:64ce87bc:::smithi15736696-20:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:39:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 129.6624894, 2024-09-16T03:39:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40437787200000003, 2024-09-16T03:39:30.657 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 419 2024-09-16T03:39:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:39:30.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:39:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913687+0000", 2024-09-16T03:39:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3299999999999999e-06 2024-09-16T03:39:30.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913700+0000", 2024-09-16T03:39:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2830000000000001e-05 2024-09-16T03:39:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913733+0000", 2024-09-16T03:39:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2815999999999999e-05 2024-09-16T03:39:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913750+0000", 2024-09-16T03:39:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7073000000000001e-05 2024-09-16T03:39:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913920+0000", 2024-09-16T03:39:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000170015 2024-09-16T03:39:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913981+0000", 2024-09-16T03:39:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1631999999999995e-05 2024-09-16T03:39:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216580+0000", 2024-09-16T03:39:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30259888200000001 2024-09-16T03:39:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217205+0000", 2024-09-16T03:39:30.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062461099999999996 2024-09-16T03:39:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317915+0000", 2024-09-16T03:39:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10071053000000001 2024-09-16T03:39:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317996+0000", 2024-09-16T03:39:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0440999999999998e-05 2024-09-16T03:39:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318032+0000", 2024-09-16T03:39:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5944e-05 2024-09-16T03:39:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318061+0000", 2024-09-16T03:39:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9768e-05 2024-09-16T03:39:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.664 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:479 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:39:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 128.85463325399999, 2024-09-16T03:39:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.504011706, 2024-09-16T03:39:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.665 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 479 2024-09-16T03:39:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:39:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:39:30.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.667 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721542+0000", 2024-09-16T03:39:30.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3199999999999998e-06 2024-09-16T03:39:30.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.667 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721553+0000", 2024-09-16T03:39:30.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1413000000000001e-05 2024-09-16T03:39:30.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721570+0000", 2024-09-16T03:39:30.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.685e-05 2024-09-16T03:39:30.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721596+0000", 2024-09-16T03:39:30.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5647999999999999e-05 2024-09-16T03:39:30.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.921950+0000", 2024-09-16T03:39:30.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20035402899999999 2024-09-16T03:39:30.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.922009+0000", 2024-09-16T03:39:30.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9046999999999999e-05 2024-09-16T03:39:30.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:39:30.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.023850+0000", 2024-09-16T03:39:30.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10184130199999999 2024-09-16T03:39:30.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.024503+0000", 2024-09-16T03:39:30.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065257399999999995 2024-09-16T03:39:30.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.125105+0000", 2024-09-16T03:39:30.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100602046 2024-09-16T03:39:30.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225461+0000", 2024-09-16T03:39:30.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100355835 2024-09-16T03:39:30.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225512+0000", 2024-09-16T03:39:30.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0862999999999997e-05 2024-09-16T03:39:30.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.672 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225551+0000", 2024-09-16T03:39:30.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9779e-05 2024-09-16T03:39:30.672 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.672 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.672 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.673 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:996 4.8 4:148e627d:::smithi15736696-23:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:30.673 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:39:30.673 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 119.528830347, 2024-09-16T03:39:30.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.73588397999999999, 2024-09-16T03:39:30.673 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.673 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.673 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:30.673 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.673 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.673 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.674 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 996 2024-09-16T03:39:30.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.674 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:39:30.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:39:30.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047344+0000", 2024-09-16T03:39:30.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.846e-06 2024-09-16T03:39:30.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047351+0000", 2024-09-16T03:39:30.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.973e-06 2024-09-16T03:39:30.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047365+0000", 2024-09-16T03:39:30.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3556e-05 2024-09-16T03:39:30.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047382+0000", 2024-09-16T03:39:30.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6994e-05 2024-09-16T03:39:30.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047453+0000", 2024-09-16T03:39:30.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1458000000000006e-05 2024-09-16T03:39:30.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:30.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047465+0000", 2024-09-16T03:39:30.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1119e-05 2024-09-16T03:39:30.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449411+0000", 2024-09-16T03:39:30.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40194680399999999 2024-09-16T03:39:30.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449535+0000", 2024-09-16T03:39:30.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124043 2024-09-16T03:39:30.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783227+0000", 2024-09-16T03:39:30.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.333691187 2024-09-16T03:39:30.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.679 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.679 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1048 4.16 4:6df3bbbe:::smithi15736696-13:head [write 1622009~726683] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:30.679 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:30.679 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 119.00038623, 2024-09-16T03:39:30.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40682847599999999, 2024-09-16T03:39:30.679 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.679 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.680 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.680 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.680 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.680 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.680 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1048 2024-09-16T03:39:30.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.680 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:30.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:30.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.681 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.681 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:30.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-16T03:39:30.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576941+0000", 2024-09-16T03:39:30.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011532230000000001 2024-09-16T03:39:30.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576949+0000", 2024-09-16T03:39:30.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1089999999999998e-06 2024-09-16T03:39:30.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576956+0000", 2024-09-16T03:39:30.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000001e-06 2024-09-16T03:39:30.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878710+0000", 2024-09-16T03:39:30.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30175434299999998 2024-09-16T03:39:30.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878765+0000", 2024-09-16T03:39:30.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4988000000000003e-05 2024-09-16T03:39:30.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:30.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878852+0000", 2024-09-16T03:39:30.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7201999999999996e-05 2024-09-16T03:39:30.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881535+0000", 2024-09-16T03:39:30.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026828310000000001 2024-09-16T03:39:30.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883968+0000", 2024-09-16T03:39:30.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024330659999999998 2024-09-16T03:39:30.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982541+0000", 2024-09-16T03:39:30.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098572876000000004 2024-09-16T03:39:30.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982580+0000", 2024-09-16T03:39:30.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8714000000000002e-05 2024-09-16T03:39:30.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.686 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982615+0000", 2024-09-16T03:39:30.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5438000000000002e-05 2024-09-16T03:39:30.687 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.687 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.687 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.687 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1049 4.16 4:6df3bbbe:::smithi15736696-13:head [write 3121996~104915] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:30.687 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:39:30.687 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 118.99920808100001, 2024-09-16T03:39:30.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40579890600000001, 2024-09-16T03:39:30.687 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.688 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.688 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.688 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.688 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.688 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.688 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1049 2024-09-16T03:39:30.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.688 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:39:30.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:39:30.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576966+0000", 2024-09-16T03:39:30.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8199999999999999e-07 2024-09-16T03:39:30.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577069+0000", 2024-09-16T03:39:30.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103583 2024-09-16T03:39:30.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577076+0000", 2024-09-16T03:39:30.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8190000000000003e-06 2024-09-16T03:39:30.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577082+0000", 2024-09-16T03:39:30.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5130000000000004e-06 2024-09-16T03:39:30.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880130+0000", 2024-09-16T03:39:30.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.303048127 2024-09-16T03:39:30.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880183+0000", 2024-09-16T03:39:30.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3127999999999997e-05 2024-09-16T03:39:30.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:30.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880282+0000", 2024-09-16T03:39:30.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8881000000000004e-05 2024-09-16T03:39:30.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883101+0000", 2024-09-16T03:39:30.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028194180000000002 2024-09-16T03:39:30.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.979716+0000", 2024-09-16T03:39:30.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.096614899000000004 2024-09-16T03:39:30.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982707+0000", 2024-09-16T03:39:30.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029914249999999998 2024-09-16T03:39:30.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982739+0000", 2024-09-16T03:39:30.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1250000000000001e-05 2024-09-16T03:39:30.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982764+0000", 2024-09-16T03:39:30.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5080999999999999e-05 2024-09-16T03:39:30.694 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.695 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.695 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.695 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1050 4.16 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 3226911] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:30.695 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:39:30.695 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 118.999083486, 2024-09-16T03:39:30.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40784258099999998, 2024-09-16T03:39:30.695 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.695 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.696 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.696 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.696 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.696 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.696 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1050 2024-09-16T03:39:30.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.696 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:39:30.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:39:30.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577090+0000", 2024-09-16T03:39:30.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000003e-07 2024-09-16T03:39:30.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577094+0000", 2024-09-16T03:39:30.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4340000000000001e-06 2024-09-16T03:39:30.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577099+0000", 2024-09-16T03:39:30.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8599999999999998e-06 2024-09-16T03:39:30.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577104+0000", 2024-09-16T03:39:30.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0010000000000001e-06 2024-09-16T03:39:30.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881242+0000", 2024-09-16T03:39:30.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30413779000000002 2024-09-16T03:39:30.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881290+0000", 2024-09-16T03:39:30.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7298e-05 2024-09-16T03:39:30.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:30.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881367+0000", 2024-09-16T03:39:30.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7750000000000006e-05 2024-09-16T03:39:30.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883117+0000", 2024-09-16T03:39:30.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001750096 2024-09-16T03:39:30.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.980218+0000", 2024-09-16T03:39:30.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097100724999999999 2024-09-16T03:39:30.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984859+0000", 2024-09-16T03:39:30.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046412950000000001 2024-09-16T03:39:30.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984900+0000", 2024-09-16T03:39:30.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0300999999999999e-05 2024-09-16T03:39:30.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984932+0000", 2024-09-16T03:39:30.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2332999999999997e-05 2024-09-16T03:39:30.702 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.702 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.703 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1051 4.16 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:30.703 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:30.703 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 118.99906120599999, 2024-09-16T03:39:30.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50807376999999998, 2024-09-16T03:39:30.703 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.703 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.703 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:30.703 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.704 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.704 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.704 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1051 2024-09-16T03:39:30.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.704 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:30.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:30.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:30.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:39:30.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577115+0000", 2024-09-16T03:39:30.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3980000000000002e-06 2024-09-16T03:39:30.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577120+0000", 2024-09-16T03:39:30.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5589999999999996e-06 2024-09-16T03:39:30.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577125+0000", 2024-09-16T03:39:30.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9910000000000002e-06 2024-09-16T03:39:30.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882780+0000", 2024-09-16T03:39:30.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30565438499999997 2024-09-16T03:39:30.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:30.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882825+0000", 2024-09-16T03:39:30.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5739999999999999e-05 2024-09-16T03:39:30.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085039+0000", 2024-09-16T03:39:30.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20221376499999999 2024-09-16T03:39:30.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085096+0000", 2024-09-16T03:39:30.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6666000000000003e-05 2024-09-16T03:39:30.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085186+0000", 2024-09-16T03:39:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9716000000000006e-05 2024-09-16T03:39:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1359 4.6 4:64ce87bc:::smithi15736696-20:head [write 1768928~405354] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T03:39:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:39:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 110.41003806800001, 2024-09-16T03:39:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40239613499999999, 2024-09-16T03:39:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1359 2024-09-16T03:39:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:39:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:39:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166136+0000", 2024-09-16T03:39:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:39:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166923+0000", 2024-09-16T03:39:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00078683500000000005 2024-09-16T03:39:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166933+0000", 2024-09-16T03:39:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0445000000000001e-05 2024-09-16T03:39:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166938+0000", 2024-09-16T03:39:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0119999999999996e-06 2024-09-16T03:39:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264901+0000", 2024-09-16T03:39:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097962831 2024-09-16T03:39:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264963+0000", 2024-09-16T03:39:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1812999999999999e-05 2024-09-16T03:39:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.265085+0000", 2024-09-16T03:39:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121914 2024-09-16T03:39:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.269430+0000", 2024-09-16T03:39:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0043455000000000004 2024-09-16T03:39:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.372259+0000", 2024-09-16T03:39:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102828645 2024-09-16T03:39:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568289+0000", 2024-09-16T03:39:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19603034999999999 2024-09-16T03:39:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568475+0000", 2024-09-16T03:39:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018613599999999999 2024-09-16T03:39:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568531+0000", 2024-09-16T03:39:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5729000000000001e-05 2024-09-16T03:39:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.717 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1676 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v29,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:30.717 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:39:30.717 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 100.446209205, 2024-09-16T03:39:30.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50424139300000004, 2024-09-16T03:39:30.717 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.717 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.717 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.717 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:30.717 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:30.718 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:30.718 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1676 2024-09-16T03:39:30.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.718 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:39:30.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:39:30.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129968+0000", 2024-09-16T03:39:30.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8430000000000003e-06 2024-09-16T03:39:30.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129982+0000", 2024-09-16T03:39:30.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4514e-05 2024-09-16T03:39:30.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130013+0000", 2024-09-16T03:39:30.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1081999999999998e-05 2024-09-16T03:39:30.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130029+0000", 2024-09-16T03:39:30.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5849999999999999e-05 2024-09-16T03:39:30.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130169+0000", 2024-09-16T03:39:30.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000139973 2024-09-16T03:39:30.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130240+0000", 2024-09-16T03:39:30.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0638999999999999e-05 2024-09-16T03:39:30.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:39:30.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.532892+0000", 2024-09-16T03:39:30.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402652654 2024-09-16T03:39:30.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:30.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533546+0000", 2024-09-16T03:39:30.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065390700000000001 2024-09-16T03:39:30.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533943+0000", 2024-09-16T03:39:30.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039703999999999998 2024-09-16T03:39:30.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:30.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634089+0000", 2024-09-16T03:39:30.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100146098 2024-09-16T03:39:30.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634153+0000", 2024-09-16T03:39:30.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3581e-05 2024-09-16T03:39:30.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634205+0000", 2024-09-16T03:39:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2212000000000001e-05 2024-09-16T03:39:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:77 3.2 e73/68 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head v 73'117, mlcod=66'115)", 2024-09-16T03:39:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:39:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 90.397170927000005, 2024-09-16T03:39:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.401393467, 2024-09-16T03:39:30.725 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:39:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:39:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179005+0000", 2024-09-16T03:39:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.553e-06 2024-09-16T03:39:30.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179029+0000", 2024-09-16T03:39:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4039000000000001e-05 2024-09-16T03:39:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179046+0000", 2024-09-16T03:39:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7139999999999999e-05 2024-09-16T03:39:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.279144+0000", 2024-09-16T03:39:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100098245 2024-09-16T03:39:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379323+0000", 2024-09-16T03:39:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100178621 2024-09-16T03:39:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379348+0000", 2024-09-16T03:39:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5315000000000001e-05 2024-09-16T03:39:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580315+0000", 2024-09-16T03:39:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20096734599999999 2024-09-16T03:39:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:30.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:30.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580395+0000", 2024-09-16T03:39:30.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0208000000000002e-05 2024-09-16T03:39:30.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.731 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:30.731 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:30.731 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:30.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:39:30.734 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:30.734 INFO:tasks.rados.rados.0.smithi157.stdout:1907: expect (ObjNum 342 snap 0 seq_num 342) 2024-09-16T03:39:30.734 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 260115 2024-09-16T03:39:30.735 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:39:30.735 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:30.651+0000 7fedb5739640 -1 osd.9 150 handle_osd_map injecting map crc failure 2024-09-16T03:39:30.735 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:30.731+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:30.778 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1140442 2024-09-16T03:39:30.826 INFO:tasks.rados.rados.0.smithi157.stdout:1899: finishing write tid 1 to smithi15736696-49 2024-09-16T03:39:30.826 INFO:tasks.rados.rados.0.smithi157.stdout:1899: finishing write tid 2 to smithi15736696-49 2024-09-16T03:39:30.826 INFO:tasks.rados.rados.0.smithi157.stdout:1900: finishing write tid 1 to smithi15736696-35 2024-09-16T03:39:30.826 INFO:tasks.rados.rados.0.smithi157.stdout:1900: finishing write tid 2 to smithi15736696-35 2024-09-16T03:39:30.826 INFO:tasks.rados.rados.0.smithi157.stdout:1900: finishing write tid 3 to smithi15736696-35 2024-09-16T03:39:30.826 INFO:tasks.rados.rados.0.smithi157.stdout:1896: finishing set_chunk to oid 17 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 139 (ObjNum 406 snap 0 seq_num 406) dirty exists 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:1904: finishing set_chunk to oid 29 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 136 (ObjNum 389 snap 0 seq_num 389) dirty exists 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:1906: finishing write tid 1 to smithi15736696-32 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:1899: finishing write tid 3 to smithi15736696-49 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:1896: done (15 left) 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:1898: done (14 left) 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:1903: done (13 left) 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:1904: done (12 left) 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:1907: done (11 left) 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:1914: read oid 48 snap -1 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:1914: expect (ObjNum 367 snap 0 seq_num 367) 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:1915: write oid 40 current snap is 0 2024-09-16T03:39:30.827 INFO:tasks.rados.rados.0.smithi157.stdout:1899: finishing write tid 4 to smithi15736696-49 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:1899: oid 49 updating version 0 to 79 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:1899: oid 49 updating version 79 to 80 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:1899: oid 49 version 80 is already newer than 78 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 80 (ObjNum 421 snap 0 seq_num 421) dirty exists 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:1899: left oid 49 (ObjNum 421 snap 0 seq_num 421) 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:1900: finishing write tid 4 to smithi15736696-35 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:1900: finishing write tid 5 to smithi15736696-35 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:1900: oid 35 updating version 0 to 129 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:1900: oid 35 updating version 129 to 132 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:1900: oid 35 version 132 is already newer than 131 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:1900: oid 35 version 132 is already newer than 130 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 35 v 132 (ObjNum 422 snap 0 seq_num 422) dirty exists 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:1900: left oid 35 (ObjNum 422 snap 0 seq_num 422) 2024-09-16T03:39:30.828 INFO:tasks.rados.rados.0.smithi157.stdout:1906: finishing write tid 2 to smithi15736696-32 2024-09-16T03:39:30.829 INFO:tasks.rados.rados.0.smithi157.stdout:1915: seq_num 424 ranges {726168=704991,1895071=683474} 2024-09-16T03:39:30.833 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:30.831+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:30.833 INFO:tasks.rados.rados.0.smithi157.stdout:1915: writing smithi15736696-40 from 726168 to 1431159 tid 1 2024-09-16T03:39:30.839 INFO:tasks.rados.rados.0.smithi157.stdout:1915: writing smithi15736696-40 from 1895071 to 2578545 tid 2 2024-09-16T03:39:30.839 INFO:tasks.rados.rados.0.smithi157.stdout:1899: done (12 left) 2024-09-16T03:39:30.839 INFO:tasks.rados.rados.0.smithi157.stdout:1900: done (11 left) 2024-09-16T03:39:30.839 INFO:tasks.rados.rados.0.smithi157.stdout:1916: write oid 18 current snap is 0 2024-09-16T03:39:30.840 INFO:tasks.rados.rados.0.smithi157.stdout:1916: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2024-09-16T03:39:30.847 INFO:tasks.rados.rados.0.smithi157.stdout:1916: writing smithi15736696-18 from 742975 to 1523215 tid 1 2024-09-16T03:39:30.851 INFO:tasks.rados.rados.0.smithi157.stdout:1916: writing smithi15736696-18 from 1953553 to 2380685 tid 2 2024-09-16T03:39:30.856 INFO:tasks.rados.rados.0.smithi157.stdout:1916: writing smithi15736696-18 from 2971713 to 3142975 tid 3 2024-09-16T03:39:30.856 INFO:tasks.rados.rados.0.smithi157.stdout:1917: read oid 22 snap -1 2024-09-16T03:39:30.856 INFO:tasks.rados.rados.0.smithi157.stdout:1917: expect deleted 2024-09-16T03:39:30.856 INFO:tasks.rados.rados.0.smithi157.stdout:1918: write (excl) oid 17 current snap is 0 2024-09-16T03:39:30.856 INFO:tasks.rados.rados.0.smithi157.stdout:1905: expect (ObjNum 418 snap 0 seq_num 418) 2024-09-16T03:39:30.862 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 639907 2024-09-16T03:39:30.880 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:30.880 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:30.881 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:30.881 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:30.890 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:39:30.933 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:30.931+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:31.006 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:31.032 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:31.032 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:31.032 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:31.033 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:31.033 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:31.034 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:31.031+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:31.041 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:39:31.084 INFO:tasks.rados.rados.0.smithi157.stdout:1906: finishing write tid 3 to smithi15736696-32 2024-09-16T03:39:31.085 INFO:tasks.rados.rados.0.smithi157.stdout:1906: finishing write tid 4 to smithi15736696-32 2024-09-16T03:39:31.085 INFO:tasks.rados.rados.0.smithi157.stdout:1906: finishing write tid 5 to smithi15736696-32 2024-09-16T03:39:31.085 INFO:tasks.rados.rados.0.smithi157.stdout:1906: oid 32 updating version 0 to 25 2024-09-16T03:39:31.085 INFO:tasks.rados.rados.0.smithi157.stdout:1906: oid 32 version 25 is already newer than 23 2024-09-16T03:39:31.085 INFO:tasks.rados.rados.0.smithi157.stdout:1906: oid 32 version 25 is already newer than 22 2024-09-16T03:39:31.085 INFO:tasks.rados.rados.0.smithi157.stdout:1906: oid 32 version 25 is already newer than 24 2024-09-16T03:39:31.085 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 32 v 25 (ObjNum 423 snap 0 seq_num 423) dirty exists 2024-09-16T03:39:31.085 INFO:tasks.rados.rados.0.smithi157.stdout:1906: left oid 32 (ObjNum 423 snap 0 seq_num 423) 2024-09-16T03:39:31.085 INFO:tasks.rados.rados.0.smithi157.stdout:1908: finishing set_chunk to oid 24 2024-09-16T03:39:31.085 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 768512 length: 1316352 2024-09-16T03:39:31.085 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 0 (ObjNum 333 snap 0 seq_num 333) dirty exists 2024-09-16T03:39:31.086 INFO:tasks.rados.rados.0.smithi157.stdout:1918: seq_num 426 ranges {759782=571842,2012035=687143} 2024-09-16T03:39:31.090 INFO:tasks.rados.rados.0.smithi157.stdout:1918: writing smithi15736696-17 from 759782 to 1331624 tid 1 2024-09-16T03:39:31.097 INFO:tasks.rados.rados.0.smithi157.stdout:1918: writing smithi15736696-17 from 2012035 to 2699178 tid 2 2024-09-16T03:39:31.098 INFO:tasks.rados.rados.0.smithi157.stdout:1905: done (13 left) 2024-09-16T03:39:31.098 INFO:tasks.rados.rados.0.smithi157.stdout:1906: done (12 left) 2024-09-16T03:39:31.098 INFO:tasks.rados.rados.0.smithi157.stdout:1908: done (11 left) 2024-09-16T03:39:31.098 INFO:tasks.rados.rados.0.smithi157.stdout:1919: set_chunk oid 26 target oid 2024-09-16T03:39:31.135 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:31.133+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:31.147 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:3 3.0 3.f934d18 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:39:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 135.745452524, 2024-09-16T03:39:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78381636700000001, 2024-09-16T03:39:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:39:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:39:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399429+0000", 2024-09-16T03:39:31.149 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399434+0000", 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1050000000000003e-06 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399442+0000", 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0160000000000008e-06 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.150 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399448+0000", 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.879e-06 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399489+0000", 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0815999999999997e-05 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399495+0000", 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3439999999999997e-06 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.151 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183226+0000", 2024-09-16T03:39:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78373077400000002 2024-09-16T03:39:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183244+0000", 2024-09-16T03:39:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8128999999999999e-05 2024-09-16T03:39:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.152 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:5 3.0 3.aefbbec0 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 135.64568506200001, 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68447923499999996, 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.153 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:39:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499196+0000", 2024-09-16T03:39:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3480000000000001e-06 2024-09-16T03:39:31.154 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499202+0000", 2024-09-16T03:39:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4219999999999999e-06 2024-09-16T03:39:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499212+0000", 2024-09-16T03:39:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:39:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.155 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499218+0000", 2024-09-16T03:39:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4920000000000001e-06 2024-09-16T03:39:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499283+0000", 2024-09-16T03:39:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4906000000000005e-05 2024-09-16T03:39:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499293+0000", 2024-09-16T03:39:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0513000000000001e-05 2024-09-16T03:39:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.156 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183665+0000", 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68437196600000005 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183674+0000", 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9700000000000005e-06 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.157 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:6 3.0 3.3bcda120 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 135.64562445300001, 2024-09-16T03:39:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78474037699999999, 2024-09-16T03:39:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0500000000000002e-07 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499260+0000", 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6669999999999998e-06 2024-09-16T03:39:31.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499266+0000", 2024-09-16T03:39:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:39:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499274+0000", 2024-09-16T03:39:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2250000000000003e-06 2024-09-16T03:39:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.160 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499316+0000", 2024-09-16T03:39:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2670000000000003e-05 2024-09-16T03:39:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499323+0000", 2024-09-16T03:39:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.37e-06 2024-09-16T03:39:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283987+0000", 2024-09-16T03:39:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78466433700000005 2024-09-16T03:39:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283996+0000", 2024-09-16T03:39:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8650000000000007e-06 2024-09-16T03:39:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:15 3.0 3.7ec93288 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:39:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 135.536432443, 2024-09-16T03:39:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57518172599999995, 2024-09-16T03:39:31.162 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608449+0000", 2024-09-16T03:39:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.184e-06 2024-09-16T03:39:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608459+0000", 2024-09-16T03:39:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0553999999999999e-05 2024-09-16T03:39:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.164 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608470+0000", 2024-09-16T03:39:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0524e-05 2024-09-16T03:39:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608493+0000", 2024-09-16T03:39:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2980999999999999e-05 2024-09-16T03:39:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608537+0000", 2024-09-16T03:39:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3850000000000002e-05 2024-09-16T03:39:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608542+0000", 2024-09-16T03:39:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:39:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183623+0000", 2024-09-16T03:39:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.575080649 2024-09-16T03:39:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183629+0000", 2024-09-16T03:39:31.166 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6950000000000003e-06 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:11 3.0 3.77005160 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 135.442932133, 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.48153733399999998, 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:39:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:39:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701949+0000", 2024-09-16T03:39:31.168 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.314e-06 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701954+0000", 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701964+0000", 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9550000000000004e-06 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.169 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701986+0000", 2024-09-16T03:39:31.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1931999999999999e-05 2024-09-16T03:39:31.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802129+0000", 2024-09-16T03:39:31.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100142986 2024-09-16T03:39:31.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.170 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:31.170 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802135+0000", 2024-09-16T03:39:31.170 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6350000000000003e-06 2024-09-16T03:39:31.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183475+0000", 2024-09-16T03:39:31.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38133958099999998 2024-09-16T03:39:31.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.171 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.171 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183485+0000", 2024-09-16T03:39:31.171 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0642000000000001e-05 2024-09-16T03:39:31.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.171 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:6 3.0 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 134.16202822, 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60469085300000003, 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982853+0000", 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.393e-06 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982858+0000", 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.062e-06 2024-09-16T03:39:31.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982867+0000", 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982879+0000", 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2241e-05 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982996+0000", 2024-09-16T03:39:31.174 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116996 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983007+0000", 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0694e-05 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284499+0000", 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30149171400000002 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.175 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284690+0000", 2024-09-16T03:39:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000191032 2024-09-16T03:39:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284908+0000", 2024-09-16T03:39:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021827399999999999 2024-09-16T03:39:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487092+0000", 2024-09-16T03:39:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.202184156 2024-09-16T03:39:31.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487110+0000", 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7856999999999999e-05 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587335+0000", 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100224802 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587410+0000", 2024-09-16T03:39:31.177 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5343e-05 2024-09-16T03:39:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587543+0000", 2024-09-16T03:39:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000132492 2024-09-16T03:39:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:2 3.0 3:16550cfd:::smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:39:31.178 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 134.16195602799999, 2024-09-16T03:39:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40289696000000003, 2024-09-16T03:39:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:39:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.179 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:39:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982925+0000", 2024-09-16T03:39:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:39:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982928+0000", 2024-09-16T03:39:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4130000000000002e-06 2024-09-16T03:39:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982936+0000", 2024-09-16T03:39:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.875e-06 2024-09-16T03:39:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982942+0000", 2024-09-16T03:39:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:39:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.181 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983093+0000", 2024-09-16T03:39:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015144400000000001 2024-09-16T03:39:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983098+0000", 2024-09-16T03:39:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4909999999999999e-06 2024-09-16T03:39:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.182 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385695+0000", 2024-09-16T03:39:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402597539 2024-09-16T03:39:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385821+0000", 2024-09-16T03:39:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125418 2024-09-16T03:39:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.184 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:9 3.0 3:2c9b9e7c:::smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:39:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:39:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 134.16193983900001, 2024-09-16T03:39:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60615655800000001, 2024-09-16T03:39:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.185 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:39:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:39:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.186 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982941+0000", 2024-09-16T03:39:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0860000000000001e-06 2024-09-16T03:39:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982946+0000", 2024-09-16T03:39:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:39:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982954+0000", 2024-09-16T03:39:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8520000000000004e-06 2024-09-16T03:39:31.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982958+0000", 2024-09-16T03:39:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4870000000000004e-06 2024-09-16T03:39:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983116+0000", 2024-09-16T03:39:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015774699999999999 2024-09-16T03:39:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:31.188 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983120+0000", 2024-09-16T03:39:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:39:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486023+0000", 2024-09-16T03:39:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50290281400000003 2024-09-16T03:39:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486423+0000", 2024-09-16T03:39:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040079299999999998 2024-09-16T03:39:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486707+0000", 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028336600000000001 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588861+0000", 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102154756 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588952+0000", 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0630000000000005e-05 2024-09-16T03:39:31.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588984+0000", 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2116999999999998e-05 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589022+0000", 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7910000000000001e-05 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589097+0000", 2024-09-16T03:39:31.191 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4536999999999995e-05 2024-09-16T03:39:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:12 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:39:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 134.16186449099999, 2024-09-16T03:39:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60616102900000002, 2024-09-16T03:39:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:31.192 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:39:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:39:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.193 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983017+0000", 2024-09-16T03:39:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.483e-06 2024-09-16T03:39:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983022+0000", 2024-09-16T03:39:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:39:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983032+0000", 2024-09-16T03:39:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0036e-05 2024-09-16T03:39:31.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983038+0000", 2024-09-16T03:39:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6800000000000004e-06 2024-09-16T03:39:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983155+0000", 2024-09-16T03:39:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116401 2024-09-16T03:39:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:31.195 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983163+0000", 2024-09-16T03:39:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6219999999999998e-06 2024-09-16T03:39:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589120+0000", 2024-09-16T03:39:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60595653199999999 2024-09-16T03:39:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589176+0000", 2024-09-16T03:39:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6542000000000002e-05 2024-09-16T03:39:31.196 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:30 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 129.928545985, 2024-09-16T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40312788999999999, 2024-09-16T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.197 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.198 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216338+0000", 2024-09-16T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0799999999999999e-06 2024-09-16T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216354+0000", 2024-09-16T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6439e-05 2024-09-16T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216392+0000", 2024-09-16T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7162000000000001e-05 2024-09-16T03:39:31.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216444+0000", 2024-09-16T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2082e-05 2024-09-16T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417447+0000", 2024-09-16T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20100358400000001 2024-09-16T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.200 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417583+0000", 2024-09-16T03:39:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135537 2024-09-16T03:39:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417777+0000", 2024-09-16T03:39:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019431200000000001 2024-09-16T03:39:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418479+0000", 2024-09-16T03:39:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00070148499999999996 2024-09-16T03:39:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.201 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418755+0000", 2024-09-16T03:39:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027627100000000003 2024-09-16T03:39:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619285+0000", 2024-09-16T03:39:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200530138 2024-09-16T03:39:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619370+0000", 2024-09-16T03:39:31.202 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4567999999999994e-05 2024-09-16T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619462+0000", 2024-09-16T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2231999999999994e-05 2024-09-16T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:34 3.0 3:08307d25:::2b3cb797ae2d3abc24b2cae4a012f26325debe2fc38ccb08bd4503e949d305ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:39:31.203 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 119.670089928, 2024-09-16T03:39:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30798127600000003, 2024-09-16T03:39:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:39:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.204 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474791+0000", 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6000000000000003e-07 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474865+0000", 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4504000000000004e-05 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.205 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474882+0000", 2024-09-16T03:39:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6765999999999998e-05 2024-09-16T03:39:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474890+0000", 2024-09-16T03:39:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.14e-06 2024-09-16T03:39:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681242+0000", 2024-09-16T03:39:31.206 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20635221200000001 2024-09-16T03:39:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681355+0000", 2024-09-16T03:39:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000112211 2024-09-16T03:39:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681505+0000", 2024-09-16T03:39:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015040100000000001 2024-09-16T03:39:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.207 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682352+0000", 2024-09-16T03:39:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084686400000000002 2024-09-16T03:39:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682989+0000", 2024-09-16T03:39:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063749800000000001 2024-09-16T03:39:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782695+0000", 2024-09-16T03:39:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099706063999999997 2024-09-16T03:39:31.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782735+0000", 2024-09-16T03:39:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9304000000000002e-05 2024-09-16T03:39:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782771+0000", 2024-09-16T03:39:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6551999999999998e-05 2024-09-16T03:39:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:87 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 100.34089520400001, 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402368742, 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:39:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803986+0000", 2024-09-16T03:39:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9400000000000004e-07 2024-09-16T03:39:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804061+0000", 2024-09-16T03:39:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5230999999999999e-05 2024-09-16T03:39:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804069+0000", 2024-09-16T03:39:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7530000000000001e-06 2024-09-16T03:39:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804073+0000", 2024-09-16T03:39:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9660000000000004e-06 2024-09-16T03:39:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907236+0000", 2024-09-16T03:39:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103162566 2024-09-16T03:39:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907324+0000", 2024-09-16T03:39:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7894999999999998e-05 2024-09-16T03:39:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907474+0000", 2024-09-16T03:39:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014983999999999999 2024-09-16T03:39:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.908330+0000", 2024-09-16T03:39:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085600100000000001 2024-09-16T03:39:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107220+0000", 2024-09-16T03:39:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198889812 2024-09-16T03:39:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206269+0000", 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099049611999999995 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206305+0000", 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5679000000000002e-05 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206353+0000", 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8593e-05 2024-09-16T03:39:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:3086 3.0 3:1e42685c:::6a9007270aa32b63b7f7e5355768bf791fa2d1832cb8adf59666d35d1b037cad:head [write 0~1679360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:39:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.733057676999998, 2024-09-16T03:39:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57631983099999995, 2024-09-16T03:39:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 3086 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:39:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411824+0000", 2024-09-16T03:39:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.849e-06 2024-09-16T03:39:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414291+0000", 2024-09-16T03:39:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024666900000000001 2024-09-16T03:39:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.218 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414304+0000", 2024-09-16T03:39:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2886e-05 2024-09-16T03:39:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414310+0000", 2024-09-16T03:39:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4219999999999997e-06 2024-09-16T03:39:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783484+0000", 2024-09-16T03:39:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.36917337700000002 2024-09-16T03:39:31.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783601+0000", 2024-09-16T03:39:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011796899999999999 2024-09-16T03:39:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783711+0000", 2024-09-16T03:39:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110012 2024-09-16T03:39:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.220 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887652+0000", 2024-09-16T03:39:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103940373 2024-09-16T03:39:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887669+0000", 2024-09-16T03:39:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6864e-05 2024-09-16T03:39:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942330+0000", 2024-09-16T03:39:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.054661333999999999 2024-09-16T03:39:31.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987927+0000", 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045596722999999999 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987993+0000", 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6141000000000003e-05 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.988142+0000", 2024-09-16T03:39:31.222 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014919099999999999 2024-09-16T03:39:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:211 3.0 3:09d857e3:::85925bf7ff4b61513b387a03c5bd27d81ce6bd960cfaf104686b99338ce39220:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:39:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.506878567999998, 2024-09-16T03:39:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.55082720799999996, 2024-09-16T03:39:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.223 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:39:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:39:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.224 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638003+0000", 2024-09-16T03:39:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8250000000000001e-06 2024-09-16T03:39:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638010+0000", 2024-09-16T03:39:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4130000000000003e-06 2024-09-16T03:39:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638022+0000", 2024-09-16T03:39:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2016e-05 2024-09-16T03:39:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638028+0000", 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0390000000000001e-06 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886321+0000", 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.248293081 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886428+0000", 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000106791 2024-09-16T03:39:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886630+0000", 2024-09-16T03:39:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020280499999999999 2024-09-16T03:39:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889701+0000", 2024-09-16T03:39:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030706650000000002 2024-09-16T03:39:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.227 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942392+0000", 2024-09-16T03:39:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.052691156000000003 2024-09-16T03:39:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987675+0000", 2024-09-16T03:39:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045283229000000001 2024-09-16T03:39:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188710+0000", 2024-09-16T03:39:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20103479899999999 2024-09-16T03:39:31.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188753+0000", 2024-09-16T03:39:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2762000000000002e-05 2024-09-16T03:39:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188829+0000", 2024-09-16T03:39:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5626999999999999e-05 2024-09-16T03:39:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.229 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:214 3.0 3:2fb3d035:::5dba7feb0009e61b5e6a50f88cc7ba99127287f7eeb210342a4b40e996304a6b:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:39:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.506764494000002, 2024-09-16T03:39:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.55085424699999996, 2024-09-16T03:39:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:39:31.230 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:39:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638116+0000", 2024-09-16T03:39:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1200000000000002e-07 2024-09-16T03:39:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638120+0000", 2024-09-16T03:39:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3400000000000002e-06 2024-09-16T03:39:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638126+0000", 2024-09-16T03:39:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2339999999999999e-06 2024-09-16T03:39:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.232 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638130+0000", 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6349999999999997e-06 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887101+0000", 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.248970462 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887191+0000", 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0087000000000007e-05 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.233 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887375+0000", 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018401600000000001 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889706+0000", 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023305869999999998 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942851+0000", 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.053145489999999997 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.088369+0000", 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.14551832200000001 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188856+0000", 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100486244 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188902+0000", 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6134999999999998e-05 2024-09-16T03:39:31.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188970+0000", 2024-09-16T03:39:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7983000000000003e-05 2024-09-16T03:39:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:219 3.0 3:21e0c4e8:::f1ac9f6b1092b2dbfccdb2f32bdf689fb55e405da82aefeec91d0d73cdce51e1:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:39:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 55.406958211999999, 2024-09-16T03:39:31.236 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.71756203500000004, 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737924+0000", 2024-09-16T03:39:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8500000000000001e-06 2024-09-16T03:39:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737931+0000", 2024-09-16T03:39:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9879999999999998e-06 2024-09-16T03:39:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.238 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737942+0000", 2024-09-16T03:39:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1837e-05 2024-09-16T03:39:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.838013+0000", 2024-09-16T03:39:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100070438 2024-09-16T03:39:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887695+0000", 2024-09-16T03:39:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.049682624000000002 2024-09-16T03:39:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887831+0000", 2024-09-16T03:39:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013565499999999999 2024-09-16T03:39:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887948+0000", 2024-09-16T03:39:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000117353 2024-09-16T03:39:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889709+0000", 2024-09-16T03:39:31.240 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017605629999999999 2024-09-16T03:39:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.043145+0000", 2024-09-16T03:39:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.153436136 2024-09-16T03:39:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188664+0000", 2024-09-16T03:39:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.14551857000000001 2024-09-16T03:39:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.241 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455373+0000", 2024-09-16T03:39:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.26670951199999998 2024-09-16T03:39:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455413+0000", 2024-09-16T03:39:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9517999999999997e-05 2024-09-16T03:39:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455484+0000", 2024-09-16T03:39:31.242 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0990999999999998e-05 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:3100 3.0 3:2db88641:::a63ac782fa4bce1e96397c8301f2734ce3d5850a2dc454a54c42fc5451a8d72d:head [write 0~1584128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 54.873377955000002, 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38491842399999998, 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:31.243 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 3100 2024-09-16T03:39:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:39:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:39:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271523+0000", 2024-09-16T03:39:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0649000000000002e-05 2024-09-16T03:39:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273460+0000", 2024-09-16T03:39:31.245 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019373820000000001 2024-09-16T03:39:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273488+0000", 2024-09-16T03:39:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7537000000000001e-05 2024-09-16T03:39:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273503+0000", 2024-09-16T03:39:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5438999999999999e-05 2024-09-16T03:39:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.246 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273645+0000", 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000142013 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273820+0000", 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017546200000000001 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273943+0000", 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012269099999999999 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.276532+0000", 2024-09-16T03:39:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025884010000000002 2024-09-16T03:39:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455527+0000", 2024-09-16T03:39:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.17899536199999999 2024-09-16T03:39:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.556259+0000", 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100731821 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656291+0000", 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100032144 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656330+0000", 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8600000000000003e-05 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656420+0000", 2024-09-16T03:39:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0922999999999997e-05 2024-09-16T03:39:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:54 3.0 3:04086e73:::a30f14ebc2b326cba32a5aae93ab4cf09bd2a972b24cfcba49e65d7da3ac52de:head [call cas.chunk_create_or_get_ref in=131071b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:39:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 49.78957158, 2024-09-16T03:39:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30746864499999998, 2024-09-16T03:39:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.250 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:39:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:39:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355311+0000", 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3250000000000002e-06 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355502+0000", 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019106 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355515+0000", 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3730000000000001e-05 2024-09-16T03:39:31.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355526+0000", 2024-09-16T03:39:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0473e-05 2024-09-16T03:39:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355567+0000", 2024-09-16T03:39:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1455000000000002e-05 2024-09-16T03:39:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:31.253 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355576+0000", 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2840000000000006e-06 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.457986+0000", 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10241070200000001 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.458087+0000", 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000100715 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.254 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.458252+0000", 2024-09-16T03:39:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016499 2024-09-16T03:39:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560797+0000", 2024-09-16T03:39:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10254485300000001 2024-09-16T03:39:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662591+0000", 2024-09-16T03:39:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10179377100000001 2024-09-16T03:39:31.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662693+0000", 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000102174 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662745+0000", 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1887e-05 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662777+0000", 2024-09-16T03:39:31.256 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2225999999999999e-05 2024-09-16T03:39:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:58 3.0 3:07e63342:::0385a732ea8ffd270f01137d752687ebb6ebb56bc894ea96f00cc7a27fcd2563:head [call cas.chunk_create_or_get_ref in=284238b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:39:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 49.789338745999999, 2024-09-16T03:39:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30738563099999999, 2024-09-16T03:39:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.257 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355542+0000", 2024-09-16T03:39:31.258 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1699999999999997e-07 2024-09-16T03:39:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355761+0000", 2024-09-16T03:39:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021889699999999999 2024-09-16T03:39:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355769+0000", 2024-09-16T03:39:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.695e-06 2024-09-16T03:39:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.259 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355775+0000", 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5239999999999997e-06 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355851+0000", 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6103e-05 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355860+0000", 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0710000000000001e-06 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459143+0000", 2024-09-16T03:39:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10328306399999999 2024-09-16T03:39:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459292+0000", 2024-09-16T03:39:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000148221 2024-09-16T03:39:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459540+0000", 2024-09-16T03:39:31.261 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024846100000000001 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662661+0000", 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.203120936 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662669+0000", 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6520000000000006e-06 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.262 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662824+0000", 2024-09-16T03:39:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015512699999999999 2024-09-16T03:39:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662885+0000", 2024-09-16T03:39:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0789999999999999e-05 2024-09-16T03:39:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662927+0000", 2024-09-16T03:39:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2172999999999998e-05 2024-09-16T03:39:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.263 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:60 3.0 3:16d0c248:::a2b7d44d389d3ac8752762fa1d216869a8025d7e5ae15e8811d7e7f2543095f6:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:39:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 49.789093674999997, 2024-09-16T03:39:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50513552500000003, 2024-09-16T03:39:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.264 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:39:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:39:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355787+0000", 2024-09-16T03:39:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-16T03:39:31.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355791+0000", 2024-09-16T03:39:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.631e-06 2024-09-16T03:39:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355797+0000", 2024-09-16T03:39:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7379999999999999e-06 2024-09-16T03:39:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.266 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355802+0000", 2024-09-16T03:39:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.322e-06 2024-09-16T03:39:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.356898+0000", 2024-09-16T03:39:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010963710000000001 2024-09-16T03:39:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.356902+0000", 2024-09-16T03:39:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0110000000000002e-06 2024-09-16T03:39:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.267 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560272+0000", 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20336954199999999 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560376+0000", 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103885 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560508+0000", 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013254599999999999 2024-09-16T03:39:31.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662786+0000", 2024-09-16T03:39:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10227757699999999 2024-09-16T03:39:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860745+0000", 2024-09-16T03:39:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19795950700000001 2024-09-16T03:39:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:31.269 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860843+0000", 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7829000000000001e-05 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860890+0000", 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7005999999999997e-05 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860922+0000", 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1680000000000002e-05 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.270 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.271 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.271 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:31.272 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:39:31.273 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:31.271+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:39:31.373 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:31.370+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:31.377 INFO:tasks.rados.rados.0.smithi157.stdout:1919: set_chunk oid 26 offset: 61440 length: 1213440 target oid 854bc0225a3bb690bfac097dfa6f255fbb6cdea230ec8761b6d08a4474b3799a offset: 0 snap -1 2024-09-16T03:39:31.377 INFO:tasks.rados.rados.0.smithi157.stdout:1920: write (excl) oid 31 current snap is 0 2024-09-16T03:39:31.377 INFO:tasks.rados.rados.0.smithi157.stdout:1920: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2024-09-16T03:39:31.384 INFO:tasks.rados.rados.0.smithi157.stdout:1920: writing smithi15736696-31 from 776589 to 1423680 tid 1 2024-09-16T03:39:31.392 INFO:tasks.rados.rados.0.smithi157.stdout:1920: writing smithi15736696-31 from 2070517 to 2501318 tid 2 2024-09-16T03:39:31.394 INFO:tasks.rados.rados.0.smithi157.stdout:1920: writing smithi15736696-31 from 3026559 to 3176589 tid 3 2024-09-16T03:39:31.394 INFO:tasks.rados.rados.0.smithi157.stdout:1921: read oid 5 snap -1 2024-09-16T03:39:31.394 INFO:tasks.rados.rados.0.smithi157.stdout:1921: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-16T03:39:31.394 INFO:tasks.rados.rados.0.smithi157.stdout:1922: read oid 32 snap -1 2024-09-16T03:39:31.394 INFO:tasks.rados.rados.0.smithi157.stdout:1922: expect (ObjNum 423 snap 0 seq_num 423) 2024-09-16T03:39:31.395 INFO:tasks.rados.rados.0.smithi157.stdout:1923: write (excl) oid 11 current snap is 0 2024-09-16T03:39:31.395 INFO:tasks.rados.rados.0.smithi157.stdout:1910: expect (ObjNum 297 snap 0 seq_num 297) 2024-09-16T03:39:31.420 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:31.420 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:31.428 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:39:31.474 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:31.472+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:31.486 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2719424 2024-09-16T03:39:31.494 INFO:tasks.rados.rados.0.smithi157.stdout:1909: expect (ObjNum 387 snap 0 seq_num 387) 2024-09-16T03:39:31.494 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:31.532 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:31.533 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:31.533 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:31.533 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:31.533 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:31.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:39:31.542 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:39:31.575 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:31.574+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:31.671 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:31.672 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:31.677 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:31.675+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:31.688 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:31.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:57 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'1, mlcod=0'0)", 2024-09-16T03:39:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:39:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 139.561276749, 2024-09-16T03:39:31.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20372090900000001, 2024-09-16T03:39:31.691 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.691 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.691 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.691 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:39:31.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:39:31.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125313+0000", 2024-09-16T03:39:31.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7180000000000001e-06 2024-09-16T03:39:31.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126730+0000", 2024-09-16T03:39:31.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00141731 2024-09-16T03:39:31.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.695 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126746+0000", 2024-09-16T03:39:31.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6455000000000001e-05 2024-09-16T03:39:31.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126756+0000", 2024-09-16T03:39:31.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:39:31.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227068+0000", 2024-09-16T03:39:31.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100312308 2024-09-16T03:39:31.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227085+0000", 2024-09-16T03:39:31.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6898e-05 2024-09-16T03:39:31.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328995+0000", 2024-09-16T03:39:31.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101909903 2024-09-16T03:39:31.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329031+0000", 2024-09-16T03:39:31.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5717999999999999e-05 2024-09-16T03:39:31.701 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.701 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.701 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.701 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:58 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:31.702 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:39:31.702 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 139.55794361900001, 2024-09-16T03:39:31.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084524500000001, 2024-09-16T03:39:31.702 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.702 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.703 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.703 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:39:31.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:39:31.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128644+0000", 2024-09-16T03:39:31.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.251e-06 2024-09-16T03:39:31.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128913+0000", 2024-09-16T03:39:31.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026856200000000002 2024-09-16T03:39:31.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128925+0000", 2024-09-16T03:39:31.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2109999999999999e-05 2024-09-16T03:39:31.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128933+0000", 2024-09-16T03:39:31.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6260000000000003e-06 2024-09-16T03:39:31.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328196+0000", 2024-09-16T03:39:31.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19926303200000001 2024-09-16T03:39:31.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328253+0000", 2024-09-16T03:39:31.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7262999999999997e-05 2024-09-16T03:39:31.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329458+0000", 2024-09-16T03:39:31.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012051939999999999 2024-09-16T03:39:31.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329488+0000", 2024-09-16T03:39:31.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0207000000000001e-05 2024-09-16T03:39:31.712 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.712 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.713 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.713 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:59 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'3, mlcod=0'0)", 2024-09-16T03:39:31.713 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:39:31.714 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 139.55744818599999, 2024-09-16T03:39:31.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036954900000001, 2024-09-16T03:39:31.714 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.714 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.714 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.715 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:39:31.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:39:31.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129139+0000", 2024-09-16T03:39:31.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6099999999999999e-07 2024-09-16T03:39:31.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129158+0000", 2024-09-16T03:39:31.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088e-05 2024-09-16T03:39:31.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129168+0000", 2024-09-16T03:39:31.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.048e-05 2024-09-16T03:39:31.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.229233+0000", 2024-09-16T03:39:31.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100065168 2024-09-16T03:39:31.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328815+0000", 2024-09-16T03:39:31.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099581694999999998 2024-09-16T03:39:31.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328845+0000", 2024-09-16T03:39:31.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0193e-05 2024-09-16T03:39:31.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329492+0000", 2024-09-16T03:39:31.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064671800000000001 2024-09-16T03:39:31.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329508+0000", 2024-09-16T03:39:31.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6246e-05 2024-09-16T03:39:31.724 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.725 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.726 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.0:15 4.d e47/32 4:b3fe05b2:::smithi15736696-15:head v 47'5, mlcod=39'3)", 2024-09-16T03:39:31.726 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:39:31.726 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 125.38415345999999, 2024-09-16T03:39:31.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100920751, 2024-09-16T03:39:31.726 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.726 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.727 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.727 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:39:31.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:39:31.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302435+0000", 2024-09-16T03:39:31.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2340000000000001e-06 2024-09-16T03:39:31.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302446+0000", 2024-09-16T03:39:31.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0596e-05 2024-09-16T03:39:31.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302461+0000", 2024-09-16T03:39:31.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5082000000000001e-05 2024-09-16T03:39:31.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402592+0000", 2024-09-16T03:39:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100130488 2024-09-16T03:39:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402712+0000", 2024-09-16T03:39:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012013100000000001 2024-09-16T03:39:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402741+0000", 2024-09-16T03:39:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9717999999999999e-05 2024-09-16T03:39:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403309+0000", 2024-09-16T03:39:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056725700000000005 2024-09-16T03:39:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403354+0000", 2024-09-16T03:39:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5244999999999998e-05 2024-09-16T03:39:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.737 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:39:31.737 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:39:31.737 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 101.536654751, 2024-09-16T03:39:31.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032593800000000001, 2024-09-16T03:39:31.738 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.738 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.738 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.738 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:39:31.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.740 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.740 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:39:31.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149943+0000", 2024-09-16T03:39:31.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0817e-05 2024-09-16T03:39:31.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151026+0000", 2024-09-16T03:39:31.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010835350000000001 2024-09-16T03:39:31.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151040+0000", 2024-09-16T03:39:31.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3884e-05 2024-09-16T03:39:31.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151049+0000", 2024-09-16T03:39:31.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:39:31.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151107+0000", 2024-09-16T03:39:31.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8199000000000003e-05 2024-09-16T03:39:31.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151124+0000", 2024-09-16T03:39:31.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7756000000000001e-05 2024-09-16T03:39:31.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153142+0000", 2024-09-16T03:39:31.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020177569999999998 2024-09-16T03:39:31.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153191+0000", 2024-09-16T03:39:31.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9051999999999997e-05 2024-09-16T03:39:31.748 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.748 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.748 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.749 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1682 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'7, mlcod=40'4)", 2024-09-16T03:39:31.749 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:39:31.749 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 101.433057561, 2024-09-16T03:39:31.749 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10169502499999999, 2024-09-16T03:39:31.749 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.750 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.750 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.750 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.750 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:39:31.751 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.751 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.751 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:39:31.752 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.752 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.752 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253540+0000", 2024-09-16T03:39:31.753 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0888e-05 2024-09-16T03:39:31.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.753 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.753 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254015+0000", 2024-09-16T03:39:31.754 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047485499999999999 2024-09-16T03:39:31.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.754 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.755 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254030+0000", 2024-09-16T03:39:31.755 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4899000000000001e-05 2024-09-16T03:39:31.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.755 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.756 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254039+0000", 2024-09-16T03:39:31.756 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5000000000000005e-06 2024-09-16T03:39:31.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.757 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.757 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254107+0000", 2024-09-16T03:39:31.757 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7298000000000005e-05 2024-09-16T03:39:31.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.758 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254122+0000", 2024-09-16T03:39:31.758 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5928999999999999e-05 2024-09-16T03:39:31.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.758 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.759 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355184+0000", 2024-09-16T03:39:31.759 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101061703 2024-09-16T03:39:31.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.759 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.760 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355224+0000", 2024-09-16T03:39:31.760 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9953000000000002e-05 2024-09-16T03:39:31.760 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.760 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.760 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.761 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1683 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'8, mlcod=40'4)", 2024-09-16T03:39:31.761 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:39:31.761 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 101.432520013, 2024-09-16T03:39:31.761 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101630885, 2024-09-16T03:39:31.762 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.762 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.762 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.762 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.762 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.763 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:39:31.763 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.763 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.764 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:39:31.764 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.764 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.765 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254068+0000", 2024-09-16T03:39:31.765 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5700000000000001e-07 2024-09-16T03:39:31.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.765 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254075+0000", 2024-09-16T03:39:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2710000000000003e-06 2024-09-16T03:39:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254083+0000", 2024-09-16T03:39:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0220000000000004e-06 2024-09-16T03:39:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254089+0000", 2024-09-16T03:39:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8699999999999997e-06 2024-09-16T03:39:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354968+0000", 2024-09-16T03:39:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100879625 2024-09-16T03:39:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354992+0000", 2024-09-16T03:39:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-16T03:39:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355665+0000", 2024-09-16T03:39:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00067371100000000001 2024-09-16T03:39:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355698+0000", 2024-09-16T03:39:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2291e-05 2024-09-16T03:39:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.0:177 4.10 e65/57 4:0a55bbf2:::smithi15736696-18:head v 65'47, mlcod=0'0)", 2024-09-16T03:39:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:31.769 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 100.479573425, 2024-09-16T03:39:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074461099999999995, 2024-09-16T03:39:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.770 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5300000000000001e-07 2024-09-16T03:39:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207018+0000", 2024-09-16T03:39:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7210000000000002e-06 2024-09-16T03:39:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207031+0000", 2024-09-16T03:39:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2767e-05 2024-09-16T03:39:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207040+0000", 2024-09-16T03:39:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5400000000000001e-06 2024-09-16T03:39:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.772 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207107+0000", 2024-09-16T03:39:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6625999999999995e-05 2024-09-16T03:39:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207131+0000", 2024-09-16T03:39:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4128e-05 2024-09-16T03:39:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207712+0000", 2024-09-16T03:39:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00058124700000000001 2024-09-16T03:39:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207758+0000", 2024-09-16T03:39:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5429000000000002e-05 2024-09-16T03:39:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1723 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'48, mlcod=53'46)", 2024-09-16T03:39:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:39:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 100.037506244, 2024-09-16T03:39:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00088284599999999996, 2024-09-16T03:39:31.774 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:39:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:39:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649081+0000", 2024-09-16T03:39:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8299999999999995e-07 2024-09-16T03:39:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649101+0000", 2024-09-16T03:39:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0126e-05 2024-09-16T03:39:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.776 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649141+0000", 2024-09-16T03:39:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9609e-05 2024-09-16T03:39:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649161+0000", 2024-09-16T03:39:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9599999999999999e-05 2024-09-16T03:39:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649299+0000", 2024-09-16T03:39:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013822199999999999 2024-09-16T03:39:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649318+0000", 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9273999999999999e-05 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649919+0000", 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060074100000000001 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649963+0000", 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4391000000000002e-05 2024-09-16T03:39:31.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.779 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.779 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1724 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'49, mlcod=53'46)", 2024-09-16T03:39:31.779 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:39:31.779 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 100.033841279, 2024-09-16T03:39:31.779 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064536300000000003, 2024-09-16T03:39:31.779 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.779 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.779 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.780 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:39:31.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.780 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.780 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:39:31.780 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652747+0000", 2024-09-16T03:39:31.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.291e-06 2024-09-16T03:39:31.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652753+0000", 2024-09-16T03:39:31.781 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4450000000000001e-06 2024-09-16T03:39:31.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.781 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.781 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652782+0000", 2024-09-16T03:39:31.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9094999999999998e-05 2024-09-16T03:39:31.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652797+0000", 2024-09-16T03:39:31.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5021999999999999e-05 2024-09-16T03:39:31.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.782 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.782 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652854+0000", 2024-09-16T03:39:31.782 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6586999999999999e-05 2024-09-16T03:39:31.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652885+0000", 2024-09-16T03:39:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.146e-05 2024-09-16T03:39:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653350+0000", 2024-09-16T03:39:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046443599999999998 2024-09-16T03:39:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653391+0000", 2024-09-16T03:39:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1026999999999999e-05 2024-09-16T03:39:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1725 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'50, mlcod=53'46)", 2024-09-16T03:39:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:39:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 100.033048456, 2024-09-16T03:39:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056324100000000002, 2024-09-16T03:39:31.784 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:39:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:39:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:31.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653539+0000", 2024-09-16T03:39:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9699999999999994e-07 2024-09-16T03:39:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653545+0000", 2024-09-16T03:39:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9309999999999996e-06 2024-09-16T03:39:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:31.786 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653557+0000", 2024-09-16T03:39:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2340999999999999e-05 2024-09-16T03:39:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653564+0000", 2024-09-16T03:39:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2779999999999999e-06 2024-09-16T03:39:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653605+0000", 2024-09-16T03:39:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1301000000000003e-05 2024-09-16T03:39:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653621+0000", 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5951e-05 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654064+0000", 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000442843 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654101+0000", 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7598999999999997e-05 2024-09-16T03:39:31.788 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:31.789 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:31.790 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:31.791 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:31.792 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:31.792 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:31.792 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:39:31.793 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:31.788+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:31.793 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:31.794 INFO:tasks.rados.rados.0.smithi157.stdout:1912: finishing set_chunk to oid 8 2024-09-16T03:39:31.794 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 8 set_chunk returned error code -95 offset: 636928 length: 1108480 2024-09-16T03:39:31.794 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 0 (ObjNum 359 snap 0 seq_num 359) dirty exists 2024-09-16T03:39:31.794 INFO:tasks.rados.rados.0.smithi157.stdout:1923: seq_num 428 ranges {793396=722340,2128999=690812} 2024-09-16T03:39:31.794 INFO:tasks.rados.rados.0.smithi157.stdout:1923: writing smithi15736696-11 from 793396 to 1515736 tid 1 2024-09-16T03:39:31.794 INFO:tasks.rados.rados.0.smithi157.stdout:1923: writing smithi15736696-11 from 2128999 to 2819811 tid 2 2024-09-16T03:39:31.794 INFO:tasks.rados.rados.0.smithi157.stdout:1909: done (15 left) 2024-09-16T03:39:31.794 INFO:tasks.rados.rados.0.smithi157.stdout:1910: done (14 left) 2024-09-16T03:39:31.794 INFO:tasks.rados.rados.0.smithi157.stdout:1912: done (13 left) 2024-09-16T03:39:31.794 INFO:tasks.rados.rados.0.smithi157.stdout:1924: write (excl) oid 29 current snap is 0 2024-09-16T03:39:31.794 INFO:tasks.rados.rados.0.smithi157.stdout:1911: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-16T03:39:31.795 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:31.795 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:31.795 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:31.796 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:31.803 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:31.803 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:31.820 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:31.821 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:31.837 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:31.837 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:31.847 INFO:tasks.rados.rados.0.smithi157.stdout:1914: expect (ObjNum 367 snap 0 seq_num 367) 2024-09-16T03:39:31.853 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:31.853 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:31.870 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:31.870 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:31.887 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:31.887 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:31.891 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:31.890+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:31.904 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:31.904 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:31.909 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1222089 2024-09-16T03:39:31.914 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:31.914 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:31.919 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:31.920 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:31.923 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:39:31.926 INFO:tasks.rados.rados.0.smithi157.stdout:1913: got (22) Invalid argument 2024-09-16T03:39:31.926 INFO:tasks.rados.rados.0.smithi157.stdout:1916: finishing write tid 1 to smithi15736696-18 2024-09-16T03:39:31.926 INFO:tasks.rados.rados.0.smithi157.stdout:1916: finishing write tid 2 to smithi15736696-18 2024-09-16T03:39:31.926 INFO:tasks.rados.rados.0.smithi157.stdout:1916: finishing write tid 3 to smithi15736696-18 2024-09-16T03:39:31.926 INFO:tasks.rados.rados.0.smithi157.stdout:1915: finishing write tid 1 to smithi15736696-40 2024-09-16T03:39:31.926 INFO:tasks.rados.rados.0.smithi157.stdout:1924: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-09-16T03:39:31.932 INFO:tasks.rados.rados.0.smithi157.stdout:1924: writing smithi15736696-29 from 410203 to 1207792 tid 1 2024-09-16T03:39:31.938 INFO:tasks.rados.rados.0.smithi157.stdout:1924: writing smithi15736696-29 from 1787481 to 2221951 tid 2 2024-09-16T03:39:31.944 INFO:tasks.rados.rados.0.smithi157.stdout:1924: writing smithi15736696-29 from 2681405 to 3210203 tid 3 2024-09-16T03:39:31.945 INFO:tasks.rados.rados.0.smithi157.stdout:1911: done (13 left) 2024-09-16T03:39:31.945 INFO:tasks.rados.rados.0.smithi157.stdout:1913: done (12 left) 2024-09-16T03:39:31.945 INFO:tasks.rados.rados.0.smithi157.stdout:1914: done (11 left) 2024-09-16T03:39:31.945 INFO:tasks.rados.rados.0.smithi157.stdout:1917: done (10 left) 2024-09-16T03:39:31.945 INFO:tasks.rados.rados.0.smithi157.stdout:1925: write oid 12 current snap is 0 2024-09-16T03:39:31.945 INFO:tasks.rados.rados.0.smithi157.stdout:1925: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-09-16T03:39:31.946 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:31.946 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:31.948 INFO:tasks.rados.rados.0.smithi157.stdout:1925: writing smithi15736696-12 from 427010 to 899848 tid 1 2024-09-16T03:39:31.955 INFO:tasks.rados.rados.0.smithi157.stdout:1925: writing smithi15736696-12 from 1329610 to 2024091 tid 2 2024-09-16T03:39:31.961 INFO:tasks.rados.rados.0.smithi157.stdout:1925: writing smithi15736696-12 from 2708828 to 3227010 tid 3 2024-09-16T03:39:31.961 INFO:tasks.rados.rados.0.smithi157.stdout:1926: read oid 13 snap -1 2024-09-16T03:39:31.961 INFO:tasks.rados.rados.0.smithi157.stdout:1926: expect (ObjNum 401 snap 0 seq_num 401) 2024-09-16T03:39:31.961 INFO:tasks.rados.rados.0.smithi157.stdout:1927: write oid 3 current snap is 0 2024-09-16T03:39:31.962 INFO:tasks.rados.rados.0.smithi157.stdout:1927: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2024-09-16T03:39:31.964 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:31.964 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:31.967 INFO:tasks.rados.rados.0.smithi157.stdout:1927: writing smithi15736696-3 from 443817 to 991904 tid 1 2024-09-16T03:39:31.974 INFO:tasks.rados.rados.0.smithi157.stdout:1927: writing smithi15736696-3 from 1788092 to 2226231 tid 2 2024-09-16T03:39:31.979 INFO:tasks.rados.rados.0.smithi157.stdout:1927: writing smithi15736696-3 from 3019898 to 3243817 tid 3 2024-09-16T03:39:31.979 INFO:tasks.rados.rados.0.smithi157.stdout:1928: tier_promote oid 49 2024-09-16T03:39:31.980 INFO:tasks.rados.rados.0.smithi157.stdout:1915: finishing write tid 2 to smithi15736696-40 2024-09-16T03:39:31.980 INFO:tasks.rados.rados.0.smithi157.stdout:1929: read oid 16 snap -1 2024-09-16T03:39:31.980 INFO:tasks.rados.rados.0.smithi157.stdout:1929: expect (ObjNum 415 snap 0 seq_num 415) 2024-09-16T03:39:31.980 INFO:tasks.rados.rados.0.smithi157.stdout:1930: write (excl) oid 9 current snap is 0 2024-09-16T03:39:31.980 INFO:tasks.rados.rados.0.smithi157.stdout:1915: finishing write tid 3 to smithi15736696-40 2024-09-16T03:39:31.980 INFO:tasks.rados.rados.0.smithi157.stdout:1901: got (0) Success 2024-09-16T03:39:31.980 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 0 (ObjNum 386 snap 0 seq_num 386) dirty exists 2024-09-16T03:39:31.980 INFO:tasks.rados.rados.0.smithi157.stdout:1918: finishing write tid 1 to smithi15736696-17 2024-09-16T03:39:31.980 INFO:tasks.rados.rados.0.smithi157.stdout:1921: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-16T03:39:31.983 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:31.983 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:31.992 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:31.991+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:32.028 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1115582 2024-09-16T03:39:32.030 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:32.030 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:32.041 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:39:32.049 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:32.087 INFO:tasks.rados.rados.0.smithi157.stdout:1922: expect (ObjNum 423 snap 0 seq_num 423) 2024-09-16T03:39:32.088 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:32.093 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:32.091+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:32.144 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:32.145 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:32.154 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:39:32.194 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:32.193+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:32.218 INFO:tasks.rados.rados.0.smithi157.stdout:1916: finishing write tid 4 to smithi15736696-18 2024-09-16T03:39:32.219 INFO:tasks.rados.rados.0.smithi157.stdout:1915: finishing write tid 4 to smithi15736696-40 2024-09-16T03:39:32.219 INFO:tasks.rados.rados.0.smithi157.stdout:1915: oid 40 updating version 0 to 153 2024-09-16T03:39:32.219 INFO:tasks.rados.rados.0.smithi157.stdout:1915: oid 40 updating version 153 to 154 2024-09-16T03:39:32.219 INFO:tasks.rados.rados.0.smithi157.stdout:1915: oid 40 version 154 is already newer than 152 2024-09-16T03:39:32.219 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 154 (ObjNum 424 snap 0 seq_num 424) dirty exists 2024-09-16T03:39:32.219 INFO:tasks.rados.rados.0.smithi157.stdout:1915: left oid 40 (ObjNum 424 snap 0 seq_num 424) 2024-09-16T03:39:32.219 INFO:tasks.rados.rados.0.smithi157.stdout:1918: finishing write tid 2 to smithi15736696-17 2024-09-16T03:39:32.219 INFO:tasks.rados.rados.0.smithi157.stdout:1916: finishing write tid 5 to smithi15736696-18 2024-09-16T03:39:32.219 INFO:tasks.rados.rados.0.smithi157.stdout:1916: oid 18 updating version 0 to 75 2024-09-16T03:39:32.219 INFO:tasks.rados.rados.0.smithi157.stdout:1916: oid 18 version 75 is already newer than 74 2024-09-16T03:39:32.219 INFO:tasks.rados.rados.0.smithi157.stdout:1916: oid 18 updating version 75 to 76 2024-09-16T03:39:32.219 INFO:tasks.rados.rados.0.smithi157.stdout:1916: oid 18 version 76 is already newer than 73 2024-09-16T03:39:32.220 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 76 (ObjNum 425 snap 0 seq_num 425) dirty exists 2024-09-16T03:39:32.220 INFO:tasks.rados.rados.0.smithi157.stdout:1916: left oid 18 (ObjNum 425 snap 0 seq_num 425) 2024-09-16T03:39:32.220 INFO:tasks.rados.rados.0.smithi157.stdout:1930: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-09-16T03:39:32.224 INFO:tasks.rados.rados.0.smithi157.stdout:1930: writing smithi15736696-9 from 460624 to 1083960 tid 1 2024-09-16T03:39:32.233 INFO:tasks.rados.rados.0.smithi157.stdout:1930: writing smithi15736696-9 from 1846574 to 2544724 tid 2 2024-09-16T03:39:32.236 INFO:tasks.rados.rados.0.smithi157.stdout:1930: writing smithi15736696-9 from 3163674 to 3260624 tid 3 2024-09-16T03:39:32.236 INFO:tasks.rados.rados.0.smithi157.stdout:1901: done (15 left) 2024-09-16T03:39:32.236 INFO:tasks.rados.rados.0.smithi157.stdout:1915: done (14 left) 2024-09-16T03:39:32.236 INFO:tasks.rados.rados.0.smithi157.stdout:1916: done (13 left) 2024-09-16T03:39:32.237 INFO:tasks.rados.rados.0.smithi157.stdout:1921: done (12 left) 2024-09-16T03:39:32.237 INFO:tasks.rados.rados.0.smithi157.stdout:1922: done (11 left) 2024-09-16T03:39:32.237 INFO:tasks.rados.rados.0.smithi157.stdout:1931: read oid 32 snap -1 2024-09-16T03:39:32.237 INFO:tasks.rados.rados.0.smithi157.stdout:1931: expect (ObjNum 423 snap 0 seq_num 423) 2024-09-16T03:39:32.237 INFO:tasks.rados.rados.0.smithi157.stdout:1932: read oid 28 snap -1 2024-09-16T03:39:32.237 INFO:tasks.rados.rados.0.smithi157.stdout:1932: expect (ObjNum 387 snap 0 seq_num 387) 2024-09-16T03:39:32.237 INFO:tasks.rados.rados.0.smithi157.stdout:1933: set_chunk oid 14 target oid 2024-09-16T03:39:32.259 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:32.259 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:32.268 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:39:32.312 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:32.310+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:32.366 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:32.365+0000 7f6528e68640 -1 osd.1 152 handle_osd_map injecting map crc failure 2024-09-16T03:39:32.374 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:32.374 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:32.383 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:39:32.406 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:32.404+0000 7f9288166640 -1 osd.14 152 handle_osd_map injecting map crc failure 2024-09-16T03:39:32.407 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:32.404+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:32.456 INFO:tasks.rados.rados.0.smithi157.stdout:1933: set_chunk oid 14 offset: 212992 length: 1948672 target oid d60bd2b8228bd3a678a5ef552346d8a9cc0326254c12a9fa397ef3c67c0b22d1 offset: 0 snap -1 2024-09-16T03:39:32.456 INFO:tasks.rados.rados.0.smithi157.stdout:1934: delete oid 47 current snap is 0 2024-09-16T03:39:32.456 INFO:tasks.rados.rados.0.smithi157.stdout:1918: finishing write tid 3 to smithi15736696-17 2024-09-16T03:39:32.456 INFO:tasks.rados.rados.0.smithi157.stdout:1918: finishing write tid 4 to smithi15736696-17 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:1918: oid 17 updating version 0 to 143 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:1918: oid 17 updating version 143 to 144 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:1918: oid 17 version 144 is already newer than 142 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 144 (ObjNum 426 snap 0 seq_num 426) dirty exists 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:1918: left oid 17 (ObjNum 426 snap 0 seq_num 426) 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:1920: finishing write tid 1 to smithi15736696-31 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:1920: finishing write tid 2 to smithi15736696-31 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:1920: finishing write tid 3 to smithi15736696-31 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:1920: finishing write tid 4 to smithi15736696-31 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:1920: finishing write tid 5 to smithi15736696-31 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:1920: oid 31 updating version 0 to 87 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:1920: oid 31 updating version 87 to 89 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:1920: oid 31 version 89 is already newer than 88 2024-09-16T03:39:32.457 INFO:tasks.rados.rados.0.smithi157.stdout:1920: oid 31 version 89 is already newer than 86 2024-09-16T03:39:32.458 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 89 (ObjNum 427 snap 0 seq_num 427) dirty exists 2024-09-16T03:39:32.458 INFO:tasks.rados.rados.0.smithi157.stdout:1920: left oid 31 (ObjNum 427 snap 0 seq_num 427) 2024-09-16T03:39:32.458 INFO:tasks.rados.rados.0.smithi157.stdout:1919: finishing set_chunk to oid 26 2024-09-16T03:39:32.458 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 119 (ObjNum 379 snap 0 seq_num 379) dirty exists 2024-09-16T03:39:32.458 INFO:tasks.rados.rados.0.smithi157.stdout:1926: expect (ObjNum 401 snap 0 seq_num 401) 2024-09-16T03:39:32.497 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:32.496+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:32.510 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:32.510 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:32.519 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:39:32.599 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:32.597+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:32.624 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:32.624 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:32.631 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:39:32.699 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:32.697+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:32.715 INFO:tasks.rados.rados.0.smithi157.stdout:1923: finishing write tid 1 to smithi15736696-11 2024-09-16T03:39:32.715 INFO:tasks.rados.rados.0.smithi157.stdout:1923: finishing write tid 2 to smithi15736696-11 2024-09-16T03:39:32.715 INFO:tasks.rados.rados.0.smithi157.stdout:1923: finishing write tid 3 to smithi15736696-11 2024-09-16T03:39:32.715 INFO:tasks.rados.rados.0.smithi157.stdout:1927: finishing write tid 1 to smithi15736696-3 2024-09-16T03:39:32.715 INFO:tasks.rados.rados.0.smithi157.stdout:1923: finishing write tid 4 to smithi15736696-11 2024-09-16T03:39:32.715 INFO:tasks.rados.rados.0.smithi157.stdout:1923: oid 11 updating version 0 to 77 2024-09-16T03:39:32.715 INFO:tasks.rados.rados.0.smithi157.stdout:1923: oid 11 version 77 is already newer than 76 2024-09-16T03:39:32.715 INFO:tasks.rados.rados.0.smithi157.stdout:1923: oid 11 updating version 77 to 78 2024-09-16T03:39:32.715 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 78 (ObjNum 428 snap 0 seq_num 428) dirty exists 2024-09-16T03:39:32.715 INFO:tasks.rados.rados.0.smithi157.stdout:1923: left oid 11 (ObjNum 428 snap 0 seq_num 428) 2024-09-16T03:39:32.715 INFO:tasks.rados.rados.0.smithi157.stdout:1929: expect (ObjNum 415 snap 0 seq_num 415) 2024-09-16T03:39:32.721 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 636352 2024-09-16T03:39:32.736 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:32.736 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:32.744 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:39:32.800 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:32.798+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:32.840 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 923740 2024-09-16T03:39:32.847 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:32.847 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:32.857 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:39:32.890 INFO:tasks.rados.rados.0.smithi157.stdout:1928: got (0) Success 2024-09-16T03:39:32.890 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 80 (ObjNum 421 snap 0 seq_num 421) dirty exists 2024-09-16T03:39:32.890 INFO:tasks.rados.rados.0.smithi157.stdout:1925: finishing write tid 1 to smithi15736696-12 2024-09-16T03:39:32.890 INFO:tasks.rados.rados.0.smithi157.stdout:1925: finishing write tid 2 to smithi15736696-12 2024-09-16T03:39:32.890 INFO:tasks.rados.rados.0.smithi157.stdout:1918: done (14 left) 2024-09-16T03:39:32.890 INFO:tasks.rados.rados.0.smithi157.stdout:1919: done (13 left) 2024-09-16T03:39:32.890 INFO:tasks.rados.rados.0.smithi157.stdout:1920: done (12 left) 2024-09-16T03:39:32.890 INFO:tasks.rados.rados.0.smithi157.stdout:1923: done (11 left) 2024-09-16T03:39:32.890 INFO:tasks.rados.rados.0.smithi157.stdout:1926: done (10 left) 2024-09-16T03:39:32.890 INFO:tasks.rados.rados.0.smithi157.stdout:1928: done (9 left) 2024-09-16T03:39:32.890 INFO:tasks.rados.rados.0.smithi157.stdout:1929: done (8 left) 2024-09-16T03:39:32.891 INFO:tasks.rados.rados.0.smithi157.stdout:1934: done (7 left) 2024-09-16T03:39:32.891 INFO:tasks.rados.rados.0.smithi157.stdout:1935: read oid 7 snap -1 2024-09-16T03:39:32.891 INFO:tasks.rados.rados.0.smithi157.stdout:1935: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-16T03:39:32.891 INFO:tasks.rados.rados.0.smithi157.stdout:1936: tier_promote oid 42 2024-09-16T03:39:32.891 INFO:tasks.rados.rados.0.smithi157.stdout:1925: finishing write tid 3 to smithi15736696-12 2024-09-16T03:39:32.891 INFO:tasks.rados.rados.0.smithi157.stdout:1925: finishing write tid 4 to smithi15736696-12 2024-09-16T03:39:32.891 INFO:tasks.rados.rados.0.smithi157.stdout:1925: finishing write tid 5 to smithi15736696-12 2024-09-16T03:39:32.891 INFO:tasks.rados.rados.0.smithi157.stdout:1925: oid 12 updating version 0 to 18 2024-09-16T03:39:32.891 INFO:tasks.rados.rados.0.smithi157.stdout:1925: oid 12 updating version 18 to 20 2024-09-16T03:39:32.891 INFO:tasks.rados.rados.0.smithi157.stdout:1925: oid 12 version 20 is already newer than 19 2024-09-16T03:39:32.891 INFO:tasks.rados.rados.0.smithi157.stdout:1925: oid 12 updating version 20 to 21 2024-09-16T03:39:32.891 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 21 (ObjNum 430 snap 0 seq_num 430) dirty exists 2024-09-16T03:39:32.891 INFO:tasks.rados.rados.0.smithi157.stdout:1925: left oid 12 (ObjNum 430 snap 0 seq_num 430) 2024-09-16T03:39:32.892 INFO:tasks.rados.rados.0.smithi157.stdout:1925: done (8 left) 2024-09-16T03:39:32.892 INFO:tasks.rados.rados.0.smithi157.stdout:1937: read oid 43 snap -1 2024-09-16T03:39:32.892 INFO:tasks.rados.rados.0.smithi157.stdout:1937: expect (ObjNum 418 snap 0 seq_num 418) 2024-09-16T03:39:32.892 INFO:tasks.rados.rados.0.smithi157.stdout:1938: read oid 20 snap -1 2024-09-16T03:39:32.892 INFO:tasks.rados.rados.0.smithi157.stdout:1938: expect (ObjNum 399 snap 0 seq_num 399) 2024-09-16T03:39:32.892 INFO:tasks.rados.rados.0.smithi157.stdout:1939: write oid 19 current snap is 0 2024-09-16T03:39:32.892 INFO:tasks.rados.rados.0.smithi157.stdout:1939: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2024-09-16T03:39:32.894 INFO:tasks.rados.rados.0.smithi157.stdout:1939: writing smithi15736696-19 from 494238 to 984425 tid 1 2024-09-16T03:39:32.901 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:32.899+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:32.901 INFO:tasks.rados.rados.0.smithi157.stdout:1939: writing smithi15736696-19 from 1563538 to 2265357 tid 2 2024-09-16T03:39:32.907 INFO:tasks.rados.rados.0.smithi157.stdout:1939: writing smithi15736696-19 from 2818520 to 3294238 tid 3 2024-09-16T03:39:32.908 INFO:tasks.rados.rados.0.smithi157.stdout:1940: read oid 22 snap -1 2024-09-16T03:39:32.908 INFO:tasks.rados.rados.0.smithi157.stdout:1940: expect deleted 2024-09-16T03:39:32.908 INFO:tasks.rados.rados.0.smithi157.stdout:1941: read oid 21 snap -1 2024-09-16T03:39:32.908 INFO:tasks.rados.rados.0.smithi157.stdout:1941: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-16T03:39:32.908 INFO:tasks.rados.rados.0.smithi157.stdout:1942: read oid 46 snap -1 2024-09-16T03:39:32.908 INFO:tasks.rados.rados.0.smithi157.stdout:1942: expect (ObjNum 383 snap 0 seq_num 383) 2024-09-16T03:39:32.908 INFO:tasks.rados.rados.0.smithi157.stdout:1943: read oid 10 snap -1 2024-09-16T03:39:32.908 INFO:tasks.rados.rados.0.smithi157.stdout:1943: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-16T03:39:32.908 INFO:tasks.rados.rados.0.smithi157.stdout:1944: tier_evict oid 44 2024-09-16T03:39:32.908 INFO:tasks.rados.rados.0.smithi157.stdout:1944: tier_evict oid 44 snap -1 2024-09-16T03:39:32.908 INFO:tasks.rados.rados.0.smithi157.stdout:1932: expect (ObjNum 387 snap 0 seq_num 387) 2024-09-16T03:39:32.963 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:32.963 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:32.971 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:39:33.002 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:33.000+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:33.090 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2276982 2024-09-16T03:39:33.100 INFO:tasks.rados.rados.0.smithi157.stdout:1931: expect (ObjNum 423 snap 0 seq_num 423) 2024-09-16T03:39:33.102 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:33.100+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:33.116 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:33.116 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:33.125 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:39:33.195 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:33.209 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:33.207+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:33.245 INFO:tasks.rados.rados.0.smithi157.stdout:1924: finishing write tid 1 to smithi15736696-29 2024-09-16T03:39:33.245 INFO:tasks.rados.rados.0.smithi157.stdout:1924: finishing write tid 2 to smithi15736696-29 2024-09-16T03:39:33.245 INFO:tasks.rados.rados.0.smithi157.stdout:1924: finishing write tid 3 to smithi15736696-29 2024-09-16T03:39:33.245 INFO:tasks.rados.rados.0.smithi157.stdout:1924: finishing write tid 4 to smithi15736696-29 2024-09-16T03:39:33.245 INFO:tasks.rados.rados.0.smithi157.stdout:1924: finishing write tid 5 to smithi15736696-29 2024-09-16T03:39:33.245 INFO:tasks.rados.rados.0.smithi157.stdout:1924: oid 29 updating version 0 to 146 2024-09-16T03:39:33.245 INFO:tasks.rados.rados.0.smithi157.stdout:1924: oid 29 updating version 146 to 148 2024-09-16T03:39:33.245 INFO:tasks.rados.rados.0.smithi157.stdout:1924: oid 29 version 148 is already newer than 147 2024-09-16T03:39:33.246 INFO:tasks.rados.rados.0.smithi157.stdout:1924: oid 29 version 148 is already newer than 145 2024-09-16T03:39:33.246 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 148 (ObjNum 429 snap 0 seq_num 429) dirty exists 2024-09-16T03:39:33.246 INFO:tasks.rados.rados.0.smithi157.stdout:1924: left oid 29 (ObjNum 429 snap 0 seq_num 429) 2024-09-16T03:39:33.246 INFO:tasks.rados.rados.0.smithi157.stdout:1927: finishing write tid 2 to smithi15736696-3 2024-09-16T03:39:33.246 INFO:tasks.rados.rados.0.smithi157.stdout:1927: finishing write tid 3 to smithi15736696-3 2024-09-16T03:39:33.246 INFO:tasks.rados.rados.0.smithi157.stdout:1930: finishing write tid 1 to smithi15736696-9 2024-09-16T03:39:33.246 INFO:tasks.rados.rados.0.smithi157.stdout:1930: finishing write tid 2 to smithi15736696-9 2024-09-16T03:39:33.246 INFO:tasks.rados.rados.0.smithi157.stdout:1927: finishing write tid 4 to smithi15736696-3 2024-09-16T03:39:33.246 INFO:tasks.rados.rados.0.smithi157.stdout:1927: finishing write tid 5 to smithi15736696-3 2024-09-16T03:39:33.246 INFO:tasks.rados.rados.0.smithi157.stdout:1927: oid 3 updating version 0 to 79 2024-09-16T03:39:33.246 INFO:tasks.rados.rados.0.smithi157.stdout:1927: oid 3 updating version 79 to 82 2024-09-16T03:39:33.246 INFO:tasks.rados.rados.0.smithi157.stdout:1927: oid 3 version 82 is already newer than 81 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:1927: oid 3 version 82 is already newer than 80 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 3 v 82 (ObjNum 431 snap 0 seq_num 431) dirty exists 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:1927: left oid 3 (ObjNum 431 snap 0 seq_num 431) 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:1930: finishing write tid 3 to smithi15736696-9 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:1930: finishing write tid 4 to smithi15736696-9 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:1930: finishing write tid 5 to smithi15736696-9 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:1930: oid 9 updating version 0 to 90 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:1930: oid 9 version 90 is already newer than 89 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:1930: oid 9 updating version 90 to 91 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:1930: oid 9 updating version 91 to 92 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 92 (ObjNum 432 snap 0 seq_num 432) dirty exists 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:1930: left oid 9 (ObjNum 432 snap 0 seq_num 432) 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:1933: finishing set_chunk to oid 14 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 14 v 55 (ObjNum 398 snap 0 seq_num 398) dirty exists 2024-09-16T03:39:33.247 INFO:tasks.rados.rados.0.smithi157.stdout:1924: done (15 left) 2024-09-16T03:39:33.248 INFO:tasks.rados.rados.0.smithi157.stdout:1927: done (14 left) 2024-09-16T03:39:33.248 INFO:tasks.rados.rados.0.smithi157.stdout:1930: done (13 left) 2024-09-16T03:39:33.248 INFO:tasks.rados.rados.0.smithi157.stdout:1931: done (12 left) 2024-09-16T03:39:33.248 INFO:tasks.rados.rados.0.smithi157.stdout:1932: done (11 left) 2024-09-16T03:39:33.248 INFO:tasks.rados.rados.0.smithi157.stdout:1933: done (10 left) 2024-09-16T03:39:33.248 INFO:tasks.rados.rados.0.smithi157.stdout:1945: delete oid 16 current snap is 0 2024-09-16T03:39:33.248 INFO:tasks.rados.rados.0.smithi157.stdout:1944: got (22) Invalid argument 2024-09-16T03:39:33.248 INFO:tasks.rados.rados.0.smithi157.stdout:1935: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-16T03:39:33.270 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:33.270 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:33.279 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:39:33.323 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:33.322+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:33.357 INFO:tasks.rados.rados.0.smithi157.stdout:1937: expect (ObjNum 418 snap 0 seq_num 418) 2024-09-16T03:39:33.367 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:33.365+0000 7f4214d04640 -1 osd.8 153 handle_osd_map injecting map crc failure 2024-09-16T03:39:33.382 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:39:33.385 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:33.385 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:33.394 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:39:33.405 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:33.403+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:33.429 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:33.505 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:33.503+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:33.506 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:33.504+0000 7f9288166640 -1 osd.14 153 handle_osd_map injecting map crc failure 2024-09-16T03:39:33.534 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:33.535 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:33.543 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:39:33.564 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2835784 2024-09-16T03:39:33.572 INFO:tasks.rados.rados.0.smithi157.stdout:1938: expect (ObjNum 399 snap 0 seq_num 399) 2024-09-16T03:39:33.607 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:33.605+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:33.611 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:33.648 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:33.648 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:33.656 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:39:33.666 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 9, 1] out_osds: [5, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:39:33.666 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:39:33.666 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:39:33.666 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-16T03:39:33.667 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-16T03:39:33.683 INFO:tasks.rados.rados.0.smithi157.stdout:1939: finishing write tid 1 to smithi15736696-19 2024-09-16T03:39:33.683 INFO:tasks.rados.rados.0.smithi157.stdout:1941: expect (ObjNum 366 snap 0 seq_num 366) 2024-09-16T03:39:33.683 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:33.726 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:33.725+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:33.729 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1436073 2024-09-16T03:39:33.777 INFO:tasks.rados.rados.0.smithi157.stdout:1942: expect (ObjNum 383 snap 0 seq_num 383) 2024-09-16T03:39:33.777 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:33.789 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:33.789 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:33.798 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:39:33.804 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:33.804 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:33.804 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:33.804 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:33.804 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:33.804 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:33.804 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:33.804 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:33.804 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:33.804 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:33.804 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:33.804 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:33.807 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:33.807 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:33.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-09-16T03:39:33.809 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:33.807+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:33.851 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:33.892 INFO:tasks.rados.rados.0.smithi157.stdout:1936: got (2) No such file or directory 2024-09-16T03:39:33.892 INFO:tasks.rados.rados.0.smithi157.stdout:1936: got expected ENOENT (src dne) 2024-09-16T03:39:33.893 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 77 (ObjNum 129 snap 0 seq_num 3487771760) clean dne 2024-09-16T03:39:33.893 INFO:tasks.rados.rados.0.smithi157.stdout:1939: finishing write tid 2 to smithi15736696-19 2024-09-16T03:39:33.893 INFO:tasks.rados.rados.0.smithi157.stdout:1935: done (10 left) 2024-09-16T03:39:33.893 INFO:tasks.rados.rados.0.smithi157.stdout:1936: done (9 left) 2024-09-16T03:39:33.893 INFO:tasks.rados.rados.0.smithi157.stdout:1937: done (8 left) 2024-09-16T03:39:33.893 INFO:tasks.rados.rados.0.smithi157.stdout:1938: done (7 left) 2024-09-16T03:39:33.893 INFO:tasks.rados.rados.0.smithi157.stdout:1940: done (6 left) 2024-09-16T03:39:33.893 INFO:tasks.rados.rados.0.smithi157.stdout:1941: done (5 left) 2024-09-16T03:39:33.893 INFO:tasks.rados.rados.0.smithi157.stdout:1942: done (4 left) 2024-09-16T03:39:33.893 INFO:tasks.rados.rados.0.smithi157.stdout:1944: done (3 left) 2024-09-16T03:39:33.893 INFO:tasks.rados.rados.0.smithi157.stdout:1945: done (2 left) 2024-09-16T03:39:33.893 INFO:tasks.rados.rados.0.smithi157.stdout:1946: read oid 40 snap -1 2024-09-16T03:39:33.893 INFO:tasks.rados.rados.0.smithi157.stdout:1946: expect (ObjNum 424 snap 0 seq_num 424) 2024-09-16T03:39:33.894 INFO:tasks.rados.rados.0.smithi157.stdout:1947: read oid 43 snap -1 2024-09-16T03:39:33.894 INFO:tasks.rados.rados.0.smithi157.stdout:1947: expect (ObjNum 418 snap 0 seq_num 418) 2024-09-16T03:39:33.894 INFO:tasks.rados.rados.0.smithi157.stdout:1948: write oid 31 current snap is 0 2024-09-16T03:39:33.894 INFO:tasks.rados.rados.0.smithi157.stdout:1939: finishing write tid 3 to smithi15736696-19 2024-09-16T03:39:33.894 INFO:tasks.rados.rados.0.smithi157.stdout:1939: finishing write tid 4 to smithi15736696-19 2024-09-16T03:39:33.894 INFO:tasks.rados.rados.0.smithi157.stdout:1939: finishing write tid 5 to smithi15736696-19 2024-09-16T03:39:33.894 INFO:tasks.rados.rados.0.smithi157.stdout:1939: oid 19 updating version 0 to 166 2024-09-16T03:39:33.894 INFO:tasks.rados.rados.0.smithi157.stdout:1939: oid 19 updating version 166 to 168 2024-09-16T03:39:33.894 INFO:tasks.rados.rados.0.smithi157.stdout:1939: oid 19 version 168 is already newer than 167 2024-09-16T03:39:33.894 INFO:tasks.rados.rados.0.smithi157.stdout:1939: oid 19 version 168 is already newer than 165 2024-09-16T03:39:33.894 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 19 v 168 (ObjNum 434 snap 0 seq_num 434) dirty exists 2024-09-16T03:39:33.894 INFO:tasks.rados.rados.0.smithi157.stdout:1939: left oid 19 (ObjNum 434 snap 0 seq_num 434) 2024-09-16T03:39:33.894 INFO:tasks.rados.rados.0.smithi157.stdout:1948: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2024-09-16T03:39:33.900 INFO:tasks.rados.rados.0.smithi157.stdout:1948: writing smithi15736696-31 from 527852 to 1168537 tid 1 2024-09-16T03:39:33.909 INFO:tasks.rados.rados.0.smithi157.stdout:1948: writing smithi15736696-31 from 1680502 to 2269637 tid 2 2024-09-16T03:39:33.909 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:33.907+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:33.915 INFO:tasks.rados.rados.0.smithi157.stdout:1948: writing smithi15736696-31 from 2757013 to 3327852 tid 3 2024-09-16T03:39:33.916 INFO:tasks.rados.rados.0.smithi157.stdout:1939: done (4 left) 2024-09-16T03:39:33.916 INFO:tasks.rados.rados.0.smithi157.stdout:1949: set_chunk oid 33 target oid 2024-09-16T03:39:33.948 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:33.948 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:33.948 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:33.948 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:33.948 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:33.948 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:33.948 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:33.948 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:33.948 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:33.949 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:33.949 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:33.949 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:33.951 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:33.951 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:33.952 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-16T03:39:33.952 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:33.952 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:33.960 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:39:33.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:39:34.010 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:34.008+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:34.018 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:34.015+0000 7fd101e6f640 -1 osd.2 153 handle_osd_map injecting map crc failure 2024-09-16T03:39:34.019 INFO:tasks.rados.rados.0.smithi157.stdout:1949: set_chunk oid 33 offset: 1116160 length: 389120 target oid 4342a13fc2af79ea36f7ba8672e7ec1dfe0433b5ce6796e6689d75b33a6eed07 offset: 0 snap -1 2024-09-16T03:39:34.019 INFO:tasks.rados.rados.0.smithi157.stdout:1950: tier_promote oid 42 2024-09-16T03:39:34.019 INFO:tasks.rados.rados.0.smithi157.stdout:1946: expect (ObjNum 424 snap 0 seq_num 424) 2024-09-16T03:39:34.036 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1363126 2024-09-16T03:39:34.097 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1660023 2024-09-16T03:39:34.108 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:34.108 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:34.111 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:34.109+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:34.117 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:39:34.125 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:34.125 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:34.132 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:34.132 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:34.132 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:34.132 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:34.132 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:34.132 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:34.132 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:34.132 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:34.132 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:34.132 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:34.133 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:34.133 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:34.134 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:34.135 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:34.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 'trim stale osdmaps' 2024-09-16T03:39:34.137 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:34.143 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:34.144 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:34.161 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:34.161 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:34.178 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:34.194 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:34.194 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:34.203 INFO:tasks.rados.rados.0.smithi157.stdout:1943: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-16T03:39:34.203 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:34.211 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:34.211 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:34.212 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:34.210+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:34.228 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:34.228 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:34.241 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:34.241 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:34.248 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:34.248 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:34.250 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:39:34.265 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:34.265 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:34.274 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:34.275 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:34.275 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:34.275 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:34.275 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:34.275 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:34.276 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:34.276 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:34.276 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:34.276 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:34.276 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:34.276 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:34.277 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:34.277 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:34.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 'trim stale osdmaps' 2024-09-16T03:39:34.282 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:34.283 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:34.299 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:34.300 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:34.310 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 556788 2024-09-16T03:39:34.313 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:34.311+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:34.317 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:34.317 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:34.334 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:34.334 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:34.351 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:34.351 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:34.354 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:34.354 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:34.354 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:34.354 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:34.354 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:34.356 INFO:tasks.rados.rados.0.smithi157.stdout:1947: expect (ObjNum 418 snap 0 seq_num 418) 2024-09-16T03:39:34.356 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:34.363 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:39:34.368 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:34.368 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:34.386 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:34.386 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:34.413 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:34.412+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:34.421 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:34.421 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:34.422 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:34.422 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:34.422 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:34.422 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:34.422 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:34.422 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:34.422 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:34.422 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:34.422 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:34.422 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:34.424 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:34.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 'trim stale osdmaps' 2024-09-16T03:39:34.434 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:34.470 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:34.470 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:34.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:34.470 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:34.470 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.470 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T03:39:34.470 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:39:34.470 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 163.390253782, 2024-09-16T03:39:34.470 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.88203620400000005, 2024-09-16T03:39:34.470 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.470 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.470 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.471 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.471 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.471 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.471 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:39:34.471 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.471 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.471 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.471 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.471 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:39:34.471 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.472 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.472 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:39:34.472 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.472 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.472 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076767+0000", 2024-09-16T03:39:34.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9300000000000002e-06 2024-09-16T03:39:34.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076774+0000", 2024-09-16T03:39:34.473 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6630000000000004e-06 2024-09-16T03:39:34.473 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.473 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.473 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.473 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076785+0000", 2024-09-16T03:39:34.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1221e-05 2024-09-16T03:39:34.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076796+0000", 2024-09-16T03:39:34.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0428000000000001e-05 2024-09-16T03:39:34.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.474 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.474 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.474 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076856+0000", 2024-09-16T03:39:34.474 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:39:34.474 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.475 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:34.475 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076863+0000", 2024-09-16T03:39:34.475 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9840000000000004e-06 2024-09-16T03:39:34.475 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.475 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855770+0000", 2024-09-16T03:39:34.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.77890736999999999 2024-09-16T03:39:34.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855902+0000", 2024-09-16T03:39:34.476 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013207 2024-09-16T03:39:34.476 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.476 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.476 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.476 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.856054+0000", 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015220899999999999 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.858407+0000", 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023530069999999998 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958598+0000", 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100190194 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.477 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958716+0000", 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118589 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958761+0000", 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5120000000000002e-05 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958802+0000", 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0383999999999998e-05 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.478 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 159.871837605, 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70508991300000001, 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.479 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 52 2024-09-16T03:39:34.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.480 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:34.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.480 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.480 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:34.480 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.480 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.480 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6300000000000005e-07 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595187+0000", 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9790000000000001e-06 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595194+0000", 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6490000000000002e-06 2024-09-16T03:39:34.481 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595208+0000", 2024-09-16T03:39:34.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4139e-05 2024-09-16T03:39:34.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.482 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.482 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801931+0000", 2024-09-16T03:39:34.482 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.206723083 2024-09-16T03:39:34.482 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.482 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801990+0000", 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8625999999999997e-05 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802066+0000", 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6553999999999999e-05 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802504+0000", 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043775200000000002 2024-09-16T03:39:34.483 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.199998+0000", 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39749365599999997 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300151+0000", 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015310099999999 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300210+0000", 2024-09-16T03:39:34.484 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9376999999999997e-05 2024-09-16T03:39:34.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.485 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.485 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.485 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300271+0000", 2024-09-16T03:39:34.485 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1334000000000006e-05 2024-09-16T03:39:34.485 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.485 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.485 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.485 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.486 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.486 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.486 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:34.486 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 158.09694662199999, 2024-09-16T03:39:34.486 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60623559900000001, 2024-09-16T03:39:34.486 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.486 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.486 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.486 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.486 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.486 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.486 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 71 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.487 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370077+0000", 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9940000000000002e-06 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370081+0000", 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5850000000000001e-06 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.488 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370087+0000", 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673522+0000", 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.303434969 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673567+0000", 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5080000000000002e-05 2024-09-16T03:39:34.489 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673651+0000", 2024-09-16T03:39:34.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3652999999999993e-05 2024-09-16T03:39:34.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775115+0000", 2024-09-16T03:39:34.490 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101464178 2024-09-16T03:39:34.490 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.490 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.490 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.490 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975918+0000", 2024-09-16T03:39:34.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20080318699999999 2024-09-16T03:39:34.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976254+0000", 2024-09-16T03:39:34.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033619200000000002 2024-09-16T03:39:34.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.491 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.491 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976285+0000", 2024-09-16T03:39:34.491 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0679000000000002e-05 2024-09-16T03:39:34.491 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.491 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.492 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.492 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976308+0000", 2024-09-16T03:39:34.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3051e-05 2024-09-16T03:39:34.492 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.492 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.492 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.492 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.492 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.492 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.492 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:34.492 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 158.09692326199999, 2024-09-16T03:39:34.492 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60632925100000001, 2024-09-16T03:39:34.492 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 72 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.493 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370150+0000", 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3906999999999997e-05 2024-09-16T03:39:34.494 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370157+0000", 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370162+0000", 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7249999999999997e-06 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673969+0000", 2024-09-16T03:39:34.495 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30380713199999998 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674030+0000", 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1141000000000003e-05 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674124+0000", 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3859999999999999e-05 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.496 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775687+0000", 2024-09-16T03:39:34.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10156293299999999 2024-09-16T03:39:34.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976103+0000", 2024-09-16T03:39:34.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20041631400000001 2024-09-16T03:39:34.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.497 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.497 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976355+0000", 2024-09-16T03:39:34.497 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000251611 2024-09-16T03:39:34.497 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.497 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976385+0000", 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0883999999999998e-05 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976425+0000", 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9666000000000003e-05 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.498 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.499 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:39:34.499 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 158.096850778, 2024-09-16T03:39:34.499 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60763126999999995, 2024-09-16T03:39:34.499 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.499 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.499 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.499 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.499 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.499 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.499 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 73 2024-09-16T03:39:34.499 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.499 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:39:34.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:39:34.500 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.500 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.500 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.500 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.500 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370169+0000", 2024-09-16T03:39:34.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6199999999999998e-07 2024-09-16T03:39:34.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370171+0000", 2024-09-16T03:39:34.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:39:34.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.501 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.501 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.501 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370175+0000", 2024-09-16T03:39:34.501 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.596e-06 2024-09-16T03:39:34.501 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470295+0000", 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100119967 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775148+0000", 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30485311799999998 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775203+0000", 2024-09-16T03:39:34.502 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4592999999999997e-05 2024-09-16T03:39:34.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775293+0000", 2024-09-16T03:39:34.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0859999999999994e-05 2024-09-16T03:39:34.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.503 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976223+0000", 2024-09-16T03:39:34.503 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20092996599999999 2024-09-16T03:39:34.503 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.503 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.503 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976463+0000", 2024-09-16T03:39:34.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000239695 2024-09-16T03:39:34.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977730+0000", 2024-09-16T03:39:34.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012667690000000001 2024-09-16T03:39:34.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.504 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.504 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.504 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977773+0000", 2024-09-16T03:39:34.504 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2604000000000001e-05 2024-09-16T03:39:34.504 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.505 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.505 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977800+0000", 2024-09-16T03:39:34.505 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7137000000000001e-05 2024-09-16T03:39:34.505 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.505 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.505 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.505 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.505 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.505 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.505 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:39:34.505 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 157.99671192299999, 2024-09-16T03:39:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60633810099999996, 2024-09-16T03:39:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 74 2024-09-16T03:39:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470308+0000", 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0950000000000001e-06 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.507 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470313+0000", 2024-09-16T03:39:34.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:39:34.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470320+0000", 2024-09-16T03:39:34.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5089999999999999e-06 2024-09-16T03:39:34.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.508 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.508 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.508 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470327+0000", 2024-09-16T03:39:34.508 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:39:34.508 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976483+0000", 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50615579499999996 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976533+0000", 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0096000000000003e-05 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976643+0000", 2024-09-16T03:39:34.509 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109481 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977808+0000", 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011649869999999999 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.981699+0000", 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003891579 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.510 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076496+0000", 2024-09-16T03:39:34.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.094797333999999997 2024-09-16T03:39:34.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076583+0000", 2024-09-16T03:39:34.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6571999999999997e-05 2024-09-16T03:39:34.511 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.511 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.511 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.511 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076645+0000", 2024-09-16T03:39:34.511 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2331999999999999e-05 2024-09-16T03:39:34.511 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.511 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.512 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.512 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.512 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.512 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.512 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:39:34.512 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 157.99668178799999, 2024-09-16T03:39:34.512 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71251765099999997, 2024-09-16T03:39:34.512 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.512 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.512 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.512 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.512 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.513 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.513 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 75 2024-09-16T03:39:34.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.513 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:39:34.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.513 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.513 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.513 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:39:34.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470338+0000", 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2500000000000005e-07 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470389+0000", 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.134e-05 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470397+0000", 2024-09-16T03:39:34.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8140000000000001e-06 2024-09-16T03:39:34.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570537+0000", 2024-09-16T03:39:34.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100139425 2024-09-16T03:39:34.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977018+0000", 2024-09-16T03:39:34.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40648102899999999 2024-09-16T03:39:34.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977063+0000", 2024-09-16T03:39:34.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5247000000000002e-05 2024-09-16T03:39:34.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977143+0000", 2024-09-16T03:39:34.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0054000000000003e-05 2024-09-16T03:39:34.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.978132+0000", 2024-09-16T03:39:34.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00098896399999999995 2024-09-16T03:39:34.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182725+0000", 2024-09-16T03:39:34.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.204592626 2024-09-16T03:39:34.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182783+0000", 2024-09-16T03:39:34.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8285000000000002e-05 2024-09-16T03:39:34.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182825+0000", 2024-09-16T03:39:34.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2398999999999998e-05 2024-09-16T03:39:34.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182855+0000", 2024-09-16T03:39:34.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9842999999999999e-05 2024-09-16T03:39:34.518 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.518 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.518 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.519 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.519 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:39:34.519 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 157.896455221, 2024-09-16T03:39:34.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71263271800000005, 2024-09-16T03:39:34.519 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.519 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.519 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.519 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.519 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.519 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.519 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 76 2024-09-16T03:39:34.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.520 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:39:34.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.520 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:39:34.520 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.520 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.520 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.520 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570566+0000", 2024-09-16T03:39:34.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:39:34.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570577+0000", 2024-09-16T03:39:34.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0672999999999999e-05 2024-09-16T03:39:34.521 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.521 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.521 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.521 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570593+0000", 2024-09-16T03:39:34.521 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5979999999999999e-05 2024-09-16T03:39:34.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570604+0000", 2024-09-16T03:39:34.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.077e-05 2024-09-16T03:39:34.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.522 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979890+0000", 2024-09-16T03:39:34.522 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40928624200000002 2024-09-16T03:39:34.522 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.522 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.522 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979945+0000", 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5651e-05 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980038+0000", 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2603999999999996e-05 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980669+0000", 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063121900000000003 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.523 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283050+0000", 2024-09-16T03:39:34.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30238120800000001 2024-09-16T03:39:34.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283132+0000", 2024-09-16T03:39:34.524 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1841999999999993e-05 2024-09-16T03:39:34.524 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.524 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.524 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.524 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283168+0000", 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5438999999999997e-05 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283197+0000", 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8960000000000001e-05 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:39:34.525 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 156.08192012200001, 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70799990400000001, 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 94 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:39:34.526 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:39:34.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.527 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.527 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385100+0000", 2024-09-16T03:39:34.527 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9599999999999998e-07 2024-09-16T03:39:34.527 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.527 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385104+0000", 2024-09-16T03:39:34.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7579999999999998e-06 2024-09-16T03:39:34.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385128+0000", 2024-09-16T03:39:34.528 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4700999999999999e-05 2024-09-16T03:39:34.528 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.528 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.528 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.528 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385139+0000", 2024-09-16T03:39:34.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0587e-05 2024-09-16T03:39:34.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787499+0000", 2024-09-16T03:39:34.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40235976099999998 2024-09-16T03:39:34.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.529 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.529 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.529 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787545+0000", 2024-09-16T03:39:34.529 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6418999999999997e-05 2024-09-16T03:39:34.529 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787623+0000", 2024-09-16T03:39:34.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7897000000000003e-05 2024-09-16T03:39:34.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.530 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.530 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.530 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888605+0000", 2024-09-16T03:39:34.530 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100982321 2024-09-16T03:39:34.530 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990721+0000", 2024-09-16T03:39:34.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102115922 2024-09-16T03:39:34.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093019+0000", 2024-09-16T03:39:34.531 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10229788500000001 2024-09-16T03:39:34.531 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.531 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.531 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.531 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093051+0000", 2024-09-16T03:39:34.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2255999999999997e-05 2024-09-16T03:39:34.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.532 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.532 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093099+0000", 2024-09-16T03:39:34.532 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7500999999999998e-05 2024-09-16T03:39:34.532 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.532 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.532 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.532 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.532 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.532 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.533 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:39:34.533 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 156.08186879900001, 2024-09-16T03:39:34.533 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80404594699999998, 2024-09-16T03:39:34.533 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.533 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.533 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.533 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.533 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.533 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.533 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 95 2024-09-16T03:39:34.533 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.533 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.533 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385151+0000", 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.534 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385165+0000", 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4175999999999999e-05 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385175+0000", 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385183+0000", 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:39:34.535 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888643+0000", 2024-09-16T03:39:34.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50345988200000003 2024-09-16T03:39:34.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888695+0000", 2024-09-16T03:39:34.536 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2834999999999998e-05 2024-09-16T03:39:34.536 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.536 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.536 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.536 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888802+0000", 2024-09-16T03:39:34.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010697899999999999 2024-09-16T03:39:34.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990658+0000", 2024-09-16T03:39:34.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185585699999999 2024-09-16T03:39:34.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.537 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.537 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990917+0000", 2024-09-16T03:39:34.537 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025853600000000002 2024-09-16T03:39:34.537 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.537 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189054+0000", 2024-09-16T03:39:34.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198136812 2024-09-16T03:39:34.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.538 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189135+0000", 2024-09-16T03:39:34.538 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1482000000000004e-05 2024-09-16T03:39:34.538 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.538 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.538 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.539 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189196+0000", 2024-09-16T03:39:34.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1175999999999998e-05 2024-09-16T03:39:34.539 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.539 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.539 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.539 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.539 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.539 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.539 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:39:34.539 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 156.081828172, 2024-09-16T03:39:34.539 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80412488400000004, 2024-09-16T03:39:34.540 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.540 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.540 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.540 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.540 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.540 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.540 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 96 2024-09-16T03:39:34.540 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.540 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.540 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.540 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.540 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:39:34.540 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:39:34.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.541 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385192+0000", 2024-09-16T03:39:34.541 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:39:34.541 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.541 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.541 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385267+0000", 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5535000000000001e-05 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385277+0000", 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6239999999999997e-06 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385284+0000", 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4800000000000004e-06 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.542 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989283+0000", 2024-09-16T03:39:34.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60399849500000002 2024-09-16T03:39:34.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989410+0000", 2024-09-16T03:39:34.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127298 2024-09-16T03:39:34.543 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.543 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.543 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.543 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989615+0000", 2024-09-16T03:39:34.543 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000205045 2024-09-16T03:39:34.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991036+0000", 2024-09-16T03:39:34.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001421042 2024-09-16T03:39:34.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.544 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189228+0000", 2024-09-16T03:39:34.544 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19819188500000001 2024-09-16T03:39:34.544 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.544 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.544 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189249+0000", 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0996999999999999e-05 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189293+0000", 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3912000000000002e-05 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189316+0000", 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2739e-05 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.545 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 156.08172444100001, 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80642264100000005, 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.546 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 97 2024-09-16T03:39:34.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.547 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:39:34.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.547 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.547 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.547 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:39:34.547 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.547 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385296+0000", 2024-09-16T03:39:34.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.006e-06 2024-09-16T03:39:34.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385301+0000", 2024-09-16T03:39:34.548 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2630000000000003e-06 2024-09-16T03:39:34.548 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.548 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.548 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.548 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385308+0000", 2024-09-16T03:39:34.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6030000000000001e-06 2024-09-16T03:39:34.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485459+0000", 2024-09-16T03:39:34.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150929 2024-09-16T03:39:34.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.549 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.549 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990238+0000", 2024-09-16T03:39:34.549 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50477992000000005 2024-09-16T03:39:34.549 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.549 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990287+0000", 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8863999999999998e-05 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990377+0000", 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9900000000000003e-05 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093109+0000", 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102732055 2024-09-16T03:39:34.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189351+0000", 2024-09-16T03:39:34.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096241766000000006 2024-09-16T03:39:34.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191570+0000", 2024-09-16T03:39:34.551 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00221915 2024-09-16T03:39:34.551 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.551 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.551 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.551 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191653+0000", 2024-09-16T03:39:34.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3046000000000004e-05 2024-09-16T03:39:34.552 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.552 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.552 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.552 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191717+0000", 2024-09-16T03:39:34.552 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4139000000000004e-05 2024-09-16T03:39:34.552 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.552 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.552 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.553 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.553 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.553 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:39:34.553 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 155.98153398400001, 2024-09-16T03:39:34.553 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90688294999999997, 2024-09-16T03:39:34.553 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.554 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.554 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.554 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.554 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.554 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.554 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 98 2024-09-16T03:39:34.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.554 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.554 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.554 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.554 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:39:34.554 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:39:34.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.555 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485487+0000", 2024-09-16T03:39:34.555 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1799999999999999e-06 2024-09-16T03:39:34.555 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.555 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.555 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485498+0000", 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0648999999999999e-05 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485514+0000", 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5943999999999998e-05 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485525+0000", 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.065e-05 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.556 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991102+0000", 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50557748400000002 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991161+0000", 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8801999999999997e-05 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991255+0000", 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3936000000000003e-05 2024-09-16T03:39:34.557 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093112+0000", 2024-09-16T03:39:34.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185709499999999 2024-09-16T03:39:34.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291911+0000", 2024-09-16T03:39:34.558 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19879897699999999 2024-09-16T03:39:34.558 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.558 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.558 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.558 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392167+0000", 2024-09-16T03:39:34.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10025634999999999 2024-09-16T03:39:34.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392249+0000", 2024-09-16T03:39:34.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2175000000000005e-05 2024-09-16T03:39:34.559 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.559 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.559 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.559 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392368+0000", 2024-09-16T03:39:34.559 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118708 2024-09-16T03:39:34.559 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.560 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.560 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.560 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.560 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.560 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.560 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:39:34.560 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 155.981480361, 2024-09-16T03:39:34.560 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90707203199999997, 2024-09-16T03:39:34.560 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.560 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.561 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.561 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.561 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 99 2024-09-16T03:39:34.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.561 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.561 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.561 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.561 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:39:34.561 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.561 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:39:34.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.562 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485540+0000", 2024-09-16T03:39:34.562 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-06 2024-09-16T03:39:34.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.562 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.562 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485661+0000", 2024-09-16T03:39:34.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120835 2024-09-16T03:39:34.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485680+0000", 2024-09-16T03:39:34.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8885999999999998e-05 2024-09-16T03:39:34.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.563 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.563 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.563 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485693+0000", 2024-09-16T03:39:34.563 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3297e-05 2024-09-16T03:39:34.563 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991546+0000", 2024-09-16T03:39:34.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.505853098 2024-09-16T03:39:34.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991598+0000", 2024-09-16T03:39:34.564 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1921e-05 2024-09-16T03:39:34.564 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.564 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.564 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.564 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991681+0000", 2024-09-16T03:39:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2502000000000004e-05 2024-09-16T03:39:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093115+0000", 2024-09-16T03:39:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10143424299999999 2024-09-16T03:39:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291977+0000", 2024-09-16T03:39:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198862172 2024-09-16T03:39:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.565 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392541+0000", 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100563758 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392579+0000", 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7796999999999997e-05 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392611+0000", 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2073999999999998e-05 2024-09-16T03:39:34.566 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.567 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.567 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.567 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.567 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.567 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:34.567 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 155.981301468, 2024-09-16T03:39:34.567 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0073149939999999, 2024-09-16T03:39:34.567 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.567 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.567 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.567 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.568 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.568 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.568 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 100 2024-09-16T03:39:34.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.568 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:34.568 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.568 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.568 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.568 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.568 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7400000000000003e-07 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485721+0000", 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4530000000000001e-06 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.569 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485726+0000", 2024-09-16T03:39:34.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0590000000000002e-06 2024-09-16T03:39:34.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485729+0000", 2024-09-16T03:39:34.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:39:34.570 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.570 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.570 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.570 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092342+0000", 2024-09-16T03:39:34.570 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60661343700000003 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092469+0000", 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012658799999999999 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092671+0000", 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020206 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.571 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392452+0000", 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29978163699999999 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392486+0000", 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3736999999999999e-05 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492812+0000", 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100325621 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.572 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492901+0000", 2024-09-16T03:39:34.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8702000000000006e-05 2024-09-16T03:39:34.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.573 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.573 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.493033+0000", 2024-09-16T03:39:34.573 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000132185 2024-09-16T03:39:34.573 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.573 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.573 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.573 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.573 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.574 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.574 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:39:34.574 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 153.80615094699999, 2024-09-16T03:39:34.574 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60874581500000002, 2024-09-16T03:39:34.574 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.574 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.574 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.574 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.574 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.574 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.574 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 123 2024-09-16T03:39:34.574 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660869+0000", 2024-09-16T03:39:34.575 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9400000000000005e-07 2024-09-16T03:39:34.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660921+0000", 2024-09-16T03:39:34.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2312000000000003e-05 2024-09-16T03:39:34.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.576 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660927+0000", 2024-09-16T03:39:34.576 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9329999999999998e-06 2024-09-16T03:39:34.576 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.576 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.576 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660931+0000", 2024-09-16T03:39:34.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:39:34.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.967991+0000", 2024-09-16T03:39:34.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30706032700000002 2024-09-16T03:39:34.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.577 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.577 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.577 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968039+0000", 2024-09-16T03:39:34.577 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7477e-05 2024-09-16T03:39:34.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968117+0000", 2024-09-16T03:39:34.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8589000000000003e-05 2024-09-16T03:39:34.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.578 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067678+0000", 2024-09-16T03:39:34.578 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099560406000000004 2024-09-16T03:39:34.578 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.578 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.578 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170175+0000", 2024-09-16T03:39:34.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102496927 2024-09-16T03:39:34.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269491+0000", 2024-09-16T03:39:34.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099316205000000005 2024-09-16T03:39:34.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.579 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.579 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.579 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269574+0000", 2024-09-16T03:39:34.579 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2903000000000002e-05 2024-09-16T03:39:34.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.580 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.580 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269614+0000", 2024-09-16T03:39:34.580 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0188000000000002e-05 2024-09-16T03:39:34.580 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.580 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.580 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.580 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.580 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.580 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:34.580 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 153.80608224900001, 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60879245500000001, 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 124 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.581 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:39:34.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:39:34.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.582 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.582 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660938+0000", 2024-09-16T03:39:34.582 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:39:34.582 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.582 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660940+0000", 2024-09-16T03:39:34.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2529999999999999e-06 2024-09-16T03:39:34.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660943+0000", 2024-09-16T03:39:34.583 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.01e-06 2024-09-16T03:39:34.583 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.583 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.583 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.583 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660946+0000", 2024-09-16T03:39:34.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8159999999999998e-06 2024-09-16T03:39:34.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012417+0000", 2024-09-16T03:39:34.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.35147120599999998 2024-09-16T03:39:34.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.584 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.584 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012497+0000", 2024-09-16T03:39:34.584 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9963000000000007e-05 2024-09-16T03:39:34.584 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.584 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012585+0000", 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067680+0000", 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055095203000000002 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170243+0000", 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10256290799999999 2024-09-16T03:39:34.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269662+0000", 2024-09-16T03:39:34.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099418896000000007 2024-09-16T03:39:34.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.586 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269701+0000", 2024-09-16T03:39:34.586 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8829999999999999e-05 2024-09-16T03:39:34.586 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.586 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.586 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269729+0000", 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8569e-05 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1550 4.13 4:c8492bfc:::smithi15736696-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 108.800354109, 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62706974999999998, 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.587 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1550 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.588 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:34.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:34.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0699999999999998e-07 2024-09-16T03:39:34.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.589 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.589 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666669+0000", 2024-09-16T03:39:34.589 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.393e-06 2024-09-16T03:39:34.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.589 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666673+0000", 2024-09-16T03:39:34.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:39:34.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666682+0000", 2024-09-16T03:39:34.590 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6510000000000007e-06 2024-09-16T03:39:34.590 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.590 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.590 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.590 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.913940+0000", 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.24725763000000001 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.914034+0000", 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3923000000000002e-05 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293650+0000", 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.37961662200000001 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.591 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293688+0000", 2024-09-16T03:39:34.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8096000000000003e-05 2024-09-16T03:39:34.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.592 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.592 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293735+0000", 2024-09-16T03:39:34.592 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6451e-05 2024-09-16T03:39:34.592 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.592 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.592 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.592 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.592 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.592 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:77 3.2 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:39:34.593 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:39:34.593 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 94.388477488000007, 2024-09-16T03:39:34.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.61452436499999996, 2024-09-16T03:39:34.593 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.593 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.593 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:34.593 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.593 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.593 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:39:34.593 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.593 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.593 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:39:34.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.594 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.594 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078543+0000", 2024-09-16T03:39:34.594 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.553e-06 2024-09-16T03:39:34.594 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.594 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078551+0000", 2024-09-16T03:39:34.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0790000000000001e-06 2024-09-16T03:39:34.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.595 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078564+0000", 2024-09-16T03:39:34.595 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2943e-05 2024-09-16T03:39:34.595 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.595 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.595 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178635+0000", 2024-09-16T03:39:34.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10007044599999999 2024-09-16T03:39:34.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178684+0000", 2024-09-16T03:39:34.596 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9592999999999998e-05 2024-09-16T03:39:34.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.596 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.596 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.596 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178747+0000", 2024-09-16T03:39:34.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2286000000000006e-05 2024-09-16T03:39:34.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4,5", 2024-09-16T03:39:34.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178857+0000", 2024-09-16T03:39:34.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110807 2024-09-16T03:39:34.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.597 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.597 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280364+0000", 2024-09-16T03:39:34.597 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101506518 2024-09-16T03:39:34.597 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.597 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280373+0000", 2024-09-16T03:39:34.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6899999999999998e-06 2024-09-16T03:39:34.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.369249+0000", 2024-09-16T03:39:34.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.088876172000000003 2024-09-16T03:39:34.598 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.598 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.598 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.598 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693044+0000", 2024-09-16T03:39:34.598 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.32379486499999999 2024-09-16T03:39:34.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.599 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.599 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693066+0000", 2024-09-16T03:39:34.599 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2413e-05 2024-09-16T03:39:34.599 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.599 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.599 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.599 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.599 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.599 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3119 4.13 4:c8492bfc:::smithi15736696-42:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:39:34.599 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:39:34.599 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 57.535580432000003, 2024-09-16T03:39:34.600 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.64286237700000004, 2024-09-16T03:39:34.600 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.600 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.600 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:34.600 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.600 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:34.600 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:34.600 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3119 2024-09-16T03:39:34.600 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.600 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.600 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.600 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.600 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:39:34.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:39:34.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.601 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.601 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931441+0000", 2024-09-16T03:39:34.601 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3599999999999999e-06 2024-09-16T03:39:34.601 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.601 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931449+0000", 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4080000000000001e-06 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931463+0000", 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.488e-05 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931473+0000", 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8109999999999997e-06 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.602 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.031604+0000", 2024-09-16T03:39:34.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013079499999999 2024-09-16T03:39:34.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.031668+0000", 2024-09-16T03:39:34.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4226999999999993e-05 2024-09-16T03:39:34.603 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.603 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.603 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.603 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.574301+0000", 2024-09-16T03:39:34.603 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54263289599999998 2024-09-16T03:39:34.604 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.604 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.604 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.604 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.604 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.604 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:34.605 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:34.604+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:34.606 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:34.607 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:39:34.607 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:34.608 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:34.609 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:34.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 'trim stale osdmaps' 2024-09-16T03:39:34.609 INFO:tasks.rados.rados.0.smithi157.stdout:1948: finishing write tid 1 to smithi15736696-31 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:1948: finishing write tid 2 to smithi15736696-31 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:1948: finishing write tid 3 to smithi15736696-31 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:1948: finishing write tid 4 to smithi15736696-31 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:1948: finishing write tid 5 to smithi15736696-31 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:1948: oid 31 updating version 0 to 92 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:1948: oid 31 updating version 92 to 93 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:1948: oid 31 version 93 is already newer than 90 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:1948: oid 31 version 93 is already newer than 91 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 93 (ObjNum 436 snap 0 seq_num 436) dirty exists 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:1948: left oid 31 (ObjNum 436 snap 0 seq_num 436) 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:1950: got (2) No such file or directory 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:1950: got expected ENOENT (src dne) 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 77 (ObjNum 129 snap 0 seq_num 3487771760) clean dne 2024-09-16T03:39:34.610 INFO:tasks.rados.rados.0.smithi157.stdout:1949: finishing set_chunk to oid 33 2024-09-16T03:39:34.611 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 62 (ObjNum 328 snap 0 seq_num 328) dirty exists 2024-09-16T03:39:34.611 INFO:tasks.rados.rados.0.smithi157.stdout:1943: done (5 left) 2024-09-16T03:39:34.611 INFO:tasks.rados.rados.0.smithi157.stdout:1946: done (4 left) 2024-09-16T03:39:34.611 INFO:tasks.rados.rados.0.smithi157.stdout:1947: done (3 left) 2024-09-16T03:39:34.611 INFO:tasks.rados.rados.0.smithi157.stdout:1948: done (2 left) 2024-09-16T03:39:34.611 INFO:tasks.rados.rados.0.smithi157.stdout:1949: done (1 left) 2024-09-16T03:39:34.611 INFO:tasks.rados.rados.0.smithi157.stdout:1950: done (0 left) 2024-09-16T03:39:34.611 INFO:tasks.rados.rados.0.smithi157.stdout:1951: set_chunk oid 42 target oid 2024-09-16T03:39:34.611 INFO:tasks.rados.rados.0.smithi157.stdout:1951: set_chunk oid 42 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:39:34.611 INFO:tasks.rados.rados.0.smithi157.stdout:1952: set_chunk oid 50 target oid 2024-09-16T03:39:34.664 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:39:34.665 INFO:tasks.rados.rados.0.smithi157.stdout:1952: set_chunk oid 50 offset: 335872 length: 485376 target oid f216c361ccde0f9185c4f5bc70cc212424f68cb9cc78b90b80f7b5a64a0c2daf offset: 0 snap -1 2024-09-16T03:39:34.665 INFO:tasks.rados.rados.0.smithi157.stdout:1953: set_chunk oid 24 target oid 2024-09-16T03:39:34.706 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:34.704+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:34.715 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:34.715 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:34.715 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.715 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4782 3.3 3.4692fa03 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-16T03:39:34.715 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:34.667135+0000", 2024-09-16T03:39:34.715 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.047228085000000003, 2024-09-16T03:39:34.716 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.047246983999999999, 2024-09-16T03:39:34.716 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.716 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.716 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:39:34.716 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.716 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:34.716 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:34.716 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4782 2024-09-16T03:39:34.716 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.716 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.716 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.716 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.716 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:34.667135+0000", 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:34.667135+0000", 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:34.667136+0000", 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2860000000000001e-06 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.717 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:34.667346+0000", 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000210039 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:34.667358+0000", 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1969e-05 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:34.667365+0000", 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3810000000000001e-06 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.718 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.719 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.719 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.719 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-16T03:39:34.719 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 1 2024-09-16T03:39:34.719 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:34.724 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:39:34.750 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:34.750 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:34.751 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:34.753 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:34.754 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:34.754 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 'trim stale osdmaps' 2024-09-16T03:39:34.806 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:34.805+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:34.829 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:34.829 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:34.829 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:34.829 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:34.829 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:34.837 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:39:34.873 INFO:tasks.rados.rados.0.smithi157.stdout:1953: set_chunk oid 24 offset: 846336 length: 1106432 target oid 0b5767827c86785865ef8b21d1633748651a6614501adb73b041067e87948aa8 offset: 0 snap -1 2024-09-16T03:39:34.873 INFO:tasks.rados.rados.0.smithi157.stdout:1954: read oid 10 snap -1 2024-09-16T03:39:34.873 INFO:tasks.rados.rados.0.smithi157.stdout:1954: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-16T03:39:34.873 INFO:tasks.rados.rados.0.smithi157.stdout:1955: set_chunk oid 25 target oid 2024-09-16T03:39:34.891 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:34.891 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:34.891 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:34.891 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:34.892 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:34.892 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:34.892 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:34.892 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:34.892 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:34.892 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:34.892 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:34.892 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:34.895 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:34.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 'trim stale osdmaps' 2024-09-16T03:39:34.936 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:34.934+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:34.945 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:34.945 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:34.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:34.945 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:34.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.945 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:465 4.5 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:34.945 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:39:34.945 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 133.92620638899999, 2024-09-16T03:39:34.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60529649299999999, 2024-09-16T03:39:34.945 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.945 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.946 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.946 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.946 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:34.946 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:34.946 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 465 2024-09-16T03:39:34.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.946 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:39:34.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:39:34.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015706+0000", 2024-09-16T03:39:34.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.091e-06 2024-09-16T03:39:34.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015712+0000", 2024-09-16T03:39:34.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-16T03:39:34.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015724+0000", 2024-09-16T03:39:34.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1408e-05 2024-09-16T03:39:34.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015731+0000", 2024-09-16T03:39:34.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:39:34.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216049+0000", 2024-09-16T03:39:34.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200317895 2024-09-16T03:39:34.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216178+0000", 2024-09-16T03:39:34.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012848800000000001 2024-09-16T03:39:34.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:34.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620010+0000", 2024-09-16T03:39:34.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40383207199999999 2024-09-16T03:39:34.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620710+0000", 2024-09-16T03:39:34.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069988400000000001 2024-09-16T03:39:34.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620902+0000", 2024-09-16T03:39:34.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019197800000000001 2024-09-16T03:39:34.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620933+0000", 2024-09-16T03:39:34.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1513999999999997e-05 2024-09-16T03:39:34.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620968+0000", 2024-09-16T03:39:34.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4524999999999999e-05 2024-09-16T03:39:34.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.621002+0000", 2024-09-16T03:39:34.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3955000000000003e-05 2024-09-16T03:39:34.951 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.951 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.952 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.952 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:10 3.3 3:deac805c:::smithi15736696-44 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:34.952 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:39:34.952 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 128.534711413, 2024-09-16T03:39:34.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53191748100000003, 2024-09-16T03:39:34.952 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.952 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.952 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.952 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:39:34.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:39:34.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407202+0000", 2024-09-16T03:39:34.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.855e-06 2024-09-16T03:39:34.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407208+0000", 2024-09-16T03:39:34.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9819999999999996e-06 2024-09-16T03:39:34.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407220+0000", 2024-09-16T03:39:34.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2191999999999999e-05 2024-09-16T03:39:34.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407236+0000", 2024-09-16T03:39:34.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5698000000000001e-05 2024-09-16T03:39:34.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538167+0000", 2024-09-16T03:39:34.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.13093111499999999 2024-09-16T03:39:34.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538229+0000", 2024-09-16T03:39:34.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2441999999999996e-05 2024-09-16T03:39:34.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:39:34.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538346+0000", 2024-09-16T03:39:34.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011716400000000001 2024-09-16T03:39:34.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540757+0000", 2024-09-16T03:39:34.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024105089999999999 2024-09-16T03:39:34.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.639501+0000", 2024-09-16T03:39:34.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098743545000000002 2024-09-16T03:39:34.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939038+0000", 2024-09-16T03:39:34.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29953765700000001 2024-09-16T03:39:34.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939075+0000", 2024-09-16T03:39:34.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6600999999999997e-05 2024-09-16T03:39:34.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939118+0000", 2024-09-16T03:39:34.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2721e-05 2024-09-16T03:39:34.958 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.958 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.958 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.958 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:984 4.15 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:34.958 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:39:34.958 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 124.569791485, 2024-09-16T03:39:34.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.97836159599999994, 2024-09-16T03:39:34.958 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 984 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:39:34.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372122+0000", 2024-09-16T03:39:34.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.934e-06 2024-09-16T03:39:34.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372129+0000", 2024-09-16T03:39:34.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8029999999999999e-06 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372141+0000", 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2006e-05 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472234+0000", 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100092985 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472318+0000", 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4257000000000003e-05 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472326+0000", 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8960000000000003e-06 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350293+0000", 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.87796699899999997 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350433+0000", 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013999600000000001 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350482+0000", 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8720000000000001e-05 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1000 4.15 4:aff4cc41:::smithi15736696-11:head [read 0~2873964,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:34.963 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:39:34.964 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 124.3513524, 2024-09-16T03:39:34.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.761707841, 2024-09-16T03:39:34.964 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.964 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.964 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:34.964 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.964 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:34.964 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:34.964 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1000 2024-09-16T03:39:34.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.964 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:39:34.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:39:34.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590560+0000", 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590562+0000", 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.17e-06 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590568+0000", 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7939999999999997e-06 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590572+0000", 2024-09-16T03:39:34.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1300000000000001e-06 2024-09-16T03:39:34.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590619+0000", 2024-09-16T03:39:34.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7148999999999999e-05 2024-09-16T03:39:34.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:34.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590625+0000", 2024-09-16T03:39:34.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5679999999999996e-06 2024-09-16T03:39:34.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350858+0000", 2024-09-16T03:39:34.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76023295000000002 2024-09-16T03:39:34.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350973+0000", 2024-09-16T03:39:34.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115023 2024-09-16T03:39:34.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352267+0000", 2024-09-16T03:39:34.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012935970000000001 2024-09-16T03:39:34.969 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.969 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.969 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.969 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1001 4.15 4:aff4cc41:::smithi15736696-11:head [read 1254674~1619290,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:34.969 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:34.969 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 124.351333815, 2024-09-16T03:39:34.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76275827799999996, 2024-09-16T03:39:34.969 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.969 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.970 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:34.970 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.970 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:34.970 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:34.970 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1001 2024-09-16T03:39:34.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.970 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:34.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:34.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:34.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:39:34.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590580+0000", 2024-09-16T03:39:34.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590584+0000", 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8800000000000001e-06 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590587+0000", 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6630000000000002e-06 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590650+0000", 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3950999999999998e-05 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590654+0000", 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.89e-06 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352309+0000", 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76165495699999997 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352407+0000", 2024-09-16T03:39:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7188999999999994e-05 2024-09-16T03:39:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.353336+0000", 2024-09-16T03:39:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092939400000000005 2024-09-16T03:39:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.974 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1002 4.15 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~1527025] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 124.35130456100001, 2024-09-16T03:39:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85963523600000002, 2024-09-16T03:39:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1002 2024-09-16T03:39:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.15e-07 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590609+0000", 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.606e-06 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590615+0000", 2024-09-16T03:39:34.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3040000000000001e-06 2024-09-16T03:39:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590618+0000", 2024-09-16T03:39:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.309e-06 2024-09-16T03:39:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590669+0000", 2024-09-16T03:39:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1048000000000003e-05 2024-09-16T03:39:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590673+0000", 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.061e-06 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.449932+0000", 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85925979699999999 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450032+0000", 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9858000000000005e-05 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450242+0000", 2024-09-16T03:39:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020993800000000001 2024-09-16T03:39:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2467 4.5 4:a11e1d7c:::smithi15736696-43:head [write 1772395~538068] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:39:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.950628190000003, 2024-09-16T03:39:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.56288137400000005, 2024-09-16T03:39:34.980 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2467 2024-09-16T03:39:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:39:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991285+0000", 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0990000000000001e-06 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991447+0000", 2024-09-16T03:39:34.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016170099999999999 2024-09-16T03:39:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991458+0000", 2024-09-16T03:39:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.128e-05 2024-09-16T03:39:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991464+0000", 2024-09-16T03:39:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3800000000000002e-06 2024-09-16T03:39:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150823+0000", 2024-09-16T03:39:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15935943799999999 2024-09-16T03:39:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150899+0000", 2024-09-16T03:39:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5359999999999997e-05 2024-09-16T03:39:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151007+0000", 2024-09-16T03:39:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108317 2024-09-16T03:39:34.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253992+0000", 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102985249 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.353797+0000", 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099805039999999998 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554039+0000", 2024-09-16T03:39:34.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20024175899999999 2024-09-16T03:39:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554129+0000", 2024-09-16T03:39:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0067000000000004e-05 2024-09-16T03:39:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554165+0000", 2024-09-16T03:39:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5683999999999999e-05 2024-09-16T03:39:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.986 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2468 4.5 4:a11e1d7c:::smithi15736696-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.945207410999998, 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.55765803899999999, 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2468 2024-09-16T03:39:34.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996708+0000", 2024-09-16T03:39:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5149999999999998e-06 2024-09-16T03:39:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996921+0000", 2024-09-16T03:39:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000213692 2024-09-16T03:39:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996937+0000", 2024-09-16T03:39:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5699999999999999e-05 2024-09-16T03:39:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996946+0000", 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7159999999999992e-06 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152362+0000", 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15541682800000001 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152421+0000", 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-16T03:39:34.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152515+0000", 2024-09-16T03:39:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3659999999999994e-05 2024-09-16T03:39:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253995+0000", 2024-09-16T03:39:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10148022499999999 2024-09-16T03:39:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.453908+0000", 2024-09-16T03:39:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19991299700000001 2024-09-16T03:39:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554321+0000", 2024-09-16T03:39:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100412977 2024-09-16T03:39:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554344+0000", 2024-09-16T03:39:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.262e-05 2024-09-16T03:39:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554362+0000", 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8539999999999999e-05 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2469 4.5 4:a11e1d7c:::smithi15736696-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.944952048000005, 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.75785074799999996, 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:34.993 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2469 2024-09-16T03:39:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:39:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996960+0000", 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3000000000000001e-07 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996963+0000", 2024-09-16T03:39:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8679999999999999e-06 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996967+0000", 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.622e-06 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996971+0000", 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1429999999999998e-06 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:34.996 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253536+0000", 2024-09-16T03:39:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25656582900000002 2024-09-16T03:39:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253632+0000", 2024-09-16T03:39:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6067e-05 2024-09-16T03:39:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253745+0000", 2024-09-16T03:39:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011256699999999999 2024-09-16T03:39:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254218+0000", 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00047321400000000001 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554397+0000", 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30017854399999999 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754557+0000", 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200159848 2024-09-16T03:39:34.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754646+0000", 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9370999999999995e-05 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754810+0000", 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016414500000000001 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:34.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2470 4.5 4:a11e1d7c:::smithi15736696-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.944935719, 2024-09-16T03:39:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76022711799999998, 2024-09-16T03:39:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2470 2024-09-16T03:39:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.000 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:35.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.65e-07 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996977+0000", 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4079999999999999e-06 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996981+0000", 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.002 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996988+0000", 2024-09-16T03:39:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0890000000000002e-06 2024-09-16T03:39:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253947+0000", 2024-09-16T03:39:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25695931100000002 2024-09-16T03:39:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253985+0000", 2024-09-16T03:39:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7939999999999999e-05 2024-09-16T03:39:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757090+0000", 2024-09-16T03:39:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50310477499999995 2024-09-16T03:39:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757140+0000", 2024-09-16T03:39:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9567999999999997e-05 2024-09-16T03:39:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757203+0000", 2024-09-16T03:39:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3215999999999999e-05 2024-09-16T03:39:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2487 4.15 4:ad0b9852:::smithi15736696-16:head [write 1947841~685395] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:39:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.525891129000001, 2024-09-16T03:39:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.74140762199999999, 2024-09-16T03:39:35.005 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2487 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:39:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416023+0000", 2024-09-16T03:39:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:39:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416341+0000", 2024-09-16T03:39:35.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031881699999999999 2024-09-16T03:39:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416353+0000", 2024-09-16T03:39:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.189e-05 2024-09-16T03:39:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416360+0000", 2024-09-16T03:39:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5719999999999999e-06 2024-09-16T03:39:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755578+0000", 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33921859199999999 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755639+0000", 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0615000000000002e-05 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755735+0000", 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5798999999999995e-05 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858633+0000", 2024-09-16T03:39:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10289859599999999 2024-09-16T03:39:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858695+0000", 2024-09-16T03:39:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2000000000000003e-05 2024-09-16T03:39:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157315+0000", 2024-09-16T03:39:35.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29861931600000002 2024-09-16T03:39:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157375+0000", 2024-09-16T03:39:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0268e-05 2024-09-16T03:39:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157428+0000", 2024-09-16T03:39:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2939999999999998e-05 2024-09-16T03:39:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:35.011 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2488 4.15 4:ad0b9852:::smithi15736696-16:head [write 3383215~196676] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.520218478999993, 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73589615900000005, 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2488 2024-09-16T03:39:35.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:39:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:39:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421695+0000", 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1469999999999999e-06 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421975+0000", 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028008299999999998 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421987+0000", 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1294000000000001e-05 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421992+0000", 2024-09-16T03:39:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9349999999999999e-06 2024-09-16T03:39:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757205+0000", 2024-09-16T03:39:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33521242800000001 2024-09-16T03:39:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757269+0000", 2024-09-16T03:39:35.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3761000000000002e-05 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757376+0000", 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010706599999999999 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858684+0000", 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101308255 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:35.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858700+0000", 2024-09-16T03:39:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6138999999999999e-05 2024-09-16T03:39:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157491+0000", 2024-09-16T03:39:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29879056999999998 2024-09-16T03:39:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157546+0000", 2024-09-16T03:39:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5668999999999998e-05 2024-09-16T03:39:35.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157589+0000", 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2812000000000003e-05 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2489 4.15 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=7b,truncate 3579891] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.519909656999999, 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73575981700000004, 2024-09-16T03:39:35.018 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2489 2024-09-16T03:39:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0200000000000002e-07 2024-09-16T03:39:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422005+0000", 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7379999999999999e-06 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422009+0000", 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6219999999999998e-06 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422016+0000", 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5789999999999998e-06 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858071+0000", 2024-09-16T03:39:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.436054302 2024-09-16T03:39:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858130+0000", 2024-09-16T03:39:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9493000000000001e-05 2024-09-16T03:39:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:35.022 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858226+0000", 2024-09-16T03:39:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6174000000000004e-05 2024-09-16T03:39:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:39:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10106483199999999 2024-09-16T03:39:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959639+0000", 2024-09-16T03:39:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034802999999999999 2024-09-16T03:39:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157640+0000", 2024-09-16T03:39:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19800045299999999 2024-09-16T03:39:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157702+0000", 2024-09-16T03:39:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2218000000000006e-05 2024-09-16T03:39:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157762+0000", 2024-09-16T03:39:35.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9774000000000001e-05 2024-09-16T03:39:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2490 4.15 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:39:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.519891064999996, 2024-09-16T03:39:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73597824999999994, 2024-09-16T03:39:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:35.025 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2490 2024-09-16T03:39:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:39:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:35.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:39:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422021+0000", 2024-09-16T03:39:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8099999999999999e-07 2024-09-16T03:39:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422022+0000", 2024-09-16T03:39:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.573e-06 2024-09-16T03:39:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422027+0000", 2024-09-16T03:39:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3250000000000001e-06 2024-09-16T03:39:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422030+0000", 2024-09-16T03:39:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.948e-06 2024-09-16T03:39:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858510+0000", 2024-09-16T03:39:35.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.43648032599999997 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858588+0000", 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7888999999999999e-05 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157803+0000", 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.299215642 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:35.029 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157893+0000", 2024-09-16T03:39:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9565000000000001e-05 2024-09-16T03:39:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157999+0000", 2024-09-16T03:39:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105701 2024-09-16T03:39:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2492 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:35.030 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.519878555999995, 2024-09-16T03:39:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53721099400000005, 2024-09-16T03:39:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2492 2024-09-16T03:39:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:39:35.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422036+0000", 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5900000000000002e-06 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422043+0000", 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3069999999999997e-06 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422048+0000", 2024-09-16T03:39:35.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6619999999999996e-06 2024-09-16T03:39:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958774+0000", 2024-09-16T03:39:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53672593499999999 2024-09-16T03:39:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958868+0000", 2024-09-16T03:39:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4408999999999997e-05 2024-09-16T03:39:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959244+0000", 2024-09-16T03:39:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037581099999999998 2024-09-16T03:39:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2493 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.519858776999996, 2024-09-16T03:39:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53757571299999996, 2024-09-16T03:39:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:35.035 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2493 2024-09-16T03:39:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3600000000000004e-07 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422055+0000", 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7460000000000001e-06 2024-09-16T03:39:35.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422058+0000", 2024-09-16T03:39:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:39:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422061+0000", 2024-09-16T03:39:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6759999999999999e-06 2024-09-16T03:39:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53723057900000004 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959367+0000", 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5158000000000001e-05 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959628+0000", 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026175800000000001 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:35.039 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2494 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 1738052~782998] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.519847747, 2024-09-16T03:39:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53787640199999998, 2024-09-16T03:39:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.040 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2494 2024-09-16T03:39:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6100000000000002e-07 2024-09-16T03:39:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422065+0000", 2024-09-16T03:39:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2839999999999999e-06 2024-09-16T03:39:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422069+0000", 2024-09-16T03:39:35.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7239999999999998e-06 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422072+0000", 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6560000000000001e-06 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959661+0000", 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53758982600000005 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:35.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959739+0000", 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7581000000000001e-05 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959940+0000", 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020107 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3987 4.36 4:6ef3b53d:::smithi15736696-48:head [setxattr _header (56) in=7b,truncate 2168169] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:39:35.044 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.565793862, 2024-09-16T03:39:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50651914200000003, 2024-09-16T03:39:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3987 2024-09-16T03:39:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:35.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:39:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:39:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376119+0000", 2024-09-16T03:39:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3799999999999996e-07 2024-09-16T03:39:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376122+0000", 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.202e-06 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376251+0000", 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012896699999999999 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376259+0000", 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.32e-06 2024-09-16T03:39:35.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379368+0000", 2024-09-16T03:39:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031084989999999998 2024-09-16T03:39:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379441+0000", 2024-09-16T03:39:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3331999999999995e-05 2024-09-16T03:39:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,10", 2024-09-16T03:39:35.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379559+0000", 2024-09-16T03:39:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118068 2024-09-16T03:39:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380350+0000", 2024-09-16T03:39:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00079103699999999997 2024-09-16T03:39:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.580657+0000", 2024-09-16T03:39:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20030706600000001 2024-09-16T03:39:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882497+0000", 2024-09-16T03:39:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30183998899999998 2024-09-16T03:39:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882560+0000", 2024-09-16T03:39:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3372000000000002e-05 2024-09-16T03:39:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882637+0000", 2024-09-16T03:39:35.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6352000000000004e-05 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3988 4.36 4:6ef3b53d:::smithi15736696-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 29.565640014, 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60662574199999997, 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:35.051 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3988 2024-09-16T03:39:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:39:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:39:35.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376273+0000", 2024-09-16T03:39:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3170000000000001e-06 2024-09-16T03:39:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376278+0000", 2024-09-16T03:39:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7849999999999999e-06 2024-09-16T03:39:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376286+0000", 2024-09-16T03:39:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2009999999999999e-06 2024-09-16T03:39:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376291+0000", 2024-09-16T03:39:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4029999999999998e-06 2024-09-16T03:39:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380020+0000", 2024-09-16T03:39:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037289630000000001 2024-09-16T03:39:35.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380065+0000", 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4666999999999998e-05 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982800+0000", 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60273539099999995 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982838+0000", 2024-09-16T03:39:35.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8155999999999999e-05 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982897+0000", 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8859e-05 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4506 4.1a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v129,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e146)", 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:25.237761+0000", 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 9.7041500860000003, 2024-09-16T03:39:35.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.51319717499999995, 2024-09-16T03:39:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4506 2024-09-16T03:39:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237761+0000", 2024-09-16T03:39:35.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237761+0000", 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237764+0000", 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2819999999999999e-06 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:35.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237772+0000", 2024-09-16T03:39:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9929999999999995e-06 2024-09-16T03:39:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237786+0000", 2024-09-16T03:39:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4508999999999999e-05 2024-09-16T03:39:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237795+0000", 2024-09-16T03:39:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2070000000000001e-06 2024-09-16T03:39:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237845+0000", 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.952e-05 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237911+0000", 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6434999999999997e-05 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.649713+0000", 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.41180149399999999 2024-09-16T03:39:35.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.650287+0000", 2024-09-16T03:39:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057374399999999995 2024-09-16T03:39:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.650476+0000", 2024-09-16T03:39:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018991900000000001 2024-09-16T03:39:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.750876+0000", 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100399355 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.750914+0000", 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8226999999999998e-05 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.750959+0000", 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4490000000000003e-05 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:35.062 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:35.063 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:35.063 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:35.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:39:35.065 INFO:tasks.rados.rados.0.smithi157.stdout:1955: set_chunk oid 25 offset: 419328 length: 2210816 target oid b0055f59be8f5d11e343bc1fd7c1a99122c204f35d2b2bbe678e8f84c1191c09 offset: 0 snap -1 2024-09-16T03:39:35.065 INFO:tasks.rados.rados.0.smithi157.stdout:1956: read oid 15 snap -1 2024-09-16T03:39:35.065 INFO:tasks.rados.rados.0.smithi157.stdout:1956: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-16T03:39:35.065 INFO:tasks.rados.rados.0.smithi157.stdout:1957: set_chunk oid 12 target oid 2024-09-16T03:39:35.066 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:35.063+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:35.072 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:35.072 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:35.072 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:35.072 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:35.072 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:35.073 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:35.073 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:35.073 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:35.073 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:35.073 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:35.073 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:35.073 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:35.077 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:35.077 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 'trim stale osdmaps' 2024-09-16T03:39:35.093 INFO:tasks.rados.rados.0.smithi157.stdout:1957: set_chunk oid 12 offset: 533504 length: 1126912 target oid f8f1d6d29636a020a389a4bb9fa19775cb5949fcc06a80858e4997822fde2403 offset: 0 snap -1 2024-09-16T03:39:35.093 INFO:tasks.rados.rados.0.smithi157.stdout:1958: set_chunk oid 26 target oid 2024-09-16T03:39:35.165 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:35.164+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:35.176 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:35.176 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:35.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.176 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4628.0:4789 4.20 e155/111)", 2024-09-16T03:39:35.176 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:35.094712+0000", 2024-09-16T03:39:35.176 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.080398118000000005, 2024-09-16T03:39:35.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.080423258999999997, 2024-09-16T03:39:35.176 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.177 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:39:35.177 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:35.094712+0000", 2024-09-16T03:39:35.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:35.094712+0000", 2024-09-16T03:39:35.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:35.094714+0000", 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.897e-06 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:35.094722+0000", 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2500000000000006e-06 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:35.094738+0000", 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6163000000000001e-05 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:35.094746+0000", 2024-09-16T03:39:35.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2420000000000003e-06 2024-09-16T03:39:35.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.179 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.179 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:39:35.179 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-09-16T03:39:35.179 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:35.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:39:35.256 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:35.256 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:35.256 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:35.256 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:35.257 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:35.257 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:35.257 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:35.257 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:35.257 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:35.257 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:35.257 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:35.257 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:35.259 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:35.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 'trim stale osdmaps' 2024-09-16T03:39:35.266 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:35.265+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:35.296 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:35.296 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:35.296 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:35.296 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:35.296 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:35.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:39:35.310 INFO:tasks.rados.rados.0.smithi157.stdout:1958: set_chunk oid 26 offset: 82432 length: 2217984 target oid 70fb138f6741c35bf9b80ea29c1675be6e30345d7d21aa8701e0feea60724be7 offset: 0 snap -1 2024-09-16T03:39:35.310 INFO:tasks.rados.rados.0.smithi157.stdout:1959: set_chunk oid 9 target oid 2024-09-16T03:39:35.390 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:35.388+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:35.397 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:35.397 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:35.398 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:35.398 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:35.398 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:35.398 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:35.398 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:35.398 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:35.398 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:35.398 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:35.398 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:35.398 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:35.401 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:35.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 'trim stale osdmaps' 2024-09-16T03:39:35.457 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:39 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'21, mlcod=26'19)", 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 160.85887276400001, 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30197182500000003, 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:39:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595320+0000", 2024-09-16T03:39:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.962e-06 2024-09-16T03:39:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595336+0000", 2024-09-16T03:39:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5568000000000001e-05 2024-09-16T03:39:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595349+0000", 2024-09-16T03:39:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3203e-05 2024-09-16T03:39:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695424+0000", 2024-09-16T03:39:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007542899999999 2024-09-16T03:39:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695495+0000", 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0450999999999993e-05 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695538+0000", 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3433000000000002e-05 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897256+0000", 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20171840699999999 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897290+0000", 2024-09-16T03:39:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3371999999999998e-05 2024-09-16T03:39:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1948 4.8 4:146f84f0:::smithi15736696-6:head [setxattr _header (55) in=7b,truncate 2756348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:39:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.329383957999994, 2024-09-16T03:39:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40326984799999999, 2024-09-16T03:39:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1948 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:39:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124808+0000", 2024-09-16T03:39:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9099999999999991e-07 2024-09-16T03:39:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124813+0000", 2024-09-16T03:39:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9599999999999999e-06 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124821+0000", 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0490000000000004e-06 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124827+0000", 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3339999999999998e-06 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126739+0000", 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00191147 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126830+0000", 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1751999999999998e-05 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127005+0000", 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174227 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127981+0000", 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097622799999999999 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.128193+0000", 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021176999999999999 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.135281+0000", 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070882150000000001 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.427785+0000", 2024-09-16T03:39:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29250390599999998 2024-09-16T03:39:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527932+0000", 2024-09-16T03:39:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10014759500000001 2024-09-16T03:39:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527980+0000", 2024-09-16T03:39:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7725000000000001e-05 2024-09-16T03:39:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528077+0000", 2024-09-16T03:39:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6626000000000006e-05 2024-09-16T03:39:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1949 4.8 4:146f84f0:::smithi15736696-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.329353213999994, 2024-09-16T03:39:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40346921899999999, 2024-09-16T03:39:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1949 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:39:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124841+0000", 2024-09-16T03:39:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5890000000000001e-06 2024-09-16T03:39:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124847+0000", 2024-09-16T03:39:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0820000000000004e-06 2024-09-16T03:39:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124852+0000", 2024-09-16T03:39:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1070000000000004e-06 2024-09-16T03:39:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127536+0000", 2024-09-16T03:39:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026844619999999999 2024-09-16T03:39:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127585+0000", 2024-09-16T03:39:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8915999999999997e-05 2024-09-16T03:39:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528126+0000", 2024-09-16T03:39:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40054061899999999 2024-09-16T03:39:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528193+0000", 2024-09-16T03:39:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6717999999999994e-05 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528307+0000", 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011413600000000001 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1979 4.18 4:1f63731d:::smithi15736696-30:head [write 789962~534747] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.818345902000004, 2024-09-16T03:39:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.506754182, 2024-09-16T03:39:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1979 2024-09-16T03:39:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:39:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:39:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635850+0000", 2024-09-16T03:39:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6670000000000004e-06 2024-09-16T03:39:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636429+0000", 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000579745 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636441+0000", 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1965e-05 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736594+0000", 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100152291 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837326+0000", 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10073201399999999 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837437+0000", 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111914 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837625+0000", 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000187391 2024-09-16T03:39:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939449+0000", 2024-09-16T03:39:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10182371599999999 2024-09-16T03:39:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939476+0000", 2024-09-16T03:39:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7117000000000001e-05 2024-09-16T03:39:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142297+0000", 2024-09-16T03:39:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20282091499999999 2024-09-16T03:39:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142347+0000", 2024-09-16T03:39:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0800000000000002e-05 2024-09-16T03:39:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142485+0000", 2024-09-16T03:39:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013799999999999999 2024-09-16T03:39:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142526+0000", 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0269999999999999e-05 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142599+0000", 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3376999999999999e-05 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.482 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1980 4.18 4:1f63731d:::smithi15736696-30:head [write 1780952~690056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:35.483 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:39:35.483 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.717544740999998, 2024-09-16T03:39:35.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40615011400000001, 2024-09-16T03:39:35.483 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.483 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.483 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.483 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1980 2024-09-16T03:39:35.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.483 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:39:35.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:39:35.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736656+0000", 2024-09-16T03:39:35.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0339999999999999e-05 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737092+0000", 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435957 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737102+0000", 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7860000000000008e-06 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737107+0000", 2024-09-16T03:39:35.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:39:35.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839282+0000", 2024-09-16T03:39:35.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10217523100000001 2024-09-16T03:39:35.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.486 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.486 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839389+0000", 2024-09-16T03:39:35.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010619399999999999 2024-09-16T03:39:35.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:35.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839567+0000", 2024-09-16T03:39:35.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017886200000000001 2024-09-16T03:39:35.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:35.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939485+0000", 2024-09-16T03:39:35.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099917168000000001 2024-09-16T03:39:35.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943655+0000", 2024-09-16T03:39:35.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004170287 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142373+0000", 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19871839599999999 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142431+0000", 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7880999999999997e-05 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142627+0000", 2024-09-16T03:39:35.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019580500000000001 2024-09-16T03:39:35.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:35.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142723+0000", 2024-09-16T03:39:35.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6209e-05 2024-09-16T03:39:35.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142796+0000", 2024-09-16T03:39:35.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2978000000000005e-05 2024-09-16T03:39:35.489 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.489 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.490 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.490 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2277 4.18 4:1f63731d:::smithi15736696-30:head [write 460558~707131] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:35.490 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:39:35.490 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 85.031409041000003, 2024-09-16T03:39:35.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39421539100000003, 2024-09-16T03:39:35.490 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.490 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.490 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.490 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.490 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.490 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2277 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422785+0000", 2024-09-16T03:39:35.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9809999999999999e-06 2024-09-16T03:39:35.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423586+0000", 2024-09-16T03:39:35.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080133600000000004 2024-09-16T03:39:35.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423607+0000", 2024-09-16T03:39:35.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1039e-05 2024-09-16T03:39:35.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423616+0000", 2024-09-16T03:39:35.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0189999999999995e-06 2024-09-16T03:39:35.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523776+0000", 2024-09-16T03:39:35.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015945399999999 2024-09-16T03:39:35.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523851+0000", 2024-09-16T03:39:35.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5549999999999993e-05 2024-09-16T03:39:35.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:35.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523972+0000", 2024-09-16T03:39:35.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120683 2024-09-16T03:39:35.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527436+0000", 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034638870000000001 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636588+0000", 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109152601 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636665+0000", 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6661000000000003e-05 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637128+0000", 2024-09-16T03:39:35.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046316199999999998 2024-09-16T03:39:35.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816886+0000", 2024-09-16T03:39:35.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17975745200000001 2024-09-16T03:39:35.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:35.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816927+0000", 2024-09-16T03:39:35.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1529000000000002e-05 2024-09-16T03:39:35.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.497 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816997+0000", 2024-09-16T03:39:35.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0036999999999994e-05 2024-09-16T03:39:35.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.497 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.498 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2278 4.18 4:1f63731d:::smithi15736696-30:head [write 1869389~508716] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:35.498 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:39:35.498 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 85.024840040000001, 2024-09-16T03:39:35.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48788153400000001, 2024-09-16T03:39:35.499 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.499 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.499 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.499 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.499 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2278 2024-09-16T03:39:35.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.499 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429352+0000", 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.604e-06 2024-09-16T03:39:35.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429895+0000", 2024-09-16T03:39:35.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054308899999999996 2024-09-16T03:39:35.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429909+0000", 2024-09-16T03:39:35.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3518e-05 2024-09-16T03:39:35.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429917+0000", 2024-09-16T03:39:35.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-09-16T03:39:35.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525208+0000", 2024-09-16T03:39:35.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095291736000000002 2024-09-16T03:39:35.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525303+0000", 2024-09-16T03:39:35.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4304999999999999e-05 2024-09-16T03:39:35.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525410+0000", 2024-09-16T03:39:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010755399999999999 2024-09-16T03:39:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636785+0000", 2024-09-16T03:39:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111375037 2024-09-16T03:39:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:35.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637092+0000", 2024-09-16T03:39:35.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030685599999999999 2024-09-16T03:39:35.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637157+0000", 2024-09-16T03:39:35.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4593999999999999e-05 2024-09-16T03:39:35.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637236+0000", 2024-09-16T03:39:35.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9563999999999999e-05 2024-09-16T03:39:35.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917119+0000", 2024-09-16T03:39:35.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.279882931 2024-09-16T03:39:35.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:35.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917158+0000", 2024-09-16T03:39:35.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9096e-05 2024-09-16T03:39:35.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917232+0000", 2024-09-16T03:39:35.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4091000000000006e-05 2024-09-16T03:39:35.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.506 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.506 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2279 4.18 4:1f63731d:::smithi15736696-30:head [write 3094884~165674] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:35.506 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:39:35.506 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 85.020151244999994, 2024-09-16T03:39:35.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483372209, 2024-09-16T03:39:35.506 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.506 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.506 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.506 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.507 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.507 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2279 2024-09-16T03:39:35.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.507 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:39:35.507 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.507 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.507 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.507 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:39:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434041+0000", 2024-09-16T03:39:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.195e-06 2024-09-16T03:39:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434191+0000", 2024-09-16T03:39:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000150488 2024-09-16T03:39:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.508 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434202+0000", 2024-09-16T03:39:35.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0978000000000001e-05 2024-09-16T03:39:35.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434208+0000", 2024-09-16T03:39:35.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-09-16T03:39:35.509 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.509 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.509 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.509 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526820+0000", 2024-09-16T03:39:35.509 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092612229000000004 2024-09-16T03:39:35.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526894+0000", 2024-09-16T03:39:35.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3435000000000004e-05 2024-09-16T03:39:35.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:35.510 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526990+0000", 2024-09-16T03:39:35.510 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6112000000000004e-05 2024-09-16T03:39:35.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.510 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.510 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637050+0000", 2024-09-16T03:39:35.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110059859 2024-09-16T03:39:35.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:35.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637097+0000", 2024-09-16T03:39:35.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6848000000000003e-05 2024-09-16T03:39:35.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.511 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.511 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.511 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637183+0000", 2024-09-16T03:39:35.511 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6798000000000005e-05 2024-09-16T03:39:35.511 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637262+0000", 2024-09-16T03:39:35.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9049999999999997e-05 2024-09-16T03:39:35.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917287+0000", 2024-09-16T03:39:35.512 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280025036 2024-09-16T03:39:35.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.512 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.512 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:35.512 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917326+0000", 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.871e-05 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917412+0000", 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5638999999999998e-05 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2280 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 3260558] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:35.513 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 85.019972964000004, 2024-09-16T03:39:35.514 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48353995599999999, 2024-09-16T03:39:35.514 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.514 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.514 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.514 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.514 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.514 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.514 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2280 2024-09-16T03:39:35.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.514 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.514 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.514 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.514 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.515 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434221+0000", 2024-09-16T03:39:35.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8329999999999998e-06 2024-09-16T03:39:35.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434225+0000", 2024-09-16T03:39:35.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7589999999999999e-06 2024-09-16T03:39:35.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.516 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.516 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.516 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434228+0000", 2024-09-16T03:39:35.516 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.971e-06 2024-09-16T03:39:35.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527459+0000", 2024-09-16T03:39:35.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093231249000000002 2024-09-16T03:39:35.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527519+0000", 2024-09-16T03:39:35.517 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9471e-05 2024-09-16T03:39:35.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.517 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.517 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:35.517 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527606+0000", 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6776999999999999e-05 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637084+0000", 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109478228 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637100+0000", 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6711999999999999e-05 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.518 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637210+0000", 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010909200000000001 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637296+0000", 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6718000000000006e-05 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917637+0000", 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280340537 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.519 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:35.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917671+0000", 2024-09-16T03:39:35.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3983999999999999e-05 2024-09-16T03:39:35.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.520 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.520 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917758+0000", 2024-09-16T03:39:35.520 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7017999999999999e-05 2024-09-16T03:39:35.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.520 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.520 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.520 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2281 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 85.019958669000005, 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483761683, 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2281 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.521 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.96e-07 2024-09-16T03:39:35.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434234+0000", 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4050000000000001e-06 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434237+0000", 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1939999999999998e-06 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434240+0000", 2024-09-16T03:39:35.523 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.74e-06 2024-09-16T03:39:35.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530037+0000", 2024-09-16T03:39:35.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095796830999999999 2024-09-16T03:39:35.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:35.524 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530089+0000", 2024-09-16T03:39:35.524 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:39:35.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.524 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.524 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917823+0000", 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38773380000000002 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917891+0000", 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7676999999999996e-05 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917994+0000", 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010286200000000001 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.525 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.526 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.526 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.526 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2790 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v39,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-09-16T03:39:35.526 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:39:35.526 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.050111213999998, 2024-09-16T03:39:35.526 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42581759000000002, 2024-09-16T03:39:35.526 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.526 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.526 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.526 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.527 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2790 2024-09-16T03:39:35.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.527 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:39:35.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.527 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.527 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.527 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:39:35.527 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404083+0000", 2024-09-16T03:39:35.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2770000000000001e-06 2024-09-16T03:39:35.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.528 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404095+0000", 2024-09-16T03:39:35.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1780000000000001e-05 2024-09-16T03:39:35.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.528 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.528 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404116+0000", 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0882000000000001e-05 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404127+0000", 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1078e-05 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489837+0000", 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085709977000000007 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.529 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489956+0000", 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119856 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.592758+0000", 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10280165400000001 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.593693+0000", 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000935285 2024-09-16T03:39:35.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.594164+0000", 2024-09-16T03:39:35.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047097000000000003 2024-09-16T03:39:35.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.531 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.628426+0000", 2024-09-16T03:39:35.531 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034262070999999998 2024-09-16T03:39:35.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.531 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.531 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.732623+0000", 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10419628 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829726+0000", 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097103098999999998 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829790+0000", 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4677000000000004e-05 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829897+0000", 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106704 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2850 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 2252171] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.533570932000003, 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30323033799999999, 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.533 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.534 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.534 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.534 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.534 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.534 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2850 2024-09-16T03:39:35.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.534 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:39:35.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:39:35.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920621+0000", 2024-09-16T03:39:35.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.47e-07 2024-09-16T03:39:35.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920625+0000", 2024-09-16T03:39:35.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1620000000000001e-06 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920632+0000", 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0180000000000004e-06 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920638+0000", 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0950000000000001e-06 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922161+0000", 2024-09-16T03:39:35.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015229880000000001 2024-09-16T03:39:35.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922222+0000", 2024-09-16T03:39:35.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0903999999999998e-05 2024-09-16T03:39:35.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:35.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922323+0000", 2024-09-16T03:39:35.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101425 2024-09-16T03:39:35.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:35.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.023371+0000", 2024-09-16T03:39:35.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101047493 2024-09-16T03:39:35.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123227+0000", 2024-09-16T03:39:35.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099856057999999998 2024-09-16T03:39:35.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123840+0000", 2024-09-16T03:39:35.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061324999999999997 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123902+0000", 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1447999999999998e-05 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223497+0000", 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099595499000000004 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:35.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223665+0000", 2024-09-16T03:39:35.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016807399999999999 2024-09-16T03:39:35.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223850+0000", 2024-09-16T03:39:35.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018507699999999999 2024-09-16T03:39:35.540 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.540 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.540 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.540 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2851 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:39:35.540 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:39:35.541 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.533534836000001, 2024-09-16T03:39:35.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30354833399999998, 2024-09-16T03:39:35.541 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.541 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.541 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:35.541 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.541 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.541 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.541 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2851 2024-09-16T03:39:35.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.541 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:39:35.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:39:35.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920657+0000", 2024-09-16T03:39:35.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0100000000000004e-07 2024-09-16T03:39:35.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920660+0000", 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6519999999999999e-06 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920668+0000", 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1919999999999999e-06 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920674+0000", 2024-09-16T03:39:35.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126299+0000", 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20562534700000001 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126358+0000", 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8459999999999999e-05 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223913+0000", 2024-09-16T03:39:35.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097555263000000003 2024-09-16T03:39:35.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224009+0000", 2024-09-16T03:39:35.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5723999999999994e-05 2024-09-16T03:39:35.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224204+0000", 2024-09-16T03:39:35.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195498 2024-09-16T03:39:35.545 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.545 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3570 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.448401511999997, 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.353157053, 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.546 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3570 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005796+0000", 2024-09-16T03:39:35.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9229999999999996e-06 2024-09-16T03:39:35.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006222+0000", 2024-09-16T03:39:35.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000426282 2024-09-16T03:39:35.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006240+0000", 2024-09-16T03:39:35.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7858000000000001e-05 2024-09-16T03:39:35.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006249+0000", 2024-09-16T03:39:35.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7120000000000006e-06 2024-09-16T03:39:35.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006323+0000", 2024-09-16T03:39:35.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3808999999999998e-05 2024-09-16T03:39:35.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:35.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006333+0000", 2024-09-16T03:39:35.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0488e-05 2024-09-16T03:39:35.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358910+0000", 2024-09-16T03:39:35.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.35257616400000003 2024-09-16T03:39:35.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358946+0000", 2024-09-16T03:39:35.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6816999999999997e-05 2024-09-16T03:39:35.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.551 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.551 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3571 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:35.551 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:39:35.551 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.441896356000001, 2024-09-16T03:39:35.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34670431699999998, 2024-09-16T03:39:35.551 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.551 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.551 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:35.552 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.552 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.552 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.552 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3571 2024-09-16T03:39:35.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.552 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:39:35.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:39:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012298+0000", 2024-09-16T03:39:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.18e-06 2024-09-16T03:39:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012947+0000", 2024-09-16T03:39:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064929600000000005 2024-09-16T03:39:35.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012961+0000", 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4341e-05 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012970+0000", 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2740000000000007e-06 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.013025+0000", 2024-09-16T03:39:35.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5353999999999999e-05 2024-09-16T03:39:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.013033+0000", 2024-09-16T03:39:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4619999999999996e-06 2024-09-16T03:39:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358989+0000", 2024-09-16T03:39:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34595562699999999 2024-09-16T03:39:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358999+0000", 2024-09-16T03:39:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7829999999999993e-06 2024-09-16T03:39:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3572 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:39:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.435875600000003, 2024-09-16T03:39:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34073019300000001, 2024-09-16T03:39:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3572 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:39:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018317+0000", 2024-09-16T03:39:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4610000000000001e-06 2024-09-16T03:39:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018779+0000", 2024-09-16T03:39:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046202500000000002 2024-09-16T03:39:35.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018793+0000", 2024-09-16T03:39:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4629e-05 2024-09-16T03:39:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018799+0000", 2024-09-16T03:39:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1589999999999998e-06 2024-09-16T03:39:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018867+0000", 2024-09-16T03:39:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7879000000000005e-05 2024-09-16T03:39:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018888+0000", 2024-09-16T03:39:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0747999999999999e-05 2024-09-16T03:39:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359036+0000", 2024-09-16T03:39:35.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34014758 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359045+0000", 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7119999999999995e-06 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3573 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.435375905999997, 2024-09-16T03:39:35.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34027060599999998, 2024-09-16T03:39:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3573 2024-09-16T03:39:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:39:35.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018816+0000", 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1400000000000001e-06 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018821+0000", 2024-09-16T03:39:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4179999999999996e-06 2024-09-16T03:39:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018829+0000", 2024-09-16T03:39:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6320000000000008e-06 2024-09-16T03:39:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018835+0000", 2024-09-16T03:39:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0399999999999998e-06 2024-09-16T03:39:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018953+0000", 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117713 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018967+0000", 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4334e-05 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359078+0000", 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34011044499999998 2024-09-16T03:39:35.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359086+0000", 2024-09-16T03:39:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8839999999999994e-06 2024-09-16T03:39:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3574 4.28 4.f21f628 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:39:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.435342751, 2024-09-16T03:39:35.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.340272882, 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3574 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018849+0000", 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9599999999999998e-07 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018852+0000", 2024-09-16T03:39:35.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9749999999999999e-06 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018867+0000", 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5102e-05 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018882+0000", 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5319e-05 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.569 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018993+0000", 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110457 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018998+0000", 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.028e-06 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359114+0000", 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34011593099999998 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359121+0000", 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2740000000000002e-06 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3996 4.8 4:13b03fc9:::smithi15736696-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:05.315045+0000", 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.139145341999999, 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30191814700000003, 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.571 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3996 2024-09-16T03:39:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315045+0000", 2024-09-16T03:39:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315045+0000", 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315046+0000", 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9100000000000003e-07 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315049+0000", 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1020000000000001e-06 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315056+0000", 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5400000000000001e-06 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315062+0000", 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.908e-06 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.515314+0000", 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20025267999999999 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.515437+0000", 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122955 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.616750+0000", 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10131272199999999 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.616842+0000", 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2408e-05 2024-09-16T03:39:35.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.616964+0000", 2024-09-16T03:39:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121141 2024-09-16T03:39:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4658 4.28 4:146f84f0:::smithi15736696-6:head [cache-evict] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e150)", 2024-09-16T03:39:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.652963+0000", 2024-09-16T03:39:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8012279270000002, 2024-09-16T03:39:35.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30247697200000001, 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4658 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652963+0000", 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652963+0000", 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652966+0000", 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8090000000000001e-06 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652974+0000", 2024-09-16T03:39:35.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8490000000000006e-06 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652990+0000", 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.653002+0000", 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1895999999999999e-05 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.753468+0000", 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100466079 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.753489+0000", 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0479000000000002e-05 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.955202+0000", 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201713432 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.955360+0000", 2024-09-16T03:39:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000158102 2024-09-16T03:39:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.955440+0000", 2024-09-16T03:39:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9550999999999998e-05 2024-09-16T03:39:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.581 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.581 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:35.584 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:39:35.585 INFO:tasks.rados.rados.0.smithi157.stdout:1959: set_chunk oid 9 offset: 2203136 length: 563200 target oid 876508206c1ed0ad0189ebcb0460759e6dc37cd09ceb12376f98de4b74116035 offset: 0 snap -1 2024-09-16T03:39:35.585 INFO:tasks.rados.rados.0.smithi157.stdout:1960: read oid 31 snap -1 2024-09-16T03:39:35.585 INFO:tasks.rados.rados.0.smithi157.stdout:1960: expect (ObjNum 436 snap 0 seq_num 436) 2024-09-16T03:39:35.585 INFO:tasks.rados.rados.0.smithi157.stdout:1961: write (excl) oid 30 current snap is 0 2024-09-16T03:39:35.585 INFO:tasks.rados.rados.0.smithi157.stdout:1951: finishing set_chunk to oid 42 2024-09-16T03:39:35.585 INFO:tasks.rados.rados.0.smithi157.stdout:1951: got expected ENOENT (src dne) 2024-09-16T03:39:35.585 INFO:tasks.rados.rados.0.smithi157.stdout:1961: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-09-16T03:39:35.586 INFO:tasks.rados.rados.0.smithi157.stdout:1961: writing smithi15736696-30 from 544659 to 1260593 tid 1 2024-09-16T03:39:35.586 INFO:tasks.rados.rados.0.smithi157.stdout:1961: writing smithi15736696-30 from 1738984 to 2188130 tid 2 2024-09-16T03:39:35.586 INFO:tasks.rados.rados.0.smithi157.stdout:1961: writing smithi15736696-30 from 2784436 to 3344659 tid 3 2024-09-16T03:39:35.586 INFO:tasks.rados.rados.0.smithi157.stdout:1951: done (10 left) 2024-09-16T03:39:35.586 INFO:tasks.rados.rados.0.smithi157.stdout:1962: set_chunk oid 11 target oid 2024-09-16T03:39:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:35.586 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:35.581+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:35.586 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:35.587 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:35.587 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:35.587 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:35.587 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:35.587 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:35.587 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:35.587 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:35.587 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:35.587 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:35.587 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:35.587 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:35.587 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:35.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 'trim stale osdmaps' 2024-09-16T03:39:35.663 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:35.662+0000 7ff26836a640 -1 osd.11 155 handle_osd_map injecting map crc failure 2024-09-16T03:39:35.683 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:35.682+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:35.738 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:35.738 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:35.738 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:39:35.738 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:35.748 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:39:35.767 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:35.767 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:35.767 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:35.767 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:35.767 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:35.767 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:35.767 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:35.767 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:35.767 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:35.768 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:35.768 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:35.768 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:35.770 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:35.770 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-16T03:39:35.785 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:35.783+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:35.857 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:35.857 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:35.858 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:35.858 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:35.858 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:35.867 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:39:35.886 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:35.884+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:35.950 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-16T03:39:35.950 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-16T03:39:35.950 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-16T03:39:35.950 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-16T03:39:35.950 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-16T03:39:35.950 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-16T03:39:35.950 INFO:teuthology.orchestra.run.smithi064.stderr:bench [] [] [] [] 2024-09-16T03:39:35.951 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros 2024-09-16T03:39:35.951 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list 2024-09-16T03:39:35.951 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats 2024-09-16T03:39:35.951 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-16T03:39:35.951 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command 2024-09-16T03:39:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T03:39:35.953 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-16T03:39:35.978 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:35.978 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:35.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:35.978 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:35.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.979 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:34 4.e 4:773b934b:::smithi15736696-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:35.979 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:39:35.979 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 143.92064166700001, 2024-09-16T03:39:35.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50346527299999999, 2024-09-16T03:39:35.979 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.979 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.979 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.979 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.979 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-16T03:39:35.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.980 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:39:35.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:39:35.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054413+0000", 2024-09-16T03:39:35.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0669999999999999e-06 2024-09-16T03:39:35.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054525+0000", 2024-09-16T03:39:35.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111462 2024-09-16T03:39:35.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054538+0000", 2024-09-16T03:39:35.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3341e-05 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054545+0000", 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.584e-06 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054608+0000", 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2953999999999994e-05 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:35.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054613+0000", 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6110000000000001e-06 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154852+0000", 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10023838 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154915+0000", 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3138000000000004e-05 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:39:35.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155001+0000", 2024-09-16T03:39:35.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5730999999999996e-05 2024-09-16T03:39:35.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:35.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156166+0000", 2024-09-16T03:39:35.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001165485 2024-09-16T03:39:35.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255909+0000", 2024-09-16T03:39:35.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099742558999999995 2024-09-16T03:39:35.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557800+0000", 2024-09-16T03:39:35.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30189130600000003 2024-09-16T03:39:35.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:35.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557836+0000", 2024-09-16T03:39:35.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6566000000000002e-05 2024-09-16T03:39:35.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557877+0000", 2024-09-16T03:39:35.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0089000000000002e-05 2024-09-16T03:39:35.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.986 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.986 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:35 4.e 4:773b934b:::smithi15736696-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:35.986 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:39:35.986 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 143.92049333599999, 2024-09-16T03:39:35.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50344320499999995, 2024-09-16T03:39:35.986 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.986 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.987 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:35.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.987 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-16T03:39:35.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.987 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:39:35.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:39:35.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:39:35.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:39:35.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054564+0000", 2024-09-16T03:39:35.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5710000000000002e-06 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054569+0000", 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0490000000000003e-06 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054578+0000", 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054639+0000", 2024-09-16T03:39:35.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1061000000000004e-05 2024-09-16T03:39:35.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:35.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054644+0000", 2024-09-16T03:39:35.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999996e-06 2024-09-16T03:39:35.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155607+0000", 2024-09-16T03:39:35.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100963372 2024-09-16T03:39:35.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155652+0000", 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155722+0000", 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0130999999999997e-05 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.157921+0000", 2024-09-16T03:39:35.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021990159999999998 2024-09-16T03:39:35.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255940+0000", 2024-09-16T03:39:35.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098018569999999999 2024-09-16T03:39:35.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:35.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557930+0000", 2024-09-16T03:39:35.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.301990333 2024-09-16T03:39:35.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557963+0000", 2024-09-16T03:39:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2348000000000003e-05 2024-09-16T03:39:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558003+0000", 2024-09-16T03:39:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:39:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.993 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:36 4.e 4:773b934b:::smithi15736696-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 143.92045464700001, 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50353811800000003, 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:35.994 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054599+0000", 2024-09-16T03:39:35.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:39:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054601+0000", 2024-09-16T03:39:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4360000000000001e-06 2024-09-16T03:39:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054606+0000", 2024-09-16T03:39:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1019999999999996e-06 2024-09-16T03:39:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:35.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054610+0000", 2024-09-16T03:39:35.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.681e-06 2024-09-16T03:39:35.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054662+0000", 2024-09-16T03:39:35.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1545000000000002e-05 2024-09-16T03:39:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054664+0000", 2024-09-16T03:39:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1919999999999999e-06 2024-09-16T03:39:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155903+0000", 2024-09-16T03:39:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101239068 2024-09-16T03:39:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155970+0000", 2024-09-16T03:39:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6934999999999995e-05 2024-09-16T03:39:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558029+0000", 2024-09-16T03:39:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40205888899999997 2024-09-16T03:39:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:35.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558078+0000", 2024-09-16T03:39:36.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9842999999999997e-05 2024-09-16T03:39:36.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558136+0000", 2024-09-16T03:39:36.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7985000000000001e-05 2024-09-16T03:39:36.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.000 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.000 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.000 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:90 3.0 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:39:36.000 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:39:36.001 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 100.41844147800001, 2024-09-16T03:39:36.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62288137399999999, 2024-09-16T03:39:36.001 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.001 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.001 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:39:36.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556614+0000", 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3439999999999999e-06 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556621+0000", 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7440000000000001e-06 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556632+0000", 2024-09-16T03:39:36.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1725e-05 2024-09-16T03:39:36.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556640+0000", 2024-09-16T03:39:36.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0239999999999997e-06 2024-09-16T03:39:36.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556709+0000", 2024-09-16T03:39:36.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8787000000000004e-05 2024-09-16T03:39:36.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:36.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556718+0000", 2024-09-16T03:39:36.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6920000000000008e-06 2024-09-16T03:39:36.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758133+0000", 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201414977 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758273+0000", 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140684 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758479+0000", 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020508599999999999 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862766+0000", 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104287952 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.878363+0000", 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015596601999999999 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960000+0000", 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081636630000000002 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960162+0000", 2024-09-16T03:39:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000162597 2024-09-16T03:39:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179366+0000", 2024-09-16T03:39:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21920357100000001 2024-09-16T03:39:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179411+0000", 2024-09-16T03:39:36.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5461999999999999e-05 2024-09-16T03:39:36.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179493+0000", 2024-09-16T03:39:36.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1496999999999996e-05 2024-09-16T03:39:36.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.008 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.008 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.008 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:91 3.0 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:39:36.008 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:39:36.008 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 100.41839652, 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70460932600000004, 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:39:36.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556657+0000", 2024-09-16T03:39:36.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:39:36.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556663+0000", 2024-09-16T03:39:36.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6579999999999999e-06 2024-09-16T03:39:36.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556670+0000", 2024-09-16T03:39:36.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3189999999999997e-06 2024-09-16T03:39:36.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556675+0000", 2024-09-16T03:39:36.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000002e-06 2024-09-16T03:39:36.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656929+0000", 2024-09-16T03:39:36.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10025371299999999 2024-09-16T03:39:36.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:36.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656944+0000", 2024-09-16T03:39:36.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5460000000000001e-05 2024-09-16T03:39:36.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861436+0000", 2024-09-16T03:39:36.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.204491969 2024-09-16T03:39:36.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861676+0000", 2024-09-16T03:39:36.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024051899999999999 2024-09-16T03:39:36.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:36.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861828+0000", 2024-09-16T03:39:36.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015110199999999999 2024-09-16T03:39:36.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862775+0000", 2024-09-16T03:39:36.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094718600000000003 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079130+0000", 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21635549200000001 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179557+0000", 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10042638399999999 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179972+0000", 2024-09-16T03:39:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041574 2024-09-16T03:39:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261034+0000", 2024-09-16T03:39:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081061604999999995 2024-09-16T03:39:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261142+0000", 2024-09-16T03:39:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107688 2024-09-16T03:39:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261266+0000", 2024-09-16T03:39:36.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124082 2024-09-16T03:39:36.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.016 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:94 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:39:36.016 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:39:36.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 100.418370395, 2024-09-16T03:39:36.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70479513699999996, 2024-09-16T03:39:36.016 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.017 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.017 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:39:36.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:39:36.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:39:36.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-09-16T03:39:36.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556685+0000", 2024-09-16T03:39:36.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4279999999999999e-06 2024-09-16T03:39:36.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556689+0000", 2024-09-16T03:39:36.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.709e-06 2024-09-16T03:39:36.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556693+0000", 2024-09-16T03:39:36.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2939999999999999e-06 2024-09-16T03:39:36.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657036+0000", 2024-09-16T03:39:36.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10034209600000001 2024-09-16T03:39:36.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:36.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657054+0000", 2024-09-16T03:39:36.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8162999999999999e-05 2024-09-16T03:39:36.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862279+0000", 2024-09-16T03:39:36.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20522560200000001 2024-09-16T03:39:36.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862380+0000", 2024-09-16T03:39:36.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101178 2024-09-16T03:39:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862509+0000", 2024-09-16T03:39:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128909 2024-09-16T03:39:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.863063+0000", 2024-09-16T03:39:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055327600000000001 2024-09-16T03:39:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079172+0000", 2024-09-16T03:39:36.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21610910899999999 2024-09-16T03:39:36.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179932+0000", 2024-09-16T03:39:36.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100760575 2024-09-16T03:39:36.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179997+0000", 2024-09-16T03:39:36.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5018999999999995e-05 2024-09-16T03:39:36.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261318+0000", 2024-09-16T03:39:36.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081320838000000006 2024-09-16T03:39:36.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261385+0000", 2024-09-16T03:39:36.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7115999999999999e-05 2024-09-16T03:39:36.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261478+0000", 2024-09-16T03:39:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2360000000000003e-05 2024-09-16T03:39:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1742 3.0 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head [write 0~1659392] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T03:39:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:39:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 100.41834199500001, 2024-09-16T03:39:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62317617400000003, 2024-09-16T03:39:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.025 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.025 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1742 2024-09-16T03:39:36.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.025 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:39:36.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556713+0000", 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7209999999999999e-06 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558531+0000", 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018184480000000001 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558539+0000", 2024-09-16T03:39:36.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0830000000000004e-06 2024-09-16T03:39:36.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558546+0000", 2024-09-16T03:39:36.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3369999999999996e-06 2024-09-16T03:39:36.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759148+0000", 2024-09-16T03:39:36.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20060235500000001 2024-09-16T03:39:36.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759230+0000", 2024-09-16T03:39:36.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2391000000000004e-05 2024-09-16T03:39:36.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:36.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759329+0000", 2024-09-16T03:39:36.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9147000000000005e-05 2024-09-16T03:39:36.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862771+0000", 2024-09-16T03:39:36.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103441085 2024-09-16T03:39:36.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960102+0000", 2024-09-16T03:39:36.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097331259000000003 2024-09-16T03:39:36.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960218+0000", 2024-09-16T03:39:36.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011581100000000001 2024-09-16T03:39:36.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079074+0000", 2024-09-16T03:39:36.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118856436 2024-09-16T03:39:36.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179758+0000", 2024-09-16T03:39:36.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100683884 2024-09-16T03:39:36.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179800+0000", 2024-09-16T03:39:36.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1928000000000002e-05 2024-09-16T03:39:36.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179887+0000", 2024-09-16T03:39:36.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7288999999999997e-05 2024-09-16T03:39:36.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.032 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.032 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.032 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:89 3.0 3:1c07e18b:::d3cef63b99e391058a88d444a56984ba6fc25b22efbc98e24ba274246c542d3c:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:36.032 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:39:36.032 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.638939499000003, 2024-09-16T03:39:36.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50388158999999999, 2024-09-16T03:39:36.032 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:39:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:39:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336115+0000", 2024-09-16T03:39:36.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.866e-06 2024-09-16T03:39:36.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336123+0000", 2024-09-16T03:39:36.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6990000000000003e-06 2024-09-16T03:39:36.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336136+0000", 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3342e-05 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336143+0000", 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2099999999999998e-06 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336214+0000", 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1538000000000005e-05 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336342+0000", 2024-09-16T03:39:36.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127408 2024-09-16T03:39:36.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:36.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336593+0000", 2024-09-16T03:39:36.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025150500000000001 2024-09-16T03:39:36.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337284+0000", 2024-09-16T03:39:36.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069060900000000001 2024-09-16T03:39:36.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338068+0000", 2024-09-16T03:39:36.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078438900000000003 2024-09-16T03:39:36.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338107+0000", 2024-09-16T03:39:36.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8858e-05 2024-09-16T03:39:36.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538164+0000", 2024-09-16T03:39:36.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200056971 2024-09-16T03:39:36.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839843+0000", 2024-09-16T03:39:36.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30167921599999997 2024-09-16T03:39:36.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839905+0000", 2024-09-16T03:39:36.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1542000000000001e-05 2024-09-16T03:39:36.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839995+0000", 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0437000000000002e-05 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:98 3.0 3:04dfaaf9:::3de53e1d584ced9b0f970855867dbc9a14b3fc90962aa20a81c1ed25ff4b961f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.637611598000007, 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70310345299999999, 2024-09-16T03:39:36.039 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.040 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:39:36.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:39:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337442+0000", 2024-09-16T03:39:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0729999999999999e-06 2024-09-16T03:39:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337447+0000", 2024-09-16T03:39:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3259999999999997e-06 2024-09-16T03:39:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337457+0000", 2024-09-16T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-09-16T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337462+0000", 2024-09-16T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0780000000000003e-06 2024-09-16T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337511+0000", 2024-09-16T03:39:36.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:39:36.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337574+0000", 2024-09-16T03:39:36.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3114999999999995e-05 2024-09-16T03:39:36.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:36.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337723+0000", 2024-09-16T03:39:36.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014882299999999999 2024-09-16T03:39:36.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.339039+0000", 2024-09-16T03:39:36.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001315524 2024-09-16T03:39:36.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538225+0000", 2024-09-16T03:39:36.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19918607199999999 2024-09-16T03:39:36.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638838+0000", 2024-09-16T03:39:36.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100613466 2024-09-16T03:39:36.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.739726+0000", 2024-09-16T03:39:36.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100887854 2024-09-16T03:39:36.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040319+0000", 2024-09-16T03:39:36.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30059265699999999 2024-09-16T03:39:36.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040398+0000", 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8852999999999999e-05 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040545+0000", 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147236 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:101 3.0 3:090a0a71:::0ecec3e4e81d718f460a9826fb4cef2398bf4c951ee653be84827f33946eef5f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:39:36.046 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.538440851999994, 2024-09-16T03:39:36.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60435370600000005, 2024-09-16T03:39:36.047 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.047 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.047 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:39:36.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436614+0000", 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.844e-06 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436623+0000", 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0739999999999999e-06 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436636+0000", 2024-09-16T03:39:36.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2629999999999999e-05 2024-09-16T03:39:36.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436643+0000", 2024-09-16T03:39:36.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7489999999999998e-06 2024-09-16T03:39:36.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536906+0000", 2024-09-16T03:39:36.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10026262599999999 2024-09-16T03:39:36.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536979+0000", 2024-09-16T03:39:36.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3311000000000003e-05 2024-09-16T03:39:36.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:36.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537109+0000", 2024-09-16T03:39:36.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013015800000000001 2024-09-16T03:39:36.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538241+0000", 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001131726 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638904+0000", 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066240799999999 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.940200+0000", 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30129597299999999 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040604+0000", 2024-09-16T03:39:36.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040478999999999 2024-09-16T03:39:36.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040759+0000", 2024-09-16T03:39:36.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015445500000000001 2024-09-16T03:39:36.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040824+0000", 2024-09-16T03:39:36.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5633e-05 2024-09-16T03:39:36.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040966+0000", 2024-09-16T03:39:36.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014132899999999999 2024-09-16T03:39:36.053 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.053 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.053 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.053 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:102 3.0 3:17742417:::632266e231c4c385ace473a8f4464285cf620fbc8799f68869353e2aa4660567:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:36.053 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.538394026000006, 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.604540823, 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:39:36.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436660+0000", 2024-09-16T03:39:36.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:39:36.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436663+0000", 2024-09-16T03:39:36.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1599999999999998e-06 2024-09-16T03:39:36.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436670+0000", 2024-09-16T03:39:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.742e-06 2024-09-16T03:39:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436675+0000", 2024-09-16T03:39:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3839999999999996e-06 2024-09-16T03:39:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537499+0000", 2024-09-16T03:39:36.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100823987 2024-09-16T03:39:36.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537602+0000", 2024-09-16T03:39:36.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102837 2024-09-16T03:39:36.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:36.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537793+0000", 2024-09-16T03:39:36.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019123499999999999 2024-09-16T03:39:36.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638962+0000", 2024-09-16T03:39:36.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101168944 2024-09-16T03:39:36.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.840035+0000", 2024-09-16T03:39:36.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201073481 2024-09-16T03:39:36.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040706+0000", 2024-09-16T03:39:36.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200670606 2024-09-16T03:39:36.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041009+0000", 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030271600000000002 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041087+0000", 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8310000000000001e-05 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041127+0000", 2024-09-16T03:39:36.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:39:36.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041200+0000", 2024-09-16T03:39:36.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2381999999999999e-05 2024-09-16T03:39:36.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.060 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.060 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:103 3.0 3:28447854:::c811732c201b76a96e1532c4d9a18d4f4f2a8ddec3bf79945689a1a62c8a5a25:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:36.060 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.538348776999996, 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60467526100000002, 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:39:36.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436705+0000", 2024-09-16T03:39:36.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.23e-07 2024-09-16T03:39:36.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436719+0000", 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3957999999999999e-05 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436725+0000", 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6100000000000002e-06 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436730+0000", 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0609999999999997e-06 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538257+0000", 2024-09-16T03:39:36.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10152739500000001 2024-09-16T03:39:36.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538314+0000", 2024-09-16T03:39:36.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7355000000000002e-05 2024-09-16T03:39:36.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:36.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538435+0000", 2024-09-16T03:39:36.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120967 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638968+0000", 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100532994 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040662+0000", 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40169386299999998 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041059+0000", 2024-09-16T03:39:36.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039723900000000002 2024-09-16T03:39:36.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041247+0000", 2024-09-16T03:39:36.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018756900000000001 2024-09-16T03:39:36.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041286+0000", 2024-09-16T03:39:36.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8971999999999999e-05 2024-09-16T03:39:36.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041318+0000", 2024-09-16T03:39:36.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1628999999999998e-05 2024-09-16T03:39:36.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041379+0000", 2024-09-16T03:39:36.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1926000000000003e-05 2024-09-16T03:39:36.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.068 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2473 4.e 4:765e9179:::smithi15736696-44:head [write 1830877~681726] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:36.068 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:39:36.068 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.964363551999995, 2024-09-16T03:39:36.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50285070099999996, 2024-09-16T03:39:36.068 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.068 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.068 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2473 2024-09-16T03:39:36.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.068 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:39:36.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:39:36.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010691+0000", 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-06 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011744+0000", 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001052486 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011755+0000", 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1073000000000001e-05 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011761+0000", 2024-09-16T03:39:36.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1600000000000003e-06 2024-09-16T03:39:36.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011804+0000", 2024-09-16T03:39:36.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2697999999999997e-05 2024-09-16T03:39:36.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011885+0000", 2024-09-16T03:39:36.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.119e-05 2024-09-16T03:39:36.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:39:36.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.012013+0000", 2024-09-16T03:39:36.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128066 2024-09-16T03:39:36.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.013923+0000", 2024-09-16T03:39:36.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019104739999999999 2024-09-16T03:39:36.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.412920+0000", 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39899685200000001 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513347+0000", 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100427167 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513490+0000", 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142163 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513540+0000", 2024-09-16T03:39:36.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0482000000000002e-05 2024-09-16T03:39:36.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.074 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.074 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:136 3.0 3:1b68d773:::374e1d62491a062bcb5530630b4c28c4696d618a79860b96215d227206cd62aa:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:36.074 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:39:36.074 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.860559013, 2024-09-16T03:39:36.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50236015199999995, 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114494+0000", 2024-09-16T03:39:36.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4499999999999997e-07 2024-09-16T03:39:36.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114497+0000", 2024-09-16T03:39:36.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:39:36.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114506+0000", 2024-09-16T03:39:36.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5759999999999995e-06 2024-09-16T03:39:36.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114511+0000", 2024-09-16T03:39:36.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0649999999999998e-06 2024-09-16T03:39:36.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115277+0000", 2024-09-16T03:39:36.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076592800000000003 2024-09-16T03:39:36.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115368+0000", 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1037000000000003e-05 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115573+0000", 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000204594 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116773+0000", 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001200062 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.121514+0000", 2024-09-16T03:39:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047407270000000001 2024-09-16T03:39:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414056+0000", 2024-09-16T03:39:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29254249700000001 2024-09-16T03:39:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414192+0000", 2024-09-16T03:39:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013572000000000001 2024-09-16T03:39:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616663+0000", 2024-09-16T03:39:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20247094800000001 2024-09-16T03:39:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616728+0000", 2024-09-16T03:39:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5217999999999998e-05 2024-09-16T03:39:36.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616854+0000", 2024-09-16T03:39:36.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126064 2024-09-16T03:39:36.081 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.081 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.081 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.081 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:137 3.0 3:20b08a9e:::0ba23f501764ab3447cc323c825ee8bff84c3c6e340c68c08117252e1f0b39bb:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:36.081 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:39:36.081 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 79.860515067999998, 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50256677599999999, 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:39:36.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:39:36.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1200000000000003e-07 2024-09-16T03:39:36.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114542+0000", 2024-09-16T03:39:36.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7400000000000002e-06 2024-09-16T03:39:36.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114547+0000", 2024-09-16T03:39:36.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.42e-06 2024-09-16T03:39:36.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114550+0000", 2024-09-16T03:39:36.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4470000000000002e-06 2024-09-16T03:39:36.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115951+0000", 2024-09-16T03:39:36.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014005459999999999 2024-09-16T03:39:36.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116047+0000", 2024-09-16T03:39:36.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6899999999999997e-05 2024-09-16T03:39:36.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:36.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116270+0000", 2024-09-16T03:39:36.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022291699999999999 2024-09-16T03:39:36.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.117236+0000", 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096522299999999997 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.218932+0000", 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10169644799999999 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414271+0000", 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19533916300000001 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.514717+0000", 2024-09-16T03:39:36.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100445879 2024-09-16T03:39:36.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616919+0000", 2024-09-16T03:39:36.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10220199100000001 2024-09-16T03:39:36.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616980+0000", 2024-09-16T03:39:36.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0977999999999998e-05 2024-09-16T03:39:36.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.617105+0000", 2024-09-16T03:39:36.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012461199999999999 2024-09-16T03:39:36.088 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.088 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.088 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.088 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3402 4.1b 4:dbb349e8:::smithi15736696-46:head [stat,delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T03:39:36.088 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:46.357580+0000", 2024-09-16T03:39:36.088 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.617473136999998, 2024-09-16T03:39:36.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.80350077600000003, 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3402 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357580+0000", 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357580+0000", 2024-09-16T03:39:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357582+0000", 2024-09-16T03:39:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1890000000000001e-06 2024-09-16T03:39:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357588+0000", 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.896e-06 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357606+0000", 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7841000000000001e-05 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357617+0000", 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1698999999999999e-05 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357697+0000", 2024-09-16T03:39:36.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9822999999999998e-05 2024-09-16T03:39:36.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:36.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357708+0000", 2024-09-16T03:39:36.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1119e-05 2024-09-16T03:39:36.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.962316+0000", 2024-09-16T03:39:36.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60460744899999996 2024-09-16T03:39:36.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.962430+0000", 2024-09-16T03:39:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011423 2024-09-16T03:39:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,6,8,9,15", 2024-09-16T03:39:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.962530+0000", 2024-09-16T03:39:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100005 2024-09-16T03:39:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063659+0000", 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101128564 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063691+0000", 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2413999999999998e-05 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063701+0000", 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9180000000000006e-06 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063733+0000", 2024-09-16T03:39:36.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2416000000000003e-05 2024-09-16T03:39:36.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063758+0000", 2024-09-16T03:39:36.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.421e-05 2024-09-16T03:39:36.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.160869+0000", 2024-09-16T03:39:36.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097111571999999993 2024-09-16T03:39:36.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.160922+0000", 2024-09-16T03:39:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3025999999999997e-05 2024-09-16T03:39:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.161081+0000", 2024-09-16T03:39:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000158405 2024-09-16T03:39:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.097 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4662 4.20 4:049ac94d:::smithi15736696-40:head [write 726168~704991] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:36.097 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.832696+0000", 2024-09-16T03:39:36.097 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1423573060000001, 2024-09-16T03:39:36.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52767523100000002, 2024-09-16T03:39:36.097 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.097 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.097 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.097 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.097 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4662 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.832696+0000", 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.832696+0000", 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.832699+0000", 2024-09-16T03:39:36.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4599999999999999e-06 2024-09-16T03:39:36.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.833419+0000", 2024-09-16T03:39:36.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072038500000000004 2024-09-16T03:39:36.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.833441+0000", 2024-09-16T03:39:36.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1135e-05 2024-09-16T03:39:36.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.833460+0000", 2024-09-16T03:39:36.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9296e-05 2024-09-16T03:39:36.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.955818+0000", 2024-09-16T03:39:36.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12235790100000001 2024-09-16T03:39:36.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.955889+0000", 2024-09-16T03:39:36.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1039999999999997e-05 2024-09-16T03:39:36.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,2,8,10,15", 2024-09-16T03:39:36.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.956001+0000", 2024-09-16T03:39:36.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111692 2024-09-16T03:39:36.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.056147+0000", 2024-09-16T03:39:36.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100146179 2024-09-16T03:39:36.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.056166+0000", 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9656999999999998e-05 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.056217+0000", 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0238999999999998e-05 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.057817+0000", 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001600193 2024-09-16T03:39:36.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360100+0000", 2024-09-16T03:39:36.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30228278800000002 2024-09-16T03:39:36.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360244+0000", 2024-09-16T03:39:36.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014397800000000001 2024-09-16T03:39:36.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360289+0000", 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5374999999999998e-05 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360371+0000", 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1913e-05 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4663 4.20 4:049ac94d:::smithi15736696-40:head [write 1895071~683474] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.838668+0000", 2024-09-16T03:39:36.104 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1363845599999998, 2024-09-16T03:39:36.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52198192799999998, 2024-09-16T03:39:36.105 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.105 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.105 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.105 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.105 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4663 2024-09-16T03:39:36.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.105 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.838668+0000", 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.838668+0000", 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.838670+0000", 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.401e-06 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839234+0000", 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056420899999999998 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839248+0000", 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4208e-05 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839255+0000", 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.742e-06 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.057859+0000", 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21860384799999999 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.057934+0000", 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5638999999999999e-05 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,2,8,10,15", 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.058042+0000", 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107314 2024-09-16T03:39:36.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.059148+0000", 2024-09-16T03:39:36.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011060079999999999 2024-09-16T03:39:36.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159745+0000", 2024-09-16T03:39:36.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10059675 2024-09-16T03:39:36.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159780+0000", 2024-09-16T03:39:36.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5351000000000001e-05 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.259887+0000", 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10010670200000001 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360172+0000", 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100285759 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360535+0000", 2024-09-16T03:39:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036267599999999998 2024-09-16T03:39:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360570+0000", 2024-09-16T03:39:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5138999999999997e-05 2024-09-16T03:39:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360650+0000", 2024-09-16T03:39:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0181999999999999e-05 2024-09-16T03:39:36.114 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4664 4.20 4:049ac94d:::smithi15736696-40:head [setxattr _header (56) in=7b,truncate 3126168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.839266+0000", 2024-09-16T03:39:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1357874299999997, 2024-09-16T03:39:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52151806899999997, 2024-09-16T03:39:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.115 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4664 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839266+0000", 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839266+0000", 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839266+0000", 2024-09-16T03:39:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:39:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839270+0000", 2024-09-16T03:39:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2100000000000002e-06 2024-09-16T03:39:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839273+0000", 2024-09-16T03:39:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.715e-06 2024-09-16T03:39:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839276+0000", 2024-09-16T03:39:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2200000000000001e-06 2024-09-16T03:39:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159242+0000", 2024-09-16T03:39:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31996554700000002 2024-09-16T03:39:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159311+0000", 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9419999999999996e-05 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,2,8,10,15", 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159423+0000", 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011188600000000001 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360380+0000", 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20095712900000001 2024-09-16T03:39:36.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360414+0000", 2024-09-16T03:39:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3176000000000002e-05 2024-09-16T03:39:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360449+0000", 2024-09-16T03:39:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4872000000000001e-05 2024-09-16T03:39:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360479+0000", 2024-09-16T03:39:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0402000000000001e-05 2024-09-16T03:39:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360509+0000", 2024-09-16T03:39:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9907999999999999e-05 2024-09-16T03:39:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360685+0000", 2024-09-16T03:39:36.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175897 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360717+0000", 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2342999999999998e-05 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360784+0000", 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6514999999999996e-05 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4665 4.20 4:049ac94d:::smithi15736696-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.839281+0000", 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1357721830000003, 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.82253416700000004, 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4665 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.123 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839281+0000", 2024-09-16T03:39:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839281+0000", 2024-09-16T03:39:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839281+0000", 2024-09-16T03:39:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3299999999999998e-07 2024-09-16T03:39:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839283+0000", 2024-09-16T03:39:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6470000000000001e-06 2024-09-16T03:39:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839286+0000", 2024-09-16T03:39:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.506e-06 2024-09-16T03:39:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839290+0000", 2024-09-16T03:39:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9559999999999999e-06 2024-09-16T03:39:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.661604+0000", 2024-09-16T03:39:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.822313819 2024-09-16T03:39:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.661677+0000", 2024-09-16T03:39:36.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2712000000000004e-05 2024-09-16T03:39:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.661815+0000", 2024-09-16T03:39:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013819400000000001 2024-09-16T03:39:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.127 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.127 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:36.128 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:36.035+0000 7fa3886e6640 -1 osd.10 155 handle_osd_map injecting map crc failure 2024-09-16T03:39:36.129 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:39:36.130 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:36.127+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:36.230 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:36.229+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:36.278 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:36.279 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:39:36.279 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:36.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:39:36.331 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:36.329+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:36.394 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:36.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:36.394 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:36.394 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:36.395 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:36.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:39:36.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:39:36.431 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:36.430+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:36.525 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:36.526 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:36.532 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:36.531+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:36.543 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:36.544 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:36.555 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:36.555 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:36.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:36.555 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:36.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.556 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:303 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:36.556 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:39:36.556 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 140.14085806200001, 2024-09-16T03:39:36.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50956986000000004, 2024-09-16T03:39:36.556 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.556 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.556 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.556 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.556 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.556 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 303 2024-09-16T03:39:36.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.557 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:39:36.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:39:36.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411721+0000", 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0200000000000001e-06 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411729+0000", 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1240000000000005e-06 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411744+0000", 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099000000000001e-05 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411751+0000", 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.029e-06 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511892+0000", 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100140879 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511961+0000", 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9379000000000001e-05 2024-09-16T03:39:36.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:36.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.613647+0000", 2024-09-16T03:39:36.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101685994 2024-09-16T03:39:36.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.614294+0000", 2024-09-16T03:39:36.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064692799999999996 2024-09-16T03:39:36.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.714962+0000", 2024-09-16T03:39:36.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066765 2024-09-16T03:39:36.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921157+0000", 2024-09-16T03:39:36.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206195463 2024-09-16T03:39:36.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921233+0000", 2024-09-16T03:39:36.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5446999999999998e-05 2024-09-16T03:39:36.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921288+0000", 2024-09-16T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5847999999999998e-05 2024-09-16T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:503 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 134.73078351000001, 2024-09-16T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60521621599999997, 2024-09-16T03:39:36.562 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.563 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.563 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.563 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.563 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 503 2024-09-16T03:39:36.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.563 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:39:36.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:39:36.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821794+0000", 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821798+0000", 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2030000000000002e-06 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821804+0000", 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821810+0000", 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4249999999999997e-06 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922212+0000", 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040228800000001 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922307+0000", 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5089999999999999e-05 2024-09-16T03:39:36.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.126007+0000", 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20369931499999999 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.127605+0000", 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001598213 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.327263+0000", 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199657905 2024-09-16T03:39:36.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426900+0000", 2024-09-16T03:39:36.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099637217 2024-09-16T03:39:36.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426963+0000", 2024-09-16T03:39:36.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:39:36.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.427009+0000", 2024-09-16T03:39:36.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6533000000000003e-05 2024-09-16T03:39:36.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.569 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1227 4.a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v18,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:39:36.569 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:39:36.569 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 119.87660724600001, 2024-09-16T03:39:36.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.44160531400000003, 2024-09-16T03:39:36.569 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.569 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.570 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.570 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1227 2024-09-16T03:39:36.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.570 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:39:36.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:39:36.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675972+0000", 2024-09-16T03:39:36.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7269999999999999e-06 2024-09-16T03:39:36.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675981+0000", 2024-09-16T03:39:36.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.166e-06 2024-09-16T03:39:36.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675996+0000", 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4901e-05 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676005+0000", 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4579999999999993e-06 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676081+0000", 2024-09-16T03:39:36.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6157999999999998e-05 2024-09-16T03:39:36.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676188+0000", 2024-09-16T03:39:36.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106871 2024-09-16T03:39:36.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:36.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.016594+0000", 2024-09-16T03:39:36.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34040622300000001 2024-09-16T03:39:36.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017089+0000", 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049523899999999997 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017779+0000", 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069032200000000005 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117504+0000", 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099724207999999995 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117543+0000", 2024-09-16T03:39:36.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9208999999999997e-05 2024-09-16T03:39:36.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117575+0000", 2024-09-16T03:39:36.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1832000000000003e-05 2024-09-16T03:39:36.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.575 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.575 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1523 4.a 4:50107449:::smithi15736696-27:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:39:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:39:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 111.18956592000001, 2024-09-16T03:39:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.410522742, 2024-09-16T03:39:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1523 2024-09-16T03:39:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.576 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:39:36.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:39:36.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363012+0000", 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.269e-06 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363016+0000", 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8920000000000002e-06 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363027+0000", 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1079e-05 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463160+0000", 2024-09-16T03:39:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10013267100000001 2024-09-16T03:39:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463297+0000", 2024-09-16T03:39:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013717699999999999 2024-09-16T03:39:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463381+0000", 2024-09-16T03:39:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3701000000000004e-05 2024-09-16T03:39:36.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.773533+0000", 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31015295300000001 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1690 4.a 4:51338165:::smithi15736696-4:head [assert-version v63,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 106.102293253, 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52002349699999995, 2024-09-16T03:39:36.580 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.581 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.581 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.581 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.581 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.581 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1690 2024-09-16T03:39:36.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.581 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:39:36.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:39:36.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450286+0000", 2024-09-16T03:39:36.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-16T03:39:36.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450293+0000", 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5089999999999996e-06 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450308+0000", 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4955e-05 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450316+0000", 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7079999999999997e-06 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450386+0000", 2024-09-16T03:39:36.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9715999999999995e-05 2024-09-16T03:39:36.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450478+0000", 2024-09-16T03:39:36.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2610999999999998e-05 2024-09-16T03:39:36.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:36.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.665802+0000", 2024-09-16T03:39:36.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21532342700000001 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668724+0000", 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029220600000000002 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668758+0000", 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.434e-05 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970243+0000", 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30148501 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970278+0000", 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4965000000000002e-05 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970307+0000", 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8968000000000001e-05 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.587 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1654 4.12 4:4a9d235d:::smithi15736696-1:head [write 652138~461367 in=461367b,stat out=16b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:36.587 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:39:36.587 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 106.01053976199999, 2024-09-16T03:39:36.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42790198699999998, 2024-09-16T03:39:36.587 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.587 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.587 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.587 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.587 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1654 2024-09-16T03:39:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:39:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:39:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542039+0000", 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0250000000000001e-06 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542682+0000", 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064305099999999995 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542698+0000", 2024-09-16T03:39:36.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6469000000000001e-05 2024-09-16T03:39:36.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542710+0000", 2024-09-16T03:39:36.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1173e-05 2024-09-16T03:39:36.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542767+0000", 2024-09-16T03:39:36.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7584000000000003e-05 2024-09-16T03:39:36.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:36.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542775+0000", 2024-09-16T03:39:36.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6149999999999999e-06 2024-09-16T03:39:36.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666597+0000", 2024-09-16T03:39:36.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123821815 2024-09-16T03:39:36.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666745+0000", 2024-09-16T03:39:36.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148675 2024-09-16T03:39:36.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:39:36.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666935+0000", 2024-09-16T03:39:36.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018963899999999999 2024-09-16T03:39:36.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969207+0000", 2024-09-16T03:39:36.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302272031 2024-09-16T03:39:36.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969374+0000", 2024-09-16T03:39:36.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016686699999999999 2024-09-16T03:39:36.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969533+0000", 2024-09-16T03:39:36.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015909299999999999 2024-09-16T03:39:36.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969560+0000", 2024-09-16T03:39:36.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7148000000000002e-05 2024-09-16T03:39:36.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969631+0000", 2024-09-16T03:39:36.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1174000000000007e-05 2024-09-16T03:39:36.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969685+0000", 2024-09-16T03:39:36.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3613000000000003e-05 2024-09-16T03:39:36.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969764+0000", 2024-09-16T03:39:36.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9281000000000002e-05 2024-09-16T03:39:36.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969939+0000", 2024-09-16T03:39:36.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473400000000001 2024-09-16T03:39:36.595 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.595 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.595 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.596 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1655 4.12 4:4a9d235d:::smithi15736696-1:head [write 1574940~677198 in=677198b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:36.596 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:39:36.596 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 106.009847644, 2024-09-16T03:39:36.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42773147700000003, 2024-09-16T03:39:36.596 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.596 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.596 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.596 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.596 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.596 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.596 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1655 2024-09-16T03:39:36.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.597 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:39:36.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:39:36.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542730+0000", 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.085e-06 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543338+0000", 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060826100000000002 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543351+0000", 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2789000000000001e-05 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543360+0000", 2024-09-16T03:39:36.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1970000000000002e-06 2024-09-16T03:39:36.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543420+0000", 2024-09-16T03:39:36.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9562999999999999e-05 2024-09-16T03:39:36.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:36.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543431+0000", 2024-09-16T03:39:36.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0866999999999999e-05 2024-09-16T03:39:36.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.768963+0000", 2024-09-16T03:39:36.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.225532487 2024-09-16T03:39:36.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769029+0000", 2024-09-16T03:39:36.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5881000000000002e-05 2024-09-16T03:39:36.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:39:36.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769134+0000", 2024-09-16T03:39:36.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104923 2024-09-16T03:39:36.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969639+0000", 2024-09-16T03:39:36.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20050517900000001 2024-09-16T03:39:36.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969978+0000", 2024-09-16T03:39:36.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033848599999999998 2024-09-16T03:39:36.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970206+0000", 2024-09-16T03:39:36.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022867300000000001 2024-09-16T03:39:36.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970248+0000", 2024-09-16T03:39:36.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1433999999999997e-05 2024-09-16T03:39:36.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970299+0000", 2024-09-16T03:39:36.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0677999999999998e-05 2024-09-16T03:39:36.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970338+0000", 2024-09-16T03:39:36.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9646e-05 2024-09-16T03:39:36.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970377+0000", 2024-09-16T03:39:36.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8439999999999998e-05 2024-09-16T03:39:36.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970460+0000", 2024-09-16T03:39:36.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3887999999999994e-05 2024-09-16T03:39:36.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.604 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.604 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1656 4.12 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2252138] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:36.604 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:39:36.604 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 106.00920157199999, 2024-09-16T03:39:36.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72904596899999996, 2024-09-16T03:39:36.605 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.605 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.605 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.605 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.605 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1656 2024-09-16T03:39:36.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.605 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:39:36.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:39:36.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543376+0000", 2024-09-16T03:39:36.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4200000000000004e-07 2024-09-16T03:39:36.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543383+0000", 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6290000000000004e-06 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543390+0000", 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543396+0000", 2024-09-16T03:39:36.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.02e-06 2024-09-16T03:39:36.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543461+0000", 2024-09-16T03:39:36.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4876999999999996e-05 2024-09-16T03:39:36.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:36.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543465+0000", 2024-09-16T03:39:36.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2139999999999998e-06 2024-09-16T03:39:36.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970490+0000", 2024-09-16T03:39:36.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42702478300000002 2024-09-16T03:39:36.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970556+0000", 2024-09-16T03:39:36.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6137999999999996e-05 2024-09-16T03:39:36.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:39:36.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970670+0000", 2024-09-16T03:39:36.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114416 2024-09-16T03:39:36.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971375+0000", 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070496000000000003 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071773+0000", 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100397945 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071823+0000", 2024-09-16T03:39:36.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:39:36.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.072067+0000", 2024-09-16T03:39:36.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243856 2024-09-16T03:39:36.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.171991+0000", 2024-09-16T03:39:36.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099924052999999999 2024-09-16T03:39:36.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272176+0000", 2024-09-16T03:39:36.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100184963 2024-09-16T03:39:36.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272251+0000", 2024-09-16T03:39:36.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5902000000000006e-05 2024-09-16T03:39:36.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272421+0000", 2024-09-16T03:39:36.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016962800000000001 2024-09-16T03:39:36.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.613 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.613 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1657 4.12 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:36.613 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:39:36.613 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 106.009169586, 2024-09-16T03:39:36.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72934181899999995, 2024-09-16T03:39:36.613 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.613 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.613 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:36.613 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.614 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1657 2024-09-16T03:39:36.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.614 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:39:36.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:39:36.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543408+0000", 2024-09-16T03:39:36.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.13e-07 2024-09-16T03:39:36.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543410+0000", 2024-09-16T03:39:36.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7030000000000002e-06 2024-09-16T03:39:36.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543417+0000", 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.567e-06 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543423+0000", 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0619999999999997e-06 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543489+0000", 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6315999999999993e-05 2024-09-16T03:39:36.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:36.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543498+0000", 2024-09-16T03:39:36.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7239999999999998e-06 2024-09-16T03:39:36.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970952+0000", 2024-09-16T03:39:36.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42745424399999998 2024-09-16T03:39:36.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:36.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971012+0000", 2024-09-16T03:39:36.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9771000000000001e-05 2024-09-16T03:39:36.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272463+0000", 2024-09-16T03:39:36.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30145090800000002 2024-09-16T03:39:36.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272558+0000", 2024-09-16T03:39:36.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4863000000000003e-05 2024-09-16T03:39:36.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272749+0000", 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191048 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2106 4.a 4:51338165:::smithi15736696-4:head [sparse-read 0~887916] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 91.503644632999993, 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40254453200000001, 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2106 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:39:36.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048933+0000", 2024-09-16T03:39:36.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:39:36.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048936+0000", 2024-09-16T03:39:36.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3890000000000001e-06 2024-09-16T03:39:36.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048943+0000", 2024-09-16T03:39:36.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1250000000000004e-06 2024-09-16T03:39:36.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048953+0000", 2024-09-16T03:39:36.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8190000000000003e-06 2024-09-16T03:39:36.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149113+0000", 2024-09-16T03:39:36.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100160007 2024-09-16T03:39:36.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:36.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149118+0000", 2024-09-16T03:39:36.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7650000000000001e-06 2024-09-16T03:39:36.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451341+0000", 2024-09-16T03:39:36.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30222344699999998 2024-09-16T03:39:36.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451381+0000", 2024-09-16T03:39:36.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9517000000000001e-05 2024-09-16T03:39:36.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451477+0000", 2024-09-16T03:39:36.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5656999999999996e-05 2024-09-16T03:39:36.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.624 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.625 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2107 4.a 4:51338165:::smithi15736696-4:head [read 1698310~486600,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:39:36.625 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:39:36.625 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 91.503614569999996, 2024-09-16T03:39:36.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40288642400000002, 2024-09-16T03:39:36.625 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.625 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.625 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:36.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.625 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2107 2024-09-16T03:39:36.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.625 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:39:36.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:39:36.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048963+0000", 2024-09-16T03:39:36.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:39:36.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048967+0000", 2024-09-16T03:39:36.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9060000000000004e-06 2024-09-16T03:39:36.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048972+0000", 2024-09-16T03:39:36.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5910000000000003e-06 2024-09-16T03:39:36.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048977+0000", 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9629999999999997e-06 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149135+0000", 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015828 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149138+0000", 2024-09-16T03:39:36.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.779e-06 2024-09-16T03:39:36.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451506+0000", 2024-09-16T03:39:36.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302367891 2024-09-16T03:39:36.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451558+0000", 2024-09-16T03:39:36.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2374999999999999e-05 2024-09-16T03:39:36.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451849+0000", 2024-09-16T03:39:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029009899999999999 2024-09-16T03:39:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2987 4.a 4:5a5e9bc2:::smithi15736696-36:head [write 1607751~551752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:39:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:39:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.633629080999995, 2024-09-16T03:39:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49279943700000001, 2024-09-16T03:39:36.630 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.631 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.631 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.631 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.631 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.631 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.631 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2987 2024-09-16T03:39:36.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.631 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:39:36.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:39:36.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918949+0000", 2024-09-16T03:39:36.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.86e-06 2024-09-16T03:39:36.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919579+0000", 2024-09-16T03:39:36.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062909000000000001 2024-09-16T03:39:36.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919591+0000", 2024-09-16T03:39:36.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2078e-05 2024-09-16T03:39:36.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919596+0000", 2024-09-16T03:39:36.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:39:36.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019705+0000", 2024-09-16T03:39:36.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100108405 2024-09-16T03:39:36.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019754+0000", 2024-09-16T03:39:36.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9258999999999997e-05 2024-09-16T03:39:36.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:36.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019847+0000", 2024-09-16T03:39:36.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2954999999999993e-05 2024-09-16T03:39:36.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.023003+0000", 2024-09-16T03:39:36.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031561110000000001 2024-09-16T03:39:36.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.122544+0000", 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099540553000000004 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411658+0000", 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.28911426699999998 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411696+0000", 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8430000000000003e-05 2024-09-16T03:39:36.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411747+0000", 2024-09-16T03:39:36.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0745999999999998e-05 2024-09-16T03:39:36.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.637 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.637 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2988 4.a 4:5a5e9bc2:::smithi15736696-36:head [setxattr _header (56) in=7b,truncate 2487469] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:39:36.637 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:39:36.637 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.632967342000001, 2024-09-16T03:39:36.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59236646500000001, 2024-09-16T03:39:36.638 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.638 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.638 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.638 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2988 2024-09-16T03:39:36.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.638 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:39:36.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919610+0000", 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.048e-06 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919615+0000", 2024-09-16T03:39:36.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8239999999999999e-06 2024-09-16T03:39:36.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919621+0000", 2024-09-16T03:39:36.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4380000000000003e-06 2024-09-16T03:39:36.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919626+0000", 2024-09-16T03:39:36.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2000000000000002e-06 2024-09-16T03:39:36.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.020922+0000", 2024-09-16T03:39:36.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101295782 2024-09-16T03:39:36.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.020995+0000", 2024-09-16T03:39:36.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3256999999999993e-05 2024-09-16T03:39:36.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:36.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.021088+0000", 2024-09-16T03:39:36.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3419999999999997e-05 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.023007+0000", 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019186380000000001 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.222530+0000", 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19952276899999999 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511887+0000", 2024-09-16T03:39:36.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.289357378 2024-09-16T03:39:36.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511927+0000", 2024-09-16T03:39:36.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0357e-05 2024-09-16T03:39:36.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511976+0000", 2024-09-16T03:39:36.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8353999999999998e-05 2024-09-16T03:39:36.643 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.643 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3258 4.1a 4:5e3c0760:::smithi15736696-24:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:39:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:39:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.122516466999997, 2024-09-16T03:39:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3302292470000001, 2024-09-16T03:39:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.644 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.645 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3258 2024-09-16T03:39:36.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.645 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:39:36.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:39:36.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430061+0000", 2024-09-16T03:39:36.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.27e-07 2024-09-16T03:39:36.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430062+0000", 2024-09-16T03:39:36.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6780000000000001e-06 2024-09-16T03:39:36.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430066+0000", 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5269999999999999e-06 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430069+0000", 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430119+0000", 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0212e-05 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:39:36.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430122+0000", 2024-09-16T03:39:36.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7319999999999998e-06 2024-09-16T03:39:36.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.631493+0000", 2024-09-16T03:39:36.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20137142499999999 2024-09-16T03:39:36.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.631579+0000", 2024-09-16T03:39:36.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5418000000000001e-05 2024-09-16T03:39:36.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:41.760289+0000", 2024-09-16T03:39:36.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1287106680000001 2024-09-16T03:39:36.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.649 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.649 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3701 4.a 4:51338165:::smithi15736696-4:head [assert-version v115,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e121)", 2024-09-16T03:39:36.649 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:39:36.649 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.181669927999998, 2024-09-16T03:39:36.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50414302200000005, 2024-09-16T03:39:36.650 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.650 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.650 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.650 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.650 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.650 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.650 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3701 2024-09-16T03:39:36.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.650 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:39:36.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:39:36.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370910+0000", 2024-09-16T03:39:36.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1200000000000002e-06 2024-09-16T03:39:36.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370923+0000", 2024-09-16T03:39:36.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3078000000000001e-05 2024-09-16T03:39:36.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370940+0000", 2024-09-16T03:39:36.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7461000000000001e-05 2024-09-16T03:39:36.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370950+0000", 2024-09-16T03:39:36.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0091999999999999e-05 2024-09-16T03:39:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.371029+0000", 2024-09-16T03:39:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8879000000000002e-05 2024-09-16T03:39:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.371136+0000", 2024-09-16T03:39:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106983 2024-09-16T03:39:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.573316+0000", 2024-09-16T03:39:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20217934500000001 2024-09-16T03:39:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.573914+0000", 2024-09-16T03:39:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059857900000000002 2024-09-16T03:39:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.763374+0000", 2024-09-16T03:39:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18945932200000001 2024-09-16T03:39:36.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.874936+0000", 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111562256 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.875002+0000", 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6050000000000006e-05 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.875050+0000", 2024-09-16T03:39:36.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7856999999999999e-05 2024-09-16T03:39:36.656 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.656 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.656 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.656 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4258 4.a 4:50107449:::smithi15736696-27:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-16T03:39:36.656 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:16.528667+0000", 2024-09-16T03:39:36.656 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.023909716999999, 2024-09-16T03:39:36.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.47672299499999998, 2024-09-16T03:39:36.656 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.656 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.656 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:36.656 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.657 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.657 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.657 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4258 2024-09-16T03:39:36.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.657 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528667+0000", 2024-09-16T03:39:36.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528667+0000", 2024-09-16T03:39:36.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528669+0000", 2024-09-16T03:39:36.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.883e-06 2024-09-16T03:39:36.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528674+0000", 2024-09-16T03:39:36.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2599999999999996e-06 2024-09-16T03:39:36.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528687+0000", 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3220999999999999e-05 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528695+0000", 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528754+0000", 2024-09-16T03:39:36.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9555000000000002e-05 2024-09-16T03:39:36.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528864+0000", 2024-09-16T03:39:36.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010949 2024-09-16T03:39:36.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:17.005390+0000", 2024-09-16T03:39:36.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.47652628499999999 2024-09-16T03:39:36.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.661 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4368 4.a 4:51338165:::smithi15736696-4:head [write 773221~492056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:36.661 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:20.368068+0000", 2024-09-16T03:39:36.661 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.184509045999999, 2024-09-16T03:39:36.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41474344400000002, 2024-09-16T03:39:36.661 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.661 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.661 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.661 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4368 2024-09-16T03:39:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.368068+0000", 2024-09-16T03:39:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.368068+0000", 2024-09-16T03:39:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.368070+0000", 2024-09-16T03:39:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3829999999999999e-06 2024-09-16T03:39:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.368716+0000", 2024-09-16T03:39:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064629800000000001 2024-09-16T03:39:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.368734+0000", 2024-09-16T03:39:36.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7453999999999999e-05 2024-09-16T03:39:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.368742+0000", 2024-09-16T03:39:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8159999999999994e-06 2024-09-16T03:39:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374166+0000", 2024-09-16T03:39:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054248109999999999 2024-09-16T03:39:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:39:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374171+0000", 2024-09-16T03:39:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9060000000000001e-06 2024-09-16T03:39:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.678206+0000", 2024-09-16T03:39:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30403448500000002 2024-09-16T03:39:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.678318+0000", 2024-09-16T03:39:36.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112181 2024-09-16T03:39:36.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:36.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.678441+0000", 2024-09-16T03:39:36.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122683 2024-09-16T03:39:36.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782676+0000", 2024-09-16T03:39:36.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104235049 2024-09-16T03:39:36.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782690+0000", 2024-09-16T03:39:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4796e-05 2024-09-16T03:39:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782734+0000", 2024-09-16T03:39:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3662999999999999e-05 2024-09-16T03:39:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782776+0000", 2024-09-16T03:39:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1495999999999997e-05 2024-09-16T03:39:36.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782811+0000", 2024-09-16T03:39:36.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5423000000000003e-05 2024-09-16T03:39:36.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.668 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.668 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4369 4.a 4:51338165:::smithi15736696-4:head [write 1816008~503126] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:36.668 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:20.373551+0000", 2024-09-16T03:39:36.668 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.179025952, 2024-09-16T03:39:36.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.75753581699999994, 2024-09-16T03:39:36.669 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.669 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.669 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.669 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.669 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.669 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.669 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4369 2024-09-16T03:39:36.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.669 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.373551+0000", 2024-09-16T03:39:36.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.373551+0000", 2024-09-16T03:39:36.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.373552+0000", 2024-09-16T03:39:36.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2550000000000001e-06 2024-09-16T03:39:36.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374063+0000", 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00051145499999999996 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374077+0000", 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3535e-05 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374087+0000", 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0086e-05 2024-09-16T03:39:36.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374194+0000", 2024-09-16T03:39:36.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010661 2024-09-16T03:39:36.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:36.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374196+0000", 2024-09-16T03:39:36.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7980000000000001e-06 2024-09-16T03:39:36.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.780445+0000", 2024-09-16T03:39:36.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40624829800000001 2024-09-16T03:39:36.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.780585+0000", 2024-09-16T03:39:36.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014048700000000001 2024-09-16T03:39:36.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:36.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.780748+0000", 2024-09-16T03:39:36.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000162798 2024-09-16T03:39:36.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782933+0000", 2024-09-16T03:39:36.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002184994 2024-09-16T03:39:36.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.030573+0000", 2024-09-16T03:39:36.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.247640049 2024-09-16T03:39:36.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.130869+0000", 2024-09-16T03:39:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100296195 2024-09-16T03:39:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131031+0000", 2024-09-16T03:39:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016220899999999999 2024-09-16T03:39:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131087+0000", 2024-09-16T03:39:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5047999999999999e-05 2024-09-16T03:39:36.675 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.676 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.676 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.676 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4370 4.a 4:51338165:::smithi15736696-4:head [setxattr _header (55) in=7b,truncate 2773221] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:36.676 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:20.374103+0000", 2024-09-16T03:39:36.676 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.178473433000001, 2024-09-16T03:39:36.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.757112648, 2024-09-16T03:39:36.676 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.676 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.676 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:36.676 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4370 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374103+0000", 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374103+0000", 2024-09-16T03:39:36.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374104+0000", 2024-09-16T03:39:36.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.192e-06 2024-09-16T03:39:36.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374109+0000", 2024-09-16T03:39:36.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7280000000000003e-06 2024-09-16T03:39:36.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374117+0000", 2024-09-16T03:39:36.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2039999999999997e-06 2024-09-16T03:39:36.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374124+0000", 2024-09-16T03:39:36.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3380000000000001e-06 2024-09-16T03:39:36.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374212+0000", 2024-09-16T03:39:36.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8663000000000001e-05 2024-09-16T03:39:36.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:36.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374215+0000", 2024-09-16T03:39:36.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3929999999999998e-06 2024-09-16T03:39:36.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782160+0000", 2024-09-16T03:39:36.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.407945112 2024-09-16T03:39:36.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782254+0000", 2024-09-16T03:39:36.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4152000000000003e-05 2024-09-16T03:39:36.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:36.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782407+0000", 2024-09-16T03:39:36.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015309500000000001 2024-09-16T03:39:36.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:36.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.785130+0000", 2024-09-16T03:39:36.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027231320000000001 2024-09-16T03:39:36.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.130717+0000", 2024-09-16T03:39:36.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34558722600000003 2024-09-16T03:39:36.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:36.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131123+0000", 2024-09-16T03:39:36.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040580199999999997 2024-09-16T03:39:36.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:36.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131164+0000", 2024-09-16T03:39:36.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0487e-05 2024-09-16T03:39:36.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131216+0000", 2024-09-16T03:39:36.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2123999999999998e-05 2024-09-16T03:39:36.683 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.683 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.683 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.684 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4371 4.a 4:51338165:::smithi15736696-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:36.684 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:20.374154+0000", 2024-09-16T03:39:36.684 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.178422533999999, 2024-09-16T03:39:36.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.75721231200000005, 2024-09-16T03:39:36.684 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:36.684 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:36.684 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:36.684 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:36.684 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:36.684 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:36.684 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4371 2024-09-16T03:39:36.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.684 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:36.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:36.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374154+0000", 2024-09-16T03:39:36.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:36.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374154+0000", 2024-09-16T03:39:36.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:36.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:36.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374155+0000", 2024-09-16T03:39:36.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4099999999999997e-07 2024-09-16T03:39:36.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:36.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374159+0000", 2024-09-16T03:39:36.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.506e-06 2024-09-16T03:39:36.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:36.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374167+0000", 2024-09-16T03:39:36.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0870000000000007e-06 2024-09-16T03:39:36.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:36.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374173+0000", 2024-09-16T03:39:36.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.652e-06 2024-09-16T03:39:36.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.468957+0000", 2024-09-16T03:39:36.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094784050999999994 2024-09-16T03:39:36.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:36.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.468967+0000", 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6169999999999996e-06 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782856+0000", 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.313889311 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782918+0000", 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2055000000000001e-05 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:36.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131245+0000", 2024-09-16T03:39:36.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34832644699999998 2024-09-16T03:39:36.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:36.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131301+0000", 2024-09-16T03:39:36.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5721999999999999e-05 2024-09-16T03:39:36.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:36.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:36.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:36.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131366+0000", 2024-09-16T03:39:36.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5922999999999999e-05 2024-09-16T03:39:36.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.690 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.690 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:36.690 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:36.690 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:36.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:39:36.692 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:36.666+0000 7f652ae6c640 -1 osd.1 156 handle_osd_map injecting map crc failure 2024-09-16T03:39:36.692 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:36.692 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:36.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:39:36.693 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:36.694 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:36.694 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:36.694 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:36.694 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:36.695 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:36.695 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:36.695 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:36.696 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:36.696 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:36.696 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:36.696 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:36.697 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:36.697 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:36.697 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:36.689+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:36.699 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:36.700 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:36.716 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:36.717 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:36.733 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:36.733 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:36.750 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:36.751 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:36.790 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:36.791 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:36.792 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:36.790+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:36.807 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:36.808 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:36.814 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:36.814 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:36.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:39:36.893 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:36.891+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:36.972 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:36.972 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:36.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:39:37.024 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:37.023+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:37.088 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:37.089 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:37.094 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:37.093+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:37.097 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:39:37.195 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:37.194+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:37.213 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:37.213 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:37.222 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:39:37.296 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:37.295+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:37.325 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:37.325 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:37.334 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:39:37.397 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:37.395+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:37.436 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:37.436 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:37.446 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:39:37.497 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:37.496+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:37.549 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:37.549 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:37.557 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:39:37.598 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:37.596+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:37.675 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:37.673+0000 7fedb773d640 -1 osd.9 157 handle_osd_map injecting map crc failure 2024-09-16T03:39:37.676 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:39:37.704 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:37.704 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:37.713 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:39:37.735 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:37.733+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:37.800 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:37.798+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:37.815 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:37.816 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:37.824 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:39:37.901 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:37.900+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:37.927 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:37.927 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:37.935 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:39:38.002 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:38.001+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:38.079 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:38.079 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:38.087 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:39:38.103 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:38.102+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:38.190 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:38.190 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:38.199 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:39:38.203 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:38.202+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:38.302 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:38.302 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:38.305 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:38.303+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:38.311 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:39:38.405 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:38.404+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:38.413 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:38.413 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:38.422 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:39:38.506 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:38.505+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:38.524 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:38.525 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:38.533 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:39:38.608 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:38.606+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:38.650 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:38.650 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:38.659 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:39:38.708 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:38.706+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:38.768 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:38.768 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:38.777 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:39:38.807 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:38.805+0000 7f12e2a1a640 -1 osd.5 158 handle_osd_map injecting map crc failure 2024-09-16T03:39:38.809 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:38.808+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:38.822 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:39:38.910 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:38.908+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:38.924 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:38.924 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:38.933 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:39:38.960 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:38.960 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:38.978 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:38.978 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:38.996 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:38.996 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:39.011 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:39.009+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:39.013 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:39.013 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:39.029 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:39.029 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:39.046 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:39.046 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:39.063 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:39.063 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:39.078 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:39.078 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:39.078 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:39:39.078 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:39.080 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:39.081 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:39.087 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:39:39.097 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:39.097 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:39.112 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:39.110+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:39.114 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:39.114 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:39.132 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:39.132 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:39.148 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:39.148 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:39.164 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:39.165 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:39.181 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:39.181 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:39.192 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:39.192 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:39.192 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:39.192 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:39.192 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:39.201 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:39.201 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:39.202 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:39:39.213 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:39.212+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:39.218 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:39.218 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:39.308 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:39.308 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:39.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:39.308 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:39.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.308 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.15.23:39 3.1 3:86477bf9:::smithi15736696-46 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:39.308 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:39:39.308 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 132.46549012200001, 2024-09-16T03:39:39.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40113680400000001, 2024-09-16T03:39:39.309 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.309 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.309 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.309 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:39:39.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839697+0000", 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839700+0000", 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.571e-06 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839704+0000", 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8399999999999997e-06 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.311 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839712+0000", 2024-09-16T03:39:39.311 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9990000000000008e-06 2024-09-16T03:39:39.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.311 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.311 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839720+0000", 2024-09-16T03:39:39.311 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0099999999999995e-06 2024-09-16T03:39:39.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.311 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.311 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839783+0000", 2024-09-16T03:39:39.311 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3014000000000003e-05 2024-09-16T03:39:39.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.839870+0000", 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7392999999999995e-05 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840049+0000", 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017825800000000001 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.840874+0000", 2024-09-16T03:39:39.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082532099999999995 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.040285+0000", 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19941088400000001 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240748+0000", 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20046322699999999 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240786+0000", 2024-09-16T03:39:39.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8158000000000003e-05 2024-09-16T03:39:39.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:27.240834+0000", 2024-09-16T03:39:39.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8128999999999999e-05 2024-09-16T03:39:39.314 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.314 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.314 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.314 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:983 3.1 3:902be6e8:::d2002292ea0d491a73c3f0803ee54a42e170a9ae5aae2b49191b9c752a601fa2:head [write 0~2138112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-16T03:39:39.314 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 129.24158718199999, 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30819799799999997, 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 983 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:39:39.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063600+0000", 2024-09-16T03:39:39.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.063610+0000", 2024-09-16T03:39:39.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2990000000000002e-06 2024-09-16T03:39:39.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065735+0000", 2024-09-16T03:39:39.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0021257799999999999 2024-09-16T03:39:39.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065752+0000", 2024-09-16T03:39:39.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6895999999999999e-05 2024-09-16T03:39:39.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065762+0000", 2024-09-16T03:39:39.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6830000000000003e-06 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.065871+0000", 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010904900000000001 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066005+0000", 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013369799999999999 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.066135+0000", 2024-09-16T03:39:39.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000130481 2024-09-16T03:39:39.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.069822+0000", 2024-09-16T03:39:39.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036863759999999999 2024-09-16T03:39:39.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272370+0000", 2024-09-16T03:39:39.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20254802599999999 2024-09-16T03:39:39.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.320 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.320 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371671+0000", 2024-09-16T03:39:39.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099300893000000001 2024-09-16T03:39:39.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.320 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.320 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371754+0000", 2024-09-16T03:39:39.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3075e-05 2024-09-16T03:39:39.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.320 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.320 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.371798+0000", 2024-09-16T03:39:39.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4742e-05 2024-09-16T03:39:39.321 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.321 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.321 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.321 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.321 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1704 3.1 3:b6a94245:::f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99:head [write 0~453120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:39.321 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:39:39.321 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 108.51235009, 2024-09-16T03:39:39.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40403313699999999, 2024-09-16T03:39:39.321 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.321 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.321 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.321 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.322 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.322 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.322 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1704 2024-09-16T03:39:39.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.322 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:39:39.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792841+0000", 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.433e-06 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793329+0000", 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048839200000000004 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793345+0000", 2024-09-16T03:39:39.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6079e-05 2024-09-16T03:39:39.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.324 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793354+0000", 2024-09-16T03:39:39.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.32e-06 2024-09-16T03:39:39.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.324 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793418+0000", 2024-09-16T03:39:39.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3944999999999998e-05 2024-09-16T03:39:39.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793549+0000", 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013159900000000001 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793666+0000", 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116532 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.797707+0000", 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040409629999999998 2024-09-16T03:39:39.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897288+0000", 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099581449000000002 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196775+0000", 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948696800000002 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196814+0000", 2024-09-16T03:39:39.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8336e-05 2024-09-16T03:39:39.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196871+0000", 2024-09-16T03:39:39.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7120999999999997e-05 2024-09-16T03:39:39.327 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.327 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.327 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.327 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:84 3.1 3:b24f289d:::e84532c28f467d2f2a6e2cf44904fa0d4a555e1304bbe1b25db645af42152bd7:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:39.327 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:39:39.327 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 108.503578151, 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.395455581, 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801609+0000", 2024-09-16T03:39:39.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.329 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.801613+0000", 2024-09-16T03:39:39.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9559999999999999e-06 2024-09-16T03:39:39.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.329 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802117+0000", 2024-09-16T03:39:39.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050401600000000003 2024-09-16T03:39:39.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802141+0000", 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3618e-05 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802155+0000", 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4358e-05 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802229+0000", 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3930999999999994e-05 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802355+0000", 2024-09-16T03:39:39.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012527299999999999 2024-09-16T03:39:39.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.331 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.331 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802567+0000", 2024-09-16T03:39:39.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021250199999999999 2024-09-16T03:39:39.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.331 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.331 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806218+0000", 2024-09-16T03:39:39.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0036512189999999998 2024-09-16T03:39:39.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.332 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.332 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897323+0000", 2024-09-16T03:39:39.332 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091104892000000007 2024-09-16T03:39:39.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.332 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.332 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196959+0000", 2024-09-16T03:39:39.332 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299635971 2024-09-16T03:39:39.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.332 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.332 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197021+0000", 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1938000000000002e-05 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197065+0000", 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3906999999999998e-05 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:85 3.1 3:b380f269:::e65351e8a65a0b1a517e3b1a85e04b84c0ab7cfa6655f11fb0da31f56b4dd9af:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:39:39.333 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 108.502999254, 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39502727300000001, 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802188+0000", 2024-09-16T03:39:39.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802190+0000", 2024-09-16T03:39:39.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2419999999999999e-06 2024-09-16T03:39:39.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802464+0000", 2024-09-16T03:39:39.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00027463699999999999 2024-09-16T03:39:39.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802478+0000", 2024-09-16T03:39:39.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3375e-05 2024-09-16T03:39:39.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802484+0000", 2024-09-16T03:39:39.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0990000000000004e-06 2024-09-16T03:39:39.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803475+0000", 2024-09-16T03:39:39.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00099113099999999991 2024-09-16T03:39:39.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803651+0000", 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017654300000000001 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.803923+0000", 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000271842 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806228+0000", 2024-09-16T03:39:39.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0023049590000000001 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897563+0000", 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091334724000000006 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197123+0000", 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299559992 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197177+0000", 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4447999999999998e-05 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197216+0000", 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8281000000000002e-05 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:86 3.1 3:b3acb53e:::e951eb69ce5151a3f54107fa22c70f80729eff845da1613447a91536a24046e8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:39:39.339 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 108.50269215900001, 2024-09-16T03:39:39.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.394851071, 2024-09-16T03:39:39.340 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.340 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.340 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.340 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:39:39.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802495+0000", 2024-09-16T03:39:39.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.341 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.341 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802496+0000", 2024-09-16T03:39:39.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:39:39.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.341 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.341 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802782+0000", 2024-09-16T03:39:39.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00028523600000000002 2024-09-16T03:39:39.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802796+0000", 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4594e-05 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.802804+0000", 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3790000000000006e-06 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804787+0000", 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019825060000000002 2024-09-16T03:39:39.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.804895+0000", 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010813499999999999 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.805046+0000", 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000150972 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.806681+0000", 2024-09-16T03:39:39.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0016353839999999999 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.005712+0000", 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19903093199999999 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197278+0000", 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.191565295 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197315+0000", 2024-09-16T03:39:39.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6946000000000002e-05 2024-09-16T03:39:39.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.197347+0000", 2024-09-16T03:39:39.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1863000000000003e-05 2024-09-16T03:39:39.345 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.345 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.345 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.345 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:140 3.1 3:845e3b2c:::21d7b947791b34f7e3fcd39ad119c9f923ad2f3cc7a703501eac7a68189d5437:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:39.345 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:39:39.345 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 83.190512905000006, 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39914108999999998, 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114675+0000", 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114677+0000", 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1610000000000001e-06 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114684+0000", 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3150000000000003e-06 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114699+0000", 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4996e-05 2024-09-16T03:39:39.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114705+0000", 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0580000000000003e-06 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.214862+0000", 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015700199999999 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.214922+0000", 2024-09-16T03:39:39.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0053999999999998e-05 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215023+0000", 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100432 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.216100+0000", 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010776819999999999 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.499919+0000", 2024-09-16T03:39:39.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.28381845900000002 2024-09-16T03:39:39.350 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.350 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.350 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513755+0000", 2024-09-16T03:39:39.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.013836477 2024-09-16T03:39:39.350 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.350 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.350 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513790+0000", 2024-09-16T03:39:39.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4295999999999998e-05 2024-09-16T03:39:39.350 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.351 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.351 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513816+0000", 2024-09-16T03:39:39.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6157999999999999e-05 2024-09-16T03:39:39.351 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.351 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.351 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.351 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.351 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:141 3.1 3:9387c678:::99a87d3b555b50e905a1ff55d99eb4aa025bd0070f07a4910d03cf1af95bbc44:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:39.351 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 83.190462208, 2024-09-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50160030300000003, 2024-09-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114726+0000", 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4399999999999999e-07 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114730+0000", 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1479999999999999e-06 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114738+0000", 2024-09-16T03:39:39.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3200000000000002e-06 2024-09-16T03:39:39.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114743+0000", 2024-09-16T03:39:39.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.891e-06 2024-09-16T03:39:39.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215320+0000", 2024-09-16T03:39:39.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100576463 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215376+0000", 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6310999999999997e-05 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215473+0000", 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7103999999999997e-05 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.216108+0000", 2024-09-16T03:39:39.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00063422800000000003 2024-09-16T03:39:39.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513851+0000", 2024-09-16T03:39:39.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29774376400000002 2024-09-16T03:39:39.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616086+0000", 2024-09-16T03:39:39.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10223441 2024-09-16T03:39:39.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616240+0000", 2024-09-16T03:39:39.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015453799999999999 2024-09-16T03:39:39.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616326+0000", 2024-09-16T03:39:39.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5382000000000003e-05 2024-09-16T03:39:39.357 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.357 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.357 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:142 3.1 3:a93b7b74:::70cd8f47898d416018aded7de9c29e0fe002b2336cb6cfabbf3eb6238766c3b1:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 83.190432821000002, 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50166088799999997, 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:39:39.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:39:39.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3600000000000003e-07 2024-09-16T03:39:39.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114762+0000", 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6039999999999998e-06 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114769+0000", 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5939999999999999e-06 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114774+0000", 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2179999999999998e-06 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215687+0000", 2024-09-16T03:39:39.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100912974 2024-09-16T03:39:39.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215733+0000", 2024-09-16T03:39:39.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6419999999999999e-05 2024-09-16T03:39:39.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215836+0000", 2024-09-16T03:39:39.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102981 2024-09-16T03:39:39.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316766+0000", 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100929693 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513877+0000", 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19711067800000001 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616358+0000", 2024-09-16T03:39:39.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10248180699999999 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616391+0000", 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2551000000000001e-05 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616416+0000", 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4732e-05 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.364 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.364 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:143 3.1 3:ac848b49:::adc4f0c8a4d0bc7e52f9d5c3b4981447737a7dbe58b8313a369b04383e10893f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:39.364 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:39:39.364 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 83.190402814999999, 2024-09-16T03:39:39.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60333394699999998, 2024-09-16T03:39:39.364 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.364 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.364 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.364 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.364 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.364 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:39:39.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:39:39.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:39:39.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7899999999999999e-07 2024-09-16T03:39:39.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114788+0000", 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4569999999999999e-06 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114792+0000", 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5730000000000001e-06 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114796+0000", 2024-09-16T03:39:39.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5449999999999999e-06 2024-09-16T03:39:39.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316237+0000", 2024-09-16T03:39:39.367 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20144099300000001 2024-09-16T03:39:39.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316417+0000", 2024-09-16T03:39:39.367 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017981099999999999 2024-09-16T03:39:39.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316564+0000", 2024-09-16T03:39:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014693700000000001 2024-09-16T03:39:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316993+0000", 2024-09-16T03:39:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042985499999999998 2024-09-16T03:39:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.717965+0000", 2024-09-16T03:39:39.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40097123699999998 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718005+0000", 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0083000000000002e-05 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718064+0000", 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9020000000000001e-05 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718119+0000", 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4957000000000003e-05 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:149 3.1 3:8eb88b02:::fa73f0ab3a7395fcfc19219899ff8c0a4a3c96caf8327f9da9656e2450435b27:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 82.176494167000001, 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.348655887, 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:39:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128693+0000", 2024-09-16T03:39:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128698+0000", 2024-09-16T03:39:39.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.527e-06 2024-09-16T03:39:39.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.372 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128714+0000", 2024-09-16T03:39:39.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6019e-05 2024-09-16T03:39:39.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.372 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128757+0000", 2024-09-16T03:39:39.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3318000000000001e-05 2024-09-16T03:39:39.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128764+0000", 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.28e-06 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.228916+0000", 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10015252199999999 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.228976+0000", 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0390999999999998e-05 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.229077+0000", 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010053 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.274759+0000", 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.045682404000000003 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.330314+0000", 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.055554916000000003 2024-09-16T03:39:39.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477196+0000", 2024-09-16T03:39:39.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14688130099999999 2024-09-16T03:39:39.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477291+0000", 2024-09-16T03:39:39.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5144999999999998e-05 2024-09-16T03:39:39.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.477349+0000", 2024-09-16T03:39:39.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8533999999999999e-05 2024-09-16T03:39:39.376 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.376 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.376 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.376 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:139 3.1 3:bd600e93:::b81929c6ed6df2fe181809e10b331bcd3c2df1a1077e06024548babcdcd8f550:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:39:39.376 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:39:39.376 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.263614054000001, 2024-09-16T03:39:39.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.43180016300000001, 2024-09-16T03:39:39.376 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.376 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.376 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.377 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.377 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:39:39.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.377 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.377 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:39:39.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.377 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.377 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2300000000000001e-07 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041700+0000", 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000126367 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041707+0000", 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1519999999999997e-06 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041710+0000", 2024-09-16T03:39:39.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7869999999999999e-06 2024-09-16T03:39:39.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043365+0000", 2024-09-16T03:39:39.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001654914 2024-09-16T03:39:39.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043453+0000", 2024-09-16T03:39:39.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7954000000000005e-05 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043596+0000", 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014304500000000001 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.145131+0000", 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10153440599999999 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.344273+0000", 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19914232200000001 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473290+0000", 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12901701900000001 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473325+0000", 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4656000000000002e-05 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473374+0000", 2024-09-16T03:39:39.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9017999999999997e-05 2024-09-16T03:39:39.382 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.382 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.382 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.382 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:144 3.1 3:91c40fc0:::f2d8823355177e50718038dafd7a8313c1a25b0231461a8672e684a7d4e587b4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:39:39.382 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:39:39.382 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.263469796999999, 2024-09-16T03:39:39.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.431801096, 2024-09-16T03:39:39.383 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.383 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.383 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.383 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:39:39.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:39:39.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:39:39.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.75e-07 2024-09-16T03:39:39.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042077+0000", 2024-09-16T03:39:39.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035865099999999998 2024-09-16T03:39:39.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042084+0000", 2024-09-16T03:39:39.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2400000000000001e-06 2024-09-16T03:39:39.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042088+0000", 2024-09-16T03:39:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1570000000000002e-06 2024-09-16T03:39:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144179+0000", 2024-09-16T03:39:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10209033300000001 2024-09-16T03:39:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144427+0000", 2024-09-16T03:39:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000247879 2024-09-16T03:39:39.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144656+0000", 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022919800000000001 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.145866+0000", 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001210046 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.344293+0000", 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.198427353 2024-09-16T03:39:39.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473432+0000", 2024-09-16T03:39:39.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12913908800000001 2024-09-16T03:39:39.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473478+0000", 2024-09-16T03:39:39.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6165000000000003e-05 2024-09-16T03:39:39.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473519+0000", 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0510999999999999e-05 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:165 3.1 3:8166b007:::3555453b0ab6ec5b2f29a1e76cc26fdaf7f55d077a227627cdf2200eb9ca342d:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 57.477952995000003, 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40346431100000002, 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.389 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827235+0000", 2024-09-16T03:39:39.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5999999999999999e-07 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827239+0000", 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8920000000000002e-06 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827243+0000", 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2760000000000002e-06 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.391 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827246+0000", 2024-09-16T03:39:39.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1990000000000002e-06 2024-09-16T03:39:39.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.029284+0000", 2024-09-16T03:39:39.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20203769499999999 2024-09-16T03:39:39.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.029375+0000", 2024-09-16T03:39:39.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1419e-05 2024-09-16T03:39:39.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.393 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.393 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.029542+0000", 2024-09-16T03:39:39.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016613199999999999 2024-09-16T03:39:39.393 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.393 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.393 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.393 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.130201+0000", 2024-09-16T03:39:39.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10065989 2024-09-16T03:39:39.393 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.393 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.393 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.393 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230566+0000", 2024-09-16T03:39:39.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100364563 2024-09-16T03:39:39.394 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.394 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.394 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.394 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230623+0000", 2024-09-16T03:39:39.394 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6848000000000002e-05 2024-09-16T03:39:39.394 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.394 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.394 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.394 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230670+0000", 2024-09-16T03:39:39.394 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7117999999999998e-05 2024-09-16T03:39:39.396 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.396 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.396 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.396 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230699+0000", 2024-09-16T03:39:39.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8918999999999999e-05 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:166 3.1 3:91c40fc0:::f2d8823355177e50718038dafd7a8313c1a25b0231461a8672e684a7d4e587b4:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.827250+0000", 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 57.477937283999999, 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40362257099999999, 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827250+0000", 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827250+0000", 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827251+0000", 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827253+0000", 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.928e-06 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827257+0000", 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0249999999999996e-06 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.827260+0000", 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3359999999999999e-06 2024-09-16T03:39:39.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.090467+0000", 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.26320751100000001 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.090543+0000", 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5740000000000003e-05 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.090651+0000", 2024-09-16T03:39:39.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000107415 2024-09-16T03:39:39.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.401 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.130204+0000", 2024-09-16T03:39:39.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.039553455000000001 2024-09-16T03:39:39.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.401 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230753+0000", 2024-09-16T03:39:39.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100549096 2024-09-16T03:39:39.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230809+0000", 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6243999999999999e-05 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230848+0000", 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9045999999999999e-05 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:42.230873+0000", 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4433000000000001e-05 2024-09-16T03:39:39.402 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4365 3.1 3:8dceab4f:::b595f19538cd60ceade5ba11d0db1f07eff23820ff1cafbf439b875b0bc21add:head [write 0~243200] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:19.957638+0000", 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.347549755999999, 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40324459000000001, 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.403 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4365 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:19.957638+0000", 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:19.957638+0000", 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.405 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:19.957640+0000", 2024-09-16T03:39:39.405 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.967e-06 2024-09-16T03:39:39.405 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.405 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.405 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.405 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:19.957896+0000", 2024-09-16T03:39:39.405 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000256641 2024-09-16T03:39:39.405 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.405 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.406 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.406 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:19.957908+0000", 2024-09-16T03:39:39.406 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1402e-05 2024-09-16T03:39:39.406 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.406 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.406 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.406 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.058035+0000", 2024-09-16T03:39:39.406 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100127323 2024-09-16T03:39:39.406 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.406 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.406 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.406 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.158314+0000", 2024-09-16T03:39:39.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100278943 2024-09-16T03:39:39.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.407 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.158401+0000", 2024-09-16T03:39:39.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7225000000000006e-05 2024-09-16T03:39:39.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.407 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.158516+0000", 2024-09-16T03:39:39.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000115127 2024-09-16T03:39:39.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.159492+0000", 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00097529099999999998 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.159994+0000", 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050216400000000004 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.360804+0000", 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20081049200000001 2024-09-16T03:39:39.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.409 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.409 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.409 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.360836+0000", 2024-09-16T03:39:39.409 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1967000000000001e-05 2024-09-16T03:39:39.409 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.409 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.409 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.409 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.360882+0000", 2024-09-16T03:39:39.409 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6047999999999997e-05 2024-09-16T03:39:39.409 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.409 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.409 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.409 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4373 3.1 3:86e0c90a:::e78062fe11bded5b0dfbe3bf79dfdc9dee286cfe94341e7d51857de1af7a4ac6:head [write 0~1226752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:20.379803+0000", 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 18.925384881999999, 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.306323646, 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4373 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.410 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.379803+0000", 2024-09-16T03:39:39.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.379803+0000", 2024-09-16T03:39:39.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.379813+0000", 2024-09-16T03:39:39.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0329999999999999e-05 2024-09-16T03:39:39.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.381263+0000", 2024-09-16T03:39:39.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014503610000000001 2024-09-16T03:39:39.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.381275+0000", 2024-09-16T03:39:39.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1182999999999999e-05 2024-09-16T03:39:39.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.481404+0000", 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012908399999999 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.481488+0000", 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4145000000000002e-05 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.481582+0000", 2024-09-16T03:39:39.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3883999999999998e-05 2024-09-16T03:39:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.481673+0000", 2024-09-16T03:39:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1337999999999999e-05 2024-09-16T03:39:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.483542+0000", 2024-09-16T03:39:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001869227 2024-09-16T03:39:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.414 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.415 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.685985+0000", 2024-09-16T03:39:39.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202442873 2024-09-16T03:39:39.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.415 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.415 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.415 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.686056+0000", 2024-09-16T03:39:39.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1166000000000003e-05 2024-09-16T03:39:39.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.415 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.415 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.415 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.686096+0000", 2024-09-16T03:39:39.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9249999999999999e-05 2024-09-16T03:39:39.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.686126+0000", 2024-09-16T03:39:39.416 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0805000000000001e-05 2024-09-16T03:39:39.416 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.416 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.416 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.416 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.416 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.12.21:67 3.1 3:9eefaeba:::6a3dd5a76815e0ee5654a8dedb996b0b45131a021371296859100c694e8b8ffb:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:39.416 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:30.867424+0000", 2024-09-16T03:39:39.416 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 8.4377631330000007, 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30644011599999998, 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867424+0000", 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867424+0000", 2024-09-16T03:39:39.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867426+0000", 2024-09-16T03:39:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.079e-06 2024-09-16T03:39:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867522+0000", 2024-09-16T03:39:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6187999999999995e-05 2024-09-16T03:39:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867533+0000", 2024-09-16T03:39:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.112e-05 2024-09-16T03:39:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867538+0000", 2024-09-16T03:39:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3900000000000001e-06 2024-09-16T03:39:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.170652+0000", 2024-09-16T03:39:39.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.303113617 2024-09-16T03:39:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.170763+0000", 2024-09-16T03:39:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011136 2024-09-16T03:39:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.170951+0000", 2024-09-16T03:39:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000187307 2024-09-16T03:39:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.420 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.173710+0000", 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027594059999999998 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.173779+0000", 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8893000000000006e-05 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.173796+0000", 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6759e-05 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.173835+0000", 2024-09-16T03:39:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9249000000000003e-05 2024-09-16T03:39:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.173865+0000", 2024-09-16T03:39:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9748e-05 2024-09-16T03:39:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.422 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.12.21:68 3.1 3:85b1a0cb:::cbec311fd3dcf9951ac1aa2c731e21adc6221934f894d1c2b9cc0e954f5d34ba:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:30.867550+0000", 2024-09-16T03:39:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 8.4376379840000002, 2024-09-16T03:39:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50821847499999995, 2024-09-16T03:39:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867550+0000", 2024-09-16T03:39:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867550+0000", 2024-09-16T03:39:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867550+0000", 2024-09-16T03:39:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9899999999999999e-07 2024-09-16T03:39:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.424 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867846+0000", 2024-09-16T03:39:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029565 2024-09-16T03:39:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867858+0000", 2024-09-16T03:39:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1623e-05 2024-09-16T03:39:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867866+0000", 2024-09-16T03:39:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5900000000000008e-06 2024-09-16T03:39:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.171695+0000", 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30382852100000002 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.171800+0000", 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104832 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.172004+0000", 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020439000000000001 2024-09-16T03:39:39.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.173873+0000", 2024-09-16T03:39:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001868724 2024-09-16T03:39:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.173911+0000", 2024-09-16T03:39:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7997999999999998e-05 2024-09-16T03:39:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.375626+0000", 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.201715532 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.375727+0000", 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010069999999999999 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.375768+0000", 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1115999999999997e-05 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.428 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.12.21:71 3.1 3:b5d59aa7:::333d73059656e9a14b425bc3214236b65d87405f6459482d3b8927f96fbd6395:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:30.867879+0000", 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 8.4373082299999993, 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50807818800000004, 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867879+0000", 2024-09-16T03:39:39.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867879+0000", 2024-09-16T03:39:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867880+0000", 2024-09-16T03:39:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6799999999999999e-07 2024-09-16T03:39:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.868431+0000", 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00055086899999999995 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.868451+0000", 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9639e-05 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.868457+0000", 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0310000000000004e-06 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.172722+0000", 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30426540899999999 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.172882+0000", 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000160094 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.173126+0000", 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024391500000000001 2024-09-16T03:39:39.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.179336+0000", 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0062094810000000002 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.274806+0000", 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.095470151000000003 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.375843+0000", 2024-09-16T03:39:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101037217 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.375901+0000", 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7683000000000003e-05 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.375958+0000", 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6931000000000001e-05 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.434 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.435 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.435 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:39.436 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:39:39.437 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:39.434+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:39.537 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:39.536+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:39.582 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:39.582 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:39.582 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:39:39.582 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:39.590 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:39:39.637 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:39.636+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:39.687 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:39.685+0000 7f6528e68640 -1 osd.1 159 handle_osd_map injecting map crc failure 2024-09-16T03:39:39.688 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:39.685+0000 7eff7884b640 -1 osd.3 159 handle_osd_map injecting map crc failure 2024-09-16T03:39:39.693 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:39:39.697 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:39.697 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:39.697 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:39.697 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:39.697 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:39.707 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:39:39.750 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:39.748+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:39.816 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:39.816 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:39.816 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:39.816 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:39.816 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.816 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:416 4.0 4:023abada:::smithi15736696-17:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:39.816 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:39:39.817 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 139.50369394500001, 2024-09-16T03:39:39.817 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60401983699999995, 2024-09-16T03:39:39.817 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.817 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.817 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.817 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.817 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.817 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.817 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 416 2024-09-16T03:39:39.817 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.817 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.817 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.817 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309243+0000", 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.178e-06 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.818 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309248+0000", 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309257+0000", 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4720000000000001e-06 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309262+0000", 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0839999999999999e-06 2024-09-16T03:39:39.819 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309297+0000", 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4997999999999999e-05 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309367+0000", 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0124999999999997e-05 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812055+0000", 2024-09-16T03:39:39.820 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50268796900000001 2024-09-16T03:39:39.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.821 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.821 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812614+0000", 2024-09-16T03:39:39.821 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000558309 2024-09-16T03:39:39.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.821 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.821 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.813040+0000", 2024-09-16T03:39:39.821 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042652800000000002 2024-09-16T03:39:39.821 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.821 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.821 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.822 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913189+0000", 2024-09-16T03:39:39.822 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10014880399999999 2024-09-16T03:39:39.822 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.822 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.822 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.822 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913220+0000", 2024-09-16T03:39:39.822 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1353000000000003e-05 2024-09-16T03:39:39.822 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.822 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.822 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.822 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913262+0000", 2024-09-16T03:39:39.822 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1284e-05 2024-09-16T03:39:39.822 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:494 4.f 4:f817def9:::smithi15736696-45:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 138.09075421, 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60451760399999999, 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.823 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 494 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.824 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.825 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722183+0000", 2024-09-16T03:39:39.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:39:39.825 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.825 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.825 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722189+0000", 2024-09-16T03:39:39.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9429999999999997e-06 2024-09-16T03:39:39.825 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.825 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.825 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.825 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722203+0000", 2024-09-16T03:39:39.825 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4367e-05 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722210+0000", 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822367+0000", 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100156933 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.826 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822461+0000", 2024-09-16T03:39:39.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3216999999999999e-05 2024-09-16T03:39:39.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.827 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.827 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:39:39.827 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125593+0000", 2024-09-16T03:39:39.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30313253699999998 2024-09-16T03:39:39.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.827 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.827 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.827 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126307+0000", 2024-09-16T03:39:39.827 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071350000000000005 2024-09-16T03:39:39.827 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.827 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.828 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.828 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126825+0000", 2024-09-16T03:39:39.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051846100000000001 2024-09-16T03:39:39.828 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.828 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.828 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.828 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326612+0000", 2024-09-16T03:39:39.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19978721999999999 2024-09-16T03:39:39.828 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.828 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.828 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.828 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326663+0000", 2024-09-16T03:39:39.828 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0460000000000001e-05 2024-09-16T03:39:39.829 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.829 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.829 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.829 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326699+0000", 2024-09-16T03:39:39.829 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6340000000000001e-05 2024-09-16T03:39:39.829 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.829 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.829 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.829 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.829 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.829 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:784 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=63b,truncate 2840350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:39.829 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:39:39.829 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 134.006346334, 2024-09-16T03:39:39.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63358405200000001, 2024-09-16T03:39:39.830 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.830 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.830 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.830 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.830 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.830 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.830 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 784 2024-09-16T03:39:39.830 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.830 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.830 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.830 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.830 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:39:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:39:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806590+0000", 2024-09-16T03:39:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6000000000000003e-07 2024-09-16T03:39:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.832 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806593+0000", 2024-09-16T03:39:39.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6649999999999999e-06 2024-09-16T03:39:39.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.832 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806597+0000", 2024-09-16T03:39:39.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.245e-06 2024-09-16T03:39:39.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.832 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806600+0000", 2024-09-16T03:39:39.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.23e-06 2024-09-16T03:39:39.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099381+0000", 2024-09-16T03:39:39.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29278119400000002 2024-09-16T03:39:39.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099438+0000", 2024-09-16T03:39:39.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6198e-05 2024-09-16T03:39:39.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:39.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099527+0000", 2024-09-16T03:39:39.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9505000000000004e-05 2024-09-16T03:39:39.834 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.834 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200775+0000", 2024-09-16T03:39:39.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101247904 2024-09-16T03:39:39.834 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.834 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200889+0000", 2024-09-16T03:39:39.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114301 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440066+0000", 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.239176367 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440109+0000", 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2941000000000001e-05 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440173+0000", 2024-09-16T03:39:39.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4842e-05 2024-09-16T03:39:39.836 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.836 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.836 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.836 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:785 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:39.836 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:39:39.836 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 134.00633040899999, 2024-09-16T03:39:39.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63376373200000002, 2024-09-16T03:39:39.836 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.836 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 785 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.838 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:39:39.838 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.838 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.838 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.838 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.838 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806606+0000", 2024-09-16T03:39:39.838 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:39:39.838 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.838 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.838 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.838 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806607+0000", 2024-09-16T03:39:39.838 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.424e-06 2024-09-16T03:39:39.838 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806611+0000", 2024-09-16T03:39:39.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1080000000000003e-06 2024-09-16T03:39:39.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806614+0000", 2024-09-16T03:39:39.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.067e-06 2024-09-16T03:39:39.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099813+0000", 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29319853699999998 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099858+0000", 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5667999999999997e-05 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440236+0000", 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34037791499999998 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440297+0000", 2024-09-16T03:39:39.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0988e-05 2024-09-16T03:39:39.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440369+0000", 2024-09-16T03:39:39.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1682999999999998e-05 2024-09-16T03:39:39.841 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.841 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.841 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.841 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:801 4.0 4:07ad0560:::smithi15736696-25:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:39:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 133.80685251099999, 2024-09-16T03:39:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.534510445, 2024-09-16T03:39:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 801 2024-09-16T03:39:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006084+0000", 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.322e-06 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.843 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006088+0000", 2024-09-16T03:39:39.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000001e-06 2024-09-16T03:39:39.844 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006098+0000", 2024-09-16T03:39:39.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0186999999999999e-05 2024-09-16T03:39:39.844 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006104+0000", 2024-09-16T03:39:39.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.271e-06 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.199952+0000", 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.193847935 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200027+0000", 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4987000000000006e-05 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.540594+0000", 2024-09-16T03:39:39.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34056708299999999 2024-09-16T03:39:39.846 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.846 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.846 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.846 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.846 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1023 4.0 4:06b6d050:::smithi15736696-50:head [write 2246563~579356] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:39.846 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:39:39.846 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 128.50170344099999, 2024-09-16T03:39:39.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60727534699999997, 2024-09-16T03:39:39.846 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.846 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.846 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1023 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.847 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:39:39.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311233+0000", 2024-09-16T03:39:39.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1039999999999999e-06 2024-09-16T03:39:39.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311692+0000", 2024-09-16T03:39:39.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045908800000000002 2024-09-16T03:39:39.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.849 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311704+0000", 2024-09-16T03:39:39.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1844e-05 2024-09-16T03:39:39.849 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.849 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.849 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311713+0000", 2024-09-16T03:39:39.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2500000000000006e-06 2024-09-16T03:39:39.849 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.849 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.849 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.849 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359204+0000", 2024-09-16T03:39:39.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.047491646999999998 2024-09-16T03:39:39.850 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.850 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.850 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359258+0000", 2024-09-16T03:39:39.850 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4082000000000002e-05 2024-09-16T03:39:39.850 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.850 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:39.850 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359354+0000", 2024-09-16T03:39:39.850 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5686999999999994e-05 2024-09-16T03:39:39.850 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.850 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512245+0000", 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15289135700000001 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512956+0000", 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071110300000000004 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918361+0000", 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40540489400000002 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.851 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918455+0000", 2024-09-16T03:39:39.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3412999999999995e-05 2024-09-16T03:39:39.852 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918508+0000", 2024-09-16T03:39:39.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:39:39.852 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.852 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.852 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.852 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.853 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1024 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3176490] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:39.853 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:39:39.853 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 128.50121046000001, 2024-09-16T03:39:39.853 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60701227000000002, 2024-09-16T03:39:39.853 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.853 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.853 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.853 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.853 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.853 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.853 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1024 2024-09-16T03:39:39.853 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.853 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.854 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.854 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.854 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:39:39.854 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.854 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.854 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.854 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.854 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:39:39.854 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.854 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.854 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.854 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.854 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311726+0000", 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.02e-07 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311730+0000", 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1099999999999996e-06 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311735+0000", 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4680000000000003e-06 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.855 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311739+0000", 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9380000000000002e-06 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361468+0000", 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.049729450000000001 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361539+0000", 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0896999999999996e-05 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361661+0000", 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000122404 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512943+0000", 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.151281481 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512959+0000", 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:39:39.857 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.858 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.858 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.858 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918588+0000", 2024-09-16T03:39:39.858 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40562815499999999 2024-09-16T03:39:39.858 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.858 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.858 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.858 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918646+0000", 2024-09-16T03:39:39.858 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8566999999999997e-05 2024-09-16T03:39:39.858 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.858 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.858 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.858 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918737+0000", 2024-09-16T03:39:39.859 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1308000000000001e-05 2024-09-16T03:39:39.859 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.859 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.859 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.859 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.859 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.859 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1025 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:39.859 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:39.859 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 128.50119196200001, 2024-09-16T03:39:39.859 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60723649599999996, 2024-09-16T03:39:39.859 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.859 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.859 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1025 2024-09-16T03:39:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.860 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9400000000000001e-07 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311746+0000", 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2670000000000001e-06 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.861 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311752+0000", 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6389999999999997e-06 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311757+0000", 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3639999999999998e-06 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362018+0000", 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.050260969000000003 2024-09-16T03:39:39.862 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.863 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.863 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:39.863 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362089+0000", 2024-09-16T03:39:39.863 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0735000000000006e-05 2024-09-16T03:39:39.863 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.863 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.863 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.863 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918781+0000", 2024-09-16T03:39:39.863 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55669215400000005 2024-09-16T03:39:39.863 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.863 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.863 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.864 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918867+0000", 2024-09-16T03:39:39.864 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6019000000000004e-05 2024-09-16T03:39:39.864 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.864 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.864 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.864 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918980+0000", 2024-09-16T03:39:39.864 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011295500000000001 2024-09-16T03:39:39.864 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.864 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.864 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.864 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.864 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.864 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1055 4.10 4:0a55bbf2:::smithi15736696-18:head [setxattr _header (56) in=7b,truncate 3243718] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:39.865 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:39:39.865 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 128.299199254, 2024-09-16T03:39:39.865 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61098759599999997, 2024-09-16T03:39:39.865 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.865 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.865 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.865 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.865 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.865 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.865 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1055 2024-09-16T03:39:39.865 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.865 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.865 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:39:39.866 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.866 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.866 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:39:39.866 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.866 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.866 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513737+0000", 2024-09-16T03:39:39.866 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9399999999999993e-07 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513759+0000", 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1903999999999999e-05 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513773+0000", 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3244000000000001e-05 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513780+0000", 2024-09-16T03:39:39.867 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999999e-06 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714016+0000", 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20023635000000001 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714024+0000", 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8359999999999992e-06 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.868 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918736+0000", 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471236700000001 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918830+0000", 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3769000000000003e-05 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919003+0000", 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000173198 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.869 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921589+0000", 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002586419 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019655+0000", 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098065801999999994 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124538+0000", 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10488272899999999 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124635+0000", 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7125000000000003e-05 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124724+0000", 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8858999999999997e-05 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1056 4.10 4:0a55bbf2:::smithi15736696-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:39.871 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 128.299146683, 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61111632800000004, 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1056 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.872 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2499999999999995e-07 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513792+0000", 2024-09-16T03:39:39.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:39:39.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513797+0000", 2024-09-16T03:39:39.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4940000000000003e-06 2024-09-16T03:39:39.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513802+0000", 2024-09-16T03:39:39.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5390000000000001e-06 2024-09-16T03:39:39.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919221+0000", 2024-09-16T03:39:39.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40541913899999998 2024-09-16T03:39:39.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:39.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919268+0000", 2024-09-16T03:39:39.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7027000000000002e-05 2024-09-16T03:39:39.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124766+0000", 2024-09-16T03:39:39.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20549842099999999 2024-09-16T03:39:39.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124828+0000", 2024-09-16T03:39:39.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1803000000000004e-05 2024-09-16T03:39:39.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.876 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124905+0000", 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7163000000000006e-05 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2160 4.17 4:ed874e32:::smithi15736696-14:head [assert-version v36,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 92.982790273000006, 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55377054800000003, 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.877 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2160 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:39:39.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830149+0000", 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.743e-06 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830162+0000", 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3283e-05 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830191+0000", 2024-09-16T03:39:39.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8466000000000002e-05 2024-09-16T03:39:39.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830207+0000", 2024-09-16T03:39:39.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6453e-05 2024-09-16T03:39:39.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830362+0000", 2024-09-16T03:39:39.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015450699999999999 2024-09-16T03:39:39.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830608+0000", 2024-09-16T03:39:39.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024587399999999998 2024-09-16T03:39:39.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:39:39.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.233387+0000", 2024-09-16T03:39:39.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40277970099999999 2024-09-16T03:39:39.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234208+0000", 2024-09-16T03:39:39.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082053900000000001 2024-09-16T03:39:39.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234370+0000", 2024-09-16T03:39:39.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000161645 2024-09-16T03:39:39.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383821+0000", 2024-09-16T03:39:39.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14945145300000001 2024-09-16T03:39:39.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383867+0000", 2024-09-16T03:39:39.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5498999999999999e-05 2024-09-16T03:39:39.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383916+0000", 2024-09-16T03:39:39.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9385000000000002e-05 2024-09-16T03:39:39.883 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.883 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.883 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.884 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2532 4.0 4:06b6d050:::smithi15736696-50:head [write 2603977~725887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:39.884 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:39:39.884 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 83.218642884000005, 2024-09-16T03:39:39.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53439557900000001, 2024-09-16T03:39:39.884 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.884 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.884 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.884 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.884 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.884 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.884 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2532 2024-09-16T03:39:39.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.884 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:39:39.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594298+0000", 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6030000000000004e-06 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594876+0000", 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00057757800000000003 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594885+0000", 2024-09-16T03:39:39.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4150000000000005e-06 2024-09-16T03:39:39.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594892+0000", 2024-09-16T03:39:39.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7229999999999998e-06 2024-09-16T03:39:39.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628138+0000", 2024-09-16T03:39:39.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033246407999999998 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628252+0000", 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113755 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628344+0000", 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2028999999999996e-05 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730215+0000", 2024-09-16T03:39:39.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101870761 2024-09-16T03:39:39.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.732105+0000", 2024-09-16T03:39:39.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001889951 2024-09-16T03:39:39.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128411+0000", 2024-09-16T03:39:39.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.396305764 2024-09-16T03:39:39.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128611+0000", 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019990599999999999 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128688+0000", 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7686000000000001e-05 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2534 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 83.218021109000006, 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53406219200000005, 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2534 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8999999999999998e-07 2024-09-16T03:39:39.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594916+0000", 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.344e-06 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594919+0000", 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0580000000000002e-06 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594922+0000", 2024-09-16T03:39:39.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4389999999999999e-06 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729999+0000", 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.13507707399999999 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730046+0000", 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7382999999999997e-05 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128879+0000", 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39883298499999997 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128924+0000", 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128977+0000", 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2648999999999997e-05 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2621 4.f 4:fa163101:::smithi15736696-10:head [setxattr _header (56) in=7b,truncate 3815189] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 79.570244731000003, 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55430782499999998, 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2621 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242692+0000", 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7599999999999996e-07 2024-09-16T03:39:39.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242695+0000", 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.529e-06 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242699+0000", 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1010000000000002e-06 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242703+0000", 2024-09-16T03:39:39.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.518e-06 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.494935+0000", 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25223231200000001 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495011+0000", 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5692999999999995e-05 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495117+0000", 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000105944 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.596308+0000", 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101190825 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796852+0000", 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20054443199999999 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796893+0000", 2024-09-16T03:39:39.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1221999999999999e-05 2024-09-16T03:39:39.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.901 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796946+0000", 2024-09-16T03:39:39.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2435000000000002e-05 2024-09-16T03:39:39.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.901 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796999+0000", 2024-09-16T03:39:39.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3038000000000003e-05 2024-09-16T03:39:39.901 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.901 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.902 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.902 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2622 4.f 4:fa163101:::smithi15736696-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:39:39.902 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:39.902 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 79.570228876000002, 2024-09-16T03:39:39.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55448354399999999, 2024-09-16T03:39:39.902 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.902 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.902 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:39.902 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.902 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.902 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2622 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9999999999999999e-07 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242709+0000", 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.429e-06 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242712+0000", 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6090000000000001e-06 2024-09-16T03:39:39.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242716+0000", 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6480000000000001e-06 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495371+0000", 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25265485700000001 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495424+0000", 2024-09-16T03:39:39.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2927999999999999e-05 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797042+0000", 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30161821900000002 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797114+0000", 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2445000000000002e-05 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797190+0000", 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6108999999999999e-05 2024-09-16T03:39:39.906 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.907 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.907 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.907 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4669 4.10 4:0a55bbf2:::smithi15736696-18:head [setxattr _header (56) in=7b,truncate 3142975] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:39.907 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:30.855601+0000", 2024-09-16T03:39:39.907 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 8.9573347569999999, 2024-09-16T03:39:39.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.70839343799999999, 2024-09-16T03:39:39.907 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.907 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.907 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.907 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.907 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4669 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855601+0000", 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855601+0000", 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855602+0000", 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8400000000000003e-07 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855605+0000", 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5619999999999999e-06 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855609+0000", 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8909999999999997e-06 2024-09-16T03:39:39.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855613+0000", 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1980000000000003e-06 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.054797+0000", 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19918406499999999 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.054867+0000", 2024-09-16T03:39:39.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9103000000000006e-05 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.054969+0000", 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010247999999999999 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.155829+0000", 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10086023400000001 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.157532+0000", 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017031310000000001 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.563800+0000", 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40626808800000003 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.563926+0000", 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012529800000000001 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.563994+0000", 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8603999999999996e-05 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4670 4.10 4:0a55bbf2:::smithi15736696-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:30.855618+0000", 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 8.9573174120000001, 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.80871130099999999, 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:39.913 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4670 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855618+0000", 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855618+0000", 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855619+0000", 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.22e-07 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855620+0000", 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7099999999999999e-06 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855624+0000", 2024-09-16T03:39:39.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.287e-06 2024-09-16T03:39:39.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855627+0000", 2024-09-16T03:39:39.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.05e-06 2024-09-16T03:39:39.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.155366+0000", 2024-09-16T03:39:39.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299739861 2024-09-16T03:39:39.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:39.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.155514+0000", 2024-09-16T03:39:39.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014730099999999999 2024-09-16T03:39:39.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.664164+0000", 2024-09-16T03:39:39.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50864991100000001 2024-09-16T03:39:39.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.664247+0000", 2024-09-16T03:39:39.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3811000000000001e-05 2024-09-16T03:39:39.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.664330+0000", 2024-09-16T03:39:39.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2047999999999998e-05 2024-09-16T03:39:39.918 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.918 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.918 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.918 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4675 4.0 4:023abada:::smithi15736696-17:head [write 2012035~687143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:39.918 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:31.097063+0000", 2024-09-16T03:39:39.918 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 8.7158724020000005, 2024-09-16T03:39:39.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.56545287600000005, 2024-09-16T03:39:39.919 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.919 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.919 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.919 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.919 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.919 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.919 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4675 2024-09-16T03:39:39.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.919 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097063+0000", 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097063+0000", 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097065+0000", 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.407e-06 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097709+0000", 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00064482299999999999 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097721+0000", 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1764e-05 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097728+0000", 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4289999999999998e-06 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.159378+0000", 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.061650127999999998 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.159513+0000", 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013524699999999999 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.159613+0000", 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9952999999999997e-05 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.162018+0000", 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0024047539999999998 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.256225+0000", 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.094207338000000002 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.662322+0000", 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40609676 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.662479+0000", 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015684999999999999 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.662516+0000", 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7422999999999997e-05 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4676 4.0 4:023abada:::smithi15736696-17:head [setxattr _header (56) in=7b,truncate 3159782] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:31.097738+0000", 2024-09-16T03:39:39.924 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 8.7151973970000007, 2024-09-16T03:39:39.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.66492440100000005, 2024-09-16T03:39:39.925 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.925 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.925 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:39.925 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.925 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.925 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.925 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4676 2024-09-16T03:39:39.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.925 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097738+0000", 2024-09-16T03:39:39.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097738+0000", 2024-09-16T03:39:39.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097739+0000", 2024-09-16T03:39:39.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6199999999999997e-07 2024-09-16T03:39:39.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097742+0000", 2024-09-16T03:39:39.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.202e-06 2024-09-16T03:39:39.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097746+0000", 2024-09-16T03:39:39.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5420000000000001e-06 2024-09-16T03:39:39.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097749+0000", 2024-09-16T03:39:39.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2289999999999999e-06 2024-09-16T03:39:39.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.161472+0000", 2024-09-16T03:39:39.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.063722665999999997 2024-09-16T03:39:39.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.161547+0000", 2024-09-16T03:39:39.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5673999999999994e-05 2024-09-16T03:39:39.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.161650+0000", 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010220100000000001 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.263634+0000", 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101984436 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.263667+0000", 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2660000000000002e-05 2024-09-16T03:39:39.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.762551+0000", 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.49888468899999999 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.762595+0000", 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3498000000000003e-05 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.762663+0000", 2024-09-16T03:39:39.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7842000000000005e-05 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4677 4.0 4:023abada:::smithi15736696-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:31.097753+0000", 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 8.7151827819999994, 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.66511091099999997, 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:39.931 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4677 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097753+0000", 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097753+0000", 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097753+0000", 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7099999999999998e-07 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097755+0000", 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5349999999999999e-06 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097758+0000", 2024-09-16T03:39:39.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3749999999999999e-06 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097762+0000", 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5700000000000001e-06 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.256017+0000", 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15825552900000001 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:39.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.256149+0000", 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013168500000000001 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.762695+0000", 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50654610200000005 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.762756+0000", 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0946999999999998e-05 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:39.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:39.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.762864+0000", 2024-09-16T03:39:39.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000107897 2024-09-16T03:39:39.936 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.936 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.936 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.936 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:39.936 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:39.936 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:39.937 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:39.935+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:39.938 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:39:40.038 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:40.037+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:40.082 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:40.083 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:40.083 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:39:40.083 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:40.091 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:39:40.139 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:40.137+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:40.194 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:40.194 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:40.194 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:40.203 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:39:40.240 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:40.239+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:40.341 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:40.339+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:40.348 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:40.348 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:40.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:40.348 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:40.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.349 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:40.349 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:40.349 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 148.20211363799999, 2024-09-16T03:39:40.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:39:40.349 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.349 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.349 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.349 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.350 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.351 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:39:40.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:39:40.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.351 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.351 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.351 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:39:40.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:39:40.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.351 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.351 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.351 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:39:40.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:39:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:39:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:39:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:39:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:39:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:39:40.353 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:40.354 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 148.20176027799999, 2024-09-16T03:39:40.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:39:40.354 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.354 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.354 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.354 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:40.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:39:40.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:39:40.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:39:40.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:39:40.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:39:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:39:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:39:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:39:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:39:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:39:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 148.20063248, 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.358 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.359 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.359 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:40.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:40.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:39:40.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:39:40.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:39:40.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:39:40.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.362 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.363 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.363 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:39:40.363 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:40.363 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 148.19964503899999, 2024-09-16T03:39:40.363 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:39:40.363 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.363 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.363 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.363 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.364 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.364 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:40.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.364 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.364 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.364 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.364 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:40.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.364 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.364 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.364 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.364 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:39:40.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:39:40.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:39:40.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:39:40.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:39:40.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:39:40.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.367 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:39:40.368 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:40.368 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 148.19888829499999, 2024-09-16T03:39:40.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:39:40.368 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.368 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.368 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.368 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:40.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:39:40.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:39:40.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:989 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'31, mlcod=46'28)", 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 129.958036935, 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40332266900000002, 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.372 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387812+0000", 2024-09-16T03:39:40.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388467+0000", 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065539500000000002 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388481+0000", 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3905e-05 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488552+0000", 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10007120899999999 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688818+0000", 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20026627599999999 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688845+0000", 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791091+0000", 2024-09-16T03:39:40.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10224588699999999 2024-09-16T03:39:40.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791131+0000", 2024-09-16T03:39:40.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0293000000000002e-05 2024-09-16T03:39:40.376 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.376 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.376 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.377 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.377 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:990 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'32, mlcod=46'28)", 2024-09-16T03:39:40.377 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:40.377 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 129.85725326299999, 2024-09-16T03:39:40.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30256419200000001, 2024-09-16T03:39:40.377 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.377 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.377 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.377 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.377 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:40.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:40.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488594+0000", 2024-09-16T03:39:40.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:39:40.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488616+0000", 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2419e-05 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488627+0000", 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1251e-05 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588753+0000", 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012544399999999 2024-09-16T03:39:40.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690679+0000", 2024-09-16T03:39:40.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101926027 2024-09-16T03:39:40.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690695+0000", 2024-09-16T03:39:40.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6528000000000002e-05 2024-09-16T03:39:40.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791133+0000", 2024-09-16T03:39:40.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100438233 2024-09-16T03:39:40.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791157+0000", 2024-09-16T03:39:40.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-05 2024-09-16T03:39:40.381 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.381 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.381 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.381 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3081 4.1f e102/100 4:fa163101:::smithi15736696-10:head v 102'93, mlcod=102'91)", 2024-09-16T03:39:40.381 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:40.382 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 65.012130795000004, 2024-09-16T03:39:40.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20437049600000001, 2024-09-16T03:39:40.382 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.382 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.382 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.382 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:40.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:40.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333718+0000", 2024-09-16T03:39:40.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3720000000000001e-06 2024-09-16T03:39:40.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334338+0000", 2024-09-16T03:39:40.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061959099999999996 2024-09-16T03:39:40.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334355+0000", 2024-09-16T03:39:40.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.677e-05 2024-09-16T03:39:40.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334364+0000", 2024-09-16T03:39:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3780000000000007e-06 2024-09-16T03:39:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334424+0000", 2024-09-16T03:39:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0084000000000003e-05 2024-09-16T03:39:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334447+0000", 2024-09-16T03:39:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3113e-05 2024-09-16T03:39:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538030+0000", 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20358272399999999 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538085+0000", 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5464000000000003e-05 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.386 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4007 4.37 4:ed874e32:::smithi15736696-14:head [write 601783~551825,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:39:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.812646948999998, 2024-09-16T03:39:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29242759600000001, 2024-09-16T03:39:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4007 2024-09-16T03:39:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:39:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:39:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533200+0000", 2024-09-16T03:39:40.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3600000000000002e-07 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533841+0000", 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00064141100000000004 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533862+0000", 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0687e-05 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634001+0000", 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100139413 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721351+0000", 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.087349914000000001 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721357+0000", 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7100000000000004e-06 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721891+0000", 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053461299999999997 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.722024+0000", 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013274599999999999 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.722158+0000", 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013362399999999999 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.391 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:40.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725271+0000", 2024-09-16T03:39:40.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003113345 2024-09-16T03:39:40.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825258+0000", 2024-09-16T03:39:40.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099987260999999994 2024-09-16T03:39:40.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825453+0000", 2024-09-16T03:39:40.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019453999999999999 2024-09-16T03:39:40.393 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.393 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.393 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.393 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825591+0000", 2024-09-16T03:39:40.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013849700000000001 2024-09-16T03:39:40.393 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.393 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.393 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.393 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825626+0000", 2024-09-16T03:39:40.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4999000000000002e-05 2024-09-16T03:39:40.393 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.393 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.394 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.394 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.394 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.394 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4008 4.37 4:ed874e32:::smithi15736696-14:head [write 1776679~425104] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:40.394 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:39:40.394 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.711826825999999, 2024-09-16T03:39:40.394 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29371618199999999, 2024-09-16T03:39:40.394 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.394 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.394 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.394 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:40.394 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:40.394 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.395 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4008 2024-09-16T03:39:40.395 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.395 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.395 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.395 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.395 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:39:40.395 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.395 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.395 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.395 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.395 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:39:40.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.396 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.396 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.396 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.396 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634021+0000", 2024-09-16T03:39:40.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9989999999999998e-06 2024-09-16T03:39:40.396 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.396 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.396 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.396 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634187+0000", 2024-09-16T03:39:40.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016655199999999999 2024-09-16T03:39:40.396 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.396 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634197+0000", 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7520000000000008e-06 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634205+0000", 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8210000000000002e-06 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725407+0000", 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091202249999999999 2024-09-16T03:39:40.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725506+0000", 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8493999999999996e-05 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725648+0000", 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014187299999999999 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.727142+0000", 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001494478 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825761+0000", 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098619340999999999 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927671+0000", 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10190922700000001 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927705+0000", 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4523000000000001e-05 2024-09-16T03:39:40.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927735+0000", 2024-09-16T03:39:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9872000000000001e-05 2024-09-16T03:39:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4009 4.37 4:ed874e32:::smithi15736696-14:head [setxattr _header (56) in=7b,truncate 2201783] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:39:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.711629899000002, 2024-09-16T03:39:40.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29364004500000002, 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4009 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634217+0000", 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7199999999999998e-07 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634221+0000", 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9650000000000002e-06 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634226+0000", 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7830000000000004e-06 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634232+0000", 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2909999999999998e-06 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726750+0000", 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.09251827 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726830+0000", 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9808999999999994e-05 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726916+0000", 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6055000000000002e-05 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.727785+0000", 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086898599999999998 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.926315+0000", 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19852985000000001 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927777+0000", 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001462138 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927818+0000", 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1006e-05 2024-09-16T03:39:40.405 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927856+0000", 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8120000000000001e-05 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4010 4.37 4:ed874e32:::smithi15736696-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.711606205999999, 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39386449699999998, 2024-09-16T03:39:40.406 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4010 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634243+0000", 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.869e-06 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634249+0000", 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9449999999999998e-06 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634255+0000", 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1979999999999998e-06 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.734247+0000", 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099991570000000002 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.409 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.734386+0000", 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013961600000000001 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.027978+0000", 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29359164599999998 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.028036+0000", 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7802e-05 2024-09-16T03:39:40.410 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.028104+0000", 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8264999999999998e-05 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4034 4.17 4:eb60a28d:::smithi15736696-41:head [write 635397~702323,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 34.309184760999997, 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471451500000001, 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.411 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4034 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036663+0000", 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.855e-06 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037227+0000", 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00056451099999999996 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037240+0000", 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2208999999999999e-05 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037245+0000", 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6319999999999996e-06 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037305+0000", 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0272000000000002e-05 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037384+0000", 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8330000000000004e-05 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037491+0000", 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010730699999999999 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.042284+0000", 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0047932649999999997 2024-09-16T03:39:40.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.140922+0000", 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098637552000000003 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241267+0000", 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100344866 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241330+0000", 2024-09-16T03:39:40.416 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2953000000000005e-05 2024-09-16T03:39:40.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241375+0000", 2024-09-16T03:39:40.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5763000000000002e-05 2024-09-16T03:39:40.417 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.417 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.417 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.417 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "rep_scrubmap(4.32 e135 from shard 11)", 2024-09-16T03:39:40.417 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:11.840602+0000", 2024-09-16T03:39:40.417 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.505243530000001, 2024-09-16T03:39:40.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.26655261099999999, 2024-09-16T03:39:40.418 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.418 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.418 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:40.418 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840602+0000", 2024-09-16T03:39:40.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840602+0000", 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840605+0000", 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840616+0000", 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0672e-05 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.419 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.420 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840632+0000", 2024-09-16T03:39:40.420 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6268000000000001e-05 2024-09-16T03:39:40.420 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.420 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.420 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.420 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.940701+0000", 2024-09-16T03:39:40.420 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10006937 2024-09-16T03:39:40.420 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.420 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.420 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.420 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.107074+0000", 2024-09-16T03:39:40.420 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.16637296600000001 2024-09-16T03:39:40.420 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.107101+0000", 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6225999999999999e-05 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.107155+0000", 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4311999999999998e-05 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.422 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4101 4.17 4:eb60a28d:::smithi15736696-41:head [write 719432~794921] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T03:39:40.422 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:11.929701+0000", 2024-09-16T03:39:40.422 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.416144406000001, 2024-09-16T03:39:40.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30975548400000003, 2024-09-16T03:39:40.422 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.422 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.422 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.422 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:40.422 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:40.422 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.422 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4101 2024-09-16T03:39:40.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.422 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.929701+0000", 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.929701+0000", 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.929704+0000", 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.21e-06 2024-09-16T03:39:40.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.930544+0000", 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00084025300000000001 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.930561+0000", 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7601999999999999e-05 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.930571+0000", 2024-09-16T03:39:40.424 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1260000000000004e-06 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.030702+0000", 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10013184899999999 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.030769+0000", 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6742000000000006e-05 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:40.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.030873+0000", 2024-09-16T03:39:40.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104005 2024-09-16T03:39:40.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:40.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.033373+0000", 2024-09-16T03:39:40.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0025000489999999998 2024-09-16T03:39:40.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.138540+0000", 2024-09-16T03:39:40.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10516709 2024-09-16T03:39:40.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.239289+0000", 2024-09-16T03:39:40.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10074846599999999 2024-09-16T03:39:40.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.239382+0000", 2024-09-16T03:39:40.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3195000000000005e-05 2024-09-16T03:39:40.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.239457+0000", 2024-09-16T03:39:40.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4896999999999998e-05 2024-09-16T03:39:40.428 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.428 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.428 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.428 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.428 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.428 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4366 4.23 4:c512d873:::smithi15736696-38:head [assert-version v41,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:40.428 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:20.361285+0000", 2024-09-16T03:39:40.428 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 19.984560493, 2024-09-16T03:39:40.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.303627655, 2024-09-16T03:39:40.428 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.428 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.428 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.428 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:40.429 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:40.429 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.429 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4366 2024-09-16T03:39:40.429 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.429 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.429 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361285+0000", 2024-09-16T03:39:40.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.429 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.429 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361285+0000", 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361287+0000", 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.626e-06 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361292+0000", 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3279999999999996e-06 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361304+0000", 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1574e-05 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361310+0000", 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1990000000000003e-06 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361362+0000", 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2459e-05 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361448+0000", 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5928999999999997e-05 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.463126+0000", 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10167792 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.463778+0000", 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000652123 2024-09-16T03:39:40.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.564004+0000", 2024-09-16T03:39:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100225143 2024-09-16T03:39:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.664718+0000", 2024-09-16T03:39:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100714673 2024-09-16T03:39:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:40.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.664865+0000", 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014715099999999999 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.664913+0000", 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7530000000000001e-05 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4526 4.12 4:4a9d235d:::smithi15736696-1:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:39:40.434 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.774075432, 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30344452999999999, 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4526 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:39:40.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.436 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.436 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.436 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.436 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:39:40.436 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.436 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.436 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.436 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.436 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571773+0000", 2024-09-16T03:39:40.436 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3750000000000001e-06 2024-09-16T03:39:40.436 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.436 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.436 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571780+0000", 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5480000000000004e-06 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571797+0000", 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6320999999999998e-05 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571821+0000", 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4111e-05 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.437 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571884+0000", 2024-09-16T03:39:40.438 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3608000000000005e-05 2024-09-16T03:39:40.438 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.438 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571981+0000", 2024-09-16T03:39:40.438 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6255999999999995e-05 2024-09-16T03:39:40.438 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.438 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.875215+0000", 2024-09-16T03:39:40.438 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30323431099999998 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4539 4.12 4:4a9d235d:::smithi15736696-1:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:26.069749+0000", 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 14.276096976, 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.31900961999999999, 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:40.439 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4539 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069749+0000", 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069749+0000", 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.440 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069752+0000", 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4800000000000001e-06 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069779+0000", 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6336e-05 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069794+0000", 2024-09-16T03:39:40.441 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5648e-05 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069805+0000", 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0403e-05 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069884+0000", 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9660000000000006e-05 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069982+0000", 2024-09-16T03:39:40.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7637e-05 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.388758+0000", 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.31877645599999999 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4561 4.23 4:c512d873:::smithi15736696-38:head [read 0~2672379,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e148)", 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:26.675782+0000", 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.670063552, 2024-09-16T03:39:40.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30481955599999999, 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4561 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675782+0000", 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675782+0000", 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675783+0000", 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4900000000000001e-07 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675785+0000", 2024-09-16T03:39:40.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6429999999999999e-06 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675790+0000", 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4859999999999999e-06 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675793+0000", 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0019999999999998e-06 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.446 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.776027+0000", 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10023393899999999 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.776034+0000", 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.939e-06 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.978683+0000", 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264916399999999 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.978742+0000", 2024-09-16T03:39:40.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8699000000000002e-05 2024-09-16T03:39:40.448 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.448 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.448 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.980602+0000", 2024-09-16T03:39:40.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001860235 2024-09-16T03:39:40.448 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.448 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.448 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.448 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.448 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4562 4.23 4:c512d873:::smithi15736696-38:head [read 0~2672379,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e148)", 2024-09-16T03:39:40.448 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:26.675811+0000", 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 13.670035219000001, 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30577821199999999, 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4562 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:40.449 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675811+0000", 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675811+0000", 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675811+0000", 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5400000000000002e-07 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.450 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675816+0000", 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7539999999999997e-06 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675821+0000", 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8160000000000001e-06 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.775929+0000", 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100108076 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.776062+0000", 2024-09-16T03:39:40.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013323400000000001 2024-09-16T03:39:40.452 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:40.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.776067+0000", 2024-09-16T03:39:40.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1669999999999998e-06 2024-09-16T03:39:40.452 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:40.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.980656+0000", 2024-09-16T03:39:40.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.204588772 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.980711+0000", 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4898999999999997e-05 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.981589+0000", 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00087803999999999996 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.453 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:40.454 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:40.454 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:40.455 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:39:40.456 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:40.453+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:40.508 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:40.506+0000 7efeef6d5640 -1 osd.13 159 handle_osd_map injecting map crc failure 2024-09-16T03:39:40.556 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:40.554+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:40.570 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:40.570 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:39:40.571 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:40.579 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:39:40.657 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:40.655+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:40.683 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:40.683 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:40.683 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:40.683 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:40.683 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:40.691 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:39:40.694 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:39:40.694 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:40.692+0000 7fedb5739640 -1 osd.9 160 handle_osd_map injecting map crc failure 2024-09-16T03:39:40.697 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:40.695+0000 7eff7884b640 -1 osd.3 160 handle_osd_map injecting map crc failure 2024-09-16T03:39:40.698 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:40.696+0000 7eff7884b640 -1 osd.3 160 handle_osd_map injecting map crc failure 2024-09-16T03:39:40.758 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:40.756+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:40.799 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:40.799 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:40.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:40.800 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:40.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.800 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3402 4.1b 4.1792cddb (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e110)", 2024-09-16T03:39:40.800 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:44.983846+0000", 2024-09-16T03:39:40.800 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 55.812559401999998, 2024-09-16T03:39:40.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53764449299999995, 2024-09-16T03:39:40.800 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.800 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.800 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:40.800 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.800 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.800 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.801 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3402 2024-09-16T03:39:40.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.801 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983846+0000", 2024-09-16T03:39:40.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983846+0000", 2024-09-16T03:39:40.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983855+0000", 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2059999999999996e-06 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983860+0000", 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3990000000000003e-06 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983869+0000", 2024-09-16T03:39:40.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5609999999999999e-06 2024-09-16T03:39:40.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983875+0000", 2024-09-16T03:39:40.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9900000000000002e-06 2024-09-16T03:39:40.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983933+0000", 2024-09-16T03:39:40.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8519e-05 2024-09-16T03:39:40.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:40.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:44.983940+0000", 2024-09-16T03:39:40.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1579999999999999e-06 2024-09-16T03:39:40.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:45.521480+0000", 2024-09-16T03:39:40.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53753933300000001 2024-09-16T03:39:40.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:45.521490+0000", 2024-09-16T03:39:40.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0327e-05 2024-09-16T03:39:40.805 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.805 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.805 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.805 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.805 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3590 4.1b 4:dcb6965e:::smithi15736696-28:head [write 2019164~716432 in=716432b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:40.805 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.052101+0000", 2024-09-16T03:39:40.805 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 49.744303791, 2024-09-16T03:39:40.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50534822099999999, 2024-09-16T03:39:40.805 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.805 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.805 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.805 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3590 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052101+0000", 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052101+0000", 2024-09-16T03:39:40.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052103+0000", 2024-09-16T03:39:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.44e-06 2024-09-16T03:39:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052404+0000", 2024-09-16T03:39:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00030129000000000001 2024-09-16T03:39:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052416+0000", 2024-09-16T03:39:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1868999999999999e-05 2024-09-16T03:39:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.052422+0000", 2024-09-16T03:39:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:39:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.103424+0000", 2024-09-16T03:39:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.051001643999999999 2024-09-16T03:39:40.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.103503+0000", 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9871999999999997e-05 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.103590+0000", 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6268999999999997e-05 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.105959+0000", 2024-09-16T03:39:40.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023694179999999999 2024-09-16T03:39:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.356869+0000", 2024-09-16T03:39:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.250909726 2024-09-16T03:39:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.557321+0000", 2024-09-16T03:39:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200451935 2024-09-16T03:39:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.557381+0000", 2024-09-16T03:39:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9960000000000002e-05 2024-09-16T03:39:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.557449+0000", 2024-09-16T03:39:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8621000000000006e-05 2024-09-16T03:39:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.811 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3591 4.1b 4:dcb6965e:::smithi15736696-28:head [write 3335341~194162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 49.739999644999997, 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60146457399999997, 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3591 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.812 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:39:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:39:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056407+0000", 2024-09-16T03:39:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8920000000000001e-06 2024-09-16T03:39:40.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056502+0000", 2024-09-16T03:39:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4838000000000002e-05 2024-09-16T03:39:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056511+0000", 2024-09-16T03:39:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3579999999999992e-06 2024-09-16T03:39:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156581+0000", 2024-09-16T03:39:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100069254 2024-09-16T03:39:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256413+0000", 2024-09-16T03:39:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099832613000000001 2024-09-16T03:39:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256479+0000", 2024-09-16T03:39:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5388000000000005e-05 2024-09-16T03:39:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256582+0000", 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000102876 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257889+0000", 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001307119 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657701+0000", 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39981235999999998 2024-09-16T03:39:40.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657769+0000", 2024-09-16T03:39:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7508000000000005e-05 2024-09-16T03:39:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657827+0000", 2024-09-16T03:39:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8878000000000001e-05 2024-09-16T03:39:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657870+0000", 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2490000000000001e-05 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3592 4.1b 4:dcb6965e:::smithi15736696-28:head [setxattr _header (56) in=7b,truncate 3529503] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 49.639791492999997, 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60139765499999998, 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.818 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3592 2024-09-16T03:39:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:39:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.819 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:39:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156615+0000", 2024-09-16T03:39:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.714e-06 2024-09-16T03:39:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156623+0000", 2024-09-16T03:39:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5939999999999996e-06 2024-09-16T03:39:40.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156634+0000", 2024-09-16T03:39:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.093e-05 2024-09-16T03:39:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156640+0000", 2024-09-16T03:39:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1890000000000004e-06 2024-09-16T03:39:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257172+0000", 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100532525 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257233+0000", 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0736999999999998e-05 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257319+0000", 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5870000000000003e-05 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.822 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257891+0000", 2024-09-16T03:39:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057231100000000004 2024-09-16T03:39:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657732+0000", 2024-09-16T03:39:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39984067699999998 2024-09-16T03:39:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.757877+0000", 2024-09-16T03:39:40.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100145238 2024-09-16T03:39:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.757947+0000", 2024-09-16T03:39:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9821999999999997e-05 2024-09-16T03:39:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758011+0000", 2024-09-16T03:39:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4047999999999994e-05 2024-09-16T03:39:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.824 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3593 4.1b 4:dcb6965e:::smithi15736696-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 49.639756759000001, 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60159266700000003, 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3593 2024-09-16T03:39:40.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:39:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:39:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.826 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156649+0000", 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6400000000000002e-07 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156652+0000", 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9579999999999999e-06 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156658+0000", 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4729999999999997e-06 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.827 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156667+0000", 2024-09-16T03:39:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1940000000000004e-06 2024-09-16T03:39:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257836+0000", 2024-09-16T03:39:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101168444 2024-09-16T03:39:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257881+0000", 2024-09-16T03:39:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4910999999999997e-05 2024-09-16T03:39:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758073+0000", 2024-09-16T03:39:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50019239699999996 2024-09-16T03:39:40.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758147+0000", 2024-09-16T03:39:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3360000000000002e-05 2024-09-16T03:39:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758241+0000", 2024-09-16T03:39:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4265999999999995e-05 2024-09-16T03:39:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4013 4.1f 4:fcb75e43:::smithi15736696-26:head [assert-version v111,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.540779+0000", 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 35.255625905999999, 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40351445699999999, 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4013 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.831 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.540779+0000", 2024-09-16T03:39:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.540779+0000", 2024-09-16T03:39:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.832 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.540781+0000", 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.787e-06 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.540787+0000", 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2709999999999998e-06 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.540798+0000", 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1185e-05 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.540804+0000", 2024-09-16T03:39:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1090000000000003e-06 2024-09-16T03:39:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.640931+0000", 2024-09-16T03:39:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10012628599999999 2024-09-16T03:39:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.641041+0000", 2024-09-16T03:39:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011045299999999999 2024-09-16T03:39:40.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 10,12", 2024-09-16T03:39:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.743114+0000", 2024-09-16T03:39:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102072786 2024-09-16T03:39:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.743478+0000", 2024-09-16T03:39:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00036393900000000003 2024-09-16T03:39:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.743878+0000", 2024-09-16T03:39:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00040044100000000002 2024-09-16T03:39:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.944223+0000", 2024-09-16T03:39:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200344627 2024-09-16T03:39:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.944256+0000", 2024-09-16T03:39:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3222999999999997e-05 2024-09-16T03:39:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.836 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.944294+0000", 2024-09-16T03:39:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.735e-05 2024-09-16T03:39:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.837 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4069 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:39:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548281+0000", 2024-09-16T03:39:40.840 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.248124042000001, 2024-09-16T03:39:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62975188199999999, 2024-09-16T03:39:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4069 2024-09-16T03:39:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.841 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548281+0000", 2024-09-16T03:39:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548281+0000", 2024-09-16T03:39:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548284+0000", 2024-09-16T03:39:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:39:40.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548292+0000", 2024-09-16T03:39:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2169999999999994e-06 2024-09-16T03:39:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548308+0000", 2024-09-16T03:39:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5602000000000001e-05 2024-09-16T03:39:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548322+0000", 2024-09-16T03:39:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-05 2024-09-16T03:39:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548706+0000", 2024-09-16T03:39:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038430199999999999 2024-09-16T03:39:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548721+0000", 2024-09-16T03:39:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4440999999999999e-05 2024-09-16T03:39:40.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.177929+0000", 2024-09-16T03:39:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62920806500000004 2024-09-16T03:39:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.178033+0000", 2024-09-16T03:39:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103824 2024-09-16T03:39:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.845 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4070 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:39:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548345+0000", 2024-09-16T03:39:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.248060185, 2024-09-16T03:39:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.729858275, 2024-09-16T03:39:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.846 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4070 2024-09-16T03:39:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548345+0000", 2024-09-16T03:39:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548345+0000", 2024-09-16T03:39:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548346+0000", 2024-09-16T03:39:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.64e-07 2024-09-16T03:39:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548354+0000", 2024-09-16T03:39:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.875e-06 2024-09-16T03:39:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548377+0000", 2024-09-16T03:39:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3612999999999999e-05 2024-09-16T03:39:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548390+0000", 2024-09-16T03:39:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2836999999999999e-05 2024-09-16T03:39:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548761+0000", 2024-09-16T03:39:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037075600000000003 2024-09-16T03:39:40.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548769+0000", 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5110000000000003e-06 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278121+0000", 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72935201800000005 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278203+0000", 2024-09-16T03:39:40.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1700999999999996e-05 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4071 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548403+0000", 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.248002194000001, 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73003053100000004, 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4071 2024-09-16T03:39:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548403+0000", 2024-09-16T03:39:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548403+0000", 2024-09-16T03:39:40.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548404+0000", 2024-09-16T03:39:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0500000000000002e-07 2024-09-16T03:39:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548408+0000", 2024-09-16T03:39:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2699999999999998e-06 2024-09-16T03:39:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548415+0000", 2024-09-16T03:39:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7340000000000002e-06 2024-09-16T03:39:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548421+0000", 2024-09-16T03:39:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.84e-06 2024-09-16T03:39:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548798+0000", 2024-09-16T03:39:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000377916 2024-09-16T03:39:40.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548806+0000", 2024-09-16T03:39:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5599999999999996e-06 2024-09-16T03:39:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278371+0000", 2024-09-16T03:39:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.729564616 2024-09-16T03:39:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278433+0000", 2024-09-16T03:39:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2689999999999998e-05 2024-09-16T03:39:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4079 3.0 3:18759fb9:::13e91a6c9fb63c85209a4d9065cc1e1225929f8ddc4955a7e63272d6c7055a80:head [write 0~1183232] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:39:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548428+0000", 2024-09-16T03:39:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 30.247976908999998, 2024-09-16T03:39:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0303502819999999, 2024-09-16T03:39:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.856 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4079 2024-09-16T03:39:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548428+0000", 2024-09-16T03:39:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548428+0000", 2024-09-16T03:39:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548429+0000", 2024-09-16T03:39:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2399999999999998e-07 2024-09-16T03:39:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549249+0000", 2024-09-16T03:39:40.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00081996300000000005 2024-09-16T03:39:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549263+0000", 2024-09-16T03:39:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4319e-05 2024-09-16T03:39:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549271+0000", 2024-09-16T03:39:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1640000000000001e-06 2024-09-16T03:39:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549342+0000", 2024-09-16T03:39:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0638999999999999e-05 2024-09-16T03:39:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549353+0000", 2024-09-16T03:39:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1653e-05 2024-09-16T03:39:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.274940+0000", 2024-09-16T03:39:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72558676 2024-09-16T03:39:40.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.275056+0000", 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011581 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,6,9", 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.275176+0000", 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119963 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.377903+0000", 2024-09-16T03:39:40.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10272743199999999 2024-09-16T03:39:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578407+0000", 2024-09-16T03:39:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200503758 2024-09-16T03:39:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578475+0000", 2024-09-16T03:39:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7718000000000005e-05 2024-09-16T03:39:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578663+0000", 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018870599999999999 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578706+0000", 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2459000000000001e-05 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578778+0000", 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2414000000000002e-05 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.863 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4212 4.35 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e140)", 2024-09-16T03:39:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:15.308792+0000", 2024-09-16T03:39:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.487613206999999, 2024-09-16T03:39:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.49743623399999998, 2024-09-16T03:39:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.864 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4212 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308792+0000", 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308792+0000", 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308792+0000", 2024-09-16T03:39:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.75e-07 2024-09-16T03:39:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308796+0000", 2024-09-16T03:39:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3160000000000001e-06 2024-09-16T03:39:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308805+0000", 2024-09-16T03:39:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2560000000000008e-06 2024-09-16T03:39:40.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.308812+0000", 2024-09-16T03:39:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.635e-06 2024-09-16T03:39:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.310568+0000", 2024-09-16T03:39:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001756075 2024-09-16T03:39:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:40.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.310639+0000", 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0964999999999996e-05 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.806106+0000", 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.49546743700000001 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.806160+0000", 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3767000000000002e-05 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:15.806228+0000", 2024-09-16T03:39:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8108000000000006e-05 2024-09-16T03:39:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:173 3.0 3:3f1f54cb:::df21b27e4940b754b9c0285d87bcbc8dbfcdd0059bba94745a59fbc37acf1ebe:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e146)", 2024-09-16T03:39:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:25.238052+0000", 2024-09-16T03:39:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.558352736, 2024-09-16T03:39:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.41153400800000001, 2024-09-16T03:39:40.869 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238052+0000", 2024-09-16T03:39:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238052+0000", 2024-09-16T03:39:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238054+0000", 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0600000000000002e-06 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238060+0000", 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5169999999999999e-06 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238071+0000", 2024-09-16T03:39:40.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1182999999999999e-05 2024-09-16T03:39:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238079+0000", 2024-09-16T03:39:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0499999999999992e-06 2024-09-16T03:39:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238125+0000", 2024-09-16T03:39:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6146000000000002e-05 2024-09-16T03:39:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238185+0000", 2024-09-16T03:39:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9463000000000003e-05 2024-09-16T03:39:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,6,9", 2024-09-16T03:39:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238324+0000", 2024-09-16T03:39:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013895100000000001 2024-09-16T03:39:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.238802+0000", 2024-09-16T03:39:40.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00047803899999999998 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.239099+0000", 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029756100000000001 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.439494+0000", 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200395028 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.649394+0000", 2024-09-16T03:39:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20989983100000001 2024-09-16T03:39:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.649465+0000", 2024-09-16T03:39:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.1077999999999999e-05 2024-09-16T03:39:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.649586+0000", 2024-09-16T03:39:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000121101 2024-09-16T03:39:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.875 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4534 4.35 4:ad0b9852:::smithi15736696-16:head [write 574905~711397 in=711397b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.060664+0000", 2024-09-16T03:39:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.735741183, 2024-09-16T03:39:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62894715300000004, 2024-09-16T03:39:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.876 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4534 2024-09-16T03:39:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060664+0000", 2024-09-16T03:39:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060664+0000", 2024-09-16T03:39:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060665+0000", 2024-09-16T03:39:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2330000000000001e-06 2024-09-16T03:39:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060808+0000", 2024-09-16T03:39:40.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014335500000000001 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060819+0000", 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0828e-05 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060826+0000", 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7209999999999997e-06 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.879 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.237917+0000", 2024-09-16T03:39:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17709092100000001 2024-09-16T03:39:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.237985+0000", 2024-09-16T03:39:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8270999999999997e-05 2024-09-16T03:39:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:39:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.238092+0000", 2024-09-16T03:39:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000107324 2024-09-16T03:39:40.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241761+0000", 2024-09-16T03:39:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036689190000000001 2024-09-16T03:39:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243578+0000", 2024-09-16T03:39:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018165029999999999 2024-09-16T03:39:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.689508+0000", 2024-09-16T03:39:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.44593033199999998 2024-09-16T03:39:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.689561+0000", 2024-09-16T03:39:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2401000000000002e-05 2024-09-16T03:39:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.689611+0000", 2024-09-16T03:39:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0345e-05 2024-09-16T03:39:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.882 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4535 4.35 4:ad0b9852:::smithi15736696-16:head [write 1885086~525140 in=525140b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.066720+0000", 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.729684981, 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72311801499999995, 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.883 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4535 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066720+0000", 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066720+0000", 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.884 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066721+0000", 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.111e-06 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066954+0000", 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023295100000000001 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066965+0000", 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0845e-05 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.885 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066970+0000", 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5459999999999999e-06 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239646+0000", 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17267560400000001 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239715+0000", 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8566999999999996e-05 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239843+0000", 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000128861 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241769+0000", 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001925544 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243609+0000", 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018399340000000001 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789655+0000", 2024-09-16T03:39:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54604627900000002 2024-09-16T03:39:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789709+0000", 2024-09-16T03:39:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3479000000000001e-05 2024-09-16T03:39:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789838+0000", 2024-09-16T03:39:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012929400000000001 2024-09-16T03:39:40.888 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4536 4.35 4:ad0b9852:::smithi15736696-16:head [write 2930189~44716 in=44716b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.069712+0000", 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.726693371, 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72027944300000002, 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.889 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4536 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069712+0000", 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069712+0000", 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069713+0000", 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1209999999999999e-06 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069748+0000", 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5626000000000001e-05 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069765+0000", 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7104000000000001e-05 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069771+0000", 2024-09-16T03:39:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4310000000000002e-06 2024-09-16T03:39:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241008+0000", 2024-09-16T03:39:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17123754899999999 2024-09-16T03:39:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241070+0000", 2024-09-16T03:39:40.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1351000000000003e-05 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241165+0000", 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5372999999999996e-05 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241774+0000", 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000609258 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243890+0000", 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021156669999999999 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789905+0000", 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54601474800000005 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789952+0000", 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7539e-05 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789991+0000", 2024-09-16T03:39:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8676e-05 2024-09-16T03:39:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4537 4.35 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=63b,truncate 2974905] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.069780+0000", 2024-09-16T03:39:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.726624877999999, 2024-09-16T03:39:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72043964199999999, 2024-09-16T03:39:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.895 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4537 2024-09-16T03:39:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069780+0000", 2024-09-16T03:39:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069780+0000", 2024-09-16T03:39:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069781+0000", 2024-09-16T03:39:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:39:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069784+0000", 2024-09-16T03:39:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9170000000000002e-06 2024-09-16T03:39:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069788+0000", 2024-09-16T03:39:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7230000000000004e-06 2024-09-16T03:39:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069791+0000", 2024-09-16T03:39:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0060000000000001e-06 2024-09-16T03:39:40.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241479+0000", 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17168776899999999 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241523+0000", 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4270000000000001e-05 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241598+0000", 2024-09-16T03:39:40.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4264000000000006e-05 2024-09-16T03:39:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241973+0000", 2024-09-16T03:39:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037565999999999999 2024-09-16T03:39:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.344072+0000", 2024-09-16T03:39:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10209850500000001 2024-09-16T03:39:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790106+0000", 2024-09-16T03:39:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.44603410500000001 2024-09-16T03:39:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790149+0000", 2024-09-16T03:39:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3204000000000002e-05 2024-09-16T03:39:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790220+0000", 2024-09-16T03:39:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0640000000000001e-05 2024-09-16T03:39:40.901 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4538 4.35 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:39:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.726609351, 2024-09-16T03:39:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72066842900000005, 2024-09-16T03:39:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:40.902 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4538 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:39:40.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:39:40.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.03e-07 2024-09-16T03:39:40.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069797+0000", 2024-09-16T03:39:40.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4500000000000001e-06 2024-09-16T03:39:40.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069801+0000", 2024-09-16T03:39:40.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8850000000000001e-06 2024-09-16T03:39:40.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.169935+0000", 2024-09-16T03:39:40.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013362100000001 2024-09-16T03:39:40.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241791+0000", 2024-09-16T03:39:40.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.071856512999999997 2024-09-16T03:39:40.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:40.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241834+0000", 2024-09-16T03:39:40.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2824999999999997e-05 2024-09-16T03:39:40.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790284+0000", 2024-09-16T03:39:40.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54844967 2024-09-16T03:39:40.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790369+0000", 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5453999999999999e-05 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790464+0000", 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4708000000000002e-05 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4644 3.0 3:1f968072:::a9957f29a7f946349b562c449f43fa1bc2e3c48b00925943f7b23963f3ffcd03:head [write 0~1316352] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e149)", 2024-09-16T03:39:40.907 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:29.647871+0000", 2024-09-16T03:39:40.908 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 11.148533469, 2024-09-16T03:39:40.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.707065583, 2024-09-16T03:39:40.908 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.908 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.908 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.908 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:40.908 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:40.908 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:40.908 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4644 2024-09-16T03:39:40.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.908 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.908 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.647871+0000", 2024-09-16T03:39:40.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.647871+0000", 2024-09-16T03:39:40.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.647873+0000", 2024-09-16T03:39:40.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6050000000000001e-06 2024-09-16T03:39:40.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648606+0000", 2024-09-16T03:39:40.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073310499999999995 2024-09-16T03:39:40.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648619+0000", 2024-09-16T03:39:40.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3023000000000001e-05 2024-09-16T03:39:40.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648631+0000", 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.173e-05 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.748799+0000", 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100167847 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.748813+0000", 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3913999999999999e-05 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.950128+0000", 2024-09-16T03:39:40.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20131515899999999 2024-09-16T03:39:40.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.950233+0000", 2024-09-16T03:39:40.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010524 2024-09-16T03:39:40.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1,4,6,9", 2024-09-16T03:39:40.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.950411+0000", 2024-09-16T03:39:40.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000177937 2024-09-16T03:39:40.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153899+0000", 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20348797699999999 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153953+0000", 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3919999999999999e-05 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153964+0000", 2024-09-16T03:39:40.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1089e-05 2024-09-16T03:39:40.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254327+0000", 2024-09-16T03:39:40.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100362865 2024-09-16T03:39:40.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.354738+0000", 2024-09-16T03:39:40.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100410991 2024-09-16T03:39:40.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:40.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.354824+0000", 2024-09-16T03:39:40.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6557999999999994e-05 2024-09-16T03:39:40.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.354937+0000", 2024-09-16T03:39:40.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112623 2024-09-16T03:39:40.915 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.915 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.915 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.915 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:179 3.0 3:004ec411:::10547772e2dbc1bd82bb7d7e420056cffb9d849bbc3fd015ab311821f554ea0c:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e150)", 2024-09-16T03:39:40.916 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:29.648695+0000", 2024-09-16T03:39:40.916 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 11.147710050000001, 2024-09-16T03:39:40.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70650748100000005, 2024-09-16T03:39:40.916 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.916 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.916 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.916 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648695+0000", 2024-09-16T03:39:40.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648695+0000", 2024-09-16T03:39:40.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648696+0000", 2024-09-16T03:39:40.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2900000000000003e-07 2024-09-16T03:39:40.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648700+0000", 2024-09-16T03:39:40.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.873e-06 2024-09-16T03:39:40.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648708+0000", 2024-09-16T03:39:40.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1969999999999996e-06 2024-09-16T03:39:40.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648715+0000", 2024-09-16T03:39:40.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.17e-06 2024-09-16T03:39:40.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.748978+0000", 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100262658 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.748981+0000", 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3859999999999999e-06 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.953320+0000", 2024-09-16T03:39:40.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20433948599999999 2024-09-16T03:39:40.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.953414+0000", 2024-09-16T03:39:40.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3813000000000005e-05 2024-09-16T03:39:40.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1,4,6,9", 2024-09-16T03:39:40.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.953542+0000", 2024-09-16T03:39:40.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012773799999999999 2024-09-16T03:39:40.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:40.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153969+0000", 2024-09-16T03:39:40.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200426624 2024-09-16T03:39:40.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254193+0000", 2024-09-16T03:39:40.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100223953 2024-09-16T03:39:40.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254393+0000", 2024-09-16T03:39:40.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000200745 2024-09-16T03:39:40.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254424+0000", 2024-09-16T03:39:40.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0825999999999999e-05 2024-09-16T03:39:40.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355004+0000", 2024-09-16T03:39:40.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100579794 2024-09-16T03:39:40.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:40.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355070+0000", 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5999000000000003e-05 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355202+0000", 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013249 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:180 3.0 3:0569db51:::61d0b13b504667b91c94c752c1e35844395f8745f67042f596605bdf34d7970d:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e150)", 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:29.648725+0000", 2024-09-16T03:39:40.923 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 11.147680038000001, 2024-09-16T03:39:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70679689899999998, 2024-09-16T03:39:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648725+0000", 2024-09-16T03:39:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:40.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648725+0000", 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648726+0000", 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0800000000000004e-07 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648729+0000", 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4589999999999998e-06 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648737+0000", 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7880000000000007e-06 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648740+0000", 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6729999999999998e-06 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.849392+0000", 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20065119000000001 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.926 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:40.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.849397+0000", 2024-09-16T03:39:40.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3809999999999998e-06 2024-09-16T03:39:40.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:40.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153085+0000", 2024-09-16T03:39:40.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30368795100000001 2024-09-16T03:39:40.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:40.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153326+0000", 2024-09-16T03:39:40.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000241396 2024-09-16T03:39:40.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1,4,6,9", 2024-09-16T03:39:40.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153489+0000", 2024-09-16T03:39:40.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000162492 2024-09-16T03:39:40.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:40.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254432+0000", 2024-09-16T03:39:40.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100943038 2024-09-16T03:39:40.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254454+0000", 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2212999999999999e-05 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254476+0000", 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1781e-05 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355284+0000", 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100808457 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:40.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355335+0000", 2024-09-16T03:39:40.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0864e-05 2024-09-16T03:39:40.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:40.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355396+0000", 2024-09-16T03:39:40.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1135000000000003e-05 2024-09-16T03:39:40.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:40.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:40.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:40.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355522+0000", 2024-09-16T03:39:40.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000125473 2024-09-16T03:39:40.931 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.931 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.931 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.931 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:40.931 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:40.931 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:40.933 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:39:40.933 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:40.931+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:40.954 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 9, 1] out_osds: [5, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:39:40.955 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:39:40.955 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:39:40.955 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-16T03:39:40.955 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T03:39:40.955 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T03:39:40.955 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-16T03:39:41.033 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:41.031+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:41.069 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:41.069 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T03:39:41.069 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:41.076 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:41.076 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:41.076 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:39:41.076 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:41.084 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:39:41.133 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:41.132+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:41.227 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:41.227 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:41.227 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:41.227 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:41.227 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:41.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:39:41.235 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:39:41.236 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:41.233+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:41.336 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:41.334+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:41.340 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:41.340 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:41.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:41.340 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:41.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.340 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:39:41.341 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:39:41.341 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 149.15191103500001, 2024-09-16T03:39:41.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102630163, 2024-09-16T03:39:41.341 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.341 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.341 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.341 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:39:41.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186081+0000", 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5600000000000005e-07 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186970+0000", 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00088889000000000004 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186979+0000", 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0620000000000007e-06 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186986+0000", 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287184+0000", 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100198036 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287199+0000", 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5458999999999999e-05 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288659+0000", 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014601900000000001 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288711+0000", 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1088000000000003e-05 2024-09-16T03:39:41.344 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:39:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 149.15087460199999, 2024-09-16T03:39:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10205370900000001, 2024-09-16T03:39:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.345 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187118+0000", 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8599999999999998e-07 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188062+0000", 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094447300000000004 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188070+0000", 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.695e-06 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188074+0000", 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0709999999999996e-06 2024-09-16T03:39:41.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287842+0000", 2024-09-16T03:39:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099768478999999993 2024-09-16T03:39:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287864+0000", 2024-09-16T03:39:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1679999999999999e-05 2024-09-16T03:39:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289137+0000", 2024-09-16T03:39:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012732430000000001 2024-09-16T03:39:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289171+0000", 2024-09-16T03:39:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3382e-05 2024-09-16T03:39:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:42 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 149.080792694, 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10172858, 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:39:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257201+0000", 2024-09-16T03:39:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:39:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257536+0000", 2024-09-16T03:39:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033513899999999998 2024-09-16T03:39:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257555+0000", 2024-09-16T03:39:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8547999999999999e-05 2024-09-16T03:39:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257566+0000", 2024-09-16T03:39:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1379999999999999e-05 2024-09-16T03:39:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357720+0000", 2024-09-16T03:39:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100153532 2024-09-16T03:39:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357734+0000", 2024-09-16T03:39:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4491000000000001e-05 2024-09-16T03:39:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358896+0000", 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011620599999999999 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358927+0000", 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1158999999999998e-05 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:43 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'3, mlcod=0'0)", 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:39:41.354 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 149.07900878199999, 2024-09-16T03:39:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101764453, 2024-09-16T03:39:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:39:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:39:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258984+0000", 2024-09-16T03:39:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5689999999999999e-06 2024-09-16T03:39:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258994+0000", 2024-09-16T03:39:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-16T03:39:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259008+0000", 2024-09-16T03:39:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4151999999999999e-05 2024-09-16T03:39:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359073+0000", 2024-09-16T03:39:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100065628 2024-09-16T03:39:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359110+0000", 2024-09-16T03:39:41.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7045999999999997e-05 2024-09-16T03:39:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359124+0000", 2024-09-16T03:39:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.382e-05 2024-09-16T03:39:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360716+0000", 2024-09-16T03:39:41.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015919829999999999 2024-09-16T03:39:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360747+0000", 2024-09-16T03:39:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0874999999999998e-05 2024-09-16T03:39:41.359 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.360 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.360 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:492 4.5 e45/32 4:a11e1d7c:::smithi15736696-43:head v 45'22, mlcod=45'20)", 2024-09-16T03:39:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:39:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 139.51464049500001, 2024-09-16T03:39:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100797151, 2024-09-16T03:39:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:39:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823351+0000", 2024-09-16T03:39:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823353+0000", 2024-09-16T03:39:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2450000000000001e-06 2024-09-16T03:39:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823363+0000", 2024-09-16T03:39:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0074999999999999e-05 2024-09-16T03:39:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.823378+0000", 2024-09-16T03:39:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4534e-05 2024-09-16T03:39:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923461+0000", 2024-09-16T03:39:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100082859 2024-09-16T03:39:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923525+0000", 2024-09-16T03:39:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4616999999999995e-05 2024-09-16T03:39:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.923549+0000", 2024-09-16T03:39:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.357e-05 2024-09-16T03:39:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924106+0000", 2024-09-16T03:39:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000556753 2024-09-16T03:39:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.924148+0000", 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2497999999999999e-05 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1570 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'31, mlcod=48'29)", 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 115.163681884, 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10253683500000001, 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.366 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8500000000000002e-07 2024-09-16T03:39:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174689+0000", 2024-09-16T03:39:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037958600000000001 2024-09-16T03:39:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174699+0000", 2024-09-16T03:39:41.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0015e-05 2024-09-16T03:39:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174706+0000", 2024-09-16T03:39:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6250000000000001e-06 2024-09-16T03:39:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274865+0000", 2024-09-16T03:39:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100159157 2024-09-16T03:39:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274879+0000", 2024-09-16T03:39:41.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3983e-05 2024-09-16T03:39:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276820+0000", 2024-09-16T03:39:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001941248 2024-09-16T03:39:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276846+0000", 2024-09-16T03:39:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5936e-05 2024-09-16T03:39:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1571 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'32, mlcod=48'29)", 2024-09-16T03:39:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:39:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 115.16323113, 2024-09-16T03:39:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10249543, 2024-09-16T03:39:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.372 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:39:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:39:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174761+0000", 2024-09-16T03:39:41.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0800000000000004e-07 2024-09-16T03:39:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175053+0000", 2024-09-16T03:39:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029151600000000002 2024-09-16T03:39:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175063+0000", 2024-09-16T03:39:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0538e-05 2024-09-16T03:39:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175070+0000", 2024-09-16T03:39:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7819999999999996e-06 2024-09-16T03:39:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275895+0000", 2024-09-16T03:39:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10082516 2024-09-16T03:39:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275909+0000", 2024-09-16T03:39:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4197e-05 2024-09-16T03:39:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277232+0000", 2024-09-16T03:39:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013231009999999999 2024-09-16T03:39:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277256+0000", 2024-09-16T03:39:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3428e-05 2024-09-16T03:39:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1572 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'33, mlcod=48'29)", 2024-09-16T03:39:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:41.378 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 115.16288983, 2024-09-16T03:39:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10216886, 2024-09-16T03:39:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8400000000000004e-07 2024-09-16T03:39:41.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175107+0000", 2024-09-16T03:39:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.578e-06 2024-09-16T03:39:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175114+0000", 2024-09-16T03:39:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.888e-06 2024-09-16T03:39:41.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175119+0000", 2024-09-16T03:39:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.074e-06 2024-09-16T03:39:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276662+0000", 2024-09-16T03:39:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101542753 2024-09-16T03:39:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276674+0000", 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1960000000000001e-05 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277258+0000", 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058431900000000003 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277270+0000", 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2604e-05 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2454 4.15 e84/50 4:aff4cc41:::smithi15736696-11:head v 84'41, mlcod=79'39)", 2024-09-16T03:39:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:39:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 85.874752467999997, 2024-09-16T03:39:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101244889, 2024-09-16T03:39:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:39:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:39:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463239+0000", 2024-09-16T03:39:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2899999999999999e-07 2024-09-16T03:39:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463249+0000", 2024-09-16T03:39:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8039999999999996e-06 2024-09-16T03:39:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.463267+0000", 2024-09-16T03:39:41.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7955e-05 2024-09-16T03:39:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563340+0000", 2024-09-16T03:39:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100072999 2024-09-16T03:39:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563396+0000", 2024-09-16T03:39:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6141999999999999e-05 2024-09-16T03:39:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.563414+0000", 2024-09-16T03:39:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8088999999999999e-05 2024-09-16T03:39:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.564427+0000", 2024-09-16T03:39:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0010127230000000001 2024-09-16T03:39:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.564484+0000", 2024-09-16T03:39:41.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6747999999999999e-05 2024-09-16T03:39:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2467 4.5 e85/50)", 2024-09-16T03:39:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:39:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 85.186860342000003, 2024-09-16T03:39:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102439455, 2024-09-16T03:39:41.389 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:39:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:39:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151134+0000", 2024-09-16T03:39:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2030000000000001e-06 2024-09-16T03:39:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151480+0000", 2024-09-16T03:39:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034613999999999999 2024-09-16T03:39:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151500+0000", 2024-09-16T03:39:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9582999999999999e-05 2024-09-16T03:39:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151513+0000", 2024-09-16T03:39:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:39:41.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251653+0000", 2024-09-16T03:39:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10014007900000001 2024-09-16T03:39:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251679+0000", 2024-09-16T03:39:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5721e-05 2024-09-16T03:39:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253532+0000", 2024-09-16T03:39:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001852701 2024-09-16T03:39:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253571+0000", 2024-09-16T03:39:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8677999999999998e-05 2024-09-16T03:39:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2468 4.5 e85/50)", 2024-09-16T03:39:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:39:41.395 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 85.185391812999995, 2024-09-16T03:39:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101505239, 2024-09-16T03:39:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:39:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:39:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152601+0000", 2024-09-16T03:39:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6869999999999999e-06 2024-09-16T03:39:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152719+0000", 2024-09-16T03:39:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117508 2024-09-16T03:39:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152732+0000", 2024-09-16T03:39:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:39:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152740+0000", 2024-09-16T03:39:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7090000000000002e-06 2024-09-16T03:39:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252704+0000", 2024-09-16T03:39:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099964259999999999 2024-09-16T03:39:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252727+0000", 2024-09-16T03:39:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3601e-05 2024-09-16T03:39:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254075+0000", 2024-09-16T03:39:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00134772 2024-09-16T03:39:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254105+0000", 2024-09-16T03:39:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9802e-05 2024-09-16T03:39:41.400 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2623 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'62, mlcod=85'60)", 2024-09-16T03:39:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 80.988754182999998, 2024-09-16T03:39:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.104590794, 2024-09-16T03:39:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.401 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:41.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1899999999999999e-07 2024-09-16T03:39:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349496+0000", 2024-09-16T03:39:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000258934 2024-09-16T03:39:41.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349506+0000", 2024-09-16T03:39:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9389999999999992e-06 2024-09-16T03:39:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349513+0000", 2024-09-16T03:39:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6209999999999998e-06 2024-09-16T03:39:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449668+0000", 2024-09-16T03:39:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015550099999999 2024-09-16T03:39:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449695+0000", 2024-09-16T03:39:41.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6253000000000001e-05 2024-09-16T03:39:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453784+0000", 2024-09-16T03:39:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040896609999999996 2024-09-16T03:39:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453828+0000", 2024-09-16T03:39:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3665999999999998e-05 2024-09-16T03:39:41.406 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2624 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'63, mlcod=85'60)", 2024-09-16T03:39:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:39:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 80.987406945000004, 2024-09-16T03:39:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103286796, 2024-09-16T03:39:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:39:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:39:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350586+0000", 2024-09-16T03:39:41.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0020000000000001e-06 2024-09-16T03:39:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350794+0000", 2024-09-16T03:39:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020824099999999999 2024-09-16T03:39:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350805+0000", 2024-09-16T03:39:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1443999999999999e-05 2024-09-16T03:39:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450876+0000", 2024-09-16T03:39:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100070612 2024-09-16T03:39:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450960+0000", 2024-09-16T03:39:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4444999999999995e-05 2024-09-16T03:39:41.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450980+0000", 2024-09-16T03:39:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9728000000000002e-05 2024-09-16T03:39:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453831+0000", 2024-09-16T03:39:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002850923 2024-09-16T03:39:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453871+0000", 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0401000000000002e-05 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2626 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'65, mlcod=85'60)", 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 80.886702967999994, 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100625902, 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.412 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:39:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451288+0000", 2024-09-16T03:39:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451289+0000", 2024-09-16T03:39:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7100000000000002e-07 2024-09-16T03:39:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451295+0000", 2024-09-16T03:39:41.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2890000000000003e-06 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.451300+0000", 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0710000000000002e-06 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551399+0000", 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10009881 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551482+0000", 2024-09-16T03:39:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2738000000000006e-05 2024-09-16T03:39:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551496+0000", 2024-09-16T03:39:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4394e-05 2024-09-16T03:39:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551881+0000", 2024-09-16T03:39:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038522300000000002 2024-09-16T03:39:41.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.551914+0000", 2024-09-16T03:39:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2906e-05 2024-09-16T03:39:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.417 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2661 4.15 e91/50 4:aff4cc41:::smithi15736696-11:head v 91'52, mlcod=88'48)", 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 79.928806714999993, 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10164756699999999, 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409186+0000", 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-06 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409568+0000", 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038196600000000002 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409578+0000", 2024-09-16T03:39:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.781e-06 2024-09-16T03:39:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409583+0000", 2024-09-16T03:39:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3070000000000002e-06 2024-09-16T03:39:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409633+0000", 2024-09-16T03:39:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0358000000000001e-05 2024-09-16T03:39:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409649+0000", 2024-09-16T03:39:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5875999999999998e-05 2024-09-16T03:39:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510800+0000", 2024-09-16T03:39:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101150728 2024-09-16T03:39:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510832+0000", 2024-09-16T03:39:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2500999999999999e-05 2024-09-16T03:39:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2662 4.15 e91/50 4:aff4cc41:::smithi15736696-11:head v 91'53, mlcod=88'48)", 2024-09-16T03:39:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:39:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 79.927598764999999, 2024-09-16T03:39:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100650287, 2024-09-16T03:39:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:39:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410393+0000", 2024-09-16T03:39:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410394+0000", 2024-09-16T03:39:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6299999999999993e-07 2024-09-16T03:39:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410399+0000", 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9400000000000001e-06 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410407+0000", 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8859999999999993e-06 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.410412+0000", 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9339999999999997e-06 2024-09-16T03:39:41.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510516+0000", 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100103159 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510536+0000", 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0072999999999999e-05 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.511018+0000", 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00048191899999999998 2024-09-16T03:39:41.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.511043+0000", 2024-09-16T03:39:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5412999999999999e-05 2024-09-16T03:39:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2914 4.5 e98/50 4:a4fbca73:::smithi15736696-39:head v 98'72, mlcod=96'69)", 2024-09-16T03:39:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:30.471084+0000", 2024-09-16T03:39:41.428 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 70.866907193000003, 2024-09-16T03:39:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10075181900000001, 2024-09-16T03:39:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471084+0000", 2024-09-16T03:39:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471084+0000", 2024-09-16T03:39:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471087+0000", 2024-09-16T03:39:41.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3580000000000001e-06 2024-09-16T03:39:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471099+0000", 2024-09-16T03:39:41.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.255e-05 2024-09-16T03:39:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471116+0000", 2024-09-16T03:39:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6867999999999998e-05 2024-09-16T03:39:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.471128+0000", 2024-09-16T03:39:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2313999999999999e-05 2024-09-16T03:39:41.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571266+0000", 2024-09-16T03:39:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013743999999999 2024-09-16T03:39:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571286+0000", 2024-09-16T03:39:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0313999999999999e-05 2024-09-16T03:39:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571787+0000", 2024-09-16T03:39:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00050111600000000002 2024-09-16T03:39:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:30.571836+0000", 2024-09-16T03:39:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8859000000000001e-05 2024-09-16T03:39:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.434 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:3970 4.5 e130/110 4:a11e1d7c:::smithi15736696-43:head v 130'83, mlcod=113'81)", 2024-09-16T03:39:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:39:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 36.075335133999999, 2024-09-16T03:39:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102913225, 2024-09-16T03:39:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.435 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262657+0000", 2024-09-16T03:39:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.72e-07 2024-09-16T03:39:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262783+0000", 2024-09-16T03:39:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012681199999999999 2024-09-16T03:39:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262792+0000", 2024-09-16T03:39:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3839999999999997e-06 2024-09-16T03:39:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262799+0000", 2024-09-16T03:39:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4100000000000002e-06 2024-09-16T03:39:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.362958+0000", 2024-09-16T03:39:41.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100158395 2024-09-16T03:39:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.362977+0000", 2024-09-16T03:39:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9741e-05 2024-09-16T03:39:41.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.365511+0000", 2024-09-16T03:39:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0025332499999999999 2024-09-16T03:39:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.365570+0000", 2024-09-16T03:39:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8961e-05 2024-09-16T03:39:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.441 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:4805 4.18 e155/112 4:1f63731d:::smithi15736696-30:head v 155'52, mlcod=128'50)", 2024-09-16T03:39:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:35.524640+0000", 2024-09-16T03:39:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.8133512889999999, 2024-09-16T03:39:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.104400324, 2024-09-16T03:39:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.442 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.524640+0000", 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.524640+0000", 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.524640+0000", 2024-09-16T03:39:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8699999999999999e-07 2024-09-16T03:39:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.526714+0000", 2024-09-16T03:39:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002073835 2024-09-16T03:39:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.444 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.526725+0000", 2024-09-16T03:39:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1148000000000001e-05 2024-09-16T03:39:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.526734+0000", 2024-09-16T03:39:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6899999999999998e-06 2024-09-16T03:39:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.626909+0000", 2024-09-16T03:39:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100174737 2024-09-16T03:39:41.445 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.626945+0000", 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6316000000000003e-05 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.629006+0000", 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020604239999999999 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.629040+0000", 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4987000000000002e-05 2024-09-16T03:39:41.446 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:4806 4.18 e155/112 4:1f63731d:::smithi15736696-30:head v 155'53, mlcod=128'50)", 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:35.528446+0000", 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 5.8095457020000003, 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10102397, 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:41.447 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.528446+0000", 2024-09-16T03:39:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.528446+0000", 2024-09-16T03:39:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.528447+0000", 2024-09-16T03:39:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4399999999999999e-07 2024-09-16T03:39:41.448 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.449 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.449 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:41.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.529209+0000", 2024-09-16T03:39:41.453 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00076229399999999995 2024-09-16T03:39:41.453 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.453 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.453 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:41.453 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.529217+0000", 2024-09-16T03:39:41.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5129999999999996e-06 2024-09-16T03:39:41.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:41.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.529222+0000", 2024-09-16T03:39:41.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8550000000000001e-06 2024-09-16T03:39:41.454 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.454 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.454 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:41.454 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.628131+0000", 2024-09-16T03:39:41.454 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098908877000000006 2024-09-16T03:39:41.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:41.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.628147+0000", 2024-09-16T03:39:41.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.558e-05 2024-09-16T03:39:41.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.455 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.455 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:41.455 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.629438+0000", 2024-09-16T03:39:41.455 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012912010000000001 2024-09-16T03:39:41.455 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.629470+0000", 2024-09-16T03:39:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1806e-05 2024-09-16T03:39:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:41.456 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:41.456 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:41.457 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:41.458 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:41.456+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:41.458 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:41.459 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:41.459 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:41.459 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:39:41.460 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:41.460 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:41.487 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:41.487 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:41.503 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:41.503 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:41.520 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:41.521 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:41.537 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:41.537 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:41.555 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:41.555 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:41.558 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:41.557+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:41.566 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:41.566 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:41.566 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:39:41.566 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:41.571 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:41.571 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:41.575 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:39:41.588 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:41.588 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:41.605 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:41.605 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:41.622 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:41.622 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:41.638 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:41.638 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:41.657 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:41.657 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:41.659 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:41.658+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:41.672 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:41.673 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:41.679 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:41.680 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:41.688 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:39:41.689 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:41.689 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:41.760 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:41.758+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:41.792 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:41.792 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:41.801 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:39:41.885 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:41.884+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:41.945 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:41.945 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:41.954 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:39:41.963 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:41.962+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:42.065 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:42.064+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:42.096 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:42.096 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:42.105 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:39:42.165 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:42.164+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:42.203 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:42.201+0000 7f8be56cd640 -1 osd.7 161 handle_osd_map injecting map crc failure 2024-09-16T03:39:42.211 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:42.211 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:42.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:39:42.267 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:42.266+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:42.331 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:42.331 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:42.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:39:42.368 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:42.366+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:42.447 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:42.447 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:42.455 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:39:42.468 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:42.467+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:42.562 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:42.563 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:42.569 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:42.567+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:42.571 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:39:42.669 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:42.668+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:42.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:39:42.720 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:42.720 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:42.729 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:39:42.771 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:42.769+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:42.871 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:42.870+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:42.878 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:42.878 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:42.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:39:42.972 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:42.971+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:43.035 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:43.035 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:43.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:39:43.073 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:43.072+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:43.150 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:43.150 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:43.159 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:39:43.173 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:43.172+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:43.265 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:43.266 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:43.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:39:43.275 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:43.272+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:43.375 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:43.373+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:43.422 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:43.422 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:43.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:39:43.515 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:43.513+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:43.576 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:43.575+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:43.578 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:43.578 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:43.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:39:43.678 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:43.676+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:43.699 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:43.698+0000 7f49439d4640 -1 osd.4 162 handle_osd_map injecting map crc failure 2024-09-16T03:39:43.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:39:43.703 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:39:43.739 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:43.739 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:43.748 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:39:43.778 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:43.777+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:43.830 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:43.830 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:43.864 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:43.865 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:43.879 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:43.878+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:43.897 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:43.898 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:43.925 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:43.926 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:43.930 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:43.930 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:43.939 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:39:43.942 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:43.943 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:43.959 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:43.959 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:43.976 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:43.976 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:43.979 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:43.978+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:43.993 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:43.994 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:44.009 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:44.010 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:44.026 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:44.026 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:44.044 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:44.044 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:44.055 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:44.055 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:44.055 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:44.055 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:44.061 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:44.061 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:44.064 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:39:44.078 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:44.078 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:44.080 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:44.079+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:44.094 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:44.094 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:44.111 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:44.112 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:44.129 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:44.129 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:44.167 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:44.167 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:44.167 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:44.167 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:44.168 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:44.176 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:39:44.181 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:44.180+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:44.282 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:44.282 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:44.282 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:44.282 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:44.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.282 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.21:5 3.0 e45/44 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'39, mlcod=45'37)", 2024-09-16T03:39:44.282 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:39:44.283 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 143.36487245800001, 2024-09-16T03:39:44.283 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20119541799999999, 2024-09-16T03:39:44.283 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.283 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.283 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.283 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:39:44.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:39:44.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.284 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.284 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.284 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914597+0000", 2024-09-16T03:39:44.284 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.494e-06 2024-09-16T03:39:44.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.285 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914603+0000", 2024-09-16T03:39:44.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7349999999999999e-06 2024-09-16T03:39:44.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914614+0000", 2024-09-16T03:39:44.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0995999999999999e-05 2024-09-16T03:39:44.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.286 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.286 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914623+0000", 2024-09-16T03:39:44.286 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3899999999999993e-06 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.114941+0000", 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20031812399999999 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115001+0000", 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0161999999999998e-05 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.287 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115750+0000", 2024-09-16T03:39:44.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074851699999999998 2024-09-16T03:39:44.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.288 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.288 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115791+0000", 2024-09-16T03:39:44.288 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1e-05 2024-09-16T03:39:44.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.288 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.288 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.5.21:36 3.0 e52/44 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 52'62, mlcod=51'60)", 2024-09-16T03:39:44.288 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:39:44.289 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 132.39484179900001, 2024-09-16T03:39:44.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20104840199999999, 2024-09-16T03:39:44.289 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.289 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.289 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.289 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.289 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:39:44.289 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.289 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.289 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:39:44.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884627+0000", 2024-09-16T03:39:44.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.322e-06 2024-09-16T03:39:44.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.290 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.290 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884634+0000", 2024-09-16T03:39:44.290 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:39:44.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884643+0000", 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7680000000000003e-06 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984740+0000", 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100096066 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084931+0000", 2024-09-16T03:39:44.291 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100191408 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084952+0000", 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.128e-05 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085645+0000", 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00069309500000000002 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.292 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085674+0000", 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9025000000000001e-05 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 114.12954320199999, 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20169899299999999, 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149927+0000", 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9299999999999999e-06 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151090+0000", 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011624459999999999 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151105+0000", 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5238999999999999e-05 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251193+0000", 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100088025 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251556+0000", 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036245599999999999 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251578+0000", 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2651e-05 2024-09-16T03:39:44.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351582+0000", 2024-09-16T03:39:44.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100003811 2024-09-16T03:39:44.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.297 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.297 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351624+0000", 2024-09-16T03:39:44.297 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1434999999999999e-05 2024-09-16T03:39:44.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.297 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1742 3.0 e70/69 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head v 70'104, mlcod=0'0)", 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.50366822300001, 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20271830699999999, 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:44.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:44.299 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:39:44.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.299 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.299 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.299 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777455+0000", 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001655524 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777468+0000", 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2486999999999999e-05 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777474+0000", 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.300 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876577+0000", 2024-09-16T03:39:44.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099103360000000001 2024-09-16T03:39:44.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876594+0000", 2024-09-16T03:39:44.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6449000000000001e-05 2024-09-16T03:39:44.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.301 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.301 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978469+0000", 2024-09-16T03:39:44.301 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101875521 2024-09-16T03:39:44.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978518+0000", 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8488000000000001e-05 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:91 3.0 e70/69 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head v 70'105, mlcod=0'0)", 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.417456354, 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.116532025, 2024-09-16T03:39:44.302 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862015+0000", 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3569999999999998e-06 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862029+0000", 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4082e-05 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862055+0000", 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6570000000000001e-05 2024-09-16T03:39:44.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862070+0000", 2024-09-16T03:39:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4473e-05 2024-09-16T03:39:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877631+0000", 2024-09-16T03:39:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.015560744 2024-09-16T03:39:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.305 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877674+0000", 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3615999999999997e-05 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978525+0000", 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100850696 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978543+0000", 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8487000000000001e-05 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:94 3.0 e70/69 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head v 70'106, mlcod=0'0)", 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.416920135, 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11627603, 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:39:44.307 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:39:44.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.308 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862549+0000", 2024-09-16T03:39:44.308 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1650000000000001e-06 2024-09-16T03:39:44.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.308 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862556+0000", 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8709999999999999e-06 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862566+0000", 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0271e-05 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862573+0000", 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.861e-06 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978123+0000", 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11554993199999999 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978163+0000", 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0201999999999999e-05 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978796+0000", 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063316700000000002 2024-09-16T03:39:44.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978824+0000", 2024-09-16T03:39:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7560999999999999e-05 2024-09-16T03:39:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1947 4.8 e74/68 4:146f84f0:::smithi15736696-6:head v 74'47, mlcod=73'45)", 2024-09-16T03:39:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:39:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 103.154303997, 2024-09-16T03:39:44.311 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103866244, 2024-09-16T03:39:44.312 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.312 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.312 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.312 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:39:44.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.312 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.312 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125164+0000", 2024-09-16T03:39:44.312 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.313 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.313 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.125165+0000", 2024-09-16T03:39:44.316 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.513e-06 2024-09-16T03:39:44.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126855+0000", 2024-09-16T03:39:44.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0016901010000000001 2024-09-16T03:39:44.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.317 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.317 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.126870+0000", 2024-09-16T03:39:44.317 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4487999999999999e-05 2024-09-16T03:39:44.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227009+0000", 2024-09-16T03:39:44.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100138888 2024-09-16T03:39:44.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.318 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.318 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227117+0000", 2024-09-16T03:39:44.318 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107984 2024-09-16T03:39:44.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.227136+0000", 2024-09-16T03:39:44.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9247999999999999e-05 2024-09-16T03:39:44.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.319 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229002+0000", 2024-09-16T03:39:44.319 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001866143 2024-09-16T03:39:44.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.319 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.320 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.229030+0000", 2024-09-16T03:39:44.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7878999999999999e-05 2024-09-16T03:39:44.320 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.320 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.320 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.320 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1979 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'32, mlcod=0'0)", 2024-09-16T03:39:44.320 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:39:44.320 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 102.44172945, 2024-09-16T03:39:44.320 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20429768500000001, 2024-09-16T03:39:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:39:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:39:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837740+0000", 2024-09-16T03:39:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.779e-06 2024-09-16T03:39:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838356+0000", 2024-09-16T03:39:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061634500000000002 2024-09-16T03:39:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838371+0000", 2024-09-16T03:39:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5007e-05 2024-09-16T03:39:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838380+0000", 2024-09-16T03:39:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1009999999999992e-06 2024-09-16T03:39:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038674+0000", 2024-09-16T03:39:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20029450500000001 2024-09-16T03:39:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038710+0000", 2024-09-16T03:39:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6100999999999999e-05 2024-09-16T03:39:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042001+0000", 2024-09-16T03:39:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032909430000000002 2024-09-16T03:39:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042036+0000", 2024-09-16T03:39:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4904000000000003e-05 2024-09-16T03:39:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1980 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'33, mlcod=0'0)", 2024-09-16T03:39:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:39:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 102.43985285399999, 2024-09-16T03:39:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20296795100000001, 2024-09-16T03:39:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:39:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:39:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839616+0000", 2024-09-16T03:39:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3099999999999996e-07 2024-09-16T03:39:44.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840706+0000", 2024-09-16T03:39:44.328 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010900980000000001 2024-09-16T03:39:44.328 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.328 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.328 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840720+0000", 2024-09-16T03:39:44.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.378e-05 2024-09-16T03:39:44.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840727+0000", 2024-09-16T03:39:44.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.525e-06 2024-09-16T03:39:44.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.329 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.329 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.329 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039940+0000", 2024-09-16T03:39:44.329 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19921313199999999 2024-09-16T03:39:44.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039967+0000", 2024-09-16T03:39:44.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6393999999999998e-05 2024-09-16T03:39:44.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.330 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.330 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042552+0000", 2024-09-16T03:39:44.330 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025850999999999999 2024-09-16T03:39:44.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.331 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.331 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042583+0000", 2024-09-16T03:39:44.331 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0991000000000002e-05 2024-09-16T03:39:44.331 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.331 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.331 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.331 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.331 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1981 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'34, mlcod=0'0)", 2024-09-16T03:39:44.331 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:39:44.332 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 102.437830499, 2024-09-16T03:39:44.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201270175, 2024-09-16T03:39:44.332 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.332 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.332 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.332 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.332 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.332 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:39:44.332 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:39:44.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.333 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.333 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841638+0000", 2024-09-16T03:39:44.333 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-16T03:39:44.333 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841644+0000", 2024-09-16T03:39:44.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8050000000000003e-06 2024-09-16T03:39:44.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841653+0000", 2024-09-16T03:39:44.334 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6580000000000008e-06 2024-09-16T03:39:44.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.334 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.334 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841660+0000", 2024-09-16T03:39:44.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9290000000000001e-06 2024-09-16T03:39:44.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041673+0000", 2024-09-16T03:39:44.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20001311499999999 2024-09-16T03:39:44.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.335 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.335 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041696+0000", 2024-09-16T03:39:44.335 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3365000000000001e-05 2024-09-16T03:39:44.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042878+0000", 2024-09-16T03:39:44.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011814499999999999 2024-09-16T03:39:44.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.336 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.336 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042907+0000", 2024-09-16T03:39:44.336 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9655999999999999e-05 2024-09-16T03:39:44.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.337 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.337 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2117 3.0 e76/69 3:3c8eb54b:::593314a3a723c40dec2b25351adb282089b02c0264cb0891ee3f17edde05dfe9:head v 76'138, mlcod=75'136)", 2024-09-16T03:39:44.337 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:39:44.337 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 98.627422811000002, 2024-09-16T03:39:44.337 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10374235, 2024-09-16T03:39:44.337 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.338 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.338 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.338 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:39:44.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.338 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.338 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:39:44.338 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.652045+0000", 2024-09-16T03:39:44.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8099999999999999e-07 2024-09-16T03:39:44.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.339 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.339 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654214+0000", 2024-09-16T03:39:44.339 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.002169191 2024-09-16T03:39:44.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654230+0000", 2024-09-16T03:39:44.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6144e-05 2024-09-16T03:39:44.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.340 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.340 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.654242+0000", 2024-09-16T03:39:44.340 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1748e-05 2024-09-16T03:39:44.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.754387+0000", 2024-09-16T03:39:44.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10014505899999999 2024-09-16T03:39:44.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.341 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.341 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.754409+0000", 2024-09-16T03:39:44.341 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2181999999999999e-05 2024-09-16T03:39:44.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.755736+0000", 2024-09-16T03:39:44.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0013261690000000001 2024-09-16T03:39:44.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.342 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.342 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:05.755787+0000", 2024-09-16T03:39:44.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1675999999999998e-05 2024-09-16T03:39:44.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.342 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.343 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2536 3.0 e86/69 3:10a17b1b:::13dc2fab1f287170fabf96291b912bf80e0b081f291f2c6d41d494566f2d1f8c:head v 86'176, mlcod=86'174)", 2024-09-16T03:39:44.343 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:39:44.343 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 87.439956120999994, 2024-09-16T03:39:44.343 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20664157499999999, 2024-09-16T03:39:44.343 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.343 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.343 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.343 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:39:44.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.344 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:39:44.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839515+0000", 2024-09-16T03:39:44.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3330000000000001e-06 2024-09-16T03:39:44.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843392+0000", 2024-09-16T03:39:44.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038769329999999999 2024-09-16T03:39:44.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843412+0000", 2024-09-16T03:39:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0608e-05 2024-09-16T03:39:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.943546+0000", 2024-09-16T03:39:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100133176 2024-09-16T03:39:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043824+0000", 2024-09-16T03:39:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100278785 2024-09-16T03:39:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043869+0000", 2024-09-16T03:39:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4845999999999997e-05 2024-09-16T03:39:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046096+0000", 2024-09-16T03:39:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022262559999999998 2024-09-16T03:39:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046153+0000", 2024-09-16T03:39:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7637999999999999e-05 2024-09-16T03:39:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2654 4.8 e90/68 4:148e627d:::smithi15736696-23:head v 90'58, mlcod=80'56)", 2024-09-16T03:39:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:39:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 82.904568763, 2024-09-16T03:39:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.106604212, 2024-09-16T03:39:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:39:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:39:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374901+0000", 2024-09-16T03:39:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1749999999999999e-06 2024-09-16T03:39:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375810+0000", 2024-09-16T03:39:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00090911400000000004 2024-09-16T03:39:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375827+0000", 2024-09-16T03:39:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7010000000000001e-05 2024-09-16T03:39:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476010+0000", 2024-09-16T03:39:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100182879 2024-09-16T03:39:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476184+0000", 2024-09-16T03:39:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017408899999999999 2024-09-16T03:39:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476206+0000", 2024-09-16T03:39:44.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2013000000000001e-05 2024-09-16T03:39:44.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481458+0000", 2024-09-16T03:39:44.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052519120000000001 2024-09-16T03:39:44.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481503+0000", 2024-09-16T03:39:44.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5019999999999999e-05 2024-09-16T03:39:44.354 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.354 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.354 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.354 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "pg_update_log_missing(4.c epoch 99/68 rep_tid 69 entries 99'56 (0'0) error 4:34492f38:::smithi15736696-33:head by client.4628.0:3000 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 90'53 roll_forward_to 90'54)", 2024-09-16T03:39:44.354 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:39:44.354 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 72.359610611999997, 2024-09-16T03:39:44.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201155684, 2024-09-16T03:39:44.354 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.355 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.355 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:44.355 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:39:44.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:39:44.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919858+0000", 2024-09-16T03:39:44.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:39:44.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919871+0000", 2024-09-16T03:39:44.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3433e-05 2024-09-16T03:39:44.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919929+0000", 2024-09-16T03:39:44.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7543000000000001e-05 2024-09-16T03:39:44.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919942+0000", 2024-09-16T03:39:44.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3122000000000001e-05 2024-09-16T03:39:44.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.120302+0000", 2024-09-16T03:39:44.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036052800000001 2024-09-16T03:39:44.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.121013+0000", 2024-09-16T03:39:44.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071046800000000002 2024-09-16T03:39:44.358 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.358 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.359 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:4644 3.0 e150/149 3:1f968072:::a9957f29a7f946349b562c449f43fa1bc2e3c48b00925943f7b23963f3ffcd03:head v 150'271, mlcod=0'0)", 2024-09-16T03:39:44.359 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:29.950532+0000", 2024-09-16T03:39:44.359 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.328935358000001, 2024-09-16T03:39:44.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30108226199999999, 2024-09-16T03:39:44.359 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.359 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.359 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.359 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950532+0000", 2024-09-16T03:39:44.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950532+0000", 2024-09-16T03:39:44.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950533+0000", 2024-09-16T03:39:44.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-16T03:39:44.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950538+0000", 2024-09-16T03:39:44.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5170000000000001e-06 2024-09-16T03:39:44.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950544+0000", 2024-09-16T03:39:44.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0580000000000003e-06 2024-09-16T03:39:44.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950550+0000", 2024-09-16T03:39:44.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2330000000000003e-06 2024-09-16T03:39:44.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.050698+0000", 2024-09-16T03:39:44.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100147758 2024-09-16T03:39:44.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.050724+0000", 2024-09-16T03:39:44.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:39:44.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251572+0000", 2024-09-16T03:39:44.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084784999999999 2024-09-16T03:39:44.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251615+0000", 2024-09-16T03:39:44.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2317999999999997e-05 2024-09-16T03:39:44.364 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.364 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.364 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.364 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:179 3.0 e150/149 3:004ec411:::10547772e2dbc1bd82bb7d7e420056cffb9d849bbc3fd015ab311821f554ea0c:head v 150'272, mlcod=0'0)", 2024-09-16T03:39:44.364 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:29.953579+0000", 2024-09-16T03:39:44.365 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.325888894, 2024-09-16T03:39:44.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.29807083000000001, 2024-09-16T03:39:44.365 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.365 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.365 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.365 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953579+0000", 2024-09-16T03:39:44.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953579+0000", 2024-09-16T03:39:44.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953579+0000", 2024-09-16T03:39:44.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5600000000000005e-07 2024-09-16T03:39:44.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953584+0000", 2024-09-16T03:39:44.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.126e-06 2024-09-16T03:39:44.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953590+0000", 2024-09-16T03:39:44.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0680000000000002e-06 2024-09-16T03:39:44.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953594+0000", 2024-09-16T03:39:44.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5480000000000003e-06 2024-09-16T03:39:44.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.151091+0000", 2024-09-16T03:39:44.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.197496745 2024-09-16T03:39:44.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.151114+0000", 2024-09-16T03:39:44.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3045e-05 2024-09-16T03:39:44.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251621+0000", 2024-09-16T03:39:44.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10050719800000001 2024-09-16T03:39:44.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251650+0000", 2024-09-16T03:39:44.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8444000000000001e-05 2024-09-16T03:39:44.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.370 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.370 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "pg_scan(get_digest 4.8 MIN-MIN e 150/149)", 2024-09-16T03:39:44.370 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:30.051498+0000", 2024-09-16T03:39:44.370 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 14.227969947, 2024-09-16T03:39:44.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20012074299999999, 2024-09-16T03:39:44.370 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.370 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.371 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:44.371 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051498+0000", 2024-09-16T03:39:44.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051498+0000", 2024-09-16T03:39:44.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051499+0000", 2024-09-16T03:39:44.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6899999999999996e-07 2024-09-16T03:39:44.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051509+0000", 2024-09-16T03:39:44.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0907000000000001e-05 2024-09-16T03:39:44.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051517+0000", 2024-09-16T03:39:44.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3690000000000001e-06 2024-09-16T03:39:44.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051523+0000", 2024-09-16T03:39:44.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7950000000000004e-06 2024-09-16T03:39:44.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251377+0000", 2024-09-16T03:39:44.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19985429800000001 2024-09-16T03:39:44.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251392+0000", 2024-09-16T03:39:44.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4603000000000001e-05 2024-09-16T03:39:44.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251619+0000", 2024-09-16T03:39:44.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022700199999999999 2024-09-16T03:39:44.375 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.375 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.375 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.375 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "pg_scan(get_digest 3.0 MIN-MIN e 150/149)", 2024-09-16T03:39:44.376 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:30.456069+0000", 2024-09-16T03:39:44.376 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.823399179999999, 2024-09-16T03:39:44.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200403153, 2024-09-16T03:39:44.376 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.376 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.376 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:44.376 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456069+0000", 2024-09-16T03:39:44.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456069+0000", 2024-09-16T03:39:44.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456070+0000", 2024-09-16T03:39:44.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3829999999999999e-06 2024-09-16T03:39:44.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456074+0000", 2024-09-16T03:39:44.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2289999999999997e-06 2024-09-16T03:39:44.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456083+0000", 2024-09-16T03:39:44.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9789999999999999e-06 2024-09-16T03:39:44.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.556184+0000", 2024-09-16T03:39:44.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100101147 2024-09-16T03:39:44.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.656320+0000", 2024-09-16T03:39:44.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100135347 2024-09-16T03:39:44.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.656328+0000", 2024-09-16T03:39:44.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4610000000000008e-06 2024-09-16T03:39:44.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.656472+0000", 2024-09-16T03:39:44.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000143607 2024-09-16T03:39:44.380 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.380 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.380 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.380 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "MOSDPGPush(3.0 151/149 [PushOp(3:0a2ac06d:::fa4d51829b626c1d967af933073bb2ae733ab2be6a82b2fb4183d28c22e9c453:head, version: 122'238, data_included: [0~589824], data_size: 589824, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:0a2ac06d:::fa4d51829b626c1d967af933073bb2ae733ab2be6a82b2fb4183d28c22e9c453:head@122'238, size: 693248, copy_subset: [(0, 693248)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 693248, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T03:39:44.381 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:31.172695+0000", 2024-09-16T03:39:44.381 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 13.106772325, 2024-09-16T03:39:44.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103831614, 2024-09-16T03:39:44.381 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.381 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.381 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:44.381 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:31.172695+0000", 2024-09-16T03:39:44.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:31.172695+0000", 2024-09-16T03:39:44.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:31.172697+0000", 2024-09-16T03:39:44.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2219999999999999e-06 2024-09-16T03:39:44.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:31.173480+0000", 2024-09-16T03:39:44.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00078297999999999996 2024-09-16T03:39:44.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:31.173496+0000", 2024-09-16T03:39:44.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6821999999999999e-05 2024-09-16T03:39:44.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:31.173508+0000", 2024-09-16T03:39:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1163e-05 2024-09-16T03:39:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:31.273637+0000", 2024-09-16T03:39:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10012892599999999 2024-09-16T03:39:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:31.273645+0000", 2024-09-16T03:39:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3820000000000004e-06 2024-09-16T03:39:44.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:31.276527+0000", 2024-09-16T03:39:44.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028821189999999998 2024-09-16T03:39:44.385 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.385 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.385 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.385 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:4811 3.0 e155/149 3:3c95911f:::e0ef79a5931af77a89e00e45b270ee95266718455ec934bcc09140d5d71025ab:head v 155'275, mlcod=150'273)", 2024-09-16T03:39:44.385 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:35.639297+0000", 2024-09-16T03:39:44.385 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 8.6401710860000005, 2024-09-16T03:39:44.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.105075237, 2024-09-16T03:39:44.386 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.386 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.386 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.386 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.639297+0000", 2024-09-16T03:39:44.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.639297+0000", 2024-09-16T03:39:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.639298+0000", 2024-09-16T03:39:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8059999999999999e-06 2024-09-16T03:39:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.642391+0000", 2024-09-16T03:39:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030922050000000002 2024-09-16T03:39:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.642407+0000", 2024-09-16T03:39:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6393e-05 2024-09-16T03:39:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.642416+0000", 2024-09-16T03:39:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8079999999999994e-06 2024-09-16T03:39:44.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.742572+0000", 2024-09-16T03:39:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10015578999999999 2024-09-16T03:39:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.742596+0000", 2024-09-16T03:39:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4406e-05 2024-09-16T03:39:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.744316+0000", 2024-09-16T03:39:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017202090000000001 2024-09-16T03:39:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.744372+0000", 2024-09-16T03:39:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5619999999999999e-05 2024-09-16T03:39:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.390 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.391 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:44.392 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:44.281+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:44.393 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:39:44.393 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:44.391+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:44.493 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:44.491+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:44.536 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:44.536 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:44.544 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:39:44.629 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:44.628+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:44.688 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:44.688 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:44.688 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:44.688 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:44.688 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:44.695 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:44.693+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:44.696 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:39:44.795 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:44.793+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:44.841 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:44.842 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:44.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:44.842 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:44.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.842 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:25 4.6 4:6ece7830:::smithi15736696-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:44.842 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:39:44.842 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 152.71906845699999, 2024-09-16T03:39:44.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40559452699999998, 2024-09-16T03:39:44.842 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.842 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.842 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.842 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.843 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.843 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.843 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 25 2024-09-16T03:39:44.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.843 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:39:44.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:39:44.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119830+0000", 2024-09-16T03:39:44.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9200000000000001e-07 2024-09-16T03:39:44.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120013+0000", 2024-09-16T03:39:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018367200000000001 2024-09-16T03:39:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120021+0000", 2024-09-16T03:39:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9179999999999994e-06 2024-09-16T03:39:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120027+0000", 2024-09-16T03:39:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6710000000000004e-06 2024-09-16T03:39:44.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220194+0000", 2024-09-16T03:39:44.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10016689500000001 2024-09-16T03:39:44.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220399+0000", 2024-09-16T03:39:44.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020502899999999999 2024-09-16T03:39:44.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:44.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220613+0000", 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021383900000000001 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224640+0000", 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040268099999999996 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423348+0000", 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19870853399999999 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525343+0000", 2024-09-16T03:39:44.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101995081 2024-09-16T03:39:44.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525381+0000", 2024-09-16T03:39:44.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7842000000000001e-05 2024-09-16T03:39:44.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525424+0000", 2024-09-16T03:39:44.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2644e-05 2024-09-16T03:39:44.849 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.849 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.849 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.849 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:26 4.6 4:6ece7830:::smithi15736696-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:44.849 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:39:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 152.71885623, 2024-09-16T03:39:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550111300000002, 2024-09-16T03:39:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 26 2024-09-16T03:39:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:39:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:39:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120042+0000", 2024-09-16T03:39:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7999999999999995e-07 2024-09-16T03:39:44.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120210+0000", 2024-09-16T03:39:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016779499999999999 2024-09-16T03:39:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120219+0000", 2024-09-16T03:39:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6859999999999995e-06 2024-09-16T03:39:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.852 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120223+0000", 2024-09-16T03:39:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0330000000000002e-06 2024-09-16T03:39:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221494+0000", 2024-09-16T03:39:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101271812 2024-09-16T03:39:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221572+0000", 2024-09-16T03:39:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7733999999999998e-05 2024-09-16T03:39:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.853 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221697+0000", 2024-09-16T03:39:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012474900000000001 2024-09-16T03:39:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224670+0000", 2024-09-16T03:39:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029729589999999998 2024-09-16T03:39:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.854 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423398+0000", 2024-09-16T03:39:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19872784900000001 2024-09-16T03:39:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525465+0000", 2024-09-16T03:39:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102067557 2024-09-16T03:39:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525494+0000", 2024-09-16T03:39:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9113999999999999e-05 2024-09-16T03:39:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525542+0000", 2024-09-16T03:39:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8145000000000001e-05 2024-09-16T03:39:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:27 4.6 4:6ece7830:::smithi15736696-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:39:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 152.71866864500001, 2024-09-16T03:39:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40541507399999999, 2024-09-16T03:39:44.856 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27 2024-09-16T03:39:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:39:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:39:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120230+0000", 2024-09-16T03:39:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5799999999999999e-07 2024-09-16T03:39:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.858 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120233+0000", 2024-09-16T03:39:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.055e-06 2024-09-16T03:39:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120237+0000", 2024-09-16T03:39:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7099999999999998e-06 2024-09-16T03:39:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120244+0000", 2024-09-16T03:39:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4810000000000003e-06 2024-09-16T03:39:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222660+0000", 2024-09-16T03:39:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102416089 2024-09-16T03:39:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222705+0000", 2024-09-16T03:39:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5368999999999999e-05 2024-09-16T03:39:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222782+0000", 2024-09-16T03:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6623000000000001e-05 2024-09-16T03:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224696+0000", 2024-09-16T03:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019143929999999999 2024-09-16T03:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423402+0000", 2024-09-16T03:39:44.861 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198705305 2024-09-16T03:39:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525582+0000", 2024-09-16T03:39:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102180178 2024-09-16T03:39:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525615+0000", 2024-09-16T03:39:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3303000000000003e-05 2024-09-16T03:39:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.862 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525644+0000", 2024-09-16T03:39:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8909999999999999e-05 2024-09-16T03:39:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:28 4.6 4:6ece7830:::smithi15736696-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 152.71864789700001, 2024-09-16T03:39:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550787799999999, 2024-09-16T03:39:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.863 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 28 2024-09-16T03:39:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.864 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7999999999999996e-07 2024-09-16T03:39:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.865 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120252+0000", 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.249e-06 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120257+0000", 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1969999999999998e-06 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120261+0000", 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1130000000000003e-06 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222976+0000", 2024-09-16T03:39:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102715397 2024-09-16T03:39:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223049+0000", 2024-09-16T03:39:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2846e-05 2024-09-16T03:39:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525666+0000", 2024-09-16T03:39:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30261668800000002 2024-09-16T03:39:44.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525704+0000", 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8025999999999998e-05 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525758+0000", 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3881999999999997e-05 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:49 4.6 4:6df3bbbe:::smithi15736696-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 152.71863283299999, 2024-09-16T03:39:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40556222800000002, 2024-09-16T03:39:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49 2024-09-16T03:39:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.869 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8700000000000002e-07 2024-09-16T03:39:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120454+0000", 2024-09-16T03:39:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018909299999999999 2024-09-16T03:39:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.871 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120465+0000", 2024-09-16T03:39:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.083e-05 2024-09-16T03:39:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120468+0000", 2024-09-16T03:39:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3869999999999999e-06 2024-09-16T03:39:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223082+0000", 2024-09-16T03:39:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102613379 2024-09-16T03:39:44.872 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223189+0000", 2024-09-16T03:39:44.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107728 2024-09-16T03:39:44.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.873 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:44.873 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223368+0000", 2024-09-16T03:39:44.873 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179011 2024-09-16T03:39:44.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.226087+0000", 2024-09-16T03:39:44.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027183390000000002 2024-09-16T03:39:44.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.874 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.874 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423408+0000", 2024-09-16T03:39:44.874 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19732106499999999 2024-09-16T03:39:44.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525779+0000", 2024-09-16T03:39:44.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10237133 2024-09-16T03:39:44.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.875 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525805+0000", 2024-09-16T03:39:44.875 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5967000000000001e-05 2024-09-16T03:39:44.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.875 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.876 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525827+0000", 2024-09-16T03:39:44.876 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1812000000000001e-05 2024-09-16T03:39:44.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.876 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.876 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:50 4.6 4:6df3bbbe:::smithi15736696-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:44.876 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:39:44.876 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 152.718424524, 2024-09-16T03:39:44.877 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50526462999999999, 2024-09-16T03:39:44.877 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.877 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.877 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.877 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.877 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.877 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.877 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 50 2024-09-16T03:39:44.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.877 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.877 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:39:44.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:39:44.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.878 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.878 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120474+0000", 2024-09-16T03:39:44.878 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4600000000000005e-07 2024-09-16T03:39:44.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120901+0000", 2024-09-16T03:39:44.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00042690000000000002 2024-09-16T03:39:44.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120907+0000", 2024-09-16T03:39:44.879 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7710000000000001e-06 2024-09-16T03:39:44.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.879 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.879 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120911+0000", 2024-09-16T03:39:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0500000000000002e-06 2024-09-16T03:39:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224717+0000", 2024-09-16T03:39:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103806175 2024-09-16T03:39:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224763+0000", 2024-09-16T03:39:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5629e-05 2024-09-16T03:39:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224838+0000", 2024-09-16T03:39:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4548999999999994e-05 2024-09-16T03:39:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423390+0000", 2024-09-16T03:39:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19855242400000001 2024-09-16T03:39:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423411+0000", 2024-09-16T03:39:44.881 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0888000000000001e-05 2024-09-16T03:39:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625680+0000", 2024-09-16T03:39:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20226923099999999 2024-09-16T03:39:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625709+0000", 2024-09-16T03:39:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9232000000000001e-05 2024-09-16T03:39:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.882 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.883 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625738+0000", 2024-09-16T03:39:44.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8235e-05 2024-09-16T03:39:44.883 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.883 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.883 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.883 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:51 4.6 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:44.883 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:44.883 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 152.717979942, 2024-09-16T03:39:44.883 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50490003699999997, 2024-09-16T03:39:44.883 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.883 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 51 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.884 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:44.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:44.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8699999999999995e-07 2024-09-16T03:39:44.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.885 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.885 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120920+0000", 2024-09-16T03:39:44.885 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0820000000000001e-06 2024-09-16T03:39:44.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120924+0000", 2024-09-16T03:39:44.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:39:44.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120927+0000", 2024-09-16T03:39:44.886 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5809999999999999e-06 2024-09-16T03:39:44.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.886 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.886 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225783+0000", 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10485588699999999 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225832+0000", 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9750000000000003e-05 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225910+0000", 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7084999999999998e-05 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.887 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423414+0000", 2024-09-16T03:39:44.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19750438200000001 2024-09-16T03:39:44.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423456+0000", 2024-09-16T03:39:44.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2386999999999999e-05 2024-09-16T03:39:44.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.888 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.888 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625766+0000", 2024-09-16T03:39:44.888 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20231011300000001 2024-09-16T03:39:44.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625790+0000", 2024-09-16T03:39:44.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3672999999999998e-05 2024-09-16T03:39:44.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.889 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.889 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625818+0000", 2024-09-16T03:39:44.889 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7576000000000002e-05 2024-09-16T03:39:44.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.889 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:52 4.6 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 152.717967746, 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50502012900000004, 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 52 2024-09-16T03:39:44.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.891 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:44.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.891 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:44.891 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.891 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:39:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120932+0000", 2024-09-16T03:39:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.181e-06 2024-09-16T03:39:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120935+0000", 2024-09-16T03:39:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7690000000000001e-06 2024-09-16T03:39:44.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120938+0000", 2024-09-16T03:39:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6639999999999998e-06 2024-09-16T03:39:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423188+0000", 2024-09-16T03:39:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30224989200000002 2024-09-16T03:39:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:44.893 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423318+0000", 2024-09-16T03:39:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013045999999999999 2024-09-16T03:39:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625844+0000", 2024-09-16T03:39:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20252547600000001 2024-09-16T03:39:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625895+0000", 2024-09-16T03:39:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1242000000000002e-05 2024-09-16T03:39:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625950+0000", 2024-09-16T03:39:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5164999999999998e-05 2024-09-16T03:39:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:279 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:39:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 149.64201595899999, 2024-09-16T03:39:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50446570800000001, 2024-09-16T03:39:44.895 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 279 2024-09-16T03:39:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:39:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196883+0000", 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.48e-06 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196889+0000", 2024-09-16T03:39:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6239999999999999e-06 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196902+0000", 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2804e-05 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196909+0000", 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9379999999999992e-06 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196963+0000", 2024-09-16T03:39:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3437999999999999e-05 2024-09-16T03:39:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op must wait for map", 2024-09-16T03:39:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196969+0000", 2024-09-16T03:39:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1709999999999999e-06 2024-09-16T03:39:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302538+0000", 2024-09-16T03:39:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10556889899999999 2024-09-16T03:39:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302659+0000", 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121441 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500040+0000", 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19738082900000001 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500589+0000", 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00054827000000000001 2024-09-16T03:39:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701118+0000", 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20052917100000001 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701205+0000", 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7120999999999995e-05 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701289+0000", 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4457999999999997e-05 2024-09-16T03:39:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701347+0000", 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8063999999999998e-05 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:306 4.6 4:6ece7830:::smithi15736696-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 148.326013644, 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40927214699999998, 2024-09-16T03:39:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 306 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:39:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512885+0000", 2024-09-16T03:39:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.277e-06 2024-09-16T03:39:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512890+0000", 2024-09-16T03:39:44.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6689999999999997e-06 2024-09-16T03:39:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512900+0000", 2024-09-16T03:39:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0393e-05 2024-09-16T03:39:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512906+0000", 2024-09-16T03:39:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4569999999999996e-06 2024-09-16T03:39:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512951+0000", 2024-09-16T03:39:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4891000000000001e-05 2024-09-16T03:39:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.513000+0000", 2024-09-16T03:39:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9187999999999997e-05 2024-09-16T03:39:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.720758+0000", 2024-09-16T03:39:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20775775699999999 2024-09-16T03:39:44.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.721960+0000", 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001202107 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922010+0000", 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20004993900000001 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922055+0000", 2024-09-16T03:39:44.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4981000000000002e-05 2024-09-16T03:39:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922103+0000", 2024-09-16T03:39:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8097999999999999e-05 2024-09-16T03:39:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922156+0000", 2024-09-16T03:39:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3390000000000002e-05 2024-09-16T03:39:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:419 4.6 4:64ce87bc:::smithi15736696-20:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:39:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 143.92521403000001, 2024-09-16T03:39:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40437787200000003, 2024-09-16T03:39:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.909 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 419 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913687+0000", 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3299999999999999e-06 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913700+0000", 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2830000000000001e-05 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913733+0000", 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2815999999999999e-05 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913750+0000", 2024-09-16T03:39:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7073000000000001e-05 2024-09-16T03:39:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913920+0000", 2024-09-16T03:39:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000170015 2024-09-16T03:39:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913981+0000", 2024-09-16T03:39:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1631999999999995e-05 2024-09-16T03:39:44.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216580+0000", 2024-09-16T03:39:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30259888200000001 2024-09-16T03:39:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217205+0000", 2024-09-16T03:39:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062461099999999996 2024-09-16T03:39:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.913 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317915+0000", 2024-09-16T03:39:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10071053000000001 2024-09-16T03:39:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317996+0000", 2024-09-16T03:39:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0440999999999998e-05 2024-09-16T03:39:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318032+0000", 2024-09-16T03:39:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5944e-05 2024-09-16T03:39:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318061+0000", 2024-09-16T03:39:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9768e-05 2024-09-16T03:39:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:479 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:39:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 143.117357884, 2024-09-16T03:39:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.504011706, 2024-09-16T03:39:44.915 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 479 2024-09-16T03:39:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:39:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:39:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721542+0000", 2024-09-16T03:39:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3199999999999998e-06 2024-09-16T03:39:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721553+0000", 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1413000000000001e-05 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721570+0000", 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.685e-05 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721596+0000", 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5647999999999999e-05 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.921950+0000", 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20035402899999999 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.922009+0000", 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9046999999999999e-05 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.023850+0000", 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10184130199999999 2024-09-16T03:39:44.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.024503+0000", 2024-09-16T03:39:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065257399999999995 2024-09-16T03:39:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.125105+0000", 2024-09-16T03:39:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100602046 2024-09-16T03:39:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225461+0000", 2024-09-16T03:39:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100355835 2024-09-16T03:39:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225512+0000", 2024-09-16T03:39:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0862999999999997e-05 2024-09-16T03:39:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225551+0000", 2024-09-16T03:39:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9779e-05 2024-09-16T03:39:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.921 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:996 4.8 4:148e627d:::smithi15736696-23:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:39:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 133.791554977, 2024-09-16T03:39:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.73588397999999999, 2024-09-16T03:39:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.922 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 996 2024-09-16T03:39:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:39:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:39:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047344+0000", 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.846e-06 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047351+0000", 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.973e-06 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047365+0000", 2024-09-16T03:39:44.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3556e-05 2024-09-16T03:39:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047382+0000", 2024-09-16T03:39:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6994e-05 2024-09-16T03:39:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047453+0000", 2024-09-16T03:39:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1458000000000006e-05 2024-09-16T03:39:44.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047465+0000", 2024-09-16T03:39:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1119e-05 2024-09-16T03:39:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449411+0000", 2024-09-16T03:39:44.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40194680399999999 2024-09-16T03:39:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449535+0000", 2024-09-16T03:39:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124043 2024-09-16T03:39:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783227+0000", 2024-09-16T03:39:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.333691187 2024-09-16T03:39:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.929 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1048 4.16 4:6df3bbbe:::smithi15736696-13:head [write 1622009~726683] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 133.26311086000001, 2024-09-16T03:39:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40682847599999999, 2024-09-16T03:39:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.930 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1048 2024-09-16T03:39:44.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.931 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:44.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:44.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-16T03:39:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576941+0000", 2024-09-16T03:39:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011532230000000001 2024-09-16T03:39:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576949+0000", 2024-09-16T03:39:44.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1089999999999998e-06 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576956+0000", 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000001e-06 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878710+0000", 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30175434299999998 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878765+0000", 2024-09-16T03:39:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4988000000000003e-05 2024-09-16T03:39:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878852+0000", 2024-09-16T03:39:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7201999999999996e-05 2024-09-16T03:39:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881535+0000", 2024-09-16T03:39:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026828310000000001 2024-09-16T03:39:44.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883968+0000", 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024330659999999998 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982541+0000", 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098572876000000004 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982580+0000", 2024-09-16T03:39:44.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8714000000000002e-05 2024-09-16T03:39:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982615+0000", 2024-09-16T03:39:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5438000000000002e-05 2024-09-16T03:39:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1049 4.16 4:6df3bbbe:::smithi15736696-13:head [write 3121996~104915] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:44.936 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 133.26193271099999, 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40579890600000001, 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1049 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:39:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:39:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576966+0000", 2024-09-16T03:39:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8199999999999999e-07 2024-09-16T03:39:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577069+0000", 2024-09-16T03:39:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103583 2024-09-16T03:39:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577076+0000", 2024-09-16T03:39:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8190000000000003e-06 2024-09-16T03:39:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577082+0000", 2024-09-16T03:39:44.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5130000000000004e-06 2024-09-16T03:39:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880130+0000", 2024-09-16T03:39:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.303048127 2024-09-16T03:39:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880183+0000", 2024-09-16T03:39:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3127999999999997e-05 2024-09-16T03:39:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880282+0000", 2024-09-16T03:39:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8881000000000004e-05 2024-09-16T03:39:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883101+0000", 2024-09-16T03:39:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028194180000000002 2024-09-16T03:39:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.979716+0000", 2024-09-16T03:39:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.096614899000000004 2024-09-16T03:39:44.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982707+0000", 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029914249999999998 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982739+0000", 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1250000000000001e-05 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982764+0000", 2024-09-16T03:39:44.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5080999999999999e-05 2024-09-16T03:39:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1050 4.16 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 3226911] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:39:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 133.261808116, 2024-09-16T03:39:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40784258099999998, 2024-09-16T03:39:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.943 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1050 2024-09-16T03:39:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:39:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.944 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577090+0000", 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000003e-07 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577094+0000", 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4340000000000001e-06 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577099+0000", 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8599999999999998e-06 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577104+0000", 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0010000000000001e-06 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881242+0000", 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30413779000000002 2024-09-16T03:39:44.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881290+0000", 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7298e-05 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881367+0000", 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7750000000000006e-05 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883117+0000", 2024-09-16T03:39:44.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001750096 2024-09-16T03:39:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.980218+0000", 2024-09-16T03:39:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097100724999999999 2024-09-16T03:39:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984859+0000", 2024-09-16T03:39:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046412950000000001 2024-09-16T03:39:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984900+0000", 2024-09-16T03:39:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0300999999999999e-05 2024-09-16T03:39:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984932+0000", 2024-09-16T03:39:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2332999999999997e-05 2024-09-16T03:39:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1051 4.16 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 133.261785836, 2024-09-16T03:39:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50807376999999998, 2024-09-16T03:39:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1051 2024-09-16T03:39:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.950 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:44.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:39:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577115+0000", 2024-09-16T03:39:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3980000000000002e-06 2024-09-16T03:39:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577120+0000", 2024-09-16T03:39:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5589999999999996e-06 2024-09-16T03:39:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577125+0000", 2024-09-16T03:39:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9910000000000002e-06 2024-09-16T03:39:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882780+0000", 2024-09-16T03:39:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30565438499999997 2024-09-16T03:39:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882825+0000", 2024-09-16T03:39:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5739999999999999e-05 2024-09-16T03:39:44.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085039+0000", 2024-09-16T03:39:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20221376499999999 2024-09-16T03:39:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085096+0000", 2024-09-16T03:39:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6666000000000003e-05 2024-09-16T03:39:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085186+0000", 2024-09-16T03:39:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9716000000000006e-05 2024-09-16T03:39:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1359 4.6 4:64ce87bc:::smithi15736696-20:head [write 1768928~405354] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T03:39:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:39:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 124.672762698, 2024-09-16T03:39:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40239613499999999, 2024-09-16T03:39:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.955 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1359 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:39:44.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166136+0000", 2024-09-16T03:39:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:39:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166923+0000", 2024-09-16T03:39:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00078683500000000005 2024-09-16T03:39:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.957 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166933+0000", 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0445000000000001e-05 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166938+0000", 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0119999999999996e-06 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264901+0000", 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097962831 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264963+0000", 2024-09-16T03:39:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1812999999999999e-05 2024-09-16T03:39:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.265085+0000", 2024-09-16T03:39:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121914 2024-09-16T03:39:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.269430+0000", 2024-09-16T03:39:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0043455000000000004 2024-09-16T03:39:44.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.372259+0000", 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102828645 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568289+0000", 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19603034999999999 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568475+0000", 2024-09-16T03:39:44.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018613599999999999 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568531+0000", 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5729000000000001e-05 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1676 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v29,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 114.708933835, 2024-09-16T03:39:44.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50424139300000004, 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1676 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:39:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129968+0000", 2024-09-16T03:39:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8430000000000003e-06 2024-09-16T03:39:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129982+0000", 2024-09-16T03:39:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4514e-05 2024-09-16T03:39:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130013+0000", 2024-09-16T03:39:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1081999999999998e-05 2024-09-16T03:39:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130029+0000", 2024-09-16T03:39:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5849999999999999e-05 2024-09-16T03:39:44.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130169+0000", 2024-09-16T03:39:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000139973 2024-09-16T03:39:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130240+0000", 2024-09-16T03:39:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0638999999999999e-05 2024-09-16T03:39:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:39:44.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.532892+0000", 2024-09-16T03:39:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402652654 2024-09-16T03:39:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533546+0000", 2024-09-16T03:39:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065390700000000001 2024-09-16T03:39:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533943+0000", 2024-09-16T03:39:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039703999999999998 2024-09-16T03:39:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634089+0000", 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100146098 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634153+0000", 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3581e-05 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634205+0000", 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2212000000000001e-05 2024-09-16T03:39:44.967 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:77 3.2 e73/68 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head v 73'117, mlcod=66'115)", 2024-09-16T03:39:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:39:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.659895557, 2024-09-16T03:39:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.401393467, 2024-09-16T03:39:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:44.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:39:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:39:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179005+0000", 2024-09-16T03:39:44.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.553e-06 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179029+0000", 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4039000000000001e-05 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179046+0000", 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7139999999999999e-05 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:44.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.279144+0000", 2024-09-16T03:39:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100098245 2024-09-16T03:39:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379323+0000", 2024-09-16T03:39:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100178621 2024-09-16T03:39:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379348+0000", 2024-09-16T03:39:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5315000000000001e-05 2024-09-16T03:39:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580315+0000", 2024-09-16T03:39:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20096734599999999 2024-09-16T03:39:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580395+0000", 2024-09-16T03:39:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0208000000000002e-05 2024-09-16T03:39:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:44.972 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:44.973 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:44.974 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:39:44.975 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:44.973+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:45.075 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:45.072+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:45.118 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:45.118 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:45.118 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:45.118 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:45.127 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:39:45.175 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:45.174+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:45.229 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:45.229 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:45.229 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:45.229 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:45.229 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:45.237 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:39:45.277 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:45.274+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:45.342 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:45.342 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:45.342 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:45.343 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:45.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.343 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:3 3.0 3.f934d18 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:45.343 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:39:45.343 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 149.93953407500001, 2024-09-16T03:39:45.344 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78381636700000001, 2024-09-16T03:39:45.344 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.344 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.344 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:45.344 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.344 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:39:45.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:39:45.345 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.345 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.345 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399429+0000", 2024-09-16T03:39:45.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:39:45.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.346 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.346 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399434+0000", 2024-09-16T03:39:45.346 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1050000000000003e-06 2024-09-16T03:39:45.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399442+0000", 2024-09-16T03:39:45.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0160000000000008e-06 2024-09-16T03:39:45.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.347 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.347 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399448+0000", 2024-09-16T03:39:45.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.879e-06 2024-09-16T03:39:45.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399489+0000", 2024-09-16T03:39:45.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0815999999999997e-05 2024-09-16T03:39:45.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:45.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399495+0000", 2024-09-16T03:39:45.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3439999999999997e-06 2024-09-16T03:39:45.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183226+0000", 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78373077400000002 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183244+0000", 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8128999999999999e-05 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:5 3.0 3.aefbbec0 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:45.349 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:39:45.350 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 149.83976661299999, 2024-09-16T03:39:45.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68447923499999996, 2024-09-16T03:39:45.350 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.350 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.350 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:45.350 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499196+0000", 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3480000000000001e-06 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499202+0000", 2024-09-16T03:39:45.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4219999999999999e-06 2024-09-16T03:39:45.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499212+0000", 2024-09-16T03:39:45.352 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:39:45.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.352 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499218+0000", 2024-09-16T03:39:45.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4920000000000001e-06 2024-09-16T03:39:45.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499283+0000", 2024-09-16T03:39:45.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4906000000000005e-05 2024-09-16T03:39:45.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.353 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:45.353 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499293+0000", 2024-09-16T03:39:45.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0513000000000001e-05 2024-09-16T03:39:45.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183665+0000", 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68437196600000005 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183674+0000", 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9700000000000005e-06 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:6 3.0 3.3bcda120 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 149.83970600399999, 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78474037699999999, 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:45.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:45.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0500000000000002e-07 2024-09-16T03:39:45.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499260+0000", 2024-09-16T03:39:45.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6669999999999998e-06 2024-09-16T03:39:45.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499266+0000", 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499274+0000", 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2250000000000003e-06 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499316+0000", 2024-09-16T03:39:45.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2670000000000003e-05 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499323+0000", 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.37e-06 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283987+0000", 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78466433700000005 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283996+0000", 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8650000000000007e-06 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:15 3.0 3.7ec93288 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 149.73051399400001, 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57518172599999995, 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:45.359 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608449+0000", 2024-09-16T03:39:45.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.184e-06 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608459+0000", 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0553999999999999e-05 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608470+0000", 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0524e-05 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608493+0000", 2024-09-16T03:39:45.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2980999999999999e-05 2024-09-16T03:39:45.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608537+0000", 2024-09-16T03:39:45.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3850000000000002e-05 2024-09-16T03:39:45.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:45.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608542+0000", 2024-09-16T03:39:45.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:39:45.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183623+0000", 2024-09-16T03:39:45.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.575080649 2024-09-16T03:39:45.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183629+0000", 2024-09-16T03:39:45.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6950000000000003e-06 2024-09-16T03:39:45.363 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.363 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:11 3.0 3.77005160 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 149.63701368400001, 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.48153733399999998, 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:39:45.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:39:45.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701949+0000", 2024-09-16T03:39:45.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.314e-06 2024-09-16T03:39:45.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701954+0000", 2024-09-16T03:39:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:39:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701964+0000", 2024-09-16T03:39:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9550000000000004e-06 2024-09-16T03:39:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701986+0000", 2024-09-16T03:39:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1931999999999999e-05 2024-09-16T03:39:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802129+0000", 2024-09-16T03:39:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100142986 2024-09-16T03:39:45.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802135+0000", 2024-09-16T03:39:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6350000000000003e-06 2024-09-16T03:39:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183475+0000", 2024-09-16T03:39:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38133958099999998 2024-09-16T03:39:45.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183485+0000", 2024-09-16T03:39:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0642000000000001e-05 2024-09-16T03:39:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.369 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:6 3.0 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 148.35610977100001, 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60469085300000003, 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:39:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982853+0000", 2024-09-16T03:39:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.393e-06 2024-09-16T03:39:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982858+0000", 2024-09-16T03:39:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.062e-06 2024-09-16T03:39:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982867+0000", 2024-09-16T03:39:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:39:45.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982879+0000", 2024-09-16T03:39:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2241e-05 2024-09-16T03:39:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982996+0000", 2024-09-16T03:39:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116996 2024-09-16T03:39:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983007+0000", 2024-09-16T03:39:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0694e-05 2024-09-16T03:39:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284499+0000", 2024-09-16T03:39:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30149171400000002 2024-09-16T03:39:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284690+0000", 2024-09-16T03:39:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000191032 2024-09-16T03:39:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284908+0000", 2024-09-16T03:39:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021827399999999999 2024-09-16T03:39:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487092+0000", 2024-09-16T03:39:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.202184156 2024-09-16T03:39:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:45.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487110+0000", 2024-09-16T03:39:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7856999999999999e-05 2024-09-16T03:39:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587335+0000", 2024-09-16T03:39:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100224802 2024-09-16T03:39:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587410+0000", 2024-09-16T03:39:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5343e-05 2024-09-16T03:39:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587543+0000", 2024-09-16T03:39:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000132492 2024-09-16T03:39:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:2 3.0 3:16550cfd:::smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:45.378 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:39:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 148.356037579, 2024-09-16T03:39:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40289696000000003, 2024-09-16T03:39:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:39:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982925+0000", 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982928+0000", 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4130000000000002e-06 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982936+0000", 2024-09-16T03:39:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.875e-06 2024-09-16T03:39:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982942+0000", 2024-09-16T03:39:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:39:45.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983093+0000", 2024-09-16T03:39:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015144400000000001 2024-09-16T03:39:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983098+0000", 2024-09-16T03:39:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4909999999999999e-06 2024-09-16T03:39:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385695+0000", 2024-09-16T03:39:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402597539 2024-09-16T03:39:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385821+0000", 2024-09-16T03:39:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125418 2024-09-16T03:39:45.383 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:9 3.0 3:2c9b9e7c:::smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:39:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:39:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 148.35602139, 2024-09-16T03:39:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60615655800000001, 2024-09-16T03:39:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.384 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:39:45.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:39:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982941+0000", 2024-09-16T03:39:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0860000000000001e-06 2024-09-16T03:39:45.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982946+0000", 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982954+0000", 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8520000000000004e-06 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982958+0000", 2024-09-16T03:39:45.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4870000000000004e-06 2024-09-16T03:39:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983116+0000", 2024-09-16T03:39:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015774699999999999 2024-09-16T03:39:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983120+0000", 2024-09-16T03:39:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:39:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486023+0000", 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50290281400000003 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486423+0000", 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040079299999999998 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486707+0000", 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028336600000000001 2024-09-16T03:39:45.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588861+0000", 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102154756 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588952+0000", 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0630000000000005e-05 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588984+0000", 2024-09-16T03:39:45.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2116999999999998e-05 2024-09-16T03:39:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589022+0000", 2024-09-16T03:39:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7910000000000001e-05 2024-09-16T03:39:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589097+0000", 2024-09-16T03:39:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4536999999999995e-05 2024-09-16T03:39:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:12 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:39:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 148.355946042, 2024-09-16T03:39:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60616102900000002, 2024-09-16T03:39:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:39:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:39:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983017+0000", 2024-09-16T03:39:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.483e-06 2024-09-16T03:39:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983022+0000", 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983032+0000", 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0036e-05 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983038+0000", 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6800000000000004e-06 2024-09-16T03:39:45.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983155+0000", 2024-09-16T03:39:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116401 2024-09-16T03:39:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983163+0000", 2024-09-16T03:39:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6219999999999998e-06 2024-09-16T03:39:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589120+0000", 2024-09-16T03:39:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60595653199999999 2024-09-16T03:39:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589176+0000", 2024-09-16T03:39:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6542000000000002e-05 2024-09-16T03:39:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:30 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:39:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 144.12262753600001, 2024-09-16T03:39:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40312788999999999, 2024-09-16T03:39:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:39:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:39:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216338+0000", 2024-09-16T03:39:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0799999999999999e-06 2024-09-16T03:39:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216354+0000", 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6439e-05 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216392+0000", 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7162000000000001e-05 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216444+0000", 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2082e-05 2024-09-16T03:39:45.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417447+0000", 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20100358400000001 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417583+0000", 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135537 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417777+0000", 2024-09-16T03:39:45.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019431200000000001 2024-09-16T03:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418479+0000", 2024-09-16T03:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00070148499999999996 2024-09-16T03:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418755+0000", 2024-09-16T03:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027627100000000003 2024-09-16T03:39:45.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619285+0000", 2024-09-16T03:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200530138 2024-09-16T03:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619370+0000", 2024-09-16T03:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4567999999999994e-05 2024-09-16T03:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619462+0000", 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2231999999999994e-05 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:34 3.0 3:08307d25:::2b3cb797ae2d3abc24b2cae4a012f26325debe2fc38ccb08bd4503e949d305ac:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 133.86417147899999, 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30798127600000003, 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.403 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474790+0000", 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474791+0000", 2024-09-16T03:39:45.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6000000000000003e-07 2024-09-16T03:39:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474865+0000", 2024-09-16T03:39:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4504000000000004e-05 2024-09-16T03:39:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474882+0000", 2024-09-16T03:39:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6765999999999998e-05 2024-09-16T03:39:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.405 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.474890+0000", 2024-09-16T03:39:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.14e-06 2024-09-16T03:39:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681242+0000", 2024-09-16T03:39:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20635221200000001 2024-09-16T03:39:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681355+0000", 2024-09-16T03:39:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000112211 2024-09-16T03:39:45.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.681505+0000", 2024-09-16T03:39:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015040100000000001 2024-09-16T03:39:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682352+0000", 2024-09-16T03:39:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00084686400000000002 2024-09-16T03:39:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.682989+0000", 2024-09-16T03:39:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063749800000000001 2024-09-16T03:39:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782695+0000", 2024-09-16T03:39:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099706063999999997 2024-09-16T03:39:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782735+0000", 2024-09-16T03:39:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9304000000000002e-05 2024-09-16T03:39:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.782771+0000", 2024-09-16T03:39:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6551999999999998e-05 2024-09-16T03:39:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:87 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:39:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 114.534976755, 2024-09-16T03:39:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402368742, 2024-09-16T03:39:45.409 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803986+0000", 2024-09-16T03:39:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9400000000000004e-07 2024-09-16T03:39:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804061+0000", 2024-09-16T03:39:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5230999999999999e-05 2024-09-16T03:39:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804069+0000", 2024-09-16T03:39:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7530000000000001e-06 2024-09-16T03:39:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804073+0000", 2024-09-16T03:39:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9660000000000004e-06 2024-09-16T03:39:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907236+0000", 2024-09-16T03:39:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103162566 2024-09-16T03:39:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907324+0000", 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7894999999999998e-05 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907474+0000", 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014983999999999999 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.908330+0000", 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085600100000000001 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107220+0000", 2024-09-16T03:39:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198889812 2024-09-16T03:39:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206269+0000", 2024-09-16T03:39:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099049611999999995 2024-09-16T03:39:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206305+0000", 2024-09-16T03:39:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5679000000000002e-05 2024-09-16T03:39:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206353+0000", 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8593e-05 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:3086 3.0 3:1e42685c:::6a9007270aa32b63b7f7e5355768bf791fa2d1832cb8adf59666d35d1b037cad:head [write 0~1679360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.927139228000001, 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57631983099999995, 2024-09-16T03:39:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 3086 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411824+0000", 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.849e-06 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414291+0000", 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024666900000000001 2024-09-16T03:39:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414304+0000", 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2886e-05 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414310+0000", 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4219999999999997e-06 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783484+0000", 2024-09-16T03:39:45.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.36917337700000002 2024-09-16T03:39:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783601+0000", 2024-09-16T03:39:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011796899999999999 2024-09-16T03:39:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783711+0000", 2024-09-16T03:39:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110012 2024-09-16T03:39:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887652+0000", 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103940373 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887669+0000", 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6864e-05 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942330+0000", 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.054661333999999999 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987927+0000", 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045596722999999999 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987993+0000", 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6141000000000003e-05 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.988142+0000", 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014919099999999999 2024-09-16T03:39:45.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:211 3.0 3:09d857e3:::85925bf7ff4b61513b387a03c5bd27d81ce6bd960cfaf104686b99338ce39220:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.700960119000001, 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.55082720799999996, 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638003+0000", 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8250000000000001e-06 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638010+0000", 2024-09-16T03:39:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4130000000000003e-06 2024-09-16T03:39:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638022+0000", 2024-09-16T03:39:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2016e-05 2024-09-16T03:39:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638028+0000", 2024-09-16T03:39:45.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0390000000000001e-06 2024-09-16T03:39:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886321+0000", 2024-09-16T03:39:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.248293081 2024-09-16T03:39:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886428+0000", 2024-09-16T03:39:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000106791 2024-09-16T03:39:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886630+0000", 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020280499999999999 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889701+0000", 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030706650000000002 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942392+0000", 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.052691156000000003 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987675+0000", 2024-09-16T03:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045283229000000001 2024-09-16T03:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188710+0000", 2024-09-16T03:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20103479899999999 2024-09-16T03:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188753+0000", 2024-09-16T03:39:45.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2762000000000002e-05 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188829+0000", 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5626999999999999e-05 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:214 3.0 3:2fb3d035:::5dba7feb0009e61b5e6a50f88cc7ba99127287f7eeb210342a4b40e996304a6b:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:39:45.428 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.700846045000006, 2024-09-16T03:39:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.55085424699999996, 2024-09-16T03:39:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:39:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:39:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638116+0000", 2024-09-16T03:39:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1200000000000002e-07 2024-09-16T03:39:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638120+0000", 2024-09-16T03:39:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3400000000000002e-06 2024-09-16T03:39:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638126+0000", 2024-09-16T03:39:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2339999999999999e-06 2024-09-16T03:39:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638130+0000", 2024-09-16T03:39:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6349999999999997e-06 2024-09-16T03:39:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887101+0000", 2024-09-16T03:39:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.248970462 2024-09-16T03:39:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887191+0000", 2024-09-16T03:39:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0087000000000007e-05 2024-09-16T03:39:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887375+0000", 2024-09-16T03:39:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018401600000000001 2024-09-16T03:39:45.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889706+0000", 2024-09-16T03:39:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023305869999999998 2024-09-16T03:39:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942851+0000", 2024-09-16T03:39:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.053145489999999997 2024-09-16T03:39:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.088369+0000", 2024-09-16T03:39:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.14551832200000001 2024-09-16T03:39:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188856+0000", 2024-09-16T03:39:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100486244 2024-09-16T03:39:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188902+0000", 2024-09-16T03:39:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6134999999999998e-05 2024-09-16T03:39:45.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188970+0000", 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7983000000000003e-05 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:219 3.0 3:21e0c4e8:::f1ac9f6b1092b2dbfccdb2f32bdf689fb55e405da82aefeec91d0d73cdce51e1:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.601039763000003, 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.71756203500000004, 2024-09-16T03:39:45.435 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:39:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:39:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737924+0000", 2024-09-16T03:39:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8500000000000001e-06 2024-09-16T03:39:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737931+0000", 2024-09-16T03:39:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9879999999999998e-06 2024-09-16T03:39:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737942+0000", 2024-09-16T03:39:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1837e-05 2024-09-16T03:39:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.838013+0000", 2024-09-16T03:39:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100070438 2024-09-16T03:39:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887695+0000", 2024-09-16T03:39:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.049682624000000002 2024-09-16T03:39:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887831+0000", 2024-09-16T03:39:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013565499999999999 2024-09-16T03:39:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887948+0000", 2024-09-16T03:39:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000117353 2024-09-16T03:39:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:45.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889709+0000", 2024-09-16T03:39:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017605629999999999 2024-09-16T03:39:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.043145+0000", 2024-09-16T03:39:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.153436136 2024-09-16T03:39:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188664+0000", 2024-09-16T03:39:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.14551857000000001 2024-09-16T03:39:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455373+0000", 2024-09-16T03:39:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.26670951199999998 2024-09-16T03:39:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455413+0000", 2024-09-16T03:39:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9517999999999997e-05 2024-09-16T03:39:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455484+0000", 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0990999999999998e-05 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:3100 3.0 3:2db88641:::a63ac782fa4bce1e96397c8301f2734ce3d5850a2dc454a54c42fc5451a8d72d:head [write 0~1584128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 69.067459506000006, 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38491842399999998, 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.442 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 3100 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:39:45.443 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271523+0000", 2024-09-16T03:39:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0649000000000002e-05 2024-09-16T03:39:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273460+0000", 2024-09-16T03:39:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019373820000000001 2024-09-16T03:39:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.444 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273488+0000", 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7537000000000001e-05 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273503+0000", 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5438999999999999e-05 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273645+0000", 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000142013 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273820+0000", 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017546200000000001 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273943+0000", 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012269099999999999 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.276532+0000", 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025884010000000002 2024-09-16T03:39:45.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455527+0000", 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.17899536199999999 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.556259+0000", 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100731821 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656291+0000", 2024-09-16T03:39:45.447 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100032144 2024-09-16T03:39:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656330+0000", 2024-09-16T03:39:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8600000000000003e-05 2024-09-16T03:39:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656420+0000", 2024-09-16T03:39:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0922999999999997e-05 2024-09-16T03:39:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.448 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:54 3.0 3:04086e73:::a30f14ebc2b326cba32a5aae93ab4cf09bd2a972b24cfcba49e65d7da3ac52de:head [call cas.chunk_create_or_get_ref in=131071b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:39:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 63.983653130999997, 2024-09-16T03:39:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30746864499999998, 2024-09-16T03:39:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355308+0000", 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355311+0000", 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3250000000000002e-06 2024-09-16T03:39:45.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355502+0000", 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019106 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355515+0000", 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3730000000000001e-05 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355526+0000", 2024-09-16T03:39:45.451 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0473e-05 2024-09-16T03:39:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355567+0000", 2024-09-16T03:39:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1455000000000002e-05 2024-09-16T03:39:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355576+0000", 2024-09-16T03:39:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.2840000000000006e-06 2024-09-16T03:39:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.452 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.457986+0000", 2024-09-16T03:39:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10241070200000001 2024-09-16T03:39:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.458087+0000", 2024-09-16T03:39:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000100715 2024-09-16T03:39:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.458252+0000", 2024-09-16T03:39:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016499 2024-09-16T03:39:45.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560797+0000", 2024-09-16T03:39:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10254485300000001 2024-09-16T03:39:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662591+0000", 2024-09-16T03:39:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10179377100000001 2024-09-16T03:39:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.454 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662693+0000", 2024-09-16T03:39:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000102174 2024-09-16T03:39:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662745+0000", 2024-09-16T03:39:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1887e-05 2024-09-16T03:39:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662777+0000", 2024-09-16T03:39:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2225999999999999e-05 2024-09-16T03:39:45.455 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:58 3.0 3:07e63342:::0385a732ea8ffd270f01137d752687ebb6ebb56bc894ea96f00cc7a27fcd2563:head [call cas.chunk_create_or_get_ref in=284238b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:39:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 63.983420297000002, 2024-09-16T03:39:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30738563099999999, 2024-09-16T03:39:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.456 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355541+0000", 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355542+0000", 2024-09-16T03:39:45.457 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.1699999999999997e-07 2024-09-16T03:39:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355761+0000", 2024-09-16T03:39:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021889699999999999 2024-09-16T03:39:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355769+0000", 2024-09-16T03:39:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.695e-06 2024-09-16T03:39:45.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355775+0000", 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5239999999999997e-06 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355851+0000", 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6103e-05 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355860+0000", 2024-09-16T03:39:45.459 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0710000000000001e-06 2024-09-16T03:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459143+0000", 2024-09-16T03:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10328306399999999 2024-09-16T03:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459292+0000", 2024-09-16T03:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000148221 2024-09-16T03:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.460 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.459540+0000", 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00024846100000000001 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662661+0000", 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.203120936 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662669+0000", 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6520000000000006e-06 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662824+0000", 2024-09-16T03:39:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015512699999999999 2024-09-16T03:39:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662885+0000", 2024-09-16T03:39:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0789999999999999e-05 2024-09-16T03:39:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662927+0000", 2024-09-16T03:39:45.462 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2172999999999998e-05 2024-09-16T03:39:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:60 3.0 3:16d0c248:::a2b7d44d389d3ac8752762fa1d216869a8025d7e5ae15e8811d7e7f2543095f6:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:39:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 63.983175226, 2024-09-16T03:39:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50513552500000003, 2024-09-16T03:39:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.463 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:39:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:39:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.464 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355787+0000", 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355791+0000", 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.631e-06 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355797+0000", 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7379999999999999e-06 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.465 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355802+0000", 2024-09-16T03:39:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.322e-06 2024-09-16T03:39:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.356898+0000", 2024-09-16T03:39:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010963710000000001 2024-09-16T03:39:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.356902+0000", 2024-09-16T03:39:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0110000000000002e-06 2024-09-16T03:39:45.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560272+0000", 2024-09-16T03:39:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20336954199999999 2024-09-16T03:39:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560376+0000", 2024-09-16T03:39:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103885 2024-09-16T03:39:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:45.467 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560508+0000", 2024-09-16T03:39:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013254599999999999 2024-09-16T03:39:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662786+0000", 2024-09-16T03:39:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10227757699999999 2024-09-16T03:39:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860745+0000", 2024-09-16T03:39:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19795950700000001 2024-09-16T03:39:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860843+0000", 2024-09-16T03:39:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7829000000000001e-05 2024-09-16T03:39:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860890+0000", 2024-09-16T03:39:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7005999999999997e-05 2024-09-16T03:39:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860922+0000", 2024-09-16T03:39:45.469 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1680000000000002e-05 2024-09-16T03:39:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.470 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.470 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:45.472 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:39:45.472 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:45.470+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:45.573 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:45.570+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:45.576 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:45.576 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:45.576 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:45.576 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:45.585 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:39:45.674 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:45.672+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:45.688 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:45.688 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:45.688 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:45.688 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:45.688 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:45.696 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:39:45.719 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:39:45.775 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:45.773+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:45.801 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:45.801 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:45.801 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:45.801 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:45.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.801 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:57 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'1, mlcod=0'0)", 2024-09-16T03:39:45.801 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 153.67410710600001, 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20372090900000001, 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.802 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125313+0000", 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7180000000000001e-06 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126730+0000", 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00141731 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.803 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126746+0000", 2024-09-16T03:39:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6455000000000001e-05 2024-09-16T03:39:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126756+0000", 2024-09-16T03:39:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:39:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227068+0000", 2024-09-16T03:39:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100312308 2024-09-16T03:39:45.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227085+0000", 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6898e-05 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328995+0000", 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101909903 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329031+0000", 2024-09-16T03:39:45.805 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5717999999999999e-05 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:58 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 153.67077397599999, 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084524500000001, 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:39:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:39:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128644+0000", 2024-09-16T03:39:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.251e-06 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128913+0000", 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026856200000000002 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128925+0000", 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2109999999999999e-05 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128933+0000", 2024-09-16T03:39:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6260000000000003e-06 2024-09-16T03:39:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328196+0000", 2024-09-16T03:39:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19926303200000001 2024-09-16T03:39:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328253+0000", 2024-09-16T03:39:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7262999999999997e-05 2024-09-16T03:39:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329458+0000", 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012051939999999999 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329488+0000", 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0207000000000001e-05 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.810 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:59 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'3, mlcod=0'0)", 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 153.67027854299999, 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036954900000001, 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.811 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129139+0000", 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6099999999999999e-07 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129158+0000", 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088e-05 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129168+0000", 2024-09-16T03:39:45.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.048e-05 2024-09-16T03:39:45.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.813 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.229233+0000", 2024-09-16T03:39:45.813 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100065168 2024-09-16T03:39:45.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.813 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328815+0000", 2024-09-16T03:39:45.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099581694999999998 2024-09-16T03:39:45.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328845+0000", 2024-09-16T03:39:45.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0193e-05 2024-09-16T03:39:45.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.814 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.814 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329492+0000", 2024-09-16T03:39:45.814 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064671800000000001 2024-09-16T03:39:45.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329508+0000", 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6246e-05 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.0:15 4.d e47/32 4:b3fe05b2:::smithi15736696-15:head v 47'5, mlcod=39'3)", 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 139.496983817, 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100920751, 2024-09-16T03:39:45.815 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.816 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.816 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.816 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:39:45.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.816 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.816 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:39:45.816 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302435+0000", 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2340000000000001e-06 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302446+0000", 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0596e-05 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302461+0000", 2024-09-16T03:39:45.817 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5082000000000001e-05 2024-09-16T03:39:45.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402592+0000", 2024-09-16T03:39:45.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100130488 2024-09-16T03:39:45.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.818 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402712+0000", 2024-09-16T03:39:45.818 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012013100000000001 2024-09-16T03:39:45.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.818 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402741+0000", 2024-09-16T03:39:45.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9717999999999999e-05 2024-09-16T03:39:45.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403309+0000", 2024-09-16T03:39:45.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056725700000000005 2024-09-16T03:39:45.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.819 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.819 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403354+0000", 2024-09-16T03:39:45.819 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5244999999999998e-05 2024-09-16T03:39:45.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.820 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.820 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:39:45.820 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:39:45.820 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 115.64948510799999, 2024-09-16T03:39:45.820 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032593800000000001, 2024-09-16T03:39:45.820 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.820 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.820 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.820 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149943+0000", 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0817e-05 2024-09-16T03:39:45.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151026+0000", 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010835350000000001 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151040+0000", 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3884e-05 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151049+0000", 2024-09-16T03:39:45.822 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151107+0000", 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8199000000000003e-05 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151124+0000", 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7756000000000001e-05 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.823 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153142+0000", 2024-09-16T03:39:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020177569999999998 2024-09-16T03:39:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153191+0000", 2024-09-16T03:39:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9051999999999997e-05 2024-09-16T03:39:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1682 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'7, mlcod=40'4)", 2024-09-16T03:39:45.824 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 115.54588791800001, 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10169502499999999, 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.825 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:39:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253540+0000", 2024-09-16T03:39:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0888e-05 2024-09-16T03:39:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254015+0000", 2024-09-16T03:39:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047485499999999999 2024-09-16T03:39:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254030+0000", 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4899000000000001e-05 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254039+0000", 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5000000000000005e-06 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254107+0000", 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7298000000000005e-05 2024-09-16T03:39:45.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254122+0000", 2024-09-16T03:39:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5928999999999999e-05 2024-09-16T03:39:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355184+0000", 2024-09-16T03:39:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101061703 2024-09-16T03:39:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.828 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355224+0000", 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9953000000000002e-05 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1683 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'8, mlcod=40'4)", 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 115.54535036999999, 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101630885, 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.829 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:39:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:39:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.830 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254068+0000", 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5700000000000001e-07 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254075+0000", 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2710000000000003e-06 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254083+0000", 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0220000000000004e-06 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.831 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254089+0000", 2024-09-16T03:39:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8699999999999997e-06 2024-09-16T03:39:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354968+0000", 2024-09-16T03:39:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100879625 2024-09-16T03:39:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354992+0000", 2024-09-16T03:39:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-16T03:39:45.832 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355665+0000", 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00067371100000000001 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355698+0000", 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2291e-05 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.0:177 4.10 e65/57 4:0a55bbf2:::smithi15736696-18:head v 65'47, mlcod=0'0)", 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 114.59240378200001, 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074461099999999995, 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5300000000000001e-07 2024-09-16T03:39:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207018+0000", 2024-09-16T03:39:45.835 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7210000000000002e-06 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207031+0000", 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2767e-05 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207040+0000", 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5400000000000001e-06 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.836 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207107+0000", 2024-09-16T03:39:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6625999999999995e-05 2024-09-16T03:39:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207131+0000", 2024-09-16T03:39:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4128e-05 2024-09-16T03:39:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207712+0000", 2024-09-16T03:39:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00058124700000000001 2024-09-16T03:39:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207758+0000", 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5429000000000002e-05 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1723 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'48, mlcod=53'46)", 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 114.15033660100001, 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00088284599999999996, 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.838 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.839 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649081+0000", 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8299999999999995e-07 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649101+0000", 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0126e-05 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649141+0000", 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9609e-05 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649161+0000", 2024-09-16T03:39:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9599999999999999e-05 2024-09-16T03:39:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649299+0000", 2024-09-16T03:39:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013822199999999999 2024-09-16T03:39:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649318+0000", 2024-09-16T03:39:45.841 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9273999999999999e-05 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649919+0000", 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060074100000000001 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649963+0000", 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4391000000000002e-05 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1724 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'49, mlcod=53'46)", 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 114.14667163599999, 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064536300000000003, 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:39:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652747+0000", 2024-09-16T03:39:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.291e-06 2024-09-16T03:39:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.844 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652753+0000", 2024-09-16T03:39:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4450000000000001e-06 2024-09-16T03:39:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652782+0000", 2024-09-16T03:39:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9094999999999998e-05 2024-09-16T03:39:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652797+0000", 2024-09-16T03:39:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5021999999999999e-05 2024-09-16T03:39:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652854+0000", 2024-09-16T03:39:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6586999999999999e-05 2024-09-16T03:39:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652885+0000", 2024-09-16T03:39:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.146e-05 2024-09-16T03:39:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653350+0000", 2024-09-16T03:39:45.846 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046443599999999998 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653391+0000", 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1026999999999999e-05 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1725 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'50, mlcod=53'46)", 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 114.145878813, 2024-09-16T03:39:45.847 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056324100000000002, 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:45.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653539+0000", 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9699999999999994e-07 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653545+0000", 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9309999999999996e-06 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653557+0000", 2024-09-16T03:39:45.849 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2340999999999999e-05 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653564+0000", 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2779999999999999e-06 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653605+0000", 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1301000000000003e-05 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:45.850 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653621+0000", 2024-09-16T03:39:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5951e-05 2024-09-16T03:39:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654064+0000", 2024-09-16T03:39:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000442843 2024-09-16T03:39:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654101+0000", 2024-09-16T03:39:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7598999999999997e-05 2024-09-16T03:39:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.851 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:45.852 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:45.852 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:45.853 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:39:45.894 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:45.892+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:45.970 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:45.970 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:45.970 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:39:45.970 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:45.976 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:45.974+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:45.980 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:39:46.077 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:46.076+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:46.079 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 9, 1] out_osds: [5, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:39:46.080 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:39:46.080 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:39:46.080 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-16T03:39:46.080 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T03:39:46.080 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T03:39:46.081 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2024-09-16T03:39:46.084 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:46.084 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:46.084 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:46.085 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:46.085 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:46.094 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:39:46.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:39:46.178 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:46.177+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:46.186 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:46.186 INFO:teuthology.orchestra.run.smithi155.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T03:39:46.186 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:46.241 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:46.242 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:46.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:46.242 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:46.242 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.242 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:5 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'1, mlcod=0'0)", 2024-09-16T03:39:46.242 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:39:46.242 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 154.27104905900001, 2024-09-16T03:39:46.242 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.105742802, 2024-09-16T03:39:46.242 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.242 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.242 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.242 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968147+0000", 2024-09-16T03:39:46.243 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4e-07 2024-09-16T03:39:46.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.244 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968752+0000", 2024-09-16T03:39:46.244 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060437900000000005 2024-09-16T03:39:46.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.244 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968762+0000", 2024-09-16T03:39:46.244 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0284e-05 2024-09-16T03:39:46.244 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.244 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.244 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968778+0000", 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6124e-05 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069797+0000", 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10101895700000001 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069833+0000", 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6013000000000002e-05 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.245 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.246 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.246 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073838+0000", 2024-09-16T03:39:46.246 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040049049999999996 2024-09-16T03:39:46.246 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.246 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.246 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.246 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073889+0000", 2024-09-16T03:39:46.246 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.13e-05 2024-09-16T03:39:46.246 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.246 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.246 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.246 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.246 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.247 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:7 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'3, mlcod=0'0)", 2024-09-16T03:39:46.247 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:39:46.247 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 154.26415656, 2024-09-16T03:39:46.247 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20061241399999999, 2024-09-16T03:39:46.247 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.247 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.247 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.248 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.248 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.248 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.248 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:39:46.248 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.248 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.248 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.248 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.249 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:39:46.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.249 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.249 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975042+0000", 2024-09-16T03:39:46.249 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5289999999999998e-06 2024-09-16T03:39:46.249 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.249 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975079+0000", 2024-09-16T03:39:46.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7626999999999997e-05 2024-09-16T03:39:46.250 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.250 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.250 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.250 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975104+0000", 2024-09-16T03:39:46.250 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4958999999999999e-05 2024-09-16T03:39:46.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.251 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.251 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.075211+0000", 2024-09-16T03:39:46.251 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100106814 2024-09-16T03:39:46.251 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.251 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.251 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080573+0000", 2024-09-16T03:39:46.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0053618650000000004 2024-09-16T03:39:46.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.252 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080592+0000", 2024-09-16T03:39:46.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9079000000000001e-05 2024-09-16T03:39:46.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.252 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.252 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.252 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175497+0000", 2024-09-16T03:39:46.252 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.094904832999999994 2024-09-16T03:39:46.252 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.253 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.253 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175651+0000", 2024-09-16T03:39:46.253 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015470800000000001 2024-09-16T03:39:46.253 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.253 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.253 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.253 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.253 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.253 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:44 3.3 e49/31 3:dc284e8d:::smithi15736696-28 low_tier:head v 49'55, mlcod=49'53)", 2024-09-16T03:39:46.253 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:39:46.253 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 136.116232164, 2024-09-16T03:39:46.253 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.149341897, 2024-09-16T03:39:46.254 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.254 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.254 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.254 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.254 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.254 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:39:46.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.254 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.254 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.254 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.254 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:39:46.254 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.255 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122964+0000", 2024-09-16T03:39:46.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.91e-07 2024-09-16T03:39:46.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.255 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122968+0000", 2024-09-16T03:39:46.255 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1470000000000003e-06 2024-09-16T03:39:46.255 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.255 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.255 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.256 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122979+0000", 2024-09-16T03:39:46.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1090000000000001e-05 2024-09-16T03:39:46.256 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.256 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.256 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.256 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.223110+0000", 2024-09-16T03:39:46.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100131357 2024-09-16T03:39:46.256 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.256 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.256 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.256 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271688+0000", 2024-09-16T03:39:46.256 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048578192999999999 2024-09-16T03:39:46.256 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271712+0000", 2024-09-16T03:39:46.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4066e-05 2024-09-16T03:39:46.257 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.257 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272260+0000", 2024-09-16T03:39:46.257 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054737000000000004 2024-09-16T03:39:46.257 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.257 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.257 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.258 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272305+0000", 2024-09-16T03:39:46.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5482999999999998e-05 2024-09-16T03:39:46.258 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.258 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.258 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.258 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.258 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.258 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1007 3.3 e50/31 3:fda66a0d:::920f29fe85d0745d4cd24a887714ac53f1b9a978ec2208ff473c99287fdf8feb:head v 50'57, mlcod=49'55)", 2024-09-16T03:39:46.258 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:39:46.258 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 135.64460561000001, 2024-09-16T03:39:46.258 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20236485200000001, 2024-09-16T03:39:46.258 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.258 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.259 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.259 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.259 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.259 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:39:46.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.259 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.259 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.259 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.259 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:39:46.259 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.259 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.260 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.260 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.260 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594592+0000", 2024-09-16T03:39:46.260 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0700000000000001e-06 2024-09-16T03:39:46.260 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.260 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.260 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.260 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595508+0000", 2024-09-16T03:39:46.260 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00091599499999999996 2024-09-16T03:39:46.260 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.260 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.260 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.260 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595521+0000", 2024-09-16T03:39:46.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:39:46.261 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595531+0000", 2024-09-16T03:39:46.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2499999999999995e-06 2024-09-16T03:39:46.261 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.261 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.261 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795762+0000", 2024-09-16T03:39:46.261 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200230977 2024-09-16T03:39:46.261 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.261 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795778+0000", 2024-09-16T03:39:46.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:39:46.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796912+0000", 2024-09-16T03:39:46.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011338520000000001 2024-09-16T03:39:46.262 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.262 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.262 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.262 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796955+0000", 2024-09-16T03:39:46.262 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2768000000000001e-05 2024-09-16T03:39:46.263 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.263 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.263 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.263 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.263 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.263 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1247 4.11 4:89839b7e:::smithi15736696-12:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-16T03:39:46.263 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:39:46.263 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 129.283618429, 2024-09-16T03:39:46.263 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202294525, 2024-09-16T03:39:46.263 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.263 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.263 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.264 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:46.264 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:46.264 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:46.264 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1247 2024-09-16T03:39:46.264 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.264 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.264 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.264 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.264 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:39:46.264 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.264 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.264 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955579+0000", 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3240000000000001e-06 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955587+0000", 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.266 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955600+0000", 2024-09-16T03:39:46.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2723e-05 2024-09-16T03:39:46.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.266 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955612+0000", 2024-09-16T03:39:46.266 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2422999999999999e-05 2024-09-16T03:39:46.266 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.266 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.266 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055837+0000", 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10022539699999999 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055974+0000", 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000136995 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,7,8,11,12", 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.056092+0000", 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011738 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.267 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057209+0000", 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011170329999999999 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057262+0000", 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2859000000000003e-05 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157594+0000", 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10033241699999999 2024-09-16T03:39:46.268 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.269 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:46.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157648+0000", 2024-09-16T03:39:46.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3863999999999998e-05 2024-09-16T03:39:46.269 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.269 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:46.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157692+0000", 2024-09-16T03:39:46.269 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4406999999999997e-05 2024-09-16T03:39:46.269 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.269 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.269 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:46.269 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157732+0000", 2024-09-16T03:39:46.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9156000000000003e-05 2024-09-16T03:39:46.270 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.270 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.270 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157780+0000", 2024-09-16T03:39:46.270 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8801999999999998e-05 2024-09-16T03:39:46.270 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.270 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.270 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.271 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157872+0000", 2024-09-16T03:39:46.271 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1311999999999996e-05 2024-09-16T03:39:46.271 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.271 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.271 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.271 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.271 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.271 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1518 3.3 e59/31 3:df1be05b:::e06092e68680905b5745d5057e93f6c7581518aed07ca5db2802546e9c29aeef:head v 59'95, mlcod=58'93)", 2024-09-16T03:39:46.271 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:39:46.271 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 120.981174756, 2024-09-16T03:39:46.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104523454, 2024-09-16T03:39:46.272 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.272 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.272 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.272 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.272 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.272 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.272 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:39:46.272 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.273 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.273 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.273 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:39:46.273 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.273 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.273 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.273 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.274 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258022+0000", 2024-09-16T03:39:46.274 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.372e-06 2024-09-16T03:39:46.274 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.274 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.274 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.274 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258462+0000", 2024-09-16T03:39:46.274 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00043970100000000002 2024-09-16T03:39:46.274 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.274 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.274 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.275 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258476+0000", 2024-09-16T03:39:46.275 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.376e-05 2024-09-16T03:39:46.275 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.275 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.275 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.275 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258483+0000", 2024-09-16T03:39:46.275 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.294e-06 2024-09-16T03:39:46.275 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.275 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.275 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.275 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358679+0000", 2024-09-16T03:39:46.275 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100196177 2024-09-16T03:39:46.275 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358718+0000", 2024-09-16T03:39:46.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9295999999999998e-05 2024-09-16T03:39:46.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.276 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362496+0000", 2024-09-16T03:39:46.276 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037772069999999999 2024-09-16T03:39:46.276 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.276 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.276 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362544+0000", 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8646999999999997e-05 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1923 4.4 e73/50 4:239dbd85:::smithi15736696-2:head v 73'19, mlcod=62'17)", 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 106.275313934, 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104144848, 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.277 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963882+0000", 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.963884+0000", 2024-09-16T03:39:46.278 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964597+0000", 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071314700000000002 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964615+0000", 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8209000000000001e-05 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.279 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:59.964626+0000", 2024-09-16T03:39:46.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0511e-05 2024-09-16T03:39:46.280 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.280 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.280 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.280 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064779+0000", 2024-09-16T03:39:46.280 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100153495 2024-09-16T03:39:46.281 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.281 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.281 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.281 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.064794+0000", 2024-09-16T03:39:46.281 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4946000000000001e-05 2024-09-16T03:39:46.281 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.281 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.281 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.067999+0000", 2024-09-16T03:39:46.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0032050709999999999 2024-09-16T03:39:46.282 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.282 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.282 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.282 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:00.068026+0000", 2024-09-16T03:39:46.282 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7066000000000001e-05 2024-09-16T03:39:46.283 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.283 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.283 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.283 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.283 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.283 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2445 4.11 4:89839b7e:::smithi15736696-12:head [sparse-read 1818507~599510,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:39:46.283 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:39:46.283 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 90.795032105999994, 2024-09-16T03:39:46.283 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200613392, 2024-09-16T03:39:46.284 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.284 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.284 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:46.284 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:46.284 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:46.284 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:46.284 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2445 2024-09-16T03:39:46.284 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.284 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.284 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:39:46.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:39:46.285 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.285 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.285 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.285 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.285 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444167+0000", 2024-09-16T03:39:46.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6650000000000001e-06 2024-09-16T03:39:46.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444182+0000", 2024-09-16T03:39:46.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4678000000000001e-05 2024-09-16T03:39:46.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.286 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.286 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444207+0000", 2024-09-16T03:39:46.286 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4768000000000001e-05 2024-09-16T03:39:46.286 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.286 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.544359+0000", 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100152804 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644561+0000", 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10020214500000001 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644624+0000", 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2444e-05 2024-09-16T03:39:46.287 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.288 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.288 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.288 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644777+0000", 2024-09-16T03:39:46.288 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000152888 2024-09-16T03:39:46.288 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.288 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.288 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.288 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.288 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.288 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2629 3.3 e89/31 3:e23364e2:::e447860b11e9f9fde68b75cb1d2f204a74952889ec51a2417ef18c1427ff9a70:head v 89'173, mlcod=87'171)", 2024-09-16T03:39:46.288 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:39:46.288 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 85.787289412999996, 2024-09-16T03:39:46.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10848095100000001, 2024-09-16T03:39:46.289 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.289 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.289 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.289 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:39:46.289 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.289 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.289 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.289 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.289 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451907+0000", 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6599999999999996e-07 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456443+0000", 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0045365509999999998 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.290 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456470+0000", 2024-09-16T03:39:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6897e-05 2024-09-16T03:39:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456496+0000", 2024-09-16T03:39:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.618e-05 2024-09-16T03:39:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.291 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556666+0000", 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100169678 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556686+0000", 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9740000000000001e-05 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560345+0000", 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003658821 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560387+0000", 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2518000000000002e-05 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:209 3.3 e99/31 3:f2e0c7d0:::6bf49184c86dbc717b1f919afcf6cb40a4ca9c2465eb100f7ba59687540c40ed:head v 99'201, mlcod=98'194)", 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:39:46.293 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 74.033159216000001, 2024-09-16T03:39:46.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200559603, 2024-09-16T03:39:46.294 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.294 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.294 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.294 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.294 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:39:46.294 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.294 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.294 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.294 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.294 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:39:46.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206037+0000", 2024-09-16T03:39:46.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9999999999999999e-07 2024-09-16T03:39:46.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.295 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.295 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206039+0000", 2024-09-16T03:39:46.295 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2579999999999998e-06 2024-09-16T03:39:46.295 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.295 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206041+0000", 2024-09-16T03:39:46.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.542e-06 2024-09-16T03:39:46.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306162+0000", 2024-09-16T03:39:46.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100120348 2024-09-16T03:39:46.296 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.296 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.296 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.296 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306250+0000", 2024-09-16T03:39:46.296 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306268+0000", 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8127000000000001e-05 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.406549+0000", 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10028121199999999 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.297 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.406596+0000", 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6700999999999999e-05 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3274 4.11 4:89839b7e:::smithi15736696-12:head [write 1813152~490434] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 65.041396046000003, 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.28154463400000002, 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.298 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3274 2024-09-16T03:39:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:39:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.299 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197801+0000", 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9539999999999998e-06 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.198205+0000", 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00040328199999999998 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.300 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.198218+0000", 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3176000000000001e-05 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298318+0000", 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100100562 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298419+0000", 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100667 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298489+0000", 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0023000000000004e-05 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,12", 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298583+0000", 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3634000000000005e-05 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300283+0000", 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00170049 2024-09-16T03:39:46.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.307072+0000", 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0067889209999999998 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479243+0000", 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.17217031799999999 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479300+0000", 2024-09-16T03:39:46.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7837999999999997e-05 2024-09-16T03:39:46.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479344+0000", 2024-09-16T03:39:46.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3769000000000001e-05 2024-09-16T03:39:46.304 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.304 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.304 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.304 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3275 4.11 4:89839b7e:::smithi15736696-12:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:39:46.304 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:39:46.304 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 64.940850632999997, 2024-09-16T03:39:46.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.276974636, 2024-09-16T03:39:46.305 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.305 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.305 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.305 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:46.305 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:46.305 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:46.305 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3275 2024-09-16T03:39:46.305 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.305 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298364+0000", 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9273e-05 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298373+0000", 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5809999999999997e-06 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298382+0000", 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6269999999999995e-06 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298390+0000", 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2259999999999999e-06 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299770+0000", 2024-09-16T03:39:46.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013803159999999999 2024-09-16T03:39:46.308 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299852+0000", 2024-09-16T03:39:46.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.174e-05 2024-09-16T03:39:46.308 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,12", 2024-09-16T03:39:46.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299948+0000", 2024-09-16T03:39:46.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6669000000000006e-05 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300496+0000", 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054762799999999998 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.401930+0000", 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101434495 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:46.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575189+0000", 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.173258996 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575247+0000", 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7651000000000001e-05 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575320+0000", 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2434000000000005e-05 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.310 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3276 4.11 4:89839b7e:::smithi15736696-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 64.940781821000002, 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.37724456499999998, 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3276 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.311 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:39:46.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:39:46.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298415+0000", 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298418+0000", 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.766e-06 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298426+0000", 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1359999999999999e-06 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298433+0000", 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0899999999999999e-06 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300168+0000", 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017353869999999999 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300215+0000", 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6702000000000001e-05 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675524+0000", 2024-09-16T03:39:46.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.375309585 2024-09-16T03:39:46.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675596+0000", 2024-09-16T03:39:46.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1921000000000005e-05 2024-09-16T03:39:46.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675658+0000", 2024-09-16T03:39:46.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2052999999999997e-05 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3985 4.36 e130/126 4:6ef3b53d:::smithi15736696-48:head v 130'88, mlcod=124'86)", 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.376369+0000", 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 40.862826140000003, 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20392002300000001, 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376369+0000", 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376369+0000", 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376370+0000", 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1299999999999999e-07 2024-09-16T03:39:46.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376877+0000", 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050726199999999997 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376888+0000", 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0798999999999999e-05 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376896+0000", 2024-09-16T03:39:46.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1380000000000007e-06 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376950+0000", 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4580000000000003e-05 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376970+0000", 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9607e-05 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.319 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580253+0000", 2024-09-16T03:39:46.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20328323600000001 2024-09-16T03:39:46.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.320 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.320 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580289+0000", 2024-09-16T03:39:46.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6188e-05 2024-09-16T03:39:46.320 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.320 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.320 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.320 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3986 4.36 e130/126 4:6ef3b53d:::smithi15736696-48:head v 130'89, mlcod=124'86)", 2024-09-16T03:39:46.320 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.377664+0000", 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 40.861531409999998, 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264148100000001, 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.377664+0000", 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.377664+0000", 2024-09-16T03:39:46.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.377665+0000", 2024-09-16T03:39:46.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.540000000000001e-07 2024-09-16T03:39:46.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378852+0000", 2024-09-16T03:39:46.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001187097 2024-09-16T03:39:46.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378863+0000", 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0721e-05 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378868+0000", 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5650000000000001e-06 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378913+0000", 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4867999999999998e-05 2024-09-16T03:39:46.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378929+0000", 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5752000000000001e-05 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580291+0000", 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20136209599999999 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580306+0000", 2024-09-16T03:39:46.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4428e-05 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3987 4.36 e130/126 4:6ef3b53d:::smithi15736696-48:head v 130'90, mlcod=124'86)", 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.379666+0000", 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 40.859529948000002, 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20094095300000001, 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379666+0000", 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379666+0000", 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379667+0000", 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.11e-06 2024-09-16T03:39:46.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379675+0000", 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4489999999999999e-06 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379683+0000", 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0660000000000004e-06 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379690+0000", 2024-09-16T03:39:46.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4010000000000002e-06 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.579898+0000", 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20020810999999999 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.579914+0000", 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6198e-05 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580572+0000", 2024-09-16T03:39:46.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065845699999999999 2024-09-16T03:39:46.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.329 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580607+0000", 2024-09-16T03:39:46.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4162000000000003e-05 2024-09-16T03:39:46.329 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.329 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.329 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.329 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:4043 4.36 e131/126 4:6ece7830:::smithi15736696-7:head v 131'93, mlcod=130'89)", 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.055626+0000", 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 40.183569167000002, 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19612945100000001, 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055626+0000", 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055626+0000", 2024-09-16T03:39:46.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.331 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.331 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055627+0000", 2024-09-16T03:39:46.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0869999999999999e-06 2024-09-16T03:39:46.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.331 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.331 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055633+0000", 2024-09-16T03:39:46.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.101e-06 2024-09-16T03:39:46.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.332 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.332 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055641+0000", 2024-09-16T03:39:46.332 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9759999999999995e-06 2024-09-16T03:39:46.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.332 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.332 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055646+0000", 2024-09-16T03:39:46.332 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4360000000000001e-06 2024-09-16T03:39:46.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.333 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.333 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251121+0000", 2024-09-16T03:39:46.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.195475068 2024-09-16T03:39:46.333 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.333 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.333 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251137+0000", 2024-09-16T03:39:46.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6271999999999999e-05 2024-09-16T03:39:46.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:46.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251719+0000", 2024-09-16T03:39:46.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058167799999999997 2024-09-16T03:39:46.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251756+0000", 2024-09-16T03:39:46.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6832999999999998e-05 2024-09-16T03:39:46.335 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.335 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.336 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.336 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "replica_scrub(pg: 4.12,from:0'0,to:127'20,epoch:139/138,start:4:48000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:39:46.336 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:14.698983+0000", 2024-09-16T03:39:46.336 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 31.54021212, 2024-09-16T03:39:46.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12536847100000001, 2024-09-16T03:39:46.337 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.337 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.337 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:46.337 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.337 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.337 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.698983+0000", 2024-09-16T03:39:46.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.698983+0000", 2024-09-16T03:39:46.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.698985+0000", 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4899999999999999e-06 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.698991+0000", 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9440000000000002e-06 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.699003+0000", 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1708999999999999e-05 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.699009+0000", 2024-09-16T03:39:46.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4849999999999997e-06 2024-09-16T03:39:46.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.824228+0000", 2024-09-16T03:39:46.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12521855800000001 2024-09-16T03:39:46.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.824262+0000", 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4513e-05 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.824352+0000", 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9771999999999994e-05 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDPGPush(4.28 160/158 [PushOp(4:148e627d:::smithi15736696-23:head, version: 129'68, data_included: [658874~799064,1976979~249512], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:148e627d:::smithi15736696-23:head@129'68, size: 3899224, copy_subset: [(0, 3899224)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2226491, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:39.703233+0000", 2024-09-16T03:39:46.341 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.5359621700000003, 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104655188, 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.703233+0000", 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.703233+0000", 2024-09-16T03:39:46.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.703236+0000", 2024-09-16T03:39:46.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.61e-06 2024-09-16T03:39:46.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.705967+0000", 2024-09-16T03:39:46.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027309679999999999 2024-09-16T03:39:46.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.705989+0000", 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2200000000000001e-05 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.706002+0000", 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3010000000000001e-05 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.806175+0000", 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100172785 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.806191+0000", 2024-09-16T03:39:46.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5865000000000001e-05 2024-09-16T03:39:46.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.807889+0000", 2024-09-16T03:39:46.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00169775 2024-09-16T03:39:46.345 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.345 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.345 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.346 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "pg_backfill(progress 4.28 e 160/160 lb 4:146f84f0:::smithi15736696-6:head)", 2024-09-16T03:39:46.346 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:39.706023+0000", 2024-09-16T03:39:46.346 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.533172682, 2024-09-16T03:39:46.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20046278000000001, 2024-09-16T03:39:46.346 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:46.346 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:46.346 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:46.346 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:46.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:46.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.706023+0000", 2024-09-16T03:39:46.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:46.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.706023+0000", 2024-09-16T03:39:46.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:46.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:46.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.706023+0000", 2024-09-16T03:39:46.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2099999999999996e-07 2024-09-16T03:39:46.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:46.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.706029+0000", 2024-09-16T03:39:46.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2399999999999998e-06 2024-09-16T03:39:46.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:46.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.706033+0000", 2024-09-16T03:39:46.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2470000000000002e-06 2024-09-16T03:39:46.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:46.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.806100+0000", 2024-09-16T03:39:46.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10006743 2024-09-16T03:39:46.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:46.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.906349+0000", 2024-09-16T03:39:46.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100248381 2024-09-16T03:39:46.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:46.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.906370+0000", 2024-09-16T03:39:46.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1455999999999999e-05 2024-09-16T03:39:46.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:46.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:46.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:46.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.906486+0000", 2024-09-16T03:39:46.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000115405 2024-09-16T03:39:46.350 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.350 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.350 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.350 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:46.350 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:46.350 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:46.351 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:46.352 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:46.352 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:46.353 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:46.353 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:39:46.353 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:46.354 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:46.354 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:46.354 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:46.355 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:46.355 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:46.355 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:46.356 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:46.356 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:46.349+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:46.365 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:46.366 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:46.370 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:39:46.384 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:46.384 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:46.400 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:46.400 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:46.418 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:46.418 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:46.434 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:46.434 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:46.450 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:46.451 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:46.452 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:46.451+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:46.461 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:46.461 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:46.466 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:46.467 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:46.470 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:39:46.484 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:46.484 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:46.501 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:46.501 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:46.518 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:46.518 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:46.571 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:46.569+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:46.613 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:46.613 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:46.622 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:39:46.653 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:46.652+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:46.702 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:46.700+0000 7fd101e6f640 -1 osd.2 163 handle_osd_map injecting map crc failure 2024-09-16T03:39:46.726 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:46.727 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:46.735 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:39:46.755 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:46.753+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:46.838 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:46.838 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:46.847 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:39:46.891 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:46.890+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:46.930 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:46.929+0000 7f3a1f71e640 -1 osd.6 163 handle_osd_map injecting map crc failure 2024-09-16T03:39:46.930 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:46.929+0000 7efeef6d5640 -1 osd.13 163 handle_osd_map injecting map crc failure 2024-09-16T03:39:46.951 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:46.952 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:46.956 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:46.954+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:46.960 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:39:47.033 INFO:tasks.rados.rados.0.smithi157.stdout:1962: set_chunk oid 11 offset: 1035264 length: 1317888 target oid e0ef79a5931af77a89e00e45b270ee95266718455ec934bcc09140d5d71025ab offset: 0 snap -1 2024-09-16T03:39:47.033 INFO:tasks.rados.rados.0.smithi157.stdout:1963: tier_flush oid 45 2024-09-16T03:39:47.033 INFO:tasks.rados.rados.0.smithi157.stdout:1963: tier_flush oid 45 snap -1 2024-09-16T03:39:47.033 INFO:tasks.rados.rados.0.smithi157.stdout:1953: finishing set_chunk to oid 24 2024-09-16T03:39:47.033 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 24 set_chunk returned error code -95 offset: 846336 length: 1106432 2024-09-16T03:39:47.033 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 0 (ObjNum 333 snap 0 seq_num 333) dirty exists 2024-09-16T03:39:47.033 INFO:tasks.rados.rados.0.smithi157.stdout:1954: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-16T03:39:47.057 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:47.054+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:47.106 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:47.106 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:47.114 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:39:47.131 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1323132 2024-09-16T03:39:47.158 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:47.157+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:47.163 INFO:tasks.rados.rados.0.smithi157.stdout:1952: finishing set_chunk to oid 50 2024-09-16T03:39:47.163 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 50 v 143 (ObjNum 355 snap 0 seq_num 355) dirty exists 2024-09-16T03:39:47.164 INFO:tasks.rados.rados.0.smithi157.stdout:1956: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-16T03:39:47.219 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:47.219 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:47.228 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:39:47.232 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:47.230+0000 7f8be56cd640 -1 osd.7 163 handle_osd_map injecting map crc failure 2024-09-16T03:39:47.235 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:47.259 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:47.257+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:47.272 INFO:tasks.rados.rados.0.smithi157.stdout:1957: finishing set_chunk to oid 12 2024-09-16T03:39:47.272 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 21 (ObjNum 430 snap 0 seq_num 430) dirty exists 2024-09-16T03:39:47.273 INFO:tasks.rados.rados.0.smithi157.stdout:1955: finishing set_chunk to oid 25 2024-09-16T03:39:47.273 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 151 (ObjNum 412 snap 0 seq_num 412) dirty exists 2024-09-16T03:39:47.273 INFO:tasks.rados.rados.0.smithi157.stdout:1958: finishing set_chunk to oid 26 2024-09-16T03:39:47.273 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 26 set_chunk returned error code -95 offset: 82432 length: 2217984 2024-09-16T03:39:47.273 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 0 (ObjNum 379 snap 0 seq_num 379) dirty exists 2024-09-16T03:39:47.273 INFO:tasks.rados.rados.0.smithi157.stdout:1952: done (11 left) 2024-09-16T03:39:47.273 INFO:tasks.rados.rados.0.smithi157.stdout:1953: done (10 left) 2024-09-16T03:39:47.273 INFO:tasks.rados.rados.0.smithi157.stdout:1954: done (9 left) 2024-09-16T03:39:47.273 INFO:tasks.rados.rados.0.smithi157.stdout:1955: done (8 left) 2024-09-16T03:39:47.273 INFO:tasks.rados.rados.0.smithi157.stdout:1956: done (7 left) 2024-09-16T03:39:47.274 INFO:tasks.rados.rados.0.smithi157.stdout:1957: done (6 left) 2024-09-16T03:39:47.274 INFO:tasks.rados.rados.0.smithi157.stdout:1958: done (5 left) 2024-09-16T03:39:47.274 INFO:tasks.rados.rados.0.smithi157.stdout:1964: read oid 49 snap -1 2024-09-16T03:39:47.274 INFO:tasks.rados.rados.0.smithi157.stdout:1964: expect (ObjNum 421 snap 0 seq_num 421) 2024-09-16T03:39:47.274 INFO:tasks.rados.rados.0.smithi157.stdout:1965: tier_evict oid 32 2024-09-16T03:39:47.274 INFO:tasks.rados.rados.0.smithi157.stdout:1965: tier_evict oid 32 snap -1 2024-09-16T03:39:47.274 INFO:tasks.rados.rados.0.smithi157.stdout:1960: expect (ObjNum 436 snap 0 seq_num 436) 2024-09-16T03:39:47.359 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:47.358+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:47.372 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:47.372 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:47.381 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:39:47.393 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2946965 2024-09-16T03:39:47.396 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:47.461 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:47.458+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:47.466 INFO:tasks.rados.rados.0.smithi157.stdout:1961: finishing write tid 1 to smithi15736696-30 2024-09-16T03:39:47.466 INFO:tasks.rados.rados.0.smithi157.stdout:1961: finishing write tid 2 to smithi15736696-30 2024-09-16T03:39:47.466 INFO:tasks.rados.rados.0.smithi157.stdout:1961: finishing write tid 3 to smithi15736696-30 2024-09-16T03:39:47.467 INFO:tasks.rados.rados.0.smithi157.stdout:1961: finishing write tid 4 to smithi15736696-30 2024-09-16T03:39:47.467 INFO:tasks.rados.rados.0.smithi157.stdout:1959: finishing set_chunk to oid 9 2024-09-16T03:39:47.467 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 9 v 92 (ObjNum 432 snap 0 seq_num 432) dirty exists 2024-09-16T03:39:47.467 INFO:tasks.rados.rados.0.smithi157.stdout:1961: finishing write tid 5 to smithi15736696-30 2024-09-16T03:39:47.467 INFO:tasks.rados.rados.0.smithi157.stdout:1961: oid 30 updating version 0 to 52 2024-09-16T03:39:47.467 INFO:tasks.rados.rados.0.smithi157.stdout:1961: oid 30 updating version 52 to 54 2024-09-16T03:39:47.467 INFO:tasks.rados.rados.0.smithi157.stdout:1961: oid 30 updating version 54 to 55 2024-09-16T03:39:47.467 INFO:tasks.rados.rados.0.smithi157.stdout:1961: oid 30 version 55 is already newer than 53 2024-09-16T03:39:47.467 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 30 v 55 (ObjNum 437 snap 0 seq_num 437) dirty exists 2024-09-16T03:39:47.467 INFO:tasks.rados.rados.0.smithi157.stdout:1961: left oid 30 (ObjNum 437 snap 0 seq_num 437) 2024-09-16T03:39:47.468 INFO:tasks.rados.rados.0.smithi157.stdout:1964: expect (ObjNum 421 snap 0 seq_num 421) 2024-09-16T03:39:47.487 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:47.487 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:47.496 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:39:47.561 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:47.559+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:47.601 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:47.601 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:47.610 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:39:47.616 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 877716 2024-09-16T03:39:47.663 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:47.661+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:47.675 INFO:tasks.rados.rados.0.smithi157.stdout:1965: got (22) Invalid argument 2024-09-16T03:39:47.675 INFO:tasks.rados.rados.0.smithi157.stdout:1962: finishing set_chunk to oid 11 2024-09-16T03:39:47.675 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 78 (ObjNum 428 snap 0 seq_num 428) dirty exists 2024-09-16T03:39:47.675 INFO:tasks.rados.rados.0.smithi157.stdout:1959: done (6 left) 2024-09-16T03:39:47.675 INFO:tasks.rados.rados.0.smithi157.stdout:1960: done (5 left) 2024-09-16T03:39:47.675 INFO:tasks.rados.rados.0.smithi157.stdout:1961: done (4 left) 2024-09-16T03:39:47.675 INFO:tasks.rados.rados.0.smithi157.stdout:1962: done (3 left) 2024-09-16T03:39:47.675 INFO:tasks.rados.rados.0.smithi157.stdout:1964: done (2 left) 2024-09-16T03:39:47.675 INFO:tasks.rados.rados.0.smithi157.stdout:1965: done (1 left) 2024-09-16T03:39:47.676 INFO:tasks.rados.rados.0.smithi157.stdout:1966: read oid 50 snap -1 2024-09-16T03:39:47.676 INFO:tasks.rados.rados.0.smithi157.stdout:1966: expect (ObjNum 355 snap 0 seq_num 355) 2024-09-16T03:39:47.676 INFO:tasks.rados.rados.0.smithi157.stdout:1967: write (excl) oid 17 current snap is 0 2024-09-16T03:39:47.676 INFO:tasks.rados.rados.0.smithi157.stdout:1967: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2024-09-16T03:39:47.683 INFO:tasks.rados.rados.0.smithi157.stdout:1967: writing smithi15736696-17 from 561466 to 1352649 tid 1 2024-09-16T03:39:47.691 INFO:tasks.rados.rados.0.smithi157.stdout:1967: writing smithi15736696-17 from 2081113 to 2673917 tid 2 2024-09-16T03:39:47.696 INFO:tasks.rados.rados.0.smithi157.stdout:1967: writing smithi15736696-17 from 3095506 to 3361466 tid 3 2024-09-16T03:39:47.697 INFO:tasks.rados.rados.0.smithi157.stdout:1968: read oid 23 snap -1 2024-09-16T03:39:47.697 INFO:tasks.rados.rados.0.smithi157.stdout:1968: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:39:47.697 INFO:tasks.rados.rados.0.smithi157.stdout:1969: read oid 37 snap -1 2024-09-16T03:39:47.697 INFO:tasks.rados.rados.0.smithi157.stdout:1969: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-16T03:39:47.697 INFO:tasks.rados.rados.0.smithi157.stdout:1970: read oid 14 snap -1 2024-09-16T03:39:47.697 INFO:tasks.rados.rados.0.smithi157.stdout:1970: expect (ObjNum 398 snap 0 seq_num 398) 2024-09-16T03:39:47.697 INFO:tasks.rados.rados.0.smithi157.stdout:1971: read oid 10 snap -1 2024-09-16T03:39:47.697 INFO:tasks.rados.rados.0.smithi157.stdout:1971: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-16T03:39:47.697 INFO:tasks.rados.rados.0.smithi157.stdout:1972: delete oid 40 current snap is 0 2024-09-16T03:39:47.698 INFO:tasks.rados.rados.0.smithi157.stdout:1971: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-16T03:39:47.710 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1103718 2024-09-16T03:39:47.731 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:47.731 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:47.740 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:39:47.756 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1048442 2024-09-16T03:39:47.763 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:47.761+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:47.840 INFO:tasks.rados.rados.0.smithi157.stdout:1969: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-16T03:39:47.845 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:47.845 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:47.854 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:39:47.888 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2354472 2024-09-16T03:39:47.888 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:47.885+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:47.944 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:47.965 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:47.964+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:47.991 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:47.991 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:48.000 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:39:48.063 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:48.065 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:48.064+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:48.105 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:48.106 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:48.114 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:39:48.167 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:48.164+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:48.182 INFO:tasks.rados.rados.0.smithi157.stdout:1968: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:39:48.217 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:48.217 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:48.225 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:39:48.268 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:48.265+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:48.369 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:48.369 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:48.369 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:48.367+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:48.377 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:39:48.385 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:39:48.458 INFO:tasks.rados.rados.0.smithi157.stdout:1963: got (0) Success 2024-09-16T03:39:48.458 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 45 v 0 (ObjNum 325 snap 0 seq_num 325) dirty exists 2024-09-16T03:39:48.458 INFO:tasks.rados.rados.0.smithi157.stdout:1967: finishing write tid 1 to smithi15736696-17 2024-09-16T03:39:48.458 INFO:tasks.rados.rados.0.smithi157.stdout:1970: expect (ObjNum 398 snap 0 seq_num 398) 2024-09-16T03:39:48.469 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:48.468+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:48.470 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1072966 2024-09-16T03:39:48.481 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:48.481 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:48.490 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:39:48.523 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2394073 2024-09-16T03:39:48.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:39:48.531 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:48.574 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:48.573+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:48.576 INFO:tasks.rados.rados.0.smithi157.stdout:1967: finishing write tid 2 to smithi15736696-17 2024-09-16T03:39:48.576 INFO:tasks.rados.rados.0.smithi157.stdout:1967: finishing write tid 3 to smithi15736696-17 2024-09-16T03:39:48.576 INFO:tasks.rados.rados.0.smithi157.stdout:1967: finishing write tid 4 to smithi15736696-17 2024-09-16T03:39:48.576 INFO:tasks.rados.rados.0.smithi157.stdout:1966: expect (ObjNum 355 snap 0 seq_num 355) 2024-09-16T03:39:48.593 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:48.593 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:48.602 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:39:48.656 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:48.657 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:48.671 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:48.670+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:48.672 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3866354 2024-09-16T03:39:48.674 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:48.674 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:48.691 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:48.691 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:48.708 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:48.708 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:48.724 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:48.725 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:48.741 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:48.741 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:48.746 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:48.746 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:48.754 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:39:48.758 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:48.758 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:48.771 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:48.770+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:48.775 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:48.775 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:48.783 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:39:48.784 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:48.782+0000 7f0ae754b640 -1 osd.0 164 handle_osd_map injecting map crc failure 2024-09-16T03:39:48.794 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:48.795 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:48.812 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:48.812 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:48.829 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:48.829 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:48.845 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:48.846 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:48.861 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:48.861 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:48.872 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:48.869+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:48.878 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:48.878 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:48.886 INFO:tasks.rados.rados.0.smithi157.stdout:1967: finishing write tid 5 to smithi15736696-17 2024-09-16T03:39:48.886 INFO:tasks.rados.rados.0.smithi157.stdout:1967: oid 17 updating version 0 to 150 2024-09-16T03:39:48.886 INFO:tasks.rados.rados.0.smithi157.stdout:1967: oid 17 version 150 is already newer than 149 2024-09-16T03:39:48.886 INFO:tasks.rados.rados.0.smithi157.stdout:1967: oid 17 updating version 150 to 152 2024-09-16T03:39:48.886 INFO:tasks.rados.rados.0.smithi157.stdout:1967: oid 17 version 152 is already newer than 151 2024-09-16T03:39:48.886 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 17 v 152 (ObjNum 438 snap 0 seq_num 438) dirty exists 2024-09-16T03:39:48.886 INFO:tasks.rados.rados.0.smithi157.stdout:1967: left oid 17 (ObjNum 438 snap 0 seq_num 438) 2024-09-16T03:39:48.886 INFO:tasks.rados.rados.0.smithi157.stdout:1963: done (7 left) 2024-09-16T03:39:48.887 INFO:tasks.rados.rados.0.smithi157.stdout:1966: done (6 left) 2024-09-16T03:39:48.887 INFO:tasks.rados.rados.0.smithi157.stdout:1967: done (5 left) 2024-09-16T03:39:48.887 INFO:tasks.rados.rados.0.smithi157.stdout:1968: done (4 left) 2024-09-16T03:39:48.887 INFO:tasks.rados.rados.0.smithi157.stdout:1969: done (3 left) 2024-09-16T03:39:48.887 INFO:tasks.rados.rados.0.smithi157.stdout:1970: done (2 left) 2024-09-16T03:39:48.887 INFO:tasks.rados.rados.0.smithi157.stdout:1971: done (1 left) 2024-09-16T03:39:48.887 INFO:tasks.rados.rados.0.smithi157.stdout:1972: done (0 left) 2024-09-16T03:39:48.887 INFO:tasks.rados.rados.0.smithi157.stdout:1973: write oid 10 current snap is 0 2024-09-16T03:39:48.887 INFO:tasks.rados.rados.0.smithi157.stdout:1973: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-09-16T03:39:48.893 INFO:tasks.rados.rados.0.smithi157.stdout:1973: writing smithi15736696-10 from 595080 to 1136761 tid 1 2024-09-16T03:39:48.895 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:48.896 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:48.901 INFO:tasks.rados.rados.0.smithi157.stdout:1973: writing smithi15736696-10 from 1798077 to 2394550 tid 2 2024-09-16T03:39:48.907 INFO:tasks.rados.rados.0.smithi157.stdout:1973: writing smithi15736696-10 from 3150352 to 3395080 tid 3 2024-09-16T03:39:48.907 INFO:tasks.rados.rados.0.smithi157.stdout:1974: read oid 34 snap -1 2024-09-16T03:39:48.908 INFO:tasks.rados.rados.0.smithi157.stdout:1974: expect (ObjNum 342 snap 0 seq_num 342) 2024-09-16T03:39:48.908 INFO:tasks.rados.rados.0.smithi157.stdout:1975: write oid 18 current snap is 0 2024-09-16T03:39:48.908 INFO:tasks.rados.rados.0.smithi157.stdout:1975: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2024-09-16T03:39:48.910 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:48.910 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:48.913 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:48.913 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:48.916 INFO:tasks.rados.rados.0.smithi157.stdout:1975: writing smithi15736696-18 from 611887 to 1345170 tid 1 2024-09-16T03:39:48.923 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:39:48.923 INFO:tasks.rados.rados.0.smithi157.stdout:1975: writing smithi15736696-18 from 1972912 to 2429396 tid 2 2024-09-16T03:39:48.928 INFO:tasks.rados.rados.0.smithi157.stdout:1975: writing smithi15736696-18 from 2894128 to 3411887 tid 3 2024-09-16T03:39:48.928 INFO:tasks.rados.rados.0.smithi157.stdout:1976: read oid 37 snap -1 2024-09-16T03:39:48.928 INFO:tasks.rados.rados.0.smithi157.stdout:1976: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-16T03:39:48.928 INFO:tasks.rados.rados.0.smithi157.stdout:1977: set_chunk oid 31 target oid 2024-09-16T03:39:48.973 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:48.972+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:49.036 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:465 4.5 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:39:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 148.01682840399999, 2024-09-16T03:39:49.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60529649299999999, 2024-09-16T03:39:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 465 2024-09-16T03:39:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015706+0000", 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.091e-06 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015712+0000", 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015724+0000", 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1408e-05 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015731+0000", 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216049+0000", 2024-09-16T03:39:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200317895 2024-09-16T03:39:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216178+0000", 2024-09-16T03:39:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012848800000000001 2024-09-16T03:39:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620010+0000", 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40383207199999999 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620710+0000", 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069988400000000001 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620902+0000", 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019197800000000001 2024-09-16T03:39:49.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620933+0000", 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1513999999999997e-05 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620968+0000", 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4524999999999999e-05 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.621002+0000", 2024-09-16T03:39:49.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3955000000000003e-05 2024-09-16T03:39:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:10 3.3 3:deac805c:::smithi15736696-44 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:39:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 142.625333428, 2024-09-16T03:39:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53191748100000003, 2024-09-16T03:39:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.043 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:39:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:39:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407202+0000", 2024-09-16T03:39:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.855e-06 2024-09-16T03:39:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407208+0000", 2024-09-16T03:39:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9819999999999996e-06 2024-09-16T03:39:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407220+0000", 2024-09-16T03:39:49.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2191999999999999e-05 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407236+0000", 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5698000000000001e-05 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538167+0000", 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.13093111499999999 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538229+0000", 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2441999999999996e-05 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538346+0000", 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011716400000000001 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540757+0000", 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024105089999999999 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.639501+0000", 2024-09-16T03:39:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098743545000000002 2024-09-16T03:39:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939038+0000", 2024-09-16T03:39:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29953765700000001 2024-09-16T03:39:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.048 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939075+0000", 2024-09-16T03:39:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6600999999999997e-05 2024-09-16T03:39:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939118+0000", 2024-09-16T03:39:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2721e-05 2024-09-16T03:39:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.049 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:984 4.15 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:39:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 138.6604135, 2024-09-16T03:39:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.97836159599999994, 2024-09-16T03:39:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 984 2024-09-16T03:39:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.050 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372122+0000", 2024-09-16T03:39:49.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.934e-06 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372129+0000", 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8029999999999999e-06 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372141+0000", 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2006e-05 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472234+0000", 2024-09-16T03:39:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100092985 2024-09-16T03:39:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472318+0000", 2024-09-16T03:39:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4257000000000003e-05 2024-09-16T03:39:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472326+0000", 2024-09-16T03:39:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8960000000000003e-06 2024-09-16T03:39:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350293+0000", 2024-09-16T03:39:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.87796699899999997 2024-09-16T03:39:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350433+0000", 2024-09-16T03:39:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013999600000000001 2024-09-16T03:39:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350482+0000", 2024-09-16T03:39:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8720000000000001e-05 2024-09-16T03:39:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1000 4.15 4:aff4cc41:::smithi15736696-11:head [read 0~2873964,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:39:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 138.441974415, 2024-09-16T03:39:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.761707841, 2024-09-16T03:39:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.055 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.056 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:49.056 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.056 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.056 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.056 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1000 2024-09-16T03:39:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:39:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:39:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590560+0000", 2024-09-16T03:39:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:39:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590562+0000", 2024-09-16T03:39:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.17e-06 2024-09-16T03:39:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590568+0000", 2024-09-16T03:39:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7939999999999997e-06 2024-09-16T03:39:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590572+0000", 2024-09-16T03:39:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1300000000000001e-06 2024-09-16T03:39:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590619+0000", 2024-09-16T03:39:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7148999999999999e-05 2024-09-16T03:39:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.061 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590625+0000", 2024-09-16T03:39:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5679999999999996e-06 2024-09-16T03:39:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350858+0000", 2024-09-16T03:39:49.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76023295000000002 2024-09-16T03:39:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350973+0000", 2024-09-16T03:39:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115023 2024-09-16T03:39:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352267+0000", 2024-09-16T03:39:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012935970000000001 2024-09-16T03:39:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1001 4.15 4:aff4cc41:::smithi15736696-11:head [read 1254674~1619290,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 138.44195583000001, 2024-09-16T03:39:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76275827799999996, 2024-09-16T03:39:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.064 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1001 2024-09-16T03:39:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:39:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:39:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.066 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590580+0000", 2024-09-16T03:39:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:39:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590584+0000", 2024-09-16T03:39:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8800000000000001e-06 2024-09-16T03:39:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590587+0000", 2024-09-16T03:39:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6630000000000002e-06 2024-09-16T03:39:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590650+0000", 2024-09-16T03:39:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3950999999999998e-05 2024-09-16T03:39:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590654+0000", 2024-09-16T03:39:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.89e-06 2024-09-16T03:39:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352309+0000", 2024-09-16T03:39:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76165495699999997 2024-09-16T03:39:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352407+0000", 2024-09-16T03:39:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7188999999999994e-05 2024-09-16T03:39:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.353336+0000", 2024-09-16T03:39:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092939400000000005 2024-09-16T03:39:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1002 4.15 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~1527025] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 138.44192657599999, 2024-09-16T03:39:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85963523600000002, 2024-09-16T03:39:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1002 2024-09-16T03:39:49.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.072 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:39:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.15e-07 2024-09-16T03:39:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590609+0000", 2024-09-16T03:39:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.606e-06 2024-09-16T03:39:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590615+0000", 2024-09-16T03:39:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3040000000000001e-06 2024-09-16T03:39:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590618+0000", 2024-09-16T03:39:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.309e-06 2024-09-16T03:39:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590669+0000", 2024-09-16T03:39:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1048000000000003e-05 2024-09-16T03:39:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590673+0000", 2024-09-16T03:39:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.061e-06 2024-09-16T03:39:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.449932+0000", 2024-09-16T03:39:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85925979699999999 2024-09-16T03:39:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450032+0000", 2024-09-16T03:39:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9858000000000005e-05 2024-09-16T03:39:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450242+0000", 2024-09-16T03:39:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020993800000000001 2024-09-16T03:39:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.076 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2467 4.5 4:a11e1d7c:::smithi15736696-43:head [write 1772395~538068] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:39:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 93.041250204999997, 2024-09-16T03:39:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.56288137400000005, 2024-09-16T03:39:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2467 2024-09-16T03:39:49.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:39:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:39:49.078 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991285+0000", 2024-09-16T03:39:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0990000000000001e-06 2024-09-16T03:39:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991447+0000", 2024-09-16T03:39:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016170099999999999 2024-09-16T03:39:49.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991458+0000", 2024-09-16T03:39:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.128e-05 2024-09-16T03:39:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991464+0000", 2024-09-16T03:39:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3800000000000002e-06 2024-09-16T03:39:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.080 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150823+0000", 2024-09-16T03:39:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15935943799999999 2024-09-16T03:39:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150899+0000", 2024-09-16T03:39:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5359999999999997e-05 2024-09-16T03:39:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:49.081 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151007+0000", 2024-09-16T03:39:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108317 2024-09-16T03:39:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253992+0000", 2024-09-16T03:39:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102985249 2024-09-16T03:39:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.353797+0000", 2024-09-16T03:39:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099805039999999998 2024-09-16T03:39:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.082 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554039+0000", 2024-09-16T03:39:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20024175899999999 2024-09-16T03:39:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554129+0000", 2024-09-16T03:39:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0067000000000004e-05 2024-09-16T03:39:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.083 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554165+0000", 2024-09-16T03:39:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5683999999999999e-05 2024-09-16T03:39:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2468 4.5 4:a11e1d7c:::smithi15736696-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:39:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 93.035829426000006, 2024-09-16T03:39:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.55765803899999999, 2024-09-16T03:39:49.084 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2468 2024-09-16T03:39:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:39:49.085 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:39:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996708+0000", 2024-09-16T03:39:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5149999999999998e-06 2024-09-16T03:39:49.086 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996921+0000", 2024-09-16T03:39:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000213692 2024-09-16T03:39:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.087 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996937+0000", 2024-09-16T03:39:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5699999999999999e-05 2024-09-16T03:39:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996946+0000", 2024-09-16T03:39:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7159999999999992e-06 2024-09-16T03:39:49.088 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152362+0000", 2024-09-16T03:39:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15541682800000001 2024-09-16T03:39:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152421+0000", 2024-09-16T03:39:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-16T03:39:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.089 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152515+0000", 2024-09-16T03:39:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3659999999999994e-05 2024-09-16T03:39:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253995+0000", 2024-09-16T03:39:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10148022499999999 2024-09-16T03:39:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.090 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.453908+0000", 2024-09-16T03:39:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19991299700000001 2024-09-16T03:39:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554321+0000", 2024-09-16T03:39:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100412977 2024-09-16T03:39:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554344+0000", 2024-09-16T03:39:49.091 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.262e-05 2024-09-16T03:39:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554362+0000", 2024-09-16T03:39:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8539999999999999e-05 2024-09-16T03:39:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2469 4.5 4:a11e1d7c:::smithi15736696-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:49.092 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:39:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 93.035574062999999, 2024-09-16T03:39:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.75785074799999996, 2024-09-16T03:39:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2469 2024-09-16T03:39:49.093 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:39:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:39:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.094 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996960+0000", 2024-09-16T03:39:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3000000000000001e-07 2024-09-16T03:39:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996963+0000", 2024-09-16T03:39:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8679999999999999e-06 2024-09-16T03:39:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.095 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996967+0000", 2024-09-16T03:39:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.622e-06 2024-09-16T03:39:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996971+0000", 2024-09-16T03:39:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1429999999999998e-06 2024-09-16T03:39:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.096 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253536+0000", 2024-09-16T03:39:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25656582900000002 2024-09-16T03:39:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253632+0000", 2024-09-16T03:39:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6067e-05 2024-09-16T03:39:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:49.097 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253745+0000", 2024-09-16T03:39:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011256699999999999 2024-09-16T03:39:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254218+0000", 2024-09-16T03:39:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00047321400000000001 2024-09-16T03:39:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.098 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554397+0000", 2024-09-16T03:39:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30017854399999999 2024-09-16T03:39:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754557+0000", 2024-09-16T03:39:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200159848 2024-09-16T03:39:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754646+0000", 2024-09-16T03:39:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9370999999999995e-05 2024-09-16T03:39:49.099 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754810+0000", 2024-09-16T03:39:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016414500000000001 2024-09-16T03:39:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2470 4.5 4:a11e1d7c:::smithi15736696-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:49.100 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 93.035557733999994, 2024-09-16T03:39:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76022711799999998, 2024-09-16T03:39:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2470 2024-09-16T03:39:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.101 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.102 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:39:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.65e-07 2024-09-16T03:39:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996977+0000", 2024-09-16T03:39:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4079999999999999e-06 2024-09-16T03:39:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996981+0000", 2024-09-16T03:39:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-16T03:39:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.103 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996988+0000", 2024-09-16T03:39:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0890000000000002e-06 2024-09-16T03:39:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253947+0000", 2024-09-16T03:39:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25695931100000002 2024-09-16T03:39:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:49.104 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253985+0000", 2024-09-16T03:39:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7939999999999999e-05 2024-09-16T03:39:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757090+0000", 2024-09-16T03:39:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50310477499999995 2024-09-16T03:39:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757140+0000", 2024-09-16T03:39:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9567999999999997e-05 2024-09-16T03:39:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.105 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757203+0000", 2024-09-16T03:39:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3215999999999999e-05 2024-09-16T03:39:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2487 4.15 4:ad0b9852:::smithi15736696-16:head [write 1947841~685395] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:39:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.616513143999995, 2024-09-16T03:39:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.74140762199999999, 2024-09-16T03:39:49.106 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2487 2024-09-16T03:39:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:39:49.107 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:39:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416023+0000", 2024-09-16T03:39:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:39:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.108 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416341+0000", 2024-09-16T03:39:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031881699999999999 2024-09-16T03:39:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416353+0000", 2024-09-16T03:39:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.189e-05 2024-09-16T03:39:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.109 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416360+0000", 2024-09-16T03:39:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5719999999999999e-06 2024-09-16T03:39:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755578+0000", 2024-09-16T03:39:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33921859199999999 2024-09-16T03:39:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755639+0000", 2024-09-16T03:39:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0615000000000002e-05 2024-09-16T03:39:49.110 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755735+0000", 2024-09-16T03:39:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5798999999999995e-05 2024-09-16T03:39:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858633+0000", 2024-09-16T03:39:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10289859599999999 2024-09-16T03:39:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.111 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858695+0000", 2024-09-16T03:39:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2000000000000003e-05 2024-09-16T03:39:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157315+0000", 2024-09-16T03:39:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29861931600000002 2024-09-16T03:39:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157375+0000", 2024-09-16T03:39:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0268e-05 2024-09-16T03:39:49.112 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157428+0000", 2024-09-16T03:39:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2939999999999998e-05 2024-09-16T03:39:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2488 4.15 4:ad0b9852:::smithi15736696-16:head [write 3383215~196676] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:39:49.113 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.610840494000001, 2024-09-16T03:39:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73589615900000005, 2024-09-16T03:39:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2488 2024-09-16T03:39:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.114 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:39:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:39:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.115 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421695+0000", 2024-09-16T03:39:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1469999999999999e-06 2024-09-16T03:39:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421975+0000", 2024-09-16T03:39:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028008299999999998 2024-09-16T03:39:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421987+0000", 2024-09-16T03:39:49.116 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1294000000000001e-05 2024-09-16T03:39:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421992+0000", 2024-09-16T03:39:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9349999999999999e-06 2024-09-16T03:39:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757205+0000", 2024-09-16T03:39:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33521242800000001 2024-09-16T03:39:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.117 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757269+0000", 2024-09-16T03:39:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3761000000000002e-05 2024-09-16T03:39:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757376+0000", 2024-09-16T03:39:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010706599999999999 2024-09-16T03:39:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.118 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858684+0000", 2024-09-16T03:39:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101308255 2024-09-16T03:39:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858700+0000", 2024-09-16T03:39:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6138999999999999e-05 2024-09-16T03:39:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157491+0000", 2024-09-16T03:39:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29879056999999998 2024-09-16T03:39:49.119 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157546+0000", 2024-09-16T03:39:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5668999999999998e-05 2024-09-16T03:39:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157589+0000", 2024-09-16T03:39:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2812000000000003e-05 2024-09-16T03:39:49.120 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2489 4.15 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=7b,truncate 3579891] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.610531671999993, 2024-09-16T03:39:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73575981700000004, 2024-09-16T03:39:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.121 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2489 2024-09-16T03:39:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.122 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:39:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0200000000000002e-07 2024-09-16T03:39:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.123 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422005+0000", 2024-09-16T03:39:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7379999999999999e-06 2024-09-16T03:39:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422009+0000", 2024-09-16T03:39:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6219999999999998e-06 2024-09-16T03:39:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422016+0000", 2024-09-16T03:39:49.124 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5789999999999998e-06 2024-09-16T03:39:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858071+0000", 2024-09-16T03:39:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.436054302 2024-09-16T03:39:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858130+0000", 2024-09-16T03:39:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9493000000000001e-05 2024-09-16T03:39:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.125 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:39:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858226+0000", 2024-09-16T03:39:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6174000000000004e-05 2024-09-16T03:39:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:39:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10106483199999999 2024-09-16T03:39:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:49.126 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959639+0000", 2024-09-16T03:39:49.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034802999999999999 2024-09-16T03:39:49.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157640+0000", 2024-09-16T03:39:49.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19800045299999999 2024-09-16T03:39:49.127 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.127 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.127 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:49.127 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157702+0000", 2024-09-16T03:39:49.127 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2218000000000006e-05 2024-09-16T03:39:49.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.128 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.128 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157762+0000", 2024-09-16T03:39:49.128 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9774000000000001e-05 2024-09-16T03:39:49.128 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.128 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.128 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.128 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.128 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.128 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2490 4.15 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:49.129 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:39:49.129 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.610513080000004, 2024-09-16T03:39:49.129 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73597824999999994, 2024-09-16T03:39:49.129 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.129 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.129 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:49.129 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.129 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.129 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.129 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2490 2024-09-16T03:39:49.129 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.130 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:39:49.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.130 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.130 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.130 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:39:49.130 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.130 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422021+0000", 2024-09-16T03:39:49.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8099999999999999e-07 2024-09-16T03:39:49.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.131 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.131 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422022+0000", 2024-09-16T03:39:49.131 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.573e-06 2024-09-16T03:39:49.131 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.131 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422027+0000", 2024-09-16T03:39:49.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3250000000000001e-06 2024-09-16T03:39:49.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.132 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.132 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422030+0000", 2024-09-16T03:39:49.132 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.948e-06 2024-09-16T03:39:49.132 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.132 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858510+0000", 2024-09-16T03:39:49.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.43648032599999997 2024-09-16T03:39:49.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:49.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858588+0000", 2024-09-16T03:39:49.133 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7888999999999999e-05 2024-09-16T03:39:49.133 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.133 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.133 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.133 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157803+0000", 2024-09-16T03:39:49.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.299215642 2024-09-16T03:39:49.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157893+0000", 2024-09-16T03:39:49.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9565000000000001e-05 2024-09-16T03:39:49.134 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.134 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.134 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.134 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157999+0000", 2024-09-16T03:39:49.134 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105701 2024-09-16T03:39:49.134 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.134 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.135 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.135 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.135 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.135 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2492 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:49.135 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:49.135 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.610500571000003, 2024-09-16T03:39:49.135 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53721099400000005, 2024-09-16T03:39:49.135 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.135 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.135 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:49.135 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.136 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.136 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.136 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2492 2024-09-16T03:39:49.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.136 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.136 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.136 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:49.136 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.136 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.136 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.137 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.137 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:49.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.137 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.137 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.137 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:39:49.137 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:39:49.137 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.137 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.138 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.138 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422036+0000", 2024-09-16T03:39:49.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5900000000000002e-06 2024-09-16T03:39:49.138 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.138 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.138 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.138 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422043+0000", 2024-09-16T03:39:49.138 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3069999999999997e-06 2024-09-16T03:39:49.138 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.138 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.139 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.139 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422048+0000", 2024-09-16T03:39:49.139 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6619999999999996e-06 2024-09-16T03:39:49.139 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.139 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.139 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.139 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958774+0000", 2024-09-16T03:39:49.139 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53672593499999999 2024-09-16T03:39:49.139 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.139 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.139 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.139 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958868+0000", 2024-09-16T03:39:49.140 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4408999999999997e-05 2024-09-16T03:39:49.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.140 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.140 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959244+0000", 2024-09-16T03:39:49.140 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037581099999999998 2024-09-16T03:39:49.140 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.140 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.140 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.140 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.140 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.140 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2493 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:49.140 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:49.141 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.610480792000004, 2024-09-16T03:39:49.141 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53757571299999996, 2024-09-16T03:39:49.141 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.141 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.141 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:49.141 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.141 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.141 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.141 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2493 2024-09-16T03:39:49.141 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.141 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:49.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.142 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:49.142 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.142 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.142 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.142 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:39:49.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3600000000000004e-07 2024-09-16T03:39:49.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422055+0000", 2024-09-16T03:39:49.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7460000000000001e-06 2024-09-16T03:39:49.143 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.143 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.143 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.143 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422058+0000", 2024-09-16T03:39:49.143 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:39:49.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.144 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.144 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422061+0000", 2024-09-16T03:39:49.144 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6759999999999999e-06 2024-09-16T03:39:49.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.144 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.144 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:39:49.144 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53723057900000004 2024-09-16T03:39:49.144 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.144 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.144 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959367+0000", 2024-09-16T03:39:49.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5158000000000001e-05 2024-09-16T03:39:49.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.145 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.145 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.145 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959628+0000", 2024-09-16T03:39:49.145 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026175800000000001 2024-09-16T03:39:49.145 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.145 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.145 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.145 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.146 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.146 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2494 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 1738052~782998] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:49.146 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:49.146 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.610469761999994, 2024-09-16T03:39:49.146 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53787640199999998, 2024-09-16T03:39:49.146 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.146 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.146 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:49.146 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.146 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.146 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.147 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2494 2024-09-16T03:39:49.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.147 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:49.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.147 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.147 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.147 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.147 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:49.147 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:39:49.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6100000000000002e-07 2024-09-16T03:39:49.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.148 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.148 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422065+0000", 2024-09-16T03:39:49.148 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2839999999999999e-06 2024-09-16T03:39:49.148 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.148 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422069+0000", 2024-09-16T03:39:49.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7239999999999998e-06 2024-09-16T03:39:49.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422072+0000", 2024-09-16T03:39:49.149 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6560000000000001e-06 2024-09-16T03:39:49.149 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.149 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.149 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.149 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959661+0000", 2024-09-16T03:39:49.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53758982600000005 2024-09-16T03:39:49.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959739+0000", 2024-09-16T03:39:49.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7581000000000001e-05 2024-09-16T03:39:49.150 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.150 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.150 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.150 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959940+0000", 2024-09-16T03:39:49.150 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020107 2024-09-16T03:39:49.150 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.151 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.151 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.151 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.151 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.151 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3987 4.36 4:6ef3b53d:::smithi15736696-48:head [setxattr _header (56) in=7b,truncate 2168169] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:49.151 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:39:49.151 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.656415877000001, 2024-09-16T03:39:49.151 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50651914200000003, 2024-09-16T03:39:49.151 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.151 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.151 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.152 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.152 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.152 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.152 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3987 2024-09-16T03:39:49.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.152 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.152 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.152 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:39:49.152 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.152 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.152 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:39:49.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376119+0000", 2024-09-16T03:39:49.153 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3799999999999996e-07 2024-09-16T03:39:49.153 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.153 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.153 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.153 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376122+0000", 2024-09-16T03:39:49.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.202e-06 2024-09-16T03:39:49.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376251+0000", 2024-09-16T03:39:49.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012896699999999999 2024-09-16T03:39:49.154 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.154 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.154 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.154 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376259+0000", 2024-09-16T03:39:49.154 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.32e-06 2024-09-16T03:39:49.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379368+0000", 2024-09-16T03:39:49.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031084989999999998 2024-09-16T03:39:49.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.155 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.155 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379441+0000", 2024-09-16T03:39:49.155 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3331999999999995e-05 2024-09-16T03:39:49.155 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.155 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,10", 2024-09-16T03:39:49.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379559+0000", 2024-09-16T03:39:49.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118068 2024-09-16T03:39:49.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:49.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380350+0000", 2024-09-16T03:39:49.156 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00079103699999999997 2024-09-16T03:39:49.156 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.156 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.156 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.156 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.580657+0000", 2024-09-16T03:39:49.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20030706600000001 2024-09-16T03:39:49.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882497+0000", 2024-09-16T03:39:49.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30183998899999998 2024-09-16T03:39:49.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.157 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.157 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:49.157 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882560+0000", 2024-09-16T03:39:49.157 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3372000000000002e-05 2024-09-16T03:39:49.157 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.158 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.158 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882637+0000", 2024-09-16T03:39:49.158 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6352000000000004e-05 2024-09-16T03:39:49.158 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.158 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.158 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.158 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.158 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.158 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3988 4.36 4:6ef3b53d:::smithi15736696-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:49.158 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:39:49.159 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 43.656262028999997, 2024-09-16T03:39:49.159 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60662574199999997, 2024-09-16T03:39:49.159 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.159 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.159 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:49.159 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.159 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.159 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.159 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3988 2024-09-16T03:39:49.159 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.159 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:39:49.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.160 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.160 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:39:49.160 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.160 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.160 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376273+0000", 2024-09-16T03:39:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3170000000000001e-06 2024-09-16T03:39:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376278+0000", 2024-09-16T03:39:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7849999999999999e-06 2024-09-16T03:39:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376286+0000", 2024-09-16T03:39:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2009999999999999e-06 2024-09-16T03:39:49.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376291+0000", 2024-09-16T03:39:49.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4029999999999998e-06 2024-09-16T03:39:49.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.162 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.162 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380020+0000", 2024-09-16T03:39:49.162 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037289630000000001 2024-09-16T03:39:49.162 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.162 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:49.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380065+0000", 2024-09-16T03:39:49.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4666999999999998e-05 2024-09-16T03:39:49.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.163 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982800+0000", 2024-09-16T03:39:49.163 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60273539099999995 2024-09-16T03:39:49.163 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.163 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.163 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982838+0000", 2024-09-16T03:39:49.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8155999999999999e-05 2024-09-16T03:39:49.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.164 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.164 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.164 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982897+0000", 2024-09-16T03:39:49.164 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8859e-05 2024-09-16T03:39:49.164 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.164 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.164 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.164 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.165 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.165 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4506 4.1a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v129,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e146)", 2024-09-16T03:39:49.165 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:25.237761+0000", 2024-09-16T03:39:49.165 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 23.794772101, 2024-09-16T03:39:49.165 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.51319717499999995, 2024-09-16T03:39:49.165 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:49.165 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:49.165 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.165 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:49.165 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:49.165 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.166 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4506 2024-09-16T03:39:49.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.166 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:49.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:49.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237761+0000", 2024-09-16T03:39:49.166 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.166 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.166 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.166 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:49.166 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237761+0000", 2024-09-16T03:39:49.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:49.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:49.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237764+0000", 2024-09-16T03:39:49.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2819999999999999e-06 2024-09-16T03:39:49.167 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.167 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.167 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:49.167 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237772+0000", 2024-09-16T03:39:49.167 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9929999999999995e-06 2024-09-16T03:39:49.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:49.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237786+0000", 2024-09-16T03:39:49.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4508999999999999e-05 2024-09-16T03:39:49.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.168 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.168 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237795+0000", 2024-09-16T03:39:49.168 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2070000000000001e-06 2024-09-16T03:39:49.168 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.168 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:49.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237845+0000", 2024-09-16T03:39:49.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.952e-05 2024-09-16T03:39:49.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:49.169 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237911+0000", 2024-09-16T03:39:49.169 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6434999999999997e-05 2024-09-16T03:39:49.169 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.169 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.169 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:39:49.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.649713+0000", 2024-09-16T03:39:49.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.41180149399999999 2024-09-16T03:39:49.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:49.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.650287+0000", 2024-09-16T03:39:49.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057374399999999995 2024-09-16T03:39:49.170 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.170 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.170 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.170 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.650476+0000", 2024-09-16T03:39:49.170 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018991900000000001 2024-09-16T03:39:49.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.750876+0000", 2024-09-16T03:39:49.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100399355 2024-09-16T03:39:49.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.171 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:49.171 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.750914+0000", 2024-09-16T03:39:49.171 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8226999999999998e-05 2024-09-16T03:39:49.171 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:49.171 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:49.172 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:49.172 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.750959+0000", 2024-09-16T03:39:49.172 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4490000000000003e-05 2024-09-16T03:39:49.172 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.172 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.172 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.172 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:49.172 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:49.172 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:49.174 INFO:tasks.rados.rados.0.smithi157.stdout:1977: set_chunk oid 31 offset: 1551872 length: 735232 target oid 634684704d9ba26a9b1af7b23ebff1c4c48c2efe8aac61d8789c2ca07e5cbd72 offset: 0 snap -1 2024-09-16T03:39:49.174 INFO:tasks.rados.rados.0.smithi157.stdout:1978: set_chunk oid 47 target oid 2024-09-16T03:39:49.174 INFO:tasks.rados.rados.0.smithi157.stdout:1978: set_chunk oid 47 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:39:49.174 INFO:tasks.rados.rados.0.smithi157.stdout:1979: write oid 15 current snap is 0 2024-09-16T03:39:49.174 INFO:tasks.rados.rados.0.smithi157.stdout:1979: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2024-09-16T03:39:49.174 INFO:tasks.rados.rados.0.smithi157.stdout:1979: writing smithi15736696-15 from 628694 to 1037226 tid 1 2024-09-16T03:39:49.176 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:39:49.177 INFO:tasks.rados.rados.0.smithi157.stdout:1979: writing smithi15736696-15 from 1515041 to 2115183 tid 2 2024-09-16T03:39:49.177 INFO:tasks.rados.rados.0.smithi157.stdout:1979: writing smithi15736696-15 from 2805198 to 3357310 tid 3 2024-09-16T03:39:49.177 INFO:tasks.rados.rados.0.smithi157.stdout:1980: tier_flush oid 11 2024-09-16T03:39:49.177 INFO:tasks.rados.rados.0.smithi157.stdout:1980: tier_flush oid 11 snap -1 2024-09-16T03:39:49.177 INFO:tasks.rados.rados.0.smithi157.stdout:1974: expect (ObjNum 342 snap 0 seq_num 342) 2024-09-16T03:39:49.177 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:49.177 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:49.178 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:49.172+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:49.216 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2001087 2024-09-16T03:39:49.244 INFO:tasks.rados.rados.0.smithi157.stdout:1973: finishing write tid 1 to smithi15736696-10 2024-09-16T03:39:49.244 INFO:tasks.rados.rados.0.smithi157.stdout:1973: finishing write tid 2 to smithi15736696-10 2024-09-16T03:39:49.245 INFO:tasks.rados.rados.0.smithi157.stdout:1975: finishing write tid 1 to smithi15736696-18 2024-09-16T03:39:49.245 INFO:tasks.rados.rados.0.smithi157.stdout:1975: finishing write tid 2 to smithi15736696-18 2024-09-16T03:39:49.245 INFO:tasks.rados.rados.0.smithi157.stdout:1975: finishing write tid 3 to smithi15736696-18 2024-09-16T03:39:49.245 INFO:tasks.rados.rados.0.smithi157.stdout:1976: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-16T03:39:49.274 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:49.272+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:49.288 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:49.288 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:49.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.289 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.3.22:58 3.2 e165/74 3:6d11fd67:::151202294a8e2e887e3610e0069526a79a1233b1b59d154d57eda7c0823a8bd6:head v 165'298, mlcod=165'294)", 2024-09-16T03:39:49.289 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.176085+0000", 2024-09-16T03:39:49.289 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.11122082, 2024-09-16T03:39:49.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111237584, 2024-09-16T03:39:49.289 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.289 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.289 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:49.289 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.176085+0000", 2024-09-16T03:39:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.176085+0000", 2024-09-16T03:39:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.176086+0000", 2024-09-16T03:39:49.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7000000000000004e-07 2024-09-16T03:39:49.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.176239+0000", 2024-09-16T03:39:49.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015368099999999999 2024-09-16T03:39:49.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.176254+0000", 2024-09-16T03:39:49.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4569999999999999e-05 2024-09-16T03:39:49.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.176260+0000", 2024-09-16T03:39:49.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8459999999999996e-06 2024-09-16T03:39:49.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.276436+0000", 2024-09-16T03:39:49.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10017605 2024-09-16T03:39:49.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.276454+0000", 2024-09-16T03:39:49.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8496e-05 2024-09-16T03:39:49.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.293 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.294 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.3.22:59 3.2 e165/74 3:5573d8b2:::c40fdc5e707a3e740431f326cea6a27fc346a55a14ed7af96c95f9637cca0901:head v 165'299, mlcod=165'294)", 2024-09-16T03:39:49.294 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.176930+0000", 2024-09-16T03:39:49.294 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.110375787, 2024-09-16T03:39:49.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110425723, 2024-09-16T03:39:49.294 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.294 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.294 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:49.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.176930+0000", 2024-09-16T03:39:49.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.176930+0000", 2024-09-16T03:39:49.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.176931+0000", 2024-09-16T03:39:49.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.359e-06 2024-09-16T03:39:49.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.177046+0000", 2024-09-16T03:39:49.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011516899999999999 2024-09-16T03:39:49.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.177058+0000", 2024-09-16T03:39:49.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1211e-05 2024-09-16T03:39:49.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.177064+0000", 2024-09-16T03:39:49.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6460000000000004e-06 2024-09-16T03:39:49.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.276826+0000", 2024-09-16T03:39:49.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099761557000000001 2024-09-16T03:39:49.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.276841+0000", 2024-09-16T03:39:49.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4899000000000001e-05 2024-09-16T03:39:49.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.298 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.3.22:66 3.2 e165/74 3:6c26570c:::9eb3274fcfa870fef5ac57044c6dff4f50e170c7007f812e577892dd932ac9a6:head v 165'300, mlcod=165'296)", 2024-09-16T03:39:49.299 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.178585+0000", 2024-09-16T03:39:49.299 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.108721022, 2024-09-16T03:39:49.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.108799464, 2024-09-16T03:39:49.299 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.299 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.299 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:49.299 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.178585+0000", 2024-09-16T03:39:49.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.178585+0000", 2024-09-16T03:39:49.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.178586+0000", 2024-09-16T03:39:49.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4619999999999999e-06 2024-09-16T03:39:49.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.178852+0000", 2024-09-16T03:39:49.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026565699999999998 2024-09-16T03:39:49.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.178863+0000", 2024-09-16T03:39:49.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1110999999999999e-05 2024-09-16T03:39:49.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.178877+0000", 2024-09-16T03:39:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3997e-05 2024-09-16T03:39:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.277113+0000", 2024-09-16T03:39:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098236352999999998 2024-09-16T03:39:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.277134+0000", 2024-09-16T03:39:49.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0632000000000002e-05 2024-09-16T03:39:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.3.22:67 3.2 e165/74)", 2024-09-16T03:39:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.179487+0000", 2024-09-16T03:39:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.107818732, 2024-09-16T03:39:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10792212299999999, 2024-09-16T03:39:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:39:49.304 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.179487+0000", 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.179487+0000", 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.179488+0000", 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2099999999999995e-07 2024-09-16T03:39:49.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.179539+0000", 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0973999999999997e-05 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.179556+0000", 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7130999999999999e-05 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.179563+0000", 2024-09-16T03:39:49.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4889999999999998e-06 2024-09-16T03:39:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.3.22:68 3.2 e165/74)", 2024-09-16T03:39:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.280177+0000", 2024-09-16T03:39:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0071290169999999996, 2024-09-16T03:39:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0072537629999999999, 2024-09-16T03:39:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.307 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:39:49.308 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.280177+0000", 2024-09-16T03:39:49.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.280177+0000", 2024-09-16T03:39:49.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.280179+0000", 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2139999999999999e-06 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.280897+0000", 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071766600000000005 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.280929+0000", 2024-09-16T03:39:49.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.29e-05 2024-09-16T03:39:49.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.280939+0000", 2024-09-16T03:39:49.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2969999999999992e-06 2024-09-16T03:39:49.310 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.310 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.310 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.310 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.310 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:39:49.310 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 5 2024-09-16T03:39:49.310 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:49.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:39:49.375 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:49.374+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:49.460 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:49.460 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:49.460 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:49.460 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:49.460 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:49.469 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:39:49.476 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:49.475+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:49.539 INFO:tasks.rados.rados.0.smithi157.stdout:1973: finishing write tid 3 to smithi15736696-10 2024-09-16T03:39:49.539 INFO:tasks.rados.rados.0.smithi157.stdout:1978: finishing set_chunk to oid 47 2024-09-16T03:39:49.539 INFO:tasks.rados.rados.0.smithi157.stdout:1978: got expected ENOENT (src dne) 2024-09-16T03:39:49.539 INFO:tasks.rados.rados.0.smithi157.stdout:1975: finishing write tid 4 to smithi15736696-18 2024-09-16T03:39:49.539 INFO:tasks.rados.rados.0.smithi157.stdout:1975: finishing write tid 5 to smithi15736696-18 2024-09-16T03:39:49.539 INFO:tasks.rados.rados.0.smithi157.stdout:1975: oid 18 updating version 0 to 80 2024-09-16T03:39:49.539 INFO:tasks.rados.rados.0.smithi157.stdout:1975: oid 18 version 80 is already newer than 79 2024-09-16T03:39:49.539 INFO:tasks.rados.rados.0.smithi157.stdout:1975: oid 18 version 80 is already newer than 78 2024-09-16T03:39:49.539 INFO:tasks.rados.rados.0.smithi157.stdout:1975: oid 18 version 80 is already newer than 77 2024-09-16T03:39:49.539 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 80 (ObjNum 441 snap 0 seq_num 441) dirty exists 2024-09-16T03:39:49.539 INFO:tasks.rados.rados.0.smithi157.stdout:1975: left oid 18 (ObjNum 441 snap 0 seq_num 441) 2024-09-16T03:39:49.539 INFO:tasks.rados.rados.0.smithi157.stdout:1973: finishing write tid 4 to smithi15736696-10 2024-09-16T03:39:49.540 INFO:tasks.rados.rados.0.smithi157.stdout:1973: finishing write tid 5 to smithi15736696-10 2024-09-16T03:39:49.540 INFO:tasks.rados.rados.0.smithi157.stdout:1973: oid 10 updating version 0 to 126 2024-09-16T03:39:49.540 INFO:tasks.rados.rados.0.smithi157.stdout:1973: oid 10 updating version 126 to 128 2024-09-16T03:39:49.540 INFO:tasks.rados.rados.0.smithi157.stdout:1973: oid 10 version 128 is already newer than 125 2024-09-16T03:39:49.540 INFO:tasks.rados.rados.0.smithi157.stdout:1973: oid 10 version 128 is already newer than 127 2024-09-16T03:39:49.540 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 10 v 128 (ObjNum 440 snap 0 seq_num 440) dirty exists 2024-09-16T03:39:49.540 INFO:tasks.rados.rados.0.smithi157.stdout:1973: left oid 10 (ObjNum 440 snap 0 seq_num 440) 2024-09-16T03:39:49.540 INFO:tasks.rados.rados.0.smithi157.stdout:1979: finishing write tid 1 to smithi15736696-15 2024-09-16T03:39:49.540 INFO:tasks.rados.rados.0.smithi157.stdout:1973: done (7 left) 2024-09-16T03:39:49.540 INFO:tasks.rados.rados.0.smithi157.stdout:1974: done (6 left) 2024-09-16T03:39:49.540 INFO:tasks.rados.rados.0.smithi157.stdout:1975: done (5 left) 2024-09-16T03:39:49.540 INFO:tasks.rados.rados.0.smithi157.stdout:1976: done (4 left) 2024-09-16T03:39:49.540 INFO:tasks.rados.rados.0.smithi157.stdout:1978: done (3 left) 2024-09-16T03:39:49.541 INFO:tasks.rados.rados.0.smithi157.stdout:1981: read oid 36 snap -1 2024-09-16T03:39:49.541 INFO:tasks.rados.rados.0.smithi157.stdout:1981: expect (ObjNum 420 snap 0 seq_num 420) 2024-09-16T03:39:49.541 INFO:tasks.rados.rados.0.smithi157.stdout:1982: write (excl) oid 42 current snap is 0 2024-09-16T03:39:49.541 INFO:tasks.rados.rados.0.smithi157.stdout:1979: finishing write tid 2 to smithi15736696-15 2024-09-16T03:39:49.541 INFO:tasks.rados.rados.0.smithi157.stdout:1982: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2024-09-16T03:39:49.544 INFO:tasks.rados.rados.0.smithi157.stdout:1982: writing smithi15736696-42 from 645501 to 1129282 tid 1 2024-09-16T03:39:49.549 INFO:tasks.rados.rados.0.smithi157.stdout:1982: writing smithi15736696-42 from 1573523 to 2033676 tid 2 2024-09-16T03:39:49.556 INFO:tasks.rados.rados.0.smithi157.stdout:1982: writing smithi15736696-42 from 2548974 to 3312358 tid 3 2024-09-16T03:39:49.556 INFO:tasks.rados.rados.0.smithi157.stdout:1983: read oid 2 snap -1 2024-09-16T03:39:49.556 INFO:tasks.rados.rados.0.smithi157.stdout:1983: expect deleted 2024-09-16T03:39:49.556 INFO:tasks.rados.rados.0.smithi157.stdout:1984: write (excl) oid 22 current snap is 0 2024-09-16T03:39:49.556 INFO:tasks.rados.rados.0.smithi157.stdout:1984: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2024-09-16T03:39:49.562 INFO:tasks.rados.rados.0.smithi157.stdout:1984: writing smithi15736696-22 from 662308 to 1221338 tid 1 2024-09-16T03:39:49.568 INFO:tasks.rados.rados.0.smithi157.stdout:1984: writing smithi15736696-22 from 1632005 to 2235816 tid 2 2024-09-16T03:39:49.575 INFO:tasks.rados.rados.0.smithi157.stdout:1984: writing smithi15736696-22 from 2860044 to 3462308 tid 3 2024-09-16T03:39:49.575 INFO:tasks.rados.rados.0.smithi157.stdout:1985: read oid 29 snap -1 2024-09-16T03:39:49.575 INFO:tasks.rados.rados.0.smithi157.stdout:1985: expect (ObjNum 429 snap 0 seq_num 429) 2024-09-16T03:39:49.575 INFO:tasks.rados.rados.0.smithi157.stdout:1986: read oid 23 snap -1 2024-09-16T03:39:49.575 INFO:tasks.rados.rados.0.smithi157.stdout:1986: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:39:49.575 INFO:tasks.rados.rados.0.smithi157.stdout:1987: write (excl) oid 37 current snap is 0 2024-09-16T03:39:49.575 INFO:tasks.rados.rados.0.smithi157.stdout:1987: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2024-09-16T03:39:49.577 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:49.574+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:49.580 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:49.580 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:49.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:49.580 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:49.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.581 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4260.0:39 1.0 e27/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 27'21, mlcod=26'19)", 2024-09-16T03:39:49.581 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:39:49.581 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 174.982198251, 2024-09-16T03:39:49.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30197182500000003, 2024-09-16T03:39:49.581 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.581 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.581 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.581 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:39:49.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595318+0000", 2024-09-16T03:39:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595320+0000", 2024-09-16T03:39:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.962e-06 2024-09-16T03:39:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595336+0000", 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5568000000000001e-05 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.595349+0000", 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3203e-05 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695424+0000", 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10007542899999999 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695495+0000", 2024-09-16T03:39:49.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0450999999999993e-05 2024-09-16T03:39:49.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.695538+0000", 2024-09-16T03:39:49.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3433000000000002e-05 2024-09-16T03:39:49.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:49.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897256+0000", 2024-09-16T03:39:49.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20171840699999999 2024-09-16T03:39:49.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:36:54.897290+0000", 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3371999999999998e-05 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1948 4.8 4:146f84f0:::smithi15736696-6:head [setxattr _header (55) in=7b,truncate 2756348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.452709445, 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40326984799999999, 2024-09-16T03:39:49.585 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1948 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124808+0000", 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9099999999999991e-07 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124813+0000", 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9599999999999999e-06 2024-09-16T03:39:49.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124821+0000", 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0490000000000004e-06 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124827+0000", 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3339999999999998e-06 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126739+0000", 2024-09-16T03:39:49.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00191147 2024-09-16T03:39:49.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126830+0000", 2024-09-16T03:39:49.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1751999999999998e-05 2024-09-16T03:39:49.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:49.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127005+0000", 2024-09-16T03:39:49.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174227 2024-09-16T03:39:49.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127981+0000", 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097622799999999999 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.128193+0000", 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021176999999999999 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.135281+0000", 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070882150000000001 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.427785+0000", 2024-09-16T03:39:49.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29250390599999998 2024-09-16T03:39:49.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527932+0000", 2024-09-16T03:39:49.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10014759500000001 2024-09-16T03:39:49.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:49.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527980+0000", 2024-09-16T03:39:49.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7725000000000001e-05 2024-09-16T03:39:49.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528077+0000", 2024-09-16T03:39:49.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6626000000000006e-05 2024-09-16T03:39:49.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.592 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.592 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1949 4.8 4:146f84f0:::smithi15736696-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.452678701, 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40346921899999999, 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1949 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:39:49.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124841+0000", 2024-09-16T03:39:49.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5890000000000001e-06 2024-09-16T03:39:49.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124847+0000", 2024-09-16T03:39:49.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0820000000000004e-06 2024-09-16T03:39:49.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124852+0000", 2024-09-16T03:39:49.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1070000000000004e-06 2024-09-16T03:39:49.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127536+0000", 2024-09-16T03:39:49.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026844619999999999 2024-09-16T03:39:49.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:49.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127585+0000", 2024-09-16T03:39:49.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8915999999999997e-05 2024-09-16T03:39:49.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528126+0000", 2024-09-16T03:39:49.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40054061899999999 2024-09-16T03:39:49.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528193+0000", 2024-09-16T03:39:49.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6717999999999994e-05 2024-09-16T03:39:49.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528307+0000", 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011413600000000001 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1979 4.18 4:1f63731d:::smithi15736696-30:head [write 789962~534747] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 107.94167138900001, 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.506754182, 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.598 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.599 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.599 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1979 2024-09-16T03:39:49.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.599 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:39:49.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635850+0000", 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6670000000000004e-06 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636429+0000", 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000579745 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636441+0000", 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1965e-05 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736594+0000", 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100152291 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837326+0000", 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10073201399999999 2024-09-16T03:39:49.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837437+0000", 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111914 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837625+0000", 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000187391 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939449+0000", 2024-09-16T03:39:49.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10182371599999999 2024-09-16T03:39:49.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:49.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939476+0000", 2024-09-16T03:39:49.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7117000000000001e-05 2024-09-16T03:39:49.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142297+0000", 2024-09-16T03:39:49.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20282091499999999 2024-09-16T03:39:49.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142347+0000", 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0800000000000002e-05 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142485+0000", 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013799999999999999 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142526+0000", 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0269999999999999e-05 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142599+0000", 2024-09-16T03:39:49.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3376999999999999e-05 2024-09-16T03:39:49.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.605 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.605 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.605 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1980 4.18 4:1f63731d:::smithi15736696-30:head [write 1780952~690056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:49.605 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:39:49.605 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 107.840870228, 2024-09-16T03:39:49.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40615011400000001, 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1980 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:39:49.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736656+0000", 2024-09-16T03:39:49.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0339999999999999e-05 2024-09-16T03:39:49.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737092+0000", 2024-09-16T03:39:49.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435957 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737102+0000", 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7860000000000008e-06 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737107+0000", 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839282+0000", 2024-09-16T03:39:49.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10217523100000001 2024-09-16T03:39:49.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839389+0000", 2024-09-16T03:39:49.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010619399999999999 2024-09-16T03:39:49.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:49.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839567+0000", 2024-09-16T03:39:49.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017886200000000001 2024-09-16T03:39:49.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939485+0000", 2024-09-16T03:39:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099917168000000001 2024-09-16T03:39:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943655+0000", 2024-09-16T03:39:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004170287 2024-09-16T03:39:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142373+0000", 2024-09-16T03:39:49.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19871839599999999 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142431+0000", 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7880999999999997e-05 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142627+0000", 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019580500000000001 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:49.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142723+0000", 2024-09-16T03:39:49.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6209e-05 2024-09-16T03:39:49.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142796+0000", 2024-09-16T03:39:49.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2978000000000005e-05 2024-09-16T03:39:49.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.612 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2277 4.18 4:1f63731d:::smithi15736696-30:head [write 460558~707131] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 99.154734528000006, 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39421539100000003, 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2277 2024-09-16T03:39:49.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:39:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:39:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422785+0000", 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9809999999999999e-06 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423586+0000", 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080133600000000004 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423607+0000", 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1039e-05 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423616+0000", 2024-09-16T03:39:49.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0189999999999995e-06 2024-09-16T03:39:49.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523776+0000", 2024-09-16T03:39:49.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015945399999999 2024-09-16T03:39:49.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523851+0000", 2024-09-16T03:39:49.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5549999999999993e-05 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523972+0000", 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120683 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527436+0000", 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034638870000000001 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636588+0000", 2024-09-16T03:39:49.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109152601 2024-09-16T03:39:49.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636665+0000", 2024-09-16T03:39:49.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6661000000000003e-05 2024-09-16T03:39:49.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637128+0000", 2024-09-16T03:39:49.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046316199999999998 2024-09-16T03:39:49.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816886+0000", 2024-09-16T03:39:49.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17975745200000001 2024-09-16T03:39:49.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:49.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816927+0000", 2024-09-16T03:39:49.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1529000000000002e-05 2024-09-16T03:39:49.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816997+0000", 2024-09-16T03:39:49.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0036999999999994e-05 2024-09-16T03:39:49.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.620 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.620 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2278 4.18 4:1f63731d:::smithi15736696-30:head [write 1869389~508716] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:49.620 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:39:49.620 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 99.148165527000003, 2024-09-16T03:39:49.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48788153400000001, 2024-09-16T03:39:49.620 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.620 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.620 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2278 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429352+0000", 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.604e-06 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429895+0000", 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054308899999999996 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429909+0000", 2024-09-16T03:39:49.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3518e-05 2024-09-16T03:39:49.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429917+0000", 2024-09-16T03:39:49.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-09-16T03:39:49.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525208+0000", 2024-09-16T03:39:49.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095291736000000002 2024-09-16T03:39:49.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525303+0000", 2024-09-16T03:39:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4304999999999999e-05 2024-09-16T03:39:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525410+0000", 2024-09-16T03:39:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010755399999999999 2024-09-16T03:39:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636785+0000", 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111375037 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637092+0000", 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030685599999999999 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637157+0000", 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4593999999999999e-05 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637236+0000", 2024-09-16T03:39:49.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9563999999999999e-05 2024-09-16T03:39:49.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917119+0000", 2024-09-16T03:39:49.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.279882931 2024-09-16T03:39:49.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:49.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917158+0000", 2024-09-16T03:39:49.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9096e-05 2024-09-16T03:39:49.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917232+0000", 2024-09-16T03:39:49.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4091000000000006e-05 2024-09-16T03:39:49.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.629 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.630 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2279 4.18 4:1f63731d:::smithi15736696-30:head [write 3094884~165674] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:49.630 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:39:49.630 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 99.143476731999996, 2024-09-16T03:39:49.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483372209, 2024-09-16T03:39:49.630 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.630 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.630 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.630 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2279 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434041+0000", 2024-09-16T03:39:49.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.195e-06 2024-09-16T03:39:49.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434191+0000", 2024-09-16T03:39:49.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000150488 2024-09-16T03:39:49.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434202+0000", 2024-09-16T03:39:49.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0978000000000001e-05 2024-09-16T03:39:49.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434208+0000", 2024-09-16T03:39:49.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-09-16T03:39:49.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526820+0000", 2024-09-16T03:39:49.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092612229000000004 2024-09-16T03:39:49.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526894+0000", 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3435000000000004e-05 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526990+0000", 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6112000000000004e-05 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637050+0000", 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110059859 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:49.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637097+0000", 2024-09-16T03:39:49.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6848000000000003e-05 2024-09-16T03:39:49.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637183+0000", 2024-09-16T03:39:49.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6798000000000005e-05 2024-09-16T03:39:49.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637262+0000", 2024-09-16T03:39:49.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9049999999999997e-05 2024-09-16T03:39:49.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917287+0000", 2024-09-16T03:39:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280025036 2024-09-16T03:39:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917326+0000", 2024-09-16T03:39:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.871e-05 2024-09-16T03:39:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917412+0000", 2024-09-16T03:39:49.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5638999999999998e-05 2024-09-16T03:39:49.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.637 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.637 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2280 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 3260558] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:49.637 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:49.637 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 99.143298451000007, 2024-09-16T03:39:49.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48353995599999999, 2024-09-16T03:39:49.637 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.637 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2280 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434221+0000", 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8329999999999998e-06 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434225+0000", 2024-09-16T03:39:49.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7589999999999999e-06 2024-09-16T03:39:49.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434228+0000", 2024-09-16T03:39:49.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.971e-06 2024-09-16T03:39:49.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527459+0000", 2024-09-16T03:39:49.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093231249000000002 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527519+0000", 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9471e-05 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527606+0000", 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6776999999999999e-05 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637084+0000", 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109478228 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637100+0000", 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6711999999999999e-05 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637210+0000", 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010909200000000001 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637296+0000", 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6718000000000006e-05 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917637+0000", 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280340537 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917671+0000", 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3983999999999999e-05 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917758+0000", 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7017999999999999e-05 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2281 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 99.143284155999993, 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483761683, 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2281 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.96e-07 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434234+0000", 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4050000000000001e-06 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434237+0000", 2024-09-16T03:39:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1939999999999998e-06 2024-09-16T03:39:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434240+0000", 2024-09-16T03:39:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.74e-06 2024-09-16T03:39:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530037+0000", 2024-09-16T03:39:49.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095796830999999999 2024-09-16T03:39:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530089+0000", 2024-09-16T03:39:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:39:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917823+0000", 2024-09-16T03:39:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38773380000000002 2024-09-16T03:39:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917891+0000", 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7676999999999996e-05 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917994+0000", 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010286200000000001 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2790 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v39,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-09-16T03:39:49.649 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:39:49.650 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.173436701, 2024-09-16T03:39:49.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42581759000000002, 2024-09-16T03:39:49.650 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.650 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.650 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.650 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.650 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.650 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.650 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2790 2024-09-16T03:39:49.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.650 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404083+0000", 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2770000000000001e-06 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404095+0000", 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1780000000000001e-05 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404116+0000", 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0882000000000001e-05 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404127+0000", 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1078e-05 2024-09-16T03:39:49.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489837+0000", 2024-09-16T03:39:49.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085709977000000007 2024-09-16T03:39:49.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489956+0000", 2024-09-16T03:39:49.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119856 2024-09-16T03:39:49.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:49.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.592758+0000", 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10280165400000001 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.593693+0000", 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000935285 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.594164+0000", 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047097000000000003 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.628426+0000", 2024-09-16T03:39:49.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034262070999999998 2024-09-16T03:39:49.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.732623+0000", 2024-09-16T03:39:49.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10419628 2024-09-16T03:39:49.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829726+0000", 2024-09-16T03:39:49.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097103098999999998 2024-09-16T03:39:49.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:49.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829790+0000", 2024-09-16T03:39:49.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4677000000000004e-05 2024-09-16T03:39:49.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829897+0000", 2024-09-16T03:39:49.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106704 2024-09-16T03:39:49.656 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.656 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2850 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 2252171] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 82.656896419000006, 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30323033799999999, 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.657 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2850 2024-09-16T03:39:49.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.658 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:39:49.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:39:49.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920621+0000", 2024-09-16T03:39:49.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.47e-07 2024-09-16T03:39:49.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920625+0000", 2024-09-16T03:39:49.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1620000000000001e-06 2024-09-16T03:39:49.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920632+0000", 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0180000000000004e-06 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920638+0000", 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0950000000000001e-06 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922161+0000", 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015229880000000001 2024-09-16T03:39:49.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922222+0000", 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0903999999999998e-05 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922323+0000", 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101425 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.023371+0000", 2024-09-16T03:39:49.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101047493 2024-09-16T03:39:49.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123227+0000", 2024-09-16T03:39:49.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099856057999999998 2024-09-16T03:39:49.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123840+0000", 2024-09-16T03:39:49.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061324999999999997 2024-09-16T03:39:49.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123902+0000", 2024-09-16T03:39:49.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1447999999999998e-05 2024-09-16T03:39:49.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223497+0000", 2024-09-16T03:39:49.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099595499000000004 2024-09-16T03:39:49.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:49.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223665+0000", 2024-09-16T03:39:49.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016807399999999999 2024-09-16T03:39:49.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223850+0000", 2024-09-16T03:39:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018507699999999999 2024-09-16T03:39:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2851 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:39:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:39:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 82.656860323000004, 2024-09-16T03:39:49.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30354833399999998, 2024-09-16T03:39:49.665 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.665 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.665 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:49.665 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.665 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.665 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.665 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2851 2024-09-16T03:39:49.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.665 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:39:49.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:39:49.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920657+0000", 2024-09-16T03:39:49.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0100000000000004e-07 2024-09-16T03:39:49.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920660+0000", 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6519999999999999e-06 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920668+0000", 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1919999999999999e-06 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920674+0000", 2024-09-16T03:39:49.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-16T03:39:49.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126299+0000", 2024-09-16T03:39:49.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20562534700000001 2024-09-16T03:39:49.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:49.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126358+0000", 2024-09-16T03:39:49.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8459999999999999e-05 2024-09-16T03:39:49.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223913+0000", 2024-09-16T03:39:49.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097555263000000003 2024-09-16T03:39:49.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224009+0000", 2024-09-16T03:39:49.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5723999999999994e-05 2024-09-16T03:39:49.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224204+0000", 2024-09-16T03:39:49.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195498 2024-09-16T03:39:49.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.670 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.670 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3570 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:49.670 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:39:49.670 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.571726998999999, 2024-09-16T03:39:49.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.353157053, 2024-09-16T03:39:49.670 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.670 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.670 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:49.671 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.671 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.671 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.671 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3570 2024-09-16T03:39:49.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.671 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:39:49.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:39:49.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005796+0000", 2024-09-16T03:39:49.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9229999999999996e-06 2024-09-16T03:39:49.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006222+0000", 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000426282 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006240+0000", 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7858000000000001e-05 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006249+0000", 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7120000000000006e-06 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006323+0000", 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3808999999999998e-05 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006333+0000", 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0488e-05 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358910+0000", 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.35257616400000003 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358946+0000", 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6816999999999997e-05 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3571 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:39:49.675 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.565221843000003, 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34670431699999998, 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3571 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:39:49.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:39:49.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012298+0000", 2024-09-16T03:39:49.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.18e-06 2024-09-16T03:39:49.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012947+0000", 2024-09-16T03:39:49.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064929600000000005 2024-09-16T03:39:49.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012961+0000", 2024-09-16T03:39:49.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4341e-05 2024-09-16T03:39:49.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012970+0000", 2024-09-16T03:39:49.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2740000000000007e-06 2024-09-16T03:39:49.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.013025+0000", 2024-09-16T03:39:49.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5353999999999999e-05 2024-09-16T03:39:49.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:49.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.013033+0000", 2024-09-16T03:39:49.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4619999999999996e-06 2024-09-16T03:39:49.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358989+0000", 2024-09-16T03:39:49.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34595562699999999 2024-09-16T03:39:49.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358999+0000", 2024-09-16T03:39:49.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7829999999999993e-06 2024-09-16T03:39:49.680 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.680 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.680 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.681 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3572 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:49.681 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:39:49.681 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.559201086999998, 2024-09-16T03:39:49.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34073019300000001, 2024-09-16T03:39:49.681 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.681 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.681 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:49.681 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.681 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.681 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.681 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3572 2024-09-16T03:39:49.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.682 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:39:49.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:39:49.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018317+0000", 2024-09-16T03:39:49.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4610000000000001e-06 2024-09-16T03:39:49.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018779+0000", 2024-09-16T03:39:49.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046202500000000002 2024-09-16T03:39:49.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018793+0000", 2024-09-16T03:39:49.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4629e-05 2024-09-16T03:39:49.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018799+0000", 2024-09-16T03:39:49.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1589999999999998e-06 2024-09-16T03:39:49.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018867+0000", 2024-09-16T03:39:49.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7879000000000005e-05 2024-09-16T03:39:49.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:49.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018888+0000", 2024-09-16T03:39:49.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0747999999999999e-05 2024-09-16T03:39:49.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359036+0000", 2024-09-16T03:39:49.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34014758 2024-09-16T03:39:49.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359045+0000", 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7119999999999995e-06 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3573 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.558701393, 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34027060599999998, 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:49.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.687 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3573 2024-09-16T03:39:49.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.687 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:39:49.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018816+0000", 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1400000000000001e-06 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018821+0000", 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4179999999999996e-06 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018829+0000", 2024-09-16T03:39:49.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6320000000000008e-06 2024-09-16T03:39:49.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018835+0000", 2024-09-16T03:39:49.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0399999999999998e-06 2024-09-16T03:39:49.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018953+0000", 2024-09-16T03:39:49.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117713 2024-09-16T03:39:49.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:49.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018967+0000", 2024-09-16T03:39:49.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4334e-05 2024-09-16T03:39:49.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359078+0000", 2024-09-16T03:39:49.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34011044499999998 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359086+0000", 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8839999999999994e-06 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3574 4.28 4.f21f628 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.558668238000003, 2024-09-16T03:39:49.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.340272882, 2024-09-16T03:39:49.692 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.692 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.692 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:49.692 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.692 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.692 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.692 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3574 2024-09-16T03:39:49.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.692 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:39:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:39:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018849+0000", 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9599999999999998e-07 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018852+0000", 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9749999999999999e-06 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018867+0000", 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5102e-05 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018882+0000", 2024-09-16T03:39:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5319e-05 2024-09-16T03:39:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018993+0000", 2024-09-16T03:39:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110457 2024-09-16T03:39:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018998+0000", 2024-09-16T03:39:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.028e-06 2024-09-16T03:39:49.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359114+0000", 2024-09-16T03:39:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34011593099999998 2024-09-16T03:39:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359121+0000", 2024-09-16T03:39:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2740000000000002e-06 2024-09-16T03:39:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3996 4.8 4:13b03fc9:::smithi15736696-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:05.315045+0000", 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.262470829000002, 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30191814700000003, 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3996 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315045+0000", 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315045+0000", 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315046+0000", 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9100000000000003e-07 2024-09-16T03:39:49.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315049+0000", 2024-09-16T03:39:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1020000000000001e-06 2024-09-16T03:39:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315056+0000", 2024-09-16T03:39:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5400000000000001e-06 2024-09-16T03:39:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.315062+0000", 2024-09-16T03:39:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.908e-06 2024-09-16T03:39:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.515314+0000", 2024-09-16T03:39:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20025267999999999 2024-09-16T03:39:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.515437+0000", 2024-09-16T03:39:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122955 2024-09-16T03:39:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.616750+0000", 2024-09-16T03:39:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10131272199999999 2024-09-16T03:39:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.616842+0000", 2024-09-16T03:39:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2408e-05 2024-09-16T03:39:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.616964+0000", 2024-09-16T03:39:49.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121141 2024-09-16T03:39:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4658 4.28 4:146f84f0:::smithi15736696-6:head [cache-evict] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e150)", 2024-09-16T03:39:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.652963+0000", 2024-09-16T03:39:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.924553414000002, 2024-09-16T03:39:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30247697200000001, 2024-09-16T03:39:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:49.702 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4658 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652963+0000", 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652963+0000", 2024-09-16T03:39:49.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652966+0000", 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8090000000000001e-06 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652974+0000", 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8490000000000006e-06 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652990+0000", 2024-09-16T03:39:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6775000000000001e-05 2024-09-16T03:39:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.653002+0000", 2024-09-16T03:39:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1895999999999999e-05 2024-09-16T03:39:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.753468+0000", 2024-09-16T03:39:49.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100466079 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.753489+0000", 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0479000000000002e-05 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.955202+0000", 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201713432 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.955360+0000", 2024-09-16T03:39:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000158102 2024-09-16T03:39:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.955440+0000", 2024-09-16T03:39:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9550999999999998e-05 2024-09-16T03:39:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.707 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.707 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:49.709 INFO:tasks.rados.rados.0.smithi157.stdout:1987: writing smithi15736696-37 from 679115 to 1313394 tid 1 2024-09-16T03:39:49.709 INFO:tasks.rados.rados.0.smithi157.stdout:1987: writing smithi15736696-37 from 2090487 to 2554309 tid 2 2024-09-16T03:39:49.709 INFO:tasks.rados.rados.0.smithi157.stdout:1987: writing smithi15736696-37 from 3003820 to 3479115 tid 3 2024-09-16T03:39:49.710 INFO:tasks.rados.rados.0.smithi157.stdout:1988: set_chunk oid 5 target oid 2024-09-16T03:39:49.710 INFO:tasks.rados.rados.0.smithi157.stdout:1988: set_chunk oid 5 offset: 1001984 length: 269312 target oid 612d08aa83970a081457bcabc21187507e335ef8ad749c973e2f579badb1ec92 offset: 0 snap -1 2024-09-16T03:39:49.710 INFO:tasks.rados.rados.0.smithi157.stdout:1989: tier_promote oid 8 2024-09-16T03:39:49.710 INFO:tasks.rados.rados.0.smithi157.stdout:1981: expect (ObjNum 420 snap 0 seq_num 420) 2024-09-16T03:39:49.710 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:39:49.711 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:49.707+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:49.787 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:49.785+0000 7f3d4270c640 -1 osd.12 165 handle_osd_map injecting map crc failure 2024-09-16T03:39:49.809 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:49.806+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:49.820 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.820 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(osd.10.22:290 3.1 e165/31)", 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.772486+0000", 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.047015669000000003, 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047029806, 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "no flag points reached", 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.772486+0000", 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.772486+0000", 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.772487+0000", 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.297e-06 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.772494+0000", 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1250000000000004e-06 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.772504+0000", 2024-09-16T03:39:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9350000000000006e-06 2024-09-16T03:39:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4871 4.13 4:c8492bfc:::smithi15736696-42:head [setxattr _header (56) in=63b,truncate 3445501] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-16T03:39:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.556560+0000", 2024-09-16T03:39:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.26294109300000001, 2024-09-16T03:39:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.26298778299999997, 2024-09-16T03:39:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-09-16T03:39:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4871 2024-09-16T03:39:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.824 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.556560+0000", 2024-09-16T03:39:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.556560+0000", 2024-09-16T03:39:49.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.556562+0000", 2024-09-16T03:39:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.252e-06 2024-09-16T03:39:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.556566+0000", 2024-09-16T03:39:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7740000000000004e-06 2024-09-16T03:39:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.556575+0000", 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6559999999999998e-06 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.556582+0000", 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3980000000000002e-06 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.748141+0000", 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19155888600000001 2024-09-16T03:39:49.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.748243+0000", 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102142 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,9", 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.748400+0000", 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000156317 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.749242+0000", 2024-09-16T03:39:49.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00084229499999999996 2024-09-16T03:39:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.749539+0000", 2024-09-16T03:39:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029729700000000002 2024-09-16T03:39:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4872 4.13 4:c8492bfc:::smithi15736696-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e165)", 2024-09-16T03:39:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.556592+0000", 2024-09-16T03:39:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.26290966799999999, 2024-09-16T03:39:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.26299940500000002, 2024-09-16T03:39:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for rw locks", 2024-09-16T03:39:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:49.830 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4872 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.556592+0000", 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.556592+0000", 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.556592+0000", 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3499999999999996e-07 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.556595+0000", 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6759999999999999e-06 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.556603+0000", 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6729999999999992e-06 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.556610+0000", 2024-09-16T03:39:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3370000000000002e-06 2024-09-16T03:39:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.749149+0000", 2024-09-16T03:39:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19253874600000001 2024-09-16T03:39:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.749229+0000", 2024-09-16T03:39:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0406000000000002e-05 2024-09-16T03:39:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:39:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 3 2024-09-16T03:39:49.834 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:49.836 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:39:49.840 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 474477 2024-09-16T03:39:49.909 INFO:tasks.rados.rados.0.smithi157.stdout:1980: got (0) Success 2024-09-16T03:39:49.909 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 11 v 0 (ObjNum 428 snap 0 seq_num 428) dirty exists 2024-09-16T03:39:49.909 INFO:tasks.rados.rados.0.smithi157.stdout:1980: done (11 left) 2024-09-16T03:39:49.909 INFO:tasks.rados.rados.0.smithi157.stdout:1981: done (10 left) 2024-09-16T03:39:49.909 INFO:tasks.rados.rados.0.smithi157.stdout:1990: read oid 24 snap -1 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:1990: expect (ObjNum 333 snap 0 seq_num 333) 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:1991: tier_evict oid 46 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:1991: tier_evict oid 46 snap -1 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:1977: finishing set_chunk to oid 31 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 31 v 93 (ObjNum 436 snap 0 seq_num 436) dirty exists 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:1979: finishing write tid 3 to smithi15736696-15 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:1984: finishing write tid 1 to smithi15736696-22 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:1984: finishing write tid 2 to smithi15736696-22 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:1988: finishing set_chunk to oid 5 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:Range is overlapped: oid 5 set_chunk returned error code -95 offset: 1001984 length: 269312 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 5 v 0 (ObjNum 378 snap 0 seq_num 378) dirty exists 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:1982: finishing write tid 1 to smithi15736696-42 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:1982: finishing write tid 2 to smithi15736696-42 2024-09-16T03:39:49.910 INFO:tasks.rados.rados.0.smithi157.stdout:1977: done (11 left) 2024-09-16T03:39:49.911 INFO:tasks.rados.rados.0.smithi157.stdout:1983: done (10 left) 2024-09-16T03:39:49.911 INFO:tasks.rados.rados.0.smithi157.stdout:1988: done (9 left) 2024-09-16T03:39:49.911 INFO:tasks.rados.rados.0.smithi157.stdout:1992: tier_flush oid 47 2024-09-16T03:39:49.911 INFO:tasks.rados.rados.0.smithi157.stdout:1992: tier_flush oid 47 snap -1 2024-09-16T03:39:49.911 INFO:tasks.rados.rados.0.smithi157.stdout:1982: finishing write tid 3 to smithi15736696-42 2024-09-16T03:39:49.911 INFO:tasks.rados.rados.0.smithi157.stdout:1984: finishing write tid 3 to smithi15736696-22 2024-09-16T03:39:49.911 INFO:tasks.rados.rados.0.smithi157.stdout:1987: finishing write tid 1 to smithi15736696-37 2024-09-16T03:39:49.911 INFO:tasks.rados.rados.0.smithi157.stdout:1987: finishing write tid 2 to smithi15736696-37 2024-09-16T03:39:49.911 INFO:tasks.rados.rados.0.smithi157.stdout:1986: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-16T03:39:49.912 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:49.910+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:49.944 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:49.944 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:49.944 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:49.944 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:49.944 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:49.952 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:39:50.013 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:50.012+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:50.041 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:50.037+0000 7f8be56cd640 -1 osd.7 165 handle_osd_map injecting map crc failure 2024-09-16T03:39:50.062 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:50.062 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:50.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:50.062 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:50.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.063 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:34 4.e 4:773b934b:::smithi15736696-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:50.063 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:39:50.063 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 158.00458018, 2024-09-16T03:39:50.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50346527299999999, 2024-09-16T03:39:50.063 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.063 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-16T03:39:50.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.063 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:39:50.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:39:50.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054413+0000", 2024-09-16T03:39:50.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0669999999999999e-06 2024-09-16T03:39:50.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054525+0000", 2024-09-16T03:39:50.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111462 2024-09-16T03:39:50.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054538+0000", 2024-09-16T03:39:50.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3341e-05 2024-09-16T03:39:50.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054545+0000", 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.584e-06 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054608+0000", 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2953999999999994e-05 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054613+0000", 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6110000000000001e-06 2024-09-16T03:39:50.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154852+0000", 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10023838 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154915+0000", 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3138000000000004e-05 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155001+0000", 2024-09-16T03:39:50.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5730999999999996e-05 2024-09-16T03:39:50.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156166+0000", 2024-09-16T03:39:50.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001165485 2024-09-16T03:39:50.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255909+0000", 2024-09-16T03:39:50.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099742558999999995 2024-09-16T03:39:50.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557800+0000", 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30189130600000003 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557836+0000", 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6566000000000002e-05 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557877+0000", 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0089000000000002e-05 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:35 4.e 4:773b934b:::smithi15736696-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 158.00443184900001, 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50344320499999995, 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.070 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:39:50.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:39:50.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054564+0000", 2024-09-16T03:39:50.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5710000000000002e-06 2024-09-16T03:39:50.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054569+0000", 2024-09-16T03:39:50.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0490000000000003e-06 2024-09-16T03:39:50.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054578+0000", 2024-09-16T03:39:50.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:39:50.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054639+0000", 2024-09-16T03:39:50.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1061000000000004e-05 2024-09-16T03:39:50.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054644+0000", 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999996e-06 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155607+0000", 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100963372 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155652+0000", 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:39:50.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155722+0000", 2024-09-16T03:39:50.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0130999999999997e-05 2024-09-16T03:39:50.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.157921+0000", 2024-09-16T03:39:50.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021990159999999998 2024-09-16T03:39:50.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255940+0000", 2024-09-16T03:39:50.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098018569999999999 2024-09-16T03:39:50.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557930+0000", 2024-09-16T03:39:50.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.301990333 2024-09-16T03:39:50.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557963+0000", 2024-09-16T03:39:50.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2348000000000003e-05 2024-09-16T03:39:50.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558003+0000", 2024-09-16T03:39:50.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:39:50.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.077 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.077 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:36 4.e 4:773b934b:::smithi15736696-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:50.077 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:39:50.077 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 158.00439316000001, 2024-09-16T03:39:50.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50353811800000003, 2024-09-16T03:39:50.077 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:39:50.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054599+0000", 2024-09-16T03:39:50.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:39:50.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054601+0000", 2024-09-16T03:39:50.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4360000000000001e-06 2024-09-16T03:39:50.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054606+0000", 2024-09-16T03:39:50.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1019999999999996e-06 2024-09-16T03:39:50.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054610+0000", 2024-09-16T03:39:50.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.681e-06 2024-09-16T03:39:50.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054662+0000", 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1545000000000002e-05 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054664+0000", 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1919999999999999e-06 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155903+0000", 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101239068 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:50.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155970+0000", 2024-09-16T03:39:50.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6934999999999995e-05 2024-09-16T03:39:50.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558029+0000", 2024-09-16T03:39:50.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40205888899999997 2024-09-16T03:39:50.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558078+0000", 2024-09-16T03:39:50.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9842999999999997e-05 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558136+0000", 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7985000000000001e-05 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:90 3.0 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 114.502379991, 2024-09-16T03:39:50.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62288137399999999, 2024-09-16T03:39:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:39:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:39:50.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556614+0000", 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3439999999999999e-06 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556621+0000", 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7440000000000001e-06 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.085 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556632+0000", 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1725e-05 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556640+0000", 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0239999999999997e-06 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556709+0000", 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8787000000000004e-05 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556718+0000", 2024-09-16T03:39:50.086 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6920000000000008e-06 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758133+0000", 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201414977 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758273+0000", 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140684 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:50.087 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758479+0000", 2024-09-16T03:39:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020508599999999999 2024-09-16T03:39:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862766+0000", 2024-09-16T03:39:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104287952 2024-09-16T03:39:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.878363+0000", 2024-09-16T03:39:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015596601999999999 2024-09-16T03:39:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.088 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960000+0000", 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081636630000000002 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960162+0000", 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000162597 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179366+0000", 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21920357100000001 2024-09-16T03:39:50.089 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179411+0000", 2024-09-16T03:39:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5461999999999999e-05 2024-09-16T03:39:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179493+0000", 2024-09-16T03:39:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1496999999999996e-05 2024-09-16T03:39:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.091 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:91 3.0 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:39:50.091 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:39:50.091 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 114.50233503299999, 2024-09-16T03:39:50.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70460932600000004, 2024-09-16T03:39:50.091 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.091 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.091 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.091 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.091 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.091 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.091 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:39:50.091 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:39:50.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.092 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556657+0000", 2024-09-16T03:39:50.092 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:39:50.092 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.092 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.092 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556663+0000", 2024-09-16T03:39:50.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6579999999999999e-06 2024-09-16T03:39:50.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556670+0000", 2024-09-16T03:39:50.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3189999999999997e-06 2024-09-16T03:39:50.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.093 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.093 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.093 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556675+0000", 2024-09-16T03:39:50.093 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000002e-06 2024-09-16T03:39:50.093 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656929+0000", 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10025371299999999 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656944+0000", 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5460000000000001e-05 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861436+0000", 2024-09-16T03:39:50.094 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.204491969 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861676+0000", 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024051899999999999 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861828+0000", 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015110199999999999 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.095 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862775+0000", 2024-09-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094718600000000003 2024-09-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079130+0000", 2024-09-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21635549200000001 2024-09-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179557+0000", 2024-09-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10042638399999999 2024-09-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.096 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179972+0000", 2024-09-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041574 2024-09-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261034+0000", 2024-09-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081061604999999995 2024-09-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261142+0000", 2024-09-16T03:39:50.097 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107688 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261266+0000", 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124082 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:94 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 114.502308908, 2024-09-16T03:39:50.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70479513699999996, 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556685+0000", 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4279999999999999e-06 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556689+0000", 2024-09-16T03:39:50.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.709e-06 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556693+0000", 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2939999999999999e-06 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657036+0000", 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10034209600000001 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:50.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657054+0000", 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8162999999999999e-05 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862279+0000", 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20522560200000001 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862380+0000", 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101178 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862509+0000", 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128909 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.863063+0000", 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055327600000000001 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079172+0000", 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21610910899999999 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179932+0000", 2024-09-16T03:39:50.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100760575 2024-09-16T03:39:50.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179997+0000", 2024-09-16T03:39:50.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5018999999999995e-05 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261318+0000", 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081320838000000006 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261385+0000", 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7115999999999999e-05 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261478+0000", 2024-09-16T03:39:50.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2360000000000003e-05 2024-09-16T03:39:50.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.106 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.106 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.106 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1742 3.0 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head [write 0~1659392] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T03:39:50.106 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:39:50.106 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 114.502280508, 2024-09-16T03:39:50.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62317617400000003, 2024-09-16T03:39:50.106 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.106 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.106 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1742 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:39:50.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556713+0000", 2024-09-16T03:39:50.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7209999999999999e-06 2024-09-16T03:39:50.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558531+0000", 2024-09-16T03:39:50.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018184480000000001 2024-09-16T03:39:50.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558539+0000", 2024-09-16T03:39:50.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0830000000000004e-06 2024-09-16T03:39:50.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558546+0000", 2024-09-16T03:39:50.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3369999999999996e-06 2024-09-16T03:39:50.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.109 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.109 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759148+0000", 2024-09-16T03:39:50.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20060235500000001 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759230+0000", 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2391000000000004e-05 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759329+0000", 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9147000000000005e-05 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862771+0000", 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103441085 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960102+0000", 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097331259000000003 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960218+0000", 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011581100000000001 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079074+0000", 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118856436 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179758+0000", 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100683884 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179800+0000", 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1928000000000002e-05 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179887+0000", 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7288999999999997e-05 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:89 3.0 3:1c07e18b:::d3cef63b99e391058a88d444a56984ba6fc25b22efbc98e24ba274246c542d3c:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.722878012, 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50388158999999999, 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.113 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.114 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.114 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:39:50.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:39:50.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336115+0000", 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.866e-06 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336123+0000", 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6990000000000003e-06 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336136+0000", 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3342e-05 2024-09-16T03:39:50.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336143+0000", 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2099999999999998e-06 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336214+0000", 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1538000000000005e-05 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336342+0000", 2024-09-16T03:39:50.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127408 2024-09-16T03:39:50.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:50.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336593+0000", 2024-09-16T03:39:50.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025150500000000001 2024-09-16T03:39:50.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337284+0000", 2024-09-16T03:39:50.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069060900000000001 2024-09-16T03:39:50.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338068+0000", 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078438900000000003 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338107+0000", 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8858e-05 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538164+0000", 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200056971 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839843+0000", 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30167921599999997 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839905+0000", 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1542000000000001e-05 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839995+0000", 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0437000000000002e-05 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.119 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:98 3.0 3:04dfaaf9:::3de53e1d584ced9b0f970855867dbc9a14b3fc90962aa20a81c1ed25ff4b961f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.721550111, 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70310345299999999, 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337442+0000", 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0729999999999999e-06 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337447+0000", 2024-09-16T03:39:50.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3259999999999997e-06 2024-09-16T03:39:50.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337457+0000", 2024-09-16T03:39:50.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-09-16T03:39:50.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337462+0000", 2024-09-16T03:39:50.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0780000000000003e-06 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337511+0000", 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337574+0000", 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3114999999999995e-05 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337723+0000", 2024-09-16T03:39:50.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014882299999999999 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.339039+0000", 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001315524 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538225+0000", 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19918607199999999 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638838+0000", 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100613466 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.739726+0000", 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100887854 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040319+0000", 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30059265699999999 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040398+0000", 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8852999999999999e-05 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040545+0000", 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147236 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.126 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:101 3.0 3:090a0a71:::0ecec3e4e81d718f460a9826fb4cef2398bf4c951ee653be84827f33946eef5f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.622379365, 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60435370600000005, 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:39:50.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436614+0000", 2024-09-16T03:39:50.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.844e-06 2024-09-16T03:39:50.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436623+0000", 2024-09-16T03:39:50.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0739999999999999e-06 2024-09-16T03:39:50.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436636+0000", 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2629999999999999e-05 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436643+0000", 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7489999999999998e-06 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536906+0000", 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10026262599999999 2024-09-16T03:39:50.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536979+0000", 2024-09-16T03:39:50.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3311000000000003e-05 2024-09-16T03:39:50.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:50.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537109+0000", 2024-09-16T03:39:50.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013015800000000001 2024-09-16T03:39:50.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538241+0000", 2024-09-16T03:39:50.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001131726 2024-09-16T03:39:50.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638904+0000", 2024-09-16T03:39:50.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066240799999999 2024-09-16T03:39:50.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.940200+0000", 2024-09-16T03:39:50.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30129597299999999 2024-09-16T03:39:50.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040604+0000", 2024-09-16T03:39:50.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040478999999999 2024-09-16T03:39:50.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040759+0000", 2024-09-16T03:39:50.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015445500000000001 2024-09-16T03:39:50.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040824+0000", 2024-09-16T03:39:50.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5633e-05 2024-09-16T03:39:50.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040966+0000", 2024-09-16T03:39:50.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014132899999999999 2024-09-16T03:39:50.133 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.133 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.133 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.133 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:102 3.0 3:17742417:::632266e231c4c385ace473a8f4464285cf620fbc8799f68869353e2aa4660567:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:50.133 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:39:50.133 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.622332539, 2024-09-16T03:39:50.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.604540823, 2024-09-16T03:39:50.134 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.134 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.134 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.134 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:39:50.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436660+0000", 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436663+0000", 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1599999999999998e-06 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436670+0000", 2024-09-16T03:39:50.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.742e-06 2024-09-16T03:39:50.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436675+0000", 2024-09-16T03:39:50.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3839999999999996e-06 2024-09-16T03:39:50.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537499+0000", 2024-09-16T03:39:50.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100823987 2024-09-16T03:39:50.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537602+0000", 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102837 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537793+0000", 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019123499999999999 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638962+0000", 2024-09-16T03:39:50.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101168944 2024-09-16T03:39:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.840035+0000", 2024-09-16T03:39:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201073481 2024-09-16T03:39:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040706+0000", 2024-09-16T03:39:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200670606 2024-09-16T03:39:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041009+0000", 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030271600000000002 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041087+0000", 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8310000000000001e-05 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041127+0000", 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041200+0000", 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2381999999999999e-05 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:103 3.0 3:28447854:::c811732c201b76a96e1532c4d9a18d4f4f2a8ddec3bf79945689a1a62c8a5a25:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.62228729, 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60467526100000002, 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.140 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436705+0000", 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.23e-07 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436719+0000", 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3957999999999999e-05 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436725+0000", 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6100000000000002e-06 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436730+0000", 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0609999999999997e-06 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538257+0000", 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10152739500000001 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538314+0000", 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7355000000000002e-05 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538435+0000", 2024-09-16T03:39:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120967 2024-09-16T03:39:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638968+0000", 2024-09-16T03:39:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100532994 2024-09-16T03:39:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040662+0000", 2024-09-16T03:39:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40169386299999998 2024-09-16T03:39:50.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041059+0000", 2024-09-16T03:39:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039723900000000002 2024-09-16T03:39:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041247+0000", 2024-09-16T03:39:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018756900000000001 2024-09-16T03:39:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041286+0000", 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8971999999999999e-05 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041318+0000", 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1628999999999998e-05 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041379+0000", 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1926000000000003e-05 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2473 4.e 4:765e9179:::smithi15736696-44:head [write 1830877~681726] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:39:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 94.048302065000001, 2024-09-16T03:39:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50285070099999996, 2024-09-16T03:39:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.147 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2473 2024-09-16T03:39:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:39:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:39:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010691+0000", 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-06 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011744+0000", 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001052486 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011755+0000", 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1073000000000001e-05 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011761+0000", 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1600000000000003e-06 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011804+0000", 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2697999999999997e-05 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011885+0000", 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.119e-05 2024-09-16T03:39:50.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.012013+0000", 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128066 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.013923+0000", 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019104739999999999 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.412920+0000", 2024-09-16T03:39:50.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39899685200000001 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513347+0000", 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100427167 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513490+0000", 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142163 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513540+0000", 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0482000000000002e-05 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:137 3.0 3:20b08a9e:::0ba23f501764ab3447cc323c825ee8bff84c3c6e340c68c08117252e1f0b39bb:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 93.944453581000005, 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50256677599999999, 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.153 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:39:50.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1200000000000003e-07 2024-09-16T03:39:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114542+0000", 2024-09-16T03:39:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7400000000000002e-06 2024-09-16T03:39:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114547+0000", 2024-09-16T03:39:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.42e-06 2024-09-16T03:39:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114550+0000", 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4470000000000002e-06 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115951+0000", 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014005459999999999 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116047+0000", 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6899999999999997e-05 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116270+0000", 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022291699999999999 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.117236+0000", 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096522299999999997 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.218932+0000", 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10169644799999999 2024-09-16T03:39:50.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414271+0000", 2024-09-16T03:39:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19533916300000001 2024-09-16T03:39:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.514717+0000", 2024-09-16T03:39:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100445879 2024-09-16T03:39:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616919+0000", 2024-09-16T03:39:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10220199100000001 2024-09-16T03:39:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616980+0000", 2024-09-16T03:39:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0977999999999998e-05 2024-09-16T03:39:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.617105+0000", 2024-09-16T03:39:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012461199999999999 2024-09-16T03:39:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.159 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3402 4.1b 4:dbb349e8:::smithi15736696-46:head [stat,delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T03:39:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:46.357580+0000", 2024-09-16T03:39:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.701411649999997, 2024-09-16T03:39:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.80350077600000003, 2024-09-16T03:39:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3402 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357580+0000", 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357580+0000", 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357582+0000", 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1890000000000001e-06 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357588+0000", 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.896e-06 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357606+0000", 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7841000000000001e-05 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357617+0000", 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1698999999999999e-05 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357697+0000", 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9822999999999998e-05 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357708+0000", 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1119e-05 2024-09-16T03:39:50.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.962316+0000", 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60460744899999996 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.962430+0000", 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011423 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,6,8,9,15", 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.962530+0000", 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100005 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063659+0000", 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101128564 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063691+0000", 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2413999999999998e-05 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063701+0000", 2024-09-16T03:39:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9180000000000006e-06 2024-09-16T03:39:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063733+0000", 2024-09-16T03:39:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2416000000000003e-05 2024-09-16T03:39:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063758+0000", 2024-09-16T03:39:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.421e-05 2024-09-16T03:39:50.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.160869+0000", 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097111571999999993 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.160922+0000", 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3025999999999997e-05 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.161081+0000", 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000158405 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4662 4.20 4:049ac94d:::smithi15736696-40:head [write 726168~704991] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.832696+0000", 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.226295819000001, 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52767523100000002, 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4662 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.832696+0000", 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.832696+0000", 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.832699+0000", 2024-09-16T03:39:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4599999999999999e-06 2024-09-16T03:39:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.833419+0000", 2024-09-16T03:39:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072038500000000004 2024-09-16T03:39:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.833441+0000", 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1135e-05 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.833460+0000", 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9296e-05 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.955818+0000", 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12235790100000001 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.955889+0000", 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1039999999999997e-05 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,2,8,10,15", 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.956001+0000", 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111692 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.056147+0000", 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100146179 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.056166+0000", 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9656999999999998e-05 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.056217+0000", 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0238999999999998e-05 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.057817+0000", 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001600193 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360100+0000", 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30228278800000002 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360244+0000", 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014397800000000001 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360289+0000", 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5374999999999998e-05 2024-09-16T03:39:50.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360371+0000", 2024-09-16T03:39:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1913e-05 2024-09-16T03:39:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4663 4.20 4:049ac94d:::smithi15736696-40:head [write 1895071~683474] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.838668+0000", 2024-09-16T03:39:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.220323072999999, 2024-09-16T03:39:50.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52198192799999998, 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4663 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.838668+0000", 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.838668+0000", 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.838670+0000", 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.401e-06 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839234+0000", 2024-09-16T03:39:50.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056420899999999998 2024-09-16T03:39:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839248+0000", 2024-09-16T03:39:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4208e-05 2024-09-16T03:39:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839255+0000", 2024-09-16T03:39:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.742e-06 2024-09-16T03:39:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.057859+0000", 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21860384799999999 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.057934+0000", 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5638999999999999e-05 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,2,8,10,15", 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.058042+0000", 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107314 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.059148+0000", 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011060079999999999 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159745+0000", 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10059675 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159780+0000", 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5351000000000001e-05 2024-09-16T03:39:50.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.259887+0000", 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10010670200000001 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360172+0000", 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100285759 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360535+0000", 2024-09-16T03:39:50.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036267599999999998 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360570+0000", 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5138999999999997e-05 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360650+0000", 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0181999999999999e-05 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4664 4.20 4:049ac94d:::smithi15736696-40:head [setxattr _header (56) in=7b,truncate 3126168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.839266+0000", 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.219725943, 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52151806899999997, 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4664 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839266+0000", 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839266+0000", 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839266+0000", 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839270+0000", 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2100000000000002e-06 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839273+0000", 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.715e-06 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839276+0000", 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2200000000000001e-06 2024-09-16T03:39:50.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159242+0000", 2024-09-16T03:39:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31996554700000002 2024-09-16T03:39:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159311+0000", 2024-09-16T03:39:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9419999999999996e-05 2024-09-16T03:39:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,2,8,10,15", 2024-09-16T03:39:50.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159423+0000", 2024-09-16T03:39:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011188600000000001 2024-09-16T03:39:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360380+0000", 2024-09-16T03:39:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20095712900000001 2024-09-16T03:39:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360414+0000", 2024-09-16T03:39:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3176000000000002e-05 2024-09-16T03:39:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360449+0000", 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4872000000000001e-05 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360479+0000", 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0402000000000001e-05 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360509+0000", 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9907999999999999e-05 2024-09-16T03:39:50.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360685+0000", 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175897 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360717+0000", 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2342999999999998e-05 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360784+0000", 2024-09-16T03:39:50.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6514999999999996e-05 2024-09-16T03:39:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4665 4.20 4:049ac94d:::smithi15736696-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.839281+0000", 2024-09-16T03:39:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.219710696, 2024-09-16T03:39:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.82253416700000004, 2024-09-16T03:39:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.190 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4665 2024-09-16T03:39:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839281+0000", 2024-09-16T03:39:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839281+0000", 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839281+0000", 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3299999999999998e-07 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839283+0000", 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6470000000000001e-06 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839286+0000", 2024-09-16T03:39:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.506e-06 2024-09-16T03:39:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839290+0000", 2024-09-16T03:39:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9559999999999999e-06 2024-09-16T03:39:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.661604+0000", 2024-09-16T03:39:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.822313819 2024-09-16T03:39:50.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.661677+0000", 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2712000000000004e-05 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.661815+0000", 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013819400000000001 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4857 4.e 4:70297eed:::smithi15736696-31:head [assert-version v93,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e165)", 2024-09-16T03:39:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.036628+0000", 2024-09-16T03:39:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.022363527, 2024-09-16T03:39:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.54220494600000002, 2024-09-16T03:39:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4857 2024-09-16T03:39:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.195 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036628+0000", 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036628+0000", 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036629+0000", 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.502e-06 2024-09-16T03:39:50.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036635+0000", 2024-09-16T03:39:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5430000000000001e-06 2024-09-16T03:39:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036646+0000", 2024-09-16T03:39:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0886999999999999e-05 2024-09-16T03:39:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036654+0000", 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0839999999999992e-06 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036701+0000", 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036765+0000", 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4039000000000001e-05 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:39:50.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.178117+0000", 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.141351912 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.179109+0000", 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00099158100000000006 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.379230+0000", 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20012068599999999 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.578763+0000", 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19953386300000001 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.578801+0000", 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7880999999999999e-05 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.578833+0000", 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1608e-05 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.200 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.201 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.201 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:50.202 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:50.202 INFO:tasks.rados.rados.0.smithi157.stdout:1985: expect (ObjNum 429 snap 0 seq_num 429) 2024-09-16T03:39:50.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:39:50.203 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:50.200+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:50.303 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:50.300+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:50.314 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:50.351 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:50.351 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:50.351 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:39:50.351 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:50.360 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:39:50.398 INFO:tasks.rados.rados.0.smithi157.stdout:1979: finishing write tid 4 to smithi15736696-15 2024-09-16T03:39:50.398 INFO:tasks.rados.rados.0.smithi157.stdout:1979: finishing write tid 5 to smithi15736696-15 2024-09-16T03:39:50.398 INFO:tasks.rados.rados.0.smithi157.stdout:1979: oid 15 updating version 0 to 25 2024-09-16T03:39:50.398 INFO:tasks.rados.rados.0.smithi157.stdout:1979: oid 15 version 25 is already newer than 24 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:1979: oid 15 updating version 25 to 27 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:1979: oid 15 version 27 is already newer than 26 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 15 v 27 (ObjNum 442 snap 0 seq_num 442) dirty exists 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:1979: left oid 15 (ObjNum 442 snap 0 seq_num 442) 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:1979: done (9 left) 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:1985: done (8 left) 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:1986: done (7 left) 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:1993: read oid 27 snap -1 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:1993: expect (ObjNum 395 snap 0 seq_num 395) 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:1994: read oid 4 snap -1 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:1994: expect (ObjNum 411 snap 0 seq_num 411) 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:1995: read oid 44 snap -1 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:1995: expect (ObjNum 410 snap 0 seq_num 410) 2024-09-16T03:39:50.399 INFO:tasks.rados.rados.0.smithi157.stdout:1996: read oid 33 snap -1 2024-09-16T03:39:50.400 INFO:tasks.rados.rados.0.smithi157.stdout:1996: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-16T03:39:50.400 INFO:tasks.rados.rados.0.smithi157.stdout:1997: set_chunk oid 49 target oid 2024-09-16T03:39:50.404 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:50.401+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:50.468 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:50.468 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:50.468 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:50.468 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:50.468 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:50.476 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:39:50.505 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:50.502+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:50.512 INFO:tasks.rados.rados.0.smithi157.stdout:1997: set_chunk oid 49 offset: 1667584 length: 1217024 target oid 90126f8fa79ebc3fc3567008e3cb9d186b37bbdf19dc6c656f74ceb654c85402 offset: 0 snap -1 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout:1998: read oid 48 snap -1 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout:1998: expect (ObjNum 367 snap 0 seq_num 367) 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout:1999: read oid 45 snap -1 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout:1999: expect (ObjNum 325 snap 0 seq_num 325) 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout:2000: tier_evict oid 26 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout:2000: tier_evict oid 26 snap -1 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout:1982: finishing write tid 4 to smithi15736696-42 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout:1984: finishing write tid 4 to smithi15736696-22 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout:2001: read oid 6 snap -1 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout:2001: expect (ObjNum 327 snap 0 seq_num 327) 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout:1984: finishing write tid 5 to smithi15736696-22 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout:1984: oid 22 updating version 0 to 106 2024-09-16T03:39:50.513 INFO:tasks.rados.rados.0.smithi157.stdout:1984: oid 22 updating version 106 to 107 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout:1984: oid 22 updating version 107 to 108 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout:1984: oid 22 version 108 is already newer than 105 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 22 v 108 (ObjNum 444 snap 0 seq_num 444) dirty exists 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout:1984: left oid 22 (ObjNum 444 snap 0 seq_num 444) 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout:1984: done (15 left) 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout:2002: read oid 5 snap -1 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout:2002: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout:1987: finishing write tid 3 to smithi15736696-37 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout:1982: finishing write tid 5 to smithi15736696-42 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout:1982: oid 42 updating version 0 to 80 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout:1982: oid 42 updating version 80 to 82 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout:1982: oid 42 updating version 82 to 83 2024-09-16T03:39:50.514 INFO:tasks.rados.rados.0.smithi157.stdout:1982: oid 42 version 83 is already newer than 81 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 42 v 83 (ObjNum 443 snap 0 seq_num 443) dirty exists 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout:1982: left oid 42 (ObjNum 443 snap 0 seq_num 443) 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout:1987: finishing write tid 4 to smithi15736696-37 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout:1982: done (15 left) 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout:2003: read oid 40 snap -1 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout:2003: expect deleted 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout:1987: finishing write tid 5 to smithi15736696-37 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout:1987: oid 37 updating version 0 to 82 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout:1987: oid 37 updating version 82 to 84 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout:1987: oid 37 version 84 is already newer than 83 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout:1987: oid 37 updating version 84 to 85 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 37 v 85 (ObjNum 445 snap 0 seq_num 445) dirty exists 2024-09-16T03:39:50.515 INFO:tasks.rados.rados.0.smithi157.stdout:1987: left oid 37 (ObjNum 445 snap 0 seq_num 445) 2024-09-16T03:39:50.516 INFO:tasks.rados.rados.0.smithi157.stdout:1991: got (22) Invalid argument 2024-09-16T03:39:50.516 INFO:tasks.rados.rados.0.smithi157.stdout:1992: got (2) No such file or directory 2024-09-16T03:39:50.516 INFO:tasks.rados.rados.0.smithi157.stdout:1992: got expected ENOENT (src dne) 2024-09-16T03:39:50.516 INFO:tasks.rados.rados.0.smithi157.stdout:1990: expect (ObjNum 333 snap 0 seq_num 333) 2024-09-16T03:39:50.589 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:50.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:303 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:39:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 154.174836473, 2024-09-16T03:39:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50956986000000004, 2024-09-16T03:39:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.590 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 303 2024-09-16T03:39:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:39:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:39:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411721+0000", 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0200000000000001e-06 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411729+0000", 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1240000000000005e-06 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411744+0000", 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099000000000001e-05 2024-09-16T03:39:50.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411751+0000", 2024-09-16T03:39:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.029e-06 2024-09-16T03:39:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511892+0000", 2024-09-16T03:39:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100140879 2024-09-16T03:39:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511961+0000", 2024-09-16T03:39:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9379000000000001e-05 2024-09-16T03:39:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.613647+0000", 2024-09-16T03:39:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101685994 2024-09-16T03:39:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.614294+0000", 2024-09-16T03:39:50.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064692799999999996 2024-09-16T03:39:50.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.714962+0000", 2024-09-16T03:39:50.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066765 2024-09-16T03:39:50.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921157+0000", 2024-09-16T03:39:50.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206195463 2024-09-16T03:39:50.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921233+0000", 2024-09-16T03:39:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5446999999999998e-05 2024-09-16T03:39:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921288+0000", 2024-09-16T03:39:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5847999999999998e-05 2024-09-16T03:39:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.596 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:503 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 148.764761921, 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60521621599999997, 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 503 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821794+0000", 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-09-16T03:39:50.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821798+0000", 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2030000000000002e-06 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821804+0000", 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821810+0000", 2024-09-16T03:39:50.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4249999999999997e-06 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922212+0000", 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040228800000001 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922307+0000", 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5089999999999999e-05 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.126007+0000", 2024-09-16T03:39:50.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20369931499999999 2024-09-16T03:39:50.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.127605+0000", 2024-09-16T03:39:50.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001598213 2024-09-16T03:39:50.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.327263+0000", 2024-09-16T03:39:50.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199657905 2024-09-16T03:39:50.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426900+0000", 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099637217 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426963+0000", 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.427009+0000", 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6533000000000003e-05 2024-09-16T03:39:50.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1227 4.a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v18,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 133.91058565700001, 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.44160531400000003, 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1227 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675969+0000", 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675972+0000", 2024-09-16T03:39:50.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7269999999999999e-06 2024-09-16T03:39:50.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675981+0000", 2024-09-16T03:39:50.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.166e-06 2024-09-16T03:39:50.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.675996+0000", 2024-09-16T03:39:50.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4901e-05 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676005+0000", 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4579999999999993e-06 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676081+0000", 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6157999999999998e-05 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:36.676188+0000", 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106871 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.016594+0000", 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34040622300000001 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017089+0000", 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049523899999999997 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.017779+0000", 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069032200000000005 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117504+0000", 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099724207999999995 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117543+0000", 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9208999999999997e-05 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:37.117575+0000", 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1832000000000003e-05 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1523 4.a 4:50107449:::smithi15736696-27:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 125.223544331, 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.410522742, 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:50.609 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1523 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363011+0000", 2024-09-16T03:39:50.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363012+0000", 2024-09-16T03:39:50.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.269e-06 2024-09-16T03:39:50.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363016+0000", 2024-09-16T03:39:50.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8920000000000002e-06 2024-09-16T03:39:50.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.363027+0000", 2024-09-16T03:39:50.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1079e-05 2024-09-16T03:39:50.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463160+0000", 2024-09-16T03:39:50.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10013267100000001 2024-09-16T03:39:50.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463297+0000", 2024-09-16T03:39:50.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013717699999999999 2024-09-16T03:39:50.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.463381+0000", 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3701000000000004e-05 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:45.773533+0000", 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31015295300000001 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1690 4.a 4:51338165:::smithi15736696-4:head [assert-version v63,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 120.13627166400001, 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52002349699999995, 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1690 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450286+0000", 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-16T03:39:50.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450293+0000", 2024-09-16T03:39:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5089999999999996e-06 2024-09-16T03:39:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450308+0000", 2024-09-16T03:39:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4955e-05 2024-09-16T03:39:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450316+0000", 2024-09-16T03:39:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7079999999999997e-06 2024-09-16T03:39:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450386+0000", 2024-09-16T03:39:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9715999999999995e-05 2024-09-16T03:39:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450478+0000", 2024-09-16T03:39:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2610999999999998e-05 2024-09-16T03:39:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.665802+0000", 2024-09-16T03:39:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21532342700000001 2024-09-16T03:39:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668724+0000", 2024-09-16T03:39:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029220600000000002 2024-09-16T03:39:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668758+0000", 2024-09-16T03:39:50.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.434e-05 2024-09-16T03:39:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970243+0000", 2024-09-16T03:39:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30148501 2024-09-16T03:39:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970278+0000", 2024-09-16T03:39:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4965000000000002e-05 2024-09-16T03:39:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970307+0000", 2024-09-16T03:39:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8968000000000001e-05 2024-09-16T03:39:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1654 4.12 4:4a9d235d:::smithi15736696-1:head [write 652138~461367 in=461367b,stat out=16b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:39:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 120.044518173, 2024-09-16T03:39:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42790198699999998, 2024-09-16T03:39:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.620 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1654 2024-09-16T03:39:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:39:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542037+0000", 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542039+0000", 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0250000000000001e-06 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542682+0000", 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064305099999999995 2024-09-16T03:39:50.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542698+0000", 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6469000000000001e-05 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542710+0000", 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1173e-05 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542767+0000", 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7584000000000003e-05 2024-09-16T03:39:50.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542775+0000", 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6149999999999999e-06 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666597+0000", 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.123821815 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666745+0000", 2024-09-16T03:39:50.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148675 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.666935+0000", 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018963899999999999 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969207+0000", 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302272031 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969374+0000", 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016686699999999999 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969533+0000", 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015909299999999999 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969560+0000", 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7148000000000002e-05 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969631+0000", 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1174000000000007e-05 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969685+0000", 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3613000000000003e-05 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969764+0000", 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9281000000000002e-05 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969939+0000", 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017473400000000001 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1655 4.12 4:4a9d235d:::smithi15736696-1:head [write 1574940~677198 in=677198b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 120.043826055, 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42773147700000003, 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.628 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1655 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542729+0000", 2024-09-16T03:39:50.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.542730+0000", 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.085e-06 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543338+0000", 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060826100000000002 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543351+0000", 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2789000000000001e-05 2024-09-16T03:39:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543360+0000", 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1970000000000002e-06 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543420+0000", 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9562999999999999e-05 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543431+0000", 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0866999999999999e-05 2024-09-16T03:39:50.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.768963+0000", 2024-09-16T03:39:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.225532487 2024-09-16T03:39:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769029+0000", 2024-09-16T03:39:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5881000000000002e-05 2024-09-16T03:39:50.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:39:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.769134+0000", 2024-09-16T03:39:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104923 2024-09-16T03:39:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969639+0000", 2024-09-16T03:39:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20050517900000001 2024-09-16T03:39:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.969978+0000", 2024-09-16T03:39:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00033848599999999998 2024-09-16T03:39:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970206+0000", 2024-09-16T03:39:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022867300000000001 2024-09-16T03:39:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970248+0000", 2024-09-16T03:39:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1433999999999997e-05 2024-09-16T03:39:50.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970299+0000", 2024-09-16T03:39:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0677999999999998e-05 2024-09-16T03:39:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970338+0000", 2024-09-16T03:39:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9646e-05 2024-09-16T03:39:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970377+0000", 2024-09-16T03:39:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8439999999999998e-05 2024-09-16T03:39:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970460+0000", 2024-09-16T03:39:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3887999999999994e-05 2024-09-16T03:39:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1656 4.12 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2252138] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:50.636 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:39:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 120.043179983, 2024-09-16T03:39:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72904596899999996, 2024-09-16T03:39:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1656 2024-09-16T03:39:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543376+0000", 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4200000000000004e-07 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543383+0000", 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6290000000000004e-06 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543390+0000", 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543396+0000", 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.02e-06 2024-09-16T03:39:50.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543461+0000", 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4876999999999996e-05 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543465+0000", 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2139999999999998e-06 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970490+0000", 2024-09-16T03:39:50.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42702478300000002 2024-09-16T03:39:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970556+0000", 2024-09-16T03:39:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6137999999999996e-05 2024-09-16T03:39:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:39:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970670+0000", 2024-09-16T03:39:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114416 2024-09-16T03:39:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971375+0000", 2024-09-16T03:39:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070496000000000003 2024-09-16T03:39:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071773+0000", 2024-09-16T03:39:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100397945 2024-09-16T03:39:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071823+0000", 2024-09-16T03:39:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:39:50.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.072067+0000", 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243856 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.171991+0000", 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099924052999999999 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272176+0000", 2024-09-16T03:39:50.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100184963 2024-09-16T03:39:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272251+0000", 2024-09-16T03:39:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5902000000000006e-05 2024-09-16T03:39:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272421+0000", 2024-09-16T03:39:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016962800000000001 2024-09-16T03:39:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1657 4.12 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 120.04314799700001, 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72934181899999995, 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1657 2024-09-16T03:39:50.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543408+0000", 2024-09-16T03:39:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.13e-07 2024-09-16T03:39:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543410+0000", 2024-09-16T03:39:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7030000000000002e-06 2024-09-16T03:39:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543417+0000", 2024-09-16T03:39:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.567e-06 2024-09-16T03:39:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543423+0000", 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0619999999999997e-06 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543489+0000", 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6315999999999993e-05 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543498+0000", 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7239999999999998e-06 2024-09-16T03:39:50.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970952+0000", 2024-09-16T03:39:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42745424399999998 2024-09-16T03:39:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971012+0000", 2024-09-16T03:39:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9771000000000001e-05 2024-09-16T03:39:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272463+0000", 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30145090800000002 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272558+0000", 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4863000000000003e-05 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272749+0000", 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191048 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2106 4.a 4:51338165:::smithi15736696-4:head [sparse-read 0~887916] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 105.537623044, 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40254453200000001, 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2106 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048932+0000", 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048933+0000", 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:39:50.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048936+0000", 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3890000000000001e-06 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048943+0000", 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1250000000000004e-06 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048953+0000", 2024-09-16T03:39:50.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8190000000000003e-06 2024-09-16T03:39:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149113+0000", 2024-09-16T03:39:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100160007 2024-09-16T03:39:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149118+0000", 2024-09-16T03:39:50.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7650000000000001e-06 2024-09-16T03:39:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451341+0000", 2024-09-16T03:39:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30222344699999998 2024-09-16T03:39:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451381+0000", 2024-09-16T03:39:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9517000000000001e-05 2024-09-16T03:39:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451477+0000", 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5656999999999996e-05 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2107 4.a 4:51338165:::smithi15736696-4:head [read 1698310~486600,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 105.537592981, 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40288642400000002, 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.658 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2107 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048962+0000", 2024-09-16T03:39:50.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048963+0000", 2024-09-16T03:39:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4000000000000002e-07 2024-09-16T03:39:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048967+0000", 2024-09-16T03:39:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9060000000000004e-06 2024-09-16T03:39:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048972+0000", 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5910000000000003e-06 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.048977+0000", 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9629999999999997e-06 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149135+0000", 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015828 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.149138+0000", 2024-09-16T03:39:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.779e-06 2024-09-16T03:39:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451506+0000", 2024-09-16T03:39:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.302367891 2024-09-16T03:39:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451558+0000", 2024-09-16T03:39:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2374999999999999e-05 2024-09-16T03:39:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:05.451849+0000", 2024-09-16T03:39:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029009899999999999 2024-09-16T03:39:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2987 4.a 4:5a5e9bc2:::smithi15736696-36:head [write 1607751~551752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:39:50.663 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:39:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 78.667607492000002, 2024-09-16T03:39:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49279943700000001, 2024-09-16T03:39:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2987 2024-09-16T03:39:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918949+0000", 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.86e-06 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919579+0000", 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062909000000000001 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919591+0000", 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2078e-05 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919596+0000", 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:39:50.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019705+0000", 2024-09-16T03:39:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100108405 2024-09-16T03:39:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019754+0000", 2024-09-16T03:39:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9258999999999997e-05 2024-09-16T03:39:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:50.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019847+0000", 2024-09-16T03:39:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2954999999999993e-05 2024-09-16T03:39:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.023003+0000", 2024-09-16T03:39:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031561110000000001 2024-09-16T03:39:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.122544+0000", 2024-09-16T03:39:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099540553000000004 2024-09-16T03:39:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411658+0000", 2024-09-16T03:39:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.28911426699999998 2024-09-16T03:39:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411696+0000", 2024-09-16T03:39:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8430000000000003e-05 2024-09-16T03:39:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411747+0000", 2024-09-16T03:39:50.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0745999999999998e-05 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2988 4.a 4:5a5e9bc2:::smithi15736696-36:head [setxattr _header (56) in=7b,truncate 2487469] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 78.666945752999993, 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59236646500000001, 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.670 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2988 2024-09-16T03:39:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:39:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:39:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919610+0000", 2024-09-16T03:39:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.048e-06 2024-09-16T03:39:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919615+0000", 2024-09-16T03:39:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8239999999999999e-06 2024-09-16T03:39:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919621+0000", 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4380000000000003e-06 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919626+0000", 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2000000000000002e-06 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.020922+0000", 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101295782 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.020995+0000", 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3256999999999993e-05 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.021088+0000", 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3419999999999997e-05 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.023007+0000", 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019186380000000001 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.222530+0000", 2024-09-16T03:39:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19952276899999999 2024-09-16T03:39:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511887+0000", 2024-09-16T03:39:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.289357378 2024-09-16T03:39:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511927+0000", 2024-09-16T03:39:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0357e-05 2024-09-16T03:39:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511976+0000", 2024-09-16T03:39:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8353999999999998e-05 2024-09-16T03:39:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3258 4.1a 4:5e3c0760:::smithi15736696-24:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:39:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:39:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.156494878000004, 2024-09-16T03:39:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3302292470000001, 2024-09-16T03:39:50.676 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3258 2024-09-16T03:39:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:39:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430061+0000", 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.27e-07 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430062+0000", 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6780000000000001e-06 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430066+0000", 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5269999999999999e-06 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430069+0000", 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430119+0000", 2024-09-16T03:39:50.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0212e-05 2024-09-16T03:39:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:39:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430122+0000", 2024-09-16T03:39:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7319999999999998e-06 2024-09-16T03:39:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.631493+0000", 2024-09-16T03:39:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20137142499999999 2024-09-16T03:39:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.631579+0000", 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5418000000000001e-05 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:41.760289+0000", 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1287106680000001 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3701 4.a 4:51338165:::smithi15736696-4:head [assert-version v115,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e121)", 2024-09-16T03:39:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:39:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.215648338999998, 2024-09-16T03:39:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50414302200000005, 2024-09-16T03:39:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3701 2024-09-16T03:39:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370910+0000", 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1200000000000002e-06 2024-09-16T03:39:50.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370923+0000", 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3078000000000001e-05 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370940+0000", 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7461000000000001e-05 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370950+0000", 2024-09-16T03:39:50.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0091999999999999e-05 2024-09-16T03:39:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.371029+0000", 2024-09-16T03:39:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8879000000000002e-05 2024-09-16T03:39:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.371136+0000", 2024-09-16T03:39:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106983 2024-09-16T03:39:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.573316+0000", 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20217934500000001 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.573914+0000", 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059857900000000002 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.763374+0000", 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18945932200000001 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.874936+0000", 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111562256 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.875002+0000", 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6050000000000006e-05 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.875050+0000", 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7856999999999999e-05 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4258 4.a 4:50107449:::smithi15736696-27:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-16T03:39:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:16.528667+0000", 2024-09-16T03:39:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.057888128000002, 2024-09-16T03:39:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.47672299499999998, 2024-09-16T03:39:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.688 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4258 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528667+0000", 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528667+0000", 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528669+0000", 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.883e-06 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528674+0000", 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2599999999999996e-06 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528687+0000", 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3220999999999999e-05 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528695+0000", 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528754+0000", 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9555000000000002e-05 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528864+0000", 2024-09-16T03:39:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010949 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:17.005390+0000", 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.47652628499999999 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4368 4.a 4:51338165:::smithi15736696-4:head [write 773221~492056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:20.368068+0000", 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.218487456999998, 2024-09-16T03:39:50.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41474344400000002, 2024-09-16T03:39:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4368 2024-09-16T03:39:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.368068+0000", 2024-09-16T03:39:50.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.368068+0000", 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.368070+0000", 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3829999999999999e-06 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.368716+0000", 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064629800000000001 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.368734+0000", 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7453999999999999e-05 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.368742+0000", 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8159999999999994e-06 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374166+0000", 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054248109999999999 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374171+0000", 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9060000000000001e-06 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.678206+0000", 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30403448500000002 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.678318+0000", 2024-09-16T03:39:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112181 2024-09-16T03:39:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.678441+0000", 2024-09-16T03:39:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122683 2024-09-16T03:39:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782676+0000", 2024-09-16T03:39:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104235049 2024-09-16T03:39:50.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782690+0000", 2024-09-16T03:39:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4796e-05 2024-09-16T03:39:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782734+0000", 2024-09-16T03:39:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3662999999999999e-05 2024-09-16T03:39:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782776+0000", 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1495999999999997e-05 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782811+0000", 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5423000000000003e-05 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4369 4.a 4:51338165:::smithi15736696-4:head [write 1816008~503126] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:20.373551+0000", 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.213004363, 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.75753581699999994, 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4369 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.373551+0000", 2024-09-16T03:39:50.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.373551+0000", 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.373552+0000", 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2550000000000001e-06 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374063+0000", 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00051145499999999996 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374077+0000", 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3535e-05 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374087+0000", 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0086e-05 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374194+0000", 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010661 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374196+0000", 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7980000000000001e-06 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.780445+0000", 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40624829800000001 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.780585+0000", 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014048700000000001 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.780748+0000", 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000162798 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782933+0000", 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002184994 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.030573+0000", 2024-09-16T03:39:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.247640049 2024-09-16T03:39:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.130869+0000", 2024-09-16T03:39:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100296195 2024-09-16T03:39:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131031+0000", 2024-09-16T03:39:50.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016220899999999999 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131087+0000", 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5047999999999999e-05 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4370 4.a 4:51338165:::smithi15736696-4:head [setxattr _header (55) in=7b,truncate 2773221] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:20.374103+0000", 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.212451844, 2024-09-16T03:39:50.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.757112648, 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4370 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374103+0000", 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374103+0000", 2024-09-16T03:39:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374104+0000", 2024-09-16T03:39:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.192e-06 2024-09-16T03:39:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374109+0000", 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7280000000000003e-06 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374117+0000", 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2039999999999997e-06 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374124+0000", 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3380000000000001e-06 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374212+0000", 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8663000000000001e-05 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374215+0000", 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3929999999999998e-06 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782160+0000", 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.407945112 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782254+0000", 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4152000000000003e-05 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782407+0000", 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015309500000000001 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.785130+0000", 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027231320000000001 2024-09-16T03:39:50.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.130717+0000", 2024-09-16T03:39:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34558722600000003 2024-09-16T03:39:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131123+0000", 2024-09-16T03:39:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040580199999999997 2024-09-16T03:39:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:50.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131164+0000", 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0487e-05 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131216+0000", 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2123999999999998e-05 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4371 4.a 4:51338165:::smithi15736696-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:20.374154+0000", 2024-09-16T03:39:50.713 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.212400944999999, 2024-09-16T03:39:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.75721231200000005, 2024-09-16T03:39:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4371 2024-09-16T03:39:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:50.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374154+0000", 2024-09-16T03:39:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374154+0000", 2024-09-16T03:39:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374155+0000", 2024-09-16T03:39:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4099999999999997e-07 2024-09-16T03:39:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374159+0000", 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.506e-06 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374167+0000", 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0870000000000007e-06 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374173+0000", 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.652e-06 2024-09-16T03:39:50.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.468957+0000", 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094784050999999994 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.468967+0000", 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6169999999999996e-06 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782856+0000", 2024-09-16T03:39:50.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.313889311 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782918+0000", 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2055000000000001e-05 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131245+0000", 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34832644699999998 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:50.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131301+0000", 2024-09-16T03:39:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5721999999999999e-05 2024-09-16T03:39:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131366+0000", 2024-09-16T03:39:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5922999999999999e-05 2024-09-16T03:39:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:50.719 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:50.719 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:50.721 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 3342062 2024-09-16T03:39:50.721 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:50.721 INFO:tasks.rados.rados.0.smithi157.stdout:1989: got (0) Success 2024-09-16T03:39:50.721 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 97 (ObjNum 359 snap 0 seq_num 359) dirty exists 2024-09-16T03:39:50.721 INFO:tasks.rados.rados.0.smithi157.stdout:1996: expect (ObjNum 328 snap 0 seq_num 328) 2024-09-16T03:39:50.721 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:50.722 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:50.718+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:50.722 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:39:50.745 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:50.786 INFO:tasks.rados.rados.0.smithi157.stdout:1993: expect (ObjNum 395 snap 0 seq_num 395) 2024-09-16T03:39:50.797 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:39:50.821 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:50.819+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:50.831 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:50.831 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:50.831 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:39:50.831 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:50.841 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:39:50.845 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 481379 2024-09-16T03:39:50.922 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:50.920+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:50.926 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:39:50.965 INFO:tasks.rados.rados.0.smithi157.stdout:1995: expect (ObjNum 410 snap 0 seq_num 410) 2024-09-16T03:39:50.965 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:50.990 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:39:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:39:50.990 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:39:50.990 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:50.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:39:51.023 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:51.021+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:51.053 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:51.053 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:51.071 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:51.071 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:51.078 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:51.088 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:51.088 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:51.105 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:51.105 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:51.112 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:39:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:39:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:39:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:39:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:790 4.d 4:b3fe05b2:::smithi15736696-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:51.112 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:39:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 145.22151955699999, 2024-09-16T03:39:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41456747799999999, 2024-09-16T03:39:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 790 2024-09-16T03:39:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887975+0000", 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0369999999999999e-06 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887987+0000", 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1606e-05 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888016+0000", 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.917e-05 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888032+0000", 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5616000000000001e-05 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888178+0000", 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612100000000001 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888307+0000", 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012953899999999999 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.302540+0000", 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41423238899999998 2024-09-16T03:39:51.116 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1418 4.c 4:3313e97c:::smithi15736696-21:head [write 615156~555834] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:39:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 129.883408385, 2024-09-16T03:39:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40891108799999998, 2024-09-16T03:39:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.117 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1418 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226087+0000", 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8890000000000004e-06 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226827+0000", 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000739298 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226848+0000", 2024-09-16T03:39:51.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1418000000000001e-05 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226858+0000", 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0434e-05 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327036+0000", 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10017769 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327098+0000", 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1760999999999994e-05 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327230+0000", 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131839 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434369+0000", 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107139788 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434444+0000", 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4171999999999994e-05 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634830+0000", 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20038660599999999 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634937+0000", 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010716499999999999 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634994+0000", 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7027999999999997e-05 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1421 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 129.870060236, 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.497492088, 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.123 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1421 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239432+0000", 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5999999999999991e-07 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239436+0000", 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239443+0000", 2024-09-16T03:39:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.968e-06 2024-09-16T03:39:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239447+0000", 2024-09-16T03:39:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6969999999999999e-06 2024-09-16T03:39:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635225+0000", 2024-09-16T03:39:51.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39577807399999998 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635290+0000", 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4583000000000002e-05 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635410+0000", 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120684 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:51.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636180+0000", 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076997899999999995 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636799+0000", 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061906600000000004 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736742+0000", 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099943041999999996 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736843+0000", 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010076799999999999 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736924+0000", 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0419000000000004e-05 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1422 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:39:51.129 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 129.870038956, 2024-09-16T03:39:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49764818700000002, 2024-09-16T03:39:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1422 2024-09-16T03:39:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239455+0000", 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9860000000000001e-06 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239461+0000", 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239464+0000", 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2619999999999999e-06 2024-09-16T03:39:51.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635736+0000", 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.396272287 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635790+0000", 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4252000000000002e-05 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736964+0000", 2024-09-16T03:39:51.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101173319 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737024+0000", 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9975000000000001e-05 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737101+0000", 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7303000000000002e-05 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2643 4.c 4:34492f38:::smithi15736696-33:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 90.228638207000003, 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50004631700000002, 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2643 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880854+0000", 2024-09-16T03:39:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6599999999999994e-07 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881441+0000", 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058613000000000003 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881450+0000", 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6260000000000007e-06 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881455+0000", 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.515e-06 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979409+0000", 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097954645000000007 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979512+0000", 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102701 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979684+0000", 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017176299999999999 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084699+0000", 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10501545599999999 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084714+0000", 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4738e-05 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185858+0000", 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101144074 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380720+0000", 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.194862072 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380783+0000", 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3262000000000005e-05 2024-09-16T03:39:51.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380900+0000", 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116369 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2644 4.c 4:34492f38:::smithi15736696-33:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 90.223961712000005, 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.495569859, 2024-09-16T03:39:51.141 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2644 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885531+0000", 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885813+0000", 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028188399999999999 2024-09-16T03:39:51.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885824+0000", 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885830+0000", 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6289999999999998e-06 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983163+0000", 2024-09-16T03:39:51.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097333127000000005 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983273+0000", 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109868 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983458+0000", 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018554899999999999 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:51.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084722+0000", 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101263244 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185717+0000", 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100995575 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185904+0000", 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186682 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380965+0000", 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.195060921 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381030+0000", 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5174999999999999e-05 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381100+0000", 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9877999999999997e-05 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.147 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2645 4.c 4:34492f38:::smithi15736696-33:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 90.223652211000001, 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59563304800000005, 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.148 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2645 2024-09-16T03:39:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:39:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:39:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885840+0000", 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885843+0000", 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.785e-06 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885848+0000", 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3979999999999998e-06 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885852+0000", 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6509999999999999e-06 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084768+0000", 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19891652000000001 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084831+0000", 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2717000000000003e-05 2024-09-16T03:39:51.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084964+0000", 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132731 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185772+0000", 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10080797399999999 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185790+0000", 2024-09-16T03:39:51.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7966999999999999e-05 2024-09-16T03:39:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185959+0000", 2024-09-16T03:39:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168777 2024-09-16T03:39:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481235+0000", 2024-09-16T03:39:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29527644800000002 2024-09-16T03:39:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481307+0000", 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1539999999999996e-05 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481473+0000", 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016595600000000001 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2646 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:39:51.154 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:39:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 90.223635521999995, 2024-09-16T03:39:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59596245699999995, 2024-09-16T03:39:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2646 2024-09-16T03:39:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885857+0000", 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885860+0000", 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8940000000000003e-06 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885864+0000", 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.602e-06 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885867+0000", 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0089999999999999e-06 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085196+0000", 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19932923399999999 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085287+0000", 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0480000000000001e-05 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481521+0000", 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39623439799999999 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481613+0000", 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2276999999999998e-05 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481819+0000", 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020521499999999999 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.159 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3554 4.c 4.3e97c8cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.659462497, 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2069473150000001, 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3554 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.160 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:39:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:39:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450032+0000", 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9639999999999999e-06 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450041+0000", 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450059+0000", 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450069+0000", 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8320000000000001e-06 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.467620+0000", 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017550676000000001 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.467626+0000", 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3430000000000001e-06 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656941+0000", 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1893151820000001 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656977+0000", 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5491999999999998e-05 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.164 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3581 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.083955684000003, 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.63145314900000005, 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3581 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025539+0000", 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4510000000000001e-06 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026168+0000", 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062808199999999997 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026186+0000", 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8694000000000001e-05 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026196+0000", 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8800000000000003e-06 2024-09-16T03:39:51.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035407+0000", 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092111020000000005 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035416+0000", 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9870000000000005e-06 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.557740+0000", 2024-09-16T03:39:51.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.522323918 2024-09-16T03:39:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656989+0000", 2024-09-16T03:39:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099249034999999999 2024-09-16T03:39:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3582 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:39:51.169 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.076713839, 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62429066099999997, 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3582 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032779+0000", 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2720000000000001e-06 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035239+0000", 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024594180000000001 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035261+0000", 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2246e-05 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035273+0000", 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2289999999999999e-05 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038105+0000", 2024-09-16T03:39:51.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028319780000000002 2024-09-16T03:39:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038111+0000", 2024-09-16T03:39:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3099999999999997e-06 2024-09-16T03:39:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657045+0000", 2024-09-16T03:39:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61893337199999998 2024-09-16T03:39:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657069+0000", 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3774999999999999e-05 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3583 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.071605709000004, 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61919224299999998, 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.174 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3583 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:39:51.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037888+0000", 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-06 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038032+0000", 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014443799999999999 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038045+0000", 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3417999999999999e-05 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038052+0000", 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3890000000000002e-06 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038133+0000", 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0856e-05 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038136+0000", 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7160000000000001e-06 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657073+0000", 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61893632899999995 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657078+0000", 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4770000000000002e-06 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.178 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3584 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.071427415000002, 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61903161699999998, 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3584 2024-09-16T03:39:51.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038065+0000", 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3499999999999996e-07 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038069+0000", 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038075+0000", 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0549999999999996e-06 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038078+0000", 2024-09-16T03:39:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4939999999999999e-06 2024-09-16T03:39:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.126379+0000", 2024-09-16T03:39:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088300141999999998 2024-09-16T03:39:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.126393+0000", 2024-09-16T03:39:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4827999999999999e-05 2024-09-16T03:39:51.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657092+0000", 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.53069909500000001 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657096+0000", 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4139999999999998e-06 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3585 4.2c 4.1cf4922c (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.071406854999999, 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61902921799999999, 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3585 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1899999999999998e-07 2024-09-16T03:39:51.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038087+0000", 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038091+0000", 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1590000000000003e-06 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038095+0000", 2024-09-16T03:39:51.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5779999999999999e-06 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.138249+0000", 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100154101 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.138256+0000", 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4209999999999998e-06 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657111+0000", 2024-09-16T03:39:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.51885431800000004 2024-09-16T03:39:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657114+0000", 2024-09-16T03:39:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5190000000000001e-06 2024-09-16T03:39:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3983 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 2151362] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:51.188 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.817498090999997, 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49254613699999999, 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3983 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9500000000000002e-07 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291997+0000", 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9679999999999998e-06 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292002+0000", 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7700000000000001e-06 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292005+0000", 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3019999999999999e-06 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.382133+0000", 2024-09-16T03:39:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.090127600000000002 2024-09-16T03:39:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.382184+0000", 2024-09-16T03:39:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1304999999999998e-05 2024-09-16T03:39:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 11,15", 2024-09-16T03:39:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.382301+0000", 2024-09-16T03:39:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116673 2024-09-16T03:39:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.383337+0000", 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00103619 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.583745+0000", 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20040787600000001 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784456+0000", 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20071103700000001 2024-09-16T03:39:51.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784496+0000", 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9740999999999998e-05 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784540+0000", 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4079999999999998e-05 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3984 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:39:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.817481909999998, 2024-09-16T03:39:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49270909200000002, 2024-09-16T03:39:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3984 2024-09-16T03:39:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.195 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:39:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:39:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4900000000000001e-07 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292012+0000", 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9250000000000002e-06 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292015+0000", 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4199999999999999e-06 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292018+0000", 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.994e-06 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.388062+0000", 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096043327999999997 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.388110+0000", 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8272999999999997e-05 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784615+0000", 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39650470300000001 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784664+0000", 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8841000000000002e-05 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784719+0000", 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5259000000000001e-05 2024-09-16T03:39:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4624 4.d 4:b3fe05b2:::smithi15736696-15:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e150)", 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.652818+0000", 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.456673600999999, 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72344784900000003, 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.200 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4624 2024-09-16T03:39:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652818+0000", 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652818+0000", 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652820+0000", 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.357e-06 2024-09-16T03:39:51.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652827+0000", 2024-09-16T03:39:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6429999999999998e-06 2024-09-16T03:39:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652840+0000", 2024-09-16T03:39:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2571e-05 2024-09-16T03:39:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652855+0000", 2024-09-16T03:39:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5671e-05 2024-09-16T03:39:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.657070+0000", 2024-09-16T03:39:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042144820000000003 2024-09-16T03:39:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:39:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.657088+0000", 2024-09-16T03:39:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7995999999999998e-05 2024-09-16T03:39:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.854710+0000", 2024-09-16T03:39:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19762168499999999 2024-09-16T03:39:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.854863+0000", 2024-09-16T03:39:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000153568 2024-09-16T03:39:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.376266+0000", 2024-09-16T03:39:51.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52140287600000002 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4861 4.d 4:b3fe05b2:::smithi15736696-15:head [write 2805198~552112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.054939+0000", 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.0545522749999998, 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59583329399999996, 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4861 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.054939+0000", 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.054939+0000", 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.054940+0000", 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3099999999999997e-07 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055449+0000", 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050923899999999998 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055457+0000", 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7030000000000006e-06 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055461+0000", 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9779999999999999e-06 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.243176+0000", 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18771517600000001 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.243181+0000", 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.51e-06 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.348586+0000", 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10540505 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.348658+0000", 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2310000000000004e-05 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3", 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.348770+0000", 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111573 2024-09-16T03:39:51.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.350138+0000", 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001368282 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.549595+0000", 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19945716899999999 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.650700+0000", 2024-09-16T03:39:51.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101104609 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.650738+0000", 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8643000000000003e-05 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.650773+0000", 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4320999999999999e-05 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4862 4.d 4:b3fe05b2:::smithi15736696-15:head [setxattr _header (56) in=7b,truncate 3428694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-16T03:39:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.055468+0000", 2024-09-16T03:39:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.0540239300000001, 2024-09-16T03:39:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.795963953, 2024-09-16T03:39:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4862 2024-09-16T03:39:51.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055468+0000", 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055468+0000", 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055468+0000", 2024-09-16T03:39:51.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6800000000000001e-07 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055471+0000", 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5239999999999999e-06 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055474+0000", 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4800000000000001e-06 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055477+0000", 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.756e-06 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.243208+0000", 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18773089400000001 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.243210+0000", 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2390000000000001e-06 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.349668+0000", 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10645742499999999 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.349742+0000", 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4036999999999996e-05 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3", 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.349890+0000", 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148238 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.350269+0000", 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037942199999999997 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.750976+0000", 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.400706442 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.851331+0000", 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100355047 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.851381+0000", 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0121999999999999e-05 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.851432+0000", 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0859000000000002e-05 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.222 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4863 4.d 4:b3fe05b2:::smithi15736696-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e165)", 2024-09-16T03:39:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.055481+0000", 2024-09-16T03:39:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 2.0540109050000002, 2024-09-16T03:39:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.79613757299999999, 2024-09-16T03:39:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:39:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:39:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:39:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:39:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:39:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4863 2024-09-16T03:39:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:39:51.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055481+0000", 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055481+0000", 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055481+0000", 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4400000000000001e-07 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:39:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055483+0000", 2024-09-16T03:39:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.409e-06 2024-09-16T03:39:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:39:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055486+0000", 2024-09-16T03:39:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.162e-06 2024-09-16T03:39:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:39:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055489+0000", 2024-09-16T03:39:51.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9859999999999999e-06 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.243248+0000", 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18775898999999999 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.243252+0000", 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5419999999999999e-06 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.350067+0000", 2024-09-16T03:39:51.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106814669 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.350122+0000", 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4787999999999998e-05 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.851497+0000", 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.501375499 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:39:51.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.851558+0000", 2024-09-16T03:39:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1223000000000007e-05 2024-09-16T03:39:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:39:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:39:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:39:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.851618+0000", 2024-09-16T03:39:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9960999999999997e-05 2024-09-16T03:39:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:39:51.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:39:51.228 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:39:51.229 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 9, 1] out_osds: [5, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:39:51.229 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:39:51.229 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:39:51.229 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-16T03:39:51.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 0.05 2024-09-16T03:39:51.231 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:39:51.231 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:51.231 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:51.232 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:51.232 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:51.232 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:51.232 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:51.233 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:51.233 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:51.233 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:51.233 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:51.234 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:51.234 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:51.234 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:51.234 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:51.235 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:51.228+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:51.240 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:51.240 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:51.257 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:51.257 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:51.275 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:51.275 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:51.292 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:51.292 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:51.308 INFO:tasks.rados.rados.0.smithi157.stdout:1994: expect (ObjNum 411 snap 0 seq_num 411) 2024-09-16T03:39:51.309 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:51.309 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:51.331 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:51.328+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:51.351 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:51.352 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:51.361 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:39:51.404 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:51.431 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:51.430+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:51.457 INFO:tasks.rados.rados.0.smithi157.stdout:1997: finishing set_chunk to oid 49 2024-09-16T03:39:51.457 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 80 (ObjNum 421 snap 0 seq_num 421) dirty exists 2024-09-16T03:39:51.457 INFO:tasks.rados.rados.0.smithi157.stdout:1998: expect (ObjNum 367 snap 0 seq_num 367) 2024-09-16T03:39:51.457 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:51.465 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:51.465 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:51.478 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:39:51.532 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:51.529+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:51.545 INFO:tasks.rados.rados.0.smithi157.stdout:2001: expect (ObjNum 327 snap 0 seq_num 327) 2024-09-16T03:39:51.545 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:51.580 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:51.580 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:51.588 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:39:51.592 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2527697 2024-09-16T03:39:51.615 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2911430 2024-09-16T03:39:51.628 INFO:tasks.rados.rados.0.smithi157.stdout:1999: expect (ObjNum 325 snap 0 seq_num 325) 2024-09-16T03:39:51.628 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 489471 2024-09-16T03:39:51.633 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:51.630+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:51.668 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1028058 2024-09-16T03:39:51.692 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:51.692 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:51.700 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:39:51.733 INFO:tasks.rados.rados.0.smithi157.stdout:2000: got (22) Invalid argument 2024-09-16T03:39:51.733 INFO:tasks.rados.rados.0.smithi157.stdout:2002: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-16T03:39:51.734 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:51.732+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:51.740 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1408297 2024-09-16T03:39:51.762 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:51.792 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:39:51.792 INFO:teuthology.orchestra.run.smithi064.stdout:moved 4 / 207 (1.93237%) 2024-09-16T03:39:51.792 INFO:teuthology.orchestra.run.smithi064.stdout:avg 14.7857 2024-09-16T03:39:51.792 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 4.05636 -> 4.14347 (expected baseline 3.70535) 2024-09-16T03:39:51.792 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.7 with 7 -> 7 pgs (0.47343 -> 0.47343 * mean) 2024-09-16T03:39:51.792 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.10 with 20 -> 20 pgs (1.35266 -> 1.35266 * mean) 2024-09-16T03:39:51.792 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:51.793 INFO:teuthology.orchestra.run.smithi064.stdout:oload 110 2024-09-16T03:39:51.793 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 0.05 2024-09-16T03:39:51.793 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-09-16T03:39:51.793 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0007 2024-09-16T03:39:51.793 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0007 2024-09-16T03:39:51.793 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 weight 1.0000 -> 0.9500 2024-09-16T03:39:51.793 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15 weight 1.0000 -> 0.9500 2024-09-16T03:39:51.793 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 weight 1.0000 -> 0.9500 2024-09-16T03:39:51.793 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 weight 1.0000 -> 0.9500 2024-09-16T03:39:51.807 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-16T03:39:51.807 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 3.0 2024-09-16T03:39:51.834 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:51.832+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:51.839 INFO:tasks.rados.rados.0.smithi157.stdout:1987: done (15 left) 2024-09-16T03:39:51.839 INFO:tasks.rados.rados.0.smithi157.stdout:1989: done (14 left) 2024-09-16T03:39:51.839 INFO:tasks.rados.rados.0.smithi157.stdout:1990: done (13 left) 2024-09-16T03:39:51.839 INFO:tasks.rados.rados.0.smithi157.stdout:1991: done (12 left) 2024-09-16T03:39:51.839 INFO:tasks.rados.rados.0.smithi157.stdout:1992: done (11 left) 2024-09-16T03:39:51.839 INFO:tasks.rados.rados.0.smithi157.stdout:1993: done (10 left) 2024-09-16T03:39:51.839 INFO:tasks.rados.rados.0.smithi157.stdout:1994: done (9 left) 2024-09-16T03:39:51.839 INFO:tasks.rados.rados.0.smithi157.stdout:1995: done (8 left) 2024-09-16T03:39:51.839 INFO:tasks.rados.rados.0.smithi157.stdout:1996: done (7 left) 2024-09-16T03:39:51.839 INFO:tasks.rados.rados.0.smithi157.stdout:1997: done (6 left) 2024-09-16T03:39:51.840 INFO:tasks.rados.rados.0.smithi157.stdout:1998: done (5 left) 2024-09-16T03:39:51.840 INFO:tasks.rados.rados.0.smithi157.stdout:1999: done (4 left) 2024-09-16T03:39:51.840 INFO:tasks.rados.rados.0.smithi157.stdout:2000: done (3 left) 2024-09-16T03:39:51.840 INFO:tasks.rados.rados.0.smithi157.stdout:2001: done (2 left) 2024-09-16T03:39:51.840 INFO:tasks.rados.rados.0.smithi157.stdout:2002: done (1 left) 2024-09-16T03:39:51.840 INFO:tasks.rados.rados.0.smithi157.stdout:2003: done (0 left) 2024-09-16T03:39:51.840 INFO:tasks.rados.rados.0.smithi157.stdout:2004: write (excl) oid 33 current snap is 0 2024-09-16T03:39:51.840 INFO:tasks.rados.rados.0.smithi157.stdout:2004: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2024-09-16T03:39:51.844 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:51.844 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:51.846 INFO:tasks.rados.rados.0.smithi157.stdout:2004: writing smithi15736696-33 from 695922 to 1405450 tid 1 2024-09-16T03:39:51.853 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:39:51.853 INFO:tasks.rados.rados.0.smithi157.stdout:2004: writing smithi15736696-33 from 2032616 to 2640096 tid 2 2024-09-16T03:39:51.857 INFO:tasks.rados.rados.0.smithi157.stdout:2004: writing smithi15736696-33 from 3198537 to 3495922 tid 3 2024-09-16T03:39:51.857 INFO:tasks.rados.rados.0.smithi157.stdout:2005: write oid 21 current snap is 0 2024-09-16T03:39:51.857 INFO:tasks.rados.rados.0.smithi157.stdout:2005: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-09-16T03:39:51.865 INFO:tasks.rados.rados.0.smithi157.stdout:2005: writing smithi15736696-21 from 712729 to 1497506 tid 1 2024-09-16T03:39:51.870 INFO:tasks.rados.rados.0.smithi157.stdout:2005: writing smithi15736696-21 from 2091098 to 2558589 tid 2 2024-09-16T03:39:51.876 INFO:tasks.rados.rados.0.smithi157.stdout:2005: writing smithi15736696-21 from 3342313 to 3512729 tid 3 2024-09-16T03:39:51.876 INFO:tasks.rados.rados.0.smithi157.stdout:2006: read oid 7 snap -1 2024-09-16T03:39:51.876 INFO:tasks.rados.rados.0.smithi157.stdout:2006: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-16T03:39:51.876 INFO:tasks.rados.rados.0.smithi157.stdout:2007: tier_promote oid 18 2024-09-16T03:39:51.876 INFO:tasks.rados.rados.0.smithi157.stdout:2008: read oid 16 snap -1 2024-09-16T03:39:51.877 INFO:tasks.rados.rados.0.smithi157.stdout:2008: expect deleted 2024-09-16T03:39:51.877 INFO:tasks.rados.rados.0.smithi157.stdout:2009: write (excl) oid 26 current snap is 0 2024-09-16T03:39:51.877 INFO:tasks.rados.rados.0.smithi157.stdout:2009: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2024-09-16T03:39:51.882 INFO:tasks.rados.rados.0.smithi157.stdout:2009: writing smithi15736696-26 from 729536 to 1189562 tid 1 2024-09-16T03:39:51.888 INFO:tasks.rados.rados.0.smithi157.stdout:2009: writing smithi15736696-26 from 1749580 to 2360729 tid 2 2024-09-16T03:39:51.894 INFO:tasks.rados.rados.0.smithi157.stdout:2009: writing smithi15736696-26 from 2853383 to 3305833 tid 3 2024-09-16T03:39:51.894 INFO:tasks.rados.rados.0.smithi157.stdout:2010: read oid 6 snap -1 2024-09-16T03:39:51.894 INFO:tasks.rados.rados.0.smithi157.stdout:2010: expect (ObjNum 327 snap 0 seq_num 327) 2024-09-16T03:39:51.894 INFO:tasks.rados.rados.0.smithi157.stdout:2011: tier_evict oid 10 2024-09-16T03:39:51.894 INFO:tasks.rados.rados.0.smithi157.stdout:2011: tier_evict oid 10 snap -1 2024-09-16T03:39:51.894 INFO:tasks.rados.rados.0.smithi157.stdout:2012: read oid 37 snap -1 2024-09-16T03:39:51.894 INFO:tasks.rados.rados.0.smithi157.stdout:2012: expect (ObjNum 445 snap 0 seq_num 445) 2024-09-16T03:39:51.895 INFO:tasks.rados.rados.0.smithi157.stdout:2013: write (excl) oid 43 current snap is 0 2024-09-16T03:39:51.895 INFO:tasks.rados.rados.0.smithi157.stdout:2013: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2024-09-16T03:39:51.902 INFO:tasks.rados.rados.0.smithi157.stdout:2013: writing smithi15736696-43 from 746343 to 1397971 tid 1 2024-09-16T03:39:51.908 INFO:tasks.rados.rados.0.smithi157.stdout:2013: writing smithi15736696-43 from 1924415 to 2679222 tid 2 2024-09-16T03:39:51.914 INFO:tasks.rados.rados.0.smithi157.stdout:2013: writing smithi15736696-43 from 3397159 to 3546343 tid 3 2024-09-16T03:39:51.914 INFO:tasks.rados.rados.0.smithi157.stdout:2014: read oid 32 snap -1 2024-09-16T03:39:51.914 INFO:tasks.rados.rados.0.smithi157.stdout:2014: expect (ObjNum 423 snap 0 seq_num 423) 2024-09-16T03:39:51.914 INFO:tasks.rados.rados.0.smithi157.stdout:2015: read oid 1 snap -1 2024-09-16T03:39:51.914 INFO:tasks.rados.rados.0.smithi157.stdout:2015: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-16T03:39:51.914 INFO:tasks.rados.rados.0.smithi157.stdout:2016: write (excl) oid 34 current snap is 0 2024-09-16T03:39:51.914 INFO:tasks.rados.rados.0.smithi157.stdout:2016: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2024-09-16T03:39:51.921 INFO:tasks.rados.rados.0.smithi157.stdout:2016: writing smithi15736696-34 from 763150 to 1490027 tid 1 2024-09-16T03:39:51.927 INFO:tasks.rados.rados.0.smithi157.stdout:2016: writing smithi15736696-34 from 1982897 to 2597715 tid 2 2024-09-16T03:39:51.933 INFO:tasks.rados.rados.0.smithi157.stdout:2016: writing smithi15736696-34 from 3024582 to 3499576 tid 3 2024-09-16T03:39:51.933 INFO:tasks.rados.rados.0.smithi157.stdout:2017: write (excl) oid 12 current snap is 0 2024-09-16T03:39:51.933 INFO:tasks.rados.rados.0.smithi157.stdout:2017: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-09-16T03:39:51.935 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:51.933+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:51.939 INFO:tasks.rados.rados.0.smithi157.stdout:2017: writing smithi15736696-12 from 779957 to 1182083 tid 1 2024-09-16T03:39:51.947 INFO:tasks.rados.rados.0.smithi157.stdout:2017: writing smithi15736696-12 from 1925026 to 2683502 tid 2 2024-09-16T03:39:51.952 INFO:tasks.rados.rados.0.smithi157.stdout:2017: writing smithi15736696-12 from 3335652 to 3579957 tid 3 2024-09-16T03:39:51.952 INFO:tasks.rados.rados.0.smithi157.stdout:2018: read oid 9 snap -1 2024-09-16T03:39:51.952 INFO:tasks.rados.rados.0.smithi157.stdout:2018: expect (ObjNum 432 snap 0 seq_num 432) 2024-09-16T03:39:51.953 INFO:tasks.rados.rados.0.smithi157.stdout:2019: write oid 29 current snap is 0 2024-09-16T03:39:51.953 INFO:tasks.rados.rados.0.smithi157.stdout:2019: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2024-09-16T03:39:51.957 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:51.957 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:51.958 INFO:tasks.rados.rados.0.smithi157.stdout:2019: writing smithi15736696-29 from 796764 to 1274139 tid 1 2024-09-16T03:39:51.966 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:39:51.966 INFO:tasks.rados.rados.0.smithi157.stdout:2019: writing smithi15736696-29 from 1983508 to 2601995 tid 2 2024-09-16T03:39:51.971 INFO:tasks.rados.rados.0.smithi157.stdout:2019: writing smithi15736696-29 from 3363075 to 3596764 tid 3 2024-09-16T03:39:51.971 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:39:51.971 INFO:tasks.rados.rados.0.smithi157.stdout:2015: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-16T03:39:51.975 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 454640 2024-09-16T03:39:52.035 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1389631 2024-09-16T03:39:52.036 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:52.035+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:52.069 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:52.069 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:52.078 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:39:52.108 INFO:tasks.rados.rados.0.smithi157.stdout:2005: finishing write tid 1 to smithi15736696-21 2024-09-16T03:39:52.108 INFO:tasks.rados.rados.0.smithi157.stdout:2015: done (15 left) 2024-09-16T03:39:52.108 INFO:tasks.rados.rados.0.smithi157.stdout:2020: read oid 4 snap -1 2024-09-16T03:39:52.108 INFO:tasks.rados.rados.0.smithi157.stdout:2020: expect (ObjNum 411 snap 0 seq_num 411) 2024-09-16T03:39:52.108 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:39:52.108 INFO:tasks.rados.rados.0.smithi157.stdout:2005: finishing write tid 2 to smithi15736696-21 2024-09-16T03:39:52.108 INFO:tasks.rados.rados.0.smithi157.stdout:2010: expect (ObjNum 327 snap 0 seq_num 327) 2024-09-16T03:39:52.137 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:52.134+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:52.158 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-09-16T03:39:52.157+0000 7f5019b9e640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-16T03:39:52.159 INFO:teuthology.orchestra.run.smithi064.stdout:moved 12 / 207 (5.7971%) 2024-09-16T03:39:52.159 INFO:teuthology.orchestra.run.smithi064.stdout:avg 14.7857 2024-09-16T03:39:52.159 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 4.14347 -> 4.14347 (expected baseline 3.70535) 2024-09-16T03:39:52.159 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.7 with 7 -> 10 pgs (0.47343 -> 0.676328 * mean) 2024-09-16T03:39:52.159 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.10 with 20 -> 21 pgs (1.35266 -> 1.42029 * mean) 2024-09-16T03:39:52.159 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:52.159 INFO:teuthology.orchestra.run.smithi064.stdout:oload 110 2024-09-16T03:39:52.159 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 3 2024-09-16T03:39:52.159 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-09-16T03:39:52.159 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0007 2024-09-16T03:39:52.159 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0008 2024-09-16T03:39:52.159 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 weight 0.9500 -> 0.5253 2024-09-16T03:39:52.159 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15 weight 0.9500 -> 0.7906 2024-09-16T03:39:52.160 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 weight 0.9500 -> 0.7466 2024-09-16T03:39:52.160 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8 weight 1.0000 -> 0.8549 2024-09-16T03:39:52.160 INFO:teuthology.orchestra.run.smithi064.stderr:no change 2024-09-16T03:39:52.172 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-16T03:39:52.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2024-09-16T03:39:52.180 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:52.180 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:52.189 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:39:52.208 INFO:tasks.rados.rados.0.smithi157.stdout:2012: expect (ObjNum 445 snap 0 seq_num 445) 2024-09-16T03:39:52.208 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:52.255 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:52.254+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:52.333 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:52.333 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:52.338 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:52.335+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:52.341 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:39:52.439 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:52.437+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:39:52.444 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:52.444 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:52.453 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:39:52.511 INFO:tasks.ceph.mgr.y.smithi064.stderr:2024-09-16T03:39:52.509+0000 7f5019b9e640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-16T03:39:52.511 INFO:teuthology.orchestra.run.smithi064.stdout:moved 12 / 207 (5.7971%) 2024-09-16T03:39:52.511 INFO:teuthology.orchestra.run.smithi064.stdout:avg 14.7857 2024-09-16T03:39:52.512 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 4.14347 -> 4.14347 (expected baseline 3.70535) 2024-09-16T03:39:52.512 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.7 with 7 -> 10 pgs (0.47343 -> 0.676328 * mean) 2024-09-16T03:39:52.512 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.10 with 20 -> 21 pgs (1.35266 -> 1.42029 * mean) 2024-09-16T03:39:52.512 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:52.512 INFO:teuthology.orchestra.run.smithi064.stdout:oload 110 2024-09-16T03:39:52.512 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 1 2024-09-16T03:39:52.512 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-09-16T03:39:52.512 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0007 2024-09-16T03:39:52.512 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0008 2024-09-16T03:39:52.512 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 weight 0.9500 -> 0.5253 2024-09-16T03:39:52.512 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15 weight 0.9500 -> 0.7906 2024-09-16T03:39:52.512 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 weight 0.9500 -> 0.7466 2024-09-16T03:39:52.513 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8 weight 1.0000 -> 0.8549 2024-09-16T03:39:52.513 INFO:teuthology.orchestra.run.smithi064.stderr:no change 2024-09-16T03:39:52.524 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-16T03:39:52.525 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 3.0 2024-09-16T03:39:52.540 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:52.537+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:52.548 INFO:tasks.rados.rados.0.smithi157.stdout:2014: expect (ObjNum 423 snap 0 seq_num 423) 2024-09-16T03:39:52.555 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:52.555 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:52.564 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:39:52.573 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2983761 2024-09-16T03:39:52.623 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:52.641 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:52.639+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:52.671 INFO:tasks.rados.rados.0.smithi157.stdout:2005: finishing write tid 3 to smithi15736696-21 2024-09-16T03:39:52.672 INFO:tasks.rados.rados.0.smithi157.stdout:2005: finishing write tid 4 to smithi15736696-21 2024-09-16T03:39:52.672 INFO:tasks.rados.rados.0.smithi157.stdout:2005: finishing write tid 5 to smithi15736696-21 2024-09-16T03:39:52.672 INFO:tasks.rados.rados.0.smithi157.stdout:2005: oid 21 updating version 0 to 67 2024-09-16T03:39:52.672 INFO:tasks.rados.rados.0.smithi157.stdout:2005: oid 21 updating version 67 to 68 2024-09-16T03:39:52.672 INFO:tasks.rados.rados.0.smithi157.stdout:2005: oid 21 version 68 is already newer than 66 2024-09-16T03:39:52.672 INFO:tasks.rados.rados.0.smithi157.stdout:2005: oid 21 updating version 68 to 69 2024-09-16T03:39:52.672 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 21 v 69 (ObjNum 447 snap 0 seq_num 447) dirty exists 2024-09-16T03:39:52.672 INFO:tasks.rados.rados.0.smithi157.stdout:2005: left oid 21 (ObjNum 447 snap 0 seq_num 447) 2024-09-16T03:39:52.672 INFO:tasks.rados.rados.0.smithi157.stdout:2018: expect (ObjNum 432 snap 0 seq_num 432) 2024-09-16T03:39:52.675 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 590417 2024-09-16T03:39:52.709 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:52.709 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:52.717 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:39:52.721 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:52.742 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:52.740+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:52.758 INFO:tasks.rados.rados.0.smithi157.stdout:2013: finishing write tid 1 to smithi15736696-43 2024-09-16T03:39:52.758 INFO:tasks.rados.rados.0.smithi157.stdout:2019: finishing write tid 1 to smithi15736696-29 2024-09-16T03:39:52.758 INFO:tasks.rados.rados.0.smithi157.stdout:2005: done (15 left) 2024-09-16T03:39:52.758 INFO:tasks.rados.rados.0.smithi157.stdout:2010: done (14 left) 2024-09-16T03:39:52.758 INFO:tasks.rados.rados.0.smithi157.stdout:2012: done (13 left) 2024-09-16T03:39:52.758 INFO:tasks.rados.rados.0.smithi157.stdout:2014: done (12 left) 2024-09-16T03:39:52.758 INFO:tasks.rados.rados.0.smithi157.stdout:2018: done (11 left) 2024-09-16T03:39:52.758 INFO:tasks.rados.rados.0.smithi157.stdout:2021: read oid 35 snap -1 2024-09-16T03:39:52.758 INFO:tasks.rados.rados.0.smithi157.stdout:2021: expect (ObjNum 422 snap 0 seq_num 422) 2024-09-16T03:39:52.758 INFO:tasks.rados.rados.0.smithi157.stdout:2022: read oid 31 snap -1 2024-09-16T03:39:52.758 INFO:tasks.rados.rados.0.smithi157.stdout:2022: expect (ObjNum 436 snap 0 seq_num 436) 2024-09-16T03:39:52.758 INFO:tasks.rados.rados.0.smithi157.stdout:2023: tier_evict oid 42 2024-09-16T03:39:52.759 INFO:tasks.rados.rados.0.smithi157.stdout:2023: tier_evict oid 42 snap -1 2024-09-16T03:39:52.759 INFO:tasks.rados.rados.0.smithi157.stdout:2004: finishing write tid 1 to smithi15736696-33 2024-09-16T03:39:52.759 INFO:tasks.rados.rados.0.smithi157.stdout:2004: finishing write tid 2 to smithi15736696-33 2024-09-16T03:39:52.759 INFO:tasks.rados.rados.0.smithi157.stdout:2004: finishing write tid 3 to smithi15736696-33 2024-09-16T03:39:52.759 INFO:tasks.rados.rados.0.smithi157.stdout:2004: finishing write tid 4 to smithi15736696-33 2024-09-16T03:39:52.759 INFO:tasks.rados.rados.0.smithi157.stdout:2004: finishing write tid 5 to smithi15736696-33 2024-09-16T03:39:52.759 INFO:tasks.rados.rados.0.smithi157.stdout:2004: oid 33 updating version 0 to 66 2024-09-16T03:39:52.759 INFO:tasks.rados.rados.0.smithi157.stdout:2004: oid 33 updating version 66 to 68 2024-09-16T03:39:52.759 INFO:tasks.rados.rados.0.smithi157.stdout:2004: oid 33 updating version 68 to 69 2024-09-16T03:39:52.759 INFO:tasks.rados.rados.0.smithi157.stdout:2004: oid 33 version 69 is already newer than 67 2024-09-16T03:39:52.759 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 33 v 69 (ObjNum 446 snap 0 seq_num 446) dirty exists 2024-09-16T03:39:52.759 INFO:tasks.rados.rados.0.smithi157.stdout:2004: left oid 33 (ObjNum 446 snap 0 seq_num 446) 2024-09-16T03:39:52.759 INFO:tasks.rados.rados.0.smithi157.stdout:2006: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-16T03:39:52.795 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:52.793+0000 7f3a21722640 -1 osd.6 167 handle_osd_map injecting map crc failure 2024-09-16T03:39:52.799 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:52.798+0000 7f9288166640 -1 osd.14 167 handle_osd_map injecting map crc failure 2024-09-16T03:39:52.835 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:52.837 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:52.837 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:52.843 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:52.842+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:52.847 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:39:52.876 INFO:tasks.rados.rados.0.smithi157.stdout:2017: finishing write tid 1 to smithi15736696-12 2024-09-16T03:39:52.876 INFO:tasks.rados.rados.0.smithi157.stdout:2013: finishing write tid 2 to smithi15736696-43 2024-09-16T03:39:52.876 INFO:tasks.rados.rados.0.smithi157.stdout:2013: finishing write tid 3 to smithi15736696-43 2024-09-16T03:39:52.877 INFO:tasks.rados.rados.0.smithi157.stdout:2019: finishing write tid 2 to smithi15736696-29 2024-09-16T03:39:52.877 INFO:tasks.rados.rados.0.smithi157.stdout:2007: got (0) Success 2024-09-16T03:39:52.877 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 18 v 80 (ObjNum 441 snap 0 seq_num 441) dirty exists 2024-09-16T03:39:52.877 INFO:tasks.rados.rados.0.smithi157.stdout:2019: finishing write tid 3 to smithi15736696-29 2024-09-16T03:39:52.877 INFO:tasks.rados.rados.0.smithi157.stdout:2004: done (13 left) 2024-09-16T03:39:52.877 INFO:tasks.rados.rados.0.smithi157.stdout:2006: done (12 left) 2024-09-16T03:39:52.877 INFO:tasks.rados.rados.0.smithi157.stdout:2007: done (11 left) 2024-09-16T03:39:52.877 INFO:tasks.rados.rados.0.smithi157.stdout:2008: done (10 left) 2024-09-16T03:39:52.877 INFO:tasks.rados.rados.0.smithi157.stdout:2024: tier_flush oid 6 2024-09-16T03:39:52.877 INFO:tasks.rados.rados.0.smithi157.stdout:2024: tier_flush oid 6 snap -1 2024-09-16T03:39:52.877 INFO:tasks.rados.rados.0.smithi157.stdout:2019: finishing write tid 4 to smithi15736696-29 2024-09-16T03:39:52.877 INFO:tasks.rados.rados.0.smithi157.stdout:2017: finishing write tid 2 to smithi15736696-12 2024-09-16T03:39:52.877 INFO:tasks.rados.rados.0.smithi157.stdout:2017: finishing write tid 3 to smithi15736696-12 2024-09-16T03:39:52.878 INFO:tasks.rados.rados.0.smithi157.stdout:2013: finishing write tid 4 to smithi15736696-43 2024-09-16T03:39:52.878 INFO:tasks.rados.rados.0.smithi157.stdout:2013: finishing write tid 5 to smithi15736696-43 2024-09-16T03:39:52.878 INFO:tasks.rados.rados.0.smithi157.stdout:2013: oid 43 updating version 0 to 93 2024-09-16T03:39:52.878 INFO:tasks.rados.rados.0.smithi157.stdout:2013: oid 43 updating version 93 to 94 2024-09-16T03:39:52.878 INFO:tasks.rados.rados.0.smithi157.stdout:2013: oid 43 updating version 94 to 96 2024-09-16T03:39:52.878 INFO:tasks.rados.rados.0.smithi157.stdout:2013: oid 43 version 96 is already newer than 95 2024-09-16T03:39:52.878 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 96 (ObjNum 449 snap 0 seq_num 449) dirty exists 2024-09-16T03:39:52.878 INFO:tasks.rados.rados.0.smithi157.stdout:2013: left oid 43 (ObjNum 449 snap 0 seq_num 449) 2024-09-16T03:39:52.878 INFO:tasks.rados.rados.0.smithi157.stdout:2020: expect (ObjNum 411 snap 0 seq_num 411) 2024-09-16T03:39:52.913 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2787596 2024-09-16T03:39:52.922 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 994365 2024-09-16T03:39:52.944 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:52.941+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:52.961 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:52.991 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:52.991 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:53.000 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:39:53.016 INFO:tasks.rados.rados.0.smithi157.stdout:2019: finishing write tid 5 to smithi15736696-29 2024-09-16T03:39:53.016 INFO:tasks.rados.rados.0.smithi157.stdout:2019: oid 29 updating version 0 to 155 2024-09-16T03:39:53.016 INFO:tasks.rados.rados.0.smithi157.stdout:2019: oid 29 version 155 is already newer than 153 2024-09-16T03:39:53.016 INFO:tasks.rados.rados.0.smithi157.stdout:2019: oid 29 version 155 is already newer than 154 2024-09-16T03:39:53.016 INFO:tasks.rados.rados.0.smithi157.stdout:2019: oid 29 updating version 155 to 156 2024-09-16T03:39:53.016 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 29 v 156 (ObjNum 452 snap 0 seq_num 452) dirty exists 2024-09-16T03:39:53.016 INFO:tasks.rados.rados.0.smithi157.stdout:2019: left oid 29 (ObjNum 452 snap 0 seq_num 452) 2024-09-16T03:39:53.017 INFO:tasks.rados.rados.0.smithi157.stdout:2017: finishing write tid 4 to smithi15736696-12 2024-09-16T03:39:53.017 INFO:tasks.rados.rados.0.smithi157.stdout:2013: done (10 left) 2024-09-16T03:39:53.017 INFO:tasks.rados.rados.0.smithi157.stdout:2019: done (9 left) 2024-09-16T03:39:53.017 INFO:tasks.rados.rados.0.smithi157.stdout:2020: done (8 left) 2024-09-16T03:39:53.017 INFO:tasks.rados.rados.0.smithi157.stdout:2025: read oid 41 snap -1 2024-09-16T03:39:53.017 INFO:tasks.rados.rados.0.smithi157.stdout:2025: expect deleted 2024-09-16T03:39:53.017 INFO:tasks.rados.rados.0.smithi157.stdout:2026: write (excl) oid 49 current snap is 0 2024-09-16T03:39:53.017 INFO:tasks.rados.rados.0.smithi157.stdout:2017: finishing write tid 5 to smithi15736696-12 2024-09-16T03:39:53.017 INFO:tasks.rados.rados.0.smithi157.stdout:2017: oid 12 updating version 0 to 23 2024-09-16T03:39:53.017 INFO:tasks.rados.rados.0.smithi157.stdout:2017: oid 12 updating version 23 to 25 2024-09-16T03:39:53.017 INFO:tasks.rados.rados.0.smithi157.stdout:2017: oid 12 version 25 is already newer than 24 2024-09-16T03:39:53.017 INFO:tasks.rados.rados.0.smithi157.stdout:2017: oid 12 updating version 25 to 26 2024-09-16T03:39:53.018 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 12 v 26 (ObjNum 451 snap 0 seq_num 451) dirty exists 2024-09-16T03:39:53.018 INFO:tasks.rados.rados.0.smithi157.stdout:2017: left oid 12 (ObjNum 451 snap 0 seq_num 451) 2024-09-16T03:39:53.018 INFO:tasks.rados.rados.0.smithi157.stdout:2016: finishing write tid 1 to smithi15736696-34 2024-09-16T03:39:53.018 INFO:tasks.rados.rados.0.smithi157.stdout:2016: finishing write tid 2 to smithi15736696-34 2024-09-16T03:39:53.018 INFO:tasks.rados.rados.0.smithi157.stdout:2026: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2024-09-16T03:39:53.021 INFO:tasks.rados.rados.0.smithi157.stdout:2026: writing smithi15736696-49 from 413571 to 966195 tid 1 2024-09-16T03:39:53.028 INFO:tasks.rados.rados.0.smithi157.stdout:2026: writing smithi15736696-49 from 1641990 to 2404135 tid 2 2024-09-16T03:39:53.034 INFO:tasks.rados.rados.0.smithi157.stdout:2026: writing smithi15736696-49 from 2990498 to 3415661 tid 3 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:2017: done (9 left) 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:2027: read oid 19 snap -1 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:2027: expect (ObjNum 434 snap 0 seq_num 434) 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:2028: read oid 44 snap -1 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:2028: expect (ObjNum 410 snap 0 seq_num 410) 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:2029: write (excl) oid 4 current snap is 0 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:2016: finishing write tid 3 to smithi15736696-34 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:2016: finishing write tid 4 to smithi15736696-34 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:2016: finishing write tid 5 to smithi15736696-34 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:2016: oid 34 updating version 0 to 172 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:2016: oid 34 version 172 is already newer than 171 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:2016: oid 34 version 172 is already newer than 169 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:2016: oid 34 version 172 is already newer than 170 2024-09-16T03:39:53.035 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 34 v 172 (ObjNum 450 snap 0 seq_num 450) dirty exists 2024-09-16T03:39:53.036 INFO:tasks.rados.rados.0.smithi157.stdout:2016: left oid 34 (ObjNum 450 snap 0 seq_num 450) 2024-09-16T03:39:53.036 INFO:tasks.rados.rados.0.smithi157.stdout:2029: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-09-16T03:39:53.040 INFO:tasks.rados.rados.0.smithi157.stdout:2029: writing smithi15736696-4 from 430378 to 1058251 tid 1 2024-09-16T03:39:53.045 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:53.044+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:53.047 INFO:tasks.rados.rados.0.smithi157.stdout:2029: writing smithi15736696-4 from 1700472 to 2322628 tid 2 2024-09-16T03:39:53.054 INFO:tasks.rados.rados.0.smithi157.stdout:2029: writing smithi15736696-4 from 3017921 to 3630378 tid 3 2024-09-16T03:39:53.054 INFO:tasks.rados.rados.0.smithi157.stdout:2016: done (11 left) 2024-09-16T03:39:53.054 INFO:tasks.rados.rados.0.smithi157.stdout:2030: read oid 5 snap -1 2024-09-16T03:39:53.054 INFO:tasks.rados.rados.0.smithi157.stdout:2030: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-16T03:39:53.054 INFO:tasks.rados.rados.0.smithi157.stdout:2031: read oid 38 snap -1 2024-09-16T03:39:53.055 INFO:tasks.rados.rados.0.smithi157.stdout:2031: expect (ObjNum 397 snap 0 seq_num 397) 2024-09-16T03:39:53.055 INFO:tasks.rados.rados.0.smithi157.stdout:2032: write oid 20 current snap is 0 2024-09-16T03:39:53.055 INFO:tasks.rados.rados.0.smithi157.stdout:2021: expect (ObjNum 422 snap 0 seq_num 422) 2024-09-16T03:39:53.107 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:53.107 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:53.116 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:39:53.145 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:53.143+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:53.173 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 2626363 2024-09-16T03:39:53.183 INFO:tasks.rados.rados.0.smithi157.stdout:2023: got (22) Invalid argument 2024-09-16T03:39:53.184 INFO:tasks.rados.rados.0.smithi157.stdout:2026: finishing write tid 1 to smithi15736696-49 2024-09-16T03:39:53.184 INFO:tasks.rados.rados.0.smithi157.stdout:2026: finishing write tid 2 to smithi15736696-49 2024-09-16T03:39:53.184 INFO:tasks.rados.rados.0.smithi157.stdout:2031: expect (ObjNum 397 snap 0 seq_num 397) 2024-09-16T03:39:53.184 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:53.213 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 264868 2024-09-16T03:39:53.246 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:53.244+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:53.260 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:53.261 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:53.264 INFO:tasks.rados.rados.0.smithi157.stdout:2030: expect (ObjNum 378 snap 0 seq_num 378) 2024-09-16T03:39:53.269 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:39:53.305 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 972315 2024-09-16T03:39:53.320 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:39:53.352 INFO:tasks.rados.rados.0.smithi157.stdout:2026: finishing write tid 3 to smithi15736696-49 2024-09-16T03:39:53.352 INFO:tasks.rados.rados.0.smithi157.stdout:2026: finishing write tid 4 to smithi15736696-49 2024-09-16T03:39:53.352 INFO:tasks.rados.rados.0.smithi157.stdout:2026: finishing write tid 5 to smithi15736696-49 2024-09-16T03:39:53.352 INFO:tasks.rados.rados.0.smithi157.stdout:2026: oid 49 updating version 0 to 89 2024-09-16T03:39:53.353 INFO:tasks.rados.rados.0.smithi157.stdout:2026: oid 49 version 89 is already newer than 87 2024-09-16T03:39:53.353 INFO:tasks.rados.rados.0.smithi157.stdout:2026: oid 49 version 89 is already newer than 88 2024-09-16T03:39:53.353 INFO:tasks.rados.rados.0.smithi157.stdout:2026: oid 49 updating version 89 to 90 2024-09-16T03:39:53.353 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 49 v 90 (ObjNum 453 snap 0 seq_num 453) dirty exists 2024-09-16T03:39:53.353 INFO:tasks.rados.rados.0.smithi157.stdout:2026: left oid 49 (ObjNum 453 snap 0 seq_num 453) 2024-09-16T03:39:53.353 INFO:tasks.rados.rados.0.smithi157.stdout:2028: expect (ObjNum 410 snap 0 seq_num 410) 2024-09-16T03:39:53.353 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:53.364 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:53.363+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:53.372 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:53.372 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:53.380 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:39:53.446 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:53.446 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:53.447 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:53.445+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:53.464 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:53.464 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:53.481 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:53.485 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:53.485 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:53.494 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:39:53.497 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:53.498 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:53.513 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:53.514 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:53.530 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:53.530 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:53.546 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:53.547 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:53.549 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:53.547+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:53.562 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:53.563 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:53.578 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:53.578 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:53.595 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:53.595 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:53.598 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:53.598 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:53.598 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1193297 2024-09-16T03:39:53.607 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:39:53.611 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:53.612 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:53.628 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:53.628 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:53.644 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:53.644 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:53.649 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:53.647+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:53.660 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:53.661 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:53.677 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:53.677 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:53.692 INFO:tasks.rados.rados.0.smithi157.stdout:2027: expect (ObjNum 434 snap 0 seq_num 434) 2024-09-16T03:39:53.694 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:53.694 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:53.711 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:53.711 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:53.712 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:53.712 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:53.712 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:53.720 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:39:53.750 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:53.747+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:53.755 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1893380 2024-09-16T03:39:53.798 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:53.796+0000 7f6528e68640 -1 osd.1 168 handle_osd_map injecting map crc failure 2024-09-16T03:39:53.799 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:39:53.797+0000 7f49419d0640 -1 osd.4 168 handle_osd_map injecting map crc failure 2024-09-16T03:39:53.801 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:53.799+0000 7f9288166640 -1 osd.14 168 handle_osd_map injecting map crc failure 2024-09-16T03:39:53.801 INFO:teuthology.orchestra.run.smithi064.stdout:moved 10 / 207 (4.83092%) 2024-09-16T03:39:53.801 INFO:teuthology.orchestra.run.smithi064.stdout:avg 14.7857 2024-09-16T03:39:53.801 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 4.14347 -> 3.96734 (expected baseline 3.70535) 2024-09-16T03:39:53.802 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.7 with 7 -> 9 pgs (0.47343 -> 0.608696 * mean) 2024-09-16T03:39:53.802 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.10 with 20 -> 21 pgs (1.35266 -> 1.42029 * mean) 2024-09-16T03:39:53.802 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:53.802 INFO:teuthology.orchestra.run.smithi064.stdout:oload 110 2024-09-16T03:39:53.802 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 3 2024-09-16T03:39:53.802 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-09-16T03:39:53.802 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0007 2024-09-16T03:39:53.802 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0008 2024-09-16T03:39:53.802 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 weight 0.9500 -> 0.5237 2024-09-16T03:39:53.802 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15 weight 0.9500 -> 0.8574 2024-09-16T03:39:53.802 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 weight 0.9500 -> 0.7443 2024-09-16T03:39:53.802 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8 weight 1.0000 -> 0.8523 2024-09-16T03:39:53.807 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:39:53.817 INFO:tasks.rados.rados.0.smithi157.stdout:2029: finishing write tid 1 to smithi15736696-4 2024-09-16T03:39:53.817 INFO:tasks.rados.rados.0.smithi157.stdout:2029: finishing write tid 2 to smithi15736696-4 2024-09-16T03:39:53.817 INFO:tasks.rados.rados.0.smithi157.stdout:2029: finishing write tid 3 to smithi15736696-4 2024-09-16T03:39:53.817 INFO:tasks.rados.rados.0.smithi157.stdout:2029: finishing write tid 4 to smithi15736696-4 2024-09-16T03:39:53.817 INFO:tasks.rados.rados.0.smithi157.stdout:2032: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2024-09-16T03:39:53.818 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-16T03:39:53.819 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 0.05 2024-09-16T03:39:53.823 INFO:tasks.rados.rados.0.smithi157.stdout:2032: writing smithi15736696-20 from 447185 to 1150307 tid 1 2024-09-16T03:39:53.829 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:53.830 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:53.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:53.830 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:53.830 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.830 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:416 4.0 4:023abada:::smithi15736696-17:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:53.830 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:39:53.830 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 153.51752281, 2024-09-16T03:39:53.830 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60401983699999995, 2024-09-16T03:39:53.830 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.830 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.830 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:53.830 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.831 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.831 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.831 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 416 2024-09-16T03:39:53.831 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.831 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.831 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.831 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:39:53.831 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.831 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.831 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.831 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.831 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309242+0000", 2024-09-16T03:39:53.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.832 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309243+0000", 2024-09-16T03:39:53.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.178e-06 2024-09-16T03:39:53.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.832 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.832 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309248+0000", 2024-09-16T03:39:53.832 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:39:53.832 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.832 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309257+0000", 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4720000000000001e-06 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309262+0000", 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0839999999999999e-06 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309297+0000", 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4997999999999999e-05 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.833 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.309367+0000", 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0124999999999997e-05 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812055+0000", 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50268796900000001 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.812614+0000", 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000558309 2024-09-16T03:39:53.834 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.813040+0000", 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042652800000000002 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913189+0000", 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10014880399999999 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913220+0000", 2024-09-16T03:39:53.835 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1353000000000003e-05 2024-09-16T03:39:53.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.836 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.836 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:20.913262+0000", 2024-09-16T03:39:53.836 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1284e-05 2024-09-16T03:39:53.836 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.836 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.836 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.836 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.836 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.836 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:494 4.f 4:f817def9:::smithi15736696-45:head [assert-version v15,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:53.836 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:39:53.836 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 152.10458307499999, 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60451760399999999, 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 494 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:39:53.837 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722181+0000", 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722183+0000", 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1429999999999999e-06 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.838 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722189+0000", 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9429999999999997e-06 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722203+0000", 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4367e-05 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.722210+0000", 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.839 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822367+0000", 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100156933 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:21.822461+0000", 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3216999999999999e-05 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.125593+0000", 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30313253699999998 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.840 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126307+0000", 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071350000000000005 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.126825+0000", 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00051846100000000001 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326612+0000", 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19978721999999999 2024-09-16T03:39:53.841 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326663+0000", 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.0460000000000001e-05 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:22.326699+0000", 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6340000000000001e-05 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.842 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.843 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:784 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=63b,truncate 2840350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:53.843 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:39:53.843 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 148.02017519899999, 2024-09-16T03:39:53.843 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63358405200000001, 2024-09-16T03:39:53.843 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.843 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.843 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:53.843 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.843 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.843 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.843 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 784 2024-09-16T03:39:53.843 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.843 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806589+0000", 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806590+0000", 2024-09-16T03:39:53.844 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6000000000000003e-07 2024-09-16T03:39:53.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806593+0000", 2024-09-16T03:39:53.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6649999999999999e-06 2024-09-16T03:39:53.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.845 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806597+0000", 2024-09-16T03:39:53.845 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.245e-06 2024-09-16T03:39:53.845 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.845 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.845 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806600+0000", 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.23e-06 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099381+0000", 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29278119400000002 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099438+0000", 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6198e-05 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.846 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:53.847 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099527+0000", 2024-09-16T03:39:53.847 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9505000000000004e-05 2024-09-16T03:39:53.847 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.847 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.847 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:53.847 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200775+0000", 2024-09-16T03:39:53.847 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101247904 2024-09-16T03:39:53.847 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.847 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.847 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.847 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200889+0000", 2024-09-16T03:39:53.847 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000114301 2024-09-16T03:39:53.847 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440066+0000", 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.239176367 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440109+0000", 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2941000000000001e-05 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440173+0000", 2024-09-16T03:39:53.848 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4842e-05 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:785 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 148.02015927400001, 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.63376373200000002, 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.849 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 785 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806605+0000", 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.850 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806606+0000", 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4200000000000002e-07 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806607+0000", 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.424e-06 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806611+0000", 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1080000000000003e-06 2024-09-16T03:39:53.851 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:25.806614+0000", 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.067e-06 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099813+0000", 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29319853699999998 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.099858+0000", 2024-09-16T03:39:53.852 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5667999999999997e-05 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440236+0000", 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34037791499999998 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440297+0000", 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0988e-05 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.853 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.440369+0000", 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1682999999999998e-05 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:801 4.0 4:07ad0560:::smithi15736696-25:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 147.82068137600001, 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.534510445, 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.854 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 801 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006083+0000", 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.855 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006084+0000", 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.322e-06 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006088+0000", 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6600000000000001e-06 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006098+0000", 2024-09-16T03:39:53.856 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0186999999999999e-05 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.006104+0000", 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.271e-06 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.199952+0000", 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.193847935 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.857 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.200027+0000", 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4987000000000006e-05 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:26.540594+0000", 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.34056708299999999 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1023 4.0 4:06b6d050:::smithi15736696-50:head [write 2246563~579356] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:39:53.858 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 142.51553230600001, 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60727534699999997, 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1023 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:39:53.859 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.860 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.860 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.860 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.860 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311232+0000", 2024-09-16T03:39:53.860 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.860 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.860 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.860 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.860 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311233+0000", 2024-09-16T03:39:53.860 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1039999999999999e-06 2024-09-16T03:39:53.860 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.860 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.860 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311692+0000", 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00045908800000000002 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311704+0000", 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1844e-05 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311713+0000", 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2500000000000006e-06 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.861 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359204+0000", 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.047491646999999998 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359258+0000", 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4082000000000002e-05 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.359354+0000", 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.5686999999999994e-05 2024-09-16T03:39:53.862 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512245+0000", 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15289135700000001 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512956+0000", 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00071110300000000004 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918361+0000", 2024-09-16T03:39:53.863 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40540489400000002 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918455+0000", 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3412999999999995e-05 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918508+0000", 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.864 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1024 4.0 4:06b6d050:::smithi15736696-50:head [setxattr _header (56) in=7b,truncate 3176490] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 142.515039325, 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60701227000000002, 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1024 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.865 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311725+0000", 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311726+0000", 2024-09-16T03:39:53.866 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.02e-07 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311730+0000", 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1099999999999996e-06 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311735+0000", 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4680000000000003e-06 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.867 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311739+0000", 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9380000000000002e-06 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361468+0000", 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.049729450000000001 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361539+0000", 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0896999999999996e-05 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:53.868 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.361661+0000", 2024-09-16T03:39:53.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000122404 2024-09-16T03:39:53.869 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.869 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.869 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.869 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512943+0000", 2024-09-16T03:39:53.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.151281481 2024-09-16T03:39:53.869 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.869 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.869 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:53.869 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.512959+0000", 2024-09-16T03:39:53.869 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:39:53.869 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.869 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918588+0000", 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40562815499999999 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918646+0000", 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8566999999999997e-05 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918737+0000", 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1308000000000001e-05 2024-09-16T03:39:53.870 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1025 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 142.515020827, 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60723649599999996, 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.871 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1025 2024-09-16T03:39:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.872 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311744+0000", 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9400000000000001e-07 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311746+0000", 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2670000000000001e-06 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311752+0000", 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6389999999999997e-06 2024-09-16T03:39:53.873 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.311757+0000", 2024-09-16T03:39:53.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3639999999999998e-06 2024-09-16T03:39:53.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362018+0000", 2024-09-16T03:39:53.874 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.050260969000000003 2024-09-16T03:39:53.874 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.874 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.874 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:53.874 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.362089+0000", 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0735000000000006e-05 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918781+0000", 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55669215400000005 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918867+0000", 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6019000000000004e-05 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.875 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.876 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918980+0000", 2024-09-16T03:39:53.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011295500000000001 2024-09-16T03:39:53.876 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.876 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.876 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.876 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.876 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.876 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1055 4.10 4:0a55bbf2:::smithi15736696-18:head [setxattr _header (56) in=7b,truncate 3243718] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:53.876 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:39:53.876 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 142.31302811899999, 2024-09-16T03:39:53.876 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61098759599999997, 2024-09-16T03:39:53.876 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.876 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.877 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:53.877 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.877 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.877 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.877 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1055 2024-09-16T03:39:53.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.877 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.877 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.877 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:39:53.877 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.877 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.877 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513736+0000", 2024-09-16T03:39:53.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513737+0000", 2024-09-16T03:39:53.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9399999999999993e-07 2024-09-16T03:39:53.878 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.878 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.878 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.878 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513759+0000", 2024-09-16T03:39:53.878 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1903999999999999e-05 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513773+0000", 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3244000000000001e-05 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513780+0000", 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9999999999999999e-06 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714016+0000", 2024-09-16T03:39:53.879 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20023635000000001 2024-09-16T03:39:53.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:39:53.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.714024+0000", 2024-09-16T03:39:53.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8359999999999992e-06 2024-09-16T03:39:53.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.880 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918736+0000", 2024-09-16T03:39:53.880 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471236700000001 2024-09-16T03:39:53.880 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.880 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.880 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.918830+0000", 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3769000000000003e-05 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919003+0000", 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000173198 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.921589+0000", 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.002586419 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.881 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.019655+0000", 2024-09-16T03:39:53.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098065801999999994 2024-09-16T03:39:53.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124538+0000", 2024-09-16T03:39:53.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10488272899999999 2024-09-16T03:39:53.882 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.882 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.882 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:53.882 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124635+0000", 2024-09-16T03:39:53.882 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7125000000000003e-05 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124724+0000", 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8858999999999997e-05 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1056 4.10 4:0a55bbf2:::smithi15736696-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 142.312975548, 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.61111632800000004, 2024-09-16T03:39:53.883 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1056 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.884 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:53.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513789+0000", 2024-09-16T03:39:53.885 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2499999999999995e-07 2024-09-16T03:39:53.885 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.885 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.885 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.885 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513792+0000", 2024-09-16T03:39:53.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:39:53.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513797+0000", 2024-09-16T03:39:53.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4940000000000003e-06 2024-09-16T03:39:53.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.886 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.886 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.513802+0000", 2024-09-16T03:39:53.886 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5390000000000001e-06 2024-09-16T03:39:53.886 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.886 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919221+0000", 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40541913899999998 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:31.919268+0000", 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7027000000000002e-05 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124766+0000", 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20549842099999999 2024-09-16T03:39:53.887 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124828+0000", 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1803000000000004e-05 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:32.124905+0000", 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7163000000000006e-05 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.889 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2160 4.17 4:ed874e32:::smithi15736696-14:head [assert-version v36,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e78)", 2024-09-16T03:39:53.889 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:39:53.889 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 106.996619138, 2024-09-16T03:39:53.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55377054800000003, 2024-09-16T03:39:53.889 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.889 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.889 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:53.889 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.889 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.889 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.889 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2160 2024-09-16T03:39:53.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.889 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830145+0000", 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830149+0000", 2024-09-16T03:39:53.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.743e-06 2024-09-16T03:39:53.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830162+0000", 2024-09-16T03:39:53.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3283e-05 2024-09-16T03:39:53.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830191+0000", 2024-09-16T03:39:53.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8466000000000002e-05 2024-09-16T03:39:53.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830207+0000", 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6453e-05 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830362+0000", 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015450699999999999 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:06.830608+0000", 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024587399999999998 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.233387+0000", 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40277970099999999 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234208+0000", 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00082053900000000001 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.234370+0000", 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000161645 2024-09-16T03:39:53.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383821+0000", 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.14945145300000001 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383867+0000", 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5498999999999999e-05 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:07.383916+0000", 2024-09-16T03:39:53.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9385000000000002e-05 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2532 4.0 4:06b6d050:::smithi15736696-50:head [write 2603977~725887] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 97.232471748999998, 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53439557900000001, 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.895 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2532 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594293+0000", 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594298+0000", 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6030000000000004e-06 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594876+0000", 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00057757800000000003 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594885+0000", 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.4150000000000005e-06 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594892+0000", 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7229999999999998e-06 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628138+0000", 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.033246407999999998 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628252+0000", 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000113755 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:53.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.628344+0000", 2024-09-16T03:39:53.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2028999999999996e-05 2024-09-16T03:39:53.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:53.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730215+0000", 2024-09-16T03:39:53.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101870761 2024-09-16T03:39:53.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.732105+0000", 2024-09-16T03:39:53.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001889951 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128411+0000", 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.396305764 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128611+0000", 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019990599999999999 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128688+0000", 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7686000000000001e-05 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2534 4.0 4:06b6d050:::smithi15736696-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 97.231849973999999, 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.53406219200000005, 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:53.901 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2534 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594915+0000", 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8999999999999998e-07 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594916+0000", 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.344e-06 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594919+0000", 2024-09-16T03:39:53.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0580000000000002e-06 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.594922+0000", 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4389999999999999e-06 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.729999+0000", 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.13507707399999999 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:53.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.730046+0000", 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7382999999999997e-05 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128879+0000", 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39883298499999997 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128924+0000", 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:17.128977+0000", 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2648999999999997e-05 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2621 4.f 4:fa163101:::smithi15736696-10:head [setxattr _header (56) in=7b,truncate 3815189] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 93.584073595999996, 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55430782499999998, 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.906 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2621 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242691+0000", 2024-09-16T03:39:53.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242692+0000", 2024-09-16T03:39:53.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7599999999999996e-07 2024-09-16T03:39:53.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242695+0000", 2024-09-16T03:39:53.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.529e-06 2024-09-16T03:39:53.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242699+0000", 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1010000000000002e-06 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242703+0000", 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.518e-06 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.494935+0000", 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25223231200000001 2024-09-16T03:39:53.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495011+0000", 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5692999999999995e-05 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,14", 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495117+0000", 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000105944 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.596308+0000", 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101190825 2024-09-16T03:39:53.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796852+0000", 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20054443199999999 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796893+0000", 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1221999999999999e-05 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796946+0000", 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2435000000000002e-05 2024-09-16T03:39:53.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.796999+0000", 2024-09-16T03:39:53.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3038000000000003e-05 2024-09-16T03:39:53.912 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.912 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.912 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.912 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2622 4.f 4:fa163101:::smithi15736696-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e89)", 2024-09-16T03:39:53.912 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:53.912 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 93.584057740999995, 2024-09-16T03:39:53.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.55448354399999999, 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2622 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242707+0000", 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9999999999999999e-07 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242709+0000", 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.429e-06 2024-09-16T03:39:53.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242712+0000", 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6090000000000001e-06 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.242716+0000", 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6480000000000001e-06 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495371+0000", 2024-09-16T03:39:53.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.25265485700000001 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.495424+0000", 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2927999999999999e-05 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797042+0000", 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30161821900000002 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797114+0000", 2024-09-16T03:39:53.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2445000000000002e-05 2024-09-16T03:39:53.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.797190+0000", 2024-09-16T03:39:53.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6108999999999999e-05 2024-09-16T03:39:53.917 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.917 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.917 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.917 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4669 4.10 4:0a55bbf2:::smithi15736696-18:head [setxattr _header (56) in=7b,truncate 3142975] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:30.855601+0000", 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.971163621999999, 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.70839343799999999, 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4669 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855601+0000", 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855601+0000", 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855602+0000", 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8400000000000003e-07 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855605+0000", 2024-09-16T03:39:53.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5619999999999999e-06 2024-09-16T03:39:53.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855609+0000", 2024-09-16T03:39:53.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8909999999999997e-06 2024-09-16T03:39:53.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855613+0000", 2024-09-16T03:39:53.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1980000000000003e-06 2024-09-16T03:39:53.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.054797+0000", 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19918406499999999 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.054867+0000", 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9103000000000006e-05 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.054969+0000", 2024-09-16T03:39:53.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010247999999999999 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.155829+0000", 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10086023400000001 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.157532+0000", 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017031310000000001 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.563800+0000", 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40626808800000003 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.563926+0000", 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012529800000000001 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.563994+0000", 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8603999999999996e-05 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4670 4.10 4:0a55bbf2:::smithi15736696-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:30.855618+0000", 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.971146276999999, 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.80871130099999999, 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4670 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.924 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855618+0000", 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855618+0000", 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855619+0000", 2024-09-16T03:39:53.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.22e-07 2024-09-16T03:39:53.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855620+0000", 2024-09-16T03:39:53.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7099999999999999e-06 2024-09-16T03:39:53.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855624+0000", 2024-09-16T03:39:53.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.287e-06 2024-09-16T03:39:53.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.855627+0000", 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.05e-06 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.155366+0000", 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.299739861 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.155514+0000", 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014730099999999999 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.664164+0000", 2024-09-16T03:39:53.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50864991100000001 2024-09-16T03:39:53.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.664247+0000", 2024-09-16T03:39:53.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3811000000000001e-05 2024-09-16T03:39:53.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.664330+0000", 2024-09-16T03:39:53.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2047999999999998e-05 2024-09-16T03:39:53.929 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.929 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.929 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.932 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4675 4.0 4:023abada:::smithi15736696-17:head [write 2012035~687143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:53.932 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:31.097063+0000", 2024-09-16T03:39:53.933 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.729701266999999, 2024-09-16T03:39:53.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.56545287600000005, 2024-09-16T03:39:53.933 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.933 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.933 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:53.933 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.933 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.933 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.933 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4675 2024-09-16T03:39:53.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.933 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097063+0000", 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097063+0000", 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097065+0000", 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.407e-06 2024-09-16T03:39:53.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097709+0000", 2024-09-16T03:39:53.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00064482299999999999 2024-09-16T03:39:53.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097721+0000", 2024-09-16T03:39:53.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1764e-05 2024-09-16T03:39:53.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097728+0000", 2024-09-16T03:39:53.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4289999999999998e-06 2024-09-16T03:39:53.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.936 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.159378+0000", 2024-09-16T03:39:53.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.061650127999999998 2024-09-16T03:39:53.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.936 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.159513+0000", 2024-09-16T03:39:53.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013524699999999999 2024-09-16T03:39:53.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:53.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.159613+0000", 2024-09-16T03:39:53.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9952999999999997e-05 2024-09-16T03:39:53.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:53.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.162018+0000", 2024-09-16T03:39:53.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0024047539999999998 2024-09-16T03:39:53.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.256225+0000", 2024-09-16T03:39:53.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.094207338000000002 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.662322+0000", 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40609676 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.662479+0000", 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015684999999999999 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.662516+0000", 2024-09-16T03:39:53.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7422999999999997e-05 2024-09-16T03:39:53.939 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.939 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.939 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.939 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4676 4.0 4:023abada:::smithi15736696-17:head [setxattr _header (56) in=7b,truncate 3159782] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:53.939 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:31.097738+0000", 2024-09-16T03:39:53.939 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.729026262000001, 2024-09-16T03:39:53.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.66492440100000005, 2024-09-16T03:39:53.939 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.939 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.939 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4676 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097738+0000", 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097738+0000", 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097739+0000", 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6199999999999997e-07 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097742+0000", 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.202e-06 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097746+0000", 2024-09-16T03:39:53.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5420000000000001e-06 2024-09-16T03:39:53.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097749+0000", 2024-09-16T03:39:53.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2289999999999999e-06 2024-09-16T03:39:53.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.161472+0000", 2024-09-16T03:39:53.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.063722665999999997 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.161547+0000", 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5673999999999994e-05 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,15", 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.161650+0000", 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010220100000000001 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.263634+0000", 2024-09-16T03:39:53.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101984436 2024-09-16T03:39:53.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.944 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.944 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.944 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.263667+0000", 2024-09-16T03:39:53.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2660000000000002e-05 2024-09-16T03:39:53.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.944 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.944 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:53.944 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.762551+0000", 2024-09-16T03:39:53.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.49888468899999999 2024-09-16T03:39:53.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.944 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.944 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:53.945 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.762595+0000", 2024-09-16T03:39:53.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3498000000000003e-05 2024-09-16T03:39:53.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.945 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.945 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.762663+0000", 2024-09-16T03:39:53.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7842000000000005e-05 2024-09-16T03:39:53.945 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.945 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.945 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.945 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4677 4.0 4:023abada:::smithi15736696-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:31.097753+0000", 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 22.729011647, 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.66511091099999997, 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4677 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:53.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097753+0000", 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097753+0000", 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097753+0000", 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7099999999999998e-07 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097755+0000", 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5349999999999999e-06 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097758+0000", 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3749999999999999e-06 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.097762+0000", 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5700000000000001e-06 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.256017+0000", 2024-09-16T03:39:53.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.15825552900000001 2024-09-16T03:39:53.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:53.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.256149+0000", 2024-09-16T03:39:53.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013168500000000001 2024-09-16T03:39:53.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:53.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.762695+0000", 2024-09-16T03:39:53.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50654610200000005 2024-09-16T03:39:53.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.762756+0000", 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0946999999999998e-05 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.762864+0000", 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000107897 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:53.950 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:53.951 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:53.952 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:53.950+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:53.953 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:39:53.956 INFO:tasks.rados.rados.0.smithi157.stdout:2032: writing smithi15736696-20 from 1642601 to 2408415 tid 2 2024-09-16T03:39:53.956 INFO:tasks.rados.rados.0.smithi157.stdout:2032: writing smithi15736696-20 from 2928991 to 3376698 tid 3 2024-09-16T03:39:53.956 INFO:tasks.rados.rados.0.smithi157.stdout:2021: done (13 left) 2024-09-16T03:39:53.956 INFO:tasks.rados.rados.0.smithi157.stdout:2023: done (12 left) 2024-09-16T03:39:53.956 INFO:tasks.rados.rados.0.smithi157.stdout:2025: done (11 left) 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:2026: done (10 left) 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:2027: done (9 left) 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:2028: done (8 left) 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:2030: done (7 left) 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:2031: done (6 left) 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:2033: read oid 14 snap -1 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:2033: expect (ObjNum 398 snap 0 seq_num 398) 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:2034: write oid 25 current snap is 0 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:2029: finishing write tid 5 to smithi15736696-4 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:2029: oid 4 updating version 0 to 175 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:2029: oid 4 updating version 175 to 176 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:2029: oid 4 version 176 is already newer than 174 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:2029: oid 4 version 176 is already newer than 173 2024-09-16T03:39:53.957 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 4 v 176 (ObjNum 454 snap 0 seq_num 454) dirty exists 2024-09-16T03:39:53.958 INFO:tasks.rados.rados.0.smithi157.stdout:2029: left oid 4 (ObjNum 454 snap 0 seq_num 454) 2024-09-16T03:39:53.958 INFO:tasks.rados.rados.0.smithi157.stdout:2034: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2024-09-16T03:39:53.958 INFO:tasks.rados.rados.0.smithi157.stdout:2034: writing smithi15736696-25 from 463992 to 1242363 tid 1 2024-09-16T03:39:53.958 INFO:tasks.rados.rados.0.smithi157.stdout:2034: writing smithi15736696-25 from 1701083 to 2326908 tid 2 2024-09-16T03:39:53.958 INFO:tasks.rados.rados.0.smithi157.stdout:2034: writing smithi15736696-25 from 2956414 to 3615393 tid 3 2024-09-16T03:39:53.958 INFO:tasks.rados.rados.0.smithi157.stdout:2029: done (7 left) 2024-09-16T03:39:53.958 INFO:tasks.rados.rados.0.smithi157.stdout:2035: set_chunk oid 47 target oid 2024-09-16T03:39:53.958 INFO:tasks.rados.rados.0.smithi157.stdout:2035: set_chunk oid 47 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:39:53.958 INFO:tasks.rados.rados.0.smithi157.stdout:2036: read oid 13 snap -1 2024-09-16T03:39:53.958 INFO:tasks.rados.rados.0.smithi157.stdout:2036: expect (ObjNum 401 snap 0 seq_num 401) 2024-09-16T03:39:53.958 INFO:tasks.rados.rados.0.smithi157.stdout:2037: write oid 16 current snap is 0 2024-09-16T03:39:53.958 INFO:tasks.rados.rados.0.smithi157.stdout:2037: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2024-09-16T03:39:53.958 INFO:tasks.rados.rados.0.smithi157.stdout:2037: writing smithi15736696-16 from 480799 to 1050772 tid 1 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2037: writing smithi15736696-16 from 1475918 to 2245401 tid 2 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2037: writing smithi15736696-16 from 2700190 to 3286794 tid 3 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2038: write (excl) oid 24 current snap is 0 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2038: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2038: writing smithi15736696-24 from 497606 to 1142828 tid 1 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2038: writing smithi15736696-24 from 1934400 to 2563894 tid 2 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2038: writing smithi15736696-24 from 3243966 to 3697606 tid 3 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2039: write oid 43 current snap is 0 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2039: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2039: writing smithi15736696-43 from 514413 to 1234884 tid 1 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2039: writing smithi15736696-43 from 1876529 to 2649681 tid 2 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2039: writing smithi15736696-43 from 3438683 to 3714413 tid 3 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2040: tier_promote oid 39 2024-09-16T03:39:53.959 INFO:tasks.rados.rados.0.smithi157.stdout:2035: finishing set_chunk to oid 47 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2035: got expected ENOENT (src dne) 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2035: done (12 left) 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2041: read oid 34 snap -1 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2041: expect (ObjNum 450 snap 0 seq_num 450) 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2042: tier_promote oid 8 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2043: set_chunk oid 41 target oid 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2043: set_chunk oid 41 offset: 0 length: 4000000 target oid smithi15736696- offset: 0 snap -1 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2044: read oid 22 snap -1 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2044: expect (ObjNum 444 snap 0 seq_num 444) 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2043: finishing set_chunk to oid 41 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2043: got expected ENOENT (src dne) 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2043: done (15 left) 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2045: read oid 11 snap -1 2024-09-16T03:39:53.960 INFO:tasks.rados.rados.0.smithi157.stdout:2045: expect (ObjNum 428 snap 0 seq_num 428) 2024-09-16T03:39:53.961 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 16 2024-09-16T03:39:53.963 INFO:tasks.rados.rados.0.smithi157.stdout:2036: expect (ObjNum 401 snap 0 seq_num 401) 2024-09-16T03:39:53.963 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:54.053 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:54.051+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:54.063 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:54.068 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:54.068 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:54.068 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:39:54.068 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:54.077 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:39:54.153 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:54.150+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:54.182 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:54.182 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:39:54.183 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:39:54.183 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:39:54.183 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:54.191 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:39:54.254 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:54.252+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:54.255 INFO:tasks.rados.rados.0.smithi157.stdout:2032: finishing write tid 1 to smithi15736696-20 2024-09-16T03:39:54.255 INFO:tasks.rados.rados.0.smithi157.stdout:2032: finishing write tid 2 to smithi15736696-20 2024-09-16T03:39:54.255 INFO:tasks.rados.rados.0.smithi157.stdout:2034: finishing write tid 1 to smithi15736696-25 2024-09-16T03:39:54.255 INFO:tasks.rados.rados.0.smithi157.stdout:2041: expect (ObjNum 450 snap 0 seq_num 450) 2024-09-16T03:39:54.300 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:39:54.301 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:39:54.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:39:54.301 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:39:54.301 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.301 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:54.301 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:54.301 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 162.154471941, 2024-09-16T03:39:54.301 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:39:54.301 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.301 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.301 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.301 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:39:54.302 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.303 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:39:54.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:39:54.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:39:54.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:39:54.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.304 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.304 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:39:54.304 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:39:54.304 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.304 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.305 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:39:54.306 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:54.306 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 162.15411858100001, 2024-09-16T03:39:54.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:39:54.306 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.306 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.306 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.306 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.306 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.306 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:54.306 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.306 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.306 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:39:54.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:39:54.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:39:54.307 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.307 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.307 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.307 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:39:54.307 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.308 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:39:54.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:39:54.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:39:54.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:39:54.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.309 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.309 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:39:54.309 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:39:54.309 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.309 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 162.15299078300001, 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.310 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.311 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.312 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.313 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.314 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 162.152003342, 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.315 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.316 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.317 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:39:54.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:39:54.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:39:54.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:39:54.318 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.318 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.318 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.318 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:39:54.318 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 162.151246598, 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.319 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.320 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:39:54.321 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:39:54.322 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:39:54.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.323 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.323 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:39:54.323 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:39:54.323 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.323 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.323 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.323 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.323 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.323 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:989 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'31, mlcod=46'28)", 2024-09-16T03:39:54.323 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:54.323 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 143.91039523800001, 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40332266900000002, 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:39:54.324 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387812+0000", 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388467+0000", 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065539500000000002 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.325 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388481+0000", 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3905e-05 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488552+0000", 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10007120899999999 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688818+0000", 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20026627599999999 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.326 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688845+0000", 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791091+0000", 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10224588699999999 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791131+0000", 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0293000000000002e-05 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.327 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:990 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'32, mlcod=46'28)", 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 143.809611566, 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30256419200000001, 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.328 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.329 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:39:54.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.329 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488594+0000", 2024-09-16T03:39:54.329 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:39:54.329 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.329 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.329 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.329 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488616+0000", 2024-09-16T03:39:54.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2419e-05 2024-09-16T03:39:54.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.330 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.330 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488627+0000", 2024-09-16T03:39:54.330 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1251e-05 2024-09-16T03:39:54.330 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.330 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588753+0000", 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012544399999999 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690679+0000", 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101926027 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690695+0000", 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6528000000000002e-05 2024-09-16T03:39:54.331 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791133+0000", 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100438233 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791157+0000", 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-05 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.332 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3081 4.1f e102/100 4:fa163101:::smithi15736696-10:head v 102'93, mlcod=102'91)", 2024-09-16T03:39:54.333 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:54.333 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 78.964489098000001, 2024-09-16T03:39:54.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20437049600000001, 2024-09-16T03:39:54.333 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.333 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.333 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.333 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.333 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.333 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:54.333 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.333 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.333 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333718+0000", 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3720000000000001e-06 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334338+0000", 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061959099999999996 2024-09-16T03:39:54.334 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334355+0000", 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.677e-05 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334364+0000", 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3780000000000007e-06 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334424+0000", 2024-09-16T03:39:54.335 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0084000000000003e-05 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334447+0000", 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3113e-05 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538030+0000", 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20358272399999999 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.336 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538085+0000", 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5464000000000003e-05 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4007 4.37 4:ed874e32:::smithi15736696-14:head [write 601783~551825,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 48.765005252000002, 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29242759600000001, 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.337 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4007 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:39:54.338 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533200+0000", 2024-09-16T03:39:54.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3600000000000002e-07 2024-09-16T03:39:54.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.339 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533841+0000", 2024-09-16T03:39:54.339 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00064141100000000004 2024-09-16T03:39:54.339 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.339 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.339 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533862+0000", 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0687e-05 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634001+0000", 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100139413 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721351+0000", 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.087349914000000001 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.340 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721357+0000", 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7100000000000004e-06 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721891+0000", 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053461299999999997 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.722024+0000", 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013274599999999999 2024-09-16T03:39:54.341 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.722158+0000", 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013362399999999999 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725271+0000", 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003113345 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825258+0000", 2024-09-16T03:39:54.342 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099987260999999994 2024-09-16T03:39:54.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825453+0000", 2024-09-16T03:39:54.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019453999999999999 2024-09-16T03:39:54.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.343 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825591+0000", 2024-09-16T03:39:54.343 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013849700000000001 2024-09-16T03:39:54.343 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.343 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.343 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.344 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825626+0000", 2024-09-16T03:39:54.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4999000000000002e-05 2024-09-16T03:39:54.344 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.344 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.344 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.344 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.344 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.344 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4008 4.37 4:ed874e32:::smithi15736696-14:head [write 1776679~425104] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:54.344 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:39:54.344 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 48.664185129000003, 2024-09-16T03:39:54.344 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29371618199999999, 2024-09-16T03:39:54.344 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.344 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4008 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.345 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634021+0000", 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9989999999999998e-06 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634187+0000", 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016655199999999999 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.346 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634197+0000", 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7520000000000008e-06 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634205+0000", 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8210000000000002e-06 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725407+0000", 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091202249999999999 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.347 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725506+0000", 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8493999999999996e-05 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725648+0000", 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014187299999999999 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.727142+0000", 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001494478 2024-09-16T03:39:54.348 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825761+0000", 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098619340999999999 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927671+0000", 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10190922700000001 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927705+0000", 2024-09-16T03:39:54.349 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4523000000000001e-05 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927735+0000", 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9872000000000001e-05 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4009 4.37 4:ed874e32:::smithi15736696-14:head [setxattr _header (56) in=7b,truncate 2201783] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 48.663988201999999, 2024-09-16T03:39:54.350 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29364004500000002, 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4009 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.351 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634217+0000", 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7199999999999998e-07 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634221+0000", 2024-09-16T03:39:54.352 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9650000000000002e-06 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634226+0000", 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7830000000000004e-06 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634232+0000", 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2909999999999998e-06 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.353 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726750+0000", 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.09251827 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726830+0000", 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9808999999999994e-05 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726916+0000", 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6055000000000002e-05 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:54.354 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.727785+0000", 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086898599999999998 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.926315+0000", 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19852985000000001 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927777+0000", 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001462138 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.355 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927818+0000", 2024-09-16T03:39:54.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1006e-05 2024-09-16T03:39:54.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.356 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.356 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927856+0000", 2024-09-16T03:39:54.356 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8120000000000001e-05 2024-09-16T03:39:54.356 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.356 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.356 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.356 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.356 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.356 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4010 4.37 4:ed874e32:::smithi15736696-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:39:54.357 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:54.357 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 48.663964509000003, 2024-09-16T03:39:54.357 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39386449699999998, 2024-09-16T03:39:54.357 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.357 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.357 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:54.357 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:54.357 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:54.357 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.357 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4010 2024-09-16T03:39:54.357 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.357 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.357 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:39:54.358 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634243+0000", 2024-09-16T03:39:54.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.869e-06 2024-09-16T03:39:54.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634249+0000", 2024-09-16T03:39:54.359 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9449999999999998e-06 2024-09-16T03:39:54.359 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.359 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.359 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.359 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634255+0000", 2024-09-16T03:39:54.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1979999999999998e-06 2024-09-16T03:39:54.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.734247+0000", 2024-09-16T03:39:54.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099991570000000002 2024-09-16T03:39:54.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.360 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:54.360 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.734386+0000", 2024-09-16T03:39:54.360 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013961600000000001 2024-09-16T03:39:54.360 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.360 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.027978+0000", 2024-09-16T03:39:54.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29359164599999998 2024-09-16T03:39:54.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.028036+0000", 2024-09-16T03:39:54.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7802e-05 2024-09-16T03:39:54.361 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.361 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.361 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.361 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.028104+0000", 2024-09-16T03:39:54.361 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8264999999999998e-05 2024-09-16T03:39:54.362 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.362 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.362 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.362 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.362 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.362 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4034 4.17 4:eb60a28d:::smithi15736696-41:head [write 635397~702323,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:39:54.362 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:39:54.362 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 48.261543064000001, 2024-09-16T03:39:54.362 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471451500000001, 2024-09-16T03:39:54.362 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.362 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.362 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4034 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.363 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:39:54.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.364 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.364 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.364 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.364 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036663+0000", 2024-09-16T03:39:54.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.855e-06 2024-09-16T03:39:54.364 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.364 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.364 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.364 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037227+0000", 2024-09-16T03:39:54.364 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00056451099999999996 2024-09-16T03:39:54.364 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.364 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037240+0000", 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2208999999999999e-05 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037245+0000", 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6319999999999996e-06 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037305+0000", 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0272000000000002e-05 2024-09-16T03:39:54.365 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037384+0000", 2024-09-16T03:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8330000000000004e-05 2024-09-16T03:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037491+0000", 2024-09-16T03:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010730699999999999 2024-09-16T03:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:54.366 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.042284+0000", 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0047932649999999997 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.140922+0000", 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098637552000000003 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241267+0000", 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100344866 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241330+0000", 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2953000000000005e-05 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241375+0000", 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5763000000000002e-05 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "rep_scrubmap(4.32 e135 from shard 11)", 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:11.840602+0000", 2024-09-16T03:39:54.368 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.457601832999998, 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.26655261099999999, 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840602+0000", 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840602+0000", 2024-09-16T03:39:54.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840605+0000", 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840616+0000", 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0672e-05 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840632+0000", 2024-09-16T03:39:54.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6268000000000001e-05 2024-09-16T03:39:54.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.940701+0000", 2024-09-16T03:39:54.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10006937 2024-09-16T03:39:54.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.371 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.107074+0000", 2024-09-16T03:39:54.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.16637296600000001 2024-09-16T03:39:54.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.371 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.372 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.107101+0000", 2024-09-16T03:39:54.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6225999999999999e-05 2024-09-16T03:39:54.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.372 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.107155+0000", 2024-09-16T03:39:54.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4311999999999998e-05 2024-09-16T03:39:54.372 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.372 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.372 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.372 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4101 4.17 4:eb60a28d:::smithi15736696-41:head [write 719432~794921] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:11.929701+0000", 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.368502708999998, 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30975548400000003, 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4101 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.929701+0000", 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.929701+0000", 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.929704+0000", 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.21e-06 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.930544+0000", 2024-09-16T03:39:54.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00084025300000000001 2024-09-16T03:39:54.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.930561+0000", 2024-09-16T03:39:54.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7601999999999999e-05 2024-09-16T03:39:54.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.930571+0000", 2024-09-16T03:39:54.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1260000000000004e-06 2024-09-16T03:39:54.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.030702+0000", 2024-09-16T03:39:54.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10013184899999999 2024-09-16T03:39:54.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.376 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.030769+0000", 2024-09-16T03:39:54.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6742000000000006e-05 2024-09-16T03:39:54.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.030873+0000", 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104005 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.033373+0000", 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0025000489999999998 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.138540+0000", 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10516709 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.239289+0000", 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10074846599999999 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.239382+0000", 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3195000000000005e-05 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.239457+0000", 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4896999999999998e-05 2024-09-16T03:39:54.378 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.379 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.379 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.379 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4366 4.23 4:c512d873:::smithi15736696-38:head [assert-version v41,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:39:54.379 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:20.361285+0000", 2024-09-16T03:39:54.379 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 33.936918796, 2024-09-16T03:39:54.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.303627655, 2024-09-16T03:39:54.379 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.379 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.379 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.379 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:54.379 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4366 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361285+0000", 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361285+0000", 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361287+0000", 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.626e-06 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361292+0000", 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3279999999999996e-06 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361304+0000", 2024-09-16T03:39:54.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1574e-05 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361310+0000", 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1990000000000003e-06 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361362+0000", 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2459e-05 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361448+0000", 2024-09-16T03:39:54.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5928999999999997e-05 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.463126+0000", 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10167792 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.463778+0000", 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000652123 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.564004+0000", 2024-09-16T03:39:54.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100225143 2024-09-16T03:39:54.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.664718+0000", 2024-09-16T03:39:54.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100714673 2024-09-16T03:39:54.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:39:54.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.664865+0000", 2024-09-16T03:39:54.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014715099999999999 2024-09-16T03:39:54.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.664913+0000", 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7530000000000001e-05 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4526 4.12 4:4a9d235d:::smithi15736696-1:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.726433735000001, 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30344452999999999, 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.385 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:54.386 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:54.386 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:54.386 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.386 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4526 2024-09-16T03:39:54.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.386 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.386 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.386 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:39:54.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.386 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571773+0000", 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3750000000000001e-06 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571780+0000", 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5480000000000004e-06 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571797+0000", 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6320999999999998e-05 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571821+0000", 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4111e-05 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571884+0000", 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3608000000000005e-05 2024-09-16T03:39:54.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571981+0000", 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6255999999999995e-05 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.875215+0000", 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30323431099999998 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4539 4.12 4:4a9d235d:::smithi15736696-1:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:26.069749+0000", 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 28.228455278999999, 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.31900961999999999, 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4539 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069749+0000", 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069749+0000", 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069752+0000", 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4800000000000001e-06 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.391 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069779+0000", 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6336e-05 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069794+0000", 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5648e-05 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069805+0000", 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0403e-05 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.393 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069884+0000", 2024-09-16T03:39:54.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9660000000000006e-05 2024-09-16T03:39:54.393 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.393 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.393 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.393 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069982+0000", 2024-09-16T03:39:54.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7637e-05 2024-09-16T03:39:54.393 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.393 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.393 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.393 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.388758+0000", 2024-09-16T03:39:54.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.31877645599999999 2024-09-16T03:39:54.393 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4561 4.23 4:c512d873:::smithi15736696-38:head [read 0~2672379,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e148)", 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:26.675782+0000", 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.622421854999999, 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30481955599999999, 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:54.394 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4561 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675782+0000", 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675782+0000", 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.395 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.396 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.396 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675783+0000", 2024-09-16T03:39:54.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4900000000000001e-07 2024-09-16T03:39:54.396 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.396 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.396 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.396 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675785+0000", 2024-09-16T03:39:54.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6429999999999999e-06 2024-09-16T03:39:54.396 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.396 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.396 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.396 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675790+0000", 2024-09-16T03:39:54.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4859999999999999e-06 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675793+0000", 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0019999999999998e-06 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.776027+0000", 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10023393899999999 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:54.397 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.776034+0000", 2024-09-16T03:39:54.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.939e-06 2024-09-16T03:39:54.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.978683+0000", 2024-09-16T03:39:54.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264916399999999 2024-09-16T03:39:54.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.978742+0000", 2024-09-16T03:39:54.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8699000000000002e-05 2024-09-16T03:39:54.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.980602+0000", 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001860235 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4562 4.23 4:c512d873:::smithi15736696-38:head [read 0~2672379,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e148)", 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:26.675811+0000", 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 27.622393521999999, 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30577821199999999, 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:39:54.399 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4562 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675811+0000", 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:39:54.400 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675811+0000", 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675811+0000", 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5400000000000002e-07 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675816+0000", 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7539999999999997e-06 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675821+0000", 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8160000000000001e-06 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.775929+0000", 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100108076 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.776062+0000", 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013323400000000001 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:54.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.776067+0000", 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1669999999999998e-06 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.980656+0000", 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.204588772 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.980711+0000", 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4898999999999997e-05 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:39:54.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:39:54.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.981589+0000", 2024-09-16T03:39:54.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00087803999999999996 2024-09-16T03:39:54.404 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.404 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.404 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.404 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:39:54.404 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:39:54.404 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:39:54.405 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 414331 2024-09-16T03:39:54.405 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:54.406 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:39:54.406 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:54.404+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:54.479 INFO:tasks.rados.rados.0.smithi157.stdout:2032: finishing write tid 3 to smithi15736696-20 2024-09-16T03:39:54.479 INFO:tasks.rados.rados.0.smithi157.stdout:2032: finishing write tid 4 to smithi15736696-20 2024-09-16T03:39:54.480 INFO:tasks.rados.rados.0.smithi157.stdout:2032: finishing write tid 5 to smithi15736696-20 2024-09-16T03:39:54.480 INFO:tasks.rados.rados.0.smithi157.stdout:2032: oid 20 updating version 0 to 64 2024-09-16T03:39:54.480 INFO:tasks.rados.rados.0.smithi157.stdout:2032: oid 20 version 64 is already newer than 61 2024-09-16T03:39:54.480 INFO:tasks.rados.rados.0.smithi157.stdout:2032: oid 20 version 64 is already newer than 63 2024-09-16T03:39:54.480 INFO:tasks.rados.rados.0.smithi157.stdout:2032: oid 20 version 64 is already newer than 62 2024-09-16T03:39:54.480 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 20 v 64 (ObjNum 455 snap 0 seq_num 455) dirty exists 2024-09-16T03:39:54.480 INFO:tasks.rados.rados.0.smithi157.stdout:2032: left oid 20 (ObjNum 455 snap 0 seq_num 455) 2024-09-16T03:39:54.480 INFO:tasks.rados.rados.0.smithi157.stdout:2040: got (0) Success 2024-09-16T03:39:54.480 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 39 v 85 (ObjNum 365 snap 0 seq_num 365) dirty exists 2024-09-16T03:39:54.480 INFO:tasks.rados.rados.0.smithi157.stdout:2034: finishing write tid 2 to smithi15736696-25 2024-09-16T03:39:54.480 INFO:tasks.rados.rados.0.smithi157.stdout:2032: done (15 left) 2024-09-16T03:39:54.480 INFO:tasks.rados.rados.0.smithi157.stdout:2036: done (14 left) 2024-09-16T03:39:54.480 INFO:tasks.rados.rados.0.smithi157.stdout:2040: done (13 left) 2024-09-16T03:39:54.481 INFO:tasks.rados.rados.0.smithi157.stdout:2041: done (12 left) 2024-09-16T03:39:54.481 INFO:tasks.rados.rados.0.smithi157.stdout:2046: read oid 29 snap -1 2024-09-16T03:39:54.481 INFO:tasks.rados.rados.0.smithi157.stdout:2046: expect (ObjNum 452 snap 0 seq_num 452) 2024-09-16T03:39:54.481 INFO:tasks.rados.rados.0.smithi157.stdout:2047: read oid 12 snap -1 2024-09-16T03:39:54.481 INFO:tasks.rados.rados.0.smithi157.stdout:2047: expect (ObjNum 451 snap 0 seq_num 451) 2024-09-16T03:39:54.481 INFO:tasks.rados.rados.0.smithi157.stdout:2048: write (excl) oid 41 current snap is 0 2024-09-16T03:39:54.481 INFO:tasks.rados.rados.0.smithi157.stdout:2022: expect (ObjNum 436 snap 0 seq_num 436) 2024-09-16T03:39:54.506 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:54.505+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:54.513 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:54.513 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:54.513 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.513 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5068 4.35 4.82332ff5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:39:54.513 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.911828+0000", 2024-09-16T03:39:54.513 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.60012137899999995, 2024-09-16T03:39:54.513 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60013888900000001, 2024-09-16T03:39:54.513 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.513 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.513 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:39:54.513 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5068 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911828+0000", 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911828+0000", 2024-09-16T03:39:54.514 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911829+0000", 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1799999999999995e-07 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911831+0000", 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8499999999999998e-06 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.515 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911837+0000", 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3929999999999999e-06 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911841+0000", 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5069 4.35 4.82332ff5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.911846+0000", 2024-09-16T03:39:54.516 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.60010380299999999, 2024-09-16T03:39:54.517 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60015286400000001, 2024-09-16T03:39:54.517 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.517 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.517 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "queued for pg", 2024-09-16T03:39:54.517 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.517 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.517 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.517 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5069 2024-09-16T03:39:54.517 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.517 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.517 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.517 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.517 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911846+0000", 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911846+0000", 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911846+0000", 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1300000000000001e-07 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.518 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911847+0000", 2024-09-16T03:39:54.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.559e-06 2024-09-16T03:39:54.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911850+0000", 2024-09-16T03:39:54.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.903e-06 2024-09-16T03:39:54.519 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.519 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.519 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.519 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.011923+0000", 2024-09-16T03:39:54.519 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100072736 2024-09-16T03:39:54.519 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.519 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.520 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.520 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.520 INFO:teuthology.orchestra.run.smithi157.stdout: ], 2024-09-16T03:39:54.520 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 2 2024-09-16T03:39:54.520 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:54.521 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:39:54.607 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:54.605+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:54.630 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:54.631 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:54.631 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:54.631 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:54.631 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:54.639 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:39:54.687 INFO:tasks.rados.rados.0.smithi157.stdout:2048: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-09-16T03:39:54.695 INFO:tasks.rados.rados.0.smithi157.stdout:2048: writing smithi15736696-41 from 531220 to 1326940 tid 1 2024-09-16T03:39:54.701 INFO:tasks.rados.rados.0.smithi157.stdout:2048: writing smithi15736696-41 from 1935011 to 2451821 tid 2 2024-09-16T03:39:54.707 INFO:tasks.rados.rados.0.smithi157.stdout:2048: writing smithi15736696-41 from 3066106 to 3486526 tid 3 2024-09-16T03:39:54.707 INFO:tasks.rados.rados.0.smithi157.stdout:2022: done (14 left) 2024-09-16T03:39:54.707 INFO:tasks.rados.rados.0.smithi157.stdout:2049: write (excl) oid 40 current snap is 0 2024-09-16T03:39:54.708 INFO:tasks.rados.rados.0.smithi157.stdout:2011: got (22) Invalid argument 2024-09-16T03:39:54.708 INFO:tasks.rados.rados.0.smithi157.stdout:2049: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2024-09-16T03:39:54.708 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:54.706+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:54.712 INFO:tasks.rados.rados.0.smithi157.stdout:2049: writing smithi15736696-40 from 548027 to 1018996 tid 1 2024-09-16T03:39:54.720 INFO:tasks.rados.rados.0.smithi157.stdout:2049: writing smithi15736696-40 from 1593493 to 2370314 tid 2 2024-09-16T03:39:54.727 INFO:tasks.rados.rados.0.smithi157.stdout:2049: writing smithi15736696-40 from 3093529 to 3725221 tid 3 2024-09-16T03:39:54.727 INFO:tasks.rados.rados.0.smithi157.stdout:2011: done (14 left) 2024-09-16T03:39:54.727 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 14 2024-09-16T03:39:54.727 INFO:tasks.rados.rados.0.smithi157.stdout:2009: finishing write tid 1 to smithi15736696-26 2024-09-16T03:39:54.727 INFO:tasks.rados.rados.0.smithi157.stdout:2009: finishing write tid 2 to smithi15736696-26 2024-09-16T03:39:54.727 INFO:tasks.rados.rados.0.smithi157.stdout:2009: finishing write tid 3 to smithi15736696-26 2024-09-16T03:39:54.728 INFO:tasks.rados.rados.0.smithi157.stdout:2044: expect (ObjNum 444 snap 0 seq_num 444) 2024-09-16T03:39:54.746 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.746 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3591 4.1b 4:dcb6965e:::smithi15736696-28:head [write 3335341~194162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:39:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.687171401999997, 2024-09-16T03:39:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60146457399999997, 2024-09-16T03:39:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3591 2024-09-16T03:39:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.747 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056407+0000", 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8920000000000001e-06 2024-09-16T03:39:54.748 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056502+0000", 2024-09-16T03:39:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4838000000000002e-05 2024-09-16T03:39:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056511+0000", 2024-09-16T03:39:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3579999999999992e-06 2024-09-16T03:39:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.749 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156581+0000", 2024-09-16T03:39:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100069254 2024-09-16T03:39:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256413+0000", 2024-09-16T03:39:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099832613000000001 2024-09-16T03:39:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256479+0000", 2024-09-16T03:39:54.750 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5388000000000005e-05 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256582+0000", 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000102876 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257889+0000", 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001307119 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.751 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657701+0000", 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39981235999999998 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657769+0000", 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7508000000000005e-05 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657827+0000", 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8878000000000001e-05 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.752 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657870+0000", 2024-09-16T03:39:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2490000000000001e-05 2024-09-16T03:39:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3592 4.1b 4:dcb6965e:::smithi15736696-28:head [setxattr _header (56) in=7b,truncate 3529503] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:39:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.586963249999997, 2024-09-16T03:39:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60139765499999998, 2024-09-16T03:39:54.753 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3592 2024-09-16T03:39:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:39:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.754 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156615+0000", 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.714e-06 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156623+0000", 2024-09-16T03:39:54.755 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5939999999999996e-06 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156634+0000", 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.093e-05 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156640+0000", 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1890000000000004e-06 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.756 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257172+0000", 2024-09-16T03:39:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100532525 2024-09-16T03:39:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257233+0000", 2024-09-16T03:39:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0736999999999998e-05 2024-09-16T03:39:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:39:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257319+0000", 2024-09-16T03:39:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5870000000000003e-05 2024-09-16T03:39:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.757 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257891+0000", 2024-09-16T03:39:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057231100000000004 2024-09-16T03:39:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657732+0000", 2024-09-16T03:39:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39984067699999998 2024-09-16T03:39:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.757877+0000", 2024-09-16T03:39:54.758 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100145238 2024-09-16T03:39:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.757947+0000", 2024-09-16T03:39:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9821999999999997e-05 2024-09-16T03:39:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758011+0000", 2024-09-16T03:39:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4047999999999994e-05 2024-09-16T03:39:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.759 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3593 4.1b 4:dcb6965e:::smithi15736696-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 63.586928516, 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60159266700000003, 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3593 2024-09-16T03:39:54.760 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:39:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:39:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.761 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156649+0000", 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6400000000000002e-07 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156652+0000", 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9579999999999999e-06 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156658+0000", 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4729999999999997e-06 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.762 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156667+0000", 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1940000000000004e-06 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257836+0000", 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101168444 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257881+0000", 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4910999999999997e-05 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.763 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758073+0000", 2024-09-16T03:39:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50019239699999996 2024-09-16T03:39:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758147+0000", 2024-09-16T03:39:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3360000000000002e-05 2024-09-16T03:39:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758241+0000", 2024-09-16T03:39:54.764 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4265999999999995e-05 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4069 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548281+0000", 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.195295799, 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62975188199999999, 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.765 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4069 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548281+0000", 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548281+0000", 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.766 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548284+0000", 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548292+0000", 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2169999999999994e-06 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548308+0000", 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5602000000000001e-05 2024-09-16T03:39:54.767 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548322+0000", 2024-09-16T03:39:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-05 2024-09-16T03:39:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548706+0000", 2024-09-16T03:39:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038430199999999999 2024-09-16T03:39:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:54.768 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548721+0000", 2024-09-16T03:39:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4440999999999999e-05 2024-09-16T03:39:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.177929+0000", 2024-09-16T03:39:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62920806500000004 2024-09-16T03:39:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.178033+0000", 2024-09-16T03:39:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103824 2024-09-16T03:39:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.769 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4070 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:39:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548345+0000", 2024-09-16T03:39:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.195231941999999, 2024-09-16T03:39:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.729858275, 2024-09-16T03:39:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.770 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4070 2024-09-16T03:39:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548345+0000", 2024-09-16T03:39:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.771 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548345+0000", 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548346+0000", 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.64e-07 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548354+0000", 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.875e-06 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.772 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548377+0000", 2024-09-16T03:39:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3612999999999999e-05 2024-09-16T03:39:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548390+0000", 2024-09-16T03:39:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2836999999999999e-05 2024-09-16T03:39:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548761+0000", 2024-09-16T03:39:54.773 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037075600000000003 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548769+0000", 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5110000000000003e-06 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278121+0000", 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72935201800000005 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.774 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278203+0000", 2024-09-16T03:39:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1700999999999996e-05 2024-09-16T03:39:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4071 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:39:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548403+0000", 2024-09-16T03:39:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.195173951000001, 2024-09-16T03:39:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73003053100000004, 2024-09-16T03:39:54.775 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4071 2024-09-16T03:39:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548403+0000", 2024-09-16T03:39:54.776 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548403+0000", 2024-09-16T03:39:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548404+0000", 2024-09-16T03:39:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0500000000000002e-07 2024-09-16T03:39:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.777 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548408+0000", 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2699999999999998e-06 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548415+0000", 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7340000000000002e-06 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548421+0000", 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.84e-06 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548798+0000", 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000377916 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548806+0000", 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5599999999999996e-06 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278371+0000", 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.729564616 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278433+0000", 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2689999999999998e-05 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4079 3.0 3:18759fb9:::13e91a6c9fb63c85209a4d9065cc1e1225929f8ddc4955a7e63272d6c7055a80:head [write 0~1183232] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548428+0000", 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.195148666000001, 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0303502819999999, 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.780 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4079 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548428+0000", 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.781 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548428+0000", 2024-09-16T03:39:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548429+0000", 2024-09-16T03:39:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2399999999999998e-07 2024-09-16T03:39:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549249+0000", 2024-09-16T03:39:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00081996300000000005 2024-09-16T03:39:54.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549263+0000", 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4319e-05 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549271+0000", 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1640000000000001e-06 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549342+0000", 2024-09-16T03:39:54.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0638999999999999e-05 2024-09-16T03:39:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549353+0000", 2024-09-16T03:39:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1653e-05 2024-09-16T03:39:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.274940+0000", 2024-09-16T03:39:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72558676 2024-09-16T03:39:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.275056+0000", 2024-09-16T03:39:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011581 2024-09-16T03:39:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,6,9", 2024-09-16T03:39:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.275176+0000", 2024-09-16T03:39:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119963 2024-09-16T03:39:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.377903+0000", 2024-09-16T03:39:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10272743199999999 2024-09-16T03:39:54.785 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578407+0000", 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200503758 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578475+0000", 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7718000000000005e-05 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578663+0000", 2024-09-16T03:39:54.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018870599999999999 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578706+0000", 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2459000000000001e-05 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578778+0000", 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2414000000000002e-05 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4534 4.35 4:ad0b9852:::smithi15736696-16:head [write 574905~711397 in=711397b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.060664+0000", 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 28.682912940000001, 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62894715300000004, 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4534 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060664+0000", 2024-09-16T03:39:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060664+0000", 2024-09-16T03:39:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060665+0000", 2024-09-16T03:39:54.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2330000000000001e-06 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060808+0000", 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014335500000000001 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060819+0000", 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0828e-05 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060826+0000", 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7209999999999997e-06 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.237917+0000", 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17709092100000001 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.237985+0000", 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8270999999999997e-05 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.238092+0000", 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000107324 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241761+0000", 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036689190000000001 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243578+0000", 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018165029999999999 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.793 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.793 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.689508+0000", 2024-09-16T03:39:54.793 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.44593033199999998 2024-09-16T03:39:54.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.793 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.793 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:54.793 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.689561+0000", 2024-09-16T03:39:54.793 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2401000000000002e-05 2024-09-16T03:39:54.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.793 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.793 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.793 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.689611+0000", 2024-09-16T03:39:54.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0345e-05 2024-09-16T03:39:54.794 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.794 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.794 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.794 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.794 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4535 4.35 4:ad0b9852:::smithi15736696-16:head [write 1885086~525140 in=525140b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:54.794 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.066720+0000", 2024-09-16T03:39:54.794 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 28.676856738000001, 2024-09-16T03:39:54.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72311801499999995, 2024-09-16T03:39:54.794 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.794 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.794 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.795 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.795 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.795 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.795 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4535 2024-09-16T03:39:54.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.795 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.795 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066720+0000", 2024-09-16T03:39:54.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066720+0000", 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066721+0000", 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.111e-06 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066954+0000", 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023295100000000001 2024-09-16T03:39:54.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066965+0000", 2024-09-16T03:39:54.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0845e-05 2024-09-16T03:39:54.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066970+0000", 2024-09-16T03:39:54.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5459999999999999e-06 2024-09-16T03:39:54.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239646+0000", 2024-09-16T03:39:54.798 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17267560400000001 2024-09-16T03:39:54.798 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.798 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.798 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.798 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239715+0000", 2024-09-16T03:39:54.798 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8566999999999996e-05 2024-09-16T03:39:54.798 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.798 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.798 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:39:54.798 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239843+0000", 2024-09-16T03:39:54.798 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000128861 2024-09-16T03:39:54.798 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.798 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241769+0000", 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001925544 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243609+0000", 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018399340000000001 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789655+0000", 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54604627900000002 2024-09-16T03:39:54.799 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.800 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:54.800 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789709+0000", 2024-09-16T03:39:54.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3479000000000001e-05 2024-09-16T03:39:54.800 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.800 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.800 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789838+0000", 2024-09-16T03:39:54.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012929400000000001 2024-09-16T03:39:54.800 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.800 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.800 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.800 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.801 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4536 4.35 4:ad0b9852:::smithi15736696-16:head [write 2930189~44716 in=44716b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:54.801 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.069712+0000", 2024-09-16T03:39:54.801 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 28.673865127999999, 2024-09-16T03:39:54.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72027944300000002, 2024-09-16T03:39:54.801 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.801 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.801 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.801 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.801 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.801 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.801 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4536 2024-09-16T03:39:54.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.802 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069712+0000", 2024-09-16T03:39:54.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069712+0000", 2024-09-16T03:39:54.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069713+0000", 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1209999999999999e-06 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069748+0000", 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5626000000000001e-05 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069765+0000", 2024-09-16T03:39:54.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7104000000000001e-05 2024-09-16T03:39:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069771+0000", 2024-09-16T03:39:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4310000000000002e-06 2024-09-16T03:39:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241008+0000", 2024-09-16T03:39:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17123754899999999 2024-09-16T03:39:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241070+0000", 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1351000000000003e-05 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241165+0000", 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5372999999999996e-05 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241774+0000", 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000609258 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243890+0000", 2024-09-16T03:39:54.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021156669999999999 2024-09-16T03:39:54.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789905+0000", 2024-09-16T03:39:54.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54601474800000005 2024-09-16T03:39:54.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:54.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789952+0000", 2024-09-16T03:39:54.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7539e-05 2024-09-16T03:39:54.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.807 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789991+0000", 2024-09-16T03:39:54.807 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8676e-05 2024-09-16T03:39:54.807 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.807 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.807 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.807 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4537 4.35 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=63b,truncate 2974905] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:54.807 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.069780+0000", 2024-09-16T03:39:54.808 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 28.673796634999999, 2024-09-16T03:39:54.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72043964199999999, 2024-09-16T03:39:54.808 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.808 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.808 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.808 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.808 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.808 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.808 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4537 2024-09-16T03:39:54.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.808 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069780+0000", 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069780+0000", 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069781+0000", 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069784+0000", 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9170000000000002e-06 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069788+0000", 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7230000000000004e-06 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069791+0000", 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0060000000000001e-06 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241479+0000", 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17168776899999999 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241523+0000", 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4270000000000001e-05 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241598+0000", 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4264000000000006e-05 2024-09-16T03:39:54.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.812 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.812 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:54.812 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241973+0000", 2024-09-16T03:39:54.812 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037565999999999999 2024-09-16T03:39:54.812 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.812 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.812 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.812 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.344072+0000", 2024-09-16T03:39:54.812 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10209850500000001 2024-09-16T03:39:54.812 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.812 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.812 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790106+0000", 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.44603410500000001 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790149+0000", 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3204000000000002e-05 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790220+0000", 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0640000000000001e-05 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.813 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.814 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.814 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4538 4.35 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:39:54.814 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:39:54.814 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 28.673781108, 2024-09-16T03:39:54.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72066842900000005, 2024-09-16T03:39:54.814 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.814 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.814 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:54.814 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.814 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.814 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4538 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:39:54.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.03e-07 2024-09-16T03:39:54.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.816 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069797+0000", 2024-09-16T03:39:54.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4500000000000001e-06 2024-09-16T03:39:54.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.816 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069801+0000", 2024-09-16T03:39:54.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8850000000000001e-06 2024-09-16T03:39:54.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.169935+0000", 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013362100000001 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241791+0000", 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.071856512999999997 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241834+0000", 2024-09-16T03:39:54.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2824999999999997e-05 2024-09-16T03:39:54.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.818 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.818 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790284+0000", 2024-09-16T03:39:54.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54844967 2024-09-16T03:39:54.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.818 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.818 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790369+0000", 2024-09-16T03:39:54.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5453999999999999e-05 2024-09-16T03:39:54.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.818 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790464+0000", 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4708000000000002e-05 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4644 3.0 3:1f968072:::a9957f29a7f946349b562c449f43fa1bc2e3c48b00925943f7b23963f3ffcd03:head [write 0~1316352] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e149)", 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:29.647871+0000", 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.095705226, 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.707065583, 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.819 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.820 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.820 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.820 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.820 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4644 2024-09-16T03:39:54.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.820 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.647871+0000", 2024-09-16T03:39:54.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.647871+0000", 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.647873+0000", 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6050000000000001e-06 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648606+0000", 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073310499999999995 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.822 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.822 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648619+0000", 2024-09-16T03:39:54.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3023000000000001e-05 2024-09-16T03:39:54.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.822 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.822 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.822 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648631+0000", 2024-09-16T03:39:54.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.173e-05 2024-09-16T03:39:54.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.822 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.822 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.748799+0000", 2024-09-16T03:39:54.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100167847 2024-09-16T03:39:54.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:54.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.748813+0000", 2024-09-16T03:39:54.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3913999999999999e-05 2024-09-16T03:39:54.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.950128+0000", 2024-09-16T03:39:54.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20131515899999999 2024-09-16T03:39:54.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.950233+0000", 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010524 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1,4,6,9", 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.950411+0000", 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000177937 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153899+0000", 2024-09-16T03:39:54.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20348797699999999 2024-09-16T03:39:54.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.825 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.825 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.825 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153953+0000", 2024-09-16T03:39:54.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3919999999999999e-05 2024-09-16T03:39:54.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.825 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.825 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:54.825 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153964+0000", 2024-09-16T03:39:54.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1089e-05 2024-09-16T03:39:54.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.825 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.825 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254327+0000", 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100362865 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.354738+0000", 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100410991 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.354824+0000", 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6557999999999994e-05 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.354937+0000", 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112623 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:179 3.0 3:004ec411:::10547772e2dbc1bd82bb7d7e420056cffb9d849bbc3fd015ab311821f554ea0c:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e150)", 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:29.648695+0000", 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.094881807, 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70650748100000005, 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.827 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.828 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.828 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.828 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648695+0000", 2024-09-16T03:39:54.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.828 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.828 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648695+0000", 2024-09-16T03:39:54.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.828 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.828 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648696+0000", 2024-09-16T03:39:54.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2900000000000003e-07 2024-09-16T03:39:54.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648700+0000", 2024-09-16T03:39:54.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.873e-06 2024-09-16T03:39:54.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648708+0000", 2024-09-16T03:39:54.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1969999999999996e-06 2024-09-16T03:39:54.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.830 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.830 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648715+0000", 2024-09-16T03:39:54.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.17e-06 2024-09-16T03:39:54.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.830 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.830 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.748978+0000", 2024-09-16T03:39:54.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100262658 2024-09-16T03:39:54.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.830 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:54.831 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.748981+0000", 2024-09-16T03:39:54.831 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3859999999999999e-06 2024-09-16T03:39:54.831 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.831 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.831 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.831 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.953320+0000", 2024-09-16T03:39:54.831 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20433948599999999 2024-09-16T03:39:54.831 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.831 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.831 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.831 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.953414+0000", 2024-09-16T03:39:54.831 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3813000000000005e-05 2024-09-16T03:39:54.831 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.832 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.832 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1,4,6,9", 2024-09-16T03:39:54.832 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.953542+0000", 2024-09-16T03:39:54.832 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012773799999999999 2024-09-16T03:39:54.832 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.832 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.832 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:54.832 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153969+0000", 2024-09-16T03:39:54.832 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200426624 2024-09-16T03:39:54.832 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.832 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.832 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.832 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254193+0000", 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100223953 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254393+0000", 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000200745 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254424+0000", 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0825999999999999e-05 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355004+0000", 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100579794 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355070+0000", 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5999000000000003e-05 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355202+0000", 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013249 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.834 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:180 3.0 3:0569db51:::61d0b13b504667b91c94c752c1e35844395f8745f67042f596605bdf34d7970d:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e150)", 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:29.648725+0000", 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 25.094851795, 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70679689899999998, 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648725+0000", 2024-09-16T03:39:54.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.836 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.836 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.836 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648725+0000", 2024-09-16T03:39:54.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.836 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.836 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.836 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648726+0000", 2024-09-16T03:39:54.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0800000000000004e-07 2024-09-16T03:39:54.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.836 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648729+0000", 2024-09-16T03:39:54.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4589999999999998e-06 2024-09-16T03:39:54.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648737+0000", 2024-09-16T03:39:54.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7880000000000007e-06 2024-09-16T03:39:54.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648740+0000", 2024-09-16T03:39:54.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6729999999999998e-06 2024-09-16T03:39:54.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.849392+0000", 2024-09-16T03:39:54.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20065119000000001 2024-09-16T03:39:54.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:54.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.849397+0000", 2024-09-16T03:39:54.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3809999999999998e-06 2024-09-16T03:39:54.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153085+0000", 2024-09-16T03:39:54.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30368795100000001 2024-09-16T03:39:54.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153326+0000", 2024-09-16T03:39:54.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000241396 2024-09-16T03:39:54.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.839 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.839 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1,4,6,9", 2024-09-16T03:39:54.839 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153489+0000", 2024-09-16T03:39:54.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000162492 2024-09-16T03:39:54.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:54.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254432+0000", 2024-09-16T03:39:54.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100943038 2024-09-16T03:39:54.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254454+0000", 2024-09-16T03:39:54.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2212999999999999e-05 2024-09-16T03:39:54.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254476+0000", 2024-09-16T03:39:54.841 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1781e-05 2024-09-16T03:39:54.841 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.841 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.841 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355284+0000", 2024-09-16T03:39:54.841 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100808457 2024-09-16T03:39:54.841 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.841 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.841 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355335+0000", 2024-09-16T03:39:54.841 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0864e-05 2024-09-16T03:39:54.841 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:54.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355396+0000", 2024-09-16T03:39:54.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1135000000000003e-05 2024-09-16T03:39:54.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355522+0000", 2024-09-16T03:39:54.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000125473 2024-09-16T03:39:54.842 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.842 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.842 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.843 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5045 4.35 4:ad0b9852:::smithi15736696-16:head [write 2700190~586604] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:39:54.843 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.871612+0000", 2024-09-16T03:39:54.843 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.87196490500000001, 2024-09-16T03:39:54.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.59445419600000005, 2024-09-16T03:39:54.843 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.843 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.843 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.843 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.843 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.843 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.843 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5045 2024-09-16T03:39:54.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.844 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871612+0000", 2024-09-16T03:39:54.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871612+0000", 2024-09-16T03:39:54.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871613+0000", 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6190000000000001e-06 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871900+0000", 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000286874 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871911+0000", 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1151999999999999e-05 2024-09-16T03:39:54.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.846 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.846 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.846 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871917+0000", 2024-09-16T03:39:54.846 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9880000000000001e-06 2024-09-16T03:39:54.846 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.846 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.846 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.846 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.163760+0000", 2024-09-16T03:39:54.846 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29184254199999998 2024-09-16T03:39:54.846 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.846 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.846 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.846 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.163846+0000", 2024-09-16T03:39:54.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5944000000000002e-05 2024-09-16T03:39:54.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4,6,9,10", 2024-09-16T03:39:54.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.163968+0000", 2024-09-16T03:39:54.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000121995 2024-09-16T03:39:54.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.365568+0000", 2024-09-16T03:39:54.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20159980799999999 2024-09-16T03:39:54.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465712+0000", 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100144396 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465745+0000", 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3151000000000002e-05 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465803+0000", 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8006e-05 2024-09-16T03:39:54.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465827+0000", 2024-09-16T03:39:54.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4187999999999999e-05 2024-09-16T03:39:54.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465914+0000", 2024-09-16T03:39:54.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6172000000000001e-05 2024-09-16T03:39:54.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:54.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465957+0000", 2024-09-16T03:39:54.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3269999999999997e-05 2024-09-16T03:39:54.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466066+0000", 2024-09-16T03:39:54.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109091 2024-09-16T03:39:54.850 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.850 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.850 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.850 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5046 4.35 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=7b,truncate 3680799] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:39:54.850 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.871927+0000", 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.87164948200000003, 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.59428540500000004, 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5046 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.851 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871927+0000", 2024-09-16T03:39:54.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871927+0000", 2024-09-16T03:39:54.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.852 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.852 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871928+0000", 2024-09-16T03:39:54.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7700000000000004e-07 2024-09-16T03:39:54.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871931+0000", 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0860000000000002e-06 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871936+0000", 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.865e-06 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871939+0000", 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5250000000000001e-06 2024-09-16T03:39:54.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.164841+0000", 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29290207299999999 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.164901+0000", 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9176999999999999e-05 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4,6,9,10", 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.165000+0000", 2024-09-16T03:39:54.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9374000000000001e-05 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465794+0000", 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30079360999999999 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465807+0000", 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.377e-05 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465850+0000", 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2719000000000002e-05 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465872+0000", 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1784999999999999e-05 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465893+0000", 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0811000000000001e-05 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.856 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:54.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466102+0000", 2024-09-16T03:39:54.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020906399999999999 2024-09-16T03:39:54.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:54.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466132+0000", 2024-09-16T03:39:54.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0663000000000001e-05 2024-09-16T03:39:54.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466213+0000", 2024-09-16T03:39:54.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0205999999999997e-05 2024-09-16T03:39:54.857 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5047 4.35 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.871944+0000", 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.87163289200000005, 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.59445495699999995, 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.858 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5047 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871944+0000", 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871944+0000", 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871944+0000", 2024-09-16T03:39:54.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1199999999999999e-07 2024-09-16T03:39:54.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871946+0000", 2024-09-16T03:39:54.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6479999999999999e-06 2024-09-16T03:39:54.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871950+0000", 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.177e-06 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871953+0000", 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1389999999999999e-06 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466235+0000", 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.59428207899999996 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466296+0000", 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0498999999999998e-05 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466399+0000", 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010310300000000001 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5068 4.35 4:aff4cc41:::smithi15736696-11:head [read 548290~2645106,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:39:54.862 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.911828+0000", 2024-09-16T03:39:54.863 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.83174877400000002, 2024-09-16T03:39:54.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.65552754499999999, 2024-09-16T03:39:54.863 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.863 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.863 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:54.863 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.863 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.863 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.863 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5068 2024-09-16T03:39:54.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.863 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911828+0000", 2024-09-16T03:39:54.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.864 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911828+0000", 2024-09-16T03:39:54.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.864 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911829+0000", 2024-09-16T03:39:54.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1799999999999995e-07 2024-09-16T03:39:54.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911831+0000", 2024-09-16T03:39:54.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8499999999999998e-06 2024-09-16T03:39:54.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911837+0000", 2024-09-16T03:39:54.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3929999999999999e-06 2024-09-16T03:39:54.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911841+0000", 2024-09-16T03:39:54.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:39:54.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.565938+0000", 2024-09-16T03:39:54.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.65409697600000005 2024-09-16T03:39:54.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.566115+0000", 2024-09-16T03:39:54.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017720999999999999 2024-09-16T03:39:54.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.567355+0000", 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00124065 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5069 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~1413152] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.911846+0000", 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.83173119799999995, 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.65602458500000005, 2024-09-16T03:39:54.867 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:54.868 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:54.868 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:54.868 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:54.868 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:54.868 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:54.868 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5069 2024-09-16T03:39:54.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.868 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:54.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:54.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911846+0000", 2024-09-16T03:39:54.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.869 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:54.869 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911846+0000", 2024-09-16T03:39:54.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:54.869 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.869 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:54.869 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911846+0000", 2024-09-16T03:39:54.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1300000000000001e-07 2024-09-16T03:39:54.869 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.869 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:54.869 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911847+0000", 2024-09-16T03:39:54.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.559e-06 2024-09-16T03:39:54.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:54.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911850+0000", 2024-09-16T03:39:54.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.903e-06 2024-09-16T03:39:54.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:54.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.011923+0000", 2024-09-16T03:39:54.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100072736 2024-09-16T03:39:54.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.567391+0000", 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.55546773699999996 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.567508+0000", 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117143 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.567870+0000", 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00036219399999999998 2024-09-16T03:39:54.871 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.872 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.872 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.872 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:54.872 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:54.872 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:54.873 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:54.820+0000 7fd101e6f640 -1 osd.2 169 handle_osd_map injecting map crc failure 2024-09-16T03:39:54.873 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:39:54.875 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:39:54.875 INFO:teuthology.orchestra.run.smithi064.stdout:moved 6 / 207 (2.89855%) 2024-09-16T03:39:54.875 INFO:teuthology.orchestra.run.smithi064.stdout:avg 14.7857 2024-09-16T03:39:54.875 INFO:teuthology.orchestra.run.smithi064.stdout:stddev 3.96734 -> 4.03872 (expected baseline 3.70535) 2024-09-16T03:39:54.875 INFO:teuthology.orchestra.run.smithi064.stdout:min osd.7 with 9 -> 10 pgs (0.608696 -> 0.676328 * mean) 2024-09-16T03:39:54.875 INFO:teuthology.orchestra.run.smithi064.stdout:max osd.10 with 21 -> 21 pgs (1.42029 -> 1.42029 * mean) 2024-09-16T03:39:54.875 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:39:54.875 INFO:teuthology.orchestra.run.smithi064.stdout:oload 110 2024-09-16T03:39:54.876 INFO:teuthology.orchestra.run.smithi064.stdout:max_change 0.05 2024-09-16T03:39:54.876 INFO:teuthology.orchestra.run.smithi064.stdout:max_change_osds 4 2024-09-16T03:39:54.876 INFO:teuthology.orchestra.run.smithi064.stdout:average_utilization 0.0007 2024-09-16T03:39:54.876 INFO:teuthology.orchestra.run.smithi064.stdout:overload_utilization 0.0008 2024-09-16T03:39:54.876 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 weight 0.5237 -> 0.4738 2024-09-16T03:39:54.876 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15 weight 0.8574 -> 0.8074 2024-09-16T03:39:54.876 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 weight 0.7443 -> 0.6944 2024-09-16T03:39:54.876 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8 weight 0.8523 -> 0.8024 2024-09-16T03:39:54.876 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:54.801+0000 7f66f3205640 -1 osd.15 169 handle_osd_map injecting map crc failure 2024-09-16T03:39:54.876 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:54.803+0000 7f66f3205640 -1 osd.15 169 handle_osd_map injecting map crc failure 2024-09-16T03:39:54.878 INFO:tasks.rados.rados.0.smithi157.stdout:2037: finishing write tid 1 to smithi15736696-16 2024-09-16T03:39:54.878 INFO:tasks.rados.rados.0.smithi157.stdout:2037: finishing write tid 2 to smithi15736696-16 2024-09-16T03:39:54.878 INFO:tasks.rados.rados.0.smithi157.stdout:2034: finishing write tid 3 to smithi15736696-25 2024-09-16T03:39:54.878 INFO:tasks.rados.rados.0.smithi157.stdout:2034: finishing write tid 4 to smithi15736696-25 2024-09-16T03:39:54.878 INFO:tasks.rados.rados.0.smithi157.stdout:2034: finishing write tid 5 to smithi15736696-25 2024-09-16T03:39:54.878 INFO:tasks.rados.rados.0.smithi157.stdout:2034: oid 25 updating version 0 to 159 2024-09-16T03:39:54.878 INFO:tasks.rados.rados.0.smithi157.stdout:2034: oid 25 updating version 159 to 161 2024-09-16T03:39:54.879 INFO:tasks.rados.rados.0.smithi157.stdout:2034: oid 25 version 161 is already newer than 158 2024-09-16T03:39:54.879 INFO:tasks.rados.rados.0.smithi157.stdout:2034: oid 25 version 161 is already newer than 160 2024-09-16T03:39:54.879 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 25 v 161 (ObjNum 456 snap 0 seq_num 456) dirty exists 2024-09-16T03:39:54.879 INFO:tasks.rados.rados.0.smithi157.stdout:2034: left oid 25 (ObjNum 456 snap 0 seq_num 456) 2024-09-16T03:39:54.879 INFO:tasks.rados.rados.0.smithi157.stdout:2037: finishing write tid 3 to smithi15736696-16 2024-09-16T03:39:54.879 INFO:tasks.rados.rados.0.smithi157.stdout:2037: finishing write tid 4 to smithi15736696-16 2024-09-16T03:39:54.879 INFO:tasks.rados.rados.0.smithi157.stdout:2037: finishing write tid 5 to smithi15736696-16 2024-09-16T03:39:54.879 INFO:tasks.rados.rados.0.smithi157.stdout:2037: oid 16 updating version 0 to 85 2024-09-16T03:39:54.879 INFO:tasks.rados.rados.0.smithi157.stdout:2037: oid 16 version 85 is already newer than 84 2024-09-16T03:39:54.879 INFO:tasks.rados.rados.0.smithi157.stdout:2037: oid 16 version 85 is already newer than 83 2024-09-16T03:39:54.879 INFO:tasks.rados.rados.0.smithi157.stdout:2037: oid 16 version 85 is already newer than 82 2024-09-16T03:39:54.879 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 16 v 85 (ObjNum 457 snap 0 seq_num 457) dirty exists 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout:2037: left oid 16 (ObjNum 457 snap 0 seq_num 457) 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout:2034: done (13 left) 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout:2037: done (12 left) 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout:2044: done (11 left) 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 11 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout:2038: finishing write tid 1 to smithi15736696-24 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout:2038: finishing write tid 2 to smithi15736696-24 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout:2038: finishing write tid 3 to smithi15736696-24 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout:2038: finishing write tid 4 to smithi15736696-24 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout:2038: finishing write tid 5 to smithi15736696-24 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout:2038: oid 24 updating version 0 to 135 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout:2038: oid 24 updating version 135 to 138 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout:2038: oid 24 version 138 is already newer than 136 2024-09-16T03:39:54.880 INFO:tasks.rados.rados.0.smithi157.stdout:2038: oid 24 version 138 is already newer than 137 2024-09-16T03:39:54.881 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 24 v 138 (ObjNum 458 snap 0 seq_num 458) dirty exists 2024-09-16T03:39:54.881 INFO:tasks.rados.rados.0.smithi157.stdout:2038: left oid 24 (ObjNum 458 snap 0 seq_num 458) 2024-09-16T03:39:54.881 INFO:tasks.rados.rados.0.smithi157.stdout:2045: expect (ObjNum 428 snap 0 seq_num 428) 2024-09-16T03:39:54.881 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1577264 2024-09-16T03:39:54.881 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 548290 2024-09-16T03:39:54.881 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:54.870+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:54.932 INFO:tasks.rados.rados.0.smithi157.stdout:2009: finishing write tid 4 to smithi15736696-26 2024-09-16T03:39:54.932 INFO:tasks.rados.rados.0.smithi157.stdout:2009: finishing write tid 5 to smithi15736696-26 2024-09-16T03:39:54.932 INFO:tasks.rados.rados.0.smithi157.stdout:2009: oid 26 updating version 0 to 131 2024-09-16T03:39:54.932 INFO:tasks.rados.rados.0.smithi157.stdout:2009: oid 26 version 131 is already newer than 129 2024-09-16T03:39:54.932 INFO:tasks.rados.rados.0.smithi157.stdout:2009: oid 26 updating version 131 to 132 2024-09-16T03:39:54.932 INFO:tasks.rados.rados.0.smithi157.stdout:2009: oid 26 version 132 is already newer than 130 2024-09-16T03:39:54.932 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 26 v 132 (ObjNum 448 snap 0 seq_num 448) dirty exists 2024-09-16T03:39:54.933 INFO:tasks.rados.rados.0.smithi157.stdout:2009: left oid 26 (ObjNum 448 snap 0 seq_num 448) 2024-09-16T03:39:54.933 INFO:tasks.rados.rados.0.smithi157.stdout:2046: expect (ObjNum 452 snap 0 seq_num 452) 2024-09-16T03:39:54.974 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:54.971+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:55.007 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:55.022 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:55.022 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:55.022 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:39:55.022 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:55.031 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:39:55.113 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:55.111+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:55.158 INFO:tasks.rados.rados.0.smithi157.stdout:2047: expect (ObjNum 451 snap 0 seq_num 451) 2024-09-16T03:39:55.175 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:55.175 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:55.175 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:55.176 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:55.176 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:55.176 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:55.174+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:55.184 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:39:55.276 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:55.274+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:55.294 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 0 2024-09-16T03:39:55.332 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:55.333 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:55.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:55.333 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:55.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.333 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:39:55.333 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:39:55.333 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 163.144373194, 2024-09-16T03:39:55.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102630163, 2024-09-16T03:39:55.333 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.333 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.333 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.333 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:39:55.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:39:55.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186081+0000", 2024-09-16T03:39:55.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5600000000000005e-07 2024-09-16T03:39:55.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186970+0000", 2024-09-16T03:39:55.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00088889000000000004 2024-09-16T03:39:55.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186979+0000", 2024-09-16T03:39:55.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0620000000000007e-06 2024-09-16T03:39:55.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186986+0000", 2024-09-16T03:39:55.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:39:55.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287184+0000", 2024-09-16T03:39:55.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100198036 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287199+0000", 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5458999999999999e-05 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288659+0000", 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014601900000000001 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288711+0000", 2024-09-16T03:39:55.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1088000000000003e-05 2024-09-16T03:39:55.338 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.338 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.338 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.338 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:55.338 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:39:55.338 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 163.143336761, 2024-09-16T03:39:55.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10205370900000001, 2024-09-16T03:39:55.338 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.338 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.338 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187118+0000", 2024-09-16T03:39:55.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8599999999999998e-07 2024-09-16T03:39:55.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188062+0000", 2024-09-16T03:39:55.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094447300000000004 2024-09-16T03:39:55.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188070+0000", 2024-09-16T03:39:55.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.695e-06 2024-09-16T03:39:55.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188074+0000", 2024-09-16T03:39:55.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0709999999999996e-06 2024-09-16T03:39:55.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287842+0000", 2024-09-16T03:39:55.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099768478999999993 2024-09-16T03:39:55.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287864+0000", 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1679999999999999e-05 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289137+0000", 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012732430000000001 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289171+0000", 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3382e-05 2024-09-16T03:39:55.342 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.343 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.343 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.343 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:42 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:55.343 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:39:55.343 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 163.07325485300001, 2024-09-16T03:39:55.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10172858, 2024-09-16T03:39:55.343 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.343 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.344 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.344 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:39:55.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257201+0000", 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257536+0000", 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033513899999999998 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257555+0000", 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8547999999999999e-05 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257566+0000", 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1379999999999999e-05 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357720+0000", 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100153532 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357734+0000", 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4491000000000001e-05 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358896+0000", 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011620599999999999 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358927+0000", 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1158999999999998e-05 2024-09-16T03:39:55.347 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.348 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.348 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.348 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:43 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'3, mlcod=0'0)", 2024-09-16T03:39:55.348 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:39:55.348 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 163.071470941, 2024-09-16T03:39:55.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101764453, 2024-09-16T03:39:55.348 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.348 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.348 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.348 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258984+0000", 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5689999999999999e-06 2024-09-16T03:39:55.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258994+0000", 2024-09-16T03:39:55.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-16T03:39:55.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259008+0000", 2024-09-16T03:39:55.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4151999999999999e-05 2024-09-16T03:39:55.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359073+0000", 2024-09-16T03:39:55.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100065628 2024-09-16T03:39:55.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359110+0000", 2024-09-16T03:39:55.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7045999999999997e-05 2024-09-16T03:39:55.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359124+0000", 2024-09-16T03:39:55.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.382e-05 2024-09-16T03:39:55.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360716+0000", 2024-09-16T03:39:55.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015919829999999999 2024-09-16T03:39:55.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360747+0000", 2024-09-16T03:39:55.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0874999999999998e-05 2024-09-16T03:39:55.352 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.352 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.352 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.353 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1570 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'31, mlcod=48'29)", 2024-09-16T03:39:55.353 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:55.353 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 129.15614404300001, 2024-09-16T03:39:55.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10253683500000001, 2024-09-16T03:39:55.353 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.353 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.353 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.353 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:55.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8500000000000002e-07 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174689+0000", 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037958600000000001 2024-09-16T03:39:55.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174699+0000", 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0015e-05 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174706+0000", 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6250000000000001e-06 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274865+0000", 2024-09-16T03:39:55.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100159157 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274879+0000", 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3983e-05 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276820+0000", 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001941248 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276846+0000", 2024-09-16T03:39:55.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5936e-05 2024-09-16T03:39:55.357 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.357 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.357 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.357 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.357 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1571 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'32, mlcod=48'29)", 2024-09-16T03:39:55.357 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:39:55.357 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 129.155693289, 2024-09-16T03:39:55.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10249543, 2024-09-16T03:39:55.357 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.357 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.357 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.358 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174761+0000", 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0800000000000004e-07 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175053+0000", 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029151600000000002 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175063+0000", 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0538e-05 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175070+0000", 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7819999999999996e-06 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275895+0000", 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10082516 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275909+0000", 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4197e-05 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277232+0000", 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013231009999999999 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277256+0000", 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3428e-05 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.361 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1572 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'33, mlcod=48'29)", 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 129.155351989, 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10216886, 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:55.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:39:55.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8400000000000004e-07 2024-09-16T03:39:55.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175107+0000", 2024-09-16T03:39:55.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.578e-06 2024-09-16T03:39:55.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175114+0000", 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.888e-06 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175119+0000", 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.074e-06 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276662+0000", 2024-09-16T03:39:55.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101542753 2024-09-16T03:39:55.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276674+0000", 2024-09-16T03:39:55.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1960000000000001e-05 2024-09-16T03:39:55.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277258+0000", 2024-09-16T03:39:55.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058431900000000003 2024-09-16T03:39:55.365 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277270+0000", 2024-09-16T03:39:55.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2604e-05 2024-09-16T03:39:55.366 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.366 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.366 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.366 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2467 4.5 e85/50)", 2024-09-16T03:39:55.366 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:39:55.366 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 99.179322501000001, 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102439455, 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:39:55.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151134+0000", 2024-09-16T03:39:55.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2030000000000001e-06 2024-09-16T03:39:55.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151480+0000", 2024-09-16T03:39:55.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034613999999999999 2024-09-16T03:39:55.368 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151500+0000", 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9582999999999999e-05 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151513+0000", 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251653+0000", 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10014007900000001 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251679+0000", 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5721e-05 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253532+0000", 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001852701 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253571+0000", 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8677999999999998e-05 2024-09-16T03:39:55.370 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.371 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.371 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.371 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.371 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2468 4.5 e85/50)", 2024-09-16T03:39:55.371 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:39:55.371 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 99.177853971999994, 2024-09-16T03:39:55.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101505239, 2024-09-16T03:39:55.371 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.371 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.371 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.371 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:39:55.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:39:55.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152601+0000", 2024-09-16T03:39:55.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6869999999999999e-06 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152719+0000", 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117508 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152732+0000", 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152740+0000", 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7090000000000002e-06 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252704+0000", 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099964259999999999 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252727+0000", 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3601e-05 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254075+0000", 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00134772 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254105+0000", 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9802e-05 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2623 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'62, mlcod=85'60)", 2024-09-16T03:39:55.375 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 94.981216341999996, 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.104590794, 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1899999999999999e-07 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349496+0000", 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000258934 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349506+0000", 2024-09-16T03:39:55.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9389999999999992e-06 2024-09-16T03:39:55.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349513+0000", 2024-09-16T03:39:55.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6209999999999998e-06 2024-09-16T03:39:55.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449668+0000", 2024-09-16T03:39:55.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015550099999999 2024-09-16T03:39:55.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449695+0000", 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6253000000000001e-05 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453784+0000", 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040896609999999996 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453828+0000", 2024-09-16T03:39:55.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3665999999999998e-05 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2624 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'63, mlcod=85'60)", 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 94.979869104000002, 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103286796, 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350586+0000", 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0020000000000001e-06 2024-09-16T03:39:55.381 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350794+0000", 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020824099999999999 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350805+0000", 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1443999999999999e-05 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450876+0000", 2024-09-16T03:39:55.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100070612 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450960+0000", 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4444999999999995e-05 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450980+0000", 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9728000000000002e-05 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453831+0000", 2024-09-16T03:39:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002850923 2024-09-16T03:39:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453871+0000", 2024-09-16T03:39:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0401000000000002e-05 2024-09-16T03:39:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2661 4.15 e91/50 4:aff4cc41:::smithi15736696-11:head v 91'52, mlcod=88'48)", 2024-09-16T03:39:55.384 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 93.921268874000006, 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10164756699999999, 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409186+0000", 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-06 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409568+0000", 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038196600000000002 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409578+0000", 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.781e-06 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409583+0000", 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3070000000000002e-06 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409633+0000", 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0358000000000001e-05 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409649+0000", 2024-09-16T03:39:55.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5875999999999998e-05 2024-09-16T03:39:55.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510800+0000", 2024-09-16T03:39:55.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101150728 2024-09-16T03:39:55.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510832+0000", 2024-09-16T03:39:55.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2500999999999999e-05 2024-09-16T03:39:55.388 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:3970 4.5 e130/110 4:a11e1d7c:::smithi15736696-43:head v 130'83, mlcod=113'81)", 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 50.067797292999998, 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102913225, 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262657+0000", 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.72e-07 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262783+0000", 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012681199999999999 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262792+0000", 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3839999999999997e-06 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262799+0000", 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4100000000000002e-06 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.362958+0000", 2024-09-16T03:39:55.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100158395 2024-09-16T03:39:55.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.362977+0000", 2024-09-16T03:39:55.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9741e-05 2024-09-16T03:39:55.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.392 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.365511+0000", 2024-09-16T03:39:55.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0025332499999999999 2024-09-16T03:39:55.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.365570+0000", 2024-09-16T03:39:55.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8961e-05 2024-09-16T03:39:55.393 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.393 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.393 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.393 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:4805 4.18 e155/112 4:1f63731d:::smithi15736696-30:head v 155'52, mlcod=128'50)", 2024-09-16T03:39:55.393 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:35.524640+0000", 2024-09-16T03:39:55.393 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 19.805813447999999, 2024-09-16T03:39:55.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.104400324, 2024-09-16T03:39:55.394 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.394 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.394 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.394 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.524640+0000", 2024-09-16T03:39:55.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.524640+0000", 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.524640+0000", 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8699999999999999e-07 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.526714+0000", 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002073835 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.526725+0000", 2024-09-16T03:39:55.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1148000000000001e-05 2024-09-16T03:39:55.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.526734+0000", 2024-09-16T03:39:55.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6899999999999998e-06 2024-09-16T03:39:55.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.626909+0000", 2024-09-16T03:39:55.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100174737 2024-09-16T03:39:55.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.626945+0000", 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6316000000000003e-05 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.629006+0000", 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020604239999999999 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.629040+0000", 2024-09-16T03:39:55.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4987000000000002e-05 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5053 4.5 4.3eb87885 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.897511+0000", 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.432942215, 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90556326399999998, 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.398 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:55.399 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:55.399 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5053 2024-09-16T03:39:55.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.399 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.897511+0000", 2024-09-16T03:39:55.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.897511+0000", 2024-09-16T03:39:55.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.897514+0000", 2024-09-16T03:39:55.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4789999999999999e-06 2024-09-16T03:39:55.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.897935+0000", 2024-09-16T03:39:55.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042092099999999999 2024-09-16T03:39:55.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.897949+0000", 2024-09-16T03:39:55.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4450000000000001e-05 2024-09-16T03:39:55.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.897958+0000", 2024-09-16T03:39:55.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4759999999999998e-06 2024-09-16T03:39:55.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.901396+0000", 2024-09-16T03:39:55.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034385380000000001 2024-09-16T03:39:55.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:55.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.901405+0000", 2024-09-16T03:39:55.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6209999999999993e-06 2024-09-16T03:39:55.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803045+0000", 2024-09-16T03:39:55.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90163974000000002 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803075+0000", 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0039000000000002e-05 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5054 4.5 4.3eb87885 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.905677+0000", 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.424776832, 2024-09-16T03:39:55.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.89747413799999998, 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5054 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.905677+0000", 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.905677+0000", 2024-09-16T03:39:55.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.905679+0000", 2024-09-16T03:39:55.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2299999999999998e-06 2024-09-16T03:39:55.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.906107+0000", 2024-09-16T03:39:55.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000428162 2024-09-16T03:39:55.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.906120+0000", 2024-09-16T03:39:55.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3176999999999999e-05 2024-09-16T03:39:55.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.906129+0000", 2024-09-16T03:39:55.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9460000000000004e-06 2024-09-16T03:39:55.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.407 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.407 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.906170+0000", 2024-09-16T03:39:55.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0985999999999997e-05 2024-09-16T03:39:55.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.407 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:55.407 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.906180+0000", 2024-09-16T03:39:55.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5489999999999995e-06 2024-09-16T03:39:55.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.407 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.407 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803120+0000", 2024-09-16T03:39:55.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.89694054599999995 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803151+0000", 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0542e-05 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5055 4.5 4.3eb87885 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.911301+0000", 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.4191523079999999, 2024-09-16T03:39:55.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.99196924200000003, 2024-09-16T03:39:55.409 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.409 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.409 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:55.409 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.409 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:55.409 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:55.409 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5055 2024-09-16T03:39:55.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.409 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911301+0000", 2024-09-16T03:39:55.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911301+0000", 2024-09-16T03:39:55.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911303+0000", 2024-09-16T03:39:55.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2810000000000001e-06 2024-09-16T03:39:55.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911450+0000", 2024-09-16T03:39:55.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014764699999999999 2024-09-16T03:39:55.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911461+0000", 2024-09-16T03:39:55.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0713999999999999e-05 2024-09-16T03:39:55.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911467+0000", 2024-09-16T03:39:55.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5969999999999999e-06 2024-09-16T03:39:55.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911515+0000", 2024-09-16T03:39:55.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8841000000000002e-05 2024-09-16T03:39:55.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.412 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:55.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911524+0000", 2024-09-16T03:39:55.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3299999999999999e-06 2024-09-16T03:39:55.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.903256+0000", 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.99173228800000002 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.903271+0000", 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4544e-05 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.414 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5056 4.5 4.3eb87885 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:39:55.414 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.911478+0000", 2024-09-16T03:39:55.414 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.4189760739999999, 2024-09-16T03:39:55.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.99183876800000004, 2024-09-16T03:39:55.414 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.414 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.414 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:55.414 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.414 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:55.414 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:55.415 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5056 2024-09-16T03:39:55.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.415 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911478+0000", 2024-09-16T03:39:55.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911478+0000", 2024-09-16T03:39:55.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911478+0000", 2024-09-16T03:39:55.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6799999999999999e-07 2024-09-16T03:39:55.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911482+0000", 2024-09-16T03:39:55.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6210000000000001e-06 2024-09-16T03:39:55.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911488+0000", 2024-09-16T03:39:55.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9610000000000002e-06 2024-09-16T03:39:55.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911493+0000", 2024-09-16T03:39:55.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2859999999999999e-06 2024-09-16T03:39:55.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911559+0000", 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5555999999999994e-05 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911565+0000", 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2360000000000001e-06 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.903307+0000", 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.99174187800000002 2024-09-16T03:39:55.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.903316+0000", 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4620000000000002e-06 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5057 4.5 4.3eb87885 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.911500+0000", 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 1.4189533249999999, 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.092075873, 2024-09-16T03:39:55.419 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.420 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.420 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:55.420 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.420 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:55.420 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:55.420 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5057 2024-09-16T03:39:55.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.420 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911500+0000", 2024-09-16T03:39:55.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911500+0000", 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911501+0000", 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7400000000000003e-07 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911503+0000", 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4140000000000001e-06 2024-09-16T03:39:55.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911509+0000", 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3179999999999997e-06 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911516+0000", 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2490000000000004e-06 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911583+0000", 2024-09-16T03:39:55.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6742999999999994e-05 2024-09-16T03:39:55.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:55.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911587+0000", 2024-09-16T03:39:55.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1670000000000001e-06 2024-09-16T03:39:55.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.423 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:55.003562+0000", 2024-09-16T03:39:55.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.091974939 2024-09-16T03:39:55.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.423 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:55.003576+0000", 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4469e-05 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_backoff(4.5 ack-block id 1 [4:a0000000::::0,4:a4000000::::head) e170)", 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:54.803413+0000", 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 0.52704012099999997, 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20039902400000001, 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.424 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 0 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803413+0000", 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803413+0000", 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803415+0000", 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8199999999999999e-06 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803419+0000", 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1810000000000003e-06 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803429+0000", 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7620000000000007e-06 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803438+0000", 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9060000000000008e-06 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:55.003803+0000", 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20036456899999999 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:55.003813+0000", 2024-09-16T03:39:55.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7860000000000008e-06 2024-09-16T03:39:55.428 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.428 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.428 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.428 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.428 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.428 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:55.430 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:39:55.431 INFO:tasks.rados.rados.0.smithi157.stdout:2049: finishing write tid 1 to smithi15736696-40 2024-09-16T03:39:55.431 INFO:tasks.rados.rados.0.smithi157.stdout:2048: finishing write tid 1 to smithi15736696-41 2024-09-16T03:39:55.431 INFO:tasks.rados.rados.0.smithi157.stdout:2048: finishing write tid 2 to smithi15736696-41 2024-09-16T03:39:55.431 INFO:tasks.rados.rados.0.smithi157.stdout:2048: finishing write tid 3 to smithi15736696-41 2024-09-16T03:39:55.431 INFO:tasks.rados.rados.0.smithi157.stdout:2048: finishing write tid 4 to smithi15736696-41 2024-09-16T03:39:55.431 INFO:tasks.rados.rados.0.smithi157.stdout:2048: finishing write tid 5 to smithi15736696-41 2024-09-16T03:39:55.431 INFO:tasks.rados.rados.0.smithi157.stdout:2048: oid 41 updating version 0 to 57 2024-09-16T03:39:55.431 INFO:tasks.rados.rados.0.smithi157.stdout:2048: oid 41 updating version 57 to 60 2024-09-16T03:39:55.431 INFO:tasks.rados.rados.0.smithi157.stdout:2048: oid 41 version 60 is already newer than 59 2024-09-16T03:39:55.431 INFO:tasks.rados.rados.0.smithi157.stdout:2048: oid 41 version 60 is already newer than 58 2024-09-16T03:39:55.431 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 41 v 60 (ObjNum 460 snap 0 seq_num 460) dirty exists 2024-09-16T03:39:55.431 INFO:tasks.rados.rados.0.smithi157.stdout:2048: left oid 41 (ObjNum 460 snap 0 seq_num 460) 2024-09-16T03:39:55.431 INFO:tasks.rados.rados.0.smithi157.stdout:2049: finishing write tid 2 to smithi15736696-40 2024-09-16T03:39:55.432 INFO:tasks.rados.rados.0.smithi157.stdout:2009: done (10 left) 2024-09-16T03:39:55.432 INFO:tasks.rados.rados.0.smithi157.stdout:2038: done (9 left) 2024-09-16T03:39:55.432 INFO:tasks.rados.rados.0.smithi157.stdout:2045: done (8 left) 2024-09-16T03:39:55.432 INFO:tasks.rados.rados.0.smithi157.stdout:2046: done (7 left) 2024-09-16T03:39:55.432 INFO:tasks.rados.rados.0.smithi157.stdout:2047: done (6 left) 2024-09-16T03:39:55.432 INFO:tasks.rados.rados.0.smithi157.stdout:2048: done (5 left) 2024-09-16T03:39:55.432 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 5 2024-09-16T03:39:55.432 INFO:tasks.rados.rados.0.smithi157.stdout:2049: finishing write tid 3 to smithi15736696-40 2024-09-16T03:39:55.432 INFO:tasks.rados.rados.0.smithi157.stdout:2049: finishing write tid 4 to smithi15736696-40 2024-09-16T03:39:55.432 INFO:tasks.rados.rados.0.smithi157.stdout:2049: finishing write tid 5 to smithi15736696-40 2024-09-16T03:39:55.432 INFO:tasks.rados.rados.0.smithi157.stdout:2049: oid 40 updating version 0 to 164 2024-09-16T03:39:55.432 INFO:tasks.rados.rados.0.smithi157.stdout:2049: oid 40 version 164 is already newer than 162 2024-09-16T03:39:55.432 INFO:tasks.rados.rados.0.smithi157.stdout:2049: oid 40 version 164 is already newer than 163 2024-09-16T03:39:55.433 INFO:tasks.rados.rados.0.smithi157.stdout:2049: oid 40 updating version 164 to 165 2024-09-16T03:39:55.433 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 40 v 165 (ObjNum 461 snap 0 seq_num 461) dirty exists 2024-09-16T03:39:55.433 INFO:tasks.rados.rados.0.smithi157.stdout:2049: left oid 40 (ObjNum 461 snap 0 seq_num 461) 2024-09-16T03:39:55.433 INFO:tasks.rados.rados.0.smithi157.stdout:2049: done (4 left) 2024-09-16T03:39:55.433 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 4 2024-09-16T03:39:55.433 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:55.428+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:55.507 INFO:tasks.rados.rados.0.smithi157.stdout:2033: expect (ObjNum 398 snap 0 seq_num 398) 2024-09-16T03:39:55.519 INFO:tasks.rados.rados.0.smithi157.stdout:seeking to 1065058 2024-09-16T03:39:55.530 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:55.528+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:55.574 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:55.574 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:55.574 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:39:55.574 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:55.583 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:39:55.622 INFO:tasks.rados.rados.0.smithi157.stdout:2033: done (3 left) 2024-09-16T03:39:55.622 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 3 2024-09-16T03:39:55.631 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:55.629+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:55.705 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:39:55.724 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:55.724 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:39:55.724 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:39:55.724 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:39:55.724 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:55.732 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:55.730+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:55.733 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:39:55.802 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:55.799+0000 7eff7884b640 -1 osd.3 170 handle_osd_map injecting map crc failure 2024-09-16T03:39:55.815 INFO:tasks.rados.rados.0.smithi157.stdout:2024: got (0) Success 2024-09-16T03:39:55.815 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 6 v 0 (ObjNum 327 snap 0 seq_num 327) dirty exists 2024-09-16T03:39:55.816 INFO:tasks.rados.rados.0.smithi157.stdout:2024: done (2 left) 2024-09-16T03:39:55.816 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 2 2024-09-16T03:39:55.832 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:55.830+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:55.842 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:39:55.842 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:39:55.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:39:55.842 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:39:55.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.842 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T03:39:55.842 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:39:55.842 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 184.762400808, 2024-09-16T03:39:55.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.88203620400000005, 2024-09-16T03:39:55.842 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.842 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:39:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076767+0000", 2024-09-16T03:39:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9300000000000002e-06 2024-09-16T03:39:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076774+0000", 2024-09-16T03:39:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6630000000000004e-06 2024-09-16T03:39:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076785+0000", 2024-09-16T03:39:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1221e-05 2024-09-16T03:39:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076796+0000", 2024-09-16T03:39:55.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0428000000000001e-05 2024-09-16T03:39:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076856+0000", 2024-09-16T03:39:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:39:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:39:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076863+0000", 2024-09-16T03:39:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9840000000000004e-06 2024-09-16T03:39:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855770+0000", 2024-09-16T03:39:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.77890736999999999 2024-09-16T03:39:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855902+0000", 2024-09-16T03:39:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013207 2024-09-16T03:39:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.856054+0000", 2024-09-16T03:39:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015220899999999999 2024-09-16T03:39:55.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.858407+0000", 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023530069999999998 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958598+0000", 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100190194 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958716+0000", 2024-09-16T03:39:55.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118589 2024-09-16T03:39:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958761+0000", 2024-09-16T03:39:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5120000000000002e-05 2024-09-16T03:39:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958802+0000", 2024-09-16T03:39:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0383999999999998e-05 2024-09-16T03:39:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.953 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:39:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 181.24398463099999, 2024-09-16T03:39:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70508991300000001, 2024-09-16T03:39:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:55.954 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 52 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6300000000000005e-07 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595187+0000", 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9790000000000001e-06 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595194+0000", 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6490000000000002e-06 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595208+0000", 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4139e-05 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801931+0000", 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.206723083 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801990+0000", 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8625999999999997e-05 2024-09-16T03:39:55.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802066+0000", 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6553999999999999e-05 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802504+0000", 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043775200000000002 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.199998+0000", 2024-09-16T03:39:55.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39749365599999997 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300151+0000", 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015310099999999 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300210+0000", 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9376999999999997e-05 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300271+0000", 2024-09-16T03:39:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1334000000000006e-05 2024-09-16T03:39:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 179.46909364800001, 2024-09-16T03:39:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60623559900000001, 2024-09-16T03:39:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.960 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 71 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370077+0000", 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9940000000000002e-06 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370081+0000", 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5850000000000001e-06 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370087+0000", 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673522+0000", 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.303434969 2024-09-16T03:39:55.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673567+0000", 2024-09-16T03:39:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5080000000000002e-05 2024-09-16T03:39:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673651+0000", 2024-09-16T03:39:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3652999999999993e-05 2024-09-16T03:39:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:55.964 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775115+0000", 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101464178 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975918+0000", 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20080318699999999 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976254+0000", 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033619200000000002 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976285+0000", 2024-09-16T03:39:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0679000000000002e-05 2024-09-16T03:39:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976308+0000", 2024-09-16T03:39:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3051e-05 2024-09-16T03:39:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.966 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 179.46907028800001, 2024-09-16T03:39:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60632925100000001, 2024-09-16T03:39:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 72 2024-09-16T03:39:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:39:55.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370150+0000", 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3906999999999997e-05 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370157+0000", 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370162+0000", 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7249999999999997e-06 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673969+0000", 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30380713199999998 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674030+0000", 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1141000000000003e-05 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674124+0000", 2024-09-16T03:39:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3859999999999999e-05 2024-09-16T03:39:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775687+0000", 2024-09-16T03:39:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10156293299999999 2024-09-16T03:39:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976103+0000", 2024-09-16T03:39:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20041631400000001 2024-09-16T03:39:55.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976355+0000", 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000251611 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976385+0000", 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0883999999999998e-05 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976425+0000", 2024-09-16T03:39:55.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9666000000000003e-05 2024-09-16T03:39:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:39:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 179.468997804, 2024-09-16T03:39:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60763126999999995, 2024-09-16T03:39:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.973 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 73 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:39:55.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370169+0000", 2024-09-16T03:39:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6199999999999998e-07 2024-09-16T03:39:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370171+0000", 2024-09-16T03:39:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:39:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370175+0000", 2024-09-16T03:39:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.596e-06 2024-09-16T03:39:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470295+0000", 2024-09-16T03:39:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100119967 2024-09-16T03:39:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775148+0000", 2024-09-16T03:39:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30485311799999998 2024-09-16T03:39:55.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775203+0000", 2024-09-16T03:39:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4592999999999997e-05 2024-09-16T03:39:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775293+0000", 2024-09-16T03:39:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0859999999999994e-05 2024-09-16T03:39:55.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976223+0000", 2024-09-16T03:39:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20092996599999999 2024-09-16T03:39:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976463+0000", 2024-09-16T03:39:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000239695 2024-09-16T03:39:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977730+0000", 2024-09-16T03:39:55.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012667690000000001 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977773+0000", 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2604000000000001e-05 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977800+0000", 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7137000000000001e-05 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 179.36885894900001, 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60633810099999996, 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 74 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.983 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470308+0000", 2024-09-16T03:39:55.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0950000000000001e-06 2024-09-16T03:39:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470313+0000", 2024-09-16T03:39:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:39:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470320+0000", 2024-09-16T03:39:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5089999999999999e-06 2024-09-16T03:39:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470327+0000", 2024-09-16T03:39:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:39:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976483+0000", 2024-09-16T03:39:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50615579499999996 2024-09-16T03:39:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976533+0000", 2024-09-16T03:39:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0096000000000003e-05 2024-09-16T03:39:55.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976643+0000", 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109481 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977808+0000", 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011649869999999999 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.981699+0000", 2024-09-16T03:39:55.987 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003891579 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076496+0000", 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.094797333999999997 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076583+0000", 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6571999999999997e-05 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.988 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076645+0000", 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2331999999999999e-05 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 179.36882881400001, 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71251765099999997, 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.989 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 75 2024-09-16T03:39:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:39:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.990 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:39:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470338+0000", 2024-09-16T03:39:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2500000000000005e-07 2024-09-16T03:39:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470389+0000", 2024-09-16T03:39:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.134e-05 2024-09-16T03:39:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.991 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470397+0000", 2024-09-16T03:39:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8140000000000001e-06 2024-09-16T03:39:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570537+0000", 2024-09-16T03:39:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100139425 2024-09-16T03:39:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977018+0000", 2024-09-16T03:39:55.992 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40648102899999999 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977063+0000", 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5247000000000002e-05 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977143+0000", 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0054000000000003e-05 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:55.993 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.978132+0000", 2024-09-16T03:39:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00098896399999999995 2024-09-16T03:39:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182725+0000", 2024-09-16T03:39:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.204592626 2024-09-16T03:39:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182783+0000", 2024-09-16T03:39:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8285000000000002e-05 2024-09-16T03:39:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.994 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182825+0000", 2024-09-16T03:39:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2398999999999998e-05 2024-09-16T03:39:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182855+0000", 2024-09-16T03:39:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9842999999999999e-05 2024-09-16T03:39:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.995 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:39:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 179.26860224699999, 2024-09-16T03:39:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71263271800000005, 2024-09-16T03:39:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 76 2024-09-16T03:39:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.996 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570566+0000", 2024-09-16T03:39:55.997 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570577+0000", 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0672999999999999e-05 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570593+0000", 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5979999999999999e-05 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:55.998 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570604+0000", 2024-09-16T03:39:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.077e-05 2024-09-16T03:39:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979890+0000", 2024-09-16T03:39:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40928624200000002 2024-09-16T03:39:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979945+0000", 2024-09-16T03:39:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5651e-05 2024-09-16T03:39:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:55.999 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980038+0000", 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2603999999999996e-05 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980669+0000", 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063121900000000003 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283050+0000", 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30238120800000001 2024-09-16T03:39:56.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283132+0000", 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1841999999999993e-05 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283168+0000", 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5438999999999997e-05 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283197+0000", 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8960000000000001e-05 2024-09-16T03:39:56.001 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:39:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 177.45406714800001, 2024-09-16T03:39:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70799990400000001, 2024-09-16T03:39:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:56.002 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 94 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385100+0000", 2024-09-16T03:39:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9599999999999998e-07 2024-09-16T03:39:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385104+0000", 2024-09-16T03:39:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7579999999999998e-06 2024-09-16T03:39:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:56.004 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385128+0000", 2024-09-16T03:39:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4700999999999999e-05 2024-09-16T03:39:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385139+0000", 2024-09-16T03:39:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0587e-05 2024-09-16T03:39:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787499+0000", 2024-09-16T03:39:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40235976099999998 2024-09-16T03:39:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.005 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787545+0000", 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6418999999999997e-05 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787623+0000", 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7897000000000003e-05 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888605+0000", 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100982321 2024-09-16T03:39:56.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990721+0000", 2024-09-16T03:39:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102115922 2024-09-16T03:39:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093019+0000", 2024-09-16T03:39:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10229788500000001 2024-09-16T03:39:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:56.007 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093051+0000", 2024-09-16T03:39:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2255999999999997e-05 2024-09-16T03:39:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093099+0000", 2024-09-16T03:39:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7500999999999998e-05 2024-09-16T03:39:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:56.008 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:39:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 177.454015825, 2024-09-16T03:39:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80404594699999998, 2024-09-16T03:39:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 95 2024-09-16T03:39:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.009 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:39:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:39:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385151+0000", 2024-09-16T03:39:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:39:56.010 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385165+0000", 2024-09-16T03:39:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4175999999999999e-05 2024-09-16T03:39:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385175+0000", 2024-09-16T03:39:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:39:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:56.011 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385183+0000", 2024-09-16T03:39:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:39:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888643+0000", 2024-09-16T03:39:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50345988200000003 2024-09-16T03:39:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888695+0000", 2024-09-16T03:39:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2834999999999998e-05 2024-09-16T03:39:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.012 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888802+0000", 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010697899999999999 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990658+0000", 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185585699999999 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990917+0000", 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025853600000000002 2024-09-16T03:39:56.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189054+0000", 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198136812 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189135+0000", 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1482000000000004e-05 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189196+0000", 2024-09-16T03:39:56.014 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1175999999999998e-05 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 177.45397519799999, 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80412488400000004, 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:56.015 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 96 2024-09-16T03:39:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:39:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.016 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:39:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385192+0000", 2024-09-16T03:39:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:39:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385267+0000", 2024-09-16T03:39:56.017 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5535000000000001e-05 2024-09-16T03:39:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385277+0000", 2024-09-16T03:39:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6239999999999997e-06 2024-09-16T03:39:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385284+0000", 2024-09-16T03:39:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4800000000000004e-06 2024-09-16T03:39:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.018 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989283+0000", 2024-09-16T03:39:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60399849500000002 2024-09-16T03:39:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989410+0000", 2024-09-16T03:39:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127298 2024-09-16T03:39:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989615+0000", 2024-09-16T03:39:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000205045 2024-09-16T03:39:56.019 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991036+0000", 2024-09-16T03:39:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001421042 2024-09-16T03:39:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189228+0000", 2024-09-16T03:39:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19819188500000001 2024-09-16T03:39:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.020 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189249+0000", 2024-09-16T03:39:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0996999999999999e-05 2024-09-16T03:39:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189293+0000", 2024-09-16T03:39:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3912000000000002e-05 2024-09-16T03:39:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189316+0000", 2024-09-16T03:39:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2739e-05 2024-09-16T03:39:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.021 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 177.453871467, 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80642264100000005, 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:56.022 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 97 2024-09-16T03:39:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:39:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:39:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.023 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385296+0000", 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.006e-06 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385301+0000", 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2630000000000003e-06 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385308+0000", 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6030000000000001e-06 2024-09-16T03:39:56.024 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485459+0000", 2024-09-16T03:39:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150929 2024-09-16T03:39:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990238+0000", 2024-09-16T03:39:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50477992000000005 2024-09-16T03:39:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:56.025 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990287+0000", 2024-09-16T03:39:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8863999999999998e-05 2024-09-16T03:39:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990377+0000", 2024-09-16T03:39:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9900000000000003e-05 2024-09-16T03:39:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093109+0000", 2024-09-16T03:39:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102732055 2024-09-16T03:39:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.026 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189351+0000", 2024-09-16T03:39:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096241766000000006 2024-09-16T03:39:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191570+0000", 2024-09-16T03:39:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00221915 2024-09-16T03:39:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191653+0000", 2024-09-16T03:39:56.027 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3046000000000004e-05 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191717+0000", 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4139000000000004e-05 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 177.35368101, 2024-09-16T03:39:56.028 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90688294999999997, 2024-09-16T03:39:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 98 2024-09-16T03:39:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.029 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:39:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:39:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485487+0000", 2024-09-16T03:39:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1799999999999999e-06 2024-09-16T03:39:56.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485498+0000", 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0648999999999999e-05 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485514+0000", 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5943999999999998e-05 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485525+0000", 2024-09-16T03:39:56.031 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.065e-05 2024-09-16T03:39:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991102+0000", 2024-09-16T03:39:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50557748400000002 2024-09-16T03:39:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991161+0000", 2024-09-16T03:39:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8801999999999997e-05 2024-09-16T03:39:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.032 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991255+0000", 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3936000000000003e-05 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093112+0000", 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185709499999999 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291911+0000", 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19879897699999999 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.033 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392167+0000", 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10025634999999999 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392249+0000", 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2175000000000005e-05 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392368+0000", 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118708 2024-09-16T03:39:56.034 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:39:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 177.35362738699999, 2024-09-16T03:39:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90707203199999997, 2024-09-16T03:39:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:56.035 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 99 2024-09-16T03:39:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:39:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:56.036 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:39:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485540+0000", 2024-09-16T03:39:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-06 2024-09-16T03:39:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485661+0000", 2024-09-16T03:39:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120835 2024-09-16T03:39:56.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485680+0000", 2024-09-16T03:39:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8885999999999998e-05 2024-09-16T03:39:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485693+0000", 2024-09-16T03:39:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3297e-05 2024-09-16T03:39:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.038 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991546+0000", 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.505853098 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991598+0000", 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1921e-05 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991681+0000", 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2502000000000004e-05 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.039 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093115+0000", 2024-09-16T03:39:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10143424299999999 2024-09-16T03:39:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291977+0000", 2024-09-16T03:39:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198862172 2024-09-16T03:39:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392541+0000", 2024-09-16T03:39:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100563758 2024-09-16T03:39:56.040 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392579+0000", 2024-09-16T03:39:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7796999999999997e-05 2024-09-16T03:39:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392611+0000", 2024-09-16T03:39:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2073999999999998e-05 2024-09-16T03:39:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.041 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 177.35344849399999, 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0073149939999999, 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 100 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.042 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:39:56.043 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7400000000000003e-07 2024-09-16T03:39:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485721+0000", 2024-09-16T03:39:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4530000000000001e-06 2024-09-16T03:39:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485726+0000", 2024-09-16T03:39:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0590000000000002e-06 2024-09-16T03:39:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.044 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485729+0000", 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092342+0000", 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60661343700000003 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092469+0000", 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012658799999999999 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.045 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092671+0000", 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020206 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392452+0000", 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29978163699999999 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392486+0000", 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3736999999999999e-05 2024-09-16T03:39:56.046 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492812+0000", 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100325621 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492901+0000", 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8702000000000006e-05 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.493033+0000", 2024-09-16T03:39:56.047 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000132185 2024-09-16T03:39:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:39:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 175.17829797300001, 2024-09-16T03:39:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60874581500000002, 2024-09-16T03:39:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:56.048 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 123 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:56.049 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660869+0000", 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9400000000000005e-07 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660921+0000", 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2312000000000003e-05 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.050 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660927+0000", 2024-09-16T03:39:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9329999999999998e-06 2024-09-16T03:39:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660931+0000", 2024-09-16T03:39:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:39:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.967991+0000", 2024-09-16T03:39:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30706032700000002 2024-09-16T03:39:56.051 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968039+0000", 2024-09-16T03:39:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7477e-05 2024-09-16T03:39:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968117+0000", 2024-09-16T03:39:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8589000000000003e-05 2024-09-16T03:39:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:56.052 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067678+0000", 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099560406000000004 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170175+0000", 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102496927 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269491+0000", 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099316205000000005 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.053 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269574+0000", 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2903000000000002e-05 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269614+0000", 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0188000000000002e-05 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:39:56.054 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 175.17822927500001, 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60879245500000001, 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 124 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:56.055 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:39:56.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:56.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:39:56.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.056 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:56.056 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660938+0000", 2024-09-16T03:39:56.056 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:39:56.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.056 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660940+0000", 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2529999999999999e-06 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660943+0000", 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.01e-06 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660946+0000", 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8159999999999998e-06 2024-09-16T03:39:56.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012417+0000", 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.35147120599999998 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012497+0000", 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9963000000000007e-05 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012585+0000", 2024-09-16T03:39:56.058 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:39:56.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:56.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067680+0000", 2024-09-16T03:39:56.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055095203000000002 2024-09-16T03:39:56.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.059 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170243+0000", 2024-09-16T03:39:56.059 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10256290799999999 2024-09-16T03:39:56.059 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.059 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.059 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269662+0000", 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099418896000000007 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269701+0000", 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8829999999999999e-05 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269729+0000", 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8569e-05 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.060 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.061 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.061 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.061 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.061 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1550 4.13 4:c8492bfc:::smithi15736696-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:39:56.061 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:56.061 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 130.172501135, 2024-09-16T03:39:56.061 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62706974999999998, 2024-09-16T03:39:56.061 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:56.061 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:56.061 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:56.061 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:56.061 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:56.061 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1550 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.062 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0699999999999998e-07 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666669+0000", 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.393e-06 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666673+0000", 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:39:56.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:56.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666682+0000", 2024-09-16T03:39:56.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6510000000000007e-06 2024-09-16T03:39:56.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.913940+0000", 2024-09-16T03:39:56.064 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.24725763000000001 2024-09-16T03:39:56.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.064 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.064 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:56.064 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.914034+0000", 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3923000000000002e-05 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293650+0000", 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.37961662200000001 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293688+0000", 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8096000000000003e-05 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.065 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293735+0000", 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6451e-05 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:77 3.2 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 115.760624514, 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.61452436499999996, 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:56.066 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:56.067 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078543+0000", 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.553e-06 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078551+0000", 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0790000000000001e-06 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078564+0000", 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2943e-05 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:56.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178635+0000", 2024-09-16T03:39:56.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10007044599999999 2024-09-16T03:39:56.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178684+0000", 2024-09-16T03:39:56.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9592999999999998e-05 2024-09-16T03:39:56.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.069 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.069 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:56.069 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178747+0000", 2024-09-16T03:39:56.069 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2286000000000006e-05 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4,5", 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178857+0000", 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110807 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280364+0000", 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101506518 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:39:56.070 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280373+0000", 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6899999999999998e-06 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.369249+0000", 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.088876172000000003 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693044+0000", 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.32379486499999999 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:56.071 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693066+0000", 2024-09-16T03:39:56.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2413e-05 2024-09-16T03:39:56.072 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.072 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.072 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.072 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.072 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3119 4.13 4:c8492bfc:::smithi15736696-42:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:39:56.072 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:39:56.072 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 78.907727457999997, 2024-09-16T03:39:56.072 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.64286237700000004, 2024-09-16T03:39:56.072 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:39:56.072 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:39:56.072 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:39:56.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:39:56.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:39:56.073 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:56.073 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3119 2024-09-16T03:39:56.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.073 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:39:56.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:39:56.073 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:39:56.073 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.073 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.073 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.073 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931441+0000", 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3599999999999999e-06 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931449+0000", 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4080000000000001e-06 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931463+0000", 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.488e-05 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931473+0000", 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8109999999999997e-06 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.031604+0000", 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013079499999999 2024-09-16T03:39:56.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:39:56.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.031668+0000", 2024-09-16T03:39:56.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4226999999999993e-05 2024-09-16T03:39:56.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:39:56.076 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:39:56.076 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:39:56.076 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.574301+0000", 2024-09-16T03:39:56.076 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54263289599999998 2024-09-16T03:39:56.076 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.076 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.076 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.076 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:39:56.077 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:39:56.077 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:39:56.078 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:56.078 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:56.079 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:56.079 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:56.079 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:39:56.080 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:56.080 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:56.080 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:56.081 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:56.081 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:56.081 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:56.081 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:56.082 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:56.082 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:56.082 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:56.082 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:56.083 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:56.083 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:56.083 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:56.084 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:56.084 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:56.084 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:56.076+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:56.093 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:56.093 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:56.110 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:56.110 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:56.127 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:56.127 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:56.144 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:56.144 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:56.160 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:56.161 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:56.177 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:56.177 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:56.178 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:56.177+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:56.191 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:56.191 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:56.200 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:39:56.214 INFO:tasks.rados.rados.0.smithi157.stdout:2039: finishing write tid 1 to smithi15736696-43 2024-09-16T03:39:56.214 INFO:tasks.rados.rados.0.smithi157.stdout:2039: finishing write tid 2 to smithi15736696-43 2024-09-16T03:39:56.280 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:56.278+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:56.303 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:56.304 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:56.312 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:39:56.397 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:39:56.395+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:39:56.417 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:56.415+0000 7f3d40708640 -1 osd.12 171 handle_osd_map injecting map crc failure 2024-09-16T03:39:56.458 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:56.458 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:56.467 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:39:56.510 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:56.509+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:56.512 INFO:tasks.rados.rados.0.smithi157.stdout:2039: finishing write tid 3 to smithi15736696-43 2024-09-16T03:39:56.578 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:56.578 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:56.582 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:56.581+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:56.587 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:39:56.613 INFO:tasks.rados.rados.0.smithi157.stdout:2039: finishing write tid 4 to smithi15736696-43 2024-09-16T03:39:56.613 INFO:tasks.rados.rados.0.smithi157.stdout:2039: finishing write tid 5 to smithi15736696-43 2024-09-16T03:39:56.614 INFO:tasks.rados.rados.0.smithi157.stdout:2039: oid 43 updating version 0 to 99 2024-09-16T03:39:56.614 INFO:tasks.rados.rados.0.smithi157.stdout:2039: oid 43 version 99 is already newer than 97 2024-09-16T03:39:56.614 INFO:tasks.rados.rados.0.smithi157.stdout:2039: oid 43 updating version 99 to 100 2024-09-16T03:39:56.614 INFO:tasks.rados.rados.0.smithi157.stdout:2039: oid 43 version 100 is already newer than 98 2024-09-16T03:39:56.614 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 43 v 100 (ObjNum 459 snap 0 seq_num 459) dirty exists 2024-09-16T03:39:56.614 INFO:tasks.rados.rados.0.smithi157.stdout:2039: left oid 43 (ObjNum 459 snap 0 seq_num 459) 2024-09-16T03:39:56.614 INFO:tasks.rados.rados.0.smithi157.stdout:2039: done (1 left) 2024-09-16T03:39:56.614 INFO:tasks.rados.rados.0.smithi157.stdout: waiting on 1 2024-09-16T03:39:56.684 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:56.682+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:56.723 INFO:tasks.rados.rados.0.smithi157.stdout:2042: got (0) Success 2024-09-16T03:39:56.724 INFO:tasks.rados.rados.0.smithi157.stdout:update_object_version oid 8 v 97 (ObjNum 359 snap 0 seq_num 359) dirty exists 2024-09-16T03:39:56.724 INFO:tasks.rados.rados.0.smithi157.stdout:2042: done (0 left) 2024-09-16T03:39:56.735 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:56.736 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:56.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:39:56.786 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:56.783+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:56.854 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:56.854 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:56.863 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:39:56.878 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:39:56.885 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:39:56.883+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:39:56.972 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:56.973 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:56.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:39:56.986 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:56.984+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:57.087 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:57.085+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:57.091 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:57.092 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:57.101 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:39:57.188 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:39:57.187+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:39:57.211 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:57.211 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:57.219 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:39:57.289 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:57.287+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:57.327 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:57.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:39:57.378 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:39:57.380 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:57.378+0000 7f9288166640 -1 osd.14 172 handle_osd_map injecting map crc failure 2024-09-16T03:39:57.390 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:57.389+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:57.485 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:57.486 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:57.491 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:57.489+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:57.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:39:57.542 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:57.540+0000 7f3d40708640 -1 osd.12 172 handle_osd_map injecting map crc failure 2024-09-16T03:39:57.592 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:57.591+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:57.603 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:57.603 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:57.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:39:57.693 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:39:57.691+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:39:57.721 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:57.721 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:57.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:39:57.794 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:57.793+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:57.836 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:57.837 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:57.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:39:57.895 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:39:57.893+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:39:57.955 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:57.955 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:57.963 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:39:57.995 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:57.994+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:58.079 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:58.079 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:58.088 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:39:58.132 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:58.130+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:58.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:39:58.192 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:39:58.192 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:39:58.201 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:39:58.233 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:39:58.231+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:39:58.298 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:58.296+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:58.314 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:39:58.314 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:39:58.331 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:39:58.332 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:39:58.346 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:58.347 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:58.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:58.347 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:58.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.347 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.21:5 3.0 e45/44 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'39, mlcod=45'37)", 2024-09-16T03:39:58.347 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:39:58.347 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 157.42975540399999, 2024-09-16T03:39:58.347 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20119541799999999, 2024-09-16T03:39:58.347 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.347 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.347 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.347 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:39:58.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.348 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.348 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:39:58.348 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914597+0000", 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.494e-06 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914603+0000", 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7349999999999999e-06 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914614+0000", 2024-09-16T03:39:58.349 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0995999999999999e-05 2024-09-16T03:39:58.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914623+0000", 2024-09-16T03:39:58.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3899999999999993e-06 2024-09-16T03:39:58.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.350 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.350 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.114941+0000", 2024-09-16T03:39:58.350 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20031812399999999 2024-09-16T03:39:58.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115001+0000", 2024-09-16T03:39:58.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0161999999999998e-05 2024-09-16T03:39:58.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.351 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115750+0000", 2024-09-16T03:39:58.351 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074851699999999998 2024-09-16T03:39:58.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.351 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.352 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115791+0000", 2024-09-16T03:39:58.353 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1e-05 2024-09-16T03:39:58.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.354 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.354 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.354 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.5.21:36 3.0 e52/44 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 52'62, mlcod=51'60)", 2024-09-16T03:39:58.354 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:39:58.354 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 146.45972474499999, 2024-09-16T03:39:58.354 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20104840199999999, 2024-09-16T03:39:58.354 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.354 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.354 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.354 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884627+0000", 2024-09-16T03:39:58.355 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.322e-06 2024-09-16T03:39:58.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884634+0000", 2024-09-16T03:39:58.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:39:58.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.356 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.356 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884643+0000", 2024-09-16T03:39:58.356 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7680000000000003e-06 2024-09-16T03:39:58.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984740+0000", 2024-09-16T03:39:58.357 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100096066 2024-09-16T03:39:58.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.357 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.357 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084931+0000", 2024-09-16T03:39:58.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100191408 2024-09-16T03:39:58.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084952+0000", 2024-09-16T03:39:58.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.128e-05 2024-09-16T03:39:58.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.358 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.358 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085645+0000", 2024-09-16T03:39:58.358 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00069309500000000002 2024-09-16T03:39:58.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085674+0000", 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9025000000000001e-05 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 128.19442614799999, 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20169899299999999, 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.359 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.360 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149927+0000", 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9299999999999999e-06 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151090+0000", 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011624459999999999 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151105+0000", 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5238999999999999e-05 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251193+0000", 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100088025 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251556+0000", 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036245599999999999 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251578+0000", 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2651e-05 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351582+0000", 2024-09-16T03:39:58.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100003811 2024-09-16T03:39:58.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.363 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.363 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351624+0000", 2024-09-16T03:39:58.363 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1434999999999999e-05 2024-09-16T03:39:58.363 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.363 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.363 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.364 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1742 3.0 e70/69 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head v 70'104, mlcod=0'0)", 2024-09-16T03:39:58.364 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:58.364 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 122.568551169, 2024-09-16T03:39:58.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20271830699999999, 2024-09-16T03:39:58.364 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.364 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.364 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.364 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.364 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.364 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:58.364 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:58.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:39:58.365 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:39:58.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.365 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.365 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777455+0000", 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001655524 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777468+0000", 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2486999999999999e-05 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777474+0000", 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.366 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876577+0000", 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099103360000000001 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876594+0000", 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6449000000000001e-05 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978469+0000", 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101875521 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.368 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.368 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978518+0000", 2024-09-16T03:39:58.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8488000000000001e-05 2024-09-16T03:39:58.368 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.368 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.368 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.368 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:91 3.0 e70/69 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head v 70'105, mlcod=0'0)", 2024-09-16T03:39:58.368 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:39:58.368 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 122.48233930000001, 2024-09-16T03:39:58.368 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.116532025, 2024-09-16T03:39:58.368 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862015+0000", 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3569999999999998e-06 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862029+0000", 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4082e-05 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862055+0000", 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6570000000000001e-05 2024-09-16T03:39:58.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862070+0000", 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4473e-05 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877631+0000", 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.015560744 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877674+0000", 2024-09-16T03:39:58.371 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3615999999999997e-05 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978525+0000", 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100850696 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978543+0000", 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8487000000000001e-05 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.373 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:94 3.0 e70/69 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head v 70'106, mlcod=0'0)", 2024-09-16T03:39:58.373 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:39:58.373 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 122.481803081, 2024-09-16T03:39:58.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11627603, 2024-09-16T03:39:58.373 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.373 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.373 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.373 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.373 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.373 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.373 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:39:58.373 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862549+0000", 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1650000000000001e-06 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.374 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862556+0000", 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8709999999999999e-06 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862566+0000", 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0271e-05 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862573+0000", 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.861e-06 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.375 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978123+0000", 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11554993199999999 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978163+0000", 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0201999999999999e-05 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978796+0000", 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063316700000000002 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.376 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.377 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978824+0000", 2024-09-16T03:39:58.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7560999999999999e-05 2024-09-16T03:39:58.377 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.377 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.377 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.377 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1979 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'32, mlcod=0'0)", 2024-09-16T03:39:58.377 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:39:58.377 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 116.50661239599999, 2024-09-16T03:39:58.377 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20429768500000001, 2024-09-16T03:39:58.377 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.377 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.378 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.378 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:39:58.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.378 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.378 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:39:58.378 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837740+0000", 2024-09-16T03:39:58.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.779e-06 2024-09-16T03:39:58.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838356+0000", 2024-09-16T03:39:58.379 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061634500000000002 2024-09-16T03:39:58.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.379 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.379 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838371+0000", 2024-09-16T03:39:58.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5007e-05 2024-09-16T03:39:58.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838380+0000", 2024-09-16T03:39:58.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1009999999999992e-06 2024-09-16T03:39:58.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.380 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.380 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038674+0000", 2024-09-16T03:39:58.380 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20029450500000001 2024-09-16T03:39:58.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.380 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038710+0000", 2024-09-16T03:39:58.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6100999999999999e-05 2024-09-16T03:39:58.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042001+0000", 2024-09-16T03:39:58.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032909430000000002 2024-09-16T03:39:58.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.381 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.381 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042036+0000", 2024-09-16T03:39:58.381 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4904000000000003e-05 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1980 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'33, mlcod=0'0)", 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 116.50473580000001, 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20296795100000001, 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:39:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:39:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839616+0000", 2024-09-16T03:39:58.383 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3099999999999996e-07 2024-09-16T03:39:58.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840706+0000", 2024-09-16T03:39:58.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010900980000000001 2024-09-16T03:39:58.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.384 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.384 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840720+0000", 2024-09-16T03:39:58.384 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.378e-05 2024-09-16T03:39:58.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840727+0000", 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.525e-06 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039940+0000", 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19921313199999999 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039967+0000", 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6393999999999998e-05 2024-09-16T03:39:58.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042552+0000", 2024-09-16T03:39:58.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025850999999999999 2024-09-16T03:39:58.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.386 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.386 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042583+0000", 2024-09-16T03:39:58.386 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0991000000000002e-05 2024-09-16T03:39:58.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.386 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1981 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'34, mlcod=0'0)", 2024-09-16T03:39:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:39:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 116.502713445, 2024-09-16T03:39:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201270175, 2024-09-16T03:39:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:39:58.387 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841638+0000", 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.388 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841644+0000", 2024-09-16T03:39:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8050000000000003e-06 2024-09-16T03:39:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841653+0000", 2024-09-16T03:39:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6580000000000008e-06 2024-09-16T03:39:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841660+0000", 2024-09-16T03:39:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9290000000000001e-06 2024-09-16T03:39:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041673+0000", 2024-09-16T03:39:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20001311499999999 2024-09-16T03:39:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041696+0000", 2024-09-16T03:39:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3365000000000001e-05 2024-09-16T03:39:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042878+0000", 2024-09-16T03:39:58.390 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011814499999999999 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042907+0000", 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9655999999999999e-05 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2536 3.0 e86/69 3:10a17b1b:::13dc2fab1f287170fabf96291b912bf80e0b081f291f2c6d41d494566f2d1f8c:head v 86'176, mlcod=86'174)", 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 101.50483906700001, 2024-09-16T03:39:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20664157499999999, 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839515+0000", 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3330000000000001e-06 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843392+0000", 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038769329999999999 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.393 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843412+0000", 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0608e-05 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.943546+0000", 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100133176 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043824+0000", 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100278785 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.394 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043869+0000", 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4845999999999997e-05 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046096+0000", 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022262559999999998 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046153+0000", 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7637999999999999e-05 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.395 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.396 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2654 4.8 e90/68 4:148e627d:::smithi15736696-23:head v 90'58, mlcod=80'56)", 2024-09-16T03:39:58.396 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:39:58.396 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 96.969451708999998, 2024-09-16T03:39:58.396 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.106604212, 2024-09-16T03:39:58.396 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.396 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.396 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.396 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.396 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:39:58.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:39:58.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.397 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.397 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374901+0000", 2024-09-16T03:39:58.397 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1749999999999999e-06 2024-09-16T03:39:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375810+0000", 2024-09-16T03:39:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00090911400000000004 2024-09-16T03:39:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375827+0000", 2024-09-16T03:39:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7010000000000001e-05 2024-09-16T03:39:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476010+0000", 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100182879 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476184+0000", 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017408899999999999 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476206+0000", 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2013000000000001e-05 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.399 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481458+0000", 2024-09-16T03:39:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052519120000000001 2024-09-16T03:39:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481503+0000", 2024-09-16T03:39:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5019999999999999e-05 2024-09-16T03:39:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.400 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "pg_update_log_missing(4.c epoch 99/68 rep_tid 69 entries 99'56 (0'0) error 4:34492f38:::smithi15736696-33:head by client.4628.0:3000 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 90'53 roll_forward_to 90'54)", 2024-09-16T03:39:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:39:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 86.424493557999995, 2024-09-16T03:39:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201155684, 2024-09-16T03:39:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:39:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:39:58.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.402 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919858+0000", 2024-09-16T03:39:58.402 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:39:58.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.402 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919871+0000", 2024-09-16T03:39:58.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3433e-05 2024-09-16T03:39:58.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.403 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.403 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919929+0000", 2024-09-16T03:39:58.403 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7543000000000001e-05 2024-09-16T03:39:58.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919942+0000", 2024-09-16T03:39:58.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3122000000000001e-05 2024-09-16T03:39:58.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.404 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.120302+0000", 2024-09-16T03:39:58.404 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036052800000001 2024-09-16T03:39:58.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.404 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.121013+0000", 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071046800000000002 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:4644 3.0 e150/149 3:1f968072:::a9957f29a7f946349b562c449f43fa1bc2e3c48b00925943f7b23963f3ffcd03:head v 150'271, mlcod=0'0)", 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:29.950532+0000", 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 28.393818304, 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30108226199999999, 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.405 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950532+0000", 2024-09-16T03:39:58.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950532+0000", 2024-09-16T03:39:58.406 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.406 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.406 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950533+0000", 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950538+0000", 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5170000000000001e-06 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950544+0000", 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0580000000000003e-06 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.407 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950550+0000", 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2330000000000003e-06 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.050698+0000", 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100147758 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.050724+0000", 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251572+0000", 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084784999999999 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251615+0000", 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2317999999999997e-05 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.409 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:179 3.0 e150/149 3:004ec411:::10547772e2dbc1bd82bb7d7e420056cffb9d849bbc3fd015ab311821f554ea0c:head v 150'272, mlcod=0'0)", 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:29.953579+0000", 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 28.390771839999999, 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.29807083000000001, 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953579+0000", 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.410 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953579+0000", 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953579+0000", 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5600000000000005e-07 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953584+0000", 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.126e-06 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.411 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953590+0000", 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0680000000000002e-06 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953594+0000", 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5480000000000003e-06 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.151091+0000", 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.197496745 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.151114+0000", 2024-09-16T03:39:58.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3045e-05 2024-09-16T03:39:58.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251621+0000", 2024-09-16T03:39:58.413 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10050719800000001 2024-09-16T03:39:58.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.413 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.413 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251650+0000", 2024-09-16T03:39:58.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8444000000000001e-05 2024-09-16T03:39:58.414 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.414 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.414 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.414 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "pg_scan(get_digest 4.8 MIN-MIN e 150/149)", 2024-09-16T03:39:58.414 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:30.051498+0000", 2024-09-16T03:39:58.414 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 28.292852892999999, 2024-09-16T03:39:58.414 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20012074299999999, 2024-09-16T03:39:58.414 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.414 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.414 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051498+0000", 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051498+0000", 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.415 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051499+0000", 2024-09-16T03:39:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6899999999999996e-07 2024-09-16T03:39:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051509+0000", 2024-09-16T03:39:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0907000000000001e-05 2024-09-16T03:39:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051517+0000", 2024-09-16T03:39:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3690000000000001e-06 2024-09-16T03:39:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051523+0000", 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7950000000000004e-06 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251377+0000", 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19985429800000001 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251392+0000", 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4603000000000001e-05 2024-09-16T03:39:58.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.418 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.418 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251619+0000", 2024-09-16T03:39:58.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022700199999999999 2024-09-16T03:39:58.418 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.418 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.418 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.418 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "pg_scan(get_digest 3.0 MIN-MIN e 150/149)", 2024-09-16T03:39:58.418 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:30.456069+0000", 2024-09-16T03:39:58.418 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 27.888282126, 2024-09-16T03:39:58.418 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200403153, 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456069+0000", 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456069+0000", 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456070+0000", 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3829999999999999e-06 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456074+0000", 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2289999999999997e-06 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456083+0000", 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9789999999999999e-06 2024-09-16T03:39:58.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.556184+0000", 2024-09-16T03:39:58.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100101147 2024-09-16T03:39:58.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.656320+0000", 2024-09-16T03:39:58.421 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100135347 2024-09-16T03:39:58.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.421 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.421 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.656328+0000", 2024-09-16T03:39:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4610000000000008e-06 2024-09-16T03:39:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.656472+0000", 2024-09-16T03:39:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000143607 2024-09-16T03:39:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:4811 3.0 e155/149 3:3c95911f:::e0ef79a5931af77a89e00e45b270ee95266718455ec934bcc09140d5d71025ab:head v 155'275, mlcod=150'273)", 2024-09-16T03:39:58.422 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:35.639297+0000", 2024-09-16T03:39:58.423 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 22.705054032, 2024-09-16T03:39:58.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.105075237, 2024-09-16T03:39:58.423 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.423 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.423 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.423 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.423 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.639297+0000", 2024-09-16T03:39:58.423 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.423 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.639297+0000", 2024-09-16T03:39:58.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.639298+0000", 2024-09-16T03:39:58.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8059999999999999e-06 2024-09-16T03:39:58.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.424 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.424 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.642391+0000", 2024-09-16T03:39:58.424 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030922050000000002 2024-09-16T03:39:58.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.642407+0000", 2024-09-16T03:39:58.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6393e-05 2024-09-16T03:39:58.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.425 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.425 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.642416+0000", 2024-09-16T03:39:58.425 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8079999999999994e-06 2024-09-16T03:39:58.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.426 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.742572+0000", 2024-09-16T03:39:58.426 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10015578999999999 2024-09-16T03:39:58.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.426 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.742596+0000", 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4406e-05 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.744316+0000", 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017202090000000001 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.744372+0000", 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5619999999999999e-05 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.427 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.428 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.428 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:4839 4.1f e164/153 4:fa163101:::smithi15736696-10:head v 164'125, mlcod=152'123)", 2024-09-16T03:39:58.428 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:48.893119+0000", 2024-09-16T03:39:58.428 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 9.4512317790000004, 2024-09-16T03:39:58.428 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10403327, 2024-09-16T03:39:58.428 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.428 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.428 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.428 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.428 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.893119+0000", 2024-09-16T03:39:58.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.893119+0000", 2024-09-16T03:39:58.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.429 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.429 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.893122+0000", 2024-09-16T03:39:58.429 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6280000000000001e-06 2024-09-16T03:39:58.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.894042+0000", 2024-09-16T03:39:58.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00092027899999999998 2024-09-16T03:39:58.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.894058+0000", 2024-09-16T03:39:58.430 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5956000000000001e-05 2024-09-16T03:39:58.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.430 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.430 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.894067+0000", 2024-09-16T03:39:58.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7240000000000004e-06 2024-09-16T03:39:58.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.994267+0000", 2024-09-16T03:39:58.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10019945199999999 2024-09-16T03:39:58.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.431 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.431 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.994285+0000", 2024-09-16T03:39:58.431 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7923000000000001e-05 2024-09-16T03:39:58.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.997112+0000", 2024-09-16T03:39:58.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028274749999999999 2024-09-16T03:39:58.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.432 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.432 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.997152+0000", 2024-09-16T03:39:58.432 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9833000000000003e-05 2024-09-16T03:39:58.432 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.432 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.432 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:4862 4.d e165/153 4:b3fe05b2:::smithi15736696-15:head v 165'27, mlcod=0'0)", 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:49.449186+0000", 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 8.8951642849999999, 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30053745799999998, 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.449186+0000", 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.449186+0000", 2024-09-16T03:39:58.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.449187+0000", 2024-09-16T03:39:58.434 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2999999999999999e-07 2024-09-16T03:39:58.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.434 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.434 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.449192+0000", 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0889999999999999e-06 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.449199+0000", 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2870000000000001e-06 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.549302+0000", 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100103716 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.435 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.569629+0000", 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.020326956 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.569652+0000", 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2345e-05 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.749678+0000", 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.18002673 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.436 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.749724+0000", 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5405000000000003e-05 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:4949 4.1f e169/168 4:fcb75e43:::smithi15736696-26:head v 169'129, mlcod=0'0)", 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:54.102759+0000", 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 4.2415921689999996, 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10516655599999999, 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.437 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.438 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.102759+0000", 2024-09-16T03:39:58.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.438 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.102759+0000", 2024-09-16T03:39:58.438 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.438 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.102760+0000", 2024-09-16T03:39:58.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1209999999999999e-06 2024-09-16T03:39:58.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.103993+0000", 2024-09-16T03:39:58.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012336669999999999 2024-09-16T03:39:58.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.439 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.439 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.104004+0000", 2024-09-16T03:39:58.439 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1056000000000001e-05 2024-09-16T03:39:58.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.104013+0000", 2024-09-16T03:39:58.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6100000000000006e-06 2024-09-16T03:39:58.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.440 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.201992+0000", 2024-09-16T03:39:58.440 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097979207999999998 2024-09-16T03:39:58.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.440 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.440 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.202045+0000", 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2611000000000002e-05 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.207868+0000", 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0058229090000000002 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.207925+0000", 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7374000000000003e-05 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.441 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.442 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.442 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:58.443 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:39:58.443 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:39:58.444 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:39:58.444 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:39:58.444 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:39:58.445 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:39:58.445 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:39:58.446 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:39:58.446 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:39:58.446 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:39:58.446 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:39:58.447 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:39:58.447 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:39:58.447 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:39:58.442+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:39:58.450 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:39:58.451 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:39:58.469 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:39:58.469 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:39:58.544 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:58.542+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:58.588 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:58.589 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:58.589 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:58.589 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:58.596 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:39:58.597 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:39:58.598 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:39:58.613 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:39:58.614 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:39:58.629 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:39:58.629 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:39:58.645 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:39:58.645 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:39:58.646 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:39:58.644+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:39:58.662 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:39:58.662 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:39:58.680 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:39:58.680 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:39:58.740 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:58.741 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:58.741 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:58.741 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:58.741 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:58.746 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:58.744+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:58.749 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:39:58.847 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:58.846+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:58.893 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:58.893 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:58.893 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:25 4.6 4:6ece7830:::smithi15736696-7:head [write 500842~451494,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 166.77092963000001, 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40559452699999998, 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 25 2024-09-16T03:39:58.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.895 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:39:58.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.895 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119829+0000", 2024-09-16T03:39:58.895 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.895 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.119830+0000", 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9200000000000001e-07 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120013+0000", 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018367200000000001 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120021+0000", 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9179999999999994e-06 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120027+0000", 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6710000000000004e-06 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220194+0000", 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10016689500000001 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220399+0000", 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020502899999999999 2024-09-16T03:39:58.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:58.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.220613+0000", 2024-09-16T03:39:58.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021383900000000001 2024-09-16T03:39:58.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224640+0000", 2024-09-16T03:39:58.898 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0040268099999999996 2024-09-16T03:39:58.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.898 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:58.898 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423348+0000", 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19870853399999999 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525343+0000", 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101995081 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525381+0000", 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7842000000000001e-05 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.899 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525424+0000", 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2644e-05 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:26 4.6 4:6ece7830:::smithi15736696-7:head [write 1718186~382656] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 166.77071740299999, 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550111300000002, 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.900 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 26 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.901 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120041+0000", 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120042+0000", 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7999999999999995e-07 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120210+0000", 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016779499999999999 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.902 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120219+0000", 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6859999999999995e-06 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120223+0000", 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0330000000000002e-06 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221494+0000", 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101271812 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221572+0000", 2024-09-16T03:39:58.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7733999999999998e-05 2024-09-16T03:39:58.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:58.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.221697+0000", 2024-09-16T03:39:58.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012474900000000001 2024-09-16T03:39:58.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.904 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.904 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224670+0000", 2024-09-16T03:39:58.904 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029729589999999998 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423398+0000", 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19872784900000001 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525465+0000", 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102067557 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525494+0000", 2024-09-16T03:39:58.905 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9113999999999999e-05 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525542+0000", 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8145000000000001e-05 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:27 4.6 4:6ece7830:::smithi15736696-7:head [setxattr _header (55) in=7b,truncate 2100842] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 166.770529818, 2024-09-16T03:39:58.906 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40541507399999999, 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 27 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120229+0000", 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120230+0000", 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5799999999999999e-07 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120233+0000", 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.055e-06 2024-09-16T03:39:58.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120237+0000", 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7099999999999998e-06 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120244+0000", 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4810000000000003e-06 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222660+0000", 2024-09-16T03:39:58.909 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102416089 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222705+0000", 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5368999999999999e-05 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222782+0000", 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6623000000000001e-05 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224696+0000", 2024-09-16T03:39:58.910 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019143929999999999 2024-09-16T03:39:58.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:58.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423402+0000", 2024-09-16T03:39:58.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198705305 2024-09-16T03:39:58.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.911 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525582+0000", 2024-09-16T03:39:58.911 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102180178 2024-09-16T03:39:58.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.911 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.911 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525615+0000", 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3303000000000003e-05 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525644+0000", 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8909999999999999e-05 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:28 4.6 4:6ece7830:::smithi15736696-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:58.912 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:58.913 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 166.77050907, 2024-09-16T03:39:58.913 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40550787799999999, 2024-09-16T03:39:58.913 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.913 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.913 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:58.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.913 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.913 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 28 2024-09-16T03:39:58.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.913 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.913 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:58.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:58.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.914 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.914 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120250+0000", 2024-09-16T03:39:58.914 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7999999999999996e-07 2024-09-16T03:39:58.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120252+0000", 2024-09-16T03:39:58.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.249e-06 2024-09-16T03:39:58.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.915 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120257+0000", 2024-09-16T03:39:58.915 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1969999999999998e-06 2024-09-16T03:39:58.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.915 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.915 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120261+0000", 2024-09-16T03:39:58.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1130000000000003e-06 2024-09-16T03:39:58.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.222976+0000", 2024-09-16T03:39:58.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102715397 2024-09-16T03:39:58.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.916 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:58.916 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223049+0000", 2024-09-16T03:39:58.916 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2846e-05 2024-09-16T03:39:58.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525666+0000", 2024-09-16T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30261668800000002 2024-09-16T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525704+0000", 2024-09-16T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8025999999999998e-05 2024-09-16T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.917 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525758+0000", 2024-09-16T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3881999999999997e-05 2024-09-16T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:49 4.6 4:6df3bbbe:::smithi15736696-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 166.77049400600001, 2024-09-16T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40556222800000002, 2024-09-16T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.918 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.919 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.919 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.919 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.919 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 49 2024-09-16T03:39:58.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.919 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.919 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:58.919 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.919 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.919 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120265+0000", 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8700000000000002e-07 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120454+0000", 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018909299999999999 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120465+0000", 2024-09-16T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.083e-05 2024-09-16T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120468+0000", 2024-09-16T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3869999999999999e-06 2024-09-16T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223082+0000", 2024-09-16T03:39:58.921 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102613379 2024-09-16T03:39:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223189+0000", 2024-09-16T03:39:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000107728 2024-09-16T03:39:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.223368+0000", 2024-09-16T03:39:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000179011 2024-09-16T03:39:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.922 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.226087+0000", 2024-09-16T03:39:58.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0027183390000000002 2024-09-16T03:39:58.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:58.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423408+0000", 2024-09-16T03:39:58.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19732106499999999 2024-09-16T03:39:58.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.923 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.923 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525779+0000", 2024-09-16T03:39:58.923 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10237133 2024-09-16T03:39:58.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525805+0000", 2024-09-16T03:39:58.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5967000000000001e-05 2024-09-16T03:39:58.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.924 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.924 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.525827+0000", 2024-09-16T03:39:58.924 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1812000000000001e-05 2024-09-16T03:39:58.924 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.924 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.924 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:50 4.6 4:6df3bbbe:::smithi15736696-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 166.77028569699999, 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50526462999999999, 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 50 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.925 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:39:58.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.926 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.926 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120473+0000", 2024-09-16T03:39:58.926 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120474+0000", 2024-09-16T03:39:58.927 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4600000000000005e-07 2024-09-16T03:39:58.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.927 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.927 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120901+0000", 2024-09-16T03:39:58.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00042690000000000002 2024-09-16T03:39:58.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120907+0000", 2024-09-16T03:39:58.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7710000000000001e-06 2024-09-16T03:39:58.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.928 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.928 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.928 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120911+0000", 2024-09-16T03:39:58.928 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0500000000000002e-06 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224717+0000", 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103806175 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224763+0000", 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5629e-05 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:58.929 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.224838+0000", 2024-09-16T03:39:58.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4548999999999994e-05 2024-09-16T03:39:58.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423390+0000", 2024-09-16T03:39:58.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19855242400000001 2024-09-16T03:39:58.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.930 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:58.930 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423411+0000", 2024-09-16T03:39:58.930 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0888000000000001e-05 2024-09-16T03:39:58.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625680+0000", 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20226923099999999 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625709+0000", 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9232000000000001e-05 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625738+0000", 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8235e-05 2024-09-16T03:39:58.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:51 4.6 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 2201684] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 166.76984111499999, 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50490003699999997, 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.932 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 51 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120918+0000", 2024-09-16T03:39:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8699999999999995e-07 2024-09-16T03:39:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120920+0000", 2024-09-16T03:39:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0820000000000001e-06 2024-09-16T03:39:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120924+0000", 2024-09-16T03:39:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0339999999999998e-06 2024-09-16T03:39:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120927+0000", 2024-09-16T03:39:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5809999999999999e-06 2024-09-16T03:39:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225783+0000", 2024-09-16T03:39:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10485588699999999 2024-09-16T03:39:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225832+0000", 2024-09-16T03:39:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9750000000000003e-05 2024-09-16T03:39:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.225910+0000", 2024-09-16T03:39:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7084999999999998e-05 2024-09-16T03:39:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423414+0000", 2024-09-16T03:39:58.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19750438200000001 2024-09-16T03:39:58.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423456+0000", 2024-09-16T03:39:58.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2386999999999999e-05 2024-09-16T03:39:58.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.937 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.937 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625766+0000", 2024-09-16T03:39:58.937 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20231011300000001 2024-09-16T03:39:58.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625790+0000", 2024-09-16T03:39:58.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3672999999999998e-05 2024-09-16T03:39:58.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.938 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.938 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625818+0000", 2024-09-16T03:39:58.938 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7576000000000002e-05 2024-09-16T03:39:58.938 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.938 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.938 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:52 4.6 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 166.76982891899999, 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50502012900000004, 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 52 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120930+0000", 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:39:58.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120932+0000", 2024-09-16T03:39:58.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.181e-06 2024-09-16T03:39:58.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120935+0000", 2024-09-16T03:39:58.941 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7690000000000001e-06 2024-09-16T03:39:58.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.941 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.941 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.120938+0000", 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6639999999999998e-06 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423188+0000", 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30224989200000002 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.423318+0000", 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013045999999999999 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.942 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625844+0000", 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20252547600000001 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625895+0000", 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1242000000000002e-05 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.625950+0000", 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5164999999999998e-05 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:279 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 163.69387713200001, 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50446570800000001, 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 279 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.944 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196882+0000", 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196883+0000", 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.48e-06 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.945 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196889+0000", 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6239999999999999e-06 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196902+0000", 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2804e-05 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196909+0000", 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9379999999999992e-06 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196963+0000", 2024-09-16T03:39:58.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3437999999999999e-05 2024-09-16T03:39:58.947 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.947 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.947 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op must wait for map", 2024-09-16T03:39:58.947 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.196969+0000", 2024-09-16T03:39:58.947 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1709999999999999e-06 2024-09-16T03:39:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302538+0000", 2024-09-16T03:39:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10556889899999999 2024-09-16T03:39:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.302659+0000", 2024-09-16T03:39:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121441 2024-09-16T03:39:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.948 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:39:58.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500040+0000", 2024-09-16T03:39:58.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19738082900000001 2024-09-16T03:39:58.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:58.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.500589+0000", 2024-09-16T03:39:58.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00054827000000000001 2024-09-16T03:39:58.949 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.949 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.949 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.949 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701118+0000", 2024-09-16T03:39:58.949 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20052917100000001 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701205+0000", 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7120999999999995e-05 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701289+0000", 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4457999999999997e-05 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701347+0000", 2024-09-16T03:39:58.950 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8063999999999998e-05 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:306 4.6 4:6ece7830:::smithi15736696-7:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 162.37787481699999, 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40927214699999998, 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.951 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 306 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512884+0000", 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512885+0000", 2024-09-16T03:39:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.277e-06 2024-09-16T03:39:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512890+0000", 2024-09-16T03:39:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6689999999999997e-06 2024-09-16T03:39:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512900+0000", 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0393e-05 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512906+0000", 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4569999999999996e-06 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.512951+0000", 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4891000000000001e-05 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.513000+0000", 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9187999999999997e-05 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.720758+0000", 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20775775699999999 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.721960+0000", 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001202107 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922010+0000", 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20004993900000001 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922055+0000", 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4981000000000002e-05 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922103+0000", 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8097999999999999e-05 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.922156+0000", 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3390000000000002e-05 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:419 4.6 4:64ce87bc:::smithi15736696-20:head [assert-version v9,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 157.977075203, 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40437787200000003, 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 419 2024-09-16T03:39:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:39:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913684+0000", 2024-09-16T03:39:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913687+0000", 2024-09-16T03:39:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3299999999999999e-06 2024-09-16T03:39:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913700+0000", 2024-09-16T03:39:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2830000000000001e-05 2024-09-16T03:39:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913733+0000", 2024-09-16T03:39:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2815999999999999e-05 2024-09-16T03:39:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913750+0000", 2024-09-16T03:39:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7073000000000001e-05 2024-09-16T03:39:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913920+0000", 2024-09-16T03:39:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000170015 2024-09-16T03:39:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.913981+0000", 2024-09-16T03:39:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1631999999999995e-05 2024-09-16T03:39:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:58.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216580+0000", 2024-09-16T03:39:58.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30259888200000001 2024-09-16T03:39:58.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:58.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.217205+0000", 2024-09-16T03:39:58.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00062461099999999996 2024-09-16T03:39:58.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317915+0000", 2024-09-16T03:39:58.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10071053000000001 2024-09-16T03:39:58.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.317996+0000", 2024-09-16T03:39:58.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0440999999999998e-05 2024-09-16T03:39:58.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318032+0000", 2024-09-16T03:39:58.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5944e-05 2024-09-16T03:39:58.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.318061+0000", 2024-09-16T03:39:58.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9768e-05 2024-09-16T03:39:58.964 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.964 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.964 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.964 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:479 4.8 4:1f63731d:::smithi15736696-30:head [assert-version v12,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:58.964 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:39:58.964 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 157.16921905699999, 2024-09-16T03:39:58.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.504011706, 2024-09-16T03:39:58.964 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.965 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.965 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.965 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.965 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.965 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.965 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 479 2024-09-16T03:39:58.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.965 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:39:58.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721540+0000", 2024-09-16T03:39:58.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.966 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721542+0000", 2024-09-16T03:39:58.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3199999999999998e-06 2024-09-16T03:39:58.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.966 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721553+0000", 2024-09-16T03:39:58.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1413000000000001e-05 2024-09-16T03:39:58.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721570+0000", 2024-09-16T03:39:58.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.685e-05 2024-09-16T03:39:58.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.721596+0000", 2024-09-16T03:39:58.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5647999999999999e-05 2024-09-16T03:39:58.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.921950+0000", 2024-09-16T03:39:58.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20035402899999999 2024-09-16T03:39:58.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.922009+0000", 2024-09-16T03:39:58.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9046999999999999e-05 2024-09-16T03:39:58.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:39:58.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.023850+0000", 2024-09-16T03:39:58.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10184130199999999 2024-09-16T03:39:58.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:58.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.024503+0000", 2024-09-16T03:39:58.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065257399999999995 2024-09-16T03:39:58.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.125105+0000", 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100602046 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225461+0000", 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100355835 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225512+0000", 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0862999999999997e-05 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:22.225551+0000", 2024-09-16T03:39:58.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9779e-05 2024-09-16T03:39:58.971 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.971 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.971 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.971 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:996 4.8 4:148e627d:::smithi15736696-23:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:39:58.971 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:39:58.971 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 147.84341615, 2024-09-16T03:39:58.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.73588397999999999, 2024-09-16T03:39:58.971 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.971 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.972 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:58.972 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.972 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.972 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.972 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 996 2024-09-16T03:39:58.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.972 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:39:58.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047343+0000", 2024-09-16T03:39:58.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047344+0000", 2024-09-16T03:39:58.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.846e-06 2024-09-16T03:39:58.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047351+0000", 2024-09-16T03:39:58.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.973e-06 2024-09-16T03:39:58.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047365+0000", 2024-09-16T03:39:58.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3556e-05 2024-09-16T03:39:58.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047382+0000", 2024-09-16T03:39:58.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6994e-05 2024-09-16T03:39:58.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047453+0000", 2024-09-16T03:39:58.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1458000000000006e-05 2024-09-16T03:39:58.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:58.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.047465+0000", 2024-09-16T03:39:58.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1119e-05 2024-09-16T03:39:58.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.975 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449411+0000", 2024-09-16T03:39:58.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40194680399999999 2024-09-16T03:39:58.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.449535+0000", 2024-09-16T03:39:58.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000124043 2024-09-16T03:39:58.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.783227+0000", 2024-09-16T03:39:58.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.333691187 2024-09-16T03:39:58.976 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.976 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.977 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.977 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1048 4.16 4:6df3bbbe:::smithi15736696-13:head [write 1622009~726683] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:58.977 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:58.977 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 147.314972033, 2024-09-16T03:39:58.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40682847599999999, 2024-09-16T03:39:58.977 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.977 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.977 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.977 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.978 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.978 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.978 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1048 2024-09-16T03:39:58.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.978 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:58.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:58.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.575787+0000", 2024-09-16T03:39:58.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.06e-07 2024-09-16T03:39:58.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576941+0000", 2024-09-16T03:39:58.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011532230000000001 2024-09-16T03:39:58.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576949+0000", 2024-09-16T03:39:58.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1089999999999998e-06 2024-09-16T03:39:58.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576956+0000", 2024-09-16T03:39:58.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000001e-06 2024-09-16T03:39:58.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878710+0000", 2024-09-16T03:39:58.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30175434299999998 2024-09-16T03:39:58.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878765+0000", 2024-09-16T03:39:58.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4988000000000003e-05 2024-09-16T03:39:58.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.878852+0000", 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7201999999999996e-05 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881535+0000", 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026828310000000001 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883968+0000", 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024330659999999998 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982541+0000", 2024-09-16T03:39:58.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.098572876000000004 2024-09-16T03:39:58.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982580+0000", 2024-09-16T03:39:58.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8714000000000002e-05 2024-09-16T03:39:58.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982615+0000", 2024-09-16T03:39:58.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5438000000000002e-05 2024-09-16T03:39:58.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.984 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.984 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1049 4.16 4:6df3bbbe:::smithi15736696-13:head [write 3121996~104915] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:58.984 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:39:58.984 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 147.31379388400001, 2024-09-16T03:39:58.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40579890600000001, 2024-09-16T03:39:58.984 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.984 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.984 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.984 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1049 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576965+0000", 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.576966+0000", 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8199999999999999e-07 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577069+0000", 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103583 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577076+0000", 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8190000000000003e-06 2024-09-16T03:39:58.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577082+0000", 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5130000000000004e-06 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880130+0000", 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.303048127 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880183+0000", 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3127999999999997e-05 2024-09-16T03:39:58.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:58.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.880282+0000", 2024-09-16T03:39:58.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.8881000000000004e-05 2024-09-16T03:39:58.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:58.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883101+0000", 2024-09-16T03:39:58.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0028194180000000002 2024-09-16T03:39:58.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.979716+0000", 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.096614899000000004 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982707+0000", 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0029914249999999998 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982739+0000", 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1250000000000001e-05 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.982764+0000", 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5080999999999999e-05 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1050 4.16 4:6df3bbbe:::smithi15736696-13:head [setxattr _header (56) in=7b,truncate 3226911] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 147.31366928899999, 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40784258099999998, 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.990 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1050 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577089+0000", 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577090+0000", 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9800000000000003e-07 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577094+0000", 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4340000000000001e-06 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577099+0000", 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8599999999999998e-06 2024-09-16T03:39:58.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577104+0000", 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0010000000000001e-06 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881242+0000", 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30413779000000002 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881290+0000", 2024-09-16T03:39:58.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7298e-05 2024-09-16T03:39:58.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:58.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.881367+0000", 2024-09-16T03:39:58.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7750000000000006e-05 2024-09-16T03:39:58.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:58.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.883117+0000", 2024-09-16T03:39:58.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001750096 2024-09-16T03:39:58.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.980218+0000", 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097100724999999999 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984859+0000", 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0046412950000000001 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984900+0000", 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0300999999999999e-05 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:58.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984932+0000", 2024-09-16T03:39:58.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2332999999999997e-05 2024-09-16T03:39:58.996 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.996 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:58.996 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:58.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.996 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1051 4.16 4:6df3bbbe:::smithi15736696-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T03:39:58.996 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:58.996 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 147.31364700899999, 2024-09-16T03:39:58.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50807376999999998, 2024-09-16T03:39:58.996 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:58.997 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:58.997 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:39:58.997 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:58.997 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:58.997 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:58.997 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1051 2024-09-16T03:39:58.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.997 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:58.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:58.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:58.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:58.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:58.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:58.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:58.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577112+0000", 2024-09-16T03:39:58.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5000000000000003e-07 2024-09-16T03:39:58.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:58.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577115+0000", 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3980000000000002e-06 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577120+0000", 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5589999999999996e-06 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.577125+0000", 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9910000000000002e-06 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:58.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882780+0000", 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30565438499999997 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for rw locks", 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.882825+0000", 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5739999999999999e-05 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085039+0000", 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20221376499999999 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085096+0000", 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6666000000000003e-05 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085186+0000", 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9716000000000006e-05 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1359 4.6 4:64ce87bc:::smithi15736696-20:head [write 1768928~405354] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T03:39:59.001 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 138.72462387100001, 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40239613499999999, 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1359 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166135+0000", 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166136+0000", 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166923+0000", 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00078683500000000005 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166933+0000", 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0445000000000001e-05 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.166938+0000", 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0119999999999996e-06 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264901+0000", 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097962831 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.264963+0000", 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1812999999999999e-05 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 6,10", 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.265085+0000", 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000121914 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.269430+0000", 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0043455000000000004 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.372259+0000", 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102828645 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568289+0000", 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19603034999999999 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568475+0000", 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018613599999999999 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:40.568531+0000", 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5729000000000001e-05 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:1676 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v29,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:39:59.007 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:39:59.008 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 128.760795008, 2024-09-16T03:39:59.008 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50424139300000004, 2024-09-16T03:39:59.008 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.008 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.008 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.008 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:59.008 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:59.008 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:59.008 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 1676 2024-09-16T03:39:59.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.008 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.008 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:39:59.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129964+0000", 2024-09-16T03:39:59.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.009 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.009 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129968+0000", 2024-09-16T03:39:59.009 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8430000000000003e-06 2024-09-16T03:39:59.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.129982+0000", 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4514e-05 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130013+0000", 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1081999999999998e-05 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130029+0000", 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5849999999999999e-05 2024-09-16T03:39:59.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130169+0000", 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000139973 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.130240+0000", 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0638999999999999e-05 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 0,1", 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.532892+0000", 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402652654 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.011 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533546+0000", 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00065390700000000001 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.533943+0000", 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00039703999999999998 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634089+0000", 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100146098 2024-09-16T03:39:59.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634153+0000", 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3581e-05 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.634205+0000", 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2212000000000001e-05 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:77 3.2 e73/68 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head v 73'117, mlcod=66'115)", 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 118.71175673, 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.401393467, 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179002+0000", 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179005+0000", 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.553e-06 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179029+0000", 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4039000000000001e-05 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.179046+0000", 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7139999999999999e-05 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.279144+0000", 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100098245 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379323+0000", 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100178621 2024-09-16T03:39:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.379348+0000", 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.5315000000000001e-05 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580315+0000", 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20096734599999999 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:00.580395+0000", 2024-09-16T03:39:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0208000000000002e-05 2024-09-16T03:39:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.018 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:59.020 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:39:59.020 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:59.018+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:59.119 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:39:59.118+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:39:59.164 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:59.164 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:59.164 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:59.173 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:39:59.256 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:59.255+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:39:59.314 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:59.314 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:59.315 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:59.315 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:59.315 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:59.321 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:39:59.319+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:39:59.323 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:39:59.392 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:39:59.422 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:59.421+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:59.469 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:59.470 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:59.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:59.470 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:59.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.470 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.12.21:3 3.0 3.f934d18 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:59.470 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:39:59.470 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 164.06766115100001, 2024-09-16T03:39:59.470 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78381636700000001, 2024-09-16T03:39:59.470 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.470 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.470 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:59.470 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:39:59.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399427+0000", 2024-09-16T03:39:59.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.471 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.471 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399429+0000", 2024-09-16T03:39:59.471 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3039999999999999e-06 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399434+0000", 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.1050000000000003e-06 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399442+0000", 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0160000000000008e-06 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.472 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399448+0000", 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.879e-06 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399489+0000", 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0815999999999997e-05 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.399495+0000", 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.3439999999999997e-06 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.473 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183226+0000", 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78373077400000002 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183244+0000", 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8128999999999999e-05 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:5 3.0 3.aefbbec0 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:59.474 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:39:59.475 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 163.96789368899999, 2024-09-16T03:39:59.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68447923499999996, 2024-09-16T03:39:59.475 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.475 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.475 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:59.475 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.475 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:39:59.475 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.475 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499195+0000", 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499196+0000", 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3480000000000001e-06 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499202+0000", 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4219999999999999e-06 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499212+0000", 2024-09-16T03:39:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0618000000000001e-05 2024-09-16T03:39:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499218+0000", 2024-09-16T03:39:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.4920000000000001e-06 2024-09-16T03:39:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499283+0000", 2024-09-16T03:39:59.477 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4906000000000005e-05 2024-09-16T03:39:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499293+0000", 2024-09-16T03:39:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0513000000000001e-05 2024-09-16T03:39:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183665+0000", 2024-09-16T03:39:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.68437196600000005 2024-09-16T03:39:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.478 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183674+0000", 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9700000000000005e-06 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:6 3.0 3.3bcda120 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 163.96783307999999, 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78474037699999999, 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.479 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.480 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499256+0000", 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0500000000000002e-07 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499260+0000", 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6669999999999998e-06 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499266+0000", 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499274+0000", 2024-09-16T03:39:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2250000000000003e-06 2024-09-16T03:39:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499316+0000", 2024-09-16T03:39:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2670000000000003e-05 2024-09-16T03:39:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.499323+0000", 2024-09-16T03:39:59.482 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.37e-06 2024-09-16T03:39:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283987+0000", 2024-09-16T03:39:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.78466433700000005 2024-09-16T03:39:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.283996+0000", 2024-09-16T03:39:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8650000000000007e-06 2024-09-16T03:39:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.483 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:15 3.0 3.7ec93288 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:39:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 163.85864107, 2024-09-16T03:39:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57518172599999995, 2024-09-16T03:39:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.484 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:39:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608448+0000", 2024-09-16T03:39:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608449+0000", 2024-09-16T03:39:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.184e-06 2024-09-16T03:39:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.485 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608459+0000", 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0553999999999999e-05 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608470+0000", 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0524e-05 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608493+0000", 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2980999999999999e-05 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608537+0000", 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3850000000000002e-05 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.608542+0000", 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183623+0000", 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.575080649 2024-09-16T03:39:59.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183629+0000", 2024-09-16T03:39:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6950000000000003e-06 2024-09-16T03:39:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:11 3.0 3.77005160 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e41)", 2024-09-16T03:39:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:39:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 163.76514076000001, 2024-09-16T03:39:59.488 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.48153733399999998, 2024-09-16T03:39:59.489 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.489 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.489 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:59.489 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:39:59.489 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.489 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.489 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701948+0000", 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701949+0000", 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.314e-06 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701954+0000", 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2889999999999997e-06 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.490 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701964+0000", 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9550000000000004e-06 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.701986+0000", 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1931999999999999e-05 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802129+0000", 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100142986 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.491 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:59.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:15.802135+0000", 2024-09-16T03:39:59.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6350000000000003e-06 2024-09-16T03:39:59.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183475+0000", 2024-09-16T03:39:59.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38133958099999998 2024-09-16T03:39:59.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.492 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.492 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.183485+0000", 2024-09-16T03:39:59.492 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0642000000000001e-05 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.5.21:6 3.0 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 162.48423684700001, 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60469085300000003, 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982852+0000", 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982853+0000", 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.393e-06 2024-09-16T03:39:59.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982858+0000", 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.062e-06 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982867+0000", 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7970000000000006e-06 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982879+0000", 2024-09-16T03:39:59.495 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2241e-05 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982996+0000", 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116996 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983007+0000", 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0694e-05 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284499+0000", 2024-09-16T03:39:59.496 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30149171400000002 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284690+0000", 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000191032 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.284908+0000", 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00021827399999999999 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.497 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487092+0000", 2024-09-16T03:39:59.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.202184156 2024-09-16T03:39:59.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.487110+0000", 2024-09-16T03:39:59.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7856999999999999e-05 2024-09-16T03:39:59.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.498 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.498 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587335+0000", 2024-09-16T03:39:59.498 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100224802 2024-09-16T03:39:59.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587410+0000", 2024-09-16T03:39:59.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5343e-05 2024-09-16T03:39:59.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.499 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.499 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.587543+0000", 2024-09-16T03:39:59.499 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000132492 2024-09-16T03:39:59.499 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.499 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.499 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.500 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:2 3.0 3:16550cfd:::smithi15736696-19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:59.500 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:39:59.500 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 162.484164655, 2024-09-16T03:39:59.500 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40289696000000003, 2024-09-16T03:39:59.500 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.500 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.500 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:59.500 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.500 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982924+0000", 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982925+0000", 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2900000000000003e-07 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982928+0000", 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4130000000000002e-06 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982936+0000", 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.875e-06 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982942+0000", 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1510000000000001e-06 2024-09-16T03:39:59.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983093+0000", 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015144400000000001 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983098+0000", 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4909999999999999e-06 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385695+0000", 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402597539 2024-09-16T03:39:59.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.504 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.504 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.385821+0000", 2024-09-16T03:39:59.504 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000125418 2024-09-16T03:39:59.504 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.504 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.504 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.504 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:9 3.0 3:2c9b9e7c:::smithi15736696-29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-16T03:39:59.504 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:39:59.504 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 162.48414846599999, 2024-09-16T03:39:59.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60615655800000001, 2024-09-16T03:39:59.505 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.505 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.505 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.505 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:39:59.505 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.505 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.505 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982940+0000", 2024-09-16T03:39:59.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982941+0000", 2024-09-16T03:39:59.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0860000000000001e-06 2024-09-16T03:39:59.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.506 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.506 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982946+0000", 2024-09-16T03:39:59.506 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7530000000000001e-06 2024-09-16T03:39:59.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982954+0000", 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.8520000000000004e-06 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.982958+0000", 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4870000000000004e-06 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983116+0000", 2024-09-16T03:39:59.507 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00015774699999999999 2024-09-16T03:39:59.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.508 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:59.508 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983120+0000", 2024-09-16T03:39:59.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7100000000000001e-06 2024-09-16T03:39:59.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.508 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.508 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486023+0000", 2024-09-16T03:39:59.508 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50290281400000003 2024-09-16T03:39:59.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.508 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486423+0000", 2024-09-16T03:39:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00040079299999999998 2024-09-16T03:39:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.486707+0000", 2024-09-16T03:39:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00028336600000000001 2024-09-16T03:39:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588861+0000", 2024-09-16T03:39:59.509 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.102154756 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588952+0000", 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0630000000000005e-05 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.588984+0000", 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2116999999999998e-05 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589022+0000", 2024-09-16T03:39:59.510 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7910000000000001e-05 2024-09-16T03:39:59.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.511 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.511 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589097+0000", 2024-09-16T03:39:59.511 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4536999999999995e-05 2024-09-16T03:39:59.511 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.511 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.511 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.511 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:12 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:39:59.511 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 162.484073118, 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60616102900000002, 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983015+0000", 2024-09-16T03:39:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983017+0000", 2024-09-16T03:39:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.483e-06 2024-09-16T03:39:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983022+0000", 2024-09-16T03:39:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7330000000000003e-06 2024-09-16T03:39:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983032+0000", 2024-09-16T03:39:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0036e-05 2024-09-16T03:39:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983038+0000", 2024-09-16T03:39:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6800000000000004e-06 2024-09-16T03:39:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983155+0000", 2024-09-16T03:39:59.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000116401 2024-09-16T03:39:59.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.515 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:59.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:16.983163+0000", 2024-09-16T03:39:59.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6219999999999998e-06 2024-09-16T03:39:59.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.515 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.515 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589120+0000", 2024-09-16T03:39:59.515 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.60595653199999999 2024-09-16T03:39:59.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.516 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.516 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:17.589176+0000", 2024-09-16T03:39:59.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6542000000000002e-05 2024-09-16T03:39:59.516 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.516 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.516 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.516 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.15.23:30 3.0 3:3fc1f531:::smithi15736696-16 low_tier:head [call cas.chunk_get_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:39:59.516 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:39:59.516 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 158.25075461200001, 2024-09-16T03:39:59.516 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40312788999999999, 2024-09-16T03:39:59.516 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.517 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.517 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.517 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:39:59.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.517 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.517 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216334+0000", 2024-09-16T03:39:59.517 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216338+0000", 2024-09-16T03:39:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0799999999999999e-06 2024-09-16T03:39:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216354+0000", 2024-09-16T03:39:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6439e-05 2024-09-16T03:39:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.518 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216392+0000", 2024-09-16T03:39:59.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7162000000000001e-05 2024-09-16T03:39:59.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.216444+0000", 2024-09-16T03:39:59.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2082e-05 2024-09-16T03:39:59.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.519 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.519 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417447+0000", 2024-09-16T03:39:59.519 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20100358400000001 2024-09-16T03:39:59.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417583+0000", 2024-09-16T03:39:59.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000135537 2024-09-16T03:39:59.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:59.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.417777+0000", 2024-09-16T03:39:59.520 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00019431200000000001 2024-09-16T03:39:59.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.520 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.520 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.520 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418479+0000", 2024-09-16T03:39:59.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00070148499999999996 2024-09-16T03:39:59.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.418755+0000", 2024-09-16T03:39:59.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00027627100000000003 2024-09-16T03:39:59.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.521 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.521 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619285+0000", 2024-09-16T03:39:59.521 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200530138 2024-09-16T03:39:59.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619370+0000", 2024-09-16T03:39:59.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4567999999999994e-05 2024-09-16T03:39:59.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.522 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.522 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.619462+0000", 2024-09-16T03:39:59.522 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2231999999999994e-05 2024-09-16T03:39:59.522 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.522 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.522 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:87 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 128.663103831, 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.402368742, 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803985+0000", 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.803986+0000", 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9400000000000004e-07 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804061+0000", 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5230999999999999e-05 2024-09-16T03:39:59.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804069+0000", 2024-09-16T03:39:59.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.7530000000000001e-06 2024-09-16T03:39:59.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.525 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.525 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.804073+0000", 2024-09-16T03:39:59.525 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9660000000000004e-06 2024-09-16T03:39:59.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907236+0000", 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103162566 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907324+0000", 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.7894999999999998e-05 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.907474+0000", 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014983999999999999 2024-09-16T03:39:59.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.908330+0000", 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00085600100000000001 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.107220+0000", 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.198889812 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206269+0000", 2024-09-16T03:39:59.527 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099049611999999995 2024-09-16T03:39:59.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.528 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206305+0000", 2024-09-16T03:39:59.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5679000000000002e-05 2024-09-16T03:39:59.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.528 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.528 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.206353+0000", 2024-09-16T03:39:59.528 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8593e-05 2024-09-16T03:39:59.528 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.528 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.529 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.529 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:3086 3.0 3:1e42685c:::6a9007270aa32b63b7f7e5355768bf791fa2d1832cb8adf59666d35d1b037cad:head [write 0~1679360] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:59.529 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:39:59.529 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 84.055266304, 2024-09-16T03:39:59.529 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.57631983099999995, 2024-09-16T03:39:59.529 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.529 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.529 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.530 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:59.530 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:59.530 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:59.530 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 3086 2024-09-16T03:39:59.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.530 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.530 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.530 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:39:59.530 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411822+0000", 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.411824+0000", 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.849e-06 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414291+0000", 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0024666900000000001 2024-09-16T03:39:59.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414304+0000", 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2886e-05 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.414310+0000", 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4219999999999997e-06 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783484+0000", 2024-09-16T03:39:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.36917337700000002 2024-09-16T03:39:59.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783601+0000", 2024-09-16T03:39:59.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011796899999999999 2024-09-16T03:39:59.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:59.533 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.783711+0000", 2024-09-16T03:39:59.533 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000110012 2024-09-16T03:39:59.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.533 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887652+0000", 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.103940373 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887669+0000", 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6864e-05 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942330+0000", 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.054661333999999999 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987927+0000", 2024-09-16T03:39:59.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045596722999999999 2024-09-16T03:39:59.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987993+0000", 2024-09-16T03:39:59.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6141000000000003e-05 2024-09-16T03:39:59.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.535 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.535 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.988142+0000", 2024-09-16T03:39:59.535 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00014919099999999999 2024-09-16T03:39:59.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.536 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.536 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:211 3.0 3:09d857e3:::85925bf7ff4b61513b387a03c5bd27d81ce6bd960cfaf104686b99338ce39220:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:59.536 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:39:59.536 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.829087195, 2024-09-16T03:39:59.536 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.55082720799999996, 2024-09-16T03:39:59.536 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.536 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.536 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.536 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638001+0000", 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638003+0000", 2024-09-16T03:39:59.537 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8250000000000001e-06 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638010+0000", 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4130000000000003e-06 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638022+0000", 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2016e-05 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.538 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638028+0000", 2024-09-16T03:39:59.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0390000000000001e-06 2024-09-16T03:39:59.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886321+0000", 2024-09-16T03:39:59.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.248293081 2024-09-16T03:39:59.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.539 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.539 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886428+0000", 2024-09-16T03:39:59.539 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000106791 2024-09-16T03:39:59.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.886630+0000", 2024-09-16T03:39:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00020280499999999999 2024-09-16T03:39:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889701+0000", 2024-09-16T03:39:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030706650000000002 2024-09-16T03:39:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942392+0000", 2024-09-16T03:39:59.540 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.052691156000000003 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.987675+0000", 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.045283229000000001 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188710+0000", 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20103479899999999 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.541 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188753+0000", 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2762000000000002e-05 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188829+0000", 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.5626999999999999e-05 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:214 3.0 3:2fb3d035:::5dba7feb0009e61b5e6a50f88cc7ba99127287f7eeb210342a4b40e996304a6b:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.828973121000004, 2024-09-16T03:39:59.542 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.55085424699999996, 2024-09-16T03:39:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:39:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638115+0000", 2024-09-16T03:39:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638116+0000", 2024-09-16T03:39:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.1200000000000002e-07 2024-09-16T03:39:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638120+0000", 2024-09-16T03:39:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3400000000000002e-06 2024-09-16T03:39:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638126+0000", 2024-09-16T03:39:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2339999999999999e-06 2024-09-16T03:39:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.638130+0000", 2024-09-16T03:39:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6349999999999997e-06 2024-09-16T03:39:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887101+0000", 2024-09-16T03:39:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.248970462 2024-09-16T03:39:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887191+0000", 2024-09-16T03:39:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0087000000000007e-05 2024-09-16T03:39:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887375+0000", 2024-09-16T03:39:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00018401600000000001 2024-09-16T03:39:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889706+0000", 2024-09-16T03:39:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0023305869999999998 2024-09-16T03:39:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.942851+0000", 2024-09-16T03:39:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.053145489999999997 2024-09-16T03:39:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.088369+0000", 2024-09-16T03:39:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.14551832200000001 2024-09-16T03:39:59.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188856+0000", 2024-09-16T03:39:59.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100486244 2024-09-16T03:39:59.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.548 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.548 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188902+0000", 2024-09-16T03:39:59.548 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.6134999999999998e-05 2024-09-16T03:39:59.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.549 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.549 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188970+0000", 2024-09-16T03:39:59.549 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7983000000000003e-05 2024-09-16T03:39:59.549 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.549 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.549 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.550 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:219 3.0 3:21e0c4e8:::f1ac9f6b1092b2dbfccdb2f32bdf689fb55e405da82aefeec91d0d73cdce51e1:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:59.550 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:39:59.550 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.729166839000001, 2024-09-16T03:39:59.550 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.71756203500000004, 2024-09-16T03:39:59.550 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.550 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.550 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.550 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.550 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.550 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:39:59.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737922+0000", 2024-09-16T03:39:59.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.551 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.551 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737924+0000", 2024-09-16T03:39:59.551 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8500000000000001e-06 2024-09-16T03:39:59.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737931+0000", 2024-09-16T03:39:59.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9879999999999998e-06 2024-09-16T03:39:59.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.737942+0000", 2024-09-16T03:39:59.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1837e-05 2024-09-16T03:39:59.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.552 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.552 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.552 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.838013+0000", 2024-09-16T03:39:59.552 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100070438 2024-09-16T03:39:59.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887695+0000", 2024-09-16T03:39:59.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.049682624000000002 2024-09-16T03:39:59.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.553 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.553 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887831+0000", 2024-09-16T03:39:59.553 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013565499999999999 2024-09-16T03:39:59.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:59.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.887948+0000", 2024-09-16T03:39:59.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000117353 2024-09-16T03:39:59.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:35.889709+0000", 2024-09-16T03:39:59.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017605629999999999 2024-09-16T03:39:59.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.554 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.554 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.043145+0000", 2024-09-16T03:39:59.554 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.153436136 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.188664+0000", 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.14551857000000001 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455373+0000", 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.26670951199999998 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455413+0000", 2024-09-16T03:39:59.555 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9517999999999997e-05 2024-09-16T03:39:59.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.556 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.556 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455484+0000", 2024-09-16T03:39:59.556 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.0990999999999998e-05 2024-09-16T03:39:59.556 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.556 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.556 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.556 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(client.4628.0:3100 3.0 3:2db88641:::a63ac782fa4bce1e96397c8301f2734ce3d5850a2dc454a54c42fc5451a8d72d:head [write 0~1584128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T03:39:59.556 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:39:59.557 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 83.195586582000004, 2024-09-16T03:39:59.557 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38491842399999998, 2024-09-16T03:39:59.557 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.557 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.557 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.557 INFO:teuthology.orchestra.run.smithi112.stdout: "client_info": { 2024-09-16T03:39:59.557 INFO:teuthology.orchestra.run.smithi112.stdout: "client": "client.4628", 2024-09-16T03:39:59.557 INFO:teuthology.orchestra.run.smithi112.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:39:59.557 INFO:teuthology.orchestra.run.smithi112.stdout: "tid": 3100 2024-09-16T03:39:59.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.557 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:39:59.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271502+0000", 2024-09-16T03:39:59.558 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.558 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.558 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.271523+0000", 2024-09-16T03:39:59.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0649000000000002e-05 2024-09-16T03:39:59.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273460+0000", 2024-09-16T03:39:59.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0019373820000000001 2024-09-16T03:39:59.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.559 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.559 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273488+0000", 2024-09-16T03:39:59.559 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7537000000000001e-05 2024-09-16T03:39:59.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273503+0000", 2024-09-16T03:39:59.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5438999999999999e-05 2024-09-16T03:39:59.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273645+0000", 2024-09-16T03:39:59.560 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000142013 2024-09-16T03:39:59.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.560 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.560 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273820+0000", 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017546200000000001 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,3,4", 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.273943+0000", 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012269099999999999 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.276532+0000", 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025884010000000002 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.561 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.562 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.455527+0000", 2024-09-16T03:39:59.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.17899536199999999 2024-09-16T03:39:59.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.562 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.562 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.556259+0000", 2024-09-16T03:39:59.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100731821 2024-09-16T03:39:59.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.562 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.562 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656291+0000", 2024-09-16T03:39:59.562 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100032144 2024-09-16T03:39:59.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.563 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656330+0000", 2024-09-16T03:39:59.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.8600000000000003e-05 2024-09-16T03:39:59.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.563 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.563 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:36.656420+0000", 2024-09-16T03:39:59.563 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.0922999999999997e-05 2024-09-16T03:39:59.563 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.563 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.563 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.564 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.8.22:60 3.0 3:16d0c248:::a2b7d44d389d3ac8752762fa1d216869a8025d7e5ae15e8811d7e7f2543095f6:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e108)", 2024-09-16T03:39:59.564 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:39:59.564 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 78.111302301999999, 2024-09-16T03:39:59.564 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.50513552500000003, 2024-09-16T03:39:59.564 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.564 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.564 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.564 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.564 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.564 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:39:59.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355786+0000", 2024-09-16T03:39:59.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.565 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.565 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355787+0000", 2024-09-16T03:39:59.565 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8000000000000004e-07 2024-09-16T03:39:59.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355791+0000", 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.631e-06 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355797+0000", 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7379999999999999e-06 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.355802+0000", 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.322e-06 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.356898+0000", 2024-09-16T03:39:59.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010963710000000001 2024-09-16T03:39:59.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:59.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.356902+0000", 2024-09-16T03:39:59.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0110000000000002e-06 2024-09-16T03:39:59.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.567 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.567 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560272+0000", 2024-09-16T03:39:59.567 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20336954199999999 2024-09-16T03:39:59.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560376+0000", 2024-09-16T03:39:59.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000103885 2024-09-16T03:39:59.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.568 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:39:59.568 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.560508+0000", 2024-09-16T03:39:59.568 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013254599999999999 2024-09-16T03:39:59.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.569 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.569 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.662786+0000", 2024-09-16T03:39:59.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10227757699999999 2024-09-16T03:39:59.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.569 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.569 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860745+0000", 2024-09-16T03:39:59.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19795950700000001 2024-09-16T03:39:59.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.569 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.569 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860843+0000", 2024-09-16T03:39:59.569 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7829000000000001e-05 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860890+0000", 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7005999999999997e-05 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:41.860922+0000", 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1680000000000002e-05 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.571 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.3.22:52 3.0 3:26b8133e:::1dcedb6d744c92dd878f17d66ff60f4d5ee4b2ff8b48c7b420b9d154387bca01:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e165)", 2024-09-16T03:39:59.571 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:48.998036+0000", 2024-09-16T03:39:59.571 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.469052239, 2024-09-16T03:39:59.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.38193842099999997, 2024-09-16T03:39:59.571 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.571 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.571 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.571 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.571 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.571 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.998036+0000", 2024-09-16T03:39:59.571 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.998036+0000", 2024-09-16T03:39:59.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.572 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.572 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.998039+0000", 2024-09-16T03:39:59.572 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1890000000000001e-06 2024-09-16T03:39:59.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.998048+0000", 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.4199999999999996e-06 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.998063+0000", 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5218999999999999e-05 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.998072+0000", 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6379999999999993e-06 2024-09-16T03:39:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.998145+0000", 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3268000000000004e-05 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op must wait for map", 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:48.998150+0000", 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9010000000000002e-06 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.200232+0000", 2024-09-16T03:39:59.574 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.202081548 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.200348+0000", 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00011605899999999999 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.200475+0000", 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012718500000000001 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.575 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.203119+0000", 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0026447160000000001 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.203177+0000", 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7976000000000002e-05 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.379748+0000", 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.176570587 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.576 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.379849+0000", 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000101035 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.379975+0000", 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012568 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.3.22:64 3.0 3:19e265f1:::56ead6b7bcc6d5c4548c4aca6c8d0b8ad5a705bc262502855d6f6cd40deba8a8:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e165)", 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:49.075077+0000", 2024-09-16T03:39:59.577 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 10.392012011, 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.40506330499999998, 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.075077+0000", 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.075077+0000", 2024-09-16T03:39:59.578 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.075077+0000", 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9100000000000004e-07 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.075136+0000", 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8799999999999999e-05 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.075143+0000", 2024-09-16T03:39:59.579 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2490000000000004e-06 2024-09-16T03:39:59.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.075148+0000", 2024-09-16T03:39:59.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4610000000000001e-06 2024-09-16T03:39:59.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.580 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.075180+0000", 2024-09-16T03:39:59.580 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2453000000000003e-05 2024-09-16T03:39:59.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.580 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.075183+0000", 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0939999999999999e-06 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.300321+0000", 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.225138011 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.300416+0000", 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5056999999999995e-05 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for subops from 1,4", 2024-09-16T03:39:59.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.300579+0000", 2024-09-16T03:39:59.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00016242700000000001 2024-09-16T03:39:59.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.304771+0000", 2024-09-16T03:39:59.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0041918270000000004 2024-09-16T03:39:59.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.582 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "op_commit", 2024-09-16T03:39:59.582 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.304782+0000", 2024-09-16T03:39:59.582 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1100999999999999e-05 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.480072+0000", 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.17528986399999999 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.480108+0000", 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6835999999999998e-05 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.480140+0000", 2024-09-16T03:39:59.583 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.1534e-05 2024-09-16T03:39:59.584 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.584 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.584 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.584 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.584 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_op(osd.10.22:304 3.0 3.b3bca138 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:39:59.584 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:54.205879+0000", 2024-09-16T03:39:59.584 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 5.2612094000000003, 2024-09-16T03:39:59.584 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.59718373800000002, 2024-09-16T03:39:59.584 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.584 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.584 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:39:59.584 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.205879+0000", 2024-09-16T03:39:59.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.205879+0000", 2024-09-16T03:39:59.585 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.585 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.585 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.205881+0000", 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2409999999999998e-06 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.205888+0000", 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.161e-06 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.205901+0000", 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3344e-05 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.586 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.205908+0000", 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2579999999999998e-06 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.205963+0000", 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.503e-05 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "waiting for peered", 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.205971+0000", 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.9189999999999999e-06 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.587 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.803029+0000", 2024-09-16T03:39:59.588 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.59705819400000004 2024-09-16T03:39:59.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.588 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.588 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.803063+0000", 2024-09-16T03:39:59.588 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3590999999999997e-05 2024-09-16T03:39:59.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.588 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.588 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.588 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:59.589 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:59.576+0000 7eff7884b640 -1 osd.3 174 handle_osd_map injecting map crc failure 2024-09-16T03:39:59.590 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:39:59.591 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:39:59.532+0000 7f6528e68640 -1 osd.1 174 handle_osd_map injecting map crc failure 2024-09-16T03:39:59.591 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:39:59.588+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:39:59.690 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:39:59.689+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:39:59.734 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:59.734 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:39:59.734 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:59.742 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:39:59.791 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:39:59.789+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:39:59.844 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:59.844 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:39:59.844 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:39:59.844 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:39:59.844 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:39:59.852 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:39:59.877 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 9, 1] out_osds: [5, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:39:59.877 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:39:59.877 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:39:59.877 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T03:39:59.877 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T03:39:59.892 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:39:59.890+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:39:59.956 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:39:59.957 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:39:59.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:39:59.957 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:39:59.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.957 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:57 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'1, mlcod=0'0)", 2024-09-16T03:39:59.957 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:39:59.957 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 167.82982154600001, 2024-09-16T03:39:59.957 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20372090900000001, 2024-09-16T03:39:59.957 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.957 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.957 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.957 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125310+0000", 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.125313+0000", 2024-09-16T03:39:59.958 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7180000000000001e-06 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126730+0000", 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00141731 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126746+0000", 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6455000000000001e-05 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.959 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.126756+0000", 2024-09-16T03:39:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5990000000000008e-06 2024-09-16T03:39:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227068+0000", 2024-09-16T03:39:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100312308 2024-09-16T03:39:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.227085+0000", 2024-09-16T03:39:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6898e-05 2024-09-16T03:39:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328995+0000", 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101909903 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329031+0000", 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.5717999999999999e-05 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.961 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:58 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'2, mlcod=0'0)", 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 167.82648841599999, 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084524500000001, 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.962 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128643+0000", 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128644+0000", 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.251e-06 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128913+0000", 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00026856200000000002 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128925+0000", 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2109999999999999e-05 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.128933+0000", 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6260000000000003e-06 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328196+0000", 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19926303200000001 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328253+0000", 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7262999999999997e-05 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329458+0000", 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012051939999999999 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.965 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329488+0000", 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0207000000000001e-05 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:59 4.d e39/32 4:b3fe05b2:::smithi15736696-15:head v 39'3, mlcod=0'0)", 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 167.82599298299999, 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036954900000001, 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.966 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129138+0000", 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129139+0000", 2024-09-16T03:39:59.967 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.6099999999999999e-07 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129158+0000", 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8088e-05 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.129168+0000", 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.048e-05 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.968 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.229233+0000", 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100065168 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328815+0000", 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099581694999999998 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.328845+0000", 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0193e-05 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.969 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329492+0000", 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064671800000000001 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:12.329508+0000", 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6246e-05 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.0:15 4.d e47/32 4:b3fe05b2:::smithi15736696-15:head v 47'5, mlcod=39'3)", 2024-09-16T03:39:59.970 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 153.652698257, 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100920751, 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.971 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302433+0000", 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302435+0000", 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2340000000000001e-06 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302446+0000", 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0596e-05 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.972 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.302461+0000", 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5082000000000001e-05 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402592+0000", 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100130488 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402712+0000", 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00012013100000000001 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.402741+0000", 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9717999999999999e-05 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403309+0000", 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056725700000000005 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:26.403354+0000", 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5244999999999998e-05 2024-09-16T03:39:59.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 129.80519954799999, 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032593800000000001, 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.975 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:39:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149932+0000", 2024-09-16T03:39:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149943+0000", 2024-09-16T03:39:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0817e-05 2024-09-16T03:39:59.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151026+0000", 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010835350000000001 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151040+0000", 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3884e-05 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151049+0000", 2024-09-16T03:39:59.977 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3799999999999994e-06 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151107+0000", 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8199000000000003e-05 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151124+0000", 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7756000000000001e-05 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.978 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153142+0000", 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0020177569999999998 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.153191+0000", 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9051999999999997e-05 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1682 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'7, mlcod=40'4)", 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:39:59.979 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 129.701602358, 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10169502499999999, 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253529+0000", 2024-09-16T03:39:59.980 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.253540+0000", 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0888e-05 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254015+0000", 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00047485499999999999 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.981 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254030+0000", 2024-09-16T03:39:59.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4899000000000001e-05 2024-09-16T03:39:59.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254039+0000", 2024-09-16T03:39:59.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5000000000000005e-06 2024-09-16T03:39:59.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.982 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.982 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254107+0000", 2024-09-16T03:39:59.982 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7298000000000005e-05 2024-09-16T03:39:59.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254122+0000", 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5928999999999999e-05 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355184+0000", 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101061703 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355224+0000", 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9953000000000002e-05 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.983 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1683 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'8, mlcod=40'4)", 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 129.70106480999999, 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101630885, 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.984 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:39:59.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254067+0000", 2024-09-16T03:39:59.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.985 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.985 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254068+0000", 2024-09-16T03:39:59.985 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.5700000000000001e-07 2024-09-16T03:39:59.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254075+0000", 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.2710000000000003e-06 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254083+0000", 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.0220000000000004e-06 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.254089+0000", 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8699999999999997e-06 2024-09-16T03:39:59.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354968+0000", 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100879625 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.354992+0000", 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3238e-05 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355665+0000", 2024-09-16T03:39:59.987 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00067371100000000001 2024-09-16T03:39:59.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.988 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.988 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.355698+0000", 2024-09-16T03:39:59.988 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.2291e-05 2024-09-16T03:39:59.988 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.988 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.988 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.988 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.0:177 4.10 e65/57 4:0a55bbf2:::smithi15736696-18:head v 65'47, mlcod=0'0)", 2024-09-16T03:39:59.988 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:59.988 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 128.74811822199999, 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074461099999999995, 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:59.989 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207013+0000", 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5300000000000001e-07 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207018+0000", 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.7210000000000002e-06 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.990 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207031+0000", 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2767e-05 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207040+0000", 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.5400000000000001e-06 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207107+0000", 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.6625999999999995e-05 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207131+0000", 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4128e-05 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207712+0000", 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00058124700000000001 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.207758+0000", 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5429000000000002e-05 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1723 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'48, mlcod=53'46)", 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 128.30605104099999, 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00088284599999999996, 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649080+0000", 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649081+0000", 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8299999999999995e-07 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649101+0000", 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0126e-05 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649141+0000", 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.9609e-05 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649161+0000", 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9599999999999999e-05 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:39:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649299+0000", 2024-09-16T03:39:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00013822199999999999 2024-09-16T03:39:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:39:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649318+0000", 2024-09-16T03:39:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9273999999999999e-05 2024-09-16T03:39:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:39:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649919+0000", 2024-09-16T03:39:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00060074100000000001 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.649963+0000", 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4391000000000002e-05 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1724 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'49, mlcod=53'46)", 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 128.302386076, 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00064536300000000003, 2024-09-16T03:39:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:39:59.998 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:39:59.998 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:39:59.998 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:39:59.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:39:59.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:39:59.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.998 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:39:59.998 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652745+0000", 2024-09-16T03:39:59.998 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:39:59.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652747+0000", 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.291e-06 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652753+0000", 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.4450000000000001e-06 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652782+0000", 2024-09-16T03:39:59.999 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9094999999999998e-05 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652797+0000", 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5021999999999999e-05 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652854+0000", 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.6586999999999999e-05 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:00.000 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.652885+0000", 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.146e-05 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653350+0000", 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00046443599999999998 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653391+0000", 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1026999999999999e-05 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:00.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1725 4.10 e66/57 4:0a55bbf2:::smithi15736696-18:head v 66'50, mlcod=53'46)", 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 128.30159325299999, 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00056324100000000002, 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:40:00.002 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653538+0000", 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653539+0000", 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.9699999999999994e-07 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:00.003 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653545+0000", 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.9309999999999996e-06 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653557+0000", 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2340999999999999e-05 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653564+0000", 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2779999999999999e-06 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.004 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653605+0000", 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1301000000000003e-05 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.653621+0000", 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5951e-05 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654064+0000", 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000442843 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:00.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:00.006 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:00.006 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:51.654101+0000", 2024-09-16T03:40:00.006 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.7598999999999997e-05 2024-09-16T03:40:00.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:00.006 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:00.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:00.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:00.006 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:00.006 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:40:00.008 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:40:00.008 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:40:00.006+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:40:00.109 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:40:00.107+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:40:00.124 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:40:00.124 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:40:00.124 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.124 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDScrubReserve(4.23,RELEASE,e:175,from:14,reservation_nonce:0)", 2024-09-16T03:40:00.124 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:40:00.075671+0000", 2024-09-16T03:40:00.124 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 0.047497649000000003, 2024-09-16T03:40:00.124 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.047514256999999997, 2024-09-16T03:40:00.124 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.124 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.125 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "queued for pg", 2024-09-16T03:40:00.125 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:40:00.075671+0000", 2024-09-16T03:40:00.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.125 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.125 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:40:00.075671+0000", 2024-09-16T03:40:00.125 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.125 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.125 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:40:00.075671+0000", 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9400000000000005e-07 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:40:00.075676+0000", 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3259999999999997e-06 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:40:00.075689+0000", 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3113e-05 2024-09-16T03:40:00.126 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.127 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.127 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.127 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:40:00.075697+0000", 2024-09-16T03:40:00.127 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0989999999999999e-06 2024-09-16T03:40:00.127 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.127 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.127 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.127 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.127 INFO:teuthology.orchestra.run.smithi155.stdout: ], 2024-09-16T03:40:00.127 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 1 2024-09-16T03:40:00.127 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:40:00.133 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:40:00.178 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:00.179 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:40:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "epoch": 175, 2024-09-16T03:40:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "fsid": "b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2", 2024-09-16T03:40:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "created": "2024-09-16T03:36:37.310421+0000", 2024-09-16T03:40:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "modified": "2024-09-16T03:39:59.375257+0000", 2024-09-16T03:40:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "last_up_change": "2024-09-16T03:36:50.058111+0000", 2024-09-16T03:40:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "last_in_change": "2024-09-16T03:39:28.633923+0000", 2024-09-16T03:40:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T03:40:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_num": 5804032, 2024-09-16T03:40:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_set": [ 2024-09-16T03:40:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "nodeep-scrub", 2024-09-16T03:40:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "pglog_hardlimit", 2024-09-16T03:40:00.179 INFO:teuthology.orchestra.run.smithi064.stdout: "purged_snapdirs", 2024-09-16T03:40:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_deletes", 2024-09-16T03:40:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "sortbitwise" 2024-09-16T03:40:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:40:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_version": 7, 2024-09-16T03:40:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T03:40:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T03:40:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T03:40:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_snapshot": "", 2024-09-16T03:40:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_max": 4, 2024-09-16T03:40:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "max_osd": 16, 2024-09-16T03:40:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "require_min_compat_client": "luminous", 2024-09-16T03:40:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "min_compat_client": "luminous", 2024-09-16T03:40:00.180 INFO:teuthology.orchestra.run.smithi064.stdout: "require_osd_release": "squid", 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "allow_crimson": false, 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "pools": [ 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 1, 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": ".mgr", 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-09-16T03:36:49.696307+0000", 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:40:00.181 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "is_stretch_pool": false, 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 1, 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 1, 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 1, 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 1, 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 1, 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-09-16T03:40:00.182 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-09-16T03:40:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-09-16T03:40:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-09-16T03:40:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-09-16T03:40:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "25", 2024-09-16T03:40:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-09-16T03:40:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:40:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T03:40:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-09-16T03:40:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:40:00.183 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-09-16T03:40:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-09-16T03:40:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-09-16T03:40:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-09-16T03:40:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-09-16T03:40:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-09-16T03:40:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-09-16T03:40:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-09-16T03:40:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-09-16T03:40:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-09-16T03:40:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-09-16T03:40:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-09-16T03:40:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-09-16T03:40:00.184 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:40:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:40:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:40:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-09-16T03:40:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-09-16T03:40:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-09-16T03:40:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-09-16T03:40:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-09-16T03:40:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.185 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_max": 32, 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_min": 1 2024-09-16T03:40:00.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "mgr": {} 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 15, 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 15, 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.20000000298023224, 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 3, 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 3, 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89189434051513672, 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-09-16T03:40:00.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 2, 2024-09-16T03:40:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "rbd", 2024-09-16T03:40:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-09-16T03:36:51.904455+0000", 2024-09-16T03:40:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 8193, 2024-09-16T03:40:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T03:40:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-09-16T03:40:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-09-16T03:40:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-09-16T03:40:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-09-16T03:40:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:40:00.188 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "is_stretch_pool": false, 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 8, 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 8, 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 8, 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 8, 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 8, 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "28", 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 2, 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 28, 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-09-16T03:40:00.190 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-09-16T03:40:00.191 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-09-16T03:40:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-09-16T03:40:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-09-16T03:40:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-09-16T03:40:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-09-16T03:40:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-09-16T03:40:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-09-16T03:40:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:40:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:40:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:40:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:40:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-09-16T03:40:00.192 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "rbd": {} 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 3.7699999809265137, 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 3.7699999809265137, 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.73000001907348633, 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2.75, 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.75, 2024-09-16T03:40:00.193 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89189434051513672, 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 3, 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "low_tier", 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-09-16T03:37:02.999795+0000", 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-09-16T03:40:00.194 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "is_stretch_pool": false, 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 4, 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 4, 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 4, 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 4, 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 4, 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-09-16T03:40:00.195 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-09-16T03:40:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-09-16T03:40:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-09-16T03:40:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-09-16T03:40:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-09-16T03:40:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "33", 2024-09-16T03:40:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-09-16T03:40:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:40:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T03:40:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-09-16T03:40:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:40:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-09-16T03:40:00.196 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-09-16T03:40:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-09-16T03:40:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-09-16T03:40:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-09-16T03:40:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-09-16T03:40:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-09-16T03:40:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-09-16T03:40:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-09-16T03:40:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-09-16T03:40:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-09-16T03:40:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-09-16T03:40:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-09-16T03:40:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:40:00.197 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:40:00.198 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:40:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-09-16T03:40:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-09-16T03:40:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-09-16T03:40:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-09-16T03:40:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "options": {}, 2024-09-16T03:40:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": {}, 2024-09-16T03:40:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-09-16T03:40:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-09-16T03:40:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 3.7699999809265137, 2024-09-16T03:40:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 3.7699999809265137, 2024-09-16T03:40:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.52999997138977051, 2024-09-16T03:40:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-09-16T03:40:00.199 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2, 2024-09-16T03:40:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89189434051513672, 2024-09-16T03:40:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-16T03:40:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-09-16T03:40:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "pool": 4, 2024-09-16T03:40:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_name": "unique_pool_0", 2024-09-16T03:40:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "create_time": "2024-09-16T03:37:04.279293+0000", 2024-09-16T03:40:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "flags": 1, 2024-09-16T03:40:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "flags_names": "hashpspool", 2024-09-16T03:40:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "type": 1, 2024-09-16T03:40:00.200 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 3, 2024-09-16T03:40:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "min_size": 2, 2024-09-16T03:40:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_rule": 0, 2024-09-16T03:40:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_count": 0, 2024-09-16T03:40:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_target": 0, 2024-09-16T03:40:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T03:40:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T03:40:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "is_stretch_pool": false, 2024-09-16T03:40:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "object_hash": 2, 2024-09-16T03:40:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_autoscale_mode": "off", 2024-09-16T03:40:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num": 56, 2024-09-16T03:40:00.201 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num": 56, 2024-09-16T03:40:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_placement_num_target": 56, 2024-09-16T03:40:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_target": 56, 2024-09-16T03:40:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_num_pending": 56, 2024-09-16T03:40:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "last_pg_merge_meta": { 2024-09-16T03:40:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "source_pgid": "0.0", 2024-09-16T03:40:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "ready_epoch": 0, 2024-09-16T03:40:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_started": 0, 2024-09-16T03:40:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "last_epoch_clean": 0, 2024-09-16T03:40:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "source_version": "0'0", 2024-09-16T03:40:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "target_version": "0'0" 2024-09-16T03:40:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "last_change": "157", 2024-09-16T03:40:00.202 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend": "0", 2024-09-16T03:40:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T03:40:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "last_force_op_resend_preluminous": "126", 2024-09-16T03:40:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "auid": 0, 2024-09-16T03:40:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_mode": "selfmanaged", 2024-09-16T03:40:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_seq": 0, 2024-09-16T03:40:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "snap_epoch": 0, 2024-09-16T03:40:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "pool_snaps": [], 2024-09-16T03:40:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps": "[]", 2024-09-16T03:40:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_bytes": 0, 2024-09-16T03:40:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "quota_max_objects": 0, 2024-09-16T03:40:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "tiers": [], 2024-09-16T03:40:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "tier_of": -1, 2024-09-16T03:40:00.203 INFO:teuthology.orchestra.run.smithi064.stdout: "read_tier": -1, 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "write_tier": -1, 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_mode": "none", 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_bytes": 0, 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "target_max_objects": 0, 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_flush_age": 0, 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "cache_min_evict_age": 0, 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profile": "", 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_params": { 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "none" 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.204 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_period": 0, 2024-09-16T03:40:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_count": 0, 2024-09-16T03:40:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "use_gmt_hitset": true, 2024-09-16T03:40:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "min_read_recency_for_promote": 0, 2024-09-16T03:40:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "min_write_recency_for_promote": 0, 2024-09-16T03:40:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T03:40:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "hit_set_search_last_n": 0, 2024-09-16T03:40:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "grade_table": [], 2024-09-16T03:40:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "stripe_width": 0, 2024-09-16T03:40:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "expected_num_objects": 0, 2024-09-16T03:40:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "fast_read": false, 2024-09-16T03:40:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "options": { 2024-09-16T03:40:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "dedup_cdc_chunk_size": 131072, 2024-09-16T03:40:00.205 INFO:teuthology.orchestra.run.smithi064.stdout: "dedup_chunk_algorithm": "fastcdc", 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "dedup_tier": 3, 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "fingerprint_algorithm": "sha256" 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "application_metadata": { 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "rados": {} 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "read_balance": { 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "score_type": "Fair distribution", 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "score_acting": 2.1500000953674316, 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "score_stable": 2.4200000762939453, 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "optimal_score": 0.93000000715255737, 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_acting": 2, 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "raw_score_stable": 2.25, 2024-09-16T03:40:00.206 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity_weighted": 0.89189434051513672, 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity": 0.94999998807907104, 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "average_primary_affinity_weighted": 0.88999998569488525 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "bdbe961a-78d3-4f7d-88ed-ce24475d49e3", 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:40:00.207 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:40:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 171, 2024-09-16T03:40:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6813", 2024-09-16T03:40:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3924351635 2024-09-16T03:40:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.208 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6814", 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3924351635 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6816", 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3924351635 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6815", 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3924351635 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6813/3924351635", 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6814/3924351635", 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6816/3924351635", 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6815/3924351635", 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "154c8ddd-7a3d-402c-b9d6-28538f4e6bc4", 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.211 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:40:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0, 2024-09-16T03:40:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:40:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 160, 2024-09-16T03:40:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.212 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6804", 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 624977804 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6805", 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 624977804 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.213 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6807", 2024-09-16T03:40:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 624977804 2024-09-16T03:40:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6806", 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 624977804 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.112:6804/624977804", 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.112:6805/624977804", 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.112:6807/624977804", 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.112:6806/624977804", 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.215 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-09-16T03:40:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "2a06de4e-0835-4a42-8c8a-49fb55aab566", 2024-09-16T03:40:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:40:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:40:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.216 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 23, 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 159, 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6812", 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2804648086 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.217 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6813", 2024-09-16T03:40:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2804648086 2024-09-16T03:40:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.218 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6815", 2024-09-16T03:40:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2804648086 2024-09-16T03:40:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6814", 2024-09-16T03:40:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2804648086 2024-09-16T03:40:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.155:6812/2804648086", 2024-09-16T03:40:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.155:6813/2804648086", 2024-09-16T03:40:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.155:6815/2804648086", 2024-09-16T03:40:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.155:6814/2804648086", 2024-09-16T03:40:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-09-16T03:40:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "8a87f9f0-c380-4f38-bbd2-06b2def4fd2d", 2024-09-16T03:40:00.220 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.6943511962890625, 2024-09-16T03:40:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:40:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:40:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 173, 2024-09-16T03:40:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.221 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6800", 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1935597988 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6801", 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1935597988 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6803", 2024-09-16T03:40:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1935597988 2024-09-16T03:40:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6802", 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1935597988 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.157:6800/1935597988", 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.157:6801/1935597988", 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.157:6803/1935597988", 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.157:6802/1935597988", 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-09-16T03:40:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "5ac53318-12e8-496c-a87f-ba5de79c52aa", 2024-09-16T03:40:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.4737548828125, 2024-09-16T03:40:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.9752197265625, 2024-09-16T03:40:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:40:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 171, 2024-09-16T03:40:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.225 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6809", 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675475248 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.226 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6810", 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675475248 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6812", 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675475248 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.227 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6811", 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2675475248 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6809/2675475248", 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6810/2675475248", 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6812/2675475248", 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6811/2675475248", 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.228 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-09-16T03:40:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "675c81a4-2314-463e-82a7-0060735c78fd", 2024-09-16T03:40:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-09-16T03:40:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-16T03:40:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:40:00.229 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 21, 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 67, 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6800", 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 499905276 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6801", 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 499905276 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.231 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6803", 2024-09-16T03:40:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 499905276 2024-09-16T03:40:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.232 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6802", 2024-09-16T03:40:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 499905276 2024-09-16T03:40:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.112:6800/499905276", 2024-09-16T03:40:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.112:6801/499905276", 2024-09-16T03:40:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.112:6803/499905276", 2024-09-16T03:40:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.112:6802/499905276", 2024-09-16T03:40:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "b18962a7-3b23-4e50-bf31-1d0629557423", 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.95001220703125, 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 23, 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 169, 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.234 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6808", 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2839983406 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6809", 2024-09-16T03:40:00.235 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2839983406 2024-09-16T03:40:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6811", 2024-09-16T03:40:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2839983406 2024-09-16T03:40:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.236 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6810", 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2839983406 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.155:6808/2839983406", 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.155:6809/2839983406", 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.155:6811/2839983406", 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.155:6810/2839983406", 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.237 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "9706c1e8-dac0-4e16-8bc5-7cacc6f13870", 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:40:00.238 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 173, 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6808", 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3021982871 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6809", 2024-09-16T03:40:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3021982871 2024-09-16T03:40:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6811", 2024-09-16T03:40:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3021982871 2024-09-16T03:40:00.240 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6810", 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3021982871 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.157:6808/3021982871", 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.157:6809/3021982871", 2024-09-16T03:40:00.241 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.157:6811/3021982871", 2024-09-16T03:40:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.157:6810/3021982871", 2024-09-16T03:40:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 8, 2024-09-16T03:40:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "fafd06b8-c005-4471-a128-9dd51580c0f0", 2024-09-16T03:40:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.8023529052734375, 2024-09-16T03:40:00.242 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 170, 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6805", 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3609017627 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.243 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6806", 2024-09-16T03:40:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3609017627 2024-09-16T03:40:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6808", 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3609017627 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6807", 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3609017627 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.245 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6805/3609017627", 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6806/3609017627", 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6808/3609017627", 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6807/3609017627", 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 9, 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "79e38e57-52c2-416f-9b90-75621f06d3e3", 2024-09-16T03:40:00.246 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 170, 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.247 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6812", 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 701981815 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6813", 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 701981815 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.248 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6815", 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 701981815 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6814", 2024-09-16T03:40:00.249 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 701981815 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.112:6812/701981815", 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.112:6813/701981815", 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.112:6815/701981815", 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.112:6814/701981815", 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.250 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 10, 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10", 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 174, 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6804", 2024-09-16T03:40:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4138619523 2024-09-16T03:40:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.252 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6805", 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4138619523 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6807", 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4138619523 2024-09-16T03:40:00.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6806", 2024-09-16T03:40:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4138619523 2024-09-16T03:40:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.155:6804/4138619523", 2024-09-16T03:40:00.254 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.155:6805/4138619523", 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.155:6807/4138619523", 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.155:6806/4138619523", 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 11, 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "7a2a169a-ce94-4091-a792-27d0ecf9a7fa", 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:40:00.255 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 0.2950897216796875, 2024-09-16T03:40:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 23, 2024-09-16T03:40:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 170, 2024-09-16T03:40:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6804", 2024-09-16T03:40:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3607367937 2024-09-16T03:40:00.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6805", 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3607367937 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6807", 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3607367937 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6806", 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3607367937 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.258 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.157:6804/3607367937", 2024-09-16T03:40:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.157:6805/3607367937", 2024-09-16T03:40:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.157:6807/3607367937", 2024-09-16T03:40:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.157:6806/3607367937", 2024-09-16T03:40:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 12, 2024-09-16T03:40:00.259 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "7a280cda-b740-40c8-9fae-1a1ff11651f6", 2024-09-16T03:40:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:40:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:40:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 21, 2024-09-16T03:40:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 160, 2024-09-16T03:40:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6801", 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2984461266 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6802", 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2984461266 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6804", 2024-09-16T03:40:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2984461266 2024-09-16T03:40:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.64:6803", 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2984461266 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.64:6801/2984461266", 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.64:6802/2984461266", 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.64:6804/2984461266", 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.64:6803/2984461266", 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.263 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 13, 2024-09-16T03:40:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "ce3a3148-cd90-422c-a106-045de89ab045", 2024-09-16T03:40:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 0, 2024-09-16T03:40:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0, 2024-09-16T03:40:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:40:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 22, 2024-09-16T03:40:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 0, 2024-09-16T03:40:00.264 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6808", 2024-09-16T03:40:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1616314527 2024-09-16T03:40:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.265 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6809", 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1616314527 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6811", 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1616314527 2024-09-16T03:40:00.266 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.112:6810", 2024-09-16T03:40:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 1616314527 2024-09-16T03:40:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.112:6808/1616314527", 2024-09-16T03:40:00.267 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.112:6809/1616314527", 2024-09-16T03:40:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.112:6811/1616314527", 2024-09-16T03:40:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.112:6810/1616314527", 2024-09-16T03:40:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 14, 2024-09-16T03:40:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "461c2d79-f4e7-4b21-899c-50d4a6cf12f9", 2024-09-16T03:40:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.268 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 1, 2024-09-16T03:40:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:40:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 21, 2024-09-16T03:40:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 160, 2024-09-16T03:40:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6800", 2024-09-16T03:40:00.269 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4203868121 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6801", 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4203868121 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.270 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6803", 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4203868121 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.155:6802", 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 4203868121 2024-09-16T03:40:00.271 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.155:6800/4203868121", 2024-09-16T03:40:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.155:6801/4203868121", 2024-09-16T03:40:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.155:6803/4203868121", 2024-09-16T03:40:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.155:6802/4203868121", 2024-09-16T03:40:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 15, 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "uuid": "9a8c0bbe-9389-4b2c-95e1-34d3e66461e9", 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "up": 1, 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "in": 1, 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "weight": 0.80743408203125, 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_affinity": 1, 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_begin": 0, 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "last_clean_end": 0, 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "up_from": 23, 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "up_thru": 173, 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "down_at": 0, 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "lost_at": 0, 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addrs": { 2024-09-16T03:40:00.273 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6812", 2024-09-16T03:40:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 181354529 2024-09-16T03:40:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addrs": { 2024-09-16T03:40:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6813", 2024-09-16T03:40:00.274 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 181354529 2024-09-16T03:40:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addrs": { 2024-09-16T03:40:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6815", 2024-09-16T03:40:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 181354529 2024-09-16T03:40:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.275 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addrs": { 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "addrvec": [ 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "172.21.15.157:6814", 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 181354529 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "public_addr": "172.21.15.157:6812/181354529", 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_addr": "172.21.15.157:6813/181354529", 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_back_addr": "172.21.15.157:6815/181354529", 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "heartbeat_front_addr": "172.21.15.157:6814/181354529", 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "state": [ 2024-09-16T03:40:00.276 INFO:teuthology.orchestra.run.smithi064.stdout: "exists", 2024-09-16T03:40:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "up" 2024-09-16T03:40:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:40:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_xinfo": [ 2024-09-16T03:40:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 0, 2024-09-16T03:40:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.277 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:18.425590+0000", 2024-09-16T03:40:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 1, 2024-09-16T03:40:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:06.827840+0000", 2024-09-16T03:40:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 2, 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:35.030728+0000", 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 3, 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.279 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:23.374251+0000", 2024-09-16T03:40:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 4, 2024-09-16T03:40:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.280 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:32.706176+0000", 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 5, 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:02.129117+0000", 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.281 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 6, 2024-09-16T03:40:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:19.948144+0000", 2024-09-16T03:40:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 7, 2024-09-16T03:40:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.282 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:38:56.835860+0000", 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 8, 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:10.351843+0000", 2024-09-16T03:40:00.283 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 9, 2024-09-16T03:40:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:38:58.705841+0000", 2024-09-16T03:40:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.284 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 10, 2024-09-16T03:40:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:40.632845+0000", 2024-09-16T03:40:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 11, 2024-09-16T03:40:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.285 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:05.604711+0000", 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 12, 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:25.172167+0000", 2024-09-16T03:40:00.286 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 13, 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 65536, 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:05.022970+0000", 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 14, 2024-09-16T03:40:00.287 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:30.872798+0000", 2024-09-16T03:40:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "osd": 15, 2024-09-16T03:40:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "down_stamp": "0.000000", 2024-09-16T03:40:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_probability": 0, 2024-09-16T03:40:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "laggy_interval": 0, 2024-09-16T03:40:00.288 INFO:teuthology.orchestra.run.smithi064.stdout: "features": 4540719139924082687, 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "old_weight": 0, 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "last_purged_snaps_scrub": "2024-09-16T03:39:30.112593+0000", 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "dead_epoch": 0 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap": [ 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.1", 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: 7, 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: 2 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.289 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_items": [], 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_upmap_primaries": [], 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "pg_temp": [ 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.0", 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: 9, 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: 1, 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: 4 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.290 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.1", 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: 4 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "3.3", 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: 8, 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: 15 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.6", 2024-09-16T03:40:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: 10, 2024-09-16T03:40:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-16T03:40:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: 4 2024-09-16T03:40:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.c", 2024-09-16T03:40:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: 12, 2024-09-16T03:40:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: 15, 2024-09-16T03:40:00.292 INFO:teuthology.orchestra.run.smithi064.stdout: 9 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.e", 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: 14, 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: 8 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.10", 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.293 INFO:teuthology.orchestra.run.smithi064.stdout: 10, 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: 4 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.1a", 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: 11, 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: 15 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.294 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.24", 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: 11, 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: 15 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.26", 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: 10, 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: 4 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.27", 2024-09-16T03:40:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: 6, 2024-09-16T03:40:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: 10, 2024-09-16T03:40:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: 11 2024-09-16T03:40:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.2a", 2024-09-16T03:40:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-16T03:40:00.296 INFO:teuthology.orchestra.run.smithi064.stdout: 8, 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: 10 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.2e", 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: 8, 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: 15 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.30", 2024-09-16T03:40:00.297 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: 10, 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: 3 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.31", 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: 12, 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: 4 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.33", 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: 4, 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: 9 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.35", 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: 3, 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: 9, 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: 10 2024-09-16T03:40:00.299 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "pgid": "4.36", 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "osds": [ 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: 2, 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: 10, 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: 15 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "primary_temp": [], 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklist": {}, 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "range_blocklist": {}, 2024-09-16T03:40:00.300 INFO:teuthology.orchestra.run.smithi064.stdout: "erasure_code_profiles": { 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "default": { 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "crush-failure-domain": "osd", 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "k": "2", 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "m": "1", 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "plugin": "jerasure", 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "technique": "reed_sol_van" 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "removed_snaps_queue": [], 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "new_removed_snaps": [], 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "new_purged_snaps": [], 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "crush_node_flags": {}, 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "device_class_flags": {}, 2024-09-16T03:40:00.301 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode": { 2024-09-16T03:40:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_enabled": false, 2024-09-16T03:40:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_bucket_count": 0, 2024-09-16T03:40:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: "degraded_stretch_mode": 0, 2024-09-16T03:40:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: "recovering_stretch_mode": 0, 2024-09-16T03:40:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: "stretch_mode_bucket": 0 2024-09-16T03:40:00.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:00.302 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:40:00.304 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:40:00.304 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:40:00.304 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:40:00.304 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:40:00.304 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:40:00.305 INFO:tasks.thrashosds.thrasher:j is {'epoch': 175, 'fsid': 'b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2', 'created': '2024-09-16T03:36:37.310421+0000', 'modified': '2024-09-16T03:39:59.375257+0000', 'last_up_change': '2024-09-16T03:36:50.058111+0000', 'last_in_change': '2024-09-16T03:39:28.633923+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T03:36:49.696307+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15, 'score_stable': 15, 'optimal_score': 0.20000000298023224, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8918943405151367, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T03:36:51.904455+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.7699999809265137, 'score_stable': 3.7699999809265137, 'optimal_score': 0.7300000190734863, 'raw_score_acting': 2.75, 'raw_score_stable': 2.75, 'primary_affinity_weighted': 0.8918943405151367, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 3, 'pool_name': 'low_tier', 'create_time': '2024-09-16T03:37:02.999795+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 4, 'pg_placement_num': 4, 'pg_placement_num_target': 4, 'pg_num_target': 4, 'pg_num_pending': 4, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.7699999809265137, 'score_stable': 3.7699999809265137, 'optimal_score': 0.5299999713897705, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8918943405151367, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.8899999856948853}}, {'pool': 4, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T03:37:04.279293+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 56, 'pg_placement_num': 56, 'pg_placement_num_target': 56, 'pg_num_target': 56, 'pg_num_pending': 56, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '157', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '126', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'dedup_cdc_chunk_size': 131072, 'dedup_chunk_algorithm': 'fastcdc', 'dedup_tier': 3, 'fingerprint_algorithm': 'sha256'}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1500000953674316, 'score_stable': 2.4200000762939453, 'optimal_score': 0.9300000071525574, 'raw_score_acting': 2, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8918943405151367, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.8899999856948853}}], 'osds': [{'osd': 0, 'uuid': 'bdbe961a-78d3-4f7d-88ed-ce24475d49e3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 171, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 3924351635}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6814', 'nonce': 3924351635}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6816', 'nonce': 3924351635}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 3924351635}]}, 'public_addr': '172.21.15.64:6813/3924351635', 'cluster_addr': '172.21.15.64:6814/3924351635', 'heartbeat_back_addr': '172.21.15.64:6816/3924351635', 'heartbeat_front_addr': '172.21.15.64:6815/3924351635', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '154c8ddd-7a3d-402c-b9d6-28538f4e6bc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6804', 'nonce': 624977804}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6805', 'nonce': 624977804}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6807', 'nonce': 624977804}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6806', 'nonce': 624977804}]}, 'public_addr': '172.21.15.112:6804/624977804', 'cluster_addr': '172.21.15.112:6805/624977804', 'heartbeat_back_addr': '172.21.15.112:6807/624977804', 'heartbeat_front_addr': '172.21.15.112:6806/624977804', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2a06de4e-0835-4a42-8c8a-49fb55aab566', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 159, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6812', 'nonce': 2804648086}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6813', 'nonce': 2804648086}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6815', 'nonce': 2804648086}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6814', 'nonce': 2804648086}]}, 'public_addr': '172.21.15.155:6812/2804648086', 'cluster_addr': '172.21.15.155:6813/2804648086', 'heartbeat_back_addr': '172.21.15.155:6815/2804648086', 'heartbeat_front_addr': '172.21.15.155:6814/2804648086', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '8a87f9f0-c380-4f38-bbd2-06b2def4fd2d', 'up': 1, 'in': 1, 'weight': 0.6943511962890625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 173, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6800', 'nonce': 1935597988}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6801', 'nonce': 1935597988}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6803', 'nonce': 1935597988}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6802', 'nonce': 1935597988}]}, 'public_addr': '172.21.15.157:6800/1935597988', 'cluster_addr': '172.21.15.157:6801/1935597988', 'heartbeat_back_addr': '172.21.15.157:6803/1935597988', 'heartbeat_front_addr': '172.21.15.157:6802/1935597988', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '5ac53318-12e8-496c-a87f-ba5de79c52aa', 'up': 1, 'in': 1, 'weight': 0.4737548828125, 'primary_affinity': 0.9752197265625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 171, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 2675475248}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6810', 'nonce': 2675475248}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6812', 'nonce': 2675475248}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 2675475248}]}, 'public_addr': '172.21.15.64:6809/2675475248', 'cluster_addr': '172.21.15.64:6810/2675475248', 'heartbeat_back_addr': '172.21.15.64:6812/2675475248', 'heartbeat_front_addr': '172.21.15.64:6811/2675475248', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '675c81a4-2314-463e-82a7-0060735c78fd', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6800', 'nonce': 499905276}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6801', 'nonce': 499905276}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6803', 'nonce': 499905276}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6802', 'nonce': 499905276}]}, 'public_addr': '172.21.15.112:6800/499905276', 'cluster_addr': '172.21.15.112:6801/499905276', 'heartbeat_back_addr': '172.21.15.112:6803/499905276', 'heartbeat_front_addr': '172.21.15.112:6802/499905276', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b18962a7-3b23-4e50-bf31-1d0629557423', 'up': 1, 'in': 1, 'weight': 0.95001220703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 169, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6808', 'nonce': 2839983406}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6809', 'nonce': 2839983406}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6811', 'nonce': 2839983406}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6810', 'nonce': 2839983406}]}, 'public_addr': '172.21.15.155:6808/2839983406', 'cluster_addr': '172.21.15.155:6809/2839983406', 'heartbeat_back_addr': '172.21.15.155:6811/2839983406', 'heartbeat_front_addr': '172.21.15.155:6810/2839983406', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '9706c1e8-dac0-4e16-8bc5-7cacc6f13870', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 173, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6808', 'nonce': 3021982871}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6809', 'nonce': 3021982871}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6811', 'nonce': 3021982871}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6810', 'nonce': 3021982871}]}, 'public_addr': '172.21.15.157:6808/3021982871', 'cluster_addr': '172.21.15.157:6809/3021982871', 'heartbeat_back_addr': '172.21.15.157:6811/3021982871', 'heartbeat_front_addr': '172.21.15.157:6810/3021982871', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'fafd06b8-c005-4471-a128-9dd51580c0f0', 'up': 1, 'in': 1, 'weight': 0.8023529052734375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 170, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 3609017627}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6806', 'nonce': 3609017627}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6808', 'nonce': 3609017627}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 3609017627}]}, 'public_addr': '172.21.15.64:6805/3609017627', 'cluster_addr': '172.21.15.64:6806/3609017627', 'heartbeat_back_addr': '172.21.15.64:6808/3609017627', 'heartbeat_front_addr': '172.21.15.64:6807/3609017627', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '79e38e57-52c2-416f-9b90-75621f06d3e3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 170, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6812', 'nonce': 701981815}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6813', 'nonce': 701981815}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6815', 'nonce': 701981815}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6814', 'nonce': 701981815}]}, 'public_addr': '172.21.15.112:6812/701981815', 'cluster_addr': '172.21.15.112:6813/701981815', 'heartbeat_back_addr': '172.21.15.112:6815/701981815', 'heartbeat_front_addr': '172.21.15.112:6814/701981815', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 174, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6804', 'nonce': 4138619523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6805', 'nonce': 4138619523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6807', 'nonce': 4138619523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6806', 'nonce': 4138619523}]}, 'public_addr': '172.21.15.155:6804/4138619523', 'cluster_addr': '172.21.15.155:6805/4138619523', 'heartbeat_back_addr': '172.21.15.155:6807/4138619523', 'heartbeat_front_addr': '172.21.15.155:6806/4138619523', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': '7a2a169a-ce94-4091-a792-27d0ecf9a7fa', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2950897216796875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 170, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6804', 'nonce': 3607367937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6805', 'nonce': 3607367937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6807', 'nonce': 3607367937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6806', 'nonce': 3607367937}]}, 'public_addr': '172.21.15.157:6804/3607367937', 'cluster_addr': '172.21.15.157:6805/3607367937', 'heartbeat_back_addr': '172.21.15.157:6807/3607367937', 'heartbeat_front_addr': '172.21.15.157:6806/3607367937', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': '7a280cda-b740-40c8-9fae-1a1ff11651f6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 2984461266}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6802', 'nonce': 2984461266}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6804', 'nonce': 2984461266}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 2984461266}]}, 'public_addr': '172.21.15.64:6801/2984461266', 'cluster_addr': '172.21.15.64:6802/2984461266', 'heartbeat_back_addr': '172.21.15.64:6804/2984461266', 'heartbeat_front_addr': '172.21.15.64:6803/2984461266', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': 'ce3a3148-cd90-422c-a106-045de89ab045', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6808', 'nonce': 1616314527}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6809', 'nonce': 1616314527}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6811', 'nonce': 1616314527}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.112:6810', 'nonce': 1616314527}]}, 'public_addr': '172.21.15.112:6808/1616314527', 'cluster_addr': '172.21.15.112:6809/1616314527', 'heartbeat_back_addr': '172.21.15.112:6811/1616314527', 'heartbeat_front_addr': '172.21.15.112:6810/1616314527', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '461c2d79-f4e7-4b21-899c-50d4a6cf12f9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6800', 'nonce': 4203868121}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6801', 'nonce': 4203868121}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6803', 'nonce': 4203868121}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.155:6802', 'nonce': 4203868121}]}, 'public_addr': '172.21.15.155:6800/4203868121', 'cluster_addr': '172.21.15.155:6801/4203868121', 'heartbeat_back_addr': '172.21.15.155:6803/4203868121', 'heartbeat_front_addr': '172.21.15.155:6802/4203868121', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': '9a8c0bbe-9389-4b2c-95e1-34d3e66461e9', 'up': 1, 'in': 1, 'weight': 0.80743408203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 173, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6812', 'nonce': 181354529}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6813', 'nonce': 181354529}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6815', 'nonce': 181354529}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.157:6814', 'nonce': 181354529}]}, 'public_addr': '172.21.15.157:6812/181354529', 'cluster_addr': '172.21.15.157:6813/181354529', 'heartbeat_back_addr': '172.21.15.157:6815/181354529', 'heartbeat_front_addr': '172.21.15.157:6814/181354529', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:18.425590+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:06.827840+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:35.030728+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:23.374251+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:32.706176+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-16T03:39:02.129117+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:19.948144+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:38:56.835860+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:10.351843+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:38:58.705841+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:40.632845+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:05.604711+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:25.172167+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-16T03:39:05.022970+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:30.872798+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T03:39:30.112593+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.1', 'osds': [4, 7, 2]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.0', 'osds': [9, 1, 4]}, {'pgid': '3.1', 'osds': [6, 3, 4]}, {'pgid': '3.3', 'osds': [8, 2, 15]}, {'pgid': '4.6', 'osds': [10, 6, 4]}, {'pgid': '4.c', 'osds': [12, 15, 9]}, {'pgid': '4.e', 'osds': [4, 14, 8]}, {'pgid': '4.10', 'osds': [10, 3, 4]}, {'pgid': '4.1a', 'osds': [6, 11, 15]}, {'pgid': '4.24', 'osds': [2, 11, 15]}, {'pgid': '4.26', 'osds': [6, 10, 4]}, {'pgid': '4.27', 'osds': [6, 10, 11]}, {'pgid': '4.2a', 'osds': [3, 8, 10]}, {'pgid': '4.2e', 'osds': [4, 8, 15]}, {'pgid': '4.30', 'osds': [4, 10, 3]}, {'pgid': '4.31', 'osds': [12, 2, 4]}, {'pgid': '4.33', 'osds': [3, 4, 9]}, {'pgid': '4.35', 'osds': [3, 9, 10]}, {'pgid': '4.36', 'osds': [2, 10, 15]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T03:40:00.306 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:00.306 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:40:00.306 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:40:00.302+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:40:00.384 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:40:00.382+0000 7f49439d4640 -1 osd.4 175 handle_osd_map injecting map crc failure 2024-09-16T03:40:00.386 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:40:00.389 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:40:00.388+0000 7fedb773d640 -1 osd.9 175 handle_osd_map injecting map crc failure 2024-09-16T03:40:00.404 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:40:00.403+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:40:00.414 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:40:00.415 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:40:00.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:40:00.415 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:40:00.415 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.415 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:5 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'1, mlcod=0'0)", 2024-09-16T03:40:00.415 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:40:00.415 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 168.44437930199999, 2024-09-16T03:40:00.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.105742802, 2024-09-16T03:40:00.415 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.415 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.415 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.415 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.415 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:40:00.416 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.416 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968146+0000", 2024-09-16T03:40:00.416 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.416 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968147+0000", 2024-09-16T03:40:00.416 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4e-07 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968752+0000", 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00060437900000000005 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968762+0000", 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0284e-05 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.968778+0000", 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6124e-05 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069797+0000", 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10101895700000001 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.069833+0000", 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6013000000000002e-05 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073838+0000", 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040049049999999996 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.073889+0000", 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.13e-05 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:7 4.4 e38/32 4:239dbd85:::smithi15736696-2:head v 38'3, mlcod=0'0)", 2024-09-16T03:40:00.419 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:40:00.420 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 168.43748680300001, 2024-09-16T03:40:00.420 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20061241399999999, 2024-09-16T03:40:00.420 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.420 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.420 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.420 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.420 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.420 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.420 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:40:00.420 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.420 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.420 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.420 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975039+0000", 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975042+0000", 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.5289999999999998e-06 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975079+0000", 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7626999999999997e-05 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:11.975104+0000", 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4958999999999999e-05 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.075211+0000", 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100106814 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080573+0000", 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0053618650000000004 2024-09-16T03:40:00.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.080592+0000", 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9079000000000001e-05 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175497+0000", 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.094904832999999994 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.175651+0000", 2024-09-16T03:40:00.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015470800000000001 2024-09-16T03:40:00.424 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.424 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.424 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.424 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.424 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.15.23:44 3.3 e49/31 3:dc284e8d:::smithi15736696-28 low_tier:head v 49'55, mlcod=49'53)", 2024-09-16T03:40:00.424 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:40:00.424 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 150.28956240700001, 2024-09-16T03:40:00.424 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.149341897, 2024-09-16T03:40:00.424 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.424 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.424 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.424 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122963+0000", 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122964+0000", 2024-09-16T03:40:00.425 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.91e-07 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122968+0000", 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1470000000000003e-06 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.122979+0000", 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1090000000000001e-05 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.223110+0000", 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100131357 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271688+0000", 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.048578192999999999 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.271712+0000", 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4066e-05 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272260+0000", 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054737000000000004 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.272305+0000", 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5482999999999998e-05 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1007 3.3 e50/31 3:fda66a0d:::920f29fe85d0745d4cd24a887714ac53f1b9a978ec2208ff473c99287fdf8feb:head v 50'57, mlcod=49'55)", 2024-09-16T03:40:00.428 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:40:00.429 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 149.81793585299999, 2024-09-16T03:40:00.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20236485200000001, 2024-09-16T03:40:00.429 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.429 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.429 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.429 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.429 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:40:00.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.429 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594590+0000", 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.594592+0000", 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0700000000000001e-06 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595508+0000", 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00091599499999999996 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595521+0000", 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.595531+0000", 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2499999999999995e-06 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795762+0000", 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200230977 2024-09-16T03:40:00.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.432 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.795778+0000", 2024-09-16T03:40:00.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6589999999999999e-05 2024-09-16T03:40:00.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.432 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796912+0000", 2024-09-16T03:40:00.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011338520000000001 2024-09-16T03:40:00.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.432 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.796955+0000", 2024-09-16T03:40:00.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2768000000000001e-05 2024-09-16T03:40:00.433 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.433 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.433 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.433 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1247 4.11 4:89839b7e:::smithi15736696-12:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-16T03:40:00.433 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:40:00.433 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 143.45694867200001, 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.202294525, 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1247 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:40:00.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955577+0000", 2024-09-16T03:40:00.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955579+0000", 2024-09-16T03:40:00.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3240000000000001e-06 2024-09-16T03:40:00.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955587+0000", 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4329999999999998e-06 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955600+0000", 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2723e-05 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:36.955612+0000", 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2422999999999999e-05 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.436 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055837+0000", 2024-09-16T03:40:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10022539699999999 2024-09-16T03:40:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.055974+0000", 2024-09-16T03:40:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000136995 2024-09-16T03:40:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,7,8,11,12", 2024-09-16T03:40:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.056092+0000", 2024-09-16T03:40:00.437 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011738 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057209+0000", 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0011170329999999999 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.057262+0000", 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2859000000000003e-05 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.438 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157594+0000", 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10033241699999999 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157648+0000", 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3863999999999998e-05 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157692+0000", 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4406999999999997e-05 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.439 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157732+0000", 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9156000000000003e-05 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157780+0000", 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8801999999999998e-05 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:37.157872+0000", 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1311999999999996e-05 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.440 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:1518 3.3 e59/31 3:df1be05b:::e06092e68680905b5745d5057e93f6c7581518aed07ca5db2802546e9c29aeef:head v 59'95, mlcod=58'93)", 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 135.15450499900001, 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104523454, 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.441 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258021+0000", 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258022+0000", 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.372e-06 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.442 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258462+0000", 2024-09-16T03:40:00.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00043970100000000002 2024-09-16T03:40:00.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258476+0000", 2024-09-16T03:40:00.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.376e-05 2024-09-16T03:40:00.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.258483+0000", 2024-09-16T03:40:00.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.294e-06 2024-09-16T03:40:00.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358679+0000", 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100196177 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.358718+0000", 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9295999999999998e-05 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362496+0000", 2024-09-16T03:40:00.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0037772069999999999 2024-09-16T03:40:00.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.445 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:45.362544+0000", 2024-09-16T03:40:00.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8646999999999997e-05 2024-09-16T03:40:00.445 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.445 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.445 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.445 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:2445 4.11 4:89839b7e:::smithi15736696-12:head [sparse-read 1818507~599510,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-09-16T03:40:00.445 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:40:00.445 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 104.968362349, 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200613392, 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 2445 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:40:00.446 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444163+0000", 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444167+0000", 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6650000000000001e-06 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444182+0000", 2024-09-16T03:40:00.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4678000000000001e-05 2024-09-16T03:40:00.448 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.448 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.448 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.444207+0000", 2024-09-16T03:40:00.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4768000000000001e-05 2024-09-16T03:40:00.448 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.448 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.448 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.544359+0000", 2024-09-16T03:40:00.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100152804 2024-09-16T03:40:00.448 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644561+0000", 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10020214500000001 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644624+0000", 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2444e-05 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:15.644777+0000", 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000152888 2024-09-16T03:40:00.449 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:2629 3.3 e89/31 3:e23364e2:::e447860b11e9f9fde68b75cb1d2f204a74952889ec51a2417ef18c1427ff9a70:head v 89'173, mlcod=87'171)", 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 99.960619656000006, 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10848095100000001, 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451906+0000", 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.451907+0000", 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6599999999999996e-07 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456443+0000", 2024-09-16T03:40:00.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0045365509999999998 2024-09-16T03:40:00.452 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456470+0000", 2024-09-16T03:40:00.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6897e-05 2024-09-16T03:40:00.452 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.456496+0000", 2024-09-16T03:40:00.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.618e-05 2024-09-16T03:40:00.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.453 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.453 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.453 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556666+0000", 2024-09-16T03:40:00.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100169678 2024-09-16T03:40:00.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.453 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.453 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.453 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.556686+0000", 2024-09-16T03:40:00.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9740000000000001e-05 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560345+0000", 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003658821 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:20.560387+0000", 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2518000000000002e-05 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.454 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(osd.10.22:209 3.3 e99/31 3:f2e0c7d0:::6bf49184c86dbc717b1f919afcf6cb40a4ca9c2465eb100f7ba59687540c40ed:head v 99'201, mlcod=98'194)", 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 88.206489458999997, 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200559603, 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.455 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.456 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.456 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.456 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206036+0000", 2024-09-16T03:40:00.456 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.456 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.456 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.456 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.456 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206037+0000", 2024-09-16T03:40:00.456 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9999999999999999e-07 2024-09-16T03:40:00.456 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.456 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.456 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.456 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206039+0000", 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2579999999999998e-06 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.206041+0000", 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.542e-06 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306162+0000", 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100120348 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.457 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.458 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306250+0000", 2024-09-16T03:40:00.458 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:40:00.458 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.458 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.458 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.458 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.306268+0000", 2024-09-16T03:40:00.458 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8127000000000001e-05 2024-09-16T03:40:00.458 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.458 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.458 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.458 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.406549+0000", 2024-09-16T03:40:00.458 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10028121199999999 2024-09-16T03:40:00.458 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:32.406596+0000", 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6700999999999999e-05 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3274 4.11 4:89839b7e:::smithi15736696-12:head [write 1813152~490434] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 79.214726288999998, 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.28154463400000002, 2024-09-16T03:40:00.459 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.460 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.460 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.460 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:00.460 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:00.460 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:00.460 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3274 2024-09-16T03:40:00.460 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.460 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.460 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.460 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.460 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:40:00.460 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.460 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197799+0000", 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.197801+0000", 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9539999999999998e-06 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.198205+0000", 2024-09-16T03:40:00.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00040328199999999998 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.198218+0000", 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3176000000000001e-05 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298318+0000", 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100100562 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.462 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298419+0000", 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100667 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298489+0000", 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0023000000000004e-05 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,12", 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298583+0000", 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3634000000000005e-05 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.463 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.464 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300283+0000", 2024-09-16T03:40:00.464 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00170049 2024-09-16T03:40:00.464 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.464 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.464 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.464 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.307072+0000", 2024-09-16T03:40:00.464 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0067889209999999998 2024-09-16T03:40:00.464 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.464 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.464 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.464 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479243+0000", 2024-09-16T03:40:00.464 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.17217031799999999 2024-09-16T03:40:00.464 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479300+0000", 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7837999999999997e-05 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.479344+0000", 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3769000000000001e-05 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.465 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3275 4.11 4:89839b7e:::smithi15736696-12:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 79.114180876000006, 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.276974636, 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3275 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.466 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298345+0000", 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298364+0000", 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9273e-05 2024-09-16T03:40:00.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298373+0000", 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5809999999999997e-06 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298382+0000", 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6269999999999995e-06 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298390+0000", 2024-09-16T03:40:00.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2259999999999999e-06 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299770+0000", 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0013803159999999999 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299852+0000", 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.174e-05 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 4,12", 2024-09-16T03:40:00.469 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.299948+0000", 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6669000000000006e-05 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300496+0000", 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00054762799999999998 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.401930+0000", 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101434495 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.470 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575189+0000", 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.173258996 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575247+0000", 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7651000000000001e-05 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.575320+0000", 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2434000000000005e-05 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.471 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.472 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.472 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.472 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:3276 4.11 4:89839b7e:::smithi15736696-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:40:00.472 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:40:00.472 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 79.114112063999997, 2024-09-16T03:40:00.472 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.37724456499999998, 2024-09-16T03:40:00.472 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.472 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.472 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:40:00.472 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:00.472 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:00.472 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:00.472 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 3276 2024-09-16T03:40:00.473 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.473 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.473 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.473 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.473 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:40:00.473 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.473 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.473 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.473 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.473 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298414+0000", 2024-09-16T03:40:00.473 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.473 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.473 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298415+0000", 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.2500000000000004e-07 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298418+0000", 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.766e-06 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298426+0000", 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1359999999999999e-06 2024-09-16T03:40:00.474 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.298433+0000", 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0899999999999999e-06 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300168+0000", 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0017353869999999999 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.300215+0000", 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6702000000000001e-05 2024-09-16T03:40:00.475 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.476 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.476 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675524+0000", 2024-09-16T03:40:00.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.375309585 2024-09-16T03:40:00.476 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.476 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.476 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675596+0000", 2024-09-16T03:40:00.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.1921000000000005e-05 2024-09-16T03:40:00.476 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.476 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.476 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:41.675658+0000", 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2052999999999997e-05 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3985 4.36 e130/126 4:6ef3b53d:::smithi15736696-48:head v 130'88, mlcod=124'86)", 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.376369+0000", 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 55.036156382999998, 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20392002300000001, 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376369+0000", 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376369+0000", 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376370+0000", 2024-09-16T03:40:00.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1299999999999999e-07 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376877+0000", 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00050726199999999997 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376888+0000", 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0798999999999999e-05 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376896+0000", 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.1380000000000007e-06 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376950+0000", 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4580000000000003e-05 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.376970+0000", 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9607e-05 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580253+0000", 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20328323600000001 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580289+0000", 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6188e-05 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3986 4.36 e130/126 4:6ef3b53d:::smithi15736696-48:head v 130'89, mlcod=124'86)", 2024-09-16T03:40:00.481 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.377664+0000", 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 55.034861653, 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264148100000001, 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.377664+0000", 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.482 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.377664+0000", 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.377665+0000", 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.540000000000001e-07 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378852+0000", 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001187097 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.483 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378863+0000", 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0721e-05 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378868+0000", 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5650000000000001e-06 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378913+0000", 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4867999999999998e-05 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.378929+0000", 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5752000000000001e-05 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580291+0000", 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20136209599999999 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580306+0000", 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4428e-05 2024-09-16T03:40:00.485 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3987 4.36 e130/126 4:6ef3b53d:::smithi15736696-48:head v 130'90, mlcod=124'86)", 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.379666+0000", 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 55.032860190999997, 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20094095300000001, 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.487 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379666+0000", 2024-09-16T03:40:00.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.487 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.487 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.487 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.487 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379666+0000", 2024-09-16T03:40:00.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.487 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.487 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.487 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.487 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379667+0000", 2024-09-16T03:40:00.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.11e-06 2024-09-16T03:40:00.487 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.488 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379675+0000", 2024-09-16T03:40:00.488 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4489999999999999e-06 2024-09-16T03:40:00.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.488 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379683+0000", 2024-09-16T03:40:00.488 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.0660000000000004e-06 2024-09-16T03:40:00.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.488 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.379690+0000", 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4010000000000002e-06 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.579898+0000", 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20020810999999999 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.579914+0000", 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6198e-05 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.489 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580572+0000", 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065845699999999999 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.580607+0000", 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4162000000000003e-05 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:4043 4.36 e131/126 4:6ece7830:::smithi15736696-7:head v 131'93, mlcod=130'89)", 2024-09-16T03:40:00.490 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.055626+0000", 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 54.356899409999997, 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19612945100000001, 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055626+0000", 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.491 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055626+0000", 2024-09-16T03:40:00.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.492 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.492 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.492 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055627+0000", 2024-09-16T03:40:00.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0869999999999999e-06 2024-09-16T03:40:00.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.492 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.492 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.492 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055633+0000", 2024-09-16T03:40:00.492 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.101e-06 2024-09-16T03:40:00.492 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.492 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055641+0000", 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9759999999999995e-06 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.055646+0000", 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4360000000000001e-06 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251121+0000", 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.195475068 2024-09-16T03:40:00.493 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251137+0000", 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6271999999999999e-05 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251719+0000", 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00058167799999999997 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.251756+0000", 2024-09-16T03:40:00.494 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.6832999999999998e-05 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "replica_scrub(pg: 4.12,from:0'0,to:127'20,epoch:139/138,start:4:48000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:14.698983+0000", 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 45.713542363000002, 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12536847100000001, 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.495 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.698983+0000", 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.698983+0000", 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.698985+0000", 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4899999999999999e-06 2024-09-16T03:40:00.496 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.698991+0000", 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9440000000000002e-06 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.699003+0000", 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1708999999999999e-05 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.699009+0000", 2024-09-16T03:40:00.497 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4849999999999997e-06 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.824228+0000", 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12521855800000001 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.824262+0000", 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4513e-05 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:14.824352+0000", 2024-09-16T03:40:00.498 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9771999999999994e-05 2024-09-16T03:40:00.499 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.499 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.499 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.499 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.499 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.499 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "MOSDPGPush(4.28 160/158 [PushOp(4:148e627d:::smithi15736696-23:head, version: 129'68, data_included: [658874~799064,1976979~249512], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:148e627d:::smithi15736696-23:head@129'68, size: 3899224, copy_subset: [(0, 3899224)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2226491, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-16T03:40:00.499 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:39.703233+0000", 2024-09-16T03:40:00.499 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.709292413, 2024-09-16T03:40:00.499 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104655188, 2024-09-16T03:40:00.499 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.499 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.499 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:40:00.499 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.500 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.703233+0000", 2024-09-16T03:40:00.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.500 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.703233+0000", 2024-09-16T03:40:00.500 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.500 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.500 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.500 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.500 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.703236+0000", 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.61e-06 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.705967+0000", 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0027309679999999999 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.705989+0000", 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2200000000000001e-05 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.501 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.706002+0000", 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3010000000000001e-05 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.806175+0000", 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100172785 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.806191+0000", 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5865000000000001e-05 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.502 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.807889+0000", 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00169775 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "pg_backfill(progress 4.28 e 160/160 lb 4:146f84f0:::smithi15736696-6:head)", 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:39.706023+0000", 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 20.706502924999999, 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20046278000000001, 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.503 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.504 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:40:00.504 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.504 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.504 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.504 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.706023+0000", 2024-09-16T03:40:00.504 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.504 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.504 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.504 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.504 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.706023+0000", 2024-09-16T03:40:00.504 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.504 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.504 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.706023+0000", 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2099999999999996e-07 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.706029+0000", 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2399999999999998e-06 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.706033+0000", 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2470000000000002e-06 2024-09-16T03:40:00.505 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.806100+0000", 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10006743 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.906349+0000", 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100248381 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.906370+0000", 2024-09-16T03:40:00.506 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1455999999999999e-05 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:39.906486+0000", 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000115405 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:5073 4.11 4:89839b7e:::smithi15736696-12:head [sparse-read 0~2871452,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:54.478314+0000", 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 5.9342119650000003, 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20265423399999999, 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 5073 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.478314+0000", 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.508 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.478314+0000", 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.478316+0000", 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.782e-06 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.478322+0000", 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9939999999999997e-06 2024-09-16T03:40:00.509 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.478332+0000", 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0338e-05 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.578481+0000", 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100149126 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.678642+0000", 2024-09-16T03:40:00.510 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10016080300000001 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.678693+0000", 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1282000000000001e-05 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.680968+0000", 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0022749089999999999 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.511 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.512 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.512 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:40:00.513 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:40:00.514 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:40:00.511+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:40:00.614 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:40:00.612+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:40:00.649 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:00.650 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:40:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:40:00.658 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:40:00.658 INFO:teuthology.orchestra.run.smithi155.stdout: "num_ops": 0 2024-09-16T03:40:00.658 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:40:00.660 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-16T03:40:00.384802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":269696260,"num_objects":204,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":132,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":3036,"num_read_kb":1715449,"num_write":2330,"num_write_kb":518162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":151868233,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":24,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":207,"acting":207,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":245,"num_osds":14,"num_per_pool_osds":14,"num_per_pool_omap_osds":14,"kb":1321205760,"kb_used":961928,"kb_used_data":504612,"kb_used_omap":21,"kb_used_meta":448810,"kb_avail":1320243832,"statfs":{"total":1352914698240,"available":1351929683968,"internally_reserved":0,"allocated":516722688,"data_stored":524296992,"data_compressed":14999774,"data_compressed_allocated":16445440,"data_compressed_original":34537472,"omap_allocated":22329,"internal_metadata":459581639},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-305287,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":95,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":46,"num_read_kb":445,"num_write":62,"num_write_kb":11269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":45403730,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.013789"},"pg_stats":[{"pgid":"4.37","version":"153'56","reported_seq":506,"reported_epoch":172,"state":"peering","last_fresh":"2024-09-16T03:39:56.373768+0000","last_change":"2024-09-16T03:39:56.373768+0000","last_active":"2024-09-16T03:39:56.373768+0000","last_peered":"2024-09-16T03:39:55.916257+0000","last_clean":"2024-09-16T03:39:32.454334+0000","last_became_active":"2024-09-16T03:39:55.303172+0000","last_became_peered":"2024-09-16T03:39:55.303172+0000","last_unstale":"2024-09-16T03:39:56.373768+0000","last_undegraded":"2024-09-16T03:39:56.373768+0000","last_fullsized":"2024-09-16T03:39:56.373768+0000","mapping_epoch":172,"log_start":"142'54","ondisk_log_start":"142'54","created":126,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":6,"last_scrub":"59'33","last_scrub_stamp":"2024-09-16T03:37:47.616805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:37:47.616805+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":49,"ondisk_log_size":2,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3378290,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":102,"num_read_kb":92194,"num_write":16,"num_write_kb":4585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2934066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,14,7],"acting":[15,14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.36","version":"142'96","reported_seq":547,"reported_epoch":171,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:55.917326+0000","last_change":"2024-09-16T03:39:38.887893+0000","last_active":"2024-09-16T03:39:55.917326+0000","last_peered":"2024-09-16T03:39:55.917326+0000","last_clean":"2024-09-16T03:39:19.651965+0000","last_became_active":"2024-09-16T03:39:38.786957+0000","last_became_peered":"2024-09-16T03:39:38.786957+0000","last_unstale":"2024-09-16T03:39:55.917326+0000","last_undegraded":"2024-09-16T03:39:55.917326+0000","last_fullsized":"2024-09-16T03:39:55.917326+0000","mapping_epoch":158,"log_start":"131'93","ondisk_log_start":"131'93","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.716125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.716125+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":91,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7159980,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":90,"num_read_kb":79209,"num_write":12,"num_write_kb":3200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,11],"acting":[2,10,15],"avail_no_missing":["2","10","15"],"object_location_counts":[{"shards":"2,10,15","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"169'85","reported_seq":626,"reported_epoch":175,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:59.580167+0000","last_change":"2024-09-16T03:39:56.515854+0000","last_active":"2024-09-16T03:39:59.580167+0000","last_peered":"2024-09-16T03:39:59.580167+0000","last_clean":"2024-09-16T03:39:35.664815+0000","last_became_active":"2024-09-16T03:39:56.009732+0000","last_became_peered":"2024-09-16T03:39:56.009732+0000","last_unstale":"2024-09-16T03:39:59.580167+0000","last_undegraded":"2024-09-16T03:39:59.580167+0000","last_fullsized":"2024-09-16T03:39:59.580167+0000","mapping_epoch":170,"log_start":"165'81","ondisk_log_start":"165'81","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'67","last_scrub_stamp":"2024-09-16T03:39:15.079987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:15.079987+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":79,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6874195,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":30309,"num_write":19,"num_write_kb":5766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[3,9,10],"avail_no_missing":["3","9","10"],"object_location_counts":[{"shards":"3,9,10","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"149'25","reported_seq":408,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.782593+0000","last_change":"2024-09-16T03:39:58.682198+0000","last_active":"2024-09-16T03:39:58.782593+0000","last_peered":"2024-09-16T03:39:58.782593+0000","last_clean":"2024-09-16T03:39:58.782593+0000","last_became_active":"2024-09-16T03:39:58.581739+0000","last_became_peered":"2024-09-16T03:39:58.581739+0000","last_unstale":"2024-09-16T03:39:58.782593+0000","last_undegraded":"2024-09-16T03:39:58.782593+0000","last_fullsized":"2024-09-16T03:39:58.782593+0000","mapping_epoch":173,"log_start":"149'23","ondisk_log_start":"149'23","created":126,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":6,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:02.919259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.919259+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":20,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:11.357946+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3109361,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":17899,"num_write":5,"num_write_kb":1218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2853416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,9],"acting":[10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.33","version":"148'76","reported_seq":578,"reported_epoch":175,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:59.580399+0000","last_change":"2024-09-16T03:39:39.185026+0000","last_active":"2024-09-16T03:39:59.580399+0000","last_peered":"2024-09-16T03:39:59.580399+0000","last_clean":"2024-09-16T03:39:26.364451+0000","last_became_active":"2024-09-16T03:39:38.883512+0000","last_became_peered":"2024-09-16T03:39:38.883512+0000","last_unstale":"2024-09-16T03:39:59.580399+0000","last_undegraded":"2024-09-16T03:39:59.580399+0000","last_fullsized":"2024-09-16T03:39:59.580399+0000","mapping_epoch":158,"log_start":"137'73","ondisk_log_start":"137'73","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'72","last_scrub_stamp":"2024-09-16T03:39:02.143339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.143339+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":69,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353046,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":37925,"num_write":4,"num_write_kb":955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":994759,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,10],"acting":[3,4,9],"avail_no_missing":["3","4","9"],"object_location_counts":[{"shards":"3,4,9","objects":1}],"blocked_by":[],"up_primary":14,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.32","version":"106'17","reported_seq":319,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.582292+0000","last_change":"2024-09-16T03:39:44.270851+0000","last_active":"2024-09-16T03:39:58.582292+0000","last_peered":"2024-09-16T03:39:58.582292+0000","last_clean":"2024-09-16T03:39:58.582292+0000","last_became_active":"2024-09-16T03:39:40.893486+0000","last_became_peered":"2024-09-16T03:39:40.893486+0000","last_unstale":"2024-09-16T03:39:58.582292+0000","last_undegraded":"2024-09-16T03:39:58.582292+0000","last_fullsized":"2024-09-16T03:39:58.582292+0000","mapping_epoch":160,"log_start":"82'13","ondisk_log_start":"82'13","created":126,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":6,"last_scrub":"106'17","last_scrub_stamp":"2024-09-16T03:39:12.108317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:12.108317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:40:11.560664+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1033647,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":20741,"num_write":6,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1331059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,12],"acting":[0,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.31","version":"114'17","reported_seq":286,"reported_epoch":174,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:58.377155+0000","last_change":"2024-09-16T03:39:38.988193+0000","last_active":"2024-09-16T03:39:58.377155+0000","last_peered":"2024-09-16T03:39:58.377155+0000","last_clean":"2024-09-16T03:38:48.641776+0000","last_became_active":"2024-09-16T03:39:38.987267+0000","last_became_peered":"2024-09-16T03:39:38.987267+0000","last_unstale":"2024-09-16T03:39:58.377155+0000","last_undegraded":"2024-09-16T03:39:58.377155+0000","last_fullsized":"2024-09-16T03:39:58.377155+0000","mapping_epoch":158,"log_start":"107'14","ondisk_log_start":"107'14","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:35.719895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:35.719895+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1495840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":28046,"num_write":4,"num_write_kb":1066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,11],"acting":[12,2,4],"avail_no_missing":["12","2","4"],"object_location_counts":[{"shards":"2,4,12","objects":1}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.30","version":"118'64","reported_seq":410,"reported_epoch":175,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:59.576808+0000","last_change":"2024-09-16T03:39:56.515576+0000","last_active":"2024-09-16T03:39:59.576808+0000","last_peered":"2024-09-16T03:39:59.576808+0000","last_clean":"2024-09-16T03:39:52.798009+0000","last_became_active":"2024-09-16T03:39:56.404165+0000","last_became_peered":"2024-09-16T03:39:56.404165+0000","last_unstale":"2024-09-16T03:39:59.576808+0000","last_undegraded":"2024-09-16T03:39:59.576808+0000","last_fullsized":"2024-09-16T03:39:59.576808+0000","mapping_epoch":170,"log_start":"113'61","ondisk_log_start":"113'61","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":6,"last_scrub":"118'64","last_scrub_stamp":"2024-09-16T03:39:38.383253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.383253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,3],"avail_no_missing":["4","3","9","10"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"113'94","reported_seq":464,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.477401+0000","last_change":"2024-09-16T03:39:42.292571+0000","last_active":"2024-09-16T03:39:58.477401+0000","last_peered":"2024-09-16T03:39:58.477401+0000","last_clean":"2024-09-16T03:39:58.477401+0000","last_became_active":"2024-09-16T03:39:40.802316+0000","last_became_peered":"2024-09-16T03:39:40.802316+0000","last_unstale":"2024-09-16T03:39:58.477401+0000","last_undegraded":"2024-09-16T03:39:58.477401+0000","last_fullsized":"2024-09-16T03:39:58.477401+0000","mapping_epoch":160,"log_start":"107'90","ondisk_log_start":"107'90","created":126,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":6,"last_scrub":"113'94","last_scrub_stamp":"2024-09-16T03:39:03.319012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.319012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":87,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:40:11.566141+0000","scrub_duration":319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1470629,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,3],"acting":[14,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.2e","version":"155'93","reported_seq":570,"reported_epoch":175,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:59.577135+0000","last_change":"2024-09-16T03:39:39.286505+0000","last_active":"2024-09-16T03:39:59.577135+0000","last_peered":"2024-09-16T03:39:59.577135+0000","last_clean":"2024-09-16T03:39:35.765256+0000","last_became_active":"2024-09-16T03:39:38.884986+0000","last_became_peered":"2024-09-16T03:39:38.884986+0000","last_unstale":"2024-09-16T03:39:59.577135+0000","last_undegraded":"2024-09-16T03:39:59.577135+0000","last_fullsized":"2024-09-16T03:39:59.577135+0000","mapping_epoch":158,"log_start":"152'90","ondisk_log_start":"152'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"123'75","last_scrub_stamp":"2024-09-16T03:39:04.235141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.235141+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":90,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151494,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":62325,"num_write":19,"num_write_kb":5191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,14],"acting":[4,8,15],"avail_no_missing":["4","8","15"],"object_location_counts":[{"shards":"4,8,15","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"172'112","reported_seq":583,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.493029+0000","last_change":"2024-09-16T03:39:03.917422+0000","last_active":"2024-09-16T03:39:58.493029+0000","last_peered":"2024-09-16T03:39:58.493029+0000","last_clean":"2024-09-16T03:39:58.493029+0000","last_became_active":"2024-09-16T03:39:01.801112+0000","last_became_peered":"2024-09-16T03:39:01.801112+0000","last_unstale":"2024-09-16T03:39:58.493029+0000","last_undegraded":"2024-09-16T03:39:58.493029+0000","last_fullsized":"2024-09-16T03:39:58.493029+0000","mapping_epoch":32,"log_start":"166'109","ondisk_log_start":"166'109","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'97","last_scrub_stamp":"2024-09-16T03:39:03.917355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.917355+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":106,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:06.444266+0000","scrub_duration":771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496021,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":28034,"num_write":29,"num_write_kb":7222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.e","version":"165'94","reported_seq":578,"reported_epoch":175,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:59.576801+0000","last_change":"2024-09-16T03:39:56.722563+0000","last_active":"2024-09-16T03:39:59.576801+0000","last_peered":"2024-09-16T03:39:59.576801+0000","last_clean":"2024-09-16T03:39:54.776007+0000","last_became_active":"2024-09-16T03:39:56.614366+0000","last_became_peered":"2024-09-16T03:39:56.614366+0000","last_unstale":"2024-09-16T03:39:59.576801+0000","last_undegraded":"2024-09-16T03:39:59.576801+0000","last_fullsized":"2024-09-16T03:39:59.576801+0000","mapping_epoch":171,"log_start":"154'91","ondisk_log_start":"154'91","created":32,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'93","last_scrub_stamp":"2024-09-16T03:39:35.280938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:35.280938+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":88,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327852,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":17624,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2614941,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14,3],"acting":[4,14,8],"avail_no_missing":["4","8","14","15"],"object_location_counts":[{"shards":"4,8,14","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"165'27","reported_seq":474,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.377352+0000","last_change":"2024-09-16T03:39:33.569454+0000","last_active":"2024-09-16T03:39:58.377352+0000","last_peered":"2024-09-16T03:39:58.377352+0000","last_clean":"2024-09-16T03:39:58.377352+0000","last_became_active":"2024-09-16T03:39:33.569207+0000","last_became_peered":"2024-09-16T03:39:33.569207+0000","last_unstale":"2024-09-16T03:39:58.377352+0000","last_undegraded":"2024-09-16T03:39:58.377352+0000","last_fullsized":"2024-09-16T03:39:58.377352+0000","mapping_epoch":153,"log_start":"152'23","ondisk_log_start":"152'23","created":32,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'19","last_scrub_stamp":"2024-09-16T03:39:01.149486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.149486+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":23,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:09.211597+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3428694,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":17088,"num_write":9,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1327143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,1,3],"acting":[12,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"167'69","reported_seq":533,"reported_epoch":174,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:58.377156+0000","last_change":"2024-09-16T03:39:31.255267+0000","last_active":"2024-09-16T03:39:58.377156+0000","last_peered":"2024-09-16T03:39:58.377156+0000","last_clean":"2024-09-16T03:39:27.002211+0000","last_became_active":"2024-09-16T03:39:31.053925+0000","last_became_peered":"2024-09-16T03:39:31.053925+0000","last_unstale":"2024-09-16T03:39:58.377156+0000","last_undegraded":"2024-09-16T03:39:58.377156+0000","last_fullsized":"2024-09-16T03:39:58.377156+0000","mapping_epoch":150,"log_start":"167'66","ondisk_log_start":"167'66","created":32,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'57","last_scrub_stamp":"2024-09-16T03:38:48.923184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.923184+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":64,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512729,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":30539,"num_write":14,"num_write_kb":4279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1213217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,1],"acting":[12,15,9],"avail_no_missing":["12","9","15"],"object_location_counts":[{"shards":"9,12,15","objects":1}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"153'135","reported_seq":680,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T03:39:56.373441+0000","last_change":"2024-09-16T03:39:09.748835+0000","last_active":"2024-09-16T03:39:56.373441+0000","last_peered":"2024-09-16T03:39:56.373441+0000","last_clean":"2024-09-16T03:39:56.373441+0000","last_became_active":"2024-09-16T03:39:09.748391+0000","last_became_peered":"2024-09-16T03:39:09.748391+0000","last_unstale":"2024-09-16T03:39:56.373441+0000","last_undegraded":"2024-09-16T03:39:56.373441+0000","last_fullsized":"2024-09-16T03:39:56.373441+0000","mapping_epoch":134,"log_start":"129'132","ondisk_log_start":"129'132","created":32,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:46.855462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.855462+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":128,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:58.988672+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":17762,"num_write":32,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.a","version":"168'176","reported_seq":839,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.572529+0000","last_change":"2024-09-16T03:39:01.504751+0000","last_active":"2024-09-16T03:39:59.572529+0000","last_peered":"2024-09-16T03:39:59.572529+0000","last_clean":"2024-09-16T03:39:59.572529+0000","last_became_active":"2024-09-16T03:38:46.021805+0000","last_became_peered":"2024-09-16T03:38:46.021805+0000","last_unstale":"2024-09-16T03:39:59.572529+0000","last_undegraded":"2024-09-16T03:39:59.572529+0000","last_fullsized":"2024-09-16T03:39:59.572529+0000","mapping_epoch":32,"log_start":"168'173","ondisk_log_start":"168'173","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'147","last_scrub_stamp":"2024-09-16T03:39:01.504663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.504663+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":170,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.277419+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13126531,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":161,"num_read_kb":166710,"num_write":34,"num_write_kb":9701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":337,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.477045+0000","last_change":"2024-09-16T03:39:30.330195+0000","last_active":"2024-09-16T03:39:58.477045+0000","last_peered":"2024-09-16T03:39:58.477045+0000","last_clean":"2024-09-16T03:39:58.477045+0000","last_became_active":"2024-09-16T03:39:30.229833+0000","last_became_peered":"2024-09-16T03:39:30.229833+0000","last_unstale":"2024-09-16T03:39:58.477045+0000","last_undegraded":"2024-09-16T03:39:58.477045+0000","last_fullsized":"2024-09-16T03:39:58.477045+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:46.881248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:46.881248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:00.421608+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,8],"acting":[14,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.8","version":"152'82","reported_seq":574,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.582117+0000","last_change":"2024-09-16T03:39:31.762814+0000","last_active":"2024-09-16T03:39:58.582117+0000","last_peered":"2024-09-16T03:39:58.582117+0000","last_clean":"2024-09-16T03:39:58.582117+0000","last_became_active":"2024-09-16T03:39:31.762544+0000","last_became_peered":"2024-09-16T03:39:31.762544+0000","last_unstale":"2024-09-16T03:39:58.582117+0000","last_undegraded":"2024-09-16T03:39:58.582117+0000","last_fullsized":"2024-09-16T03:39:58.582117+0000","mapping_epoch":151,"log_start":"140'78","ondisk_log_start":"140'78","created":32,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'71","last_scrub_stamp":"2024-09-16T03:39:02.002247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.002247+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":76,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.244585+0000","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243817,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":3438,"num_write":14,"num_write_kb":3035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"171'100","reported_seq":557,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.683099+0000","last_change":"2024-09-16T03:39:58.682984+0000","last_active":"2024-09-16T03:39:58.683099+0000","last_peered":"2024-09-16T03:39:58.683099+0000","last_clean":"2024-09-16T03:39:58.683099+0000","last_became_active":"2024-09-16T03:39:58.682644+0000","last_became_peered":"2024-09-16T03:39:58.682644+0000","last_unstale":"2024-09-16T03:39:58.683099+0000","last_undegraded":"2024-09-16T03:39:58.683099+0000","last_fullsized":"2024-09-16T03:39:58.683099+0000","mapping_epoch":173,"log_start":"171'97","ondisk_log_start":"171'97","created":32,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'82","last_scrub_stamp":"2024-09-16T03:39:00.900619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.900619+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":95,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.165962+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3714413,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":40079,"num_write":22,"num_write_kb":6258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2417775,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":499,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T03:39:56.377733+0000","last_change":"2024-09-16T03:38:36.757156+0000","last_active":"2024-09-16T03:39:56.377733+0000","last_peered":"2024-09-16T03:39:56.377733+0000","last_clean":"2024-09-16T03:39:56.377733+0000","last_became_active":"2024-09-16T03:38:36.756855+0000","last_became_peered":"2024-09-16T03:38:36.756855+0000","last_unstale":"2024-09-16T03:39:56.377733+0000","last_undegraded":"2024-09-16T03:39:56.377733+0000","last_fullsized":"2024-09-16T03:39:56.377733+0000","mapping_epoch":103,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:38:28.063690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:38:28.063690+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:30.945544+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":206,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.582480+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:39:58.582480+0000","last_peered":"2024-09-16T03:39:58.582480+0000","last_clean":"2024-09-16T03:39:58.582480+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:39:58.582480+0000","last_undegraded":"2024-09-16T03:39:58.582480+0000","last_fullsized":"2024-09-16T03:39:58.582480+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.436956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":715,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T03:39:56.377991+0000","last_change":"2024-09-16T03:39:23.138722+0000","last_active":"2024-09-16T03:39:56.377991+0000","last_peered":"2024-09-16T03:39:56.377991+0000","last_clean":"2024-09-16T03:39:56.377991+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:39:56.377991+0000","last_undegraded":"2024-09-16T03:39:56.377991+0000","last_fullsized":"2024-09-16T03:39:56.377991+0000","mapping_epoch":74,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.823622+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":304,"num_read_kb":9852,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.6","version":"108'44","reported_seq":470,"reported_epoch":174,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:58.478951+0000","last_change":"2024-09-16T03:39:55.909742+0000","last_active":"2024-09-16T03:39:58.478951+0000","last_peered":"2024-09-16T03:39:58.478951+0000","last_clean":"2024-09-16T03:39:52.998142+0000","last_became_active":"2024-09-16T03:39:55.908933+0000","last_became_peered":"2024-09-16T03:39:55.908933+0000","last_unstale":"2024-09-16T03:39:58.478951+0000","last_undegraded":"2024-09-16T03:39:58.478951+0000","last_fullsized":"2024-09-16T03:39:58.478951+0000","mapping_epoch":170,"log_start":"105'41","ondisk_log_start":"105'41","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:48.279224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.279224+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,12],"acting":[10,6,4],"avail_no_missing":["10","4","6","12"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":348,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.572620+0000","last_change":"2024-09-16T03:39:09.749531+0000","last_active":"2024-09-16T03:39:59.572620+0000","last_peered":"2024-09-16T03:39:59.572620+0000","last_clean":"2024-09-16T03:39:59.572620+0000","last_became_active":"2024-09-16T03:39:09.649028+0000","last_became_peered":"2024-09-16T03:39:09.649028+0000","last_unstale":"2024-09-16T03:39:59.572620+0000","last_undegraded":"2024-09-16T03:39:59.572620+0000","last_fullsized":"2024-09-16T03:39:59.572620+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:24.358339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:24.358339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:41.336202+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":597,"reported_epoch":172,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:56.375696+0000","last_change":"2024-09-16T03:39:55.213119+0000","last_active":"2024-09-16T03:39:56.375696+0000","last_peered":"2024-09-16T03:39:56.375696+0000","last_clean":"2024-09-16T03:39:49.975837+0000","last_became_active":"2024-09-16T03:39:54.908050+0000","last_became_peered":"2024-09-16T03:39:54.908050+0000","last_unstale":"2024-09-16T03:39:56.375696+0000","last_undegraded":"2024-09-16T03:39:56.375696+0000","last_fullsized":"2024-09-16T03:39:56.375696+0000","mapping_epoch":169,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'251","last_scrub_stamp":"2024-09-16T03:39:36.474408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:36.474408+0000","objects_scrubbed":37,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":78,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":232,"num_read_kb":2384,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,1],"acting":[6,3,4],"avail_no_missing":["6","3","4"],"object_location_counts":[{"shards":"3,4,6","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"168'90","reported_seq":545,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.378966+0000","last_change":"2024-09-16T03:39:01.297447+0000","last_active":"2024-09-16T03:39:58.378966+0000","last_peered":"2024-09-16T03:39:58.378966+0000","last_clean":"2024-09-16T03:39:58.378966+0000","last_became_active":"2024-09-16T03:38:45.723754+0000","last_became_peered":"2024-09-16T03:38:45.723754+0000","last_unstale":"2024-09-16T03:39:58.378966+0000","last_undegraded":"2024-09-16T03:39:58.378966+0000","last_fullsized":"2024-09-16T03:39:58.378966+0000","mapping_epoch":32,"log_start":"168'87","ondisk_log_start":"168'87","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'72","last_scrub_stamp":"2024-09-16T03:39:01.297324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.297324+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":85,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:26.128089+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7092686,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":98758,"num_write":19,"num_write_kb":5816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T03:39:56.377972+0000","last_change":"2024-09-16T03:39:56.004042+0000","last_active":"2024-09-16T03:39:56.377972+0000","last_peered":"2024-09-16T03:39:56.377972+0000","last_clean":"2024-09-16T03:39:56.377972+0000","last_became_active":"2024-09-16T03:39:56.003773+0000","last_became_peered":"2024-09-16T03:39:56.003773+0000","last_unstale":"2024-09-16T03:39:56.377972+0000","last_undegraded":"2024-09-16T03:39:56.377972+0000","last_fullsized":"2024-09-16T03:39:56.377972+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:27.167621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,7],"acting":[11,14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":904,"reported_epoch":175,"state":"active+clean+remapped","last_fresh":"2024-09-16T03:40:00.042073+0000","last_change":"2024-09-16T03:40:00.042073+0000","last_active":"2024-09-16T03:40:00.042073+0000","last_peered":"2024-09-16T03:40:00.042073+0000","last_clean":"2024-09-16T03:40:00.042073+0000","last_became_active":"2024-09-16T03:39:56.102324+0000","last_became_peered":"2024-09-16T03:39:56.102324+0000","last_unstale":"2024-09-16T03:40:00.042073+0000","last_undegraded":"2024-09-16T03:40:00.042073+0000","last_fullsized":"2024-09-16T03:40:00.042073+0000","mapping_epoch":170,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:59.044102+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":268,"num_read_kb":13655,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":57210536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,6],"acting":[9,1,4],"avail_no_missing":["9","1","4","6"],"object_location_counts":[{"shards":"1,4,9","objects":24}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"137'33","reported_seq":411,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T03:39:56.378040+0000","last_change":"2024-09-16T03:38:47.042020+0000","last_active":"2024-09-16T03:39:56.378040+0000","last_peered":"2024-09-16T03:39:56.378040+0000","last_clean":"2024-09-16T03:39:56.378040+0000","last_became_active":"2024-09-16T03:38:45.820169+0000","last_became_peered":"2024-09-16T03:38:45.820169+0000","last_unstale":"2024-09-16T03:39:56.378040+0000","last_undegraded":"2024-09-16T03:39:56.378040+0000","last_fullsized":"2024-09-16T03:39:56.378040+0000","mapping_epoch":32,"log_start":"87'30","ondisk_log_start":"87'30","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.041851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.041851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":30,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:55.806387+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":48,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.378228+0000","last_change":"2024-09-16T03:39:57.677954+0000","last_active":"2024-09-16T03:39:58.378228+0000","last_peered":"2024-09-16T03:39:58.378228+0000","last_clean":"2024-09-16T03:39:58.378228+0000","last_became_active":"2024-09-16T03:39:57.677624+0000","last_became_peered":"2024-09-16T03:39:57.677624+0000","last_unstale":"2024-09-16T03:39:58.378228+0000","last_undegraded":"2024-09-16T03:39:58.378228+0000","last_fullsized":"2024-09-16T03:39:58.378228+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:39:14.082477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.082477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.802773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":718,"reported_epoch":175,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:39:59.981642+0000","last_change":"2024-09-16T03:39:56.513325+0000","last_active":"2024-09-16T03:39:59.981642+0000","last_peered":"2024-09-16T03:39:59.981642+0000","last_clean":"2024-09-16T03:39:52.800868+0000","last_became_active":"2024-09-16T03:39:56.207437+0000","last_became_peered":"2024-09-16T03:39:56.207437+0000","last_unstale":"2024-09-16T03:39:59.981642+0000","last_undegraded":"2024-09-16T03:39:59.981642+0000","last_fullsized":"2024-09-16T03:39:59.981642+0000","mapping_epoch":170,"log_start":"168'338","ondisk_log_start":"168'338","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":350,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":9718902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[8,2,15],"avail_no_missing":["8","2","15"],"object_location_counts":[{"shards":"2,8,15","objects":39}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":355,"reported_epoch":172,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:39:56.674202+0000","last_change":"2024-09-16T03:39:56.774663+0000","last_active":"2024-09-16T03:39:56.674202+0000","last_peered":"2024-09-16T03:39:56.674202+0000","last_clean":"2024-09-16T03:39:53.801531+0000","last_became_active":"2024-09-16T03:39:56.574007+0000","last_became_peered":"2024-09-16T03:39:56.574007+0000","last_unstale":"2024-09-16T03:39:56.674202+0000","last_undegraded":"2024-09-16T03:39:56.674202+0000","last_fullsized":"2024-09-16T03:39:56.674202+0000","mapping_epoch":171,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:35.828203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.828203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,3],"acting":[15,11,8],"avail_no_missing":["15","3","8","11"],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":324,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T03:39:56.375897+0000","last_change":"2024-09-16T03:39:29.848063+0000","last_active":"2024-09-16T03:39:56.375897+0000","last_peered":"2024-09-16T03:39:56.375897+0000","last_clean":"2024-09-16T03:39:56.375897+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:39:56.375897+0000","last_undegraded":"2024-09-16T03:39:56.375897+0000","last_fullsized":"2024-09-16T03:39:56.375897+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:33.922407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"94'34","reported_seq":420,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T03:39:56.377909+0000","last_change":"2024-09-16T03:38:39.081942+0000","last_active":"2024-09-16T03:39:56.377909+0000","last_peered":"2024-09-16T03:39:56.377909+0000","last_clean":"2024-09-16T03:39:56.377909+0000","last_became_active":"2024-09-16T03:38:33.580022+0000","last_became_peered":"2024-09-16T03:38:33.580022+0000","last_unstale":"2024-09-16T03:39:56.377909+0000","last_undegraded":"2024-09-16T03:39:56.377909+0000","last_fullsized":"2024-09-16T03:39:56.377909+0000","mapping_epoch":96,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:39.081810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.081810+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:39:56.535575+0000","scrub_duration":473,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.475459+0000","last_change":"2024-09-16T03:39:09.848905+0000","last_active":"2024-09-16T03:39:59.475459+0000","last_peered":"2024-09-16T03:39:59.475459+0000","last_clean":"2024-09-16T03:39:59.475459+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:39:59.475459+0000","last_undegraded":"2024-09-16T03:39:59.475459+0000","last_fullsized":"2024-09-16T03:39:59.475459+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:52.781538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"167'156","reported_seq":710,"reported_epoch":173,"state":"peering","last_fresh":"2024-09-16T03:39:57.843752+0000","last_change":"2024-09-16T03:39:58.378783+0000","last_active":"2024-09-16T03:39:58.378783+0000","last_peered":"2024-09-16T03:39:57.843752+0000","last_clean":"2024-09-16T03:39:57.843752+0000","last_became_active":"2024-09-16T03:39:56.514715+0000","last_became_peered":"2024-09-16T03:39:56.514715+0000","last_unstale":"2024-09-16T03:39:57.843752+0000","last_undegraded":"2024-09-16T03:39:57.843752+0000","last_fullsized":"2024-09-16T03:39:57.843752+0000","mapping_epoch":174,"log_start":"167'153","ondisk_log_start":"167'153","created":32,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'118","last_scrub_stamp":"2024-09-16T03:38:39.704919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.704919+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":150,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6958230,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":93961,"num_write":45,"num_write_kb":13895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5176846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15,8],"acting":[10,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,8,15],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":336,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.377256+0000","last_change":"2024-09-16T03:39:39.004001+0000","last_active":"2024-09-16T03:39:58.377256+0000","last_peered":"2024-09-16T03:39:58.377256+0000","last_clean":"2024-09-16T03:39:58.377256+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:39:58.377256+0000","last_undegraded":"2024-09-16T03:39:58.377256+0000","last_fullsized":"2024-09-16T03:39:58.377256+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.017825+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":349,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.576952+0000","last_change":"2024-09-16T03:39:59.297976+0000","last_active":"2024-09-16T03:39:59.576952+0000","last_peered":"2024-09-16T03:39:59.576952+0000","last_clean":"2024-09-16T03:39:59.576952+0000","last_became_active":"2024-09-16T03:38:33.484091+0000","last_became_peered":"2024-09-16T03:38:33.484091+0000","last_unstale":"2024-09-16T03:39:59.576952+0000","last_undegraded":"2024-09-16T03:39:59.576952+0000","last_fullsized":"2024-09-16T03:39:59.576952+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:39:59.297888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:59.297888+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:25.801060+0000","scrub_duration":390,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":323,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.572714+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:39:59.572714+0000","last_peered":"2024-09-16T03:39:59.572714+0000","last_clean":"2024-09-16T03:39:59.572714+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:39:59.572714+0000","last_undegraded":"2024-09-16T03:39:59.572714+0000","last_fullsized":"2024-09-16T03:39:59.572714+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:29.326990+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.10","version":"167'81","reported_seq":508,"reported_epoch":174,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:58.378098+0000","last_change":"2024-09-16T03:39:56.206774+0000","last_active":"2024-09-16T03:39:58.378098+0000","last_peered":"2024-09-16T03:39:58.378098+0000","last_clean":"2024-09-16T03:39:52.997913+0000","last_became_active":"2024-09-16T03:39:56.106140+0000","last_became_peered":"2024-09-16T03:39:56.106140+0000","last_unstale":"2024-09-16T03:39:58.378098+0000","last_undegraded":"2024-09-16T03:39:58.378098+0000","last_fullsized":"2024-09-16T03:39:58.378098+0000","mapping_epoch":170,"log_start":"165'78","ondisk_log_start":"165'78","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"118'64","last_scrub_stamp":"2024-09-16T03:39:04.442556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.442556+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":75,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3411887,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":20,"num_write_kb":5652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,11],"acting":[10,3,4],"avail_no_missing":["10","3","4"],"object_location_counts":[{"shards":"3,4,10","objects":1}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"167'26","reported_seq":403,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-16T03:39:55.917282+0000","last_change":"2024-09-16T03:39:03.929961+0000","last_active":"2024-09-16T03:39:55.917282+0000","last_peered":"2024-09-16T03:39:55.917282+0000","last_clean":"2024-09-16T03:39:55.917282+0000","last_became_active":"2024-09-16T03:39:01.696690+0000","last_became_peered":"2024-09-16T03:39:01.696690+0000","last_unstale":"2024-09-16T03:39:55.917282+0000","last_undegraded":"2024-09-16T03:39:55.917282+0000","last_fullsized":"2024-09-16T03:39:55.917282+0000","mapping_epoch":63,"log_start":"155'22","ondisk_log_start":"155'22","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:03.929863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.929863+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":22,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:13.702587+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3579957,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":37538,"num_write":10,"num_write_kb":3021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"147'23","reported_seq":487,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.376976+0000","last_change":"2024-09-16T03:39:14.925442+0000","last_active":"2024-09-16T03:39:58.376976+0000","last_peered":"2024-09-16T03:39:58.376976+0000","last_clean":"2024-09-16T03:39:58.376976+0000","last_became_active":"2024-09-16T03:39:13.974170+0000","last_became_peered":"2024-09-16T03:39:13.974170+0000","last_unstale":"2024-09-16T03:39:58.376976+0000","last_undegraded":"2024-09-16T03:39:58.376976+0000","last_fullsized":"2024-09-16T03:39:58.376976+0000","mapping_epoch":138,"log_start":"127'20","ondisk_log_start":"127'20","created":50,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'20","last_scrub_stamp":"2024-09-16T03:39:14.925356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.925356+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":20,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.341117+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2000099,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5461,"num_write":1,"num_write_kb":458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.13","version":"165'83","reported_seq":573,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.577026+0000","last_change":"2024-09-16T03:39:15.224056+0000","last_active":"2024-09-16T03:39:59.577026+0000","last_peered":"2024-09-16T03:39:59.577026+0000","last_clean":"2024-09-16T03:39:59.577026+0000","last_became_active":"2024-09-16T03:39:15.223614+0000","last_became_peered":"2024-09-16T03:39:15.223614+0000","last_unstale":"2024-09-16T03:39:59.577026+0000","last_undegraded":"2024-09-16T03:39:59.577026+0000","last_fullsized":"2024-09-16T03:39:59.577026+0000","mapping_epoch":139,"log_start":"154'79","ondisk_log_start":"154'79","created":50,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'73","last_scrub_stamp":"2024-09-16T03:39:03.124583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.124583+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":77,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.132930+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445501,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":11846,"num_write":10,"num_write_kb":2874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"104'20","reported_seq":397,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-16T03:39:55.917452+0000","last_change":"2024-09-16T03:39:01.898409+0000","last_active":"2024-09-16T03:39:55.917452+0000","last_peered":"2024-09-16T03:39:55.917452+0000","last_clean":"2024-09-16T03:39:55.917452+0000","last_became_active":"2024-09-16T03:39:01.596287+0000","last_became_peered":"2024-09-16T03:39:01.596287+0000","last_unstale":"2024-09-16T03:39:55.917452+0000","last_undegraded":"2024-09-16T03:39:55.917452+0000","last_fullsized":"2024-09-16T03:39:55.917452+0000","mapping_epoch":114,"log_start":"98'17","ondisk_log_start":"98'17","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:01.898331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.898331+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.728364+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"123'65","reported_seq":559,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.680665+0000","last_change":"2024-09-16T03:39:14.175562+0000","last_active":"2024-09-16T03:39:59.680665+0000","last_peered":"2024-09-16T03:39:59.680665+0000","last_clean":"2024-09-16T03:39:59.680665+0000","last_became_active":"2024-09-16T03:39:13.671769+0000","last_became_peered":"2024-09-16T03:39:13.671769+0000","last_unstale":"2024-09-16T03:39:59.680665+0000","last_undegraded":"2024-09-16T03:39:59.680665+0000","last_fullsized":"2024-09-16T03:39:59.680665+0000","mapping_epoch":138,"log_start":"123'62","ondisk_log_start":"123'62","created":50,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"123'65","last_scrub_stamp":"2024-09-16T03:39:14.175493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.175493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:36.855683+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"124'86","reported_seq":529,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T03:39:56.373478+0000","last_change":"2024-09-16T03:39:03.073530+0000","last_active":"2024-09-16T03:39:56.373478+0000","last_peered":"2024-09-16T03:39:56.373478+0000","last_clean":"2024-09-16T03:39:56.373478+0000","last_became_active":"2024-09-16T03:39:01.797805+0000","last_became_peered":"2024-09-16T03:39:01.797805+0000","last_unstale":"2024-09-16T03:39:56.373478+0000","last_undegraded":"2024-09-16T03:39:56.373478+0000","last_fullsized":"2024-09-16T03:39:56.373478+0000","mapping_epoch":115,"log_start":"124'82","ondisk_log_start":"124'82","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.073459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.073459+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":80,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:30.483816+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.17","version":"169'60","reported_seq":504,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.477302+0000","last_change":"2024-09-16T03:39:35.812368+0000","last_active":"2024-09-16T03:39:58.477302+0000","last_peered":"2024-09-16T03:39:58.477302+0000","last_clean":"2024-09-16T03:39:58.477302+0000","last_became_active":"2024-09-16T03:39:03.717420+0000","last_became_peered":"2024-09-16T03:39:03.717420+0000","last_unstale":"2024-09-16T03:39:58.477302+0000","last_undegraded":"2024-09-16T03:39:58.477302+0000","last_fullsized":"2024-09-16T03:39:58.477302+0000","mapping_epoch":128,"log_start":"148'56","ondisk_log_start":"148'56","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"148'56","last_scrub_stamp":"2024-09-16T03:39:35.812298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:35.812298+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":55,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:54.749022+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731220,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"155'55","reported_seq":464,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.572582+0000","last_change":"2024-09-16T03:39:02.668794+0000","last_active":"2024-09-16T03:39:59.572582+0000","last_peered":"2024-09-16T03:39:59.572582+0000","last_clean":"2024-09-16T03:39:59.572582+0000","last_became_active":"2024-09-16T03:38:47.562641+0000","last_became_peered":"2024-09-16T03:38:47.562641+0000","last_unstale":"2024-09-16T03:39:59.572582+0000","last_undegraded":"2024-09-16T03:39:59.572582+0000","last_fullsized":"2024-09-16T03:39:59.572582+0000","mapping_epoch":112,"log_start":"155'52","ondisk_log_start":"155'52","created":50,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'48","last_scrub_stamp":"2024-09-16T03:39:02.668681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.668681+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:14.665737+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3344659,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":25041,"num_write":28,"num_write_kb":7736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":93,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.575338+0000","last_change":"2024-09-16T03:39:09.847101+0000","last_active":"2024-09-16T03:39:59.575338+0000","last_peered":"2024-09-16T03:39:59.575338+0000","last_clean":"2024-09-16T03:39:59.575338+0000","last_became_active":"2024-09-16T03:39:09.846691+0000","last_became_peered":"2024-09-16T03:39:09.846691+0000","last_unstale":"2024-09-16T03:39:59.575338+0000","last_undegraded":"2024-09-16T03:39:59.575338+0000","last_fullsized":"2024-09-16T03:39:59.575338+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:01.106840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.106840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:27.531290+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"169'138","reported_seq":672,"reported_epoch":172,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:56.375790+0000","last_change":"2024-09-16T03:39:54.206127+0000","last_active":"2024-09-16T03:39:56.375790+0000","last_peered":"2024-09-16T03:39:56.375790+0000","last_clean":"2024-09-16T03:39:28.503345+0000","last_became_active":"2024-09-16T03:39:54.099638+0000","last_became_peered":"2024-09-16T03:39:54.099638+0000","last_unstale":"2024-09-16T03:39:56.375790+0000","last_undegraded":"2024-09-16T03:39:56.375790+0000","last_fullsized":"2024-09-16T03:39:56.375790+0000","mapping_epoch":168,"log_start":"148'134","ondisk_log_start":"148'134","created":100,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'115","last_scrub_stamp":"2024-09-16T03:38:35.254488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.254488+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":134,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6756546,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":51953,"num_write":24,"num_write_kb":6981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3860936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,14],"acting":[6,11,15],"avail_no_missing":["6","11","15"],"object_location_counts":[{"shards":"6,11,15","objects":2}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"147'136","reported_seq":721,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.680733+0000","last_change":"2024-09-16T03:39:58.681289+0000","last_active":"2024-09-16T03:39:59.680733+0000","last_peered":"2024-09-16T03:39:59.680733+0000","last_clean":"2024-09-16T03:39:59.680733+0000","last_became_active":"2024-09-16T03:39:58.580803+0000","last_became_peered":"2024-09-16T03:39:58.580803+0000","last_unstale":"2024-09-16T03:39:59.680733+0000","last_undegraded":"2024-09-16T03:39:59.680733+0000","last_fullsized":"2024-09-16T03:39:59.680733+0000","mapping_epoch":173,"log_start":"139'133","ondisk_log_start":"139'133","created":100,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'107","last_scrub_stamp":"2024-09-16T03:38:39.905697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.905697+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":127,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4941390,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":116543,"num_write":32,"num_write_kb":9493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5788673,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,10],"acting":[3,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"99'56","reported_seq":471,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-16T03:39:56.575969+0000","last_change":"2024-09-16T03:39:33.570828+0000","last_active":"2024-09-16T03:39:56.575969+0000","last_peered":"2024-09-16T03:39:56.575969+0000","last_clean":"2024-09-16T03:39:56.575969+0000","last_became_active":"2024-09-16T03:39:33.570516+0000","last_became_peered":"2024-09-16T03:39:33.570516+0000","last_unstale":"2024-09-16T03:39:56.575969+0000","last_undegraded":"2024-09-16T03:39:56.575969+0000","last_fullsized":"2024-09-16T03:39:56.575969+0000","mapping_epoch":153,"log_start":"90'53","ondisk_log_start":"90'53","created":100,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":5,"last_scrub":"99'56","last_scrub_stamp":"2024-09-16T03:39:00.988517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.988517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.293621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"92'14","reported_seq":362,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.580593+0000","last_change":"2024-09-16T03:38:49.092747+0000","last_active":"2024-09-16T03:39:59.580593+0000","last_peered":"2024-09-16T03:39:59.580593+0000","last_clean":"2024-09-16T03:39:59.580593+0000","last_became_active":"2024-09-16T03:38:48.661213+0000","last_became_peered":"2024-09-16T03:38:48.661213+0000","last_unstale":"2024-09-16T03:39:59.580593+0000","last_undegraded":"2024-09-16T03:39:59.580593+0000","last_fullsized":"2024-09-16T03:39:59.580593+0000","mapping_epoch":113,"log_start":"91'11","ondisk_log_start":"91'11","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:49.092687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.092687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:14.647233+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,12],"acting":[3,11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"92'59","reported_seq":477,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.377224+0000","last_change":"2024-09-16T03:39:48.664382+0000","last_active":"2024-09-16T03:39:58.377224+0000","last_peered":"2024-09-16T03:39:58.377224+0000","last_clean":"2024-09-16T03:39:58.377224+0000","last_became_active":"2024-09-16T03:38:48.561170+0000","last_became_peered":"2024-09-16T03:38:48.561170+0000","last_unstale":"2024-09-16T03:39:58.377224+0000","last_undegraded":"2024-09-16T03:39:58.377224+0000","last_fullsized":"2024-09-16T03:39:58.377224+0000","mapping_epoch":113,"log_start":"89'56","ondisk_log_start":"89'56","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'59","last_scrub_stamp":"2024-09-16T03:39:48.664277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:48.664277+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.699198+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,6],"acting":[14,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1f","version":"169'132","reported_seq":681,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.378907+0000","last_change":"2024-09-16T03:39:56.619547+0000","last_active":"2024-09-16T03:39:58.378907+0000","last_peered":"2024-09-16T03:39:58.378907+0000","last_clean":"2024-09-16T03:39:58.378907+0000","last_became_active":"2024-09-16T03:39:56.619304+0000","last_became_peered":"2024-09-16T03:39:56.619304+0000","last_unstale":"2024-09-16T03:39:58.378907+0000","last_undegraded":"2024-09-16T03:39:58.378907+0000","last_fullsized":"2024-09-16T03:39:58.378907+0000","mapping_epoch":171,"log_start":"165'128","ondisk_log_start":"165'128","created":100,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'95","last_scrub_stamp":"2024-09-16T03:38:37.201959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:37.201959+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":128,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.586143+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10386891,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":131,"num_read_kb":122162,"num_write":49,"num_write_kb":13103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":13164419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,12],"acting":[10,1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.20","version":"170'165","reported_seq":768,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.572452+0000","last_change":"2024-09-16T03:39:49.993969+0000","last_active":"2024-09-16T03:39:59.572452+0000","last_peered":"2024-09-16T03:39:59.572452+0000","last_clean":"2024-09-16T03:39:59.572452+0000","last_became_active":"2024-09-16T03:39:49.893618+0000","last_became_peered":"2024-09-16T03:39:49.893618+0000","last_unstale":"2024-09-16T03:39:59.572452+0000","last_undegraded":"2024-09-16T03:39:59.572452+0000","last_fullsized":"2024-09-16T03:39:59.572452+0000","mapping_epoch":165,"log_start":"169'162","ondisk_log_start":"169'162","created":100,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'118","last_scrub_stamp":"2024-09-16T03:38:36.698150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:36.698150+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":162,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:55.225728+0000","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11378504,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":184,"num_read_kb":187520,"num_write":53,"num_write_kb":16191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6817236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[8,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.21","version":"48'7","reported_seq":222,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.376863+0000","last_change":"2024-09-16T03:38:49.064947+0000","last_active":"2024-09-16T03:39:58.376863+0000","last_peered":"2024-09-16T03:39:58.376863+0000","last_clean":"2024-09-16T03:39:58.376863+0000","last_became_active":"2024-09-16T03:38:48.559431+0000","last_became_peered":"2024-09-16T03:38:48.559431+0000","last_unstale":"2024-09-16T03:39:58.376863+0000","last_undegraded":"2024-09-16T03:39:58.376863+0000","last_fullsized":"2024-09-16T03:39:58.376863+0000","mapping_epoch":113,"log_start":"40'4","ondisk_log_start":"40'4","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":6,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:49.064870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064870+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:03.082107+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,10],"acting":[14,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.22","version":"45'6","reported_seq":227,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.475340+0000","last_change":"2024-09-16T03:39:13.974174+0000","last_active":"2024-09-16T03:39:59.475340+0000","last_peered":"2024-09-16T03:39:59.475340+0000","last_clean":"2024-09-16T03:39:59.475340+0000","last_became_active":"2024-09-16T03:39:13.973813+0000","last_became_peered":"2024-09-16T03:39:13.973813+0000","last_unstale":"2024-09-16T03:39:59.475340+0000","last_undegraded":"2024-09-16T03:39:59.475340+0000","last_fullsized":"2024-09-16T03:39:59.475340+0000","mapping_epoch":138,"log_start":"39'3","ondisk_log_start":"39'3","created":100,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:39:00.968448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.968448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:30.310727+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,6],"acting":[9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.23","version":"143'42","reported_seq":449,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.377012+0000","last_change":"2024-09-16T03:38:49.464614+0000","last_active":"2024-09-16T03:39:58.377012+0000","last_peered":"2024-09-16T03:39:58.377012+0000","last_clean":"2024-09-16T03:39:58.377012+0000","last_became_active":"2024-09-16T03:38:49.463797+0000","last_became_peered":"2024-09-16T03:38:49.463797+0000","last_unstale":"2024-09-16T03:39:58.377012+0000","last_undegraded":"2024-09-16T03:39:58.377012+0000","last_fullsized":"2024-09-16T03:39:58.377012+0000","mapping_epoch":114,"log_start":"142'39","ondisk_log_start":"142'39","created":100,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:38:34.893524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:34.893524+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":38,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672379,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":24141,"num_write":9,"num_write_kb":2300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,2],"acting":[14,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.24","version":"105'32","reported_seq":254,"reported_epoch":171,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:55.917237+0000","last_change":"2024-09-16T03:39:39.285524+0000","last_active":"2024-09-16T03:39:55.917237+0000","last_peered":"2024-09-16T03:39:55.917237+0000","last_clean":"2024-09-16T03:38:17.396491+0000","last_became_active":"2024-09-16T03:39:38.786889+0000","last_became_peered":"2024-09-16T03:39:38.786889+0000","last_unstale":"2024-09-16T03:39:55.917237+0000","last_undegraded":"2024-09-16T03:39:55.917237+0000","last_fullsized":"2024-09-16T03:39:55.917237+0000","mapping_epoch":158,"log_start":"83'29","ondisk_log_start":"83'29","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.978888+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":4728,"num_write":10,"num_write_kb":2338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,14],"acting":[2,11,15],"avail_no_missing":["2","0","3","11","14","15"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"169'86","reported_seq":464,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.377238+0000","last_change":"2024-09-16T03:39:40.798757+0000","last_active":"2024-09-16T03:39:58.377238+0000","last_peered":"2024-09-16T03:39:58.377238+0000","last_clean":"2024-09-16T03:39:58.377238+0000","last_became_active":"2024-09-16T03:39:40.798223+0000","last_became_peered":"2024-09-16T03:39:40.798223+0000","last_unstale":"2024-09-16T03:39:58.377238+0000","last_undegraded":"2024-09-16T03:39:58.377238+0000","last_fullsized":"2024-09-16T03:39:58.377238+0000","mapping_epoch":160,"log_start":"130'83","ondisk_log_start":"130'83","created":110,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'77","last_scrub_stamp":"2024-09-16T03:38:48.088768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.088768+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":80,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:14.812409+0000","scrub_duration":388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2134555,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":10,"num_write_kb":2500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1126280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,7,14],"acting":[12,7,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.26","version":"169'64","reported_seq":468,"reported_epoch":172,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:56.376012+0000","last_change":"2024-09-16T03:39:38.986370+0000","last_active":"2024-09-16T03:39:56.376012+0000","last_peered":"2024-09-16T03:39:56.376012+0000","last_clean":"2024-09-16T03:39:19.447260+0000","last_became_active":"2024-09-16T03:39:38.985600+0000","last_became_peered":"2024-09-16T03:39:38.985600+0000","last_unstale":"2024-09-16T03:39:56.376012+0000","last_undegraded":"2024-09-16T03:39:56.376012+0000","last_fullsized":"2024-09-16T03:39:56.376012+0000","mapping_epoch":158,"log_start":"142'60","ondisk_log_start":"142'60","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:49.477297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.477297+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":59,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3647185,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":24205,"num_write":24,"num_write_kb":7094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,10],"acting":[6,10,4],"avail_no_missing":["6","4","10"],"object_location_counts":[{"shards":"4,6,10","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"109'66","reported_seq":363,"reported_epoch":172,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:56.376182+0000","last_change":"2024-09-16T03:39:38.989488+0000","last_active":"2024-09-16T03:39:56.376182+0000","last_peered":"2024-09-16T03:39:56.376182+0000","last_clean":"2024-09-16T03:38:43.524716+0000","last_became_active":"2024-09-16T03:39:38.888633+0000","last_became_peered":"2024-09-16T03:39:38.888633+0000","last_unstale":"2024-09-16T03:39:56.376182+0000","last_undegraded":"2024-09-16T03:39:56.376182+0000","last_fullsized":"2024-09-16T03:39:56.376182+0000","mapping_epoch":158,"log_start":"105'63","ondisk_log_start":"105'63","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'66","last_scrub_stamp":"2024-09-16T03:38:47.511252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.511252+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2512680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":54642,"num_write":20,"num_write_kb":5643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[6,10,11],"avail_no_missing":["6","10","11"],"object_location_counts":[{"shards":"6,10,11","objects":1}],"blocked_by":[],"up_primary":8,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.28","version":"171'70","reported_seq":632,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.575382+0000","last_change":"2024-09-16T03:39:41.471715+0000","last_active":"2024-09-16T03:39:59.575382+0000","last_peered":"2024-09-16T03:39:59.575382+0000","last_clean":"2024-09-16T03:39:59.575382+0000","last_became_active":"2024-09-16T03:39:41.471298+0000","last_became_peered":"2024-09-16T03:39:41.471298+0000","last_unstale":"2024-09-16T03:39:59.575382+0000","last_undegraded":"2024-09-16T03:39:59.575382+0000","last_fullsized":"2024-09-16T03:39:59.575382+0000","mapping_epoch":161,"log_start":"118'67","ondisk_log_start":"118'67","created":110,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:48.551929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.551929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":66,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:12.543779+0000","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395113,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":70,"num_read_kb":76288,"num_write":5,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":14443776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,2],"acting":[9,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":44,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.378114+0000","last_change":"2024-09-16T03:39:37.881274+0000","last_active":"2024-09-16T03:39:58.378114+0000","last_peered":"2024-09-16T03:39:58.378114+0000","last_clean":"2024-09-16T03:39:58.378114+0000","last_became_active":"2024-09-16T03:39:37.880874+0000","last_became_peered":"2024-09-16T03:39:37.880874+0000","last_unstale":"2024-09-16T03:39:58.378114+0000","last_undegraded":"2024-09-16T03:39:58.378114+0000","last_fullsized":"2024-09-16T03:39:58.378114+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:00.834742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.834742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:21.681138+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,2],"acting":[10,14,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2a","version":"108'122","reported_seq":565,"reported_epoch":175,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:39:59.580264+0000","last_change":"2024-09-16T03:39:39.190319+0000","last_active":"2024-09-16T03:39:59.580264+0000","last_peered":"2024-09-16T03:39:59.580264+0000","last_clean":"2024-09-16T03:38:41.455596+0000","last_became_active":"2024-09-16T03:39:38.888612+0000","last_became_peered":"2024-09-16T03:39:38.888612+0000","last_unstale":"2024-09-16T03:39:59.580264+0000","last_undegraded":"2024-09-16T03:39:59.580264+0000","last_fullsized":"2024-09-16T03:39:59.580264+0000","mapping_epoch":158,"log_start":"107'115","ondisk_log_start":"107'115","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'122","last_scrub_stamp":"2024-09-16T03:38:47.964439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.964439+0000","objects_scrubbed":4,"log_size":7,"log_dups_size":112,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647218,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":6805,"num_write":7,"num_write_kb":1927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,12],"acting":[3,8,10],"avail_no_missing":["3","8","10"],"object_location_counts":[{"shards":"3,8,10","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2b","version":"149'132","reported_seq":684,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.777141+0000","last_change":"2024-09-16T03:39:41.669420+0000","last_active":"2024-09-16T03:39:59.777141+0000","last_peered":"2024-09-16T03:39:59.777141+0000","last_clean":"2024-09-16T03:39:59.777141+0000","last_became_active":"2024-09-16T03:39:41.669036+0000","last_became_peered":"2024-09-16T03:39:41.669036+0000","last_unstale":"2024-09-16T03:39:59.777141+0000","last_undegraded":"2024-09-16T03:39:59.777141+0000","last_fullsized":"2024-09-16T03:39:59.777141+0000","mapping_epoch":161,"log_start":"144'128","ondisk_log_start":"144'128","created":110,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:49.064061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064061+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":128,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:55.831512+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092554,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":5914,"num_write":15,"num_write_kb":4670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1630666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2c","version":"167'69","reported_seq":556,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-09-16T03:39:59.680501+0000","last_change":"2024-09-16T03:39:41.471606+0000","last_active":"2024-09-16T03:39:59.680501+0000","last_peered":"2024-09-16T03:39:59.680501+0000","last_clean":"2024-09-16T03:39:59.680501+0000","last_became_active":"2024-09-16T03:39:41.471156+0000","last_became_peered":"2024-09-16T03:39:41.471156+0000","last_unstale":"2024-09-16T03:39:59.680501+0000","last_undegraded":"2024-09-16T03:39:59.680501+0000","last_fullsized":"2024-09-16T03:39:59.680501+0000","mapping_epoch":161,"log_start":"154'65","ondisk_log_start":"154'65","created":110,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'58","last_scrub_stamp":"2024-09-16T03:38:47.889276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.889276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":64,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:16.605878+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3495922,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":43795,"num_write":10,"num_write_kb":2812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3787209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"92'14","reported_seq":423,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-16T03:39:58.377125+0000","last_change":"2024-09-16T03:39:40.897844+0000","last_active":"2024-09-16T03:39:58.377125+0000","last_peered":"2024-09-16T03:39:58.377125+0000","last_clean":"2024-09-16T03:39:58.377125+0000","last_became_active":"2024-09-16T03:39:40.897531+0000","last_became_peered":"2024-09-16T03:39:40.897531+0000","last_unstale":"2024-09-16T03:39:58.377125+0000","last_undegraded":"2024-09-16T03:39:58.377125+0000","last_fullsized":"2024-09-16T03:39:58.377125+0000","mapping_epoch":160,"log_start":"91'11","ondisk_log_start":"91'11","created":110,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":6,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:47.115651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.115651+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:56.558775+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,4],"acting":[14,11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":56,"stat_sum":{"num_bytes":166501715,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1833,"num_read_kb":1681230,"num_write":648,"num_write_kb":181404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":71482417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":24,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":293896192,"data_stored":293203250,"data_compressed":627765,"data_compressed_allocated":688128,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":168,"acting":168,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":96,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1117,"num_read_kb":34146,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":79327941,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211935232,"data_stored":226377329,"data_compressed":13916980,"data_compressed_allocated":14848000,"data_compressed_original":29925376,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":2644640,"data_compressed":159066,"data_compressed_allocated":204800,"data_compressed_original":2584576,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247847,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75464,"kb_used_data":42720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94296376,"statfs":{"total":96636764160,"available":96559489024,"internally_reserved":0,"allocated":43745280,"data_stored":44067988,"data_compressed":1346028,"data_compressed_allocated":1449984,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247848,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63764,"kb_used_data":31020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94308076,"statfs":{"total":96636764160,"available":96571469824,"internally_reserved":0,"allocated":31764480,"data_stored":32878054,"data_compressed":1075067,"data_compressed_allocated":1191936,"data_compressed_original":3039232,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247847,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79072,"kb_used_data":43720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94292768,"statfs":{"total":96636764160,"available":96555794432,"internally_reserved":0,"allocated":44769280,"data_stored":45637722,"data_compressed":1275314,"data_compressed_allocated":1404928,"data_compressed_original":3039232,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280553,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71728,"kb_used_data":40540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94300112,"statfs":{"total":96636764160,"available":96563314688,"internally_reserved":0,"allocated":41512960,"data_stored":43085209,"data_compressed":1695959,"data_compressed_allocated":1851392,"data_compressed_original":4214784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280553,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280552,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56024,"kb_used_data":24836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94315816,"statfs":{"total":96636764160,"available":96579395584,"internally_reserved":0,"allocated":25432064,"data_stored":25830063,"data_compressed":955046,"data_compressed_allocated":1036288,"data_compressed_original":2113536,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313257,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43124,"kb_used_data":14524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328716,"statfs":{"total":96636764160,"available":96592605184,"internally_reserved":0,"allocated":14872576,"data_stored":14462211,"data_compressed":204265,"data_compressed_allocated":249856,"data_compressed_original":499712,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280553,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83112,"kb_used_data":47764,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94288728,"statfs":{"total":96636764160,"available":96551657472,"internally_reserved":0,"allocated":48910336,"data_stored":49615599,"data_compressed":1248643,"data_compressed_allocated":1384448,"data_compressed_original":2924544,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280552,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102764,"kb_used_data":67416,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94269076,"statfs":{"total":96636764160,"available":96531533824,"internally_reserved":0,"allocated":69033984,"data_stored":71750370,"data_compressed":2961533,"data_compressed_allocated":3207168,"data_compressed_original":6787072,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313257,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280552,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41368,"kb_used_data":8616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330472,"statfs":{"total":96636764160,"available":96594403328,"internally_reserved":0,"allocated":8822784,"data_stored":8324017,"data_compressed":83668,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280552,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84568,"kb_used_data":49220,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94287272,"statfs":{"total":96636764160,"available":96550166528,"internally_reserved":0,"allocated":50401280,"data_stored":50750812,"data_compressed":1311501,"data_compressed_allocated":1417216,"data_compressed_original":2564096,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280552,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61084,"kb_used_data":29896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310756,"statfs":{"total":96636764160,"available":96574214144,"internally_reserved":0,"allocated":30613504,"data_stored":31490896,"data_compressed":1003619,"data_compressed_allocated":1097728,"data_compressed_original":2711552,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280553,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81072,"kb_used_data":48320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94290768,"statfs":{"total":96636764160,"available":96553746432,"internally_reserved":0,"allocated":49479680,"data_stored":48986715,"data_compressed":249352,"data_compressed_allocated":299008,"data_compressed_original":630784,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,2,3,4,6,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313256,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46592,"kb_used_data":15400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325248,"statfs":{"total":96636764160,"available":96589053952,"internally_reserved":0,"allocated":15769600,"data_stored":15213387,"data_compressed":106457,"data_compressed_allocated":143360,"data_compressed_original":303104,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247848,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72192,"kb_used_data":40620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94299648,"statfs":{"total":96636764160,"available":96562839552,"internally_reserved":0,"allocated":41594880,"data_stored":42203949,"data_compressed":1483322,"data_compressed_allocated":1593344,"data_compressed_original":2924544,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24895488,"data_stored":26587185,"data_compressed":1581043,"data_compressed_allocated":1691648,"data_compressed_original":3465216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15872000,"data_stored":16957807,"data_compressed":1325719,"data_compressed_allocated":1396736,"data_compressed_original":2523136,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":48046080,"data_stored":51351621,"data_compressed":2874927,"data_compressed_allocated":3084288,"data_compressed_original":6541312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":19632128,"data_stored":21049778,"data_compressed":1187240,"data_compressed_allocated":1282048,"data_compressed_original":2752512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14860288,"data_stored":14775225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9277440,"data_stored":9253748,"data_compressed":22789,"data_compressed_allocated":24576,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24104960,"data_stored":24092098,"data_compressed":85710,"data_compressed_allocated":94208,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30834688,"data_stored":30754602,"data_compressed":94382,"data_compressed_allocated":102400,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":19304448,"data_stored":19197855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":23490560,"data_stored":23414372,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7184384,"data_stored":7135967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":34025472,"data_stored":33912667,"data_compressed":9548,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14385152,"data_stored":14378215,"data_compressed":35763,"data_compressed_allocated":40960,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":47800320,"data_stored":47794347,"data_compressed":165684,"data_compressed_allocated":180224,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12890112,"data_stored":12877307,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":14069760,"data_stored":14019073,"data_compressed":22789,"data_compressed_allocated":24576,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13270816,"data_compressed":120597,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":27377664,"data_stored":27246918,"data_compressed":49947,"data_compressed_allocated":53248,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:00.663 INFO:tasks.thrashosds.thrasher:Setting 4.24 to [15, 0, 14] 2024-09-16T03:40:00.664 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.24', '15', '0', '14'] 2024-09-16T03:40:00.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 4.24 15 0 14 2024-09-16T03:40:00.667 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:40:00.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:40:00.715 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:40:00.714+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:40:00.771 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:40:00.772 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [], 2024-09-16T03:40:00.772 INFO:teuthology.orchestra.run.smithi155.stdout: "complaint_time": 30, 2024-09-16T03:40:00.772 INFO:teuthology.orchestra.run.smithi155.stdout: "num_blocked_ops": 0 2024-09-16T03:40:00.772 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:40:00.780 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:40:00.825 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:40:00.824+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:40:00.835 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:40:00.835 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:00.852 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:40:00.852 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:00.869 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:40:00.869 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:00.887 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:40:00.887 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:40:00.887 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:40:00.887 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:40:00.887 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.887 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:1704 3.1 3:b6a94245:::f84dc3041e0041180b045402a4556a8ca4a929f7ad374eab28b75e0cdc993c99:head [write 0~453120] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e65)", 2024-09-16T03:40:00.887 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:40:00.888 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 130.09093069799999, 2024-09-16T03:40:00.888 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40403313699999999, 2024-09-16T03:40:00.888 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.888 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.888 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.888 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:00.888 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:00.888 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:00.888 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 1704 2024-09-16T03:40:00.888 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.888 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.888 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.888 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:40:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792838+0000", 2024-09-16T03:40:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.792841+0000", 2024-09-16T03:40:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.433e-06 2024-09-16T03:40:00.889 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793329+0000", 2024-09-16T03:40:00.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00048839200000000004 2024-09-16T03:40:00.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793345+0000", 2024-09-16T03:40:00.890 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6079e-05 2024-09-16T03:40:00.890 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.890 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.890 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.890 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793354+0000", 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.32e-06 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793418+0000", 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3944999999999998e-05 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793549+0000", 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013159900000000001 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.891 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:40:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.793666+0000", 2024-09-16T03:40:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000116532 2024-09-16T03:40:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.797707+0000", 2024-09-16T03:40:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0040409629999999998 2024-09-16T03:40:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:50.897288+0000", 2024-09-16T03:40:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099581449000000002 2024-09-16T03:40:00.892 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196775+0000", 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948696800000002 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196814+0000", 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8336e-05 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:51.196871+0000", 2024-09-16T03:40:00.893 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7120999999999997e-05 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:141 3.1 3:9387c678:::99a87d3b555b50e905a1ff55d99eb4aa025bd0070f07a4910d03cf1af95bbc44:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 104.769042816, 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50160030300000003, 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.894 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:40:00.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.895 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114725+0000", 2024-09-16T03:40:00.895 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.895 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.895 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.895 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114726+0000", 2024-09-16T03:40:00.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4399999999999999e-07 2024-09-16T03:40:00.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114730+0000", 2024-09-16T03:40:00.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1479999999999999e-06 2024-09-16T03:40:00.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.896 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.896 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.896 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114738+0000", 2024-09-16T03:40:00.896 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3200000000000002e-06 2024-09-16T03:40:00.896 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114743+0000", 2024-09-16T03:40:00.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.891e-06 2024-09-16T03:40:00.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215320+0000", 2024-09-16T03:40:00.897 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100576463 2024-09-16T03:40:00.897 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.897 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.897 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.897 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215376+0000", 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6310999999999997e-05 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215473+0000", 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7103999999999997e-05 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.216108+0000", 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00063422800000000003 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.898 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513851+0000", 2024-09-16T03:40:00.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29774376400000002 2024-09-16T03:40:00.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616086+0000", 2024-09-16T03:40:00.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10223441 2024-09-16T03:40:00.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.899 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.899 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.899 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616240+0000", 2024-09-16T03:40:00.899 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015453799999999999 2024-09-16T03:40:00.899 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.900 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.900 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616326+0000", 2024-09-16T03:40:00.900 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5382000000000003e-05 2024-09-16T03:40:00.900 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.900 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.900 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.900 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.900 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.900 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:142 3.1 3:a93b7b74:::70cd8f47898d416018aded7de9c29e0fe002b2336cb6cfabbf3eb6238766c3b1:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 104.769013429, 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50166088799999997, 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114755+0000", 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3600000000000003e-07 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114762+0000", 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6039999999999998e-06 2024-09-16T03:40:00.902 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114769+0000", 2024-09-16T03:40:00.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5939999999999999e-06 2024-09-16T03:40:00.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114774+0000", 2024-09-16T03:40:00.903 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2179999999999998e-06 2024-09-16T03:40:00.903 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.903 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.903 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.903 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215687+0000", 2024-09-16T03:40:00.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100912974 2024-09-16T03:40:00.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215733+0000", 2024-09-16T03:40:00.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6419999999999999e-05 2024-09-16T03:40:00.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.904 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:40:00.904 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.215836+0000", 2024-09-16T03:40:00.904 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000102981 2024-09-16T03:40:00.904 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.904 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316766+0000", 2024-09-16T03:40:00.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100929693 2024-09-16T03:40:00.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.513877+0000", 2024-09-16T03:40:00.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19711067800000001 2024-09-16T03:40:00.905 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.905 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.905 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.905 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616358+0000", 2024-09-16T03:40:00.905 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10248180699999999 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616391+0000", 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.2551000000000001e-05 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.616416+0000", 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4732e-05 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.906 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:143 3.1 3:ac848b49:::adc4f0c8a4d0bc7e52f9d5c3b4981447737a7dbe58b8313a369b04383e10893f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 104.76898342299999, 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60333394699999998, 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.907 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:40:00.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114785+0000", 2024-09-16T03:40:00.908 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7899999999999999e-07 2024-09-16T03:40:00.908 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.908 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.908 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.908 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114788+0000", 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4569999999999999e-06 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114792+0000", 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5730000000000001e-06 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.114796+0000", 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.5449999999999999e-06 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.909 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316237+0000", 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20144099300000001 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316417+0000", 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017981099999999999 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316564+0000", 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014693700000000001 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.910 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.316993+0000", 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042985499999999998 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.717965+0000", 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40097123699999998 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718005+0000", 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0083000000000002e-05 2024-09-16T03:40:00.911 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718064+0000", 2024-09-16T03:40:00.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9020000000000001e-05 2024-09-16T03:40:00.912 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.912 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.912 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.912 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:16.718119+0000", 2024-09-16T03:40:00.912 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4957000000000003e-05 2024-09-16T03:40:00.912 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.912 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:139 3.1 3:bd600e93:::b81929c6ed6df2fe181809e10b331bcd3c2df1a1077e06024548babcdcd8f550:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 83.842194661999997, 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.43180016300000001, 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041574+0000", 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2300000000000001e-07 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041700+0000", 2024-09-16T03:40:00.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000126367 2024-09-16T03:40:00.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041707+0000", 2024-09-16T03:40:00.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1519999999999997e-06 2024-09-16T03:40:00.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.915 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.915 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.915 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041710+0000", 2024-09-16T03:40:00.915 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7869999999999999e-06 2024-09-16T03:40:00.915 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043365+0000", 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001654914 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043453+0000", 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7954000000000005e-05 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.043596+0000", 2024-09-16T03:40:00.916 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014304500000000001 2024-09-16T03:40:00.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.145131+0000", 2024-09-16T03:40:00.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10153440599999999 2024-09-16T03:40:00.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.917 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.344273+0000", 2024-09-16T03:40:00.917 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19914232200000001 2024-09-16T03:40:00.917 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.917 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.917 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473290+0000", 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12901701900000001 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473325+0000", 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4656000000000002e-05 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473374+0000", 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9017999999999997e-05 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.918 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.919 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.919 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.919 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.11.23:144 3.1 3:91c40fc0:::f2d8823355177e50718038dafd7a8313c1a25b0231461a8672e684a7d4e587b4:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:40:00.919 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:40:00.919 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 83.842050404999995, 2024-09-16T03:40:00.919 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.431801096, 2024-09-16T03:40:00.919 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.919 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.919 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.919 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.919 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.919 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:40:00.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:40:00.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.920 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.920 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.920 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.041718+0000", 2024-09-16T03:40:00.920 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.75e-07 2024-09-16T03:40:00.920 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042077+0000", 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00035865099999999998 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042084+0000", 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2400000000000001e-06 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.042088+0000", 2024-09-16T03:40:00.921 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1570000000000002e-06 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144179+0000", 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10209033300000001 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144427+0000", 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000247879 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:40:00.922 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.144656+0000", 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022919800000000001 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.145866+0000", 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001210046 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.344293+0000", 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.198427353 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.923 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473432+0000", 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.12913908800000001 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473478+0000", 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.6165000000000003e-05 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:37.473519+0000", 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0510999999999999e-05 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.924 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.12.21:68 3.1 3:85b1a0cb:::cbec311fd3dcf9951ac1aa2c731e21adc6221934f894d1c2b9cc0e954f5d34ba:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:30.867550+0000", 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.016218592000001, 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50821847499999995, 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.925 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867550+0000", 2024-09-16T03:40:00.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867550+0000", 2024-09-16T03:40:00.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.926 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.926 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.926 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867550+0000", 2024-09-16T03:40:00.926 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9899999999999999e-07 2024-09-16T03:40:00.926 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867846+0000", 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029565 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867858+0000", 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1623e-05 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867866+0000", 2024-09-16T03:40:00.927 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5900000000000008e-06 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.171695+0000", 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30382852100000002 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.171800+0000", 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104832 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:40:00.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.172004+0000", 2024-09-16T03:40:00.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020439000000000001 2024-09-16T03:40:00.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.173873+0000", 2024-09-16T03:40:00.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001868724 2024-09-16T03:40:00.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.929 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.929 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.929 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.173911+0000", 2024-09-16T03:40:00.929 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7997999999999998e-05 2024-09-16T03:40:00.929 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.375626+0000", 2024-09-16T03:40:00.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.201715532 2024-09-16T03:40:00.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.375727+0000", 2024-09-16T03:40:00.930 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010069999999999999 2024-09-16T03:40:00.930 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.930 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.930 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.930 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.375768+0000", 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1115999999999997e-05 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.12.21:71 3.1 3:b5d59aa7:::333d73059656e9a14b425bc3214236b65d87405f6459482d3b8927f96fbd6395:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:30.867879+0000", 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 30.015888837999999, 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.50807818800000004, 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.931 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867879+0000", 2024-09-16T03:40:00.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.932 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.932 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867879+0000", 2024-09-16T03:40:00.932 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.932 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.932 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.867880+0000", 2024-09-16T03:40:00.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6799999999999999e-07 2024-09-16T03:40:00.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.868431+0000", 2024-09-16T03:40:00.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00055086899999999995 2024-09-16T03:40:00.933 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.933 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.933 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.933 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.868451+0000", 2024-09-16T03:40:00.933 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9639e-05 2024-09-16T03:40:00.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:30.868457+0000", 2024-09-16T03:40:00.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0310000000000004e-06 2024-09-16T03:40:00.934 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.934 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.934 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.934 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.172722+0000", 2024-09-16T03:40:00.934 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30426540899999999 2024-09-16T03:40:00.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.172882+0000", 2024-09-16T03:40:00.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000160094 2024-09-16T03:40:00.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:40:00.935 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.173126+0000", 2024-09-16T03:40:00.935 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00024391500000000001 2024-09-16T03:40:00.935 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.935 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.935 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.179336+0000", 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0062094810000000002 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.274806+0000", 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.095470151000000003 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.375843+0000", 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101037217 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.936 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.375901+0000", 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7683000000000003e-05 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:31.375958+0000", 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6931000000000001e-05 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.937 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.12.21:78 3.1 3:85b1a0cb:::cbec311fd3dcf9951ac1aa2c731e21adc6221934f894d1c2b9cc0e954f5d34ba:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e165)", 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:49.448718+0000", 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 11.435050333, 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.42434692899999998, 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.448718+0000", 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.448718+0000", 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.448719+0000", 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1930000000000001e-06 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.448725+0000", 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6529999999999999e-06 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.448739+0000", 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4216000000000001e-05 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.548894+0000", 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100154836 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.579727+0000", 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.030833126999999998 2024-09-16T03:40:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.579814+0000", 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7062000000000001e-05 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 3,4", 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.579982+0000", 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016846299999999999 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.680592+0000", 2024-09-16T03:40:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10060993999999999 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.680607+0000", 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4275e-05 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.872970+0000", 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19236324699999999 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.873016+0000", 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5868999999999997e-05 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:49.873065+0000", 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.9048000000000002e-05 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:5048 4.1a 4:5e3c0760:::smithi15736696-24:head [write 497606~645222,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:53.901848+0000", 2024-09-16T03:40:00.943 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.9819197900000001, 2024-09-16T03:40:00.944 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60206692399999995, 2024-09-16T03:40:00.944 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.944 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.944 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.944 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:00.944 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:00.944 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:00.944 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 5048 2024-09-16T03:40:00.944 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.944 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.944 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.944 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.945 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.901848+0000", 2024-09-16T03:40:00.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.945 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.945 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.901848+0000", 2024-09-16T03:40:00.945 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.945 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.945 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.945 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.901849+0000", 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4999999999999998e-07 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.902947+0000", 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0010977529999999999 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.902957+0000", 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0189e-05 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.946 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.902964+0000", 2024-09-16T03:40:00.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6469999999999998e-06 2024-09-16T03:40:00.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904500+0000", 2024-09-16T03:40:00.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0015354909999999999 2024-09-16T03:40:00.947 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.947 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.947 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:40:00.947 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904511+0000", 2024-09-16T03:40:00.947 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0745e-05 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.099928+0000", 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19541708699999999 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.100038+0000", 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000109789 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 11,14,15", 2024-09-16T03:40:00.948 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.100156+0000", 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000118027 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.205142+0000", 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104986247 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.205352+0000", 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020979899999999999 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.949 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.403628+0000", 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19827620800000001 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.503699+0000", 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100070801 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.503824+0000", 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00012536000000000001 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.950 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.503915+0000", 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1330999999999997e-05 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:5049 4.1a 4:5e3c0760:::smithi15736696-24:head [write 1934400~629494] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:53.902984+0000", 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.9807839869999997, 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60268962199999998, 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.951 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.952 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.952 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:00.952 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:00.952 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:00.952 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 5049 2024-09-16T03:40:00.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.952 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.952 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.952 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.952 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.902984+0000", 2024-09-16T03:40:00.952 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.952 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.952 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.902984+0000", 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.902986+0000", 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.395e-06 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904398+0000", 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0014122690000000001 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.953 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904418+0000", 2024-09-16T03:40:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0132999999999998e-05 2024-09-16T03:40:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904425+0000", 2024-09-16T03:40:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:40:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904850+0000", 2024-09-16T03:40:00.954 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00042544600000000001 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904859+0000", 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6719999999999993e-06 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.101849+0000", 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19699007099999999 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.101922+0000", 2024-09-16T03:40:00.955 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2960999999999995e-05 2024-09-16T03:40:00.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 11,14,15", 2024-09-16T03:40:00.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.102061+0000", 2024-09-16T03:40:00.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013863600000000001 2024-09-16T03:40:00.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.956 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.205305+0000", 2024-09-16T03:40:00.956 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.103244345 2024-09-16T03:40:00.956 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.956 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.956 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.205411+0000", 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000106119 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.403761+0000", 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19835000699999999 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.505497+0000", 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101735519 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.957 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.958 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.958 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.505559+0000", 2024-09-16T03:40:00.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2298999999999994e-05 2024-09-16T03:40:00.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.958 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.958 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.958 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.505674+0000", 2024-09-16T03:40:00.958 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000115024 2024-09-16T03:40:00.958 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.958 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.958 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.958 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.958 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:5050 4.1a 4:5e3c0760:::smithi15736696-24:head [write 3243966~453640] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:53.904451+0000", 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.9793173150000003, 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60139276100000005, 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 5050 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904451+0000", 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904451+0000", 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904452+0000", 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.319e-06 2024-09-16T03:40:00.960 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904785+0000", 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00033317299999999998 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904794+0000", 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3229999999999998e-06 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904799+0000", 2024-09-16T03:40:00.961 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8099999999999997e-06 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904889+0000", 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.0276000000000001e-05 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904895+0000", 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0050000000000001e-06 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.962 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.203670+0000", 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.298775077 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.203770+0000", 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000100552 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 11,14,15", 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.203946+0000", 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00017512199999999999 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.963 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.206370+0000", 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0024247219999999998 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.206388+0000", 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7643999999999999e-05 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.403825+0000", 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.197437156 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.965 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.965 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.505721+0000", 2024-09-16T03:40:00.965 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101896111 2024-09-16T03:40:00.965 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.965 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.965 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.505769+0000", 2024-09-16T03:40:00.965 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7973000000000003e-05 2024-09-16T03:40:00.965 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.965 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.965 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.965 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.505844+0000", 2024-09-16T03:40:00.965 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4498000000000004e-05 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:5051 4.1a 4:5e3c0760:::smithi15736696-24:head [setxattr _header (56) in=7b,truncate 3697606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:53.904809+0000", 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.9789587419999997, 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60127522, 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:00.966 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 5051 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904809+0000", 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904809+0000", 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.967 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904810+0000", 2024-09-16T03:40:00.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5799999999999999e-07 2024-09-16T03:40:00.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904813+0000", 2024-09-16T03:40:00.968 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.458e-06 2024-09-16T03:40:00.968 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.968 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.968 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.968 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904817+0000", 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5909999999999998e-06 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904821+0000", 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9380000000000002e-06 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904924+0000", 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000103253 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.969 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904931+0000", 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8950000000000001e-06 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.205619+0000", 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300687869 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.205706+0000", 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6849999999999997e-05 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.970 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.971 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 11,14,15", 2024-09-16T03:40:00.971 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.205840+0000", 2024-09-16T03:40:00.971 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013404399999999999 2024-09-16T03:40:00.971 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.971 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.971 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.971 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.306578+0000", 2024-09-16T03:40:00.971 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100737965 2024-09-16T03:40:00.971 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.971 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.971 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.971 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.306604+0000", 2024-09-16T03:40:00.971 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6254e-05 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.404749+0000", 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098145011000000004 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.505888+0000", 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101138505 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.972 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.505946+0000", 2024-09-16T03:40:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8254000000000002e-05 2024-09-16T03:40:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.506085+0000", 2024-09-16T03:40:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000138675 2024-09-16T03:40:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:5052 4.1a 4:5e3c0760:::smithi15736696-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:00.973 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:53.904827+0000", 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.9789414140000003, 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.60143271899999995, 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 5052 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.974 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904827+0000", 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904827+0000", 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904827+0000", 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.8299999999999998e-07 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.975 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904828+0000", 2024-09-16T03:40:00.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.387e-06 2024-09-16T03:40:00.976 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.976 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904832+0000", 2024-09-16T03:40:00.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.332e-06 2024-09-16T03:40:00.976 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.976 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.976 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.976 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:53.904835+0000", 2024-09-16T03:40:00.976 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.614e-06 2024-09-16T03:40:00.976 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.099348+0000", 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19451236699999999 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.099360+0000", 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1933000000000001e-05 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.306481+0000", 2024-09-16T03:40:00.977 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20712161000000001 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.306549+0000", 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8118000000000001e-05 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.506114+0000", 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.199564346 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.978 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.506161+0000", 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7655999999999999e-05 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.506260+0000", 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8072999999999994e-05 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.9.22:6 3.1 3:8b45ca42:::c189c20f5a256634455be17c65464481437e6247d8cfe0b956c93f9c69b67755:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e170)", 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:54.803275+0000", 2024-09-16T03:40:00.979 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.0804928220000001, 2024-09-16T03:40:00.980 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.71069067600000002, 2024-09-16T03:40:00.980 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.980 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.980 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.980 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.980 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.980 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.980 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803275+0000", 2024-09-16T03:40:00.980 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.980 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.980 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.980 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.980 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803275+0000", 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803277+0000", 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7579999999999999e-06 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804266+0000", 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0009890229999999999 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.981 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804282+0000", 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6322000000000001e-05 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804291+0000", 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.2619999999999998e-06 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804846+0000", 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00055509400000000001 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.982 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804856+0000", 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0315e-05 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.211807+0000", 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40695035499999999 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.211962+0000", 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000155704 2024-09-16T03:40:00.983 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 1,3,4,11", 2024-09-16T03:40:00.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.212279+0000", 2024-09-16T03:40:00.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00031632200000000001 2024-09-16T03:40:00.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.213635+0000", 2024-09-16T03:40:00.984 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001356171 2024-09-16T03:40:00.984 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.984 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.984 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.984 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.214656+0000", 2024-09-16T03:40:00.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00102129 2024-09-16T03:40:00.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.215682+0000", 2024-09-16T03:40:00.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001025771 2024-09-16T03:40:00.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.985 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.985 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.215733+0000", 2024-09-16T03:40:00.985 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1526000000000001e-05 2024-09-16T03:40:00.985 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.985 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.513820+0000", 2024-09-16T03:40:00.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.298086562 2024-09-16T03:40:00.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.513888+0000", 2024-09-16T03:40:00.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8501e-05 2024-09-16T03:40:00.986 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.986 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.986 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.986 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.513966+0000", 2024-09-16T03:40:00.986 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7700000000000005e-05 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.8.22:70 3.1 3:b73c47cb:::0c28b64cb98c4375b8285829cb22a51685910510d1ba74394a66883f781db201:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e170)", 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:54.803314+0000", 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.0804545799999996, 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40843503399999997, 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.987 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803314+0000", 2024-09-16T03:40:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803314+0000", 2024-09-16T03:40:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803315+0000", 2024-09-16T03:40:00.988 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5969999999999999e-06 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803326+0000", 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0858e-05 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803340+0000", 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3869999999999999e-05 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.989 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803362+0000", 2024-09-16T03:40:00.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.1852999999999999e-05 2024-09-16T03:40:00.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.990 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803665+0000", 2024-09-16T03:40:00.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00030284300000000001 2024-09-16T03:40:00.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.990 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:40:00.990 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803674+0000", 2024-09-16T03:40:00.990 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3680000000000008e-06 2024-09-16T03:40:00.990 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.990 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.908356+0000", 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.104681502 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.908513+0000", 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00015753800000000001 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 1,3,4,11", 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.908731+0000", 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00021764199999999999 2024-09-16T03:40:00.991 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.009673+0000", 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100942744 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.009693+0000", 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9079000000000001e-05 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.009735+0000", 2024-09-16T03:40:00.992 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2519999999999999e-05 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.111236+0000", 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10150103100000001 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.211460+0000", 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100223563 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:00.993 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.211560+0000", 2024-09-16T03:40:00.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.9974000000000002e-05 2024-09-16T03:40:00.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.994 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:00.994 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.211749+0000", 2024-09-16T03:40:00.994 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00018905199999999999 2024-09-16T03:40:00.994 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.994 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:00.994 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:00.994 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.994 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.994 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.10.22:311 3.1 3:8960c1cd:::854bc0225a3bb690bfac097dfa6f255fbb6cdea230ec8761b6d08a4474b3799a:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e170)", 2024-09-16T03:40:00.994 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:54.803480+0000", 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.0802881260000001, 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.410100144, 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803480+0000", 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:00.995 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803480+0000", 2024-09-16T03:40:00.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:00.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:00.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803481+0000", 2024-09-16T03:40:00.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8199999999999999e-07 2024-09-16T03:40:00.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.996 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:00.996 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803485+0000", 2024-09-16T03:40:00.996 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.827e-06 2024-09-16T03:40:00.996 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.996 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803497+0000", 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2432000000000001e-05 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803504+0000", 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.4819999999999999e-06 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803733+0000", 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00022970299999999999 2024-09-16T03:40:00.997 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.803740+0000", 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6630000000000004e-06 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.110293+0000", 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30655313000000001 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.110432+0000", 2024-09-16T03:40:00.998 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013844199999999999 2024-09-16T03:40:00.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 1,3,4,11", 2024-09-16T03:40:00.999 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.110630+0000", 2024-09-16T03:40:00.999 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000198426 2024-09-16T03:40:00.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:00.999 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.213264+0000", 2024-09-16T03:40:00.999 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102633725 2024-09-16T03:40:00.999 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:00.999 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:00.999 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.213317+0000", 2024-09-16T03:40:01.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3703999999999999e-05 2024-09-16T03:40:01.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.000 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.000 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.213361+0000", 2024-09-16T03:40:01.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.3729000000000001e-05 2024-09-16T03:40:01.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.000 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.000 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.000 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.213409+0000", 2024-09-16T03:40:01.000 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7746e-05 2024-09-16T03:40:01.000 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.213455+0000", 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5686000000000003e-05 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.213493+0000", 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8339e-05 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.213580+0000", 2024-09-16T03:40:01.001 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7328000000000001e-05 2024-09-16T03:40:01.002 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:01.002 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:01.002 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:01.002 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.002 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.002 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.9.22:7 3.1 3:9e9cc1a8:::b5ee11f24f94d7179c7321ae7ab16b218edc88fca58c1cedd2314a17c4c69f18:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e170)", 2024-09-16T03:40:01.002 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:54.804432+0000", 2024-09-16T03:40:01.002 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.0793362220000002, 2024-09-16T03:40:01.002 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.80982263099999996, 2024-09-16T03:40:01.002 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:01.002 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:01.002 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:01.002 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804432+0000", 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804432+0000", 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804433+0000", 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0070000000000001e-06 2024-09-16T03:40:01.003 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804765+0000", 2024-09-16T03:40:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00033170600000000003 2024-09-16T03:40:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804776+0000", 2024-09-16T03:40:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0871e-05 2024-09-16T03:40:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804782+0000", 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9529999999999996e-06 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.805453+0000", 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00067132300000000004 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.805462+0000", 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.7779999999999996e-06 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.005 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.213672+0000", 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40821022099999998 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.213762+0000", 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.9680999999999997e-05 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 1,3,4,11", 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.213933+0000", 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000171674 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.006 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.215778+0000", 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0018451279999999999 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.215786+0000", 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.52e-06 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.317016+0000", 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10123038700000001 2024-09-16T03:40:01.007 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.514012+0000", 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19699576699999999 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.614147+0000", 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100135422 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.614185+0000", 2024-09-16T03:40:01.008 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7846000000000003e-05 2024-09-16T03:40:01.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.009 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:01.009 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.614255+0000", 2024-09-16T03:40:01.009 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.9346999999999999e-05 2024-09-16T03:40:01.009 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:01.009 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:01.009 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:01.009 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.009 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.009 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.9.22:11 3.1 3:837c3ce9:::e3926f6fdc186218432a42befd92c5098035f552a1b792c85b68c55f205e832d:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e170)", 2024-09-16T03:40:01.009 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:54.804801+0000", 2024-09-16T03:40:01.009 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.0789667930000002, 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.90959779699999999, 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804801+0000", 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804801+0000", 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:01.010 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:01.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.804802+0000", 2024-09-16T03:40:01.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.95e-07 2024-09-16T03:40:01.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:01.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.805375+0000", 2024-09-16T03:40:01.011 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00057209800000000005 2024-09-16T03:40:01.011 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.011 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.011 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:01.011 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.805389+0000", 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.4307e-05 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.805394+0000", 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3059999999999997e-06 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.805504+0000", 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010949499999999999 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.012 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:40:01.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.805508+0000", 2024-09-16T03:40:01.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0210000000000001e-06 2024-09-16T03:40:01.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:01.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.214700+0000", 2024-09-16T03:40:01.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40919203799999998 2024-09-16T03:40:01.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.013 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.013 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:01.013 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.214836+0000", 2024-09-16T03:40:01.013 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000136181 2024-09-16T03:40:01.013 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 1,3,4,11", 2024-09-16T03:40:01.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.215130+0000", 2024-09-16T03:40:01.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029451400000000001 2024-09-16T03:40:01.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:01.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.316960+0000", 2024-09-16T03:40:01.014 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101829241 2024-09-16T03:40:01.014 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.014 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.014 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.014 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.316991+0000", 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.1739000000000002e-05 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.317282+0000", 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000290809 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.514048+0000", 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.196765625 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.015 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.714279+0000", 2024-09-16T03:40:01.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.200230721 2024-09-16T03:40:01.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:01.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.714321+0000", 2024-09-16T03:40:01.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.2599000000000003e-05 2024-09-16T03:40:01.016 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.016 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.016 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:01.016 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.714399+0000", 2024-09-16T03:40:01.016 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8108000000000005e-05 2024-09-16T03:40:01.016 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:01.016 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(osd.9.22:15 3.1 3:964c391a:::c9c0cf4a00ae29c03b283d9cbb01cfb175b42402a28680dcd93389b12aeeb768:head [call cas.chunk_create_or_get_ref in=26b] snapc 0=[] RETRY=2 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e170)", 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:54.805404+0000", 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 6.0783637309999996, 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.009100074, 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:01.017 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.805404+0000", 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.805404+0000", 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.805405+0000", 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.02e-07 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.018 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.805445+0000", 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9631000000000001e-05 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.805452+0000", 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.2989999999999999e-06 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.805460+0000", 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.498e-06 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.019 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:01.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.901836+0000", 2024-09-16T03:40:01.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.096375892000000005 2024-09-16T03:40:01.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:40:01.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:54.901955+0000", 2024-09-16T03:40:01.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000119261 2024-09-16T03:40:01.020 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.020 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.020 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:01.020 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.315953+0000", 2024-09-16T03:40:01.020 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.41399809100000001 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.316204+0000", 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00025065299999999999 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 1,3,4,11", 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.316412+0000", 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00020808600000000001 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:01.021 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.317196+0000", 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00078460499999999996 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.317527+0000", 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00033069300000000001 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.417041+0000", 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099513631000000005 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.022 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.514072+0000", 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.097031514999999999 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.814378+0000", 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30030562799999999 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.814420+0000", 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1841000000000001e-05 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:01.023 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:01.024 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:01.024 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:55.814505+0000", 2024-09-16T03:40:01.024 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4847999999999998e-05 2024-09-16T03:40:01.024 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:01.024 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:01.024 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:01.024 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:01.024 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:01.024 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:40:01.025 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:40:01.026 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:01.026 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:40:01.026 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:01.027 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:40:01.027 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:40:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:01.028 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:40:01.028 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:01.028 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:40:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:01.029 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:40:01.029 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:01.029 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:40:01.030 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:01.030 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:40:01.024+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:40:01.039 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:40:01.039 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:01.056 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:40:01.057 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:01.072 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:40:01.072 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:01.090 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:40:01.090 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:01.107 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:40:01.107 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:01.124 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:40:01.125 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:01.126 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:40:01.124+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:40:01.171 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:01.171 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:01.179 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:40:01.227 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:40:01.225+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:40:01.282 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:01.282 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:01.291 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:40:01.334 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:40:01.333+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:40:01.392 INFO:teuthology.orchestra.run.smithi064.stderr:set 4.24 pg_upmap mapping to [15,0,14] 2024-09-16T03:40:01.398 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:01.398 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:01.407 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:40:01.448 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:40:01.447+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:40:01.494 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:40:01.493+0000 7f0ae754b640 -1 osd.0 176 handle_osd_map injecting map crc failure 2024-09-16T03:40:01.495 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:40:01.493+0000 7f66f3205640 -1 osd.15 176 handle_osd_map injecting map crc failure 2024-09-16T03:40:01.529 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:40:01.528+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:40:01.552 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:01.552 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:01.561 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:40:01.630 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:40:01.628+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:40:01.666 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:01.666 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:01.675 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:40:01.731 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:40:01.730+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:40:01.780 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:01.780 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:01.788 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:40:01.833 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:40:01.831+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:40:01.907 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:01.907 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:01.916 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:40:01.933 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:40:01.931+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:40:02.022 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:02.022 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:02.030 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:40:02.034 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:40:02.032+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:40:02.135 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:40:02.133+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:40:02.135 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:02.135 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:02.144 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:40:02.236 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:40:02.235+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:40:02.248 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:02.248 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:02.256 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:40:02.337 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:40:02.335+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:40:02.359 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:02.359 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:02.367 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:40:02.400 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T03:40:02.438 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:40:02.436+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:40:02.473 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:02.473 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:02.481 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:40:02.538 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:40:02.537+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:40:02.587 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:02.587 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:02.595 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:40:02.639 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:40:02.637+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:40:02.739 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:02.740 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:02.740 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:40:02.738+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:40:02.743 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:40:02.741+0000 7f6528e68640 -1 osd.1 177 handle_osd_map injecting map crc failure 2024-09-16T03:40:02.749 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:40:02.840 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:40:02.839+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:40:02.852 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:02.852 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:02.860 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:40:02.942 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:40:02.941+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:40:02.964 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:02.964 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:02.973 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:40:03.043 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:40:03.042+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:40:03.076 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:03.076 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:03.085 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:40:03.136 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:40:03.144 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:40:03.143+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:40:03.229 INFO:teuthology.orchestra.run.smithi157.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:03.229 INFO:teuthology.orchestra.run.smithi157.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:03.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T03:40:03.245 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:40:03.243+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:40:03.265 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:40:03.265 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:03.283 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:40:03.283 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:03.300 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:40:03.300 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:03.317 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:40:03.318 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:03.334 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:40:03.334 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:03.346 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:40:03.344+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:40:03.347 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:40:03.347 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:40:03.347 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:40:03.347 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:40:03.351 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:40:03.351 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:03.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T03:40:03.368 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:40:03.369 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:03.385 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:40:03.386 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:03.401 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is set 2024-09-16T03:40:03.403 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:40:03.403 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:03.423 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:40:03.423 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:03.439 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:40:03.440 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:03.447 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:40:03.445+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:40:03.456 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:40:03.457 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:03.473 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:40:03.474 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:03.480 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:40:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:40:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:40:03.480 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:40:03.481 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:40:03.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T03:40:03.491 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:40:03.492 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:03.509 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:40:03.509 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:03.526 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:40:03.526 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:03.547 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:40:03.546+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:40:03.602 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:40:03.602 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:40:03.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:40:03.602 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:40:03.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.602 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1948 4.8 4:146f84f0:::smithi15736696-6:head [setxattr _header (55) in=7b,truncate 2756348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:03.602 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:40:03.603 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.47441943600001, 2024-09-16T03:40:03.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40326984799999999, 2024-09-16T03:40:03.603 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.603 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.603 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:03.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.603 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1948 2024-09-16T03:40:03.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.603 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124807+0000", 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124808+0000", 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9099999999999991e-07 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124813+0000", 2024-09-16T03:40:03.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9599999999999999e-06 2024-09-16T03:40:03.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124821+0000", 2024-09-16T03:40:03.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0490000000000004e-06 2024-09-16T03:40:03.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124827+0000", 2024-09-16T03:40:03.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3339999999999998e-06 2024-09-16T03:40:03.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126739+0000", 2024-09-16T03:40:03.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00191147 2024-09-16T03:40:03.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.126830+0000", 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1751999999999998e-05 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127005+0000", 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000174227 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127981+0000", 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097622799999999999 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:03.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.128193+0000", 2024-09-16T03:40:03.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00021176999999999999 2024-09-16T03:40:03.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.135281+0000", 2024-09-16T03:40:03.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070882150000000001 2024-09-16T03:40:03.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.427785+0000", 2024-09-16T03:40:03.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29250390599999998 2024-09-16T03:40:03.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527932+0000", 2024-09-16T03:40:03.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10014759500000001 2024-09-16T03:40:03.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:03.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.527980+0000", 2024-09-16T03:40:03.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7725000000000001e-05 2024-09-16T03:40:03.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528077+0000", 2024-09-16T03:40:03.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6626000000000006e-05 2024-09-16T03:40:03.611 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.611 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.611 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.611 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1949 4.8 4:146f84f0:::smithi15736696-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:03.611 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:40:03.611 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.47438869200001, 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40346921899999999, 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1949 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:40:03.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124838+0000", 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124841+0000", 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5890000000000001e-06 2024-09-16T03:40:03.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124847+0000", 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0820000000000004e-06 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.124852+0000", 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1070000000000004e-06 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127536+0000", 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026844619999999999 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.127585+0000", 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8915999999999997e-05 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528126+0000", 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40054061899999999 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528193+0000", 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6717999999999994e-05 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.528307+0000", 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011413600000000001 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1979 4.18 4:1f63731d:::smithi15736696-30:head [write 789962~534747] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.96338138, 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.506754182, 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.616 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1979 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635845+0000", 2024-09-16T03:40:03.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.635850+0000", 2024-09-16T03:40:03.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6670000000000004e-06 2024-09-16T03:40:03.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636429+0000", 2024-09-16T03:40:03.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000579745 2024-09-16T03:40:03.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.636441+0000", 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1965e-05 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736594+0000", 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100152291 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837326+0000", 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10073201399999999 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837437+0000", 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111914 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.837625+0000", 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000187391 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939449+0000", 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10182371599999999 2024-09-16T03:40:03.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:03.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939476+0000", 2024-09-16T03:40:03.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7117000000000001e-05 2024-09-16T03:40:03.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142297+0000", 2024-09-16T03:40:03.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20282091499999999 2024-09-16T03:40:03.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142347+0000", 2024-09-16T03:40:03.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0800000000000002e-05 2024-09-16T03:40:03.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142485+0000", 2024-09-16T03:40:03.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013799999999999999 2024-09-16T03:40:03.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:03.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142526+0000", 2024-09-16T03:40:03.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0269999999999999e-05 2024-09-16T03:40:03.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142599+0000", 2024-09-16T03:40:03.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3376999999999999e-05 2024-09-16T03:40:03.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.623 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.623 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1980 4.18 4:1f63731d:::smithi15736696-30:head [write 1780952~690056] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:03.624 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:40:03.624 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 121.86258021899999, 2024-09-16T03:40:03.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40615011400000001, 2024-09-16T03:40:03.624 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.624 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.624 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:03.624 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.624 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.624 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.624 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1980 2024-09-16T03:40:03.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.624 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736646+0000", 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.736656+0000", 2024-09-16T03:40:03.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0339999999999999e-05 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737092+0000", 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435957 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737102+0000", 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7860000000000008e-06 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.737107+0000", 2024-09-16T03:40:03.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:40:03.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839282+0000", 2024-09-16T03:40:03.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10217523100000001 2024-09-16T03:40:03.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839389+0000", 2024-09-16T03:40:03.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010619399999999999 2024-09-16T03:40:03.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839567+0000", 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017886200000000001 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.939485+0000", 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099917168000000001 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.943655+0000", 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004170287 2024-09-16T03:40:03.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142373+0000", 2024-09-16T03:40:03.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19871839599999999 2024-09-16T03:40:03.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142431+0000", 2024-09-16T03:40:03.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7880999999999997e-05 2024-09-16T03:40:03.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142627+0000", 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019580500000000001 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142723+0000", 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6209e-05 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.142796+0000", 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2978000000000005e-05 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.631 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2277 4.18 4:1f63731d:::smithi15736696-30:head [write 460558~707131] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:40:03.631 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:40:03.631 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.176444519, 2024-09-16T03:40:03.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39421539100000003, 2024-09-16T03:40:03.631 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.631 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.631 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:03.631 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.631 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.631 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.631 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2277 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422782+0000", 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.422785+0000", 2024-09-16T03:40:03.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9809999999999999e-06 2024-09-16T03:40:03.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423586+0000", 2024-09-16T03:40:03.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080133600000000004 2024-09-16T03:40:03.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423607+0000", 2024-09-16T03:40:03.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1039e-05 2024-09-16T03:40:03.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.423616+0000", 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0189999999999995e-06 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523776+0000", 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10015945399999999 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523851+0000", 2024-09-16T03:40:03.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5549999999999993e-05 2024-09-16T03:40:03.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:40:03.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.523972+0000", 2024-09-16T03:40:03.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120683 2024-09-16T03:40:03.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:03.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527436+0000", 2024-09-16T03:40:03.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034638870000000001 2024-09-16T03:40:03.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636588+0000", 2024-09-16T03:40:03.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109152601 2024-09-16T03:40:03.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636665+0000", 2024-09-16T03:40:03.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6661000000000003e-05 2024-09-16T03:40:03.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637128+0000", 2024-09-16T03:40:03.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046316199999999998 2024-09-16T03:40:03.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816886+0000", 2024-09-16T03:40:03.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.17975745200000001 2024-09-16T03:40:03.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:03.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816927+0000", 2024-09-16T03:40:03.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1529000000000002e-05 2024-09-16T03:40:03.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.816997+0000", 2024-09-16T03:40:03.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0036999999999994e-05 2024-09-16T03:40:03.638 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.638 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.638 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.639 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2278 4.18 4:1f63731d:::smithi15736696-30:head [write 1869389~508716] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:40:03.639 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:40:03.639 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.169875518, 2024-09-16T03:40:03.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48788153400000001, 2024-09-16T03:40:03.639 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.639 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.639 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:03.639 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.639 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.639 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.639 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2278 2024-09-16T03:40:03.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.640 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:40:03.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429351+0000", 2024-09-16T03:40:03.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429352+0000", 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.604e-06 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429895+0000", 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054308899999999996 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429909+0000", 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3518e-05 2024-09-16T03:40:03.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.429917+0000", 2024-09-16T03:40:03.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.559e-06 2024-09-16T03:40:03.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525208+0000", 2024-09-16T03:40:03.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095291736000000002 2024-09-16T03:40:03.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525303+0000", 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4304999999999999e-05 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.525410+0000", 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010755399999999999 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.636785+0000", 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111375037 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637092+0000", 2024-09-16T03:40:03.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030685599999999999 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637157+0000", 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4593999999999999e-05 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637236+0000", 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9563999999999999e-05 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917119+0000", 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.279882931 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917158+0000", 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9096e-05 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917232+0000", 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4091000000000006e-05 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2279 4.18 4:1f63731d:::smithi15736696-30:head [write 3094884~165674] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.16518672300001, 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483372209, 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2279 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.646 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:40:03.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434040+0000", 2024-09-16T03:40:03.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434041+0000", 2024-09-16T03:40:03.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.195e-06 2024-09-16T03:40:03.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434191+0000", 2024-09-16T03:40:03.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000150488 2024-09-16T03:40:03.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434202+0000", 2024-09-16T03:40:03.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0978000000000001e-05 2024-09-16T03:40:03.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434208+0000", 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8320000000000002e-06 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526820+0000", 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.092612229000000004 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526894+0000", 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3435000000000004e-05 2024-09-16T03:40:03.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.526990+0000", 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6112000000000004e-05 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637050+0000", 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.110059859 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637097+0000", 2024-09-16T03:40:03.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6848000000000003e-05 2024-09-16T03:40:03.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637183+0000", 2024-09-16T03:40:03.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6798000000000005e-05 2024-09-16T03:40:03.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637262+0000", 2024-09-16T03:40:03.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9049999999999997e-05 2024-09-16T03:40:03.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917287+0000", 2024-09-16T03:40:03.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280025036 2024-09-16T03:40:03.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:03.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917326+0000", 2024-09-16T03:40:03.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.871e-05 2024-09-16T03:40:03.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917412+0000", 2024-09-16T03:40:03.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5638999999999998e-05 2024-09-16T03:40:03.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.653 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.653 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.653 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2280 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 3260558] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:40:03.653 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:40:03.653 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.165008442, 2024-09-16T03:40:03.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.48353995599999999, 2024-09-16T03:40:03.654 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.654 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.654 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:03.654 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.654 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.654 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.654 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2280 2024-09-16T03:40:03.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.654 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:40:03.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:40:03.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434218+0000", 2024-09-16T03:40:03.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0699999999999997e-07 2024-09-16T03:40:03.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434221+0000", 2024-09-16T03:40:03.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8329999999999998e-06 2024-09-16T03:40:03.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434225+0000", 2024-09-16T03:40:03.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7589999999999999e-06 2024-09-16T03:40:03.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434228+0000", 2024-09-16T03:40:03.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.971e-06 2024-09-16T03:40:03.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527459+0000", 2024-09-16T03:40:03.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.093231249000000002 2024-09-16T03:40:03.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527519+0000", 2024-09-16T03:40:03.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9471e-05 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.527606+0000", 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6776999999999999e-05 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637084+0000", 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.109478228 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637100+0000", 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6711999999999999e-05 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637210+0000", 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010909200000000001 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.637296+0000", 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6718000000000006e-05 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917637+0000", 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.280340537 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917671+0000", 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3983999999999999e-05 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917758+0000", 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7017999999999999e-05 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2281 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e80)", 2024-09-16T03:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 113.164994147, 2024-09-16T03:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.483761683, 2024-09-16T03:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2281 2024-09-16T03:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434232+0000", 2024-09-16T03:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.96e-07 2024-09-16T03:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434234+0000", 2024-09-16T03:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4050000000000001e-06 2024-09-16T03:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434237+0000", 2024-09-16T03:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1939999999999998e-06 2024-09-16T03:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.434240+0000", 2024-09-16T03:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.74e-06 2024-09-16T03:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530037+0000", 2024-09-16T03:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.095796830999999999 2024-09-16T03:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.530089+0000", 2024-09-16T03:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2877999999999998e-05 2024-09-16T03:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917823+0000", 2024-09-16T03:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.38773380000000002 2024-09-16T03:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917891+0000", 2024-09-16T03:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7676999999999996e-05 2024-09-16T03:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:10.917994+0000", 2024-09-16T03:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010286200000000001 2024-09-16T03:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2790 4.18 4:1f63731d:::smithi15736696-30:head [assert-version v39,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e94)", 2024-09-16T03:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 98.195146691999994, 2024-09-16T03:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42581759000000002, 2024-09-16T03:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2790 2024-09-16T03:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404080+0000", 2024-09-16T03:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404083+0000", 2024-09-16T03:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2770000000000001e-06 2024-09-16T03:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404095+0000", 2024-09-16T03:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1780000000000001e-05 2024-09-16T03:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404116+0000", 2024-09-16T03:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0882000000000001e-05 2024-09-16T03:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.404127+0000", 2024-09-16T03:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1078e-05 2024-09-16T03:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489837+0000", 2024-09-16T03:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.085709977000000007 2024-09-16T03:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.489956+0000", 2024-09-16T03:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119856 2024-09-16T03:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:40:03.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.592758+0000", 2024-09-16T03:40:03.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10280165400000001 2024-09-16T03:40:03.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:03.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.593693+0000", 2024-09-16T03:40:03.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000935285 2024-09-16T03:40:03.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.594164+0000", 2024-09-16T03:40:03.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00047097000000000003 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.628426+0000", 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034262070999999998 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.732623+0000", 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10419628 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829726+0000", 2024-09-16T03:40:03.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097103098999999998 2024-09-16T03:40:03.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:03.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829790+0000", 2024-09-16T03:40:03.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4677000000000004e-05 2024-09-16T03:40:03.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:25.829897+0000", 2024-09-16T03:40:03.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106704 2024-09-16T03:40:03.676 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.677 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.677 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.677 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2850 4.18 4:1f63731d:::smithi15736696-30:head [setxattr _header (56) in=7b,truncate 2252171] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:40:03.677 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:40:03.677 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 96.67860641, 2024-09-16T03:40:03.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30323033799999999, 2024-09-16T03:40:03.677 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.677 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.677 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:03.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2850 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920620+0000", 2024-09-16T03:40:03.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920621+0000", 2024-09-16T03:40:03.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.47e-07 2024-09-16T03:40:03.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920625+0000", 2024-09-16T03:40:03.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1620000000000001e-06 2024-09-16T03:40:03.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920632+0000", 2024-09-16T03:40:03.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0180000000000004e-06 2024-09-16T03:40:03.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920638+0000", 2024-09-16T03:40:03.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0950000000000001e-06 2024-09-16T03:40:03.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922161+0000", 2024-09-16T03:40:03.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015229880000000001 2024-09-16T03:40:03.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922222+0000", 2024-09-16T03:40:03.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0903999999999998e-05 2024-09-16T03:40:03.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,4,5,10", 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.922323+0000", 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101425 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.023371+0000", 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101047493 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123227+0000", 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099856057999999998 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123840+0000", 2024-09-16T03:40:03.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061324999999999997 2024-09-16T03:40:03.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.123902+0000", 2024-09-16T03:40:03.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1447999999999998e-05 2024-09-16T03:40:03.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:03.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223497+0000", 2024-09-16T03:40:03.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099595499000000004 2024-09-16T03:40:03.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:03.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223665+0000", 2024-09-16T03:40:03.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016807399999999999 2024-09-16T03:40:03.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223850+0000", 2024-09-16T03:40:03.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018507699999999999 2024-09-16T03:40:03.684 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.685 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.685 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.685 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2851 4.18 4:1f63731d:::smithi15736696-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-16T03:40:03.685 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:40:03.685 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 96.678570313999998, 2024-09-16T03:40:03.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30354833399999998, 2024-09-16T03:40:03.685 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.685 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.685 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2851 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920656+0000", 2024-09-16T03:40:03.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920657+0000", 2024-09-16T03:40:03.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0100000000000004e-07 2024-09-16T03:40:03.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920660+0000", 2024-09-16T03:40:03.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6519999999999999e-06 2024-09-16T03:40:03.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920668+0000", 2024-09-16T03:40:03.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1919999999999999e-06 2024-09-16T03:40:03.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:26.920674+0000", 2024-09-16T03:40:03.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-16T03:40:03.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126299+0000", 2024-09-16T03:40:03.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20562534700000001 2024-09-16T03:40:03.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:03.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.126358+0000", 2024-09-16T03:40:03.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8459999999999999e-05 2024-09-16T03:40:03.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.223913+0000", 2024-09-16T03:40:03.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097555263000000003 2024-09-16T03:40:03.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224009+0000", 2024-09-16T03:40:03.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5723999999999994e-05 2024-09-16T03:40:03.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:27.224204+0000", 2024-09-16T03:40:03.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000195498 2024-09-16T03:40:03.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.692 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.692 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3570 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:03.692 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:40:03.692 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 72.593436990000001, 2024-09-16T03:40:03.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.353157053, 2024-09-16T03:40:03.693 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.693 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.693 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:03.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.693 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3570 2024-09-16T03:40:03.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.693 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:40:03.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005789+0000", 2024-09-16T03:40:03.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.005796+0000", 2024-09-16T03:40:03.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9229999999999996e-06 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006222+0000", 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000426282 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006240+0000", 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7858000000000001e-05 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006249+0000", 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7120000000000006e-06 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006323+0000", 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3808999999999998e-05 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.006333+0000", 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0488e-05 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358910+0000", 2024-09-16T03:40:03.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.35257616400000003 2024-09-16T03:40:03.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358946+0000", 2024-09-16T03:40:03.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6816999999999997e-05 2024-09-16T03:40:03.697 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.697 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.697 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.698 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3571 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:03.698 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:40:03.698 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 72.586931833999998, 2024-09-16T03:40:03.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34670431699999998, 2024-09-16T03:40:03.698 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.698 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.698 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:03.698 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.698 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.698 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.698 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3571 2024-09-16T03:40:03.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.698 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:40:03.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012294+0000", 2024-09-16T03:40:03.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012298+0000", 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.18e-06 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012947+0000", 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064929600000000005 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012961+0000", 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4341e-05 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.012970+0000", 2024-09-16T03:40:03.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2740000000000007e-06 2024-09-16T03:40:03.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.013025+0000", 2024-09-16T03:40:03.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5353999999999999e-05 2024-09-16T03:40:03.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:03.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.013033+0000", 2024-09-16T03:40:03.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4619999999999996e-06 2024-09-16T03:40:03.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358989+0000", 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34595562699999999 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.358999+0000", 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7829999999999993e-06 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3572 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 72.580911078, 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34073019300000001, 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3572 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:40:03.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018315+0000", 2024-09-16T03:40:03.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018317+0000", 2024-09-16T03:40:03.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4610000000000001e-06 2024-09-16T03:40:03.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018779+0000", 2024-09-16T03:40:03.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00046202500000000002 2024-09-16T03:40:03.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018793+0000", 2024-09-16T03:40:03.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4629e-05 2024-09-16T03:40:03.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018799+0000", 2024-09-16T03:40:03.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1589999999999998e-06 2024-09-16T03:40:03.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018867+0000", 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7879000000000005e-05 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018888+0000", 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0747999999999999e-05 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359036+0000", 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34014758 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359045+0000", 2024-09-16T03:40:03.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7119999999999995e-06 2024-09-16T03:40:03.708 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.708 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.708 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.708 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3573 4.28 4.f21f628 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:03.708 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:40:03.708 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 72.580411384000001, 2024-09-16T03:40:03.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34027060599999998, 2024-09-16T03:40:03.708 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.708 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3573 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018815+0000", 2024-09-16T03:40:03.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018816+0000", 2024-09-16T03:40:03.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1400000000000001e-06 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018821+0000", 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4179999999999996e-06 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018829+0000", 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6320000000000008e-06 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018835+0000", 2024-09-16T03:40:03.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0399999999999998e-06 2024-09-16T03:40:03.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018953+0000", 2024-09-16T03:40:03.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117713 2024-09-16T03:40:03.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:03.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018967+0000", 2024-09-16T03:40:03.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4334e-05 2024-09-16T03:40:03.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359078+0000", 2024-09-16T03:40:03.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34011044499999998 2024-09-16T03:40:03.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359086+0000", 2024-09-16T03:40:03.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8839999999999994e-06 2024-09-16T03:40:03.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.713 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3574 4.28 4.f21f628 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 72.580378229000004, 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.340272882, 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3574 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018848+0000", 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018849+0000", 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9599999999999998e-07 2024-09-16T03:40:03.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018852+0000", 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9749999999999999e-06 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018867+0000", 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5102e-05 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018882+0000", 2024-09-16T03:40:03.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5319e-05 2024-09-16T03:40:03.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018993+0000", 2024-09-16T03:40:03.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110457 2024-09-16T03:40:03.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:03.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.018998+0000", 2024-09-16T03:40:03.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.028e-06 2024-09-16T03:40:03.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359114+0000", 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34011593099999998 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.359121+0000", 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2740000000000002e-06 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.718 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:5031 4.37 4:ed874e32:::smithi15736696-14:head [read 1065058~1624128,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e170)", 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:54.901612+0000", 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.6976141380000005, 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40269039400000001, 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5031 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901612+0000", 2024-09-16T03:40:03.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901612+0000", 2024-09-16T03:40:03.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901615+0000", 2024-09-16T03:40:03.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6649999999999999e-06 2024-09-16T03:40:03.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901621+0000", 2024-09-16T03:40:03.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5980000000000002e-06 2024-09-16T03:40:03.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901637+0000", 2024-09-16T03:40:03.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5427999999999999e-05 2024-09-16T03:40:03.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901646+0000", 2024-09-16T03:40:03.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2199999999999998e-06 2024-09-16T03:40:03.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.001856+0000", 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10021042099999999 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.001902+0000", 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5169000000000001e-05 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.303461+0000", 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30155930600000003 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.303560+0000", 2024-09-16T03:40:03.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8777000000000007e-05 2024-09-16T03:40:03.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.724 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.724 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.304303+0000", 2024-09-16T03:40:03.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00074281 2024-09-16T03:40:03.724 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.724 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.724 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:5032 4.37 4:ed874e32:::smithi15736696-14:head [sparse-read 485321~1785976] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e170)", 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:54.901664+0000", 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.6975619680000005, 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50375708399999997, 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5032 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901664+0000", 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901664+0000", 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901665+0000", 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901669+0000", 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.048e-06 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901676+0000", 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6619999999999999e-06 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901682+0000", 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9519999999999999e-06 2024-09-16T03:40:03.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.001936+0000", 2024-09-16T03:40:03.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10025371299999999 2024-09-16T03:40:03.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:03.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.001940+0000", 2024-09-16T03:40:03.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5510000000000001e-06 2024-09-16T03:40:03.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.404766+0000", 2024-09-16T03:40:03.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40282600200000002 2024-09-16T03:40:03.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.404992+0000", 2024-09-16T03:40:03.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022603599999999999 2024-09-16T03:40:03.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.730 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.405421+0000", 2024-09-16T03:40:03.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042927800000000001 2024-09-16T03:40:03.730 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.730 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.730 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.731 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:5033 4.37 4:ed874e32:::smithi15736696-14:head [sparse-read 2351971~323757] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e170)", 2024-09-16T03:40:03.731 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:54.901692+0000", 2024-09-16T03:40:03.731 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.6975339070000004, 2024-09-16T03:40:03.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60441299900000001, 2024-09-16T03:40:03.731 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:03.731 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:03.731 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:03.731 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:03.731 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:03.732 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:03.732 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5033 2024-09-16T03:40:03.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.732 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:03.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:03.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901692+0000", 2024-09-16T03:40:03.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:03.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901692+0000", 2024-09-16T03:40:03.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901693+0000", 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3200000000000005e-07 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901696+0000", 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7140000000000002e-06 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901702+0000", 2024-09-16T03:40:03.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2639999999999997e-06 2024-09-16T03:40:03.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:03.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.901707+0000", 2024-09-16T03:40:03.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3229999999999997e-06 2024-09-16T03:40:03.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.202653+0000", 2024-09-16T03:40:03.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30094582199999997 2024-09-16T03:40:03.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:03.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.202661+0000", 2024-09-16T03:40:03.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.089e-06 2024-09-16T03:40:03.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:03.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.505621+0000", 2024-09-16T03:40:03.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30295951399999999 2024-09-16T03:40:03.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:03.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.505812+0000", 2024-09-16T03:40:03.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019083500000000001 2024-09-16T03:40:03.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:03.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:03.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:03.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.506105+0000", 2024-09-16T03:40:03.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029380600000000003 2024-09-16T03:40:03.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.736 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.736 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:03.736 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:03.736 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:40:03.738 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T03:40:03.738 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:40:03.736+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:40:03.838 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:40:03.836+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:40:03.846 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:40:03.846 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:40:03.846 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:40:03.846 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:40:03.855 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T03:40:03.939 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:40:03.937+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:40:04.003 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:40:04.003 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:40:04.003 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:40:04.003 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:40:04.003 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:40:04.011 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T03:40:04.040 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:40:04.039+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:40:04.121 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:40:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:40:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:40:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:40:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:34 4.e 4:773b934b:::smithi15736696-9:head [write 1951503~182953] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:40:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:40:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 172.064156459, 2024-09-16T03:40:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50346527299999999, 2024-09-16T03:40:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.122 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-16T03:40:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:40:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054411+0000", 2024-09-16T03:40:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054413+0000", 2024-09-16T03:40:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0669999999999999e-06 2024-09-16T03:40:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054525+0000", 2024-09-16T03:40:04.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111462 2024-09-16T03:40:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054538+0000", 2024-09-16T03:40:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3341e-05 2024-09-16T03:40:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054545+0000", 2024-09-16T03:40:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.584e-06 2024-09-16T03:40:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054608+0000", 2024-09-16T03:40:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2953999999999994e-05 2024-09-16T03:40:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054613+0000", 2024-09-16T03:40:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6110000000000001e-06 2024-09-16T03:40:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154852+0000", 2024-09-16T03:40:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10023838 2024-09-16T03:40:04.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.154915+0000", 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3138000000000004e-05 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155001+0000", 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5730999999999996e-05 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.156166+0000", 2024-09-16T03:40:04.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001165485 2024-09-16T03:40:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255909+0000", 2024-09-16T03:40:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099742558999999995 2024-09-16T03:40:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557800+0000", 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30189130600000003 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557836+0000", 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6566000000000002e-05 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557877+0000", 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0089000000000002e-05 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.129 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:35 4.e 4:773b934b:::smithi15736696-9:head [setxattr _header (55) in=7b,truncate 2134456] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:40:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:40:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 172.06400812800001, 2024-09-16T03:40:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50344320499999995, 2024-09-16T03:40:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.130 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054560+0000", 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5899999999999996e-07 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054564+0000", 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5710000000000002e-06 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054569+0000", 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0490000000000003e-06 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054578+0000", 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5689999999999994e-06 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054639+0000", 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1061000000000004e-05 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054644+0000", 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3409999999999996e-06 2024-09-16T03:40:04.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155607+0000", 2024-09-16T03:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100963372 2024-09-16T03:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155652+0000", 2024-09-16T03:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4969999999999998e-05 2024-09-16T03:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:40:04.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155722+0000", 2024-09-16T03:40:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0130999999999997e-05 2024-09-16T03:40:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.157921+0000", 2024-09-16T03:40:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021990159999999998 2024-09-16T03:40:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.255940+0000", 2024-09-16T03:40:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.098018569999999999 2024-09-16T03:40:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557930+0000", 2024-09-16T03:40:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.301990333 2024-09-16T03:40:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.557963+0000", 2024-09-16T03:40:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2348000000000003e-05 2024-09-16T03:40:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558003+0000", 2024-09-16T03:40:04.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:40:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:36 4.e 4:773b934b:::smithi15736696-9:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-16T03:40:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:40:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 172.063969439, 2024-09-16T03:40:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50353811800000003, 2024-09-16T03:40:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:04.137 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-16T03:40:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:40:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054598+0000", 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054599+0000", 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4200000000000001e-07 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054601+0000", 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4360000000000001e-06 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054606+0000", 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1019999999999996e-06 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054610+0000", 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.681e-06 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054662+0000", 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1545000000000002e-05 2024-09-16T03:40:04.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.054664+0000", 2024-09-16T03:40:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1919999999999999e-06 2024-09-16T03:40:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155903+0000", 2024-09-16T03:40:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101239068 2024-09-16T03:40:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:04.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.155970+0000", 2024-09-16T03:40:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6934999999999995e-05 2024-09-16T03:40:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558029+0000", 2024-09-16T03:40:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40205888899999997 2024-09-16T03:40:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558078+0000", 2024-09-16T03:40:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9842999999999997e-05 2024-09-16T03:40:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:12.558136+0000", 2024-09-16T03:40:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7985000000000001e-05 2024-09-16T03:40:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:90 3.0 3:07a861fa:::795ff7d1dbb920ddce0e1e8e9c423222c62a5f547438edf9ea06ce4da8d102ed:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:40:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:40:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 128.56195627, 2024-09-16T03:40:04.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62288137399999999, 2024-09-16T03:40:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:40:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556611+0000", 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556614+0000", 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3439999999999999e-06 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556621+0000", 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7440000000000001e-06 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556632+0000", 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1725e-05 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556640+0000", 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0239999999999997e-06 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556709+0000", 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8787000000000004e-05 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556718+0000", 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6920000000000008e-06 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758133+0000", 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201414977 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758273+0000", 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140684 2024-09-16T03:40:04.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.758479+0000", 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020508599999999999 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862766+0000", 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.104287952 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.878363+0000", 2024-09-16T03:40:04.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015596601999999999 2024-09-16T03:40:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960000+0000", 2024-09-16T03:40:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081636630000000002 2024-09-16T03:40:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960162+0000", 2024-09-16T03:40:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000162597 2024-09-16T03:40:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179366+0000", 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21920357100000001 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179411+0000", 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5461999999999999e-05 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179493+0000", 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1496999999999996e-05 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:91 3.0 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:40:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:40:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 128.56191131200001, 2024-09-16T03:40:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70460932600000004, 2024-09-16T03:40:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556656+0000", 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556657+0000", 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1900000000000001e-07 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556663+0000", 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6579999999999999e-06 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556670+0000", 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3189999999999997e-06 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556675+0000", 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4900000000000002e-06 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656929+0000", 2024-09-16T03:40:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10025371299999999 2024-09-16T03:40:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.656944+0000", 2024-09-16T03:40:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5460000000000001e-05 2024-09-16T03:40:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861436+0000", 2024-09-16T03:40:04.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.204491969 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861676+0000", 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024051899999999999 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.861828+0000", 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015110199999999999 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862775+0000", 2024-09-16T03:40:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094718600000000003 2024-09-16T03:40:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079130+0000", 2024-09-16T03:40:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21635549200000001 2024-09-16T03:40:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179557+0000", 2024-09-16T03:40:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10042638399999999 2024-09-16T03:40:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179972+0000", 2024-09-16T03:40:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041574 2024-09-16T03:40:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261034+0000", 2024-09-16T03:40:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081061604999999995 2024-09-16T03:40:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261142+0000", 2024-09-16T03:40:04.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107688 2024-09-16T03:40:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261266+0000", 2024-09-16T03:40:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124082 2024-09-16T03:40:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:94 3.0 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head [call cas.chunk_put_ref in=16b] snapc 0=[] RETRY=3 ondisk+retry+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e70)", 2024-09-16T03:40:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:40:04.158 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 128.561885187, 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70479513699999996, 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:40:04.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556683+0000", 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6499999999999999e-07 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556685+0000", 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4279999999999999e-06 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556689+0000", 2024-09-16T03:40:04.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.709e-06 2024-09-16T03:40:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556693+0000", 2024-09-16T03:40:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2939999999999999e-06 2024-09-16T03:40:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657036+0000", 2024-09-16T03:40:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10034209600000001 2024-09-16T03:40:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.657054+0000", 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8162999999999999e-05 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862279+0000", 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20522560200000001 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862380+0000", 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101178 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862509+0000", 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128909 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.863063+0000", 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055327600000000001 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079172+0000", 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21610910899999999 2024-09-16T03:40:04.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179932+0000", 2024-09-16T03:40:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100760575 2024-09-16T03:40:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179997+0000", 2024-09-16T03:40:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5018999999999995e-05 2024-09-16T03:40:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261318+0000", 2024-09-16T03:40:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.081320838000000006 2024-09-16T03:40:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261385+0000", 2024-09-16T03:40:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7115999999999999e-05 2024-09-16T03:40:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.261478+0000", 2024-09-16T03:40:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2360000000000003e-05 2024-09-16T03:40:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.165 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1742 3.0 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head [write 0~1659392] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T03:40:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:40:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 128.56185678700001, 2024-09-16T03:40:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62317617400000003, 2024-09-16T03:40:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.166 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1742 2024-09-16T03:40:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:40:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556711+0000", 2024-09-16T03:40:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.556713+0000", 2024-09-16T03:40:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7209999999999999e-06 2024-09-16T03:40:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558531+0000", 2024-09-16T03:40:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018184480000000001 2024-09-16T03:40:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558539+0000", 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0830000000000004e-06 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.558546+0000", 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3369999999999996e-06 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759148+0000", 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20060235500000001 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759230+0000", 2024-09-16T03:40:04.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2391000000000004e-05 2024-09-16T03:40:04.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:40:04.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.759329+0000", 2024-09-16T03:40:04.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9147000000000005e-05 2024-09-16T03:40:04.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.862771+0000", 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.103441085 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960102+0000", 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097331259000000003 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:55.960218+0000", 2024-09-16T03:40:04.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011581100000000001 2024-09-16T03:40:04.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.079074+0000", 2024-09-16T03:40:04.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.118856436 2024-09-16T03:40:04.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179758+0000", 2024-09-16T03:40:04.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100683884 2024-09-16T03:40:04.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179800+0000", 2024-09-16T03:40:04.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1928000000000002e-05 2024-09-16T03:40:04.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:56.179887+0000", 2024-09-16T03:40:04.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7288999999999997e-05 2024-09-16T03:40:04.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.174 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.174 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.174 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:89 3.0 3:1c07e18b:::d3cef63b99e391058a88d444a56984ba6fc25b22efbc98e24ba274246c542d3c:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.78245429099999, 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50388158999999999, 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336113+0000", 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336115+0000", 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.866e-06 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336123+0000", 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6990000000000003e-06 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336136+0000", 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3342e-05 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336143+0000", 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2099999999999998e-06 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336214+0000", 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1538000000000005e-05 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336342+0000", 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127408 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.336593+0000", 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025150500000000001 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337284+0000", 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069060900000000001 2024-09-16T03:40:04.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338068+0000", 2024-09-16T03:40:04.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078438900000000003 2024-09-16T03:40:04.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.338107+0000", 2024-09-16T03:40:04.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8858e-05 2024-09-16T03:40:04.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538164+0000", 2024-09-16T03:40:04.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200056971 2024-09-16T03:40:04.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839843+0000", 2024-09-16T03:40:04.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30167921599999997 2024-09-16T03:40:04.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839905+0000", 2024-09-16T03:40:04.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1542000000000001e-05 2024-09-16T03:40:04.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.839995+0000", 2024-09-16T03:40:04.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0437000000000002e-05 2024-09-16T03:40:04.181 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.181 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.181 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.181 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:98 3.0 3:04dfaaf9:::3de53e1d584ced9b0f970855867dbc9a14b3fc90962aa20a81c1ed25ff4b961f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:04.181 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:40:04.181 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.78112639, 2024-09-16T03:40:04.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.70310345299999999, 2024-09-16T03:40:04.181 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.182 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.182 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.182 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:40:04.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337441+0000", 2024-09-16T03:40:04.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337442+0000", 2024-09-16T03:40:04.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0729999999999999e-06 2024-09-16T03:40:04.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337447+0000", 2024-09-16T03:40:04.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3259999999999997e-06 2024-09-16T03:40:04.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337457+0000", 2024-09-16T03:40:04.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.944e-06 2024-09-16T03:40:04.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337462+0000", 2024-09-16T03:40:04.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0780000000000003e-06 2024-09-16T03:40:04.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337511+0000", 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337574+0000", 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3114999999999995e-05 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.337723+0000", 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014882299999999999 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.339039+0000", 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001315524 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538225+0000", 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19918607199999999 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638838+0000", 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100613466 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.739726+0000", 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100887854 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040319+0000", 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30059265699999999 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040398+0000", 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8852999999999999e-05 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040545+0000", 2024-09-16T03:40:04.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000147236 2024-09-16T03:40:04.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.188 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.188 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.188 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:101 3.0 3:090a0a71:::0ecec3e4e81d718f460a9826fb4cef2398bf4c951ee653be84827f33946eef5f:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:04.188 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:40:04.188 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.681955644, 2024-09-16T03:40:04.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60435370600000005, 2024-09-16T03:40:04.188 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.189 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.189 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:40:04.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436612+0000", 2024-09-16T03:40:04.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436614+0000", 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.844e-06 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436623+0000", 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0739999999999999e-06 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436636+0000", 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2629999999999999e-05 2024-09-16T03:40:04.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436643+0000", 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7489999999999998e-06 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536906+0000", 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10026262599999999 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.536979+0000", 2024-09-16T03:40:04.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3311000000000003e-05 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537109+0000", 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013015800000000001 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538241+0000", 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001131726 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638904+0000", 2024-09-16T03:40:04.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066240799999999 2024-09-16T03:40:04.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.940200+0000", 2024-09-16T03:40:04.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30129597299999999 2024-09-16T03:40:04.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040604+0000", 2024-09-16T03:40:04.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040478999999999 2024-09-16T03:40:04.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040759+0000", 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015445500000000001 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040824+0000", 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5633e-05 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040966+0000", 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014132899999999999 2024-09-16T03:40:04.194 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.195 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.195 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.195 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:102 3.0 3:17742417:::632266e231c4c385ace473a8f4464285cf620fbc8799f68869353e2aa4660567:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:04.195 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:40:04.195 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.681908818, 2024-09-16T03:40:04.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.604540823, 2024-09-16T03:40:04.195 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.195 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.195 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.195 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436659+0000", 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436660+0000", 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2399999999999997e-07 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436663+0000", 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1599999999999998e-06 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436670+0000", 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.742e-06 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436675+0000", 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3839999999999996e-06 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537499+0000", 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100823987 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537602+0000", 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102837 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:40:04.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.537793+0000", 2024-09-16T03:40:04.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019123499999999999 2024-09-16T03:40:04.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638962+0000", 2024-09-16T03:40:04.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101168944 2024-09-16T03:40:04.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.840035+0000", 2024-09-16T03:40:04.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.201073481 2024-09-16T03:40:04.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040706+0000", 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.200670606 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041009+0000", 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030271600000000002 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041087+0000", 2024-09-16T03:40:04.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8310000000000001e-05 2024-09-16T03:40:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041127+0000", 2024-09-16T03:40:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0315000000000003e-05 2024-09-16T03:40:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041200+0000", 2024-09-16T03:40:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2381999999999999e-05 2024-09-16T03:40:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.11.23:103 3.0 3:28447854:::c811732c201b76a96e1532c4d9a18d4f4f2a8ddec3bf79945689a1a62c8a5a25:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e74)", 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 122.681863569, 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60467526100000002, 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:40:04.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436704+0000", 2024-09-16T03:40:04.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436705+0000", 2024-09-16T03:40:04.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.23e-07 2024-09-16T03:40:04.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436719+0000", 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3957999999999999e-05 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436725+0000", 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6100000000000002e-06 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.436730+0000", 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0609999999999997e-06 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538257+0000", 2024-09-16T03:40:04.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10152739500000001 2024-09-16T03:40:04.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538314+0000", 2024-09-16T03:40:04.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7355000000000002e-05 2024-09-16T03:40:04.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:40:04.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.538435+0000", 2024-09-16T03:40:04.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120967 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:01.638968+0000", 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100532994 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.040662+0000", 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40169386299999998 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041059+0000", 2024-09-16T03:40:04.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039723900000000002 2024-09-16T03:40:04.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041247+0000", 2024-09-16T03:40:04.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018756900000000001 2024-09-16T03:40:04.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041286+0000", 2024-09-16T03:40:04.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8971999999999999e-05 2024-09-16T03:40:04.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041318+0000", 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1628999999999998e-05 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:02.041379+0000", 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1926000000000003e-05 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.208 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2473 4.e 4:765e9179:::smithi15736696-44:head [write 1830877~681726] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.107878344, 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50285070099999996, 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2473 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:40:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010689+0000", 2024-09-16T03:40:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.010691+0000", 2024-09-16T03:40:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-06 2024-09-16T03:40:04.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011744+0000", 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001052486 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011755+0000", 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1073000000000001e-05 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011761+0000", 2024-09-16T03:40:04.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1600000000000003e-06 2024-09-16T03:40:04.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011804+0000", 2024-09-16T03:40:04.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2697999999999997e-05 2024-09-16T03:40:04.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.011885+0000", 2024-09-16T03:40:04.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.119e-05 2024-09-16T03:40:04.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.012013+0000", 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128066 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.013923+0000", 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019104739999999999 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.412920+0000", 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39899685200000001 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513347+0000", 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100427167 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513490+0000", 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142163 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.513540+0000", 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0482000000000002e-05 2024-09-16T03:40:04.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.215 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.215 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:137 3.0 3:20b08a9e:::0ba23f501764ab3447cc323c825ee8bff84c3c6e340c68c08117252e1f0b39bb:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:04.215 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:40:04.215 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 108.00402986, 2024-09-16T03:40:04.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50256677599999999, 2024-09-16T03:40:04.215 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.215 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.215 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:40:04.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:40:04.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114538+0000", 2024-09-16T03:40:04.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1200000000000003e-07 2024-09-16T03:40:04.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114542+0000", 2024-09-16T03:40:04.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7400000000000002e-06 2024-09-16T03:40:04.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114547+0000", 2024-09-16T03:40:04.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.42e-06 2024-09-16T03:40:04.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.114550+0000", 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4470000000000002e-06 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.115951+0000", 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014005459999999999 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116047+0000", 2024-09-16T03:40:04.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6899999999999997e-05 2024-09-16T03:40:04.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3,6,9", 2024-09-16T03:40:04.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.116270+0000", 2024-09-16T03:40:04.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022291699999999999 2024-09-16T03:40:04.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.117236+0000", 2024-09-16T03:40:04.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096522299999999997 2024-09-16T03:40:04.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.218932+0000", 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10169644799999999 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.414271+0000", 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19533916300000001 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.514717+0000", 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100445879 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616919+0000", 2024-09-16T03:40:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10220199100000001 2024-09-16T03:40:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.616980+0000", 2024-09-16T03:40:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0977999999999998e-05 2024-09-16T03:40:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:16.617105+0000", 2024-09-16T03:40:04.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012461199999999999 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3402 4.1b 4:dbb349e8:::smithi15736696-46:head [stat,delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e111)", 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:46.357580+0000", 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.760987928999995, 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.80350077600000003, 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3402 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357580+0000", 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357580+0000", 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357582+0000", 2024-09-16T03:40:04.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1890000000000001e-06 2024-09-16T03:40:04.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357588+0000", 2024-09-16T03:40:04.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.896e-06 2024-09-16T03:40:04.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357606+0000", 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7841000000000001e-05 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357617+0000", 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1698999999999999e-05 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357697+0000", 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9822999999999998e-05 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.357708+0000", 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1119e-05 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.962316+0000", 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60460744899999996 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.962430+0000", 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011423 2024-09-16T03:40:04.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,6,8,9,15", 2024-09-16T03:40:04.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:46.962530+0000", 2024-09-16T03:40:04.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100005 2024-09-16T03:40:04.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063659+0000", 2024-09-16T03:40:04.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101128564 2024-09-16T03:40:04.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063691+0000", 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2413999999999998e-05 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063701+0000", 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9180000000000006e-06 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063733+0000", 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2416000000000003e-05 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.063758+0000", 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.421e-05 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.160869+0000", 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097111571999999993 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.160922+0000", 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3025999999999997e-05 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:47.161081+0000", 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000158405 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4662 4.20 4:049ac94d:::smithi15736696-40:head [write 726168~704991] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.832696+0000", 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.285872097999999, 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52767523100000002, 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.230 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4662 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.832696+0000", 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.832696+0000", 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.832699+0000", 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4599999999999999e-06 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.833419+0000", 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00072038500000000004 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.833441+0000", 2024-09-16T03:40:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1135e-05 2024-09-16T03:40:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.833460+0000", 2024-09-16T03:40:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9296e-05 2024-09-16T03:40:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.955818+0000", 2024-09-16T03:40:04.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.12235790100000001 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.955889+0000", 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1039999999999997e-05 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,2,8,10,15", 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.956001+0000", 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111692 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.056147+0000", 2024-09-16T03:40:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100146179 2024-09-16T03:40:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.056166+0000", 2024-09-16T03:40:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9656999999999998e-05 2024-09-16T03:40:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.056217+0000", 2024-09-16T03:40:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0238999999999998e-05 2024-09-16T03:40:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.057817+0000", 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001600193 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360100+0000", 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30228278800000002 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360244+0000", 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014397800000000001 2024-09-16T03:40:04.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360289+0000", 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5374999999999998e-05 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360371+0000", 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1913e-05 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4663 4.20 4:049ac94d:::smithi15736696-40:head [write 1895071~683474] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:40:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.838668+0000", 2024-09-16T03:40:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.279899352000001, 2024-09-16T03:40:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52198192799999998, 2024-09-16T03:40:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4663 2024-09-16T03:40:04.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.838668+0000", 2024-09-16T03:40:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.838668+0000", 2024-09-16T03:40:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.838670+0000", 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.401e-06 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839234+0000", 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00056420899999999998 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839248+0000", 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4208e-05 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839255+0000", 2024-09-16T03:40:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.742e-06 2024-09-16T03:40:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.057859+0000", 2024-09-16T03:40:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21860384799999999 2024-09-16T03:40:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.057934+0000", 2024-09-16T03:40:04.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5638999999999999e-05 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,2,8,10,15", 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.058042+0000", 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107314 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.059148+0000", 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011060079999999999 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159745+0000", 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10059675 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159780+0000", 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5351000000000001e-05 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.259887+0000", 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10010670200000001 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360172+0000", 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100285759 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360535+0000", 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036267599999999998 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360570+0000", 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5138999999999997e-05 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360650+0000", 2024-09-16T03:40:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0181999999999999e-05 2024-09-16T03:40:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4664 4.20 4:049ac94d:::smithi15736696-40:head [setxattr _header (56) in=7b,truncate 3126168] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:40:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.839266+0000", 2024-09-16T03:40:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.279302221999998, 2024-09-16T03:40:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52151806899999997, 2024-09-16T03:40:04.245 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4664 2024-09-16T03:40:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839266+0000", 2024-09-16T03:40:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839266+0000", 2024-09-16T03:40:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839266+0000", 2024-09-16T03:40:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2900000000000002e-07 2024-09-16T03:40:04.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839270+0000", 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2100000000000002e-06 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839273+0000", 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.715e-06 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839276+0000", 2024-09-16T03:40:04.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2200000000000001e-06 2024-09-16T03:40:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159242+0000", 2024-09-16T03:40:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.31996554700000002 2024-09-16T03:40:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159311+0000", 2024-09-16T03:40:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9419999999999996e-05 2024-09-16T03:40:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,2,8,10,15", 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.159423+0000", 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011188600000000001 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360380+0000", 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20095712900000001 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360414+0000", 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3176000000000002e-05 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360449+0000", 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4872000000000001e-05 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360479+0000", 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0402000000000001e-05 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360509+0000", 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9907999999999999e-05 2024-09-16T03:40:04.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360685+0000", 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000175897 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360717+0000", 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2342999999999998e-05 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.360784+0000", 2024-09-16T03:40:04.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6514999999999996e-05 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4665 4.20 4:049ac94d:::smithi15736696-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e151)", 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.839281+0000", 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.279286974999998, 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.82253416700000004, 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4665 2024-09-16T03:40:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839281+0000", 2024-09-16T03:40:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839281+0000", 2024-09-16T03:40:04.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839281+0000", 2024-09-16T03:40:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3299999999999998e-07 2024-09-16T03:40:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839283+0000", 2024-09-16T03:40:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6470000000000001e-06 2024-09-16T03:40:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839286+0000", 2024-09-16T03:40:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.506e-06 2024-09-16T03:40:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.839290+0000", 2024-09-16T03:40:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9559999999999999e-06 2024-09-16T03:40:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.661604+0000", 2024-09-16T03:40:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.822313819 2024-09-16T03:40:04.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.661677+0000", 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2712000000000004e-05 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.661815+0000", 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013819400000000001 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4857 4.e 4:70297eed:::smithi15736696-31:head [assert-version v93,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e165)", 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.036628+0000", 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.081939805999999, 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.54220494600000002, 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4857 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036628+0000", 2024-09-16T03:40:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036628+0000", 2024-09-16T03:40:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036629+0000", 2024-09-16T03:40:04.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.502e-06 2024-09-16T03:40:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036635+0000", 2024-09-16T03:40:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5430000000000001e-06 2024-09-16T03:40:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036646+0000", 2024-09-16T03:40:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0886999999999999e-05 2024-09-16T03:40:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036654+0000", 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0839999999999992e-06 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036701+0000", 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.036765+0000", 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4039000000000001e-05 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 8,15", 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.178117+0000", 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.141351912 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.179109+0000", 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00099158100000000006 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.379230+0000", 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20012068599999999 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.578763+0000", 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19953386300000001 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.578801+0000", 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7880999999999999e-05 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.578833+0000", 2024-09-16T03:40:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1608e-05 2024-09-16T03:40:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.264 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.264 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:40:04.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T03:40:04.266 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:40:04.264+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:40:04.366 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:40:04.364+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:40:04.374 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:40:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:40:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:5292 3.3 3.bb668cab (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e179)", 2024-09-16T03:40:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:40:04.309162+0000", 2024-09-16T03:40:04.374 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.06425198, 2024-09-16T03:40:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.064271666000000005, 2024-09-16T03:40:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "queued for pg", 2024-09-16T03:40:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5292 2024-09-16T03:40:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:40:04.309162+0000", 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:40:04.309162+0000", 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:40:04.309163+0000", 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.333e-06 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:40:04.309168+0000", 2024-09-16T03:40:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3090000000000003e-06 2024-09-16T03:40:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:40:04.309182+0000", 2024-09-16T03:40:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3723000000000001e-05 2024-09-16T03:40:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:40:04.309190+0000", 2024-09-16T03:40:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9950000000000005e-06 2024-09-16T03:40:04.377 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-16T03:40:04.378 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-09-16T03:40:04.378 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:40:04.383 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T03:40:04.466 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:40:04.464+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:40:04.491 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:40:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:40:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:40:04.491 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:40:04.491 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:40:04.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T03:40:04.567 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:40:04.566+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:40:04.609 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:40:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:40:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:40:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:40:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:303 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-16T03:40:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:40:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 168.195081902, 2024-09-16T03:40:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50956986000000004, 2024-09-16T03:40:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.610 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 303 2024-09-16T03:40:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:40:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411719+0000", 2024-09-16T03:40:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411721+0000", 2024-09-16T03:40:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0200000000000001e-06 2024-09-16T03:40:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411729+0000", 2024-09-16T03:40:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1240000000000005e-06 2024-09-16T03:40:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411744+0000", 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5099000000000001e-05 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.411751+0000", 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.029e-06 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511892+0000", 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100140879 2024-09-16T03:40:04.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.511961+0000", 2024-09-16T03:40:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9379000000000001e-05 2024-09-16T03:40:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:40:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.613647+0000", 2024-09-16T03:40:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101685994 2024-09-16T03:40:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.614294+0000", 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00064692799999999996 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.714962+0000", 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10066765 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921157+0000", 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.206195463 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921233+0000", 2024-09-16T03:40:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5446999999999998e-05 2024-09-16T03:40:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:16.921288+0000", 2024-09-16T03:40:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5847999999999998e-05 2024-09-16T03:40:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.616 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:503 4.a 4:51338165:::smithi15736696-4:head [assert-version v3,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:40:04.617 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:40:04.617 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 162.78500735, 2024-09-16T03:40:04.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.60521621599999997, 2024-09-16T03:40:04.617 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.617 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.617 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.617 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.617 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 503 2024-09-16T03:40:04.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.617 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:40:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821793+0000", 2024-09-16T03:40:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821794+0000", 2024-09-16T03:40:04.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5899999999999995e-07 2024-09-16T03:40:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821798+0000", 2024-09-16T03:40:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2030000000000002e-06 2024-09-16T03:40:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821804+0000", 2024-09-16T03:40:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2790000000000004e-06 2024-09-16T03:40:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.821810+0000", 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4249999999999997e-06 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922212+0000", 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10040228800000001 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:21.922307+0000", 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5089999999999999e-05 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:40:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.126007+0000", 2024-09-16T03:40:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20369931499999999 2024-09-16T03:40:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.127605+0000", 2024-09-16T03:40:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001598213 2024-09-16T03:40:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.327263+0000", 2024-09-16T03:40:04.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.199657905 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426900+0000", 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099637217 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.426963+0000", 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2989000000000003e-05 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:22.427009+0000", 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6533000000000003e-05 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1690 4.a 4:51338165:::smithi15736696-4:head [assert-version v63,set-chunk in=126b] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 134.15651709299999, 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52002349699999995, 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.623 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1690 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450283+0000", 2024-09-16T03:40:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450286+0000", 2024-09-16T03:40:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2280000000000001e-06 2024-09-16T03:40:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450293+0000", 2024-09-16T03:40:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5089999999999996e-06 2024-09-16T03:40:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450308+0000", 2024-09-16T03:40:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4955e-05 2024-09-16T03:40:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450316+0000", 2024-09-16T03:40:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7079999999999997e-06 2024-09-16T03:40:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450386+0000", 2024-09-16T03:40:04.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9715999999999995e-05 2024-09-16T03:40:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.450478+0000", 2024-09-16T03:40:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2610999999999998e-05 2024-09-16T03:40:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:40:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.665802+0000", 2024-09-16T03:40:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.21532342700000001 2024-09-16T03:40:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668724+0000", 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029220600000000002 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.668758+0000", 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.434e-05 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970243+0000", 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30148501 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970278+0000", 2024-09-16T03:40:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4965000000000002e-05 2024-09-16T03:40:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970307+0000", 2024-09-16T03:40:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8968000000000001e-05 2024-09-16T03:40:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1656 4.12 4:4a9d235d:::smithi15736696-1:head [setxattr _header (55) in=7b,truncate 2252138] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 134.06342541199999, 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72904596899999996, 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1656 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543375+0000", 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543376+0000", 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4200000000000004e-07 2024-09-16T03:40:04.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543383+0000", 2024-09-16T03:40:04.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6290000000000004e-06 2024-09-16T03:40:04.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543390+0000", 2024-09-16T03:40:04.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2110000000000001e-06 2024-09-16T03:40:04.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543396+0000", 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.02e-06 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543461+0000", 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4876999999999996e-05 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543465+0000", 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2139999999999998e-06 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970490+0000", 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42702478300000002 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970556+0000", 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6137999999999996e-05 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2,9,11,14,15", 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970670+0000", 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114416 2024-09-16T03:40:04.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971375+0000", 2024-09-16T03:40:04.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070496000000000003 2024-09-16T03:40:04.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071773+0000", 2024-09-16T03:40:04.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100397945 2024-09-16T03:40:04.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.071823+0000", 2024-09-16T03:40:04.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9431999999999997e-05 2024-09-16T03:40:04.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.072067+0000", 2024-09-16T03:40:04.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000243856 2024-09-16T03:40:04.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.171991+0000", 2024-09-16T03:40:04.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099924052999999999 2024-09-16T03:40:04.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272176+0000", 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100184963 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272251+0000", 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5902000000000006e-05 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272421+0000", 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016962800000000001 2024-09-16T03:40:04.637 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.638 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.638 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.638 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1657 4.12 4:4a9d235d:::smithi15736696-1:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e64)", 2024-09-16T03:40:04.638 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:40:04.638 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 134.063393426, 2024-09-16T03:40:04.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72934181899999995, 2024-09-16T03:40:04.638 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.638 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.638 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:04.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.638 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.639 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.639 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1657 2024-09-16T03:40:04.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.639 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:40:04.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543407+0000", 2024-09-16T03:40:04.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543408+0000", 2024-09-16T03:40:04.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.13e-07 2024-09-16T03:40:04.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543410+0000", 2024-09-16T03:40:04.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7030000000000002e-06 2024-09-16T03:40:04.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543417+0000", 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.567e-06 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543423+0000", 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0619999999999997e-06 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543489+0000", 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6315999999999993e-05 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.543498+0000", 2024-09-16T03:40:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7239999999999998e-06 2024-09-16T03:40:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.970952+0000", 2024-09-16T03:40:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.42745424399999998 2024-09-16T03:40:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:50.971012+0000", 2024-09-16T03:40:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9771000000000001e-05 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272463+0000", 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.30145090800000002 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272558+0000", 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4863000000000003e-05 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:51.272749+0000", 2024-09-16T03:40:04.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191048 2024-09-16T03:40:04.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.644 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.644 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.644 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2987 4.a 4:5a5e9bc2:::smithi15736696-36:head [write 1607751~551752] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:40:04.644 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:40:04.644 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.687852921000001, 2024-09-16T03:40:04.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49279943700000001, 2024-09-16T03:40:04.644 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.644 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.644 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2987 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918948+0000", 2024-09-16T03:40:04.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.918949+0000", 2024-09-16T03:40:04.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.86e-06 2024-09-16T03:40:04.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919579+0000", 2024-09-16T03:40:04.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062909000000000001 2024-09-16T03:40:04.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919591+0000", 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2078e-05 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919596+0000", 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019705+0000", 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100108405 2024-09-16T03:40:04.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019754+0000", 2024-09-16T03:40:04.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9258999999999997e-05 2024-09-16T03:40:04.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:40:04.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.019847+0000", 2024-09-16T03:40:04.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2954999999999993e-05 2024-09-16T03:40:04.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.023003+0000", 2024-09-16T03:40:04.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031561110000000001 2024-09-16T03:40:04.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.122544+0000", 2024-09-16T03:40:04.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099540553000000004 2024-09-16T03:40:04.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411658+0000", 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.28911426699999998 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411696+0000", 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8430000000000003e-05 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.411747+0000", 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0745999999999998e-05 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2988 4.a 4:5a5e9bc2:::smithi15736696-36:head [setxattr _header (56) in=7b,truncate 2487469] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.687191182000007, 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59236646500000001, 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.651 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2988 2024-09-16T03:40:04.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.652 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:40:04.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919609+0000", 2024-09-16T03:40:04.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919610+0000", 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.048e-06 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919615+0000", 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8239999999999999e-06 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919621+0000", 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4380000000000003e-06 2024-09-16T03:40:04.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:31.919626+0000", 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2000000000000002e-06 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.020922+0000", 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101295782 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.020995+0000", 2024-09-16T03:40:04.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3256999999999993e-05 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.021088+0000", 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3419999999999997e-05 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.023007+0000", 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019186380000000001 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.222530+0000", 2024-09-16T03:40:04.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19952276899999999 2024-09-16T03:40:04.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511887+0000", 2024-09-16T03:40:04.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.289357378 2024-09-16T03:40:04.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511927+0000", 2024-09-16T03:40:04.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0357e-05 2024-09-16T03:40:04.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:32.511976+0000", 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8353999999999998e-05 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3258 4.1a 4:5e3c0760:::smithi15736696-24:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e107)", 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 84.176740307000003, 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3302292470000001, 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.657 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3258 2024-09-16T03:40:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:40:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430060+0000", 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430061+0000", 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.27e-07 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430062+0000", 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6780000000000001e-06 2024-09-16T03:40:04.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430066+0000", 2024-09-16T03:40:04.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5269999999999999e-06 2024-09-16T03:40:04.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430069+0000", 2024-09-16T03:40:04.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:40:04.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430119+0000", 2024-09-16T03:40:04.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0212e-05 2024-09-16T03:40:04.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-16T03:40:04.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.430122+0000", 2024-09-16T03:40:04.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7319999999999998e-06 2024-09-16T03:40:04.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.631493+0000", 2024-09-16T03:40:04.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20137142499999999 2024-09-16T03:40:04.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:40.631579+0000", 2024-09-16T03:40:04.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5418000000000001e-05 2024-09-16T03:40:04.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:41.760289+0000", 2024-09-16T03:40:04.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1287106680000001 2024-09-16T03:40:04.662 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.662 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3701 4.a 4:51338165:::smithi15736696-4:head [assert-version v115,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e121)", 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.235893767999997, 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50414302200000005, 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3701 2024-09-16T03:40:04.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.664 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:40:04.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370907+0000", 2024-09-16T03:40:04.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370910+0000", 2024-09-16T03:40:04.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1200000000000002e-06 2024-09-16T03:40:04.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370923+0000", 2024-09-16T03:40:04.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3078000000000001e-05 2024-09-16T03:40:04.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370940+0000", 2024-09-16T03:40:04.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7461000000000001e-05 2024-09-16T03:40:04.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.370950+0000", 2024-09-16T03:40:04.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0091999999999999e-05 2024-09-16T03:40:04.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.371029+0000", 2024-09-16T03:40:04.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8879000000000002e-05 2024-09-16T03:40:04.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.371136+0000", 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106983 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.573316+0000", 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20217934500000001 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.573914+0000", 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00059857900000000002 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.763374+0000", 2024-09-16T03:40:04.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18945932200000001 2024-09-16T03:40:04.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.874936+0000", 2024-09-16T03:40:04.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.111562256 2024-09-16T03:40:04.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.875002+0000", 2024-09-16T03:40:04.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6050000000000006e-05 2024-09-16T03:40:04.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:56.875050+0000", 2024-09-16T03:40:04.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7856999999999999e-05 2024-09-16T03:40:04.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.669 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.669 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4258 4.a 4:50107449:::smithi15736696-27:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-09-16T03:40:04.669 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:16.528667+0000", 2024-09-16T03:40:04.669 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 48.078133557000001, 2024-09-16T03:40:04.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.47672299499999998, 2024-09-16T03:40:04.670 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.670 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.670 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:04.670 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.670 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.670 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.670 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4258 2024-09-16T03:40:04.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.670 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528667+0000", 2024-09-16T03:40:04.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528667+0000", 2024-09-16T03:40:04.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528669+0000", 2024-09-16T03:40:04.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.883e-06 2024-09-16T03:40:04.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528674+0000", 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2599999999999996e-06 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528687+0000", 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3220999999999999e-05 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528695+0000", 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528754+0000", 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9555000000000002e-05 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:16.528864+0000", 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010949 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:17.005390+0000", 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.47652628499999999 2024-09-16T03:40:04.673 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.674 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.674 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.674 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4369 4.a 4:51338165:::smithi15736696-4:head [write 1816008~503126] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:40:04.674 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:20.373551+0000", 2024-09-16T03:40:04.674 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.233249792000002, 2024-09-16T03:40:04.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.75753581699999994, 2024-09-16T03:40:04.674 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.674 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.674 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.674 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.674 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4369 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.373551+0000", 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.373551+0000", 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.373552+0000", 2024-09-16T03:40:04.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2550000000000001e-06 2024-09-16T03:40:04.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374063+0000", 2024-09-16T03:40:04.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00051145499999999996 2024-09-16T03:40:04.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374077+0000", 2024-09-16T03:40:04.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3535e-05 2024-09-16T03:40:04.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374087+0000", 2024-09-16T03:40:04.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0086e-05 2024-09-16T03:40:04.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374194+0000", 2024-09-16T03:40:04.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010661 2024-09-16T03:40:04.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:04.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374196+0000", 2024-09-16T03:40:04.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7980000000000001e-06 2024-09-16T03:40:04.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.780445+0000", 2024-09-16T03:40:04.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40624829800000001 2024-09-16T03:40:04.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.780585+0000", 2024-09-16T03:40:04.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014048700000000001 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.780748+0000", 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000162798 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782933+0000", 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002184994 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.030573+0000", 2024-09-16T03:40:04.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.247640049 2024-09-16T03:40:04.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.130869+0000", 2024-09-16T03:40:04.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100296195 2024-09-16T03:40:04.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131031+0000", 2024-09-16T03:40:04.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016220899999999999 2024-09-16T03:40:04.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131087+0000", 2024-09-16T03:40:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5047999999999999e-05 2024-09-16T03:40:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4370 4.a 4:51338165:::smithi15736696-4:head [setxattr _header (55) in=7b,truncate 2773221] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:40:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:20.374103+0000", 2024-09-16T03:40:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.232697272999999, 2024-09-16T03:40:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.757112648, 2024-09-16T03:40:04.681 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4370 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374103+0000", 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374103+0000", 2024-09-16T03:40:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374104+0000", 2024-09-16T03:40:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.192e-06 2024-09-16T03:40:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374109+0000", 2024-09-16T03:40:04.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7280000000000003e-06 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374117+0000", 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2039999999999997e-06 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374124+0000", 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3380000000000001e-06 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374212+0000", 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8663000000000001e-05 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374215+0000", 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3929999999999998e-06 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782160+0000", 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.407945112 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782254+0000", 2024-09-16T03:40:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4152000000000003e-05 2024-09-16T03:40:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 3,10", 2024-09-16T03:40:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782407+0000", 2024-09-16T03:40:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015309500000000001 2024-09-16T03:40:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.785130+0000", 2024-09-16T03:40:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0027231320000000001 2024-09-16T03:40:04.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.130717+0000", 2024-09-16T03:40:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34558722600000003 2024-09-16T03:40:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131123+0000", 2024-09-16T03:40:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040580199999999997 2024-09-16T03:40:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131164+0000", 2024-09-16T03:40:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0487e-05 2024-09-16T03:40:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131216+0000", 2024-09-16T03:40:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2123999999999998e-05 2024-09-16T03:40:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.688 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4371 4.a 4:51338165:::smithi15736696-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:40:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:20.374154+0000", 2024-09-16T03:40:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.232646373999998, 2024-09-16T03:40:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.75721231200000005, 2024-09-16T03:40:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4371 2024-09-16T03:40:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374154+0000", 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374154+0000", 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374155+0000", 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4099999999999997e-07 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374159+0000", 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.506e-06 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374167+0000", 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0870000000000007e-06 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.374173+0000", 2024-09-16T03:40:04.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.652e-06 2024-09-16T03:40:04.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.468957+0000", 2024-09-16T03:40:04.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.094784050999999994 2024-09-16T03:40:04.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:04.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.468967+0000", 2024-09-16T03:40:04.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6169999999999996e-06 2024-09-16T03:40:04.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782856+0000", 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.313889311 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:20.782918+0000", 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2055000000000001e-05 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131245+0000", 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.34832644699999998 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131301+0000", 2024-09-16T03:40:04.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5721999999999999e-05 2024-09-16T03:40:04.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:21.131366+0000", 2024-09-16T03:40:04.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5922999999999999e-05 2024-09-16T03:40:04.694 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.694 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.694 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.8.22:71 3.3 3.3135f983 (undecoded) ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:54.056053+0000", 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.55074728, 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.84896896600000005, 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.056053+0000", 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.056053+0000", 2024-09-16T03:40:04.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.056055+0000", 2024-09-16T03:40:04.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4529999999999999e-06 2024-09-16T03:40:04.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.056062+0000", 2024-09-16T03:40:04.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1779999999999997e-06 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.056072+0000", 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0169999999999999e-05 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.056080+0000", 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1030000000000002e-06 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.056150+0000", 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9793000000000001e-05 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.056161+0000", 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0855e-05 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.904989+0000", 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.84882853199999997 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.905022+0000", 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2882000000000001e-05 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:302 3.3 3.e4d729e7 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:54.206353+0000", 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.400447955000001, 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.698820302, 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.699 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:04.700 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.206353+0000", 2024-09-16T03:40:04.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.206353+0000", 2024-09-16T03:40:04.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.206354+0000", 2024-09-16T03:40:04.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.525e-06 2024-09-16T03:40:04.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.206361+0000", 2024-09-16T03:40:04.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9399999999999996e-06 2024-09-16T03:40:04.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.206370+0000", 2024-09-16T03:40:04.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9779999999999994e-06 2024-09-16T03:40:04.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306439+0000", 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100069477 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306494+0000", 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4317999999999997e-05 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306500+0000", 2024-09-16T03:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5579999999999997e-06 2024-09-16T03:40:04.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.905149+0000", 2024-09-16T03:40:04.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59864861000000003 2024-09-16T03:40:04.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.905173+0000", 2024-09-16T03:40:04.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3896e-05 2024-09-16T03:40:04.703 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.703 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.703 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.704 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:306 3.3 3.1aa37e7b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:04.704 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:54.306457+0000", 2024-09-16T03:40:04.704 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.300343025, 2024-09-16T03:40:04.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59875414699999996, 2024-09-16T03:40:04.704 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.704 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.704 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:04.704 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306457+0000", 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306457+0000", 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306459+0000", 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1260000000000001e-06 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306464+0000", 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.271e-06 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306471+0000", 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3150000000000003e-06 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306475+0000", 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.296e-06 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306533+0000", 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7822000000000003e-05 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306540+0000", 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.956e-06 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.905204+0000", 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59866355400000004 2024-09-16T03:40:04.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.905212+0000", 2024-09-16T03:40:04.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.807e-06 2024-09-16T03:40:04.708 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.708 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.708 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.708 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:308 3.3 3.85e1bb3f (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:04.708 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:54.306481+0000", 2024-09-16T03:40:04.708 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.300319946, 2024-09-16T03:40:04.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.69906433599999995, 2024-09-16T03:40:04.709 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.709 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.709 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:04.709 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306481+0000", 2024-09-16T03:40:04.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306481+0000", 2024-09-16T03:40:04.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306481+0000", 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2300000000000002e-07 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306482+0000", 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.556e-06 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306486+0000", 2024-09-16T03:40:04.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.428e-06 2024-09-16T03:40:04.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306490+0000", 2024-09-16T03:40:04.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9659999999999998e-06 2024-09-16T03:40:04.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306557+0000", 2024-09-16T03:40:04.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7274999999999995e-05 2024-09-16T03:40:04.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:04.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306562+0000", 2024-09-16T03:40:04.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2530000000000004e-06 2024-09-16T03:40:04.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.005525+0000", 2024-09-16T03:40:04.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.698962996 2024-09-16T03:40:04.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.005545+0000", 2024-09-16T03:40:04.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9539e-05 2024-09-16T03:40:04.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.713 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.713 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.713 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(osd.10.22:309 3.3 3.4358aacb (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:04.713 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:54.306494+0000", 2024-09-16T03:40:04.713 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.300306694, 2024-09-16T03:40:04.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.69964763799999996, 2024-09-16T03:40:04.713 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.713 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.713 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:04.713 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306494+0000", 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306494+0000", 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306494+0000", 2024-09-16T03:40:04.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.91e-07 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306495+0000", 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.268e-06 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.306498+0000", 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0180000000000002e-06 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.406622+0000", 2024-09-16T03:40:04.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100123963 2024-09-16T03:40:04.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.406713+0000", 2024-09-16T03:40:04.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0243999999999999e-05 2024-09-16T03:40:04.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:04.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.406722+0000", 2024-09-16T03:40:04.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9250000000000001e-06 2024-09-16T03:40:04.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.006136+0000", 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59941401900000002 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.006141+0000", 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9100000000000002e-06 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.717 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:5084 4.20 4:049ac94d:::smithi15736696-40:head [setxattr _header (56) in=7b,truncate 3748027] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:54.727173+0000", 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8796279469999995, 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49852101399999998, 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5084 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.718 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.727173+0000", 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.727173+0000", 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.727173+0000", 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0900000000000001e-07 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.727176+0000", 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8609999999999998e-06 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.727180+0000", 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2119999999999997e-06 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.727184+0000", 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3450000000000002e-06 2024-09-16T03:40:04.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.924914+0000", 2024-09-16T03:40:04.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.197730024 2024-09-16T03:40:04.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.925007+0000", 2024-09-16T03:40:04.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2844999999999996e-05 2024-09-16T03:40:04.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0,2", 2024-09-16T03:40:04.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.925107+0000", 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100097 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.925658+0000", 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055138099999999998 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.926008+0000", 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034999 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.225509+0000", 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29950091099999998 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.225611+0000", 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101741 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.225694+0000", 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2898000000000005e-05 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.723 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:5085 4.20 4:049ac94d:::smithi15736696-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:54.727188+0000", 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8796121400000008, 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.498815075, 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:04.724 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5085 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.727188+0000", 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.727188+0000", 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.727189+0000", 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8900000000000001e-07 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.727190+0000", 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4559999999999999e-06 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.727194+0000", 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5640000000000001e-06 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.727197+0000", 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2349999999999999e-06 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.925340+0000", 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19814354000000001 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:54.925425+0000", 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4253999999999997e-05 2024-09-16T03:40:04.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.225777+0000", 2024-09-16T03:40:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.300352069 2024-09-16T03:40:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.225886+0000", 2024-09-16T03:40:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109035 2024-09-16T03:40:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:04.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:55.226003+0000", 2024-09-16T03:40:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117633 2024-09-16T03:40:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:04.729 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:04.729 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:40:04.730 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T03:40:04.731 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:40:04.728+0000 7f12ee01d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33266) UID: 0 2024-09-16T03:40:04.831 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:40:04.829+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:40:04.878 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:40:04.878 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:40:04.878 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-16T03:40:04.879 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:40:04.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T03:40:04.932 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:40:04.930+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:40:04.994 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:40:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-16T03:40:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-16T03:40:04.994 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-16T03:40:04.994 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:40:05.002 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T03:40:05.033 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:40:05.031+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:40:05.134 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:40:05.133+0000 7f8bf4cd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33216) UID: 0 2024-09-16T03:40:05.152 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:40:05.152 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-16T03:40:05.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-16T03:40:05.152 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-16T03:40:05.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.152 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:790 4.d 4:b3fe05b2:::smithi15736696-15:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:40:05.152 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:40:05.152 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 159.26127553399999, 2024-09-16T03:40:05.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41456747799999999, 2024-09-16T03:40:05.153 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:05.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.153 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 790 2024-09-16T03:40:05.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.153 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887972+0000", 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887975+0000", 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0369999999999999e-06 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.887987+0000", 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1606e-05 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888016+0000", 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.917e-05 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888032+0000", 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5616000000000001e-05 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888178+0000", 2024-09-16T03:40:05.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612100000000001 2024-09-16T03:40:05.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:25.888307+0000", 2024-09-16T03:40:05.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012953899999999999 2024-09-16T03:40:05.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:26.302540+0000", 2024-09-16T03:40:05.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.41423238899999998 2024-09-16T03:40:05.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.157 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.157 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1418 4.c 4:3313e97c:::smithi15736696-21:head [write 615156~555834] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:40:05.157 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:40:05.157 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 143.92316436199999, 2024-09-16T03:40:05.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.40891108799999998, 2024-09-16T03:40:05.158 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.158 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.158 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1418 2024-09-16T03:40:05.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.158 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:40:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226083+0000", 2024-09-16T03:40:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226087+0000", 2024-09-16T03:40:05.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8890000000000004e-06 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226827+0000", 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000739298 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226848+0000", 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1418000000000001e-05 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.226858+0000", 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0434e-05 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327036+0000", 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10017769 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327098+0000", 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1760999999999994e-05 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:40:05.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.327230+0000", 2024-09-16T03:40:05.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131839 2024-09-16T03:40:05.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434369+0000", 2024-09-16T03:40:05.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.107139788 2024-09-16T03:40:05.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.434444+0000", 2024-09-16T03:40:05.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4171999999999994e-05 2024-09-16T03:40:05.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634830+0000", 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20038660599999999 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634937+0000", 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010716499999999999 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.634994+0000", 2024-09-16T03:40:05.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7027999999999997e-05 2024-09-16T03:40:05.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.164 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.164 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1421 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 3815156] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:40:05.164 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:40:05.164 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 143.909816213, 2024-09-16T03:40:05.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.497492088, 2024-09-16T03:40:05.164 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.164 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.164 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.164 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1421 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239431+0000", 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239432+0000", 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5999999999999991e-07 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239436+0000", 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239443+0000", 2024-09-16T03:40:05.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.968e-06 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239447+0000", 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6969999999999999e-06 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635225+0000", 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39577807399999998 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635290+0000", 2024-09-16T03:40:05.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4583000000000002e-05 2024-09-16T03:40:05.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,15", 2024-09-16T03:40:05.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635410+0000", 2024-09-16T03:40:05.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120684 2024-09-16T03:40:05.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:05.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636180+0000", 2024-09-16T03:40:05.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076997899999999995 2024-09-16T03:40:05.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.636799+0000", 2024-09-16T03:40:05.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061906600000000004 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736742+0000", 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099943041999999996 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736843+0000", 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010076799999999999 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736924+0000", 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0419000000000004e-05 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:1422 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 143.909794933, 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49764818700000002, 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:05.171 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1422 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:40:05.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239453+0000", 2024-09-16T03:40:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7599999999999998e-07 2024-09-16T03:40:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239455+0000", 2024-09-16T03:40:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9860000000000001e-06 2024-09-16T03:40:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239461+0000", 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4269999999999999e-06 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.239464+0000", 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2619999999999999e-06 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635736+0000", 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.396272287 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:05.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.635790+0000", 2024-09-16T03:40:05.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4252000000000002e-05 2024-09-16T03:40:05.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.736964+0000", 2024-09-16T03:40:05.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101173319 2024-09-16T03:40:05.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737024+0000", 2024-09-16T03:40:05.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9975000000000001e-05 2024-09-16T03:40:05.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:37:41.737101+0000", 2024-09-16T03:40:05.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7303000000000002e-05 2024-09-16T03:40:05.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.176 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.176 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.176 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2643 4.c 4:34492f38:::smithi15736696-33:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:40:05.176 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:40:05.176 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.268394184, 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.50004631700000002, 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2643 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:40:05.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880853+0000", 2024-09-16T03:40:05.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.880854+0000", 2024-09-16T03:40:05.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6599999999999994e-07 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881441+0000", 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058613000000000003 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881450+0000", 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6260000000000007e-06 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.881455+0000", 2024-09-16T03:40:05.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.515e-06 2024-09-16T03:40:05.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979409+0000", 2024-09-16T03:40:05.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097954645000000007 2024-09-16T03:40:05.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979512+0000", 2024-09-16T03:40:05.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000102701 2024-09-16T03:40:05.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.979684+0000", 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017176299999999999 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084699+0000", 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10501545599999999 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084714+0000", 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4738e-05 2024-09-16T03:40:05.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185858+0000", 2024-09-16T03:40:05.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101144074 2024-09-16T03:40:05.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380720+0000", 2024-09-16T03:40:05.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.194862072 2024-09-16T03:40:05.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:05.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380783+0000", 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3262000000000005e-05 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380900+0000", 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116369 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2644 4.c 4:34492f38:::smithi15736696-33:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:40:05.183 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.263717689, 2024-09-16T03:40:05.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.495569859, 2024-09-16T03:40:05.184 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.184 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.184 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2644 2024-09-16T03:40:05.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.184 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885530+0000", 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885531+0000", 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885813+0000", 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028188399999999999 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885824+0000", 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885830+0000", 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6289999999999998e-06 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983163+0000", 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.097333127000000005 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983273+0000", 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000109868 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.983458+0000", 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018554899999999999 2024-09-16T03:40:05.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:05.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084722+0000", 2024-09-16T03:40:05.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101263244 2024-09-16T03:40:05.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185717+0000", 2024-09-16T03:40:05.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100995575 2024-09-16T03:40:05.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185904+0000", 2024-09-16T03:40:05.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186682 2024-09-16T03:40:05.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.380965+0000", 2024-09-16T03:40:05.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.195060921 2024-09-16T03:40:05.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:05.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381030+0000", 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5174999999999999e-05 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.381100+0000", 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9877999999999997e-05 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2645 4.c 4:34492f38:::smithi15736696-33:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:40:05.190 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.263408188, 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59563304800000005, 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2645 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:40:05.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885839+0000", 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885840+0000", 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8400000000000004e-07 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885843+0000", 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.785e-06 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885848+0000", 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3979999999999998e-06 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885852+0000", 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6509999999999999e-06 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084768+0000", 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19891652000000001 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084831+0000", 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2717000000000003e-05 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,11,15", 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.084964+0000", 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132731 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185772+0000", 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10080797399999999 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185790+0000", 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7966999999999999e-05 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.185959+0000", 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000168777 2024-09-16T03:40:05.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481235+0000", 2024-09-16T03:40:05.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.29527644800000002 2024-09-16T03:40:05.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:05.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481307+0000", 2024-09-16T03:40:05.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1539999999999996e-05 2024-09-16T03:40:05.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481473+0000", 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016595600000000001 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:2646 4.c 4:34492f38:::smithi15736696-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 104.26339149899999, 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59596245699999995, 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:05.197 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.198 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.198 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.198 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2646 2024-09-16T03:40:05.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.198 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:40:05.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885856+0000", 2024-09-16T03:40:05.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885857+0000", 2024-09-16T03:40:05.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:40:05.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885860+0000", 2024-09-16T03:40:05.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8940000000000003e-06 2024-09-16T03:40:05.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885864+0000", 2024-09-16T03:40:05.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.602e-06 2024-09-16T03:40:05.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:20.885867+0000", 2024-09-16T03:40:05.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0089999999999999e-06 2024-09-16T03:40:05.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085196+0000", 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19932923399999999 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.085287+0000", 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0480000000000001e-05 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481521+0000", 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39623439799999999 2024-09-16T03:40:05.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481613+0000", 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2276999999999998e-05 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:21.481819+0000", 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020521499999999999 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3554 4.c 4.3e97c8cc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.699218474000006, 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2069473150000001, 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3554 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:40:05.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450029+0000", 2024-09-16T03:40:05.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450032+0000", 2024-09-16T03:40:05.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9639999999999999e-06 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450041+0000", 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450059+0000", 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.450069+0000", 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8320000000000001e-06 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.467620+0000", 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.017550676000000001 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:50.467626+0000", 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3430000000000001e-06 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656941+0000", 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1893151820000001 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656977+0000", 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5491999999999998e-05 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3581 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:05.207 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.123711661000002, 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.63145314900000005, 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3581 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:40:05.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025536+0000", 2024-09-16T03:40:05.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.025539+0000", 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4510000000000001e-06 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026168+0000", 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00062808199999999997 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026186+0000", 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8694000000000001e-05 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.026196+0000", 2024-09-16T03:40:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8800000000000003e-06 2024-09-16T03:40:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035407+0000", 2024-09-16T03:40:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0092111020000000005 2024-09-16T03:40:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035416+0000", 2024-09-16T03:40:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9870000000000005e-06 2024-09-16T03:40:05.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.557740+0000", 2024-09-16T03:40:05.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.522323918 2024-09-16T03:40:05.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.656989+0000", 2024-09-16T03:40:05.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.099249034999999999 2024-09-16T03:40:05.212 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.212 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.212 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.213 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3582 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:05.213 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:40:05.213 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.116469816000006, 2024-09-16T03:40:05.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.62429066099999997, 2024-09-16T03:40:05.213 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.213 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.213 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:05.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.213 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3582 2024-09-16T03:40:05.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032778+0000", 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.032779+0000", 2024-09-16T03:40:05.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2720000000000001e-06 2024-09-16T03:40:05.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035239+0000", 2024-09-16T03:40:05.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0024594180000000001 2024-09-16T03:40:05.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035261+0000", 2024-09-16T03:40:05.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2246e-05 2024-09-16T03:40:05.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.035273+0000", 2024-09-16T03:40:05.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2289999999999999e-05 2024-09-16T03:40:05.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038105+0000", 2024-09-16T03:40:05.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028319780000000002 2024-09-16T03:40:05.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:05.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038111+0000", 2024-09-16T03:40:05.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3099999999999997e-06 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657045+0000", 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61893337199999998 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657069+0000", 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3774999999999999e-05 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.218 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3583 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:05.218 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:40:05.218 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.111361685999995, 2024-09-16T03:40:05.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61919224299999998, 2024-09-16T03:40:05.218 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.218 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.218 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:05.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.218 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3583 2024-09-16T03:40:05.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.219 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:40:05.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037886+0000", 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.037888+0000", 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6199999999999999e-06 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038032+0000", 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014443799999999999 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038045+0000", 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3417999999999999e-05 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038052+0000", 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3890000000000002e-06 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038133+0000", 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0856e-05 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038136+0000", 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7160000000000001e-06 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657073+0000", 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61893632899999995 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657078+0000", 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4770000000000002e-06 2024-09-16T03:40:05.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3584 4.2c 4.1cf4922c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.111183392000001, 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61903161699999998, 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.224 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3584 2024-09-16T03:40:05.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.224 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:40:05.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038064+0000", 2024-09-16T03:40:05.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038065+0000", 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3499999999999996e-07 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038069+0000", 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038075+0000", 2024-09-16T03:40:05.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0549999999999996e-06 2024-09-16T03:40:05.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038078+0000", 2024-09-16T03:40:05.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4939999999999999e-06 2024-09-16T03:40:05.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.126379+0000", 2024-09-16T03:40:05.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.088300141999999998 2024-09-16T03:40:05.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:05.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.126393+0000", 2024-09-16T03:40:05.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4827999999999999e-05 2024-09-16T03:40:05.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657092+0000", 2024-09-16T03:40:05.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.53069909500000001 2024-09-16T03:40:05.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657096+0000", 2024-09-16T03:40:05.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4139999999999998e-06 2024-09-16T03:40:05.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3585 4.2c 4.1cf4922c (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 74.111162832000005, 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.61902921799999999, 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.228 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3585 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038085+0000", 2024-09-16T03:40:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1899999999999998e-07 2024-09-16T03:40:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038087+0000", 2024-09-16T03:40:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-16T03:40:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038091+0000", 2024-09-16T03:40:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1590000000000003e-06 2024-09-16T03:40:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.038095+0000", 2024-09-16T03:40:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5779999999999999e-06 2024-09-16T03:40:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.138249+0000", 2024-09-16T03:40:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100154101 2024-09-16T03:40:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.138256+0000", 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4209999999999998e-06 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657111+0000", 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.51885431800000004 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:38:51.657114+0000", 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5190000000000001e-06 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3983 4.c 4:3313e97c:::smithi15736696-21:head [setxattr _header (56) in=7b,truncate 2151362] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:40:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:40:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.857254068000003, 2024-09-16T03:40:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49254613699999999, 2024-09-16T03:40:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3983 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291994+0000", 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9500000000000002e-07 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.291997+0000", 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9679999999999998e-06 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292002+0000", 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7700000000000001e-06 2024-09-16T03:40:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292005+0000", 2024-09-16T03:40:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3019999999999999e-06 2024-09-16T03:40:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.382133+0000", 2024-09-16T03:40:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.090127600000000002 2024-09-16T03:40:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.382184+0000", 2024-09-16T03:40:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1304999999999998e-05 2024-09-16T03:40:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 11,15", 2024-09-16T03:40:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.382301+0000", 2024-09-16T03:40:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116673 2024-09-16T03:40:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.383337+0000", 2024-09-16T03:40:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00103619 2024-09-16T03:40:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.583745+0000", 2024-09-16T03:40:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20040787600000001 2024-09-16T03:40:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784456+0000", 2024-09-16T03:40:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.20071103700000001 2024-09-16T03:40:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784496+0000", 2024-09-16T03:40:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9740999999999998e-05 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784540+0000", 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4079999999999998e-05 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:3984 4.c 4:3313e97c:::smithi15736696-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.857237886999997, 2024-09-16T03:40:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.49270909200000002, 2024-09-16T03:40:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3984 2024-09-16T03:40:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:40:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:40:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292010+0000", 2024-09-16T03:40:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4900000000000001e-07 2024-09-16T03:40:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292012+0000", 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9250000000000002e-06 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292015+0000", 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4199999999999999e-06 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.292018+0000", 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.994e-06 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.388062+0000", 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.096043327999999997 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.388110+0000", 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8272999999999997e-05 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784615+0000", 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.39650470300000001 2024-09-16T03:40:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784664+0000", 2024-09-16T03:40:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8841000000000002e-05 2024-09-16T03:40:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:05.784719+0000", 2024-09-16T03:40:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5259000000000001e-05 2024-09-16T03:40:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4624 4.d 4:b3fe05b2:::smithi15736696-15:head [tier-flush] snapc 0=[] RETRY=1 ondisk+retry+read+ignore_cache+known_if_redirected+supports_pool_eio e150)", 2024-09-16T03:40:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:30.652818+0000", 2024-09-16T03:40:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.496429577999997, 2024-09-16T03:40:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.72344784900000003, 2024-09-16T03:40:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4624 2024-09-16T03:40:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652818+0000", 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652818+0000", 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652820+0000", 2024-09-16T03:40:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.357e-06 2024-09-16T03:40:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652827+0000", 2024-09-16T03:40:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6429999999999998e-06 2024-09-16T03:40:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652840+0000", 2024-09-16T03:40:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2571e-05 2024-09-16T03:40:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.652855+0000", 2024-09-16T03:40:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5671e-05 2024-09-16T03:40:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.657070+0000", 2024-09-16T03:40:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0042144820000000003 2024-09-16T03:40:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-16T03:40:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.657088+0000", 2024-09-16T03:40:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7995999999999998e-05 2024-09-16T03:40:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.854710+0000", 2024-09-16T03:40:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19762168499999999 2024-09-16T03:40:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:30.854863+0000", 2024-09-16T03:40:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000153568 2024-09-16T03:40:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:31.376266+0000", 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.52140287600000002 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4861 4.d 4:b3fe05b2:::smithi15736696-15:head [write 2805198~552112] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.054939+0000", 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.094308252000001, 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.59583329399999996, 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4861 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.054939+0000", 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.054939+0000", 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.054940+0000", 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3099999999999997e-07 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055449+0000", 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00050923899999999998 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055457+0000", 2024-09-16T03:40:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7030000000000006e-06 2024-09-16T03:40:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055461+0000", 2024-09-16T03:40:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9779999999999999e-06 2024-09-16T03:40:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.243176+0000", 2024-09-16T03:40:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18771517600000001 2024-09-16T03:40:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.243181+0000", 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.51e-06 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.348586+0000", 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10540505 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.348658+0000", 2024-09-16T03:40:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2310000000000004e-05 2024-09-16T03:40:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3", 2024-09-16T03:40:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.348770+0000", 2024-09-16T03:40:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000111573 2024-09-16T03:40:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.350138+0000", 2024-09-16T03:40:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001368282 2024-09-16T03:40:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.549595+0000", 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.19945716899999999 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.650700+0000", 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.101104609 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.650738+0000", 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8643000000000003e-05 2024-09-16T03:40:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.650773+0000", 2024-09-16T03:40:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4320999999999999e-05 2024-09-16T03:40:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4862 4.d 4:b3fe05b2:::smithi15736696-15:head [setxattr _header (56) in=7b,truncate 3428694] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-16T03:40:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.055468+0000", 2024-09-16T03:40:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.093779906999998, 2024-09-16T03:40:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.795963953, 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4862 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055468+0000", 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055468+0000", 2024-09-16T03:40:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055468+0000", 2024-09-16T03:40:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6800000000000001e-07 2024-09-16T03:40:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055471+0000", 2024-09-16T03:40:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5239999999999999e-06 2024-09-16T03:40:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055474+0000", 2024-09-16T03:40:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4800000000000001e-06 2024-09-16T03:40:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055477+0000", 2024-09-16T03:40:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.756e-06 2024-09-16T03:40:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.243208+0000", 2024-09-16T03:40:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18773089400000001 2024-09-16T03:40:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.243210+0000", 2024-09-16T03:40:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2390000000000001e-06 2024-09-16T03:40:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.349668+0000", 2024-09-16T03:40:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.10645742499999999 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.349742+0000", 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4036999999999996e-05 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1,3", 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.349890+0000", 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148238 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-16T03:40:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.350269+0000", 2024-09-16T03:40:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037942199999999997 2024-09-16T03:40:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.750976+0000", 2024-09-16T03:40:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.400706442 2024-09-16T03:40:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.851331+0000", 2024-09-16T03:40:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.100355047 2024-09-16T03:40:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-16T03:40:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.851381+0000", 2024-09-16T03:40:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0121999999999999e-05 2024-09-16T03:40:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.851432+0000", 2024-09-16T03:40:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0859000000000002e-05 2024-09-16T03:40:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4628.0:4863 4.d 4:b3fe05b2:::smithi15736696-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e165)", 2024-09-16T03:40:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-16T03:39:49.055481+0000", 2024-09-16T03:40:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.093766882000001, 2024-09-16T03:40:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.79613757299999999, 2024-09-16T03:40:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-16T03:40:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-16T03:40:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-16T03:40:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-16T03:40:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4628", 2024-09-16T03:40:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 4863 2024-09-16T03:40:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-16T03:40:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-16T03:40:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055481+0000", 2024-09-16T03:40:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-16T03:40:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055481+0000", 2024-09-16T03:40:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-16T03:40:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055481+0000", 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4400000000000001e-07 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055483+0000", 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.409e-06 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055486+0000", 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.162e-06 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.055489+0000", 2024-09-16T03:40:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9859999999999999e-06 2024-09-16T03:40:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.243248+0000", 2024-09-16T03:40:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.18775898999999999 2024-09-16T03:40:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.243252+0000", 2024-09-16T03:40:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5419999999999999e-06 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.350067+0000", 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.106814669 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.350122+0000", 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4787999999999998e-05 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-16T03:40:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.851497+0000", 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.501375499 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.851558+0000", 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1223000000000007e-05 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-16T03:39:49.851618+0000", 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9960999999999997e-05 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-16T03:40:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-16T03:40:05.271 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:40:05.272 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T03:40:05.273 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:40:05.270+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:40:05.372 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:40:05.371+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:40:05.388 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:40:05.388 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:40:05.389 INFO:teuthology.orchestra.run.smithi112.stdout: "num_ops": 0 2024-09-16T03:40:05.389 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:40:05.397 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T03:40:05.417 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T03:40:05.434 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:40:05.432+0000 7fa3886e6640 -1 osd.10 180 handle_osd_map injecting map crc failure 2024-09-16T03:40:05.473 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:40:05.472+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:40:05.503 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:40:05.503 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [], 2024-09-16T03:40:05.503 INFO:teuthology.orchestra.run.smithi112.stdout: "complaint_time": 30, 2024-09-16T03:40:05.503 INFO:teuthology.orchestra.run.smithi112.stdout: "num_blocked_ops": 0 2024-09-16T03:40:05.503 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:40:05.512 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T03:40:05.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T03:40:05.574 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:40:05.573+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:40:05.657 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-16T03:40:05.657 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 20, 2024-09-16T03:40:05.657 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 600, 2024-09-16T03:40:05.657 INFO:teuthology.orchestra.run.smithi112.stdout: "ops": [ 2024-09-16T03:40:05.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.658 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.12.21:5 3.0 e45/44 3:18b2c9f0:::smithi15736696-33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 45'39, mlcod=45'37)", 2024-09-16T03:40:05.658 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:40:05.658 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 164.740357687, 2024-09-16T03:40:05.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20119541799999999, 2024-09-16T03:40:05.658 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.658 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.658 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.658 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.658 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.658 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:40:05.658 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914595+0000", 2024-09-16T03:40:05.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914597+0000", 2024-09-16T03:40:05.659 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.494e-06 2024-09-16T03:40:05.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.659 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.659 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914603+0000", 2024-09-16T03:40:05.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.7349999999999999e-06 2024-09-16T03:40:05.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914614+0000", 2024-09-16T03:40:05.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0995999999999999e-05 2024-09-16T03:40:05.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.660 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.660 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:20.914623+0000", 2024-09-16T03:40:05.660 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.3899999999999993e-06 2024-09-16T03:40:05.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.114941+0000", 2024-09-16T03:40:05.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20031812399999999 2024-09-16T03:40:05.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115001+0000", 2024-09-16T03:40:05.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0161999999999998e-05 2024-09-16T03:40:05.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.661 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.661 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.661 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115750+0000", 2024-09-16T03:40:05.661 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00074851699999999998 2024-09-16T03:40:05.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.662 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.662 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:21.115791+0000", 2024-09-16T03:40:05.662 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1e-05 2024-09-16T03:40:05.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.662 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.662 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.662 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.5.21:36 3.0 e52/44 3:255cbbb1:::smithi15736696-23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 52'62, mlcod=51'60)", 2024-09-16T03:40:05.662 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:40:05.662 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 153.770327028, 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20104840199999999, 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884626+0000", 2024-09-16T03:40:05.663 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884627+0000", 2024-09-16T03:40:05.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.322e-06 2024-09-16T03:40:05.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.664 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884634+0000", 2024-09-16T03:40:05.664 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.438e-06 2024-09-16T03:40:05.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.664 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.884643+0000", 2024-09-16T03:40:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.7680000000000003e-06 2024-09-16T03:40:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:31.984740+0000", 2024-09-16T03:40:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100096066 2024-09-16T03:40:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084931+0000", 2024-09-16T03:40:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100191408 2024-09-16T03:40:05.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.084952+0000", 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.128e-05 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085645+0000", 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00069309500000000002 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:32.085674+0000", 2024-09-16T03:40:05.666 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9025000000000001e-05 2024-09-16T03:40:05.667 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.667 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.667 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.667 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1681 4.d e64/32 4:b3fe05b2:::smithi15736696-15:head v 64'6, mlcod=40'4)", 2024-09-16T03:40:05.667 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:40:05.667 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 135.505028431, 2024-09-16T03:40:05.667 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20169899299999999, 2024-09-16T03:40:05.667 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.667 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.667 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.667 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:40:05.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149925+0000", 2024-09-16T03:40:05.668 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.668 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.668 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.149927+0000", 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9299999999999999e-06 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151090+0000", 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011624459999999999 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.151105+0000", 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5238999999999999e-05 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.669 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251193+0000", 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100088025 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251556+0000", 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00036245599999999999 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.251578+0000", 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2651e-05 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.670 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351582+0000", 2024-09-16T03:40:05.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100003811 2024-09-16T03:40:05.671 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.671 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.671 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:50.351624+0000", 2024-09-16T03:40:05.671 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.1434999999999999e-05 2024-09-16T03:40:05.671 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.671 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.671 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1742 3.0 e70/69 3:0d133666:::5f0e85150f5f3f5d44ff49ee0f29b34591100efd5324826be9a5fed21098c1e6:head v 70'104, mlcod=0'0)", 2024-09-16T03:40:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:40:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 129.879153452, 2024-09-16T03:40:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20271830699999999, 2024-09-16T03:40:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.672 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.775800+0000", 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0100000000000001e-07 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777455+0000", 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.001655524 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777468+0000", 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.2486999999999999e-05 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.777474+0000", 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.1770000000000003e-06 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876577+0000", 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.099103360000000001 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.876594+0000", 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6449000000000001e-05 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.675 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978469+0000", 2024-09-16T03:40:05.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.101875521 2024-09-16T03:40:05.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.676 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.676 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978518+0000", 2024-09-16T03:40:05.676 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.8488000000000001e-05 2024-09-16T03:40:05.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.676 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.676 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:91 3.0 e70/69 3:1b7d6d7c:::0404d11535492de014f16ad72014b225863a9752462996608d37a3144bea0af3:head v 70'105, mlcod=0'0)", 2024-09-16T03:40:05.676 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 129.79294158299999, 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.116532025, 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.677 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862011+0000", 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862015+0000", 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3569999999999998e-06 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862029+0000", 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4082e-05 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.678 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862055+0000", 2024-09-16T03:40:05.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6570000000000001e-05 2024-09-16T03:40:05.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862070+0000", 2024-09-16T03:40:05.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4473e-05 2024-09-16T03:40:05.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.679 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.679 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877631+0000", 2024-09-16T03:40:05.679 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.015560744 2024-09-16T03:40:05.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.877674+0000", 2024-09-16T03:40:05.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.3615999999999997e-05 2024-09-16T03:40:05.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978525+0000", 2024-09-16T03:40:05.680 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100850696 2024-09-16T03:40:05.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.680 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.680 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978543+0000", 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8487000000000001e-05 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.10.22:94 3.0 e70/69 3:25a84717:::aa4cf33b9105a7c6a5599c9330129918257cc62744bde0e55066f9651c3d22e9:head v 70'106, mlcod=0'0)", 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 129.79240536399999, 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11627603, 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.681 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:40:05.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.682 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.682 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862548+0000", 2024-09-16T03:40:05.682 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862549+0000", 2024-09-16T03:40:05.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1650000000000001e-06 2024-09-16T03:40:05.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862556+0000", 2024-09-16T03:40:05.683 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.8709999999999999e-06 2024-09-16T03:40:05.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.683 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.683 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862566+0000", 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0271e-05 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.862573+0000", 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.861e-06 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978123+0000", 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.11554993199999999 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.684 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978163+0000", 2024-09-16T03:40:05.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.0201999999999999e-05 2024-09-16T03:40:05.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978796+0000", 2024-09-16T03:40:05.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00063316700000000002 2024-09-16T03:40:05.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.685 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.685 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:37:55.978824+0000", 2024-09-16T03:40:05.685 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.7560999999999999e-05 2024-09-16T03:40:05.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1979 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'32, mlcod=0'0)", 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 123.817214679, 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20429768500000001, 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.686 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837738+0000", 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.837740+0000", 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.779e-06 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838356+0000", 2024-09-16T03:40:05.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00061634500000000002 2024-09-16T03:40:05.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838371+0000", 2024-09-16T03:40:05.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.5007e-05 2024-09-16T03:40:05.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.688 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.688 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.838380+0000", 2024-09-16T03:40:05.688 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.1009999999999992e-06 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038674+0000", 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20029450500000001 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.038710+0000", 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.6100999999999999e-05 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.689 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042001+0000", 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0032909430000000002 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042036+0000", 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.4904000000000003e-05 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1980 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'33, mlcod=0'0)", 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:40:05.690 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 123.815338083, 2024-09-16T03:40:05.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20296795100000001, 2024-09-16T03:40:05.691 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.691 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.691 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.691 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:40:05.691 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.691 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.691 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.691 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839615+0000", 2024-09-16T03:40:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.839616+0000", 2024-09-16T03:40:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.3099999999999996e-07 2024-09-16T03:40:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840706+0000", 2024-09-16T03:40:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0010900980000000001 2024-09-16T03:40:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840720+0000", 2024-09-16T03:40:05.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.378e-05 2024-09-16T03:40:05.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.840727+0000", 2024-09-16T03:40:05.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.525e-06 2024-09-16T03:40:05.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.693 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.693 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039940+0000", 2024-09-16T03:40:05.693 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19921313199999999 2024-09-16T03:40:05.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.039967+0000", 2024-09-16T03:40:05.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6393999999999998e-05 2024-09-16T03:40:05.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.694 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042552+0000", 2024-09-16T03:40:05.694 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0025850999999999999 2024-09-16T03:40:05.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.694 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042583+0000", 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.0991000000000002e-05 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:1981 4.18 e75/68 4:1f63731d:::smithi15736696-30:head v 75'34, mlcod=0'0)", 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 123.81331572800001, 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201270175, 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.695 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.696 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:40:05.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.696 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841637+0000", 2024-09-16T03:40:05.696 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.696 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841638+0000", 2024-09-16T03:40:05.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1969999999999999e-06 2024-09-16T03:40:05.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841644+0000", 2024-09-16T03:40:05.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8050000000000003e-06 2024-09-16T03:40:05.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.697 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.697 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841653+0000", 2024-09-16T03:40:05.697 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6580000000000008e-06 2024-09-16T03:40:05.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:01.841660+0000", 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.9290000000000001e-06 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041673+0000", 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20001311499999999 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.041696+0000", 2024-09-16T03:40:05.698 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3365000000000001e-05 2024-09-16T03:40:05.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042878+0000", 2024-09-16T03:40:05.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0011814499999999999 2024-09-16T03:40:05.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.699 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.699 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:02.042907+0000", 2024-09-16T03:40:05.699 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.9655999999999999e-05 2024-09-16T03:40:05.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.699 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.700 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2536 3.0 e86/69 3:10a17b1b:::13dc2fab1f287170fabf96291b912bf80e0b081f291f2c6d41d494566f2d1f8c:head v 86'176, mlcod=86'174)", 2024-09-16T03:40:05.700 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:40:05.700 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 108.81544135, 2024-09-16T03:40:05.700 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20664157499999999, 2024-09-16T03:40:05.700 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.700 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.700 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.700 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.700 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.700 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:40:05.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839512+0000", 2024-09-16T03:40:05.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.701 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.701 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.839515+0000", 2024-09-16T03:40:05.701 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 3.3330000000000001e-06 2024-09-16T03:40:05.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843392+0000", 2024-09-16T03:40:05.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0038769329999999999 2024-09-16T03:40:05.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.843412+0000", 2024-09-16T03:40:05.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.0608e-05 2024-09-16T03:40:05.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.702 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.702 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:16.943546+0000", 2024-09-16T03:40:05.702 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100133176 2024-09-16T03:40:05.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043824+0000", 2024-09-16T03:40:05.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100278785 2024-09-16T03:40:05.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.703 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.703 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.043869+0000", 2024-09-16T03:40:05.703 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.4845999999999997e-05 2024-09-16T03:40:05.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046096+0000", 2024-09-16T03:40:05.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0022262559999999998 2024-09-16T03:40:05.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.704 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.704 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:17.046153+0000", 2024-09-16T03:40:05.704 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7637999999999999e-05 2024-09-16T03:40:05.704 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.704 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.704 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.705 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:2654 4.8 e90/68 4:148e627d:::smithi15736696-23:head v 90'58, mlcod=80'56)", 2024-09-16T03:40:05.705 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:40:05.705 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 104.28005399200001, 2024-09-16T03:40:05.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.106604212, 2024-09-16T03:40:05.705 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.705 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.705 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.705 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.705 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.705 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:40:05.705 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374899+0000", 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.374901+0000", 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.1749999999999999e-06 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375810+0000", 2024-09-16T03:40:05.706 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00090911400000000004 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.375827+0000", 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.7010000000000001e-05 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476010+0000", 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100182879 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.707 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476184+0000", 2024-09-16T03:40:05.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00017408899999999999 2024-09-16T03:40:05.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.476206+0000", 2024-09-16T03:40:05.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2013000000000001e-05 2024-09-16T03:40:05.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.708 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.708 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481458+0000", 2024-09-16T03:40:05.708 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0052519120000000001 2024-09-16T03:40:05.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.709 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.709 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:21.481503+0000", 2024-09-16T03:40:05.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5019999999999999e-05 2024-09-16T03:40:05.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.709 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.709 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "pg_update_log_missing(4.c epoch 99/68 rep_tid 69 entries 99'56 (0'0) error 4:34492f38:::smithi15736696-33:head by client.4628.0:3000 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 90'53 roll_forward_to 90'54)", 2024-09-16T03:40:05.709 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:40:05.709 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 93.735095841000003, 2024-09-16T03:40:05.709 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.201155684, 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "reached pg", 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919857+0000", 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919858+0000", 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.8999999999999996e-07 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919871+0000", 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3433e-05 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919929+0000", 2024-09-16T03:40:05.711 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7543000000000001e-05 2024-09-16T03:40:05.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:31.919942+0000", 2024-09-16T03:40:05.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3122000000000001e-05 2024-09-16T03:40:05.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.712 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.120302+0000", 2024-09-16T03:40:05.712 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20036052800000001 2024-09-16T03:40:05.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.712 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:38:32.121013+0000", 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00071046800000000002 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:4644 3.0 e150/149 3:1f968072:::a9957f29a7f946349b562c449f43fa1bc2e3c48b00925943f7b23963f3ffcd03:head v 150'271, mlcod=0'0)", 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:29.950532+0000", 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 35.704420587000001, 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30108226199999999, 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950532+0000", 2024-09-16T03:40:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950532+0000", 2024-09-16T03:40:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950533+0000", 2024-09-16T03:40:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2099999999999996e-07 2024-09-16T03:40:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950538+0000", 2024-09-16T03:40:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5170000000000001e-06 2024-09-16T03:40:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950544+0000", 2024-09-16T03:40:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0580000000000003e-06 2024-09-16T03:40:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.950550+0000", 2024-09-16T03:40:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2330000000000003e-06 2024-09-16T03:40:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.050698+0000", 2024-09-16T03:40:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100147758 2024-09-16T03:40:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.050724+0000", 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251572+0000", 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20084784999999999 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251615+0000", 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2317999999999997e-05 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(osd.15.23:179 3.0 e150/149 3:004ec411:::10547772e2dbc1bd82bb7d7e420056cffb9d849bbc3fd015ab311821f554ea0c:head v 150'272, mlcod=0'0)", 2024-09-16T03:40:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:29.953579+0000", 2024-09-16T03:40:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 35.701374123000001, 2024-09-16T03:40:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.29807083000000001, 2024-09-16T03:40:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953579+0000", 2024-09-16T03:40:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953579+0000", 2024-09-16T03:40:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953579+0000", 2024-09-16T03:40:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.5600000000000005e-07 2024-09-16T03:40:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953584+0000", 2024-09-16T03:40:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.126e-06 2024-09-16T03:40:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953590+0000", 2024-09-16T03:40:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.0680000000000002e-06 2024-09-16T03:40:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:29.953594+0000", 2024-09-16T03:40:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5480000000000003e-06 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.151091+0000", 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.197496745 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.151114+0000", 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.3045e-05 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251621+0000", 2024-09-16T03:40:05.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10050719800000001 2024-09-16T03:40:05.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.722 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.722 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251650+0000", 2024-09-16T03:40:05.722 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.8444000000000001e-05 2024-09-16T03:40:05.722 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.722 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.722 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.722 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "pg_scan(get_digest 4.8 MIN-MIN e 150/149)", 2024-09-16T03:40:05.722 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:30.051498+0000", 2024-09-16T03:40:05.723 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 35.603455175999997, 2024-09-16T03:40:05.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20012074299999999, 2024-09-16T03:40:05.723 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.723 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.723 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:40:05.723 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.723 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051498+0000", 2024-09-16T03:40:05.723 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.723 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051498+0000", 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051499+0000", 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.6899999999999996e-07 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051509+0000", 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.0907000000000001e-05 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051517+0000", 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.3690000000000001e-06 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.051523+0000", 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7950000000000004e-06 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251377+0000", 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.19985429800000001 2024-09-16T03:40:05.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251392+0000", 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.4603000000000001e-05 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.251619+0000", 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.00022700199999999999 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "pg_scan(get_digest 3.0 MIN-MIN e 150/149)", 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:30.456069+0000", 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 35.198884409000001, 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.200403153, 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456069+0000", 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456069+0000", 2024-09-16T03:40:05.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456070+0000", 2024-09-16T03:40:05.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.3829999999999999e-06 2024-09-16T03:40:05.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.728 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.728 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456074+0000", 2024-09-16T03:40:05.728 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.2289999999999997e-06 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.456083+0000", 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.9789999999999999e-06 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.556184+0000", 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100101147 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.729 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.656320+0000", 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100135347 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.656328+0000", 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.4610000000000008e-06 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:30.656472+0000", 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.000143607 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:4811 3.0 e155/149 3:3c95911f:::e0ef79a5931af77a89e00e45b270ee95266718455ec934bcc09140d5d71025ab:head v 155'275, mlcod=150'273)", 2024-09-16T03:40:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:35.639297+0000", 2024-09-16T03:40:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 30.015656315000001, 2024-09-16T03:40:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.105075237, 2024-09-16T03:40:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.731 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.639297+0000", 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.639297+0000", 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.639298+0000", 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.8059999999999999e-06 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.732 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.642391+0000", 2024-09-16T03:40:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0030922050000000002 2024-09-16T03:40:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.642407+0000", 2024-09-16T03:40:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6393e-05 2024-09-16T03:40:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.642416+0000", 2024-09-16T03:40:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.8079999999999994e-06 2024-09-16T03:40:05.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.742572+0000", 2024-09-16T03:40:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10015578999999999 2024-09-16T03:40:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.742596+0000", 2024-09-16T03:40:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.4406e-05 2024-09-16T03:40:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.734 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.744316+0000", 2024-09-16T03:40:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0017202090000000001 2024-09-16T03:40:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:35.744372+0000", 2024-09-16T03:40:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.5619999999999999e-05 2024-09-16T03:40:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:4862 4.d e165/153 4:b3fe05b2:::smithi15736696-15:head v 165'27, mlcod=0'0)", 2024-09-16T03:40:05.735 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:49.449186+0000", 2024-09-16T03:40:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 16.205766568000001, 2024-09-16T03:40:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.30053745799999998, 2024-09-16T03:40:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.449186+0000", 2024-09-16T03:40:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.736 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.449186+0000", 2024-09-16T03:40:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.449187+0000", 2024-09-16T03:40:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 9.2999999999999999e-07 2024-09-16T03:40:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.449192+0000", 2024-09-16T03:40:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.0889999999999999e-06 2024-09-16T03:40:05.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.449199+0000", 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 6.2870000000000001e-06 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.549302+0000", 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100103716 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.569629+0000", 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.020326956 2024-09-16T03:40:05.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.569652+0000", 2024-09-16T03:40:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 2.2345e-05 2024-09-16T03:40:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.749678+0000", 2024-09-16T03:40:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.18002673 2024-09-16T03:40:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.739 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:49.749724+0000", 2024-09-16T03:40:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.5405000000000003e-05 2024-09-16T03:40:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "osd_repop(client.4628.0:4949 4.1f e169/168 4:fcb75e43:::smithi15736696-26:head v 169'129, mlcod=0'0)", 2024-09-16T03:40:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:39:54.102759+0000", 2024-09-16T03:40:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 11.552194452, 2024-09-16T03:40:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.10516655599999999, 2024-09-16T03:40:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.740 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.102759+0000", 2024-09-16T03:40:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.102759+0000", 2024-09-16T03:40:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.741 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.102760+0000", 2024-09-16T03:40:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1209999999999999e-06 2024-09-16T03:40:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.103993+0000", 2024-09-16T03:40:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0012336669999999999 2024-09-16T03:40:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.104004+0000", 2024-09-16T03:40:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.1056000000000001e-05 2024-09-16T03:40:05.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.104013+0000", 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 8.6100000000000006e-06 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.201992+0000", 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.097979207999999998 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.202045+0000", 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.2611000000000002e-05 2024-09-16T03:40:05.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "commit_sent", 2024-09-16T03:40:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.207868+0000", 2024-09-16T03:40:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.0058229090000000002 2024-09-16T03:40:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:39:54.207925+0000", 2024-09-16T03:40:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.7374000000000003e-05 2024-09-16T03:40:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "description": "MOSDScrubReserve(4.1f,QREQUEST,e:178,from:10,reservation_nonce:2)", 2024-09-16T03:40:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "initiated_at": "2024-09-16T03:40:03.720215+0000", 2024-09-16T03:40:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "age": 1.934737742, 2024-09-16T03:40:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.20043325400000001, 2024-09-16T03:40:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "continuous": false, 2024-09-16T03:40:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "type_data": { 2024-09-16T03:40:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "flag_point": "started", 2024-09-16T03:40:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "events": [ 2024-09-16T03:40:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "initiated", 2024-09-16T03:40:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:40:03.720215+0000", 2024-09-16T03:40:05.745 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "header_read", 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:40:03.720215+0000", 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "throttled", 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:40:03.720217+0000", 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.9010000000000001e-06 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "all_read", 2024-09-16T03:40:05.746 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:40:03.720225+0000", 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 7.4080000000000001e-06 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "dispatched", 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:40:03.720241+0000", 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 1.6220000000000001e-05 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "queued_for_pg", 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:40:03.820337+0000", 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100096175 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.747 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "reached_pg", 2024-09-16T03:40:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:40:03.920545+0000", 2024-09-16T03:40:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 0.100208174 2024-09-16T03:40:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "started", 2024-09-16T03:40:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:40:03.920595+0000", 2024-09-16T03:40:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 4.9444000000000003e-05 2024-09-16T03:40:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2024-09-16T03:40:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2024-09-16T03:40:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: "event": "done", 2024-09-16T03:40:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: "time": "2024-09-16T03:40:03.920648+0000", 2024-09-16T03:40:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: "duration": 5.3931999999999998e-05 2024-09-16T03:40:05.748 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.749 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.749 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.749 INFO:teuthology.orchestra.run.smithi112.stdout: } 2024-09-16T03:40:05.749 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-09-16T03:40:05.749 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-16T03:40:05.750 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:40:05.750 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:05.751 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:40:05.751 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:05.752 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T03:40:05.752 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:40:05.748+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:40:05.752 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:40:05.752 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:05.753 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:40:05.753 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:05.753 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:40:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:05.754 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:40:05.754 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:05.766 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:40:05.767 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:05.784 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:40:05.784 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:05.799 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:40:05.800 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:05.816 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:40:05.817 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:05.833 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:40:05.833 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:05.850 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:40:05.850 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:05.851 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:40:05.848+0000 7fd11147a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33438) UID: 0 2024-09-16T03:40:05.866 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:40:05.866 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:05.882 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:40:05.882 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:05.895 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:05.895 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:05.899 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:40:05.900 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:05.904 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T03:40:05.917 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:40:05.917 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:05.951 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:40:05.949+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:40:06.006 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:06.007 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:06.015 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T03:40:06.052 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:40:06.051+0000 7f494cfd3640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33441) UID: 0 2024-09-16T03:40:06.118 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:06.118 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:06.127 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T03:40:06.170 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:40:06.169+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:40:06.230 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:06.230 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:06.239 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T03:40:06.253 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:40:06.251+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:40:06.355 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:40:06.353+0000 7f3a2ed29640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33441) UID: 0 2024-09-16T03:40:06.371 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:40:06.369+0000 7fd103e73640 -1 osd.2 181 handle_osd_map injecting map crc failure 2024-09-16T03:40:06.374 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is set 2024-09-16T03:40:06.382 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:06.382 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:06.391 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T03:40:06.407 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 6, 7, 8, 10, 11, 12, 14, 15, 9, 1] out_osds: [5, 13] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T03:40:06.407 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T03:40:06.407 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-16T03:40:06.408 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-16T03:40:06.408 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:06.431 INFO:tasks.rados.rados.0.smithi157.stderr:0 errors. 2024-09-16T03:40:06.431 INFO:tasks.rados.rados.0.smithi157.stderr: 2024-09-16T03:40:06.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:40:06.474 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:40:06.473+0000 7eff87e56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33213) UID: 0 2024-09-16T03:40:06.494 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:06.494 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:06.504 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T03:40:06.555 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:40:06.554+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:40:06.574 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:40:06.572+0000 7eff7884b640 -1 osd.3 181 handle_osd_map injecting map crc failure 2024-09-16T03:40:06.607 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:06.608 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:06.616 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T03:40:06.658 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:40:06.656+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:40:06.746 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:06.746 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:40:06.757 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-16T03:40:06.371421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":269007156,"num_objects":204,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":114,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":2982,"num_read_kb":1623303,"num_write":2314,"num_write_kb":513577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":154862352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":24,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":207,"acting":207,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":241,"num_osds":14,"num_per_pool_osds":14,"num_per_pool_omap_osds":14,"kb":1321205760,"kb_used":957160,"kb_used_data":503804,"kb_used_omap":21,"kb_used_meta":452970,"kb_avail":1320248600,"statfs":{"total":1352914698240,"available":1351934566400,"internally_reserved":0,"allocated":515895296,"data_stored":523390394,"data_compressed":15582996,"data_compressed_allocated":17137664,"data_compressed_original":35647488,"omap_allocated":22329,"internal_metadata":463841479},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-18,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":5928185,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.970288"},"pg_stats":[{"pgid":"4.37","version":"153'56","reported_seq":533,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.596961+0000","last_change":"2024-09-16T03:39:57.794120+0000","last_active":"2024-09-16T03:40:01.596961+0000","last_peered":"2024-09-16T03:40:01.596961+0000","last_clean":"2024-09-16T03:40:01.596961+0000","last_became_active":"2024-09-16T03:39:57.578155+0000","last_became_peered":"2024-09-16T03:39:57.578155+0000","last_unstale":"2024-09-16T03:40:01.596961+0000","last_undegraded":"2024-09-16T03:40:01.596961+0000","last_fullsized":"2024-09-16T03:40:01.596961+0000","mapping_epoch":172,"log_start":"142'54","ondisk_log_start":"142'54","created":126,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":6,"last_scrub":"153'56","last_scrub_stamp":"2024-09-16T03:39:57.794044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.794044+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":49,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:26.249805+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2689186,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,14,7],"acting":[15,14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.36","version":"142'96","reported_seq":567,"reported_epoch":181,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:05.536030+0000","last_change":"2024-09-16T03:39:38.887893+0000","last_active":"2024-09-16T03:40:05.536030+0000","last_peered":"2024-09-16T03:40:05.536030+0000","last_clean":"2024-09-16T03:39:19.651965+0000","last_became_active":"2024-09-16T03:39:38.786957+0000","last_became_peered":"2024-09-16T03:39:38.786957+0000","last_unstale":"2024-09-16T03:40:05.536030+0000","last_undegraded":"2024-09-16T03:40:05.536030+0000","last_fullsized":"2024-09-16T03:40:05.536030+0000","mapping_epoch":158,"log_start":"131'93","ondisk_log_start":"131'93","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.716125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.716125+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":91,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7159980,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":90,"num_read_kb":79209,"num_write":12,"num_write_kb":3200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,11],"acting":[2,10,15],"avail_no_missing":["2","10","15"],"object_location_counts":[{"shards":"2,10,15","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"169'85","reported_seq":643,"reported_epoch":180,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:04.513231+0000","last_change":"2024-09-16T03:39:56.515854+0000","last_active":"2024-09-16T03:40:04.513231+0000","last_peered":"2024-09-16T03:40:04.513231+0000","last_clean":"2024-09-16T03:39:35.664815+0000","last_became_active":"2024-09-16T03:39:56.009732+0000","last_became_peered":"2024-09-16T03:39:56.009732+0000","last_unstale":"2024-09-16T03:40:04.513231+0000","last_undegraded":"2024-09-16T03:40:04.513231+0000","last_fullsized":"2024-09-16T03:40:04.513231+0000","mapping_epoch":170,"log_start":"165'81","ondisk_log_start":"165'81","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'67","last_scrub_stamp":"2024-09-16T03:39:15.079987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:15.079987+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":79,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6874195,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":30309,"num_write":19,"num_write_kb":5766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[3,9,10],"avail_no_missing":["3","9","10"],"object_location_counts":[{"shards":"3,9,10","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"149'25","reported_seq":418,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.721835+0000","last_change":"2024-09-16T03:39:58.682198+0000","last_active":"2024-09-16T03:40:03.721835+0000","last_peered":"2024-09-16T03:40:03.721835+0000","last_clean":"2024-09-16T03:40:03.721835+0000","last_became_active":"2024-09-16T03:39:58.581739+0000","last_became_peered":"2024-09-16T03:39:58.581739+0000","last_unstale":"2024-09-16T03:40:03.721835+0000","last_undegraded":"2024-09-16T03:40:03.721835+0000","last_fullsized":"2024-09-16T03:40:03.721835+0000","mapping_epoch":173,"log_start":"149'23","ondisk_log_start":"149'23","created":126,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":6,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:02.919259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.919259+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":20,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:11.357946+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3109361,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":17899,"num_write":5,"num_write_kb":1218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2853416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,9],"acting":[10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.33","version":"148'76","reported_seq":588,"reported_epoch":180,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:04.413245+0000","last_change":"2024-09-16T03:39:39.185026+0000","last_active":"2024-09-16T03:40:04.413245+0000","last_peered":"2024-09-16T03:40:04.413245+0000","last_clean":"2024-09-16T03:39:26.364451+0000","last_became_active":"2024-09-16T03:39:38.883512+0000","last_became_peered":"2024-09-16T03:39:38.883512+0000","last_unstale":"2024-09-16T03:40:04.413245+0000","last_undegraded":"2024-09-16T03:40:04.413245+0000","last_fullsized":"2024-09-16T03:40:04.413245+0000","mapping_epoch":158,"log_start":"137'73","ondisk_log_start":"137'73","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'72","last_scrub_stamp":"2024-09-16T03:39:02.143339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.143339+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":69,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353046,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":37925,"num_write":4,"num_write_kb":955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":994759,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,10],"acting":[3,4,9],"avail_no_missing":["3","4","9"],"object_location_counts":[{"shards":"3,4,9","objects":1}],"blocked_by":[],"up_primary":14,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.32","version":"106'17","reported_seq":329,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.401765+0000","last_change":"2024-09-16T03:39:44.270851+0000","last_active":"2024-09-16T03:40:03.401765+0000","last_peered":"2024-09-16T03:40:03.401765+0000","last_clean":"2024-09-16T03:40:03.401765+0000","last_became_active":"2024-09-16T03:39:40.893486+0000","last_became_peered":"2024-09-16T03:39:40.893486+0000","last_unstale":"2024-09-16T03:40:03.401765+0000","last_undegraded":"2024-09-16T03:40:03.401765+0000","last_fullsized":"2024-09-16T03:40:03.401765+0000","mapping_epoch":160,"log_start":"82'13","ondisk_log_start":"82'13","created":126,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":6,"last_scrub":"106'17","last_scrub_stamp":"2024-09-16T03:39:12.108317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:12.108317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:40:11.560664+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1033647,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":20741,"num_write":6,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1331059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,12],"acting":[0,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.31","version":"114'17","reported_seq":296,"reported_epoch":179,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:03.399538+0000","last_change":"2024-09-16T03:39:38.988193+0000","last_active":"2024-09-16T03:40:03.399538+0000","last_peered":"2024-09-16T03:40:03.399538+0000","last_clean":"2024-09-16T03:38:48.641776+0000","last_became_active":"2024-09-16T03:39:38.987267+0000","last_became_peered":"2024-09-16T03:39:38.987267+0000","last_unstale":"2024-09-16T03:40:03.399538+0000","last_undegraded":"2024-09-16T03:40:03.399538+0000","last_fullsized":"2024-09-16T03:40:03.399538+0000","mapping_epoch":158,"log_start":"107'14","ondisk_log_start":"107'14","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:35.719895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:35.719895+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1495840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":28046,"num_write":4,"num_write_kb":1066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,11],"acting":[12,2,4],"avail_no_missing":["12","2","4"],"object_location_counts":[{"shards":"2,4,12","objects":1}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.30","version":"118'64","reported_seq":420,"reported_epoch":180,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:04.504336+0000","last_change":"2024-09-16T03:39:56.515576+0000","last_active":"2024-09-16T03:40:04.504336+0000","last_peered":"2024-09-16T03:40:04.504336+0000","last_clean":"2024-09-16T03:39:52.798009+0000","last_became_active":"2024-09-16T03:39:56.404165+0000","last_became_peered":"2024-09-16T03:39:56.404165+0000","last_unstale":"2024-09-16T03:40:04.504336+0000","last_undegraded":"2024-09-16T03:40:04.504336+0000","last_fullsized":"2024-09-16T03:40:04.504336+0000","mapping_epoch":170,"log_start":"113'61","ondisk_log_start":"113'61","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":6,"last_scrub":"118'64","last_scrub_stamp":"2024-09-16T03:39:38.383253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.383253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,3],"avail_no_missing":["4","3","9","10"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"113'94","reported_seq":474,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405969+0000","last_change":"2024-09-16T03:39:42.292571+0000","last_active":"2024-09-16T03:40:03.405969+0000","last_peered":"2024-09-16T03:40:03.405969+0000","last_clean":"2024-09-16T03:40:03.405969+0000","last_became_active":"2024-09-16T03:39:40.802316+0000","last_became_peered":"2024-09-16T03:39:40.802316+0000","last_unstale":"2024-09-16T03:40:03.405969+0000","last_undegraded":"2024-09-16T03:40:03.405969+0000","last_fullsized":"2024-09-16T03:40:03.405969+0000","mapping_epoch":160,"log_start":"107'90","ondisk_log_start":"107'90","created":126,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":6,"last_scrub":"113'94","last_scrub_stamp":"2024-09-16T03:39:03.319012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.319012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":87,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:40:11.566141+0000","scrub_duration":319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1470629,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,3],"acting":[14,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.2e","version":"155'93","reported_seq":581,"reported_epoch":180,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:04.411225+0000","last_change":"2024-09-16T03:39:39.286505+0000","last_active":"2024-09-16T03:40:04.411225+0000","last_peered":"2024-09-16T03:40:04.411225+0000","last_clean":"2024-09-16T03:39:35.765256+0000","last_became_active":"2024-09-16T03:39:38.884986+0000","last_became_peered":"2024-09-16T03:39:38.884986+0000","last_unstale":"2024-09-16T03:40:04.411225+0000","last_undegraded":"2024-09-16T03:40:04.411225+0000","last_fullsized":"2024-09-16T03:40:04.411225+0000","mapping_epoch":158,"log_start":"152'90","ondisk_log_start":"152'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"123'75","last_scrub_stamp":"2024-09-16T03:39:04.235141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.235141+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":90,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151494,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":62325,"num_write":19,"num_write_kb":5191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,14],"acting":[4,8,15],"avail_no_missing":["4","8","15"],"object_location_counts":[{"shards":"4,8,15","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"172'112","reported_seq":595,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.722094+0000","last_change":"2024-09-16T03:39:03.917422+0000","last_active":"2024-09-16T03:40:03.722094+0000","last_peered":"2024-09-16T03:40:03.722094+0000","last_clean":"2024-09-16T03:40:03.722094+0000","last_became_active":"2024-09-16T03:39:01.801112+0000","last_became_peered":"2024-09-16T03:39:01.801112+0000","last_unstale":"2024-09-16T03:40:03.722094+0000","last_undegraded":"2024-09-16T03:40:03.722094+0000","last_fullsized":"2024-09-16T03:40:03.722094+0000","mapping_epoch":32,"log_start":"166'109","ondisk_log_start":"166'109","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'97","last_scrub_stamp":"2024-09-16T03:39:03.917355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.917355+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":106,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:06.444266+0000","scrub_duration":771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496021,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":28034,"num_write":29,"num_write_kb":7222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.e","version":"165'94","reported_seq":589,"reported_epoch":180,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:04.411548+0000","last_change":"2024-09-16T03:39:56.722563+0000","last_active":"2024-09-16T03:40:04.411548+0000","last_peered":"2024-09-16T03:40:04.411548+0000","last_clean":"2024-09-16T03:39:54.776007+0000","last_became_active":"2024-09-16T03:39:56.614366+0000","last_became_peered":"2024-09-16T03:39:56.614366+0000","last_unstale":"2024-09-16T03:40:04.411548+0000","last_undegraded":"2024-09-16T03:40:04.411548+0000","last_fullsized":"2024-09-16T03:40:04.411548+0000","mapping_epoch":171,"log_start":"154'91","ondisk_log_start":"154'91","created":32,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'93","last_scrub_stamp":"2024-09-16T03:39:35.280938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:35.280938+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":88,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327852,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":17624,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2614941,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14,3],"acting":[4,14,8],"avail_no_missing":["4","8","14","15"],"object_location_counts":[{"shards":"4,8,14","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"165'27","reported_seq":487,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.657080+0000","last_change":"2024-09-16T03:39:33.569454+0000","last_active":"2024-09-16T03:40:03.657080+0000","last_peered":"2024-09-16T03:40:03.657080+0000","last_clean":"2024-09-16T03:40:03.657080+0000","last_became_active":"2024-09-16T03:39:33.569207+0000","last_became_peered":"2024-09-16T03:39:33.569207+0000","last_unstale":"2024-09-16T03:40:03.657080+0000","last_undegraded":"2024-09-16T03:40:03.657080+0000","last_fullsized":"2024-09-16T03:40:03.657080+0000","mapping_epoch":153,"log_start":"152'23","ondisk_log_start":"152'23","created":32,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'19","last_scrub_stamp":"2024-09-16T03:39:01.149486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.149486+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":23,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:09.211597+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3428694,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":17088,"num_write":9,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1327143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,1,3],"acting":[12,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"167'69","reported_seq":547,"reported_epoch":179,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:03.399645+0000","last_change":"2024-09-16T03:40:01.256353+0000","last_active":"2024-09-16T03:40:03.399645+0000","last_peered":"2024-09-16T03:40:03.399645+0000","last_clean":"2024-09-16T03:39:27.002211+0000","last_became_active":"2024-09-16T03:39:31.053925+0000","last_became_peered":"2024-09-16T03:39:31.053925+0000","last_unstale":"2024-09-16T03:40:03.399645+0000","last_undegraded":"2024-09-16T03:40:03.399645+0000","last_fullsized":"2024-09-16T03:40:03.399645+0000","mapping_epoch":150,"log_start":"167'66","ondisk_log_start":"167'66","created":32,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'57","last_scrub_stamp":"2024-09-16T03:38:48.923184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.923184+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":64,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512729,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":30539,"num_write":14,"num_write_kb":4279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1213217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,1],"acting":[12,15,9],"avail_no_missing":["12","9","15"],"object_location_counts":[{"shards":"9,12,15","objects":1}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"153'135","reported_seq":698,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.596827+0000","last_change":"2024-09-16T03:40:01.105429+0000","last_active":"2024-09-16T03:40:01.596827+0000","last_peered":"2024-09-16T03:40:01.596827+0000","last_clean":"2024-09-16T03:40:01.596827+0000","last_became_active":"2024-09-16T03:39:09.748391+0000","last_became_peered":"2024-09-16T03:39:09.748391+0000","last_unstale":"2024-09-16T03:40:01.596827+0000","last_undegraded":"2024-09-16T03:40:01.596827+0000","last_fullsized":"2024-09-16T03:40:01.596827+0000","mapping_epoch":134,"log_start":"129'132","ondisk_log_start":"129'132","created":32,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'135","last_scrub_stamp":"2024-09-16T03:40:01.105343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:40:01.105343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":128,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:25.231007+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":17762,"num_write":32,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.a","version":"168'176","reported_seq":849,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.714220+0000","last_change":"2024-09-16T03:39:01.504751+0000","last_active":"2024-09-16T03:40:04.714220+0000","last_peered":"2024-09-16T03:40:04.714220+0000","last_clean":"2024-09-16T03:40:04.714220+0000","last_became_active":"2024-09-16T03:38:46.021805+0000","last_became_peered":"2024-09-16T03:38:46.021805+0000","last_unstale":"2024-09-16T03:40:04.714220+0000","last_undegraded":"2024-09-16T03:40:04.714220+0000","last_fullsized":"2024-09-16T03:40:04.714220+0000","mapping_epoch":32,"log_start":"168'173","ondisk_log_start":"168'173","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'147","last_scrub_stamp":"2024-09-16T03:39:01.504663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.504663+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":170,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.277419+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13126531,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":161,"num_read_kb":166710,"num_write":34,"num_write_kb":9701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":355,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405568+0000","last_change":"2024-09-16T03:40:00.748178+0000","last_active":"2024-09-16T03:40:03.405568+0000","last_peered":"2024-09-16T03:40:03.405568+0000","last_clean":"2024-09-16T03:40:03.405568+0000","last_became_active":"2024-09-16T03:39:30.229833+0000","last_became_peered":"2024-09-16T03:39:30.229833+0000","last_unstale":"2024-09-16T03:40:03.405568+0000","last_undegraded":"2024-09-16T03:40:03.405568+0000","last_fullsized":"2024-09-16T03:40:03.405568+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:00.748094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:40:00.748094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.254250+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,8],"acting":[14,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.8","version":"152'82","reported_seq":585,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.401782+0000","last_change":"2024-09-16T03:39:31.762814+0000","last_active":"2024-09-16T03:40:03.401782+0000","last_peered":"2024-09-16T03:40:03.401782+0000","last_clean":"2024-09-16T03:40:03.401782+0000","last_became_active":"2024-09-16T03:39:31.762544+0000","last_became_peered":"2024-09-16T03:39:31.762544+0000","last_unstale":"2024-09-16T03:40:03.401782+0000","last_undegraded":"2024-09-16T03:40:03.401782+0000","last_fullsized":"2024-09-16T03:40:03.401782+0000","mapping_epoch":151,"log_start":"140'78","ondisk_log_start":"140'78","created":32,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'71","last_scrub_stamp":"2024-09-16T03:39:02.002247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.002247+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":76,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.244585+0000","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243817,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":3438,"num_write":14,"num_write_kb":3035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"171'100","reported_seq":566,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.402456+0000","last_change":"2024-09-16T03:39:58.682984+0000","last_active":"2024-09-16T03:40:03.402456+0000","last_peered":"2024-09-16T03:40:03.402456+0000","last_clean":"2024-09-16T03:40:03.402456+0000","last_became_active":"2024-09-16T03:39:58.682644+0000","last_became_peered":"2024-09-16T03:39:58.682644+0000","last_unstale":"2024-09-16T03:40:03.402456+0000","last_undegraded":"2024-09-16T03:40:03.402456+0000","last_fullsized":"2024-09-16T03:40:03.402456+0000","mapping_epoch":173,"log_start":"171'97","ondisk_log_start":"171'97","created":32,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'82","last_scrub_stamp":"2024-09-16T03:39:00.900619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.900619+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":95,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.165962+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3714413,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":40079,"num_write":22,"num_write_kb":6258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2417775,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":509,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.495182+0000","last_change":"2024-09-16T03:38:36.757156+0000","last_active":"2024-09-16T03:40:01.495182+0000","last_peered":"2024-09-16T03:40:01.495182+0000","last_clean":"2024-09-16T03:40:01.495182+0000","last_became_active":"2024-09-16T03:38:36.756855+0000","last_became_peered":"2024-09-16T03:38:36.756855+0000","last_unstale":"2024-09-16T03:40:01.495182+0000","last_undegraded":"2024-09-16T03:40:01.495182+0000","last_fullsized":"2024-09-16T03:40:01.495182+0000","mapping_epoch":103,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:38:28.063690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:38:28.063690+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:30.945544+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":216,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.502092+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:40:03.502092+0000","last_peered":"2024-09-16T03:40:03.502092+0000","last_clean":"2024-09-16T03:40:03.502092+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:40:03.502092+0000","last_undegraded":"2024-09-16T03:40:03.502092+0000","last_fullsized":"2024-09-16T03:40:03.502092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.436956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":758,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.495470+0000","last_change":"2024-09-16T03:39:23.138722+0000","last_active":"2024-09-16T03:40:01.495470+0000","last_peered":"2024-09-16T03:40:01.495470+0000","last_clean":"2024-09-16T03:40:01.495470+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:40:01.495470+0000","last_undegraded":"2024-09-16T03:40:01.495470+0000","last_fullsized":"2024-09-16T03:40:01.495470+0000","mapping_epoch":74,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.823622+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.6","version":"108'44","reported_seq":481,"reported_epoch":179,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:03.722271+0000","last_change":"2024-09-16T03:39:55.909742+0000","last_active":"2024-09-16T03:40:03.722271+0000","last_peered":"2024-09-16T03:40:03.722271+0000","last_clean":"2024-09-16T03:39:52.998142+0000","last_became_active":"2024-09-16T03:39:55.908933+0000","last_became_peered":"2024-09-16T03:39:55.908933+0000","last_unstale":"2024-09-16T03:40:03.722271+0000","last_undegraded":"2024-09-16T03:40:03.722271+0000","last_fullsized":"2024-09-16T03:40:03.722271+0000","mapping_epoch":170,"log_start":"105'41","ondisk_log_start":"105'41","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:48.279224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.279224+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,12],"acting":[10,6,4],"avail_no_missing":["10","4","6","12"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":358,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.714303+0000","last_change":"2024-09-16T03:39:09.749531+0000","last_active":"2024-09-16T03:40:04.714303+0000","last_peered":"2024-09-16T03:40:04.714303+0000","last_clean":"2024-09-16T03:40:04.714303+0000","last_became_active":"2024-09-16T03:39:09.649028+0000","last_became_peered":"2024-09-16T03:39:09.649028+0000","last_unstale":"2024-09-16T03:40:04.714303+0000","last_undegraded":"2024-09-16T03:40:04.714303+0000","last_fullsized":"2024-09-16T03:40:04.714303+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:24.358339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:24.358339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:41.336202+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":609,"reported_epoch":177,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:01.731470+0000","last_change":"2024-09-16T03:39:55.213119+0000","last_active":"2024-09-16T03:40:01.731470+0000","last_peered":"2024-09-16T03:40:01.731470+0000","last_clean":"2024-09-16T03:39:49.975837+0000","last_became_active":"2024-09-16T03:39:54.908050+0000","last_became_peered":"2024-09-16T03:39:54.908050+0000","last_unstale":"2024-09-16T03:40:01.731470+0000","last_undegraded":"2024-09-16T03:40:01.731470+0000","last_fullsized":"2024-09-16T03:40:01.731470+0000","mapping_epoch":169,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'251","last_scrub_stamp":"2024-09-16T03:39:36.474408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:36.474408+0000","objects_scrubbed":37,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":78,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":234,"num_read_kb":2386,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,1],"acting":[6,3,4],"avail_no_missing":["6","3","4"],"object_location_counts":[{"shards":"3,4,6","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"168'90","reported_seq":555,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.822168+0000","last_change":"2024-09-16T03:39:01.297447+0000","last_active":"2024-09-16T03:40:03.822168+0000","last_peered":"2024-09-16T03:40:03.822168+0000","last_clean":"2024-09-16T03:40:03.822168+0000","last_became_active":"2024-09-16T03:38:45.723754+0000","last_became_peered":"2024-09-16T03:38:45.723754+0000","last_unstale":"2024-09-16T03:40:03.822168+0000","last_undegraded":"2024-09-16T03:40:03.822168+0000","last_fullsized":"2024-09-16T03:40:03.822168+0000","mapping_epoch":32,"log_start":"168'87","ondisk_log_start":"168'87","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'72","last_scrub_stamp":"2024-09-16T03:39:01.297324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.297324+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":85,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:26.128089+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7092686,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":98758,"num_write":19,"num_write_kb":5816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.495353+0000","last_change":"2024-09-16T03:39:56.004042+0000","last_active":"2024-09-16T03:40:01.495353+0000","last_peered":"2024-09-16T03:40:01.495353+0000","last_clean":"2024-09-16T03:40:01.495353+0000","last_became_active":"2024-09-16T03:39:56.003773+0000","last_became_peered":"2024-09-16T03:39:56.003773+0000","last_unstale":"2024-09-16T03:40:01.495353+0000","last_undegraded":"2024-09-16T03:40:01.495353+0000","last_fullsized":"2024-09-16T03:40:01.495353+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:27.167621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,7],"acting":[11,14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":922,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.776035+0000","last_change":"2024-09-16T03:40:01.694672+0000","last_active":"2024-09-16T03:40:04.776035+0000","last_peered":"2024-09-16T03:40:04.776035+0000","last_clean":"2024-09-16T03:40:04.776035+0000","last_became_active":"2024-09-16T03:40:01.594201+0000","last_became_peered":"2024-09-16T03:40:01.594201+0000","last_unstale":"2024-09-16T03:40:04.776035+0000","last_undegraded":"2024-09-16T03:40:04.776035+0000","last_fullsized":"2024-09-16T03:40:04.776035+0000","mapping_epoch":176,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:43.795369+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":268,"num_read_kb":13655,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":57210536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,6],"acting":[9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"137'33","reported_seq":421,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.595599+0000","last_change":"2024-09-16T03:38:47.042020+0000","last_active":"2024-09-16T03:40:01.595599+0000","last_peered":"2024-09-16T03:40:01.595599+0000","last_clean":"2024-09-16T03:40:01.595599+0000","last_became_active":"2024-09-16T03:38:45.820169+0000","last_became_peered":"2024-09-16T03:38:45.820169+0000","last_unstale":"2024-09-16T03:40:01.595599+0000","last_undegraded":"2024-09-16T03:40:01.595599+0000","last_fullsized":"2024-09-16T03:40:01.595599+0000","mapping_epoch":32,"log_start":"87'30","ondisk_log_start":"87'30","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.041851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.041851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":30,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:55.806387+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":58,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.821915+0000","last_change":"2024-09-16T03:39:57.677954+0000","last_active":"2024-09-16T03:40:03.821915+0000","last_peered":"2024-09-16T03:40:03.821915+0000","last_clean":"2024-09-16T03:40:03.821915+0000","last_became_active":"2024-09-16T03:39:57.677624+0000","last_became_peered":"2024-09-16T03:39:57.677624+0000","last_unstale":"2024-09-16T03:40:03.821915+0000","last_undegraded":"2024-09-16T03:40:03.821915+0000","last_fullsized":"2024-09-16T03:40:03.821915+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:39:14.082477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.082477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.802773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":786,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.917748+0000","last_change":"2024-09-16T03:40:03.603663+0000","last_active":"2024-09-16T03:40:04.917748+0000","last_peered":"2024-09-16T03:40:04.917748+0000","last_clean":"2024-09-16T03:40:04.917748+0000","last_became_active":"2024-09-16T03:40:03.603137+0000","last_became_peered":"2024-09-16T03:40:03.603137+0000","last_unstale":"2024-09-16T03:40:04.917748+0000","last_undegraded":"2024-09-16T03:40:04.917748+0000","last_fullsized":"2024-09-16T03:40:04.917748+0000","mapping_epoch":178,"log_start":"168'338","ondisk_log_start":"168'338","created":31,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":350,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:04.109039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":326,"num_read_kb":8268,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":15647087,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[8,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":376,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.696945+0000","last_change":"2024-09-16T03:39:58.680587+0000","last_active":"2024-09-16T03:40:01.696945+0000","last_peered":"2024-09-16T03:40:01.696945+0000","last_clean":"2024-09-16T03:40:01.696945+0000","last_became_active":"2024-09-16T03:39:58.680154+0000","last_became_peered":"2024-09-16T03:39:58.680154+0000","last_unstale":"2024-09-16T03:40:01.696945+0000","last_undegraded":"2024-09-16T03:40:01.696945+0000","last_fullsized":"2024-09-16T03:40:01.696945+0000","mapping_epoch":173,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:35.828203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.828203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.560048+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,3],"acting":[15,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":334,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.394775+0000","last_change":"2024-09-16T03:39:29.848063+0000","last_active":"2024-09-16T03:40:01.394775+0000","last_peered":"2024-09-16T03:40:01.394775+0000","last_clean":"2024-09-16T03:40:01.394775+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:40:01.394775+0000","last_undegraded":"2024-09-16T03:40:01.394775+0000","last_fullsized":"2024-09-16T03:40:01.394775+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:33.922407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"94'34","reported_seq":438,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.595525+0000","last_change":"2024-09-16T03:39:58.085927+0000","last_active":"2024-09-16T03:40:01.595525+0000","last_peered":"2024-09-16T03:40:01.595525+0000","last_clean":"2024-09-16T03:40:01.595525+0000","last_became_active":"2024-09-16T03:38:33.580022+0000","last_became_peered":"2024-09-16T03:38:33.580022+0000","last_unstale":"2024-09-16T03:40:01.595525+0000","last_undegraded":"2024-09-16T03:40:01.595525+0000","last_fullsized":"2024-09-16T03:40:01.595525+0000","mapping_epoch":96,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:39:58.085847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:58.085847+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.907220+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":102,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.876273+0000","last_change":"2024-09-16T03:39:09.848905+0000","last_active":"2024-09-16T03:40:04.876273+0000","last_peered":"2024-09-16T03:40:04.876273+0000","last_clean":"2024-09-16T03:40:04.876273+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:04.876273+0000","last_undegraded":"2024-09-16T03:40:04.876273+0000","last_fullsized":"2024-09-16T03:40:04.876273+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:52.781538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"167'156","reported_seq":728,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.721615+0000","last_change":"2024-09-16T03:39:59.774876+0000","last_active":"2024-09-16T03:40:03.721615+0000","last_peered":"2024-09-16T03:40:03.721615+0000","last_clean":"2024-09-16T03:40:03.721615+0000","last_became_active":"2024-09-16T03:39:59.774568+0000","last_became_peered":"2024-09-16T03:39:59.774568+0000","last_unstale":"2024-09-16T03:40:03.721615+0000","last_undegraded":"2024-09-16T03:40:03.721615+0000","last_fullsized":"2024-09-16T03:40:03.721615+0000","mapping_epoch":174,"log_start":"167'153","ondisk_log_start":"167'153","created":32,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'118","last_scrub_stamp":"2024-09-16T03:38:39.704919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.704919+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":150,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.113275+0000","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6958230,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":93961,"num_write":45,"num_write_kb":13895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5176846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15,8],"acting":[10,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":346,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.406014+0000","last_change":"2024-09-16T03:39:39.004001+0000","last_active":"2024-09-16T03:40:03.406014+0000","last_peered":"2024-09-16T03:40:03.406014+0000","last_clean":"2024-09-16T03:40:03.406014+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:40:03.406014+0000","last_undegraded":"2024-09-16T03:40:03.406014+0000","last_fullsized":"2024-09-16T03:40:03.406014+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.017825+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":359,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.410979+0000","last_change":"2024-09-16T03:39:59.297976+0000","last_active":"2024-09-16T03:40:04.410979+0000","last_peered":"2024-09-16T03:40:04.410979+0000","last_clean":"2024-09-16T03:40:04.410979+0000","last_became_active":"2024-09-16T03:38:33.484091+0000","last_became_peered":"2024-09-16T03:38:33.484091+0000","last_unstale":"2024-09-16T03:40:04.410979+0000","last_undegraded":"2024-09-16T03:40:04.410979+0000","last_fullsized":"2024-09-16T03:40:04.410979+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:39:59.297888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:59.297888+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:25.801060+0000","scrub_duration":390,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":333,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.714444+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:40:04.714444+0000","last_peered":"2024-09-16T03:40:04.714444+0000","last_clean":"2024-09-16T03:40:04.714444+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:04.714444+0000","last_undegraded":"2024-09-16T03:40:04.714444+0000","last_fullsized":"2024-09-16T03:40:04.714444+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:29.326990+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.10","version":"167'81","reported_seq":518,"reported_epoch":179,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:03.721581+0000","last_change":"2024-09-16T03:39:56.206774+0000","last_active":"2024-09-16T03:40:03.721581+0000","last_peered":"2024-09-16T03:40:03.721581+0000","last_clean":"2024-09-16T03:39:52.997913+0000","last_became_active":"2024-09-16T03:39:56.106140+0000","last_became_peered":"2024-09-16T03:39:56.106140+0000","last_unstale":"2024-09-16T03:40:03.721581+0000","last_undegraded":"2024-09-16T03:40:03.721581+0000","last_fullsized":"2024-09-16T03:40:03.721581+0000","mapping_epoch":170,"log_start":"165'78","ondisk_log_start":"165'78","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"118'64","last_scrub_stamp":"2024-09-16T03:39:04.442556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.442556+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":75,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3411887,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":20,"num_write_kb":5652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,11],"acting":[10,3,4],"avail_no_missing":["10","3","4"],"object_location_counts":[{"shards":"3,4,10","objects":1}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"167'26","reported_seq":423,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-16T03:40:05.635831+0000","last_change":"2024-09-16T03:39:03.929961+0000","last_active":"2024-09-16T03:40:05.635831+0000","last_peered":"2024-09-16T03:40:05.635831+0000","last_clean":"2024-09-16T03:40:05.635831+0000","last_became_active":"2024-09-16T03:39:01.696690+0000","last_became_peered":"2024-09-16T03:39:01.696690+0000","last_unstale":"2024-09-16T03:40:05.635831+0000","last_undegraded":"2024-09-16T03:40:05.635831+0000","last_fullsized":"2024-09-16T03:40:05.635831+0000","mapping_epoch":63,"log_start":"155'22","ondisk_log_start":"155'22","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:03.929863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.929863+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":22,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:13.702587+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3579957,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":37538,"num_write":10,"num_write_kb":3021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"147'23","reported_seq":502,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405868+0000","last_change":"2024-09-16T03:39:14.925442+0000","last_active":"2024-09-16T03:40:03.405868+0000","last_peered":"2024-09-16T03:40:03.405868+0000","last_clean":"2024-09-16T03:40:03.405868+0000","last_became_active":"2024-09-16T03:39:13.974170+0000","last_became_peered":"2024-09-16T03:39:13.974170+0000","last_unstale":"2024-09-16T03:40:03.405868+0000","last_undegraded":"2024-09-16T03:40:03.405868+0000","last_fullsized":"2024-09-16T03:40:03.405868+0000","mapping_epoch":138,"log_start":"127'20","ondisk_log_start":"127'20","created":50,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'20","last_scrub_stamp":"2024-09-16T03:39:14.925356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.925356+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":20,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.341117+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2000099,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5461,"num_write":1,"num_write_kb":458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.13","version":"165'83","reported_seq":583,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.511263+0000","last_change":"2024-09-16T03:39:15.224056+0000","last_active":"2024-09-16T03:40:04.511263+0000","last_peered":"2024-09-16T03:40:04.511263+0000","last_clean":"2024-09-16T03:40:04.511263+0000","last_became_active":"2024-09-16T03:39:15.223614+0000","last_became_peered":"2024-09-16T03:39:15.223614+0000","last_unstale":"2024-09-16T03:40:04.511263+0000","last_undegraded":"2024-09-16T03:40:04.511263+0000","last_fullsized":"2024-09-16T03:40:04.511263+0000","mapping_epoch":139,"log_start":"154'79","ondisk_log_start":"154'79","created":50,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'73","last_scrub_stamp":"2024-09-16T03:39:03.124583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.124583+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":77,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.132930+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445501,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":11846,"num_write":10,"num_write_kb":2874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"104'20","reported_seq":417,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-16T03:40:05.535903+0000","last_change":"2024-09-16T03:39:01.898409+0000","last_active":"2024-09-16T03:40:05.535903+0000","last_peered":"2024-09-16T03:40:05.535903+0000","last_clean":"2024-09-16T03:40:05.535903+0000","last_became_active":"2024-09-16T03:39:01.596287+0000","last_became_peered":"2024-09-16T03:39:01.596287+0000","last_unstale":"2024-09-16T03:40:05.535903+0000","last_undegraded":"2024-09-16T03:40:05.535903+0000","last_fullsized":"2024-09-16T03:40:05.535903+0000","mapping_epoch":114,"log_start":"98'17","ondisk_log_start":"98'17","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:01.898331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.898331+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.728364+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"123'65","reported_seq":569,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.513804+0000","last_change":"2024-09-16T03:39:14.175562+0000","last_active":"2024-09-16T03:40:04.513804+0000","last_peered":"2024-09-16T03:40:04.513804+0000","last_clean":"2024-09-16T03:40:04.513804+0000","last_became_active":"2024-09-16T03:39:13.671769+0000","last_became_peered":"2024-09-16T03:39:13.671769+0000","last_unstale":"2024-09-16T03:40:04.513804+0000","last_undegraded":"2024-09-16T03:40:04.513804+0000","last_fullsized":"2024-09-16T03:40:04.513804+0000","mapping_epoch":138,"log_start":"123'62","ondisk_log_start":"123'62","created":50,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"123'65","last_scrub_stamp":"2024-09-16T03:39:14.175493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.175493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:36.855683+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"124'86","reported_seq":539,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.597028+0000","last_change":"2024-09-16T03:39:03.073530+0000","last_active":"2024-09-16T03:40:01.597028+0000","last_peered":"2024-09-16T03:40:01.597028+0000","last_clean":"2024-09-16T03:40:01.597028+0000","last_became_active":"2024-09-16T03:39:01.797805+0000","last_became_peered":"2024-09-16T03:39:01.797805+0000","last_unstale":"2024-09-16T03:40:01.597028+0000","last_undegraded":"2024-09-16T03:40:01.597028+0000","last_fullsized":"2024-09-16T03:40:01.597028+0000","mapping_epoch":115,"log_start":"124'82","ondisk_log_start":"124'82","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.073459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.073459+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":80,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:30.483816+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.17","version":"169'60","reported_seq":514,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.406094+0000","last_change":"2024-09-16T03:39:35.812368+0000","last_active":"2024-09-16T03:40:03.406094+0000","last_peered":"2024-09-16T03:40:03.406094+0000","last_clean":"2024-09-16T03:40:03.406094+0000","last_became_active":"2024-09-16T03:39:03.717420+0000","last_became_peered":"2024-09-16T03:39:03.717420+0000","last_unstale":"2024-09-16T03:40:03.406094+0000","last_undegraded":"2024-09-16T03:40:03.406094+0000","last_fullsized":"2024-09-16T03:40:03.406094+0000","mapping_epoch":128,"log_start":"148'56","ondisk_log_start":"148'56","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"148'56","last_scrub_stamp":"2024-09-16T03:39:35.812298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:35.812298+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":55,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:54.749022+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731220,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"155'55","reported_seq":474,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.714255+0000","last_change":"2024-09-16T03:39:02.668794+0000","last_active":"2024-09-16T03:40:04.714255+0000","last_peered":"2024-09-16T03:40:04.714255+0000","last_clean":"2024-09-16T03:40:04.714255+0000","last_became_active":"2024-09-16T03:38:47.562641+0000","last_became_peered":"2024-09-16T03:38:47.562641+0000","last_unstale":"2024-09-16T03:40:04.714255+0000","last_undegraded":"2024-09-16T03:40:04.714255+0000","last_fullsized":"2024-09-16T03:40:04.714255+0000","mapping_epoch":112,"log_start":"155'52","ondisk_log_start":"155'52","created":50,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'48","last_scrub_stamp":"2024-09-16T03:39:02.668681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.668681+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:14.665737+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3344659,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":25041,"num_write":28,"num_write_kb":7736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":103,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.775890+0000","last_change":"2024-09-16T03:39:09.847101+0000","last_active":"2024-09-16T03:40:04.775890+0000","last_peered":"2024-09-16T03:40:04.775890+0000","last_clean":"2024-09-16T03:40:04.775890+0000","last_became_active":"2024-09-16T03:39:09.846691+0000","last_became_peered":"2024-09-16T03:39:09.846691+0000","last_unstale":"2024-09-16T03:40:04.775890+0000","last_undegraded":"2024-09-16T03:40:04.775890+0000","last_fullsized":"2024-09-16T03:40:04.775890+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:01.106840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.106840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:27.531290+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"169'138","reported_seq":682,"reported_epoch":177,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:01.394698+0000","last_change":"2024-09-16T03:39:54.206127+0000","last_active":"2024-09-16T03:40:01.394698+0000","last_peered":"2024-09-16T03:40:01.394698+0000","last_clean":"2024-09-16T03:39:28.503345+0000","last_became_active":"2024-09-16T03:39:54.099638+0000","last_became_peered":"2024-09-16T03:39:54.099638+0000","last_unstale":"2024-09-16T03:40:01.394698+0000","last_undegraded":"2024-09-16T03:40:01.394698+0000","last_fullsized":"2024-09-16T03:40:01.394698+0000","mapping_epoch":168,"log_start":"148'134","ondisk_log_start":"148'134","created":100,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'115","last_scrub_stamp":"2024-09-16T03:38:35.254488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.254488+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":134,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6756546,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":51953,"num_write":24,"num_write_kb":6981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3860936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,14],"acting":[6,11,15],"avail_no_missing":["6","11","15"],"object_location_counts":[{"shards":"6,11,15","objects":2}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"147'136","reported_seq":740,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.413680+0000","last_change":"2024-09-16T03:40:00.893177+0000","last_active":"2024-09-16T03:40:04.413680+0000","last_peered":"2024-09-16T03:40:04.413680+0000","last_clean":"2024-09-16T03:40:04.413680+0000","last_became_active":"2024-09-16T03:39:58.580803+0000","last_became_peered":"2024-09-16T03:39:58.580803+0000","last_unstale":"2024-09-16T03:40:04.413680+0000","last_undegraded":"2024-09-16T03:40:04.413680+0000","last_fullsized":"2024-09-16T03:40:04.413680+0000","mapping_epoch":173,"log_start":"139'133","ondisk_log_start":"139'133","created":100,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'136","last_scrub_stamp":"2024-09-16T03:40:00.893096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:40:00.893096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":127,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:29.744583+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4941390,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":116543,"num_write":32,"num_write_kb":9493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5788673,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,10],"acting":[3,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"99'56","reported_seq":481,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.394693+0000","last_change":"2024-09-16T03:39:33.570828+0000","last_active":"2024-09-16T03:40:01.394693+0000","last_peered":"2024-09-16T03:40:01.394693+0000","last_clean":"2024-09-16T03:40:01.394693+0000","last_became_active":"2024-09-16T03:39:33.570516+0000","last_became_peered":"2024-09-16T03:39:33.570516+0000","last_unstale":"2024-09-16T03:40:01.394693+0000","last_undegraded":"2024-09-16T03:40:01.394693+0000","last_fullsized":"2024-09-16T03:40:01.394693+0000","mapping_epoch":153,"log_start":"90'53","ondisk_log_start":"90'53","created":100,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":5,"last_scrub":"99'56","last_scrub_stamp":"2024-09-16T03:39:00.988517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.988517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.293621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"92'14","reported_seq":372,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.513806+0000","last_change":"2024-09-16T03:38:49.092747+0000","last_active":"2024-09-16T03:40:04.513806+0000","last_peered":"2024-09-16T03:40:04.513806+0000","last_clean":"2024-09-16T03:40:04.513806+0000","last_became_active":"2024-09-16T03:38:48.661213+0000","last_became_peered":"2024-09-16T03:38:48.661213+0000","last_unstale":"2024-09-16T03:40:04.513806+0000","last_undegraded":"2024-09-16T03:40:04.513806+0000","last_fullsized":"2024-09-16T03:40:04.513806+0000","mapping_epoch":113,"log_start":"91'11","ondisk_log_start":"91'11","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:49.092687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.092687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:14.647233+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,12],"acting":[3,11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"92'59","reported_seq":487,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405948+0000","last_change":"2024-09-16T03:39:48.664382+0000","last_active":"2024-09-16T03:40:03.405948+0000","last_peered":"2024-09-16T03:40:03.405948+0000","last_clean":"2024-09-16T03:40:03.405948+0000","last_became_active":"2024-09-16T03:38:48.561170+0000","last_became_peered":"2024-09-16T03:38:48.561170+0000","last_unstale":"2024-09-16T03:40:03.405948+0000","last_undegraded":"2024-09-16T03:40:03.405948+0000","last_fullsized":"2024-09-16T03:40:03.405948+0000","mapping_epoch":113,"log_start":"89'56","ondisk_log_start":"89'56","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'59","last_scrub_stamp":"2024-09-16T03:39:48.664277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:48.664277+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.699198+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,6],"acting":[14,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1f","version":"169'132","reported_seq":699,"reported_epoch":179,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:40:03.921588+0000","last_change":"2024-09-16T03:40:03.720048+0000","last_active":"2024-09-16T03:40:03.921588+0000","last_peered":"2024-09-16T03:40:03.921588+0000","last_clean":"2024-09-16T03:40:03.921588+0000","last_became_active":"2024-09-16T03:39:56.619304+0000","last_became_peered":"2024-09-16T03:39:56.619304+0000","last_unstale":"2024-09-16T03:40:03.921588+0000","last_undegraded":"2024-09-16T03:40:03.921588+0000","last_fullsized":"2024-09-16T03:40:03.921588+0000","mapping_epoch":171,"log_start":"165'128","ondisk_log_start":"165'128","created":100,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'95","last_scrub_stamp":"2024-09-16T03:38:37.201959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:37.201959+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":128,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10386891,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":131,"num_read_kb":122162,"num_write":49,"num_write_kb":13103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":13164419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,12],"acting":[10,1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.20","version":"170'165","reported_seq":779,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.714188+0000","last_change":"2024-09-16T03:39:49.993969+0000","last_active":"2024-09-16T03:40:04.714188+0000","last_peered":"2024-09-16T03:40:04.714188+0000","last_clean":"2024-09-16T03:40:04.714188+0000","last_became_active":"2024-09-16T03:39:49.893618+0000","last_became_peered":"2024-09-16T03:39:49.893618+0000","last_unstale":"2024-09-16T03:40:04.714188+0000","last_undegraded":"2024-09-16T03:40:04.714188+0000","last_fullsized":"2024-09-16T03:40:04.714188+0000","mapping_epoch":165,"log_start":"169'162","ondisk_log_start":"169'162","created":100,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'118","last_scrub_stamp":"2024-09-16T03:38:36.698150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:36.698150+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":162,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:55.225728+0000","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11378504,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":184,"num_read_kb":187520,"num_write":53,"num_write_kb":16191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6817236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[8,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.21","version":"48'7","reported_seq":232,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405379+0000","last_change":"2024-09-16T03:38:49.064947+0000","last_active":"2024-09-16T03:40:03.405379+0000","last_peered":"2024-09-16T03:40:03.405379+0000","last_clean":"2024-09-16T03:40:03.405379+0000","last_became_active":"2024-09-16T03:38:48.559431+0000","last_became_peered":"2024-09-16T03:38:48.559431+0000","last_unstale":"2024-09-16T03:40:03.405379+0000","last_undegraded":"2024-09-16T03:40:03.405379+0000","last_fullsized":"2024-09-16T03:40:03.405379+0000","mapping_epoch":113,"log_start":"40'4","ondisk_log_start":"40'4","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":6,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:49.064870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064870+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,10],"acting":[14,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.22","version":"45'6","reported_seq":237,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.776000+0000","last_change":"2024-09-16T03:39:13.974174+0000","last_active":"2024-09-16T03:40:04.776000+0000","last_peered":"2024-09-16T03:40:04.776000+0000","last_clean":"2024-09-16T03:40:04.776000+0000","last_became_active":"2024-09-16T03:39:13.973813+0000","last_became_peered":"2024-09-16T03:39:13.973813+0000","last_unstale":"2024-09-16T03:40:04.776000+0000","last_undegraded":"2024-09-16T03:40:04.776000+0000","last_fullsized":"2024-09-16T03:40:04.776000+0000","mapping_epoch":138,"log_start":"39'3","ondisk_log_start":"39'3","created":100,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:39:00.968448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.968448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:30.310727+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,6],"acting":[9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.23","version":"143'42","reported_seq":468,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405726+0000","last_change":"2024-09-16T03:40:00.075385+0000","last_active":"2024-09-16T03:40:03.405726+0000","last_peered":"2024-09-16T03:40:03.405726+0000","last_clean":"2024-09-16T03:40:03.405726+0000","last_became_active":"2024-09-16T03:38:49.463797+0000","last_became_peered":"2024-09-16T03:38:49.463797+0000","last_unstale":"2024-09-16T03:40:03.405726+0000","last_undegraded":"2024-09-16T03:40:03.405726+0000","last_fullsized":"2024-09-16T03:40:03.405726+0000","mapping_epoch":114,"log_start":"142'39","ondisk_log_start":"142'39","created":100,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":6,"last_scrub":"143'42","last_scrub_stamp":"2024-09-16T03:40:00.075276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:40:00.075276+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":38,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:09.661629+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672379,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":24141,"num_write":9,"num_write_kb":2300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,2],"acting":[14,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.24","version":"105'32","reported_seq":264,"reported_epoch":176,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:00.702345+0000","last_change":"2024-09-16T03:39:39.285524+0000","last_active":"2024-09-16T03:40:00.702345+0000","last_peered":"2024-09-16T03:40:00.702345+0000","last_clean":"2024-09-16T03:38:17.396491+0000","last_became_active":"2024-09-16T03:39:38.786889+0000","last_became_peered":"2024-09-16T03:39:38.786889+0000","last_unstale":"2024-09-16T03:40:00.702345+0000","last_undegraded":"2024-09-16T03:40:00.702345+0000","last_fullsized":"2024-09-16T03:40:00.702345+0000","mapping_epoch":158,"log_start":"83'29","ondisk_log_start":"83'29","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.978888+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":4728,"num_write":10,"num_write_kb":2338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,14],"acting":[2,11,15],"avail_no_missing":["2","0","3","11","14","15"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"169'86","reported_seq":474,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.399723+0000","last_change":"2024-09-16T03:39:40.798757+0000","last_active":"2024-09-16T03:40:03.399723+0000","last_peered":"2024-09-16T03:40:03.399723+0000","last_clean":"2024-09-16T03:40:03.399723+0000","last_became_active":"2024-09-16T03:39:40.798223+0000","last_became_peered":"2024-09-16T03:39:40.798223+0000","last_unstale":"2024-09-16T03:40:03.399723+0000","last_undegraded":"2024-09-16T03:40:03.399723+0000","last_fullsized":"2024-09-16T03:40:03.399723+0000","mapping_epoch":160,"log_start":"130'83","ondisk_log_start":"130'83","created":110,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'77","last_scrub_stamp":"2024-09-16T03:38:48.088768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.088768+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":80,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:14.812409+0000","scrub_duration":388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2134555,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":10,"num_write_kb":2500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1126280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,7,14],"acting":[12,7,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.26","version":"169'64","reported_seq":478,"reported_epoch":177,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:01.394791+0000","last_change":"2024-09-16T03:39:38.986370+0000","last_active":"2024-09-16T03:40:01.394791+0000","last_peered":"2024-09-16T03:40:01.394791+0000","last_clean":"2024-09-16T03:39:19.447260+0000","last_became_active":"2024-09-16T03:39:38.985600+0000","last_became_peered":"2024-09-16T03:39:38.985600+0000","last_unstale":"2024-09-16T03:40:01.394791+0000","last_undegraded":"2024-09-16T03:40:01.394791+0000","last_fullsized":"2024-09-16T03:40:01.394791+0000","mapping_epoch":158,"log_start":"142'60","ondisk_log_start":"142'60","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:49.477297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.477297+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":59,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3647185,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":24205,"num_write":24,"num_write_kb":7094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,10],"acting":[6,10,4],"avail_no_missing":["6","4","10"],"object_location_counts":[{"shards":"4,6,10","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"109'66","reported_seq":373,"reported_epoch":177,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:01.494833+0000","last_change":"2024-09-16T03:39:38.989488+0000","last_active":"2024-09-16T03:40:01.494833+0000","last_peered":"2024-09-16T03:40:01.494833+0000","last_clean":"2024-09-16T03:38:43.524716+0000","last_became_active":"2024-09-16T03:39:38.888633+0000","last_became_peered":"2024-09-16T03:39:38.888633+0000","last_unstale":"2024-09-16T03:40:01.494833+0000","last_undegraded":"2024-09-16T03:40:01.494833+0000","last_fullsized":"2024-09-16T03:40:01.494833+0000","mapping_epoch":158,"log_start":"105'63","ondisk_log_start":"105'63","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'66","last_scrub_stamp":"2024-09-16T03:38:47.511252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.511252+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2512680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":54642,"num_write":20,"num_write_kb":5643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[6,10,11],"avail_no_missing":["6","10","11"],"object_location_counts":[{"shards":"6,10,11","objects":1}],"blocked_by":[],"up_primary":8,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.28","version":"171'70","reported_seq":651,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.775953+0000","last_change":"2024-09-16T03:39:41.471715+0000","last_active":"2024-09-16T03:40:04.775953+0000","last_peered":"2024-09-16T03:40:04.775953+0000","last_clean":"2024-09-16T03:40:04.775953+0000","last_became_active":"2024-09-16T03:39:41.471298+0000","last_became_peered":"2024-09-16T03:39:41.471298+0000","last_unstale":"2024-09-16T03:40:04.775953+0000","last_undegraded":"2024-09-16T03:40:04.775953+0000","last_fullsized":"2024-09-16T03:40:04.775953+0000","mapping_epoch":161,"log_start":"118'67","ondisk_log_start":"118'67","created":110,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:48.551929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.551929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":66,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:12.543779+0000","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395113,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":70,"num_read_kb":76288,"num_write":5,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":14443776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,2],"acting":[9,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":54,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.721646+0000","last_change":"2024-09-16T03:39:37.881274+0000","last_active":"2024-09-16T03:40:03.721646+0000","last_peered":"2024-09-16T03:40:03.721646+0000","last_clean":"2024-09-16T03:40:03.721646+0000","last_became_active":"2024-09-16T03:39:37.880874+0000","last_became_peered":"2024-09-16T03:39:37.880874+0000","last_unstale":"2024-09-16T03:40:03.721646+0000","last_undegraded":"2024-09-16T03:40:03.721646+0000","last_fullsized":"2024-09-16T03:40:03.721646+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:00.834742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.834742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:21.681138+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,2],"acting":[10,14,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2a","version":"108'122","reported_seq":575,"reported_epoch":180,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:04.413143+0000","last_change":"2024-09-16T03:39:39.190319+0000","last_active":"2024-09-16T03:40:04.413143+0000","last_peered":"2024-09-16T03:40:04.413143+0000","last_clean":"2024-09-16T03:38:41.455596+0000","last_became_active":"2024-09-16T03:39:38.888612+0000","last_became_peered":"2024-09-16T03:39:38.888612+0000","last_unstale":"2024-09-16T03:40:04.413143+0000","last_undegraded":"2024-09-16T03:40:04.413143+0000","last_fullsized":"2024-09-16T03:40:04.413143+0000","mapping_epoch":158,"log_start":"107'115","ondisk_log_start":"107'115","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'122","last_scrub_stamp":"2024-09-16T03:38:47.964439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.964439+0000","objects_scrubbed":4,"log_size":7,"log_dups_size":112,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647218,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":6805,"num_write":7,"num_write_kb":1927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,12],"acting":[3,8,10],"avail_no_missing":["3","8","10"],"object_location_counts":[{"shards":"3,8,10","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2b","version":"149'132","reported_seq":695,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.411142+0000","last_change":"2024-09-16T03:39:41.669420+0000","last_active":"2024-09-16T03:40:04.411142+0000","last_peered":"2024-09-16T03:40:04.411142+0000","last_clean":"2024-09-16T03:40:04.411142+0000","last_became_active":"2024-09-16T03:39:41.669036+0000","last_became_peered":"2024-09-16T03:39:41.669036+0000","last_unstale":"2024-09-16T03:40:04.411142+0000","last_undegraded":"2024-09-16T03:40:04.411142+0000","last_fullsized":"2024-09-16T03:40:04.411142+0000","mapping_epoch":161,"log_start":"144'128","ondisk_log_start":"144'128","created":110,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:49.064061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064061+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":128,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:55.831512+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092554,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":5914,"num_write":15,"num_write_kb":4670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1630666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2c","version":"167'69","reported_seq":566,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.513289+0000","last_change":"2024-09-16T03:39:41.471606+0000","last_active":"2024-09-16T03:40:04.513289+0000","last_peered":"2024-09-16T03:40:04.513289+0000","last_clean":"2024-09-16T03:40:04.513289+0000","last_became_active":"2024-09-16T03:39:41.471156+0000","last_became_peered":"2024-09-16T03:39:41.471156+0000","last_unstale":"2024-09-16T03:40:04.513289+0000","last_undegraded":"2024-09-16T03:40:04.513289+0000","last_fullsized":"2024-09-16T03:40:04.513289+0000","mapping_epoch":161,"log_start":"154'65","ondisk_log_start":"154'65","created":110,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'58","last_scrub_stamp":"2024-09-16T03:38:47.889276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.889276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":64,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:16.605878+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3495922,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":43795,"num_write":10,"num_write_kb":2812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3787209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"92'14","reported_seq":433,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405770+0000","last_change":"2024-09-16T03:39:40.897844+0000","last_active":"2024-09-16T03:40:03.405770+0000","last_peered":"2024-09-16T03:40:03.405770+0000","last_clean":"2024-09-16T03:40:03.405770+0000","last_became_active":"2024-09-16T03:39:40.897531+0000","last_became_peered":"2024-09-16T03:39:40.897531+0000","last_unstale":"2024-09-16T03:40:03.405770+0000","last_undegraded":"2024-09-16T03:40:03.405770+0000","last_fullsized":"2024-09-16T03:40:03.405770+0000","mapping_epoch":160,"log_start":"91'11","ondisk_log_start":"91'11","created":110,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":6,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:47.115651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.115651+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:56.558775+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,4],"acting":[14,11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":56,"stat_sum":{"num_bytes":165812611,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1731,"num_read_kb":1589036,"num_write":632,"num_write_kb":176819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":68548351,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":24,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":289665024,"data_stored":288999499,"data_compressed":627765,"data_compressed_allocated":688128,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":168,"acting":168,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":78,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1165,"num_read_kb":34194,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":85256126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214495232,"data_stored":229041714,"data_compressed":14285886,"data_compressed_allocated":15241216,"data_compressed_original":30437376,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":2644640,"data_compressed":159066,"data_compressed_allocated":204800,"data_compressed_original":2584576,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247848,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75748,"kb_used_data":40400,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94296092,"statfs":{"total":96636764160,"available":96559198208,"internally_reserved":0,"allocated":41369600,"data_stored":41693615,"data_compressed":1360715,"data_compressed_allocated":1470464,"data_compressed_original":2588672,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247849,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62268,"kb_used_data":31080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94309572,"statfs":{"total":96636764160,"available":96573001728,"internally_reserved":0,"allocated":31825920,"data_stored":32921741,"data_compressed":1089754,"data_compressed_allocated":1212416,"data_compressed_original":3080192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247848,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88380,"kb_used_data":53024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94283460,"statfs":{"total":96636764160,"available":96546263040,"internally_reserved":0,"allocated":54296576,"data_stored":55900546,"data_compressed":2021943,"data_compressed_allocated":2199552,"data_compressed_original":4628480,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280554,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78592,"kb_used_data":47396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94293248,"statfs":{"total":96636764160,"available":96556285952,"internally_reserved":0,"allocated":48533504,"data_stored":50484788,"data_compressed":2303619,"data_compressed_allocated":2494464,"data_compressed_original":5292032,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280554,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280553,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56084,"kb_used_data":24888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94315756,"statfs":{"total":96636764160,"available":96579334144,"internally_reserved":0,"allocated":25485312,"data_stored":25872981,"data_compressed":969719,"data_compressed_allocated":1056768,"data_compressed_original":2154496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313258,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41616,"kb_used_data":14580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330224,"statfs":{"total":96636764160,"available":96594149376,"internally_reserved":0,"allocated":14929920,"data_stored":14505443,"data_compressed":218938,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280554,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83176,"kb_used_data":47828,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94288664,"statfs":{"total":96636764160,"available":96551591936,"internally_reserved":0,"allocated":48975872,"data_stored":49667230,"data_compressed":1266255,"data_compressed_allocated":1409024,"data_compressed_original":2973696,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280553,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88652,"kb_used_data":53296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94283188,"statfs":{"total":96636764160,"available":96545984512,"internally_reserved":0,"allocated":54575104,"data_stored":56406632,"data_compressed":2127618,"data_compressed_allocated":2334720,"data_compressed_original":5017600,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313258,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280553,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39856,"kb_used_data":8664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331984,"statfs":{"total":96636764160,"available":96595951616,"internally_reserved":0,"allocated":8871936,"data_stored":8358851,"data_compressed":95409,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280553,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84632,"kb_used_data":49276,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94287208,"statfs":{"total":96636764160,"available":96550100992,"internally_reserved":0,"allocated":50458624,"data_stored":50794032,"data_compressed":1326174,"data_compressed_allocated":1437696,"data_compressed_original":2605056,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280553,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61132,"kb_used_data":29940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310708,"statfs":{"total":96636764160,"available":96574164992,"internally_reserved":0,"allocated":30658560,"data_stored":31525432,"data_compressed":1015354,"data_compressed_allocated":1114112,"data_compressed_original":2744320,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280554,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79564,"kb_used_data":48368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94292276,"statfs":{"total":96636764160,"available":96555290624,"internally_reserved":0,"allocated":49528832,"data_stored":49021549,"data_compressed":261093,"data_compressed_allocated":315392,"data_compressed_original":663552,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,2,3,4,6,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313257,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46660,"kb_used_data":15460,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325180,"statfs":{"total":96636764160,"available":96588984320,"internally_reserved":0,"allocated":15831040,"data_stored":15256892,"data_compressed":121130,"data_compressed_allocated":163840,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247850,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70800,"kb_used_data":39604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94301040,"statfs":{"total":96636764160,"available":96564264960,"internally_reserved":0,"allocated":40554496,"data_stored":40980662,"data_compressed":1405275,"data_compressed_allocated":1523712,"data_compressed_original":2744320,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31858688,"data_stored":33943544,"data_compressed":2174030,"data_compressed_allocated":2314240,"data_compressed_original":4501504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14860288,"data_stored":14775225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9277440,"data_stored":9253748,"data_compressed":22789,"data_compressed_allocated":24576,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24104960,"data_stored":24092098,"data_compressed":85710,"data_compressed_allocated":94208,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30834688,"data_stored":30754602,"data_compressed":94382,"data_compressed_allocated":102400,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":19304448,"data_stored":19197855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21692416,"data_stored":21628396,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7184384,"data_stored":7135967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":34025472,"data_stored":33912667,"data_compressed":9548,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14385152,"data_stored":14378215,"data_compressed":35763,"data_compressed_allocated":40960,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":47800320,"data_stored":47794347,"data_compressed":165684,"data_compressed_allocated":180224,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12890112,"data_stored":12877307,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":14069760,"data_stored":14019073,"data_compressed":22789,"data_compressed_allocated":24576,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13270816,"data_compressed":120597,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":24944640,"data_stored":24829143,"data_compressed":49947,"data_compressed_allocated":53248,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:06.761 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-16T03:40:06.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 66 2024-09-16T03:40:06.761 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:40:06.755+0000 7f3d4fd13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33442) UID: 0 2024-09-16T03:40:06.762 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:06.762 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:06.768 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T03:40:06.786 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:06.787 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":182,"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","created":"2024-09-16T03:36:37.310421+0000","modified":"2024-09-16T03:40:06.369127+0000","last_up_change":"2024-09-16T03:36:50.058111+0000","last_in_change":"2024-09-16T03:39:28.633923+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:36:49.696307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15,"score_stable":15,"optimal_score":0.20000000298023224,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.89189434051513672,"average_primary_affinity":0.94999998807907104,"average_primary_affinity_weighted":0.88999998569488525}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:36:51.904455+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.7699999809265137,"score_stable":3.7699999809265137,"optimal_score":0.73000001907348633,"raw_score_acting":2.75,"raw_score_stable":2.75,"primary_affinity_weighted":0.89189434051513672,"average_primary_affinity":0.94999998807907104,"average_primary_affinity_weighted":0.88999998569488525}},{"pool":3,"pool_name":"low_tier","create_time":"2024-09-16T03:37:02.999795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.7699999809265137,"score_stable":3.7699999809265137,"optimal_score":0.52999997138977051,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.89189434051513672,"average_primary_affinity":0.94999998807907104,"average_primary_affinity_weighted":0.88999998569488525}},{"pool":4,"pool_name":"unique_pool_0","create_time":"2024-09-16T03:37:04.279293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":56,"pg_placement_num":56,"pg_placement_num_target":56,"pg_num_target":56,"pg_num_pending":56,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"157","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"126","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"dedup_cdc_chunk_size":131072,"dedup_chunk_algorithm":"fastcdc","dedup_tier":3,"fingerprint_algorithm":"sha256"},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1500000953674316,"score_stable":2.4200000762939453,"optimal_score":0.93000000715255737,"raw_score_acting":2,"raw_score_stable":2.25,"primary_affinity_weighted":0.89189434051513672,"average_primary_affinity":0.94999998807907104,"average_primary_affinity_weighted":0.88999998569488525}}],"osds":[{"osd":0,"uuid":"bdbe961a-78d3-4f7d-88ed-ce24475d49e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6813","nonce":3924351635}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6814","nonce":3924351635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6816","nonce":3924351635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6815","nonce":3924351635}]},"public_addr":"172.21.15.64:6813/3924351635","cluster_addr":"172.21.15.64:6814/3924351635","heartbeat_back_addr":"172.21.15.64:6816/3924351635","heartbeat_front_addr":"172.21.15.64:6815/3924351635","state":["exists","up"]},{"osd":1,"uuid":"154c8ddd-7a3d-402c-b9d6-28538f4e6bc4","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6804","nonce":624977804}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6805","nonce":624977804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6807","nonce":624977804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6806","nonce":624977804}]},"public_addr":"172.21.15.112:6804/624977804","cluster_addr":"172.21.15.112:6805/624977804","heartbeat_back_addr":"172.21.15.112:6807/624977804","heartbeat_front_addr":"172.21.15.112:6806/624977804","state":["exists","up"]},{"osd":2,"uuid":"2a06de4e-0835-4a42-8c8a-49fb55aab566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6812","nonce":2804648086}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6813","nonce":2804648086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6815","nonce":2804648086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6814","nonce":2804648086}]},"public_addr":"172.21.15.155:6812/2804648086","cluster_addr":"172.21.15.155:6813/2804648086","heartbeat_back_addr":"172.21.15.155:6815/2804648086","heartbeat_front_addr":"172.21.15.155:6814/2804648086","state":["exists","up"]},{"osd":3,"uuid":"8a87f9f0-c380-4f38-bbd2-06b2def4fd2d","up":1,"in":1,"weight":0.6943511962890625,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6800","nonce":1935597988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6801","nonce":1935597988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6803","nonce":1935597988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6802","nonce":1935597988}]},"public_addr":"172.21.15.157:6800/1935597988","cluster_addr":"172.21.15.157:6801/1935597988","heartbeat_back_addr":"172.21.15.157:6803/1935597988","heartbeat_front_addr":"172.21.15.157:6802/1935597988","state":["exists","up"]},{"osd":4,"uuid":"5ac53318-12e8-496c-a87f-ba5de79c52aa","up":1,"in":1,"weight":0.4737548828125,"primary_affinity":0.9752197265625,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":2675475248}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":2675475248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6812","nonce":2675475248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":2675475248}]},"public_addr":"172.21.15.64:6809/2675475248","cluster_addr":"172.21.15.64:6810/2675475248","heartbeat_back_addr":"172.21.15.64:6812/2675475248","heartbeat_front_addr":"172.21.15.64:6811/2675475248","state":["exists","up"]},{"osd":5,"uuid":"675c81a4-2314-463e-82a7-0060735c78fd","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6800","nonce":499905276}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6801","nonce":499905276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6803","nonce":499905276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6802","nonce":499905276}]},"public_addr":"172.21.15.112:6800/499905276","cluster_addr":"172.21.15.112:6801/499905276","heartbeat_back_addr":"172.21.15.112:6803/499905276","heartbeat_front_addr":"172.21.15.112:6802/499905276","state":["exists","up"]},{"osd":6,"uuid":"b18962a7-3b23-4e50-bf31-1d0629557423","up":1,"in":1,"weight":0.95001220703125,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":169,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6808","nonce":2839983406}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6809","nonce":2839983406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6811","nonce":2839983406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6810","nonce":2839983406}]},"public_addr":"172.21.15.155:6808/2839983406","cluster_addr":"172.21.15.155:6809/2839983406","heartbeat_back_addr":"172.21.15.155:6811/2839983406","heartbeat_front_addr":"172.21.15.155:6810/2839983406","state":["exists","up"]},{"osd":7,"uuid":"9706c1e8-dac0-4e16-8bc5-7cacc6f13870","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6808","nonce":3021982871}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6809","nonce":3021982871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6811","nonce":3021982871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6810","nonce":3021982871}]},"public_addr":"172.21.15.157:6808/3021982871","cluster_addr":"172.21.15.157:6809/3021982871","heartbeat_back_addr":"172.21.15.157:6811/3021982871","heartbeat_front_addr":"172.21.15.157:6810/3021982871","state":["exists","up"]},{"osd":8,"uuid":"fafd06b8-c005-4471-a128-9dd51580c0f0","up":1,"in":1,"weight":0.8023529052734375,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":178,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":3609017627}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":3609017627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":3609017627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":3609017627}]},"public_addr":"172.21.15.64:6805/3609017627","cluster_addr":"172.21.15.64:6806/3609017627","heartbeat_back_addr":"172.21.15.64:6808/3609017627","heartbeat_front_addr":"172.21.15.64:6807/3609017627","state":["exists","up"]},{"osd":9,"uuid":"79e38e57-52c2-416f-9b90-75621f06d3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6812","nonce":701981815}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6813","nonce":701981815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6815","nonce":701981815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6814","nonce":701981815}]},"public_addr":"172.21.15.112:6812/701981815","cluster_addr":"172.21.15.112:6813/701981815","heartbeat_back_addr":"172.21.15.112:6815/701981815","heartbeat_front_addr":"172.21.15.112:6814/701981815","state":["exists","up"]},{"osd":10,"uuid":"8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6804","nonce":4138619523}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6805","nonce":4138619523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6807","nonce":4138619523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6806","nonce":4138619523}]},"public_addr":"172.21.15.155:6804/4138619523","cluster_addr":"172.21.15.155:6805/4138619523","heartbeat_back_addr":"172.21.15.155:6807/4138619523","heartbeat_front_addr":"172.21.15.155:6806/4138619523","state":["exists","up"]},{"osd":11,"uuid":"7a2a169a-ce94-4091-a792-27d0ecf9a7fa","up":1,"in":1,"weight":1,"primary_affinity":0.2950897216796875,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":170,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6804","nonce":3607367937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6805","nonce":3607367937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6807","nonce":3607367937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6806","nonce":3607367937}]},"public_addr":"172.21.15.157:6804/3607367937","cluster_addr":"172.21.15.157:6805/3607367937","heartbeat_back_addr":"172.21.15.157:6807/3607367937","heartbeat_front_addr":"172.21.15.157:6806/3607367937","state":["exists","up"]},{"osd":12,"uuid":"7a280cda-b740-40c8-9fae-1a1ff11651f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":2984461266}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":2984461266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":2984461266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":2984461266}]},"public_addr":"172.21.15.64:6801/2984461266","cluster_addr":"172.21.15.64:6802/2984461266","heartbeat_back_addr":"172.21.15.64:6804/2984461266","heartbeat_front_addr":"172.21.15.64:6803/2984461266","state":["exists","up"]},{"osd":13,"uuid":"ce3a3148-cd90-422c-a106-045de89ab045","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6808","nonce":1616314527}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6809","nonce":1616314527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6811","nonce":1616314527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6810","nonce":1616314527}]},"public_addr":"172.21.15.112:6808/1616314527","cluster_addr":"172.21.15.112:6809/1616314527","heartbeat_back_addr":"172.21.15.112:6811/1616314527","heartbeat_front_addr":"172.21.15.112:6810/1616314527","state":["exists","up"]},{"osd":14,"uuid":"461c2d79-f4e7-4b21-899c-50d4a6cf12f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6800","nonce":4203868121}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6801","nonce":4203868121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6803","nonce":4203868121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6802","nonce":4203868121}]},"public_addr":"172.21.15.155:6800/4203868121","cluster_addr":"172.21.15.155:6801/4203868121","heartbeat_back_addr":"172.21.15.155:6803/4203868121","heartbeat_front_addr":"172.21.15.155:6802/4203868121","state":["exists","up"]},{"osd":15,"uuid":"9a8c0bbe-9389-4b2c-95e1-34d3e66461e9","up":1,"in":1,"weight":0.80743408203125,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6812","nonce":181354529}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6813","nonce":181354529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6815","nonce":181354529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6814","nonce":181354529}]},"public_addr":"172.21.15.157:6812/181354529","cluster_addr":"172.21.15.157:6813/181354529","heartbeat_back_addr":"172.21.15.157:6815/181354529","heartbeat_front_addr":"172.21.15.157:6814/181354529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:18.425590+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:06.827840+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:35.030728+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:23.374251+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:32.706176+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-16T03:39:02.129117+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:19.948144+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:38:56.835860+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:10.351843+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:38:58.705841+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:40.632845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:05.604711+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:25.172167+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-16T03:39:05.022970+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:30.872798+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:30.112593+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.1","osds":[4,7,2]},{"pgid":"4.24","osds":[15,0,14]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"3.1","osds":[6,3,4]},{"pgid":"4.6","osds":[10,6,4]},{"pgid":"4.c","osds":[12,15,9]},{"pgid":"4.e","osds":[4,14,8]},{"pgid":"4.10","osds":[10,3,4]},{"pgid":"4.1a","osds":[6,11,15]},{"pgid":"4.26","osds":[6,10,4]},{"pgid":"4.27","osds":[6,10,11]},{"pgid":"4.2a","osds":[3,8,10]},{"pgid":"4.2e","osds":[4,8,15]},{"pgid":"4.30","osds":[4,10,3]},{"pgid":"4.31","osds":[12,2,4]},{"pgid":"4.33","osds":[3,4,9]},{"pgid":"4.35","osds":[3,9,10]},{"pgid":"4.36","osds":[2,10,15]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:40:06.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:06.870 INFO:teuthology.orchestra.run.smithi112.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:06.871 INFO:teuthology.orchestra.run.smithi112.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:06.879 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T03:40:06.884 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:40:06.883+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:40:06.909 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:40:06.907+0000 7f0ae754b640 -1 osd.0 181 handle_osd_map injecting map crc failure 2024-09-16T03:40:06.960 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:40:06.957+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:40:06.997 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:06.997 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:07.006 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T03:40:07.060 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:40:07.058+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:40:07.150 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:07.151 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:40:07.152 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:07.153 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:07.161 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T03:40:07.163 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:40:07.159+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:40:07.163 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-16T03:40:06.371421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":269007156,"num_objects":204,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":114,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":2982,"num_read_kb":1623303,"num_write":2314,"num_write_kb":513577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":154862352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":24,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":196,"ondisk_log_size":196,"up":207,"acting":207,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":241,"num_osds":14,"num_per_pool_osds":14,"num_per_pool_omap_osds":14,"kb":1321205760,"kb_used":957160,"kb_used_data":503804,"kb_used_omap":21,"kb_used_meta":452970,"kb_avail":1320248600,"statfs":{"total":1352914698240,"available":1351934566400,"internally_reserved":0,"allocated":515895296,"data_stored":523390394,"data_compressed":15582996,"data_compressed_allocated":17137664,"data_compressed_original":35647488,"omap_allocated":22329,"internal_metadata":463841479},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-18,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":5928185,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.970288"},"pg_stats":[{"pgid":"4.37","version":"153'56","reported_seq":533,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.596961+0000","last_change":"2024-09-16T03:39:57.794120+0000","last_active":"2024-09-16T03:40:01.596961+0000","last_peered":"2024-09-16T03:40:01.596961+0000","last_clean":"2024-09-16T03:40:01.596961+0000","last_became_active":"2024-09-16T03:39:57.578155+0000","last_became_peered":"2024-09-16T03:39:57.578155+0000","last_unstale":"2024-09-16T03:40:01.596961+0000","last_undegraded":"2024-09-16T03:40:01.596961+0000","last_fullsized":"2024-09-16T03:40:01.596961+0000","mapping_epoch":172,"log_start":"142'54","ondisk_log_start":"142'54","created":126,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":6,"last_scrub":"153'56","last_scrub_stamp":"2024-09-16T03:39:57.794044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:57.794044+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":49,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:26.249805+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2689186,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,14,7],"acting":[15,14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.36","version":"142'96","reported_seq":567,"reported_epoch":181,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:05.536030+0000","last_change":"2024-09-16T03:39:38.887893+0000","last_active":"2024-09-16T03:40:05.536030+0000","last_peered":"2024-09-16T03:40:05.536030+0000","last_clean":"2024-09-16T03:39:19.651965+0000","last_became_active":"2024-09-16T03:39:38.786957+0000","last_became_peered":"2024-09-16T03:39:38.786957+0000","last_unstale":"2024-09-16T03:40:05.536030+0000","last_undegraded":"2024-09-16T03:40:05.536030+0000","last_fullsized":"2024-09-16T03:40:05.536030+0000","mapping_epoch":158,"log_start":"131'93","ondisk_log_start":"131'93","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.716125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.716125+0000","objects_scrubbed":3,"log_size":3,"log_dups_size":91,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7159980,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":90,"num_read_kb":79209,"num_write":12,"num_write_kb":3200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,11],"acting":[2,10,15],"avail_no_missing":["2","10","15"],"object_location_counts":[{"shards":"2,10,15","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.35","version":"169'85","reported_seq":643,"reported_epoch":180,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:04.513231+0000","last_change":"2024-09-16T03:39:56.515854+0000","last_active":"2024-09-16T03:40:04.513231+0000","last_peered":"2024-09-16T03:40:04.513231+0000","last_clean":"2024-09-16T03:39:35.664815+0000","last_became_active":"2024-09-16T03:39:56.009732+0000","last_became_peered":"2024-09-16T03:39:56.009732+0000","last_unstale":"2024-09-16T03:40:04.513231+0000","last_undegraded":"2024-09-16T03:40:04.513231+0000","last_fullsized":"2024-09-16T03:40:04.513231+0000","mapping_epoch":170,"log_start":"165'81","ondisk_log_start":"165'81","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'67","last_scrub_stamp":"2024-09-16T03:39:15.079987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:15.079987+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":79,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6874195,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":34,"num_read_kb":30309,"num_write":19,"num_write_kb":5766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[3,9,10],"avail_no_missing":["3","9","10"],"object_location_counts":[{"shards":"3,9,10","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.34","version":"149'25","reported_seq":418,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.721835+0000","last_change":"2024-09-16T03:39:58.682198+0000","last_active":"2024-09-16T03:40:03.721835+0000","last_peered":"2024-09-16T03:40:03.721835+0000","last_clean":"2024-09-16T03:40:03.721835+0000","last_became_active":"2024-09-16T03:39:58.581739+0000","last_became_peered":"2024-09-16T03:39:58.581739+0000","last_unstale":"2024-09-16T03:40:03.721835+0000","last_undegraded":"2024-09-16T03:40:03.721835+0000","last_fullsized":"2024-09-16T03:40:03.721835+0000","mapping_epoch":173,"log_start":"149'23","ondisk_log_start":"149'23","created":126,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":6,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:02.919259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.919259+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":20,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:11.357946+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3109361,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":17899,"num_write":5,"num_write_kb":1218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2853416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,9],"acting":[10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.33","version":"148'76","reported_seq":588,"reported_epoch":180,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:04.413245+0000","last_change":"2024-09-16T03:39:39.185026+0000","last_active":"2024-09-16T03:40:04.413245+0000","last_peered":"2024-09-16T03:40:04.413245+0000","last_clean":"2024-09-16T03:39:26.364451+0000","last_became_active":"2024-09-16T03:39:38.883512+0000","last_became_peered":"2024-09-16T03:39:38.883512+0000","last_unstale":"2024-09-16T03:40:04.413245+0000","last_undegraded":"2024-09-16T03:40:04.413245+0000","last_fullsized":"2024-09-16T03:40:04.413245+0000","mapping_epoch":158,"log_start":"137'73","ondisk_log_start":"137'73","created":126,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'72","last_scrub_stamp":"2024-09-16T03:39:02.143339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.143339+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":69,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353046,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":60,"num_read_kb":37925,"num_write":4,"num_write_kb":955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":994759,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,10],"acting":[3,4,9],"avail_no_missing":["3","4","9"],"object_location_counts":[{"shards":"3,4,9","objects":1}],"blocked_by":[],"up_primary":14,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.32","version":"106'17","reported_seq":329,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.401765+0000","last_change":"2024-09-16T03:39:44.270851+0000","last_active":"2024-09-16T03:40:03.401765+0000","last_peered":"2024-09-16T03:40:03.401765+0000","last_clean":"2024-09-16T03:40:03.401765+0000","last_became_active":"2024-09-16T03:39:40.893486+0000","last_became_peered":"2024-09-16T03:39:40.893486+0000","last_unstale":"2024-09-16T03:40:03.401765+0000","last_undegraded":"2024-09-16T03:40:03.401765+0000","last_fullsized":"2024-09-16T03:40:03.401765+0000","mapping_epoch":160,"log_start":"82'13","ondisk_log_start":"82'13","created":126,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":6,"last_scrub":"106'17","last_scrub_stamp":"2024-09-16T03:39:12.108317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:12.108317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":13,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:40:11.560664+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1033647,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":25,"num_read_kb":20741,"num_write":6,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1331059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,12],"acting":[0,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.31","version":"114'17","reported_seq":296,"reported_epoch":179,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:03.399538+0000","last_change":"2024-09-16T03:39:38.988193+0000","last_active":"2024-09-16T03:40:03.399538+0000","last_peered":"2024-09-16T03:40:03.399538+0000","last_clean":"2024-09-16T03:38:48.641776+0000","last_became_active":"2024-09-16T03:39:38.987267+0000","last_became_peered":"2024-09-16T03:39:38.987267+0000","last_unstale":"2024-09-16T03:40:03.399538+0000","last_undegraded":"2024-09-16T03:40:03.399538+0000","last_fullsized":"2024-09-16T03:40:03.399538+0000","mapping_epoch":158,"log_start":"107'14","ondisk_log_start":"107'14","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:35.719895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:35.719895+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1495840,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":32,"num_read_kb":28046,"num_write":4,"num_write_kb":1066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,11],"acting":[12,2,4],"avail_no_missing":["12","2","4"],"object_location_counts":[{"shards":"2,4,12","objects":1}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.30","version":"118'64","reported_seq":420,"reported_epoch":180,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:04.504336+0000","last_change":"2024-09-16T03:39:56.515576+0000","last_active":"2024-09-16T03:40:04.504336+0000","last_peered":"2024-09-16T03:40:04.504336+0000","last_clean":"2024-09-16T03:39:52.798009+0000","last_became_active":"2024-09-16T03:39:56.404165+0000","last_became_peered":"2024-09-16T03:39:56.404165+0000","last_unstale":"2024-09-16T03:40:04.504336+0000","last_undegraded":"2024-09-16T03:40:04.504336+0000","last_fullsized":"2024-09-16T03:40:04.504336+0000","mapping_epoch":170,"log_start":"113'61","ondisk_log_start":"113'61","created":126,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":6,"last_scrub":"118'64","last_scrub_stamp":"2024-09-16T03:39:38.383253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.383253+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":58,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,3],"avail_no_missing":["4","3","9","10"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"113'94","reported_seq":474,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405969+0000","last_change":"2024-09-16T03:39:42.292571+0000","last_active":"2024-09-16T03:40:03.405969+0000","last_peered":"2024-09-16T03:40:03.405969+0000","last_clean":"2024-09-16T03:40:03.405969+0000","last_became_active":"2024-09-16T03:39:40.802316+0000","last_became_peered":"2024-09-16T03:39:40.802316+0000","last_unstale":"2024-09-16T03:40:03.405969+0000","last_undegraded":"2024-09-16T03:40:03.405969+0000","last_fullsized":"2024-09-16T03:40:03.405969+0000","mapping_epoch":160,"log_start":"107'90","ondisk_log_start":"107'90","created":126,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":6,"last_scrub":"113'94","last_scrub_stamp":"2024-09-16T03:39:03.319012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.319012+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":87,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-16T03:40:11.566141+0000","scrub_duration":319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1470629,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,3],"acting":[14,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.2e","version":"155'93","reported_seq":581,"reported_epoch":180,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:04.411225+0000","last_change":"2024-09-16T03:39:39.286505+0000","last_active":"2024-09-16T03:40:04.411225+0000","last_peered":"2024-09-16T03:40:04.411225+0000","last_clean":"2024-09-16T03:39:35.765256+0000","last_became_active":"2024-09-16T03:39:38.884986+0000","last_became_peered":"2024-09-16T03:39:38.884986+0000","last_unstale":"2024-09-16T03:40:04.411225+0000","last_undegraded":"2024-09-16T03:40:04.411225+0000","last_fullsized":"2024-09-16T03:40:04.411225+0000","mapping_epoch":158,"log_start":"152'90","ondisk_log_start":"152'90","created":126,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":6,"last_scrub":"123'75","last_scrub_stamp":"2024-09-16T03:39:04.235141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.235141+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":90,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6151494,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":62325,"num_write":19,"num_write_kb":5191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,14],"acting":[4,8,15],"avail_no_missing":["4","8","15"],"object_location_counts":[{"shards":"4,8,15","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"172'112","reported_seq":595,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.722094+0000","last_change":"2024-09-16T03:39:03.917422+0000","last_active":"2024-09-16T03:40:03.722094+0000","last_peered":"2024-09-16T03:40:03.722094+0000","last_clean":"2024-09-16T03:40:03.722094+0000","last_became_active":"2024-09-16T03:39:01.801112+0000","last_became_peered":"2024-09-16T03:39:01.801112+0000","last_unstale":"2024-09-16T03:40:03.722094+0000","last_undegraded":"2024-09-16T03:40:03.722094+0000","last_fullsized":"2024-09-16T03:40:03.722094+0000","mapping_epoch":32,"log_start":"166'109","ondisk_log_start":"166'109","created":32,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'97","last_scrub_stamp":"2024-09-16T03:39:03.917355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.917355+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":106,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:06.444266+0000","scrub_duration":771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5496021,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":28034,"num_write":29,"num_write_kb":7222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,3],"acting":[10,14,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.e","version":"165'94","reported_seq":589,"reported_epoch":180,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:04.411548+0000","last_change":"2024-09-16T03:39:56.722563+0000","last_active":"2024-09-16T03:40:04.411548+0000","last_peered":"2024-09-16T03:40:04.411548+0000","last_clean":"2024-09-16T03:39:54.776007+0000","last_became_active":"2024-09-16T03:39:56.614366+0000","last_became_peered":"2024-09-16T03:39:56.614366+0000","last_unstale":"2024-09-16T03:40:04.411548+0000","last_undegraded":"2024-09-16T03:40:04.411548+0000","last_fullsized":"2024-09-16T03:40:04.411548+0000","mapping_epoch":171,"log_start":"154'91","ondisk_log_start":"154'91","created":32,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'93","last_scrub_stamp":"2024-09-16T03:39:35.280938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:35.280938+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":88,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3327852,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":17624,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2614941,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14,3],"acting":[4,14,8],"avail_no_missing":["4","8","14","15"],"object_location_counts":[{"shards":"4,8,14","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"165'27","reported_seq":487,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.657080+0000","last_change":"2024-09-16T03:39:33.569454+0000","last_active":"2024-09-16T03:40:03.657080+0000","last_peered":"2024-09-16T03:40:03.657080+0000","last_clean":"2024-09-16T03:40:03.657080+0000","last_became_active":"2024-09-16T03:39:33.569207+0000","last_became_peered":"2024-09-16T03:39:33.569207+0000","last_unstale":"2024-09-16T03:40:03.657080+0000","last_undegraded":"2024-09-16T03:40:03.657080+0000","last_fullsized":"2024-09-16T03:40:03.657080+0000","mapping_epoch":153,"log_start":"152'23","ondisk_log_start":"152'23","created":32,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'19","last_scrub_stamp":"2024-09-16T03:39:01.149486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.149486+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":23,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:09.211597+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3428694,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":33,"num_read_kb":17088,"num_write":9,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1327143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,1,3],"acting":[12,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.c","version":"167'69","reported_seq":547,"reported_epoch":179,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:03.399645+0000","last_change":"2024-09-16T03:40:01.256353+0000","last_active":"2024-09-16T03:40:03.399645+0000","last_peered":"2024-09-16T03:40:03.399645+0000","last_clean":"2024-09-16T03:39:27.002211+0000","last_became_active":"2024-09-16T03:39:31.053925+0000","last_became_peered":"2024-09-16T03:39:31.053925+0000","last_unstale":"2024-09-16T03:40:03.399645+0000","last_undegraded":"2024-09-16T03:40:03.399645+0000","last_fullsized":"2024-09-16T03:40:03.399645+0000","mapping_epoch":150,"log_start":"167'66","ondisk_log_start":"167'66","created":32,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'57","last_scrub_stamp":"2024-09-16T03:38:48.923184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.923184+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":64,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3512729,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":30539,"num_write":14,"num_write_kb":4279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1213217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,15,1],"acting":[12,15,9],"avail_no_missing":["12","9","15"],"object_location_counts":[{"shards":"9,12,15","objects":1}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.b","version":"153'135","reported_seq":698,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.596827+0000","last_change":"2024-09-16T03:40:01.105429+0000","last_active":"2024-09-16T03:40:01.596827+0000","last_peered":"2024-09-16T03:40:01.596827+0000","last_clean":"2024-09-16T03:40:01.596827+0000","last_became_active":"2024-09-16T03:39:09.748391+0000","last_became_peered":"2024-09-16T03:39:09.748391+0000","last_unstale":"2024-09-16T03:40:01.596827+0000","last_undegraded":"2024-09-16T03:40:01.596827+0000","last_fullsized":"2024-09-16T03:40:01.596827+0000","mapping_epoch":134,"log_start":"129'132","ondisk_log_start":"129'132","created":32,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'135","last_scrub_stamp":"2024-09-16T03:40:01.105343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:40:01.105343+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":128,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:25.231007+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":17762,"num_write":32,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,4,9],"acting":[15,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.a","version":"168'176","reported_seq":849,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.714220+0000","last_change":"2024-09-16T03:39:01.504751+0000","last_active":"2024-09-16T03:40:04.714220+0000","last_peered":"2024-09-16T03:40:04.714220+0000","last_clean":"2024-09-16T03:40:04.714220+0000","last_became_active":"2024-09-16T03:38:46.021805+0000","last_became_peered":"2024-09-16T03:38:46.021805+0000","last_unstale":"2024-09-16T03:40:04.714220+0000","last_undegraded":"2024-09-16T03:40:04.714220+0000","last_fullsized":"2024-09-16T03:40:04.714220+0000","mapping_epoch":32,"log_start":"168'173","ondisk_log_start":"168'173","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'147","last_scrub_stamp":"2024-09-16T03:39:01.504663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.504663+0000","objects_scrubbed":4,"log_size":3,"log_dups_size":170,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.277419+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13126531,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":161,"num_read_kb":166710,"num_write":34,"num_write_kb":9701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,10],"acting":[8,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":355,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405568+0000","last_change":"2024-09-16T03:40:00.748178+0000","last_active":"2024-09-16T03:40:03.405568+0000","last_peered":"2024-09-16T03:40:03.405568+0000","last_clean":"2024-09-16T03:40:03.405568+0000","last_became_active":"2024-09-16T03:39:30.229833+0000","last_became_peered":"2024-09-16T03:39:30.229833+0000","last_unstale":"2024-09-16T03:40:03.405568+0000","last_undegraded":"2024-09-16T03:40:03.405568+0000","last_fullsized":"2024-09-16T03:40:03.405568+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:00.748094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:40:00.748094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.254250+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,1,8],"acting":[14,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.8","version":"152'82","reported_seq":585,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.401782+0000","last_change":"2024-09-16T03:39:31.762814+0000","last_active":"2024-09-16T03:40:03.401782+0000","last_peered":"2024-09-16T03:40:03.401782+0000","last_clean":"2024-09-16T03:40:03.401782+0000","last_became_active":"2024-09-16T03:39:31.762544+0000","last_became_peered":"2024-09-16T03:39:31.762544+0000","last_unstale":"2024-09-16T03:40:03.401782+0000","last_undegraded":"2024-09-16T03:40:03.401782+0000","last_fullsized":"2024-09-16T03:40:03.401782+0000","mapping_epoch":151,"log_start":"140'78","ondisk_log_start":"140'78","created":32,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'71","last_scrub_stamp":"2024-09-16T03:39:02.002247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.002247+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":76,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.244585+0000","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3243817,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":3438,"num_write":14,"num_write_kb":3035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"171'100","reported_seq":566,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.402456+0000","last_change":"2024-09-16T03:39:58.682984+0000","last_active":"2024-09-16T03:40:03.402456+0000","last_peered":"2024-09-16T03:40:03.402456+0000","last_clean":"2024-09-16T03:40:03.402456+0000","last_became_active":"2024-09-16T03:39:58.682644+0000","last_became_peered":"2024-09-16T03:39:58.682644+0000","last_unstale":"2024-09-16T03:40:03.402456+0000","last_undegraded":"2024-09-16T03:40:03.402456+0000","last_fullsized":"2024-09-16T03:40:03.402456+0000","mapping_epoch":173,"log_start":"171'97","ondisk_log_start":"171'97","created":32,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'82","last_scrub_stamp":"2024-09-16T03:39:00.900619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.900619+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":95,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:19.165962+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3714413,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":40079,"num_write":22,"num_write_kb":6258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2417775,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,0],"acting":[7,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"30'112","reported_seq":509,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.495182+0000","last_change":"2024-09-16T03:38:36.757156+0000","last_active":"2024-09-16T03:40:01.495182+0000","last_peered":"2024-09-16T03:40:01.495182+0000","last_clean":"2024-09-16T03:40:01.495182+0000","last_became_active":"2024-09-16T03:38:36.756855+0000","last_became_peered":"2024-09-16T03:38:36.756855+0000","last_unstale":"2024-09-16T03:40:01.495182+0000","last_undegraded":"2024-09-16T03:40:01.495182+0000","last_fullsized":"2024-09-16T03:40:01.495182+0000","mapping_epoch":103,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:38:28.063690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:38:28.063690+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:30.945544+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":216,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.502092+0000","last_change":"2024-09-16T03:38:09.173787+0000","last_active":"2024-09-16T03:40:03.502092+0000","last_peered":"2024-09-16T03:40:03.502092+0000","last_clean":"2024-09-16T03:40:03.502092+0000","last_became_active":"2024-09-16T03:38:09.173252+0000","last_became_peered":"2024-09-16T03:38:09.173252+0000","last_unstale":"2024-09-16T03:40:03.502092+0000","last_undegraded":"2024-09-16T03:40:03.502092+0000","last_fullsized":"2024-09-16T03:40:03.502092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:38:01.233355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:01.233355+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.436956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":758,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.495470+0000","last_change":"2024-09-16T03:39:23.138722+0000","last_active":"2024-09-16T03:40:01.495470+0000","last_peered":"2024-09-16T03:40:01.495470+0000","last_clean":"2024-09-16T03:40:01.495470+0000","last_became_active":"2024-09-16T03:38:01.602925+0000","last_became_peered":"2024-09-16T03:38:01.602925+0000","last_unstale":"2024-09-16T03:40:01.495470+0000","last_undegraded":"2024-09-16T03:40:01.495470+0000","last_fullsized":"2024-09-16T03:40:01.495470+0000","mapping_epoch":74,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.823622+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":12398503,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,4],"acting":[11,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.6","version":"108'44","reported_seq":481,"reported_epoch":179,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:03.722271+0000","last_change":"2024-09-16T03:39:55.909742+0000","last_active":"2024-09-16T03:40:03.722271+0000","last_peered":"2024-09-16T03:40:03.722271+0000","last_clean":"2024-09-16T03:39:52.998142+0000","last_became_active":"2024-09-16T03:39:55.908933+0000","last_became_peered":"2024-09-16T03:39:55.908933+0000","last_unstale":"2024-09-16T03:40:03.722271+0000","last_undegraded":"2024-09-16T03:40:03.722271+0000","last_fullsized":"2024-09-16T03:40:03.722271+0000","mapping_epoch":170,"log_start":"105'41","ondisk_log_start":"105'41","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:48.279224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.279224+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":40,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,12],"acting":[10,6,4],"avail_no_missing":["10","4","6","12"],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":358,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.714303+0000","last_change":"2024-09-16T03:39:09.749531+0000","last_active":"2024-09-16T03:40:04.714303+0000","last_peered":"2024-09-16T03:40:04.714303+0000","last_clean":"2024-09-16T03:40:04.714303+0000","last_became_active":"2024-09-16T03:39:09.649028+0000","last_became_peered":"2024-09-16T03:39:09.649028+0000","last_unstale":"2024-09-16T03:40:04.714303+0000","last_undegraded":"2024-09-16T03:40:04.714303+0000","last_fullsized":"2024-09-16T03:40:04.714303+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:24.358339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:24.358339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:41.336202+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":609,"reported_epoch":177,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:01.731470+0000","last_change":"2024-09-16T03:39:55.213119+0000","last_active":"2024-09-16T03:40:01.731470+0000","last_peered":"2024-09-16T03:40:01.731470+0000","last_clean":"2024-09-16T03:39:49.975837+0000","last_became_active":"2024-09-16T03:39:54.908050+0000","last_became_peered":"2024-09-16T03:39:54.908050+0000","last_unstale":"2024-09-16T03:40:01.731470+0000","last_undegraded":"2024-09-16T03:40:01.731470+0000","last_fullsized":"2024-09-16T03:40:01.731470+0000","mapping_epoch":169,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'251","last_scrub_stamp":"2024-09-16T03:39:36.474408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:36.474408+0000","objects_scrubbed":37,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":78,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":234,"num_read_kb":2386,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,1],"acting":[6,3,4],"avail_no_missing":["6","3","4"],"object_location_counts":[{"shards":"3,4,6","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"168'90","reported_seq":555,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.822168+0000","last_change":"2024-09-16T03:39:01.297447+0000","last_active":"2024-09-16T03:40:03.822168+0000","last_peered":"2024-09-16T03:40:03.822168+0000","last_clean":"2024-09-16T03:40:03.822168+0000","last_became_active":"2024-09-16T03:38:45.723754+0000","last_became_peered":"2024-09-16T03:38:45.723754+0000","last_unstale":"2024-09-16T03:40:03.822168+0000","last_undegraded":"2024-09-16T03:40:03.822168+0000","last_fullsized":"2024-09-16T03:40:03.822168+0000","mapping_epoch":32,"log_start":"168'87","ondisk_log_start":"168'87","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'72","last_scrub_stamp":"2024-09-16T03:39:01.297324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.297324+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":85,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:26.128089+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7092686,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":98758,"num_write":19,"num_write_kb":5816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,6],"acting":[10,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.495353+0000","last_change":"2024-09-16T03:39:56.004042+0000","last_active":"2024-09-16T03:40:01.495353+0000","last_peered":"2024-09-16T03:40:01.495353+0000","last_clean":"2024-09-16T03:40:01.495353+0000","last_became_active":"2024-09-16T03:39:56.003773+0000","last_became_peered":"2024-09-16T03:39:56.003773+0000","last_unstale":"2024-09-16T03:40:01.495353+0000","last_undegraded":"2024-09-16T03:40:01.495353+0000","last_fullsized":"2024-09-16T03:40:01.495353+0000","mapping_epoch":170,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:27.167621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,7],"acting":[11,14,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":922,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.776035+0000","last_change":"2024-09-16T03:40:01.694672+0000","last_active":"2024-09-16T03:40:04.776035+0000","last_peered":"2024-09-16T03:40:04.776035+0000","last_clean":"2024-09-16T03:40:04.776035+0000","last_became_active":"2024-09-16T03:40:01.594201+0000","last_became_peered":"2024-09-16T03:40:01.594201+0000","last_unstale":"2024-09-16T03:40:04.776035+0000","last_undegraded":"2024-09-16T03:40:04.776035+0000","last_fullsized":"2024-09-16T03:40:04.776035+0000","mapping_epoch":176,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:43.795369+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":268,"num_read_kb":13655,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":57210536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,6],"acting":[9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.4","version":"137'33","reported_seq":421,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.595599+0000","last_change":"2024-09-16T03:38:47.042020+0000","last_active":"2024-09-16T03:40:01.595599+0000","last_peered":"2024-09-16T03:40:01.595599+0000","last_clean":"2024-09-16T03:40:01.595599+0000","last_became_active":"2024-09-16T03:38:45.820169+0000","last_became_peered":"2024-09-16T03:38:45.820169+0000","last_unstale":"2024-09-16T03:40:01.595599+0000","last_undegraded":"2024-09-16T03:40:01.595599+0000","last_fullsized":"2024-09-16T03:40:01.595599+0000","mapping_epoch":32,"log_start":"87'30","ondisk_log_start":"87'30","created":32,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.041851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.041851+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":30,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:55.806387+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,15],"acting":[11,2,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":58,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.821915+0000","last_change":"2024-09-16T03:39:57.677954+0000","last_active":"2024-09-16T03:40:03.821915+0000","last_peered":"2024-09-16T03:40:03.821915+0000","last_clean":"2024-09-16T03:40:03.821915+0000","last_became_active":"2024-09-16T03:39:57.677624+0000","last_became_peered":"2024-09-16T03:39:57.677624+0000","last_unstale":"2024-09-16T03:40:03.821915+0000","last_undegraded":"2024-09-16T03:40:03.821915+0000","last_fullsized":"2024-09-16T03:40:03.821915+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:39:14.082477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.082477+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:25.802773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":786,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.917748+0000","last_change":"2024-09-16T03:40:03.603663+0000","last_active":"2024-09-16T03:40:04.917748+0000","last_peered":"2024-09-16T03:40:04.917748+0000","last_clean":"2024-09-16T03:40:04.917748+0000","last_became_active":"2024-09-16T03:40:03.603137+0000","last_became_peered":"2024-09-16T03:40:03.603137+0000","last_unstale":"2024-09-16T03:40:04.917748+0000","last_undegraded":"2024-09-16T03:40:04.917748+0000","last_fullsized":"2024-09-16T03:40:04.917748+0000","mapping_epoch":178,"log_start":"168'338","ondisk_log_start":"168'338","created":31,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":350,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:04.109039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":326,"num_read_kb":8268,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":15647087,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[8,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.2","version":"45'6","reported_seq":376,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.696945+0000","last_change":"2024-09-16T03:39:58.680587+0000","last_active":"2024-09-16T03:40:01.696945+0000","last_peered":"2024-09-16T03:40:01.696945+0000","last_clean":"2024-09-16T03:40:01.696945+0000","last_became_active":"2024-09-16T03:39:58.680154+0000","last_became_peered":"2024-09-16T03:39:58.680154+0000","last_unstale":"2024-09-16T03:40:01.696945+0000","last_undegraded":"2024-09-16T03:40:01.696945+0000","last_fullsized":"2024-09-16T03:40:01.696945+0000","mapping_epoch":173,"log_start":"39'3","ondisk_log_start":"39'3","created":32,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:38:35.828203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.828203+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.560048+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,3],"acting":[15,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":334,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.394775+0000","last_change":"2024-09-16T03:39:29.848063+0000","last_active":"2024-09-16T03:40:01.394775+0000","last_peered":"2024-09-16T03:40:01.394775+0000","last_clean":"2024-09-16T03:40:01.394775+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:40:01.394775+0000","last_undegraded":"2024-09-16T03:40:01.394775+0000","last_fullsized":"2024-09-16T03:40:01.394775+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:36:52.069777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:33.922407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"94'34","reported_seq":438,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.595525+0000","last_change":"2024-09-16T03:39:58.085927+0000","last_active":"2024-09-16T03:40:01.595525+0000","last_peered":"2024-09-16T03:40:01.595525+0000","last_clean":"2024-09-16T03:40:01.595525+0000","last_became_active":"2024-09-16T03:38:33.580022+0000","last_became_peered":"2024-09-16T03:38:33.580022+0000","last_unstale":"2024-09-16T03:40:01.595525+0000","last_undegraded":"2024-09-16T03:40:01.595525+0000","last_fullsized":"2024-09-16T03:40:01.595525+0000","mapping_epoch":96,"log_start":"73'29","ondisk_log_start":"73'29","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'34","last_scrub_stamp":"2024-09-16T03:39:58.085847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:58.085847+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":28,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:17.907220+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,6],"acting":[11,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":102,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.876273+0000","last_change":"2024-09-16T03:39:09.848905+0000","last_active":"2024-09-16T03:40:04.876273+0000","last_peered":"2024-09-16T03:40:04.876273+0000","last_clean":"2024-09-16T03:40:04.876273+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:04.876273+0000","last_undegraded":"2024-09-16T03:40:04.876273+0000","last_fullsized":"2024-09-16T03:40:04.876273+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:52.781538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"167'156","reported_seq":728,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.721615+0000","last_change":"2024-09-16T03:39:59.774876+0000","last_active":"2024-09-16T03:40:03.721615+0000","last_peered":"2024-09-16T03:40:03.721615+0000","last_clean":"2024-09-16T03:40:03.721615+0000","last_became_active":"2024-09-16T03:39:59.774568+0000","last_became_peered":"2024-09-16T03:39:59.774568+0000","last_unstale":"2024-09-16T03:40:03.721615+0000","last_undegraded":"2024-09-16T03:40:03.721615+0000","last_fullsized":"2024-09-16T03:40:03.721615+0000","mapping_epoch":174,"log_start":"167'153","ondisk_log_start":"167'153","created":32,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'118","last_scrub_stamp":"2024-09-16T03:38:39.704919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:39.704919+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":150,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:05.113275+0000","scrub_duration":533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6958230,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":93961,"num_write":45,"num_write_kb":13895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5176846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,15,8],"acting":[10,15,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":346,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.406014+0000","last_change":"2024-09-16T03:39:39.004001+0000","last_active":"2024-09-16T03:40:03.406014+0000","last_peered":"2024-09-16T03:40:03.406014+0000","last_clean":"2024-09-16T03:40:03.406014+0000","last_became_active":"2024-09-16T03:37:55.560739+0000","last_became_peered":"2024-09-16T03:37:55.560739+0000","last_unstale":"2024-09-16T03:40:03.406014+0000","last_undegraded":"2024-09-16T03:40:03.406014+0000","last_fullsized":"2024-09-16T03:40:03.406014+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.017825+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,3,10],"acting":[14,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1","version":"48'7","reported_seq":359,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.410979+0000","last_change":"2024-09-16T03:39:59.297976+0000","last_active":"2024-09-16T03:40:04.410979+0000","last_peered":"2024-09-16T03:40:04.410979+0000","last_clean":"2024-09-16T03:40:04.410979+0000","last_became_active":"2024-09-16T03:38:33.484091+0000","last_became_peered":"2024-09-16T03:38:33.484091+0000","last_unstale":"2024-09-16T03:40:04.410979+0000","last_undegraded":"2024-09-16T03:40:04.410979+0000","last_fullsized":"2024-09-16T03:40:04.410979+0000","mapping_epoch":36,"log_start":"40'4","ondisk_log_start":"40'4","created":32,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:39:59.297888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:59.297888+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:25.801060+0000","scrub_duration":390,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":333,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.714444+0000","last_change":"2024-09-16T03:38:13.611310+0000","last_active":"2024-09-16T03:40:04.714444+0000","last_peered":"2024-09-16T03:40:04.714444+0000","last_clean":"2024-09-16T03:40:04.714444+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:04.714444+0000","last_undegraded":"2024-09-16T03:40:04.714444+0000","last_fullsized":"2024-09-16T03:40:04.714444+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:13.611212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:13.611212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:29.326990+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.10","version":"167'81","reported_seq":518,"reported_epoch":179,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:03.721581+0000","last_change":"2024-09-16T03:39:56.206774+0000","last_active":"2024-09-16T03:40:03.721581+0000","last_peered":"2024-09-16T03:40:03.721581+0000","last_clean":"2024-09-16T03:39:52.997913+0000","last_became_active":"2024-09-16T03:39:56.106140+0000","last_became_peered":"2024-09-16T03:39:56.106140+0000","last_unstale":"2024-09-16T03:40:03.721581+0000","last_undegraded":"2024-09-16T03:40:03.721581+0000","last_fullsized":"2024-09-16T03:40:03.721581+0000","mapping_epoch":170,"log_start":"165'78","ondisk_log_start":"165'78","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"118'64","last_scrub_stamp":"2024-09-16T03:39:04.442556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:04.442556+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":75,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3411887,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":20,"num_write_kb":5652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,11],"acting":[10,3,4],"avail_no_missing":["10","3","4"],"object_location_counts":[{"shards":"3,4,10","objects":1}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"167'26","reported_seq":423,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-16T03:40:05.635831+0000","last_change":"2024-09-16T03:39:03.929961+0000","last_active":"2024-09-16T03:40:05.635831+0000","last_peered":"2024-09-16T03:40:05.635831+0000","last_clean":"2024-09-16T03:40:05.635831+0000","last_became_active":"2024-09-16T03:39:01.696690+0000","last_became_peered":"2024-09-16T03:39:01.696690+0000","last_unstale":"2024-09-16T03:40:05.635831+0000","last_undegraded":"2024-09-16T03:40:05.635831+0000","last_fullsized":"2024-09-16T03:40:05.635831+0000","mapping_epoch":63,"log_start":"155'22","ondisk_log_start":"155'22","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"114'17","last_scrub_stamp":"2024-09-16T03:39:03.929863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.929863+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":22,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:13.702587+0000","scrub_duration":285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3579957,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":37538,"num_write":10,"num_write_kb":3021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,12],"acting":[2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"147'23","reported_seq":502,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405868+0000","last_change":"2024-09-16T03:39:14.925442+0000","last_active":"2024-09-16T03:40:03.405868+0000","last_peered":"2024-09-16T03:40:03.405868+0000","last_clean":"2024-09-16T03:40:03.405868+0000","last_became_active":"2024-09-16T03:39:13.974170+0000","last_became_peered":"2024-09-16T03:39:13.974170+0000","last_unstale":"2024-09-16T03:40:03.405868+0000","last_undegraded":"2024-09-16T03:40:03.405868+0000","last_fullsized":"2024-09-16T03:40:03.405868+0000","mapping_epoch":138,"log_start":"127'20","ondisk_log_start":"127'20","created":50,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'20","last_scrub_stamp":"2024-09-16T03:39:14.925356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.925356+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":20,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:22.341117+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2000099,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":5461,"num_write":1,"num_write_kb":458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,9,2],"acting":[14,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.13","version":"165'83","reported_seq":583,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.511263+0000","last_change":"2024-09-16T03:39:15.224056+0000","last_active":"2024-09-16T03:40:04.511263+0000","last_peered":"2024-09-16T03:40:04.511263+0000","last_clean":"2024-09-16T03:40:04.511263+0000","last_became_active":"2024-09-16T03:39:15.223614+0000","last_became_peered":"2024-09-16T03:39:15.223614+0000","last_unstale":"2024-09-16T03:40:04.511263+0000","last_undegraded":"2024-09-16T03:40:04.511263+0000","last_fullsized":"2024-09-16T03:40:04.511263+0000","mapping_epoch":139,"log_start":"154'79","ondisk_log_start":"154'79","created":50,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"127'73","last_scrub_stamp":"2024-09-16T03:39:03.124583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.124583+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":77,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:07.132930+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3445501,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":11846,"num_write":10,"num_write_kb":2874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9],"acting":[4,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"104'20","reported_seq":417,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-09-16T03:40:05.535903+0000","last_change":"2024-09-16T03:39:01.898409+0000","last_active":"2024-09-16T03:40:05.535903+0000","last_peered":"2024-09-16T03:40:05.535903+0000","last_clean":"2024-09-16T03:40:05.535903+0000","last_became_active":"2024-09-16T03:39:01.596287+0000","last_became_peered":"2024-09-16T03:39:01.596287+0000","last_unstale":"2024-09-16T03:40:05.535903+0000","last_undegraded":"2024-09-16T03:40:05.535903+0000","last_fullsized":"2024-09-16T03:40:05.535903+0000","mapping_epoch":114,"log_start":"98'17","ondisk_log_start":"98'17","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"104'20","last_scrub_stamp":"2024-09-16T03:39:01.898331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.898331+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":14,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:15.728364+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,8],"acting":[2,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"123'65","reported_seq":569,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.513804+0000","last_change":"2024-09-16T03:39:14.175562+0000","last_active":"2024-09-16T03:40:04.513804+0000","last_peered":"2024-09-16T03:40:04.513804+0000","last_clean":"2024-09-16T03:40:04.513804+0000","last_became_active":"2024-09-16T03:39:13.671769+0000","last_became_peered":"2024-09-16T03:39:13.671769+0000","last_unstale":"2024-09-16T03:40:04.513804+0000","last_undegraded":"2024-09-16T03:40:04.513804+0000","last_fullsized":"2024-09-16T03:40:04.513804+0000","mapping_epoch":138,"log_start":"123'62","ondisk_log_start":"123'62","created":50,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":5,"last_scrub":"123'65","last_scrub_stamp":"2024-09-16T03:39:14.175493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:14.175493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":60,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:36.855683+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"124'86","reported_seq":539,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.597028+0000","last_change":"2024-09-16T03:39:03.073530+0000","last_active":"2024-09-16T03:40:01.597028+0000","last_peered":"2024-09-16T03:40:01.597028+0000","last_clean":"2024-09-16T03:40:01.597028+0000","last_became_active":"2024-09-16T03:39:01.797805+0000","last_became_peered":"2024-09-16T03:39:01.797805+0000","last_unstale":"2024-09-16T03:40:01.597028+0000","last_undegraded":"2024-09-16T03:40:01.597028+0000","last_fullsized":"2024-09-16T03:40:01.597028+0000","mapping_epoch":115,"log_start":"124'82","ondisk_log_start":"124'82","created":50,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'86","last_scrub_stamp":"2024-09-16T03:39:03.073459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:03.073459+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":80,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:30.483816+0000","scrub_duration":438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2,10],"acting":[15,2,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.17","version":"169'60","reported_seq":514,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.406094+0000","last_change":"2024-09-16T03:39:35.812368+0000","last_active":"2024-09-16T03:40:03.406094+0000","last_peered":"2024-09-16T03:40:03.406094+0000","last_clean":"2024-09-16T03:40:03.406094+0000","last_became_active":"2024-09-16T03:39:03.717420+0000","last_became_peered":"2024-09-16T03:39:03.717420+0000","last_unstale":"2024-09-16T03:40:03.406094+0000","last_undegraded":"2024-09-16T03:40:03.406094+0000","last_fullsized":"2024-09-16T03:40:03.406094+0000","mapping_epoch":128,"log_start":"148'56","ondisk_log_start":"148'56","created":50,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":5,"last_scrub":"148'56","last_scrub_stamp":"2024-09-16T03:39:35.812298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:35.812298+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":55,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:54.749022+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3731220,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,0,6],"acting":[14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.18","version":"155'55","reported_seq":474,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.714255+0000","last_change":"2024-09-16T03:39:02.668794+0000","last_active":"2024-09-16T03:40:04.714255+0000","last_peered":"2024-09-16T03:40:04.714255+0000","last_clean":"2024-09-16T03:40:04.714255+0000","last_became_active":"2024-09-16T03:38:47.562641+0000","last_became_peered":"2024-09-16T03:38:47.562641+0000","last_unstale":"2024-09-16T03:40:04.714255+0000","last_undegraded":"2024-09-16T03:40:04.714255+0000","last_fullsized":"2024-09-16T03:40:04.714255+0000","mapping_epoch":112,"log_start":"155'52","ondisk_log_start":"155'52","created":50,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":5,"last_scrub":"124'48","last_scrub_stamp":"2024-09-16T03:39:02.668681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:02.668681+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":49,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:14.665737+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3344659,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":25041,"num_write":28,"num_write_kb":7736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":103,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.775890+0000","last_change":"2024-09-16T03:39:09.847101+0000","last_active":"2024-09-16T03:40:04.775890+0000","last_peered":"2024-09-16T03:40:04.775890+0000","last_clean":"2024-09-16T03:40:04.775890+0000","last_became_active":"2024-09-16T03:39:09.846691+0000","last_became_peered":"2024-09-16T03:39:09.846691+0000","last_unstale":"2024-09-16T03:40:04.775890+0000","last_undegraded":"2024-09-16T03:40:04.775890+0000","last_fullsized":"2024-09-16T03:40:04.775890+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:01.106840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:01.106840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:27.531290+0000","scrub_duration":502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"169'138","reported_seq":682,"reported_epoch":177,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:01.394698+0000","last_change":"2024-09-16T03:39:54.206127+0000","last_active":"2024-09-16T03:40:01.394698+0000","last_peered":"2024-09-16T03:40:01.394698+0000","last_clean":"2024-09-16T03:39:28.503345+0000","last_became_active":"2024-09-16T03:39:54.099638+0000","last_became_peered":"2024-09-16T03:39:54.099638+0000","last_unstale":"2024-09-16T03:40:01.394698+0000","last_undegraded":"2024-09-16T03:40:01.394698+0000","last_fullsized":"2024-09-16T03:40:01.394698+0000","mapping_epoch":168,"log_start":"148'134","ondisk_log_start":"148'134","created":100,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"101'115","last_scrub_stamp":"2024-09-16T03:38:35.254488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:35.254488+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":134,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6756546,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":51953,"num_write":24,"num_write_kb":6981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":3860936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,14],"acting":[6,11,15],"avail_no_missing":["6","11","15"],"object_location_counts":[{"shards":"6,11,15","objects":2}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"147'136","reported_seq":740,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.413680+0000","last_change":"2024-09-16T03:40:00.893177+0000","last_active":"2024-09-16T03:40:04.413680+0000","last_peered":"2024-09-16T03:40:04.413680+0000","last_clean":"2024-09-16T03:40:04.413680+0000","last_became_active":"2024-09-16T03:39:58.580803+0000","last_became_peered":"2024-09-16T03:39:58.580803+0000","last_unstale":"2024-09-16T03:40:04.413680+0000","last_undegraded":"2024-09-16T03:40:04.413680+0000","last_fullsized":"2024-09-16T03:40:04.413680+0000","mapping_epoch":173,"log_start":"139'133","ondisk_log_start":"139'133","created":100,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'136","last_scrub_stamp":"2024-09-16T03:40:00.893096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:40:00.893096+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":127,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:29.744583+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4941390,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":116543,"num_write":32,"num_write_kb":9493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":5788673,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,10],"acting":[3,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"99'56","reported_seq":481,"reported_epoch":177,"state":"active+clean","last_fresh":"2024-09-16T03:40:01.394693+0000","last_change":"2024-09-16T03:39:33.570828+0000","last_active":"2024-09-16T03:40:01.394693+0000","last_peered":"2024-09-16T03:40:01.394693+0000","last_clean":"2024-09-16T03:40:01.394693+0000","last_became_active":"2024-09-16T03:39:33.570516+0000","last_became_peered":"2024-09-16T03:39:33.570516+0000","last_unstale":"2024-09-16T03:40:01.394693+0000","last_undegraded":"2024-09-16T03:40:01.394693+0000","last_fullsized":"2024-09-16T03:40:01.394693+0000","mapping_epoch":153,"log_start":"90'53","ondisk_log_start":"90'53","created":100,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":5,"last_scrub":"99'56","last_scrub_stamp":"2024-09-16T03:39:00.988517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.988517+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":53,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:08.293621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"92'14","reported_seq":372,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.513806+0000","last_change":"2024-09-16T03:38:49.092747+0000","last_active":"2024-09-16T03:40:04.513806+0000","last_peered":"2024-09-16T03:40:04.513806+0000","last_clean":"2024-09-16T03:40:04.513806+0000","last_became_active":"2024-09-16T03:38:48.661213+0000","last_became_peered":"2024-09-16T03:38:48.661213+0000","last_unstale":"2024-09-16T03:40:04.513806+0000","last_undegraded":"2024-09-16T03:40:04.513806+0000","last_fullsized":"2024-09-16T03:40:04.513806+0000","mapping_epoch":113,"log_start":"91'11","ondisk_log_start":"91'11","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:49.092687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.092687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:14.647233+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,12],"acting":[3,11,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"92'59","reported_seq":487,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405948+0000","last_change":"2024-09-16T03:39:48.664382+0000","last_active":"2024-09-16T03:40:03.405948+0000","last_peered":"2024-09-16T03:40:03.405948+0000","last_clean":"2024-09-16T03:40:03.405948+0000","last_became_active":"2024-09-16T03:38:48.561170+0000","last_became_peered":"2024-09-16T03:38:48.561170+0000","last_unstale":"2024-09-16T03:40:03.405948+0000","last_undegraded":"2024-09-16T03:40:03.405948+0000","last_fullsized":"2024-09-16T03:40:03.405948+0000","mapping_epoch":113,"log_start":"89'56","ondisk_log_start":"89'56","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":5,"last_scrub":"92'59","last_scrub_stamp":"2024-09-16T03:39:48.664277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:48.664277+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":56,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:01.699198+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,4,6],"acting":[14,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.1f","version":"169'132","reported_seq":699,"reported_epoch":179,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:40:03.921588+0000","last_change":"2024-09-16T03:40:03.720048+0000","last_active":"2024-09-16T03:40:03.921588+0000","last_peered":"2024-09-16T03:40:03.921588+0000","last_clean":"2024-09-16T03:40:03.921588+0000","last_became_active":"2024-09-16T03:39:56.619304+0000","last_became_peered":"2024-09-16T03:39:56.619304+0000","last_unstale":"2024-09-16T03:40:03.921588+0000","last_undegraded":"2024-09-16T03:40:03.921588+0000","last_fullsized":"2024-09-16T03:40:03.921588+0000","mapping_epoch":171,"log_start":"165'128","ondisk_log_start":"165'128","created":100,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'95","last_scrub_stamp":"2024-09-16T03:38:37.201959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:37.201959+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":128,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10386891,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":131,"num_read_kb":122162,"num_write":49,"num_write_kb":13103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":13164419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,12],"acting":[10,1,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.20","version":"170'165","reported_seq":779,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.714188+0000","last_change":"2024-09-16T03:39:49.993969+0000","last_active":"2024-09-16T03:40:04.714188+0000","last_peered":"2024-09-16T03:40:04.714188+0000","last_clean":"2024-09-16T03:40:04.714188+0000","last_became_active":"2024-09-16T03:39:49.893618+0000","last_became_peered":"2024-09-16T03:39:49.893618+0000","last_unstale":"2024-09-16T03:40:04.714188+0000","last_undegraded":"2024-09-16T03:40:04.714188+0000","last_fullsized":"2024-09-16T03:40:04.714188+0000","mapping_epoch":165,"log_start":"169'162","ondisk_log_start":"169'162","created":100,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'118","last_scrub_stamp":"2024-09-16T03:38:36.698150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:36.698150+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":162,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:55.225728+0000","scrub_duration":645,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11378504,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":184,"num_read_kb":187520,"num_write":53,"num_write_kb":16191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6817236,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,0],"acting":[8,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.21","version":"48'7","reported_seq":232,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405379+0000","last_change":"2024-09-16T03:38:49.064947+0000","last_active":"2024-09-16T03:40:03.405379+0000","last_peered":"2024-09-16T03:40:03.405379+0000","last_clean":"2024-09-16T03:40:03.405379+0000","last_became_active":"2024-09-16T03:38:48.559431+0000","last_became_peered":"2024-09-16T03:38:48.559431+0000","last_unstale":"2024-09-16T03:40:03.405379+0000","last_undegraded":"2024-09-16T03:40:03.405379+0000","last_fullsized":"2024-09-16T03:40:03.405379+0000","mapping_epoch":113,"log_start":"40'4","ondisk_log_start":"40'4","created":100,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":6,"last_scrub":"48'7","last_scrub_stamp":"2024-09-16T03:38:49.064870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064870+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":4,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,10],"acting":[14,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.22","version":"45'6","reported_seq":237,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.776000+0000","last_change":"2024-09-16T03:39:13.974174+0000","last_active":"2024-09-16T03:40:04.776000+0000","last_peered":"2024-09-16T03:40:04.776000+0000","last_clean":"2024-09-16T03:40:04.776000+0000","last_became_active":"2024-09-16T03:39:13.973813+0000","last_became_peered":"2024-09-16T03:39:13.973813+0000","last_unstale":"2024-09-16T03:40:04.776000+0000","last_undegraded":"2024-09-16T03:40:04.776000+0000","last_fullsized":"2024-09-16T03:40:04.776000+0000","mapping_epoch":138,"log_start":"39'3","ondisk_log_start":"39'3","created":100,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":6,"last_scrub":"45'6","last_scrub_stamp":"2024-09-16T03:39:00.968448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.968448+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:30.310727+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,6],"acting":[9,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.23","version":"143'42","reported_seq":468,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405726+0000","last_change":"2024-09-16T03:40:00.075385+0000","last_active":"2024-09-16T03:40:03.405726+0000","last_peered":"2024-09-16T03:40:03.405726+0000","last_clean":"2024-09-16T03:40:03.405726+0000","last_became_active":"2024-09-16T03:38:49.463797+0000","last_became_peered":"2024-09-16T03:38:49.463797+0000","last_unstale":"2024-09-16T03:40:03.405726+0000","last_undegraded":"2024-09-16T03:40:03.405726+0000","last_fullsized":"2024-09-16T03:40:03.405726+0000","mapping_epoch":114,"log_start":"142'39","ondisk_log_start":"142'39","created":100,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":6,"last_scrub":"143'42","last_scrub_stamp":"2024-09-16T03:40:00.075276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:40:00.075276+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":38,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:09.661629+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2672379,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":24141,"num_write":9,"num_write_kb":2300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,8,2],"acting":[14,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"4.24","version":"105'32","reported_seq":264,"reported_epoch":176,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:00.702345+0000","last_change":"2024-09-16T03:39:39.285524+0000","last_active":"2024-09-16T03:40:00.702345+0000","last_peered":"2024-09-16T03:40:00.702345+0000","last_clean":"2024-09-16T03:38:17.396491+0000","last_became_active":"2024-09-16T03:39:38.786889+0000","last_became_peered":"2024-09-16T03:39:38.786889+0000","last_unstale":"2024-09-16T03:40:00.702345+0000","last_undegraded":"2024-09-16T03:40:00.702345+0000","last_fullsized":"2024-09-16T03:40:00.702345+0000","mapping_epoch":158,"log_start":"83'29","ondisk_log_start":"83'29","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'32","last_scrub_stamp":"2024-09-16T03:38:47.978888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.978888+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":29,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":4728,"num_write":10,"num_write_kb":2338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,14],"acting":[2,11,15],"avail_no_missing":["2","0","3","11","14","15"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.25","version":"169'86","reported_seq":474,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.399723+0000","last_change":"2024-09-16T03:39:40.798757+0000","last_active":"2024-09-16T03:40:03.399723+0000","last_peered":"2024-09-16T03:40:03.399723+0000","last_clean":"2024-09-16T03:40:03.399723+0000","last_became_active":"2024-09-16T03:39:40.798223+0000","last_became_peered":"2024-09-16T03:39:40.798223+0000","last_unstale":"2024-09-16T03:40:03.399723+0000","last_undegraded":"2024-09-16T03:40:03.399723+0000","last_fullsized":"2024-09-16T03:40:03.399723+0000","mapping_epoch":160,"log_start":"130'83","ondisk_log_start":"130'83","created":110,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'77","last_scrub_stamp":"2024-09-16T03:38:48.088768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.088768+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":80,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:14.812409+0000","scrub_duration":388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2134555,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":10,"num_write_kb":2500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1126280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,7,14],"acting":[12,7,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"4.26","version":"169'64","reported_seq":478,"reported_epoch":177,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:01.394791+0000","last_change":"2024-09-16T03:39:38.986370+0000","last_active":"2024-09-16T03:40:01.394791+0000","last_peered":"2024-09-16T03:40:01.394791+0000","last_clean":"2024-09-16T03:39:19.447260+0000","last_became_active":"2024-09-16T03:39:38.985600+0000","last_became_peered":"2024-09-16T03:39:38.985600+0000","last_unstale":"2024-09-16T03:40:01.394791+0000","last_undegraded":"2024-09-16T03:40:01.394791+0000","last_fullsized":"2024-09-16T03:40:01.394791+0000","mapping_epoch":158,"log_start":"142'60","ondisk_log_start":"142'60","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'44","last_scrub_stamp":"2024-09-16T03:38:49.477297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.477297+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":59,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3647185,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":24205,"num_write":24,"num_write_kb":7094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,10],"acting":[6,10,4],"avail_no_missing":["6","4","10"],"object_location_counts":[{"shards":"4,6,10","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.27","version":"109'66","reported_seq":373,"reported_epoch":177,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:01.494833+0000","last_change":"2024-09-16T03:39:38.989488+0000","last_active":"2024-09-16T03:40:01.494833+0000","last_peered":"2024-09-16T03:40:01.494833+0000","last_clean":"2024-09-16T03:38:43.524716+0000","last_became_active":"2024-09-16T03:39:38.888633+0000","last_became_peered":"2024-09-16T03:39:38.888633+0000","last_unstale":"2024-09-16T03:40:01.494833+0000","last_undegraded":"2024-09-16T03:40:01.494833+0000","last_fullsized":"2024-09-16T03:40:01.494833+0000","mapping_epoch":158,"log_start":"105'63","ondisk_log_start":"105'63","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'66","last_scrub_stamp":"2024-09-16T03:38:47.511252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.511252+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":61,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2512680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":54642,"num_write":20,"num_write_kb":5643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[6,10,11],"avail_no_missing":["6","10","11"],"object_location_counts":[{"shards":"6,10,11","objects":1}],"blocked_by":[],"up_primary":8,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.28","version":"171'70","reported_seq":651,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.775953+0000","last_change":"2024-09-16T03:39:41.471715+0000","last_active":"2024-09-16T03:40:04.775953+0000","last_peered":"2024-09-16T03:40:04.775953+0000","last_clean":"2024-09-16T03:40:04.775953+0000","last_became_active":"2024-09-16T03:39:41.471298+0000","last_became_peered":"2024-09-16T03:39:41.471298+0000","last_unstale":"2024-09-16T03:40:04.775953+0000","last_undegraded":"2024-09-16T03:40:04.775953+0000","last_fullsized":"2024-09-16T03:40:04.775953+0000","mapping_epoch":161,"log_start":"118'67","ondisk_log_start":"118'67","created":110,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":6,"last_scrub":"105'63","last_scrub_stamp":"2024-09-16T03:38:48.551929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:48.551929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":66,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:12.543779+0000","scrub_duration":1204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7395113,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":70,"num_read_kb":76288,"num_write":5,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":14443776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,2],"acting":[9,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":54,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.721646+0000","last_change":"2024-09-16T03:39:37.881274+0000","last_active":"2024-09-16T03:40:03.721646+0000","last_peered":"2024-09-16T03:40:03.721646+0000","last_clean":"2024-09-16T03:40:03.721646+0000","last_became_active":"2024-09-16T03:39:37.880874+0000","last_became_peered":"2024-09-16T03:39:37.880874+0000","last_unstale":"2024-09-16T03:40:03.721646+0000","last_undegraded":"2024-09-16T03:40:03.721646+0000","last_fullsized":"2024-09-16T03:40:03.721646+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":110,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:00.834742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:39:00.834742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:21.681138+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,14,2],"acting":[10,14,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2a","version":"108'122","reported_seq":575,"reported_epoch":180,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:04.413143+0000","last_change":"2024-09-16T03:39:39.190319+0000","last_active":"2024-09-16T03:40:04.413143+0000","last_peered":"2024-09-16T03:40:04.413143+0000","last_clean":"2024-09-16T03:38:41.455596+0000","last_became_active":"2024-09-16T03:39:38.888612+0000","last_became_peered":"2024-09-16T03:39:38.888612+0000","last_unstale":"2024-09-16T03:40:04.413143+0000","last_undegraded":"2024-09-16T03:40:04.413143+0000","last_fullsized":"2024-09-16T03:40:04.413143+0000","mapping_epoch":158,"log_start":"107'115","ondisk_log_start":"107'115","created":110,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":6,"last_scrub":"108'122","last_scrub_stamp":"2024-09-16T03:38:47.964439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.964439+0000","objects_scrubbed":4,"log_size":7,"log_dups_size":112,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5647218,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":6805,"num_write":7,"num_write_kb":1927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,12],"acting":[3,8,10],"avail_no_missing":["3","8","10"],"object_location_counts":[{"shards":"3,8,10","objects":2}],"blocked_by":[],"up_primary":9,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2b","version":"149'132","reported_seq":695,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.411142+0000","last_change":"2024-09-16T03:39:41.669420+0000","last_active":"2024-09-16T03:40:04.411142+0000","last_peered":"2024-09-16T03:40:04.411142+0000","last_clean":"2024-09-16T03:40:04.411142+0000","last_became_active":"2024-09-16T03:39:41.669036+0000","last_became_peered":"2024-09-16T03:39:41.669036+0000","last_unstale":"2024-09-16T03:40:04.411142+0000","last_undegraded":"2024-09-16T03:40:04.411142+0000","last_fullsized":"2024-09-16T03:40:04.411142+0000","mapping_epoch":161,"log_start":"144'128","ondisk_log_start":"144'128","created":110,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":6,"last_scrub":"109'119","last_scrub_stamp":"2024-09-16T03:38:49.064061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:49.064061+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":128,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:55.831512+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3092554,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":5914,"num_write":15,"num_write_kb":4670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1630666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2c","version":"167'69","reported_seq":566,"reported_epoch":180,"state":"active+clean","last_fresh":"2024-09-16T03:40:04.513289+0000","last_change":"2024-09-16T03:39:41.471606+0000","last_active":"2024-09-16T03:40:04.513289+0000","last_peered":"2024-09-16T03:40:04.513289+0000","last_clean":"2024-09-16T03:40:04.513289+0000","last_became_active":"2024-09-16T03:39:41.471156+0000","last_became_peered":"2024-09-16T03:39:41.471156+0000","last_unstale":"2024-09-16T03:40:04.513289+0000","last_undegraded":"2024-09-16T03:40:04.513289+0000","last_fullsized":"2024-09-16T03:40:04.513289+0000","mapping_epoch":161,"log_start":"154'65","ondisk_log_start":"154'65","created":110,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":6,"last_scrub":"112'58","last_scrub_stamp":"2024-09-16T03:38:47.889276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.889276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":64,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:16.605878+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3495922,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":43795,"num_write":10,"num_write_kb":2812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":3787209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,2],"acting":[3,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2d","version":"92'14","reported_seq":433,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-16T03:40:03.405770+0000","last_change":"2024-09-16T03:39:40.897844+0000","last_active":"2024-09-16T03:40:03.405770+0000","last_peered":"2024-09-16T03:40:03.405770+0000","last_clean":"2024-09-16T03:40:03.405770+0000","last_became_active":"2024-09-16T03:39:40.897531+0000","last_became_peered":"2024-09-16T03:39:40.897531+0000","last_unstale":"2024-09-16T03:40:03.405770+0000","last_undegraded":"2024-09-16T03:40:03.405770+0000","last_fullsized":"2024-09-16T03:40:03.405770+0000","mapping_epoch":160,"log_start":"91'11","ondisk_log_start":"91'11","created":110,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":6,"last_scrub":"92'14","last_scrub_stamp":"2024-09-16T03:38:47.115651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:04.919920+0000","last_clean_scrub_stamp":"2024-09-16T03:38:47.115651+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":11,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:56.558775+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,4],"acting":[14,11,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":56,"stat_sum":{"num_bytes":165812611,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1731,"num_read_kb":1589036,"num_write":632,"num_write_kb":176819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":68548351,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":24,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":289665024,"data_stored":288999499,"data_compressed":627765,"data_compressed_allocated":688128,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":168,"acting":168,"num_store_stats":16},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":78,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1165,"num_read_kb":34194,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":85256126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214495232,"data_stored":229041714,"data_compressed":14285886,"data_compressed_allocated":15241216,"data_compressed_original":30437376,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1057856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":2644640,"data_compressed":159066,"data_compressed_allocated":204800,"data_compressed_original":2584576,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247848,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75748,"kb_used_data":40400,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94296092,"statfs":{"total":96636764160,"available":96559198208,"internally_reserved":0,"allocated":41369600,"data_stored":41693615,"data_compressed":1360715,"data_compressed_allocated":1470464,"data_compressed_original":2588672,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247849,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62268,"kb_used_data":31080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94309572,"statfs":{"total":96636764160,"available":96573001728,"internally_reserved":0,"allocated":31825920,"data_stored":32921741,"data_compressed":1089754,"data_compressed_allocated":1212416,"data_compressed_original":3080192,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,7,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247848,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88380,"kb_used_data":53024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94283460,"statfs":{"total":96636764160,"available":96546263040,"internally_reserved":0,"allocated":54296576,"data_stored":55900546,"data_compressed":2021943,"data_compressed_allocated":2199552,"data_compressed_original":4628480,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280554,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78592,"kb_used_data":47396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94293248,"statfs":{"total":96636764160,"available":96556285952,"internally_reserved":0,"allocated":48533504,"data_stored":50484788,"data_compressed":2303619,"data_compressed_allocated":2494464,"data_compressed_original":5292032,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":0,"seq":94489280554,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280553,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56084,"kb_used_data":24888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94315756,"statfs":{"total":96636764160,"available":96579334144,"internally_reserved":0,"allocated":25485312,"data_stored":25872981,"data_compressed":969719,"data_compressed_allocated":1056768,"data_compressed_original":2154496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313258,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41616,"kb_used_data":14580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330224,"statfs":{"total":96636764160,"available":96594149376,"internally_reserved":0,"allocated":14929920,"data_stored":14505443,"data_compressed":218938,"data_compressed_allocated":270336,"data_compressed_original":540672,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280554,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83176,"kb_used_data":47828,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94288664,"statfs":{"total":96636764160,"available":96551591936,"internally_reserved":0,"allocated":48975872,"data_stored":49667230,"data_compressed":1266255,"data_compressed_allocated":1409024,"data_compressed_original":2973696,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280553,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88652,"kb_used_data":53296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94283188,"statfs":{"total":96636764160,"available":96545984512,"internally_reserved":0,"allocated":54575104,"data_stored":56406632,"data_compressed":2127618,"data_compressed_allocated":2334720,"data_compressed_original":5017600,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":90194313258,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280553,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39856,"kb_used_data":8664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331984,"statfs":{"total":96636764160,"available":96595951616,"internally_reserved":0,"allocated":8871936,"data_stored":8358851,"data_compressed":95409,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280553,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84632,"kb_used_data":49276,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94287208,"statfs":{"total":96636764160,"available":96550100992,"internally_reserved":0,"allocated":50458624,"data_stored":50794032,"data_compressed":1326174,"data_compressed_allocated":1437696,"data_compressed_original":2605056,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280553,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61132,"kb_used_data":29940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94310708,"statfs":{"total":96636764160,"available":96574164992,"internally_reserved":0,"allocated":30658560,"data_stored":31525432,"data_compressed":1015354,"data_compressed_allocated":1114112,"data_compressed_original":2744320,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280554,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79564,"kb_used_data":48368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94292276,"statfs":{"total":96636764160,"available":96555290624,"internally_reserved":0,"allocated":49528832,"data_stored":49021549,"data_compressed":261093,"data_compressed_allocated":315392,"data_compressed_original":663552,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,2,3,4,6,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313257,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46660,"kb_used_data":15460,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325180,"statfs":{"total":96636764160,"available":96588984320,"internally_reserved":0,"allocated":15831040,"data_stored":15256892,"data_compressed":121130,"data_compressed_allocated":163840,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247850,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70800,"kb_used_data":39604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94301040,"statfs":{"total":96636764160,"available":96564264960,"internally_reserved":0,"allocated":40554496,"data_stored":40980662,"data_compressed":1405275,"data_compressed_allocated":1523712,"data_compressed_original":2744320,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31858688,"data_stored":33943544,"data_compressed":2174030,"data_compressed_allocated":2314240,"data_compressed_original":4501504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11505664,"data_stored":12398503,"data_compressed":787965,"data_compressed_allocated":843776,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14860288,"data_stored":14775225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9277440,"data_stored":9253748,"data_compressed":22789,"data_compressed_allocated":24576,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24104960,"data_stored":24092098,"data_compressed":85710,"data_compressed_allocated":94208,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30834688,"data_stored":30754602,"data_compressed":94382,"data_compressed_allocated":102400,"data_compressed_original":196608,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":19304448,"data_stored":19197855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21692416,"data_stored":21628396,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7184384,"data_stored":7135967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":34025472,"data_stored":33912667,"data_compressed":9548,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14385152,"data_stored":14378215,"data_compressed":35763,"data_compressed_allocated":40960,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":47800320,"data_stored":47794347,"data_compressed":165684,"data_compressed_allocated":180224,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12890112,"data_stored":12877307,"data_compressed":10278,"data_compressed_allocated":12288,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":14069760,"data_stored":14019073,"data_compressed":22789,"data_compressed_allocated":24576,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":1080040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":13201408,"data_stored":13270816,"data_compressed":120597,"data_compressed_allocated":131072,"data_compressed_original":262144,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":24944640,"data_stored":24829143,"data_compressed":49947,"data_compressed_allocated":53248,"data_compressed_original":65536,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:07.261 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:40:07.259+0000 7efefacd8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33271) UID: 0 2024-09-16T03:40:07.269 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:07.270 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:07.279 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T03:40:07.362 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:40:07.360+0000 7f9293769640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33446) UID: 0 2024-09-16T03:40:07.383 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:07.384 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:07.392 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T03:40:07.437 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 4 pg_num to 66 2024-09-16T03:40:07.451 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-16T03:40:07.451 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-16T03:40:07.476 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:40:07.474+0000 7fa393ce9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33444) UID: 0 2024-09-16T03:40:07.535 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:07.535 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:07.544 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T03:40:07.565 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:40:07.562+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:40:07.665 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:40:07.664+0000 7f6538473640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33264) UID: 0 2024-09-16T03:40:07.689 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:07.689 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:07.698 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T03:40:07.766 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:40:07.765+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:40:07.803 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:07.803 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:07.812 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T03:40:07.868 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:40:07.866+0000 7f66fe808640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33221) UID: 0 2024-09-16T03:40:07.915 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:07.915 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:07.924 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T03:40:07.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T03:40:07.972 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:40:07.971+0000 7f422230a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33439) UID: 0 2024-09-16T03:40:08.027 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:08.027 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:08.035 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T03:40:08.053 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0: {} 2024-09-16T03:40:08.054 INFO:teuthology.orchestra.run.smithi064.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T03:40:08.070 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:40:08.068+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:40:08.071 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1: {} 2024-09-16T03:40:08.071 INFO:teuthology.orchestra.run.smithi064.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T03:40:08.088 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2: {} 2024-09-16T03:40:08.089 INFO:teuthology.orchestra.run.smithi064.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T03:40:08.106 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3: {} 2024-09-16T03:40:08.106 INFO:teuthology.orchestra.run.smithi064.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T03:40:08.121 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4: {} 2024-09-16T03:40:08.122 INFO:teuthology.orchestra.run.smithi064.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T03:40:08.138 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5: {} 2024-09-16T03:40:08.139 INFO:teuthology.orchestra.run.smithi064.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T03:40:08.139 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:08.139 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:08.148 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T03:40:08.155 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6: {} 2024-09-16T03:40:08.155 INFO:teuthology.orchestra.run.smithi064.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T03:40:08.171 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:40:08.170+0000 7fedc0d3b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33269) UID: 0 2024-09-16T03:40:08.172 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7: {} 2024-09-16T03:40:08.172 INFO:teuthology.orchestra.run.smithi064.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T03:40:08.188 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8: {} 2024-09-16T03:40:08.188 INFO:teuthology.orchestra.run.smithi064.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T03:40:08.205 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9: {} 2024-09-16T03:40:08.205 INFO:teuthology.orchestra.run.smithi064.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T03:40:08.221 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10: {} 2024-09-16T03:40:08.222 INFO:teuthology.orchestra.run.smithi064.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T03:40:08.239 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11: {} 2024-09-16T03:40:08.239 INFO:teuthology.orchestra.run.smithi064.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T03:40:08.252 INFO:teuthology.orchestra.run.smithi155.stdout:ERROR: (22) Invalid argument 2024-09-16T03:40:08.252 INFO:teuthology.orchestra.run.smithi155.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T03:40:08.255 INFO:teuthology.orchestra.run.smithi064.stdout:osd.12: {} 2024-09-16T03:40:08.255 INFO:teuthology.orchestra.run.smithi064.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T03:40:08.261 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T03:40:08.272 INFO:teuthology.orchestra.run.smithi064.stdout:osd.13: {} 2024-09-16T03:40:08.272 INFO:teuthology.orchestra.run.smithi064.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T03:40:08.272 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:40:08.271+0000 7ff271969640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33218) UID: 0 2024-09-16T03:40:08.288 INFO:teuthology.orchestra.run.smithi064.stdout:osd.14: {} 2024-09-16T03:40:08.289 INFO:teuthology.orchestra.run.smithi064.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T03:40:08.305 INFO:teuthology.orchestra.run.smithi064.stdout:osd.15: {} 2024-09-16T03:40:08.305 INFO:teuthology.orchestra.run.smithi064.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T03:40:08.365 INFO:teuthology.orchestra.run.smithi155.stdout:{ 2024-09-16T03:40:08.366 INFO:teuthology.orchestra.run.smithi155.stdout: "size": 20, 2024-09-16T03:40:08.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 600, 2024-09-16T03:40:08.366 INFO:teuthology.orchestra.run.smithi155.stdout: "ops": [ 2024-09-16T03:40:08.366 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.366 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:30 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'2, mlcod=0'0)", 2024-09-16T03:40:08.366 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:40:08.366 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 176.21941878999999, 2024-09-16T03:40:08.366 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.302960332, 2024-09-16T03:40:08.366 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.367 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.367 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.367 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:40:08.367 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.367 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.367 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.367 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.367 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143732+0000", 2024-09-16T03:40:08.367 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.143734+0000", 2024-09-16T03:40:08.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0870000000000001e-06 2024-09-16T03:40:08.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.368 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144027+0000", 2024-09-16T03:40:08.368 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00029316299999999998 2024-09-16T03:40:08.368 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.368 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.368 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144044+0000", 2024-09-16T03:40:08.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6433999999999999e-05 2024-09-16T03:40:08.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144055+0000", 2024-09-16T03:40:08.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0854e-05 2024-09-16T03:40:08.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.369 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.369 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.369 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144165+0000", 2024-09-16T03:40:08.369 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00011022399999999999 2024-09-16T03:40:08.369 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144195+0000", 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0224999999999999e-05 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446668+0000", 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30247257500000002 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446692+0000", 2024-09-16T03:40:08.370 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4769999999999998e-05 2024-09-16T03:40:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:31 4.f e39/32 4:f0d14c71:::smithi15736696-8:head v 39'3, mlcod=0'0)", 2024-09-16T03:40:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:40:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 176.21906543, 2024-09-16T03:40:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30262296599999999, 2024-09-16T03:40:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:40:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144085+0000", 2024-09-16T03:40:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.372 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144086+0000", 2024-09-16T03:40:08.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.4200000000000005e-07 2024-09-16T03:40:08.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144093+0000", 2024-09-16T03:40:08.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2500000000000003e-06 2024-09-16T03:40:08.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.373 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.373 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144100+0000", 2024-09-16T03:40:08.373 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.6369999999999999e-06 2024-09-16T03:40:08.373 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.373 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.144106+0000", 2024-09-16T03:40:08.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1639999999999998e-06 2024-09-16T03:40:08.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344300+0000", 2024-09-16T03:40:08.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20019377499999999 2024-09-16T03:40:08.374 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.374 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.374 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.374 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.344337+0000", 2024-09-16T03:40:08.374 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.7283000000000002e-05 2024-09-16T03:40:08.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446695+0000", 2024-09-16T03:40:08.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.102357453 2024-09-16T03:40:08.375 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.375 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.375 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.375 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446708+0000", 2024-09-16T03:40:08.375 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3562e-05 2024-09-16T03:40:08.375 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.375 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.376 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.376 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.376 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:37 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'4, mlcod=0'0)", 2024-09-16T03:40:08.376 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:40:08.376 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 176.217937632, 2024-09-16T03:40:08.376 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30150806400000002, 2024-09-16T03:40:08.376 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.376 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.376 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.376 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.376 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.376 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:40:08.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.377 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.377 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.377 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145213+0000", 2024-09-16T03:40:08.377 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.377 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.377 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145215+0000", 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7540000000000001e-06 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145755+0000", 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053976700000000002 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145768+0000", 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:40:08.378 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.379 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.379 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.145775+0000", 2024-09-16T03:40:08.379 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.3849999999999996e-06 2024-09-16T03:40:08.379 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.379 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444748+0000", 2024-09-16T03:40:08.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29897308099999997 2024-09-16T03:40:08.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.444771+0000", 2024-09-16T03:40:08.380 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2676000000000001e-05 2024-09-16T03:40:08.380 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.380 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.380 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.380 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446711+0000", 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0019397310000000001 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446721+0000", 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0718e-05 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:38 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'5, mlcod=0'0)", 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:40:08.381 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 176.216950191, 2024-09-16T03:40:08.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.300681643, 2024-09-16T03:40:08.382 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.382 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.382 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.382 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:40:08.382 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.382 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.382 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.382 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.382 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146201+0000", 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146202+0000", 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.209e-06 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146351+0000", 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014909100000000001 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.383 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146362+0000", 2024-09-16T03:40:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.092e-05 2024-09-16T03:40:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146370+0000", 2024-09-16T03:40:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:40:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445858+0000", 2024-09-16T03:40:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29948842399999998 2024-09-16T03:40:08.384 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.445877+0000", 2024-09-16T03:40:08.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8643000000000001e-05 2024-09-16T03:40:08.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446860+0000", 2024-09-16T03:40:08.385 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00098302200000000006 2024-09-16T03:40:08.385 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.385 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.385 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.385 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446882+0000", 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2469000000000001e-05 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:39 4.f e39/32 4:fa163101:::smithi15736696-10:head v 39'6, mlcod=0'0)", 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 176.21619344699999, 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30019617100000001, 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.386 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:40:08.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.387 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.387 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.387 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146957+0000", 2024-09-16T03:40:08.387 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.387 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146959+0000", 2024-09-16T03:40:08.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1739999999999999e-06 2024-09-16T03:40:08.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146965+0000", 2024-09-16T03:40:08.388 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.5200000000000003e-06 2024-09-16T03:40:08.388 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.388 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.388 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.388 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146976+0000", 2024-09-16T03:40:08.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0446e-05 2024-09-16T03:40:08.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.146983+0000", 2024-09-16T03:40:08.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.0999999999999998e-06 2024-09-16T03:40:08.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.389 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.389 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446552+0000", 2024-09-16T03:40:08.389 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29956972300000001 2024-09-16T03:40:08.389 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.389 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.446571+0000", 2024-09-16T03:40:08.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.8902e-05 2024-09-16T03:40:08.390 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447137+0000", 2024-09-16T03:40:08.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000566042 2024-09-16T03:40:08.390 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.390 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.390 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.390 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:12.447154+0000", 2024-09-16T03:40:08.390 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6263999999999999e-05 2024-09-16T03:40:08.391 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.391 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.391 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.391 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.391 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.391 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:989 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'31, mlcod=46'28)", 2024-09-16T03:40:08.391 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:40:08.391 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 157.975342087, 2024-09-16T03:40:08.391 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.40332266900000002, 2024-09-16T03:40:08.391 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.391 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.391 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.391 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:40:08.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387809+0000", 2024-09-16T03:40:08.392 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.392 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.392 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.392 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.392 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.387812+0000", 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388467+0000", 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00065539500000000002 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.388481+0000", 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.3905e-05 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.393 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488552+0000", 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10007120899999999 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688818+0000", 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20026627599999999 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.688845+0000", 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6906999999999998e-05 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.394 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791091+0000", 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10224588699999999 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791131+0000", 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.0293000000000002e-05 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.395 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:990 4.f e50/32 4:f817def9:::smithi15736696-45:head v 50'32, mlcod=46'28)", 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 157.874558415, 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30256419200000001, 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.396 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488592+0000", 2024-09-16T03:40:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488594+0000", 2024-09-16T03:40:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0899999999999999e-06 2024-09-16T03:40:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488616+0000", 2024-09-16T03:40:08.397 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.2419e-05 2024-09-16T03:40:08.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.488627+0000", 2024-09-16T03:40:08.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1251e-05 2024-09-16T03:40:08.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.398 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.398 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.398 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.588753+0000", 2024-09-16T03:40:08.398 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10012544399999999 2024-09-16T03:40:08.398 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.399 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690679+0000", 2024-09-16T03:40:08.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.101926027 2024-09-16T03:40:08.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.399 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.690695+0000", 2024-09-16T03:40:08.399 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6528000000000002e-05 2024-09-16T03:40:08.399 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.399 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.399 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.399 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791133+0000", 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100438233 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:37:30.791157+0000", 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3200000000000001e-05 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_repop(client.4628.0:3081 4.1f e102/100 4:fa163101:::smithi15736696-10:head v 102'93, mlcod=102'91)", 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:40:08.400 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 93.029435946999996, 2024-09-16T03:40:08.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20437049600000001, 2024-09-16T03:40:08.401 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.401 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.401 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.401 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.401 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:40:08.401 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.401 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.401 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.401 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.401 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333715+0000", 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.333718+0000", 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.3720000000000001e-06 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334338+0000", 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00061959099999999996 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.402 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334355+0000", 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.677e-05 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334364+0000", 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3780000000000007e-06 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334424+0000", 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0084000000000003e-05 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.403 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.334447+0000", 2024-09-16T03:40:08.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3113e-05 2024-09-16T03:40:08.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538030+0000", 2024-09-16T03:40:08.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20358272399999999 2024-09-16T03:40:08.404 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.404 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.404 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.404 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:38:35.538085+0000", 2024-09-16T03:40:08.404 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.5464000000000003e-05 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4007 4.37 4:ed874e32:::smithi15736696-14:head [write 601783~551825,stat] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.829952101000003, 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29242759600000001, 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:08.405 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:08.406 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.406 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4007 2024-09-16T03:40:08.406 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.406 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.406 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.406 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.406 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:40:08.406 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.406 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.406 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.406 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.406 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533199+0000", 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533200+0000", 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.3600000000000002e-07 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533841+0000", 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00064141100000000004 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.407 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.533862+0000", 2024-09-16T03:40:08.408 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.0687e-05 2024-09-16T03:40:08.408 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.408 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.408 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.408 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634001+0000", 2024-09-16T03:40:08.409 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100139413 2024-09-16T03:40:08.409 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.409 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.409 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.409 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721351+0000", 2024-09-16T03:40:08.409 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.087349914000000001 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for peered", 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721357+0000", 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7100000000000004e-06 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.721891+0000", 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00053461299999999997 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.410 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.722024+0000", 2024-09-16T03:40:08.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013274599999999999 2024-09-16T03:40:08.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:40:08.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.722158+0000", 2024-09-16T03:40:08.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013362399999999999 2024-09-16T03:40:08.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.411 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.411 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:08.411 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725271+0000", 2024-09-16T03:40:08.411 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.003113345 2024-09-16T03:40:08.411 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825258+0000", 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099987260999999994 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825453+0000", 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00019453999999999999 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825591+0000", 2024-09-16T03:40:08.412 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013849700000000001 2024-09-16T03:40:08.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.413 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.413 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825626+0000", 2024-09-16T03:40:08.413 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4999000000000002e-05 2024-09-16T03:40:08.413 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.413 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.413 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.413 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.413 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.413 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4008 4.37 4:ed874e32:::smithi15736696-14:head [write 1776679~425104] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:40:08.413 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:40:08.413 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.729131977999998, 2024-09-16T03:40:08.414 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29371618199999999, 2024-09-16T03:40:08.414 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.414 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.414 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.414 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:08.414 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:08.414 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.414 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4008 2024-09-16T03:40:08.414 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.414 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.414 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.414 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.414 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:40:08.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.415 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.415 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.415 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634019+0000", 2024-09-16T03:40:08.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.415 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.415 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.415 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634021+0000", 2024-09-16T03:40:08.415 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.9989999999999998e-06 2024-09-16T03:40:08.415 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.415 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.416 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.416 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634187+0000", 2024-09-16T03:40:08.416 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00016655199999999999 2024-09-16T03:40:08.416 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.416 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.417 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.417 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634197+0000", 2024-09-16T03:40:08.417 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7520000000000008e-06 2024-09-16T03:40:08.417 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.417 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.418 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634205+0000", 2024-09-16T03:40:08.418 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8210000000000002e-06 2024-09-16T03:40:08.418 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.418 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.418 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.419 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725407+0000", 2024-09-16T03:40:08.419 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.091202249999999999 2024-09-16T03:40:08.419 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.419 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.419 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.420 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725506+0000", 2024-09-16T03:40:08.420 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.8493999999999996e-05 2024-09-16T03:40:08.420 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.420 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.420 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:40:08.420 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.725648+0000", 2024-09-16T03:40:08.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014187299999999999 2024-09-16T03:40:08.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:08.421 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.727142+0000", 2024-09-16T03:40:08.421 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001494478 2024-09-16T03:40:08.421 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.421 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.421 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.422 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.825761+0000", 2024-09-16T03:40:08.422 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098619340999999999 2024-09-16T03:40:08.422 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.422 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.422 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927671+0000", 2024-09-16T03:40:08.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10190922700000001 2024-09-16T03:40:08.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.423 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.423 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.423 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927705+0000", 2024-09-16T03:40:08.423 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4523000000000001e-05 2024-09-16T03:40:08.423 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.424 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.424 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927735+0000", 2024-09-16T03:40:08.424 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.9872000000000001e-05 2024-09-16T03:40:08.424 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.424 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.424 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.424 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.424 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.424 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4009 4.37 4:ed874e32:::smithi15736696-14:head [setxattr _header (56) in=7b,truncate 2201783] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:40:08.424 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:40:08.424 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.728935051000001, 2024-09-16T03:40:08.424 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29364004500000002, 2024-09-16T03:40:08.425 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.425 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.425 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.425 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:08.425 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:08.425 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.425 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4009 2024-09-16T03:40:08.425 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.425 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.425 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.425 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.425 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634216+0000", 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634217+0000", 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.7199999999999998e-07 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.426 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634221+0000", 2024-09-16T03:40:08.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.9650000000000002e-06 2024-09-16T03:40:08.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634226+0000", 2024-09-16T03:40:08.427 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7830000000000004e-06 2024-09-16T03:40:08.427 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.427 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.427 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.427 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634232+0000", 2024-09-16T03:40:08.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2909999999999998e-06 2024-09-16T03:40:08.428 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.428 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.428 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.428 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726750+0000", 2024-09-16T03:40:08.428 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.09251827 2024-09-16T03:40:08.428 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.428 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.428 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.428 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726830+0000", 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9808999999999994e-05 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.726916+0000", 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.6055000000000002e-05 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.727785+0000", 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00086898599999999998 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.429 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.926315+0000", 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.19852985000000001 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927777+0000", 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001462138 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927818+0000", 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.1006e-05 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.430 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.431 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.431 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.927856+0000", 2024-09-16T03:40:08.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.8120000000000001e-05 2024-09-16T03:40:08.431 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.431 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.431 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.431 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.431 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.431 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4010 4.37 4:ed874e32:::smithi15736696-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:40:08.431 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:40:08.431 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.728911357999998, 2024-09-16T03:40:08.431 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.39386449699999998, 2024-09-16T03:40:08.432 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.432 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.432 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:40:08.432 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:08.432 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:08.432 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.432 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4010 2024-09-16T03:40:08.432 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.432 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.432 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.432 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.432 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:40:08.432 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:40:08.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.433 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.433 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634240+0000", 2024-09-16T03:40:08.433 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:40:08.433 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.433 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634243+0000", 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.869e-06 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634249+0000", 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.9449999999999998e-06 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.634255+0000", 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1979999999999998e-06 2024-09-16T03:40:08.434 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.734247+0000", 2024-09-16T03:40:08.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.099991570000000002 2024-09-16T03:40:08.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:08.435 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:05.734386+0000", 2024-09-16T03:40:08.435 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013961600000000001 2024-09-16T03:40:08.435 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.435 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.435 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.027978+0000", 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.29359164599999998 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.028036+0000", 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.7802e-05 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.028104+0000", 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.8264999999999998e-05 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.436 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.437 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.437 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.437 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.437 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4034 4.17 4:eb60a28d:::smithi15736696-41:head [write 635397~702323,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-16T03:40:08.437 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:40:08.437 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 62.326489913000003, 2024-09-16T03:40:08.437 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20471451500000001, 2024-09-16T03:40:08.437 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.437 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.438 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.438 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:08.438 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:08.438 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.438 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4034 2024-09-16T03:40:08.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.439 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.439 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.439 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.439 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:40:08.439 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.439 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.439 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.440 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.440 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036661+0000", 2024-09-16T03:40:08.440 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.440 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.440 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.440 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.036663+0000", 2024-09-16T03:40:08.440 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.855e-06 2024-09-16T03:40:08.440 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.440 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.440 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.440 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037227+0000", 2024-09-16T03:40:08.440 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00056451099999999996 2024-09-16T03:40:08.441 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.441 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.441 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037240+0000", 2024-09-16T03:40:08.441 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.2208999999999999e-05 2024-09-16T03:40:08.441 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.441 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.441 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037245+0000", 2024-09-16T03:40:08.441 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.6319999999999996e-06 2024-09-16T03:40:08.441 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.441 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.441 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.442 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037305+0000", 2024-09-16T03:40:08.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.0272000000000002e-05 2024-09-16T03:40:08.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.442 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.442 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037384+0000", 2024-09-16T03:40:08.442 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.8330000000000004e-05 2024-09-16T03:40:08.442 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.442 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:40:08.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.037491+0000", 2024-09-16T03:40:08.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00010730699999999999 2024-09-16T03:40:08.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.443 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:08.443 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.042284+0000", 2024-09-16T03:40:08.443 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0047932649999999997 2024-09-16T03:40:08.443 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.443 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.140922+0000", 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.098637552000000003 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241267+0000", 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100344866 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241330+0000", 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.2953000000000005e-05 2024-09-16T03:40:08.444 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.445 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.445 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:06.241375+0000", 2024-09-16T03:40:08.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5763000000000002e-05 2024-09-16T03:40:08.445 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.445 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.445 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.445 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.445 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.445 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "rep_scrubmap(4.32 e135 from shard 11)", 2024-09-16T03:40:08.445 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:11.840602+0000", 2024-09-16T03:40:08.445 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 56.522548682, 2024-09-16T03:40:08.445 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.26655261099999999, 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840602+0000", 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840602+0000", 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.446 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840605+0000", 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.7970000000000001e-06 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840616+0000", 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0672e-05 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.840632+0000", 2024-09-16T03:40:08.447 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6268000000000001e-05 2024-09-16T03:40:08.448 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.448 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.448 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.940701+0000", 2024-09-16T03:40:08.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10006937 2024-09-16T03:40:08.448 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.448 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.448 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.107074+0000", 2024-09-16T03:40:08.448 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.16637296600000001 2024-09-16T03:40:08.448 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.448 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.448 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.449 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.107101+0000", 2024-09-16T03:40:08.449 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6225999999999999e-05 2024-09-16T03:40:08.449 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.449 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.449 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.449 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.107155+0000", 2024-09-16T03:40:08.449 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4311999999999998e-05 2024-09-16T03:40:08.449 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.449 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.449 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.449 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.449 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.449 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4101 4.17 4:eb60a28d:::smithi15736696-41:head [write 719432~794921] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-16T03:40:08.450 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:11.929701+0000", 2024-09-16T03:40:08.450 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 56.433449558, 2024-09-16T03:40:08.450 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30975548400000003, 2024-09-16T03:40:08.450 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.450 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.450 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.450 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:08.450 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:08.450 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.450 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4101 2024-09-16T03:40:08.450 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.450 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.450 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.451 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.929701+0000", 2024-09-16T03:40:08.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.451 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.929701+0000", 2024-09-16T03:40:08.451 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.451 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.451 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.451 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.451 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.929704+0000", 2024-09-16T03:40:08.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.21e-06 2024-09-16T03:40:08.452 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.452 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.452 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.452 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.930544+0000", 2024-09-16T03:40:08.452 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00084025300000000001 2024-09-16T03:40:08.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.453 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.453 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.453 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.930561+0000", 2024-09-16T03:40:08.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.7601999999999999e-05 2024-09-16T03:40:08.453 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.453 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.453 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.453 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:11.930571+0000", 2024-09-16T03:40:08.453 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.1260000000000004e-06 2024-09-16T03:40:08.454 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.454 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.454 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.454 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.030702+0000", 2024-09-16T03:40:08.454 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10013184899999999 2024-09-16T03:40:08.454 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.454 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.454 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.454 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.030769+0000", 2024-09-16T03:40:08.454 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.6742000000000006e-05 2024-09-16T03:40:08.454 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.454 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.454 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 0,6", 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.030873+0000", 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000104005 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.033373+0000", 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.0025000489999999998 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.138540+0000", 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10516709 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.455 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.239289+0000", 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10074846599999999 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.239382+0000", 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.3195000000000005e-05 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:12.239457+0000", 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.4896999999999998e-05 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.456 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.457 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.457 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.457 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.457 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4366 4.23 4:c512d873:::smithi15736696-38:head [assert-version v41,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e143)", 2024-09-16T03:40:08.457 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:20.361285+0000", 2024-09-16T03:40:08.457 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 48.001865645000002, 2024-09-16T03:40:08.457 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.303627655, 2024-09-16T03:40:08.457 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.457 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.457 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.457 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:08.457 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:08.457 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.458 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4366 2024-09-16T03:40:08.458 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.458 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.458 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.458 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.458 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361285+0000", 2024-09-16T03:40:08.458 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.458 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.458 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.458 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.458 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361285+0000", 2024-09-16T03:40:08.459 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.459 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.459 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.459 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.459 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361287+0000", 2024-09-16T03:40:08.459 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.626e-06 2024-09-16T03:40:08.459 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.459 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.459 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.459 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361292+0000", 2024-09-16T03:40:08.459 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.3279999999999996e-06 2024-09-16T03:40:08.459 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.460 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.460 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.460 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361304+0000", 2024-09-16T03:40:08.460 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.1574e-05 2024-09-16T03:40:08.460 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.460 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.460 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.460 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361310+0000", 2024-09-16T03:40:08.460 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.1990000000000003e-06 2024-09-16T03:40:08.460 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.460 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.460 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361362+0000", 2024-09-16T03:40:08.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.2459e-05 2024-09-16T03:40:08.461 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.461 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.461 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.361448+0000", 2024-09-16T03:40:08.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 8.5928999999999997e-05 2024-09-16T03:40:08.461 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.461 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.461 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:40:08.461 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.463126+0000", 2024-09-16T03:40:08.461 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10167792 2024-09-16T03:40:08.461 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.462 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.462 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "op_commit", 2024-09-16T03:40:08.462 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.463778+0000", 2024-09-16T03:40:08.462 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.000652123 2024-09-16T03:40:08.462 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.462 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.462 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.462 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.564004+0000", 2024-09-16T03:40:08.462 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100225143 2024-09-16T03:40:08.462 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.463 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.463 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.463 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.664718+0000", 2024-09-16T03:40:08.463 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100714673 2024-09-16T03:40:08.463 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.463 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.463 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "commit_sent", 2024-09-16T03:40:08.463 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.664865+0000", 2024-09-16T03:40:08.463 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00014715099999999999 2024-09-16T03:40:08.463 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.463 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.463 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.463 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:20.664913+0000", 2024-09-16T03:40:08.464 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7530000000000001e-05 2024-09-16T03:40:08.464 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.464 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.464 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.464 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.464 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.464 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4526 4.12 4:4a9d235d:::smithi15736696-1:head [tier-promote] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-16T03:40:08.464 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:40:08.464 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.791380584000002, 2024-09-16T03:40:08.464 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30344452999999999, 2024-09-16T03:40:08.464 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.465 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.465 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:40:08.465 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:08.465 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:08.465 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.465 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4526 2024-09-16T03:40:08.465 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.465 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.465 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.465 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.465 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:40:08.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.466 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.466 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.466 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.466 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571770+0000", 2024-09-16T03:40:08.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.466 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.466 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.466 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.466 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571773+0000", 2024-09-16T03:40:08.466 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.3750000000000001e-06 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571780+0000", 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.5480000000000004e-06 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571797+0000", 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.6320999999999998e-05 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.467 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571821+0000", 2024-09-16T03:40:08.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.4111e-05 2024-09-16T03:40:08.468 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.468 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.468 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.468 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571884+0000", 2024-09-16T03:40:08.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.3608000000000005e-05 2024-09-16T03:40:08.468 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.468 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.468 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.468 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.571981+0000", 2024-09-16T03:40:08.468 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.6255999999999995e-05 2024-09-16T03:40:08.469 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.469 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.469 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.469 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:25.875215+0000", 2024-09-16T03:40:08.469 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30323431099999998 2024-09-16T03:40:08.469 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.470 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.470 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.470 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.470 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.470 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4539 4.12 4:4a9d235d:::smithi15736696-1:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:40:08.471 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:26.069749+0000", 2024-09-16T03:40:08.471 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 42.293402127999997, 2024-09-16T03:40:08.471 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.31900961999999999, 2024-09-16T03:40:08.471 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.471 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.471 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:40:08.471 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:08.471 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:08.472 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.472 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4539 2024-09-16T03:40:08.472 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.472 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.472 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.472 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.472 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069749+0000", 2024-09-16T03:40:08.472 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.472 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.472 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.472 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.472 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069749+0000", 2024-09-16T03:40:08.473 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.473 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.473 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.473 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.473 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069752+0000", 2024-09-16T03:40:08.473 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.4800000000000001e-06 2024-09-16T03:40:08.473 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.473 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.473 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.473 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069779+0000", 2024-09-16T03:40:08.473 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6336e-05 2024-09-16T03:40:08.473 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.474 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.474 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.474 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069794+0000", 2024-09-16T03:40:08.474 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.5648e-05 2024-09-16T03:40:08.474 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.474 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.474 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.474 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069805+0000", 2024-09-16T03:40:08.474 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 1.0403e-05 2024-09-16T03:40:08.474 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.474 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069884+0000", 2024-09-16T03:40:08.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 7.9660000000000006e-05 2024-09-16T03:40:08.475 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.475 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.069982+0000", 2024-09-16T03:40:08.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 9.7637e-05 2024-09-16T03:40:08.475 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.475 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.475 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.475 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.388758+0000", 2024-09-16T03:40:08.475 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.31877645599999999 2024-09-16T03:40:08.476 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.476 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.476 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.476 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.476 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.476 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4561 4.23 4:c512d873:::smithi15736696-38:head [read 0~2672379,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e148)", 2024-09-16T03:40:08.476 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:26.675782+0000", 2024-09-16T03:40:08.476 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 41.687368704000001, 2024-09-16T03:40:08.476 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30481955599999999, 2024-09-16T03:40:08.476 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.476 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.476 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:40:08.477 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:08.477 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:08.477 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.477 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4561 2024-09-16T03:40:08.477 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.477 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.477 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.477 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675782+0000", 2024-09-16T03:40:08.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.478 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.478 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.478 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675782+0000", 2024-09-16T03:40:08.478 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.478 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.478 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.478 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675783+0000", 2024-09-16T03:40:08.479 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4900000000000001e-07 2024-09-16T03:40:08.479 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.479 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.479 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.479 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675785+0000", 2024-09-16T03:40:08.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 2.6429999999999999e-06 2024-09-16T03:40:08.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.480 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675790+0000", 2024-09-16T03:40:08.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.4859999999999999e-06 2024-09-16T03:40:08.480 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.480 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.480 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.480 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675793+0000", 2024-09-16T03:40:08.480 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 3.0019999999999998e-06 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.776027+0000", 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.10023393899999999 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.776034+0000", 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 6.939e-06 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.481 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.978683+0000", 2024-09-16T03:40:08.482 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.20264916399999999 2024-09-16T03:40:08.482 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.482 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.482 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.482 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.978742+0000", 2024-09-16T03:40:08.482 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.8699000000000002e-05 2024-09-16T03:40:08.482 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.482 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.482 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.482 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.980602+0000", 2024-09-16T03:40:08.482 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.001860235 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: "description": "osd_op(client.4628.0:4562 4.23 4:c512d873:::smithi15736696-38:head [read 0~2672379,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e148)", 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: "initiated_at": "2024-09-16T03:39:26.675811+0000", 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: "age": 41.687340370999998, 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.30577821199999999, 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: "continuous": false, 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: "type_data": { 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: "flag_point": "started", 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: "client_info": { 2024-09-16T03:40:08.483 INFO:teuthology.orchestra.run.smithi155.stdout: "client": "client.4628", 2024-09-16T03:40:08.484 INFO:teuthology.orchestra.run.smithi155.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.484 INFO:teuthology.orchestra.run.smithi155.stdout: "tid": 4562 2024-09-16T03:40:08.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.484 INFO:teuthology.orchestra.run.smithi155.stdout: "events": [ 2024-09-16T03:40:08.484 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.484 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "initiated", 2024-09-16T03:40:08.484 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675811+0000", 2024-09-16T03:40:08.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.484 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.484 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.484 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "header_read", 2024-09-16T03:40:08.484 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675811+0000", 2024-09-16T03:40:08.484 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0 2024-09-16T03:40:08.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "throttled", 2024-09-16T03:40:08.485 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675811+0000", 2024-09-16T03:40:08.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.5400000000000002e-07 2024-09-16T03:40:08.485 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.485 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.485 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "all_read", 2024-09-16T03:40:08.485 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675816+0000", 2024-09-16T03:40:08.485 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.7539999999999997e-06 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "dispatched", 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.675821+0000", 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 4.8160000000000001e-06 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.775929+0000", 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.100108076 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.486 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.776062+0000", 2024-09-16T03:40:08.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00013323400000000001 2024-09-16T03:40:08.487 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.487 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.487 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "waiting_for_map not empty", 2024-09-16T03:40:08.487 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.776067+0000", 2024-09-16T03:40:08.487 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.1669999999999998e-06 2024-09-16T03:40:08.487 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.487 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.487 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "reached_pg", 2024-09-16T03:40:08.487 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.980656+0000", 2024-09-16T03:40:08.488 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.204588772 2024-09-16T03:40:08.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "started", 2024-09-16T03:40:08.488 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.980711+0000", 2024-09-16T03:40:08.488 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 5.4898999999999997e-05 2024-09-16T03:40:08.488 INFO:teuthology.orchestra.run.smithi155.stdout: }, 2024-09-16T03:40:08.488 INFO:teuthology.orchestra.run.smithi155.stdout: { 2024-09-16T03:40:08.488 INFO:teuthology.orchestra.run.smithi155.stdout: "event": "done", 2024-09-16T03:40:08.489 INFO:teuthology.orchestra.run.smithi155.stdout: "time": "2024-09-16T03:39:26.981589+0000", 2024-09-16T03:40:08.489 INFO:teuthology.orchestra.run.smithi155.stdout: "duration": 0.00087803999999999996 2024-09-16T03:40:08.489 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.489 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.489 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.489 INFO:teuthology.orchestra.run.smithi155.stdout: } 2024-09-16T03:40:08.489 INFO:teuthology.orchestra.run.smithi155.stdout: ] 2024-09-16T03:40:08.489 INFO:teuthology.orchestra.run.smithi155.stdout:} 2024-09-16T03:40:08.490 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:40:08.491 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:40:08.429+0000 7efef16d9640 -1 osd.13 183 handle_osd_map injecting map crc failure 2024-09-16T03:40:08.492 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'unique_pool_0' removedpool 'unique_pool_0' removed 2024-09-16T03:40:08.492 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T03:40:08.493 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:40:08.476+0000 7f3a1f71e640 -1 osd.6 183 handle_osd_map injecting map crc failure 2024-09-16T03:40:08.493 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:40:08.489+0000 7f0af6b56640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33435) UID: 0 2024-09-16T03:40:08.494 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-16T03:40:08.505 INFO:tasks.thrashosds:joining thrashosds 2024-09-16T03:40:08.544 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:40:08.541+0000 7fd101e6f640 -1 osd.2 183 handle_osd_map injecting map crc failure 2024-09-16T03:40:08.624 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:40:08.625 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:40:08.625 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:40:08.625 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:40:08.634 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T03:40:08.740 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:40:08.740 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:40:08.740 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:40:08.740 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:40:08.740 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:40:08.748 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T03:40:08.855 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:40:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:40:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:40:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:40:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3591 4.1b 4:dcb6965e:::smithi15736696-28:head [write 3335341~194162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:40:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 77.796341398999999, 2024-09-16T03:40:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60146457399999997, 2024-09-16T03:40:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.856 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3591 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056405+0000", 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056407+0000", 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8920000000000001e-06 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056502+0000", 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4838000000000002e-05 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.056511+0000", 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3579999999999992e-06 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156581+0000", 2024-09-16T03:40:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100069254 2024-09-16T03:40:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256413+0000", 2024-09-16T03:40:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099832613000000001 2024-09-16T03:40:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256479+0000", 2024-09-16T03:40:08.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5388000000000005e-05 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.256582+0000", 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000102876 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257889+0000", 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001307119 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657701+0000", 2024-09-16T03:40:08.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39981235999999998 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657769+0000", 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7508000000000005e-05 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657827+0000", 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8878000000000001e-05 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657870+0000", 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2490000000000001e-05 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3592 4.1b 4:dcb6965e:::smithi15736696-28:head [setxattr _header (56) in=7b,truncate 3529503] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 77.696133247000006, 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60139765499999998, 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.862 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3592 2024-09-16T03:40:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:40:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156613+0000", 2024-09-16T03:40:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156615+0000", 2024-09-16T03:40:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.714e-06 2024-09-16T03:40:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156623+0000", 2024-09-16T03:40:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5939999999999996e-06 2024-09-16T03:40:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156634+0000", 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.093e-05 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156640+0000", 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1890000000000004e-06 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257172+0000", 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100532525 2024-09-16T03:40:08.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257233+0000", 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0736999999999998e-05 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,8", 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257319+0000", 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5870000000000003e-05 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:08.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257891+0000", 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057231100000000004 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.657732+0000", 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39984067699999998 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.757877+0000", 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100145238 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:08.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.757947+0000", 2024-09-16T03:40:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9821999999999997e-05 2024-09-16T03:40:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758011+0000", 2024-09-16T03:40:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4047999999999994e-05 2024-09-16T03:40:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.868 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3593 4.1b 4:dcb6965e:::smithi15736696-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 77.696098512999995, 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60159266700000003, 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3593 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.869 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:40:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156648+0000", 2024-09-16T03:40:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156649+0000", 2024-09-16T03:40:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6400000000000002e-07 2024-09-16T03:40:08.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156652+0000", 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9579999999999999e-06 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156658+0000", 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4729999999999997e-06 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.156667+0000", 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.1940000000000004e-06 2024-09-16T03:40:08.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257836+0000", 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101168444 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.257881+0000", 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4910999999999997e-05 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758073+0000", 2024-09-16T03:40:08.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50019239699999996 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758147+0000", 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3360000000000002e-05 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:51.758241+0000", 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4265999999999995e-05 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4069 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548281+0000", 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.304465796000002, 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62975188199999999, 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4069 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.874 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548281+0000", 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548281+0000", 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548284+0000", 2024-09-16T03:40:08.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:40:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548292+0000", 2024-09-16T03:40:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2169999999999994e-06 2024-09-16T03:40:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548308+0000", 2024-09-16T03:40:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5602000000000001e-05 2024-09-16T03:40:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548322+0000", 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-05 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548706+0000", 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038430199999999999 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548721+0000", 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4440999999999999e-05 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.177929+0000", 2024-09-16T03:40:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62920806500000004 2024-09-16T03:40:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.178033+0000", 2024-09-16T03:40:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000103824 2024-09-16T03:40:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4070 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548345+0000", 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.304401939000002, 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.729858275, 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4070 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548345+0000", 2024-09-16T03:40:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548345+0000", 2024-09-16T03:40:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548346+0000", 2024-09-16T03:40:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.64e-07 2024-09-16T03:40:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548354+0000", 2024-09-16T03:40:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.875e-06 2024-09-16T03:40:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548377+0000", 2024-09-16T03:40:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3612999999999999e-05 2024-09-16T03:40:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548390+0000", 2024-09-16T03:40:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2836999999999999e-05 2024-09-16T03:40:08.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548761+0000", 2024-09-16T03:40:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037075600000000003 2024-09-16T03:40:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548769+0000", 2024-09-16T03:40:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5110000000000003e-06 2024-09-16T03:40:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278121+0000", 2024-09-16T03:40:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72935201800000005 2024-09-16T03:40:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278203+0000", 2024-09-16T03:40:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1700999999999996e-05 2024-09-16T03:40:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.884 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4071 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:40:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548403+0000", 2024-09-16T03:40:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.304343948000003, 2024-09-16T03:40:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73003053100000004, 2024-09-16T03:40:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:40:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4071 2024-09-16T03:40:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548403+0000", 2024-09-16T03:40:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548403+0000", 2024-09-16T03:40:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548404+0000", 2024-09-16T03:40:08.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0500000000000002e-07 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548408+0000", 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2699999999999998e-06 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548415+0000", 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7340000000000002e-06 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548421+0000", 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.84e-06 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548798+0000", 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000377916 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548806+0000", 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5599999999999996e-06 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278371+0000", 2024-09-16T03:40:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.729564616 2024-09-16T03:40:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.278433+0000", 2024-09-16T03:40:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2689999999999998e-05 2024-09-16T03:40:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4079 3.0 3:18759fb9:::13e91a6c9fb63c85209a4d9065cc1e1225929f8ddc4955a7e63272d6c7055a80:head [write 0~1183232] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e135)", 2024-09-16T03:40:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:10.548428+0000", 2024-09-16T03:40:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 58.304318662999997, 2024-09-16T03:40:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0303502819999999, 2024-09-16T03:40:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4079 2024-09-16T03:40:08.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548428+0000", 2024-09-16T03:40:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548428+0000", 2024-09-16T03:40:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.548429+0000", 2024-09-16T03:40:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2399999999999998e-07 2024-09-16T03:40:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549249+0000", 2024-09-16T03:40:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00081996300000000005 2024-09-16T03:40:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.892 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549263+0000", 2024-09-16T03:40:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4319e-05 2024-09-16T03:40:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549271+0000", 2024-09-16T03:40:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1640000000000001e-06 2024-09-16T03:40:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549342+0000", 2024-09-16T03:40:08.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0638999999999999e-05 2024-09-16T03:40:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:10.549353+0000", 2024-09-16T03:40:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1653e-05 2024-09-16T03:40:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.274940+0000", 2024-09-16T03:40:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72558676 2024-09-16T03:40:08.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.275056+0000", 2024-09-16T03:40:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011581 2024-09-16T03:40:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 4,6,9", 2024-09-16T03:40:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.275176+0000", 2024-09-16T03:40:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000119963 2024-09-16T03:40:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.377903+0000", 2024-09-16T03:40:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10272743199999999 2024-09-16T03:40:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578407+0000", 2024-09-16T03:40:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200503758 2024-09-16T03:40:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578475+0000", 2024-09-16T03:40:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7718000000000005e-05 2024-09-16T03:40:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578663+0000", 2024-09-16T03:40:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018870599999999999 2024-09-16T03:40:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578706+0000", 2024-09-16T03:40:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2459000000000001e-05 2024-09-16T03:40:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:11.578778+0000", 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2414000000000002e-05 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4534 4.35 4:ad0b9852:::smithi15736696-16:head [write 574905~711397 in=711397b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.060664+0000", 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 42.792082937000004, 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62894715300000004, 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.899 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4534 2024-09-16T03:40:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060664+0000", 2024-09-16T03:40:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060664+0000", 2024-09-16T03:40:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060665+0000", 2024-09-16T03:40:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2330000000000001e-06 2024-09-16T03:40:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060808+0000", 2024-09-16T03:40:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014335500000000001 2024-09-16T03:40:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060819+0000", 2024-09-16T03:40:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0828e-05 2024-09-16T03:40:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.060826+0000", 2024-09-16T03:40:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7209999999999997e-06 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.237917+0000", 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17709092100000001 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.237985+0000", 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8270999999999997e-05 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:40:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.238092+0000", 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000107324 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241761+0000", 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0036689190000000001 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243578+0000", 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018165029999999999 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.689508+0000", 2024-09-16T03:40:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.44593033199999998 2024-09-16T03:40:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.689561+0000", 2024-09-16T03:40:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2401000000000002e-05 2024-09-16T03:40:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.689611+0000", 2024-09-16T03:40:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0345e-05 2024-09-16T03:40:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4535 4.35 4:ad0b9852:::smithi15736696-16:head [write 1885086~525140 in=525140b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:40:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.066720+0000", 2024-09-16T03:40:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 42.786026735, 2024-09-16T03:40:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72311801499999995, 2024-09-16T03:40:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4535 2024-09-16T03:40:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066720+0000", 2024-09-16T03:40:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066720+0000", 2024-09-16T03:40:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066721+0000", 2024-09-16T03:40:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.111e-06 2024-09-16T03:40:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066954+0000", 2024-09-16T03:40:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00023295100000000001 2024-09-16T03:40:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066965+0000", 2024-09-16T03:40:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0845e-05 2024-09-16T03:40:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.066970+0000", 2024-09-16T03:40:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5459999999999999e-06 2024-09-16T03:40:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239646+0000", 2024-09-16T03:40:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17267560400000001 2024-09-16T03:40:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239715+0000", 2024-09-16T03:40:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8566999999999996e-05 2024-09-16T03:40:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:40:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.239843+0000", 2024-09-16T03:40:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000128861 2024-09-16T03:40:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241769+0000", 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001925544 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243609+0000", 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0018399340000000001 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789655+0000", 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54604627900000002 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789709+0000", 2024-09-16T03:40:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3479000000000001e-05 2024-09-16T03:40:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789838+0000", 2024-09-16T03:40:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012929400000000001 2024-09-16T03:40:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4536 4.35 4:ad0b9852:::smithi15736696-16:head [write 2930189~44716 in=44716b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:40:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.069712+0000", 2024-09-16T03:40:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 42.783035124999998, 2024-09-16T03:40:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72027944300000002, 2024-09-16T03:40:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4536 2024-09-16T03:40:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069712+0000", 2024-09-16T03:40:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069712+0000", 2024-09-16T03:40:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069713+0000", 2024-09-16T03:40:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1209999999999999e-06 2024-09-16T03:40:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069748+0000", 2024-09-16T03:40:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5626000000000001e-05 2024-09-16T03:40:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069765+0000", 2024-09-16T03:40:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7104000000000001e-05 2024-09-16T03:40:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069771+0000", 2024-09-16T03:40:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4310000000000002e-06 2024-09-16T03:40:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241008+0000", 2024-09-16T03:40:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17123754899999999 2024-09-16T03:40:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241070+0000", 2024-09-16T03:40:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1351000000000003e-05 2024-09-16T03:40:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:40:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241165+0000", 2024-09-16T03:40:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5372999999999996e-05 2024-09-16T03:40:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241774+0000", 2024-09-16T03:40:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000609258 2024-09-16T03:40:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.243890+0000", 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0021156669999999999 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789905+0000", 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54601474800000005 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789952+0000", 2024-09-16T03:40:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7539e-05 2024-09-16T03:40:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.789991+0000", 2024-09-16T03:40:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8676e-05 2024-09-16T03:40:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4537 4.35 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=63b,truncate 2974905] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:40:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.069780+0000", 2024-09-16T03:40:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 42.782966631999997, 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72043964199999999, 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4537 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069780+0000", 2024-09-16T03:40:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069780+0000", 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069781+0000", 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7899999999999998e-07 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069784+0000", 2024-09-16T03:40:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9170000000000002e-06 2024-09-16T03:40:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069788+0000", 2024-09-16T03:40:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7230000000000004e-06 2024-09-16T03:40:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069791+0000", 2024-09-16T03:40:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0060000000000001e-06 2024-09-16T03:40:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241479+0000", 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.17168776899999999 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241523+0000", 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4270000000000001e-05 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 9,10", 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241598+0000", 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4264000000000006e-05 2024-09-16T03:40:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241973+0000", 2024-09-16T03:40:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037565999999999999 2024-09-16T03:40:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.344072+0000", 2024-09-16T03:40:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10209850500000001 2024-09-16T03:40:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790106+0000", 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.44603410500000001 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790149+0000", 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3204000000000002e-05 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790220+0000", 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0640000000000001e-05 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4538 4.35 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e147)", 2024-09-16T03:40:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:40:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 42.782951105000002, 2024-09-16T03:40:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.72066842900000005, 2024-09-16T03:40:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4538 2024-09-16T03:40:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:40:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:40:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069796+0000", 2024-09-16T03:40:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.03e-07 2024-09-16T03:40:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069797+0000", 2024-09-16T03:40:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4500000000000001e-06 2024-09-16T03:40:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.069801+0000", 2024-09-16T03:40:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8850000000000001e-06 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.169935+0000", 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013362100000001 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241791+0000", 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.071856512999999997 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.241834+0000", 2024-09-16T03:40:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2824999999999997e-05 2024-09-16T03:40:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790284+0000", 2024-09-16T03:40:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54844967 2024-09-16T03:40:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790369+0000", 2024-09-16T03:40:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5453999999999999e-05 2024-09-16T03:40:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:26.790464+0000", 2024-09-16T03:40:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4708000000000002e-05 2024-09-16T03:40:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4644 3.0 3:1f968072:::a9957f29a7f946349b562c449f43fa1bc2e3c48b00925943f7b23963f3ffcd03:head [write 0~1316352] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e149)", 2024-09-16T03:40:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:29.647871+0000", 2024-09-16T03:40:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 39.204875223000002, 2024-09-16T03:40:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.707065583, 2024-09-16T03:40:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4644 2024-09-16T03:40:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.647871+0000", 2024-09-16T03:40:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.647871+0000", 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.647873+0000", 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6050000000000001e-06 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648606+0000", 2024-09-16T03:40:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00073310499999999995 2024-09-16T03:40:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648619+0000", 2024-09-16T03:40:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3023000000000001e-05 2024-09-16T03:40:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648631+0000", 2024-09-16T03:40:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.173e-05 2024-09-16T03:40:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.748799+0000", 2024-09-16T03:40:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100167847 2024-09-16T03:40:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.748813+0000", 2024-09-16T03:40:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3913999999999999e-05 2024-09-16T03:40:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.950128+0000", 2024-09-16T03:40:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20131515899999999 2024-09-16T03:40:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.950233+0000", 2024-09-16T03:40:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010524 2024-09-16T03:40:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1,4,6,9", 2024-09-16T03:40:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.950411+0000", 2024-09-16T03:40:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000177937 2024-09-16T03:40:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153899+0000", 2024-09-16T03:40:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20348797699999999 2024-09-16T03:40:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153953+0000", 2024-09-16T03:40:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3919999999999999e-05 2024-09-16T03:40:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153964+0000", 2024-09-16T03:40:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1089e-05 2024-09-16T03:40:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254327+0000", 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100362865 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.354738+0000", 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100410991 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.354824+0000", 2024-09-16T03:40:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6557999999999994e-05 2024-09-16T03:40:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.354937+0000", 2024-09-16T03:40:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000112623 2024-09-16T03:40:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:179 3.0 3:004ec411:::10547772e2dbc1bd82bb7d7e420056cffb9d849bbc3fd015ab311821f554ea0c:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e150)", 2024-09-16T03:40:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:29.648695+0000", 2024-09-16T03:40:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 39.204051804000002, 2024-09-16T03:40:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70650748100000005, 2024-09-16T03:40:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648695+0000", 2024-09-16T03:40:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648695+0000", 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648696+0000", 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2900000000000003e-07 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648700+0000", 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.873e-06 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648708+0000", 2024-09-16T03:40:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1969999999999996e-06 2024-09-16T03:40:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648715+0000", 2024-09-16T03:40:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.17e-06 2024-09-16T03:40:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.748978+0000", 2024-09-16T03:40:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100262658 2024-09-16T03:40:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.748981+0000", 2024-09-16T03:40:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3859999999999999e-06 2024-09-16T03:40:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.953320+0000", 2024-09-16T03:40:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20433948599999999 2024-09-16T03:40:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.953414+0000", 2024-09-16T03:40:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3813000000000005e-05 2024-09-16T03:40:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1,4,6,9", 2024-09-16T03:40:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.953542+0000", 2024-09-16T03:40:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012773799999999999 2024-09-16T03:40:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153969+0000", 2024-09-16T03:40:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200426624 2024-09-16T03:40:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254193+0000", 2024-09-16T03:40:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100223953 2024-09-16T03:40:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254393+0000", 2024-09-16T03:40:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000200745 2024-09-16T03:40:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254424+0000", 2024-09-16T03:40:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0825999999999999e-05 2024-09-16T03:40:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355004+0000", 2024-09-16T03:40:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100579794 2024-09-16T03:40:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355070+0000", 2024-09-16T03:40:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5999000000000003e-05 2024-09-16T03:40:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355202+0000", 2024-09-16T03:40:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013249 2024-09-16T03:40:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:180 3.0 3:0569db51:::61d0b13b504667b91c94c752c1e35844395f8745f67042f596605bdf34d7970d:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e150)", 2024-09-16T03:40:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:29.648725+0000", 2024-09-16T03:40:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 39.204021791999999, 2024-09-16T03:40:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70679689899999998, 2024-09-16T03:40:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648725+0000", 2024-09-16T03:40:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648725+0000", 2024-09-16T03:40:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648726+0000", 2024-09-16T03:40:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0800000000000004e-07 2024-09-16T03:40:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648729+0000", 2024-09-16T03:40:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4589999999999998e-06 2024-09-16T03:40:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648737+0000", 2024-09-16T03:40:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7880000000000007e-06 2024-09-16T03:40:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.648740+0000", 2024-09-16T03:40:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6729999999999998e-06 2024-09-16T03:40:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.849392+0000", 2024-09-16T03:40:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20065119000000001 2024-09-16T03:40:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:29.849397+0000", 2024-09-16T03:40:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3809999999999998e-06 2024-09-16T03:40:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153085+0000", 2024-09-16T03:40:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30368795100000001 2024-09-16T03:40:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153326+0000", 2024-09-16T03:40:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000241396 2024-09-16T03:40:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 1,4,6,9", 2024-09-16T03:40:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.153489+0000", 2024-09-16T03:40:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000162492 2024-09-16T03:40:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254432+0000", 2024-09-16T03:40:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100943038 2024-09-16T03:40:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254454+0000", 2024-09-16T03:40:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2212999999999999e-05 2024-09-16T03:40:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.254476+0000", 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1781e-05 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355284+0000", 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100808457 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355335+0000", 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0864e-05 2024-09-16T03:40:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355396+0000", 2024-09-16T03:40:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1135000000000003e-05 2024-09-16T03:40:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:30.355522+0000", 2024-09-16T03:40:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000125473 2024-09-16T03:40:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5045 4.35 4:ad0b9852:::smithi15736696-16:head [write 2700190~586604] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.871612+0000", 2024-09-16T03:40:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.981134902000001, 2024-09-16T03:40:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.59445419600000005, 2024-09-16T03:40:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5045 2024-09-16T03:40:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871612+0000", 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871612+0000", 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871613+0000", 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6190000000000001e-06 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871900+0000", 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000286874 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871911+0000", 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1151999999999999e-05 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871917+0000", 2024-09-16T03:40:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9880000000000001e-06 2024-09-16T03:40:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.163760+0000", 2024-09-16T03:40:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29184254199999998 2024-09-16T03:40:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.163846+0000", 2024-09-16T03:40:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.5944000000000002e-05 2024-09-16T03:40:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4,6,9,10", 2024-09-16T03:40:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.163968+0000", 2024-09-16T03:40:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000121995 2024-09-16T03:40:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.365568+0000", 2024-09-16T03:40:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20159980799999999 2024-09-16T03:40:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465712+0000", 2024-09-16T03:40:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100144396 2024-09-16T03:40:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465745+0000", 2024-09-16T03:40:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3151000000000002e-05 2024-09-16T03:40:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465803+0000", 2024-09-16T03:40:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8006e-05 2024-09-16T03:40:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465827+0000", 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4187999999999999e-05 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465914+0000", 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6172000000000001e-05 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465957+0000", 2024-09-16T03:40:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3269999999999997e-05 2024-09-16T03:40:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466066+0000", 2024-09-16T03:40:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109091 2024-09-16T03:40:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5046 4.35 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=7b,truncate 3680799] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:08.964 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.871927+0000", 2024-09-16T03:40:08.964 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.980819479000001, 2024-09-16T03:40:08.964 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.59428540500000004, 2024-09-16T03:40:08.964 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.964 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.964 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:08.964 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.964 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.964 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5046 2024-09-16T03:40:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871927+0000", 2024-09-16T03:40:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871927+0000", 2024-09-16T03:40:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871928+0000", 2024-09-16T03:40:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7700000000000004e-07 2024-09-16T03:40:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871931+0000", 2024-09-16T03:40:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0860000000000002e-06 2024-09-16T03:40:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871936+0000", 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.865e-06 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871939+0000", 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5250000000000001e-06 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.164841+0000", 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29290207299999999 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.164901+0000", 2024-09-16T03:40:08.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9176999999999999e-05 2024-09-16T03:40:08.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4,6,9,10", 2024-09-16T03:40:08.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.165000+0000", 2024-09-16T03:40:08.968 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9374000000000001e-05 2024-09-16T03:40:08.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.968 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.968 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.968 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465794+0000", 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30079360999999999 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465807+0000", 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.377e-05 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465850+0000", 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2719000000000002e-05 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465872+0000", 2024-09-16T03:40:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1784999999999999e-05 2024-09-16T03:40:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.465893+0000", 2024-09-16T03:40:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0811000000000001e-05 2024-09-16T03:40:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466102+0000", 2024-09-16T03:40:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020906399999999999 2024-09-16T03:40:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466132+0000", 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0663000000000001e-05 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466213+0000", 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0205999999999997e-05 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5047 4.35 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.871944+0000", 2024-09-16T03:40:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.980802889, 2024-09-16T03:40:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.59445495699999995, 2024-09-16T03:40:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5047 2024-09-16T03:40:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871944+0000", 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871944+0000", 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871944+0000", 2024-09-16T03:40:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1199999999999999e-07 2024-09-16T03:40:08.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871946+0000", 2024-09-16T03:40:08.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6479999999999999e-06 2024-09-16T03:40:08.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.974 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.974 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871950+0000", 2024-09-16T03:40:08.974 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.177e-06 2024-09-16T03:40:08.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.974 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.871953+0000", 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1389999999999999e-06 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466235+0000", 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.59428207899999996 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466296+0000", 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0498999999999998e-05 2024-09-16T03:40:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.466399+0000", 2024-09-16T03:40:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010310300000000001 2024-09-16T03:40:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5068 4.35 4:aff4cc41:::smithi15736696-11:head [read 548290~2645106,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.911828+0000", 2024-09-16T03:40:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.940918771, 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.65552754499999999, 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5068 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911828+0000", 2024-09-16T03:40:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911828+0000", 2024-09-16T03:40:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911829+0000", 2024-09-16T03:40:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1799999999999995e-07 2024-09-16T03:40:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911831+0000", 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8499999999999998e-06 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911837+0000", 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3929999999999999e-06 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911841+0000", 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8480000000000003e-06 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.565938+0000", 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.65409697600000005 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.566115+0000", 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00017720999999999999 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.567355+0000", 2024-09-16T03:40:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00124065 2024-09-16T03:40:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5069 4.35 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~1413152] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.911846+0000", 2024-09-16T03:40:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.940901195, 2024-09-16T03:40:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.65602458500000005, 2024-09-16T03:40:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5069 2024-09-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911846+0000", 2024-09-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911846+0000", 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911846+0000", 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1300000000000001e-07 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911847+0000", 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.559e-06 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911850+0000", 2024-09-16T03:40:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.903e-06 2024-09-16T03:40:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.011923+0000", 2024-09-16T03:40:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100072736 2024-09-16T03:40:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.567391+0000", 2024-09-16T03:40:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.55546773699999996 2024-09-16T03:40:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.567508+0000", 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117143 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.567870+0000", 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00036219399999999998 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:08.986 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:40:08.987 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T03:40:09.092 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:40:09.092 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:40:09.092 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:40:09.092 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:40:09.101 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T03:40:09.204 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:40:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:40:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:40:09.204 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:40:09.205 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:40:09.213 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T03:40:09.318 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:40:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:40:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:40:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:40:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:45 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'1, mlcod=0'0)", 2024-09-16T03:40:09.318 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:40:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 177.129821803, 2024-09-16T03:40:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102630163, 2024-09-16T03:40:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:40:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186080+0000", 2024-09-16T03:40:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186081+0000", 2024-09-16T03:40:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5600000000000005e-07 2024-09-16T03:40:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186970+0000", 2024-09-16T03:40:09.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00088889000000000004 2024-09-16T03:40:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186979+0000", 2024-09-16T03:40:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0620000000000007e-06 2024-09-16T03:40:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.186986+0000", 2024-09-16T03:40:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6819999999999997e-06 2024-09-16T03:40:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287184+0000", 2024-09-16T03:40:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100198036 2024-09-16T03:40:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287199+0000", 2024-09-16T03:40:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5458999999999999e-05 2024-09-16T03:40:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288659+0000", 2024-09-16T03:40:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0014601900000000001 2024-09-16T03:40:09.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.288711+0000", 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1088000000000003e-05 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:46 4.1 e39/36 4:89839b7e:::smithi15736696-12:head v 39'2, mlcod=0'0)", 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 177.12878537, 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10205370900000001, 2024-09-16T03:40:09.323 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187117+0000", 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.187118+0000", 2024-09-16T03:40:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8599999999999998e-07 2024-09-16T03:40:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.326 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188062+0000", 2024-09-16T03:40:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00094447300000000004 2024-09-16T03:40:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188070+0000", 2024-09-16T03:40:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.695e-06 2024-09-16T03:40:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.188074+0000", 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0709999999999996e-06 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287842+0000", 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099768478999999993 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.287864+0000", 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1679999999999999e-05 2024-09-16T03:40:09.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289137+0000", 2024-09-16T03:40:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012732430000000001 2024-09-16T03:40:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.289171+0000", 2024-09-16T03:40:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3382e-05 2024-09-16T03:40:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.329 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:42 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'2, mlcod=0'0)", 2024-09-16T03:40:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:40:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 177.05870346200001, 2024-09-16T03:40:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10172858, 2024-09-16T03:40:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:40:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257199+0000", 2024-09-16T03:40:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257201+0000", 2024-09-16T03:40:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2709999999999999e-06 2024-09-16T03:40:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257536+0000", 2024-09-16T03:40:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033513899999999998 2024-09-16T03:40:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257555+0000", 2024-09-16T03:40:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8547999999999999e-05 2024-09-16T03:40:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.257566+0000", 2024-09-16T03:40:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1379999999999999e-05 2024-09-16T03:40:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357720+0000", 2024-09-16T03:40:09.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100153532 2024-09-16T03:40:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.357734+0000", 2024-09-16T03:40:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4491000000000001e-05 2024-09-16T03:40:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358896+0000", 2024-09-16T03:40:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011620599999999999 2024-09-16T03:40:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.358927+0000", 2024-09-16T03:40:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1158999999999998e-05 2024-09-16T03:40:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:43 4.5 e39/32 4:aff4cc41:::smithi15736696-11:head v 39'3, mlcod=0'0)", 2024-09-16T03:40:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:40:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 177.05691955, 2024-09-16T03:40:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101764453, 2024-09-16T03:40:09.335 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:40:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258983+0000", 2024-09-16T03:40:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258984+0000", 2024-09-16T03:40:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5689999999999999e-06 2024-09-16T03:40:09.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.258994+0000", 2024-09-16T03:40:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.38e-06 2024-09-16T03:40:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.259008+0000", 2024-09-16T03:40:09.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4151999999999999e-05 2024-09-16T03:40:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359073+0000", 2024-09-16T03:40:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100065628 2024-09-16T03:40:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359110+0000", 2024-09-16T03:40:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7045999999999997e-05 2024-09-16T03:40:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.359124+0000", 2024-09-16T03:40:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.382e-05 2024-09-16T03:40:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360716+0000", 2024-09-16T03:40:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0015919829999999999 2024-09-16T03:40:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:12.360747+0000", 2024-09-16T03:40:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0874999999999998e-05 2024-09-16T03:40:09.340 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1570 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'31, mlcod=48'29)", 2024-09-16T03:40:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:40:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 143.14159265200001, 2024-09-16T03:40:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10253683500000001, 2024-09-16T03:40:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.341 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:40:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:40:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174310+0000", 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8500000000000002e-07 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174689+0000", 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037958600000000001 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174699+0000", 2024-09-16T03:40:09.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0015e-05 2024-09-16T03:40:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174706+0000", 2024-09-16T03:40:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6250000000000001e-06 2024-09-16T03:40:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274865+0000", 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100159157 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.274879+0000", 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3983e-05 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276820+0000", 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001941248 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276846+0000", 2024-09-16T03:40:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5936e-05 2024-09-16T03:40:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1571 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'32, mlcod=48'29)", 2024-09-16T03:40:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:40:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 143.141141898, 2024-09-16T03:40:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10249543, 2024-09-16T03:40:09.347 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:40:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174760+0000", 2024-09-16T03:40:09.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.174761+0000", 2024-09-16T03:40:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0800000000000004e-07 2024-09-16T03:40:09.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175053+0000", 2024-09-16T03:40:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00029151600000000002 2024-09-16T03:40:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175063+0000", 2024-09-16T03:40:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0538e-05 2024-09-16T03:40:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175070+0000", 2024-09-16T03:40:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7819999999999996e-06 2024-09-16T03:40:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275895+0000", 2024-09-16T03:40:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10082516 2024-09-16T03:40:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.275909+0000", 2024-09-16T03:40:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4197e-05 2024-09-16T03:40:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277232+0000", 2024-09-16T03:40:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0013231009999999999 2024-09-16T03:40:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277256+0000", 2024-09-16T03:40:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3428e-05 2024-09-16T03:40:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.357 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:1572 4.15 e60/50 4:ad0b9852:::smithi15736696-16:head v 60'33, mlcod=48'29)", 2024-09-16T03:40:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:40:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 143.140800598, 2024-09-16T03:40:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10216886, 2024-09-16T03:40:09.358 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.359 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:40:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:40:09.360 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175102+0000", 2024-09-16T03:40:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8400000000000004e-07 2024-09-16T03:40:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175107+0000", 2024-09-16T03:40:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.578e-06 2024-09-16T03:40:09.362 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175114+0000", 2024-09-16T03:40:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.888e-06 2024-09-16T03:40:09.363 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.175119+0000", 2024-09-16T03:40:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.074e-06 2024-09-16T03:40:09.364 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276662+0000", 2024-09-16T03:40:09.365 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101542753 2024-09-16T03:40:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.276674+0000", 2024-09-16T03:40:09.366 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1960000000000001e-05 2024-09-16T03:40:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277258+0000", 2024-09-16T03:40:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00058431900000000003 2024-09-16T03:40:09.367 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.277270+0000", 2024-09-16T03:40:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2604e-05 2024-09-16T03:40:09.368 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.369 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2467 4.5 e85/50)", 2024-09-16T03:40:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:40:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.16477111, 2024-09-16T03:40:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102439455, 2024-09-16T03:40:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.370 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:40:09.371 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151131+0000", 2024-09-16T03:40:09.372 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151134+0000", 2024-09-16T03:40:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2030000000000001e-06 2024-09-16T03:40:09.373 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151480+0000", 2024-09-16T03:40:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034613999999999999 2024-09-16T03:40:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.374 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151500+0000", 2024-09-16T03:40:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9582999999999999e-05 2024-09-16T03:40:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.375 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151513+0000", 2024-09-16T03:40:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3349999999999999e-05 2024-09-16T03:40:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.376 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251653+0000", 2024-09-16T03:40:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10014007900000001 2024-09-16T03:40:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.377 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.251679+0000", 2024-09-16T03:40:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5721e-05 2024-09-16T03:40:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253532+0000", 2024-09-16T03:40:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001852701 2024-09-16T03:40:09.378 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253571+0000", 2024-09-16T03:40:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8677999999999998e-05 2024-09-16T03:40:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.379 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2468 4.5 e85/50)", 2024-09-16T03:40:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:40:09.380 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.163302581, 2024-09-16T03:40:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101505239, 2024-09-16T03:40:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.381 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:40:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.382 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152600+0000", 2024-09-16T03:40:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.383 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152601+0000", 2024-09-16T03:40:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6869999999999999e-06 2024-09-16T03:40:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.384 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152719+0000", 2024-09-16T03:40:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000117508 2024-09-16T03:40:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152732+0000", 2024-09-16T03:40:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2952000000000001e-05 2024-09-16T03:40:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152740+0000", 2024-09-16T03:40:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7090000000000002e-06 2024-09-16T03:40:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.385 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252704+0000", 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099964259999999999 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.252727+0000", 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3601e-05 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254075+0000", 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00134772 2024-09-16T03:40:09.386 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254105+0000", 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9802e-05 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2623 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'62, mlcod=85'60)", 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 108.966664951, 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.104590794, 2024-09-16T03:40:09.387 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:40:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:40:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.388 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349237+0000", 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1899999999999999e-07 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349496+0000", 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000258934 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349506+0000", 2024-09-16T03:40:09.389 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9389999999999992e-06 2024-09-16T03:40:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.349513+0000", 2024-09-16T03:40:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6209999999999998e-06 2024-09-16T03:40:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449668+0000", 2024-09-16T03:40:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015550099999999 2024-09-16T03:40:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.390 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.449695+0000", 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6253000000000001e-05 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453784+0000", 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0040896609999999996 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453828+0000", 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3665999999999998e-05 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.391 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2624 4.5 e89/50 4:a4fbca73:::smithi15736696-39:head v 89'63, mlcod=85'60)", 2024-09-16T03:40:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:40:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 108.965317713, 2024-09-16T03:40:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.103286796, 2024-09-16T03:40:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.392 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350585+0000", 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350586+0000", 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0020000000000001e-06 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.393 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350794+0000", 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020824099999999999 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.350805+0000", 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1443999999999999e-05 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450876+0000", 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100070612 2024-09-16T03:40:09.394 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450960+0000", 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4444999999999995e-05 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.450980+0000", 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9728000000000002e-05 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453831+0000", 2024-09-16T03:40:09.395 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002850923 2024-09-16T03:40:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:20.453871+0000", 2024-09-16T03:40:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0401000000000002e-05 2024-09-16T03:40:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:2661 4.15 e91/50 4:aff4cc41:::smithi15736696-11:head v 91'52, mlcod=88'48)", 2024-09-16T03:40:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:40:09.396 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 107.90671748299999, 2024-09-16T03:40:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10164756699999999, 2024-09-16T03:40:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:40:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.397 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409185+0000", 2024-09-16T03:40:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409186+0000", 2024-09-16T03:40:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0499999999999999e-06 2024-09-16T03:40:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409568+0000", 2024-09-16T03:40:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00038196600000000002 2024-09-16T03:40:09.398 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409578+0000", 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.781e-06 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409583+0000", 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3070000000000002e-06 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409633+0000", 2024-09-16T03:40:09.399 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0358000000000001e-05 2024-09-16T03:40:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.409649+0000", 2024-09-16T03:40:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5875999999999998e-05 2024-09-16T03:40:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510800+0000", 2024-09-16T03:40:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101150728 2024-09-16T03:40:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.400 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:21.510832+0000", 2024-09-16T03:40:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2500999999999999e-05 2024-09-16T03:40:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:3970 4.5 e130/110 4:a11e1d7c:::smithi15736696-43:head v 130'83, mlcod=113'81)", 2024-09-16T03:40:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:40:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.053245902, 2024-09-16T03:40:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102913225, 2024-09-16T03:40:09.401 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262656+0000", 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.402 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262657+0000", 2024-09-16T03:40:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.72e-07 2024-09-16T03:40:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262783+0000", 2024-09-16T03:40:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012681199999999999 2024-09-16T03:40:09.403 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262792+0000", 2024-09-16T03:40:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3839999999999997e-06 2024-09-16T03:40:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.262799+0000", 2024-09-16T03:40:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4100000000000002e-06 2024-09-16T03:40:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.404 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.362958+0000", 2024-09-16T03:40:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100158395 2024-09-16T03:40:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.362977+0000", 2024-09-16T03:40:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9741e-05 2024-09-16T03:40:09.405 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.365511+0000", 2024-09-16T03:40:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0025332499999999999 2024-09-16T03:40:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.365570+0000", 2024-09-16T03:40:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8961e-05 2024-09-16T03:40:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.406 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_repop(client.4628.0:4805 4.18 e155/112 4:1f63731d:::smithi15736696-30:head v 155'52, mlcod=128'50)", 2024-09-16T03:40:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:35.524640+0000", 2024-09-16T03:40:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 33.791262056999997, 2024-09-16T03:40:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.104400324, 2024-09-16T03:40:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.407 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.524640+0000", 2024-09-16T03:40:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.524640+0000", 2024-09-16T03:40:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.408 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.524640+0000", 2024-09-16T03:40:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8699999999999999e-07 2024-09-16T03:40:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.526714+0000", 2024-09-16T03:40:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.002073835 2024-09-16T03:40:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.409 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.526725+0000", 2024-09-16T03:40:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1148000000000001e-05 2024-09-16T03:40:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.526734+0000", 2024-09-16T03:40:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6899999999999998e-06 2024-09-16T03:40:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.626909+0000", 2024-09-16T03:40:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100174737 2024-09-16T03:40:09.410 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.626945+0000", 2024-09-16T03:40:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6316000000000003e-05 2024-09-16T03:40:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.629006+0000", 2024-09-16T03:40:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0020604239999999999 2024-09-16T03:40:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.411 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:35.629040+0000", 2024-09-16T03:40:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4987000000000002e-05 2024-09-16T03:40:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5053 4.5 4.3eb87885 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.897511+0000", 2024-09-16T03:40:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.418390823999999, 2024-09-16T03:40:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90556326399999998, 2024-09-16T03:40:09.412 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:40:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5053 2024-09-16T03:40:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.897511+0000", 2024-09-16T03:40:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.413 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.897511+0000", 2024-09-16T03:40:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.897514+0000", 2024-09-16T03:40:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4789999999999999e-06 2024-09-16T03:40:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.414 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.897935+0000", 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00042092099999999999 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.897949+0000", 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4450000000000001e-05 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.897958+0000", 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4759999999999998e-06 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.415 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.901396+0000", 2024-09-16T03:40:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0034385380000000001 2024-09-16T03:40:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.901405+0000", 2024-09-16T03:40:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6209999999999993e-06 2024-09-16T03:40:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803045+0000", 2024-09-16T03:40:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90163974000000002 2024-09-16T03:40:09.416 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803075+0000", 2024-09-16T03:40:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0039000000000002e-05 2024-09-16T03:40:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5054 4.5 4.3eb87885 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:09.417 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.905677+0000", 2024-09-16T03:40:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.410225441, 2024-09-16T03:40:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.89747413799999998, 2024-09-16T03:40:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:40:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5054 2024-09-16T03:40:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.418 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.905677+0000", 2024-09-16T03:40:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.905677+0000", 2024-09-16T03:40:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.419 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.905679+0000", 2024-09-16T03:40:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2299999999999998e-06 2024-09-16T03:40:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.906107+0000", 2024-09-16T03:40:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000428162 2024-09-16T03:40:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.906120+0000", 2024-09-16T03:40:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3176999999999999e-05 2024-09-16T03:40:09.420 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.906129+0000", 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9460000000000004e-06 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.906170+0000", 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0985999999999997e-05 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.906180+0000", 2024-09-16T03:40:09.421 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5489999999999995e-06 2024-09-16T03:40:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803120+0000", 2024-09-16T03:40:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.89694054599999995 2024-09-16T03:40:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803151+0000", 2024-09-16T03:40:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0542e-05 2024-09-16T03:40:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.422 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5055 4.5 4.3eb87885 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.911301+0000", 2024-09-16T03:40:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.404600917, 2024-09-16T03:40:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.99196924200000003, 2024-09-16T03:40:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.423 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:40:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5055 2024-09-16T03:40:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911301+0000", 2024-09-16T03:40:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.424 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911301+0000", 2024-09-16T03:40:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911303+0000", 2024-09-16T03:40:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2810000000000001e-06 2024-09-16T03:40:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.425 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911450+0000", 2024-09-16T03:40:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00014764699999999999 2024-09-16T03:40:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911461+0000", 2024-09-16T03:40:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0713999999999999e-05 2024-09-16T03:40:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911467+0000", 2024-09-16T03:40:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5969999999999999e-06 2024-09-16T03:40:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.426 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911515+0000", 2024-09-16T03:40:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8841000000000002e-05 2024-09-16T03:40:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911524+0000", 2024-09-16T03:40:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3299999999999999e-06 2024-09-16T03:40:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.903256+0000", 2024-09-16T03:40:09.427 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.99173228800000002 2024-09-16T03:40:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.903271+0000", 2024-09-16T03:40:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4544e-05 2024-09-16T03:40:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5056 4.5 4.3eb87885 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.911478+0000", 2024-09-16T03:40:09.428 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.404424683, 2024-09-16T03:40:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.99183876800000004, 2024-09-16T03:40:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:40:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5056 2024-09-16T03:40:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.429 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911478+0000", 2024-09-16T03:40:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911478+0000", 2024-09-16T03:40:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911478+0000", 2024-09-16T03:40:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6799999999999999e-07 2024-09-16T03:40:09.430 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911482+0000", 2024-09-16T03:40:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6210000000000001e-06 2024-09-16T03:40:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911488+0000", 2024-09-16T03:40:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9610000000000002e-06 2024-09-16T03:40:09.431 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911493+0000", 2024-09-16T03:40:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2859999999999999e-06 2024-09-16T03:40:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911559+0000", 2024-09-16T03:40:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5555999999999994e-05 2024-09-16T03:40:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:09.432 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911565+0000", 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2360000000000001e-06 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.903307+0000", 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.99174187800000002 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.903316+0000", 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4620000000000002e-06 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.433 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:5057 4.5 4.3eb87885 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e169)", 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:53.911500+0000", 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 15.404401933999999, 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.092075873, 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 5057 2024-09-16T03:40:09.434 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911500+0000", 2024-09-16T03:40:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911500+0000", 2024-09-16T03:40:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.435 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911501+0000", 2024-09-16T03:40:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7400000000000003e-07 2024-09-16T03:40:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911503+0000", 2024-09-16T03:40:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4140000000000001e-06 2024-09-16T03:40:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911509+0000", 2024-09-16T03:40:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3179999999999997e-06 2024-09-16T03:40:09.436 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911516+0000", 2024-09-16T03:40:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.2490000000000004e-06 2024-09-16T03:40:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911583+0000", 2024-09-16T03:40:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6742999999999994e-05 2024-09-16T03:40:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.437 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:53.911587+0000", 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1670000000000001e-06 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:55.003562+0000", 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.091974939 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:55.003576+0000", 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4469e-05 2024-09-16T03:40:09.438 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_backoff(4.5 ack-block id 1 [4:a0000000::::0,4:a4000000::::head) e170)", 2024-09-16T03:40:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:54.803413+0000", 2024-09-16T03:40:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 14.512488729999999, 2024-09-16T03:40:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20039902400000001, 2024-09-16T03:40:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:40:09.439 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 0 2024-09-16T03:40:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803413+0000", 2024-09-16T03:40:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.440 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803413+0000", 2024-09-16T03:40:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803415+0000", 2024-09-16T03:40:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8199999999999999e-06 2024-09-16T03:40:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803419+0000", 2024-09-16T03:40:09.441 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.1810000000000003e-06 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803429+0000", 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7620000000000007e-06 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:54.803438+0000", 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9060000000000008e-06 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.442 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:55.003803+0000", 2024-09-16T03:40:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20036456899999999 2024-09-16T03:40:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:55.003813+0000", 2024-09-16T03:40:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7860000000000008e-06 2024-09-16T03:40:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.443 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:40:09.444 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:40:09.445 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T03:40:09.549 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:40:09.549 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:40:09.549 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:40:09.550 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:40:09.558 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T03:40:09.661 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:40:09.661 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:40:09.661 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:40:09.661 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:40:09.661 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:40:09.670 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T03:40:09.776 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:40:09.776 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:40:09.776 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:40:09.777 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:40:09.777 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.777 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T03:40:09.777 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:40:09.777 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 198.69682655899999, 2024-09-16T03:40:09.777 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.88203620400000005, 2024-09-16T03:40:09.777 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.777 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.777 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.777 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.777 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.777 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.778 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1 2024-09-16T03:40:09.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.778 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.778 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:40:09.778 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.778 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.778 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.778 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076765+0000", 2024-09-16T03:40:09.779 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.779 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.779 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.779 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.779 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076767+0000", 2024-09-16T03:40:09.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.9300000000000002e-06 2024-09-16T03:40:09.780 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.780 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.780 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.780 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076774+0000", 2024-09-16T03:40:09.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6630000000000004e-06 2024-09-16T03:40:09.780 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.780 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.780 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.780 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076785+0000", 2024-09-16T03:40:09.780 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1221e-05 2024-09-16T03:40:09.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.781 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.781 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.781 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076796+0000", 2024-09-16T03:40:09.781 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0428000000000001e-05 2024-09-16T03:40:09.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.781 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.781 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.781 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076856+0000", 2024-09-16T03:40:09.781 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0034999999999997e-05 2024-09-16T03:40:09.781 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.781 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:09.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.076863+0000", 2024-09-16T03:40:09.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.9840000000000004e-06 2024-09-16T03:40:09.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855770+0000", 2024-09-16T03:40:09.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.77890736999999999 2024-09-16T03:40:09.782 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.782 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.782 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.782 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.855902+0000", 2024-09-16T03:40:09.782 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013207 2024-09-16T03:40:09.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.856054+0000", 2024-09-16T03:40:09.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00015220899999999999 2024-09-16T03:40:09.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.783 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.783 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.858407+0000", 2024-09-16T03:40:09.783 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0023530069999999998 2024-09-16T03:40:09.783 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.783 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958598+0000", 2024-09-16T03:40:09.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100190194 2024-09-16T03:40:09.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958716+0000", 2024-09-16T03:40:09.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118589 2024-09-16T03:40:09.784 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.784 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.784 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.784 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958761+0000", 2024-09-16T03:40:09.784 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5120000000000002e-05 2024-09-16T03:40:09.785 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.785 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.785 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:51.958802+0000", 2024-09-16T03:40:09.785 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0383999999999998e-05 2024-09-16T03:40:09.785 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.785 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.785 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.785 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.785 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.785 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T03:40:09.785 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 195.17841038200001, 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70508991300000001, 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 52 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.786 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595182+0000", 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6300000000000005e-07 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.787 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595187+0000", 2024-09-16T03:40:09.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9790000000000001e-06 2024-09-16T03:40:09.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595194+0000", 2024-09-16T03:40:09.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6490000000000002e-06 2024-09-16T03:40:09.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.788 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.788 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.788 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.595208+0000", 2024-09-16T03:40:09.788 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4139e-05 2024-09-16T03:40:09.788 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801931+0000", 2024-09-16T03:40:09.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.206723083 2024-09-16T03:40:09.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.789 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.801990+0000", 2024-09-16T03:40:09.789 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8625999999999997e-05 2024-09-16T03:40:09.789 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.789 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.789 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802066+0000", 2024-09-16T03:40:09.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6553999999999999e-05 2024-09-16T03:40:09.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:54.802504+0000", 2024-09-16T03:40:09.790 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00043775200000000002 2024-09-16T03:40:09.790 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.790 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.790 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.790 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.199998+0000", 2024-09-16T03:40:09.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.39749365599999997 2024-09-16T03:40:09.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300151+0000", 2024-09-16T03:40:09.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10015310099999999 2024-09-16T03:40:09.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.791 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.791 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300210+0000", 2024-09-16T03:40:09.791 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9376999999999997e-05 2024-09-16T03:40:09.791 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.791 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.792 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.792 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:55.300271+0000", 2024-09-16T03:40:09.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1334000000000006e-05 2024-09-16T03:40:09.792 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.792 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.792 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.792 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.792 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.792 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:71 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.792 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:40:09.792 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 193.403519399, 2024-09-16T03:40:09.792 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60623559900000001, 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 71 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.793 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.794 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:40:09.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.794 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.794 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370073+0000", 2024-09-16T03:40:09.794 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4799999999999999e-07 2024-09-16T03:40:09.794 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.794 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.794 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.794 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370077+0000", 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9940000000000002e-06 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370081+0000", 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5850000000000001e-06 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370087+0000", 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6829999999999996e-06 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.795 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673522+0000", 2024-09-16T03:40:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.303434969 2024-09-16T03:40:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673567+0000", 2024-09-16T03:40:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5080000000000002e-05 2024-09-16T03:40:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673651+0000", 2024-09-16T03:40:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3652999999999993e-05 2024-09-16T03:40:09.796 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775115+0000", 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101464178 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.975918+0000", 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20080318699999999 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976254+0000", 2024-09-16T03:40:09.797 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00033619200000000002 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976285+0000", 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0679000000000002e-05 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976308+0000", 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3051e-05 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.798 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.799 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.799 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:72 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.799 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:40:09.799 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 193.403496039, 2024-09-16T03:40:09.799 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60632925100000001, 2024-09-16T03:40:09.799 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.799 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.799 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.799 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.799 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.799 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.799 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 72 2024-09-16T03:40:09.799 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.800 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.800 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.800 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:40:09.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.800 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.800 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.800 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:40:09.800 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.800 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.800 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.800 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370096+0000", 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5199999999999998e-07 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370150+0000", 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3906999999999997e-05 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370157+0000", 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7259999999999996e-06 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.801 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370162+0000", 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7249999999999997e-06 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.673969+0000", 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30380713199999998 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674030+0000", 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1141000000000003e-05 2024-09-16T03:40:09.802 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.674124+0000", 2024-09-16T03:40:09.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3859999999999999e-05 2024-09-16T03:40:09.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775687+0000", 2024-09-16T03:40:09.803 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10156293299999999 2024-09-16T03:40:09.803 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.803 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.803 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.803 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976103+0000", 2024-09-16T03:40:09.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20041631400000001 2024-09-16T03:40:09.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976355+0000", 2024-09-16T03:40:09.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000251611 2024-09-16T03:40:09.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.804 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.804 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976385+0000", 2024-09-16T03:40:09.804 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0883999999999998e-05 2024-09-16T03:40:09.804 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.804 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.805 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.805 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976425+0000", 2024-09-16T03:40:09.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9666000000000003e-05 2024-09-16T03:40:09.805 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.805 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.805 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.805 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.805 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.805 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:73 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.805 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:40:09.805 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 193.40342355499999, 2024-09-16T03:40:09.805 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60763126999999995, 2024-09-16T03:40:09.805 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.806 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.806 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.806 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.806 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.806 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.806 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 73 2024-09-16T03:40:09.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.806 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.806 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.806 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.806 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:40:09.806 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.806 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370168+0000", 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370169+0000", 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.6199999999999998e-07 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370171+0000", 2024-09-16T03:40:09.807 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4030000000000001e-06 2024-09-16T03:40:09.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.370175+0000", 2024-09-16T03:40:09.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.596e-06 2024-09-16T03:40:09.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.808 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.808 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470295+0000", 2024-09-16T03:40:09.808 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100119967 2024-09-16T03:40:09.808 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.808 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775148+0000", 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30485311799999998 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775203+0000", 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4592999999999997e-05 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.775293+0000", 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0859999999999994e-05 2024-09-16T03:40:09.809 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976223+0000", 2024-09-16T03:40:09.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20092996599999999 2024-09-16T03:40:09.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976463+0000", 2024-09-16T03:40:09.810 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000239695 2024-09-16T03:40:09.810 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.810 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.810 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.810 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977730+0000", 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012667690000000001 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977773+0000", 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2604000000000001e-05 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977800+0000", 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7137000000000001e-05 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.811 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.812 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.812 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.812 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.812 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:40:09.812 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 193.30328470000001, 2024-09-16T03:40:09.812 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60633810099999996, 2024-09-16T03:40:09.812 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.812 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.812 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.812 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.812 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.812 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.812 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 74 2024-09-16T03:40:09.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.813 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.813 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:40:09.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.813 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.813 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470307+0000", 2024-09-16T03:40:09.813 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.813 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.813 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470308+0000", 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0950000000000001e-06 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470313+0000", 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0200000000000002e-06 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470320+0000", 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5089999999999999e-06 2024-09-16T03:40:09.814 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470327+0000", 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3010000000000001e-06 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976483+0000", 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50615579499999996 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976533+0000", 2024-09-16T03:40:09.815 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0096000000000003e-05 2024-09-16T03:40:09.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.816 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.976643+0000", 2024-09-16T03:40:09.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000109481 2024-09-16T03:40:09.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.816 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.816 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977808+0000", 2024-09-16T03:40:09.816 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0011649869999999999 2024-09-16T03:40:09.816 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.816 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.981699+0000", 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.003891579 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076496+0000", 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.094797333999999997 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076583+0000", 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6571999999999997e-05 2024-09-16T03:40:09.817 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.076645+0000", 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2331999999999999e-05 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 193.303254565, 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71251765099999997, 2024-09-16T03:40:09.818 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.819 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.819 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.819 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.819 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.819 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.819 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 75 2024-09-16T03:40:09.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.819 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.819 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.819 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.819 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:40:09.819 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.819 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470337+0000", 2024-09-16T03:40:09.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470338+0000", 2024-09-16T03:40:09.820 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2500000000000005e-07 2024-09-16T03:40:09.820 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.820 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.820 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.820 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470389+0000", 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.134e-05 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.470397+0000", 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8140000000000001e-06 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570537+0000", 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100139425 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.821 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.822 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977018+0000", 2024-09-16T03:40:09.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40648102899999999 2024-09-16T03:40:09.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.822 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.822 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.822 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977063+0000", 2024-09-16T03:40:09.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5247000000000002e-05 2024-09-16T03:40:09.822 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.822 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.822 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.822 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.977143+0000", 2024-09-16T03:40:09.822 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0054000000000003e-05 2024-09-16T03:40:09.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.978132+0000", 2024-09-16T03:40:09.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00098896399999999995 2024-09-16T03:40:09.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.823 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182725+0000", 2024-09-16T03:40:09.823 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.204592626 2024-09-16T03:40:09.823 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.823 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.823 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182783+0000", 2024-09-16T03:40:09.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8285000000000002e-05 2024-09-16T03:40:09.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182825+0000", 2024-09-16T03:40:09.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2398999999999998e-05 2024-09-16T03:40:09.824 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.824 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.824 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.824 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.182855+0000", 2024-09-16T03:40:09.824 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9842999999999999e-05 2024-09-16T03:40:09.824 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.825 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.825 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.825 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.825 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.825 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.825 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:40:09.825 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 193.20302799800001, 2024-09-16T03:40:09.825 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.71263271800000005, 2024-09-16T03:40:09.825 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.825 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.825 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.825 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 76 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570564+0000", 2024-09-16T03:40:09.826 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.827 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.827 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.827 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.827 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570566+0000", 2024-09-16T03:40:09.827 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1299999999999999e-06 2024-09-16T03:40:09.827 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.827 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.827 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.827 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570577+0000", 2024-09-16T03:40:09.827 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0672999999999999e-05 2024-09-16T03:40:09.827 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.827 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570593+0000", 2024-09-16T03:40:09.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5979999999999999e-05 2024-09-16T03:40:09.828 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.828 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.570604+0000", 2024-09-16T03:40:09.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.077e-05 2024-09-16T03:40:09.828 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.828 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.828 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.828 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979890+0000", 2024-09-16T03:40:09.828 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40928624200000002 2024-09-16T03:40:09.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.979945+0000", 2024-09-16T03:40:09.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5651e-05 2024-09-16T03:40:09.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.829 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980038+0000", 2024-09-16T03:40:09.829 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2603999999999996e-05 2024-09-16T03:40:09.829 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.829 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.829 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:56.980669+0000", 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00063121900000000003 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283050+0000", 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30238120800000001 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283132+0000", 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1841999999999993e-05 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.830 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.831 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.831 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283168+0000", 2024-09-16T03:40:09.831 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5438999999999997e-05 2024-09-16T03:40:09.831 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.831 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.831 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.831 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:57.283197+0000", 2024-09-16T03:40:09.831 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8960000000000001e-05 2024-09-16T03:40:09.831 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.831 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.831 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.831 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.831 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.832 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:94 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.832 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:40:09.832 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 191.388492899, 2024-09-16T03:40:09.832 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.70799990400000001, 2024-09-16T03:40:09.832 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.832 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.832 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.832 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.832 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.832 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.832 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 94 2024-09-16T03:40:09.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.833 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:40:09.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.833 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.833 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.833 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385099+0000", 2024-09-16T03:40:09.833 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.833 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385100+0000", 2024-09-16T03:40:09.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9599999999999998e-07 2024-09-16T03:40:09.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385104+0000", 2024-09-16T03:40:09.834 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7579999999999998e-06 2024-09-16T03:40:09.834 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.834 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.834 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.834 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385128+0000", 2024-09-16T03:40:09.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4700999999999999e-05 2024-09-16T03:40:09.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385139+0000", 2024-09-16T03:40:09.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0587e-05 2024-09-16T03:40:09.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.835 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.835 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787499+0000", 2024-09-16T03:40:09.835 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40235976099999998 2024-09-16T03:40:09.835 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.835 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787545+0000", 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6418999999999997e-05 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.787623+0000", 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7897000000000003e-05 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888605+0000", 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100982321 2024-09-16T03:40:09.836 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990721+0000", 2024-09-16T03:40:09.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102115922 2024-09-16T03:40:09.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.837 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093019+0000", 2024-09-16T03:40:09.837 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10229788500000001 2024-09-16T03:40:09.837 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.837 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.837 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093051+0000", 2024-09-16T03:40:09.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2255999999999997e-05 2024-09-16T03:40:09.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.838 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.838 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093099+0000", 2024-09-16T03:40:09.838 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7500999999999998e-05 2024-09-16T03:40:09.838 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.838 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.838 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.838 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.838 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.839 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.839 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:40:09.839 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 191.38844157599999, 2024-09-16T03:40:09.839 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80404594699999998, 2024-09-16T03:40:09.839 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.839 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.839 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.839 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.839 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.839 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.839 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 95 2024-09-16T03:40:09.839 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.839 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:40:09.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385150+0000", 2024-09-16T03:40:09.840 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.840 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.840 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.840 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.840 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385151+0000", 2024-09-16T03:40:09.841 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0599999999999999e-07 2024-09-16T03:40:09.841 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.841 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.841 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385165+0000", 2024-09-16T03:40:09.841 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4175999999999999e-05 2024-09-16T03:40:09.841 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.841 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.841 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.841 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385175+0000", 2024-09-16T03:40:09.841 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5410000000000006e-06 2024-09-16T03:40:09.841 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385183+0000", 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8650000000000001e-06 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888643+0000", 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50345988200000003 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888695+0000", 2024-09-16T03:40:09.842 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2834999999999998e-05 2024-09-16T03:40:09.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.888802+0000", 2024-09-16T03:40:09.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010697899999999999 2024-09-16T03:40:09.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.843 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990658+0000", 2024-09-16T03:40:09.843 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185585699999999 2024-09-16T03:40:09.843 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.843 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.843 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990917+0000", 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00025853600000000002 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189054+0000", 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198136812 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189135+0000", 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.1482000000000004e-05 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.844 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189196+0000", 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1175999999999998e-05 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 191.38840094899999, 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80412488400000004, 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.845 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.846 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.846 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.846 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.846 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.846 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 96 2024-09-16T03:40:09.846 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.846 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.846 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.846 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.846 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:40:09.846 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.846 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385191+0000", 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385192+0000", 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3200000000000004e-07 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385267+0000", 2024-09-16T03:40:09.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5535000000000001e-05 2024-09-16T03:40:09.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385277+0000", 2024-09-16T03:40:09.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6239999999999997e-06 2024-09-16T03:40:09.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.848 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.848 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385284+0000", 2024-09-16T03:40:09.848 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4800000000000004e-06 2024-09-16T03:40:09.848 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.848 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989283+0000", 2024-09-16T03:40:09.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60399849500000002 2024-09-16T03:40:09.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989410+0000", 2024-09-16T03:40:09.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000127298 2024-09-16T03:40:09.849 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.849 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.849 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.849 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.989615+0000", 2024-09-16T03:40:09.849 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000205045 2024-09-16T03:40:09.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991036+0000", 2024-09-16T03:40:09.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.001421042 2024-09-16T03:40:09.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.850 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189228+0000", 2024-09-16T03:40:09.850 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19819188500000001 2024-09-16T03:40:09.850 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.850 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.850 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189249+0000", 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0996999999999999e-05 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189293+0000", 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3912000000000002e-05 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189316+0000", 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2739e-05 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.851 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.852 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.852 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.852 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.852 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.852 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:40:09.852 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 191.38829721799999, 2024-09-16T03:40:09.852 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.80642264100000005, 2024-09-16T03:40:09.852 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.852 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.852 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.852 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.852 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 97 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385295+0000", 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.853 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385296+0000", 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.006e-06 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385301+0000", 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2630000000000003e-06 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.385308+0000", 2024-09-16T03:40:09.854 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6030000000000001e-06 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485459+0000", 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100150929 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990238+0000", 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50477992000000005 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.855 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990287+0000", 2024-09-16T03:40:09.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8863999999999998e-05 2024-09-16T03:40:09.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.856 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.856 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.990377+0000", 2024-09-16T03:40:09.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9900000000000003e-05 2024-09-16T03:40:09.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.856 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.856 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.856 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093109+0000", 2024-09-16T03:40:09.856 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102732055 2024-09-16T03:40:09.856 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.189351+0000", 2024-09-16T03:40:09.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.096241766000000006 2024-09-16T03:40:09.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.857 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191570+0000", 2024-09-16T03:40:09.857 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00221915 2024-09-16T03:40:09.857 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.857 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.857 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191653+0000", 2024-09-16T03:40:09.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.3046000000000004e-05 2024-09-16T03:40:09.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.858 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.858 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.191717+0000", 2024-09-16T03:40:09.858 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4139000000000004e-05 2024-09-16T03:40:09.858 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.858 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.858 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.858 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.858 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.858 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.859 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:40:09.859 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 191.28810676099999, 2024-09-16T03:40:09.859 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90688294999999997, 2024-09-16T03:40:09.859 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.859 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.859 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.859 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.859 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.859 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.859 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 98 2024-09-16T03:40:09.859 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.859 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.859 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:40:09.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485485+0000", 2024-09-16T03:40:09.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.860 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.860 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.860 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.860 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485487+0000", 2024-09-16T03:40:09.860 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1799999999999999e-06 2024-09-16T03:40:09.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485498+0000", 2024-09-16T03:40:09.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0648999999999999e-05 2024-09-16T03:40:09.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.861 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.861 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485514+0000", 2024-09-16T03:40:09.861 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5943999999999998e-05 2024-09-16T03:40:09.861 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.861 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485525+0000", 2024-09-16T03:40:09.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.065e-05 2024-09-16T03:40:09.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991102+0000", 2024-09-16T03:40:09.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50557748400000002 2024-09-16T03:40:09.862 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.862 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.862 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.862 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991161+0000", 2024-09-16T03:40:09.862 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8801999999999997e-05 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991255+0000", 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3936000000000003e-05 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093112+0000", 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10185709499999999 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.863 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291911+0000", 2024-09-16T03:40:09.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19879897699999999 2024-09-16T03:40:09.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.864 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392167+0000", 2024-09-16T03:40:09.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10025634999999999 2024-09-16T03:40:09.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.864 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.864 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392249+0000", 2024-09-16T03:40:09.864 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2175000000000005e-05 2024-09-16T03:40:09.864 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.864 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.865 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.865 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392368+0000", 2024-09-16T03:40:09.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118708 2024-09-16T03:40:09.865 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.865 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.865 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.865 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.865 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.865 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.865 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:40:09.865 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 191.28805313800001, 2024-09-16T03:40:09.865 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.90707203199999997, 2024-09-16T03:40:09.865 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 99 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.866 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485539+0000", 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485540+0000", 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.449e-06 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485661+0000", 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000120835 2024-09-16T03:40:09.867 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485680+0000", 2024-09-16T03:40:09.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8885999999999998e-05 2024-09-16T03:40:09.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485693+0000", 2024-09-16T03:40:09.868 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3297e-05 2024-09-16T03:40:09.868 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.868 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.868 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.868 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991546+0000", 2024-09-16T03:40:09.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.505853098 2024-09-16T03:40:09.869 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.869 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.869 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991598+0000", 2024-09-16T03:40:09.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1921e-05 2024-09-16T03:40:09.869 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.869 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.869 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.991681+0000", 2024-09-16T03:40:09.869 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2502000000000004e-05 2024-09-16T03:40:09.869 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.869 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.093115+0000", 2024-09-16T03:40:09.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10143424299999999 2024-09-16T03:40:09.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.291977+0000", 2024-09-16T03:40:09.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.198862172 2024-09-16T03:40:09.870 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.870 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.870 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.870 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392541+0000", 2024-09-16T03:40:09.870 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100563758 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392579+0000", 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7796999999999997e-05 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392611+0000", 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2073999999999998e-05 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.871 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.872 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.872 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.872 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:40:09.872 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 191.28787424500001, 2024-09-16T03:40:09.872 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.0073149939999999, 2024-09-16T03:40:09.872 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.872 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.872 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.872 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.872 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.872 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.872 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 100 2024-09-16T03:40:09.872 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.873 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:40:09.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.873 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.873 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:40:09.873 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.873 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.873 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485718+0000", 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.7400000000000003e-07 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485721+0000", 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.4530000000000001e-06 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485726+0000", 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.0590000000000002e-06 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:58.485729+0000", 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.9409999999999999e-06 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092342+0000", 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60661343700000003 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092469+0000", 2024-09-16T03:40:09.875 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012658799999999999 2024-09-16T03:40:09.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.092671+0000", 2024-09-16T03:40:09.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020206 2024-09-16T03:40:09.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.876 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392452+0000", 2024-09-16T03:40:09.876 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29978163699999999 2024-09-16T03:40:09.876 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.876 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.876 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.392486+0000", 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3736999999999999e-05 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492812+0000", 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100325621 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.492901+0000", 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8702000000000006e-05 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.877 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.878 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.878 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:36:59.493033+0000", 2024-09-16T03:40:09.878 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000132185 2024-09-16T03:40:09.878 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.878 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.878 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.878 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.878 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.878 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.878 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:40:09.879 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 189.11272372400001, 2024-09-16T03:40:09.879 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60874581500000002, 2024-09-16T03:40:09.879 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.879 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.879 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.879 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.879 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.879 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.879 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 123 2024-09-16T03:40:09.879 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.879 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.879 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.879 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:40:09.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660868+0000", 2024-09-16T03:40:09.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.880 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.880 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.880 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.880 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660869+0000", 2024-09-16T03:40:09.880 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9400000000000005e-07 2024-09-16T03:40:09.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660921+0000", 2024-09-16T03:40:09.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2312000000000003e-05 2024-09-16T03:40:09.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.881 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660927+0000", 2024-09-16T03:40:09.881 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9329999999999998e-06 2024-09-16T03:40:09.881 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.881 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.881 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660931+0000", 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.9539999999999998e-06 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.967991+0000", 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30706032700000002 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968039+0000", 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7477e-05 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.882 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.968117+0000", 2024-09-16T03:40:09.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8589000000000003e-05 2024-09-16T03:40:09.883 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067678+0000", 2024-09-16T03:40:09.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099560406000000004 2024-09-16T03:40:09.883 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.883 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.883 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.883 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170175+0000", 2024-09-16T03:40:09.883 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102496927 2024-09-16T03:40:09.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.884 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269491+0000", 2024-09-16T03:40:09.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099316205000000005 2024-09-16T03:40:09.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.884 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269574+0000", 2024-09-16T03:40:09.884 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2903000000000002e-05 2024-09-16T03:40:09.884 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.884 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.884 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269614+0000", 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.0188000000000002e-05 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4260.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 189.112655026, 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60879245500000001, 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.885 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:09.886 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.886 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4260", 2024-09-16T03:40:09.886 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.64:0/2211237099", 2024-09-16T03:40:09.886 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 124 2024-09-16T03:40:09.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.886 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.886 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:40:09.886 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.886 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.886 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.886 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660937+0000", 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660938+0000", 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8599999999999998e-07 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660940+0000", 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2529999999999999e-06 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.887 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660943+0000", 2024-09-16T03:40:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.01e-06 2024-09-16T03:40:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:00.660946+0000", 2024-09-16T03:40:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8159999999999998e-06 2024-09-16T03:40:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012417+0000", 2024-09-16T03:40:09.888 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.35147120599999998 2024-09-16T03:40:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012497+0000", 2024-09-16T03:40:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9963000000000007e-05 2024-09-16T03:40:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,5", 2024-09-16T03:40:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.012585+0000", 2024-09-16T03:40:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.8214999999999995e-05 2024-09-16T03:40:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.889 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.067680+0000", 2024-09-16T03:40:09.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.055095203000000002 2024-09-16T03:40:09.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.890 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.170243+0000", 2024-09-16T03:40:09.890 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10256290799999999 2024-09-16T03:40:09.890 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.890 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.890 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269662+0000", 2024-09-16T03:40:09.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099418896000000007 2024-09-16T03:40:09.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:09.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269701+0000", 2024-09-16T03:40:09.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8829999999999999e-05 2024-09-16T03:40:09.891 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.891 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.891 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.891 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:01.269729+0000", 2024-09-16T03:40:09.891 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8569e-05 2024-09-16T03:40:09.891 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1550 4.13 4:c8492bfc:::smithi15736696-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 144.106926886, 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.62706974999999998, 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:09.892 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:09.893 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1550 2024-09-16T03:40:09.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.893 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:40:09.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.893 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.893 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.893 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:40:09.893 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.893 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666665+0000", 2024-09-16T03:40:09.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.0699999999999998e-07 2024-09-16T03:40:09.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666669+0000", 2024-09-16T03:40:09.894 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.393e-06 2024-09-16T03:40:09.894 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.894 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.894 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.894 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666673+0000", 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6770000000000003e-06 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.666682+0000", 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6510000000000007e-06 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.913940+0000", 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.24725763000000001 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.895 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:09.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:45.914034+0000", 2024-09-16T03:40:09.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3923000000000002e-05 2024-09-16T03:40:09.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293650+0000", 2024-09-16T03:40:09.896 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.37961662200000001 2024-09-16T03:40:09.896 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.896 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.896 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.896 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293688+0000", 2024-09-16T03:40:09.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8096000000000003e-05 2024-09-16T03:40:09.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.897 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.897 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:46.293735+0000", 2024-09-16T03:40:09.897 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6451e-05 2024-09-16T03:40:09.897 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.897 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.897 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.897 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.897 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.897 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.15.23:77 3.2 3:51e5f63d:::2aed316cd042a6b5a5637c70adde372bf85ddfb25c9988c63c3b89909dc045dc:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e73)", 2024-09-16T03:40:09.897 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:40:09.898 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 129.69505026499999, 2024-09-16T03:40:09.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.61452436499999996, 2024-09-16T03:40:09.898 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.898 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.898 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "reached pg", 2024-09-16T03:40:09.898 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.898 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:40:09.898 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.898 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.898 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.898 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078542+0000", 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078543+0000", 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.553e-06 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078551+0000", 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.0790000000000001e-06 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.899 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.078564+0000", 2024-09-16T03:40:09.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2943e-05 2024-09-16T03:40:09.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178635+0000", 2024-09-16T03:40:09.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10007044599999999 2024-09-16T03:40:09.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.900 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.900 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.900 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178684+0000", 2024-09-16T03:40:09.900 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9592999999999998e-05 2024-09-16T03:40:09.900 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178747+0000", 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2286000000000006e-05 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 0,4,5", 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.178857+0000", 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000110807 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280364+0000", 2024-09-16T03:40:09.901 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101506518 2024-09-16T03:40:09.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:09.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.280373+0000", 2024-09-16T03:40:09.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.6899999999999998e-06 2024-09-16T03:40:09.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:09.902 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.369249+0000", 2024-09-16T03:40:09.902 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.088876172000000003 2024-09-16T03:40:09.902 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.902 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.902 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693044+0000", 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.32379486499999999 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:00.693066+0000", 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2413e-05 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3119 4.13 4:c8492bfc:::smithi15736696-42:head [tier-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e104)", 2024-09-16T03:40:09.903 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 92.842153209000003, 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.64286237700000004, 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3119 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:09.904 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:40:09.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:09.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931439+0000", 2024-09-16T03:40:09.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:09.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.905 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:09.905 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931441+0000", 2024-09-16T03:40:09.905 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.3599999999999999e-06 2024-09-16T03:40:09.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.905 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:09.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931449+0000", 2024-09-16T03:40:09.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4080000000000001e-06 2024-09-16T03:40:09.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:09.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931463+0000", 2024-09-16T03:40:09.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.488e-05 2024-09-16T03:40:09.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.906 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.906 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:09.906 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:36.931473+0000", 2024-09-16T03:40:09.906 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.8109999999999997e-06 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.031604+0000", 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10013079499999999 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.031668+0000", 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.4226999999999993e-05 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:09.907 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:37.574301+0000", 2024-09-16T03:40:09.908 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.54263289599999998 2024-09-16T03:40:09.908 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.908 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.908 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.908 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:09.908 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:09.908 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:40:09.909 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T03:40:10.013 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:40:10.013 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:40:10.013 INFO:teuthology.orchestra.run.smithi157.stdout: "num_ops": 0 2024-09-16T03:40:10.013 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:40:10.021 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T03:40:10.125 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:40:10.125 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [], 2024-09-16T03:40:10.125 INFO:teuthology.orchestra.run.smithi157.stdout: "complaint_time": 30, 2024-09-16T03:40:10.125 INFO:teuthology.orchestra.run.smithi157.stdout: "num_blocked_ops": 0 2024-09-16T03:40:10.125 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:40:10.133 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T03:40:10.239 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-09-16T03:40:10.239 INFO:teuthology.orchestra.run.smithi157.stdout: "size": 20, 2024-09-16T03:40:10.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 600, 2024-09-16T03:40:10.239 INFO:teuthology.orchestra.run.smithi157.stdout: "ops": [ 2024-09-16T03:40:10.239 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.239 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:465 4.5 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T03:40:10.239 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:40:10.239 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 169.22037126000001, 2024-09-16T03:40:10.239 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60529649299999999, 2024-09-16T03:40:10.239 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.239 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.240 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:10.240 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.240 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.240 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.240 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 465 2024-09-16T03:40:10.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.240 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.240 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.240 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:40:10.240 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.240 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.240 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015705+0000", 2024-09-16T03:40:10.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015706+0000", 2024-09-16T03:40:10.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.091e-06 2024-09-16T03:40:10.241 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.241 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.241 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.241 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015712+0000", 2024-09-16T03:40:10.241 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.1999999999999999e-06 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015724+0000", 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1408e-05 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.015731+0000", 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.4830000000000002e-06 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.242 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216049+0000", 2024-09-16T03:40:10.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200317895 2024-09-16T03:40:10.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.216178+0000", 2024-09-16T03:40:10.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012848800000000001 2024-09-16T03:40:10.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.243 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:40:10.243 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620010+0000", 2024-09-16T03:40:10.243 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.40383207199999999 2024-09-16T03:40:10.243 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.243 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:10.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620710+0000", 2024-09-16T03:40:10.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00069988400000000001 2024-09-16T03:40:10.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620902+0000", 2024-09-16T03:40:10.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00019197800000000001 2024-09-16T03:40:10.244 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.244 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.244 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.244 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620933+0000", 2024-09-16T03:40:10.244 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1513999999999997e-05 2024-09-16T03:40:10.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:10.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.620968+0000", 2024-09-16T03:40:10.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.4524999999999999e-05 2024-09-16T03:40:10.245 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.245 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.245 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.245 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:21.621002+0000", 2024-09-16T03:40:10.245 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3955000000000003e-05 2024-09-16T03:40:10.245 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.245 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.245 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(osd.4.22:10 3.3 3:deac805c:::smithi15736696-44 low_tier:head [call cas.chunk_put_ref in=16b] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 163.82887628399999, 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53191748100000003, 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:40:10.246 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407200+0000", 2024-09-16T03:40:10.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.247 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407202+0000", 2024-09-16T03:40:10.247 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.855e-06 2024-09-16T03:40:10.247 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.247 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.247 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407208+0000", 2024-09-16T03:40:10.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9819999999999996e-06 2024-09-16T03:40:10.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407220+0000", 2024-09-16T03:40:10.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2191999999999999e-05 2024-09-16T03:40:10.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.248 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.248 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.248 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.407236+0000", 2024-09-16T03:40:10.248 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5698000000000001e-05 2024-09-16T03:40:10.248 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538167+0000", 2024-09-16T03:40:10.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.13093111499999999 2024-09-16T03:40:10.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.249 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538229+0000", 2024-09-16T03:40:10.249 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2441999999999996e-05 2024-09-16T03:40:10.249 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.249 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.249 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,8", 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.538346+0000", 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011716400000000001 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.540757+0000", 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0024105089999999999 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.639501+0000", 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.098743545000000002 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.250 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939038+0000", 2024-09-16T03:40:10.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29953765700000001 2024-09-16T03:40:10.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:10.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939075+0000", 2024-09-16T03:40:10.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6600999999999997e-05 2024-09-16T03:40:10.251 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.251 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.251 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.251 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:26.939118+0000", 2024-09-16T03:40:10.251 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2721e-05 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:984 4.15 4:ad0b9852:::smithi15736696-16:head [assert-version v6,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 159.86395635599999, 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.97836159599999994, 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.252 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.253 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.253 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 984 2024-09-16T03:40:10.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.253 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:40:10.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.253 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.253 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.253 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.253 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372120+0000", 2024-09-16T03:40:10.253 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372122+0000", 2024-09-16T03:40:10.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.934e-06 2024-09-16T03:40:10.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.254 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372129+0000", 2024-09-16T03:40:10.254 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.8029999999999999e-06 2024-09-16T03:40:10.254 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.254 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.254 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.372141+0000", 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2006e-05 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472234+0000", 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100092985 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472318+0000", 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.4257000000000003e-05 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.255 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:10.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.472326+0000", 2024-09-16T03:40:10.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.8960000000000003e-06 2024-09-16T03:40:10.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350293+0000", 2024-09-16T03:40:10.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.87796699899999997 2024-09-16T03:40:10.256 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.256 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.256 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.256 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350433+0000", 2024-09-16T03:40:10.256 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00013999600000000001 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350482+0000", 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.8720000000000001e-05 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1000 4.15 4:aff4cc41:::smithi15736696-11:head [read 0~2873964,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 159.64551727099999, 2024-09-16T03:40:10.257 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.761707841, 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1000 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.258 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590559+0000", 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590560+0000", 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.5999999999999999e-07 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590562+0000", 2024-09-16T03:40:10.259 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.17e-06 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590568+0000", 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.7939999999999997e-06 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590572+0000", 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1300000000000001e-06 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.260 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590619+0000", 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7148999999999999e-05 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590625+0000", 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5679999999999996e-06 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350858+0000", 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76023295000000002 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.261 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.350973+0000", 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000115023 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352267+0000", 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0012935970000000001 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1001 4.15 4:aff4cc41:::smithi15736696-11:head [read 1254674~1619290,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:40:10.262 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:40:10.263 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 159.645498686, 2024-09-16T03:40:10.263 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76275827799999996, 2024-09-16T03:40:10.263 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.263 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.263 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:10.263 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.263 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.263 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.263 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1001 2024-09-16T03:40:10.263 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.263 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.263 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.263 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590578+0000", 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.3000000000000002e-07 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.264 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590580+0000", 2024-09-16T03:40:10.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.024e-06 2024-09-16T03:40:10.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590584+0000", 2024-09-16T03:40:10.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8800000000000001e-06 2024-09-16T03:40:10.265 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.265 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.265 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.265 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590587+0000", 2024-09-16T03:40:10.265 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6630000000000002e-06 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590650+0000", 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3950999999999998e-05 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590654+0000", 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.89e-06 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352309+0000", 2024-09-16T03:40:10.266 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76165495699999997 2024-09-16T03:40:10.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.352407+0000", 2024-09-16T03:40:10.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.7188999999999994e-05 2024-09-16T03:40:10.267 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.267 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.267 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.267 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.353336+0000", 2024-09-16T03:40:10.267 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00092939400000000005 2024-09-16T03:40:10.267 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.267 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.267 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.268 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.268 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.268 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:1002 4.15 4:aff4cc41:::smithi15736696-11:head [sparse-read 0~1527025] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e50)", 2024-09-16T03:40:10.268 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:40:10.268 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 159.645469432, 2024-09-16T03:40:10.268 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85963523600000002, 2024-09-16T03:40:10.268 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.268 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.268 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:10.268 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.268 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.268 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.268 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 1002 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.269 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590607+0000", 2024-09-16T03:40:10.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.15e-07 2024-09-16T03:40:10.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590609+0000", 2024-09-16T03:40:10.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.606e-06 2024-09-16T03:40:10.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.270 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.270 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.270 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590615+0000", 2024-09-16T03:40:10.270 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3040000000000001e-06 2024-09-16T03:40:10.270 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590618+0000", 2024-09-16T03:40:10.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.309e-06 2024-09-16T03:40:10.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590669+0000", 2024-09-16T03:40:10.271 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.1048000000000003e-05 2024-09-16T03:40:10.271 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.271 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.271 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for peered", 2024-09-16T03:40:10.271 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:30.590673+0000", 2024-09-16T03:40:10.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.061e-06 2024-09-16T03:40:10.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.272 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.272 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.449932+0000", 2024-09-16T03:40:10.272 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.85925979699999999 2024-09-16T03:40:10.272 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.272 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.272 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450032+0000", 2024-09-16T03:40:10.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.9858000000000005e-05 2024-09-16T03:40:10.273 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.273 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.273 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.273 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:37:31.450242+0000", 2024-09-16T03:40:10.273 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020993800000000001 2024-09-16T03:40:10.273 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2467 4.5 4:a11e1d7c:::smithi15736696-43:head [write 1772395~538068] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 114.244793061, 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.56288137400000005, 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.274 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.275 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2467 2024-09-16T03:40:10.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.275 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:40:10.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.275 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.275 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.275 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991283+0000", 2024-09-16T03:40:10.275 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.275 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.276 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.276 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991285+0000", 2024-09-16T03:40:10.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.0990000000000001e-06 2024-09-16T03:40:10.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.276 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.276 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991447+0000", 2024-09-16T03:40:10.276 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016170099999999999 2024-09-16T03:40:10.276 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.276 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.276 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991458+0000", 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.128e-05 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.991464+0000", 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3800000000000002e-06 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150823+0000", 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15935943799999999 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.277 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.150899+0000", 2024-09-16T03:40:10.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5359999999999997e-05 2024-09-16T03:40:10.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:40:10.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.151007+0000", 2024-09-16T03:40:10.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000108317 2024-09-16T03:40:10.278 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.278 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.278 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:10.278 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253992+0000", 2024-09-16T03:40:10.278 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.102985249 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.353797+0000", 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.099805039999999998 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554039+0000", 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20024175899999999 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:10.279 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554129+0000", 2024-09-16T03:40:10.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.0067000000000004e-05 2024-09-16T03:40:10.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.280 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.280 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554165+0000", 2024-09-16T03:40:10.280 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5683999999999999e-05 2024-09-16T03:40:10.280 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.280 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.280 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.280 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.280 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.280 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2468 4.5 4:a11e1d7c:::smithi15736696-43:head [write 2900946~403005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:10.280 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:40:10.281 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 114.23937228200001, 2024-09-16T03:40:10.281 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.55765803899999999, 2024-09-16T03:40:10.281 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.281 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.281 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:10.281 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.281 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.281 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.281 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2468 2024-09-16T03:40:10.281 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.281 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.281 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.281 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:40:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996704+0000", 2024-09-16T03:40:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996708+0000", 2024-09-16T03:40:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.5149999999999998e-06 2024-09-16T03:40:10.282 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996921+0000", 2024-09-16T03:40:10.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000213692 2024-09-16T03:40:10.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.283 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996937+0000", 2024-09-16T03:40:10.283 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.5699999999999999e-05 2024-09-16T03:40:10.283 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.283 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.283 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996946+0000", 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.7159999999999992e-06 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152362+0000", 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.15541682800000001 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152421+0000", 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8569000000000001e-05 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.284 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:40:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.152515+0000", 2024-09-16T03:40:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3659999999999994e-05 2024-09-16T03:40:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253995+0000", 2024-09-16T03:40:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10148022499999999 2024-09-16T03:40:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.453908+0000", 2024-09-16T03:40:10.285 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19991299700000001 2024-09-16T03:40:10.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554321+0000", 2024-09-16T03:40:10.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100412977 2024-09-16T03:40:10.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.286 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:10.286 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554344+0000", 2024-09-16T03:40:10.286 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.262e-05 2024-09-16T03:40:10.286 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.286 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554362+0000", 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.8539999999999999e-05 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2469 4.5 4:a11e1d7c:::smithi15736696-43:head [setxattr _header (56) in=7b,truncate 3529470] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 114.239116919, 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.75785074799999996, 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.287 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.288 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:10.288 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.288 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.288 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.288 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2469 2024-09-16T03:40:10.288 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.288 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.288 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.288 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.288 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:40:10.288 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.288 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.288 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.289 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.289 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996959+0000", 2024-09-16T03:40:10.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.289 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.289 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996960+0000", 2024-09-16T03:40:10.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3000000000000001e-07 2024-09-16T03:40:10.289 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.289 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.289 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.289 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996963+0000", 2024-09-16T03:40:10.289 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8679999999999999e-06 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996967+0000", 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.622e-06 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996971+0000", 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.1429999999999998e-06 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.290 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253536+0000", 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25656582900000002 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253632+0000", 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6067e-05 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253745+0000", 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00011256699999999999 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.291 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:10.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.254218+0000", 2024-09-16T03:40:10.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00047321400000000001 2024-09-16T03:40:10.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.554397+0000", 2024-09-16T03:40:10.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30017854399999999 2024-09-16T03:40:10.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.292 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.292 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.292 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754557+0000", 2024-09-16T03:40:10.292 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.200159848 2024-09-16T03:40:10.292 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:10.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754646+0000", 2024-09-16T03:40:10.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9370999999999995e-05 2024-09-16T03:40:10.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.293 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.293 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.293 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.754810+0000", 2024-09-16T03:40:10.293 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00016414500000000001 2024-09-16T03:40:10.293 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.293 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.293 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.293 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2470 4.5 4:a11e1d7c:::smithi15736696-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:40:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 114.23910059000001, 2024-09-16T03:40:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.76022711799999998, 2024-09-16T03:40:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2470 2024-09-16T03:40:10.294 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.295 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.295 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.295 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:40:10.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.295 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.295 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:40:10.295 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.295 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.295 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.295 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996976+0000", 2024-09-16T03:40:10.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.65e-07 2024-09-16T03:40:10.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996977+0000", 2024-09-16T03:40:10.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4079999999999999e-06 2024-09-16T03:40:10.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.296 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.296 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.296 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996981+0000", 2024-09-16T03:40:10.296 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.546e-06 2024-09-16T03:40:10.296 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:15.996988+0000", 2024-09-16T03:40:10.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.0890000000000002e-06 2024-09-16T03:40:10.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253947+0000", 2024-09-16T03:40:10.297 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.25695931100000002 2024-09-16T03:40:10.297 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.297 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.297 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:10.297 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.253985+0000", 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7939999999999999e-05 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757090+0000", 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50310477499999995 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757140+0000", 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.9567999999999997e-05 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.298 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757203+0000", 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3215999999999999e-05 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2487 4.15 4:ad0b9852:::smithi15736696-16:head [write 1947841~685395] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.82005599999999, 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.74140762199999999, 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:10.299 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2487 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416020+0000", 2024-09-16T03:40:10.300 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416023+0000", 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2170000000000001e-06 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416341+0000", 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00031881699999999999 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416353+0000", 2024-09-16T03:40:10.301 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.189e-05 2024-09-16T03:40:10.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.416360+0000", 2024-09-16T03:40:10.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.5719999999999999e-06 2024-09-16T03:40:10.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.302 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755578+0000", 2024-09-16T03:40:10.302 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33921859199999999 2024-09-16T03:40:10.302 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.302 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.302 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755639+0000", 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0615000000000002e-05 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.755735+0000", 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.5798999999999995e-05 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858633+0000", 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10289859599999999 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.303 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858695+0000", 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2000000000000003e-05 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157315+0000", 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29861931600000002 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157375+0000", 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0268e-05 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.304 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.305 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.305 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157428+0000", 2024-09-16T03:40:10.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.2939999999999998e-05 2024-09-16T03:40:10.305 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.305 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.305 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.305 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.305 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.305 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2488 4.15 4:ad0b9852:::smithi15736696-16:head [write 3383215~196676] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:10.305 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:40:10.305 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.81438335, 2024-09-16T03:40:10.305 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73589615900000005, 2024-09-16T03:40:10.305 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2488 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.306 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421693+0000", 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421695+0000", 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.1469999999999999e-06 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421975+0000", 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00028008299999999998 2024-09-16T03:40:10.307 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421987+0000", 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.1294000000000001e-05 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.421992+0000", 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9349999999999999e-06 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757205+0000", 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.33521242800000001 2024-09-16T03:40:10.308 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757269+0000", 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3761000000000002e-05 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.757376+0000", 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00010706599999999999 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858684+0000", 2024-09-16T03:40:10.309 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.101308255 2024-09-16T03:40:10.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.310 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.310 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:10.310 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858700+0000", 2024-09-16T03:40:10.310 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.6138999999999999e-05 2024-09-16T03:40:10.310 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.311 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.311 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157491+0000", 2024-09-16T03:40:10.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.29879056999999998 2024-09-16T03:40:10.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:10.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157546+0000", 2024-09-16T03:40:10.312 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.5668999999999998e-05 2024-09-16T03:40:10.312 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.312 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.312 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.312 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157589+0000", 2024-09-16T03:40:10.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.2812000000000003e-05 2024-09-16T03:40:10.313 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.313 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.313 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.313 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.313 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.313 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2489 4.15 4:ad0b9852:::smithi15736696-16:head [setxattr _header (56) in=7b,truncate 3579891] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:10.313 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:40:10.313 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.81407452800001, 2024-09-16T03:40:10.313 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73575981700000004, 2024-09-16T03:40:10.313 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.313 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.313 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2489 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.314 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422002+0000", 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.0200000000000002e-07 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422005+0000", 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.7379999999999999e-06 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.315 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422009+0000", 2024-09-16T03:40:10.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.6219999999999998e-06 2024-09-16T03:40:10.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422016+0000", 2024-09-16T03:40:10.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5789999999999998e-06 2024-09-16T03:40:10.316 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.316 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.316 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.316 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858071+0000", 2024-09-16T03:40:10.316 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.436054302 2024-09-16T03:40:10.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858130+0000", 2024-09-16T03:40:10.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9493000000000001e-05 2024-09-16T03:40:10.317 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.317 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.317 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 7,11", 2024-09-16T03:40:10.317 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858226+0000", 2024-09-16T03:40:10.317 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.6174000000000004e-05 2024-09-16T03:40:10.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:40:10.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.10106483199999999 2024-09-16T03:40:10.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:10.318 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959639+0000", 2024-09-16T03:40:10.318 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00034802999999999999 2024-09-16T03:40:10.318 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.318 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.318 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157640+0000", 2024-09-16T03:40:10.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.19800045299999999 2024-09-16T03:40:10.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:10.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157702+0000", 2024-09-16T03:40:10.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.2218000000000006e-05 2024-09-16T03:40:10.319 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.319 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.319 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.319 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157762+0000", 2024-09-16T03:40:10.319 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.9774000000000001e-05 2024-09-16T03:40:10.320 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.320 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.320 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.320 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.320 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.320 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2490 4.15 4:ad0b9852:::smithi15736696-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:10.320 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:40:10.320 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.814055936, 2024-09-16T03:40:10.320 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.73597824999999994, 2024-09-16T03:40:10.320 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.320 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.320 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:10.320 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.321 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.321 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.321 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2490 2024-09-16T03:40:10.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.321 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:40:10.321 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.321 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.321 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.321 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.321 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422020+0000", 2024-09-16T03:40:10.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422021+0000", 2024-09-16T03:40:10.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8099999999999999e-07 2024-09-16T03:40:10.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.322 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.322 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.322 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422022+0000", 2024-09-16T03:40:10.322 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.573e-06 2024-09-16T03:40:10.322 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422027+0000", 2024-09-16T03:40:10.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.3250000000000001e-06 2024-09-16T03:40:10.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422030+0000", 2024-09-16T03:40:10.323 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.948e-06 2024-09-16T03:40:10.323 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.323 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.323 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.323 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858510+0000", 2024-09-16T03:40:10.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.43648032599999997 2024-09-16T03:40:10.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:10.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.858588+0000", 2024-09-16T03:40:10.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7888999999999999e-05 2024-09-16T03:40:10.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.324 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.324 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.324 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157803+0000", 2024-09-16T03:40:10.324 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.299215642 2024-09-16T03:40:10.324 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157893+0000", 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.9565000000000001e-05 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:17.157999+0000", 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000105701 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.325 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.326 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2492 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:10.326 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:40:10.326 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.814043427, 2024-09-16T03:40:10.326 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53721099400000005, 2024-09-16T03:40:10.326 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.326 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.326 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:10.326 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.326 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.326 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.326 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2492 2024-09-16T03:40:10.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.327 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:40:10.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.327 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.327 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.327 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.327 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:40:10.327 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422033+0000", 2024-09-16T03:40:10.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.8000000000000002e-07 2024-09-16T03:40:10.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.328 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.328 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422036+0000", 2024-09-16T03:40:10.328 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.5900000000000002e-06 2024-09-16T03:40:10.328 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.328 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422043+0000", 2024-09-16T03:40:10.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3069999999999997e-06 2024-09-16T03:40:10.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422048+0000", 2024-09-16T03:40:10.329 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.6619999999999996e-06 2024-09-16T03:40:10.329 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.329 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.329 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.329 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958774+0000", 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53672593499999999 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.958868+0000", 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.4408999999999997e-05 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959244+0000", 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00037581099999999998 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.330 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.331 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.331 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.331 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2493 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 0~1508050] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:10.331 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:40:10.331 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.814023648, 2024-09-16T03:40:10.331 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53757571299999996, 2024-09-16T03:40:10.331 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.331 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.331 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:10.331 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.331 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.331 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.331 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2493 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.332 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422053+0000", 2024-09-16T03:40:10.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.3600000000000004e-07 2024-09-16T03:40:10.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422055+0000", 2024-09-16T03:40:10.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.7460000000000001e-06 2024-09-16T03:40:10.333 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.333 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.333 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.333 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422058+0000", 2024-09-16T03:40:10.333 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.2600000000000001e-06 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422061+0000", 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6759999999999999e-06 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959291+0000", 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53723057900000004 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959367+0000", 2024-09-16T03:40:10.334 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.5158000000000001e-05 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959628+0000", 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00026175800000000001 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:2494 4.5 4:a4fbca73:::smithi15736696-39:head [sparse-read 1738052~782998] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 113.81401261800001, 2024-09-16T03:40:10.335 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53787640199999998, 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 2494 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.336 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:40:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422064+0000", 2024-09-16T03:40:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6100000000000002e-07 2024-09-16T03:40:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.337 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422065+0000", 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.2839999999999999e-06 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422069+0000", 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.7239999999999998e-06 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.422072+0000", 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.6560000000000001e-06 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.338 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959661+0000", 2024-09-16T03:40:10.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.53758982600000005 2024-09-16T03:40:10.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959739+0000", 2024-09-16T03:40:10.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.7581000000000001e-05 2024-09-16T03:40:10.339 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.339 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.339 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.339 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:38:16.959940+0000", 2024-09-16T03:40:10.339 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00020107 2024-09-16T03:40:10.339 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3987 4.36 4:6ef3b53d:::smithi15736696-48:head [setxattr _header (56) in=7b,truncate 2168169] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.859958732999999, 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.50651914200000003, 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.340 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3987 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376118+0000", 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.341 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376119+0000", 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.3799999999999996e-07 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376122+0000", 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.202e-06 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376251+0000", 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00012896699999999999 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.342 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376259+0000", 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.32e-06 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379368+0000", 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0031084989999999998 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379441+0000", 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.3331999999999995e-05 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.343 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 2,10", 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.379559+0000", 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.000118068 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380350+0000", 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00079103699999999997 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.580657+0000", 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.20030706600000001 2024-09-16T03:40:10.344 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882497+0000", 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.30183998899999998 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882560+0000", 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.3372000000000002e-05 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.882637+0000", 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.6352000000000004e-05 2024-09-16T03:40:10.345 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:3988 4.36 4:6ef3b53d:::smithi15736696-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e130)", 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 64.859804885000003, 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60662574199999997, 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "started", 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 3988 2024-09-16T03:40:10.346 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376271+0000", 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376273+0000", 2024-09-16T03:40:10.347 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.3170000000000001e-06 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376278+0000", 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.7849999999999999e-06 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376286+0000", 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 8.2009999999999999e-06 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.348 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.376291+0000", 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.4029999999999998e-06 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380020+0000", 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.0037289630000000001 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for rw locks", 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.380065+0000", 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4666999999999998e-05 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.349 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982800+0000", 2024-09-16T03:40:10.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.60273539099999995 2024-09-16T03:40:10.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982838+0000", 2024-09-16T03:40:10.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8155999999999999e-05 2024-09-16T03:40:10.350 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.350 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.350 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.350 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:05.982897+0000", 2024-09-16T03:40:10.350 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 5.8859e-05 2024-09-16T03:40:10.350 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.351 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.351 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.351 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.351 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.351 INFO:teuthology.orchestra.run.smithi157.stdout: "description": "osd_op(client.4628.0:4506 4.1a 4:5a5e9bc2:::smithi15736696-36:head [assert-version v129,set-chunk] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e146)", 2024-09-16T03:40:10.351 INFO:teuthology.orchestra.run.smithi157.stdout: "initiated_at": "2024-09-16T03:39:25.237761+0000", 2024-09-16T03:40:10.351 INFO:teuthology.orchestra.run.smithi157.stdout: "age": 44.998314956999998, 2024-09-16T03:40:10.351 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.51319717499999995, 2024-09-16T03:40:10.351 INFO:teuthology.orchestra.run.smithi157.stdout: "continuous": false, 2024-09-16T03:40:10.351 INFO:teuthology.orchestra.run.smithi157.stdout: "type_data": { 2024-09-16T03:40:10.351 INFO:teuthology.orchestra.run.smithi157.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T03:40:10.351 INFO:teuthology.orchestra.run.smithi157.stdout: "client_info": { 2024-09-16T03:40:10.351 INFO:teuthology.orchestra.run.smithi157.stdout: "client": "client.4628", 2024-09-16T03:40:10.352 INFO:teuthology.orchestra.run.smithi157.stdout: "client_addr": "v1:172.21.15.157:0/1215086814", 2024-09-16T03:40:10.352 INFO:teuthology.orchestra.run.smithi157.stdout: "tid": 4506 2024-09-16T03:40:10.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.352 INFO:teuthology.orchestra.run.smithi157.stdout: "events": [ 2024-09-16T03:40:10.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "initiated", 2024-09-16T03:40:10.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237761+0000", 2024-09-16T03:40:10.352 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.352 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.352 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.352 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "header_read", 2024-09-16T03:40:10.352 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237761+0000", 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "throttled", 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237764+0000", 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 2.2819999999999999e-06 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "all_read", 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237772+0000", 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 7.9929999999999995e-06 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.353 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "dispatched", 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237786+0000", 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 1.4508999999999999e-05 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "queued_for_pg", 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237795+0000", 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 9.2070000000000001e-06 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "reached_pg", 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237845+0000", 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.952e-05 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "started", 2024-09-16T03:40:10.354 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.237911+0000", 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 6.6434999999999997e-05 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "waiting for subops from 6,11", 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.649713+0000", 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.41180149399999999 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "op_commit", 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.650287+0000", 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00057374399999999995 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.355 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.650476+0000", 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.00018991900000000001 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "sub_op_commit_rec", 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.750876+0000", 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 0.100399355 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "commit_sent", 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.750914+0000", 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 3.8226999999999998e-05 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-09-16T03:40:10.356 INFO:teuthology.orchestra.run.smithi157.stdout: { 2024-09-16T03:40:10.357 INFO:teuthology.orchestra.run.smithi157.stdout: "event": "done", 2024-09-16T03:40:10.357 INFO:teuthology.orchestra.run.smithi157.stdout: "time": "2024-09-16T03:39:25.750959+0000", 2024-09-16T03:40:10.357 INFO:teuthology.orchestra.run.smithi157.stdout: "duration": 4.4490000000000003e-05 2024-09-16T03:40:10.357 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.357 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.357 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.357 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-09-16T03:40:10.357 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2024-09-16T03:40:10.357 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-09-16T03:40:11.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T03:40:11.508 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:40:11.505+0000 7fd101e6f640 -1 osd.2 185 handle_osd_map injecting map crc failure 2024-09-16T03:40:12.035 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:40:12.034+0000 7f4214d04640 -1 osd.8 185 handle_osd_map injecting map crc failure 2024-09-16T03:40:12.382 INFO:teuthology.orchestra.run.smithi064.stderr:noscrub is unset 2024-09-16T03:40:12.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T03:40:12.452 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-16T03:40:12.452 INFO:tasks.thrashosds.thrasher:Adding osd 13 2024-09-16T03:40:12.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 13 2024-09-16T03:40:12.768 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-09-16T03:40:12.766+0000 7fdb04aef640 -1 mon.c@2(leader).osd e187 definitely_dead 0 2024-09-16T03:40:13.462 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:40:13.460+0000 7ff266366640 -1 osd.11 187 handle_osd_map injecting map crc failure 2024-09-16T03:40:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:nodeep-scrub is unset 2024-09-16T03:40:13.463 INFO:teuthology.orchestra.run.smithi064.stderr:marked in osd.13. 2024-09-16T03:40:13.481 INFO:tasks.thrashosds.thrasher:Added osd 13 2024-09-16T03:40:13.481 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-16T03:40:13.481 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-16T03:40:13.481 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-09-16T03:40:13.573 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:40:13.572+0000 7f12e2a1a640 -1 osd.5 187 handle_osd_map injecting map crc failure 2024-09-16T03:40:13.841 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-09-16T03:40:13.839+0000 7fdb04aef640 -1 mon.c@2(leader).osd e188 definitely_dead 0 2024-09-16T03:40:14.463 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:40:14.461+0000 7f12e4a1e640 -1 osd.5 188 handle_osd_map injecting map crc failure 2024-09-16T03:40:14.464 INFO:teuthology.orchestra.run.smithi064.stderr:marked in osd.5. 2024-09-16T03:40:14.482 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-16T03:40:14.482 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-16T03:40:14.578 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:40:14.576+0000 7f3d40708640 -1 osd.12 188 handle_osd_map injecting map crc failure 2024-09-16T03:40:14.604 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a: {} 2024-09-16T03:40:14.604 INFO:teuthology.orchestra.run.smithi064.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T03:40:14.614 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b: {} 2024-09-16T03:40:14.614 INFO:teuthology.orchestra.run.smithi064.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T03:40:14.624 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c: {} 2024-09-16T03:40:14.624 INFO:teuthology.orchestra.run.smithi064.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T03:40:14.635 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T03:40:15.470 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T03:40:15.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-16T03:40:15.716 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:40:15.714+0000 7efef16d9640 -1 osd.13 189 handle_osd_map injecting map crc failure 2024-09-16T03:40:16.470 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:40:16.468+0000 7f0ae954f640 -1 osd.0 190 handle_osd_map injecting map crc failure 2024-09-16T03:40:16.471 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-16T03:40:16.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T03:40:17.486 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T03:40:17.490 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:40:17.488+0000 7f6528e68640 -1 osd.1 191 handle_osd_map injecting map crc failure 2024-09-16T03:40:17.503 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-16T03:40:17.979 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:40:17.977+0000 7f8be56cd640 -1 osd.7 191 handle_osd_map injecting map crc failure 2024-09-16T03:40:18.499 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:40:18.497+0000 7ff26836a640 -1 osd.11 192 handle_osd_map injecting map crc failure 2024-09-16T03:40:18.500 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-16T03:40:18.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-16T03:40:18.544 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:40:18.542+0000 7fa3886e6640 -1 osd.10 192 handle_osd_map injecting map crc failure 2024-09-16T03:40:19.502 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T03:40:19.520 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-16T03:40:19.535 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:40:19.532+0000 7f9288166640 -1 osd.14 193 handle_osd_map injecting map crc failure 2024-09-16T03:40:19.679 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:40:19.678+0000 7f66f3205640 -1 osd.15 193 handle_osd_map injecting map crc failure 2024-09-16T03:40:20.515 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-16T03:40:20.531 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-16T03:40:20.537 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:40:20.535+0000 7ff266366640 -1 osd.11 194 handle_osd_map injecting map crc failure 2024-09-16T03:40:21.382 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.12 to 1 (10000) 2024-09-16T03:40:21.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-09-16T03:40:21.736 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:40:21.733+0000 7f9288166640 -1 osd.14 195 handle_osd_map injecting map crc failure 2024-09-16T03:40:22.510 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:40:22.507+0000 7f3d4270c640 -1 osd.12 196 handle_osd_map injecting map crc failure 2024-09-16T03:40:22.512 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-09-16T03:40:22.529 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T03:40:23.517 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:40:23.515+0000 7f12e4a1e640 -1 osd.5 197 handle_osd_map injecting map crc failure 2024-09-16T03:40:23.519 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T03:40:23.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-16T03:40:23.716 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:40:23.715+0000 7ff26836a640 -1 osd.11 197 handle_osd_map injecting map crc failure 2024-09-16T03:40:24.086 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:40:24.084+0000 7efef16d9640 -1 osd.13 197 handle_osd_map injecting map crc failure 2024-09-16T03:40:24.528 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T03:40:24.542 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T03:40:24.709 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:40:24.707+0000 7f66f3205640 -1 osd.15 198 handle_osd_map injecting map crc failure 2024-09-16T03:40:25.535 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:40:25.533+0000 7f12e4a1e640 -1 osd.5 198 handle_osd_map injecting map crc failure 2024-09-16T03:40:25.536 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T03:40:25.552 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-16T03:40:26.544 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-16T03:40:26.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-16T03:40:26.714 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:40:26.712+0000 7fedb5739640 -1 osd.9 200 handle_osd_map injecting map crc failure 2024-09-16T03:40:27.544 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T03:40:27.557 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:40:27.555+0000 7f9288166640 -1 osd.14 201 handle_osd_map injecting map crc failure 2024-09-16T03:40:27.557 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:40:27.555+0000 7fa3886e6640 -1 osd.10 201 handle_osd_map injecting map crc failure 2024-09-16T03:40:27.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-16T03:40:28.549 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-16T03:40:28.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-16T03:40:29.555 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.13 to 1 (10000) 2024-09-16T03:40:29.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-09-16T03:40:30.568 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-09-16T03:40:30.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T03:40:30.958 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:40:30.956+0000 7f12e2a1a640 -1 osd.5 203 handle_osd_map injecting map crc failure 2024-09-16T03:40:31.247 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:40:31.245+0000 7f4212d00640 -1 osd.8 204 handle_osd_map injecting map crc failure 2024-09-16T03:40:31.381 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T03:40:31.396 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-16T03:40:31.539 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:40:31.536+0000 7f6528e68640 -1 osd.1 205 handle_osd_map injecting map crc failure 2024-09-16T03:40:31.539 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:40:31.536+0000 7fd101e6f640 -1 osd.2 204 handle_osd_map injecting map crc failure 2024-09-16T03:40:32.209 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:40:32.207+0000 7f3a1f71e640 -1 osd.6 205 handle_osd_map injecting map crc failure 2024-09-16T03:40:32.381 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:40:32.379+0000 7ff26836a640 -1 osd.11 205 handle_osd_map injecting map crc failure 2024-09-16T03:40:32.381 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:40:32.379+0000 7f49439d4640 -1 osd.4 205 handle_osd_map injecting map crc failure 2024-09-16T03:40:32.383 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T03:40:32.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T03:40:33.576 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T03:40:33.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-16T03:40:33.740 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:40:33.738+0000 7fa3886e6640 -1 osd.10 207 handle_osd_map injecting map crc failure 2024-09-16T03:40:34.591 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:40:34.590+0000 7f3a21722640 -1 osd.6 208 handle_osd_map injecting map crc failure 2024-09-16T03:40:34.591 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-16T03:40:34.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-16T03:40:35.594 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T03:40:35.611 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-16T03:40:36.603 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-16T03:40:36.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-16T03:40:36.884 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:40:36.882+0000 7f4212d00640 -1 osd.8 210 handle_osd_map injecting map crc failure 2024-09-16T03:40:37.615 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.14 to 1 (10000) 2024-09-16T03:40:37.633 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-09-16T03:40:38.622 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:40:38.620+0000 7f928a16a640 -1 osd.14 211 handle_osd_map injecting map crc failure 2024-09-16T03:40:38.624 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-09-16T03:40:38.641 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T03:40:39.041 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:40:39.039+0000 7f49419d0640 -1 osd.4 211 handle_osd_map injecting map crc failure 2024-09-16T03:40:39.627 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:40:39.625+0000 7f3d4270c640 -1 osd.12 213 handle_osd_map injecting map crc failure 2024-09-16T03:40:39.628 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:40:39.626+0000 7efef16d9640 -1 osd.13 212 handle_osd_map injecting map crc failure 2024-09-16T03:40:39.629 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T03:40:39.645 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-16T03:40:40.636 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:40:40.635+0000 7efef16d9640 -1 osd.13 214 handle_osd_map injecting map crc failure 2024-09-16T03:40:40.638 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:40:40.636+0000 7f3d4270c640 -1 osd.12 214 handle_osd_map injecting map crc failure 2024-09-16T03:40:40.639 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-16T03:40:40.655 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T03:40:40.807 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:40:40.806+0000 7fd101e6f640 -1 osd.2 213 handle_osd_map injecting map crc failure 2024-09-16T03:40:41.301 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:40:41.300+0000 7fa3886e6640 -1 osd.10 214 handle_osd_map injecting map crc failure 2024-09-16T03:40:41.301 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:40:41.299+0000 7f4212d00640 -1 osd.8 214 handle_osd_map injecting map crc failure 2024-09-16T03:40:41.386 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T03:40:41.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-16T03:40:41.415 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:40:41.413+0000 7f6528e68640 -1 osd.1 214 handle_osd_map injecting map crc failure 2024-09-16T03:40:41.486 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:40:41.484+0000 7f49419d0640 -1 osd.4 215 handle_osd_map injecting map crc failure 2024-09-16T03:40:42.642 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-16T03:40:42.658 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-16T03:40:42.838 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:40:42.837+0000 7f3a1f71e640 -1 osd.6 216 handle_osd_map injecting map crc failure 2024-09-16T03:40:43.241 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:40:43.238+0000 7ff266366640 -1 osd.11 216 handle_osd_map injecting map crc failure 2024-09-16T03:40:43.645 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:40:43.643+0000 7f12e4a1e640 -1 osd.5 217 handle_osd_map injecting map crc failure 2024-09-16T03:40:43.647 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T03:40:43.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-16T03:40:43.939 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:40:43.937+0000 7eff7884b640 -1 osd.3 216 handle_osd_map injecting map crc failure 2024-09-16T03:40:43.940 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:40:43.937+0000 7f66f3205640 -1 osd.15 216 handle_osd_map injecting map crc failure 2024-09-16T03:40:44.649 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:40:44.647+0000 7f6528e68640 -1 osd.1 217 handle_osd_map injecting map crc failure 2024-09-16T03:40:44.650 INFO:tasks.ceph.osd.1.smithi112.stderr:2024-09-16T03:40:44.649+0000 7f652ae6c640 -1 osd.1 217 handle_osd_map injecting map crc failure 2024-09-16T03:40:44.652 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-16T03:40:44.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-16T03:40:44.986 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:40:44.983+0000 7eff7884b640 -1 osd.3 218 handle_osd_map injecting map crc failure 2024-09-16T03:40:45.658 INFO:tasks.ceph.osd.15.smithi157.stderr:2024-09-16T03:40:45.655+0000 7f66f3205640 -1 osd.15 219 handle_osd_map injecting map crc failure 2024-09-16T03:40:45.661 INFO:teuthology.orchestra.run.smithi064.stderr:reweighted osd.15 to 1 (10000) 2024-09-16T03:40:45.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-09-16T03:40:46.385 INFO:teuthology.orchestra.run.smithi064.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-09-16T03:40:46.400 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-16T03:40:46.400 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-16T03:40:46.401 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-16T03:40:46.401 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-16T03:40:46.401 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T03:40:46.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:40:46.702 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:46.702 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":221,"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","created":"2024-09-16T03:36:37.310421+0000","modified":"2024-09-16T03:40:46.380109+0000","last_up_change":"2024-09-16T03:36:50.058111+0000","last_in_change":"2024-09-16T03:40:14.459006+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:36:49.696307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.789999961853027,"score_stable":15.789999961853027,"optimal_score":0.18999999761581421,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:36:51.904455+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":5.9699997901916504,"score_stable":5.9699997901916504,"optimal_score":0.87999999523162842,"raw_score_acting":5.25,"raw_score_stable":5.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-09-16T03:37:02.999795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9900000095367432,"score_stable":3.9900000095367432,"optimal_score":0.68999999761581421,"raw_score_acting":2.75,"raw_score_stable":2.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bdbe961a-78d3-4f7d-88ed-ce24475d49e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6813","nonce":3924351635}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6814","nonce":3924351635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6816","nonce":3924351635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6815","nonce":3924351635}]},"public_addr":"172.21.15.64:6813/3924351635","cluster_addr":"172.21.15.64:6814/3924351635","heartbeat_back_addr":"172.21.15.64:6816/3924351635","heartbeat_front_addr":"172.21.15.64:6815/3924351635","state":["exists","up"]},{"osd":1,"uuid":"154c8ddd-7a3d-402c-b9d6-28538f4e6bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":199,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6804","nonce":624977804}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6805","nonce":624977804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6807","nonce":624977804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6806","nonce":624977804}]},"public_addr":"172.21.15.112:6804/624977804","cluster_addr":"172.21.15.112:6805/624977804","heartbeat_back_addr":"172.21.15.112:6807/624977804","heartbeat_front_addr":"172.21.15.112:6806/624977804","state":["exists","up"]},{"osd":2,"uuid":"2a06de4e-0835-4a42-8c8a-49fb55aab566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6812","nonce":2804648086}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6813","nonce":2804648086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6815","nonce":2804648086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6814","nonce":2804648086}]},"public_addr":"172.21.15.155:6812/2804648086","cluster_addr":"172.21.15.155:6813/2804648086","heartbeat_back_addr":"172.21.15.155:6815/2804648086","heartbeat_front_addr":"172.21.15.155:6814/2804648086","state":["exists","up"]},{"osd":3,"uuid":"8a87f9f0-c380-4f38-bbd2-06b2def4fd2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6800","nonce":1935597988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6801","nonce":1935597988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6803","nonce":1935597988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6802","nonce":1935597988}]},"public_addr":"172.21.15.157:6800/1935597988","cluster_addr":"172.21.15.157:6801/1935597988","heartbeat_back_addr":"172.21.15.157:6803/1935597988","heartbeat_front_addr":"172.21.15.157:6802/1935597988","state":["exists","up"]},{"osd":4,"uuid":"5ac53318-12e8-496c-a87f-ba5de79c52aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":2675475248}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":2675475248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6812","nonce":2675475248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":2675475248}]},"public_addr":"172.21.15.64:6809/2675475248","cluster_addr":"172.21.15.64:6810/2675475248","heartbeat_back_addr":"172.21.15.64:6812/2675475248","heartbeat_front_addr":"172.21.15.64:6811/2675475248","state":["exists","up"]},{"osd":5,"uuid":"675c81a4-2314-463e-82a7-0060735c78fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6800","nonce":499905276}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6801","nonce":499905276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6803","nonce":499905276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6802","nonce":499905276}]},"public_addr":"172.21.15.112:6800/499905276","cluster_addr":"172.21.15.112:6801/499905276","heartbeat_back_addr":"172.21.15.112:6803/499905276","heartbeat_front_addr":"172.21.15.112:6802/499905276","state":["exists","up"]},{"osd":6,"uuid":"b18962a7-3b23-4e50-bf31-1d0629557423","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6808","nonce":2839983406}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6809","nonce":2839983406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6811","nonce":2839983406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6810","nonce":2839983406}]},"public_addr":"172.21.15.155:6808/2839983406","cluster_addr":"172.21.15.155:6809/2839983406","heartbeat_back_addr":"172.21.15.155:6811/2839983406","heartbeat_front_addr":"172.21.15.155:6810/2839983406","state":["exists","up"]},{"osd":7,"uuid":"9706c1e8-dac0-4e16-8bc5-7cacc6f13870","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6808","nonce":3021982871}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6809","nonce":3021982871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6811","nonce":3021982871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6810","nonce":3021982871}]},"public_addr":"172.21.15.157:6808/3021982871","cluster_addr":"172.21.15.157:6809/3021982871","heartbeat_back_addr":"172.21.15.157:6811/3021982871","heartbeat_front_addr":"172.21.15.157:6810/3021982871","state":["exists","up"]},{"osd":8,"uuid":"fafd06b8-c005-4471-a128-9dd51580c0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":3609017627}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":3609017627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":3609017627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":3609017627}]},"public_addr":"172.21.15.64:6805/3609017627","cluster_addr":"172.21.15.64:6806/3609017627","heartbeat_back_addr":"172.21.15.64:6808/3609017627","heartbeat_front_addr":"172.21.15.64:6807/3609017627","state":["exists","up"]},{"osd":9,"uuid":"79e38e57-52c2-416f-9b90-75621f06d3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6812","nonce":701981815}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6813","nonce":701981815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6815","nonce":701981815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6814","nonce":701981815}]},"public_addr":"172.21.15.112:6812/701981815","cluster_addr":"172.21.15.112:6813/701981815","heartbeat_back_addr":"172.21.15.112:6815/701981815","heartbeat_front_addr":"172.21.15.112:6814/701981815","state":["exists","up"]},{"osd":10,"uuid":"8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6804","nonce":4138619523}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6805","nonce":4138619523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6807","nonce":4138619523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6806","nonce":4138619523}]},"public_addr":"172.21.15.155:6804/4138619523","cluster_addr":"172.21.15.155:6805/4138619523","heartbeat_back_addr":"172.21.15.155:6807/4138619523","heartbeat_front_addr":"172.21.15.155:6806/4138619523","state":["exists","up"]},{"osd":11,"uuid":"7a2a169a-ce94-4091-a792-27d0ecf9a7fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6804","nonce":3607367937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6805","nonce":3607367937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6807","nonce":3607367937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6806","nonce":3607367937}]},"public_addr":"172.21.15.157:6804/3607367937","cluster_addr":"172.21.15.157:6805/3607367937","heartbeat_back_addr":"172.21.15.157:6807/3607367937","heartbeat_front_addr":"172.21.15.157:6806/3607367937","state":["exists","up"]},{"osd":12,"uuid":"7a280cda-b740-40c8-9fae-1a1ff11651f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":2984461266}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":2984461266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":2984461266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":2984461266}]},"public_addr":"172.21.15.64:6801/2984461266","cluster_addr":"172.21.15.64:6802/2984461266","heartbeat_back_addr":"172.21.15.64:6804/2984461266","heartbeat_front_addr":"172.21.15.64:6803/2984461266","state":["exists","up"]},{"osd":13,"uuid":"ce3a3148-cd90-422c-a106-045de89ab045","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6808","nonce":1616314527}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6809","nonce":1616314527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6811","nonce":1616314527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6810","nonce":1616314527}]},"public_addr":"172.21.15.112:6808/1616314527","cluster_addr":"172.21.15.112:6809/1616314527","heartbeat_back_addr":"172.21.15.112:6811/1616314527","heartbeat_front_addr":"172.21.15.112:6810/1616314527","state":["exists","up"]},{"osd":14,"uuid":"461c2d79-f4e7-4b21-899c-50d4a6cf12f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6800","nonce":4203868121}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6801","nonce":4203868121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6803","nonce":4203868121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6802","nonce":4203868121}]},"public_addr":"172.21.15.155:6800/4203868121","cluster_addr":"172.21.15.155:6801/4203868121","heartbeat_back_addr":"172.21.15.155:6803/4203868121","heartbeat_front_addr":"172.21.15.155:6802/4203868121","state":["exists","up"]},{"osd":15,"uuid":"9a8c0bbe-9389-4b2c-95e1-34d3e66461e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6812","nonce":181354529}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6813","nonce":181354529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6815","nonce":181354529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6814","nonce":181354529}]},"public_addr":"172.21.15.157:6812/181354529","cluster_addr":"172.21.15.157:6813/181354529","heartbeat_back_addr":"172.21.15.157:6815/181354529","heartbeat_front_addr":"172.21.15.157:6814/181354529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:41.866144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:15.766345+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:35.030728+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:28.164905+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:36.498999+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:11.403494+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:31.536918+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:13.198340+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:33.844989+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:07.784177+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:40.632845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:29.586378+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:25.172167+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:27.906719+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:30.872798+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:30.112593+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"3.1","osds":[6,4,11]},{"pgid":"3.2","osds":[11,0,4]},{"pgid":"3.3","osds":[0,8,2]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:40:46.715 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T03:40:46.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:40:47.017 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:47.017 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":221,"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","created":"2024-09-16T03:36:37.310421+0000","modified":"2024-09-16T03:40:46.380109+0000","last_up_change":"2024-09-16T03:36:50.058111+0000","last_in_change":"2024-09-16T03:40:14.459006+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:36:49.696307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.789999961853027,"score_stable":15.789999961853027,"optimal_score":0.18999999761581421,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:36:51.904455+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":5.9699997901916504,"score_stable":5.9699997901916504,"optimal_score":0.87999999523162842,"raw_score_acting":5.25,"raw_score_stable":5.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-09-16T03:37:02.999795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9900000095367432,"score_stable":3.9900000095367432,"optimal_score":0.68999999761581421,"raw_score_acting":2.75,"raw_score_stable":2.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bdbe961a-78d3-4f7d-88ed-ce24475d49e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6813","nonce":3924351635}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6814","nonce":3924351635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6816","nonce":3924351635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6815","nonce":3924351635}]},"public_addr":"172.21.15.64:6813/3924351635","cluster_addr":"172.21.15.64:6814/3924351635","heartbeat_back_addr":"172.21.15.64:6816/3924351635","heartbeat_front_addr":"172.21.15.64:6815/3924351635","state":["exists","up"]},{"osd":1,"uuid":"154c8ddd-7a3d-402c-b9d6-28538f4e6bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":199,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6804","nonce":624977804}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6805","nonce":624977804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6807","nonce":624977804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6806","nonce":624977804}]},"public_addr":"172.21.15.112:6804/624977804","cluster_addr":"172.21.15.112:6805/624977804","heartbeat_back_addr":"172.21.15.112:6807/624977804","heartbeat_front_addr":"172.21.15.112:6806/624977804","state":["exists","up"]},{"osd":2,"uuid":"2a06de4e-0835-4a42-8c8a-49fb55aab566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6812","nonce":2804648086}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6813","nonce":2804648086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6815","nonce":2804648086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6814","nonce":2804648086}]},"public_addr":"172.21.15.155:6812/2804648086","cluster_addr":"172.21.15.155:6813/2804648086","heartbeat_back_addr":"172.21.15.155:6815/2804648086","heartbeat_front_addr":"172.21.15.155:6814/2804648086","state":["exists","up"]},{"osd":3,"uuid":"8a87f9f0-c380-4f38-bbd2-06b2def4fd2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6800","nonce":1935597988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6801","nonce":1935597988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6803","nonce":1935597988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6802","nonce":1935597988}]},"public_addr":"172.21.15.157:6800/1935597988","cluster_addr":"172.21.15.157:6801/1935597988","heartbeat_back_addr":"172.21.15.157:6803/1935597988","heartbeat_front_addr":"172.21.15.157:6802/1935597988","state":["exists","up"]},{"osd":4,"uuid":"5ac53318-12e8-496c-a87f-ba5de79c52aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":2675475248}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":2675475248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6812","nonce":2675475248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":2675475248}]},"public_addr":"172.21.15.64:6809/2675475248","cluster_addr":"172.21.15.64:6810/2675475248","heartbeat_back_addr":"172.21.15.64:6812/2675475248","heartbeat_front_addr":"172.21.15.64:6811/2675475248","state":["exists","up"]},{"osd":5,"uuid":"675c81a4-2314-463e-82a7-0060735c78fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6800","nonce":499905276}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6801","nonce":499905276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6803","nonce":499905276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6802","nonce":499905276}]},"public_addr":"172.21.15.112:6800/499905276","cluster_addr":"172.21.15.112:6801/499905276","heartbeat_back_addr":"172.21.15.112:6803/499905276","heartbeat_front_addr":"172.21.15.112:6802/499905276","state":["exists","up"]},{"osd":6,"uuid":"b18962a7-3b23-4e50-bf31-1d0629557423","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":215,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6808","nonce":2839983406}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6809","nonce":2839983406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6811","nonce":2839983406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6810","nonce":2839983406}]},"public_addr":"172.21.15.155:6808/2839983406","cluster_addr":"172.21.15.155:6809/2839983406","heartbeat_back_addr":"172.21.15.155:6811/2839983406","heartbeat_front_addr":"172.21.15.155:6810/2839983406","state":["exists","up"]},{"osd":7,"uuid":"9706c1e8-dac0-4e16-8bc5-7cacc6f13870","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6808","nonce":3021982871}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6809","nonce":3021982871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6811","nonce":3021982871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6810","nonce":3021982871}]},"public_addr":"172.21.15.157:6808/3021982871","cluster_addr":"172.21.15.157:6809/3021982871","heartbeat_back_addr":"172.21.15.157:6811/3021982871","heartbeat_front_addr":"172.21.15.157:6810/3021982871","state":["exists","up"]},{"osd":8,"uuid":"fafd06b8-c005-4471-a128-9dd51580c0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":3609017627}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":3609017627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":3609017627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":3609017627}]},"public_addr":"172.21.15.64:6805/3609017627","cluster_addr":"172.21.15.64:6806/3609017627","heartbeat_back_addr":"172.21.15.64:6808/3609017627","heartbeat_front_addr":"172.21.15.64:6807/3609017627","state":["exists","up"]},{"osd":9,"uuid":"79e38e57-52c2-416f-9b90-75621f06d3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6812","nonce":701981815}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6813","nonce":701981815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6815","nonce":701981815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6814","nonce":701981815}]},"public_addr":"172.21.15.112:6812/701981815","cluster_addr":"172.21.15.112:6813/701981815","heartbeat_back_addr":"172.21.15.112:6815/701981815","heartbeat_front_addr":"172.21.15.112:6814/701981815","state":["exists","up"]},{"osd":10,"uuid":"8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6804","nonce":4138619523}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6805","nonce":4138619523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6807","nonce":4138619523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6806","nonce":4138619523}]},"public_addr":"172.21.15.155:6804/4138619523","cluster_addr":"172.21.15.155:6805/4138619523","heartbeat_back_addr":"172.21.15.155:6807/4138619523","heartbeat_front_addr":"172.21.15.155:6806/4138619523","state":["exists","up"]},{"osd":11,"uuid":"7a2a169a-ce94-4091-a792-27d0ecf9a7fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":219,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6804","nonce":3607367937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6805","nonce":3607367937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6807","nonce":3607367937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6806","nonce":3607367937}]},"public_addr":"172.21.15.157:6804/3607367937","cluster_addr":"172.21.15.157:6805/3607367937","heartbeat_back_addr":"172.21.15.157:6807/3607367937","heartbeat_front_addr":"172.21.15.157:6806/3607367937","state":["exists","up"]},{"osd":12,"uuid":"7a280cda-b740-40c8-9fae-1a1ff11651f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":2984461266}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":2984461266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":2984461266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":2984461266}]},"public_addr":"172.21.15.64:6801/2984461266","cluster_addr":"172.21.15.64:6802/2984461266","heartbeat_back_addr":"172.21.15.64:6804/2984461266","heartbeat_front_addr":"172.21.15.64:6803/2984461266","state":["exists","up"]},{"osd":13,"uuid":"ce3a3148-cd90-422c-a106-045de89ab045","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6808","nonce":1616314527}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6809","nonce":1616314527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6811","nonce":1616314527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6810","nonce":1616314527}]},"public_addr":"172.21.15.112:6808/1616314527","cluster_addr":"172.21.15.112:6809/1616314527","heartbeat_back_addr":"172.21.15.112:6811/1616314527","heartbeat_front_addr":"172.21.15.112:6810/1616314527","state":["exists","up"]},{"osd":14,"uuid":"461c2d79-f4e7-4b21-899c-50d4a6cf12f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6800","nonce":4203868121}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6801","nonce":4203868121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6803","nonce":4203868121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6802","nonce":4203868121}]},"public_addr":"172.21.15.155:6800/4203868121","cluster_addr":"172.21.15.155:6801/4203868121","heartbeat_back_addr":"172.21.15.155:6803/4203868121","heartbeat_front_addr":"172.21.15.155:6802/4203868121","state":["exists","up"]},{"osd":15,"uuid":"9a8c0bbe-9389-4b2c-95e1-34d3e66461e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6812","nonce":181354529}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6813","nonce":181354529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6815","nonce":181354529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6814","nonce":181354529}]},"public_addr":"172.21.15.157:6812/181354529","cluster_addr":"172.21.15.157:6813/181354529","heartbeat_back_addr":"172.21.15.157:6815/181354529","heartbeat_front_addr":"172.21.15.157:6814/181354529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:41.866144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:15.766345+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:35.030728+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:28.164905+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:36.498999+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:11.403494+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:31.536918+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:13.198340+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:33.844989+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:07.784177+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:40.632845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:29.586378+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:25.172167+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:27.906719+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:30.872798+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:39:30.112593+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"3.1","osds":[6,4,11]},{"pgid":"3.2","osds":[11,0,4]},{"pgid":"3.3","osds":[0,8,2]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:40:47.030 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T03:40:47.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T03:40:47.031 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T03:40:47.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T03:40:47.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T03:40:47.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T03:40:47.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T03:40:47.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T03:40:47.034 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-16T03:40:47.034 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-16T03:40:47.035 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-16T03:40:47.035 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-16T03:40:47.035 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-16T03:40:47.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-16T03:40:47.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-16T03:40:47.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-16T03:40:47.396 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:40:47.395+0000 7fd101e6f640 -1 osd.2 221 handle_osd_map injecting map crc failure 2024-09-16T03:40:47.402 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:40:47.399+0000 7f4212d00640 -1 osd.8 221 handle_osd_map injecting map crc failure 2024-09-16T03:40:47.485 INFO:teuthology.orchestra.run.smithi064.stdout:94489280563 2024-09-16T03:40:47.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T03:40:47.497 INFO:teuthology.orchestra.run.smithi064.stdout:98784247859 2024-09-16T03:40:47.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T03:40:47.502 INFO:teuthology.orchestra.run.smithi064.stdout:94489280562 2024-09-16T03:40:47.502 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T03:40:47.513 INFO:teuthology.orchestra.run.smithi064.stdout:94489280562 2024-09-16T03:40:47.513 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T03:40:47.517 INFO:teuthology.orchestra.run.smithi064.stdout:94489280563 2024-09-16T03:40:47.517 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T03:40:47.521 INFO:teuthology.orchestra.run.smithi064.stdout:94489280563 2024-09-16T03:40:47.522 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T03:40:47.533 INFO:teuthology.orchestra.run.smithi064.stdout:94489280562 2024-09-16T03:40:47.533 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T03:40:47.538 INFO:teuthology.orchestra.run.smithi064.stdout:90194313266 2024-09-16T03:40:47.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T03:40:47.539 INFO:teuthology.orchestra.run.smithi064.stdout:94489280562 2024-09-16T03:40:47.539 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T03:40:47.541 INFO:teuthology.orchestra.run.smithi064.stdout:90194313267 2024-09-16T03:40:47.542 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T03:40:47.545 INFO:teuthology.orchestra.run.smithi064.stdout:98784247858 2024-09-16T03:40:47.545 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T03:40:47.553 INFO:teuthology.orchestra.run.smithi064.stdout:94489280563 2024-09-16T03:40:47.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T03:40:47.588 INFO:teuthology.orchestra.run.smithi064.stdout:98784247858 2024-09-16T03:40:47.588 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T03:40:47.590 INFO:teuthology.orchestra.run.smithi064.stdout:98784247859 2024-09-16T03:40:47.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T03:40:47.624 INFO:teuthology.orchestra.run.smithi064.stdout:90194313267 2024-09-16T03:40:47.625 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T03:40:47.630 INFO:teuthology.orchestra.run.smithi064.stdout:94489280562 2024-09-16T03:40:47.630 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T03:40:48.467 INFO:teuthology.orchestra.run.smithi064.stdout:94489280562 2024-09-16T03:40:48.491 INFO:teuthology.orchestra.run.smithi064.stdout:98784247857 2024-09-16T03:40:48.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280563 got 94489280562 for osd.13 2024-09-16T03:40:48.513 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247859 got 98784247857 for osd.11 2024-09-16T03:40:48.658 INFO:teuthology.orchestra.run.smithi064.stdout:94489280561 2024-09-16T03:40:48.699 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280562 got 94489280561 for osd.8 2024-09-16T03:40:48.707 INFO:teuthology.orchestra.run.smithi064.stdout:94489280563 2024-09-16T03:40:48.717 INFO:teuthology.orchestra.run.smithi064.stdout:98784247858 2024-09-16T03:40:48.724 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280563 got 94489280563 for osd.0 2024-09-16T03:40:48.724 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:48.745 INFO:teuthology.orchestra.run.smithi064.stdout:90194313267 2024-09-16T03:40:48.750 INFO:teuthology.orchestra.run.smithi064.stdout:90194313267 2024-09-16T03:40:48.757 INFO:teuthology.orchestra.run.smithi064.stdout:94489280563 2024-09-16T03:40:48.758 INFO:teuthology.orchestra.run.smithi064.stdout:98784247858 2024-09-16T03:40:48.775 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247858 got 98784247858 for osd.6 2024-09-16T03:40:48.775 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:48.779 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313267 got 90194313267 for osd.5 2024-09-16T03:40:48.779 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:48.793 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247858 got 98784247858 for osd.15 2024-09-16T03:40:48.794 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:48.797 INFO:teuthology.orchestra.run.smithi064.stdout:94489280562 2024-09-16T03:40:48.798 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313267 got 90194313267 for osd.14 2024-09-16T03:40:48.798 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:48.803 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280563 got 94489280563 for osd.10 2024-09-16T03:40:48.803 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:48.815 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280562 got 94489280562 for osd.7 2024-09-16T03:40:48.816 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:48.819 INFO:teuthology.orchestra.run.smithi064.stdout:94489280562 2024-09-16T03:40:48.819 INFO:teuthology.orchestra.run.smithi064.stdout:90194313266 2024-09-16T03:40:48.822 INFO:teuthology.orchestra.run.smithi064.stdout:94489280562 2024-09-16T03:40:48.833 INFO:teuthology.orchestra.run.smithi064.stdout:98784247859 2024-09-16T03:40:48.835 INFO:teuthology.orchestra.run.smithi064.stdout:94489280562 2024-09-16T03:40:48.835 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313266 got 90194313266 for osd.12 2024-09-16T03:40:48.836 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:48.841 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280562 got 94489280562 for osd.4 2024-09-16T03:40:48.842 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:48.845 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280562 got 94489280562 for osd.9 2024-09-16T03:40:48.845 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:48.849 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247859 got 98784247859 for osd.2 2024-09-16T03:40:48.850 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:48.850 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280562 got 94489280562 for osd.1 2024-09-16T03:40:48.851 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:48.868 INFO:teuthology.orchestra.run.smithi064.stdout:94489280563 2024-09-16T03:40:48.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280563 got 94489280563 for osd.3 2024-09-16T03:40:48.882 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:49.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T03:40:49.513 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T03:40:49.700 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T03:40:49.808 INFO:teuthology.orchestra.run.smithi064.stdout:94489280563 2024-09-16T03:40:49.823 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280563 got 94489280563 for osd.13 2024-09-16T03:40:49.823 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:49.856 INFO:teuthology.orchestra.run.smithi064.stdout:98784247859 2024-09-16T03:40:49.869 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247859 got 98784247859 for osd.11 2024-09-16T03:40:49.870 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:49.941 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:40:49.939+0000 7ff266366640 -1 osd.11 222 handle_osd_map injecting map crc failure 2024-09-16T03:40:50.007 INFO:teuthology.orchestra.run.smithi064.stdout:94489280562 2024-09-16T03:40:50.020 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280562 got 94489280562 for osd.8 2024-09-16T03:40:50.020 DEBUG:teuthology.parallel:result is None 2024-09-16T03:40:50.020 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T03:40:50.020 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:50.320 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:50.320 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:40:50.332 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-09-16T03:40:49.726646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":103,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":99271272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":729876,"kb_used_data":214156,"kb_used_omap":24,"kb_used_meta":515303,"kb_avail":1509219564,"statfs":{"total":1546188226560,"available":1545440833536,"internally_reserved":0,"allocated":219295744,"data_stored":227841098,"data_compressed":14722464,"data_compressed_allocated":16297984,"data_compressed_original":34127872,"omap_allocated":25519,"internal_metadata":527670353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-20,"num_read_kb":-20,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":10146478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.068884"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":625,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:46.978617+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:40:46.978617+0000","last_peered":"2024-09-16T03:40:46.978617+0000","last_clean":"2024-09-16T03:40:46.978617+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:40:46.978617+0000","last_undegraded":"2024-09-16T03:40:46.978617+0000","last_fullsized":"2024-09-16T03:40:46.978617+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":144,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:46.978791+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:40:46.978791+0000","last_peered":"2024-09-16T03:40:46.978791+0000","last_clean":"2024-09-16T03:40:46.978791+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:40:46.978791+0000","last_undegraded":"2024-09-16T03:40:46.978791+0000","last_fullsized":"2024-09-16T03:40:46.978791+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":873,"reported_epoch":221,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:40:47.466343+0000","last_change":"2024-09-16T03:40:46.732184+0000","last_active":"2024-09-16T03:40:47.466343+0000","last_peered":"2024-09-16T03:40:47.466343+0000","last_clean":"2024-09-16T03:40:13.463899+0000","last_became_active":"2024-09-16T03:40:16.729881+0000","last_became_peered":"2024-09-16T03:40:16.729881+0000","last_unstale":"2024-09-16T03:40:47.466343+0000","last_undegraded":"2024-09-16T03:40:47.466343+0000","last_fullsized":"2024-09-16T03:40:47.466343+0000","mapping_epoch":190,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":15933108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,0,4],"avail_no_missing":["11","0","4"],"object_location_counts":[{"shards":"0,4,11","objects":44}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":455,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.408964+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:40:47.408964+0000","last_peered":"2024-09-16T03:40:47.408964+0000","last_clean":"2024-09-16T03:40:47.408964+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:40:47.408964+0000","last_undegraded":"2024-09-16T03:40:47.408964+0000","last_fullsized":"2024-09-16T03:40:47.408964+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":822,"reported_epoch":222,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:47.487205+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:40:47.487205+0000","last_peered":"2024-09-16T03:40:47.487205+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:40:47.487205+0000","last_undegraded":"2024-09-16T03:40:47.487205+0000","last_fullsized":"2024-09-16T03:40:47.487205+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.180070+0000","last_change":"2024-09-16T03:40:47.179954+0000","last_active":"2024-09-16T03:40:47.180070+0000","last_peered":"2024-09-16T03:40:47.180070+0000","last_clean":"2024-09-16T03:40:47.180070+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:40:47.180070+0000","last_undegraded":"2024-09-16T03:40:47.180070+0000","last_fullsized":"2024-09-16T03:40:47.180070+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":878,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:48.342569+0000","last_change":"2024-09-16T03:40:25.635449+0000","last_active":"2024-09-16T03:40:48.342569+0000","last_peered":"2024-09-16T03:40:48.342569+0000","last_clean":"2024-09-16T03:40:48.342569+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:40:48.342569+0000","last_undegraded":"2024-09-16T03:40:48.342569+0000","last_fullsized":"2024-09-16T03:40:48.342569+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:03.730604+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":154,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.408977+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:40:47.408977+0000","last_peered":"2024-09-16T03:40:47.408977+0000","last_clean":"2024-09-16T03:40:47.408977+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:40:47.408977+0000","last_undegraded":"2024-09-16T03:40:47.408977+0000","last_fullsized":"2024-09-16T03:40:47.408977+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":792,"reported_epoch":222,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:40:49.604141+0000","last_change":"2024-09-16T03:40:48.048391+0000","last_active":"2024-09-16T03:40:49.604141+0000","last_peered":"2024-09-16T03:40:49.604141+0000","last_clean":"2024-09-16T03:39:52.800868+0000","last_became_active":"2024-09-16T03:40:47.849953+0000","last_became_peered":"2024-09-16T03:40:47.849953+0000","last_unstale":"2024-09-16T03:40:49.604141+0000","last_undegraded":"2024-09-16T03:40:49.604141+0000","last_fullsized":"2024-09-16T03:40:49.604141+0000","mapping_epoch":221,"log_start":"168'338","ondisk_log_start":"168'338","created":31,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":350,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":22258960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[0,8,2],"avail_no_missing":["0","2","8"],"object_location_counts":[{"shards":"0,2,8","objects":39}],"blocked_by":[],"up_primary":15,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":428,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:46.865023+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:40:46.865023+0000","last_peered":"2024-09-16T03:40:46.865023+0000","last_clean":"2024-09-16T03:40:46.865023+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:40:46.865023+0000","last_undegraded":"2024-09-16T03:40:46.865023+0000","last_fullsized":"2024-09-16T03:40:46.865023+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":183,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:46.878610+0000","last_change":"2024-09-16T03:39:09.848905+0000","last_active":"2024-09-16T03:40:46.878610+0000","last_peered":"2024-09-16T03:40:46.878610+0000","last_clean":"2024-09-16T03:40:46.878610+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:46.878610+0000","last_undegraded":"2024-09-16T03:40:46.878610+0000","last_fullsized":"2024-09-16T03:40:46.878610+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:52.781538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":437,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:49.114710+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:40:49.114710+0000","last_peered":"2024-09-16T03:40:49.114710+0000","last_clean":"2024-09-16T03:40:49.114710+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:40:49.114710+0000","last_undegraded":"2024-09-16T03:40:49.114710+0000","last_fullsized":"2024-09-16T03:40:49.114710+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":423,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.409072+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:40:47.409072+0000","last_peered":"2024-09-16T03:40:47.409072+0000","last_clean":"2024-09-16T03:40:47.409072+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:47.409072+0000","last_undegraded":"2024-09-16T03:40:47.409072+0000","last_fullsized":"2024-09-16T03:40:47.409072+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":103,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":97684488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199913472,"data_stored":213522064,"data_compressed":13083793,"data_compressed_allocated":13971456,"data_compressed_original":28188672,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1310720,"data_stored":914947,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,3,4,6,7,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247859,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65284,"kb_used_data":34084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94306556,"statfs":{"total":96636764160,"available":96569913344,"internally_reserved":0,"allocated":34902016,"data_stored":37402304,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247858,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52612,"kb_used_data":17264,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319228,"statfs":{"total":96636764160,"available":96582889472,"internally_reserved":0,"allocated":17678336,"data_stored":18568407,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63680,"kb_used_data":32488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94308160,"statfs":{"total":96636764160,"available":96571555840,"internally_reserved":0,"allocated":33267712,"data_stored":35397581,"data_compressed":2306856,"data_compressed_allocated":2498560,"data_compressed_original":5300224,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28320,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343520,"statfs":{"total":96636764160,"available":96607764480,"internally_reserved":0,"allocated":1323008,"data_stored":921945,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280563,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46648,"kb_used_data":15456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325192,"statfs":{"total":96636764160,"available":96588996608,"internally_reserved":0,"allocated":15826944,"data_stored":16308720,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313268,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1310720,"data_stored":915584,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":1351680,"data_stored":925440,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280562,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83576,"kb_used_data":48216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94288264,"statfs":{"total":96636764160,"available":96551182336,"internally_reserved":0,"allocated":49373184,"data_stored":52268863,"data_compressed":2976505,"data_compressed_allocated":3227648,"data_compressed_original":6828032,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313268,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42088,"kb_used_data":10896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329752,"statfs":{"total":96636764160,"available":96593666048,"internally_reserved":0,"allocated":11157504,"data_stored":12046739,"data_compressed":856662,"data_compressed_allocated":954368,"data_compressed_original":2396160,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280563,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32480,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339360,"statfs":{"total":96636764160,"available":96603504640,"internally_reserved":0,"allocated":1310720,"data_stored":915044,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280562,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50992,"kb_used_data":15644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320848,"statfs":{"total":96636764160,"available":96584548352,"internally_reserved":0,"allocated":16019456,"data_stored":16562011,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280562,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46628,"kb_used_data":15432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325212,"statfs":{"total":96636764160,"available":96589017088,"internally_reserved":0,"allocated":15802368,"data_stored":16293042,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1335296,"data_stored":917370,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32472,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1310720,"data_stored":915623,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46828,"kb_used_data":15640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325012,"statfs":{"total":96636764160,"available":96588812288,"internally_reserved":0,"allocated":16015360,"data_stored":16567478,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31858688,"data_stored":33943544,"data_compressed":2174030,"data_compressed_allocated":2314240,"data_compressed_original":4501504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":48046080,"data_stored":51351621,"data_compressed":2874927,"data_compressed_allocated":3084288,"data_compressed_original":6541312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11636736,"data_stored":12525940,"data_compressed":804157,"data_compressed_allocated":860160,"data_compressed_original":1781760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:50.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:50.630 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:50.631 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:40:50.642 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-09-16T03:40:49.726646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":103,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":99271272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":729876,"kb_used_data":214156,"kb_used_omap":24,"kb_used_meta":515303,"kb_avail":1509219564,"statfs":{"total":1546188226560,"available":1545440833536,"internally_reserved":0,"allocated":219295744,"data_stored":227841098,"data_compressed":14722464,"data_compressed_allocated":16297984,"data_compressed_original":34127872,"omap_allocated":25519,"internal_metadata":527670353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-20,"num_read_kb":-20,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":10146478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.068884"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":625,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:46.978617+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:40:46.978617+0000","last_peered":"2024-09-16T03:40:46.978617+0000","last_clean":"2024-09-16T03:40:46.978617+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:40:46.978617+0000","last_undegraded":"2024-09-16T03:40:46.978617+0000","last_fullsized":"2024-09-16T03:40:46.978617+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":144,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:46.978791+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:40:46.978791+0000","last_peered":"2024-09-16T03:40:46.978791+0000","last_clean":"2024-09-16T03:40:46.978791+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:40:46.978791+0000","last_undegraded":"2024-09-16T03:40:46.978791+0000","last_fullsized":"2024-09-16T03:40:46.978791+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":873,"reported_epoch":221,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:40:47.466343+0000","last_change":"2024-09-16T03:40:46.732184+0000","last_active":"2024-09-16T03:40:47.466343+0000","last_peered":"2024-09-16T03:40:47.466343+0000","last_clean":"2024-09-16T03:40:13.463899+0000","last_became_active":"2024-09-16T03:40:16.729881+0000","last_became_peered":"2024-09-16T03:40:16.729881+0000","last_unstale":"2024-09-16T03:40:47.466343+0000","last_undegraded":"2024-09-16T03:40:47.466343+0000","last_fullsized":"2024-09-16T03:40:47.466343+0000","mapping_epoch":190,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":15933108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,0,4],"avail_no_missing":["11","0","4"],"object_location_counts":[{"shards":"0,4,11","objects":44}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":455,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.408964+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:40:47.408964+0000","last_peered":"2024-09-16T03:40:47.408964+0000","last_clean":"2024-09-16T03:40:47.408964+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:40:47.408964+0000","last_undegraded":"2024-09-16T03:40:47.408964+0000","last_fullsized":"2024-09-16T03:40:47.408964+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":822,"reported_epoch":222,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:47.487205+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:40:47.487205+0000","last_peered":"2024-09-16T03:40:47.487205+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:40:47.487205+0000","last_undegraded":"2024-09-16T03:40:47.487205+0000","last_fullsized":"2024-09-16T03:40:47.487205+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.180070+0000","last_change":"2024-09-16T03:40:47.179954+0000","last_active":"2024-09-16T03:40:47.180070+0000","last_peered":"2024-09-16T03:40:47.180070+0000","last_clean":"2024-09-16T03:40:47.180070+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:40:47.180070+0000","last_undegraded":"2024-09-16T03:40:47.180070+0000","last_fullsized":"2024-09-16T03:40:47.180070+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":878,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:48.342569+0000","last_change":"2024-09-16T03:40:25.635449+0000","last_active":"2024-09-16T03:40:48.342569+0000","last_peered":"2024-09-16T03:40:48.342569+0000","last_clean":"2024-09-16T03:40:48.342569+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:40:48.342569+0000","last_undegraded":"2024-09-16T03:40:48.342569+0000","last_fullsized":"2024-09-16T03:40:48.342569+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:03.730604+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":154,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.408977+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:40:47.408977+0000","last_peered":"2024-09-16T03:40:47.408977+0000","last_clean":"2024-09-16T03:40:47.408977+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:40:47.408977+0000","last_undegraded":"2024-09-16T03:40:47.408977+0000","last_fullsized":"2024-09-16T03:40:47.408977+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":792,"reported_epoch":222,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:40:49.604141+0000","last_change":"2024-09-16T03:40:48.048391+0000","last_active":"2024-09-16T03:40:49.604141+0000","last_peered":"2024-09-16T03:40:49.604141+0000","last_clean":"2024-09-16T03:39:52.800868+0000","last_became_active":"2024-09-16T03:40:47.849953+0000","last_became_peered":"2024-09-16T03:40:47.849953+0000","last_unstale":"2024-09-16T03:40:49.604141+0000","last_undegraded":"2024-09-16T03:40:49.604141+0000","last_fullsized":"2024-09-16T03:40:49.604141+0000","mapping_epoch":221,"log_start":"168'338","ondisk_log_start":"168'338","created":31,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":350,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":22258960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[0,8,2],"avail_no_missing":["0","2","8"],"object_location_counts":[{"shards":"0,2,8","objects":39}],"blocked_by":[],"up_primary":15,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":428,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:46.865023+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:40:46.865023+0000","last_peered":"2024-09-16T03:40:46.865023+0000","last_clean":"2024-09-16T03:40:46.865023+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:40:46.865023+0000","last_undegraded":"2024-09-16T03:40:46.865023+0000","last_fullsized":"2024-09-16T03:40:46.865023+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":183,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:46.878610+0000","last_change":"2024-09-16T03:39:09.848905+0000","last_active":"2024-09-16T03:40:46.878610+0000","last_peered":"2024-09-16T03:40:46.878610+0000","last_clean":"2024-09-16T03:40:46.878610+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:46.878610+0000","last_undegraded":"2024-09-16T03:40:46.878610+0000","last_fullsized":"2024-09-16T03:40:46.878610+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:52.781538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":437,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:49.114710+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:40:49.114710+0000","last_peered":"2024-09-16T03:40:49.114710+0000","last_clean":"2024-09-16T03:40:49.114710+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:40:49.114710+0000","last_undegraded":"2024-09-16T03:40:49.114710+0000","last_fullsized":"2024-09-16T03:40:49.114710+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":423,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.409072+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:40:47.409072+0000","last_peered":"2024-09-16T03:40:47.409072+0000","last_clean":"2024-09-16T03:40:47.409072+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:47.409072+0000","last_undegraded":"2024-09-16T03:40:47.409072+0000","last_fullsized":"2024-09-16T03:40:47.409072+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":103,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":97684488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199913472,"data_stored":213522064,"data_compressed":13083793,"data_compressed_allocated":13971456,"data_compressed_original":28188672,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1310720,"data_stored":914947,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,3,4,6,7,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247859,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65284,"kb_used_data":34084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94306556,"statfs":{"total":96636764160,"available":96569913344,"internally_reserved":0,"allocated":34902016,"data_stored":37402304,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247858,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52612,"kb_used_data":17264,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319228,"statfs":{"total":96636764160,"available":96582889472,"internally_reserved":0,"allocated":17678336,"data_stored":18568407,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63680,"kb_used_data":32488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94308160,"statfs":{"total":96636764160,"available":96571555840,"internally_reserved":0,"allocated":33267712,"data_stored":35397581,"data_compressed":2306856,"data_compressed_allocated":2498560,"data_compressed_original":5300224,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28320,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343520,"statfs":{"total":96636764160,"available":96607764480,"internally_reserved":0,"allocated":1323008,"data_stored":921945,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280563,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46648,"kb_used_data":15456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325192,"statfs":{"total":96636764160,"available":96588996608,"internally_reserved":0,"allocated":15826944,"data_stored":16308720,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313268,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1310720,"data_stored":915584,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":1351680,"data_stored":925440,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280562,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83576,"kb_used_data":48216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94288264,"statfs":{"total":96636764160,"available":96551182336,"internally_reserved":0,"allocated":49373184,"data_stored":52268863,"data_compressed":2976505,"data_compressed_allocated":3227648,"data_compressed_original":6828032,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313268,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42088,"kb_used_data":10896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329752,"statfs":{"total":96636764160,"available":96593666048,"internally_reserved":0,"allocated":11157504,"data_stored":12046739,"data_compressed":856662,"data_compressed_allocated":954368,"data_compressed_original":2396160,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280563,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32480,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339360,"statfs":{"total":96636764160,"available":96603504640,"internally_reserved":0,"allocated":1310720,"data_stored":915044,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280562,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50992,"kb_used_data":15644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320848,"statfs":{"total":96636764160,"available":96584548352,"internally_reserved":0,"allocated":16019456,"data_stored":16562011,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280562,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46628,"kb_used_data":15432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325212,"statfs":{"total":96636764160,"available":96589017088,"internally_reserved":0,"allocated":15802368,"data_stored":16293042,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1335296,"data_stored":917370,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32472,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1310720,"data_stored":915623,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46828,"kb_used_data":15640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325012,"statfs":{"total":96636764160,"available":96588812288,"internally_reserved":0,"allocated":16015360,"data_stored":16567478,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31858688,"data_stored":33943544,"data_compressed":2174030,"data_compressed_allocated":2314240,"data_compressed_original":4501504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":48046080,"data_stored":51351621,"data_compressed":2874927,"data_compressed_allocated":3084288,"data_compressed_original":6541312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11636736,"data_stored":12525940,"data_compressed":804157,"data_compressed_allocated":860160,"data_compressed_original":1781760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:50.644 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T03:40:50.850 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:40:50.848+0000 7f3a21722640 -1 osd.6 222 handle_osd_map injecting map crc failure 2024-09-16T03:40:51.012 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:51.013 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":253,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":223,"num_osds":16,"num_up_osds":16,"osd_up_since":1726457810,"num_in_osds":16,"osd_in_since":1726458014,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+remapped+backfilling","count":2},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":150,"data_bytes":103194545,"bytes_used":747393024,"bytes_avail":1545440833536,"bytes_total":1546188226560,"misplaced_objects":103,"misplaced_total":450,"misplaced_ratio":0.22888888888888889,"recovering_objects_per_sec":5,"recovering_bytes_per_sec":2493675,"recovering_keys_per_sec":0,"num_objects_recovered":21,"num_bytes_recovered":10146478,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T03:36:37:310011+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T03:37:51.688536+0000","services":{}},"progress_events":{}} 2024-09-16T03:40:51.025 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-16T03:40:51.026 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:51.216 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:40:51.214+0000 7efeef6d5640 -1 osd.13 222 handle_osd_map injecting map crc failure 2024-09-16T03:40:51.321 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:51.322 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:40:51.334 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-09-16T03:40:49.726646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":103,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":99271272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":729876,"kb_used_data":214156,"kb_used_omap":24,"kb_used_meta":515303,"kb_avail":1509219564,"statfs":{"total":1546188226560,"available":1545440833536,"internally_reserved":0,"allocated":219295744,"data_stored":227841098,"data_compressed":14722464,"data_compressed_allocated":16297984,"data_compressed_original":34127872,"omap_allocated":25519,"internal_metadata":527670353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-20,"num_read_kb":-20,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":10146478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.068884"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":625,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:46.978617+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:40:46.978617+0000","last_peered":"2024-09-16T03:40:46.978617+0000","last_clean":"2024-09-16T03:40:46.978617+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:40:46.978617+0000","last_undegraded":"2024-09-16T03:40:46.978617+0000","last_fullsized":"2024-09-16T03:40:46.978617+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":144,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:46.978791+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:40:46.978791+0000","last_peered":"2024-09-16T03:40:46.978791+0000","last_clean":"2024-09-16T03:40:46.978791+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:40:46.978791+0000","last_undegraded":"2024-09-16T03:40:46.978791+0000","last_fullsized":"2024-09-16T03:40:46.978791+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":873,"reported_epoch":221,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:40:47.466343+0000","last_change":"2024-09-16T03:40:46.732184+0000","last_active":"2024-09-16T03:40:47.466343+0000","last_peered":"2024-09-16T03:40:47.466343+0000","last_clean":"2024-09-16T03:40:13.463899+0000","last_became_active":"2024-09-16T03:40:16.729881+0000","last_became_peered":"2024-09-16T03:40:16.729881+0000","last_unstale":"2024-09-16T03:40:47.466343+0000","last_undegraded":"2024-09-16T03:40:47.466343+0000","last_fullsized":"2024-09-16T03:40:47.466343+0000","mapping_epoch":190,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":15933108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,0,4],"avail_no_missing":["11","0","4"],"object_location_counts":[{"shards":"0,4,11","objects":44}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":455,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.408964+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:40:47.408964+0000","last_peered":"2024-09-16T03:40:47.408964+0000","last_clean":"2024-09-16T03:40:47.408964+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:40:47.408964+0000","last_undegraded":"2024-09-16T03:40:47.408964+0000","last_fullsized":"2024-09-16T03:40:47.408964+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":822,"reported_epoch":222,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:47.487205+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:40:47.487205+0000","last_peered":"2024-09-16T03:40:47.487205+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:40:47.487205+0000","last_undegraded":"2024-09-16T03:40:47.487205+0000","last_fullsized":"2024-09-16T03:40:47.487205+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.180070+0000","last_change":"2024-09-16T03:40:47.179954+0000","last_active":"2024-09-16T03:40:47.180070+0000","last_peered":"2024-09-16T03:40:47.180070+0000","last_clean":"2024-09-16T03:40:47.180070+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:40:47.180070+0000","last_undegraded":"2024-09-16T03:40:47.180070+0000","last_fullsized":"2024-09-16T03:40:47.180070+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":878,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:48.342569+0000","last_change":"2024-09-16T03:40:25.635449+0000","last_active":"2024-09-16T03:40:48.342569+0000","last_peered":"2024-09-16T03:40:48.342569+0000","last_clean":"2024-09-16T03:40:48.342569+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:40:48.342569+0000","last_undegraded":"2024-09-16T03:40:48.342569+0000","last_fullsized":"2024-09-16T03:40:48.342569+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:03.730604+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":154,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.408977+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:40:47.408977+0000","last_peered":"2024-09-16T03:40:47.408977+0000","last_clean":"2024-09-16T03:40:47.408977+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:40:47.408977+0000","last_undegraded":"2024-09-16T03:40:47.408977+0000","last_fullsized":"2024-09-16T03:40:47.408977+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":792,"reported_epoch":222,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:40:49.604141+0000","last_change":"2024-09-16T03:40:48.048391+0000","last_active":"2024-09-16T03:40:49.604141+0000","last_peered":"2024-09-16T03:40:49.604141+0000","last_clean":"2024-09-16T03:39:52.800868+0000","last_became_active":"2024-09-16T03:40:47.849953+0000","last_became_peered":"2024-09-16T03:40:47.849953+0000","last_unstale":"2024-09-16T03:40:49.604141+0000","last_undegraded":"2024-09-16T03:40:49.604141+0000","last_fullsized":"2024-09-16T03:40:49.604141+0000","mapping_epoch":221,"log_start":"168'338","ondisk_log_start":"168'338","created":31,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":350,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":22258960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[0,8,2],"avail_no_missing":["0","2","8"],"object_location_counts":[{"shards":"0,2,8","objects":39}],"blocked_by":[],"up_primary":15,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":428,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:46.865023+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:40:46.865023+0000","last_peered":"2024-09-16T03:40:46.865023+0000","last_clean":"2024-09-16T03:40:46.865023+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:40:46.865023+0000","last_undegraded":"2024-09-16T03:40:46.865023+0000","last_fullsized":"2024-09-16T03:40:46.865023+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":183,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-09-16T03:40:46.878610+0000","last_change":"2024-09-16T03:39:09.848905+0000","last_active":"2024-09-16T03:40:46.878610+0000","last_peered":"2024-09-16T03:40:46.878610+0000","last_clean":"2024-09-16T03:40:46.878610+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:46.878610+0000","last_undegraded":"2024-09-16T03:40:46.878610+0000","last_fullsized":"2024-09-16T03:40:46.878610+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:52.781538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":437,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:49.114710+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:40:49.114710+0000","last_peered":"2024-09-16T03:40:49.114710+0000","last_clean":"2024-09-16T03:40:49.114710+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:40:49.114710+0000","last_undegraded":"2024-09-16T03:40:49.114710+0000","last_fullsized":"2024-09-16T03:40:49.114710+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":423,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.409072+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:40:47.409072+0000","last_peered":"2024-09-16T03:40:47.409072+0000","last_clean":"2024-09-16T03:40:47.409072+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:47.409072+0000","last_undegraded":"2024-09-16T03:40:47.409072+0000","last_fullsized":"2024-09-16T03:40:47.409072+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":103,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":97684488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199913472,"data_stored":213522064,"data_compressed":13083793,"data_compressed_allocated":13971456,"data_compressed_original":28188672,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1310720,"data_stored":914947,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,3,4,6,7,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247859,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65284,"kb_used_data":34084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94306556,"statfs":{"total":96636764160,"available":96569913344,"internally_reserved":0,"allocated":34902016,"data_stored":37402304,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247858,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52612,"kb_used_data":17264,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319228,"statfs":{"total":96636764160,"available":96582889472,"internally_reserved":0,"allocated":17678336,"data_stored":18568407,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63680,"kb_used_data":32488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94308160,"statfs":{"total":96636764160,"available":96571555840,"internally_reserved":0,"allocated":33267712,"data_stored":35397581,"data_compressed":2306856,"data_compressed_allocated":2498560,"data_compressed_original":5300224,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28320,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343520,"statfs":{"total":96636764160,"available":96607764480,"internally_reserved":0,"allocated":1323008,"data_stored":921945,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280563,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46648,"kb_used_data":15456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325192,"statfs":{"total":96636764160,"available":96588996608,"internally_reserved":0,"allocated":15826944,"data_stored":16308720,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313268,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1310720,"data_stored":915584,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":1351680,"data_stored":925440,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280562,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83576,"kb_used_data":48216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94288264,"statfs":{"total":96636764160,"available":96551182336,"internally_reserved":0,"allocated":49373184,"data_stored":52268863,"data_compressed":2976505,"data_compressed_allocated":3227648,"data_compressed_original":6828032,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313268,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42088,"kb_used_data":10896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329752,"statfs":{"total":96636764160,"available":96593666048,"internally_reserved":0,"allocated":11157504,"data_stored":12046739,"data_compressed":856662,"data_compressed_allocated":954368,"data_compressed_original":2396160,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280563,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32480,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339360,"statfs":{"total":96636764160,"available":96603504640,"internally_reserved":0,"allocated":1310720,"data_stored":915044,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280562,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50992,"kb_used_data":15644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320848,"statfs":{"total":96636764160,"available":96584548352,"internally_reserved":0,"allocated":16019456,"data_stored":16562011,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280562,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46628,"kb_used_data":15432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325212,"statfs":{"total":96636764160,"available":96589017088,"internally_reserved":0,"allocated":15802368,"data_stored":16293042,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1335296,"data_stored":917370,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32472,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1310720,"data_stored":915623,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46828,"kb_used_data":15640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325012,"statfs":{"total":96636764160,"available":96588812288,"internally_reserved":0,"allocated":16015360,"data_stored":16567478,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31858688,"data_stored":33943544,"data_compressed":2174030,"data_compressed_allocated":2314240,"data_compressed_original":4501504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":48046080,"data_stored":51351621,"data_compressed":2874927,"data_compressed_allocated":3084288,"data_compressed_original":6541312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11636736,"data_stored":12525940,"data_compressed":804157,"data_compressed_allocated":860160,"data_compressed_original":1781760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:51.386 INFO:tasks.ceph.osd.14.smithi155.stderr:2024-09-16T03:40:51.384+0000 7f928a16a640 -1 osd.14 223 handle_osd_map injecting map crc failure 2024-09-16T03:40:51.461 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:40:51.459+0000 7fedb5739640 -1 osd.9 223 handle_osd_map injecting map crc failure 2024-09-16T03:40:52.388 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:40:52.386+0000 7f12e4a1e640 -1 osd.5 224 handle_osd_map injecting map crc failure 2024-09-16T03:40:52.390 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:40:52.389+0000 7fa38a6ea640 -1 osd.10 224 handle_osd_map injecting map crc failure 2024-09-16T03:40:52.393 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:40:52.391+0000 7ff26836a640 -1 osd.11 224 handle_osd_map injecting map crc failure 2024-09-16T03:40:54.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:54.648 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:54.649 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:40:54.661 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-16T03:40:53.727435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":67,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":114033124,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":745756,"kb_used_data":225892,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509203684,"statfs":{"total":1546188226560,"available":1545424572416,"internally_reserved":0,"allocated":231313408,"data_stored":240571059,"data_compressed":15616464,"data_compressed_allocated":17248256,"data_compressed_original":35889152,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-47,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":21373725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.040458"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":632,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-16T03:40:51.388780+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:40:51.388780+0000","last_peered":"2024-09-16T03:40:51.388780+0000","last_clean":"2024-09-16T03:40:51.388780+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:40:51.388780+0000","last_undegraded":"2024-09-16T03:40:51.388780+0000","last_fullsized":"2024-09-16T03:40:51.388780+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":150,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-16T03:40:51.388739+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:40:51.388739+0000","last_peered":"2024-09-16T03:40:51.388739+0000","last_clean":"2024-09-16T03:40:51.388739+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:40:51.388739+0000","last_undegraded":"2024-09-16T03:40:51.388739+0000","last_fullsized":"2024-09-16T03:40:51.388739+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":952,"reported_epoch":223,"state":"peering","last_fresh":"2024-09-16T03:40:51.181287+0000","last_change":"2024-09-16T03:40:51.389251+0000","last_active":"2024-09-16T03:40:51.389251+0000","last_peered":"2024-09-16T03:40:51.181287+0000","last_clean":"2024-09-16T03:40:51.181287+0000","last_became_active":"2024-09-16T03:40:16.729881+0000","last_became_peered":"2024-09-16T03:40:16.729881+0000","last_unstale":"2024-09-16T03:40:51.181287+0000","last_undegraded":"2024-09-16T03:40:51.181287+0000","last_fullsized":"2024-09-16T03:40:51.181287+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4,5],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":455,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.408964+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:40:47.408964+0000","last_peered":"2024-09-16T03:40:47.408964+0000","last_clean":"2024-09-16T03:40:47.408964+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:40:47.408964+0000","last_undegraded":"2024-09-16T03:40:47.408964+0000","last_fullsized":"2024-09-16T03:40:47.408964+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":826,"reported_epoch":224,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:52.041135+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:40:52.041135+0000","last_peered":"2024-09-16T03:40:52.041135+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:40:52.041135+0000","last_undegraded":"2024-09-16T03:40:52.041135+0000","last_fullsized":"2024-09-16T03:40:52.041135+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-16T03:40:51.394688+0000","last_change":"2024-09-16T03:40:47.179954+0000","last_active":"2024-09-16T03:40:51.394688+0000","last_peered":"2024-09-16T03:40:51.394688+0000","last_clean":"2024-09-16T03:40:51.394688+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:40:51.394688+0000","last_undegraded":"2024-09-16T03:40:51.394688+0000","last_fullsized":"2024-09-16T03:40:51.394688+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":878,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:48.342569+0000","last_change":"2024-09-16T03:40:25.635449+0000","last_active":"2024-09-16T03:40:48.342569+0000","last_peered":"2024-09-16T03:40:48.342569+0000","last_clean":"2024-09-16T03:40:48.342569+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:40:48.342569+0000","last_undegraded":"2024-09-16T03:40:48.342569+0000","last_fullsized":"2024-09-16T03:40:48.342569+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:03.730604+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":154,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.408977+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:40:47.408977+0000","last_peered":"2024-09-16T03:40:47.408977+0000","last_clean":"2024-09-16T03:40:47.408977+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:40:47.408977+0000","last_undegraded":"2024-09-16T03:40:47.408977+0000","last_fullsized":"2024-09-16T03:40:47.408977+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":792,"reported_epoch":222,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:40:49.604141+0000","last_change":"2024-09-16T03:40:48.048391+0000","last_active":"2024-09-16T03:40:49.604141+0000","last_peered":"2024-09-16T03:40:49.604141+0000","last_clean":"2024-09-16T03:39:52.800868+0000","last_became_active":"2024-09-16T03:40:47.849953+0000","last_became_peered":"2024-09-16T03:40:47.849953+0000","last_unstale":"2024-09-16T03:40:49.604141+0000","last_undegraded":"2024-09-16T03:40:49.604141+0000","last_fullsized":"2024-09-16T03:40:49.604141+0000","mapping_epoch":221,"log_start":"168'338","ondisk_log_start":"168'338","created":31,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":350,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":22258960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[0,8,2],"avail_no_missing":["0","2","8"],"object_location_counts":[{"shards":"0,2,8","objects":39}],"blocked_by":[],"up_primary":15,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":434,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-16T03:40:52.041136+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:40:52.041136+0000","last_peered":"2024-09-16T03:40:52.041136+0000","last_clean":"2024-09-16T03:40:52.041136+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:40:52.041136+0000","last_undegraded":"2024-09-16T03:40:52.041136+0000","last_fullsized":"2024-09-16T03:40:52.041136+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":187,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-16T03:40:49.688029+0000","last_change":"2024-09-16T03:39:09.848905+0000","last_active":"2024-09-16T03:40:49.688029+0000","last_peered":"2024-09-16T03:40:49.688029+0000","last_clean":"2024-09-16T03:40:49.688029+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:49.688029+0000","last_undegraded":"2024-09-16T03:40:49.688029+0000","last_fullsized":"2024-09-16T03:40:49.688029+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:52.781538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":437,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:49.114710+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:40:49.114710+0000","last_peered":"2024-09-16T03:40:49.114710+0000","last_clean":"2024-09-16T03:40:49.114710+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:40:49.114710+0000","last_undegraded":"2024-09-16T03:40:49.114710+0000","last_fullsized":"2024-09-16T03:40:49.114710+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":423,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.409072+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:40:47.409072+0000","last_peered":"2024-09-16T03:40:47.409072+0000","last_clean":"2024-09-16T03:40:47.409072+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:47.409072+0000","last_undegraded":"2024-09-16T03:40:47.409072+0000","last_fullsized":"2024-09-16T03:40:47.409072+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":67,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":112446340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212029440,"data_stored":226413695,"data_compressed":13977793,"data_compressed_allocated":14921728,"data_compressed_original":29949952,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48212,"kb_used_data":12860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323628,"statfs":{"total":96636764160,"available":96587395072,"internally_reserved":0,"allocated":13168640,"data_stored":13536976,"data_compressed":995578,"data_compressed_allocated":1093632,"data_compressed_original":2048000,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247860,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69480,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302360,"statfs":{"total":96636764160,"available":96565616640,"internally_reserved":0,"allocated":34947072,"data_stored":37433067,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247859,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52656,"kb_used_data":17308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319184,"statfs":{"total":96636764160,"available":96582844416,"internally_reserved":0,"allocated":17723392,"data_stored":18599199,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63680,"kb_used_data":32488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94308160,"statfs":{"total":96636764160,"available":96571555840,"internally_reserved":0,"allocated":33267712,"data_stored":35397581,"data_compressed":2306856,"data_compressed_allocated":2498560,"data_compressed_original":5300224,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28320,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343520,"statfs":{"total":96636764160,"available":96607764480,"internally_reserved":0,"allocated":1323008,"data_stored":921945,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280563,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46648,"kb_used_data":15456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325192,"statfs":{"total":96636764160,"available":96588996608,"internally_reserved":0,"allocated":15826944,"data_stored":16308720,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313268,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1310720,"data_stored":915584,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36672,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335168,"statfs":{"total":96636764160,"available":96599212032,"internally_reserved":0,"allocated":1351680,"data_stored":925440,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280563,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83576,"kb_used_data":48228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94288264,"statfs":{"total":96636764160,"available":96551182336,"internally_reserved":0,"allocated":49385472,"data_stored":52276647,"data_compressed":2976505,"data_compressed_allocated":3227648,"data_compressed_original":6828032,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313268,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42088,"kb_used_data":10896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329752,"statfs":{"total":96636764160,"available":96593666048,"internally_reserved":0,"allocated":11157504,"data_stored":12046739,"data_compressed":856662,"data_compressed_allocated":954368,"data_compressed_original":2396160,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280563,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32480,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339360,"statfs":{"total":96636764160,"available":96603504640,"internally_reserved":0,"allocated":1310720,"data_stored":915044,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280563,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51004,"kb_used_data":15652,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320836,"statfs":{"total":96636764160,"available":96584536064,"internally_reserved":0,"allocated":16027648,"data_stored":16569493,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280563,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46660,"kb_used_data":15468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325180,"statfs":{"total":96636764160,"available":96588984320,"internally_reserved":0,"allocated":15839232,"data_stored":16316377,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1335296,"data_stored":917370,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32472,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339368,"statfs":{"total":96636764160,"available":96603512832,"internally_reserved":0,"allocated":1310720,"data_stored":915623,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46840,"kb_used_data":15652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325000,"statfs":{"total":96636764160,"available":96588800000,"internally_reserved":0,"allocated":16027648,"data_stored":16575254,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31858688,"data_stored":33943544,"data_compressed":2174030,"data_compressed_allocated":2314240,"data_compressed_original":4501504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":48046080,"data_stored":51351621,"data_compressed":2874927,"data_compressed_allocated":3084288,"data_compressed_original":6541312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11636736,"data_stored":12525940,"data_compressed":804157,"data_compressed_allocated":860160,"data_compressed_original":1781760,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":12115968,"data_stored":12891631,"data_compressed":894000,"data_compressed_allocated":950272,"data_compressed_original":1761280,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:54.663 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T03:40:55.038 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:55.038 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":257,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":227,"num_osds":16,"num_up_osds":16,"osd_up_since":1726457810,"num_in_osds":16,"osd_in_since":1726458014,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+remapped+backfill_toofull","count":1},{"state_name":"active+remapped+backfilling","count":1},{"state_name":"peering","count":1}],"num_pgs":13,"num_pools":3,"num_objects":150,"data_bytes":103194545,"bytes_used":763654144,"bytes_avail":1545424572416,"bytes_total":1546188226560,"inactive_pgs_ratio":0.076923079788684845,"misplaced_objects":67,"misplaced_total":450,"misplaced_ratio":0.14888888888888888,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":5289925,"recovering_keys_per_sec":0,"num_objects_recovered":47,"num_bytes_recovered":21373725,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T03:36:37:310011+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T03:37:51.688536+0000","services":{}},"progress_events":{}} 2024-09-16T03:40:55.051 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-16T03:40:55.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:55.354 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:55.355 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:40:55.367 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-09-16T03:40:54.718838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":67,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":114033124,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":753276,"kb_used_data":233428,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509196164,"statfs":{"total":1546188226560,"available":1545416871936,"internally_reserved":0,"allocated":239030272,"data_stored":248517661,"data_compressed":15849786,"data_compressed_allocated":17506304,"data_compressed_original":36544512,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-36,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":14761852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.992191"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":632,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-16T03:40:51.388780+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:40:51.388780+0000","last_peered":"2024-09-16T03:40:51.388780+0000","last_clean":"2024-09-16T03:40:51.388780+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:40:51.388780+0000","last_undegraded":"2024-09-16T03:40:51.388780+0000","last_fullsized":"2024-09-16T03:40:51.388780+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":150,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-16T03:40:51.388739+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:40:51.388739+0000","last_peered":"2024-09-16T03:40:51.388739+0000","last_clean":"2024-09-16T03:40:51.388739+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:40:51.388739+0000","last_undegraded":"2024-09-16T03:40:51.388739+0000","last_fullsized":"2024-09-16T03:40:51.388739+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":952,"reported_epoch":223,"state":"peering","last_fresh":"2024-09-16T03:40:51.181287+0000","last_change":"2024-09-16T03:40:51.389251+0000","last_active":"2024-09-16T03:40:51.389251+0000","last_peered":"2024-09-16T03:40:51.181287+0000","last_clean":"2024-09-16T03:40:51.181287+0000","last_became_active":"2024-09-16T03:40:16.729881+0000","last_became_peered":"2024-09-16T03:40:16.729881+0000","last_unstale":"2024-09-16T03:40:51.181287+0000","last_undegraded":"2024-09-16T03:40:51.181287+0000","last_fullsized":"2024-09-16T03:40:51.181287+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4,5],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":455,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.408964+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:40:47.408964+0000","last_peered":"2024-09-16T03:40:47.408964+0000","last_clean":"2024-09-16T03:40:47.408964+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:40:47.408964+0000","last_undegraded":"2024-09-16T03:40:47.408964+0000","last_fullsized":"2024-09-16T03:40:47.408964+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":826,"reported_epoch":224,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:52.041135+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:40:52.041135+0000","last_peered":"2024-09-16T03:40:52.041135+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:40:52.041135+0000","last_undegraded":"2024-09-16T03:40:52.041135+0000","last_fullsized":"2024-09-16T03:40:52.041135+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-16T03:40:51.394688+0000","last_change":"2024-09-16T03:40:47.179954+0000","last_active":"2024-09-16T03:40:51.394688+0000","last_peered":"2024-09-16T03:40:51.394688+0000","last_clean":"2024-09-16T03:40:51.394688+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:40:51.394688+0000","last_undegraded":"2024-09-16T03:40:51.394688+0000","last_fullsized":"2024-09-16T03:40:51.394688+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":885,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-16T03:40:53.707430+0000","last_change":"2024-09-16T03:40:25.635449+0000","last_active":"2024-09-16T03:40:53.707430+0000","last_peered":"2024-09-16T03:40:53.707430+0000","last_clean":"2024-09-16T03:40:53.707430+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:40:53.707430+0000","last_undegraded":"2024-09-16T03:40:53.707430+0000","last_fullsized":"2024-09-16T03:40:53.707430+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:03.730604+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":154,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.408977+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:40:47.408977+0000","last_peered":"2024-09-16T03:40:47.408977+0000","last_clean":"2024-09-16T03:40:47.408977+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:40:47.408977+0000","last_undegraded":"2024-09-16T03:40:47.408977+0000","last_fullsized":"2024-09-16T03:40:47.408977+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":792,"reported_epoch":222,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:40:49.604141+0000","last_change":"2024-09-16T03:40:48.048391+0000","last_active":"2024-09-16T03:40:49.604141+0000","last_peered":"2024-09-16T03:40:49.604141+0000","last_clean":"2024-09-16T03:39:52.800868+0000","last_became_active":"2024-09-16T03:40:47.849953+0000","last_became_peered":"2024-09-16T03:40:47.849953+0000","last_unstale":"2024-09-16T03:40:49.604141+0000","last_undegraded":"2024-09-16T03:40:49.604141+0000","last_fullsized":"2024-09-16T03:40:49.604141+0000","mapping_epoch":221,"log_start":"168'338","ondisk_log_start":"168'338","created":31,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:37:03.913037+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":350,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":22258960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[0,8,2],"avail_no_missing":["0","2","8"],"object_location_counts":[{"shards":"0,2,8","objects":39}],"blocked_by":[],"up_primary":15,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":434,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-16T03:40:52.041136+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:40:52.041136+0000","last_peered":"2024-09-16T03:40:52.041136+0000","last_clean":"2024-09-16T03:40:52.041136+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:40:52.041136+0000","last_undegraded":"2024-09-16T03:40:52.041136+0000","last_fullsized":"2024-09-16T03:40:52.041136+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":187,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-16T03:40:49.688029+0000","last_change":"2024-09-16T03:39:09.848905+0000","last_active":"2024-09-16T03:40:49.688029+0000","last_peered":"2024-09-16T03:40:49.688029+0000","last_clean":"2024-09-16T03:40:49.688029+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:49.688029+0000","last_undegraded":"2024-09-16T03:40:49.688029+0000","last_fullsized":"2024-09-16T03:40:49.688029+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:38:23.737833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:38:23.737833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:40:52.781538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":445,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-16T03:40:53.741300+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:40:53.741300+0000","last_peered":"2024-09-16T03:40:53.741300+0000","last_clean":"2024-09-16T03:40:53.741300+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:40:53.741300+0000","last_undegraded":"2024-09-16T03:40:53.741300+0000","last_fullsized":"2024-09-16T03:40:53.741300+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":423,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-16T03:40:47.409072+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:40:47.409072+0000","last_peered":"2024-09-16T03:40:47.409072+0000","last_clean":"2024-09-16T03:40:47.409072+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:47.409072+0000","last_undegraded":"2024-09-16T03:40:47.409072+0000","last_fullsized":"2024-09-16T03:40:47.409072+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":67,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":112446340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217538560,"data_stored":232184212,"data_compressed":14129381,"data_compressed_allocated":15089664,"data_compressed_original":30408704,"omap_allocated":0,"internal_metadata":0},"log_size":13,"ondisk_log_size":13,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247859,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48212,"kb_used_data":12860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323628,"statfs":{"total":96636764160,"available":96587395072,"internally_reserved":0,"allocated":13168640,"data_stored":13536976,"data_compressed":995578,"data_compressed_allocated":1093632,"data_compressed_original":2048000,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247860,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69480,"kb_used_data":34128,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302360,"statfs":{"total":96636764160,"available":96565616640,"internally_reserved":0,"allocated":34947072,"data_stored":37433067,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247859,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52656,"kb_used_data":17308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319184,"statfs":{"total":96636764160,"available":96582844416,"internally_reserved":0,"allocated":17723392,"data_stored":18599199,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280565,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63724,"kb_used_data":32532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94308116,"statfs":{"total":96636764160,"available":96571510784,"internally_reserved":0,"allocated":33312768,"data_stored":35428217,"data_compressed":2306856,"data_compressed_allocated":2498560,"data_compressed_original":5300224,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28360,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343480,"statfs":{"total":96636764160,"available":96607723520,"internally_reserved":0,"allocated":1355776,"data_stored":944850,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46672,"kb_used_data":15480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325168,"statfs":{"total":96636764160,"available":96588972032,"internally_reserved":0,"allocated":15851520,"data_stored":16324218,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":946294,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1388544,"data_stored":948639,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280563,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83576,"kb_used_data":48228,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94288264,"statfs":{"total":96636764160,"available":96551182336,"internally_reserved":0,"allocated":49385472,"data_stored":52276647,"data_compressed":2976505,"data_compressed_allocated":3227648,"data_compressed_original":6828032,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313269,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49288,"kb_used_data":18100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322552,"statfs":{"total":96636764160,"available":96586293248,"internally_reserved":0,"allocated":18534400,"data_stored":19770007,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1359872,"data_stored":946027,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280563,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51004,"kb_used_data":15652,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320836,"statfs":{"total":96636764160,"available":96584536064,"internally_reserved":0,"allocated":16027648,"data_stored":16569493,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280563,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46660,"kb_used_data":15468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325180,"statfs":{"total":96636764160,"available":96588984320,"internally_reserved":0,"allocated":15839232,"data_stored":16316377,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1380352,"data_stored":948101,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32532,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1372160,"data_stored":954295,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46840,"kb_used_data":15652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325000,"statfs":{"total":96636764160,"available":96588800000,"internally_reserved":0,"allocated":16027648,"data_stored":16575254,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31858688,"data_stored":33943544,"data_compressed":2174030,"data_compressed_allocated":2314240,"data_compressed_original":4501504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":48046080,"data_stored":51351621,"data_compressed":2874927,"data_compressed_allocated":3084288,"data_compressed_original":6541312,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":12115968,"data_stored":12891631,"data_compressed":894000,"data_compressed_allocated":950272,"data_compressed_original":1761280,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:56.918 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:40:56.916+0000 7f8be56cd640 -1 osd.7 227 handle_osd_map injecting map crc failure 2024-09-16T03:40:58.369 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:58.681 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:58.682 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:40:58.694 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-16T03:40:57.733690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":739668,"kb_used_data":219808,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509209772,"statfs":{"total":1546188226560,"available":1545430806528,"internally_reserved":0,"allocated":225083392,"data_stored":233476382,"data_compressed":15218326,"data_compressed_allocated":16814080,"data_compressed_original":34803712,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-28,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9035214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027705"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":640,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.918648+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:40:56.918648+0000","last_peered":"2024-09-16T03:40:56.918648+0000","last_clean":"2024-09-16T03:40:56.918648+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:40:56.918648+0000","last_undegraded":"2024-09-16T03:40:56.918648+0000","last_fullsized":"2024-09-16T03:40:56.918648+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":158,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.918696+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:40:56.918696+0000","last_peered":"2024-09-16T03:40:56.918696+0000","last_clean":"2024-09-16T03:40:56.918696+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:40:56.918696+0000","last_undegraded":"2024-09-16T03:40:56.918696+0000","last_fullsized":"2024-09-16T03:40:56.918696+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":968,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.019013+0000","last_change":"2024-09-16T03:40:52.880495+0000","last_active":"2024-09-16T03:40:57.019013+0000","last_peered":"2024-09-16T03:40:57.019013+0000","last_clean":"2024-09-16T03:40:57.019013+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:40:57.019013+0000","last_undegraded":"2024-09-16T03:40:57.019013+0000","last_fullsized":"2024-09-16T03:40:57.019013+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.728291+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:40:54.728291+0000","last_peered":"2024-09-16T03:40:54.728291+0000","last_clean":"2024-09-16T03:40:54.728291+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:40:54.728291+0000","last_undegraded":"2024-09-16T03:40:54.728291+0000","last_fullsized":"2024-09-16T03:40:54.728291+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":834,"reported_epoch":228,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:57.086305+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:40:57.086305+0000","last_peered":"2024-09-16T03:40:57.086305+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:40:57.086305+0000","last_undegraded":"2024-09-16T03:40:57.086305+0000","last_fullsized":"2024-09-16T03:40:57.086305+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.915401+0000","last_change":"2024-09-16T03:40:47.179954+0000","last_active":"2024-09-16T03:40:56.915401+0000","last_peered":"2024-09-16T03:40:56.915401+0000","last_clean":"2024-09-16T03:40:56.915401+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:40:56.915401+0000","last_undegraded":"2024-09-16T03:40:56.915401+0000","last_fullsized":"2024-09-16T03:40:56.915401+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":885,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-16T03:40:53.707430+0000","last_change":"2024-09-16T03:40:25.635449+0000","last_active":"2024-09-16T03:40:53.707430+0000","last_peered":"2024-09-16T03:40:53.707430+0000","last_clean":"2024-09-16T03:40:53.707430+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:40:53.707430+0000","last_undegraded":"2024-09-16T03:40:53.707430+0000","last_fullsized":"2024-09-16T03:40:53.707430+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:03.730604+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":163,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.828400+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:40:54.828400+0000","last_peered":"2024-09-16T03:40:54.828400+0000","last_clean":"2024-09-16T03:40:54.828400+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:40:54.828400+0000","last_undegraded":"2024-09-16T03:40:54.828400+0000","last_fullsized":"2024-09-16T03:40:54.828400+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":875,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.815393+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:40:56.815393+0000","last_peered":"2024-09-16T03:40:56.815393+0000","last_clean":"2024-09-16T03:40:56.815393+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:40:56.815393+0000","last_undegraded":"2024-09-16T03:40:56.815393+0000","last_fullsized":"2024-09-16T03:40:56.815393+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":442,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.086316+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:40:57.086316+0000","last_peered":"2024-09-16T03:40:57.086316+0000","last_clean":"2024-09-16T03:40:57.086316+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:40:57.086316+0000","last_undegraded":"2024-09-16T03:40:57.086316+0000","last_fullsized":"2024-09-16T03:40:57.086316+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.817488+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:40:54.817488+0000","last_peered":"2024-09-16T03:40:54.817488+0000","last_clean":"2024-09-16T03:40:54.817488+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:54.817488+0000","last_undegraded":"2024-09-16T03:40:54.817488+0000","last_fullsized":"2024-09-16T03:40:54.817488+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":445,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-16T03:40:53.741300+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:40:53.741300+0000","last_peered":"2024-09-16T03:40:53.741300+0000","last_clean":"2024-09-16T03:40:53.741300+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:40:53.741300+0000","last_undegraded":"2024-09-16T03:40:53.741300+0000","last_fullsized":"2024-09-16T03:40:53.741300+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":432,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.728320+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:40:54.728320+0000","last_peered":"2024-09-16T03:40:54.728320+0000","last_clean":"2024-09-16T03:40:54.728320+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:54.728320+0000","last_undegraded":"2024-09-16T03:40:54.728320+0000","last_fullsized":"2024-09-16T03:40:54.728320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202989568,"data_stored":216643211,"data_compressed":13497921,"data_compressed_allocated":14397440,"data_compressed_original":28667904,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51076,"kb_used_data":15728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320764,"statfs":{"total":96636764160,"available":96584462336,"internally_reserved":0,"allocated":16105472,"data_stored":16615882,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247861,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69516,"kb_used_data":34160,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302324,"statfs":{"total":96636764160,"available":96565579776,"internally_reserved":0,"allocated":34979840,"data_stored":37455973,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247860,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52708,"kb_used_data":17352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319132,"statfs":{"total":96636764160,"available":96582791168,"internally_reserved":0,"allocated":17768448,"data_stored":18629806,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280565,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63724,"kb_used_data":32532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94308116,"statfs":{"total":96636764160,"available":96571510784,"internally_reserved":0,"allocated":33312768,"data_stored":35428217,"data_compressed":2306856,"data_compressed_allocated":2498560,"data_compressed_original":5300224,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28360,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343480,"statfs":{"total":96636764160,"available":96607723520,"internally_reserved":0,"allocated":1355776,"data_stored":944850,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46672,"kb_used_data":15480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325168,"statfs":{"total":96636764160,"available":96588972032,"internally_reserved":0,"allocated":15851520,"data_stored":16324218,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":946294,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1388544,"data_stored":948639,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66876,"kb_used_data":31528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304964,"statfs":{"total":96636764160,"available":96568283136,"internally_reserved":0,"allocated":32284672,"data_stored":34010826,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313269,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49288,"kb_used_data":18100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322552,"statfs":{"total":96636764160,"available":96586293248,"internally_reserved":0,"allocated":18534400,"data_stored":19770007,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1359872,"data_stored":946027,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51060,"kb_used_data":15708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320780,"statfs":{"total":96636764160,"available":96584478720,"internally_reserved":0,"allocated":16084992,"data_stored":16607897,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46688,"kb_used_data":15500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325152,"statfs":{"total":96636764160,"available":96588955648,"internally_reserved":0,"allocated":15872000,"data_stored":16339216,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1380352,"data_stored":948101,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32532,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1372160,"data_stored":954295,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46896,"kb_used_data":15700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324944,"statfs":{"total":96636764160,"available":96588742656,"internally_reserved":0,"allocated":16076800,"data_stored":16606134,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31858688,"data_stored":33943544,"data_compressed":2174030,"data_compressed_allocated":2314240,"data_compressed_original":4501504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:58.695 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T03:40:58.696 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T03:40:58.702 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T03:40:58.703 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:58.998 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:58.999 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:40:59.011 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-16T03:40:57.733690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":739668,"kb_used_data":219808,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509209772,"statfs":{"total":1546188226560,"available":1545430806528,"internally_reserved":0,"allocated":225083392,"data_stored":233476382,"data_compressed":15218326,"data_compressed_allocated":16814080,"data_compressed_original":34803712,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-28,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9035214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027705"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":640,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.918648+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:40:56.918648+0000","last_peered":"2024-09-16T03:40:56.918648+0000","last_clean":"2024-09-16T03:40:56.918648+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:40:56.918648+0000","last_undegraded":"2024-09-16T03:40:56.918648+0000","last_fullsized":"2024-09-16T03:40:56.918648+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":158,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.918696+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:40:56.918696+0000","last_peered":"2024-09-16T03:40:56.918696+0000","last_clean":"2024-09-16T03:40:56.918696+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:40:56.918696+0000","last_undegraded":"2024-09-16T03:40:56.918696+0000","last_fullsized":"2024-09-16T03:40:56.918696+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":968,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.019013+0000","last_change":"2024-09-16T03:40:52.880495+0000","last_active":"2024-09-16T03:40:57.019013+0000","last_peered":"2024-09-16T03:40:57.019013+0000","last_clean":"2024-09-16T03:40:57.019013+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:40:57.019013+0000","last_undegraded":"2024-09-16T03:40:57.019013+0000","last_fullsized":"2024-09-16T03:40:57.019013+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.728291+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:40:54.728291+0000","last_peered":"2024-09-16T03:40:54.728291+0000","last_clean":"2024-09-16T03:40:54.728291+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:40:54.728291+0000","last_undegraded":"2024-09-16T03:40:54.728291+0000","last_fullsized":"2024-09-16T03:40:54.728291+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":834,"reported_epoch":228,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:57.086305+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:40:57.086305+0000","last_peered":"2024-09-16T03:40:57.086305+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:40:57.086305+0000","last_undegraded":"2024-09-16T03:40:57.086305+0000","last_fullsized":"2024-09-16T03:40:57.086305+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.915401+0000","last_change":"2024-09-16T03:40:47.179954+0000","last_active":"2024-09-16T03:40:56.915401+0000","last_peered":"2024-09-16T03:40:56.915401+0000","last_clean":"2024-09-16T03:40:56.915401+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:40:56.915401+0000","last_undegraded":"2024-09-16T03:40:56.915401+0000","last_fullsized":"2024-09-16T03:40:56.915401+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":885,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-16T03:40:53.707430+0000","last_change":"2024-09-16T03:40:25.635449+0000","last_active":"2024-09-16T03:40:53.707430+0000","last_peered":"2024-09-16T03:40:53.707430+0000","last_clean":"2024-09-16T03:40:53.707430+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:40:53.707430+0000","last_undegraded":"2024-09-16T03:40:53.707430+0000","last_fullsized":"2024-09-16T03:40:53.707430+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:03.730604+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":163,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.828400+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:40:54.828400+0000","last_peered":"2024-09-16T03:40:54.828400+0000","last_clean":"2024-09-16T03:40:54.828400+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:40:54.828400+0000","last_undegraded":"2024-09-16T03:40:54.828400+0000","last_fullsized":"2024-09-16T03:40:54.828400+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":875,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.815393+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:40:56.815393+0000","last_peered":"2024-09-16T03:40:56.815393+0000","last_clean":"2024-09-16T03:40:56.815393+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:40:56.815393+0000","last_undegraded":"2024-09-16T03:40:56.815393+0000","last_fullsized":"2024-09-16T03:40:56.815393+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":442,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.086316+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:40:57.086316+0000","last_peered":"2024-09-16T03:40:57.086316+0000","last_clean":"2024-09-16T03:40:57.086316+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:40:57.086316+0000","last_undegraded":"2024-09-16T03:40:57.086316+0000","last_fullsized":"2024-09-16T03:40:57.086316+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.817488+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:40:54.817488+0000","last_peered":"2024-09-16T03:40:54.817488+0000","last_clean":"2024-09-16T03:40:54.817488+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:54.817488+0000","last_undegraded":"2024-09-16T03:40:54.817488+0000","last_fullsized":"2024-09-16T03:40:54.817488+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":445,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-16T03:40:53.741300+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:40:53.741300+0000","last_peered":"2024-09-16T03:40:53.741300+0000","last_clean":"2024-09-16T03:40:53.741300+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:40:53.741300+0000","last_undegraded":"2024-09-16T03:40:53.741300+0000","last_fullsized":"2024-09-16T03:40:53.741300+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":432,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.728320+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:40:54.728320+0000","last_peered":"2024-09-16T03:40:54.728320+0000","last_clean":"2024-09-16T03:40:54.728320+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:54.728320+0000","last_undegraded":"2024-09-16T03:40:54.728320+0000","last_fullsized":"2024-09-16T03:40:54.728320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202989568,"data_stored":216643211,"data_compressed":13497921,"data_compressed_allocated":14397440,"data_compressed_original":28667904,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51076,"kb_used_data":15728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320764,"statfs":{"total":96636764160,"available":96584462336,"internally_reserved":0,"allocated":16105472,"data_stored":16615882,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247861,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69516,"kb_used_data":34160,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302324,"statfs":{"total":96636764160,"available":96565579776,"internally_reserved":0,"allocated":34979840,"data_stored":37455973,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247860,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52708,"kb_used_data":17352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319132,"statfs":{"total":96636764160,"available":96582791168,"internally_reserved":0,"allocated":17768448,"data_stored":18629806,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280565,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63724,"kb_used_data":32532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94308116,"statfs":{"total":96636764160,"available":96571510784,"internally_reserved":0,"allocated":33312768,"data_stored":35428217,"data_compressed":2306856,"data_compressed_allocated":2498560,"data_compressed_original":5300224,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28360,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343480,"statfs":{"total":96636764160,"available":96607723520,"internally_reserved":0,"allocated":1355776,"data_stored":944850,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46672,"kb_used_data":15480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325168,"statfs":{"total":96636764160,"available":96588972032,"internally_reserved":0,"allocated":15851520,"data_stored":16324218,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":946294,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1388544,"data_stored":948639,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66876,"kb_used_data":31528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304964,"statfs":{"total":96636764160,"available":96568283136,"internally_reserved":0,"allocated":32284672,"data_stored":34010826,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313269,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49288,"kb_used_data":18100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322552,"statfs":{"total":96636764160,"available":96586293248,"internally_reserved":0,"allocated":18534400,"data_stored":19770007,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1359872,"data_stored":946027,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51060,"kb_used_data":15708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320780,"statfs":{"total":96636764160,"available":96584478720,"internally_reserved":0,"allocated":16084992,"data_stored":16607897,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46688,"kb_used_data":15500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325152,"statfs":{"total":96636764160,"available":96588955648,"internally_reserved":0,"allocated":15872000,"data_stored":16339216,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1380352,"data_stored":948101,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32532,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1372160,"data_stored":954295,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46896,"kb_used_data":15700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324944,"statfs":{"total":96636764160,"available":96588742656,"internally_reserved":0,"allocated":16076800,"data_stored":16606134,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31858688,"data_stored":33943544,"data_compressed":2174030,"data_compressed_allocated":2314240,"data_compressed_original":4501504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:59.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:59.308 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:59.308 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:40:59.320 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-16T03:40:57.733690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":739668,"kb_used_data":219808,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509209772,"statfs":{"total":1546188226560,"available":1545430806528,"internally_reserved":0,"allocated":225083392,"data_stored":233476382,"data_compressed":15218326,"data_compressed_allocated":16814080,"data_compressed_original":34803712,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-28,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9035214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027705"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":640,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.918648+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:40:56.918648+0000","last_peered":"2024-09-16T03:40:56.918648+0000","last_clean":"2024-09-16T03:40:56.918648+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:40:56.918648+0000","last_undegraded":"2024-09-16T03:40:56.918648+0000","last_fullsized":"2024-09-16T03:40:56.918648+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":158,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.918696+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:40:56.918696+0000","last_peered":"2024-09-16T03:40:56.918696+0000","last_clean":"2024-09-16T03:40:56.918696+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:40:56.918696+0000","last_undegraded":"2024-09-16T03:40:56.918696+0000","last_fullsized":"2024-09-16T03:40:56.918696+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":968,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.019013+0000","last_change":"2024-09-16T03:40:52.880495+0000","last_active":"2024-09-16T03:40:57.019013+0000","last_peered":"2024-09-16T03:40:57.019013+0000","last_clean":"2024-09-16T03:40:57.019013+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:40:57.019013+0000","last_undegraded":"2024-09-16T03:40:57.019013+0000","last_fullsized":"2024-09-16T03:40:57.019013+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.728291+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:40:54.728291+0000","last_peered":"2024-09-16T03:40:54.728291+0000","last_clean":"2024-09-16T03:40:54.728291+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:40:54.728291+0000","last_undegraded":"2024-09-16T03:40:54.728291+0000","last_fullsized":"2024-09-16T03:40:54.728291+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":834,"reported_epoch":228,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:57.086305+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:40:57.086305+0000","last_peered":"2024-09-16T03:40:57.086305+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:40:57.086305+0000","last_undegraded":"2024-09-16T03:40:57.086305+0000","last_fullsized":"2024-09-16T03:40:57.086305+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.915401+0000","last_change":"2024-09-16T03:40:47.179954+0000","last_active":"2024-09-16T03:40:56.915401+0000","last_peered":"2024-09-16T03:40:56.915401+0000","last_clean":"2024-09-16T03:40:56.915401+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:40:56.915401+0000","last_undegraded":"2024-09-16T03:40:56.915401+0000","last_fullsized":"2024-09-16T03:40:56.915401+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":885,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-16T03:40:53.707430+0000","last_change":"2024-09-16T03:40:25.635449+0000","last_active":"2024-09-16T03:40:53.707430+0000","last_peered":"2024-09-16T03:40:53.707430+0000","last_clean":"2024-09-16T03:40:53.707430+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:40:53.707430+0000","last_undegraded":"2024-09-16T03:40:53.707430+0000","last_fullsized":"2024-09-16T03:40:53.707430+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:03.730604+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":163,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.828400+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:40:54.828400+0000","last_peered":"2024-09-16T03:40:54.828400+0000","last_clean":"2024-09-16T03:40:54.828400+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:40:54.828400+0000","last_undegraded":"2024-09-16T03:40:54.828400+0000","last_fullsized":"2024-09-16T03:40:54.828400+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":875,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.815393+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:40:56.815393+0000","last_peered":"2024-09-16T03:40:56.815393+0000","last_clean":"2024-09-16T03:40:56.815393+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:40:56.815393+0000","last_undegraded":"2024-09-16T03:40:56.815393+0000","last_fullsized":"2024-09-16T03:40:56.815393+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":442,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.086316+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:40:57.086316+0000","last_peered":"2024-09-16T03:40:57.086316+0000","last_clean":"2024-09-16T03:40:57.086316+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:40:57.086316+0000","last_undegraded":"2024-09-16T03:40:57.086316+0000","last_fullsized":"2024-09-16T03:40:57.086316+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.817488+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:40:54.817488+0000","last_peered":"2024-09-16T03:40:54.817488+0000","last_clean":"2024-09-16T03:40:54.817488+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:54.817488+0000","last_undegraded":"2024-09-16T03:40:54.817488+0000","last_fullsized":"2024-09-16T03:40:54.817488+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":445,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-16T03:40:53.741300+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:40:53.741300+0000","last_peered":"2024-09-16T03:40:53.741300+0000","last_clean":"2024-09-16T03:40:53.741300+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:40:53.741300+0000","last_undegraded":"2024-09-16T03:40:53.741300+0000","last_fullsized":"2024-09-16T03:40:53.741300+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":432,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.728320+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:40:54.728320+0000","last_peered":"2024-09-16T03:40:54.728320+0000","last_clean":"2024-09-16T03:40:54.728320+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:54.728320+0000","last_undegraded":"2024-09-16T03:40:54.728320+0000","last_fullsized":"2024-09-16T03:40:54.728320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202989568,"data_stored":216643211,"data_compressed":13497921,"data_compressed_allocated":14397440,"data_compressed_original":28667904,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51076,"kb_used_data":15728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320764,"statfs":{"total":96636764160,"available":96584462336,"internally_reserved":0,"allocated":16105472,"data_stored":16615882,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247861,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69516,"kb_used_data":34160,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302324,"statfs":{"total":96636764160,"available":96565579776,"internally_reserved":0,"allocated":34979840,"data_stored":37455973,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247860,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52708,"kb_used_data":17352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319132,"statfs":{"total":96636764160,"available":96582791168,"internally_reserved":0,"allocated":17768448,"data_stored":18629806,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280565,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63724,"kb_used_data":32532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94308116,"statfs":{"total":96636764160,"available":96571510784,"internally_reserved":0,"allocated":33312768,"data_stored":35428217,"data_compressed":2306856,"data_compressed_allocated":2498560,"data_compressed_original":5300224,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28360,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343480,"statfs":{"total":96636764160,"available":96607723520,"internally_reserved":0,"allocated":1355776,"data_stored":944850,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46672,"kb_used_data":15480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325168,"statfs":{"total":96636764160,"available":96588972032,"internally_reserved":0,"allocated":15851520,"data_stored":16324218,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":946294,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1388544,"data_stored":948639,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66876,"kb_used_data":31528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304964,"statfs":{"total":96636764160,"available":96568283136,"internally_reserved":0,"allocated":32284672,"data_stored":34010826,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313269,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49288,"kb_used_data":18100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322552,"statfs":{"total":96636764160,"available":96586293248,"internally_reserved":0,"allocated":18534400,"data_stored":19770007,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1359872,"data_stored":946027,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51060,"kb_used_data":15708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320780,"statfs":{"total":96636764160,"available":96584478720,"internally_reserved":0,"allocated":16084992,"data_stored":16607897,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46688,"kb_used_data":15500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325152,"statfs":{"total":96636764160,"available":96588955648,"internally_reserved":0,"allocated":15872000,"data_stored":16339216,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1380352,"data_stored":948101,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32532,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1372160,"data_stored":954295,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46896,"kb_used_data":15700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324944,"statfs":{"total":96636764160,"available":96588742656,"internally_reserved":0,"allocated":16076800,"data_stored":16606134,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31858688,"data_stored":33943544,"data_compressed":2174030,"data_compressed_allocated":2314240,"data_compressed_original":4501504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:59.322 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:40:59.617 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:40:59.618 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:40:59.630 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-16T03:40:57.733690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":739668,"kb_used_data":219808,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509209772,"statfs":{"total":1546188226560,"available":1545430806528,"internally_reserved":0,"allocated":225083392,"data_stored":233476382,"data_compressed":15218326,"data_compressed_allocated":16814080,"data_compressed_original":34803712,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-28,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9035214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027705"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":640,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.918648+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:40:56.918648+0000","last_peered":"2024-09-16T03:40:56.918648+0000","last_clean":"2024-09-16T03:40:56.918648+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:40:56.918648+0000","last_undegraded":"2024-09-16T03:40:56.918648+0000","last_fullsized":"2024-09-16T03:40:56.918648+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":158,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.918696+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:40:56.918696+0000","last_peered":"2024-09-16T03:40:56.918696+0000","last_clean":"2024-09-16T03:40:56.918696+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:40:56.918696+0000","last_undegraded":"2024-09-16T03:40:56.918696+0000","last_fullsized":"2024-09-16T03:40:56.918696+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":968,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.019013+0000","last_change":"2024-09-16T03:40:52.880495+0000","last_active":"2024-09-16T03:40:57.019013+0000","last_peered":"2024-09-16T03:40:57.019013+0000","last_clean":"2024-09-16T03:40:57.019013+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:40:57.019013+0000","last_undegraded":"2024-09-16T03:40:57.019013+0000","last_fullsized":"2024-09-16T03:40:57.019013+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.728291+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:40:54.728291+0000","last_peered":"2024-09-16T03:40:54.728291+0000","last_clean":"2024-09-16T03:40:54.728291+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:40:54.728291+0000","last_undegraded":"2024-09-16T03:40:54.728291+0000","last_fullsized":"2024-09-16T03:40:54.728291+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":834,"reported_epoch":228,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:57.086305+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:40:57.086305+0000","last_peered":"2024-09-16T03:40:57.086305+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:40:57.086305+0000","last_undegraded":"2024-09-16T03:40:57.086305+0000","last_fullsized":"2024-09-16T03:40:57.086305+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.915401+0000","last_change":"2024-09-16T03:40:47.179954+0000","last_active":"2024-09-16T03:40:56.915401+0000","last_peered":"2024-09-16T03:40:56.915401+0000","last_clean":"2024-09-16T03:40:56.915401+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:40:56.915401+0000","last_undegraded":"2024-09-16T03:40:56.915401+0000","last_fullsized":"2024-09-16T03:40:56.915401+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":885,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-16T03:40:53.707430+0000","last_change":"2024-09-16T03:40:25.635449+0000","last_active":"2024-09-16T03:40:53.707430+0000","last_peered":"2024-09-16T03:40:53.707430+0000","last_clean":"2024-09-16T03:40:53.707430+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:40:53.707430+0000","last_undegraded":"2024-09-16T03:40:53.707430+0000","last_fullsized":"2024-09-16T03:40:53.707430+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:03.730604+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":163,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.828400+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:40:54.828400+0000","last_peered":"2024-09-16T03:40:54.828400+0000","last_clean":"2024-09-16T03:40:54.828400+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:40:54.828400+0000","last_undegraded":"2024-09-16T03:40:54.828400+0000","last_fullsized":"2024-09-16T03:40:54.828400+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":875,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.815393+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:40:56.815393+0000","last_peered":"2024-09-16T03:40:56.815393+0000","last_clean":"2024-09-16T03:40:56.815393+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:40:56.815393+0000","last_undegraded":"2024-09-16T03:40:56.815393+0000","last_fullsized":"2024-09-16T03:40:56.815393+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":442,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.086316+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:40:57.086316+0000","last_peered":"2024-09-16T03:40:57.086316+0000","last_clean":"2024-09-16T03:40:57.086316+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:40:57.086316+0000","last_undegraded":"2024-09-16T03:40:57.086316+0000","last_fullsized":"2024-09-16T03:40:57.086316+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.817488+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:40:54.817488+0000","last_peered":"2024-09-16T03:40:54.817488+0000","last_clean":"2024-09-16T03:40:54.817488+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:54.817488+0000","last_undegraded":"2024-09-16T03:40:54.817488+0000","last_fullsized":"2024-09-16T03:40:54.817488+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":445,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-16T03:40:53.741300+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:40:53.741300+0000","last_peered":"2024-09-16T03:40:53.741300+0000","last_clean":"2024-09-16T03:40:53.741300+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:40:53.741300+0000","last_undegraded":"2024-09-16T03:40:53.741300+0000","last_fullsized":"2024-09-16T03:40:53.741300+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":432,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.728320+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:40:54.728320+0000","last_peered":"2024-09-16T03:40:54.728320+0000","last_clean":"2024-09-16T03:40:54.728320+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:54.728320+0000","last_undegraded":"2024-09-16T03:40:54.728320+0000","last_fullsized":"2024-09-16T03:40:54.728320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202989568,"data_stored":216643211,"data_compressed":13497921,"data_compressed_allocated":14397440,"data_compressed_original":28667904,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51076,"kb_used_data":15728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320764,"statfs":{"total":96636764160,"available":96584462336,"internally_reserved":0,"allocated":16105472,"data_stored":16615882,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247861,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69516,"kb_used_data":34160,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302324,"statfs":{"total":96636764160,"available":96565579776,"internally_reserved":0,"allocated":34979840,"data_stored":37455973,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247860,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52708,"kb_used_data":17352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319132,"statfs":{"total":96636764160,"available":96582791168,"internally_reserved":0,"allocated":17768448,"data_stored":18629806,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280565,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63724,"kb_used_data":32532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94308116,"statfs":{"total":96636764160,"available":96571510784,"internally_reserved":0,"allocated":33312768,"data_stored":35428217,"data_compressed":2306856,"data_compressed_allocated":2498560,"data_compressed_original":5300224,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28360,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343480,"statfs":{"total":96636764160,"available":96607723520,"internally_reserved":0,"allocated":1355776,"data_stored":944850,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46672,"kb_used_data":15480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325168,"statfs":{"total":96636764160,"available":96588972032,"internally_reserved":0,"allocated":15851520,"data_stored":16324218,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":946294,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36704,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335136,"statfs":{"total":96636764160,"available":96599179264,"internally_reserved":0,"allocated":1388544,"data_stored":948639,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66876,"kb_used_data":31528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304964,"statfs":{"total":96636764160,"available":96568283136,"internally_reserved":0,"allocated":32284672,"data_stored":34010826,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313269,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49288,"kb_used_data":18100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322552,"statfs":{"total":96636764160,"available":96586293248,"internally_reserved":0,"allocated":18534400,"data_stored":19770007,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1359872,"data_stored":946027,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51060,"kb_used_data":15708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320780,"statfs":{"total":96636764160,"available":96584478720,"internally_reserved":0,"allocated":16084992,"data_stored":16607897,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46688,"kb_used_data":15500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325152,"statfs":{"total":96636764160,"available":96588955648,"internally_reserved":0,"allocated":15872000,"data_stored":16339216,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1380352,"data_stored":948101,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313268,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32532,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1372160,"data_stored":954295,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46896,"kb_used_data":15700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324944,"statfs":{"total":96636764160,"available":96588742656,"internally_reserved":0,"allocated":16076800,"data_stored":16606134,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31858688,"data_stored":33943544,"data_compressed":2174030,"data_compressed_allocated":2314240,"data_compressed_original":4501504,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:40:59.631 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-09-16T03:40:59.631 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "170'267", 'reported_seq': 834, 'reported_epoch': 228, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-09-16T03:40:57.086305+0000', 'last_change': '2024-09-16T03:40:41.720636+0000', 'last_active': '2024-09-16T03:40:57.086305+0000', 'last_peered': '2024-09-16T03:40:57.086305+0000', 'last_clean': '2024-09-16T03:40:39.185069+0000', 'last_became_active': '2024-09-16T03:40:41.619774+0000', 'last_became_peered': '2024-09-16T03:40:41.619774+0000', 'last_unstale': '2024-09-16T03:40:57.086305+0000', 'last_undegraded': '2024-09-16T03:40:57.086305+0000', 'last_fullsized': '2024-09-16T03:40:57.086305+0000', 'mapping_epoch': 215, 'log_start': "170'263", 'ondisk_log_start': "170'263", 'created': 31, 'last_epoch_clean': 201, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "170'267", 'last_scrub_stamp': '2024-09-16T03:40:38.134247+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T03:37:03.913037+0000', 'last_clean_scrub_stamp': '2024-09-16T03:40:38.134247+0000', 'objects_scrubbed': 39, 'log_size': 4, 'log_dups_size': 274, 'ondisk_log_size': 4, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 406, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28703148, 'num_objects': 39, 'num_object_clones': 0, 'num_object_copies': 117, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 39, 'num_objects_unfound': 0, 'num_objects_dirty': 39, 'num_whiteouts': 0, 'num_read': 249, 'num_read_kb': 2401, 'num_write': 343, 'num_write_kb': 78301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 17668524, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4, 3], 'acting': [6, 4, 11], 'avail_no_missing': ['6', '4', '11'], 'object_location_counts': [{'shards': '4,6,11', 'objects': 39}], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T03:40:59.632 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T03:41:00.004 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:00.005 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":262,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":229,"num_osds":16,"num_up_osds":16,"osd_up_since":1726457810,"num_in_osds":16,"osd_in_since":1726458014,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":150,"data_bytes":103194545,"bytes_used":743112704,"bytes_avail":1545445113856,"bytes_total":1546188226560,"misplaced_objects":39,"misplaced_total":450,"misplaced_ratio":0.086666666666666684,"recovering_objects_per_sec":4,"recovering_bytes_per_sec":1505506,"recovering_keys_per_sec":0,"num_objects_recovered":27,"num_bytes_recovered":9035214,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T03:36:37:310011+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T03:37:51.688536+0000","services":{}},"progress_events":{}} 2024-09-16T03:41:00.018 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-16T03:41:00.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:00.312 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:00.313 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:00.324 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-09-16T03:40:59.728882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":725696,"kb_used_data":205844,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509223744,"statfs":{"total":1546188226560,"available":1545445113856,"internally_reserved":0,"allocated":210784256,"data_stored":218091107,"data_compressed":14000041,"data_compressed_allocated":15527936,"data_compressed_original":32542720,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-28,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9035214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001446"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":640,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.918648+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:40:56.918648+0000","last_peered":"2024-09-16T03:40:56.918648+0000","last_clean":"2024-09-16T03:40:56.918648+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:40:56.918648+0000","last_undegraded":"2024-09-16T03:40:56.918648+0000","last_fullsized":"2024-09-16T03:40:56.918648+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":158,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.918696+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:40:56.918696+0000","last_peered":"2024-09-16T03:40:56.918696+0000","last_clean":"2024-09-16T03:40:56.918696+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:40:56.918696+0000","last_undegraded":"2024-09-16T03:40:56.918696+0000","last_fullsized":"2024-09-16T03:40:56.918696+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":968,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.019013+0000","last_change":"2024-09-16T03:40:52.880495+0000","last_active":"2024-09-16T03:40:57.019013+0000","last_peered":"2024-09-16T03:40:57.019013+0000","last_clean":"2024-09-16T03:40:57.019013+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:40:57.019013+0000","last_undegraded":"2024-09-16T03:40:57.019013+0000","last_fullsized":"2024-09-16T03:40:57.019013+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.728291+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:40:54.728291+0000","last_peered":"2024-09-16T03:40:54.728291+0000","last_clean":"2024-09-16T03:40:54.728291+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:40:54.728291+0000","last_undegraded":"2024-09-16T03:40:54.728291+0000","last_fullsized":"2024-09-16T03:40:54.728291+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":834,"reported_epoch":228,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:57.086305+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:40:57.086305+0000","last_peered":"2024-09-16T03:40:57.086305+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:40:57.086305+0000","last_undegraded":"2024-09-16T03:40:57.086305+0000","last_fullsized":"2024-09-16T03:40:57.086305+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.915401+0000","last_change":"2024-09-16T03:40:47.179954+0000","last_active":"2024-09-16T03:40:56.915401+0000","last_peered":"2024-09-16T03:40:56.915401+0000","last_clean":"2024-09-16T03:40:56.915401+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:40:56.915401+0000","last_undegraded":"2024-09-16T03:40:56.915401+0000","last_fullsized":"2024-09-16T03:40:56.915401+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":891,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.941023+0000","last_change":"2024-09-16T03:40:25.635449+0000","last_active":"2024-09-16T03:40:57.941023+0000","last_peered":"2024-09-16T03:40:57.941023+0000","last_clean":"2024-09-16T03:40:57.941023+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:40:57.941023+0000","last_undegraded":"2024-09-16T03:40:57.941023+0000","last_fullsized":"2024-09-16T03:40:57.941023+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:03.730604+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":163,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.828400+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:40:54.828400+0000","last_peered":"2024-09-16T03:40:54.828400+0000","last_clean":"2024-09-16T03:40:54.828400+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:40:54.828400+0000","last_undegraded":"2024-09-16T03:40:54.828400+0000","last_fullsized":"2024-09-16T03:40:54.828400+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":875,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.815393+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:40:56.815393+0000","last_peered":"2024-09-16T03:40:56.815393+0000","last_clean":"2024-09-16T03:40:56.815393+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:40:56.815393+0000","last_undegraded":"2024-09-16T03:40:56.815393+0000","last_fullsized":"2024-09-16T03:40:56.815393+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":442,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.086316+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:40:57.086316+0000","last_peered":"2024-09-16T03:40:57.086316+0000","last_clean":"2024-09-16T03:40:57.086316+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:40:57.086316+0000","last_undegraded":"2024-09-16T03:40:57.086316+0000","last_fullsized":"2024-09-16T03:40:57.086316+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":203,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.817488+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:40:54.817488+0000","last_peered":"2024-09-16T03:40:54.817488+0000","last_clean":"2024-09-16T03:40:54.817488+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:54.817488+0000","last_undegraded":"2024-09-16T03:40:54.817488+0000","last_fullsized":"2024-09-16T03:40:54.817488+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":451,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.942348+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:40:57.942348+0000","last_peered":"2024-09-16T03:40:57.942348+0000","last_clean":"2024-09-16T03:40:57.942348+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:40:57.942348+0000","last_undegraded":"2024-09-16T03:40:57.942348+0000","last_fullsized":"2024-09-16T03:40:57.942348+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":432,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-16T03:40:54.728320+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:40:54.728320+0000","last_peered":"2024-09-16T03:40:54.728320+0000","last_clean":"2024-09-16T03:40:54.728320+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:54.728320+0000","last_undegraded":"2024-09-16T03:40:54.728320+0000","last_fullsized":"2024-09-16T03:40:54.728320+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51076,"kb_used_data":15728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320764,"statfs":{"total":96636764160,"available":96584462336,"internally_reserved":0,"allocated":16105472,"data_stored":16615882,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247861,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69516,"kb_used_data":34160,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302324,"statfs":{"total":96636764160,"available":96565579776,"internally_reserved":0,"allocated":34979840,"data_stored":37455973,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247860,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52708,"kb_used_data":17352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319132,"statfs":{"total":96636764160,"available":96582791168,"internally_reserved":0,"allocated":17768448,"data_stored":18629806,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280566,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49392,"kb_used_data":18200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322448,"statfs":{"total":96636764160,"available":96586186752,"internally_reserved":0,"allocated":18636800,"data_stored":19804268,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28388,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343452,"statfs":{"total":96636764160,"available":96607694848,"internally_reserved":0,"allocated":1392640,"data_stored":967967,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46732,"kb_used_data":15540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325108,"statfs":{"total":96636764160,"available":96588910592,"internally_reserved":0,"allocated":15912960,"data_stored":16362746,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313270,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1404928,"data_stored":977121,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36752,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335088,"statfs":{"total":96636764160,"available":96599130112,"internally_reserved":0,"allocated":1437696,"data_stored":979466,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280564,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66876,"kb_used_data":31528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304964,"statfs":{"total":96636764160,"available":96568283136,"internally_reserved":0,"allocated":32284672,"data_stored":34010826,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313270,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49336,"kb_used_data":18148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322504,"statfs":{"total":96636764160,"available":96586244096,"internally_reserved":0,"allocated":18583552,"data_stored":19800834,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280565,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1404928,"data_stored":976610,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51060,"kb_used_data":15708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320780,"statfs":{"total":96636764160,"available":96584478720,"internally_reserved":0,"allocated":16084992,"data_stored":16607897,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46688,"kb_used_data":15500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325152,"statfs":{"total":96636764160,"available":96588955648,"internally_reserved":0,"allocated":15872000,"data_stored":16339216,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1429504,"data_stored":978928,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313269,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32564,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1409024,"data_stored":977433,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46896,"kb_used_data":15700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324944,"statfs":{"total":96636764160,"available":96588742656,"internally_reserved":0,"allocated":16076800,"data_stored":16606134,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:01.942 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:41:01.940+0000 7eff7884b640 -1 osd.3 229 handle_osd_map injecting map crc failure 2024-09-16T03:41:03.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:03.638 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:03.639 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:03.651 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-16T03:41:01.729252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":725820,"kb_used_data":205964,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509223620,"statfs":{"total":1546188226560,"available":1545444986880,"internally_reserved":0,"allocated":210907136,"data_stored":218168123,"data_compressed":14000041,"data_compressed_allocated":15527936,"data_compressed_original":32542720,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-28,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9035214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001317"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":640,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.918648+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:40:56.918648+0000","last_peered":"2024-09-16T03:40:56.918648+0000","last_clean":"2024-09-16T03:40:56.918648+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:40:56.918648+0000","last_undegraded":"2024-09-16T03:40:56.918648+0000","last_fullsized":"2024-09-16T03:40:56.918648+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":158,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.918696+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:40:56.918696+0000","last_peered":"2024-09-16T03:40:56.918696+0000","last_clean":"2024-09-16T03:40:56.918696+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:40:56.918696+0000","last_undegraded":"2024-09-16T03:40:56.918696+0000","last_fullsized":"2024-09-16T03:40:56.918696+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":968,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.019013+0000","last_change":"2024-09-16T03:40:52.880495+0000","last_active":"2024-09-16T03:40:57.019013+0000","last_peered":"2024-09-16T03:40:57.019013+0000","last_clean":"2024-09-16T03:40:57.019013+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:40:57.019013+0000","last_undegraded":"2024-09-16T03:40:57.019013+0000","last_fullsized":"2024-09-16T03:40:57.019013+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'274","last_scrub_stamp":"2024-09-16T03:39:23.138651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:23.138651+0000","objects_scrubbed":22,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":468,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:58.041656+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:40:58.041656+0000","last_peered":"2024-09-16T03:40:58.041656+0000","last_clean":"2024-09-16T03:40:58.041656+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:40:58.041656+0000","last_undegraded":"2024-09-16T03:40:58.041656+0000","last_fullsized":"2024-09-16T03:40:58.041656+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":834,"reported_epoch":228,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:40:57.086305+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:40:57.086305+0000","last_peered":"2024-09-16T03:40:57.086305+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:40:57.086305+0000","last_undegraded":"2024-09-16T03:40:57.086305+0000","last_fullsized":"2024-09-16T03:40:57.086305+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.915401+0000","last_change":"2024-09-16T03:40:47.179954+0000","last_active":"2024-09-16T03:40:56.915401+0000","last_peered":"2024-09-16T03:40:56.915401+0000","last_clean":"2024-09-16T03:40:56.915401+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:40:56.915401+0000","last_undegraded":"2024-09-16T03:40:56.915401+0000","last_fullsized":"2024-09-16T03:40:56.915401+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:25.836387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:25.836387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":891,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.941023+0000","last_change":"2024-09-16T03:40:25.635449+0000","last_active":"2024-09-16T03:40:57.941023+0000","last_peered":"2024-09-16T03:40:57.941023+0000","last_clean":"2024-09-16T03:40:57.941023+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:40:57.941023+0000","last_undegraded":"2024-09-16T03:40:57.941023+0000","last_fullsized":"2024-09-16T03:40:57.941023+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:03.730604+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":167,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:58.041646+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:40:58.041646+0000","last_peered":"2024-09-16T03:40:58.041646+0000","last_clean":"2024-09-16T03:40:58.041646+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:40:58.041646+0000","last_undegraded":"2024-09-16T03:40:58.041646+0000","last_fullsized":"2024-09-16T03:40:58.041646+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":875,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:56.815393+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:40:56.815393+0000","last_peered":"2024-09-16T03:40:56.815393+0000","last_clean":"2024-09-16T03:40:56.815393+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:40:56.815393+0000","last_undegraded":"2024-09-16T03:40:56.815393+0000","last_fullsized":"2024-09-16T03:40:56.815393+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":442,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.086316+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:40:57.086316+0000","last_peered":"2024-09-16T03:40:57.086316+0000","last_clean":"2024-09-16T03:40:57.086316+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:40:57.086316+0000","last_undegraded":"2024-09-16T03:40:57.086316+0000","last_fullsized":"2024-09-16T03:40:57.086316+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":207,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.940916+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:40:57.940916+0000","last_peered":"2024-09-16T03:40:57.940916+0000","last_clean":"2024-09-16T03:40:57.940916+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:57.940916+0000","last_undegraded":"2024-09-16T03:40:57.940916+0000","last_fullsized":"2024-09-16T03:40:57.940916+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":451,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.942348+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:40:57.942348+0000","last_peered":"2024-09-16T03:40:57.942348+0000","last_clean":"2024-09-16T03:40:57.942348+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:40:57.942348+0000","last_undegraded":"2024-09-16T03:40:57.942348+0000","last_fullsized":"2024-09-16T03:40:57.942348+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":436,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:58.041673+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:40:58.041673+0000","last_peered":"2024-09-16T03:40:58.041673+0000","last_clean":"2024-09-16T03:40:58.041673+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:58.041673+0000","last_undegraded":"2024-09-16T03:40:58.041673+0000","last_fullsized":"2024-09-16T03:40:58.041673+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247860,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51076,"kb_used_data":15728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320764,"statfs":{"total":96636764160,"available":96584462336,"internally_reserved":0,"allocated":16105472,"data_stored":16615882,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247861,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69516,"kb_used_data":34160,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302324,"statfs":{"total":96636764160,"available":96565579776,"internally_reserved":0,"allocated":34979840,"data_stored":37455973,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247860,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52708,"kb_used_data":17352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319132,"statfs":{"total":96636764160,"available":96582791168,"internally_reserved":0,"allocated":17768448,"data_stored":18629806,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280566,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49392,"kb_used_data":18200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322448,"statfs":{"total":96636764160,"available":96586186752,"internally_reserved":0,"allocated":18636800,"data_stored":19804268,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28388,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343452,"statfs":{"total":96636764160,"available":96607694848,"internally_reserved":0,"allocated":1392640,"data_stored":967967,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46732,"kb_used_data":15540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325108,"statfs":{"total":96636764160,"available":96588910592,"internally_reserved":0,"allocated":15912960,"data_stored":16362746,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313270,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1404928,"data_stored":977121,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36752,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335088,"statfs":{"total":96636764160,"available":96599130112,"internally_reserved":0,"allocated":1437696,"data_stored":979466,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66916,"kb_used_data":31564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304924,"statfs":{"total":96636764160,"available":96568242176,"internally_reserved":0,"allocated":32321536,"data_stored":34033964,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313270,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49336,"kb_used_data":18148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322504,"statfs":{"total":96636764160,"available":96586244096,"internally_reserved":0,"allocated":18583552,"data_stored":19800834,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280565,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1404928,"data_stored":976610,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280565,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51084,"kb_used_data":15732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320756,"statfs":{"total":96636764160,"available":96584454144,"internally_reserved":0,"allocated":16109568,"data_stored":16623267,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46728,"kb_used_data":15536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325112,"statfs":{"total":96636764160,"available":96588914688,"internally_reserved":0,"allocated":15908864,"data_stored":16362354,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1429504,"data_stored":978928,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313269,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32564,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1409024,"data_stored":977433,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46916,"kb_used_data":15724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324924,"statfs":{"total":96636764160,"available":96588722176,"internally_reserved":0,"allocated":16101376,"data_stored":16621504,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:03.652 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:03.948 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:03.949 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:03.961 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-16T03:41:03.729825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":725880,"kb_used_data":206036,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509223560,"statfs":{"total":1546188226560,"available":1545444925440,"internally_reserved":0,"allocated":210980864,"data_stored":218214233,"data_compressed":14000041,"data_compressed_allocated":15527936,"data_compressed_original":32542720,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-28,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9035214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.340150"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":980,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910389+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910389+0000","last_peered":"2024-09-16T03:41:01.910389+0000","last_clean":"2024-09-16T03:41:01.910389+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910389+0000","last_undegraded":"2024-09-16T03:41:01.910389+0000","last_fullsized":"2024-09-16T03:41:01.910389+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":468,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:58.041656+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:40:58.041656+0000","last_peered":"2024-09-16T03:40:58.041656+0000","last_clean":"2024-09-16T03:40:58.041656+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:40:58.041656+0000","last_undegraded":"2024-09-16T03:40:58.041656+0000","last_fullsized":"2024-09-16T03:40:58.041656+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":838,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:41:01.746005+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:41:01.746005+0000","last_peered":"2024-09-16T03:41:01.746005+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:01.746005+0000","last_undegraded":"2024-09-16T03:41:01.746005+0000","last_fullsized":"2024-09-16T03:41:01.746005+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":891,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.941023+0000","last_change":"2024-09-16T03:40:25.635449+0000","last_active":"2024-09-16T03:40:57.941023+0000","last_peered":"2024-09-16T03:40:57.941023+0000","last_clean":"2024-09-16T03:40:57.941023+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:40:57.941023+0000","last_undegraded":"2024-09-16T03:40:57.941023+0000","last_fullsized":"2024-09-16T03:40:57.941023+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:03.730604+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":167,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:58.041646+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:40:58.041646+0000","last_peered":"2024-09-16T03:40:58.041646+0000","last_clean":"2024-09-16T03:40:58.041646+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:40:58.041646+0000","last_undegraded":"2024-09-16T03:40:58.041646+0000","last_fullsized":"2024-09-16T03:40:58.041646+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":446,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645871+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645871+0000","last_peered":"2024-09-16T03:41:01.645871+0000","last_clean":"2024-09-16T03:41:01.645871+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645871+0000","last_undegraded":"2024-09-16T03:41:01.645871+0000","last_fullsized":"2024-09-16T03:41:01.645871+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":207,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.940916+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:40:57.940916+0000","last_peered":"2024-09-16T03:40:57.940916+0000","last_clean":"2024-09-16T03:40:57.940916+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:57.940916+0000","last_undegraded":"2024-09-16T03:40:57.940916+0000","last_fullsized":"2024-09-16T03:40:57.940916+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":451,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.942348+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:40:57.942348+0000","last_peered":"2024-09-16T03:40:57.942348+0000","last_clean":"2024-09-16T03:40:57.942348+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:40:57.942348+0000","last_undegraded":"2024-09-16T03:40:57.942348+0000","last_fullsized":"2024-09-16T03:40:57.942348+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":436,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:58.041673+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:40:58.041673+0000","last_peered":"2024-09-16T03:40:58.041673+0000","last_clean":"2024-09-16T03:40:58.041673+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:58.041673+0000","last_undegraded":"2024-09-16T03:40:58.041673+0000","last_fullsized":"2024-09-16T03:40:58.041673+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247862,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69532,"kb_used_data":34184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302308,"statfs":{"total":96636764160,"available":96565563392,"internally_reserved":0,"allocated":35004416,"data_stored":37471343,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247861,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52728,"kb_used_data":17376,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319112,"statfs":{"total":96636764160,"available":96582770688,"internally_reserved":0,"allocated":17793024,"data_stored":18645176,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280566,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49392,"kb_used_data":18200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322448,"statfs":{"total":96636764160,"available":96586186752,"internally_reserved":0,"allocated":18636800,"data_stored":19804268,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28388,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343452,"statfs":{"total":96636764160,"available":96607694848,"internally_reserved":0,"allocated":1392640,"data_stored":967967,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46732,"kb_used_data":15540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325108,"statfs":{"total":96636764160,"available":96588910592,"internally_reserved":0,"allocated":15912960,"data_stored":16362746,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313270,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1404928,"data_stored":977121,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36752,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335088,"statfs":{"total":96636764160,"available":96599130112,"internally_reserved":0,"allocated":1437696,"data_stored":979466,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66916,"kb_used_data":31564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304924,"statfs":{"total":96636764160,"available":96568242176,"internally_reserved":0,"allocated":32321536,"data_stored":34033964,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313270,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49336,"kb_used_data":18148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322504,"statfs":{"total":96636764160,"available":96586244096,"internally_reserved":0,"allocated":18583552,"data_stored":19800834,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280565,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1404928,"data_stored":976610,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280565,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51084,"kb_used_data":15732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320756,"statfs":{"total":96636764160,"available":96584454144,"internally_reserved":0,"allocated":16109568,"data_stored":16623267,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46728,"kb_used_data":15536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325112,"statfs":{"total":96636764160,"available":96588914688,"internally_reserved":0,"allocated":15908864,"data_stored":16362354,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1429504,"data_stored":978928,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313269,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32564,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1409024,"data_stored":977433,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46916,"kb_used_data":15724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324924,"statfs":{"total":96636764160,"available":96588722176,"internally_reserved":0,"allocated":16101376,"data_stored":16621504,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:03.962 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-09-16T03:41:03.962 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "170'267", 'reported_seq': 838, 'reported_epoch': 230, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-09-16T03:41:01.746005+0000', 'last_change': '2024-09-16T03:40:41.720636+0000', 'last_active': '2024-09-16T03:41:01.746005+0000', 'last_peered': '2024-09-16T03:41:01.746005+0000', 'last_clean': '2024-09-16T03:40:39.185069+0000', 'last_became_active': '2024-09-16T03:40:41.619774+0000', 'last_became_peered': '2024-09-16T03:40:41.619774+0000', 'last_unstale': '2024-09-16T03:41:01.746005+0000', 'last_undegraded': '2024-09-16T03:41:01.746005+0000', 'last_fullsized': '2024-09-16T03:41:01.746005+0000', 'mapping_epoch': 215, 'log_start': "170'263", 'ondisk_log_start': "170'263", 'created': 31, 'last_epoch_clean': 201, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "170'267", 'last_scrub_stamp': '2024-09-16T03:40:38.134247+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T03:37:03.913037+0000', 'last_clean_scrub_stamp': '2024-09-16T03:40:38.134247+0000', 'objects_scrubbed': 39, 'log_size': 4, 'log_dups_size': 274, 'ondisk_log_size': 4, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 406, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28703148, 'num_objects': 39, 'num_object_clones': 0, 'num_object_copies': 117, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 39, 'num_objects_unfound': 0, 'num_objects_dirty': 39, 'num_whiteouts': 0, 'num_read': 249, 'num_read_kb': 2401, 'num_write': 343, 'num_write_kb': 78301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 17668524, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4, 3], 'acting': [6, 4, 11], 'avail_no_missing': ['6', '4', '11'], 'object_location_counts': [{'shards': '4,6,11', 'objects': 39}], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T03:41:03.963 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T03:41:04.332 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:04.332 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":267,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":230,"num_osds":16,"num_up_osds":16,"osd_up_since":1726457810,"num_in_osds":16,"osd_in_since":1726458014,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":150,"data_bytes":103194545,"bytes_used":743239680,"bytes_avail":1545444986880,"bytes_total":1546188226560,"misplaced_objects":39,"misplaced_total":450,"misplaced_ratio":0.086666666666666684,"recovering_objects_per_sec":4,"recovering_bytes_per_sec":1505538,"recovering_keys_per_sec":0,"num_objects_recovered":27,"num_bytes_recovered":9035214,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T03:36:37:310011+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T03:37:51.688536+0000","services":{}},"progress_events":{}} 2024-09-16T03:41:04.345 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-16T03:41:04.346 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:04.642 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:04.642 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:04.654 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-16T03:41:03.729825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":725880,"kb_used_data":206036,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509223560,"statfs":{"total":1546188226560,"available":1545444925440,"internally_reserved":0,"allocated":210980864,"data_stored":218214233,"data_compressed":14000041,"data_compressed_allocated":15527936,"data_compressed_original":32542720,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-28,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9035214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.340150"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":980,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910389+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910389+0000","last_peered":"2024-09-16T03:41:01.910389+0000","last_clean":"2024-09-16T03:41:01.910389+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910389+0000","last_undegraded":"2024-09-16T03:41:01.910389+0000","last_fullsized":"2024-09-16T03:41:01.910389+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":468,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:58.041656+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:40:58.041656+0000","last_peered":"2024-09-16T03:40:58.041656+0000","last_clean":"2024-09-16T03:40:58.041656+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:40:58.041656+0000","last_undegraded":"2024-09-16T03:40:58.041656+0000","last_fullsized":"2024-09-16T03:40:58.041656+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":838,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:41:01.746005+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:41:01.746005+0000","last_peered":"2024-09-16T03:41:01.746005+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:01.746005+0000","last_undegraded":"2024-09-16T03:41:01.746005+0000","last_fullsized":"2024-09-16T03:41:01.746005+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":891,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.941023+0000","last_change":"2024-09-16T03:40:25.635449+0000","last_active":"2024-09-16T03:40:57.941023+0000","last_peered":"2024-09-16T03:40:57.941023+0000","last_clean":"2024-09-16T03:40:57.941023+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:40:57.941023+0000","last_undegraded":"2024-09-16T03:40:57.941023+0000","last_fullsized":"2024-09-16T03:40:57.941023+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:03.730604+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":167,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:58.041646+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:40:58.041646+0000","last_peered":"2024-09-16T03:40:58.041646+0000","last_clean":"2024-09-16T03:40:58.041646+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:40:58.041646+0000","last_undegraded":"2024-09-16T03:40:58.041646+0000","last_fullsized":"2024-09-16T03:40:58.041646+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":446,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645871+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645871+0000","last_peered":"2024-09-16T03:41:01.645871+0000","last_clean":"2024-09-16T03:41:01.645871+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645871+0000","last_undegraded":"2024-09-16T03:41:01.645871+0000","last_fullsized":"2024-09-16T03:41:01.645871+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":207,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.940916+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:40:57.940916+0000","last_peered":"2024-09-16T03:40:57.940916+0000","last_clean":"2024-09-16T03:40:57.940916+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:40:57.940916+0000","last_undegraded":"2024-09-16T03:40:57.940916+0000","last_fullsized":"2024-09-16T03:40:57.940916+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":451,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:57.942348+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:40:57.942348+0000","last_peered":"2024-09-16T03:40:57.942348+0000","last_clean":"2024-09-16T03:40:57.942348+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:40:57.942348+0000","last_undegraded":"2024-09-16T03:40:57.942348+0000","last_fullsized":"2024-09-16T03:40:57.942348+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":436,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-16T03:40:58.041673+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:40:58.041673+0000","last_peered":"2024-09-16T03:40:58.041673+0000","last_clean":"2024-09-16T03:40:58.041673+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:40:58.041673+0000","last_undegraded":"2024-09-16T03:40:58.041673+0000","last_fullsized":"2024-09-16T03:40:58.041673+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247862,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69532,"kb_used_data":34184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302308,"statfs":{"total":96636764160,"available":96565563392,"internally_reserved":0,"allocated":35004416,"data_stored":37471343,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247861,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52728,"kb_used_data":17376,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319112,"statfs":{"total":96636764160,"available":96582770688,"internally_reserved":0,"allocated":17793024,"data_stored":18645176,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280566,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49392,"kb_used_data":18200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322448,"statfs":{"total":96636764160,"available":96586186752,"internally_reserved":0,"allocated":18636800,"data_stored":19804268,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28388,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343452,"statfs":{"total":96636764160,"available":96607694848,"internally_reserved":0,"allocated":1392640,"data_stored":967967,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46732,"kb_used_data":15540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325108,"statfs":{"total":96636764160,"available":96588910592,"internally_reserved":0,"allocated":15912960,"data_stored":16362746,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313270,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1404928,"data_stored":977121,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36752,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335088,"statfs":{"total":96636764160,"available":96599130112,"internally_reserved":0,"allocated":1437696,"data_stored":979466,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66916,"kb_used_data":31564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304924,"statfs":{"total":96636764160,"available":96568242176,"internally_reserved":0,"allocated":32321536,"data_stored":34033964,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313270,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49336,"kb_used_data":18148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322504,"statfs":{"total":96636764160,"available":96586244096,"internally_reserved":0,"allocated":18583552,"data_stored":19800834,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280565,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1404928,"data_stored":976610,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280565,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51084,"kb_used_data":15732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320756,"statfs":{"total":96636764160,"available":96584454144,"internally_reserved":0,"allocated":16109568,"data_stored":16623267,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46728,"kb_used_data":15536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325112,"statfs":{"total":96636764160,"available":96588914688,"internally_reserved":0,"allocated":15908864,"data_stored":16362354,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1429504,"data_stored":978928,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313269,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32564,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1409024,"data_stored":977433,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46916,"kb_used_data":15724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324924,"statfs":{"total":96636764160,"available":96588722176,"internally_reserved":0,"allocated":16101376,"data_stored":16621504,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:07.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:07.966 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:07.967 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:07.978 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-16T03:41:07.730721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":726032,"kb_used_data":206224,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509223408,"statfs":{"total":1546188226560,"available":1545444769792,"internally_reserved":0,"allocated":211173376,"data_stored":218336599,"data_compressed":14000041,"data_compressed_allocated":15527936,"data_compressed_original":32542720,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.997031"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":980,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910389+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910389+0000","last_peered":"2024-09-16T03:41:01.910389+0000","last_clean":"2024-09-16T03:41:01.910389+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910389+0000","last_undegraded":"2024-09-16T03:41:01.910389+0000","last_fullsized":"2024-09-16T03:41:01.910389+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":470,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010819+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:02.010819+0000","last_peered":"2024-09-16T03:41:02.010819+0000","last_clean":"2024-09-16T03:41:02.010819+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:02.010819+0000","last_undegraded":"2024-09-16T03:41:02.010819+0000","last_fullsized":"2024-09-16T03:41:02.010819+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":838,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:41:01.746005+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:41:01.746005+0000","last_peered":"2024-09-16T03:41:01.746005+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:01.746005+0000","last_undegraded":"2024-09-16T03:41:01.746005+0000","last_fullsized":"2024-09-16T03:41:01.746005+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":894,"reported_epoch":230,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:41:01.487908+0000","last_change":"2024-09-16T03:41:04.127002+0000","last_active":"2024-09-16T03:41:01.487908+0000","last_peered":"2024-09-16T03:41:01.487908+0000","last_clean":"2024-09-16T03:41:01.487908+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:01.487908+0000","last_undegraded":"2024-09-16T03:41:01.487908+0000","last_fullsized":"2024-09-16T03:41:01.487908+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":169,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010817+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:02.010817+0000","last_peered":"2024-09-16T03:41:02.010817+0000","last_clean":"2024-09-16T03:41:02.010817+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:02.010817+0000","last_undegraded":"2024-09-16T03:41:02.010817+0000","last_fullsized":"2024-09-16T03:41:02.010817+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":447,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645907+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645907+0000","last_peered":"2024-09-16T03:41:01.645907+0000","last_clean":"2024-09-16T03:41:01.645907+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645907+0000","last_undegraded":"2024-09-16T03:41:01.645907+0000","last_fullsized":"2024-09-16T03:41:01.645907+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":209,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645061+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:01.645061+0000","last_peered":"2024-09-16T03:41:01.645061+0000","last_clean":"2024-09-16T03:41:01.645061+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:01.645061+0000","last_undegraded":"2024-09-16T03:41:01.645061+0000","last_fullsized":"2024-09-16T03:41:01.645061+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.746155+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:41:01.746155+0000","last_peered":"2024-09-16T03:41:01.746155+0000","last_clean":"2024-09-16T03:41:01.746155+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:01.746155+0000","last_undegraded":"2024-09-16T03:41:01.746155+0000","last_fullsized":"2024-09-16T03:41:01.746155+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":438,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010829+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:02.010829+0000","last_peered":"2024-09-16T03:41:02.010829+0000","last_clean":"2024-09-16T03:41:02.010829+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:02.010829+0000","last_undegraded":"2024-09-16T03:41:02.010829+0000","last_fullsized":"2024-09-16T03:41:02.010829+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247863,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69544,"kb_used_data":34196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302296,"statfs":{"total":96636764160,"available":96565551104,"internally_reserved":0,"allocated":35016704,"data_stored":37479003,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247862,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":17388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":17805312,"data_stored":18652836,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280567,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28412,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343428,"statfs":{"total":96636764160,"available":96607670272,"internally_reserved":0,"allocated":1417216,"data_stored":983337,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46740,"kb_used_data":15552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325100,"statfs":{"total":96636764160,"available":96588902400,"internally_reserved":0,"allocated":15925248,"data_stored":16370406,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313271,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984781,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1445888,"data_stored":986882,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66924,"kb_used_data":31576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304916,"statfs":{"total":96636764160,"available":96568233984,"internally_reserved":0,"allocated":32333824,"data_stored":34041624,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313271,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49348,"kb_used_data":18160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322492,"statfs":{"total":96636764160,"available":96586231808,"internally_reserved":0,"allocated":18595840,"data_stored":19808494,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280566,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280566,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51092,"kb_used_data":15744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320748,"statfs":{"total":96636764160,"available":96584445952,"internally_reserved":0,"allocated":16121856,"data_stored":16630927,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15921152,"data_stored":16370014,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1441792,"data_stored":986588,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313270,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":985093,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46924,"kb_used_data":15736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324916,"statfs":{"total":96636764160,"available":96588713984,"internally_reserved":0,"allocated":16113664,"data_stored":16629164,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:07.980 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:08.276 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:08.288 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-16T03:41:07.730721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":726032,"kb_used_data":206224,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509223408,"statfs":{"total":1546188226560,"available":1545444769792,"internally_reserved":0,"allocated":211173376,"data_stored":218336599,"data_compressed":14000041,"data_compressed_allocated":15527936,"data_compressed_original":32542720,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.997031"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":980,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910389+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910389+0000","last_peered":"2024-09-16T03:41:01.910389+0000","last_clean":"2024-09-16T03:41:01.910389+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910389+0000","last_undegraded":"2024-09-16T03:41:01.910389+0000","last_fullsized":"2024-09-16T03:41:01.910389+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":470,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010819+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:02.010819+0000","last_peered":"2024-09-16T03:41:02.010819+0000","last_clean":"2024-09-16T03:41:02.010819+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:02.010819+0000","last_undegraded":"2024-09-16T03:41:02.010819+0000","last_fullsized":"2024-09-16T03:41:02.010819+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":838,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:41:01.746005+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:41:01.746005+0000","last_peered":"2024-09-16T03:41:01.746005+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:01.746005+0000","last_undegraded":"2024-09-16T03:41:01.746005+0000","last_fullsized":"2024-09-16T03:41:01.746005+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":894,"reported_epoch":230,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:41:01.487908+0000","last_change":"2024-09-16T03:41:04.127002+0000","last_active":"2024-09-16T03:41:01.487908+0000","last_peered":"2024-09-16T03:41:01.487908+0000","last_clean":"2024-09-16T03:41:01.487908+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:01.487908+0000","last_undegraded":"2024-09-16T03:41:01.487908+0000","last_fullsized":"2024-09-16T03:41:01.487908+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":169,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010817+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:02.010817+0000","last_peered":"2024-09-16T03:41:02.010817+0000","last_clean":"2024-09-16T03:41:02.010817+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:02.010817+0000","last_undegraded":"2024-09-16T03:41:02.010817+0000","last_fullsized":"2024-09-16T03:41:02.010817+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":447,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645907+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645907+0000","last_peered":"2024-09-16T03:41:01.645907+0000","last_clean":"2024-09-16T03:41:01.645907+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645907+0000","last_undegraded":"2024-09-16T03:41:01.645907+0000","last_fullsized":"2024-09-16T03:41:01.645907+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":209,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645061+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:01.645061+0000","last_peered":"2024-09-16T03:41:01.645061+0000","last_clean":"2024-09-16T03:41:01.645061+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:01.645061+0000","last_undegraded":"2024-09-16T03:41:01.645061+0000","last_fullsized":"2024-09-16T03:41:01.645061+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.746155+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:41:01.746155+0000","last_peered":"2024-09-16T03:41:01.746155+0000","last_clean":"2024-09-16T03:41:01.746155+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:01.746155+0000","last_undegraded":"2024-09-16T03:41:01.746155+0000","last_fullsized":"2024-09-16T03:41:01.746155+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":438,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010829+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:02.010829+0000","last_peered":"2024-09-16T03:41:02.010829+0000","last_clean":"2024-09-16T03:41:02.010829+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:02.010829+0000","last_undegraded":"2024-09-16T03:41:02.010829+0000","last_fullsized":"2024-09-16T03:41:02.010829+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247863,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69544,"kb_used_data":34196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302296,"statfs":{"total":96636764160,"available":96565551104,"internally_reserved":0,"allocated":35016704,"data_stored":37479003,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247862,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":17388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":17805312,"data_stored":18652836,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280567,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28412,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343428,"statfs":{"total":96636764160,"available":96607670272,"internally_reserved":0,"allocated":1417216,"data_stored":983337,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46740,"kb_used_data":15552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325100,"statfs":{"total":96636764160,"available":96588902400,"internally_reserved":0,"allocated":15925248,"data_stored":16370406,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313271,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984781,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1445888,"data_stored":986882,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66924,"kb_used_data":31576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304916,"statfs":{"total":96636764160,"available":96568233984,"internally_reserved":0,"allocated":32333824,"data_stored":34041624,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313271,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49348,"kb_used_data":18160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322492,"statfs":{"total":96636764160,"available":96586231808,"internally_reserved":0,"allocated":18595840,"data_stored":19808494,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280566,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280566,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51092,"kb_used_data":15744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320748,"statfs":{"total":96636764160,"available":96584445952,"internally_reserved":0,"allocated":16121856,"data_stored":16630927,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15921152,"data_stored":16370014,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1441792,"data_stored":986588,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313270,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":985093,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46924,"kb_used_data":15736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324916,"statfs":{"total":96636764160,"available":96588713984,"internally_reserved":0,"allocated":16113664,"data_stored":16629164,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:08.290 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-09-16T03:41:08.290 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "170'267", 'reported_seq': 838, 'reported_epoch': 230, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-09-16T03:41:01.746005+0000', 'last_change': '2024-09-16T03:40:41.720636+0000', 'last_active': '2024-09-16T03:41:01.746005+0000', 'last_peered': '2024-09-16T03:41:01.746005+0000', 'last_clean': '2024-09-16T03:40:39.185069+0000', 'last_became_active': '2024-09-16T03:40:41.619774+0000', 'last_became_peered': '2024-09-16T03:40:41.619774+0000', 'last_unstale': '2024-09-16T03:41:01.746005+0000', 'last_undegraded': '2024-09-16T03:41:01.746005+0000', 'last_fullsized': '2024-09-16T03:41:01.746005+0000', 'mapping_epoch': 215, 'log_start': "170'263", 'ondisk_log_start': "170'263", 'created': 31, 'last_epoch_clean': 201, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "170'267", 'last_scrub_stamp': '2024-09-16T03:40:38.134247+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T03:37:03.913037+0000', 'last_clean_scrub_stamp': '2024-09-16T03:40:38.134247+0000', 'objects_scrubbed': 39, 'log_size': 4, 'log_dups_size': 274, 'ondisk_log_size': 4, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 406, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28703148, 'num_objects': 39, 'num_object_clones': 0, 'num_object_copies': 117, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 39, 'num_objects_unfound': 0, 'num_objects_dirty': 39, 'num_whiteouts': 0, 'num_read': 249, 'num_read_kb': 2401, 'num_write': 343, 'num_write_kb': 78301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 17668524, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4, 3], 'acting': [6, 4, 11], 'avail_no_missing': ['6', '4', '11'], 'object_location_counts': [{'shards': '4,6,11', 'objects': 39}], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T03:41:08.290 INFO:tasks.ceph.ceph_manager.ceph:PG 3.0 is not active+clean 2024-09-16T03:41:08.290 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.0', 'version': "168'284", 'reported_seq': 894, 'reported_epoch': 230, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T03:41:01.487908+0000', 'last_change': '2024-09-16T03:41:04.127002+0000', 'last_active': '2024-09-16T03:41:01.487908+0000', 'last_peered': '2024-09-16T03:41:01.487908+0000', 'last_clean': '2024-09-16T03:41:01.487908+0000', 'last_became_active': '2024-09-16T03:40:25.634914+0000', 'last_became_peered': '2024-09-16T03:40:25.634914+0000', 'last_unstale': '2024-09-16T03:41:01.487908+0000', 'last_undegraded': '2024-09-16T03:41:01.487908+0000', 'last_fullsized': '2024-09-16T03:41:01.487908+0000', 'mapping_epoch': 199, 'log_start': "165'281", 'ondisk_log_start': "165'281", 'created': 31, 'last_epoch_clean': 200, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "155'275", 'last_scrub_stamp': '2024-09-16T03:39:38.548772+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T03:37:03.913037+0000', 'last_clean_scrub_stamp': '2024-09-16T03:39:38.548772+0000', 'objects_scrubbed': 20, 'log_size': 3, 'log_dups_size': 302, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 303, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21970448, 'num_objects': 24, 'num_object_clones': 0, 'num_object_copies': 72, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 24, 'num_whiteouts': 0, 'num_read': 258, 'num_read_kb': 13466, 'num_write': 352, 'num_write_kb': 87672, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 71, 'num_bytes_recovered': 41823896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 9, 4], 'acting': [1, 9, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-16T03:41:08.290 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T03:41:08.658 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:08.658 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":271,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":230,"num_osds":16,"num_up_osds":16,"osd_up_since":1726457810,"num_in_osds":16,"osd_in_since":1726458014,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+remapped+backfill_toofull","count":1},{"state_name":"active+clean+scrubbing","count":1}],"num_pgs":13,"num_pools":3,"num_objects":150,"data_bytes":103194545,"bytes_used":743456768,"bytes_avail":1545444769792,"bytes_total":1546188226560,"misplaced_objects":39,"misplaced_total":450,"misplaced_ratio":0.086666666666666684},"fsmap":{"epoch":1,"btime":"2024-09-16T03:36:37:310011+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T03:37:51.688536+0000","services":{}},"progress_events":{}} 2024-09-16T03:41:08.671 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-16T03:41:08.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:08.970 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:08.970 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:08.982 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-16T03:41:07.730721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":726032,"kb_used_data":206224,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509223408,"statfs":{"total":1546188226560,"available":1545444769792,"internally_reserved":0,"allocated":211173376,"data_stored":218336599,"data_compressed":14000041,"data_compressed_allocated":15527936,"data_compressed_original":32542720,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.997031"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":980,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910389+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910389+0000","last_peered":"2024-09-16T03:41:01.910389+0000","last_clean":"2024-09-16T03:41:01.910389+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910389+0000","last_undegraded":"2024-09-16T03:41:01.910389+0000","last_fullsized":"2024-09-16T03:41:01.910389+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":470,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010819+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:02.010819+0000","last_peered":"2024-09-16T03:41:02.010819+0000","last_clean":"2024-09-16T03:41:02.010819+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:02.010819+0000","last_undegraded":"2024-09-16T03:41:02.010819+0000","last_fullsized":"2024-09-16T03:41:02.010819+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":838,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:41:01.746005+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:41:01.746005+0000","last_peered":"2024-09-16T03:41:01.746005+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:01.746005+0000","last_undegraded":"2024-09-16T03:41:01.746005+0000","last_fullsized":"2024-09-16T03:41:01.746005+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":894,"reported_epoch":230,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T03:41:01.487908+0000","last_change":"2024-09-16T03:41:04.127002+0000","last_active":"2024-09-16T03:41:01.487908+0000","last_peered":"2024-09-16T03:41:01.487908+0000","last_clean":"2024-09-16T03:41:01.487908+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:01.487908+0000","last_undegraded":"2024-09-16T03:41:01.487908+0000","last_fullsized":"2024-09-16T03:41:01.487908+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'275","last_scrub_stamp":"2024-09-16T03:39:38.548772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:39:38.548772+0000","objects_scrubbed":20,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":169,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010817+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:02.010817+0000","last_peered":"2024-09-16T03:41:02.010817+0000","last_clean":"2024-09-16T03:41:02.010817+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:02.010817+0000","last_undegraded":"2024-09-16T03:41:02.010817+0000","last_fullsized":"2024-09-16T03:41:02.010817+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":447,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645907+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645907+0000","last_peered":"2024-09-16T03:41:01.645907+0000","last_clean":"2024-09-16T03:41:01.645907+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645907+0000","last_undegraded":"2024-09-16T03:41:01.645907+0000","last_fullsized":"2024-09-16T03:41:01.645907+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":209,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645061+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:01.645061+0000","last_peered":"2024-09-16T03:41:01.645061+0000","last_clean":"2024-09-16T03:41:01.645061+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:01.645061+0000","last_undegraded":"2024-09-16T03:41:01.645061+0000","last_fullsized":"2024-09-16T03:41:01.645061+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.746155+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:41:01.746155+0000","last_peered":"2024-09-16T03:41:01.746155+0000","last_clean":"2024-09-16T03:41:01.746155+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:01.746155+0000","last_undegraded":"2024-09-16T03:41:01.746155+0000","last_fullsized":"2024-09-16T03:41:01.746155+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":438,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010829+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:02.010829+0000","last_peered":"2024-09-16T03:41:02.010829+0000","last_clean":"2024-09-16T03:41:02.010829+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:02.010829+0000","last_undegraded":"2024-09-16T03:41:02.010829+0000","last_fullsized":"2024-09-16T03:41:02.010829+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247863,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69544,"kb_used_data":34196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302296,"statfs":{"total":96636764160,"available":96565551104,"internally_reserved":0,"allocated":35016704,"data_stored":37479003,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247862,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":17388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":17805312,"data_stored":18652836,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280567,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28412,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343428,"statfs":{"total":96636764160,"available":96607670272,"internally_reserved":0,"allocated":1417216,"data_stored":983337,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46740,"kb_used_data":15552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325100,"statfs":{"total":96636764160,"available":96588902400,"internally_reserved":0,"allocated":15925248,"data_stored":16370406,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313271,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984781,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1445888,"data_stored":986882,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66924,"kb_used_data":31576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304916,"statfs":{"total":96636764160,"available":96568233984,"internally_reserved":0,"allocated":32333824,"data_stored":34041624,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313271,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49348,"kb_used_data":18160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322492,"statfs":{"total":96636764160,"available":96586231808,"internally_reserved":0,"allocated":18595840,"data_stored":19808494,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280566,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280566,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51092,"kb_used_data":15744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320748,"statfs":{"total":96636764160,"available":96584445952,"internally_reserved":0,"allocated":16121856,"data_stored":16630927,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280566,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15921152,"data_stored":16370014,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1441792,"data_stored":986588,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313270,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":985093,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46924,"kb_used_data":15736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324916,"statfs":{"total":96636764160,"available":96588713984,"internally_reserved":0,"allocated":16113664,"data_stored":16629164,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:11.984 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:12.297 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:12.297 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:12.310 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-16T03:41:11.731783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":726028,"kb_used_data":206224,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509223412,"statfs":{"total":1546188226560,"available":1545444773888,"internally_reserved":0,"allocated":211173376,"data_stored":218336599,"data_compressed":14000041,"data_compressed_allocated":15527936,"data_compressed_original":32542720,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.345506"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":980,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910389+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910389+0000","last_peered":"2024-09-16T03:41:01.910389+0000","last_clean":"2024-09-16T03:41:01.910389+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910389+0000","last_undegraded":"2024-09-16T03:41:01.910389+0000","last_fullsized":"2024-09-16T03:41:01.910389+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":470,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010819+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:02.010819+0000","last_peered":"2024-09-16T03:41:02.010819+0000","last_clean":"2024-09-16T03:41:02.010819+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:02.010819+0000","last_undegraded":"2024-09-16T03:41:02.010819+0000","last_fullsized":"2024-09-16T03:41:02.010819+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":838,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:41:01.746005+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:41:01.746005+0000","last_peered":"2024-09-16T03:41:01.746005+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:01.746005+0000","last_undegraded":"2024-09-16T03:41:01.746005+0000","last_fullsized":"2024-09-16T03:41:01.746005+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":901,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:04.232326+0000","last_change":"2024-09-16T03:41:04.232326+0000","last_active":"2024-09-16T03:41:04.232326+0000","last_peered":"2024-09-16T03:41:04.232326+0000","last_clean":"2024-09-16T03:41:04.232326+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:04.232326+0000","last_undegraded":"2024-09-16T03:41:04.232326+0000","last_fullsized":"2024-09-16T03:41:04.232326+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'284","last_scrub_stamp":"2024-09-16T03:41:04.232204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:41:04.232204+0000","objects_scrubbed":24,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:14.971430+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":169,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010817+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:02.010817+0000","last_peered":"2024-09-16T03:41:02.010817+0000","last_clean":"2024-09-16T03:41:02.010817+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:02.010817+0000","last_undegraded":"2024-09-16T03:41:02.010817+0000","last_fullsized":"2024-09-16T03:41:02.010817+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":447,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645907+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645907+0000","last_peered":"2024-09-16T03:41:01.645907+0000","last_clean":"2024-09-16T03:41:01.645907+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645907+0000","last_undegraded":"2024-09-16T03:41:01.645907+0000","last_fullsized":"2024-09-16T03:41:01.645907+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":209,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645061+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:01.645061+0000","last_peered":"2024-09-16T03:41:01.645061+0000","last_clean":"2024-09-16T03:41:01.645061+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:01.645061+0000","last_undegraded":"2024-09-16T03:41:01.645061+0000","last_fullsized":"2024-09-16T03:41:01.645061+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.746155+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:41:01.746155+0000","last_peered":"2024-09-16T03:41:01.746155+0000","last_clean":"2024-09-16T03:41:01.746155+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:01.746155+0000","last_undegraded":"2024-09-16T03:41:01.746155+0000","last_fullsized":"2024-09-16T03:41:01.746155+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":438,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010829+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:02.010829+0000","last_peered":"2024-09-16T03:41:02.010829+0000","last_clean":"2024-09-16T03:41:02.010829+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:02.010829+0000","last_undegraded":"2024-09-16T03:41:02.010829+0000","last_fullsized":"2024-09-16T03:41:02.010829+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247863,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69544,"kb_used_data":34196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302296,"statfs":{"total":96636764160,"available":96565551104,"internally_reserved":0,"allocated":35016704,"data_stored":37479003,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247862,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":17388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":17805312,"data_stored":18652836,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280568,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28412,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343428,"statfs":{"total":96636764160,"available":96607670272,"internally_reserved":0,"allocated":1417216,"data_stored":983337,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15925248,"data_stored":16370406,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313272,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984781,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1445888,"data_stored":986882,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66924,"kb_used_data":31576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304916,"statfs":{"total":96636764160,"available":96568233984,"internally_reserved":0,"allocated":32333824,"data_stored":34041624,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313272,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49348,"kb_used_data":18160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322492,"statfs":{"total":96636764160,"available":96586231808,"internally_reserved":0,"allocated":18595840,"data_stored":19808494,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280567,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280567,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51092,"kb_used_data":15744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320748,"statfs":{"total":96636764160,"available":96584445952,"internally_reserved":0,"allocated":16121856,"data_stored":16630927,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15921152,"data_stored":16370014,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1441792,"data_stored":986588,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313271,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":985093,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46924,"kb_used_data":15736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324916,"statfs":{"total":96636764160,"available":96588713984,"internally_reserved":0,"allocated":16113664,"data_stored":16629164,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:12.312 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:12.608 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:12.620 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-16T03:41:11.731783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":726028,"kb_used_data":206224,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509223412,"statfs":{"total":1546188226560,"available":1545444773888,"internally_reserved":0,"allocated":211173376,"data_stored":218336599,"data_compressed":14000041,"data_compressed_allocated":15527936,"data_compressed_original":32542720,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.345506"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":980,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910389+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910389+0000","last_peered":"2024-09-16T03:41:01.910389+0000","last_clean":"2024-09-16T03:41:01.910389+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910389+0000","last_undegraded":"2024-09-16T03:41:01.910389+0000","last_fullsized":"2024-09-16T03:41:01.910389+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":470,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010819+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:02.010819+0000","last_peered":"2024-09-16T03:41:02.010819+0000","last_clean":"2024-09-16T03:41:02.010819+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:02.010819+0000","last_undegraded":"2024-09-16T03:41:02.010819+0000","last_fullsized":"2024-09-16T03:41:02.010819+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":838,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:41:01.746005+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:41:01.746005+0000","last_peered":"2024-09-16T03:41:01.746005+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:01.746005+0000","last_undegraded":"2024-09-16T03:41:01.746005+0000","last_fullsized":"2024-09-16T03:41:01.746005+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":901,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:04.232326+0000","last_change":"2024-09-16T03:41:04.232326+0000","last_active":"2024-09-16T03:41:04.232326+0000","last_peered":"2024-09-16T03:41:04.232326+0000","last_clean":"2024-09-16T03:41:04.232326+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:04.232326+0000","last_undegraded":"2024-09-16T03:41:04.232326+0000","last_fullsized":"2024-09-16T03:41:04.232326+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'284","last_scrub_stamp":"2024-09-16T03:41:04.232204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:41:04.232204+0000","objects_scrubbed":24,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:14.971430+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":169,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010817+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:02.010817+0000","last_peered":"2024-09-16T03:41:02.010817+0000","last_clean":"2024-09-16T03:41:02.010817+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:02.010817+0000","last_undegraded":"2024-09-16T03:41:02.010817+0000","last_fullsized":"2024-09-16T03:41:02.010817+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":447,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645907+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645907+0000","last_peered":"2024-09-16T03:41:01.645907+0000","last_clean":"2024-09-16T03:41:01.645907+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645907+0000","last_undegraded":"2024-09-16T03:41:01.645907+0000","last_fullsized":"2024-09-16T03:41:01.645907+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":209,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645061+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:01.645061+0000","last_peered":"2024-09-16T03:41:01.645061+0000","last_clean":"2024-09-16T03:41:01.645061+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:01.645061+0000","last_undegraded":"2024-09-16T03:41:01.645061+0000","last_fullsized":"2024-09-16T03:41:01.645061+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.746155+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:41:01.746155+0000","last_peered":"2024-09-16T03:41:01.746155+0000","last_clean":"2024-09-16T03:41:01.746155+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:01.746155+0000","last_undegraded":"2024-09-16T03:41:01.746155+0000","last_fullsized":"2024-09-16T03:41:01.746155+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":438,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010829+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:02.010829+0000","last_peered":"2024-09-16T03:41:02.010829+0000","last_clean":"2024-09-16T03:41:02.010829+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:02.010829+0000","last_undegraded":"2024-09-16T03:41:02.010829+0000","last_fullsized":"2024-09-16T03:41:02.010829+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247863,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69544,"kb_used_data":34196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302296,"statfs":{"total":96636764160,"available":96565551104,"internally_reserved":0,"allocated":35016704,"data_stored":37479003,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247862,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":17388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":17805312,"data_stored":18652836,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280568,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28412,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343428,"statfs":{"total":96636764160,"available":96607670272,"internally_reserved":0,"allocated":1417216,"data_stored":983337,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15925248,"data_stored":16370406,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313272,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984781,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1445888,"data_stored":986882,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66924,"kb_used_data":31576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304916,"statfs":{"total":96636764160,"available":96568233984,"internally_reserved":0,"allocated":32333824,"data_stored":34041624,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313272,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49348,"kb_used_data":18160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322492,"statfs":{"total":96636764160,"available":96586231808,"internally_reserved":0,"allocated":18595840,"data_stored":19808494,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280567,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280567,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51092,"kb_used_data":15744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320748,"statfs":{"total":96636764160,"available":96584445952,"internally_reserved":0,"allocated":16121856,"data_stored":16630927,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15921152,"data_stored":16370014,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1441792,"data_stored":986588,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313271,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":985093,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46924,"kb_used_data":15736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324916,"statfs":{"total":96636764160,"available":96588713984,"internally_reserved":0,"allocated":16113664,"data_stored":16629164,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:12.621 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-09-16T03:41:12.621 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "170'267", 'reported_seq': 838, 'reported_epoch': 230, 'state': 'active+remapped+backfill_toofull', 'last_fresh': '2024-09-16T03:41:01.746005+0000', 'last_change': '2024-09-16T03:40:41.720636+0000', 'last_active': '2024-09-16T03:41:01.746005+0000', 'last_peered': '2024-09-16T03:41:01.746005+0000', 'last_clean': '2024-09-16T03:40:39.185069+0000', 'last_became_active': '2024-09-16T03:40:41.619774+0000', 'last_became_peered': '2024-09-16T03:40:41.619774+0000', 'last_unstale': '2024-09-16T03:41:01.746005+0000', 'last_undegraded': '2024-09-16T03:41:01.746005+0000', 'last_fullsized': '2024-09-16T03:41:01.746005+0000', 'mapping_epoch': 215, 'log_start': "170'263", 'ondisk_log_start': "170'263", 'created': 31, 'last_epoch_clean': 201, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "170'267", 'last_scrub_stamp': '2024-09-16T03:40:38.134247+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T03:37:03.913037+0000', 'last_clean_scrub_stamp': '2024-09-16T03:40:38.134247+0000', 'objects_scrubbed': 39, 'log_size': 4, 'log_dups_size': 274, 'ondisk_log_size': 4, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 406, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28703148, 'num_objects': 39, 'num_object_clones': 0, 'num_object_copies': 117, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 39, 'num_objects_unfound': 0, 'num_objects_dirty': 39, 'num_whiteouts': 0, 'num_read': 249, 'num_read_kb': 2401, 'num_write': 343, 'num_write_kb': 78301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 17668524, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4, 3], 'acting': [6, 4, 11], 'avail_no_missing': ['6', '4', '11'], 'object_location_counts': [{'shards': '4,6,11', 'objects': 39}], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T03:41:12.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T03:41:12.998 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:12.998 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","health":{"status":"HEALTH_WARN","checks":{"PG_BACKFILL_FULL":{"severity":"HEALTH_WARN","summary":{"message":"Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":275,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":230,"num_osds":16,"num_up_osds":16,"osd_up_since":1726457810,"num_in_osds":16,"osd_in_since":1726458014,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfill_toofull","count":1}],"num_pgs":13,"num_pools":3,"num_objects":150,"data_bytes":103194545,"bytes_used":743452672,"bytes_avail":1545444773888,"bytes_total":1546188226560,"misplaced_objects":39,"misplaced_total":450,"misplaced_ratio":0.086666666666666684},"fsmap":{"epoch":1,"btime":"2024-09-16T03:36:37:310011+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T03:37:51.688536+0000","services":{}},"progress_events":{}} 2024-09-16T03:41:13.011 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-16T03:41:13.011 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:13.307 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:13.308 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:13.320 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-16T03:41:11.731783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":123068338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":726028,"kb_used_data":206224,"kb_used_omap":24,"kb_used_meta":519463,"kb_avail":1509223412,"statfs":{"total":1546188226560,"available":1545444773888,"internally_reserved":0,"allocated":211173376,"data_stored":218336599,"data_compressed":14000041,"data_compressed_allocated":15527936,"data_compressed_original":32542720,"omap_allocated":25519,"internal_metadata":531930193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.345506"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":980,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910389+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910389+0000","last_peered":"2024-09-16T03:41:01.910389+0000","last_clean":"2024-09-16T03:41:01.910389+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910389+0000","last_undegraded":"2024-09-16T03:41:01.910389+0000","last_fullsized":"2024-09-16T03:41:01.910389+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":470,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010819+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:02.010819+0000","last_peered":"2024-09-16T03:41:02.010819+0000","last_clean":"2024-09-16T03:41:02.010819+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:02.010819+0000","last_undegraded":"2024-09-16T03:41:02.010819+0000","last_fullsized":"2024-09-16T03:41:02.010819+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":838,"reported_epoch":230,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-16T03:41:01.746005+0000","last_change":"2024-09-16T03:40:41.720636+0000","last_active":"2024-09-16T03:41:01.746005+0000","last_peered":"2024-09-16T03:41:01.746005+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:01.746005+0000","last_undegraded":"2024-09-16T03:41:01.746005+0000","last_fullsized":"2024-09-16T03:41:01.746005+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":17668524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":901,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:04.232326+0000","last_change":"2024-09-16T03:41:04.232326+0000","last_active":"2024-09-16T03:41:04.232326+0000","last_peered":"2024-09-16T03:41:04.232326+0000","last_clean":"2024-09-16T03:41:04.232326+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:04.232326+0000","last_undegraded":"2024-09-16T03:41:04.232326+0000","last_fullsized":"2024-09-16T03:41:04.232326+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'284","last_scrub_stamp":"2024-09-16T03:41:04.232204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:41:04.232204+0000","objects_scrubbed":24,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:14.971430+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":169,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010817+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:02.010817+0000","last_peered":"2024-09-16T03:41:02.010817+0000","last_clean":"2024-09-16T03:41:02.010817+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:02.010817+0000","last_undegraded":"2024-09-16T03:41:02.010817+0000","last_fullsized":"2024-09-16T03:41:02.010817+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":447,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645907+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645907+0000","last_peered":"2024-09-16T03:41:01.645907+0000","last_clean":"2024-09-16T03:41:01.645907+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645907+0000","last_undegraded":"2024-09-16T03:41:01.645907+0000","last_fullsized":"2024-09-16T03:41:01.645907+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":209,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645061+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:01.645061+0000","last_peered":"2024-09-16T03:41:01.645061+0000","last_clean":"2024-09-16T03:41:01.645061+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:01.645061+0000","last_undegraded":"2024-09-16T03:41:01.645061+0000","last_fullsized":"2024-09-16T03:41:01.645061+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.746155+0000","last_change":"2024-09-16T03:40:14.787291+0000","last_active":"2024-09-16T03:41:01.746155+0000","last_peered":"2024-09-16T03:41:01.746155+0000","last_clean":"2024-09-16T03:41:01.746155+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:01.746155+0000","last_undegraded":"2024-09-16T03:41:01.746155+0000","last_fullsized":"2024-09-16T03:41:01.746155+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:39:39.003943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:39:39.003943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:07.262184+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":438,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010829+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:02.010829+0000","last_peered":"2024-09-16T03:41:02.010829+0000","last_clean":"2024-09-16T03:41:02.010829+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:02.010829+0000","last_undegraded":"2024-09-16T03:41:02.010829+0000","last_fullsized":"2024-09-16T03:41:02.010829+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":121481554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247863,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69544,"kb_used_data":34196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302296,"statfs":{"total":96636764160,"available":96565551104,"internally_reserved":0,"allocated":35016704,"data_stored":37479003,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247862,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":17388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":17805312,"data_stored":18652836,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280568,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28412,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343428,"statfs":{"total":96636764160,"available":96607670272,"internally_reserved":0,"allocated":1417216,"data_stored":983337,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15925248,"data_stored":16370406,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313272,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984781,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36760,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335080,"statfs":{"total":96636764160,"available":96599121920,"internally_reserved":0,"allocated":1445888,"data_stored":986882,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66924,"kb_used_data":31576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304916,"statfs":{"total":96636764160,"available":96568233984,"internally_reserved":0,"allocated":32333824,"data_stored":34041624,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313272,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49348,"kb_used_data":18160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322492,"statfs":{"total":96636764160,"available":96586231808,"internally_reserved":0,"allocated":18595840,"data_stored":19808494,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280567,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280567,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51092,"kb_used_data":15744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320748,"statfs":{"total":96636764160,"available":96584445952,"internally_reserved":0,"allocated":16121856,"data_stored":16630927,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280567,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15921152,"data_stored":16370014,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1441792,"data_stored":986588,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313271,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":985093,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46924,"kb_used_data":15736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324916,"statfs":{"total":96636764160,"available":96588713984,"internally_reserved":0,"allocated":16113664,"data_stored":16629164,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:16.322 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:16.632 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:16.632 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:16.644 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-09-16T03:41:15.732725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":124237954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":737748,"kb_used_data":213792,"kb_used_omap":24,"kb_used_meta":523623,"kb_avail":1509211692,"statfs":{"total":1546188226560,"available":1545432772608,"internally_reserved":0,"allocated":218923008,"data_stored":226659408,"data_compressed":14431384,"data_compressed_allocated":15998976,"data_compressed_original":33611776,"omap_allocated":25519,"internal_metadata":536190033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1169616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002900"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":980,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910389+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910389+0000","last_peered":"2024-09-16T03:41:01.910389+0000","last_clean":"2024-09-16T03:41:01.910389+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910389+0000","last_undegraded":"2024-09-16T03:41:01.910389+0000","last_fullsized":"2024-09-16T03:41:01.910389+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":470,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010819+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:02.010819+0000","last_peered":"2024-09-16T03:41:02.010819+0000","last_clean":"2024-09-16T03:41:02.010819+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:02.010819+0000","last_undegraded":"2024-09-16T03:41:02.010819+0000","last_fullsized":"2024-09-16T03:41:02.010819+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":845,"reported_epoch":230,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:41:12.233840+0000","last_change":"2024-09-16T03:41:11.721909+0000","last_active":"2024-09-16T03:41:12.233840+0000","last_peered":"2024-09-16T03:41:12.233840+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:12.233840+0000","last_undegraded":"2024-09-16T03:41:12.233840+0000","last_fullsized":"2024-09-16T03:41:12.233840+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":18838140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":901,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:04.232326+0000","last_change":"2024-09-16T03:41:04.232326+0000","last_active":"2024-09-16T03:41:04.232326+0000","last_peered":"2024-09-16T03:41:04.232326+0000","last_clean":"2024-09-16T03:41:04.232326+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:04.232326+0000","last_undegraded":"2024-09-16T03:41:04.232326+0000","last_fullsized":"2024-09-16T03:41:04.232326+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'284","last_scrub_stamp":"2024-09-16T03:41:04.232204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:41:04.232204+0000","objects_scrubbed":24,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:14.971430+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":169,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010817+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:02.010817+0000","last_peered":"2024-09-16T03:41:02.010817+0000","last_clean":"2024-09-16T03:41:02.010817+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:02.010817+0000","last_undegraded":"2024-09-16T03:41:02.010817+0000","last_fullsized":"2024-09-16T03:41:02.010817+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":447,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645907+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645907+0000","last_peered":"2024-09-16T03:41:01.645907+0000","last_clean":"2024-09-16T03:41:01.645907+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645907+0000","last_undegraded":"2024-09-16T03:41:01.645907+0000","last_fullsized":"2024-09-16T03:41:01.645907+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":210,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645096+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:01.645096+0000","last_peered":"2024-09-16T03:41:01.645096+0000","last_clean":"2024-09-16T03:41:01.645096+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:01.645096+0000","last_undegraded":"2024-09-16T03:41:01.645096+0000","last_fullsized":"2024-09-16T03:41:01.645096+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":462,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:12.745026+0000","last_change":"2024-09-16T03:41:12.744967+0000","last_active":"2024-09-16T03:41:12.745026+0000","last_peered":"2024-09-16T03:41:12.745026+0000","last_clean":"2024-09-16T03:41:12.745026+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:12.745026+0000","last_undegraded":"2024-09-16T03:41:12.745026+0000","last_fullsized":"2024-09-16T03:41:12.745026+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:12.744887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:41:12.744887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.656161+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":439,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010885+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:02.010885+0000","last_peered":"2024-09-16T03:41:02.010885+0000","last_clean":"2024-09-16T03:41:02.010885+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:02.010885+0000","last_undegraded":"2024-09-16T03:41:02.010885+0000","last_fullsized":"2024-09-16T03:41:02.010885+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":122651170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196968448,"data_stored":210367509,"data_compressed":12792038,"data_compressed_allocated":13672448,"data_compressed_original":27672576,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247864,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69544,"kb_used_data":34196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302296,"statfs":{"total":96636764160,"available":96565551104,"internally_reserved":0,"allocated":35016704,"data_stored":37479003,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247863,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":17388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":17805312,"data_stored":18652836,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280569,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28412,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343428,"statfs":{"total":96636764160,"available":96607670272,"internally_reserved":0,"allocated":1417216,"data_stored":983337,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15925248,"data_stored":16370406,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313273,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1417216,"data_stored":984781,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48484,"kb_used_data":8980,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94323356,"statfs":{"total":96636764160,"available":96587116544,"internally_reserved":0,"allocated":9195520,"data_stored":9309691,"data_compressed":532921,"data_compressed_allocated":614400,"data_compressed_original":1355776,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66924,"kb_used_data":31576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304916,"statfs":{"total":96636764160,"available":96568233984,"internally_reserved":0,"allocated":32333824,"data_stored":34041624,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313273,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49348,"kb_used_data":18160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322492,"statfs":{"total":96636764160,"available":96586231808,"internally_reserved":0,"allocated":18595840,"data_stored":19808494,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280568,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280568,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51092,"kb_used_data":15744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320748,"statfs":{"total":96636764160,"available":96584445952,"internally_reserved":0,"allocated":16121856,"data_stored":16630927,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15921152,"data_stored":16370014,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1441792,"data_stored":986588,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313272,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":985093,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46924,"kb_used_data":15736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324916,"statfs":{"total":96636764160,"available":96588713984,"internally_reserved":0,"allocated":16113664,"data_stored":16629164,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8691712,"data_stored":9371385,"data_compressed":512402,"data_compressed_allocated":561152,"data_compressed_original":1265664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:16.646 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:16.941 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:16.942 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:16.954 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-09-16T03:41:15.732725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":124237954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":737748,"kb_used_data":213792,"kb_used_omap":24,"kb_used_meta":523623,"kb_avail":1509211692,"statfs":{"total":1546188226560,"available":1545432772608,"internally_reserved":0,"allocated":218923008,"data_stored":226659408,"data_compressed":14431384,"data_compressed_allocated":15998976,"data_compressed_original":33611776,"omap_allocated":25519,"internal_metadata":536190033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1169616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002900"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":980,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910389+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910389+0000","last_peered":"2024-09-16T03:41:01.910389+0000","last_clean":"2024-09-16T03:41:01.910389+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910389+0000","last_undegraded":"2024-09-16T03:41:01.910389+0000","last_fullsized":"2024-09-16T03:41:01.910389+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":470,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010819+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:02.010819+0000","last_peered":"2024-09-16T03:41:02.010819+0000","last_clean":"2024-09-16T03:41:02.010819+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:02.010819+0000","last_undegraded":"2024-09-16T03:41:02.010819+0000","last_fullsized":"2024-09-16T03:41:02.010819+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":845,"reported_epoch":230,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:41:12.233840+0000","last_change":"2024-09-16T03:41:11.721909+0000","last_active":"2024-09-16T03:41:12.233840+0000","last_peered":"2024-09-16T03:41:12.233840+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:12.233840+0000","last_undegraded":"2024-09-16T03:41:12.233840+0000","last_fullsized":"2024-09-16T03:41:12.233840+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":18838140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":901,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:04.232326+0000","last_change":"2024-09-16T03:41:04.232326+0000","last_active":"2024-09-16T03:41:04.232326+0000","last_peered":"2024-09-16T03:41:04.232326+0000","last_clean":"2024-09-16T03:41:04.232326+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:04.232326+0000","last_undegraded":"2024-09-16T03:41:04.232326+0000","last_fullsized":"2024-09-16T03:41:04.232326+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'284","last_scrub_stamp":"2024-09-16T03:41:04.232204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:41:04.232204+0000","objects_scrubbed":24,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:14.971430+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":169,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010817+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:02.010817+0000","last_peered":"2024-09-16T03:41:02.010817+0000","last_clean":"2024-09-16T03:41:02.010817+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:02.010817+0000","last_undegraded":"2024-09-16T03:41:02.010817+0000","last_fullsized":"2024-09-16T03:41:02.010817+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":447,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645907+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645907+0000","last_peered":"2024-09-16T03:41:01.645907+0000","last_clean":"2024-09-16T03:41:01.645907+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645907+0000","last_undegraded":"2024-09-16T03:41:01.645907+0000","last_fullsized":"2024-09-16T03:41:01.645907+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":210,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645096+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:01.645096+0000","last_peered":"2024-09-16T03:41:01.645096+0000","last_clean":"2024-09-16T03:41:01.645096+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:01.645096+0000","last_undegraded":"2024-09-16T03:41:01.645096+0000","last_fullsized":"2024-09-16T03:41:01.645096+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":462,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:12.745026+0000","last_change":"2024-09-16T03:41:12.744967+0000","last_active":"2024-09-16T03:41:12.745026+0000","last_peered":"2024-09-16T03:41:12.745026+0000","last_clean":"2024-09-16T03:41:12.745026+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:12.745026+0000","last_undegraded":"2024-09-16T03:41:12.745026+0000","last_fullsized":"2024-09-16T03:41:12.745026+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:12.744887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:41:12.744887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.656161+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":439,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010885+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:02.010885+0000","last_peered":"2024-09-16T03:41:02.010885+0000","last_clean":"2024-09-16T03:41:02.010885+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:02.010885+0000","last_undegraded":"2024-09-16T03:41:02.010885+0000","last_fullsized":"2024-09-16T03:41:02.010885+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":122651170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196968448,"data_stored":210367509,"data_compressed":12792038,"data_compressed_allocated":13672448,"data_compressed_original":27672576,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247864,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69544,"kb_used_data":34196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302296,"statfs":{"total":96636764160,"available":96565551104,"internally_reserved":0,"allocated":35016704,"data_stored":37479003,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247863,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":17388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":17805312,"data_stored":18652836,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280569,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28412,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343428,"statfs":{"total":96636764160,"available":96607670272,"internally_reserved":0,"allocated":1417216,"data_stored":983337,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15925248,"data_stored":16370406,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313273,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1417216,"data_stored":984781,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48484,"kb_used_data":8980,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94323356,"statfs":{"total":96636764160,"available":96587116544,"internally_reserved":0,"allocated":9195520,"data_stored":9309691,"data_compressed":532921,"data_compressed_allocated":614400,"data_compressed_original":1355776,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66924,"kb_used_data":31576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304916,"statfs":{"total":96636764160,"available":96568233984,"internally_reserved":0,"allocated":32333824,"data_stored":34041624,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313273,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49348,"kb_used_data":18160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322492,"statfs":{"total":96636764160,"available":96586231808,"internally_reserved":0,"allocated":18595840,"data_stored":19808494,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280568,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280568,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51092,"kb_used_data":15744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320748,"statfs":{"total":96636764160,"available":96584445952,"internally_reserved":0,"allocated":16121856,"data_stored":16630927,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15921152,"data_stored":16370014,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1441792,"data_stored":986588,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313272,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":985093,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46924,"kb_used_data":15736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324916,"statfs":{"total":96636764160,"available":96588713984,"internally_reserved":0,"allocated":16113664,"data_stored":16629164,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8691712,"data_stored":9371385,"data_compressed":512402,"data_compressed_allocated":561152,"data_compressed_original":1265664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:16.956 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-09-16T03:41:16.956 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "170'267", 'reported_seq': 845, 'reported_epoch': 230, 'state': 'active+remapped+backfilling', 'last_fresh': '2024-09-16T03:41:12.233840+0000', 'last_change': '2024-09-16T03:41:11.721909+0000', 'last_active': '2024-09-16T03:41:12.233840+0000', 'last_peered': '2024-09-16T03:41:12.233840+0000', 'last_clean': '2024-09-16T03:40:39.185069+0000', 'last_became_active': '2024-09-16T03:40:41.619774+0000', 'last_became_peered': '2024-09-16T03:40:41.619774+0000', 'last_unstale': '2024-09-16T03:41:12.233840+0000', 'last_undegraded': '2024-09-16T03:41:12.233840+0000', 'last_fullsized': '2024-09-16T03:41:12.233840+0000', 'mapping_epoch': 215, 'log_start': "170'263", 'ondisk_log_start': "170'263", 'created': 31, 'last_epoch_clean': 201, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "170'267", 'last_scrub_stamp': '2024-09-16T03:40:38.134247+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T03:37:03.913037+0000', 'last_clean_scrub_stamp': '2024-09-16T03:40:38.134247+0000', 'objects_scrubbed': 39, 'log_size': 4, 'log_dups_size': 274, 'ondisk_log_size': 4, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 406, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28703148, 'num_objects': 39, 'num_object_clones': 0, 'num_object_copies': 117, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 38, 'num_objects_unfound': 0, 'num_objects_dirty': 39, 'num_whiteouts': 0, 'num_read': 249, 'num_read_kb': 2401, 'num_write': 343, 'num_write_kb': 78301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 18838140, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4, 3], 'acting': [6, 4, 11], 'avail_no_missing': ['6', '4', '11'], 'object_location_counts': [{'shards': '4,6,11', 'objects': 39}], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T03:41:16.956 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:17.324 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","health":{"status":"HEALTH_WARN","checks":{"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":280,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":230,"num_osds":16,"num_up_osds":16,"osd_up_since":1726457810,"num_in_osds":16,"osd_in_since":1726458014,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfilling","count":1}],"num_pgs":13,"num_pools":3,"num_objects":150,"data_bytes":103194545,"bytes_used":755453952,"bytes_avail":1545432772608,"bytes_total":1546188226560,"misplaced_objects":38,"misplaced_total":450,"misplaced_ratio":0.084444444444444447,"recovering_objects_per_sec":0,"recovering_bytes_per_sec":97444,"recovering_keys_per_sec":0,"num_objects_recovered":2,"num_bytes_recovered":1169616,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T03:36:37:310011+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T03:37:51.688536+0000","services":{}},"progress_events":{}} 2024-09-16T03:41:17.336 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-16T03:41:17.337 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:17.634 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:17.634 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:17.647 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-09-16T03:41:15.732725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":124237954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":737748,"kb_used_data":213792,"kb_used_omap":24,"kb_used_meta":523623,"kb_avail":1509211692,"statfs":{"total":1546188226560,"available":1545432772608,"internally_reserved":0,"allocated":218923008,"data_stored":226659408,"data_compressed":14431384,"data_compressed_allocated":15998976,"data_compressed_original":33611776,"omap_allocated":25519,"internal_metadata":536190033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1169616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002900"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":980,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910389+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910389+0000","last_peered":"2024-09-16T03:41:01.910389+0000","last_clean":"2024-09-16T03:41:01.910389+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910389+0000","last_undegraded":"2024-09-16T03:41:01.910389+0000","last_fullsized":"2024-09-16T03:41:01.910389+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":470,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010819+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:02.010819+0000","last_peered":"2024-09-16T03:41:02.010819+0000","last_clean":"2024-09-16T03:41:02.010819+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:02.010819+0000","last_undegraded":"2024-09-16T03:41:02.010819+0000","last_fullsized":"2024-09-16T03:41:02.010819+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":845,"reported_epoch":230,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:41:12.233840+0000","last_change":"2024-09-16T03:41:11.721909+0000","last_active":"2024-09-16T03:41:12.233840+0000","last_peered":"2024-09-16T03:41:12.233840+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:12.233840+0000","last_undegraded":"2024-09-16T03:41:12.233840+0000","last_fullsized":"2024-09-16T03:41:12.233840+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":18838140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":901,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:04.232326+0000","last_change":"2024-09-16T03:41:04.232326+0000","last_active":"2024-09-16T03:41:04.232326+0000","last_peered":"2024-09-16T03:41:04.232326+0000","last_clean":"2024-09-16T03:41:04.232326+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:04.232326+0000","last_undegraded":"2024-09-16T03:41:04.232326+0000","last_fullsized":"2024-09-16T03:41:04.232326+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'284","last_scrub_stamp":"2024-09-16T03:41:04.232204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:41:04.232204+0000","objects_scrubbed":24,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:14.971430+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":169,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010817+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:02.010817+0000","last_peered":"2024-09-16T03:41:02.010817+0000","last_clean":"2024-09-16T03:41:02.010817+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:02.010817+0000","last_undegraded":"2024-09-16T03:41:02.010817+0000","last_fullsized":"2024-09-16T03:41:02.010817+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":447,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645907+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645907+0000","last_peered":"2024-09-16T03:41:01.645907+0000","last_clean":"2024-09-16T03:41:01.645907+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645907+0000","last_undegraded":"2024-09-16T03:41:01.645907+0000","last_fullsized":"2024-09-16T03:41:01.645907+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":210,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645096+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:01.645096+0000","last_peered":"2024-09-16T03:41:01.645096+0000","last_clean":"2024-09-16T03:41:01.645096+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:01.645096+0000","last_undegraded":"2024-09-16T03:41:01.645096+0000","last_fullsized":"2024-09-16T03:41:01.645096+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":462,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:12.745026+0000","last_change":"2024-09-16T03:41:12.744967+0000","last_active":"2024-09-16T03:41:12.745026+0000","last_peered":"2024-09-16T03:41:12.745026+0000","last_clean":"2024-09-16T03:41:12.745026+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:12.745026+0000","last_undegraded":"2024-09-16T03:41:12.745026+0000","last_fullsized":"2024-09-16T03:41:12.745026+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:12.744887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:41:12.744887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.656161+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":439,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010885+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:02.010885+0000","last_peered":"2024-09-16T03:41:02.010885+0000","last_clean":"2024-09-16T03:41:02.010885+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:02.010885+0000","last_undegraded":"2024-09-16T03:41:02.010885+0000","last_fullsized":"2024-09-16T03:41:02.010885+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":122651170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196968448,"data_stored":210367509,"data_compressed":12792038,"data_compressed_allocated":13672448,"data_compressed_original":27672576,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247864,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69544,"kb_used_data":34196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302296,"statfs":{"total":96636764160,"available":96565551104,"internally_reserved":0,"allocated":35016704,"data_stored":37479003,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247863,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":17388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":17805312,"data_stored":18652836,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280569,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28412,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343428,"statfs":{"total":96636764160,"available":96607670272,"internally_reserved":0,"allocated":1417216,"data_stored":983337,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15925248,"data_stored":16370406,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313273,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1417216,"data_stored":984781,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48484,"kb_used_data":8980,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94323356,"statfs":{"total":96636764160,"available":96587116544,"internally_reserved":0,"allocated":9195520,"data_stored":9309691,"data_compressed":532921,"data_compressed_allocated":614400,"data_compressed_original":1355776,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66924,"kb_used_data":31576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304916,"statfs":{"total":96636764160,"available":96568233984,"internally_reserved":0,"allocated":32333824,"data_stored":34041624,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313273,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49348,"kb_used_data":18160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322492,"statfs":{"total":96636764160,"available":96586231808,"internally_reserved":0,"allocated":18595840,"data_stored":19808494,"data_compressed":1089984,"data_compressed_allocated":1212416,"data_compressed_original":3051520,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280568,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280568,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51092,"kb_used_data":15744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320748,"statfs":{"total":96636764160,"available":96584445952,"internally_reserved":0,"allocated":16121856,"data_stored":16630927,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15921152,"data_stored":16370014,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1441792,"data_stored":986588,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313272,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":985093,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46924,"kb_used_data":15736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324916,"statfs":{"total":96636764160,"available":96588713984,"internally_reserved":0,"allocated":16113664,"data_stored":16629164,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8691712,"data_stored":9371385,"data_compressed":512402,"data_compressed_allocated":561152,"data_compressed_original":1265664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:17.953 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:41:17.951+0000 7fa3886e6640 -1 osd.10 230 handle_osd_map injecting map crc failure 2024-09-16T03:41:20.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:20.960 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:20.972 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-16T03:41:19.733654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":140147038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":737844,"kb_used_data":212860,"kb_used_omap":24,"kb_used_meta":523623,"kb_avail":1509211596,"statfs":{"total":1546188226560,"available":1545432674304,"internally_reserved":0,"allocated":217968640,"data_stored":225941405,"data_compressed":14328808,"data_compressed_allocated":15851520,"data_compressed_original":33316864,"omap_allocated":25519,"internal_metadata":536190033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-38,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":17078700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001871"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":981,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910433+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910433+0000","last_peered":"2024-09-16T03:41:01.910433+0000","last_clean":"2024-09-16T03:41:01.910433+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910433+0000","last_undegraded":"2024-09-16T03:41:01.910433+0000","last_fullsized":"2024-09-16T03:41:01.910433+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":470,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010819+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:02.010819+0000","last_peered":"2024-09-16T03:41:02.010819+0000","last_clean":"2024-09-16T03:41:02.010819+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:02.010819+0000","last_undegraded":"2024-09-16T03:41:02.010819+0000","last_fullsized":"2024-09-16T03:41:02.010819+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":918,"reported_epoch":230,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:41:17.169968+0000","last_change":"2024-09-16T03:41:11.721909+0000","last_active":"2024-09-16T03:41:17.169968+0000","last_peered":"2024-09-16T03:41:17.169968+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:17.169968+0000","last_undegraded":"2024-09-16T03:41:17.169968+0000","last_fullsized":"2024-09-16T03:41:17.169968+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":34747224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":903,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.448284+0000","last_change":"2024-09-16T03:41:04.232326+0000","last_active":"2024-09-16T03:41:18.448284+0000","last_peered":"2024-09-16T03:41:18.448284+0000","last_clean":"2024-09-16T03:41:18.448284+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:18.448284+0000","last_undegraded":"2024-09-16T03:41:18.448284+0000","last_fullsized":"2024-09-16T03:41:18.448284+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'284","last_scrub_stamp":"2024-09-16T03:41:04.232204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:41:04.232204+0000","objects_scrubbed":24,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:14.971430+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":169,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010817+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:02.010817+0000","last_peered":"2024-09-16T03:41:02.010817+0000","last_clean":"2024-09-16T03:41:02.010817+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:02.010817+0000","last_undegraded":"2024-09-16T03:41:02.010817+0000","last_fullsized":"2024-09-16T03:41:02.010817+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":447,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645907+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645907+0000","last_peered":"2024-09-16T03:41:01.645907+0000","last_clean":"2024-09-16T03:41:01.645907+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645907+0000","last_undegraded":"2024-09-16T03:41:01.645907+0000","last_fullsized":"2024-09-16T03:41:01.645907+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":210,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645096+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:01.645096+0000","last_peered":"2024-09-16T03:41:01.645096+0000","last_clean":"2024-09-16T03:41:01.645096+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:01.645096+0000","last_undegraded":"2024-09-16T03:41:01.645096+0000","last_fullsized":"2024-09-16T03:41:01.645096+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":466,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.817248+0000","last_change":"2024-09-16T03:41:12.744967+0000","last_active":"2024-09-16T03:41:18.817248+0000","last_peered":"2024-09-16T03:41:18.817248+0000","last_clean":"2024-09-16T03:41:18.817248+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:18.817248+0000","last_undegraded":"2024-09-16T03:41:18.817248+0000","last_fullsized":"2024-09-16T03:41:18.817248+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:12.744887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:41:12.744887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.656161+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":439,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010885+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:02.010885+0000","last_peered":"2024-09-16T03:41:02.010885+0000","last_clean":"2024-09-16T03:41:02.010885+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:02.010885+0000","last_undegraded":"2024-09-16T03:41:02.010885+0000","last_fullsized":"2024-09-16T03:41:02.010885+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":138560254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196968448,"data_stored":210367509,"data_compressed":12792038,"data_compressed_allocated":13672448,"data_compressed_original":27672576,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247865,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69544,"kb_used_data":34196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302296,"statfs":{"total":96636764160,"available":96565551104,"internally_reserved":0,"allocated":35016704,"data_stored":37479003,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247864,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":17388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":17805312,"data_stored":18652836,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280570,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28424,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343416,"statfs":{"total":96636764160,"available":96607657984,"internally_reserved":0,"allocated":1269760,"data_stored":865113,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46752,"kb_used_data":15396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325088,"statfs":{"total":96636764160,"available":96588890112,"internally_reserved":0,"allocated":15765504,"data_stored":16250926,"data_compressed":933071,"data_compressed_allocated":1011712,"data_compressed_original":2048000,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313274,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1253376,"data_stored":864734,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48484,"kb_used_data":8980,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94323356,"statfs":{"total":96636764160,"available":96587116544,"internally_reserved":0,"allocated":9195520,"data_stored":9309691,"data_compressed":532921,"data_compressed_allocated":614400,"data_compressed_original":1355776,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66924,"kb_used_data":31576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304916,"statfs":{"total":96636764160,"available":96568233984,"internally_reserved":0,"allocated":32333824,"data_stored":34041624,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313274,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49360,"kb_used_data":18012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322480,"statfs":{"total":96636764160,"available":96586219520,"internally_reserved":0,"allocated":18444288,"data_stored":19689819,"data_compressed":1072888,"data_compressed_allocated":1187840,"data_compressed_original":3002368,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280569,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280568,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51092,"kb_used_data":15744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320748,"statfs":{"total":96636764160,"available":96584445952,"internally_reserved":0,"allocated":16121856,"data_stored":16630927,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15921152,"data_stored":16370014,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1269760,"data_stored":865900,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313273,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32604,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1261568,"data_stored":864204,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46924,"kb_used_data":15736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324916,"statfs":{"total":96636764160,"available":96588713984,"internally_reserved":0,"allocated":16113664,"data_stored":16629164,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8691712,"data_stored":9371385,"data_compressed":512402,"data_compressed_allocated":561152,"data_compressed_original":1265664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:20.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:21.273 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:21.274 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:21.286 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-16T03:41:19.733654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":140147038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":737844,"kb_used_data":212860,"kb_used_omap":24,"kb_used_meta":523623,"kb_avail":1509211596,"statfs":{"total":1546188226560,"available":1545432674304,"internally_reserved":0,"allocated":217968640,"data_stored":225941405,"data_compressed":14328808,"data_compressed_allocated":15851520,"data_compressed_original":33316864,"omap_allocated":25519,"internal_metadata":536190033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-38,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":17078700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001871"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":981,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910433+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910433+0000","last_peered":"2024-09-16T03:41:01.910433+0000","last_clean":"2024-09-16T03:41:01.910433+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910433+0000","last_undegraded":"2024-09-16T03:41:01.910433+0000","last_fullsized":"2024-09-16T03:41:01.910433+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":470,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010819+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:02.010819+0000","last_peered":"2024-09-16T03:41:02.010819+0000","last_clean":"2024-09-16T03:41:02.010819+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:02.010819+0000","last_undegraded":"2024-09-16T03:41:02.010819+0000","last_fullsized":"2024-09-16T03:41:02.010819+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":918,"reported_epoch":230,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:41:17.169968+0000","last_change":"2024-09-16T03:41:11.721909+0000","last_active":"2024-09-16T03:41:17.169968+0000","last_peered":"2024-09-16T03:41:17.169968+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:17.169968+0000","last_undegraded":"2024-09-16T03:41:17.169968+0000","last_fullsized":"2024-09-16T03:41:17.169968+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":34747224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":903,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.448284+0000","last_change":"2024-09-16T03:41:04.232326+0000","last_active":"2024-09-16T03:41:18.448284+0000","last_peered":"2024-09-16T03:41:18.448284+0000","last_clean":"2024-09-16T03:41:18.448284+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:18.448284+0000","last_undegraded":"2024-09-16T03:41:18.448284+0000","last_fullsized":"2024-09-16T03:41:18.448284+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'284","last_scrub_stamp":"2024-09-16T03:41:04.232204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:41:04.232204+0000","objects_scrubbed":24,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:14.971430+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":169,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010817+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:02.010817+0000","last_peered":"2024-09-16T03:41:02.010817+0000","last_clean":"2024-09-16T03:41:02.010817+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:02.010817+0000","last_undegraded":"2024-09-16T03:41:02.010817+0000","last_fullsized":"2024-09-16T03:41:02.010817+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":447,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645907+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645907+0000","last_peered":"2024-09-16T03:41:01.645907+0000","last_clean":"2024-09-16T03:41:01.645907+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645907+0000","last_undegraded":"2024-09-16T03:41:01.645907+0000","last_fullsized":"2024-09-16T03:41:01.645907+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":210,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645096+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:01.645096+0000","last_peered":"2024-09-16T03:41:01.645096+0000","last_clean":"2024-09-16T03:41:01.645096+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:01.645096+0000","last_undegraded":"2024-09-16T03:41:01.645096+0000","last_fullsized":"2024-09-16T03:41:01.645096+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":466,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.817248+0000","last_change":"2024-09-16T03:41:12.744967+0000","last_active":"2024-09-16T03:41:18.817248+0000","last_peered":"2024-09-16T03:41:18.817248+0000","last_clean":"2024-09-16T03:41:18.817248+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:18.817248+0000","last_undegraded":"2024-09-16T03:41:18.817248+0000","last_fullsized":"2024-09-16T03:41:18.817248+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:12.744887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:41:12.744887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.656161+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":439,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010885+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:02.010885+0000","last_peered":"2024-09-16T03:41:02.010885+0000","last_clean":"2024-09-16T03:41:02.010885+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:02.010885+0000","last_undegraded":"2024-09-16T03:41:02.010885+0000","last_fullsized":"2024-09-16T03:41:02.010885+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":138560254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196968448,"data_stored":210367509,"data_compressed":12792038,"data_compressed_allocated":13672448,"data_compressed_original":27672576,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247865,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69544,"kb_used_data":34196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302296,"statfs":{"total":96636764160,"available":96565551104,"internally_reserved":0,"allocated":35016704,"data_stored":37479003,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247864,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":17388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":17805312,"data_stored":18652836,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280570,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28424,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343416,"statfs":{"total":96636764160,"available":96607657984,"internally_reserved":0,"allocated":1269760,"data_stored":865113,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46752,"kb_used_data":15396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325088,"statfs":{"total":96636764160,"available":96588890112,"internally_reserved":0,"allocated":15765504,"data_stored":16250926,"data_compressed":933071,"data_compressed_allocated":1011712,"data_compressed_original":2048000,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313274,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1253376,"data_stored":864734,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48484,"kb_used_data":8980,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94323356,"statfs":{"total":96636764160,"available":96587116544,"internally_reserved":0,"allocated":9195520,"data_stored":9309691,"data_compressed":532921,"data_compressed_allocated":614400,"data_compressed_original":1355776,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66924,"kb_used_data":31576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304916,"statfs":{"total":96636764160,"available":96568233984,"internally_reserved":0,"allocated":32333824,"data_stored":34041624,"data_compressed":2020760,"data_compressed_allocated":2199552,"data_compressed_original":4587520,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313274,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49360,"kb_used_data":18012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322480,"statfs":{"total":96636764160,"available":96586219520,"internally_reserved":0,"allocated":18444288,"data_stored":19689819,"data_compressed":1072888,"data_compressed_allocated":1187840,"data_compressed_original":3002368,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280569,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280568,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51092,"kb_used_data":15744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320748,"statfs":{"total":96636764160,"available":96584445952,"internally_reserved":0,"allocated":16121856,"data_stored":16630927,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46736,"kb_used_data":15548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325104,"statfs":{"total":96636764160,"available":96588906496,"internally_reserved":0,"allocated":15921152,"data_stored":16370014,"data_compressed":950167,"data_compressed_allocated":1036288,"data_compressed_original":2097152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1269760,"data_stored":865900,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313273,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32604,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1261568,"data_stored":864204,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46924,"kb_used_data":15736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324916,"statfs":{"total":96636764160,"available":96588713984,"internally_reserved":0,"allocated":16113664,"data_stored":16629164,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8691712,"data_stored":9371385,"data_compressed":512402,"data_compressed_allocated":561152,"data_compressed_original":1265664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:21.287 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean 2024-09-16T03:41:21.287 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "170'267", 'reported_seq': 918, 'reported_epoch': 230, 'state': 'active+remapped+backfilling', 'last_fresh': '2024-09-16T03:41:17.169968+0000', 'last_change': '2024-09-16T03:41:11.721909+0000', 'last_active': '2024-09-16T03:41:17.169968+0000', 'last_peered': '2024-09-16T03:41:17.169968+0000', 'last_clean': '2024-09-16T03:40:39.185069+0000', 'last_became_active': '2024-09-16T03:40:41.619774+0000', 'last_became_peered': '2024-09-16T03:40:41.619774+0000', 'last_unstale': '2024-09-16T03:41:17.169968+0000', 'last_undegraded': '2024-09-16T03:41:17.169968+0000', 'last_fullsized': '2024-09-16T03:41:17.169968+0000', 'mapping_epoch': 215, 'log_start': "170'263", 'ondisk_log_start': "170'263", 'created': 31, 'last_epoch_clean': 201, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "170'267", 'last_scrub_stamp': '2024-09-16T03:40:38.134247+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T03:37:03.913037+0000', 'last_clean_scrub_stamp': '2024-09-16T03:40:38.134247+0000', 'objects_scrubbed': 39, 'log_size': 4, 'log_dups_size': 274, 'ondisk_log_size': 4, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 406, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28703148, 'num_objects': 39, 'num_object_clones': 0, 'num_object_copies': 117, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 39, 'num_whiteouts': 0, 'num_read': 249, 'num_read_kb': 2401, 'num_write': 343, 'num_write_kb': 78301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 77, 'num_bytes_recovered': 34747224, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4, 3], 'acting': [6, 4, 11], 'avail_no_missing': ['6', '4', '11'], 'object_location_counts': [{'shards': '4,6,11', 'objects': 39}], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-16T03:41:21.288 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T03:41:21.659 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:21.660 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","health":{"status":"HEALTH_WARN","checks":{"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":284,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":232,"num_osds":16,"num_up_osds":16,"osd_up_since":1726457810,"num_in_osds":16,"osd_in_since":1726458014,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+remapped+backfilling","count":1}],"num_pgs":13,"num_pools":3,"num_objects":150,"data_bytes":103194545,"bytes_used":755552256,"bytes_avail":1545432674304,"bytes_total":1546188226560,"misplaced_objects":1,"misplaced_total":450,"misplaced_ratio":0.0022222222222222222,"recovering_objects_per_sec":4,"recovering_bytes_per_sec":2134338,"recovering_keys_per_sec":0,"num_objects_recovered":38,"num_bytes_recovered":17078700,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-16T03:36:37:310011+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T03:37:51.688536+0000","services":{}},"progress_events":{}} 2024-09-16T03:41:21.673 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-16T03:41:21.673 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:21.971 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:21.972 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:21.984 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-16T03:41:21.734236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":140147038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":746424,"kb_used_data":220488,"kb_used_omap":24,"kb_used_meta":523623,"kb_avail":1509203016,"statfs":{"total":1546188226560,"available":1545423888384,"internally_reserved":0,"allocated":225779712,"data_stored":234680693,"data_compressed":14882578,"data_compressed_allocated":16420864,"data_compressed_original":34492416,"omap_allocated":25519,"internal_metadata":536190033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-37,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":15909084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001962"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":644,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:02.010486+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:02.010486+0000","last_peered":"2024-09-16T03:41:02.010486+0000","last_clean":"2024-09-16T03:41:02.010486+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:02.010486+0000","last_undegraded":"2024-09-16T03:41:02.010486+0000","last_fullsized":"2024-09-16T03:41:02.010486+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":162,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.961126+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:01.961126+0000","last_peered":"2024-09-16T03:41:01.961126+0000","last_clean":"2024-09-16T03:41:01.961126+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:01.961126+0000","last_undegraded":"2024-09-16T03:41:01.961126+0000","last_fullsized":"2024-09-16T03:41:01.961126+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":981,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.910433+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:01.910433+0000","last_peered":"2024-09-16T03:41:01.910433+0000","last_clean":"2024-09-16T03:41:01.910433+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:01.910433+0000","last_undegraded":"2024-09-16T03:41:01.910433+0000","last_fullsized":"2024-09-16T03:41:01.910433+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":474,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.984348+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:18.984348+0000","last_peered":"2024-09-16T03:41:18.984348+0000","last_clean":"2024-09-16T03:41:18.984348+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:18.984348+0000","last_undegraded":"2024-09-16T03:41:18.984348+0000","last_fullsized":"2024-09-16T03:41:18.984348+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":918,"reported_epoch":230,"state":"active+remapped+backfilling","last_fresh":"2024-09-16T03:41:17.169968+0000","last_change":"2024-09-16T03:41:11.721909+0000","last_active":"2024-09-16T03:41:17.169968+0000","last_peered":"2024-09-16T03:41:17.169968+0000","last_clean":"2024-09-16T03:40:39.185069+0000","last_became_active":"2024-09-16T03:40:41.619774+0000","last_became_peered":"2024-09-16T03:40:41.619774+0000","last_unstale":"2024-09-16T03:41:17.169968+0000","last_undegraded":"2024-09-16T03:41:17.169968+0000","last_fullsized":"2024-09-16T03:41:17.169968+0000","mapping_epoch":215,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":34747224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,11],"avail_no_missing":["6","4","11"],"object_location_counts":[{"shards":"4,6,11","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645625+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:01.645625+0000","last_peered":"2024-09-16T03:41:01.645625+0000","last_clean":"2024-09-16T03:41:01.645625+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:01.645625+0000","last_undegraded":"2024-09-16T03:41:01.645625+0000","last_fullsized":"2024-09-16T03:41:01.645625+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":903,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.448284+0000","last_change":"2024-09-16T03:41:04.232326+0000","last_active":"2024-09-16T03:41:18.448284+0000","last_peered":"2024-09-16T03:41:18.448284+0000","last_clean":"2024-09-16T03:41:18.448284+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:18.448284+0000","last_undegraded":"2024-09-16T03:41:18.448284+0000","last_fullsized":"2024-09-16T03:41:18.448284+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'284","last_scrub_stamp":"2024-09-16T03:41:04.232204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:41:04.232204+0000","objects_scrubbed":24,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:14.971430+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":173,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.884186+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:18.884186+0000","last_peered":"2024-09-16T03:41:18.884186+0000","last_clean":"2024-09-16T03:41:18.884186+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:18.884186+0000","last_undegraded":"2024-09-16T03:41:18.884186+0000","last_fullsized":"2024-09-16T03:41:18.884186+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":879,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645631+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:01.645631+0000","last_peered":"2024-09-16T03:41:01.645631+0000","last_clean":"2024-09-16T03:41:01.645631+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:01.645631+0000","last_undegraded":"2024-09-16T03:41:01.645631+0000","last_fullsized":"2024-09-16T03:41:01.645631+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":447,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-16T03:41:01.645907+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:01.645907+0000","last_peered":"2024-09-16T03:41:01.645907+0000","last_clean":"2024-09-16T03:41:01.645907+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:01.645907+0000","last_undegraded":"2024-09-16T03:41:01.645907+0000","last_fullsized":"2024-09-16T03:41:01.645907+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":214,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.884800+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:18.884800+0000","last_peered":"2024-09-16T03:41:18.884800+0000","last_clean":"2024-09-16T03:41:18.884800+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:18.884800+0000","last_undegraded":"2024-09-16T03:41:18.884800+0000","last_fullsized":"2024-09-16T03:41:18.884800+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":466,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.817248+0000","last_change":"2024-09-16T03:41:12.744967+0000","last_active":"2024-09-16T03:41:18.817248+0000","last_peered":"2024-09-16T03:41:18.817248+0000","last_clean":"2024-09-16T03:41:18.817248+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:18.817248+0000","last_undegraded":"2024-09-16T03:41:18.817248+0000","last_fullsized":"2024-09-16T03:41:18.817248+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:12.744887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:41:12.744887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.656161+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":443,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.884231+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:18.884231+0000","last_peered":"2024-09-16T03:41:18.884231+0000","last_clean":"2024-09-16T03:41:18.884231+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:18.884231+0000","last_undegraded":"2024-09-16T03:41:18.884231+0000","last_fullsized":"2024-09-16T03:41:18.884231+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":138560254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204664832,"data_stored":218664648,"data_compressed":13350229,"data_compressed_allocated":14274560,"data_compressed_original":28897280,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51100,"kb_used_data":15752,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320740,"statfs":{"total":96636764160,"available":96584437760,"internally_reserved":0,"allocated":16130048,"data_stored":16631252,"data_compressed":1319863,"data_compressed_allocated":1429504,"data_compressed_original":2547712,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247865,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69544,"kb_used_data":34196,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94302296,"statfs":{"total":96636764160,"available":96565551104,"internally_reserved":0,"allocated":35016704,"data_stored":37479003,"data_compressed":2159164,"data_compressed_allocated":2375680,"data_compressed_original":5529600,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247864,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52736,"kb_used_data":17388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319104,"statfs":{"total":96636764160,"available":96582762496,"internally_reserved":0,"allocated":17805312,"data_stored":18652836,"data_compressed":1172171,"data_compressed_allocated":1306624,"data_compressed_original":2777088,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280570,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28424,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343416,"statfs":{"total":96636764160,"available":96607657984,"internally_reserved":0,"allocated":1269760,"data_stored":865113,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46752,"kb_used_data":15396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325088,"statfs":{"total":96636764160,"available":96588890112,"internally_reserved":0,"allocated":15765504,"data_stored":16250926,"data_compressed":933071,"data_compressed_allocated":1011712,"data_compressed_original":2048000,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313274,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1253376,"data_stored":864734,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56952,"kb_used_data":17252,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94314888,"statfs":{"total":96636764160,"available":96578445312,"internally_reserved":0,"allocated":17666048,"data_stored":18533913,"data_compressed":1155075,"data_compressed_allocated":1282048,"data_compressed_original":2727936,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66952,"kb_used_data":31412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304888,"statfs":{"total":96636764160,"available":96568205312,"internally_reserved":0,"allocated":32165888,"data_stored":33920131,"data_compressed":2003664,"data_compressed_allocated":2174976,"data_compressed_original":4538368,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313274,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49360,"kb_used_data":18012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322480,"statfs":{"total":96636764160,"available":96586219520,"internally_reserved":0,"allocated":18444288,"data_stored":19689819,"data_compressed":1072888,"data_compressed_allocated":1187840,"data_compressed_original":3002368,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280569,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280569,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51120,"kb_used_data":15580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320720,"statfs":{"total":96636764160,"available":96584417280,"internally_reserved":0,"allocated":15953920,"data_stored":16509434,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46764,"kb_used_data":15388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325076,"statfs":{"total":96636764160,"available":96588877824,"internally_reserved":0,"allocated":15757312,"data_stored":16248860,"data_compressed":933071,"data_compressed_allocated":1011712,"data_compressed_original":2048000,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1269760,"data_stored":865900,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313273,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32604,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1261568,"data_stored":864204,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46952,"kb_used_data":15580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324888,"statfs":{"total":96636764160,"available":96588685312,"internally_reserved":0,"allocated":15953920,"data_stored":16508370,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":33533952,"data_stored":35964981,"data_compressed":2026338,"data_compressed_allocated":2191360,"data_compressed_original":4730880,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:24.985 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:25.296 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:25.307 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-09-16T03:41:23.734630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":303,"num_bytes_recovered":140736862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":730312,"kb_used_data":204016,"kb_used_omap":24,"kb_used_meta":523623,"kb_avail":1509219128,"statfs":{"total":1546188226560,"available":1545440387072,"internally_reserved":0,"allocated":208912384,"data_stored":216650474,"data_compressed":13760697,"data_compressed_allocated":15183872,"data_compressed_original":31854592,"omap_allocated":25519,"internal_metadata":536190033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-38,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":16498908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001904"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":648,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.983853+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:18.983853+0000","last_peered":"2024-09-16T03:41:18.983853+0000","last_clean":"2024-09-16T03:41:18.983853+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:18.983853+0000","last_undegraded":"2024-09-16T03:41:18.983853+0000","last_fullsized":"2024-09-16T03:41:18.983853+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":166,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.983870+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:18.983870+0000","last_peered":"2024-09-16T03:41:18.983870+0000","last_clean":"2024-09-16T03:41:18.983870+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:18.983870+0000","last_undegraded":"2024-09-16T03:41:18.983870+0000","last_fullsized":"2024-09-16T03:41:18.983870+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":985,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.983870+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:18.983870+0000","last_peered":"2024-09-16T03:41:18.983870+0000","last_clean":"2024-09-16T03:41:18.983870+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:18.983870+0000","last_undegraded":"2024-09-16T03:41:18.983870+0000","last_fullsized":"2024-09-16T03:41:18.983870+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":474,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.984348+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:18.984348+0000","last_peered":"2024-09-16T03:41:18.984348+0000","last_clean":"2024-09-16T03:41:18.984348+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:18.984348+0000","last_undegraded":"2024-09-16T03:41:18.984348+0000","last_fullsized":"2024-09-16T03:41:18.984348+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":935,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:19.120848+0000","last_change":"2024-09-16T03:41:19.120744+0000","last_active":"2024-09-16T03:41:19.120848+0000","last_peered":"2024-09-16T03:41:19.120848+0000","last_clean":"2024-09-16T03:41:19.120848+0000","last_became_active":"2024-09-16T03:41:19.120312+0000","last_became_peered":"2024-09-16T03:41:19.120312+0000","last_unstale":"2024-09-16T03:41:19.120848+0000","last_undegraded":"2024-09-16T03:41:19.120848+0000","last_fullsized":"2024-09-16T03:41:19.120848+0000","mapping_epoch":231,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.358509+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":35337048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.884178+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:18.884178+0000","last_peered":"2024-09-16T03:41:18.884178+0000","last_clean":"2024-09-16T03:41:18.884178+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:18.884178+0000","last_undegraded":"2024-09-16T03:41:18.884178+0000","last_fullsized":"2024-09-16T03:41:18.884178+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":903,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.448284+0000","last_change":"2024-09-16T03:41:04.232326+0000","last_active":"2024-09-16T03:41:18.448284+0000","last_peered":"2024-09-16T03:41:18.448284+0000","last_clean":"2024-09-16T03:41:18.448284+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:18.448284+0000","last_undegraded":"2024-09-16T03:41:18.448284+0000","last_fullsized":"2024-09-16T03:41:18.448284+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'284","last_scrub_stamp":"2024-09-16T03:41:04.232204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:41:04.232204+0000","objects_scrubbed":24,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:14.971430+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":173,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.884186+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:18.884186+0000","last_peered":"2024-09-16T03:41:18.884186+0000","last_clean":"2024-09-16T03:41:18.884186+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:18.884186+0000","last_undegraded":"2024-09-16T03:41:18.884186+0000","last_fullsized":"2024-09-16T03:41:18.884186+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":883,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.884174+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:18.884174+0000","last_peered":"2024-09-16T03:41:18.884174+0000","last_clean":"2024-09-16T03:41:18.884174+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:18.884174+0000","last_undegraded":"2024-09-16T03:41:18.884174+0000","last_fullsized":"2024-09-16T03:41:18.884174+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":451,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.819087+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:18.819087+0000","last_peered":"2024-09-16T03:41:18.819087+0000","last_clean":"2024-09-16T03:41:18.819087+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:18.819087+0000","last_undegraded":"2024-09-16T03:41:18.819087+0000","last_fullsized":"2024-09-16T03:41:18.819087+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":214,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.884800+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:18.884800+0000","last_peered":"2024-09-16T03:41:18.884800+0000","last_clean":"2024-09-16T03:41:18.884800+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:18.884800+0000","last_undegraded":"2024-09-16T03:41:18.884800+0000","last_fullsized":"2024-09-16T03:41:18.884800+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":466,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.817248+0000","last_change":"2024-09-16T03:41:12.744967+0000","last_active":"2024-09-16T03:41:18.817248+0000","last_peered":"2024-09-16T03:41:18.817248+0000","last_clean":"2024-09-16T03:41:18.817248+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:18.817248+0000","last_undegraded":"2024-09-16T03:41:18.817248+0000","last_fullsized":"2024-09-16T03:41:18.817248+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:12.744887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:41:12.744887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.656161+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":443,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.884231+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:18.884231+0000","last_peered":"2024-09-16T03:41:18.884231+0000","last_clean":"2024-09-16T03:41:18.884231+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:18.884231+0000","last_undegraded":"2024-09-16T03:41:18.884231+0000","last_fullsized":"2024-09-16T03:41:18.884231+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":139150078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51128,"kb_used_data":15596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320712,"statfs":{"total":96636764160,"available":96584409088,"internally_reserved":0,"allocated":15970304,"data_stored":16510765,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247866,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53376,"kb_used_data":18032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94318464,"statfs":{"total":96636764160,"available":96582107136,"internally_reserved":0,"allocated":18464768,"data_stored":19689325,"data_compressed":1071475,"data_compressed_allocated":1187840,"data_compressed_original":2990080,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247865,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52764,"kb_used_data":17236,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319076,"statfs":{"total":96636764160,"available":96582733824,"internally_reserved":0,"allocated":17649664,"data_stored":18532782,"data_compressed":1155075,"data_compressed_allocated":1282048,"data_compressed_original":2727936,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280570,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28424,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343416,"statfs":{"total":96636764160,"available":96607657984,"internally_reserved":0,"allocated":1269760,"data_stored":865113,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46752,"kb_used_data":15396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325088,"statfs":{"total":96636764160,"available":96588890112,"internally_reserved":0,"allocated":15765504,"data_stored":16250926,"data_compressed":933071,"data_compressed_allocated":1011712,"data_compressed_original":2048000,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313274,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1253376,"data_stored":864734,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56952,"kb_used_data":17252,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94314888,"statfs":{"total":96636764160,"available":96578445312,"internally_reserved":0,"allocated":17666048,"data_stored":18533913,"data_compressed":1155075,"data_compressed_allocated":1282048,"data_compressed_original":2727936,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66952,"kb_used_data":31412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304888,"statfs":{"total":96636764160,"available":96568205312,"internally_reserved":0,"allocated":32165888,"data_stored":33920131,"data_compressed":2003664,"data_compressed_allocated":2174976,"data_compressed_original":4538368,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313274,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49360,"kb_used_data":18012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322480,"statfs":{"total":96636764160,"available":96586219520,"internally_reserved":0,"allocated":18444288,"data_stored":19689819,"data_compressed":1072888,"data_compressed_allocated":1187840,"data_compressed_original":3002368,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280569,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280569,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51120,"kb_used_data":15580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320720,"statfs":{"total":96636764160,"available":96584417280,"internally_reserved":0,"allocated":15953920,"data_stored":16509434,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46764,"kb_used_data":15388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325076,"statfs":{"total":96636764160,"available":96588877824,"internally_reserved":0,"allocated":15757312,"data_stored":16248860,"data_compressed":933071,"data_compressed_allocated":1011712,"data_compressed_original":2048000,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1269760,"data_stored":865900,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313273,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32604,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1261568,"data_stored":864204,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46952,"kb_used_data":15580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324888,"statfs":{"total":96636764160,"available":96588685312,"internally_reserved":0,"allocated":15953920,"data_stored":16508370,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:25.308 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T03:41:25.309 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:25.605 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:25.606 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:25.617 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-09-16T03:41:23.734630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":303,"num_bytes_recovered":140736862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":40,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":730312,"kb_used_data":204016,"kb_used_omap":24,"kb_used_meta":523623,"kb_avail":1509219128,"statfs":{"total":1546188226560,"available":1545440387072,"internally_reserved":0,"allocated":208912384,"data_stored":216650474,"data_compressed":13760697,"data_compressed_allocated":15183872,"data_compressed_original":31854592,"omap_allocated":25519,"internal_metadata":536190033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-38,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":16498908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001904"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":648,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.983853+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:18.983853+0000","last_peered":"2024-09-16T03:41:18.983853+0000","last_clean":"2024-09-16T03:41:18.983853+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:18.983853+0000","last_undegraded":"2024-09-16T03:41:18.983853+0000","last_fullsized":"2024-09-16T03:41:18.983853+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":166,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.983870+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:18.983870+0000","last_peered":"2024-09-16T03:41:18.983870+0000","last_clean":"2024-09-16T03:41:18.983870+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:18.983870+0000","last_undegraded":"2024-09-16T03:41:18.983870+0000","last_fullsized":"2024-09-16T03:41:18.983870+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":985,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.983870+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:18.983870+0000","last_peered":"2024-09-16T03:41:18.983870+0000","last_clean":"2024-09-16T03:41:18.983870+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:18.983870+0000","last_undegraded":"2024-09-16T03:41:18.983870+0000","last_fullsized":"2024-09-16T03:41:18.983870+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":474,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.984348+0000","last_change":"2024-09-16T03:40:24.388456+0000","last_active":"2024-09-16T03:41:18.984348+0000","last_peered":"2024-09-16T03:41:18.984348+0000","last_clean":"2024-09-16T03:41:18.984348+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:18.984348+0000","last_undegraded":"2024-09-16T03:41:18.984348+0000","last_fullsized":"2024-09-16T03:41:18.984348+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:24.388390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.388390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:30.673577+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":935,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:19.120848+0000","last_change":"2024-09-16T03:41:19.120744+0000","last_active":"2024-09-16T03:41:19.120848+0000","last_peered":"2024-09-16T03:41:19.120848+0000","last_clean":"2024-09-16T03:41:19.120848+0000","last_became_active":"2024-09-16T03:41:19.120312+0000","last_became_peered":"2024-09-16T03:41:19.120312+0000","last_unstale":"2024-09-16T03:41:19.120848+0000","last_undegraded":"2024-09-16T03:41:19.120848+0000","last_fullsized":"2024-09-16T03:41:19.120848+0000","mapping_epoch":231,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.358509+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":35337048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.884178+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:18.884178+0000","last_peered":"2024-09-16T03:41:18.884178+0000","last_clean":"2024-09-16T03:41:18.884178+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:18.884178+0000","last_undegraded":"2024-09-16T03:41:18.884178+0000","last_fullsized":"2024-09-16T03:41:18.884178+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":903,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.448284+0000","last_change":"2024-09-16T03:41:04.232326+0000","last_active":"2024-09-16T03:41:18.448284+0000","last_peered":"2024-09-16T03:41:18.448284+0000","last_clean":"2024-09-16T03:41:18.448284+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:18.448284+0000","last_undegraded":"2024-09-16T03:41:18.448284+0000","last_fullsized":"2024-09-16T03:41:18.448284+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'284","last_scrub_stamp":"2024-09-16T03:41:04.232204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:41:04.232204+0000","objects_scrubbed":24,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:14.971430+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":173,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.884186+0000","last_change":"2024-09-16T03:40:41.323887+0000","last_active":"2024-09-16T03:41:18.884186+0000","last_peered":"2024-09-16T03:41:18.884186+0000","last_clean":"2024-09-16T03:41:18.884186+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:18.884186+0000","last_undegraded":"2024-09-16T03:41:18.884186+0000","last_fullsized":"2024-09-16T03:41:18.884186+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.332703+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":883,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.884174+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:18.884174+0000","last_peered":"2024-09-16T03:41:18.884174+0000","last_clean":"2024-09-16T03:41:18.884174+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:18.884174+0000","last_undegraded":"2024-09-16T03:41:18.884174+0000","last_fullsized":"2024-09-16T03:41:18.884174+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":451,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.819087+0000","last_change":"2024-09-16T03:40:35.127216+0000","last_active":"2024-09-16T03:41:18.819087+0000","last_peered":"2024-09-16T03:41:18.819087+0000","last_clean":"2024-09-16T03:41:18.819087+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:18.819087+0000","last_undegraded":"2024-09-16T03:41:18.819087+0000","last_fullsized":"2024-09-16T03:41:18.819087+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:35.127139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:35.127139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:59.069197+0000","scrub_duration":305,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":214,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.884800+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:18.884800+0000","last_peered":"2024-09-16T03:41:18.884800+0000","last_clean":"2024-09-16T03:41:18.884800+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:18.884800+0000","last_undegraded":"2024-09-16T03:41:18.884800+0000","last_fullsized":"2024-09-16T03:41:18.884800+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":466,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.817248+0000","last_change":"2024-09-16T03:41:12.744967+0000","last_active":"2024-09-16T03:41:18.817248+0000","last_peered":"2024-09-16T03:41:18.817248+0000","last_clean":"2024-09-16T03:41:18.817248+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:18.817248+0000","last_undegraded":"2024-09-16T03:41:18.817248+0000","last_fullsized":"2024-09-16T03:41:18.817248+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:12.744887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:41:12.744887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.656161+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":443,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-16T03:41:18.884231+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:18.884231+0000","last_peered":"2024-09-16T03:41:18.884231+0000","last_clean":"2024-09-16T03:41:18.884231+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:18.884231+0000","last_undegraded":"2024-09-16T03:41:18.884231+0000","last_fullsized":"2024-09-16T03:41:18.884231+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":139150078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247865,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51128,"kb_used_data":15596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320712,"statfs":{"total":96636764160,"available":96584409088,"internally_reserved":0,"allocated":15970304,"data_stored":16510765,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247866,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53376,"kb_used_data":18032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94318464,"statfs":{"total":96636764160,"available":96582107136,"internally_reserved":0,"allocated":18464768,"data_stored":19689325,"data_compressed":1071475,"data_compressed_allocated":1187840,"data_compressed_original":2990080,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247865,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52764,"kb_used_data":17236,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319076,"statfs":{"total":96636764160,"available":96582733824,"internally_reserved":0,"allocated":17649664,"data_stored":18532782,"data_compressed":1155075,"data_compressed_allocated":1282048,"data_compressed_original":2727936,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280570,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49400,"kb_used_data":18212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322440,"statfs":{"total":96636764160,"available":96586178560,"internally_reserved":0,"allocated":18649088,"data_stored":19811928,"data_compressed":1088571,"data_compressed_allocated":1212416,"data_compressed_original":3039232,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28424,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343416,"statfs":{"total":96636764160,"available":96607657984,"internally_reserved":0,"allocated":1269760,"data_stored":865113,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46752,"kb_used_data":15396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325088,"statfs":{"total":96636764160,"available":96588890112,"internally_reserved":0,"allocated":15765504,"data_stored":16250926,"data_compressed":933071,"data_compressed_allocated":1011712,"data_compressed_original":2048000,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313274,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1253376,"data_stored":864734,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56952,"kb_used_data":17252,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94314888,"statfs":{"total":96636764160,"available":96578445312,"internally_reserved":0,"allocated":17666048,"data_stored":18533913,"data_compressed":1155075,"data_compressed_allocated":1282048,"data_compressed_original":2727936,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66952,"kb_used_data":31412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304888,"statfs":{"total":96636764160,"available":96568205312,"internally_reserved":0,"allocated":32165888,"data_stored":33920131,"data_compressed":2003664,"data_compressed_allocated":2174976,"data_compressed_original":4538368,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313274,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49360,"kb_used_data":18012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322480,"statfs":{"total":96636764160,"available":96586219520,"internally_reserved":0,"allocated":18444288,"data_stored":19689819,"data_compressed":1072888,"data_compressed_allocated":1187840,"data_compressed_original":3002368,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280569,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1417216,"data_stored":984270,"data_compressed":101578,"data_compressed_allocated":143360,"data_compressed_original":286720,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280569,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51120,"kb_used_data":15580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320720,"statfs":{"total":96636764160,"available":96584417280,"internally_reserved":0,"allocated":15953920,"data_stored":16509434,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280569,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46764,"kb_used_data":15388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325076,"statfs":{"total":96636764160,"available":96588877824,"internally_reserved":0,"allocated":15757312,"data_stored":16248860,"data_compressed":933071,"data_compressed_allocated":1011712,"data_compressed_original":2048000,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1269760,"data_stored":865900,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313273,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32604,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339236,"statfs":{"total":96636764160,"available":96603377664,"internally_reserved":0,"allocated":1261568,"data_stored":864204,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46952,"kb_used_data":15580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324888,"statfs":{"total":96636764160,"available":96588685312,"internally_reserved":0,"allocated":15953920,"data_stored":16508370,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:25.619 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:25.920 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":232,"fsid":"b0e1ff9d-cd24-45dc-a2b1-cc545b2ad2b2","created":"2024-09-16T03:36:37.310421+0000","modified":"2024-09-16T03:41:18.810635+0000","last_up_change":"2024-09-16T03:36:50.058111+0000","last_in_change":"2024-09-16T03:40:14.459006+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:36:49.696307+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.789999961853027,"score_stable":15.789999961853027,"optimal_score":0.18999999761581421,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:36:51.904455+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":5.9699997901916504,"score_stable":5.9699997901916504,"optimal_score":0.87999999523162842,"raw_score_acting":5.25,"raw_score_stable":5.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"low_tier","create_time":"2024-09-16T03:37:02.999795+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":3.9900000095367432,"score_stable":3.9900000095367432,"optimal_score":0.68999999761581421,"raw_score_acting":2.75,"raw_score_stable":2.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bdbe961a-78d3-4f7d-88ed-ce24475d49e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":225,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6813","nonce":3924351635}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6814","nonce":3924351635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6816","nonce":3924351635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6815","nonce":3924351635}]},"public_addr":"172.21.15.64:6813/3924351635","cluster_addr":"172.21.15.64:6814/3924351635","heartbeat_back_addr":"172.21.15.64:6816/3924351635","heartbeat_front_addr":"172.21.15.64:6815/3924351635","state":["exists","up"]},{"osd":1,"uuid":"154c8ddd-7a3d-402c-b9d6-28538f4e6bc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":199,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6804","nonce":624977804}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6805","nonce":624977804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6807","nonce":624977804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6806","nonce":624977804}]},"public_addr":"172.21.15.112:6804/624977804","cluster_addr":"172.21.15.112:6805/624977804","heartbeat_back_addr":"172.21.15.112:6807/624977804","heartbeat_front_addr":"172.21.15.112:6806/624977804","state":["exists","up"]},{"osd":2,"uuid":"2a06de4e-0835-4a42-8c8a-49fb55aab566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6812","nonce":2804648086}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6813","nonce":2804648086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6815","nonce":2804648086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6814","nonce":2804648086}]},"public_addr":"172.21.15.155:6812/2804648086","cluster_addr":"172.21.15.155:6813/2804648086","heartbeat_back_addr":"172.21.15.155:6815/2804648086","heartbeat_front_addr":"172.21.15.155:6814/2804648086","state":["exists","up"]},{"osd":3,"uuid":"8a87f9f0-c380-4f38-bbd2-06b2def4fd2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6800","nonce":1935597988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6801","nonce":1935597988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6803","nonce":1935597988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6802","nonce":1935597988}]},"public_addr":"172.21.15.157:6800/1935597988","cluster_addr":"172.21.15.157:6801/1935597988","heartbeat_back_addr":"172.21.15.157:6803/1935597988","heartbeat_front_addr":"172.21.15.157:6802/1935597988","state":["exists","up"]},{"osd":4,"uuid":"5ac53318-12e8-496c-a87f-ba5de79c52aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":171,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6809","nonce":2675475248}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6810","nonce":2675475248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6812","nonce":2675475248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6811","nonce":2675475248}]},"public_addr":"172.21.15.64:6809/2675475248","cluster_addr":"172.21.15.64:6810/2675475248","heartbeat_back_addr":"172.21.15.64:6812/2675475248","heartbeat_front_addr":"172.21.15.64:6811/2675475248","state":["exists","up"]},{"osd":5,"uuid":"675c81a4-2314-463e-82a7-0060735c78fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6800","nonce":499905276}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6801","nonce":499905276}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6803","nonce":499905276}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6802","nonce":499905276}]},"public_addr":"172.21.15.112:6800/499905276","cluster_addr":"172.21.15.112:6801/499905276","heartbeat_back_addr":"172.21.15.112:6803/499905276","heartbeat_front_addr":"172.21.15.112:6802/499905276","state":["exists","up"]},{"osd":6,"uuid":"b18962a7-3b23-4e50-bf31-1d0629557423","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6808","nonce":2839983406}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6809","nonce":2839983406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6811","nonce":2839983406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6810","nonce":2839983406}]},"public_addr":"172.21.15.155:6808/2839983406","cluster_addr":"172.21.15.155:6809/2839983406","heartbeat_back_addr":"172.21.15.155:6811/2839983406","heartbeat_front_addr":"172.21.15.155:6810/2839983406","state":["exists","up"]},{"osd":7,"uuid":"9706c1e8-dac0-4e16-8bc5-7cacc6f13870","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":173,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6808","nonce":3021982871}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6809","nonce":3021982871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6811","nonce":3021982871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6810","nonce":3021982871}]},"public_addr":"172.21.15.157:6808/3021982871","cluster_addr":"172.21.15.157:6809/3021982871","heartbeat_back_addr":"172.21.15.157:6811/3021982871","heartbeat_front_addr":"172.21.15.157:6810/3021982871","state":["exists","up"]},{"osd":8,"uuid":"fafd06b8-c005-4471-a128-9dd51580c0f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6805","nonce":3609017627}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6806","nonce":3609017627}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6808","nonce":3609017627}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6807","nonce":3609017627}]},"public_addr":"172.21.15.64:6805/3609017627","cluster_addr":"172.21.15.64:6806/3609017627","heartbeat_back_addr":"172.21.15.64:6808/3609017627","heartbeat_front_addr":"172.21.15.64:6807/3609017627","state":["exists","up"]},{"osd":9,"uuid":"79e38e57-52c2-416f-9b90-75621f06d3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6812","nonce":701981815}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6813","nonce":701981815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6815","nonce":701981815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6814","nonce":701981815}]},"public_addr":"172.21.15.112:6812/701981815","cluster_addr":"172.21.15.112:6813/701981815","heartbeat_back_addr":"172.21.15.112:6815/701981815","heartbeat_front_addr":"172.21.15.112:6814/701981815","state":["exists","up"]},{"osd":10,"uuid":"8fcb06e3-d21f-4bbb-9f25-b3e3b4da1a10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6804","nonce":4138619523}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6805","nonce":4138619523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6807","nonce":4138619523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6806","nonce":4138619523}]},"public_addr":"172.21.15.155:6804/4138619523","cluster_addr":"172.21.15.155:6805/4138619523","heartbeat_back_addr":"172.21.15.155:6807/4138619523","heartbeat_front_addr":"172.21.15.155:6806/4138619523","state":["exists","up"]},{"osd":11,"uuid":"7a2a169a-ce94-4091-a792-27d0ecf9a7fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":224,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6804","nonce":3607367937}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6805","nonce":3607367937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6807","nonce":3607367937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6806","nonce":3607367937}]},"public_addr":"172.21.15.157:6804/3607367937","cluster_addr":"172.21.15.157:6805/3607367937","heartbeat_back_addr":"172.21.15.157:6807/3607367937","heartbeat_front_addr":"172.21.15.157:6806/3607367937","state":["exists","up"]},{"osd":12,"uuid":"7a280cda-b740-40c8-9fae-1a1ff11651f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6801","nonce":2984461266}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6802","nonce":2984461266}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6804","nonce":2984461266}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.64:6803","nonce":2984461266}]},"public_addr":"172.21.15.64:6801/2984461266","cluster_addr":"172.21.15.64:6802/2984461266","heartbeat_back_addr":"172.21.15.64:6804/2984461266","heartbeat_front_addr":"172.21.15.64:6803/2984461266","state":["exists","up"]},{"osd":13,"uuid":"ce3a3148-cd90-422c-a106-045de89ab045","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6808","nonce":1616314527}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6809","nonce":1616314527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6811","nonce":1616314527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.112:6810","nonce":1616314527}]},"public_addr":"172.21.15.112:6808/1616314527","cluster_addr":"172.21.15.112:6809/1616314527","heartbeat_back_addr":"172.21.15.112:6811/1616314527","heartbeat_front_addr":"172.21.15.112:6810/1616314527","state":["exists","up"]},{"osd":14,"uuid":"461c2d79-f4e7-4b21-899c-50d4a6cf12f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":188,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6800","nonce":4203868121}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6801","nonce":4203868121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6803","nonce":4203868121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.155:6802","nonce":4203868121}]},"public_addr":"172.21.15.155:6800/4203868121","cluster_addr":"172.21.15.155:6801/4203868121","heartbeat_back_addr":"172.21.15.155:6803/4203868121","heartbeat_front_addr":"172.21.15.155:6802/4203868121","state":["exists","up"]},{"osd":15,"uuid":"9a8c0bbe-9389-4b2c-95e1-34d3e66461e9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":226,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6812","nonce":181354529}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6813","nonce":181354529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6815","nonce":181354529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.157:6814","nonce":181354529}]},"public_addr":"172.21.15.157:6812/181354529","cluster_addr":"172.21.15.157:6813/181354529","heartbeat_back_addr":"172.21.15.157:6815/181354529","heartbeat_front_addr":"172.21.15.157:6814/181354529","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:41.866144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:15.766345+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:56.870843+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:28.164905+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:36.498999+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:11.403494+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:31.536918+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:13.198340+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:33.844989+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:07.784177+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:54.633717+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:29.586378+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:46.813546+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:27.906719+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:54.122447+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:40:49.074735+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:41:26.838 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:41:26.837+0000 7f3a21722640 -1 osd.6 232 handle_osd_map injecting map crc failure 2024-09-16T03:41:26.934 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T03:41:26.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:27.074 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:27.075 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:27.075 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:27.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T03:41:27.385 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T03:41:27.398 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T03:41:27.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:27.526 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:27.526 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:27.526 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:27.536 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T03:41:27.549 INFO:tasks.ceph.osd.0.smithi064.stderr:2024-09-16T03:41:27.548+0000 7f0ae754b640 -1 osd.0 232 handle_osd_map injecting map crc failure 2024-09-16T03:41:27.550 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:41:27.548+0000 7fa3886e6640 -1 osd.10 232 handle_osd_map injecting map crc failure 2024-09-16T03:41:27.837 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T03:41:27.851 INFO:tasks.ceph:Scrubbing osd.8 2024-09-16T03:41:27.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:28.018 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:28.018 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:28.018 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:28.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-16T03:41:28.327 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 8 to deep-scrub 2024-09-16T03:41:28.340 INFO:tasks.ceph:Scrubbing osd.12 2024-09-16T03:41:28.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:28.467 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:28.468 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:28.468 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:28.478 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-09-16T03:41:28.780 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 12 to deep-scrub 2024-09-16T03:41:28.793 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T03:41:28.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:28.920 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:28.920 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:28.920 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:28.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T03:41:29.229 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T03:41:29.242 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T03:41:29.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:29.371 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:29.371 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:29.371 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:29.382 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T03:41:29.681 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T03:41:29.694 INFO:tasks.ceph:Scrubbing osd.9 2024-09-16T03:41:29.694 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:29.821 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:29.821 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:29.821 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:29.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-16T03:41:30.130 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 9 to deep-scrub 2024-09-16T03:41:30.143 INFO:tasks.ceph:Scrubbing osd.13 2024-09-16T03:41:30.143 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:30.148 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:41:30.147+0000 7eff7884b640 -1 osd.3 233 handle_osd_map injecting map crc failure 2024-09-16T03:41:30.156 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:41:30.154+0000 7f8be56cd640 -1 osd.7 233 handle_osd_map injecting map crc failure 2024-09-16T03:41:30.271 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:30.271 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:30.271 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:30.282 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-09-16T03:41:30.583 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 13 to deep-scrub 2024-09-16T03:41:30.596 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T03:41:30.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:30.723 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:30.723 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:30.723 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:30.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T03:41:30.864 INFO:tasks.ceph.osd.11.smithi157.stderr:2024-09-16T03:41:30.863+0000 7ff266366640 -1 osd.11 234 handle_osd_map injecting map crc failure 2024-09-16T03:41:30.985 INFO:tasks.ceph.osd.6.smithi155.stderr:2024-09-16T03:41:30.984+0000 7f3a1f71e640 -1 osd.6 234 handle_osd_map injecting map crc failure 2024-09-16T03:41:31.038 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T03:41:31.051 INFO:tasks.ceph:Scrubbing osd.6 2024-09-16T03:41:31.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:31.179 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:31.179 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:31.179 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:31.189 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-16T03:41:31.492 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 6 to deep-scrub 2024-09-16T03:41:31.506 INFO:tasks.ceph:Scrubbing osd.10 2024-09-16T03:41:31.506 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:31.579 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:41:31.577+0000 7f49419d0640 -1 osd.4 235 handle_osd_map injecting map crc failure 2024-09-16T03:41:31.586 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:41:31.584+0000 7fedb5739640 -1 osd.9 235 handle_osd_map injecting map crc failure 2024-09-16T03:41:31.645 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:31.645 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:31.645 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:31.664 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-16T03:41:31.763 INFO:tasks.ceph.osd.8.smithi064.stderr:2024-09-16T03:41:31.744+0000 7f4212d00640 -1 osd.8 235 handle_osd_map injecting map crc failure 2024-09-16T03:41:31.848 INFO:tasks.ceph.osd.13.smithi112.stderr:2024-09-16T03:41:31.846+0000 7efeef6d5640 -1 osd.13 235 handle_osd_map injecting map crc failure 2024-09-16T03:41:31.967 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 10 to deep-scrub 2024-09-16T03:41:31.980 INFO:tasks.ceph:Scrubbing osd.14 2024-09-16T03:41:31.981 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:32.109 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:32.109 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:32.109 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:32.119 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-09-16T03:41:32.405 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-09-16T03:41:32.402+0000 7f8be76d1640 -1 osd.7 236 handle_osd_map injecting map crc failure 2024-09-16T03:41:32.419 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 14 to deep-scrub 2024-09-16T03:41:32.433 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T03:41:32.433 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:32.562 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:32.562 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:32.562 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:32.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T03:41:32.609 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:41:32.606+0000 7eff7884b640 -1 osd.3 236 handle_osd_map injecting map crc failure 2024-09-16T03:41:32.874 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T03:41:32.887 INFO:tasks.ceph:Scrubbing osd.7 2024-09-16T03:41:32.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:33.017 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:33.017 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:33.017 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:33.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-16T03:41:33.328 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 7 to deep-scrub 2024-09-16T03:41:33.341 INFO:tasks.ceph:Scrubbing osd.11 2024-09-16T03:41:33.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:33.469 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:33.469 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:33.470 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:33.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-16T03:41:33.780 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 11 to deep-scrub 2024-09-16T03:41:33.793 INFO:tasks.ceph:Scrubbing osd.15 2024-09-16T03:41:33.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-09-16T03:41:33.922 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-16T03:41:33.923 INFO:teuthology.orchestra.run.smithi064.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T03:41:33.923 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-16T03:41:33.933 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-09-16T03:41:34.236 INFO:teuthology.orchestra.run.smithi064.stderr:instructed osd(s) 15 to deep-scrub 2024-09-16T03:41:34.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:34.546 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:34.547 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:34.558 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-09-16T03:41:33.736412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":303,"num_bytes_recovered":140736862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":727948,"kb_used_data":203972,"kb_used_omap":24,"kb_used_meta":523623,"kb_avail":1509221492,"statfs":{"total":1546188226560,"available":1545442807808,"internally_reserved":0,"allocated":208867328,"data_stored":216592817,"data_compressed":13726505,"data_compressed_allocated":15134720,"data_compressed_original":31756288,"omap_allocated":25519,"internal_metadata":536190033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.880117"},"pg_stats":[{"pgid":"1.0","version":"30'112","reported_seq":656,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T03:41:31.848825+0000","last_change":"2024-09-16T03:40:28.582089+0000","last_active":"2024-09-16T03:41:31.848825+0000","last_peered":"2024-09-16T03:41:31.848825+0000","last_clean":"2024-09-16T03:41:31.848825+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:31.848825+0000","last_undegraded":"2024-09-16T03:41:31.848825+0000","last_fullsized":"2024-09-16T03:41:31.848825+0000","mapping_epoch":192,"log_start":"30'109","ondisk_log_start":"30'109","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"30'112","last_scrub_stamp":"2024-09-16T03:40:28.581990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:50.058111+0000","last_clean_scrub_stamp":"2024-09-16T03:40:28.581990+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":109,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.255971+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":174,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T03:41:31.848890+0000","last_change":"2024-09-16T03:40:45.759836+0000","last_active":"2024-09-16T03:41:31.848890+0000","last_peered":"2024-09-16T03:41:31.848890+0000","last_clean":"2024-09-16T03:41:31.848890+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:31.848890+0000","last_undegraded":"2024-09-16T03:41:31.848890+0000","last_fullsized":"2024-09-16T03:41:31.848890+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-16T03:40:24.423252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:24.423252+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:46.548341+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"169'318","reported_seq":993,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T03:41:31.848870+0000","last_change":"2024-09-16T03:40:59.901673+0000","last_active":"2024-09-16T03:41:31.848870+0000","last_peered":"2024-09-16T03:41:31.848870+0000","last_clean":"2024-09-16T03:41:31.848870+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:31.848870+0000","last_undegraded":"2024-09-16T03:41:31.848870+0000","last_fullsized":"2024-09-16T03:41:31.848870+0000","mapping_epoch":224,"log_start":"168'315","ondisk_log_start":"168'315","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"169'318","last_scrub_stamp":"2024-09-16T03:40:59.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.901560+0000","objects_scrubbed":44,"log_size":3,"log_dups_size":324,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:05.391036+0000","scrub_duration":507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":484,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-16T03:41:29.267840+0000","last_change":"2024-09-16T03:41:29.267840+0000","last_active":"2024-09-16T03:41:29.267840+0000","last_peered":"2024-09-16T03:41:29.267840+0000","last_clean":"2024-09-16T03:41:29.267840+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:29.267840+0000","last_undegraded":"2024-09-16T03:41:29.267840+0000","last_fullsized":"2024-09-16T03:41:29.267840+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:29.267751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:29.267751+0000","last_clean_scrub_stamp":"2024-09-16T03:41:29.267751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.409746+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"170'267","reported_seq":943,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T03:41:31.746644+0000","last_change":"2024-09-16T03:41:19.120744+0000","last_active":"2024-09-16T03:41:31.746644+0000","last_peered":"2024-09-16T03:41:31.746644+0000","last_clean":"2024-09-16T03:41:31.746644+0000","last_became_active":"2024-09-16T03:41:19.120312+0000","last_became_peered":"2024-09-16T03:41:19.120312+0000","last_unstale":"2024-09-16T03:41:31.746644+0000","last_undegraded":"2024-09-16T03:41:31.746644+0000","last_fullsized":"2024-09-16T03:41:31.746644+0000","mapping_epoch":231,"log_start":"170'263","ondisk_log_start":"170'263","created":31,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'267","last_scrub_stamp":"2024-09-16T03:40:38.134247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:38.134247+0000","objects_scrubbed":39,"log_size":4,"log_dups_size":274,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:41:47.358509+0000","scrub_duration":406,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":35337048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":50,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T03:41:31.849311+0000","last_change":"2024-09-16T03:40:59.229980+0000","last_active":"2024-09-16T03:41:31.849311+0000","last_peered":"2024-09-16T03:41:31.849311+0000","last_clean":"2024-09-16T03:41:31.849311+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:31.849311+0000","last_undegraded":"2024-09-16T03:41:31.849311+0000","last_fullsized":"2024-09-16T03:41:31.849311+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:59.229884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:59.229884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:11.853721+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"168'284","reported_seq":907,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-16T03:41:27.049450+0000","last_change":"2024-09-16T03:41:04.232326+0000","last_active":"2024-09-16T03:41:27.049450+0000","last_peered":"2024-09-16T03:41:27.049450+0000","last_clean":"2024-09-16T03:41:27.049450+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:27.049450+0000","last_undegraded":"2024-09-16T03:41:27.049450+0000","last_fullsized":"2024-09-16T03:41:27.049450+0000","mapping_epoch":199,"log_start":"165'281","ondisk_log_start":"165'281","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'284","last_scrub_stamp":"2024-09-16T03:41:04.232204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:41:04.232204+0000","objects_scrubbed":24,"log_size":3,"log_dups_size":302,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:14.971430+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":179,"reported_epoch":233,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T03:41:30.007694+0000","last_change":"2024-09-16T03:41:30.007493+0000","last_active":"2024-09-16T03:41:30.007694+0000","last_peered":"2024-09-16T03:41:30.007694+0000","last_clean":"2024-09-16T03:41:30.007694+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:30.007694+0000","last_undegraded":"2024-09-16T03:41:30.007694+0000","last_fullsized":"2024-09-16T03:41:30.007694+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T03:40:41.323782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:41.323782+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"172'341","reported_seq":891,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T03:41:31.849305+0000","last_change":"2024-09-16T03:40:55.942604+0000","last_active":"2024-09-16T03:41:31.849305+0000","last_peered":"2024-09-16T03:41:31.849305+0000","last_clean":"2024-09-16T03:41:31.849305+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:31.849305+0000","last_undegraded":"2024-09-16T03:41:31.849305+0000","last_fullsized":"2024-09-16T03:41:31.849305+0000","mapping_epoch":226,"log_start":"168'339","ondisk_log_start":"168'339","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'341","last_scrub_stamp":"2024-09-16T03:40:55.942490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:37:03.913037+0000","last_clean_scrub_stamp":"2024-09-16T03:40:55.942490+0000","objects_scrubbed":39,"log_size":2,"log_dups_size":351,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:08.438758+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":467,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-16T03:41:31.852227+0000","last_change":"2024-09-16T03:41:31.852227+0000","last_active":"2024-09-16T03:41:31.852227+0000","last_peered":"2024-09-16T03:41:31.852227+0000","last_clean":"2024-09-16T03:41:31.852227+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:31.852227+0000","last_undegraded":"2024-09-16T03:41:31.852227+0000","last_fullsized":"2024-09-16T03:41:31.852227+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:31.852140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:31.852140+0000","last_clean_scrub_stamp":"2024-09-16T03:41:31.852140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:41.976166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":218,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-16T03:41:30.148837+0000","last_change":"2024-09-16T03:40:52.997921+0000","last_active":"2024-09-16T03:41:30.148837+0000","last_peered":"2024-09-16T03:41:30.148837+0000","last_clean":"2024-09-16T03:41:30.148837+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:30.148837+0000","last_undegraded":"2024-09-16T03:41:30.148837+0000","last_fullsized":"2024-09-16T03:41:30.148837+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:52.997837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:52.997837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:13.584880+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":468,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-16T03:41:27.550675+0000","last_change":"2024-09-16T03:41:12.744967+0000","last_active":"2024-09-16T03:41:27.550675+0000","last_peered":"2024-09-16T03:41:27.550675+0000","last_clean":"2024-09-16T03:41:27.550675+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:27.550675+0000","last_undegraded":"2024-09-16T03:41:27.550675+0000","last_fullsized":"2024-09-16T03:41:27.550675+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:12.744887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:41:12.744887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.656161+0000","scrub_duration":302,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":445,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-16T03:41:27.447424+0000","last_change":"2024-09-16T03:40:32.512943+0000","last_active":"2024-09-16T03:41:27.447424+0000","last_peered":"2024-09-16T03:41:27.447424+0000","last_clean":"2024-09-16T03:41:27.447424+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:27.447424+0000","last_undegraded":"2024-09-16T03:41:27.447424+0000","last_fullsized":"2024-09-16T03:41:27.447424+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:40:32.512789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:36:52.069777+0000","last_clean_scrub_stamp":"2024-09-16T03:40:32.512789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:00.423087+0000","scrub_duration":504,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":139150078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247867,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50992,"kb_used_data":15644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320848,"statfs":{"total":96636764160,"available":96584548352,"internally_reserved":0,"allocated":16019456,"data_stored":16541423,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53416,"kb_used_data":18064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94318424,"statfs":{"total":96636764160,"available":96582066176,"internally_reserved":0,"allocated":18497536,"data_stored":19712062,"data_compressed":1071475,"data_compressed_allocated":1187840,"data_compressed_original":2990080,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52632,"kb_used_data":17276,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319208,"statfs":{"total":96636764160,"available":96582868992,"internally_reserved":0,"allocated":17690624,"data_stored":18562852,"data_compressed":1155075,"data_compressed_allocated":1282048,"data_compressed_original":2727936,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280572,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49244,"kb_used_data":18056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322596,"statfs":{"total":96636764160,"available":96586338304,"internally_reserved":0,"allocated":18489344,"data_stored":19697818,"data_compressed":1071475,"data_compressed_allocated":1187840,"data_compressed_original":2990080,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280572,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28280,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343560,"statfs":{"total":96636764160,"available":96607805440,"internally_reserved":0,"allocated":1282048,"data_stored":871675,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280571,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46596,"kb_used_data":15408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325244,"statfs":{"total":96636764160,"available":96589049856,"internally_reserved":0,"allocated":15777792,"data_stored":16257488,"data_compressed":933071,"data_compressed_allocated":1011712,"data_compressed_original":2048000,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313276,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1265664,"data_stored":871296,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280572,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56772,"kb_used_data":17264,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94315068,"statfs":{"total":96636764160,"available":96578629632,"internally_reserved":0,"allocated":17678336,"data_stored":18541590,"data_compressed":1155075,"data_compressed_allocated":1282048,"data_compressed_original":2727936,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280571,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66772,"kb_used_data":31424,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94305068,"statfs":{"total":96636764160,"available":96568389632,"internally_reserved":0,"allocated":32178176,"data_stored":33927808,"data_compressed":2003664,"data_compressed_allocated":2174976,"data_compressed_original":4538368,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313276,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49212,"kb_used_data":18024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322628,"statfs":{"total":96636764160,"available":96586371072,"internally_reserved":0,"allocated":18456576,"data_stored":19696381,"data_compressed":1072888,"data_compressed_allocated":1187840,"data_compressed_original":3002368,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280571,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1265664,"data_stored":871003,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280571,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50936,"kb_used_data":15592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320904,"statfs":{"total":96636764160,"available":96584605696,"internally_reserved":0,"allocated":15966208,"data_stored":16517111,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280571,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46588,"kb_used_data":15400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325252,"statfs":{"total":96636764160,"available":96589058048,"internally_reserved":0,"allocated":15769600,"data_stored":16256537,"data_compressed":933071,"data_compressed_allocated":1011712,"data_compressed_original":2048000,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280572,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32436,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339404,"statfs":{"total":96636764160,"available":96603549696,"internally_reserved":0,"allocated":1277952,"data_stored":872168,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32432,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1273856,"data_stored":871881,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247868,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46792,"kb_used_data":15604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325048,"statfs":{"total":96636764160,"available":96588849152,"internally_reserved":0,"allocated":15978496,"data_stored":16523724,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:34.563 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T03:40:28.581990+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=40, tm_sec=28, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:41:34.563 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T03:40:24.423252+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=40, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:41:34.563 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-16T03:40:59.901560+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=40, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:41:34.563 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-16T03:40:38.134247+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=40, tm_sec=38, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:41:34.563 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T03:40:59.229884+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=40, tm_sec=59, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:41:34.563 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-16T03:41:04.232204+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=4, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:41:34.563 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T03:40:41.323782+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=40, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:41:34.564 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-16T03:40:55.942490+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=40, tm_sec=55, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:41:34.564 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T03:40:52.997837+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=40, tm_sec=52, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:41:34.564 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T03:41:12.744887+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=12, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:41:34.564 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T03:40:32.512789+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=40, tm_sec=32, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=41, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T03:41:34.564 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T03:41:37.749 INFO:tasks.ceph.osd.3.smithi157.stderr:2024-09-16T03:41:37.746+0000 7eff7884b640 -1 osd.3 239 handle_osd_map injecting map crc failure 2024-09-16T03:41:38.880 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-16T03:41:38.878+0000 7f49439d4640 -1 osd.4 240 handle_osd_map injecting map crc failure 2024-09-16T03:41:40.216 INFO:tasks.ceph.osd.9.smithi112.stderr:2024-09-16T03:41:40.214+0000 7fedb5739640 -1 osd.9 240 handle_osd_map injecting map crc failure 2024-09-16T03:41:40.217 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-09-16T03:41:40.215+0000 7fd101e6f640 -1 osd.2 240 handle_osd_map injecting map crc failure 2024-09-16T03:41:41.400 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:41:41.398+0000 7f3d4270c640 -1 osd.12 241 handle_osd_map injecting map crc failure 2024-09-16T03:41:41.849 INFO:tasks.ceph.osd.10.smithi155.stderr:2024-09-16T03:41:41.848+0000 7fa3886e6640 -1 osd.10 241 handle_osd_map injecting map crc failure 2024-09-16T03:41:41.987 INFO:tasks.ceph.osd.5.smithi112.stderr:2024-09-16T03:41:41.985+0000 7f12e2a1a640 -1 osd.5 241 handle_osd_map injecting map crc failure 2024-09-16T03:41:52.908 INFO:tasks.ceph.osd.12.smithi064.stderr:2024-09-16T03:41:52.906+0000 7f3d4270c640 -1 osd.12 242 handle_osd_map injecting map crc failure 2024-09-16T03:41:54.565 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:41:54.877 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-16T03:41:54.878 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-09-16T03:41:54.890 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-16T03:41:53.741376+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103194545,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":1243,"num_read_kb":34080,"num_write":1682,"num_write_kb":336758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":303,"num_bytes_recovered":140736862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53,"ondisk_log_size":53,"up":39,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":729512,"kb_used_data":205524,"kb_used_omap":24,"kb_used_meta":523623,"kb_avail":1509219928,"statfs":{"total":1546188226560,"available":1545441206272,"internally_reserved":0,"allocated":210456576,"data_stored":217623014,"data_compressed":13726505,"data_compressed_allocated":15134720,"data_compressed_original":31756288,"omap_allocated":25519,"internal_metadata":536190033},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002465"},"pg_stats":[{"pgid":"1.0","version":"240'114","reported_seq":678,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-16T03:41:42.028637+0000","last_change":"2024-09-16T03:41:36.899654+0000","last_active":"2024-09-16T03:41:42.028637+0000","last_peered":"2024-09-16T03:41:42.028637+0000","last_clean":"2024-09-16T03:41:42.028637+0000","last_became_active":"2024-09-16T03:40:18.805252+0000","last_became_peered":"2024-09-16T03:40:18.805252+0000","last_unstale":"2024-09-16T03:41:42.028637+0000","last_undegraded":"2024-09-16T03:41:42.028637+0000","last_fullsized":"2024-09-16T03:41:42.028637+0000","mapping_epoch":192,"log_start":"30'111","ondisk_log_start":"30'111","created":23,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'114","last_scrub_stamp":"2024-09-16T03:41:36.899544+0000","last_deep_scrub":"240'114","last_deep_scrub_stamp":"2024-09-16T03:41:36.899544+0000","last_clean_scrub_stamp":"2024-09-16T03:41:36.899544+0000","objects_scrubbed":2,"log_size":3,"log_dups_size":111,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:01.619968+0000","scrub_duration":411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"238'2","reported_seq":195,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-16T03:41:41.928396+0000","last_change":"2024-09-16T03:41:34.952410+0000","last_active":"2024-09-16T03:41:41.928396+0000","last_peered":"2024-09-16T03:41:41.928396+0000","last_clean":"2024-09-16T03:41:41.928396+0000","last_became_active":"2024-09-16T03:40:45.759458+0000","last_became_peered":"2024-09-16T03:40:45.759458+0000","last_unstale":"2024-09-16T03:41:41.928396+0000","last_undegraded":"2024-09-16T03:41:41.928396+0000","last_fullsized":"2024-09-16T03:41:41.928396+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"238'2","last_scrub_stamp":"2024-09-16T03:41:34.952309+0000","last_deep_scrub":"238'2","last_deep_scrub_stamp":"2024-09-16T03:41:34.952309+0000","last_clean_scrub_stamp":"2024-09-16T03:41:34.952309+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:43.945572+0000","scrub_duration":404,"objects_trimmed":0,"snaptrim_duration":0.00043501599999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,8],"acting":[11,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"240'362","reported_seq":1060,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-16T03:41:42.028652+0000","last_change":"2024-09-16T03:41:39.241123+0000","last_active":"2024-09-16T03:41:42.028652+0000","last_peered":"2024-09-16T03:41:42.028652+0000","last_clean":"2024-09-16T03:41:42.028652+0000","last_became_active":"2024-09-16T03:40:52.880022+0000","last_became_peered":"2024-09-16T03:40:52.880022+0000","last_unstale":"2024-09-16T03:41:42.028652+0000","last_undegraded":"2024-09-16T03:41:42.028652+0000","last_fullsized":"2024-09-16T03:41:42.028652+0000","mapping_epoch":224,"log_start":"240'347","ondisk_log_start":"240'347","created":31,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'362","last_scrub_stamp":"2024-09-16T03:41:39.240999+0000","last_deep_scrub":"240'362","last_deep_scrub_stamp":"2024-09-16T03:41:39.240999+0000","last_clean_scrub_stamp":"2024-09-16T03:41:39.240999+0000","objects_scrubbed":44,"log_size":15,"log_dups_size":356,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:00.070296+0000","scrub_duration":3768,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27308854,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":337,"num_read_kb":9885,"num_write":407,"num_write_kb":81624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":30694960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,0],"acting":[11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":502,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-16T03:41:41.850231+0000","last_change":"2024-09-16T03:41:29.267840+0000","last_active":"2024-09-16T03:41:41.850231+0000","last_peered":"2024-09-16T03:41:41.850231+0000","last_clean":"2024-09-16T03:41:41.850231+0000","last_became_active":"2024-09-16T03:40:15.923891+0000","last_became_peered":"2024-09-16T03:40:15.923891+0000","last_unstale":"2024-09-16T03:41:41.850231+0000","last_undegraded":"2024-09-16T03:41:41.850231+0000","last_fullsized":"2024-09-16T03:41:41.850231+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:29.267751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:29.267751+0000","last_clean_scrub_stamp":"2024-09-16T03:41:29.267751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:31.409746+0000","scrub_duration":300,"objects_trimmed":0,"snaptrim_duration":0.00054816000000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,13],"acting":[8,5,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"239'306","reported_seq":1004,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-16T03:41:41.494915+0000","last_change":"2024-09-16T03:41:37.192242+0000","last_active":"2024-09-16T03:41:41.494915+0000","last_peered":"2024-09-16T03:41:41.494915+0000","last_clean":"2024-09-16T03:41:41.494915+0000","last_became_active":"2024-09-16T03:41:19.120312+0000","last_became_peered":"2024-09-16T03:41:19.120312+0000","last_unstale":"2024-09-16T03:41:41.494915+0000","last_undegraded":"2024-09-16T03:41:41.494915+0000","last_fullsized":"2024-09-16T03:41:41.494915+0000","mapping_epoch":231,"log_start":"238'296","ondisk_log_start":"238'296","created":31,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'306","last_scrub_stamp":"2024-09-16T03:41:37.192073+0000","last_deep_scrub":"239'306","last_deep_scrub_stamp":"2024-09-16T03:41:37.192073+0000","last_clean_scrub_stamp":"2024-09-16T03:41:37.192073+0000","objects_scrubbed":39,"log_size":10,"log_dups_size":307,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:57.031600+0000","scrub_duration":4459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28703148,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":249,"num_read_kb":2401,"num_write":343,"num_write_kb":78301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":35337048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":70,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-16T03:41:41.850108+0000","last_change":"2024-09-16T03:41:34.976741+0000","last_active":"2024-09-16T03:41:41.850108+0000","last_peered":"2024-09-16T03:41:41.850108+0000","last_clean":"2024-09-16T03:41:41.850108+0000","last_became_active":"2024-09-16T03:40:47.179531+0000","last_became_peered":"2024-09-16T03:40:47.179531+0000","last_unstale":"2024-09-16T03:41:41.850108+0000","last_undegraded":"2024-09-16T03:41:41.850108+0000","last_fullsized":"2024-09-16T03:41:41.850108+0000","mapping_epoch":220,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:34.976667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:34.976667+0000","last_clean_scrub_stamp":"2024-09-16T03:41:34.976667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:41.259090+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11,14],"acting":[15,11,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"236'308","reported_seq":958,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-16T03:41:43.448320+0000","last_change":"2024-09-16T03:41:32.289882+0000","last_active":"2024-09-16T03:41:43.448320+0000","last_peered":"2024-09-16T03:41:43.448320+0000","last_clean":"2024-09-16T03:41:43.448320+0000","last_became_active":"2024-09-16T03:40:25.634914+0000","last_became_peered":"2024-09-16T03:40:25.634914+0000","last_unstale":"2024-09-16T03:41:43.448320+0000","last_undegraded":"2024-09-16T03:41:43.448320+0000","last_fullsized":"2024-09-16T03:41:43.448320+0000","mapping_epoch":199,"log_start":"234'298","ondisk_log_start":"234'298","created":31,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'308","last_scrub_stamp":"2024-09-16T03:41:32.289733+0000","last_deep_scrub":"236'308","last_deep_scrub_stamp":"2024-09-16T03:41:32.289733+0000","last_clean_scrub_stamp":"2024-09-16T03:41:32.289733+0000","objects_scrubbed":24,"log_size":10,"log_dups_size":319,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:38.396959+0000","scrub_duration":2247,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21970448,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":258,"num_read_kb":13466,"num_write":352,"num_write_kb":87672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":41823896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,4],"acting":[1,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"233'3","reported_seq":203,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-16T03:41:41.850243+0000","last_change":"2024-09-16T03:41:30.211050+0000","last_active":"2024-09-16T03:41:41.850243+0000","last_peered":"2024-09-16T03:41:41.850243+0000","last_clean":"2024-09-16T03:41:41.850243+0000","last_became_active":"2024-09-16T03:40:23.622019+0000","last_became_peered":"2024-09-16T03:40:23.622019+0000","last_unstale":"2024-09-16T03:41:41.850243+0000","last_undegraded":"2024-09-16T03:41:41.850243+0000","last_fullsized":"2024-09-16T03:41:41.850243+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'3","last_scrub_stamp":"2024-09-16T03:41:30.210925+0000","last_deep_scrub":"233'3","last_deep_scrub_stamp":"2024-09-16T03:41:30.210925+0000","last_clean_scrub_stamp":"2024-09-16T03:41:30.210925+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:52.198893+0000","scrub_duration":202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,0],"acting":[8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"240'380","reported_seq":953,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-16T03:41:41.950273+0000","last_change":"2024-09-16T03:41:38.385662+0000","last_active":"2024-09-16T03:41:41.950273+0000","last_peered":"2024-09-16T03:41:41.950273+0000","last_clean":"2024-09-16T03:41:41.950273+0000","last_became_active":"2024-09-16T03:40:55.029339+0000","last_became_peered":"2024-09-16T03:40:55.029339+0000","last_unstale":"2024-09-16T03:41:41.950273+0000","last_undegraded":"2024-09-16T03:41:41.950273+0000","last_fullsized":"2024-09-16T03:41:41.950273+0000","mapping_epoch":226,"log_start":"239'370","ondisk_log_start":"239'370","created":31,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'380","last_scrub_stamp":"2024-09-16T03:41:38.385490+0000","last_deep_scrub":"240'380","last_deep_scrub_stamp":"2024-09-16T03:41:38.385490+0000","last_clean_scrub_stamp":"2024-09-16T03:41:38.385490+0000","objects_scrubbed":39,"log_size":10,"log_dups_size":382,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:40.426270+0000","scrub_duration":2560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24621708,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":313,"num_read_kb":8255,"num_write":433,"num_write_kb":86487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":31294174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,2],"acting":[15,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":479,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-16T03:41:41.494920+0000","last_change":"2024-09-16T03:41:31.852227+0000","last_active":"2024-09-16T03:41:41.494920+0000","last_peered":"2024-09-16T03:41:41.494920+0000","last_clean":"2024-09-16T03:41:41.494920+0000","last_became_active":"2024-09-16T03:39:29.847660+0000","last_became_peered":"2024-09-16T03:39:29.847660+0000","last_unstale":"2024-09-16T03:41:41.494920+0000","last_undegraded":"2024-09-16T03:41:41.494920+0000","last_fullsized":"2024-09-16T03:41:41.494920+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:31.852140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:31.852140+0000","last_clean_scrub_stamp":"2024-09-16T03:41:31.852140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:41.976166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100625145,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,10],"acting":[6,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":242,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-16T03:41:41.850310+0000","last_change":"2024-09-16T03:41:30.935745+0000","last_active":"2024-09-16T03:41:41.850310+0000","last_peered":"2024-09-16T03:41:41.850310+0000","last_clean":"2024-09-16T03:41:41.850310+0000","last_became_active":"2024-09-16T03:39:09.848580+0000","last_became_peered":"2024-09-16T03:39:09.848580+0000","last_unstale":"2024-09-16T03:41:41.850310+0000","last_undegraded":"2024-09-16T03:41:41.850310+0000","last_fullsized":"2024-09-16T03:41:41.850310+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:30.935660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:30.935660+0000","last_clean_scrub_stamp":"2024-09-16T03:41:30.935660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:42.406073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":494,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-16T03:41:41.395889+0000","last_change":"2024-09-16T03:41:33.569113+0000","last_active":"2024-09-16T03:41:41.395889+0000","last_peered":"2024-09-16T03:41:41.395889+0000","last_clean":"2024-09-16T03:41:41.395889+0000","last_became_active":"2024-09-16T03:40:14.786897+0000","last_became_peered":"2024-09-16T03:40:14.786897+0000","last_unstale":"2024-09-16T03:41:41.395889+0000","last_undegraded":"2024-09-16T03:41:41.395889+0000","last_fullsized":"2024-09-16T03:41:41.395889+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:33.568931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:33.568931+0000","last_clean_scrub_stamp":"2024-09-16T03:41:33.568931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:42:47.849328+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0.00039038700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,13,3],"acting":[14,13,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":471,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-09-16T03:41:41.950410+0000","last_change":"2024-09-16T03:41:31.085662+0000","last_active":"2024-09-16T03:41:41.950410+0000","last_peered":"2024-09-16T03:41:41.950410+0000","last_clean":"2024-09-16T03:41:41.950410+0000","last_became_active":"2024-09-16T03:36:53.813150+0000","last_became_peered":"2024-09-16T03:36:53.813150+0000","last_unstale":"2024-09-16T03:41:41.950410+0000","last_undegraded":"2024-09-16T03:41:41.950410+0000","last_fullsized":"2024-09-16T03:41:41.950410+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:41:31.085573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:41:31.085573+0000","last_clean_scrub_stamp":"2024-09-16T03:41:31.085573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:43:00.463443+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0.00052539600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":102604158,"num_objects":146,"num_object_clones":0,"num_object_copies":438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":146,"num_whiteouts":0,"num_read":1157,"num_read_kb":34007,"num_write":1535,"num_write_kb":334084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":139150078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188276736,"data_stored":200996124,"data_compressed":12279636,"data_compressed_allocated":13111296,"data_compressed_original":26406912,"omap_allocated":0,"internal_metadata":0},"log_size":45,"ondisk_log_size":45,"up":12,"acting":12,"num_store_stats":11},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":24,"acting":24,"num_store_stats":14},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":2644640,"data_compressed":160479,"data_compressed_allocated":204800,"data_compressed_original":2596864,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51068,"kb_used_data":15716,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320772,"statfs":{"total":96636764160,"available":96584470528,"internally_reserved":0,"allocated":16093184,"data_stored":16587435,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":23,"seq":98784247872,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53496,"kb_used_data":18148,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94318344,"statfs":{"total":96636764160,"available":96581984256,"internally_reserved":0,"allocated":18583552,"data_stored":19765701,"data_compressed":1071475,"data_compressed_allocated":1187840,"data_compressed_original":2990080,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,2,4,5,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":23,"seq":98784247871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52696,"kb_used_data":17348,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319144,"statfs":{"total":96636764160,"available":96582803456,"internally_reserved":0,"allocated":17764352,"data_stored":18608864,"data_compressed":1155075,"data_compressed_allocated":1282048,"data_compressed_original":2727936,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,3,4,5,7,8,10,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280576,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49352,"kb_used_data":18164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322488,"statfs":{"total":96636764160,"available":96586227712,"internally_reserved":0,"allocated":18599936,"data_stored":19766811,"data_compressed":1071475,"data_compressed_allocated":1187840,"data_compressed_original":2990080,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,7,8,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280576,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28388,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343452,"statfs":{"total":96636764160,"available":96607694848,"internally_reserved":0,"allocated":1388544,"data_stored":940424,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280575,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46700,"kb_used_data":15512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325140,"statfs":{"total":96636764160,"available":96588943360,"internally_reserved":0,"allocated":15884288,"data_stored":16326237,"data_compressed":933071,"data_compressed_allocated":1011712,"data_compressed_original":2048000,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,6,7,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313280,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32532,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339308,"statfs":{"total":96636764160,"available":96603451392,"internally_reserved":0,"allocated":1376256,"data_stored":940289,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,6,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280576,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56872,"kb_used_data":17360,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94314968,"statfs":{"total":96636764160,"available":96578527232,"internally_reserved":0,"allocated":17776640,"data_stored":18609701,"data_compressed":1155075,"data_compressed_allocated":1282048,"data_compressed_original":2727936,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[2,4,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280575,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66872,"kb_used_data":31524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94304968,"statfs":{"total":96636764160,"available":96568287232,"internally_reserved":0,"allocated":32280576,"data_stored":33996213,"data_compressed":2003664,"data_compressed_allocated":2174976,"data_compressed_original":4538368,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[3,5,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":21,"seq":90194313280,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49320,"kb_used_data":18128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322520,"statfs":{"total":96636764160,"available":96586260480,"internally_reserved":0,"allocated":18563072,"data_stored":19765130,"data_compressed":1072888,"data_compressed_allocated":1187840,"data_compressed_original":3002368,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280575,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32520,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339320,"statfs":{"total":96636764160,"available":96603463680,"internally_reserved":0,"allocated":1363968,"data_stored":939164,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,2,6,8,9,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280575,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51044,"kb_used_data":15696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320796,"statfs":{"total":96636764160,"available":96584495104,"internally_reserved":0,"allocated":16072704,"data_stored":16585860,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,5,6,7,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280575,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46688,"kb_used_data":15500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325152,"statfs":{"total":96636764160,"available":96588955648,"internally_reserved":0,"allocated":15872000,"data_stored":16324942,"data_compressed":933071,"data_compressed_allocated":1011712,"data_compressed_original":2048000,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,4,6,7,8,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280576,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32540,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1384448,"data_stored":940917,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,4,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":21,"seq":90194313279,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32540,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339300,"statfs":{"total":96636764160,"available":96603443200,"internally_reserved":0,"allocated":1380352,"data_stored":940630,"data_compressed":84482,"data_compressed_allocated":118784,"data_compressed_original":237568,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":23,"seq":98784247872,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46884,"kb_used_data":15696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324956,"statfs":{"total":96636764160,"available":96588754944,"internally_reserved":0,"allocated":16072704,"data_stored":16584696,"data_compressed":1302767,"data_compressed_allocated":1404928,"data_compressed_original":2498560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":528928,"data_compressed":32661,"data_compressed_allocated":40960,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":31248,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":30900224,"data_stored":33055164,"data_compressed":1919182,"data_compressed_allocated":2056192,"data_compressed_original":4300800,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16388096,"data_stored":17668524,"data_compressed":1070593,"data_compressed_allocated":1163264,"data_compressed_original":2490368,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14512128,"data_stored":15386640,"data_compressed":848589,"data_compressed_allocated":892928,"data_compressed_original":1810432,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":17145856,"data_stored":18296457,"data_compressed":955745,"data_compressed_allocated":1028096,"data_compressed_original":2240512,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":14712832,"data_stored":15647087,"data_compressed":1218285,"data_compressed_allocated":1286144,"data_compressed_original":2260992,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:41:54.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T03:41:55.212 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T03:41:55.212 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T03:41:55.213 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T03:41:55.213 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:55.255 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T03:41:55.256 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T03:41:55.256 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:55.303 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T03:41:55.304 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-16T03:41:55.304 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:55.336 INFO:tasks.ceph.osd.8:Stopped 2024-09-16T03:41:55.336 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-16T03:41:55.336 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:55.412 INFO:tasks.ceph.osd.12:Stopped 2024-09-16T03:41:55.412 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T03:41:55.412 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:55.446 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T03:41:55.447 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T03:41:55.447 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:55.522 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T03:41:55.522 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-16T03:41:55.522 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:55.591 INFO:tasks.ceph.osd.9:Stopped 2024-09-16T03:41:55.591 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-16T03:41:55.591 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:55.656 INFO:tasks.ceph.osd.13:Stopped 2024-09-16T03:41:55.656 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T03:41:55.656 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:55.690 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T03:41:55.690 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-16T03:41:55.691 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:55.764 INFO:tasks.ceph.osd.6:Stopped 2024-09-16T03:41:55.764 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-16T03:41:55.764 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:55.838 INFO:tasks.ceph.osd.10:Stopped 2024-09-16T03:41:55.838 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-16T03:41:55.838 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:55.905 INFO:tasks.ceph.osd.14:Stopped 2024-09-16T03:41:55.905 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T03:41:55.905 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:55.937 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T03:41:55.938 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-16T03:41:55.938 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:56.006 INFO:tasks.ceph.osd.7:Stopped 2024-09-16T03:41:56.006 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-16T03:41:56.006 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:56.081 INFO:tasks.ceph.osd.11:Stopped 2024-09-16T03:41:56.081 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-16T03:41:56.081 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:56.150 INFO:tasks.ceph.osd.15:Stopped 2024-09-16T03:41:56.151 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T03:41:56.151 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-16T03:41:56.151 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:56.184 INFO:tasks.ceph.mgr.y:Stopped 2024-09-16T03:41:56.184 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T03:41:56.184 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:56.220 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T03:41:56.220 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T03:41:56.220 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T03:41:56.221 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:56.237 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T03:41:56.237 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:41:56.238 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:56.252 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:41:56.252 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:41:56.252 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:41:56.267 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:41:56.267 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T03:41:56.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'failed to encode map' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-16T03:41:56.315 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-16T03:41:56.316 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T03:41:56.440 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-16T03:41:56.440 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T03:41:56.504 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-16T03:41:56.505 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-16T03:41:56.556 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-16T03:41:56.557 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-09-16T03:41:56.612 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-16T03:41:56.612 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T03:41:56.779 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-16T03:41:56.779 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T03:41:56.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-16T03:41:56.841 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-16T03:41:56.892 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-16T03:41:56.892 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-09-16T03:41:56.941 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi155.front.sepia.ceph.com 2024-09-16T03:41:56.941 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T03:41:57.104 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi155.front.sepia.ceph.com 2024-09-16T03:41:57.104 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-16T03:41:57.218 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi155.front.sepia.ceph.com 2024-09-16T03:41:57.218 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-16T03:41:57.274 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi155.front.sepia.ceph.com 2024-09-16T03:41:57.274 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-09-16T03:41:57.333 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-16T03:41:57.334 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T03:41:57.436 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-16T03:41:57.436 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-16T03:41:57.523 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-16T03:41:57.524 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-16T03:41:57.606 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi157.front.sepia.ceph.com 2024-09-16T03:41:57.606 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-09-16T03:41:57.665 INFO:tasks.ceph:Archiving mon data... 2024-09-16T03:41:57.666 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/data/mon.a.tgz 2024-09-16T03:41:57.666 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-09-16T03:41:57.685 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.r3Ajq1Fseu 2024-09-16T03:41:57.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.r3Ajq1Fseu 2024-09-16T03:41:57.900 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.r3Ajq1Fseu 2024-09-16T03:41:57.985 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.r3Ajq1Fseu is 708KB 2024-09-16T03:41:58.088 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.r3Ajq1Fseu 2024-09-16T03:41:58.103 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/data/mon.b.tgz 2024-09-16T03:41:58.103 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-09-16T03:41:58.122 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.ogUXVVZGwx 2024-09-16T03:41:58.122 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.ogUXVVZGwx 2024-09-16T03:41:58.353 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.ogUXVVZGwx 2024-09-16T03:41:58.436 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.ogUXVVZGwx is 708KB 2024-09-16T03:41:58.536 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.ogUXVVZGwx 2024-09-16T03:41:58.552 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/data/mon.c.tgz 2024-09-16T03:41:58.552 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-09-16T03:41:58.571 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.hAFpEAAdAW 2024-09-16T03:41:58.571 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.hAFpEAAdAW 2024-09-16T03:41:58.981 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.hAFpEAAdAW 2024-09-16T03:41:59.064 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.hAFpEAAdAW is 697KB 2024-09-16T03:41:59.136 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.hAFpEAAdAW 2024-09-16T03:41:59.151 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T03:41:59.151 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T03:41:59.153 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T03:41:59.156 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T03:41:59.192 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T03:41:59.241 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T03:41:59.257 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T03:41:59.272 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T03:41:59.291 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T03:41:59.291 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T03:41:59.293 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/remote/smithi064/crash 2024-09-16T03:41:59.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T03:41:59.323 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/remote/smithi112/crash 2024-09-16T03:41:59.324 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T03:41:59.350 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/remote/smithi155/crash 2024-09-16T03:41:59.351 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T03:41:59.375 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/remote/smithi157/crash 2024-09-16T03:41:59.376 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T03:41:59.485 INFO:tasks.ceph:Compressing logs... 2024-09-16T03:41:59.485 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T03:41:59.488 DEBUG:teuthology.orchestra.run.smithi112:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T03:41:59.490 DEBUG:teuthology.orchestra.run.smithi155:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T03:41:59.493 DEBUG:teuthology.orchestra.run.smithi157:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T03:41:59.514 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T03:41:59.514 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-16T03:41:59.514 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-09-16T03:41:59.515 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T03:41:59.515 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph-osd.13.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T03:41:59.515 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T03:41:59.517 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T03:41:59.517 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-16T03:41:59.518 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T03:41:59.518 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T03:41:59.518 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.10.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T03:41:59.518 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-09-16T03:41:59.518 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-16T03:41:59.518 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-osd.14.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T03:41:59.519 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T03:41:59.519 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T03:41:59.524 INFO:teuthology.orchestra.run.smithi155.stderr: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T03:41:59.524 INFO:teuthology.orchestra.run.smithi112.stderr: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T03:41:59.541 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32644.log 2024-09-16T03:41:59.541 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32924.log 2024-09-16T03:41:59.541 INFO:teuthology.orchestra.run.smithi157.stderr:gzip/var/log/ceph/ceph-client.admin.32644.log: -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32644.log.gz 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.32924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33084.log 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32924.log.gz 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32964.log 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.33084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33124.log 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33084.log.gz 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.32964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32724.log 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32964.log.gz 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.33124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33004.log 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33124.log.gz 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.32724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32684.log 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32724.log.gz 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-09-16T03:41:59.542 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.33004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33004.log.gz 2024-09-16T03:41:59.543 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.32684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32884.log 2024-09-16T03:41:59.543 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32684.log.gz 2024-09-16T03:41:59.543 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36673.log 2024-09-16T03:41:59.543 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.32884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33044.log 2024-09-16T03:41:59.543 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32884.log.gz 2024-09-16T03:41:59.543 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.36673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32844.log 2024-09-16T03:41:59.543 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36673.log.gz 2024-09-16T03:41:59.543 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.33044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33044.log.gz 2024-09-16T03:41:59.543 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-09-16T03:41:59.543 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.32844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32513.log 2024-09-16T03:41:59.543 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32844.log.gz 2024-09-16T03:41:59.543 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32764.log 2024-09-16T03:41:59.543 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.32513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32513.log.gz 2024-09-16T03:41:59.544 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32553.log 2024-09-16T03:41:59.544 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T03:41:59.544 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.32764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32764.log.gz 2024-09-16T03:41:59.544 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.32553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36696.log 2024-09-16T03:41:59.544 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32553.log.gz 2024-09-16T03:41:59.544 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32597.log 2024-09-16T03:41:59.544 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.0.36696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32804.log 2024-09-16T03:41:59.544 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.36696.log.gz 2024-09-16T03:41:59.544 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.32597.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T03:41:59.544 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32597.log.gz 2024-09-16T03:41:59.544 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.32804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32804.log.gz 2024-09-16T03:41:59.544 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.3.log: 93.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T03:41:59.716 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46820.log 2024-09-16T03:41:59.717 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T03:41:59.717 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59142.log 2024-09-16T03:41:59.717 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46820.log.gz 2024-09-16T03:41:59.717 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38853.log 2024-09-16T03:41:59.717 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44466.log 2024-09-16T03:41:59.717 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59142.log.gz 2024-09-16T03:41:59.717 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58848.log 2024-09-16T03:41:59.717 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38853.log.gz 2024-09-16T03:41:59.717 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51370.log 2024-09-16T03:41:59.717 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44466.log.gz 2024-09-16T03:41:59.717 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54311.log 2024-09-16T03:41:59.718 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58848.log.gz 2024-09-16T03:41:59.718 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64182.log 2024-09-16T03:41:59.718 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51370.log.gz 2024-09-16T03:41:59.718 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61574.log 2024-09-16T03:41:59.718 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54311.log.gz 2024-09-16T03:41:59.718 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57712.log 2024-09-16T03:41:59.718 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64182.log.gz 2024-09-16T03:41:59.718 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63258.log 2024-09-16T03:41:59.718 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61574.log.gz 2024-09-16T03:41:59.718 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57712.log.gz 2024-09-16T03:41:59.718 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40964.log 2024-09-16T03:41:59.718 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61114.log 2024-09-16T03:41:59.718 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63258.log.gz 2024-09-16T03:41:59.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59688.log 2024-09-16T03:41:59.719 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40964.log.gz 2024-09-16T03:41:59.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52441.log 2024-09-16T03:41:59.719 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61114.log.gz 2024-09-16T03:41:59.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61065.log 2024-09-16T03:41:59.719 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59688.log.gz 2024-09-16T03:41:59.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45095.log 2024-09-16T03:41:59.719 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52441.log.gz 2024-09-16T03:41:59.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49613.log 2024-09-16T03:41:59.719 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61065.log.gz 2024-09-16T03:41:59.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50668.log 2024-09-16T03:41:59.719 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45095.log.gz 2024-09-16T03:41:59.719 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54087.log 2024-09-16T03:41:59.720 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49613.log.gz 2024-09-16T03:41:59.720 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37640.log 2024-09-16T03:41:59.720 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50668.log.gz 2024-09-16T03:41:59.720 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53221.log 2024-09-16T03:41:59.720 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54087.log.gz 2024-09-16T03:41:59.720 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37640.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.50845.log -- replaced with /var/log/ceph/ceph-client.admin.37640.log.gz 2024-09-16T03:41:59.720 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.720 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53641.log 2024-09-16T03:41:59.720 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53221.log.gz 2024-09-16T03:41:59.720 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45335.log 2024-09-16T03:41:59.720 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50845.log.gz 2024-09-16T03:41:59.720 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45848.log 2024-09-16T03:41:59.720 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53641.log.gz 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58932.log 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45335.log.gz 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45848.log.gz 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61662.log 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38225.log 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58932.log.gz 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50198.log 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61662.log.gz 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44344.log 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38225.log.gz 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41320.log 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50198.log.gz 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52345.log 2024-09-16T03:41:59.721 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44344.log.gz 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59100.log 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41320.log.gz 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52345.log.gz 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46469.log 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38187.log 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59100.log.gz 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43423.log 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46469.log.gz 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60973.log 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38187.log.gz 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43111.log 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43423.log.gz 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43987.log 2024-09-16T03:41:59.722 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60973.log.gz 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52709.log 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43111.log.gz 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60108.log 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43987.log.gz 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50083.log 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52709.log.gz 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60435.log 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60108.log.gz 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50083.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.50083.log.gz 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63636.log 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63426.log 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60435.log.gz 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41489.log 2024-09-16T03:41:59.723 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63636.log.gz 2024-09-16T03:41:59.724 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38265.log 2024-09-16T03:41:59.724 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63426.log.gz 2024-09-16T03:41:59.724 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54823.log 2024-09-16T03:41:59.724 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41489.log.gz 2024-09-16T03:41:59.724 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38259.log 2024-09-16T03:41:59.724 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38265.log.gz 2024-09-16T03:41:59.724 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38977.log 2024-09-16T03:41:59.724 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54823.log.gz 2024-09-16T03:41:59.724 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38318.log 2024-09-16T03:41:59.724 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38259.log.gz 2024-09-16T03:41:59.724 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37612.log 2024-09-16T03:41:59.724 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38977.log.gz 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59772.log 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38318.log.gz 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56801.log 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37612.log.gz 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46758.log 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59772.log.gz 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54954.log 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56801.log.gz 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51808.log 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46758.log.gz 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64224.log 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54954.log.gz 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64518.log 2024-09-16T03:41:59.725 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51808.log.gz 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64224.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.64224.log.gz /var/log/ceph/ceph-client.admin.47909.log 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63216.log 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64518.log.gz 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58633.log 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47909.log.gz 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38772.log 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63216.log.gz 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38935.log 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58633.log.gz 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53932.log 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38772.log.gz 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59856.log 2024-09-16T03:41:59.726 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38935.log.gz 2024-09-16T03:41:59.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45252.log 2024-09-16T03:41:59.727 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53932.log.gz 2024-09-16T03:41:59.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60423.log 2024-09-16T03:41:59.727 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59856.log.gz 2024-09-16T03:41:59.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37602.log 2024-09-16T03:41:59.727 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45252.log.gz 2024-09-16T03:41:59.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40132.log 2024-09-16T03:41:59.727 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60423.log.gz 2024-09-16T03:41:59.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40871.log 2024-09-16T03:41:59.727 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37602.log.gz 2024-09-16T03:41:59.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60024.log 2024-09-16T03:41:59.727 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40132.log.gz 2024-09-16T03:41:59.727 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49686.log 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40871.log.gz 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56132.log 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60024.log.gz 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58764.log 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49686.log.gz 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55498.log 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56132.log.gz 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52510.log 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58764.log.gz 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59016.log 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55498.log.gz 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59982.log 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52510.log.gz 2024-09-16T03:41:59.728 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59016.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27341.log 2024-09-16T03:41:59.729 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59016.log.gz 2024-09-16T03:41:59.729 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37642.log 2024-09-16T03:41:59.729 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59982.log.gz 2024-09-16T03:41:59.729 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.tmp-client.admin.27341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59478.log 2024-09-16T03:41:59.729 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27341.log.gz 2024-09-16T03:41:59.729 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60433.log 2024-09-16T03:41:59.729 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37642.log.gz 2024-09-16T03:41:59.729 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63972.log 2024-09-16T03:41:59.729 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59478.log.gz 2024-09-16T03:41:59.729 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62754.log 2024-09-16T03:41:59.729 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60433.log.gz 2024-09-16T03:41:59.729 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39491.log 2024-09-16T03:41:59.730 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63972.log.gz 2024-09-16T03:41:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44583.log 2024-09-16T03:41:59.730 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62754.log.gz 2024-09-16T03:41:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59436.log 2024-09-16T03:41:59.730 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39491.log.gz 2024-09-16T03:41:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61033.log 2024-09-16T03:41:59.730 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44583.log.gz 2024-09-16T03:41:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49159.log 2024-09-16T03:41:59.730 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59436.log.gz 2024-09-16T03:41:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51868.log 2024-09-16T03:41:59.730 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61033.log.gz 2024-09-16T03:41:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59604.log 2024-09-16T03:41:59.730 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49159.log.gz 2024-09-16T03:41:59.731 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62544.log 2024-09-16T03:41:59.731 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51868.log.gz 2024-09-16T03:41:59.731 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48390.log 2024-09-16T03:41:59.731 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59604.log.gz 2024-09-16T03:41:59.731 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56958.log 2024-09-16T03:41:59.731 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62544.log.gz 2024-09-16T03:41:59.731 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62208.log 2024-09-16T03:41:59.731 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48390.log.gz 2024-09-16T03:41:59.731 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61027.log 2024-09-16T03:41:59.731 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56958.log.gz 2024-09-16T03:41:59.731 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37610.log 2024-09-16T03:41:59.731 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62208.log.gz 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47280.log 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61027.log.gz 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62040.log 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37610.log.gz 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62628.log 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47280.log.gz 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61998.log 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62040.log.gz 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45208.log 0.0% 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62628.log.gz 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58350.log 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61998.log.gz 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.45208.log: -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-09-16T03:41:59.732 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45208.log.gz 2024-09-16T03:41:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47239.log 2024-09-16T03:41:59.733 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58350.log.gz 2024-09-16T03:41:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45115.log 2024-09-16T03:41:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58569.log 2024-09-16T03:41:59.733 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47239.log.gz 2024-09-16T03:41:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47676.log 2024-09-16T03:41:59.733 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45115.log.gz 2024-09-16T03:41:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58569.log.gz 2024-09-16T03:41:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61011.log 2024-09-16T03:41:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63468.log 2024-09-16T03:41:59.733 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47676.log.gz 2024-09-16T03:41:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61011.log.gz 2024-09-16T03:41:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48745.log 2024-09-16T03:41:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41447.log 2024-09-16T03:41:59.734 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63468.log.gz 2024-09-16T03:41:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48745.log.gzgzip 2024-09-16T03:41:59.734 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61914.log 2024-09-16T03:41:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63132.log 2024-09-16T03:41:59.734 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41447.log.gz 2024-09-16T03:41:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61914.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61914.log.gz 2024-09-16T03:41:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61872.log 2024-09-16T03:41:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42128.log 2024-09-16T03:41:59.734 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63132.log.gz 2024-09-16T03:41:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61872.log.gz 2024-09-16T03:41:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33674.log 2024-09-16T03:41:59.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39216.log 2024-09-16T03:41:59.735 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42128.log.gz 2024-09-16T03:41:59.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33674.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.58182.log -- replaced with /var/log/ceph/ceph-client.admin.33674.log.gz 2024-09-16T03:41:59.735 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44202.log 2024-09-16T03:41:59.735 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39216.log.gz 2024-09-16T03:41:59.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58182.log.gz 2024-09-16T03:41:59.735 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64098.log 2024-09-16T03:41:59.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46005.log 2024-09-16T03:41:59.735 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44202.log.gz 2024-09-16T03:41:59.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47677.log 2024-09-16T03:41:59.735 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64098.log.gz 2024-09-16T03:41:59.735 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44893.log 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46005.log.gz 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47677.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.58286.log -- replaced with /var/log/ceph/ceph-client.admin.47677.log.gz 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47780.log 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44893.log.gz 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47396.log 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58286.log.gz 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39113.log 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47780.log.gz 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47396.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.58890.log -- replaced with /var/log/ceph/ceph-client.admin.47396.log.gz 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63888.log 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39113.log.gz 2024-09-16T03:41:59.736 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59184.log 2024-09-16T03:41:59.737 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58890.log.gz 2024-09-16T03:41:59.737 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63300.log 2024-09-16T03:41:59.737 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63888.log.gz 2024-09-16T03:41:59.737 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59184.log.gz 2024-09-16T03:41:59.737 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38288.log 2024-09-16T03:41:59.737 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48275.log 2024-09-16T03:41:59.737 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63300.log.gz 2024-09-16T03:41:59.737 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61788.log 2024-09-16T03:41:59.737 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38288.log.gz 2024-09-16T03:41:59.737 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46674.log 2024-09-16T03:41:59.737 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48275.log.gz 2024-09-16T03:41:59.737 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56287.log 2024-09-16T03:41:59.737 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61788.log.gz 2024-09-16T03:41:59.738 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60421.log 2024-09-16T03:41:59.738 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46674.log.gz 2024-09-16T03:41:59.738 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41551.log 2024-09-16T03:41:59.738 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56287.log.gz 2024-09-16T03:41:59.738 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45888.log 2024-09-16T03:41:59.738 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60421.log.gz 2024-09-16T03:41:59.738 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41551.log.gz 2024-09-16T03:41:59.738 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51206.log 2024-09-16T03:41:59.738 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62502.log 2024-09-16T03:41:59.738 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45888.log.gz 2024-09-16T03:41:59.738 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51206.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51206.log.gz 2024-09-16T03:41:59.738 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59268.log 2024-09-16T03:41:59.738 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55294.log 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62502.log.gz 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59268.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.59268.log.gz /var/log/ceph/ceph-client.admin.63090.log 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48787.log 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55294.log.gz 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37620.log 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63090.log.gz 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43755.log 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48787.log.gz 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37620.log.gz 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60446.log 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39027.log 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43755.log.gz 2024-09-16T03:41:59.739 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43914.log 2024-09-16T03:41:59.740 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60446.log.gz 2024-09-16T03:41:59.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60417.log 2024-09-16T03:41:59.740 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39027.log.gz 2024-09-16T03:41:59.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43914.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T03:41:59.740 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43914.log.gz 2024-09-16T03:41:59.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61080.log 2024-09-16T03:41:59.740 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60417.log.gz 2024-09-16T03:41:59.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51166.log 2024-09-16T03:41:59.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50751.log 2024-09-16T03:41:59.740 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61080.log.gz 2024-09-16T03:41:59.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37027.log 2024-09-16T03:41:59.740 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51166.log.gz 2024-09-16T03:41:59.740 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50751.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50751.log.gz 2024-09-16T03:41:59.741 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49571.log 2024-09-16T03:41:59.741 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38238.log 2024-09-16T03:41:59.741 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37027.log.gz 2024-09-16T03:41:59.741 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49571.log.gz 2024-09-16T03:41:59.741 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37083.log 2024-09-16T03:41:59.741 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60436.log 2024-09-16T03:41:59.741 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38238.log.gz 2024-09-16T03:41:59.741 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63762.log 2024-09-16T03:41:59.741 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37083.log.gz 2024-09-16T03:41:59.741 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38285.log 2024-09-16T03:41:59.741 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60436.log.gz 2024-09-16T03:41:59.741 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63762.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63762.log.gz -5 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50996.log 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50909.log 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38285.log.gz 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62250.log 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50996.log.gz 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50909.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50909.log.gz 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62250.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.62250.log.gz 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33467.log 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52025.log 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.33467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33467.log.gz 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54680.log 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62712.log 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52025.log.gz 2024-09-16T03:41:59.742 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64558.log 2024-09-16T03:41:59.743 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54680.log.gz 2024-09-16T03:41:59.743 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52583.log 2024-09-16T03:41:59.743 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62712.log.gz 2024-09-16T03:41:59.743 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59352.log 2024-09-16T03:41:59.743 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64558.log.gz 2024-09-16T03:41:59.743 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37176.log 2024-09-16T03:41:59.743 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52583.log.gz 2024-09-16T03:41:59.743 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59352.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.59352.log.gz -5 2024-09-16T03:41:59.743 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46599.log 2024-09-16T03:41:59.743 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39689.log 2024-09-16T03:41:59.743 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37176.log.gz 2024-09-16T03:41:59.743 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46599.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T03:41:59.743 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39215.log 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46599.log.gz 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40400.log 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39689.log.gz 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64056.log 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39215.log.gz 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40400.log.gz 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50593.log 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62964.log 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64056.log.gz 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62082.log 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50593.log.gz 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57195.log 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62964.log.gz 2024-09-16T03:41:59.744 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64308.log 2024-09-16T03:41:59.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62082.log.gz 2024-09-16T03:41:59.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63342.log 2024-09-16T03:41:59.745 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57195.log.gz 2024-09-16T03:41:59.745 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54640.log 2024-09-16T03:41:59.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64308.log.gz 2024-09-16T03:41:59.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60450.log 2024-09-16T03:41:59.745 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63342.log.gz 2024-09-16T03:41:59.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58974.log 2024-09-16T03:41:59.745 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54640.log.gz 2024-09-16T03:41:59.745 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61038.log 2024-09-16T03:41:59.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60450.log.gz 2024-09-16T03:41:59.745 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58391.log 2024-09-16T03:41:59.745 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58974.log.gz 2024-09-16T03:41:59.746 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61038.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45695.log 2024-09-16T03:41:59.746 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61038.log.gz 2024-09-16T03:41:59.746 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49483.log 2024-09-16T03:41:59.746 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58391.log.gz 2024-09-16T03:41:59.746 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54700.log 2024-09-16T03:41:59.746 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45695.log.gz 2024-09-16T03:41:59.746 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49483.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.62292.log -- replaced with /var/log/ceph/ceph-client.admin.49483.log.gz 2024-09-16T03:41:59.746 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.746 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51567.log 2024-09-16T03:41:59.746 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54700.log.gz 2024-09-16T03:41:59.746 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51547.log 2024-09-16T03:41:59.746 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62292.log.gz 2024-09-16T03:41:59.746 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60430.log 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51567.log.gz 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59562.log 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51547.log.gz 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60430.log.gz 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58507.log 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.59562.log: -- /var/log/ceph/ceph-client.admin.37634.log 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59562.log.gz 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58507.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.58507.log.gz -5 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51505.log 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41154.log 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37634.log.gz 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58266.log 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51505.log.gz 2024-09-16T03:41:59.747 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41154.log.gz 2024-09-16T03:41:59.748 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38688.log 2024-09-16T03:41:59.748 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62670.log 2024-09-16T03:41:59.748 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58266.log.gz 2024-09-16T03:41:59.748 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39846.log 2024-09-16T03:41:59.748 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38688.log.gz 2024-09-16T03:41:59.748 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62670.log.gz 2024-09-16T03:41:59.748 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43285.log 2024-09-16T03:41:59.748 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55662.log 2024-09-16T03:41:59.748 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39846.log.gz 2024-09-16T03:41:59.748 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43285.log.gz 2024-09-16T03:41:59.748 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58107.log 2024-09-16T03:41:59.748 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55662.log.gz 2024-09-16T03:41:59.748 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37638.log 2024-09-16T03:41:59.749 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55114.log 2024-09-16T03:41:59.749 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58107.log.gz 2024-09-16T03:41:59.749 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60066.log 2024-09-16T03:41:59.749 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37638.log.gz 2024-09-16T03:41:59.749 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55114.log.gz 2024-09-16T03:41:59.749 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62838.log 2024-09-16T03:41:59.749 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46415.log 2024-09-16T03:41:59.749 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60066.log.gz 2024-09-16T03:41:59.749 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48018.log 2024-09-16T03:41:59.749 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62838.log.gz 2024-09-16T03:41:59.749 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46415.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46415.log.gz -5 2024-09-16T03:41:59.749 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.60916.log 2024-09-16T03:41:59.749 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52227.log 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48018.log.gz 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49528.log 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60916.log.gz 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52227.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.52227.log.gz 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55054.log 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49414.log 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49528.log.gz 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56843.log 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55054.log.gz 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49414.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.49414.log.gz /var/log/ceph/ceph-client.admin.54873.log 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48952.log 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56843.log.gz 2024-09-16T03:41:59.750 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53974.log 2024-09-16T03:41:59.751 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54873.log.gz 2024-09-16T03:41:59.751 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48952.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.48952.log.gz 2024-09-16T03:41:59.751 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59310.log 2024-09-16T03:41:59.751 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53106.log 2024-09-16T03:41:59.751 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53974.log.gz 2024-09-16T03:41:59.751 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59310.log.gz 2024-09-16T03:41:59.751 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57554.log 2024-09-16T03:41:59.751 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5/var/log/ceph/ceph-client.admin.53106.log: --verbose -- /var/log/ceph/ceph-client.admin.40285.log 2024-09-16T03:41:59.751 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53106.log.gz 2024-09-16T03:41:59.751 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37260.log 2024-09-16T03:41:59.751 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57554.log.gz 2024-09-16T03:41:59.751 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40285.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40285.log.gz 2024-09-16T03:41:59.751 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52871.log 2024-09-16T03:41:59.752 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51683.log 2024-09-16T03:41:59.752 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37260.log.gz 2024-09-16T03:41:59.752 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63678.log 2024-09-16T03:41:59.752 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52871.log.gz 2024-09-16T03:41:59.752 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51683.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.63174.log 2024-09-16T03:41:59.752 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51683.log.gz 2024-09-16T03:41:59.752 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52269.log 2024-09-16T03:41:59.752 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63678.log.gz 2024-09-16T03:41:59.752 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44670.log 2024-09-16T03:41:59.752 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63174.log.gz 2024-09-16T03:41:59.752 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52269.log.gz 2024-09-16T03:41:59.752 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64014.log 2024-09-16T03:41:59.752 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41070.log 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44670.log.gz 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61830.log 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64014.log.gz 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41070.log.gz 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49017.log 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55233.log 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61830.log.gz 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60434.log 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49017.log.gz 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55233.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.55233.log.gz 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41024.log 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62166.log 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60434.log.gz 2024-09-16T03:41:59.753 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37218.log 2024-09-16T03:41:59.754 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41024.log.gz 2024-09-16T03:41:59.754 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62166.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62166.log.gz -5 2024-09-16T03:41:59.754 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42682.log 2024-09-16T03:41:59.754 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37627.log 2024-09-16T03:41:59.754 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37218.log.gz 2024-09-16T03:41:59.754 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53346.log 2024-09-16T03:41:59.754 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42682.log.gz 2024-09-16T03:41:59.754 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37627.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37627.log.gz 2024-09-16T03:41:59.754 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62334.log 2024-09-16T03:41:59.754 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58202.log 2024-09-16T03:41:59.754 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53346.log.gz 2024-09-16T03:41:59.754 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62922.log 2024-09-16T03:41:59.754 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62334.log.gz 2024-09-16T03:41:59.755 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58202.log.gz 2024-09-16T03:41:59.755 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61746.log 2024-09-16T03:41:59.755 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41739.log 2024-09-16T03:41:59.755 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62922.log.gz 2024-09-16T03:41:59.755 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40326.log 2024-09-16T03:41:59.755 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61746.log.gz 2024-09-16T03:41:59.755 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41739.log.gz 2024-09-16T03:41:59.755 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41196.log 2024-09-16T03:41:59.755 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55589.log 2024-09-16T03:41:59.755 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40326.log.gz 2024-09-16T03:41:59.755 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38813.log 2024-09-16T03:41:59.755 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41196.log.gz 2024-09-16T03:41:59.755 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59814.log 2024-09-16T03:41:59.756 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55589.log.gz 2024-09-16T03:41:59.756 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64266.log 2024-09-16T03:41:59.756 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38813.log.gz 2024-09-16T03:41:59.756 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54559.log 2024-09-16T03:41:59.756 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59814.log.gz 2024-09-16T03:41:59.756 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64266.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64266.log.gz -5 2024-09-16T03:41:59.756 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51431.log 2024-09-16T03:41:59.756 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45377.log 2024-09-16T03:41:59.756 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54559.log.gz 2024-09-16T03:41:59.756 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60993.log 2024-09-16T03:41:59.756 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51431.log.gz 2024-09-16T03:41:59.756 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45377.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45377.log.gzgzip 2024-09-16T03:41:59.756 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41362.log 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57617.log 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60993.log.gz 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41659.log 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41362.log.gz 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57617.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.57617.log.gz 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61620.log 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53460.log 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41659.log.gz 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61620.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61620.log.gz 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53460.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.64392.log 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53460.log.gz 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46360.log 2024-09-16T03:41:59.757 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.64392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43245.log 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64392.log.gz 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46360.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.44029.log 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46360.log.gz 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44541.log 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43245.log.gz 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44029.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44029.log.gz 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38893.log 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44541.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44541.log.gz 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59058.log 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38893.log.gz 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55704.log 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47416.log 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59058.log.gz 2024-09-16T03:41:59.758 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55704.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55704.log.gz 2024-09-16T03:41:59.759 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.60426.log 2024-09-16T03:41:59.759 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47416.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54047.log 2024-09-16T03:41:59.759 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47416.log.gz 2024-09-16T03:41:59.759 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60426.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.52399.log 2024-09-16T03:41:59.759 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60426.log.gz 2024-09-16T03:41:59.759 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41238.log 2024-09-16T03:41:59.759 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54047.log.gz 2024-09-16T03:41:59.759 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52399.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52399.log.gz 2024-09-16T03:41:59.759 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.61026.log 2024-09-16T03:41:59.759 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41238.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57753.log 2024-09-16T03:41:59.759 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41238.log.gz 2024-09-16T03:41:59.759 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60447.log 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61026.log.gz 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59226.log 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57753.log.gz 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60447.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60447.log.gz 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46694.log 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59226.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61704.log 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59226.log.gz 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39574.log 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46694.log.gz 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46819.log 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61704.log.gz 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61956.log 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39574.log.gz 2024-09-16T03:41:59.760 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46819.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.46819.log.gz --verbose 2024-09-16T03:41:59.761 INFO:teuthology.orchestra.run.smithi064.stderr: -- /var/log/ceph/ceph-client.admin.37644.log 2024-09-16T03:41:59.761 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57470.log 2024-09-16T03:41:59.761 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61956.log.gz 2024-09-16T03:41:59.761 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41572.log 2024-09-16T03:41:59.761 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37644.log.gz 2024-09-16T03:41:59.761 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57470.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41112.log 2024-09-16T03:41:59.761 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57470.log.gz 2024-09-16T03:41:59.761 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50825.log 2024-09-16T03:41:59.761 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41572.log.gz 2024-09-16T03:41:59.761 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63594.log 2024-09-16T03:41:59.761 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41112.log.gz 2024-09-16T03:41:59.761 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50825.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42285.log 2024-09-16T03:41:59.761 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50825.log.gz 2024-09-16T03:41:59.762 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63552.log 2024-09-16T03:41:59.762 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63594.log.gz 2024-09-16T03:41:59.762 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53305.log 2024-09-16T03:41:59.762 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42285.log.gz 2024-09-16T03:41:59.762 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63804.log 2024-09-16T03:41:59.762 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63552.log.gz 2024-09-16T03:41:59.762 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50950.log 2024-09-16T03:41:59.762 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53305.log.gz 2024-09-16T03:41:59.762 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53151.log 2024-09-16T03:41:59.762 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63804.log.gz 2024-09-16T03:41:59.762 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56562.log 2024-09-16T03:41:59.762 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50950.log.gz 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr:gzip/var/log/ceph/ceph-client.admin.53151.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.58549.log 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53151.log.gz 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56562.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56562.log.gz 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42242.log 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48860.log 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58549.log.gz 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42242.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42242.log.gz -5 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5/var/log/ceph/ceph-client.admin.48860.log: --verbose -- /var/log/ceph/ceph-client.admin.62586.log 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48860.log.gz 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42755.log 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62586.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62586.log.gz 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr: /var/log/ceph/ceph-client.admin.41405.log 2024-09-16T03:41:59.763 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42755.log.gz 2024-09-16T03:41:59.764 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58067.log 2024-09-16T03:41:59.764 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.41405.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41405.log.gz 2024-09-16T03:41:59.764 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48511.log 2024-09-16T03:41:59.764 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58067.log.gz 2024-09-16T03:41:59.764 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62376.log 2024-09-16T03:41:59.764 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48511.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48511.log.gz 2024-09-16T03:41:59.764 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64476.log 2024-09-16T03:41:59.764 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62376.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62376.log.gz 2024-09-16T03:41:59.764 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50041.log 2024-09-16T03:41:59.764 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64476.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64476.log.gz 2024-09-16T03:41:59.764 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54767.log 2024-09-16T03:41:59.764 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50041.log.gz 2024-09-16T03:41:59.764 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51746.log 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54767.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54767.log.gz 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54444.log 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51746.log.gz 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44364.log 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54444.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54444.log.gz 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.63384.log 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44364.log.gz 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47049.log 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63384.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63384.log.gz 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph.log 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47049.log.gz 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63930.log 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58465.log 2024-09-16T03:41:59.765 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63930.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.44740.log 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63930.log.gz 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42013.log 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58465.log: /var/log/ceph/ceph-client.admin.44740.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58465.log.gz 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44740.log.gz 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57597.log 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42054.log 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42013.log.gz 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57597.log.gz 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45930.log 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37643.log 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42054.log.gz 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45930.log.gzgzip 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62460.log 2024-09-16T03:41:59.766 INFO:teuthology.orchestra.run.smithi064.stderr: 86.3% -- replaced with /var/log/ceph/ceph.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33200.log 2024-09-16T03:41:59.767 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.767 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37643.log: /var/log/ceph/ceph-client.admin.62460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37643.log.gz 2024-09-16T03:41:59.767 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38235.log 2024-09-16T03:41:59.767 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62460.log.gz 2024-09-16T03:41:59.767 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.33200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38730.log 2024-09-16T03:41:59.767 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33200.log.gz 2024-09-16T03:41:59.767 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.38235.log: -- /var/log/ceph/ceph-client.admin.61125.log 2024-09-16T03:41:59.767 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38235.log.gz 2024-09-16T03:41:59.767 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38730.log.gz 2024-09-16T03:41:59.767 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48180.log 2024-09-16T03:41:59.767 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61125.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61125.log.gz 2024-09-16T03:41:59.767 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38281.log 2024-09-16T03:41:59.767 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48180.log.gz 2024-09-16T03:41:59.768 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43695.log 2024-09-16T03:41:59.768 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38281.log.gz 2024-09-16T03:41:59.768 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63006.log 2024-09-16T03:41:59.768 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43695.log.gz 2024-09-16T03:41:59.768 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37302.log 2024-09-16T03:41:59.768 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63006.log.gz 2024-09-16T03:41:59.768 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51807.log 2024-09-16T03:41:59.768 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37302.log.gz 2024-09-16T03:41:59.768 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38211.log 2024-09-16T03:41:59.768 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51807.log.gz 2024-09-16T03:41:59.768 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61514.log 2024-09-16T03:41:59.768 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38211.log.gz 2024-09-16T03:41:59.768 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61008.log 2024-09-16T03:41:59.769 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61514.log.gz 2024-09-16T03:41:59.769 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59520.log 2024-09-16T03:41:59.769 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61008.log.gz 2024-09-16T03:41:59.769 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42168.log 2024-09-16T03:41:59.769 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59520.log.gz 2024-09-16T03:41:59.769 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47521.log 2024-09-16T03:41:59.769 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42168.log.gz 2024-09-16T03:41:59.769 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38279.log 2024-09-16T03:41:59.769 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47521.log.gz 2024-09-16T03:41:59.769 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56247.log 2024-09-16T03:41:59.769 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38279.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38279.log.gz 2024-09-16T03:41:59.769 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63846.log 2024-09-16T03:41:59.770 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56247.log.gz 2024-09-16T03:41:59.770 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49259.log 2024-09-16T03:41:59.770 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63846.log.gz 2024-09-16T03:41:59.770 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62124.log 2024-09-16T03:41:59.770 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49259.log.gz 2024-09-16T03:41:59.770 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61126.log 2024-09-16T03:41:59.770 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.62124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62124.log.gz 2024-09-16T03:41:59.770 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39614.log 2024-09-16T03:41:59.770 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61126.log.gz 2024-09-16T03:41:59.770 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38293.log 2024-09-16T03:41:59.770 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39614.log.gz 2024-09-16T03:41:59.770 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50239.log 2024-09-16T03:41:59.771 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38293.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38303.log 2024-09-16T03:41:59.771 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38293.log.gz 2024-09-16T03:41:59.771 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56884.log 2024-09-16T03:41:59.771 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50239.log.gz 2024-09-16T03:41:59.771 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.38303.log: -- /var/log/ceph/ceph-client.admin.61110.log 2024-09-16T03:41:59.771 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38303.log.gz 2024-09-16T03:41:59.771 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.56884.log: -- /var/log/ceph/ceph-client.admin.60408.log 2024-09-16T03:41:59.771 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56884.log.gz/var/log/ceph/ceph-client.admin.61110.log: 2024-09-16T03:41:59.771 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.61110.log.gz -5 2024-09-16T03:41:59.771 INFO:teuthology.orchestra.run.smithi064.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48179.log 2024-09-16T03:41:59.771 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52311.log 2024-09-16T03:41:59.771 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60408.log: /var/log/ceph/ceph-client.admin.48179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42447.log 2024-09-16T03:41:59.771 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48179.log.gz 2024-09-16T03:41:59.772 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60408.log.gz 2024-09-16T03:41:59.772 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52311.log.gz 2024-09-16T03:41:59.772 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47840.log 2024-09-16T03:41:59.772 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42447.log.gz 2024-09-16T03:41:59.772 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47354.log 2024-09-16T03:41:59.772 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47840.log.gz 2024-09-16T03:41:59.772 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42599.log 2024-09-16T03:41:59.772 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47354.log.gz 2024-09-16T03:41:59.772 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47055.log 2024-09-16T03:41:59.772 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42599.log.gz 2024-09-16T03:41:59.772 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56622.log 2024-09-16T03:41:59.772 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56710.log 2024-09-16T03:41:59.772 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63510.log 2024-09-16T03:41:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47055.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56622.log.gz 2024-09-16T03:41:59.773 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56710.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-16T03:41:59.773 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56710.log.gz 2024-09-16T03:41:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63048.log 2024-09-16T03:41:59.773 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63510.log.gz 2024-09-16T03:41:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57397.log 2024-09-16T03:41:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63048.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56039.log 2024-09-16T03:41:59.773 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63048.log.gz 2024-09-16T03:41:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46517.log 2024-09-16T03:41:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57397.log: /var/log/ceph/ceph-client.admin.56039.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57397.log.gz 2024-09-16T03:41:59.773 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56039.log.gz 2024-09-16T03:41:59.773 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48901.log 2024-09-16T03:41:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.46517.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40462.log 2024-09-16T03:41:59.774 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46517.log.gz 2024-09-16T03:41:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45286.log 2024-09-16T03:41:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48901.log.gz 2024-09-16T03:41:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40462.log.gz 2024-09-16T03:41:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55857.log 2024-09-16T03:41:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45286.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42817.log 2024-09-16T03:41:59.774 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45286.log.gz 2024-09-16T03:41:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59394.log 2024-09-16T03:41:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55857.log: /var/log/ceph/ceph-client.admin.42817.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55857.log.gz 2024-09-16T03:41:59.774 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42817.log.gz 2024-09-16T03:41:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38221.log 2024-09-16T03:41:59.774 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59394.log.gz 2024-09-16T03:41:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52624.log 2024-09-16T03:41:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63720.log 2024-09-16T03:41:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52624.log: /var/log/ceph/ceph-client.admin.38221.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38221.log.gz 2024-09-16T03:41:59.775 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52624.log.gz 2024-09-16T03:41:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58806.log 2024-09-16T03:41:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.63720.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46045.log 2024-09-16T03:41:59.775 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63720.log.gz 2024-09-16T03:41:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58675.log 2024-09-16T03:41:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58806.log: /var/log/ceph/ceph-client.admin.46045.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58806.log.gz 2024-09-16T03:41:59.775 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46045.log.gz 2024-09-16T03:41:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61034.log 2024-09-16T03:41:59.775 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58675.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55394.log 2024-09-16T03:41:59.775 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43872.log 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55394.log: /var/log/ceph/ceph-client.admin.61034.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61034.log.gz 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55394.log.gz 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40798.log 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43872.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47595.log 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44624.log 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40798.log.gz/var/log/ceph/ceph-client.admin.47595.log: 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47595.log.gz 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37633.log 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.44624.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62418.log 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44624.log.gz 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52666.log 2024-09-16T03:41:59.776 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37633.log: /var/log/ceph/ceph-client.admin.62418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37633.log.gz 2024-09-16T03:41:59.777 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62418.log.gz 2024-09-16T03:41:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60418.log 2024-09-16T03:41:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.52666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52666.log.gz 2024-09-16T03:41:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37636.log 2024-09-16T03:41:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56759.log 2024-09-16T03:41:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37636.log: /var/log/ceph/ceph-client.admin.60418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37636.log.gz 2024-09-16T03:41:59.777 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60418.log.gz 2024-09-16T03:41:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62796.log 2024-09-16T03:41:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56759.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39731.log 2024-09-16T03:41:59.777 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56759.log.gz 2024-09-16T03:41:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58722.log 2024-09-16T03:41:59.777 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39731.log: /var/log/ceph/ceph-client.admin.62796.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39731.log.gz 2024-09-16T03:41:59.777 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62796.log.gz 2024-09-16T03:41:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37583.log 2024-09-16T03:41:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.58722.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42970.log 2024-09-16T03:41:59.778 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58722.log.gz 2024-09-16T03:41:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64434.log 2024-09-16T03:41:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37583.log.gz 2024-09-16T03:41:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.42970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42970.log.gz 2024-09-16T03:41:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37616.log 2024-09-16T03:41:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64434.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62880.log 2024-09-16T03:41:59.778 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64434.log.gz 2024-09-16T03:41:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54913.log 2024-09-16T03:41:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37616.log: /var/log/ceph/ceph-client.admin.62880.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37616.log.gz 2024-09-16T03:41:59.778 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62880.log.gz 2024-09-16T03:41:59.778 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53859.log 2024-09-16T03:41:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54913.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64350.log 2024-09-16T03:41:59.779 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54913.log.gz 2024-09-16T03:41:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54486.log 2024-09-16T03:41:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53859.log: /var/log/ceph/ceph-client.admin.64350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64350.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53859.log.gz 2024-09-16T03:41:59.779 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50553.log 2024-09-16T03:41:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54486.log.gz 2024-09-16T03:41:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38250.log 2024-09-16T03:41:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57042.log 2024-09-16T03:41:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.38250.log: /var/log/ceph/ceph-client.admin.50553.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38250.log.gz 2024-09-16T03:41:59.779 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50553.log.gz 2024-09-16T03:41:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50710.log 2024-09-16T03:41:59.779 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57042.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45420.log 2024-09-16T03:41:59.780 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57042.log.gz 2024-09-16T03:41:59.780 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43360.log 2024-09-16T03:41:59.780 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50710.log.gz 2024-09-16T03:41:59.780 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45420.log.gz 2024-09-16T03:41:59.780 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48348.log 2024-09-16T03:41:59.780 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43360.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43806.log 2024-09-16T03:41:59.780 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43360.log.gz 2024-09-16T03:41:59.780 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.48348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45493.log 2024-09-16T03:41:59.780 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48348.log.gz 2024-09-16T03:41:59.780 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.43806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64140.log 2024-09-16T03:41:59.780 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.45493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43806.log.gz 2024-09-16T03:41:59.780 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45493.log.gz 2024-09-16T03:41:59.780 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59646.log 2024-09-16T03:41:59.781 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59730.log 2024-09-16T03:41:59.781 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.64140.log: /var/log/ceph/ceph-client.admin.59646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64140.log.gz 2024-09-16T03:41:59.781 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59646.log.gz 2024-09-16T03:41:59.781 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53263.log 2024-09-16T03:41:59.781 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41258.log 2024-09-16T03:41:59.781 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59730.log.gz 2024-09-16T03:41:59.781 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53263.log.gz 2024-09-16T03:41:59.781 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59940.log 2024-09-16T03:41:59.781 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53064.log 2024-09-16T03:41:59.781 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59940.log: /var/log/ceph/ceph-client.admin.41258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59940.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41258.log.gz 2024-09-16T03:41:59.781 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.781 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40441.log 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53562.log 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.53064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53064.log.gz 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.40441.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40441.log.gz 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37107.log 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54994.log 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37107.log: /var/log/ceph/ceph-client.admin.53562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37107.log.gz 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53562.log.gz 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53542.log 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54994.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39782.log 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39930.log 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39782.log: /var/log/ceph/ceph-client.admin.53542.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39782.log.gz 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53542.log.gz 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60377.log 2024-09-16T03:41:59.782 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39930.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55174.log 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39930.log.gz 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57511.log 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.55174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60377.log.gz 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55174.log.gz 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60444.log 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.57511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57511.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41809.log 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56998.log 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.60444.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60444.log.gz/var/log/ceph/ceph-client.admin.41809.log: 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41809.log.gz 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61494.log 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59898.log 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56998.log.gz 2024-09-16T03:41:59.783 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.61494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61494.log.gz 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46559.log 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51629.log 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.59898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59898.log.gz/var/log/ceph/ceph-client.admin.46559.log: 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46559.log.gz 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56173.log 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54599.log 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.51629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51629.log.gz 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.56173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56173.log.gz 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47478.log 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40913.log 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.54599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54599.log.gz 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.47478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47478.log.gz 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49908.log 2024-09-16T03:41:59.784 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37617.log 2024-09-16T03:41:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.49908.log: /var/log/ceph/ceph-client.admin.40913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49908.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40913.log.gz 2024-09-16T03:41:59.785 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:41:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39888.log 2024-09-16T03:41:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50103.log 2024-09-16T03:41:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.37617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37617.log.gz 2024-09-16T03:41:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.39888.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39888.log.gz 2024-09-16T03:41:59.785 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.50103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50103.log.gz 2024-09-16T03:41:59.786 INFO:teuthology.orchestra.run.smithi064.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-16T03:41:59.786 INFO:teuthology.orchestra.run.smithi112.stderr: 89.9% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-09-16T03:41:59.860 INFO:teuthology.orchestra.run.smithi112.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T03:41:59.924 INFO:teuthology.orchestra.run.smithi064.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T03:41:59.955 INFO:teuthology.orchestra.run.smithi155.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T03:42:00.106 INFO:teuthology.orchestra.run.smithi155.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-09-16T03:42:00.148 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T03:42:00.325 INFO:teuthology.orchestra.run.smithi112.stderr: 91.1% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-16T03:42:00.373 INFO:teuthology.orchestra.run.smithi157.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-16T03:42:00.460 INFO:teuthology.orchestra.run.smithi112.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T03:42:00.462 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-16T03:42:00.462 INFO:teuthology.orchestra.run.smithi112.stderr:real 0m0.958s 2024-09-16T03:42:00.462 INFO:teuthology.orchestra.run.smithi112.stderr:user 0m2.903s 2024-09-16T03:42:00.462 INFO:teuthology.orchestra.run.smithi112.stderr:sys 0m0.104s 2024-09-16T03:42:00.593 INFO:teuthology.orchestra.run.smithi155.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T03:42:00.727 INFO:teuthology.orchestra.run.smithi064.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-09-16T03:42:00.754 INFO:teuthology.orchestra.run.smithi157.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-16T03:42:00.795 INFO:teuthology.orchestra.run.smithi064.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T03:42:00.932 INFO:teuthology.orchestra.run.smithi157.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-09-16T03:42:01.004 INFO:teuthology.orchestra.run.smithi157.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T03:42:01.006 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-09-16T03:42:01.006 INFO:teuthology.orchestra.run.smithi157.stderr:real 0m1.496s 2024-09-16T03:42:01.006 INFO:teuthology.orchestra.run.smithi157.stderr:user 0m4.818s 2024-09-16T03:42:01.006 INFO:teuthology.orchestra.run.smithi157.stderr:sys 0m0.183s 2024-09-16T03:42:01.013 INFO:teuthology.orchestra.run.smithi155.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-16T03:42:01.119 INFO:teuthology.orchestra.run.smithi064.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T03:42:01.142 INFO:teuthology.orchestra.run.smithi155.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-16T03:42:01.144 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-09-16T03:42:01.144 INFO:teuthology.orchestra.run.smithi155.stderr:real 0m1.639s 2024-09-16T03:42:01.144 INFO:teuthology.orchestra.run.smithi155.stderr:user 0m5.079s 2024-09-16T03:42:01.144 INFO:teuthology.orchestra.run.smithi155.stderr:sys 0m0.174s 2024-09-16T03:42:01.285 INFO:teuthology.orchestra.run.smithi064.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-16T03:42:01.287 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-16T03:42:01.287 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m1.783s 2024-09-16T03:42:01.287 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m6.033s 2024-09-16T03:42:01.287 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.449s 2024-09-16T03:42:01.287 INFO:tasks.ceph:Archiving logs... 2024-09-16T03:42:01.288 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/remote/smithi064/log 2024-09-16T03:42:01.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T03:42:03.957 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/remote/smithi112/log 2024-09-16T03:42:03.958 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T03:42:04.452 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/remote/smithi155/log 2024-09-16T03:42:04.453 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T03:42:05.269 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/remote/smithi157/log 2024-09-16T03:42:05.270 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T03:42:06.122 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T03:42:06.132 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T03:42:06.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T03:42:06.136 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T03:42:06.138 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T03:42:06.141 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T03:42:06.273 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T03:42:06.282 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T03:42:06.283 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:42:06.285 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:42:06.288 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:42:06.291 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:42:06.301 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:42:06.303 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:42:06.305 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:42:06.305 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-09-16T03:42:06.306 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv01.front.sepia.ceph.com 3 6 377 30 -608us[ -599us] +/- 47ms 2024-09-16T03:42:06.306 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 +947us[ +947us] +/- 66ms 2024-09-16T03:42:06.306 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -58us[ -58us] +/- 88ms 2024-09-16T03:42:06.306 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:42:06.306 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:42:06.308 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:42:06.308 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-09-16T03:42:06.308 INFO:teuthology.orchestra.run.smithi112.stdout:^* hv01.front.sepia.ceph.com 3 6 377 30 -593us[ -573us] +/- 47ms 2024-09-16T03:42:06.308 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 +970us[ +970us] +/- 66ms 2024-09-16T03:42:06.308 INFO:teuthology.orchestra.run.smithi112.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 -34us[ -34us] +/- 88ms 2024-09-16T03:42:06.308 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:42:06.309 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-09-16T03:42:06.310 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:42:06.310 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-09-16T03:42:06.310 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv01.front.sepia.ceph.com 3 6 377 30 -588us[ -554us] +/- 47ms 2024-09-16T03:42:06.310 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 28 +975us[ +975us] +/- 66ms 2024-09-16T03:42:06.310 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 -21us[ -21us] +/- 88ms 2024-09-16T03:42:06.310 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:42:06.314 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T03:42:06.314 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-09-16T03:42:06.314 INFO:teuthology.orchestra.run.smithi157.stdout:^* hv01.front.sepia.ceph.com 3 6 377 30 -618us[ -569us] +/- 47ms 2024-09-16T03:42:06.314 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 25 +951us[ +951us] +/- 66ms 2024-09-16T03:42:06.314 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -65us[ -65us] +/- 88ms 2024-09-16T03:42:06.314 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T03:42:06.315 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T03:42:06.339 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T03:42:06.339 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T03:42:06.459 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T03:42:06.548 DEBUG:teuthology.orchestra.run.smithi112:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T03:42:06.622 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T03:42:06.696 DEBUG:teuthology.orchestra.run.smithi157:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T03:42:06.771 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T03:42:06.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:06.806 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T03:42:06.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:06.840 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T03:42:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:06.870 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T03:42:06.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:06.901 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T03:42:06.912 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T03:42:06.972 INFO:teuthology.task.internal:Duration was 785.922686 seconds 2024-09-16T03:42:06.973 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T03:42:07.017 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T03:42:07.017 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T03:42:07.020 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T03:42:07.023 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T03:42:07.025 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T03:42:07.065 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:42:07.065 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:42:07.066 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:42:07.066 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T03:42:07.536 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T03:42:07.537 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-09-16T03:42:07.537 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T03:42:07.562 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2024-09-16T03:42:07.562 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T03:42:07.594 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-09-16T03:42:07.594 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T03:42:07.620 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2024-09-16T03:42:07.620 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T03:42:07.651 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T03:42:07.652 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T03:42:07.655 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T03:42:07.657 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T03:42:07.663 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T03:42:07.718 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T03:42:07.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T03:42:07.729 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T03:42:07.731 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T03:42:07.736 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T03:42:07.871 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T03:42:07.887 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T03:42:07.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T03:42:07.914 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T03:42:07.939 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T03:42:07.965 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T03:42:07.990 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T03:42:08.001 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:08.003 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:08.006 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:08.008 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:08.025 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-09-16T03:42:08.028 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2024-09-16T03:42:08.029 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-09-16T03:42:08.053 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2024-09-16T03:42:08.081 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:08.132 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:08.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:08.147 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:08.162 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T03:42:08.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:08.177 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T03:42:08.187 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T03:42:08.188 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/remote/smithi064 2024-09-16T03:42:08.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T03:42:08.224 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/remote/smithi112 2024-09-16T03:42:08.224 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T03:42:08.258 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/remote/smithi155 2024-09-16T03:42:08.258 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T03:42:08.291 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905882/remote/smithi157 2024-09-16T03:42:08.292 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T03:42:08.325 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T03:42:08.325 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T03:42:08.328 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T03:42:08.330 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T03:42:08.333 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T03:42:08.376 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T03:42:08.388 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T03:42:08.388 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T03:42:08.421 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T03:42:08.421 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T03:42:08.423 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T03:42:08.426 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T03:42:08.428 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T03:42:08.437 INFO:teuthology.orchestra.run.smithi064.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 03:42 /home/ubuntu/cephtest 2024-09-16T03:42:08.441 INFO:teuthology.orchestra.run.smithi112.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 03:42 /home/ubuntu/cephtest 2024-09-16T03:42:08.442 INFO:teuthology.orchestra.run.smithi155.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 03:42 /home/ubuntu/cephtest 2024-09-16T03:42:08.444 INFO:teuthology.orchestra.run.smithi157.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 03:42 /home/ubuntu/cephtest 2024-09-16T03:42:08.445 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T03:42:08.456 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T03:42:08.608 INFO:teuthology.run:Summary data: description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/careful_host thrashosds-health workloads/dedup-io-mixed} duration: 785.922685623169 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T03:42:08.608 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T03:42:08.687 INFO:teuthology.run:pass